E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

## Abstract

Abstract In the realm of Large Language Models (LLMs), the ability to process long contexts is increasingly crucial for tasks such as multi-round dialogues, code generation, and document summarization. This paper addresses the challenges of enhancing the long-context performance, reducing computational complexity, and leveraging pretrained models—collectively termed the "impossible triangle." We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. The method involves splitting long contexts into chunks, compressing each into embedding vectors via a pretrained text encoder, and utilizing an adapter to align these representations with a decoder-only LLM. Two training objectives, focusing on reconstruction of the encoder output and long-context instruction fine-tuning, are employed to facilitate the understanding of soft prompts by the LLM. Experimental results demonstrate that E2LLM achieves superior performance in long-context scenarios while balancing efficiency, performance, and compatibility with pretrained models. Our framework thus represents a significant advancement in the field, contributing to effective long-text modeling. Code will be available upon publication.

## 1 Introduction

Understanding and reasoning about long context has become essential for Large Language Models (LLMs), especially for tasks like multi-round dialogues Bai et al. (), (multi)-repository code generation Zhang et al. (), and (multi)-document summarization  and question answering Singh et al. (). These tasks often require processing thousands or even millions of tokens to ensure coherence and accuracy. On the other hand, to boost the performance of LLMs, techniques that effectively prompt LLMs to activate the domain-specific knowledge—such as chain-of-thought reasoning Wei et al. (), in-context learning Dong et al. (), and retrieving relevant documents or historical conversations Ding et al. ()—are also pushing the demand for longer sequence lengths.

Considerable effort has been and is still being put into developing models that can increase the context length of LLMs, aiming at achieving strong performance for longer contexts (T1), while reducing the training and inference complexity (T2), and at the same time being compatible with pretrained models (T3) such that the pretrained knowledge in these models can be effectively exploited. However, achieving all three targets simultaneously presents a formidable challenge that often leads to some compromises, a phenomenon we refer to as the “impossible triangle”, as illustrated in Figure 2.

Figure: Figure 1: E2LLM solves all the challenges of the impossible triangle at the same time, namely Performance, Efficiency and Compatibility.
Refer to caption: x1.png

Currently, research in this field has primarily focused on three main avenues: modifying position embeddings, attention mechanisms, and the long input sequence itself. The first group of methods, known as length extension, involves adjusting the position embeddings of LLMs to accommodate longer context extensions. This typically involves selecting a large base value for RoPE Su et al. () and then continuing pretraining or fine-tuning on the target length. While these methods effectively extend the length of LLMs with minimal model changes (T1&T3), they typically incur substantial computational costs during both training and inference (T2). For instance, even with the ability to extend sequence lengths to 2M, as seen in LongRoPE , enormous resources are required to train and deploy the model, and inference times can be prohibitively long for extended sequences. As opposed to the first group, the second one, dubbed sparse attention, replaces full attention in LLMs with local attention or a combination of global and local attention. This approach significantly reduces the quadratic complexity associated with full attention, even achieving linear complexity in theory (T2). However, a notable concern with sparse attention is its potential to neglect informative history, as certain tokens may not be attended to during the attention calculations (T1). Moreover, since LLMs are not originally pretrained with sparse attention, adapting them to sparse attention may require extensive training or fine-tuning (T3). Different from the previous two groups that change the LLMs, the third group of strategies directly compresses the input sequence to reduce its length (T2), which can be further divided into two subcategories. The first subgroup, known as hard prompt compression—exemplified by methods such as Retrieval-Augmented Generation (RAG) Ding et al. () and LLMLingua Jiang et al. ()—tends to process compression and inference in a two-step manner. As a result, any loss of information or introduction of irrelevant content during the compression stage may adversely affect performance in the subsequent inference step (T1). Alternatively, the second subgroup considers soft prompt compression, which summarizes long contexts into embedding vectors. However, utilizing LLMs in these approaches to directly generate sentence-level embeddings diverges from their original pretraining objective of next token prediction. Consequently, achieving satisfactory performance in this context often demands rigorous training or fine-tuning to align the model’s capabilities with the new objective (T3).

In this paper, we propose a novel compression based method named E2LLM (Encoder Elongated Large Language Models) that adeptly navigates the complexities of the “impossible triangle”. Specifically, as shown in Figure 2, our method first splits a long context into chunks and compresses each chunk into an embedding vector using a pre-trained text encoder (e.g., BERT Kenton and Toutanova ()). Then, an adapter aligns the encoder’s output with the input embedding space of a decoder-only LLM, such that the LLM can understand the embedding vectors resulting from the encoder. Finally, we set up two training objectives to align the encoder and decoder, including reconstructing the input text encoded by the encoder (“understanding”) and long-context instruction fine-tuning (“reasoning”). We postulate that LLMs are inherently rich in knowledge; thus, properly compressed soft prompts (or the embedding vectors) can succinctly convey adequate information for LLMs to generate accurate answers. Moreover, since pre-trained encoder models are inherently crafted to produce sentence embeddings, this design allows E2LLM to capitalize on both pre-trained encoders and decoders, minimizing the requirement for extensive additional training (T3). Additionally, compressing each original chunk into a vector (i.e., a single chunk token) not only enhances training and inference efficiency (T2) but also scales up the context length significantly (T1). Indeed, the theoretical sequence length equals the product of the sequence lengths of the encoder and the decoder. The experimental results provide compelling evidence of E2LLM’s superior performance in long-context scenarios, demonstrating our method’s efficacy in maintaining a delicate balance between performance, efficiency, and compatibility.

To summarize, the main contributions of our work are:

- •
We propose E2LLM, a novel LLM long-text modeling framework built upon pretrained sentence embedding models and decoder-only LLMs, effectively addressing the challenges posed by the “impossible triangle”.
- •
We introduce two training objectives, including reconstructing the soft prompt given by the encoder and the long-context instruction fine-tuning, enabling the LLM to understand the soft prompt while generating accurate outputs for long inputs.
- •
Comprehensive experiments conducted on diverse tasks and datasets demonstrate the efficiency and practicality of our proposed model and reveal its competitive performance with the state-of-the-art.

## 2 Related Works

As mentioned in the introduction, prevalent methods can be categorized into three groups: modifying the position embedding (i.e., length extension), the attention mechanism (i.e., sparse attention), and the input sequence (i.e., prompt compression).

#### Length Extension

Training LLMs on sequences with limited maximum sequence lengths while ensuring generalization for longer sequences is challenging. To address this, positional extrapolation and interpolation methods have been proposed. Positional extrapolation extends positional encoding beyond the training length; for instance, ALiBi  enhances attention with linear biases that adjust scores based on the distance between key and query positions. Instead, xPOS Sun et al. () utilizes relative position embeddings for better attention resolution and extended lengths. However, these methods have not been integrated into recent LLMs such as Llama2 Touvron et al. () and Qwen2 Bai et al. (), primarily due to the suboptimal performance of positional extrapolation. Positional interpolation, on the other hand, scales down input position indices and expands context windows to maintain performance across longer sequences. For example, Chen et al. () applies linear interpolation to RoPE to align maximum position indices with pre-training constraints. NTK interpolation bloc97. () modifies the base of RoPE to adjust the rotational velocity of its dimensions. To combine the strengths of these approaches, YaRN Peng et al. () merges linear and NTK interpolation with a ramp function and temperature factor, mitigating distribution shifts in the attention matrix with longer inputs. LongRoPE  further enhances performance by exploiting two forms of non-uniformities in RoPE positional embedding via an efficient evolutionary search.

Despite these advancements, most approaches require continual pre-training or fine-tuning to achieve the desired length, thus entailing a considerable training burden. Additionally, inference on these extended models can be slow due to the quadratic complexity of full attention. In contrast, the proposed E2LLM does not alter the original LLM’s length but compresses the input sequence into chunks of embedding vectors. This allows E2LLM to maintain the efficiency of the original LLM during both training and inference.

#### Sparse Attention

This category of methods aims to decrease the inference complexity of large language models (LLMs) by manipulating attention mechanisms with novel attention masks, enabling these models to handle longer sequences. StreamingLLM Xiao et al. () demonstrates that focusing on the beginning of the sequence and the most recent tokens within a defined window (i.e., local attention) during inference maintains performance while significantly reducing computational costs to a linear scale. However, these training-free methods often fall short in various scenarios Anagnostidis et al. (); Lou et al. (), as they may neglect informative tokens situated in the middle of the sequence. To improve performance, LM-Infinite Han et al. () reintroduces top-k tokens from the middle, but this approach necessitates the computation of all attention scores, thereby increasing computational demands. As a solution, Lou et al. () propose SparseK attention, which employs an additional scoring network to assess the importance of each key-value pair and select the top-k pairs. Alternatively, LongLoRA Chen et al. () utilizes shifted sparse attention (a variant of local attention) and fine-tunes LLMs with LoRA Hu et al. () to adapt to this mechanism. Unfortunately, as noted by Tan et al. (), there remains a significant gap between sparse and full attention, which complicates the fine-tuning of pre-trained LLMs to new attention paradigms. In contrast, the E2LLM approach summarizes long-context input into soft prompt vectors, thereby reducing sequence length without altering the full attention mechanism in LLMs.

#### Prompt Compression

Prompt compression enhances the efficiency of LLM input processing by either condensing lengthy prompts (hard prompt compression) or learning compact prompt representations (soft prompt compression). Hard prompt compression techniques include RAG Ding et al. (), LLMlingua Jiang et al. (), Selective-Context Li (), and LongLLMlingua Jiang et al. (). RAG optimizes input by retrieving only query-relevant passages, while LLMlingua and Selective-Context compress lengthy context without referring to the query. LongLLMlingua bridges these approaches by employing question-aware coarse-to-fine compression, document reordering, dynamic ratios, and sub-sequence recovery to improve performance. However, these methods separate compression and inference into distinct steps, leading to potential error propagation that degrades performance. In contrast, E2LLM is trained end-to-end, effectively mitigating the above issue.

Soft prompt compression, proposed by Mu et al. () and Ge et al. (), involves training LLMs to distill prompts into a more concise set of tokens that encapsulate the original prompt’s knowledge for future use. Chevalier et al. () extend this by developing AutoCompressor, which converts longer textual contexts into summary vectors that serve as soft prompts, which expands the LLM’s context window and reduces computational costs, as examplified in LLoCO Tan et al. (). However, directly using LLMs to generate sentence-level embeddings diverges from their original objective of next-token prediction. As a result, achieving satisfactory performance in this context often requires extensive training or fine-tuning to align the model with the new objective. To overcome this problem, our E2LLM leverages a pretrained sentence embedding model to represent prompts, aligning with the original training objectives of embedding models.

## 3 Our Approach: E2LLM

In this section, we detail the proposed E2LLM framework for understanding and reasoning over long contexts, which effectively combines the strengths of pretrained text encoders and LLM decoders.

### 3.1 Model Architecture

Figure 1 illustrates the architecture of the E2LLM framework, which comprises four key components: a Chunker, a Text Encoder ${\mathcal{E}}_{\theta}$, an Adapter ${\mathcal{A}}_{\phi}$, and an LLM Decoder ${\mathcal{D}}_{\eta}$. Here, $\theta$, $\phi$, and $\eta$ denote the (learnable) parameters specific to each component. It is essential to note that the choice of models for the encoder and decoder, the method of chunking, and the network architecture of the adapter can be customized to meet the needs of different domains. E2LLM serves as a flexible framework, seamlessly integrating these components to effectively manage long contexts while being capable of leveraging the power of more advanced components when available. We will now introduce each component in detail, following the data flow during inference in E2LLM.

#### Chunker

The Chunker is responsible for dividing long contexts into smaller, manageable chunks while ensuring that the token length of each chunk does not exceed the maximum sequence length of the text encoder. Similar to RAG, the choice of chunking strategy can impact the overall performance of E2LLM. Here, we adopt a straightforward yet effective approach: we first define a chunk size, extract the initial chunk, and then backtrack within this chunk to locate breakpoints, such as periods or line breaks. Following this, we begin a new chunk at the end of the previous one and apply the backtracking method again. We repeat this process until all text is chunked. This method helps to maintain the semantic integrity of the original texts. Note that other methods such as introducing overlap between chunks can also benefit E2LLM. Additionally, the size of the chunks is crucial for E2LLM’s performance. Our experiments indicate that including excessive context in a single chunk can degrade performance, primarily because a high compression ratio may render the embedding vector too generic, compromising specificity.

#### Text Encoder ℰ ℰ {\mathcal{E}} caligraphic_E

After chunking, we input each chunk into the text encoder to generate the corresponding embedding vector. Notably, most pretrained encoders, such as GTE Li et al. () and BGE Xiao et al. (), are trained via contrastive learning. This means the [CLS] token, which serves as the embedding vector, typically captures only the discriminative information necessary for differentiating between chunks, while information essential for the LLM decoder to answer the query may be discarded. To mitigate this limitation, we adopt low-rank adaptation (LoRA) Hu et al. () to make text encoder trainable during the alignment process. This allows the encoder to preserve the information that is beneficial for the LLM’s performance.

#### Adapter 𝒜 𝒜 {\mathcal{A}} caligraphic_A

To facilitate the LLM’s understanding of the chunk-wise semantics derived from the encoder’s output, we employ an Adapter to map the encoder’s output into the input embedding of the LLM. Since the hidden dimensions of the text encoder and the LLM decoder may differ, the Adapter is a vital component. Specifically, we utilize a two-layer Multi-Layer Perceptron (MLP) with the GELU activation function Hendrycks and Gimpel () as the adapter network. This Adapter is applied to each chunk embedding individually, and we refer to its output as the *chunk token* or *soft prompts*, which are then processed by the subsequent LLM. The Adapter is initialized randomly and trained from scratch during the alignment phase.

#### LLM Decoder 𝒟 𝒟 {\mathcal{D}} caligraphic_D

Finally, we concatenate the chunk tokens (the green tokens in Figure 2) and the text tokens corresponding to the prompt and query together, and ask the LLM to generate the answer for the query. In our experiments, we select Llama2 Touvron et al. () as the LLM Decoder due to its widespread usage in both academic research and industry applications. Additionally, we employ LoRA to further train the Decoder as part of the alignment process between the encoder and decoder.

### 3.2 Training Tasks

Now we focus on training the last two layers of the encoder, the adapter, and the LoRA branch of the decoder to enhance the E2LLM’s ability to comprehend lengthy input contexts and effectively reason about the corresponding answers. To accomplish this, we introduce two distinct training tasks.

The first task is designed to improve the LLM’s understanding of the input. As depicted in Figure 2, once the LLM receives chunk tokens from the adapter, we prompt it to restate or reconstruct the input. We refer to this as the “understanding” task. The specific prompt used is "Given the contexts: [chunk token]\n
Please follow the instruction:\nRestate the aforementioned context". Notably, this task is self-supervised, allowing us to curate a significant amount of training data to ensure that the LLM comprehensively grasps the embeddings provided by the adapter. However, in our experiments, we utilize only the input from long-context instruction fine-tuning data for this task. Given that these inputs are often too lengthy to be fully reconstructed at once, we employ a sliding window approach, reconstructing the original context in segments based on a few consecutive chunks until the entire input has been restated.

On the other hand, the second training task enables the LLM to generate answers based on the chunk tokens (i.e., the long context) and the user’s query. We refer to this as the “reasoning” task, and the prompt crafted for this purpose is "Given the contexts: [chunk token]\n
Please follow the instruction: \n
Answer the question: {query}".

#### Maximum Sequence Length

Theoretically, the maximum sequence length of E2LLM equals the product of the encoder and decoder’s sequence lengths. However, as previously mentioned, setting the chunk size to match the encoder’s sequence length presents challenges, as it may hinder the encoder’s ability to retain all pertinent information within a single chunk. Consequently, it is crucial to establish an appropriate chunk size. Thus, the effective sequence length of E2LLM is determined to be the chunk size multiplied by the sequence length of the LLM’s decoder. In actuality, we set the maximum chunk size of 512 characters in experiments, which is approximately equivalent to 100 tokens. Hence, the context length has been expanded by nearly 100 times.

#### Time and Space Complexity during Inference

Let us denote the original input length as $L$ and the chunk size in E2LLM as $C$. Therefore, the total number of chunks becomes $L/C$. For each chunk, the resulting time and space complexity from the text encoder is ${\mathcal{O}}(C^{2})$. Given that there are $L/C$ chunks, the overall complexity for the encoding step is ${\mathcal{O}}(CL)$. In practice, since all chunks can be processed in parallel, the time complexity can be further reduced by a constant factor. Subsequently, we pass the $L/C$ chunk tokens to the LLM decoder, which yields a complexity of ${\mathcal{O}}(L^{2}/C^{2})$. In summary, the total time and space complexity is ${\mathcal{O}}(LC+L^{2}/C^{2})$. To substantiate the efficiency of E2LLM during inference, we conduct empirical experiments that assess both inference time and memory usage. Further details can be found in Section 4.5.

### 3.3 Relation to Other Methods

#### Relation to VLMs

E2LLM draws inspiration from recent advancements in Vision-Language Models (VLMs) Zhang et al. (), including mini-GPT4 , LLaVa Liu et al. (), Qwen-VL Bai et al. (), and InternVL Chen et al. (). These VLMs utilize adapters to align pretrained vision encoders with LLM decoders, enabling the LLMs to process image tokens outputted by the vision encoders. In this framework, both the vision encoder and LLM decoder are pretrained independently, offering a flexible approach that allows for the alignment of high-performing vision and language models, thereby maximizing their capabilities. Notably, VLMs excel at performing OCR (Optical Character Recognition) Islam et al. () tasks, effectively recognizing and outputting text present within images. Motivated by the success of VLMs, we propose that by aligning text encoders (i.e., embedding models) with LLM decoders using an adapter, LLMs can similarly interpret sentences encoded by the text encoders and draw inferences based on this comprehension. Furthermore, as both the encoder and decoder in our approach operate within the same modality, we anticipate that the alignment process will be more straightforward than that required for models functioning across different modalities, potentially reducing the amount of data needed for alignment. Conversely, the reconstruction task employed in training E2LLM is self-supervised, enabling us to amass a vast dataset of text to enhance the LLM’s contextual understanding. In contrast, the alignment task in VLMs relies on supervised image-text pairs, which are notably more challenging to collect.

#### Relation to RAG

RAG (Retrieval-Augmented Generation) Ding et al. () typically operates by employing a text encoder-based retriever to identify pertinent passages from a knowledge base in response to user queries. These retrieved texts are then fed into an LLM (i.e., generator) to enhance the responses. RAG can augment E2LLM by retrieving the most relevant passages, while E2LLM can extend the context length of the generator in RAG. In addition, a significant challenge for RAG is the inconsistency that can arise between how the retriever and generator interpret the same text Li et al. (); Ding et al. (). E2LLM addresses this issue by aligning the retriever (text encoder) with the generator (LLM decoder), facilitating better coherence in interpretation. Moreover, E2LLM enables more efficient communication between the retriever and generator through embedding vectors instead of raw texts.

#### Relation to LLoCO

In comparison to E2LLM, LLoCO Tan et al. () utilizes AutoCompressor Chevalier et al. () as its long text encoder and omits the adapter since it employs the same LLM (i.e., Llama2) as AutoCompressor. As a result, it can effectively understand the summary vectors—similar to chunk tokens or soft prompts—generated by AutoCompressor after fine-tuning the LLM with LoRA. One advantage of LLoCO is that its text encoder, AutoCompressor, considers the interdependencies of long-context chunks. However, this also presents a limitation: the long context can only be processed sequentially, one chunk after another. By contrast, E2LLM can process all chunks in parallel and is more suitable for long context. Furthermore, the encoder is restricted to AutoCompressor, making it challenging to enhance LLoCO’s performance without updating AutoCompressor. Notably, AutoCompressor requires an extensive fine-tuning process on the original Llama2 using 2 billion tokens to enable Llama2 to generate summary tokens. In contrast, E2LLM is designed to easily incorporate stronger text encoders and LLM decoders as they become available throughout the year.

## 4 Experiments

In this section, we evaluate the performance of E2LLM on two key tasks, including document question answering (QA) and document summarization.
For our comparison, we benchmark E2LLM against four baselines, including Yarn Peng et al. (), LongLoRA Chen et al. (), RAG Gao et al. (), and LLoCO Tan et al. (), which respectively represent the state-of-the-art (SOTA) methods in length extension, sparse attention, hard prompt compression, and soft prompt compression. Note that except RAG which is free of training, we set the rank of LoRA for the LLM to be the same in all methods, resulting in 17M trainable parameters in YaRN and LLoCO and 140M for LongLoRA.

**Table 1: Dataset Statistics.**
| Dataset | Task Type | #Train. Samp. | #Eval. Samp. | Samp. Len. |
| --- | --- | --- | --- | --- |
| QMSum | Summarization | 1,257 | 272 | 14,428.78 |
| GovReport | Summarization | 10,000 | 500 | 11,204.00 |
| Quality | DocumentQA | 5,046 | 2,086 | 6,797.66 |
| NarrativeQA | DocumentQA | 3,000 | 200 | 52,158.88 |
| TriviaQA | DocumentQA | 10,000 | 500 | 1,075.90 |

### 4.1 Dataset Description

In order to evaluate the effectiveness of E2LLM, we leverage five publicly available datasets that encompass both Summarization and Document Question-Answering (DocumentQA) tasks. The data statistics are shown in Table 1.

- •
QMSum(^1^11https://github.com/Yale-LILY/QMSum) Zhong et al. () is a newly devised, human-annotated benchmark designed for the query-based multidomain meeting summarization task. It comprises an extensive range of query-summary pairs across 232 meetings in diverse fields. Specifically, we included 1,257 training samples and used 272 samples for inference. The average length of the samples in this dataset is 14,428.78 tokens.
- •
GovReport(^2^22[https://huggingface.co/datasets/ccdv/govreport-summarization](https://huggingface.co/datasets/ccdv/govreport-summarization)) Huang et al. () contains elongated reports by the U.S. Government Accountability Offices and the Congressional Research Service, complemented by summaries and abstracts hand-written by experts, which is of the summarization task genre. For training purposes, 10,000 random samples were utilized, and for inference, 500 samples were arbitrarily selected from the validation sets. The average length of the sampled data is 11,204.00 tokens.
- •
Quality(^3^33[https://huggingface.co/datasets/emozilla/quality](https://huggingface.co/datasets/emozilla/quality)) Bowman et al. () is a DocumentQA dataset comprising 5,046 training samples and 2,086 inference samples with contexts that have an average length of 6,797.66 tokens.
Further, we convert the original single-choice data format of the dataset into the QA format.
- •
NarrativeQA(^4^44[https://github.com/google-deepmind/narrativeqa](https://github.com/google-deepmind/narrativeqa)) Kovcisky et al. () is another DocumentQA dataset, primarily extracted from comprehensive book texts and film scripts from varied sources. The challenge here lies in generating concise answers from potentially disordered and lengthier texts. We randomly sample 3,000 pieces of data for training, while randomly choosing 200 samples for inference. The average sample length is 52,158.88 tokens.
- •
TriviaQA(^5^55[https://huggingface.co/datasets/mandarjoshi/trivia˙qa](https://huggingface.co/datasets/mandarjoshi/trivia_qa)) Joshi et al. ()is also a high-quality DocumentQA dataset that houses over 650K question-answer-evidence triples. It includes 95K question-answer pairings authored by trivia enthusiasts and independently sourced evidence documents. We selected 10,000 and 500 samples for training and inference respectively, with the average sample length amounting to 1,075.90 tokens.

### 4.2 Evaluation Metrics

For the task of Summarization, the performance of all methods is measured using the Rouge Lin () metric, which operates by comparing the n-gram of the generated text with that of the reference text. Specifically, we leverage Rouge-1, Rouge-2, and Rouge-L to assess the overlap between the single-token, consecutive dual-tokens, and the longest common subsequence (LCS) in the generated text by LLM and the reference text. We also compute their geometric mean, denoted as G-mean, and higher values reflect higher quality of the generated summaries.

Concerning the task of DocumentQA, we adopt the method demonstrated by Shaham et al. (), which computes the unigram overlap between the generated and reference answers. This is accomplished by normalizing white-spaces, lower-casing, excluding stopwords and punctuation. Based on the number of unigram tokens, in conjunction with the token quantity of the generated and reference answers, we calculate precision, recall, and F1. Again, a higher value indicates a more precise answer by the model.

### 4.3 Baselines and Implementaion Details

Here we describe the baselines and their implementaion details as follows:

- •
Llama2-7B Touvron et al. () refers to the original 7B version of the Llama2 model without fine-tuning. We use Llama2-7b-Chat(^6^66[https://huggingface.co/meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)) for experiments and use it as the backbone of other methods.
- •
YaRN Peng et al. () proposes a novel segmented interpolation method based on the period of different dimensions to extend the context window size. We set the scale factor to 16, and LoRA Hu et al. () is applied on the self-attention module with a rank of 16. The number of the trainable parameters is 17M.
- •
LongLoRA Chen et al. () proposes to use shift short attention instead of original full attention during training, and use Position Interpolation Chen et al. () and LoRA to fine-tune LLM to extend context window. We set the LoRA rank to 16, and fine-tune the self-attention, embeddings, and normalization modules during training according to the original paper, which leads to 140M trainable parameters overall.
- •
RAG Gao et al. () consists of two core processes: retrieval and generation. During the retrieval phase, we adopt GTE-Large-en Li et al. () as the retriever and recall the top-40 relevant context chunks with a maximum length of 512 characters based on cosine similarity, and then they are leveraged as prompts for LLMs during generation, and it does not involve training process.
- •
LLoCO Tan et al. () is a soft compression method that uses Autocompressors Chevalier et al. () to encode long context offline to achieve context window expansion. Consistent with other methods, we employ LoRA on self-attention module with a rank of 16, resulting in the number of trainable parameters to be 17M.
- •
E2LLM For our E2LLM, we utilize the GTE-Large-en Li et al. () as the encoder, which is subsequently fine-tuned using LoRA with a rank parameter set to 8. Additionally, we utilize a two-layer mlp network with a GeLU Hendrycks and Gimpel () activation function as the adapter. As for the decoder component, we leverage Llama2-7B-Chat, also fine-tuning it through LoRA with a rank of 8, and the final number of trainable parameters is 16M.

**Table 2: Performance on Long-Context datasets.**
|  | Trainable<br>Parameters | Context<br>Window | Extension<br>Method | Qmsum | GovReport | Quality | NarrativeQA | TriviaQA |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | R1 | R2 | RL | G-mean | R1 | R2 | RL | G-mean | Prec. | Recall | F1 | Prec. | Recall | F1 | Prec. | Recall | F1 |  |  |  |
| Llama2-7B | 0M | 4K | - | 0.2190 | 0.0491 | 0.1421 | 0.1151 | 0.1068 | 0.0286 | 0.0546 | 0.0550 | 0.0616 | 0.2546 | 0.0938 | 0.0304 | 0.1352 | 0.0465 | 0.0672 | 0.7666 | 0.1206 |
| LongLoRA | 140M | 100K | Sparse Attn. | 0.0741 | 0.0999 | 0.0765 | 0.0741 | 0.2704 | 0.0992 | 0.1629 | 0.1635 | 0.0741 | 0.0999 | 0.0765 | OOM | OOM | OOM | 0.1303 | 0.4928 | 0.1969 |
| YaRN | 17M | 64K | Len. Ext. | 0.2154 | 0.0534 | 0.1624 | 0.1231 | 0.1293 | 0.0413 | 0.0569 | 0.0672 | 0.1320 | 0.1942 | 0.1380 | OOM | OOM | OOM | 0.1353 | 0.4945 | 0.2022 |
| RAG | 0M | $+\infty$ | Hard Comp. | 0.0981 | 0.0172 | 0.0799 | 0.0512 | 0.0893 | 0.0229 | 0.0490 | 0.0465 | 0.0341 | 0.3408 | 0.0583 | 0.0041 | 0.0097 | 0.0055 | 0.0582 | 0.6743 | 0.1047 |
| LLoCO | 17M | 128K | Soft Comp. | 0.2371 | 0.0551 | 0.1679 | 0.1299 | 0.1169 | 0.0311 | 0.0518 | 0.0573 | 0.1681 | 0.1503 | 0.1437 | 0.1185 | 0.1134 | 0.1087 | 0.6404 | 0.6403 | 0.6321 |
| Ours | 16M | 400K | Soft Comp. | 0.2537 | 0.0655 | 0.1875 | 0.1461 | 0.3314 | 0.1075 | 0.1859 | 0.1878 | 0.1344 | 0.1495 | 0.1294 | 0.1353 | 0.1379 | 0.1235 | 0.3322 | 0.3451 | 0.3337 |

### 4.4 Performance Comparison

Here, we utilize three datasets for QA, that is, Quality, NarrativeQA, and TriviaQA, as well as two datasets for summarization, that is, QMsum and GovReport. The details of these datasets are summarized in Table 1. Note that Quality and TriviaQA have shorter lengths compared to the other datasets, but NarrativeQA has a longer length. For our experiments, we employ the validation sets of each dataset for testing, and we split the training sets into training and validation subsets using a 95:5 ratio. We further include the original LLama2-7B-chat as a baseline. To assess performance, we employ the Rouge-N score for summarization tasks, while for QA tasks, we utilize Precision, Recall, and F1-score metrics (see the relevant definitions in Section 4.2). The results for all baseline methods are presented in Table 2.

It is apparent that the proposed E2LLM consistently achieves either the best or the second-best performance across all methods evaluated. Interestingly, Yarn shows promising results in both tasks, however, it encounters out-of-memory (OOM) issues when the sequence length exceeds approximately 74,000 on an A100 GPU with 80G, a consequence of the quadratic space complexity inherent in LLMs. Moreover, Yarn’s performance falters on shorter sequences, such as the found in TriviaQA. As noted in prior research Chen et al. (), attention mechanisms can become dispersed in exceedingly long contexts, spreading too thinly across numerous token positions and thereby degrading performance on shorter contexts. On a related note, LongLoRA, which employs shifted sparse attention during training, exhibits comparable challenges. It tends to overlook informative history, resulting in poorer performance compared to E2LLM across all datasets. LongLoRA also faces OOM issues with the extensive NarrativeQA dataset due to its use of full attention during inference. On the other hand, the original Llama2-7B-Chat performs poorly on QA tasks, primarily due to its limited context length of 4K tokens. RAG shows the worst performance in QA tasks, may be comes from the lost crucial information necessary for specific query, besides, it underperforms the original Llama2-7B in summarization task because the noise introduced during the retrieval process may adversely affect the generation capabilities of LLMs Wu et al. (). In comparison to Yarn, LongLoRA, and RAG, E2LLM demonstrates a superior ability to discern essential information from long contexts while effectively discarding irrelevant data.

Lastly, we observe that LLoCO performs reasonably well in QA tasks, particularly for Quality and TriviaQA, which feature relatively short contexts. However, its performance drops significantly in summarization tasks, aligning with findings in its own publication (see Table 1 in Tan et al. ()). LLoCO employs AutoCompressor Chevalier et al. () as its text encoder, which utilizes Llama2 to generate summary vectors for each chunk. These vectors are designed to retain only the information pertinent to the subsequent chunks, discarding other potentially valuable content, as pointed out in Rau et al. (). In QA tasks, only relevant parts of the long context are required to prompt the LLM for accurate answers, which aligns reasonably with AutoCompressor’s training objectives. However, summarization tasks demand a comprehensive understanding of the entire long context. Since the summary vectors generated by AutoCompressor do not encapsulate all information within each chunk, LLoCO’s performance in summarization suffers. In contrast, E2LLM excels in capturing information across all chunks due to its unique “understanding” or reconstruction task, which contributes to its superior performance in summarization.

Figure: Figure 3: Performance on inference efficiency, including the run-time consumption (left) and gpu memory usage (right).
Refer to caption: x3.png

### 4.5 Inference Efficiency

In this subsection, we examine the inference efficiency of various methods. We begin by selecting seven context lengths evenly distributed between 1K and 73K, as both Yarn and LongLoRA encounter out-of-memory (OOM) issues at a context length of 74K. For each selected context length, we randomly choose ten samples and truncate them to the predefined length. We then average the runtime and GPU memory costs over these ten samples and illustrate the results as a function of context length in Figure 3.

Our model, E2LLM, exhibits the lowest runtime and memory usage, particularly for very long sequences at 73K. In contrast, both YaRN and LongLoRA demonstrate significantly higher resource consumption, primarily due to the quadratic complexity of full attention during inference. Similar to E2LLM, LLoCO also reduces inference time through soft prompt compression. However, its text encoder, AutoCompressor, can compress the original text by a maximum of 32 times, compared to E2LLM’s compression capability of around 100 times. Additionally, AutoCompressor processes all chunks sequentially, whereas E2LLM can handle them in parallel, further reducing inference time. It’s worth noting that the retriever in the RAG model retrieves the 40 most relevant chunks from the long context, regardless of the context length. Consequently, its inference time does not depend on the context length but incurs a greater time cost than E2LLM due to the additional retrieval process. Moreover, retrieving only 40 passages, regardless of the context length, could negatively impact performance, particularly in summarization tasks, as demonstrated in Table 2.

### 4.6 Ablation Studies

In this subsection, we conduct ablation studies of E2LLM using the QMSum and NarrativeQA datasets, which serve as representative benchmarks for long-context summarization and document question-answering tasks, respectively.
Details of each variant examined in Table 3 are outlined below.

- •
$-$und variant entails excluding the “understanding” task from our model and only employing the “reasoning” task for training purposes, which emphasis on the critical role that the “understanding” task plays within the model’s performance.
- •
$-$$\mathbf{\mathcal{E}}$ denotes the freezing of encoder parameters, thereby allowing only the adapter and the decoder-only LLM to be trainable. This configuration aims to substantiate our hypothesis that a pretrained encoder alone is incapable of preserving the pertinent information that significantly impacts the performance of the LLM. Hence, maintaining the encoder’s parameters as trainable is crucial.
- •
$-\mathbf{\mathcal{D}}$ entails keeping the decoder-only LLM frozen, in order to test whether the LLM can still adequately comprehend the output tokens from the adapter in the absence of any dedicated training.
- •
$+$overlap variant introduces an overlap of 30% of the chunk size between sequential chunks during the chunking process. Moreover, within the scope of the “understanding” task’s restatement operation, the model is required to restate the overlapping section of these chunks once.
- •
$+$bge variant test, on the other hand, involves replacing the GTE-Large-en model with the bge-m3 model as the encoder. This study seeks to affirm that our model maintains compatable with different sentence-embedding models serving as encoders.
- •
$+$Llama2$-$13B configuration, similar in testing to the $+$bge variant, is designed to verify the compatibility of our E2LLM with other LLMs serving as decoders.

**Table 3: Ablation Study on QMSum and NarrativeQA.**
|  | QMsum | NarrativeQA | Avg.<br>Rel. Diff. |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | R1 | R2 | RL | G-mean | Prec. | Recall | F1 |  |
| E2LLM | 0.2537 | 0.0655 | 0.1875 | 0.1461 | 0.1353 | 0.1379 | 0.1235 | - |
| -Und | 0.2264 | 0.0486 | 0.1620 | 0.1213 | 0.1113 | 0.1004 | 0.0994 | -16.39% |
| -${\mathcal{E}}$ | 0.2343 | 0.0541 | 0.1731 | 0.1299 | 0.1247 | 0.1125 | 0.1083 | -9.08% |
| -${\mathcal{D}}$ | 0.2309 | 0.0493 | 0.1711 | 0.1249 | 0.1223 | 0.1095 | 0.1046 | -12.03% |
| +Overlap | 0.2523 | 0.0639 | 0.1795 | 0.1425 | 0.1328 | 0.1394 | 0.1241 | +1.78% |
| +BGE | 0.2377 | 0.0607 | 0.1784 | 0.1370 | 0.1289 | 0.1203 | 0.1136 | -4.33% |
| +Llama2-13B | 0.2577 | 0.0672 | 0.1889 | 0.1484 | 0.1375 | 0.1374 | 0.1268 | +4.70% |

First, we assess the significance of the “understanding” task within E2LLM. Our findings indicate a substantial decrease in performance—by 16.39%—when this task is omitted, highlighting its crucial role in helping E2LLM interpret the chunk embeddings produced by the encoder and further enhancing the performance of the “reasoning” task. Next, we examine the necessity of training the LoRA branches of the encoder and the decoder during alignment. As shown in Table 3, the results for configurations -${\mathcal{E}}$ and -${\mathcal{D}}$ underscore the importance of training these components; without this training, E2LLM’s performance diminishes by 9.08% and 12.03%, respectively. Finally, we explore the impact of replacing the chunker, text encoder, and LLM decoder within E2LLM (notated as +overlap, +bge, and +Llama2-13B). Our analysis reveals that chunkers with overlapping segments (e.g., 30% overlap) provide a modest performance boost. Additionally, employing more advanced encoders and decoders further enhances E2LLM’s performance, suggesting that improvements in individual components can positively affect the overall system.

Figure: Figure 4: Effect of the hyperparameter. (a) the loss weight of “understanding” task. (b) the number of layers in adapter network. (c) the number of trainable layers of the encoder.
Refer to caption: x4.png

### 4.7 Hyperparameter Sensitivity

In this section, we explore the effects of hyperparameters on the performance of E2LLM, specifically focusing on the weight assigned to the “understanding” task, the LoRA rank of the encoder and decoder, and the number of layers in the adapter network.

The weight assigned to the “understanding” task indicates its relative importance compared to the “reasoning” task. Recall that the input context typically has a much longer length than answers, making it too long to be fully reconstructed at once. To address this, we employ a sliding window approach, reconstructing the original context in segments based on a few consecutive chunks until the entire input has been reconstructed. Consequently, the samples for the “understanding” task are significantly more numerous than those for the “reasoning” tasks. To maintain sample balance, we usually assign a smaller weight to the restatement task. As depicted in Figure 4, the optimal weight may vary across different datasets, which may be influenced by factors such as context length and the sentence embedding model’s capacity to comprehend the specific semantics of the context.

Moreover, we investigate the optimal LoRA rank of the encoder (i.e., GTE-Large-en) and the decoder (i.e., Llama2-7B-Chat) within the range of {0, 4, 8, 12, 16, 20, 24} and {0, 2, 4, 6, 8, 10, 12}, respectively. The findings suggest that having no trainable parameters—in other words, completely "freezing" the encoder and decoder—hinders the effective extraction of original context content and alignment between the encoder and decoder, as discussed in Section 3.1. As the rank of the two modules increases, a corresponding improvement in performance is observed, thereby underscoring the importance of training. Performance enhancement continues until it reaches a peak within a specific range of ranks. However, beyond this optimal range, further increases in rank lead to a decline in performance, attributable to overfitting on the training datasets.

We also examine the impact of the number of layers in the adapter network. Figure 4 shows that a two-layer MLP consistently delivers superior performance across different datasets, indicating stability in results. We hypothesize that a single-layer MLP may struggle with the alignment task, while a three-layer MLP might lead to overfitting on the training data.

## 5 Conclusion

In this paper, we present E2LLM, a novel approach to address the challenges of enhancing long-context performance in LLMs.
It effectively navigates the “impossible triangle” by strategically splitting long contexts into chunks, compressing them into embedding vectors, and utilizing an adapter to align these representations with a decoder-only LLM.
Two training objectives are employed to facilitate the understanding of soft prompts by the LLMs, resulting in superior performance in long-context scenarios.
Experimental findings reveal that E2LLM effectively outperforms existing approaches in balancing the long-context performance, computational efficiency, and model compatibility.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: Q1. What's the adapter is chosen and why? 

Q2. How the chunk size will affect results


See weakness for most of the comments and questions.
soundness: 3
strengths: 1. Solving long-context language modeling is an important problem

2. Compression yields benefits in computation efficiency

3. In their selected benchmarks, their model yield good performance.
summary: The paper presents E2LLM, an architecture that addresses long-context processing in LLMs through a chunk-based approach. 

At its core, E2LLM employs a pretrained text encoder to compress text chunks into embedding vectors, which are then aligned with a decoder-only LLM (LLaMA-2) through a two-layer MLP adapter. 

E2LLM also propose a two-stage training process: 1. an "understanding" task where the model reconstructs the original text from compressed representations, and 2. a "reasoning" task focused on answering queries using these compressed contexts. 

The authors position their work as solving the "impossible triangle"(I am not so sure about the reference here) of maintaining strong performance, computational efficiency, and compatibility with pretrained models.

I am concerned with unclear explanations of the core model architectures, not well collected benchmarks, and lack of in-depth analysis
weaknesses: ## Benchmark results are not tailored for the long context length argued in the paper. 

As far as I understand, most of benchmarks stated in this paper, their AVG length is not up-to-128k. And benchmarks are not on par with the recent advances, for example, RULER where there are way more challenging tasks mentioned in recent papers for example HELMET[1] and long context controlled study[3]

-  I would encourage authors to break down the task accuracy by length to show the real benefit of their method.

- Also adding some recent benchmarks, for example Ruler[1], and HELMET[2] to have real long context evaluation thoroughly.

- I think the Needle task is not well presented. I understand that authors wanna focus on the accuracy, but the heat map might give us a clear image on where the model fails, and the trend per position more clearly w.r.t needle depth. Is it possible to have a similar analysis?


## Presentation can be improved.

- I am not so sure that I understand the compression process. The major compression is depending on the chunk size in this paper, but it is not well studies how this chunk size will influence the results, or how will influence the results, semantically for example. I would encourage authors break down the Negative LL per token position to demonstrate the effect of chunk size, particularly at boundaries. 

- Model architecture details are missing. Maybe this can be benefit from adding a clear model architecture figure and justifying the design choice. Figure 1 demonstrates the idea at very high level and paragraph descriptions are not very clear in 3.1. 

References

[1] RULER: What's the Real Context Size of Your Long-Context Language Models?

[2] HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

[3] A Controlled Study on Long Context Extension and Generalization in LLMs

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: In addition to the questions listed in the Weakness section: 

- In Figure 4, why do the inference time and memory usage increase for StreamingLLM? It keeps a constant size of starting and recent KV caches, so it should stay constant past a certain length. 
- Why is E2LLM better than YaRN and LLoCO on NarrativeQA in Table 1 but not in Table 2? 
- In Sec 4.1, the datasets listed are QMSum, GovReport, HotpotQA, NarrativeQA, and TriviaQA, but the corresponding Table 4 lists Quality instead of HotpotQA for the statistics, and Table 1 shows Quality instead of HotpotQA.
- How important are the two training tasks in Sec 3.2? It's not clear how they are used during training and how they affect the performance and the training efficiency, as reconstructing the inputs from the encoder introduces additional costs.
soundness: 2
strengths: - E2LLM is a novel method with significant inference benefits compared to previous works. It shows that encoder and decoder can be connected with an adapter in the middle. 
- The method shows strong empirical results on downstream tasks in the supervised training settings., and includes a number of tasks in the evaluation setting. 
- The paper also includes a number of ablation studies that reveal more insights regarding the method.
summary: The paper introduces E2LLM, a method for long-context language modeling. The main components consist of an encoder, an adapter, and a decoder-only LLM. It uses Llama 2 7B chat as the backbone LLM and GTE as the encoder. It’s evaluated on several long-context and QA tasks like QMSum, GovReport, Quality, NarrativeQA, and TriviaQA, and achieves competitive results. It’s also evaluated on the needle-in-a-haystack and LongBench.
weaknesses: - About the baseline methods, Appendix B states that Llama 2 7B is “the backbone for the other methods”, but E2LLM uses Llama 2 7B Chat as the decoder. It doesn’t seem fair to compare methods that use different backbone models. What would the performance of E2LLM be if Llama 2 7B were used as the backbone model instead? Or if the other methods use the chat model as the backbone? 
- A key difference between previous methods and E2LLM is that they are typically studied in the pre-training/continual pre-training settings where they are first trained on a corpus like RedPajama (such as in the case of YaRN, LongLoRA, CEPE, and others), and then evaluated either zero-shot or with ICL on downstream tasks. It would be important to see how E2LLM would perform in such settings in addition to the supervised fine-tuning setting. 

On the writing, there are some citation errors, for example: 
Line 033: Giorgi et al. → Giorgi et al., 2023. 
Line 053: Ding et al. → Ding et al., 2024.
Line 090: Kenton & Toutanova, 2019 → Devlin et al., 2019.
There are others throughout the paper.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: Please refer to the above.
soundness: 3
strengths: 1. The presentation is pretty clear to me and the T1, T2 and T3 categories make lot of sense and easy to follow. They summarize current methods in terms of them.
2. They do a comprehensive evaluation on their model to show the effectiveness.
3. The model aligns the encoder and decoder via a trainable adapter, minimizing the need for extensive retraining and making it easier to implement.
summary: In this paper, authors try to propose a framework to support LLM at long context side. They chunk the input text into different parts and try to compress then by an encoder and a LORA adaptor is used to improve the representation. They propose two training objectives to handle the understanding and reasoning at the same time. They conduct experiments on several benchmarks and show the effectiveness of their method.
weaknesses: 1. The encoder-decoder framework is reasonable but I am concerned that what if the LLM is a large scale one like llama 70B. will the train be feasible. Seems author just conductor experiments on small scales.
2. The chunk size is pretty heuristic and with different tasks, the performance varies with chunk size. Could we make it learnable as well?
3. Maybe another concern is the real time application, how do we plan for that given this framework.
4. With the encoder and adapter, how does the inference speed compared to the baseline?
5. Can we do the eval on RULER?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Other than the problems mentioned in the weaknesses section, below are some other questions:
1. In line 396, you mentioned "E2LLM addresses these challenges...", which includes the challenge that StreamingLLM requires training. However, E2LLM also requires training to align the encoder, adaptor, and decoder. Please explain this claim.
2. In line 404, you mentioned that YaRN faces the challenge that "attention mechanisms can become dispersed in exceedingly long contexts". However, YaRN proposes the dynamic scaling method for the attention scores to alleviate this problem.
3. In line 265, it would be easier to understand if "sequence length" could be replaced by "context window" for the input capacity of the model.
soundness: 2
strengths: 1. The paper is overall well-written and easy to follow.
2. The integration of an encoder to compress long contexts while preserving essential information is innovative and leverages existing pretrained architectures efficiently.
3. The extensive experiments show the proposed method's performance, efficiency, and robustness. The method could serve as a strong baseline for future works.
summary: This paper introduces E2LLM, designed to efficiently and effectively handle long-context understanding in LLMs. The method balances long-context performance, computational efficiency, and compatibility. E2LLM employs a soft prompt compression technique, segmenting long contexts into chunks, compressing them into embeddings using a pretrained text encoder, and aligning these embeddings with a decoder-only LLM through an adapter. The proposed training objectives of the model focus on reconstructing encoder outputs and fine-tuning for long-context instructions. Experiments demonstrate that E2LLM outperforms existing SoTA methods across multiple long-context benchmarks including summarization, QA, needle-in-a-haystack, and LongBench, while achieving low inference time and memory usage.
weaknesses: 1. The authors would like to introduce E2LLM as a general framework that could work with any LLMs, which is intended to show the method's flexibility. However, when compared to similar methods like CEPE, both the encoder and the LoRA rank used are different. This both makes the "Trainable Parameters" in Table 1 incomparable and makes me doubt whether the improvement comes from the more powerful GTE-Large-en encoder or the proposed method.
2. It is unsure how the authors trained E2LLM and the baselines on the tasks specified in Appendix C. How were the hyperparameters for each method decided?
3. The analysis of E2LLM's complexity has flaws. As per Figure 2, the input contains a span (prompt + query) that is directly put into the decoder. The complexity needs to separate the computation of the "context" part and the "prompt" part.
4. The paper misses some related works. For example, for position embedding extension methods, there have been better methods [1] or improvements to YaRN [2]. For methods similar to CEPE, the concurrent work FocusLLM [3] has a very similar design to E2LLM.

[1] Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2023. CLEX: Continuous Length Extrapolation for Large Language Models. In ICLR 2024.

[2] Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. 2024. Resonance RoPE: Improving Context Length Generalization of Large Language Models. In ACL 2024 Findings.

[3] Zhenyu Li, Yike Zhang, Tengyu Pan, Yutao Sun, Zhichao Duan, Junjie Fang, Rong Han, Zixuan Wang, and Jianyong Wang. 2024. FocusLLM: Scaling LLM’s Context by Parallel Decoding. arXiv Preprint.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the maximum compression rate of E2LLM such that the context can be recovered losslessly?

2. Can E2LLM effectively harness the longer context windows? In other words, does the PPL/performance of E2LLM continue decreasing/increasing when the context window becomes longer?
soundness: 3
strengths: 1. E2LLM achieves a good balance between performance, efficiency, and compatibility;

2. The model is trained end-to-end, ensuring the alignment between the encoder and decoder and minimizing information loss;

3. Extensive experiments demonstrate that E2LLM outperforms several existing methods on long-context tasks such as document summarization, QA, Needle-in-a-Haystack, as well as LongBench;

4.  E2LLM achieves the lowest inference time and memory usage among compared methods.
summary: This paper presents E2LLM, a novel approach to tackle the challenge of processing long contexts in LLMs. The authors identify three key challenges: performance (T1), efficiency (T2), and compatibility (T3). E2LLM addresses these challenges by (1) splitting long contexts into chunks; (2) compressing each chunk into an embedding vector using a pre-trained text encoder; (3) utilizing an adapter to align the encoder’s output with the decoder’s input embedding space; (4) training two objectives: reconstructing the original text from the encoder’s output and fine-tuning the LLM on long-context instruction tasks.
weaknesses: 1. The authors seem to overclaim that "E2LLM solves all the challenges of the impossible triangle" since the performance on long-context tasks of E2LLM still falls far behind "continual pertaining on longer texts + long-context SFT" paradigm [1, 2] (though there is no comparison between E2LLM and this approach in the paper), which is the mainstream and SoTA approach now and widely used in most advanced LLMs such as Llama3.1 and GLM4. This makes the real-world applicability of E2LLM doubtful. A comparison between E2LLM and "continual pretraining + SFT" approach should be presented to support the claim.

2. The figure of Needle-in-a-Haystack is deceptive since the color is all green (the score less than 5 should be marked in red). Though outperforms the baseline methods, E2LLM still struggles in recalling the needle with only 30% success rate， while current long-context LLMs can achieve nearly 100% performance. 

[1] Effective Long-Context Scaling of Foundation Models
[2] LongAlign: A Recipe for Long Context Alignment of Large Language Models

Meta-review: This paper proposes a sub-quadratic transformer intended for generating text conditional on some long input context. The architecture, resemblant of the legacy encoder-decoder models, first compresses the context by chunking and applying a local encoder and temporal pooling. These compressed context representations are then prepended to the input of the decoder.

Authors proposed an approach to leverage existing encoders and decoders and combine them by means of fine tuning with adapters.

The proposal is simple and interesting, and the reported evaluations suggest it has potential to yield performant transformer variants. However, the manuscript has important issues at its current state even after back and forth with reviewers that prevent a recommendation for acceptance. To list a few examples:

- The presentation is lacking in various aspects and some important details are missing. For instance, section 3.2 doesn't provide details on training such as the datasets that are used to train the adapters and for how long they trained.
- Chunking approaches to yield sub-quadratic models was studied in the past and authors missed important references. One such example would be MegaByte and variants of it that were proposed more recently.
- The choice of baselines seems unfair sometimes since the proposed models were fine-tuned in such a way that targeted the reported evaluations, while not all baselines are fine-tuned. In fact, results would be more reliable if comparisons were carried out against baseline performances computed by others (e.g., using some leader board maintained by a third party) in addition to the baselines prepared by the authors.
- The motivation for the exact proposed architecture is unclear. There are several possible configurations that would leverage chunking and an encoder, and it's unclear why the authors picked the exact one reported in the paper.
- The main issue in my opinion is the choice of chunk sizes, which seem too small to yield significant improvements. The complexity under chunking $O(LC+L^2/C^2)$ will only be relevantly sub-quadratic if $C$ is large with respect to $L$. E.g., if one picks $C=L^{1/3}$, then we'd have $O(C^{4/3})$, but the chunk sizes in the experiments are much smaller than that. This also makes the performance results in Figure 4 a bit surprising since one would expect worse  scaling (almost as bad as quadratic) for such small chunks (512 chars).

All in all, while the approach is interesting, the manuscript needs further work prior to publication in terms of justifying the design choices better (either formally or via ablations), comparing to existing chunking-based approaches, and comparing agains results reported by others in addition to the authors implementations of baselines.

Decision: Reject