Pretraining Context Compressor for Large Language Models with Embedding-Based Memory
Authors: Yuhong Dai, Jianxun Lian, Yitian Huang, Wei Zhang, Mingyang Zhou, Mingqi Wu, Xing Xie, Hao Liao
Accepted at: acl25

Pretraining Context Compressor for Large Language Models with Embedding-Based Memory

Yuhong Dai1, Jianxun Lian2, Yitian Huang1, Wei Zhang1, Mingyang Zhou1, Mingqi Wu3, Xing Xie2, Hao Liao1
1College of Computer Science and Software Engineering, Shenzhen University, China
2Microsoft Research Asia
3Microsoft Gaming
jianxun.lian@outlook.com, haoliao@szu.edu.cn

Abstract

Efficient processing of long contexts in large language models (LLMs) is essential for real-world applications like retrieval-augmented generation and in-context learning, especially in resource-constrained environments such as edge computing. This paper explores the embedding-based context compression to reduce inference costs while preserving the downstream LLM configurations. We propose a decoupled compressor-LLM framework, pretrained on text reconstruction and completion tasks, designed to effectively preserve essential contextual information within condensed embedding representations. Our extensive experiments investigate pretraining, model configurations, compression rates, efficiency across tasks, and adaptability to various LLMs. Results demonstrate that our approach outperforms competitive baselines in three domains and across eight datasets while being adaptable to different downstream LLMs. We find that thorough pretraining and carefully selected compression rates, such as 4x and 16x, enable a lightweight compressor to achieve a good balance between accuracy and speed. These findings underscore the potential of embedding-based compression to enhance LLM efficiency and motivate further research in this area.

1 Introduction

Processing long context as input in large language models (LLMs) is essential for numerous real-world tasks, such as retrieval-augmented generation (RAG) (Gao et al., 2023), in-context learning (Li et al., 2024a) scenarios with extensive content in each demonstration, and life-long AI companions (Zhong et al., 2024) requiring the preservation of past interactions for a consistent experience. Consequently, enabling LLMs with long context processing capabilities (Chen et al., 2023b; Ding et al., 2024) has become a significant trend in newly released models. However, inference with long context poses severe challenges, as the computational cost increases quadratically with the sequence length (as illustrated in Figure A1), which is particularly unfriendly for low-resource devices such as consumer endpoints.

To mitigate these costs without altering LLM architecture, explicit and implicit compression methods have been proposed. Explicit compression methods, such as Selective Context (Li et al., 2023) and LLMLingua-2 (Pan et al., 2024), compact the context by removing redundant or unimportant tokens from the input. Implicit compression methods, such as AutoCompressor (Chevalier et al., 2023), xRAG (Cheng et al., 2024), and ICAE (Ge et al., 2024), compress the input into several dense embedding vectors (aka memory slots). This approach is inspired by the idea that embedding representations act as a new type of compact information modality, conveying denser information than explicit text tokens.

In this paper, we focus on implicit compression, as it has the potential to achieve higher compression rates, leading to more efficient LLM inference. While there are several pioneering works in this direction, we observe that some fundamental questions remain understudied. For instance, can we develop a universal compressor that benefits various downstream LLMs? Can we decouple the compressor from the downstream LLMs, allowing the compressor architecture to be much more lightweight while still maintaining high performance? Additionally, how does the performance change with varying compression rates? Addressing these questions, our research aims to provide insights into designing and optimizing implicit compression methods for efficient LLMs.

We propose a context compression architecture, as illustrated in Figure 1. Our design principle is to decouple the context compressor from the downstream LLM, ensuring that the compressor can be lighter than the LLM. The compressor and
the LLM generator are connected via a converter, which projects compressed memory slots into compatible vectors for the downstream LLM generator. We pretrain the context compressor using two tasks: the reconstruction task and the language completion task. The goal of these tasks is to enable the compressed slots to effectively memorize useful information from the context, thereby supporting high-quality language generation. We conduct thorough experiments, examining key aspects of the context compressor, such as pretraining tasks, model and data size, compression rate, and efficiency in various downstream tasks. These experiments lead to the following key findings:

Pretraining: Thorough pretraining of the compressor model is critical for achieving optimal performance. Both content reconstruction and language completion tasks are important for generating effective memory slots.

Suggested Compression Rates: 4x and 16x compression rates are recommended for current applications. A 4x compression achieves near-perfect content reconstruction, while 16x offers significant speed gains with minimal information loss.

Higher Compression Rates: Rates like 128x and 256x can still condense useful information but result in higher information loss, highlighted by the challenges in compressor pretraining.

Model Size: Larger backbone models for compressors generally produce higher quality memory slots, indicating the untapped capacity of memory slots.

Scaling to Longer Contexts: To scale to longer context compression with multiple memory slots, it is important to wrap each segment of memory slots with special tokens to clearly delineate memory boundaries.

These findings suggest that embedding-based context compression holds significant potential for improving the efficiency of LLM inference, though its full capabilities have yet to be fully explored. We hope this paper serves as a foundation and motivates further research on pretraining universal context compressors for LLMs. To summarize, the main contributions of this paper are:

• We conduct empirical studies on context compression for efficient LLM inference, utilizing a decoupled compressor-LLM framework and two pretraining tasks. We emphasize that embedding-based compression is a promising and underexplored direction in current literature.

• We conduct a comprehensive investigation into the impact of pretraining and other key configurations, such as compression rate and memory boundary tokens, on the effectiveness of context compression. Based on our findings, we provide key insights for best practices in this area.

• Our method outperforms competitive baselines across three different domains and eight datasets. Additionally, we demonstrate the high adaptability of our method to various downstream LLMs. Source code are released at https://aka.ms/memorycompressor.

2 Methodology

2.1 Problem Definition

This paper focuses on implicit context compression. Given an input sequence \( \mathbf{x} = (x_1, \ldots, x_t) \) of \( t \) tokens, a compressor model converts \( \mathbf{x} \) into a dense memory representation:

\[
\tilde{\mathbf{h}} = f_{comp}(x_1, \ldots, x_t)
\]

where \( \tilde{\mathbf{h}} = (\mathbf{h}_1, \ldots, \mathbf{h}_m) \) is a concatenation of \( m \) embedding vectors, and the compression rate is \( r = t/m \). The dense memory can then substitute the original context \( \mathbf{x} \) for downstream tasks in an LLM for text generation:

\[
f_{lm} : (\tilde{\mathbf{h}}, prompt) \rightarrow \mathcal{X}
\]

Our approach involves a two-stage training process that includes both pretraining and fine-tuning of the compressor, while keeping the downstream LLM frozen. An overview of the framework is presented in Figure 1.

2.2 Model Architecture

To enhance adaptability to different LLMs, our compressor consists of two components: an encoder and a converter. The encoder condenses the context into embedding representations, with the embedding dimension determined by the encoder’s backbone model. These representations are then processed by the converter, which scales their dimensions and adjust the embedding semantics. This ensures that the memory vectors are compatible with a wide range of downstream LLMs, allowing for more efficient and flexible inference across diverse models and tasks.

Memory Encoder. The memory encoder transforms textual input into high-information-density memory embeddings. As mainstream LLMs have
increasingly adopted the decoder-only transformer structure in recent years, we use this structure as the backbone in experiments, utilizing two variants: GPT2-Large\(^1\) and Llama-3-8B-Instruct\(^2\). Specifically, given a sequence of input tokens to be compressed, \((x_1, \ldots, x_t)\), we first append \(m\) special memory tokens (\(<mem_1>, ..., <mem_m>\)) to the end. We then take the hidden representations of these memory tokens from the final transformer layer as the encoded memory slots. In our experiments, \(t\) is set to 256 as we explore the maximum compression rate of 256x. If the original context is longer than \(t\), it will be divided into multiple segments, each generating memory slots independently.

**Memory Converter.** The converter acts as an intermediary to adapt the memory embeddings, ensuring compatibility with the target LLM decoder. It addresses potential mismatches in both the embedding dimensions and the semantics produced by the memory encoder, aligning them with the requirements of the downstream LLM decoder. Additionally, the converter facilitates efficient adaptability to various downstream LLMs without large-scale pre-training. This strategy utilizes the pre-trained knowledge embedded in the compressor while enabling efficient adaptation to new decoders. Specifically, the converter is a two-layer MLP, with RM-

SNorm (Zhang and Sennrich, 2019) applied to the memory embeddings and GELU (Hendrycks and Gimpel, 2016) activations in the first hidden layer.

**2.3 Pre-training**

Unlike other scenarios such as dense retrieval (Xiao et al., 2022) or generative language modeling (Radford et al., 2018), where pre-training tasks typically involve either token reconstruction or next token prediction, compressed memories play a unique role. On one hand, we want the memories to recall the original content, and on the other hand, we want them to assist in the generation of future tokens. Therefore, we use both types of pre-training tasks: Auto-Encoding for text reconstruction and Auto-Regression for text completion. Specifically, the compressor first compresses the original lengthy context x into memory slots h. To distinguish the boundaries of memory slots from different context segments (as mentioned in the *Memory Encoder* Section, lengthy context will be divided into segments), a pair of special tokens <MEM> and </MEM>, which indicate the beginning and end of a memory representation, are used to wrap each memory slot. Containing memory slots in the input prompt, an LLM performs the following pre-training tasks:

**Text Completion:** The purpose of text completion is to allow memory representation to extract information from the original text that is beneficial to LLMs. Using the memory representation \( \tilde{h} \) of
the input sequence prefix \((x_1, ..., x_{k-1})\), the LLM completes the rest part of sequence:

\[
\mathcal{L}_{TC} = -\frac{1}{n-k} \sum_{i=k+1}^{n} \log f_{lm}(x_i|\tilde{\mathbf{h}}, x_k, ...x_{i-1})
\]

(3)

**Text Reconstruction:** Instead of using a prompt like "Please repeat the sentence:" to instruct the LLM to perform the text reconstruction task, we follow (Ge et al., 2024) and use a new token <AE> after the memory \( \mathbf{h} \) to prompt the LLM to conduct text reconstruction:

\[
\mathcal{L}_{TR} = -\frac{1}{n} \sum_{i=1}^{n} \log f_{lm}(x_i|\tilde{\mathbf{h}}, <\mathrm{AE}>, x_1, ...x_{i-1})
\]

(4)

The final loss function, \( \mathcal{L} \), is defined as:

\[
\mathcal{L} = \lambda \mathcal{L}_{TC} + (1-\lambda) \mathcal{L}_{TR}
\]

(5)

We observe that the difficulty of text reconstruction from memories increases with the length of the original context. To facilitate training, we propose a two-step strategy. In the first stage, we train the model on a warm-up dataset containing shorter texts and utilizing a lower compression rate. Specifically, texts are divided into fixed-length sequences of 128 tokens and a 4x compression rate is applied. This configuration is used solely for the text reconstruction task and utilize 32M tokens for training. In the second stage, training is conducted on a larger dataset with 5B tokens with extended text segments of 256 tokens. Both text completion and text reconstruction tasks are carried out concurrently, with the weighting coefficient \( \lambda \) set to 0.5. Since pre-training is crucial for achieving a high-quality context compressor, we denote our method as **PCC**, short for "Pre-training Context Compressor."

**2.4 Fine-tuning**

While pre-training enables the compressor to condense information in a general manner, different specific scenarios, such as RAG-based QA or in-context learning, may require slightly different memorization patterns. To capture this nuance, we conduct domain-level fine-tuning on the compressor. Typically, the fine-tuning stage requires only a small amount of domain-specific data to achieve superior performance. Here, "domain-level" indicates that we do not fine-tune the compressor on a dataset-by-dataset basis. Instead, we group datasets by their application type (such as RAG) and fine-tune on only one of the datasets within the same domain to observe whether the compressor can generalize to other datasets within this domain.

<table>
  <tr>
    <th>Stage</th>
    <th>Source</th>
    <th>Train</th>
    <th>Test</th>
  </tr>
  <tr>
    <td rowspan="2">Pre-training</td>
    <td>FineWeb</td>
    <td>19,167,479</td>
    <td>576</td>
  </tr>
  <tr>
    <td>SQuAD</td>
    <td>86,821</td>
    <td>5,928</td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>GSM8K</td>
    <td>6,725</td>
    <td>748</td>
  </tr>
  <tr>
    <td>HPD</td>
    <td>987</td>
    <td>110</td>
  </tr>
</table>

Table 1: Basic Statistics on the number of data samples of the Datasets

3 **Experiments**

**3.1 Experimental Setup**

**3.1.1 Datasets**

This study leverages multiple datasets, including FineWeb (Penedo et al., 2024), SQuAD (Rajpurkar et al., 2018), and GSM8K (Cobbe et al., 2021), to support training and evaluation. A detailed summary of the datasets utilized at different training stages is provided in Table 1. During the pre-training phase, the FineWeb dataset is employed. We utilize 5 billion tokens from the FineWeb. For the fine-tuning phase, three datasets are incorporated: SQuAD, with 86,821 training examples and 5,928 test examples; GSM8K, consisting of 6,725 training examples and 748 test examples; and HPD (Harry Potter Dialogue dataset) (Chen et al., 2023a), containing 987 training examples and 110 test examples. The training sets of SQuAD, GSM8K and HPD are employed as the datasets for the QA tasks and in-context learning tasks, respectively. The selection of these datasets ensures a diverse set of tasks, enabling robust and comprehensive evaluation across the pre-training and fine-tuning stages.

**3.1.2 Experimental Details**

During pretraining, we employ Llama-3-8B-Instruct as the target LLM and train two types of compressors: a lightweight compressor and a large compressor, which uses the same decoder-only architecture. For the lightweight compressor, we utilize GPT2-Large as the base model, trained using a full-parameter fine-tuning method. The large compressor employs Llama-3-8B-Instruct as its base model, trained with Low-Rank Adaptation (LoRA) (Hu et al., 2021) with a parameter setting of \( r = 64 \). The learning rates are set to
\(1 \times 10^{-4}\) during the pre-training phase. The trainable parameters remain consistent between the pre-training and fine-tuning phases. Pre-training takes 123 hours for PCC (lite) and 204 hours for PCC (large) using four A100 (80GB) GPUs on a node. In all experiments, the decoding method for the LLM decoder is greedy decoding. Further implementation details can be found in Section A.1.

3.1.3 Baselines
We adopt five competitive context compression models as the baselines:
AutoCompressor (Chevalier et al., 2023) We use the princeton-nlp/AutoCompressor-Llama-2-7b-6k checkpoint, which pre-train from Llama-2-7b-hf\(^3\), to generate summary vectors, achieving a compression rate of 40x.
XRAG (Cheng et al., 2024) We use the Hannibal046/xrag-7b checkpoint. It employs Mistral-7B-Instruct-v0.2\(^4\) as its target LLM and is extensively fine-tuned on multiple downstream datasets, including SQuAD, NQ, and TriviaQA.
COCOM (Rau et al., 2024) We adopt the 4x, 16x, and 128x Lite compression models released by the authors, along with the fine-tuned target LLM Mistral-7B-Instruct-v0.2.
ICAE (Ge et al., 2024) We use the sgge-tao/icae/mistral_7b_ft_icae checkpoint with 4x compression rate. It employs Mistral-7B-Instruct-v0.2 as its target LLM.
LLMLingua-2 (Pan et al., 2024) LLMLingua-2 is an explicit compression method. For our experiments, we use the microsoft/llmlingua-2-xlm-roberta-large-meetingbank checkpoint. The compression rate is set to 4x, and we conduct experiments with Llama-3-8B-Instruct as the target LLM.

3.2 Pre-Training Dynamics
We begin by examining the convergence of pre-training using different configurations.
Text Reconstruction Figure 2a illustrates the ability to reconstruct original text from memory slots. We use commonly-used metrics in machine translation, namely BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). The results indicate that with compression rates of 4x and 16x, the compressor captures almost all the original content. However, the reconstruction quality drops sharply at 64x compression. Figure 2b presents the training curves, showing that lower compression rates, such as 4x and 16x, quickly converge to a low loss state, while higher compression rates face difficulties in convergence. Switching to a larger compressor only slightly improves the situation but does not change the overall pattern, as shown in Figure 2c. We offer a case study on text reconstruction with PCC in Table A5 in Appendix.
Text Completion Figure 3 presents the training curves for the text completion task. The main conclusions are similar to those in the text reconstruction task, but with two unique characteristics. First, compressors with lower compression rates take longer to converge. For example, the 4x compressor takes the longest time to converge, while compression rates of 64x and above converge quickly but remain at a high loss value. Second, the advantage of larger compressors over smaller ones is much more significant in the text completion task compared to the text reconstruction task. Additionally, we evaluate the perplexity with various number of memory slots in the context, and the results are presented in Table A1 in Appendix A.2.

![The converged metrics and pre-training curves for various configurations in the text reconstruction task.](page_97_120_1297_370.png)

Figure 2: The converged metrics and pre-training curves for various configurations in the text reconstruction task.

\(^3\)https://huggingface.co/meta-llama/Llama-2-7b-hf
\(^4\)https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">SQuAD</th>
    <th colspan="4">HotPotQA</th>
    <th colspan="4">AdversarialQA</th>
    <th colspan="4">NQ</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>Metrics</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
  </tr>
  <tr>
    <td rowspan="2"><b>Reference</b></td>
    <td>w/o Context</td>
    <td>14.84</td>
    <td>3.41</td>
    <td>24.80</td>
    <td>14.90</td>
    <td>12.05</td>
    <td>6.43</td>
    <td>27.48</td>
    <td>15.02</td>
    <td>19.79</td>
    <td>9.94</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ Context</td>
    <td>79.81</td>
    <td>59.97</td>
    <td>64.60</td>
    <td>51.12</td>
    <td>56.10</td>
    <td>38.67</td>
    <td>64.64</td>
    <td>51.38</td>
    <td>66.29</td>
    <td>50.29</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="7"><b>Baselines</b></td>
    <td>AutoCompressor</td>
    <td>21.46</td>
    <td>0.35</td>
    <td>16.29</td>
    <td>0.29</td>
    <td>14.09</td>
    <td>2.00</td>
    <td>25.57</td>
    <td>0.63</td>
    <td>19.35</td>
    <td>0.82</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>xRAG</td>
    <td>18.19</td>
    <td>3.46</td>
    <td>27.51</td>
    <td>16.29</td>
    <td>13.75</td>
    <td>3.47</td>
    <td>38.06</td>
    <td>20.80</td>
    <td>24.38</td>
    <td>11.01</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>4x COCOM-Lite</td>
    <td>21.70</td>
    <td>9.17</td>
    <td>40.07</td>
    <td>32.32</td>
    <td>19.45</td>
    <td>13.90</td>
    <td>50.45</td>
    <td>41.87</td>
    <td>32.92</td>
    <td>24.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>16x COCOM-Lite</td>
    <td>19.23</td>
    <td>8.13</td>
    <td>31.94</td>
    <td>25.27</td>
    <td>19.35</td>
    <td>14.73</td>
    <td>26.36</td>
    <td>20.66</td>
    <td>24.22</td>
    <td>17.20</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>128x COCOM-Lite</td>
    <td>19.56</td>
    <td>7.61</td>
    <td>23.63</td>
    <td>18.68</td>
    <td>19.47</td>
    <td>15.13</td>
    <td>18.79</td>
    <td>14.36</td>
    <td>20.36</td>
    <td>13.95</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ICAE</td>
    <td>45.69</td>
    <td>21.63</td>
    <td>35.16</td>
    <td>26.68</td>
    <td>27.98</td>
    <td>11.70</td>
    <td>59.15</td>
    <td>47.35</td>
    <td>42.00</td>
    <td>26.84</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLMLingua2</td>
    <td>51.20</td>
    <td>32.18</td>
    <td><b>55.72</b></td>
    <td><b>44.18</b></td>
    <td>35.41</td>
    <td>24.80</td>
    <td>68.44</td>
    <td>55.85</td>
    <td>52.69</td>
    <td>39.25</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="7"><b>PCC (Lite)</b></td>
    <td>4x Memory Context</td>
    <td>75.83</td>
    <td>57.44</td>
    <td>50.37</td>
    <td>42.20</td>
    <td>50.36</td>
    <td>37.83</td>
    <td>70.51</td>
    <td>61.97</td>
    <td>61.77</td>
    <td>49.86</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>16x Memory Context</td>
    <td>56.82</td>
    <td>37.72</td>
    <td>35.67</td>
    <td>27.88</td>
    <td>40.94</td>
    <td>28.07</td>
    <td>62.56</td>
    <td>52.73</td>
    <td>49.00</td>
    <td>36.60</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>64x Memory Context</td>
    <td>37.89</td>
    <td>22.66</td>
    <td>20.62</td>
    <td>14.27</td>
    <td>27.02</td>
    <td>16.30</td>
    <td>36.82</td>
    <td>29.18</td>
    <td>30.59</td>
    <td>20.60</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>128x Memory Context</td>
    <td>32.24</td>
    <td>15.92</td>
    <td>22.98</td>
    <td>16.92</td>
    <td>25.18</td>
    <td>15.73</td>
    <td>44.25</td>
    <td>36.08</td>
    <td>31.16</td>
    <td>21.16</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>256x Memory Context</td>
    <td>26.71</td>
    <td>12.63</td>
    <td>18.69</td>
    <td>12.92</td>
    <td>22.32</td>
    <td>13.93</td>
    <td>35.69</td>
    <td>27.86</td>
    <td>25.85</td>
    <td>16.84</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3"><b>PCC (Large)</b></td>
    <td>4x Memory Context</td>
    <td>77.76</td>
    <td><b>60.04</b></td>
    <td>48.19</td>
    <td>39.97</td>
    <td><b>52.56</b></td>
    <td><b>39.37</b></td>
    <td><b>75.96</b></td>
    <td><b>67.71</b></td>
    <td><b>63.62</b></td>
    <td><b>51.77</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>16x Memory Context</td>
    <td>55.32</td>
    <td>37.72</td>
    <td>33.76</td>
    <td>25.27</td>
    <td>38.96</td>
    <td>26.47</td>
    <td>71.09</td>
    <td>61.97</td>
    <td>49.78</td>
    <td>37.86</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>128x Memory Context</td>
    <td>33.72</td>
    <td>18.86</td>
    <td>27.60</td>
    <td>20.44</td>
    <td>29.75</td>
    <td>21.13</td>
    <td>62.57</td>
    <td>53.30</td>
    <td>38.41</td>
    <td>28.43</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Performance comparison of various methods on the QA task across four datasets. NQ stands for Natural Questions dataset, EM indicates Exact Match. The best-performing results on each dataset are highlighted in bold, while the second-best results are underlined. To address potential inconsistencies in decoder configurations across baselines, we further evaluate our method using alternative decoders to ensure fairness. We provide additional detailed results in Table A2 in the Appendix.

The results show that our model can generalize to multiple segments well.

3.3 Results of Downstream Tasks

3.3.1 RAG-based QA

The first type of downstream task is retrieval-augmented generative question-answering (RAG-based QA). Each data sample consists of a triple: (context, question, answer). Reference (w/o Context) means prompting the target LLM only with the question, while Reference (w/ Context) means concatenating the context and question as the prompt. For all compression models, including both our PCC and baselines, they compress the context and then concatenate the compressed memory with the original question as the prompt. To demonstrate the generalization ability of our model, for PCC models, we fine-tune only using the training set of SQuAD and then evaluate on SQuAD, HotPotQA (Yang et al., 2018), AdversarialQA (Bartolo et al., 2020), and NQ (Karpukhin et al., 2020). For baselines, we use their released settings, so they may be fine-tuned with multiple training sets, such as xRAG.

We evaluate performance using F1 and Exact Match (EM) metrics, where F1 score denotes the harmonic mean of precision and recall, and EM evaluates whether the predicted response precisely aligns with the ground truth answer.

Table 2 reports the overall results. First, in most cases and in terms of average scores, PCC (large) performs best, followed by PCC (lite). This verifies the efficacy of our method. The only exception is on HotPotQA, where LLMLingua2 outperforms our models. However, our PCC (Large) 4x outperforms LLMLingua2 in all the other three datasets. Second, 4x compression preserves the information from the context well for both lite and large compressors, sacrificing only slightly in performance compared to the reference model with the full context as explicit input. With larger compression rates, the performance drops more significantly. However, even with a compression rate as high as 256x, PCC still outperforms Reference (w/ Context), indicating that the memory still captures useful information in the highly compressed memory slots. Furthermore, our models demonstrate strong generalization ability, performing well on the HotPotQA, AdversarialQA, and NQ datasets without fine-tuning on their training sets.

3.3.2 In-Context Learning

The challenge of RAG-based QA lies in learning to locate useful evidence from the augmenting context for answer generation. In contrast, the in-context learning (ICL) task requires the ability to infer latent patterns from demonstrations in the augmenting context and apply these patterns to the current task. This task is more challenging than direct evidence identification and thus warrants a separate
(a) Comparison across different compression rates (Lite)

(b) Comparison between Lite and Large compressors

Figure 3: The pre-training curves with various configurations in the text completion task.

testing scenario. We use three datasets for evaluation: GSM8K (mathematical reasoning), SST-2 (sentiment analysis) (Socher et al., 2013), and WSC (Winograd Schema Challenge) (Levesque et al., 2012). PCC (Large) is fine-tuned with the training set of GSM8K. We compare zero-shot generation, few-shot ICL with a maximum context of 150 tokens and 750 tokens, and PCC (Large) with 4x and 16x compressed in-context examples. Results are reported in Table 3. PCC performs very well in the ICL scenario, with a 4x compression rate performing on par with or even better than explicit ICL with 750 tokens.

<table>
  <tr>
    <th>Methods</th>
    <th>GSM8K</th>
    <th>SST-2</th>
    <th>WSC</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>64.82</td>
    <td>87.04</td>
    <td>53.85</td>
  </tr>
  <tr>
    <td>ICL 150 Tokens</td>
    <td>71.72</td>
    <td>93.92</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>ICL 750 Tokens</td>
    <td><b>78.92</b></td>
    <td>94.19</td>
    <td>45.83</td>
  </tr>
  <tr>
    <td>4x Comp. 750 Tokens</td>
    <td>74.91</td>
    <td><b>94.61</b></td>
    <td><u>69.55</u></td>
  </tr>
  <tr>
    <td>4x Comp. 1500 Tokens</td>
    <td>63.76</td>
    <td>93.50</td>
    <td>64.42</td>
  </tr>
  <tr>
    <td>16x Comp. 750 Tokens</td>
    <td>71.65</td>
    <td>93.16</td>
    <td>62.18</td>
  </tr>
  <tr>
    <td>16x Comp. 1500 Tokens</td>
    <td>67.48</td>
    <td>89.99</td>
    <td>61.54</td>
  </tr>
</table>

Table 3: Comparison of performance in three ICL datasets. Accuracy (%) is employed as the metrics. Bold values indicate the highest and underlined values denote the second-highest accuracy. The compressor employed here is PCC (Large).

<table>
  <tr>
    <th>Category</th>
    <th>Methods</th>
    <th>PPL</th>
  </tr>
  <tr>
    <td rowspan="2">Reference</td>
    <td>Zero-shot</td>
    <td>36.45</td>
  </tr>
  <tr>
    <td>750 tokens context</td>
    <td>27.74</td>
  </tr>
  <tr>
    <td></td>
    <td>1500 tokens context</td>
    <td>26.08</td>
  </tr>
  <tr>
    <td rowspan="2">SFT</td>
    <td>Zero-shot</td>
    <td>24.39</td>
  </tr>
  <tr>
    <td>750 tokens context</td>
    <td><b>17.38</b></td>
  </tr>
  <tr>
    <td></td>
    <td>1500 tokens context</td>
    <td>19.18</td>
  </tr>
  <tr>
    <td rowspan="2">PCC</td>
    <td>4x 750 tokens context</td>
    <td>19.57</td>
  </tr>
  <tr>
    <td>4x 1500 tokens context</td>
    <td>20.31</td>
  </tr>
  <tr>
    <td rowspan="2">(Lite)</td>
    <td>16x 750 tokens context</td>
    <td>32.46</td>
  </tr>
  <tr>
    <td>16x 1500 tokens context</td>
    <td>30.49</td>
  </tr>
</table>

Table 4: Perplexity (PPL) in the role-play task. Lower PPL indicates better performance in generating the target character’s responses.

ideal playground for memory representation. To evaluate the effectiveness of context compression in the role-playing scenario, we use the Harry Potter Dialogue dataset. PCC (Lite) is fine-tuned on a hold-out training set and trained with the compressed 6-shot examples, while SFT (Supervised Fine-Tuning) is fine-tuned on the same number of examples but without compressing the 6-shot examples. A lower perplexity (PPL) indicates better capture of the character’s target utterance by the model. The results, presented in Table 4, shows that PCC demonstrates excellent efficacy in this scenario. With a 4x compression rate, it outperforms the explicit prompt-based role-playing strategy, with only a slight performance degradation compared to the SFT method. At 16x compression, its performance lies between zero-shot role-playing and few-shot role-playing.

3.4 Connecting Various LLMs

To assess whether the pre-trained compressor can enhance the performance of various LLMs, we con-
Figure 4: Ablation study: pre-training curves for PCC (Large) with 16x compression rate when some components are removed during the pre-training stage. And Origin represents original LLM.

<table>
  <tr>
    <th>Model</th>
    <th>Methods</th>
    <th>F1</th>
    <th>EM</th>
  </tr>
  <tr>
    <td rowspan="5">Mistral-7B</td>
    <td>w/o Context</td>
    <td>11.33</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>w Context</td>
    <td>49.02</td>
    <td>23.89</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>46.57</td>
    <td>35.15</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>35.29</td>
    <td>20.55</td>
  </tr>
  <tr>
    <td>w/o Context</td>
    <td>25.91</td>
    <td>16.48</td>
  </tr>
  <tr>
    <td rowspan="5">Qwen2.5-7B</td>
    <td>w Context</td>
    <td>66.51</td>
    <td>49.95</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>46.39</td>
    <td>32.77</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>31.52</td>
    <td>16.90</td>
  </tr>
  <tr>
    <td>w/o Context</td>
    <td>11.45</td>
    <td>2.24</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini</td>
    <td>w Context</td>
    <td>60.53</td>
    <td>40.66</td>
  </tr>
  <tr>
    <td rowspan="5">Phi-3.5-mini</td>
    <td>4x Comp.</td>
    <td>48.01</td>
    <td>34.97</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>39.30</td>
    <td>27.30</td>
  </tr>
  <tr>
    <td>w/o Context</td>
    <td>19.79</td>
    <td>9.94</td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>w Context</td>
    <td>66.29</td>
    <td>50.29</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>61.77</td>
    <td>49.86</td>
  </tr>
  <tr>
    <td></td>
    <td>16x Comp.</td>
    <td>49.00</td>
    <td>36.60</td>
  </tr>
</table>

Table 5: Performance of four different downstream LLMs. The metrics employed are the average F1 score and Exact Match (EM) on four datasets: SQuAD, HotPotQA, AdversarialQA, and Natural Questions (NQ). The compressor employed here is PCC (Lite).

<table>
  <tr>
    <th>Methods</th>
    <th>F1</th>
    <th>EM</th>
  </tr>
  <tr>
    <td>Reference (w/o Context)</td>
    <td>19.79</td>
    <td>9.94</td>
  </tr>
  <tr>
    <td><b>PCC (Large) 16x</b></td>
    <td><b>49.78</b></td>
    <td><b>37.86</b></td>
  </tr>
  <tr>
    <td>w/o Pretrain Stage</td>
    <td>23.51</td>
    <td>15.96</td>
  </tr>
  <tr>
    <td>w/o Text Reconstruction Task</td>
    <td>47.65</td>
    <td>36.74</td>
  </tr>
  <tr>
    <td>w/o Text Completion Task</td>
    <td>46.79</td>
    <td>35.65</td>
  </tr>
  <tr>
    <td>w/o Mutil Special Tokens</td>
    <td>47.87</td>
    <td>36.36</td>
  </tr>
</table>

Table 6: Ablation study: Average performance of four datasets on the RAG-based QA domain. The metrics reported are in percentages (%).

sented in Table 5 (for the complete table, please refer to Table A2 in the Appendix). Across all downstream LLMs, the pre-trained compressor (PCC) demonstrates its effectiveness by providing valuable context from the compressed memory slots, showing significant improvement compared to the models’ performance without context.

3.5 Ablation Study

In addition to the compression rate and compressor model size, which have been extensively studied in previous experiments, other important components of PCC include the pre-training stage, specific pre-training tasks, and the memory slots wrapper (<MEM> </MEM>). To further investigate these components, we conduct an ablation study. For example, w/o PretrainStage indicates removing the entire pre-training stage and retaining only the fine-tuning stage with the SQuAD training set. Results are reported in Table 6 (for the full table, please refer to Table A3 in the Appendix). We find
that the pre-training stage is particularly crucial for generating high-quality memory slots. Removing any component from the PCC framework results in a noticeable performance drop, highlighting the importance of this comprehensive approach to pre-training and memory representation. We further demonstrate the changes in the pre-training loss curves when either the text reconstruction or the text completion tasks are removed in Figure 4. It is interesting to observe that when only the text completion task is used in pre-training (as shown in Figure 4a), the text reconstruction ability degrades significantly. Conversely, when only the text reconstruction task is used in pre-training (as shown in Figure 4b), the text completion task fails to converge. The synergy of these two tasks makes the pre-training of the compressor more robust.

4 Related Work

LLMs with Long Context. Perceiving long context in large language models (LLMs) has become ubiquitous with numerous real-world applications, such as RAG systems (Gao et al., 2023), information extraction from social platforms (Shang et al., 2024), agentic frameworks for task automation (Wang et al., 2024; Huang et al., 2023), and role-playing agents (Zhong et al., 2024; Liu et al., 2024), where LLMs need to remember extensive character profiles, experiences, and past conversations. Pretraining LLMs to be context-aware over long sequences is expensive, so researchers often develop context extension techniques to expand the context windows of well-pretrained LLMs (Chen et al., 2023b; Ding et al., 2024; Zhang et al., 2024b). Given the high computational and storage costs associated with long-sequence inference, researchers have explored various methods to reduce these expenses (Zhang et al., 2024a; Jiang et al., 2024; Fang et al., 2024; Munkhdalai et al., 2024). However, these methods typically require modifications or fine-tuning of the LLMs. In this paper, we examine methods that compress long contexts into shorter versions while keeping the LLMs frozen. Thus, our focus is on context compression.

Context Compression. There are primarily two approaches for context compression: explicit compression and implicit compression. In the explicit compression approach, researchers develop methods to identify and remove redundant and non-essential tokens in the context, retaining only the important and compact information (Li et al., 2023; Jiang et al., 2023; Pan et al., 2024). This approach has the advantage of explainability, as it is possible to see which tokens are removed from the original context. However, the utilization of context can be suboptimal since it is limited to explicit tokens. In the implicit compression approach, researchers compress the original context into implicit memory slots, leveraging the fact that embedding representations can effectively condense key information (Chevalier et al., 2023; Cheng et al., 2024; Ge et al., 2024; Li et al., 2024b; Rau et al., 2024). This paper aims to push the frontier of the second approach forward by addressing key questions in context compression from a systematic perspective, including pretraining, universal and decoupled module design, compressor size and rate, and efficacy across different types of downstream applications.

5 Conclusion

In this paper, we focus on efficient long contexts processing for LLMs using embedding-based compression. We propose a decoupled compressor-LLM framework and advocate for pre-training with two language modeling tasks. The resulting method, PCC, serves as a universal and lightweight compressor adaptable to various downstream LLM decoders. Through rigorous experiments, we demonstrate that compression rates of 4x and 16x strike a good balance between accuracy and efficiency, while higher rates like 256x capture useful context at the expense of some information loss. Notably, our method outperforms competitive context compression baselines across diverse downstream tasks. We hope this research highlights the potential of implicit context compression to enhance LLM efficiency and lays the groundwork for future innovations in scalable LLM inference.

6 Limitation

While our work demonstrates promising results in embedding-based compression for LLMs, we acknowledge several limitations that provide opportunities for future improvement.

• Investigating different configurations of compressor pre-training is both time and GPU consuming. For the current version, we can only afford up to 8B LLM’s pretraining experiments. While this paper advocates for a lightweight compressor, we hope in the next version, we can launch larger compressors such as 70B models to investigate
the upper bounds of implicit compression. This may break the dilemma of higher compression rates, like 256x, being highly efficient but losing too much useful information.

• The current design of PCC compresses segments of a long context into even memory slots. However, considering that different segments of text may convey different levels of information mass and redundancy, an even compression is not an optimal approach. In the future, we plan to investigate a dynamic implicit compression technique that can determine the compression rate for each segment adaptively.

• Although currently the adaptation of the compressor to various downstream LLMs only requires a few steps of fine-tuning, it still necessitates parameter updates on both the compressor encoder and compressor converter to achieve desirable performance. In the future, we hope to develop a solution where only the converter needs to be fine-tuned, allowing the compressor encoder to remain frozen, thereby making the adaptation process even more lightweight.

7 Acknowledgments

The authors from SZU acknowledge the financial support from the National Natural Science Foundation of China (Grant Nos. 62276171, 62476173, 62002233, 61972145), the Shenzhen Fundamental Research-General Project (Grant Nos. JCYJ20240813142610014, JCYJ20240813141503005, JCYJ2022081115580-3001), Guangdong Basic and Applied Basic Research Foundation (Grant Nos. 2024A1515011938 and 2020B1515120028), Guangdong Peral River Recruitment Program of Talents (2019ZT08X603). Hao Liao is the corresponding author.

References

Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678.

Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, and Jia Li. 2023a. Large language models meet harry potter: A dataset for aligning dialogue agents with characters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8506–8520.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307.

Xin Cheng, Xun Wang, Xinxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. arXiv preprint arXiv:2405.13792.

Alexis Chevalier, Alexander Wetting, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3829–3846. Association for Computational Linguistics.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Yiran Ding, Li Lyna Zhang, Chengrudong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753.

Junjie Fang, Likai Tang, Hongzhe Bi, Yujia Qin, Si Sun, Zhenyu Li, Haolun Li, Yongjian Li, Xin Cong, Yankai Lin, et al. 2024. Unimem: Towards a unified view of long-context large language models. arXiv preprint arXiv:2402.03009.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie. 2023. Recommender ai agent: Integrating large language models for interactive recommendations. arXiv preprint arXiv:2308.16505.
Huiqiang Jiang, Yucheng Li, Chengrudong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490.

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Llmlingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 13358–13376. Association for Computational Linguistics.

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.

Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning.

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhui Chen. 2024a. Long-context llms struggle with long in-context learning. Preprint, arXiv:2404.02060.

Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6342–6353. Association for Computational Linguistics.

Zongqian Li, Yixuan Su, and Nigel Collier. 2024b. 500xcompressor: Generalized prompt compression for large language models. arXiv preprint arXiv:2408.03094.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. Barcelona, Spain. Association for Computational Linguistics.

Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2024. Llms+ persona-plug= personalized llms. arXiv preprint arXiv:2409.11901.

Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. 2024. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143.

Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 963–981. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.

Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.

David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. 2024. Context embeddings for efficient answer generation in rag. arXiv preprint arXiv:2407.09252.

Tianqi Shang, Weiqing He, Tianlong Chen, Ying Ding, Huanmei Wu, Kaixiong Zhou, and Li Shen. 2024. Integrating social determinants of health into knowledge graphs: Evaluating prediction bias and fairness in healthcare. Preprint, arXiv:2412.00245.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.

Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. Retromae: Pre-training retrieval-oriented language models via masked auto-encoder. arXiv preprint arXiv:2205.12035.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32.

Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024a. Long context compression with activation beacon. arXiv preprint arXiv:2401.03462.

Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. 2024b. Extending llama-3’s context ten-fold overnight. arXiv preprint arXiv:2404.19553.

Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731.
A Appendix

A.1 Pipline Implementation Details

Table A4 presents the hyperparameters utilized for training both the lightweight and large compressor models. Both models are optimized using the AdamW algorithm in conjunction with a cosine learning rate scheduler. The learning rate is adapted for different training phases, including pretraining and fine-tuning on the SQuAD and GSM8K datasets. Batch sizes and warmup steps are specifically configured for each model and task, with a maximum gradient norm of 0.5 consistently applied. For the lightweight compressor, all parameters are trained directly, whereas the large compressor employs LoRA with a rank of 64, a scaling factor of \( \alpha = 32 \), and a dropout rate of 0.1. Training is conducted on 4xA100 and 8xH100 GPUs, each equipped with 80 GB of memory, ensuring efficient computation for both model types. The lightweight compressor completes one epoch during pretraining and three epochs during fine-tuning, while the large compressor adheres to the same schedule for fine-tuning tasks.

A.2 Perplexity Evaluation

The effect of memory embedding on the perplexity of the target LLM is evaluated by calculating the perplexity of the second half of the input tokens. In the Plain method, as described in Reference, the first half of the input tokens is discarded, whereas the Context method retains the complete input sequence. Conversely, the PCC method compresses the first half of the input tokens into memory embeddings. Detailed experimental results are provided in Table A1.

A.3 Inference Efficiency

We evaluate the enhancement in inference efficiency of the target LLM using the lightweight compressor across various input combinations. Given that our method allows for the pre-caching of context memory embeddings, we also investigate the impact of caching these embeddings on the improvement of inference efficiency. The experimental results are presented in Table A7.

The machine configuration used for testing latency. The hardware setup included an AMD EPYC 7V13 64-Core Processor for the CPU, an A100 80GB PCIe GPU for computation, and 216GB of RAM for memory-intensive operations.

The version of software are Python 3.11, PyTorch 2.7.0, Transformers 4.41.2 and CUDA 12.6.

A.4 Case Study

The performance of the model on the text reconstruction task is evaluated by compressing a 256-token text sequence with compression rates of 4x, 16x, and 64x, respectively. The original text and the reconstructed text are presented in Table A5. In the 4x and 16x compression scenarios, the reconstructed text closely aligned with the original, achieving BLEU scores of 100% and 98.8%, respectively, demonstrating the model’s high fidelity in text restoration at low and moderate compression rates. However, as the compression rate increased to 64x, performance deteriorated significantly, with the BLEU score dropping to 27.8%.

Samples from the NQ dataset are selected for testing to evaluate performance. Unlike COCOM, which is fine-tuned on the target LLM, our proposed method is not fine-tuned on the target model. Despite this, our method outperforms COCOM in the evaluation. However, it can be observed that as the compression rate increases, the ability of the memory embedding to effectively extract and retain useful information gradually diminish.
<table>
  <tr>
    <th rowspan="2">Reference</th>
    <th colspan="5">Input Tokens</th>
  </tr>
  <tr>
    <th>512</th>
    <th>1024</th>
    <th>2048</th>
    <th>4096</th>
  </tr>
  <tr>
    <td>w/o Context</td>
    <td>17.31</td>
    <td>14.70</td>
    <td>13.07</td>
    <td>11.99</td>
  </tr>
  <tr>
    <td>w/ Context</td>
    <td>12.38</td>
    <td>11.57</td>
    <td>11.00</td>
    <td>10.57</td>
  </tr>
  <tr>
    <td rowspan="5">PCC (Lite)</td>
    <td>4x Comp.</td>
    <td>11.22</td>
    <td>10.97</td>
    <td>10.82</td>
    <td>11.30</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>12.14</td>
    <td>11.78</td>
    <td>11.48</td>
    <td>11.78</td>
  </tr>
  <tr>
    <td>64x Comp.</td>
    <td>13.05</td>
    <td>12.46</td>
    <td>12.42</td>
    <td>13.02</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>13.13</td>
    <td>12.42</td>
    <td>11.88</td>
    <td>11.57</td>
  </tr>
  <tr>
    <td>256x Comp.</td>
    <td>13.24</td>
    <td>13.15</td>
    <td>12.69</td>
    <td>12.13</td>
  </tr>
  <tr>
    <td rowspan="3">PCC (Large)</td>
    <td>4x Comp.</td>
    <td><b>10.66</b></td>
    <td><b>10.82</b></td>
    <td><b>10.79</b></td>
    <td><b>11.10</b></td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>11.84</td>
    <td>11.64</td>
    <td>11.28</td>
    <td>11.23</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>13.01</td>
    <td>12.51</td>
    <td>12.26</td>
    <td>12.28</td>
  </tr>
</table>

Table A1: Perplexity (PPL) trends across different context lengths and memory slot configurations for the pre-training text completion task. Each context length is evenly divided, with the first half used to generate memory slots and the second half used for PPL computation.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">SQuAD</th>
    <th colspan="2">HotPotQA</th>
    <th colspan="2">AdversarialQA</th>
    <th colspan="2">NQ</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
  </tr>
  <tr>
    <td rowspan="4">Mistral</td>
    <td>w/o Context</td>
    <td>10.34</td>
    <td>0.69</td>
    <td>8.44</td>
    <td>0.13</td>
    <td>7.80</td>
    <td>0.83</td>
    <td>18.75</td>
    <td>3.36</td>
    <td>11.33</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>w Context</td>
    <td>56.30</td>
    <td>27.33</td>
    <td>52.32</td>
    <td>30.97</td>
    <td>36.96</td>
    <td>12.60</td>
    <td>50.51</td>
    <td>24.64</td>
    <td>49.02</td>
    <td>23.89</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>62.40</td>
    <td>44.12</td>
    <td>31.97</td>
    <td>24.52</td>
    <td>42.51</td>
    <td>30.70</td>
    <td>49.41</td>
    <td>41.28</td>
    <td>46.57</td>
    <td>35.15</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>45.89</td>
    <td>25.61</td>
    <td>23.91</td>
    <td>17.92</td>
    <td>30.23</td>
    <td>19.02</td>
    <td>41.12</td>
    <td>19.63</td>
    <td>35.29</td>
    <td>20.55</td>
  </tr>
  <tr>
    <td rowspan="4">Qwen</td>
    <td>w/o Context</td>
    <td>21.03</td>
    <td>7.09</td>
    <td>30.66</td>
    <td>23.29</td>
    <td>16.55</td>
    <td>10.53</td>
    <td>35.41</td>
    <td>25.00</td>
    <td>25.91</td>
    <td>16.48</td>
  </tr>
  <tr>
    <td>w Context</td>
    <td>78.30</td>
    <td>57.44</td>
    <td>63.30</td>
    <td>51.73</td>
    <td>55.10</td>
    <td>36.17</td>
    <td>69.33</td>
    <td>54.47</td>
    <td>66.51</td>
    <td>49.95</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>56.79</td>
    <td>36.16</td>
    <td>38.09</td>
    <td>28.84</td>
    <td>37.55</td>
    <td>25.70</td>
    <td>53.14</td>
    <td>40.40</td>
    <td>46.39</td>
    <td>32.77</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>34.01</td>
    <td>14.53</td>
    <td>30.98</td>
    <td>20.64</td>
    <td>25.01</td>
    <td>12.97</td>
    <td>36.06</td>
    <td>19.46</td>
    <td>31.52</td>
    <td>16.90</td>
  </tr>
  <tr>
    <td rowspan="4">Phi</td>
    <td>w/o Context</td>
    <td>11.49</td>
    <td>1.04</td>
    <td>14.11</td>
    <td>4.14</td>
    <td>8.64</td>
    <td>1.70</td>
    <td>11.55</td>
    <td>2.08</td>
    <td>11.45</td>
    <td>2.24</td>
  </tr>
  <tr>
    <td>w Context</td>
    <td>73.12</td>
    <td>46.71</td>
    <td>59.03</td>
    <td>44.31</td>
    <td>50.22</td>
    <td>29.27</td>
    <td>59.75</td>
    <td>42.34</td>
    <td>60.53</td>
    <td>40.66</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>57.63</td>
    <td>36.68</td>
    <td>36.43</td>
    <td>27.52</td>
    <td>41.63</td>
    <td>29.20</td>
    <td>56.33</td>
    <td>46.50</td>
    <td>48.01</td>
    <td>34.97</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>44.29</td>
    <td>26.47</td>
    <td>32.81</td>
    <td>24.74</td>
    <td>32.14</td>
    <td>20.33</td>
    <td>47.94</td>
    <td>37.65</td>
    <td>39.30</td>
    <td>27.30</td>
  </tr>
</table>

Table A2: Performance evaluation of three LLMs—Mistral 7B v0.2 Instruct, Qwen-2.5-instruct, and Phi-3.5-mini-instruct—on four widely-used question-answering datasets: SQuAD, HotPotQA, AdversarialQA, and NQ. F1 score (%) and Exact Match (EM, %) are utilized as evaluation metrics. Each model is assessed under three conditions: (1) Without Context, where no additional context is provided during evaluation; (2) With Context, where relevant context is included; and (3) 4x/16x Compression, which evaluates model performance after applying a 4x/16x compressor. The average scores across all datasets are presented in the last columns for a comprehensive comparison.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="4">SQuAD</th>
    <th colspan="4">HotPotQA</th>
    <th colspan="4">AdversarialQA</th>
    <th colspan="4">NQ</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>Metrics</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>Average</th>
    <th>F1</th>
    <th>EM</th>
  </tr>
  <tr>
    <td rowspan="2"><b>Reference</b></td>
    <td>w/o Context</td>
    <td>14.84</td>
    <td>3.41</td>
    <td>24.80</td>
    <td>14.90</td>
    <td>12.05</td>
    <td>6.43</td>
    <td>27.48</td>
    <td>15.02</td>
    <td>19.79</td>
    <td>9.94</td>
  </tr>
  <tr>
    <td>w/ Context</td>
    <td><b>79.81</b></td>
    <td><b>59.97</b></td>
    <td><b>64.60</b></td>
    <td><b>51.12</b></td>
    <td><b>56.10</b></td>
    <td><b>38.67</b></td>
    <td><b>64.64</b></td>
    <td><b>51.38</b></td>
    <td><b>66.29</b></td>
    <td><b>50.29</b></td>
  </tr>
  <tr>
    <td rowspan="6"><b>Baselines</b></td>
    <td>AutoCompressor</td>
    <td>21.46</td>
    <td>0.35</td>
    <td>16.29</td>
    <td>0.29</td>
    <td>14.09</td>
    <td>2.00</td>
    <td>25.57</td>
    <td>0.63</td>
    <td>19.35</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>xRAG</td>
    <td>18.19</td>
    <td>3.46</td>
    <td>27.51</td>
    <td>16.29</td>
    <td>13.75</td>
    <td>3.47</td>
    <td>38.06</td>
    <td>20.80</td>
    <td>24.38</td>
    <td>11.01</td>
  </tr>
  <tr>
    <td>4x COCOM-Lite</td>
    <td>21.70</td>
    <td>9.17</td>
    <td>40.07</td>
    <td>32.32</td>
    <td>19.45</td>
    <td>13.90</td>
    <td>50.45</td>
    <td>41.87</td>
    <td>32.92</td>
    <td>24.32</td>
  </tr>
  <tr>
    <td>16x COCOM-Lite</td>
    <td>19.23</td>
    <td>8.13</td>
    <td>31.94</td>
    <td>25.27</td>
    <td>19.35</td>
    <td>14.73</td>
    <td>26.36</td>
    <td>20.66</td>
    <td>24.22</td>
    <td>17.20</td>
  </tr>
  <tr>
    <td>128x COCOM-Lite</td>
    <td>19.56</td>
    <td>7.61</td>
    <td>23.63</td>
    <td>18.68</td>
    <td>19.47</td>
    <td>15.13</td>
    <td>18.79</td>
    <td>14.36</td>
    <td>20.36</td>
    <td>13.95</td>
  </tr>
  <tr>
    <td>ICAE</td>
    <td>45.69</td>
    <td>21.63</td>
    <td>35.16</td>
    <td>26.68</td>
    <td>27.98</td>
    <td>11.70</td>
    <td>59.15</td>
    <td>47.35</td>
    <td>42.00</td>
    <td>26.84</td>
  </tr>
  <tr>
    <td>LLMLingua2</td>
    <td>51.20</td>
    <td>32.18</td>
    <td><b>55.72</b></td>
    <td><b>44.18</b></td>
    <td>35.41</td>
    <td>24.80</td>
    <td>68.44</td>
    <td>55.85</td>
    <td>52.69</td>
    <td>39.25</td>
  </tr>
  <tr>
    <td rowspan="5"><b>Ablation</b></td>
    <td>16x Memory Context</td>
    <td><b>55.32</b></td>
    <td><b>37.72</b></td>
    <td>33.76</td>
    <td>25.27</td>
    <td><b>38.96</b></td>
    <td>26.47</td>
    <td><b>71.09</b></td>
    <td>61.97</td>
    <td><b>49.78</b></td>
    <td><b>37.86</b></td>
  </tr>
  <tr>
    <td>w/o Pretrain Stage</td>
    <td>19.80</td>
    <td>8.13</td>
    <td>25.21</td>
    <td>19.10</td>
    <td>15.93</td>
    <td>11.00</td>
    <td>33.10</td>
    <td>25.61</td>
    <td>23.51</td>
    <td>15.96</td>
  </tr>
  <tr>
    <td>w/o Text Completion Task</td>
    <td>47.76</td>
    <td>29.24</td>
    <td>33.54</td>
    <td>25.72</td>
    <td>35.71</td>
    <td>25.33</td>
    <td>70.15</td>
    <td>62.29</td>
    <td>46.79</td>
    <td>35.65</td>
  </tr>
  <tr>
    <td>w/o Text Reconstruction Task</td>
    <td>44.86</td>
    <td>28.55</td>
    <td>36.34</td>
    <td>28.39</td>
    <td>38.53</td>
    <td><b>27.53</b></td>
    <td>70.88</td>
    <td><b>62.47</b></td>
    <td>47.65</td>
    <td>36.74</td>
  </tr>
  <tr>
    <td>w/o Mutil Special Tokens</td>
    <td>51.92</td>
    <td>33.39</td>
    <td>31.25</td>
    <td>23.71</td>
    <td>38.27</td>
    <td><b>27.17</b></td>
    <td>70.05</td>
    <td>61.18</td>
    <td><b>47.87</b></td>
    <td>36.36</td>
  </tr>
</table>

Table A3: Detailed results of the 16x compression configuration using the large compressor in the ablation study for QA tasks. The table presents the performance (F1 and Exact Match (EM)) across four datasets: SQuAD, HotPotQA, AdversarialQA, and NQ, along with the average metrics. Methods compared include baseline models, ablation variations (e.g., removal of pretraining, text completion, text reconstruction tasks, and multi-special tokens), and the full model configurations. Best-performing results for each dataset are highlighted in bold, and second-best results are underlined.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>PCC (Lite)</th>
    <th>PCC (Large)</th>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e-4 (Pre-train)<br>5e-5 (SQuAD Fine-tune)<br>1e-5 (HPD Fine-tune)</td>
    <td>1e-4 (Pre-train)<br>5e-5 (SQuAD Fine-tune)<br>1e-5 (GSM8K Fine-tune)</td>
  </tr>
  <tr>
    <td>Lr scheduler</td>
    <td>Cosine</td>
    <td>Cosine</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>256 (Pre-train)<br>128 (SQuAD Fine-tune)<br>8 (HPD Fine-tune)</td>
    <td>256 (Pre-train)<br>128 (SQuAD Fine-tune)<br>32 (GSM8K Fine-tune)</td>
  </tr>
  <tr>
    <td>Warmup steps</td>
    <td>300 (Pre-train)<br>100 (Fine-tune)</td>
    <td>300 (Pre-train)<br>100 (Fine-tune)</td>
  </tr>
  <tr>
    <td>Max grad norm</td>
    <td>0.5</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1 (Pre-train)<br>3 (Fine-tune)</td>
    <td>1 (Pre-train)<br>3 (Fine-tune)</td>
  </tr>
  <tr>
    <td>Training method</td>
    <td>Full parameters</td>
    <td>LoRA<br>(\( r = 64, \alpha = 32, \) dropout=0.1)</td>
  </tr>
  <tr>
    <td>Pre-training Time</td>
    <td>123 Hours(4 × A100 80GB)<br>_</td>
    <td>204 Hours(4 × A100 80GB)<br>67 Hours(8 × H100 80GB)</td>
  </tr>
</table>

Table A4: Hyperparameters for training the light-weight and large compressor models.
Origin
the numbers used in the formula on your IRS Form 1040. Here’s where each input you’ll need is located: -Total tax: Line 24 -Taxable income: Line 15 Worth noting, income refers to both earned and unearned income: -Earned income: Wages, commissions, salary, and bonuses -Unearned income: Interest income on saving accounts, dividends from stocks, and bond interest As an example, consider an individual who paid $24,000 in total tax and had a taxable income of $95,000 in salary, a $5,000 bonus, and $1,000 in income from savings interest and dividends. This person’s total taxable income would be $95,000 + $5,000 + $1,000 = $101,000. Individual effective tax rate = 24,000 / 101,000 = 0.2376 You can then express this as a percentage by multiplying it by 100 to arrive at 23.76%.

16x
the numbers used in the formula on your IRS Form 1040. Here’s where each input you’ll need is located: -Total tax: Line 24 -Taxable income: Line 15 Worth noting, income refers to both earned and unearned income: -Earned income: Wages, commissions, salary, and bonuses -Unearned income: Interest income on saving accounts, dividends from stocks, and bond interest As an example, consider an individual who paid $24,000 in total tax and had a taxable income of $95,000 in salary, a $5,000 bonus, and $1,000 in income from savings interest and dividends. This person’s total taxable income would be $95,000 + $5,000 + $1,000 = $101,000 |Individual effective tax rate = 24,000 / 101,000 = 0.2376 You can then express this as a percentage by multiplying it by 100 to arrive at 23.76%. (Almost Exactly recover.)

4x
the numbers used in the formula on your IRS Form 1040. Here’s where each input you’ll need is located: -Total tax: Line 24 -Taxable income: Line 15 Worth noting, income refers to both earned and unearned income: -Earned income: Wages, commissions, salary, and bonuses -Unearned income: Interest income on saving accounts, dividends from stocks, and bond interest As an example, consider an individual who paid $24,000 in total tax and had a taxable income of $95,000 in salary, a $5,000 bonus, and $1,000 in income from savings interest and dividends. This person’s total taxable income would be $95,000 + $5,000 + $1,000 = $101,000. Individual effective tax rate = 24,000 / 101,000 = 0.2376 You can then express this as a percentage by multiplying it by 100 to arrive at 23.76%. (Exactly recover.)

64x
the numbers you’ll use on the IRS Form 1040. Here’s where each line is located: - Total income: Taxable income. - Worthy of note: Taxable income includes both earned and unearned income. Taxable income includes: - Wages; Earned income from salaries, commissions, and bonuses on an hourly basis. - Unearned income: Interest, dividends, and capital gains from saving and investing in stocks, bonds, and a total return account. As an example, consider a person who had $25,000 in taxable income and paid $5,000 in taxes on an income of $100,000 in salary and interest. Total person’s taxable income would be $100,000 + $25,000 = $125,000. This person’s effective tax rate would be 100,000 / 125,000 = 80%. Individuals can then express this as a percentage by multiplying 80% by 100 to get 80. Effective tax rate = 80%

Table A5: Case study: Reconstructed text using different compression rates (4, 16, and 64). The compressor employed here is PCC (Lite). And Origin represents original LLM. Dataset: FineWeb. Highlighted words indicate inconsistencies between the restored text and the original text.
Model input

Question: When did the hawks win the NBA championship?

Passage 1: Atlanta Hawks
the NBA as part of the merger between the NBL and the Basketball Association of America (BAA), and briefly had Red Auerbach as coach. In 1951, Kerner moved the team to Milwaukee, where they changed their name to the Hawks. Kerner and the team moved again in 1955 to St. Louis, where they won their only NBA Championship in 1958 and qualified to play in the NBA Finals in 1957, 1960 and 1961. The Hawks played the Boston Celtics in all four of their trips to the NBA Finals. The St. Louis Hawks moved to Atlanta in 1968, when Kerner

Passage 2: 1958 NBA Finals
The 1958 NBA World Championship Series was the championship series for the 1957–58 National Basketball Association (NBA) season, and the conclusion of the season’s playoffs. It pitted the Western Division champion St. Louis Hawks against the Eastern Division champion Boston Celtics. The Hawks won the series in six games to win the club’s first and so far only NBA championship title. "Hawks win series 4–2" After suffering a heartbreaking loss to the Celtics in Game 7 of the 1957 NBA Finals, St. Louis survived a sometimes difficult 1957-58 NBA season, returning to the NBA Finals to face

Passage 3: 1971 NBA Finals
The 1971 NBA World Championship Series was the championship series played at the conclusion of the National Basketball Association (NBA)’s 25th anniversary season of 1970–71. The Western Conference champion Milwaukee Bucks, who were founded just three years earlier, swept the Eastern Conference champion Baltimore Bullets in four games. Baltimore had dethroned the 1969–70 NBA champion New York Knicks. The Bucks were the first Western Conference champions to win the league’s finals since the St. Louis Hawks did so in 1958. This was the first NBA Finals not played in the state of California in 10 years. It

Generated Response

Label: 1958

LLM w/o passages: The Atlanta Hawks have never won an NBA championship.

LLM w/ passages: 1958

PCC Lite 4x Comp: 1958
PCC Lite 16x Comp: 1958
PCC Lite 128x Comp: 1958
PCC Lite 256x Comp: 1957
COCOM 4x Comp: 1958
COCOM 16x Comp: 2006
COCOM 128x Comp: The Nba is a professional wrestling championship

Table A6: Case Study: Generated responses using different methods. The LLM used is Llama-3-8B-Instruct, and the compressors applied are the lightweight versions of PCC and COCOM. Dataset: NQ.
<table>
  <tr>
    <th>Batch×Input</th>
    <th>Model</th>
    <th>Compression Time</th>
    <th>Prefilling Time</th>
    <th>Decoding Time</th>
    <th>Total Time</th>
  </tr>
  <tr>
    <td rowspan="4"><b>16×1024</b></td>
    <td>Origin</td>
    <td>-</td>
    <td>1378.79</td>
    <td>1437.60</td>
    <td>2816.39</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>604.71</td>
    <td>359.20</td>
    <td>956.97</td>
    <td>1920.88</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>509.83</td>
    <td>100.95</td>
    <td>1090.76</td>
    <td>1701.54</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>500.88</td>
    <td>35.87</td>
    <td>1051.63</td>
    <td>1588.38</td>
  </tr>
  <tr>
    <td rowspan="4"><b>32×1024</b></td>
    <td>Origin</td>
    <td>-</td>
    <td>2774.99</td>
    <td>2372.02</td>
    <td>5147.01</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>1189.60</td>
    <td>709.19</td>
    <td>1087.64</td>
    <td>2986.43</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>998.91</td>
    <td>188.27</td>
    <td>1047.38</td>
    <td>2234.56</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>975.19</td>
    <td>37.49</td>
    <td>1057.84</td>
    <td>2070.52</td>
  </tr>
  <tr>
    <td rowspan="4"><b>16×2048</b></td>
    <td>Origin</td>
    <td>-</td>
    <td>2830.40</td>
    <td>2311.84</td>
    <td>5142.24</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>1217.97</td>
    <td>718.31</td>
    <td>1037.90</td>
    <td>2974.18</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>1025.82</td>
    <td>188.61</td>
    <td>1069.69</td>
    <td>2284.12</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>1000.76</td>
    <td>37.30</td>
    <td>1059.60</td>
    <td>2097.66</td>
  </tr>
  <tr>
    <td rowspan="4"><b>8×4096</b></td>
    <td>Origin</td>
    <td>-</td>
    <td>2910.01</td>
    <td>2320.38</td>
    <td>5230.39</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>1257.87</td>
    <td>720.62</td>
    <td>1062.48</td>
    <td>3040.97</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>1076.08</td>
    <td>189.72</td>
    <td>1068.69</td>
    <td>2334.49</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>1047.36</td>
    <td>37.13</td>
    <td>1082.19</td>
    <td>2166.68</td>
  </tr>
  <tr>
    <td rowspan="4"><b>4×8192</b></td>
    <td>Origin</td>
    <td>-</td>
    <td>3091.50</td>
    <td>2145.15</td>
    <td>5236.65</td>
  </tr>
  <tr>
    <td>4x Comp.</td>
    <td>1343.40</td>
    <td>730.81</td>
    <td>1084.96</td>
    <td>3159.17</td>
  </tr>
  <tr>
    <td>16x Comp.</td>
    <td>1145.68</td>
    <td>187.20</td>
    <td>1032.63</td>
    <td>2365.51</td>
  </tr>
  <tr>
    <td>128x Comp.</td>
    <td>1114.67</td>
    <td>36.75</td>
    <td>1058.20</td>
    <td>2209.62</td>
  </tr>
</table>

Table A7: Inference speed test experiment for lightweight compressor. The unit used in the table is milliseconds. The decoding time is the time it takes to generate 32 tokens. The value on the left side of the bracket represents the acceleration factor of uncached memory embedding, and the value on the right side represents the acceleration factor of cached memory embedding. And Origin represents the original LLM.

![Line chart comparing GPU memory usage between Origin, 4x, 16x, and 128x compression methods across different input sequences](page_1012_1342_420_312.png)

Figure A1: We compare the memory usage during inference between the original LLM method and the PCC method. The original LLM method, represented by Origin, runs out of memory (OOM) when the input text has a batch size of 8 and a length of 8192 tokens, exceeding 80 GB of GPU memory. In contrast, the PCC method maintains a much lower memory footprint under the same conditions.

Reviewer paper summaries: This paper studies about embedding-based compression of long context in an implicit manner i.e use embeddings as context.  
They explore two variants of encoders - 0.8B and 8B over multiple compression rates - 4x to 256x and 2 tasks: text reconstruction and completion.

The paper proposes a lightweight, decoupled context compressor framework, pre-trained on text reconstruction and completion tasks, to efficiently process long contexts in large language models (LLMs) while preserving performance across diverse downstream tasks and achieving significant computational cost reductions.

The paper proposes training a context compressor to efficiently encode longer contexts for LLM prompting. They empirically study the impact of various design choices and the limits along different parameters in using this method by evaluating on three different downstream tasks including RAG-based QA and in-context learning. Their results show almost no-loss from compression up to about 16x times while there are evident quality trade-off for higher compression rates. Effect of other factos like strength of base model, etc are also explored.
Summary of strengths: 1. Well motivated, high relevance for RAG based applications.
2. Decoupling of compressor is interesting as it can be used as plug-and-play for other models. 
3. Well written and is easy to follow at most places.

1. The paper proposes a decoupled compressor-LLM framework to efficiently process long contexts in large language models (LLMs) without altering their architecture.
2. The approach demonstrates strong adaptability across various LLMs and tasks, achieving optimal accuracy and computational efficiency at compression rates of 4x and 16x.
3. The authors commit to releasing pre-trained checkpoints and source code, emphasizing the potential of embedding-based context compression for scalable LLM applications.

1. The work explores a very relevant direction given the importance of long context for LLM applications. 
2. Experiments exploring the effect of compression, impact of design choices and limitations, and ablation studies, are well studied.
Summary of weaknesses: - Table 4 would be better if 4x is also reported as that will show w/ context and your method.
- Figure A1 shows that the lines with 'w/o' is better than origin for both the plots. I personally didn't understand if that's the case.
- L405-409: Does the authors have an idea or intuition on why LLMLingua2 is better than your method?
- Regarding baselines: The models in baselines have Mistral as the decoder but your method uses Llama in Table 1. A more apt comparison would be to bring the values from Mistral in Table A3 to the main table so the comparison will be for same decoders.
    - For eg: On SQuAD, PCC (Large) 16x for Mistral LLMLingua2 has an F1 of 45.89 while LLMLingua2 has 51.20 and ICAE has comparable (45.69).
    - Following the previous point, if there's a pre-training involved, the comparison should highlight that as well (models with pre-training vs w/o pre-training) to distinguish clearly.

I still think this method is too complex for being useful in reality.

1. There is no evaluation of latency or other processing gains from switching to this method over using the vanilla full-context or other compression alternative. This would be necessary to understand the trade-offs because it seems that even at the lowest compression rate of 4x there will be a likely dip in metrics (as read from Table 1). 
1. It would also be helpful to understand impact of compression on tasks that requiring generating long text similar to text completion form experiments but also summarisation or chat but when the output is fairly long (to the order of thousand words) since the impact loss in context compression could be increasingly visible in such scenarios.
Comments, suggestions and typos: N/A

1. "We then take the hidden representations of these memory tokens from the final transformer layer as the encoded memory slots." Why do not we consider all tokens include inputs?
2. Please replot "training" and "inference" of figure 1. It is now unclear what is the relationship between memory embedding and text reconstruction and text competition. I believe that the input is at the bottom of LLM and the output is at the top of LLM. Please make it more clear.
3. Are both encoder and converter trained in the pretraining and fine-tuning stage? Why do not pretrained on encoder and fine-tuned on converter?

The draft reads fine generally but could benefit from multiple passes of proof-reading to simplify language in some parts.


Response to rebuttal: I acknowledge the authors' comments in the rebuttal and thanks for pointing me to relevant results in the appendix that I missed in earlier reading. 
1. The latency gains are indeed good that would be an helpful indicator if space can be managed for it in the main draft.
2. Regarding the long-text generation, I meant more of measuring task-relevant metrics like ROUGE for summarisation etc that are more indicative of actual quality that perplexity does not do a good job revealing. This might a good test to evaluate how far a compression rate good stretch given the full models themselves struggle significantly. But this is not a common test in other compression works and also long-text evaluation itself is a challenge, so I don't particularly hold it against this work as a weakness.

I retain my overall positive score while improving on other parameters and think this is a good contribution for the community.
Overall assessment scores: 3.5

3

3.5