Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs (2024)

Abstract: The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closedbook level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput1.

heading: 1 Introduction
text: The emergence of chatGPT (Ouyang et al., 2022) and GPT4 (OpenAI, 2023), along with other Large Language Models (LLMs) (Touvron et al., 2023a,b) has sparked a global sensation. The success of LLMs is closely tied to the long context capabilities of LLMs (Dong et al., 2022; Lewis et al., 2020), especially in the field of multi-document question answering. However, the utilization of long context also introduces challenges such as higher inference cost, longer latency, and inferior perfor-
*These authors contributed equally. This work was done when Zhiwei Cao was interning at ByteDance.
†Corresponding author. 1Our code is available at https://github.com/
DeepLearnXMU/QGC.
mance caused by redundant information (Jiang et al., 2023).
Many efforts have been made to compress the long context by directly removing a certain percentage of less important words, such as LongLLMLingua (Jiang et al., 2023) and Selective-Context (Li et al., 2023). Another common method is to generate a text summary of the given context (Xu et al., 2023; Wang et al., 2023b). Unlike deleting or reordering the word in the context, AutoCompressor (Chevalier et al., 2023) compresses long documents into multiple vectors as soft prompts, which are optimized with full parameters of LLMs. However, our preliminary study shows that these methods have a common flaw: as the compression ratio increases, the compressed context fails to retain key information, resulting in a significant decrease in the performance of LLMs.
The key to solve this problem is query, which defines what key information is. We aim to preserve this query-related key information even at a high compression ratio. Specifically, we propose the Query-Guided Compressor (QGC) to fully utilize query information throughout each compression step. We first feed the query and the documents together into a context encoder to learn the queryguide document representations. We then compress these document representations into n-gram representations guided by the importance of each word in relation to the query. Subsequently, we propose to augment the n-gram representations by reviewing the query and document, which are finally aligned to the embedding space of the LLMs. We further propose dynamically adjusting the compression ratio of each document based on its relevance to the query. Compared to previous methods, QGC has several advantages: 1) high compression ratios by retaining most query-related information during compression, 2) low training costs by optimizing the compressor only instead of finetuning the entire LLM, and 3) better semantic consistency by com-
12685
pressing the n-gram structure rather than deleting words.
We validate the effectiveness of QGC on the multi-document Question Answering task, including three datasets: NaturalQuestions, TriviaQA, and HotpotQA. Experimental results on the QA task indicate that, compared to LongLLMLingua, QGC exhibits a 2.75 times higher compression ratio and a 2.42 times higher throughput. Additionally, its accuracy has improved by an average of 5 points. We further investigated the loss of key information throughout the compression process. The findings reveal that under high compression ratios and high noise conditions, QGC only incurs a performance loss of about 10%, while LongLLMLingua suffers a loss of approximately 47%. This validates the effectiveness of QGC in retaining key information.

heading: 2 Preliminary Study
text: In this section, we first briefly formulate the long context compression on the Question Answering task, and then present an analysis on the key information loss in previous compression methods.

heading: 2.1 Task Formulation
text: Given a LLM input with augmented context x = (xins,xd1 , ...,xdk , ...,xdK ,xq), which consists of the instruction xins, K documents {xdk}Kk=1, and the query xq, the objective of context compression can be formulated as:
min x̃
d(LLM(y|x),LLM(ỹ|x̃)), (1)
where y is the ground-truth answer and ỹ represents the output of the LLM with the compressed context x̃ as the input. d(·, ·) is a function measuring the distance between two distributions, such as KL divergence. In this work, we focus on compressing K retrieved documents that greatly determine the length of the input.

heading: 2.2 Key Information Loss in Compression
text: We study the effectiveness of two representative methods, LongLLMLingua (Jiang et al., 2023) and AutoCompressor (Chevalier et al., 2023). We conduct experiments on the NaturalQuestions dataset (Liu et al., 2023) and use accuracy as the evaluation metric, which judges whether any correct answers appear in the LLM prediction.
For LongLLMLingua, we apply LLaMA-2-7BChat2 as the small language model for compression, and use LongChat-13B-16K3 as the target LLM. We use the open-source AutoCompressor4, which fine-tunes LLaMA-2-7B to compress context and generate answers. Here, we consider four settings:
• Closed-book. It takes the query as the LLM input with no additional documents.
• Oracle. The query and only the document containing the ground truth are used as inputs to the LLM.
• Base. Based on Oracle, we compress the document directly with various compression ratios for LongLLMLingua. However, since AutoCompressor is set to compress documents to fixed length vectors, we change the compression ratio by adding external documents.
2https://ai.meta.com/llama/ 3https://huggingface.co/lmsys/longchat-13b-16k 4https://github.com/princeton-nlp/AutoCompressors
• Base w/ answer. We manually add key information to the compressed results by concatenating the answer with the compressed word sequence in LongLLMLingua. Note that this setting is impractical for AutoCompressor where the compressed results are vectors that cannot be changed directly.
From Figure 1, we find that the performance of both methods degrades significantly with increasing compression ratios. As shown in Figure 1(a), the performance of LongLLMLingua decreases by 47% as the compression ratio increases from 1.53x to 3.44x. Even worse, the accuracy of LongLLMLingua at 3.44x compression ratio is equivalent to that of the closed-book setting. The same findings are illustrated in Figure 1(b) for AutoCompressor.
More importantly, we observe that adding key information to the compressed result can greatly alleviate the performance degradation that typically occurs at high compression ratios. Back to Figure 1(a), the accuracy line fluctuates little as the compression ratio increases from 1.5x to 3.5x with the help of additional key information, which is a decrease of 3.87% compared to the former 47% with the loss of key information. These observations validate the need to preserve key information during compression, which motivates us to explore a better method to fully exploit query information for context compression.

heading: 3 Query-Guided Compression
text: As shown in Figure 2, we equip the LLM with the Query-Guided Compressor to compress long documents into a much shorter sequence of continuous representations, which are then concatenated with the corresponding instruction and query as the input for the LLM. In the following, we first introduce the architecture of Query-Guided Compressor and then its training objective. Then, we propose a dynamic compression strategy that assigns higher compression ratios for irrelevant documents to further improve the compressed representations.

heading: 3.1 Compressor Architecture
text: Figure 3 illustrates the basic architecture of our Query-Guided Compressor. Using the compressor, we adopt the following steps to produce compressed representations of each document: 1) learning the query-aware document representations; 2) compressing the document representations into ngram representations by weighted pooling; 3) aug-
menting the n-gram representations by reviewing the query and the entire document; 4) aligning the obtained representations into the embedding space of the LLM. Particularly, these four steps correspond exactly to the four key components of our compressor, which are all boxed in Figure 3. Note that we perform the above operations on each document, thus omitting the index k of the document for simplicity.
Query-Guided Context Encoder At the first step, we feed the concatenation of the query xq and the document xd into query-aware context encoder to learn the representations of the query and the document.
The encoder consists of two Transformer encoder layers. Formally, these representations can be obtained in the following way:
[hq;hd] = ContextEncoder([xq;xd]). (2)
Here, hq={hqi } Nq i=1 and h d={hdi }Ndi=1 are the corresponding representation sequences of the query and the document with the lengths of Nq and Nd, respectively. By allowing the query and the document to see each other during encoding, we can facilitate the extraction of the key information relevant to the query in the document.
Query-Guided Pooling Layer In the next step, we split the entire document into several n-grams and compress the information of each n-gram into a vector based on their correlation to the query.
To this end, document representations are organized as follows:
hd = [hdG1 , ...,h d Gj , ...,h d GNg ] (3)
= [hd1:n, ...,h d (j−1)×n:j×n, ...,h d Nd−n+1:Nd ],
where Gj represent the indices of the j-th n-gram. Ng= Nd n is the number of n-grams.
Then, we measure the weight of each token in Gj by calculating its relevance with the mean representation h q of query tokens:
h q =
1
Nq
∑ hqi , (4)
wi,Gj = exp s(h q , hdi )∑
i′∈Gj exp s(h q , hd i′ ) , (5)
where s(·, ·) is the dot-product function, and wi,Gj represents the weight of the i-th token representation hdi in the document, which belongs to the j-th n-gram.
Finally, we acquire the compressed n-gram representations ĥdGj as the weighted sum of token representations in the n-gram:
ĥdGj = ∑
i∈Gj wi,Gj · hdi . (6)
Query-Document Reviewing Layer To further prevent the key information loss in compression, we introduce a novel reviewing module to perfect the compressed n-gram representations by revising both the query and the document representations.
Concretely, this encoder consists of two Transformer encoder layers, which takes the query representations hq, the document representations hd, and the compressed n-gram representations ĥd as inputs, and outputs the improved document n-gram representations h̃d:
h̃d = ReviewingLayer([hq;hd; ĥd]). (7)
Semantic Alignment Layer Since h̃d lie in a different embedding space with the inputs of the LLM, we use a fully-connected semantic alignment layer to map the n-gram representations into the embedding space of the LLM. The aligned n-gram representations ed can be formulated as follows:
ed = W · h̃d + b, (8) where W and b are learnable parameters.

heading: 3.2 Compressor Training
text: Unlike AutoCompressor (Chevalier et al., 2023), we fix the parameter of the LLM and only fine-tune the compressor.
Through the above steps, each long document is compressed into a shorter sequence of continuous representations ed. Thus, the inputs of the LLM are finally formated as x̃ = (xins, ed1 , ..., edk , ..., edK ,xq). To avoid missing the key information during compression, we define the training objective of the compressor in the following way:
L = LCE + LKL (9) = − log p(y|x̃) + KL[p(y|x)||p(y|x̃)],
where KL[·||·] represents the Kullback–Leibler divergence. By introducing the KL loss, we encourage the LLM to generate the correct answer even with compressed representations as input.

heading: 3.3 Dynamically Compressing Strategy
text: Due to the different importance of retrieved documents, we propose to dynamically adjust the compression ratios for different retrieved documents. Specifically, we assign the n-gram size nk for the k-th document based on the importance ranking:
nk = { min(2 ·Ok, 16) Sk ≥ ϵ ∞ Sk < ϵ , (10)
where Sk and Ok is the score and rank of the k-th document acquired by the existing reranker, such as Contriever (Izacard et al., 2022a). ϵ is the score threshold for filtering low-score documents. Note that when the assigned n-gram size nk is set to ∞, the corresponding document will be discarded.

heading: 4 Experiments
text: In this section, we conduct extensive experiments to investigate the effectiveness of QGC.
Datasets & Evaluation Metric The experiments are carried out based on the three datasets:
• NaturalQuestions We select the processed version (Liu et al., 2023) where each question has 20 related documents and only one of them contains the correct answer. We follow Liu et al. (2023) to use accuracy (Acc) as the evaluation metric, which judges whether the correct answer appears in the prediction.
• TriviaQA We employ the adversarial Con-
triever (Izacard et al., 2022a) to retrieve the top 10 documents from all Wikipedia passages. Following Lewis et al. (2020), we use the Exact Match (EM) metric to evaluate the LLM prediction.
• HotpotQA Different from the above two datasets, HotpotQA (Yang et al.) is a multihop dataset where the answer lies in more than one document. Specifically, each question has 10 related documents and two of them are ground-truth documents. Following Yang et al., we use the F1 score to measure the correctness of the LLM.
Besides, we calculate the compression ratio (CR)
for different methods, which is defined as the length rate of the original context to the compressed context. We also provide the inference throughput (TP) on a single A100-80G GPU, including compression and generation.
Baselines Following (Jiang et al., 2023), we include two sets of methods as our baselines.
1) Reranker-based Methods. It simply uses a reranker method to sort documents based on importance and discards unimportant ones. We select the following reranker: Sentence-BERT (Reimers and Gurevych, 2020), BGE-Reranker (Xiao et al., 2023), and Cond.PPL proposed by Jiang et al. (2023) to measure the association between the query and documents. Then, we discard documents with low association until the compression ratio is met and sort the remaining documents according to the association from high to low.
2) Compression-based Methods. Compared with reranker-based methods, they further compress the sorted documents, retaining more information while satisfying a higher compression ratio. We select the following methods as our baselines:
• Selective-Context (Li et al., 2023) It uses selfinformation estimated by an external language model to prune redundant words.
• LongLLMLingua (Jiang et al., 2023) It is the state-of-the-art method for long context compression. It first uses a language model to quantify the importance of each document as its question-aware perplexity, and then designs a question-aware coarse-to-fine compression method to delete unimportant tokens.
• AutoCompressor (Chevalier et al., 2023) It fine-tunes LLaMA-2-7B to recursively compress long context into summary vectors, which are used as soft prompts to generate the answer. We use the released AutoCompressorLlama-2-7B-6K for experiments.
• ICAE (Ge et al., 2023) Similar to AutoCompressor, it generates compact and informative memory slots to represent the original context. We use the released ICAE model pre-trained on Llama-2-7B-Chat for experiments 5.
Implementation Details We use LongChat-13B16K and LLaMA-2-7B as the LLMs for evaluation,
5https://github.com/getao/icae
which are frozen during the optimization of QGC. To ensure stable and reproducible results, we employ greedy decoding and set the temperature to 0 in all experiments. Following Jiang et al. (2023), we use LLaMA-2-7B-Chat as the external language model for Selective-Context and LongLLMLingua. For QGC, both the query-guided context encoder and query-document reviewing layer consist of two Transformer encoder layers. All these layers and word embeddings are initialized with LLaMA-27B where MLP parameters are all fixed during training. Our rationale behind this approach stems from our belief that the MLP plays a crucial role in knowledge retention, while our focus lies in adjusting the acquired knowledge based on query. Thus, the trainable parameters in QGC are only 3.5% of LongChat-13B-16K. Besides the ground-truth document, we concatenate 1-4 random documents to build the long context. We also randomly set the n-gram size from the candidate list (4, 6, 8, 10) for each training batch to make the compressor more robust. We train QGC on downstream datasets for 15 epochs, using a learning rate of 5e-5 with the Adam optimizer and batch size of 64. During inference, we use the Cond.PPL proposed by Jiang et al. (2023) to sort retrieved documents for all compression-based methods and QGC, and set the ϵ as 0.35. Following (Liu et al., 2023; Bai et al., 2023) the maximum generation tokens is 100 for NaturalQuestions, and 32 for both TriviaQA and HotpotQA. All experiments are conducted on 8 NVIDIA A100 GPUs.
Main Results Table 1 reports the performance, compression ratios, and throughput of various methods or models on different datasets. Overall, QGC achieves higher compression ratios and greater throughput while achieving comparable or even better performance with LongLLMLingua. These results demonstrate that QGC can effectively compress context into shorter inputs.
Specifically, the performance and compression ratio of the reranker-based methods are limited
because no compression operation is used within the document. Compared to AutoCompressor and ICAE, our method achieves better accuracy with comparable compression ratios. Compared with LongLLMLingua, QGC achieves average +5.03 and +12.87 performance improvements when using LongChat-13B and LLaMA-2-7B as the target LLMs. On average, the compression ratio and throughput of QGC are 2.75 times and 2.47 times that of LongLLMLingua on all datasets and target LLMs, respectively.
Ablation Study To explore the effect of different components on QGC, we use LongChat-13B as the target LLM and introduce the following variants of QGC for ablation study: 1) w/o queryguided context encoder. In this variant, the query and document are independently encoded; 2) w/o query-guided pooling layer. When establishing this variant, we directly replace the weighted sum of token representations in each n-gram with their mean representation; 3) w/o query-document reviewing layer. This variant no longer refines the compressed representations of n-grams; 4) w/o dynamically compressing strategy. We fix the n-gram size as 4 for comparable comparison.
As shown in Table 2, the absence of the querydocument reviewing layer and dynamically compressing strategy lead to a 5.05 and 7.04 accuracy loss respectively. The more substantial loss is observed after removing the query-guided context encoder and query-guided pooling layer, resulting
in a significant performance accuracy drop of 18.83 and 13.85 respectively, highlighting the importance of employing the query to guide compression.

heading: 5 Analysis
text: In this section, we conduct in-depth analyses to explore the performance of QGC in terms of key information loss, demonstration compression, detailed throughput and reranker impact. All analyses are conducted on NaturalQuestions with target LLM as LongChat-13B.
Key Information Loss in QGC As described in Section 2.2, previous methods dramatically lose key information as the compression ratio increases. For comparison, we experiment with QGC using the same setting.
Compared to LongLLMLingua in Figure 4(a), the performance of QGC only decreases 10% as the compression ratio increases from 1x to 4x, and is even comparable to that of LongLLMLingua containing the correct answer in the compressed result. As seen in Figure 4(b), we observe that the performance of QGC slightly degrades with more documents, which is only a 12% decrease with 4 documents (27% for AutoCompressor). These results demonstrate that QGC can effectively retain key information even in much longer context and higher compression ratio scenarios.
Demonstration Compression for In-Context Learning To further validate the effectiveness of QGC in a broader context, we conduct experiments on both SST-2 and GSM8K datasets. We adopt the approach of previous studies (Chevalier et al., 2023; Wei et al., 2022) which utilizing demonstrations as the document, while maintaining consistency with their experimental setup. The results in Table 3 reveals notable insights. On the SST2 dataset, our method surpasses autocompressor in both compression ratio and accuracy. Meanwhile, on the GSM8K dataset, our accuracy performance remains on par with the original prompt
at the same compression ratio as autocompressor. This suggests that QGC strikes an excellent balance between model performance and compression ratio. These results showcases QGC’s proficiency in preserving information from demonstrations and fostering the in-context learning capacity of the target LLM.
Detailed Throughput Evaluation To evaluate the throughput of various methods or models, encompassing both compression and generation, we perform testing on a single A100-80G GPU.
The results presented in Figure 5 indicate that QGC is obviously higher than LongLLMLingua in both compression throughput and generation throughput. Moreover, by adjusting the hyperparameter ϵ (See Equation 10) to increase the compression ratio, QGC can achieve a higher compression ratio while minimizing the impact on LLM performance and further improving throughput. Furthermore, our higher compression ratios lead to shorter LLM input, which also significantly improves the generation throughput of the target LLM. As for LongLLMLingua, since it additionally introduces LLaMA-2-7B for compression, the compression throughput is significantly lower than ours. Besides, although LongLLMLingua can also improve compression ratio by adjusting hyper-parameters, its performance will significantly drop, while QGC still maintains excellent performance.
Impact of Different Rerankers The compression ratio for each document is determined by the corresponding correlation with the query obtained by a reranker. Here, we analyze the impact of using different rerankers in this process. In addition to the three methods introduced in reranker-based methods, we also include BM25 and Gzip (Jiang et al., b) for comparison.
Experimental results are shown in Figure 6. It can be found that QGC performs better with more competitive rerankers. Besides, compared with di-
rectly using rerankers for compression, QGC not only achieves an average 2.65 times higher compression ratio but also maintains lossless or even improved performance.

heading: 6 Related Work
text: Long Context for LLMs Recently, there have been a lot of studies focusing on expanding the context length of LLMs (Press et al., 2021; Peng et al., 2023; Bertsch et al., 2023). Existing efforts primarily involve gradually increasing the window size during pre-training (Nijkamp et al., 2023), interpolating position embeddings (Chen et al., 2023), and modifying the attention mechanism (Ding et al., 2023). Unlike these works, we do not directly aim to expand the context window of LLMs. Hence, the QGC that we proposed can complement these techniques by enabling LLMs to access a broader context with reduced cost and shorter latency.
Retrieval-augmented LMs Combined with a standalone retriever to augment LMs are gaining popularity for benefiting various knowledgeintensive tasks. Previous studies have achieved remarkable results in improving perplexity (Wang et al., 2023a), factual accuracy (Nakano et al., 2022), downstream task performance (Izacard et al., 2022b), and in-context learning (Huang et al., 2023). Besides, many works focus on cooperating LLMs and retrieved documents, such as reranking retrieved documents (Mao et al.) and discarding irrelevant documents (Mallen et al.). QGC is also a retrieval augmentation method for LLMs, which efficiently compresses the retrieved documents into shorter inputs while maintaining no significant performance degradation.
Context Compression With the growing context length in LLMs, the demand for higher ef-
ficiency, lower cost, and reduced latency has attracted much attention. As a promising solution, compression techniques can be broadly categorized into two types: black-box compression (Xu et al., 2023) and white-box compression (Wang et al., 2023b). Black-box compression primarily involves token pruning based on different importance measures, such as self-information (Li et al., 2023) and the LLM perplexity (Jiang et al., a, 2023). On the other hand, white-box compression focuses on generating summarization or compressing the context into soft prompt through fine-tuning or Low-Rank Adaptation (LoRA). For instance, Wang et al. (2023b) autoregressively generates filtered content and fine-tunes target LLM to use it for generation. Mu et al. (2023) trains LLMs to compress instructions into concise key-value attention prefixes. Chevalier et al. (2023) recursively compresses lengthy text into summary vectors, while Ge et al. (2023) generates memory slots to represent the original context. Compared with the above-mentioned compression studies, QGC’s design fully takes into account the query, which leads to the enhanced retention of key information, higher compression ratios, higher throughput, and improved overall performance.

heading: 7 Conclusion and Future Work
text: In this paper, we have presented a query-guided compressor QGC for LLMs to solve the loss of key information under high compression ratios. It consists of four essential components: query-guided context encoder, query-guided pooling layer, querydocument reviewing layer, and semantic alignment layer. In addition, we also propose a dynamically compressing strategy during inference. Extensive experiments on multi-document QA tasks demonstrate that QGC outperforms previous state-of-theart compression methods in both accuracy and compression ratios. Analyses reveal that this is primarily due to our retention of key information throughout the compression process.
In the future, we aim to validate our approach on more advanced LLMs, while also expanding its application to additional tasks like document summarization. Besides, we will try to further improve our approach by combining previous studies (Zhang et al., a; Hu et al., 2022; Zhang et al., 2022, b).
Limitations
QGC is a white-box compressor that necessitates access to the internal parameters of LLMs, which restricts its applicability. Furthermore, we have solely validated the effectiveness of QGC on QA and ICL task, and its performance on other tasks that differ significantly from QA task, such as summarization, remains to be verified.

heading: Acknowledgements
text: The project was supported by National Key R&D Program of China (No. 2022ZD0160501), National Natural Science Foundation of China (No. 62276219), and the Public Technology Service Platform Project of Xiamen (No. 3502Z20231043). We also thank the reviewers for their insightful comments.

heading: Future Work
text: In the future, we aim to validate our approach on more advanced LLMs, while also expanding its application to additional tasks like document summarization. Besides, we will try to further improve our approach by combining previous studies (Zhang et al. , a; Hu et al. , 2022; Zhang et al. , 2022, b). Limitations
QGC is a white-box compressor that necessitates access to the internal parameters of LLMs, which restricts its applicability. Furthermore, we have solely validated the effectiveness of QGC on QA and ICL task, and its performance on other tasks that differ significantly from QA task, such as summarization, remains to be verified