LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024)

Abstract: This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective. To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTalarge and mBERT. We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x.1 †Work during internship at Microsoft. ‡Corresponding author. Code: https://aka.ms/LLMLingua-2

text: Findings of the Association for Computational Linguistics ACL 2024, pages 963–981 August 11-16, 2024 ©2024 Association for Computational Linguistics
This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units according to their information entropy obtained from a causal language model such as LLaMa7B. The challenge is that information entropy may be a suboptimal compression metric: (i) it only leverages unidirectional context and may fail to capture all essential information needed for prompt compression; (ii) it is not aligned with the prompt compression objective.
To address these issues, we propose a data distillation procedure to derive knowledge from an LLM to compress prompts without losing crucial information, and meantime, introduce an extractive text compression dataset. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one, and use a Transformer encoder as the base architecture to capture all essential information for prompt compression from the full bidirectional context. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTalarge and mBERT.
We evaluate our method on both in-domain and out-of-domain datasets, including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its small size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability across different LLMs. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x.1
†Work during internship at Microsoft. ‡Corresponding author. 1Code: https://aka.ms/LLMLingua-2

heading: 1 Introduction
text: Recent years have witnessed the emergence of various prompting techniques for large language models (LLMs), such as Chain-of-Thought (COT) (Wei et al., 2022), In-context Learning (ICL) (Dong et al., 2023), and Retrieval Augmented Generation (RAG) (Lewis et al., 2020). These techniques empower LLMs to handle complex and varied tasks through rich and informative prompts that may exceed tens of thousands of tokens. However, the benefits of such lengthy prompts come at a cost of increased computational and financial overhead, as well as the degraded information perception ability of LLMs. Prompt compression is a straightforward solution to address these issues, which attempts to shorten the original prompts without losing essential information.
Several methods have been proposed to compress prompts in a task-aware manner (Jiang et al., 2023b; Xu et al., 2024; Jung and Kim, 2023; Huang et al., 2023). These techniques aim to generate compressed prompts tailored to the specific task or query, typically resulting in enhanced performance on downstream tasks, particularly in question answering. However, the dependency on task-specific features presents challenges in terms of efficiency and generalizability when deploying these methods. For example, in RAG-style applications, it may become necessary to compress the same documents multiple times depending on the associated queries with task-aware prompt compression. More details are discussed in Sec. 2.
Some works have explored task-agnostic prompt compression methods for better generalizability and efficiency (Jiang et al., 2023a; Li et al., 2023). The underlying assumption is that natural language contains redundancy (Shannon, 1951) that may be useful for human understanding but might not be necessary for LLMs. Therefore, they propose to compress prompts by removing tokens (Jiang et al.,
963
2023a) or lexical units (Li et al., 2023) according to their information entropy obtained from a causal small language model (SLM), regardless of the downstream task or question information. However, these task-agnostic methods face two challenges: (i) Information entropy is an empirical metric for prompt compression. Relying on it for prompt trimming may be suboptimal, as it is not aligned with the prompt compression objective. (ii) Causal LMs only leverage unidirectional context, which may fail to capture all essential information needed for prompt compression within the context.
The challenges lead to the following research questions:
Q1. How can we identify or build a suitable dataset to align the SLM towards effective prompt compression?
Q2. How can we design a compression algorithm that effectively leverages the full bidirectional context for better performance?
For Q1, most text compression datasets are abstractive (Toutanova et al., 2016; Koupaee and Wang, 2018; Kim et al., 2019), meaning that they treat prompt compression as a generative task where the original prompts are rephrased into condensed ones. However, this autoregressive generation process is slow and it may produce hallucinated content (Zhao et al., 2020). On the other hand, extractive compression datasets such as SentComp (Filippova and Altun, 2013) and DebateSum (Roush and Balaji, 2020) are usually created for the summarization task and often lack detailed information. In the case of prompt compression, this will hurt the performance of LLM inference in downstream applications such as QA (see Appendix G for some examples). Therefore, it is necessary to construct an extractive text compression dataset that retains essential information.
Contributions. We present this paper to address the above challenges for task-agnostic prompt compression. We make the following contributions.
• We propose a data distillation procedure to derive knowledge from an LLM (GPT-4) to compress the prompts without losing crucial information. We introduce an extractive text compression dataset, containing pairs of original texts from MeetingBank (Hu et al., 2023) and their compressed versions. We publicly release the dataset.
• We approach prompt compression as a token classification task (i.e., preserve or discard), and take the predicted probability of each token being labeled as preserve as the compression metric. The benefits are three folds: (1) It can capture all essential information needed for prompt compression from the full bidirectional context by using a Transformer encoder for feature extraction. (2) It can lead to lower latency, due to the use of smaller models to explicitly learn the compression objective. (3) It guarantees faithfulness of the compressed prompt to the original content.
• We conduct extensive experiments and analysis on both in-domain (i.e., MeetingBank) and out-of-domain datasets (i.e., LongBench, ZeroScrolls, GSM8K, and Big Bench Hard). Despite small in size, our model shows significant performance gains over strong baselines and demonstrates robust generalization ability from GPT-3.5-Turbo to Mistral-7B. Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.6x-2.9x with compression ratios of 2x-5x.

heading: 2 Related Works
text: Depending on whether task information is used for compression, prompt compression methods can be categorized into task-aware and task-agnostic compression approaches.
Task-aware compression compresses the context based on the downstream task or the current query. For example, LongLLMLingua (Jiang et al., 2023b) applies a question-aware coarse-to-fine compression approach to estimate the information entropy of the tokens and adapts the estimation according to the question. Reinforcement Learning (RL) based methods (Jung and Kim, 2023; Huang et al., 2023) usually train a model for prompt compression with reward signals from downstream tasks. Soft prompt tuning methods (Wingate et al., 2022; Mu et al., 2023) typically require fine-tuning for the specific task. Xu et al. (2024) trains a summarization model to compress the context depending on the question. Task-aware compression approaches are usually tailored for specific tasks and compression ratios, which may limit their generalizability in real-world applications.
Task-agnostic methods compress the prompt without considering the specific task, making it
more adaptable to a range of applications and blackbox LLMs. However, producing compressed text that can generalize well to different tasks is not trivial. Typical methods involve using information entropy-based metrics to remove redundant information in the prompt (Li et al., 2023; Jiang et al., 2023a). They employ a small language model to estimate token importance from the information metrics. Despite being training-free, these methods may not effectively capture the token importance distribution optimized for specific LLMs and often entail high computation overhead. Summarizationbased methods are also leveraged for task-agnostic compression (Chen et al., 2023; Packer et al., 2023). However, they often omit crucial details and do not generalize well. An alternative approach is to compress or trim the context hidden or KV caches (Chevalier et al., 2023; Ge et al., 2024; Zhang et al., 2023; Liu et al., 2023; Xiao et al., 2024). However, this is orthogonal to our work and cannot be easily applied to black-box LLMs.

heading: 3 Dataset Construction
text: In this section, we outline the process of dataset construction for prompt compression. We first introduce our data distillation procedure, which involves extracting knowledge from an LLM (GPT-4 ) to compress texts without losing crucial information or introducing hallucinated content (Sec. 3.1). Leveraging the distilled knowledge from the LLM, we explain our data annotation algorithm, which assigns labels to each word in the original text to indicate whether it should be preserved after compression (Sec. 3.2). To ensure the dataset’s quality, we propose two quality control metrics for filtering low-quality samples (Sec. 3.3).

heading: 3.1 Data Distillation
text: To extract knowledge from the LLM for effective prompt compression, our goal is to prompt GPT4 to generate compressed texts from original texts that meet the following criteria: (i) Token reduction: Compressed prompts should be short in length to reduce cost and speed up inference. (ii) Informativeness: Essential information should be retained. (iii) Faithfulness: Compressed prompts should remain faithful and avoid introducing hallucinated content to ensure accuracy when prompting LLMs in downstream tasks.
However, distilling such data from GPT-4 is challenging, as it does not consistently follow the instructions. For instance, Jiang et al. (2023a) experimented with different prompts for compression and found that GPT-4 struggles to retain essential information from original texts. In our preliminary experiments, we have also observed that GPT-4 tends to modify expressions used in the original texts and sometimes generates hallucinated content. To address this challenge, we propose the following dataset distillation procedure.
Instruction Design A well-crafted instruction is the key to unveiling the compression capabilities of GPT-4. To ensure that the generated texts stay faithful to the original, we explicitly instruct GPT4 to compress the text by discarding unimportant words in the original texts only and not adding any new words during generation.
To ensure token reduction and informativeness, previous studies (Jiang et al., 2023a; Huang et al., 2023) have specified either a compression ratio or a target number of compressed tokens in the instructions. However, GPT-4 often fails to adhere to these restrictions. Additionally, the information
density of text can vary significantly depending on its genre, style, etc. For instance, news articles typically contain denser information compared to meeting transcripts. Furthermore, even within the domain of meeting transcripts, the information density from different speakers may vary. These factors suggest that a fixed compression ratio may not be optimal. Therefore, we remove the compression ratio restriction from our instructions and instead prompt GPT-4 to compress the origin text as short as possible while retaining as much information as possible. As shown in Fig. 3, GPT-4 assigns varying compression ratios to different sentences and discards some sentences entirely. For a comparison between our instruction and those of Jiang et al. (2023a), please refer to Table 7.
Chunk-Wise Compression Empirically, we have found that the length of the original text has a notable influence on the compression performance. As shown in Fig. 4, GPT-4 tends to apply a high compression ratio when processing very long context, which might be due to GPT-4’s limited ability to handle long context. This aggressive compres-
sion leads to substantial information loss, significantly impacting the performance of downstream tasks. To mitigate this issue, we first segment each long context into multiple chunks, each containing no more than 512 tokens and ending with a period. We then instruct GPT-4 to compress each chunk individually.

heading: 3.2 Data Annotation
text: Having obtained pairs of original texts and their compressed versions from data distillation (Sec. 3.1), the goal of data annotation is to assign a binary label to each token in the original texts to determine if it should be preserved or discarded after compression. Fig. 5 describes the three primary obstacles encountered here, which arise from GPT-4’s inability to precisely comply with the instruction in Fig. 9. Alg. 1 outlines the overall procedure of the proposed annotation algorithm designed to deal with these obstacles. For more detailed information, please refer to Appendix B.

heading: 3.3 Quality Control
text: We introduce two quality control metrics to assess the quality of the compressed texts generated by GPT-4 distillation, as well as the quality of the automatically annotated labels. We then filter the examples by their scores.
Variation Rate As GPT-4 may fail to follow the instructions, we introduce the metric Variation Rate (VR) to evaluate the quality of the compressed texts generated from data distillation. VR measures the proportion of words in the compressed text that are

heading: Original Texts

heading: Compressed Texts
text: (ii) Variation: GPT-4 may modify the original words in tense, plural form, etc. during compression. (iii) Reordering: The order of words may be changed after compression.
Algorithm 1: Data Annotation Input :original string Sori, compressed
string Scomp, window size s. Split original string Sori to word list Sori. Split compressed Scomp to word list Scomp. Initialize labels of original words to False. Initialize previous match index prev to 0.
for w ∈ Scomp do for i = 1, 2, ..., s2 do
right = min(|Sori|, prev + i) if fuzzy_match(w, Sori[right]) then
L[right] = True. prev = right. Break.
end left = max(0, prev − i) if fuzzy_match(w, Sori[left]) then
L[left] = True. Break.
end end
end Output : labels of original words L(Sori).
absent in the original text. Specifically, let Scomp be the set of words in the compressed text and Sori be that of the original text. VR is defined as:
VR = 1 |Scomp| ∑
w∈Scomp I(w /∈ Sori), (1)
where | · | is the cardinality of a set. A higher variation rate implies a higher likelihood of encountering hallucinated content. Therefore, we exclude the
examples with the top 5% highest variation rates.
Alignment Gap We propose Alignment Gap (AG) to evaluate the quality of the automatically annotated labels. Let l(·) represent the annotation function, where l(w) = True signifies that word w ∈ Sori corresponds to a word in Scomp. We firstly define the matching rate (MR) as:
MR = 1 |Sori| ∑
w∈Sori I(l(w) = True). (2)
Since there exists a many-to-one word mapping from Sori to Scomp (i.e., the "Ambiguity" challenge presented in Sec. 3.2), we further present a hitting rate (HR) as a regularization term to measure the proportion of words in Scomp that are found in Sori. HR is defined as:
HR = 1 |Sori| ∑
w∈Scomp I(w ∈ Sori). (3)
Finally, the Alignment Gap (AG) is defined as:
AG = HR − MR. (4)
The alignment gap of a perfect annotation should be 0. A large AG indicates a high hitting rate with a poor matching rate, implying low-quality annotation for this example. Therefore, we discard examples of the highest 10% alignment gap to ensure quality control of the dataset.

heading: 4 Compressor
text: We formulate prompt compression as a binary token classification problem (i.e., preserve or discard) to guarantee the faithfulness of the compressed prompt to the original content, and meantime ensure the low latency of the compression model itself. For the token classification model, we employ a Transformer encoder as the feature extractor to leverage information from the bidirectional contexts of each token. We train the classification model on the dataset constructed in Sec. 3 from MeetingBank (Hu et al., 2023). During inference, we determine whether to preserve or discard each token in the original prompt based on its probability calculated by our classification model.

heading: 4.1 Token Classification Model
text: Architecture We utilize a Transformer encoder (Devlin et al., 2019) as the feature encoder fθ and add a linear classification layer on top. Given
an original prompt consisting of N words x = {xi}Ni=1, this can be formulated as:
h = fθ(x), (5)
p(xi,Θ) = softmax(Whi + b), (6)
where h = {hi}Ni=1 denotes feature vectors for all words, p(xi,Θ) ∈ R2 denotes the probability distribution of labels {preserve, discard} for the i-th word xi, and Θ = {θ,W, b} represent all the trainable parameters.
Training Let y = {yi}Ni=1 denote the corresponding labels for all words in x, then we employ cross entropy loss to train the model. The loss function L w.r.t. x is:
L(Θ) = 1 N
N∑
i=1
CrossEntropy(yi, p(xi,Θ)). (7)

heading: 4.2 Compression Strategy
text: Our approach to compressing the original prompt x = {xi}Ni=1 with a target compression ratio 1/τ involves a three-step process, where τ is defined as the quotient of the number of words in the compressed prompt and the number of words in the original prompt x. First, we derive the target number of tokens to be preserved in the compressed prompt x̃: Ñ = τN . Next, we use the token classification model to predict the probability pi of each word xi being labeled as preserve2. Finally, we retain the top Ñ words in the original prompt x with the highest pi and maintain their original order to form the compressed prompt x̃.
It’s worth noting that our approach can be readily integrated into the coarse-to-fine framework proposed in LLMLingua (Jiang et al., 2023a), allowing
2To address tokenization-related challenges that arise when applying our approach across various LLMs and SLMs, we preserve the integrity of multi-token words and represent the probability of a word by averaging over the predicted probabilities of all subword tokens.
for a higher compression ratio of ∼15x for tasks involving multiple demonstrations or documents. Particularly, we can replace the perplexity-based iterative token compression module in LLMLingua with our token-classification-based compressor, while keeping the budget controller unchanged. Detailed information can be found in Appendix K.

heading: 5 Experiment
text: Implementation Details We construct our extractive text compression dataset using training examples from MeetingBank (Hu et al., 2023) with implementation details in Appendix A. Our approach is implemented using Huggingface’s Transformers and PyTorch 2.0.1 with CUDA-11.7. We use xlm-roberta-large (Conneau et al., 2020) and multilingual-BERT (Devlin et al., 2019) for the feature encoder fθ in our compressor, which we refer to as LLMLingua-2 and LLMLingua-2-small, respectively. We finetune both models for 10 epochs, using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1e-5 and a batch size of 10. Unless specified otherwise, all reported metrics use GPT-3.5Turbo-06133 as the target LLM for downstream tasks, with greedy decoding at a temperature of 0 for enhanced stability across experiments.
Datasets & Evaluation Metrics We conduct five groups of experiments to evaluate the compressed prompts on two groups of datasets.
(i) In-Domain: As we train our compressor using the dataset built with training examples from MeetingBank (Hu et al., 2023), we use the MeetingBank test examples for in-domain evaluation. In addition to the summarization task, we further introduce a QA task by prompting GPT-4 to generate 3 question-answer pairs for each example distributed across the whole context (see Appendix F
3https://platform.openai.com/

heading: Methods LongBench ZeroSCROLLS

heading: Task(Question)-Aware Compression

heading: Task(Question)-Agnostic Compression

heading: Task(Question)-Aware Compression

heading: Task(Question)-Agnostic Compression
text: for more details). For the summarization task, we use the same evaluation metric as in LLMLingua (Jiang et al., 2023a). For QA task, we utilize the Exact Match as the evaluation metric.
(ii) Out-of-Domain: For long-context scenarios, we use LongBench (Bai et al., 2023) and ZeroSCROLLS (Shaham et al., 2023), and we employ the same evaluation metric as in LongLLMLingua (Jiang et al., 2023b). For reasoning and in-context learning, we use GSM8K (Cobbe et al., 2021) and Big Bench Hard (BBH) (bench authors, 2023), with evaluation metrics consistent with LLMLin-
gua (Jiang et al., 2023a).
Baselines We take two state-of-the-art prompt compression methods as primary baselines for comparison: Selective-Context (Li et al., 2023) and LLMLingua (Jiang et al., 2023a), both are based on LLaMA-2-7B. Additionally, we compare our approach with some task-aware prompt compression methods, such as retrieval-based methods and LongLLMLingua (Jiang et al., 2023b).
Results on In-Domain Benchmark In Table 1, we first present the results of our proposed method
compared to the strong baselines on MeetingBank. Despite the fact that our compressors are much smaller than the LLaMa-2-7B used in the baselines, our approach achieves significantly better performance on both the QA and Summary tasks, and comes close to matching the performance of the original prompt. This demonstrates the effectiveness of our constructed dataset, and highlights the importance and benefit of optimizing the compression model using prompt compression knowledge.
Results on Out-of-Domain Benchmarks As our model is trained on meeting transcripts data from MeetingBank, here we explore its generalization ability across various benchmarks of long-context scenarios, reasoning, and in-context learning. Table 2 and 3 show the results on LongBench, ZeroSCROLLS, GSM8K, and BBH: Our model has demonstrated superior performance compared to other task-agnostic baselines. Even our smaller model, which is of BERT-base size, has been able to achieve comparable, and in some cases, even slightly higher performance than the original prompt. While our approach has shown promising results, it falls short when compared to other taskaware compression methods like LongLLMlingua (Jiang et al., 2023a) on Longbench. We attribute this performance gap to the additional information that they leverage from the question. However, the task-agnostic characteristics of our model make it an efficient option with good generalizability when deployed across different scenarios.
Mistral-7B as the Target LLM Table 4 presents the results of different methods using Mistral-7Bv0.14 as the target LLM. Our method demonstrates significant performance gain over other baselines, showcasing its good generalization ability across target LLMs. Notably, LLMLingua-2 yields even better performance than the original prompt. We
4https://mistral.ai/
speculate that Mistral-7B might be less adept at managing long contexts than GPT-3.5-Turbo. Our method, by offering shorter prompts with higher information density, effectively improves Mistral7B’s final inference performance.
Latency Evaluation Table 5 shows the latency of different systems on a V100-32G GPU with different compression ratios. It shows that LLMLingua2 has a much smaller computation overhead than other compression methods, and can achieve an end-to-end speedup ranging from 1.6x to 2.9x. Additionally, our method can reduce GPU memory costs by 8x, lowering the demand for hardware resources. For details, see the Appendix I.
Observation on Context Awareness We have observed that LLMLingua-2 can effectively maintain the most informative words with respect to the full context as the compression ratio increases. We owe this to the adoption of the bidirectional contextaware feature extractor, as well as the strategy of explicitly optimizing toward the prompt compression objective. See Figure 6 for more details.
Prompt Reconstruction We have conducted experiments of prompting GPT-4 to reconstruct the original prompt from the LLMLingua-2 compressed prompt. The results show that GPT-4 can effectively reconstruct the original prompt, suggesting that there is no essential information loss during the compression process of LLMLingua-2. Figure 7 and 8 in Appendix E present some examples.

heading: Methods LongBench ZeroSCROLLS
text: Ablation Study on Chunk-Wise Compression and Instruction Design Table 7 shows that both the designed instruction and the chunk-wise compression strategy proposed in this paper significantly contribute to the success of LLMLingua-2.

heading: 6 Conclusion
text: This paper targets task-agnostic prompt compression for better generalizability and efficiency. In this paper, we identify the challenges encountered in existing methods and address them accordingly. We conduct extensive experiments and analysis on five benchmarks across different tasks and domains. Our model shows superiority over strong baselines in terms of performance and compression latency. We publicly release the dataset of text compression with no essential information loss in this paper.

heading: Limitations
text: Our text compression dataset was constructed using only training examples from MeetingBank, a dataset of summarization over meeting transcripts. This raises concerns about the generalization ability of our compressor. Here we discuss this question
from two perspectives. Firstly, we have conducted extensive out-ofdomain evaluation on four benchmarks in the paper, including LongBench (Bai et al., 2023), ZeroSCROLLS (Shaham et al., 2023), GSM8K (Cobbe et al., 2021), and Big Bench Hard (BBH) (bench authors, 2023), which cover multiple tasks from document QA to math problems and in-context learning. The experimental results show that even our LLMLingua-2-small model that is of BERT-base size achieves superior performance than the two LLaMA-2-7B based baselines Selective-Context (Li et al., 2023) and LLMLingua (Jiang et al., 2023a). This demonstrates that our learned prompt compression model has good generalization ability to data from different domains.
Secondly, we expand the constructed text compression dataset using 50k examples from TriviaQA-wiki. Then train an LLMLingua-2 compressor with the expanded dataset to see whether there would be further performance gain. Table 6 shows the results under the 2,000-token constraint. We can see that training the compressor with more data does bring further performance gain (LLMLingua-2‡). However, the improvement seems not that significant. We conjecture that this is because although the semantics of texts from different domains may vary a lot, their redundancy pattern might be similar. Such pattern or knowledge may be learned during in-domain training, and then act as an anchor that can transfer across different domains. We leave this for future work.

heading: A Details of Data Distillation
text: To construct the extractive compression dataset, we use GPT-4-32k to compress the original meeting transcript. Each transcript is divided into chunks first, with each chunk terminating at the end of a complete sentence and not exceeding 512 tokens. We employ the default parameter settings with a temperature of 0.3 and a top_p of 1.0. The maximum number of generated tokens is set to 4096. Transcripts exceeding 28K tokens are truncated, allowing a 4K token budget for generation. Fig. 9 presents the full instruction used in GPT-4 compression. Tab. 8 shows the statistics of our MeetingBank compression dataset.

heading: B Details of Data Annotation
text: Based on the compressed prompt, we design a word annotation algorithm to automatically assign each word a label indicating whether the word in the original prompt should be retained. Initially, all labels of the original words are set to False. Then, for every word in the compressed prompt, we search for its corresponding word in the original prompt, which is then assigned a True label.
Sliding Window: To assign labels to the appropriate words in the original prompt, we utilize a sliding window approach, constraining the search scope within a local window centered on the previously matched word in the original prompt.
The search initiates from the last matching position. The True label is then assigned to the first matched word in the original prompt. Furthermore, the search is bidirectional to prevent mismatches caused by GPT-4’s reordering, as shown in Fig. 5. Moreover, if GPT-4 introduces new words during compression, the sliding window restricts the search scope, preventing mismatches between the newly added words in the compressed prompt and words in the original prompt.
Fuzzy Matching: Another challenge arises from the “variation" misbehavior of GPT-4, as illustrated in Fig 5. GPT-4 may alter the original words in tense, voice, and singular/plural forms during compression, even when we request GPT-4 to compress by discarding words only. To address this issue, we first apply lemmatization to reduce words to their base form using Spacy5, and then perform word matching using the sliding window approach.

heading: C Context Aware Compression
text: Fig. 6 presents some compression results of our LLMLingua-2 under different compression ratios. Our method effectively maintains the most meaningful words as the compression ratio increases.

heading: D Comparison with Baselines
text: In Fig. 11 and Fig. 12, we qualitatively compare the compressed prompts of our methods with those of baseline method on GSM8K and BBH datasets. Note our LLMLingua-2 here is only trained on
5https://spacy.io/api/lemmatizer
MeetingBank, but also yields more reasonable compressed prompt than baseline methods on the transferred domain data.

heading: E Prompt Reconstruction
text: Fig. 7 and Fig. 8 show two reconstructed prompts from the compressed prompts using GPT-4. Specifically, we prepend a simple reconstruction instruction: "I have asked you to compress a meeting transcript by dropping word only. Now, reconstruct the original meeting transcript based on the following compressed transcript." to the compressed prompt. With the key information preserved in the compressed prompt, the reconstructed prompt closely resembles the original prompt.

heading: F Details of MeetingBank QA and MeetingBank Summary
text: The MeetingBank QA dataset consists of 862 meeting transcripts from the MeetingBank test set. Initially, we generate 10 question-answer pairs for each meeting transcript using GPT-4-32K. The instruction used in generating QA pairs is: "Create 10 questions/answer pairs from the given meeting transcript. The answer should be short and concise. The question should start with Q: and answser should start with A: . The meeting transcript is as follows.". To ensure the quality of the generated QA pairs, we discard the question-answer pairs with answer lengths exceeding 50 tokens. Subsequently, we carefully examine the remaining QA pairs to ensure that the answers actually appear in the original transcripts, instead of being products of GPT-4’s hallucinations. After the aforemen-
tioned filtering process, we retain 3 high-quality question-answer pairs for each meeting transcript. Additionally, we instruct GPT-4-32K to summarize each meeting transcript. The summaries generated by GPT-4 are used as ground truth to evaluate the summary performance.

heading: G Drawback of Existing Text Compression Dataset
text: Existing extractive compression datasets such as SentComp (Filippova and Altun, 2013) and Debate-
Sum (Roush and Balaji, 2020) are mainly created for summarization task. The compressed texts provided in their dataset are usually too concise, only maintaining the main idea of the original text and lacking detailed information. This information loss inevitably hinders the downstream tasks such as document-based QA, as illustrated in Fig. 13 and Fig. 14

heading: Our GPT-4 Instruction for Compression: System Prompt:

heading: Instruction1:

heading: H Model Size and Training Details
text: We use xlm-roberta-large which has 355M parameters as the feature encoder fθ in LLMLingua-2. The training process takes approximately 23 hours on our MeetingBank compression dataset. For LLMLingua-2-small, the feature encoder is the multilingual-BERT which has 110M parameters. It takes roughly 16 hours to train the multilingual-BERT model.

heading: I GPU Memory Usage
text: LLMLingua-2 enjoys a smaller GPU memory overhead because of its lightweight. The peak GPU memory usage of LLMLingua-2 on MeetingBank is only 2.1GB, while LLMLingua and SelectiveContext, which utilize LLAMA-2-7B as the SLM, consume 16.6GB and 26.5GB of GPU memory, respectively.

heading: J Multilingual Generalization Ability
text: In Table 10, we assess the performance of LLMLingua-2 on the Chinese benchmarks of
LongBench, comprising 5 tasks with a total of 1000 samples. Despite being trained solely on the MeetingBank data, which consists of English corpus only, LLMLingua-2 also outperforms LLMLingua on Chinese benchmarks. We attribute this performance gain to the multilingual capabilities of the xlm-roberta-large or multilingual-BERT compressor acquired from the pre-training phase.
K Integration with LongLLMLingua
In retrieval-augmented generation (RAG) and Multi-Documents Question-Answer (MDQA) scenarios, the primary challenge is to identify the document that contains the key information relevant to the question. In these scenarios, LongLLMLingua improves the key information preservation by utilizing the information provided in the question.
While LLMLingua-2 is designed for questionagnostic compression, it can also be integrated with LongLLMLingua to preserve more key information relevant to the question in these scenarios. Specifically, we utilize LongLLMLingua’s coarse-grained
compression to assign varying compression ratios to different documents based on the question’s perplexity conditioned on each document. Consequently, it allocates more token budgets to the documents which are more relevant to the question.
As illustrated in Table 11, LLMLingua-2 with LongLLMLingua coarse-grained compression achieves an average performance gain of 25.3% on NaturalQuestions (Liu et al., 2024) compared to LLMLingua-2.

heading: L Sample-Wise Dynamic Compression Ratio
text: By default, LLMLingua-2 applies fixed compression rate to all samples in the benchmark. However, this approach may not be optimal due to variations in the density of key information across different samples. To address this problem, we allow LLMLingua-2 to dynamically adjust the compression rate for each sample under the overall compression rate constraint. Specifically, we employ the compressor to predict each token’s preservation probability of all samples. We then set a probability threshold to achieve the overall compression rate constraint. For all samples, tokens with preservation probabilities higher than this threshold are retained.
Table 12 presents the performance of LLMLingua-2 using the sample-wise dynamic compression ratio, showcasing a 4.4% and 4.5% performance improvement under 7x and 5x compression ratios, respectively, compared to
LLMLingua-2 with a fixed compression ratio.
M Performance w.r.t Compression Ratio
Fig 15 presents the performance w.r.t compression ratio on a subset of 100 samples from Meetingbank. As depicted, LLMLingua-2 exhibits superior robustness compared to other baselines as the compression ratio increases.

heading: N Preservation Priority in GPT-4 Compression
text: To gain insight into GPT-4’s compression patterns, we analyze the distribution of different POS categories. Fig 16 suggests that GPT-4 prioritizes the preservation of nouns, adjectives, and numerals, which typically play a more important role in the comprehension of the overall context.

heading: O Comparison With GPT-4 Compression
text: Table 9 shows the comparison between LLMLingua-2 compressed prompts and GPT-4 compressed prompts. For GPT-4 compression, We use the same compression instruction as the one used in training data collection. The same chunking technique is also adopted with the chunk size setting to 512. It is shown that LLMLingua-2 achieves higher performance than GPT-4 compression on MeetingBank QA. We conjecture that LLMLingua-2’s ability to learn compression knowledge from the entire dataset helps mitigate the influence of noise and information loss present
in each GPT-4 compressed example, leading to superior performance.

heading: P Performance of Mistral-7B on 8K Token Subset
text: As the Mistral 7B model is trained with an 8k context length 6 , its performance may drop if the input context is too long. Therefore, we conduct additional experiments on subsets containing only examples with original prompts shorter than 8k tokens. The results, shown in Table 13, demonstrate that LLMLingua-2 continues to outperform strong baselines and even the original prompts in this subset.
6https://huggingface.co/docs/transformers/main/en/model_ doc/mistral

heading: Document:

heading: Document:
text: Methods LongBench-Zh
SingleDoc MultiDoc Summ. FewShot Synth. AVG Tokens 1/τ
Task(Question)-Agnostic Compression
LLMLingua 35.2 20.4 11.8 24.3 51.4 28.6 3,060 5x LLMLingua-2 46.7 23.0 15.3 32.8 72.6 38.1 3,023 5x
Original Prompt 61.2 28.7 16.0 29.2 77.5 42.5 14,940 -
(a) QA performance w.r.t compression ratio on a 100 samples subset of MeetingBank.
(b) Summary performance w.r.t compression ratio on a 100 samples subset of MeetingBank.
Figure 15: A plot of performance w.r.t compression ratio on a 100 samples subset of MeetingBank.

heading: Question-Agnostic Compression

heading: Future Work
text: We leave this for future work