Extending Context Window of Large Language Models via Semantic Compression (2024)

Abstract: Transformer based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses due to the quadratic complexity. These constraints restrict their applicability in long text scenarios. In this paper, we propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer without incurring significant computational costs or requiring finetuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

heading: 1 Introduction
text: The recent successful release of large language models (LLMs) such as ChatGPT (Radford et al., 2019) and LLaMA (Touvron et al., 2023) have sparked significant research efforts from both industry and academia. These LLMs have demonstrated the ability to engage in fluent and coherent conversations with human users, and have shown exceptional performance across various tasks, including document summarization, question answering, dialogue bots, and code generation.
One critical issue faced by state-of-the-art (SoTA) LLMs is the restriction on the length of text that can be inputted into the model at once. When the input context exceeds the limit of the context
*Correspondence to: niuxueyan3@huawei.com.
window, the performance of these models declines rapidly. This limitation poses a challenge to current LLMs when it comes to handling long texts such as scientific papers, novels, and legal contracts. As a result, there has been a growing interest in finding ways to extend the input length without significantly compromising the models’ performance.
The limitation on the context window primarily stems from the quadratic computation of the selfattention mechanism in the transformer. Handling lengthy texts significantly increases the computational costs in terms of both memory and time. Typically, models are trained on shorter contexts, so the maximum sequence length (i.e., the context window) is fixed. If the models are compelled to generate contexts that exceed the context window, they tend to compromise the quality of the output due to the lack of position encoding information that is not learned from the training. Furthermore, generating long sequences imposes substantial memory requirements on the computational device. This accumulation of memory requirements and the lack of effective position encoding can result in length generalization failure (Anil et al., 2022), where the models struggle to generate meaningful and coherent text beyond a certain context window size.
Some techniques have been developed to address the aforementioned challenges. One approach is to devise architectures with nearly linear complexity, which enables efficient scaling to handle very long sequences. However, training a large model from scratch incurs substantial costs. Another strategy involves employing interpolation and fine-tuning techniques to adapt the position encoding to unseen sequence lengths. While this method has the potential to improve the overall performance of LLMs on long sequences, it still demands significant time and GPU resources for fine-tuning and inference. Therefore, methods that do not necessitate altering the parameters of the pre-trained models are more efficient and resource-friendly.
5169
While most previous algorithms involve modifying the pre-trained models, we instead exploit the statistical properties of the natural language input. One empirical phenomenon, known as Zipf’s law (Zipf, 2016), observes that a small set of the most frequent word tokens in a large corpus of natural language account for almost all occurrences. This pattern arises from the tendency of language users to minimize effort in their daily conversations. Consequently, by utilizing an expanded vocabulary, sentences can be significantly shortened while preserving the semantic meaning. Moreover, it is common for language users to include redundant words during communication (Strunk Jr, 2007). These language habits are prevalent among users, and we propose to include a semantic compression module to mitigate the associated redundancies.
Our proposed semantic compression method, reminiscent of lossy source coding in information theory, extends the context window by equivalently shortening the long text input while preserving the semantic meaning. This procedure is conducted before inputting the tokens into the pre-trained LLMs. As illustrated in Fig. 1, the input undergoes compression before being transmitted to the LLM for potential downstream tasks. The semantic compression method can be customized and optimized for downstream tasks, taking into consideration practical constraints such as time and memory resources. The implementation of the semantic compression module is straightforward and can be easily incorporated into other interpolation-based context window extension methods and black box APIs. Our method demonstrates enhanced performance compared to SoTA interpolation-based methods on a range of tasks, including single-document question answering, multi-document question answering, summarization, few-shot learning, and information retrieval, using real-world datasets while incurring no extra parameter updates or memory consumption. Empirically, the proposed method is computationally efficient and achieves 6-8 times context window extension.
Our contributions:
• We propose a context window extension framework for LLMs that serves as a plugand-play tool to mitigate redundancies in input texts by efficiently performing topic modeling and semantic compression.
• We construct a graph representation of the input to identify distinct sections of the text that pertain to different topics. The result is the segmentation of long texts into separate chunks, each focusing on a specific topic. We then conquer each chunk independently, resulting in a concise version of the original texts. This compression technique helps to condense the information while preserving the key ideas and contexts.
• We demonstrate the applicability of our proposed semantic compression method through extensive experiments. The results highlight the advantages of our method in several key applications, including single-document question answering, multi-document question answering, summarization, few-shot learning, and information retrieval.

heading: 2 Related Work

heading: 2.1 Efficient Attention Operations
text: Due to the self-attention mechanism, the inference cost of LLMs grows quadratically with respect to the sequence length. One line of research aims to reduce the complexity when handling long contexts. Dai et al. (2019) present Transformer-XL which utilizes segment-level recurrence agency and a novel positional encoding scheme. Beltagy et al. (2020) introduce Longformer with a sparse attention mechanism that scales linearly with sequence length. Bo (2021) provides a faster transformer, RWKV, which exploits the strength of RNN and has linear complexity during inference. Dao et al. (2022) propose FlashAttention, a chunking strategy for the input, and utilize recomputation to avoid the quadratic complexity of attention computation.
While these methods have the potential to handle longer input sequences (Ding et al., 2023), training new models can be very costly. Moreover, these methods are still not effective when dealing with out-of-distribution content lengths.

heading: 2.2 Position Extrapolation and Interpolation
text: A larger group of research focuses on adapting existing LLMs trained on short texts to accommodate longer text during inference (Anil et al., 2022). The key challenge resides in the position encoding of the input, which has solely been trained on short texts, therefore is inadequate for handling long texts. Current studies are usually based on the relative positional encoding, Rotary Position Embeddings (RoPE) (Su et al., 2024) which is widely adopted due to its strong extrapolation capabilities. Chen et al. (2023b) develop the Position Interpolation (PI) method to linearly scale the positional encoding of long text into trained encoding. Their experiments show that an effective long-context model can be achieved with just 1000 fine-tuning steps. Peng et al. (2023) introduce YaRN, an efficient extrapolation mechanism using the neural tangent kernel which dynamically scales the logits. Chen et al. (2023a) further proposes continuous dynamics and utilizes ordinary differential equations to fit the length scaling factor.
The introduction of new positional embeddings requires fine-tuning long sequences to adapt to the unseen increased length, which can be computationally expensive. To address this problem, LongLoRA is introduced by Chen et al. (2023c), offering an efficient fine-tuning method with limited computational costs. Unlike the above methods, our approach does not require fine-tuning on long texts and is a low-cost length-extending solution.

heading: 2.3 Prompting Compression
text: There are ongoing efforts to extend the context window through smart prompting designs. Wingate et al. (2022) employ soft prompts to compress the context so that the compressed prompts can retain a substantive amount of information. Chevalier et al. (2023) further presents AutoCompressor, which utilizes unsupervised learning to adapt LLMs to compress long contexts into compact summary vectors and then extends the original length of the base model by conditional language modeling. Li (2023) leverage mutual information to eliminate redundant tokens and enhance inference efficiency. Similarly, Jiang et al. (2023) employ a small language model
to identify and remove non-essential tokens in the input prompts. These methods collectively showcase the potential of prompt compression in extending the context window of LLM. Both Zhou et al. (2023) and Wang et al. (2023a) recurrently apply LLMs to summarize the input texts to maintain long short-term memory for specific purposes such as story writing and dialogue generation, respectively.
More details on dealing with long text in LLMs are provided in the survey by Huang et al. (2023).

heading: 3 Methodology
text: We now introduce our semantic compression method for extending the context window. The core idea is to compress the input into shorter texts without losing the key information or important details. This enables us to effectively include more content within the fixed input length constrained by the base LLM. Fig. 2 provides an overview of our method, which leverages pre-trained summarization models commonly used in Natural Language Processing (NLP).
Like LLMs, existing summarization methods also have limitations regarding the length of the input. Here, we propose a divide-and-conquer approach that takes into account the topic structure of the text. By identifying distinct topics in the lengthy text and dividing the text into blocks that exhibit a certain level of mutual independence, the content within each block can be compressed efficiently owing to their statistical correlation. The blocks can then be processed in parallel using pretrained models, and the results are combined to create a condensed textual input that can be processed by the base LLM. This approach provides a computationally efficient way of removing redundancies in long texts which effectively extend the context window of the input.

heading: 3.1 Model
text: Real-world textual content, such as speech and books, frequently displays hierarchical structures, wherein each section is structured around a particular topic, and different sections differ in a topic in a sequential manner. This hierarchical structure, based on topics, bears resemblance to cliques in graphs. To identify this structure within long texts, we utilize weighted graphs to represent them and employ clustering methods to detect cliques in these graphs. The cliques can then be utilized
to represent the topic-based content of the text, allowing us to obtain chunks based on the semantic relevance of the topics.
We begin by sequentially constructing sentencelevel blocks within given lengths and representing them as nodes in our graph. In this step, we parse the text into different sentences or sub-sentences based on punctuation marks. Next, we sequentially fill the sentence-level blocks until they exceed the desired length before proceeding to the next blocks. Once we have obtained the sentencelevel blocks, we build the graph representation of long text G based on a pre-trained sentence embedding model (e.g., MiniLM (Wang et al., 2020)), where the weight G[i][j] represents the semantic similarity between the i-th and j-th sentence-level blocks. Typically, this similarity is computed using cosine similarity, which measures the cosine of the angle between two embeddings. If the similarity between two blocks is higher, it indicates that they are closer in topics.

heading: 3.2 Topic-Based Chunking
text: We then apply spectral clustering algorithms 1 on the graph to identify the underlying topic structure. Within each cluster, we group the sentencelevel blocks sequentially to obtain the topic-based chunks, which can then be handled simultaneously by the pre-trained model chosen according to the downstream task. The number of clusters can be
1We present the detailed introduction of our used clustering method in Appendix C.2.
adjusted to regulate the length of the text following semantic compression. If these semantic chunks still surpass the predetermined length, the identical procedure is repeated to acquire sub-level topic structures.
The obtained topic structures are tree-like, which can be flattened in accordance with the order of the original content. As per the model, each chunk is semantically centered around a specific topic, and these topics are mutually exclusive. Consequently, these chunks can be compressed independently by utilizing a pre-trained summarization model. Choosing from different pre-trained summarization models allows a trade-off between efficiency and effectiveness. Consequently, we can opt to selectively substitute the original chunks with the output of these pre-trained models to ensure the preservation of the underlying topic structure. The semantic compressed text can be forwarded to the LLM directly or in combination with other SoTA extension schemes to further enhance the overall outcome.

heading: 3.3 Complexity
text: To implement our method, two language models may be involved.2 The first one is the base LLM of interest whose context window is to be extended, and the other serves the purpose of semantic compression, which can be adapted from pre-trained
2Compared to the language model, the computational overhead of other components is significantly smaller. Please refer to Table 5 for more information regarding this.
summarization models. Since most of the current models are transformer-based, we can assume that their complexities are k1n2 and k2n2 respectively, where k1 and k2 are coefficients that depend on the number of the parameters of the models.
Empirically, the ratio between the summaries generated by the summary model and the original text remains relatively stable. Therefore, we assume a fixed compression coefficient α for the summary model. Given an input text with length L, the origin complexity is k1L2. Our method utilizes a divide-and-conquer strategy, dividing the long text into chunks of lengths l1, . . . , lm so that L = l1 + · · · + lm. During inference using the compressed context, the complexity is
k1(
m∑
i=1
αli) 2 = k1(α
m∑
i=1
li) 2 = α2k1L 2. (1)
Let γ1 and γ2 be the minimum and maximum input lengths of the summary model, where γ1 ≤ li ≤ γ2. Since mγ1 ≤ L, we have
m∑
i=1
k2l 2 i ≤
m∑
i=1
k2γ 2 2 = mk2γ 2 2 ≤ k2 γ22 γ1 L. (2)
Thus the complexity of compressing length-L context can be bounded by k2 γ22 γ1 L, which is linear in L. Furthermore, the summary models usually have smaller parameter sizes, and the entire compression process can be parallelized, allowing for further acceleration. The proposed compression significantly reduces the inference time for long texts. We also present the detailed empirical running time in Appendix D.

heading: 4 Experiments
text: We demonstrate that the proposed method of semantic compression can effectively extend the context window by up to 7-8 times without modifying the parameters of the pre-trained models. Furthermore, the semantic compression module can be seamlessly integrated into existing methods, allowing for further extension of the context window. This versatility enables our approach to be adapted and combined with other techniques such as YaRN, enhancing the overall performance and flexibility. To evaluate the performance of our method, we conduct experiments on several language tasks that require an understanding of long contexts. These tasks include passkey retrieval, single-document
question answering, multi-document question answering, summarization, and few-shot learning. In each task, the model is provided with a sequence of context C (typically lengthy texts) and a sequence of text Q (e.g., a prompt), and it is expected to generate the output answer A. Then we conduct the ablation study of the summary model to explore how much parameter work. We conducted an ablation study on the semantic compression module to investigate the parameter size at which the module becomes effective. Additionally, we also investigate the perplexity of generated text by our method in Appendix E.

heading: 4.1 Evaluating Tasks
text: We evaluate our proposed method on various standard benchmark tasks in the long context. Our length extension implementation is based on the pre-trained 7B LLaMA model (Touvron et al., 2023), where the context window size of this model is 4096. The semantic compression module utilizes bart-large-cnn and please refer to Appendix C for further details. We begin with the evaluation of passkey retrieval because this task can present the procedure of our methods. Then we assess the comprehensive ability on four kinds3 of natural language task from LongBench (Bai et al., 2023). Lastly, we conduct the ablation study of the semantic compression module based on LongBench.
Passkey Retrieval Retrieval has been an important application of LLMs. We evaluate the proposed method using a synthetic task for passkey retrieval introduced by Mohtashami and Jaggi (2023), where prompts are synthesized to conceal a generated passkey within a randomly chosen section of a long document. This task assesses the capacity to extract important information from any position within lengthy contexts. An illustration of the task is shown in Fig. 3. The synthetic long text incorporates the passkey digits, and the task for the LLM is to retrieve these digits from the input text. Further specifics can be found in Appendix B.
General NLP Tasks LongBench (Bai et al., 2023) is a multi-task benchmark designed for long text scenarios, consisting of six distinct
3Since our semantic compression module relies on a summarization model, the current implementation is not suitable for the remaining code and synthetic tasks in LongBench. We present the remaining results in Appendix A. Additionally, the summarization results for Llama2 7b are also provided in Appendix A.
tasks. In this study, we focus on the three English tasks from the set of four natural language tasks, namely single-document question answering, multi-document question answering, summarization, and few-shot learning. Each of the selected datasets contains 200 instances. Further information can be found in Appendix B.

heading: 4.2 Baselines
text: We choose SoTA solutions from each mainstream approach as our baselines.
Fixed-size chunking To accommodate long context within a fixed-size context window, chunking is a straightforward yet efficient approach. In NLPrelated applications, large pieces of text are usually broken down into smaller segments for targeted applications. When the input length exceeds the context window, the fixed-size chunking method (Bai et al., 2023) truncates the input sequence from the middle. This is because the most significant information typically resides at the beginning and end of the sequence.
Interpolation-based method YaRN (Peng et al., 2023) is a computationally efficient method for interpolating position encoding, which dynamically adjusts the Relative Positional Encoding RoPE over dimensions and scales the attention. YaRN offers multiple length-extended models for different versions of Llama2, with the models being trained on
a total of 64 GPUs from 8 × A100 machines. In order to ensure a fair comparison, we choose the model based on Llama2 7B, adjusted from 4k to 64k, as our baseline.
Fine-tuning approach LongLoRA (Chen et al., 2023c) is an efficient approach for fine-tuning that combines LoRA and shifts sparse attention to reduce computational costs. LongLoRA applies this technique to Llama2 models of different sizes, ranging from Llama2 7B, Llama2 13B, to Llama2 70B, with token lengths extended from 4k to 32k on a single 8 ×A100 device. In order to ensure a fair and unbiased comparison, we choose the Llama2 7B model with context extension achieved through improved LoRA fine-tuning as our baseline.

heading: 5 Results
text: In this section, we report the results of the above experiments along with a comprehensive analysis.
Passkey Retrieval We present the results of the passkey retrieval task in Fig. 4. We also showcase the compression results of our method for this task in Fig. 3. When employing Llama2 for passkey retrieval, we observe a rapid drop in accuracy to zero once the input length surpasses the window size of 4096. However, by utilizing our method, the retrieval accuracy of the Llama2 model remains above 90% even for inputs with lengths
of up to 30,000. This indicates that the semantic compression method extends the context window size of the language model by approximately 7-8 times. Furthermore, we combine our method with the SoTA interpolation-based method, YaRN, to further expand the context window size to up to 60,000, while consistently maintaining an accuracy above 90%. This demonstrates that our method can further extend the context window by integrating with current SoTA interpolation approaches.
General NLP Tasks We present our results on various general NLP tasks in Table 1, including single-document question answering, multidocument question answering, summarization, and few-shot learning. When the token length is less than 4k, there is no need to compress the context, and our method performs at the same level as the original Llama2 model. However, both the interpolation-based method YaRN and the finetuning approach LongLoRA negatively impact the performance of the Llama2 model across almost all tasks. In the 4k-8k range, our method outperforms others in 6 out of 11 tasks. It is worth noting that our model performs slightly worse in the few-shot learning task. This can be attributed to the fact that few-shot learning necessitates more detailed information, whereas our compression scheme maintains information within a fixed window. Moving on to the 8k-16k range, our method achieves the best results in 9 out of 12 tasks, exhibiting similar performance to the 4k-8k range. In the 16k-32k range, our method outperforms others in 8 out of 11 tasks. In the 32k+ range, other methods fail due to out-of-memory issues, while our method still maintains 70% of the performance achieved in the 4k range.
Ablation Study The summary model in the semantic compression module plays a crucial role. The larger the parameter size of the model, the better its ability to generalize, and the same principle applies to the task of summarization as well. In this section, we investigate the effect of parameter size on the performance of the semantic compression module. We adopt three models with parameter sizes of 406M for bart-largecnn, 320M for distilbart-cnn-12-6, and 60M for text_summarization, which we refer to as “Large”, “Medium”, and “Small”, respectively.
By only changing the summary model, we evaluate their performance on the English tasks in LongBench dataset. The results in Table 2 demonstrate that as the size of the summary model decreases, the average scores of the compression module built by the three summary models slowly decrease. Though the “large” did not achieve the best performance in the summarization task, it exceeded the average scores in the other three tasks. Even with the small model, the performance remains competitive. The above discussion indicates the robustness and reliability of our approach.

heading: 6 Conclusion
text: In this work, we propose a novel approach to addressing the limitation of context window length in large language models using semantic compression. By leveraging the statistical properties of natural language and exploiting redundancy in communication, we are able to significantly shorten texts while preserving their semantic meaning. This allows for a 6-8 time extension of the context window without the need for modifying the parameters of the pre-trained model or incurring additional computational costs. The experiments demonstrate that our proposed semantic compression method outperforms current SoTA approaches on the multitask long context understanding benchmark, Long-
Bench. Furthermore, the implementation of our semantic compression module is straightforward and can be easily integrated into other interpolationbased methods and black box APIs. This provides flexibility and adaptability to different downstream tasks, considering practical constraints such as time and memory resources. We believe our work can lead to a simpler context window extension method to be used in practice, thereby reducing the cost of large language models.

heading: 7 Limitation
text: The semantic compression method we propose is a lossy compression that retains only key semantic information. When faced with fine-grained tasks such as counting the occurrences of a specific name in the full text, our method may struggle to provide accurate answers. The current implementation just supports English since there are many effective summary models. We will introduce the multilingual version in the future.

heading: 8 Potential Risks
text: As a context window extension approach for large language models, our proposed semantic compression approach enables the model to process long texts that are 6-8 times the length of the original context window. The experiments and evaluations we conducted utilized publicly available academic datasets, thus avoiding direct ethical concerns. However, it is worth noting that our method could potentially be used to extract inferred privacy information from long texts in commercial settings. This is a common ethical concern associated with long text models.

heading: 9 Future Work
text: The core assumption of our compression approach is that long text remains redundant (Li, 2023). An apparent direction involves a task-agnostic compression strategy that considers the semantic information from task instructions. The evaluated tasks primarily focus on information retrieval from long texts. Reasoning is a fundamental ability of artificial intelligence systems (Wang et al., 2021; Yin et al., 2024b; Wang et al., 2024), and exploring performance on long text reasoning is well-deserved. The evaluation is based on automatic metrics in NLP, which may be biased. Therefore, it could be beneficial to conduct human evaluations through a
crowdsourcing platform or use GPT-4 scores (Sottana et al., 2023).

heading: A Other results
text: We present the results of all the tasks on LongBench dataset in Table 3. Compared with the baselines, it can be observed that our method performs well in Single-Doc QA, Multi-Doc QA, and Summarization tasks. We also present the performance of our method using prompting LLM as a neural compressor in Table 4. The summary prompts are the following. Although letting the model perform the summarization task itself can reduce the need for a separate summarization model, it generally results in average performance while increasing complexity.
Write a concise summary of the
following text. '''{text}'''SUMMARY:

heading: B Datasets
text: Single-Doc QA assesses the model’s ability to retrieve local information, while the summarization task requires the model to know the full text. Multi-Doc QA involves answering multihop questions (Wang et al., 2023b; Yin et al., 2024a) across long document content, which poses a significant challenge as it requires sophisticated reasoning abilities (Fei et al., 2024; Yin et al., 2023; Wang et al., 2021). However, there is still room for improvement in Few-shot Learning, Code Completion, and Synthetic tasks.
Single-Doc QA
• NarrativeQA (Kočiskỳ et al., 2018) is a standard question-answering dataset that includes books from Project Gutenberg3 and movie screenplays from a list of websites. Question-answer pairs were provided by annotators, so that each of the 1,567 books and scripts has about 30 questions and answers, and two reference answers are given for each question.
• Qasper (Dasigi et al., 2021) is a question-answering dataset of NLP publications containing abstractive, extractive, and yes/no questions.
• MultiFieldQA-en (Bai et al., 2023) is a dataset created from multiple sources including legal documents, government reports, encyclopedias, and academic publications. Doctoral students were requested to annotate each article’s queries and responses.
Multi-Doc QA
• HotpotQA (Yang et al., 2018) includes many 2-hop questions written by native speakers based on two related paragraphs.
• 2WikiMultihopQA (Ho et al., 2020) involves up-to 5-hop questions systematacially constructed by manual templates. Answering these questions requires reasoning paths and can not be solved by local content.
• MuSiQue(Trivedi et al., 2022) consists of up to 4-hop questions and removes shortcuts and naturalness questions. Each question contains 2-4 supplement paragraphs which present the reasoning path and related paragraphs.
Summarization
• GovReport (Huang et al., 2021) collects detailed reports containing human-written summaries from the U.S. Government Accountability Office and Congressional Research Service. These reports span a wide variety of national policy issues.
• QMSum (Zhong et al., 2021) contains annotated meeting-summary pairs across many domains including including product, academic, and committee meetings.
• MultiNews(Fabbri et al., 2019) is a multi-document summarization dataset. (Bai et al., 2023) cluster 2-10 news articles discussing the same event or topic, each paired with a human-written summary and form a new long text summarization task.
Few-Shot Learning To construct few-shot learning with long text, (Bai et al., 2023) select a range of training examples in the following datasets to concatenate the context in LongBench.
• TREC (Li and Roth, 2002) is a classification dataset with fine-grained class label.
• TriviaQA (Zhong et al., 2021) is a classification dataset and involves messenger-like conversations with humanwritten summaries.
• SAMSum (Fabbri et al., 2019) reading comprehension dataset and consists of question-answer pairs annotated with evidence passages.
Passkey The randomly generated prompts of the passkey retrieval task are in the format of Fig. 5.
The datasets we discussed are openly available for academic purposes and are licensed under CC-BY 4.0. Our proposed method is also licensed under CC-BY 4.0 and can be used for academic purposes. However, it is important to note that derivatives of data accessed for research purposes should not be deployed in the real world as anything other than a research prototype, especially for commercial purposes.
C Implementation Details In this section, we provide details of our algorithm implementation. Our experiments can be conducted on a single V100 GPU with 32GB of memory. Each task on LongBench takes approximately one hour to complete.
C.1 Open-source Models Our algorithm utilizes several mature open-source models. For graph representation, we make use of the sentence similarity models all-MiniLM-L6-v2 provided by the Sentence Transformer platform, which can be found at the following link: https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2. For semantic compression, we employ the pre-trained model bart-cnn-1264. In most of our experiments, we utilize Llama2-7B-chat-4k as the base large language model (Touvron et al., 2023).
C.2 Clustering Algorithms In our clustering implementation, we utilize Spectral Clustering as a standard method for graph clustering. For example, when there are two clusters, Spectral Clustering solves a convex relaxation of the normalized cuts problem on the similarity graph. It achieves this by dividing the graph into two parts in such a way that the weight of the cut edges is minimized relative to the weights of the edges within each cluster. You can refer to https://scikit-learn.org/stable/modules/ clustering.html#spectral-clustering for more details. This approach aligns closely with our objective of identifying topic structures. Furthermore, this method is both simple and effective, as it only requires one parameter: the cluster number. This parameter allows us to control the length of topic-based chunks.

heading: D Running Time
text: We present the running time analysis of three datasets in MultiDoc QA in Table 5. The average word lengths of 2wikimqa,
4Available at: https://huggingface.co/facebook/ bart-large-cnn
hotpotqa, and musique are 4,887, 9,151, and 11,214, respectively. These lengths represent word counts, not token counts, and are sourced from LongBench. However, it is important to note that due to the limitation of a single GPU, we can only process sequences up to 8k in length, which makes direct inference with yarn more advantageous. The results in the table indicate that our proposed approach has a longer inference time compared to direct inference, with the main time consumption occurring during the compression process. Nevertheless, the compression step can be parallelized, leaving room for engineering optimizations. It can be observed that the runtime of the clustering algorithm is relatively minimal compared to other modules, thus we did not delve into detailed discussions on it.

heading: E Fluency
text: We evaluate the fluency of our semantic compression method using the perplexity score, which is defined as the exponential of the average negative log-likelihood of the probabilistic model P on the distribution D, i.e.,
PPL(D,P ) := exp(−Ex∈D logP (x)).
A smaller perplexity score indicates more fluent sequences that are consistent with the model. We utilize the Llama2 model as our baseline to evaluate the fluency of generated texts by calculating the perplexity (PPL) score. Samples from the GovReport dataset are selected at varying lengths, and the reference texts are compared to the generated texts during
the computation. In cases where the length of the input text exceeds the context window of Llama2, our semantic compression module shortens the input, thereby allowing the model to continue generating new content fluently. The resulting scores are depicted in Fig. 6. The plots indicate that the perplexity of Llama2 initially decreases, but once it surpasses the window length, it rapidly increases. However, when our semantic compression method is employed, the PPL remains consistently low. This suggests that our approach successfully extends the context window up to three times without compromising the generation quality of the language model.
(Peng et al., 2023), which measures the model’s ability to predict the text and serves as an indicator of the fluency of the generated output. This analysis allows us to assess not only the effectiveness but also the quality of the generated output.

heading: Future Work
text: We will introduce the multilingual version in the future