Compressing Context to Enhance Inference Efficiency of Large Language Models
Authors: YUCHENG LI
BO DONG
Frank Guerin
Chenghua Lin
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Input Compression

Abstract: Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length.  This paper proposes a method called \textit{Selective Context} that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used.  Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

Compressing Context to Enhance Inference Efficiency of Large Language Models
Yucheng Li', Bo Dong', Frank Guerin', Chenghua Lin
Department of Computer Science, University of Surrey, UK
Department of Computer Science, The University of Manchester, UK Department of Computer Science, The University of Sheffield, UK
(yucheng.o 1i, bd00531, f. guerin/ @surrey. ac uk chenghua.. 1 Limemnnceetter.ae ac..beeeeeerree...u....
Abstract
Large Languages Models (LLMs) have shown their ability to perform new tasks, resulting in line of work that focuses on further scaling these models. These efforts are based on the assumption (thest-mree parameters will lead te-beeter performeeef
Context:
Large language models (LLMs) achieved re- markable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due significantly increased computational re- quirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Con- text that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conver- sations, tasks of summarisation, question answering, and response generation. Experi- mental results show that Selective Context sig- nificantly reduces memory cost and decreases generation latency while maintaining compa- rable performance compared to that achieved when full context is used. Specifically, achieve a 50% reduction in context cost, re- sulting in 36% reduction in inference mem- ory usage and a 32% reduction in inference time, while observing only minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance. Code and data are available at /sttps://githut com/ Soncchnngo//Seeettivee ext
Query: What's the assumption behind the efforts to further scale LLMs?
LLMs: Further scaling Large Language Models will lead to better performance on a wide range of tasks.
Figure I: Some context is redundant because L.MS have learned that knowledge. LLMs can generate the correct answer even when these redundancies are deleted.
tal in scenarios such as having long conversations, document summarisation, and question answering given long documents. However, it is very compu- tationally expensive, particularly with Transformer based LLMs, due to the quadratic growth of mem- ory and computation associated with the 2-D at- tention matrix (Vaswani et al., 2017). This makes LLMs less accessible and sometimes leads con- text truncation during inference. Moreover, due to the above limitation, existing LLMs were usually pre-trained with fixed-context windows, which fur- ther constrains their capability in processing longer context.
There active attempts in reducing the compu- tation and memory cost of the Transformer archi- tecture with sparse attention (Child et al., 2019) or local dense attention (Beltagy et al., 2020). There are also efforts to learn soft prompts with further distillation to save context cost during inference (Mu et al., 2023; Chevalier et al., 2023). In contrast to existing approaches that primarily focus on ar- chitectures distillations, we introduce fresh per- spective tackle the redundancy in the input con- text itself, thus proposing a complementary, model- agnostic approach that can be potentially combined with other architecture optimisation methods to fur- ther enhance inference efficiency.
1 Introduction
Large language models (LLMs) have demonstrated remarkable power and impressive generalisation abilities across a wide range of natural language processing tasks, as well as reae-life applications (Brown et al., 2020; Touvron et al., 2023; Bubeck et al., 2023). However, major challenge for ex- isting LLMs is processing longer context. Deal- ing with longer context with LLMs is fundamen-
Corresponding author

---

The proposed method is motivated by the po- tential redundancy and repetition in human lan- guage, which has two main sources. The first is the inherent redundancy of natural language. For example, in the conversation "A: Did you get the chance to pick up groceries today?", "B: Yes, I did get the groceries." ", the underlined part can be seen common redundancy in communication. Linguistic studies suggest redundaney is ubiqui- tous in language (Wit and Gillette, 1999). The other type of input redundancy from the overlap with training material. As the example in Fig. I shows, some parts of input have already been included in the pre-training stage of LLMs, then it is safe to delete them and the model can still gen- erate the correct answer. In summary, redundancy in the input context, while beneficial for human comprehension. can be superfiuous for LLMs and might lead unnecessary computational expense.
corresponds to its output distribution. So the self- information of token can be defined as the nega- tive log likelihood:
(xx) log, P(zz [T, (1,,,,,111... T1, Zt-
(1)
where (xx represents the self-information of token
x and P(x) denotes its output probability.
In information theory, self-information mea- sures the level of surprise or uncertainty associated with an event; rare events convey more informa- tion and thus have higher self-information, while common events convey less information and have lower reff-information. In the context of language modelling, self-information can be used to assess the informativeness of lexical units, e.g., words, phrases, or sentences. Lexical units with lower are less informative and thus are more likely to be inferred from the context. As sult, we may treat these parts of input redundant during LLM inference.
In this paper, we propose Selective Context, which prunes redundant content in given input context, thereby reducing the computational cost and making better use of the fixed context length in LLMs. Selective Context evaluates informativeness of lexical units (i.e., tokens, phrases, or sentences) with self-information (Shannon, 1948) computed by a base causal language model. By selectively retaining content with higher self-information, our method provides more compact and efficient con- text representation for LLMs to process without compromising their performance various appli- cations.
In NLP, self-information has been used measure surprise in creative language artefacts (Bunescu and Uduchi, 2022). In addition, related concepts of self-information such as entropy and perplexity are widely used in language model opti- misation and evaluation.
H(S) N Ex)(x))
(2)
PP(S) (S)
(3)
where the entropy H(S) of the sentence S (ro, ..., F,)) the average self-information of words in the sentence, and perplexity PP(S) of the sentence can be calculated with entropy. The property of self-information that is especially rele- vant to our method is the additivity.
We evaluate the effectiveness and different set- tings of Selective Context on arXiv papers, BBC News, and real conversation on ShareGPT.com with four NLP tasks: summarisation, question an- swering, original context reconstruction, and con- versation. Experimental results demonstrate that our proposed method can significantly enhance con- text efficiency of LLMs during inference while maintaining comparable performance compared to that achieved when full context is used.
I [zz. æ1) = log2 P(ro,
log2 ((zo))((1ll
(4)
log, P(xo) - log, P(zı(z0)
= (xT)
This means we can calculate the self-information of a lexical unit by simply summing the self- information of the tokens in it.
2 Self-Information
Self-information, also known as surprisal or in- formation content, is fundamental concept in information theory that quantifies the amount of information conveyed by an event given distribu- tion (Shannon, 1948). In the context of language modelling, the event can be regarded as one step of generation (i.e., a token) and the distribution
3 Method
Selective Context optimises the input context by filtering out redundant or non-essential content reduce computational cost and make better use the limited context window. In implementation,

---

Original: INTRODUCTION Continual Learning CL ) also known as Lifelong Learning a promising learning paradigm to design models thett have to Heurn how to perform moltiple across different environments over lifetie To uniform the language and euhance the-theeeeee readability of the-peper we adopi the unique term continual learning (Cb). Ideal CL models in the real world should -be deal -with domain shifts researchers hhvve recently started to |sampl tasks from Lwo different datasets. For instance. proposed train and evaluate a model Imagenet [irst and then challenge #t-perffommance - the Places365 datavet considers more scenarios starting with Imagenet or Places363, and then moving on to the VOCICUB/Scenes datasets. Few works propose more advanced scenarios built-en top-ef more than two datasets-
Filtered: INTRODUCTION Continual Learning promising learning paradigm design models
have to how across over To uniformm the language and enhance adopt the unique term continual learning
Ideal CL models in should deal domain shifts researchers recently started sample tasks
Iwo different datasets For instance proposed train and evaluate on Imagenet first challenge
Places365 considers more scenarios starting Imagenet or Places365 the VOC/CUB/Scenes datasets Few
works propose more advanced scenarios built top more than two datasets
Figure 2: A visualisation of selective context. Darker colour indicates larger value of self-information.
we first 1) employ a causal language model such as GPT (Radford et al., 2019; Brown et al., 2020), OPT (Zhang et al., 2022), or LLaMA (Touvron et al., 2023), computing self-information for each token. We then 2) merge tokens, along with their corresponding self-information values, into lexical units, which can be phrases or sentences. This step is optional if tokens are being used as the basic units. Finally, 3) we eliminate content that is deemed least necessary to render the input more compact.
to the additivity property of self-information:
[(u)= Σ (ax))
(6)
The NLTK sentence tokenizer employed to obtain sentence level lexical units. And we use spacy' to merge tokens into noun phrases. We do not merge verb phrases as it might produce very long phrases.
3.3 Selective Retention of Informative Context
With the self-information of each lexical unit com- puted, we can now evaluate their informativeness. Instead of using fixed threshold retaining a fixed number of top k lexical units, we design a filtering approach adaptively select the most informative content.
3.1 Computing Self-Information
Given a context C TD, Ii, Tn. where I; de- notes token, we use base language model M to compute the self-information for each token 24 as follows:
percentile-
First, we rank the lexical units based on their self-information values in descending order. Then, we compute the p-th percentile of self-innonnttion values among all lexical units.
1 [I;;) = - logg P(ra)ao, 21,..,,,; 12- (5)
3.2 Merging into Lexical Units
Ip np. ww..ppp.pee.iiee (ww.pereeeeee((((e)))) [(x)),
If the content filtering of selective context is di- rectly performed on the token level, it might lead to very disjoint context. Therefore apart from token level filtering, also conduct the filtering proce- dure on phrase and sentence level. We call a basic unit in our filtering lexical unit, which could be a token, a phrase or a sentence in our setting.
(7)
Next, we selectively retain lexical units with self- information values greater than or equal to the p-th percentile, constructing a filtered context C²:
C' U, (UU) 4p, 1iisn
(8)
To enable selective context to work phrases and sentences, we merge tokens and their self- information into lexical units. Each lexical unit consists of multiple tokens (Izt..., Zt a), and we can calculate its self inf-infoomaaton by summing the ielf-information of its individual tokens according
The percentile-based filtering a more flexible approach to retain the most informative content depending on the distribution of self-information values in the given context. In Figure 2, we present
'nttps:/ /tt:://ppppccc 1o/ap// pipeline- -oe-ffucctioon + mmwrgeeergguuu www__oooonuuuueeeeeee

---

an example on phrase level where p is set to 50, which means half of phrases are filtered out. In this case, the context after processing by selective context only retains 57.2% of tokens, which saves 42.8% of context length.
the release of all LLMS in our experiments. Con- sidering some of baseline models are continually being updated, we employ the latest version re- leased before 30 March 2023 to make sure models have never seen our test set their pre-training and fine-tuning stage. In addition, as some of LLMs in our experiments have max_length of 2048 tokens, we do not include articles or conversations exceeding this length.
4 Experiments
The goal of Selective Context reduce the redun- daney in the input context without compromising the generation quality LLMs. As a result, we are expecting the answers given both selective context and the original context to be as close as possible. We take the generated answer given full context the reference answer, and compare to the gen- erated answer given the selective context in our experiments.
4.2 Models
We test Selective Context the following models: GPT-3.5 and GPT-4: GPT-3.5 also known as Chat- GPT, which is likely to be further fine-tuned from GPT-3 and InstructGPT. GPT-4 is the latest model from OpenAI, which has demonstrated substan- tially improved capability on complex reasoning compared to its predecessor. GPT-3.5 and GPT- 4 are unfortunately not open-source, we can only access these models via web api?.
4.1 Datasets
Selective Context prunes redundancy in the input context to allow very long context processing for LLMs. However, existing benchmarks for LLMs, such as MMLU (Hendrycks et al., 2020) and ARC (Clark et al., 2018), are mostly single round ques- tion answering and are thus not suitable to evaluate our proposed method. Therefore, we collect three test sets consisting of long documents and conver- sations evaluate Selective Context. Statistics in detail are presented in Table 4.
LLaMA-7B, 13B, 30B: LLaMA is a family of open-source language models released by Meta, which is reported outperform GPT-3 with less parameters. The LLaMA family includes models with size ranging from 7B to 65B. To investigate the effect of scaling law to Selective Context, we experiment with LLaMA with 7B, 13B, and 30B parameters.
Vicuna-7B, 13B: Vicuna (Chiang et al., 2023) is family of open-source language models instruct- tuned from LLaMA. According their technical report, Vicuna models perform quite well on a list of multitasking benchmarks.
BBC News: A dataset containing news articles col- lected from the British Broadcasting Corporation (BBC). This dataset covers wide range of topics, including politics, business, sports, and technology. We use the full content of each news article in our experiments.
4.3 Tasks and Metries
arXiv Articles: dataset consisting of latest aca- demic papers, spaning various scientific disciplines, such as computer science, physics, and mathemat- ics. As arXiv articles can be quite long, we only process the first two sections (usually introduction and background) for each paper in our experiments. ShareGPT.com: ShareGPT.com is platform where ChatGPT users share their surprising and in- teresting conversation with ChatGPT. This dataset consists conversations in different languages and in various scenarios (c.g., coding, chitchat, writing assistant, etc.). We use the ShareGPT dataset for the conversation task in our experiments.
We evaluate Selective Context on four tasks: Original Context Reconstruction: Given a com- pressed context produced by Selective Context, this task aims to evaluate whether models are able to reconstruct the original context. This task assesses how well the filtered context retains the essential information from the original context. In exper- iments, the compressed contexts are used as input, and the original contexts are used as reference an- swers.
Given a context, the task is to generate summary that captures the main points of the document. This task aims to evaluate whether Selective Context affects the overall un- derstanding of models on the input contexts. In
Summarisation:
The three evaluation datasets were created care- fully to avoid data contamination. Data samples in the BBC News, arXiv, and ShareGPT.com datasets were all created after March 2023, which is after
Pnttps://pla form. openal. com/ com/docs/ docs/ api - rreference

---

our experiments, the input and output the com- pressed context and the summaries generated based on the compressed contexts. Summaries based on the original (full) contexts are treated as the refer- ence answers.
ence answers. In our experiment, we set the number of reference answers to
4,4 Experimental Settings
We use the smaller base causal language model for self-information computing our experiments. For the I.LMMA family and vicuna family, we em- ploy LLaMA-7B to compute self-information. For the OpenAl family, we use a smaller GPT-3 variant curie for self-information computing, which is available on OpenAl web API. In self-information computing, we do not process the entire context at once. This is due to observation the ten- dency of LLMs give later lexical units lower self- information. Instead, we compute self-information sentence by sentence our experiments.
Question Answering (QA): Given a document and a set of questions, the task is to generate answers based on the information available in the document. This task aims evaluate models' understanding of specific query. Here we first generate questions and answers based on the original context, where these answers are treated reference answers, and then ask LLMs to answer these questions with se- lective context.
This task is only for the ShareGPT dataset. Given a conversation history and a user query, the task is to generate a response to the query based on the previous conversation history. This task aims evaluate selective context's perfor- mance on conversation. Specifically, ask LLMs answer the users' last query of ShareGPT con- versation instances with selective context applied on the previous conversation history.
Conversation:
In our experiments, we compare the two dif- ferent dimensions that are adjustable in Selective Context.
We experiment with differ- ent content reduction ratios in Selective Context: 0.2, 0.35, 0.5, 0.65, and 0.8. These ratios determine the proportion of content to be filtered out, allow- ing us to study the trade-off between efficiency and performance as the amount of retained information varies.
Compression Ratios:
We employ four metrics to assess the perfor- mance of our models on the tasks: BLEU, ME- TEOR, ROUGE, and BERTScore. BLEU (Pa- pineni et al.. 2002) calculates n-gram precision, which is the proportion of n-grams in the generated text that also present in the reference text. ME- TEOR (Banerjee and Lavie, 2005) takes additional features such as synonymy, stemming and word or- der into consideration, which leads to more compre- hensive evaluation. ROUGE (Lin, 2004) focuses on how much of the important information in the reference text is present in the generated summary. BERTScore (Zhang et al., 2019) leverages con- textualised embeddings from pre-trained language models like BERT, computing the cosine similar- ity between the generated text and reference text embeddings to capture semantic similarity more effectively than traditional n- n-gram-basee metrics.
Lexical Units: Lexical units the basic element of content reduction in Selective Context. It can be sentence, phrases, or tokens. As mentioned in $3.2, we remove the redundancy in input context by specific lexical unit level.
Results
Except for $5.5, all results of selective context pre- sented are at the phrase level (the optimal).
5.1 Overview
In Table 1, we first compare the performance of Selective Context against the Original Context to how well Selective Context preserves useful in- formation when redueing context cost. The metrics are averaged across all models mentioned in $4.2. The performance drop is shown in parentheses. As demonstrated in the table, using Selective Context only leads to a marginal drop when the reduction ratio is set to 0.2 or 0.35, despite sig- nificantly reducing the context cost. The BLEU score drops by only 0.05 when 20% of the con- tent is reduced. And the number even smaller when it comes to ROUGE-I, where the drop is just 0.03. This indicate a high level consistency be-
As mentioned before, we use the generated an- swers given the full contexts as the reference an- swers. When testing the deterministic decoding strategy (greedy decoding), we take one sin- gle run on full context the reference answer. When testing the non-deterministic decoding strat- egy (t (teperature times on full context to obtain multiple reference
7) we run multiple
answers to address the randomness in decoding. The metrics are computed based on the set of refer-

---

ROUGE
BFRTScore
Method
Ratio
BLEU
METEOR
Rouge-1
Ruuge-2
Rouge-L
Precision
Recall
F1
Original
.347
.496
.511
.383
.471
.910
.909
.909
0.2
.295 (.05)
.460 (.04)
.540 (.03)
.346 (.04)
.438 (.03)
.905 (.005)
.900 (.009)
.922 (.007)
0.35
.243 (.10)
.421 (.08)
.504 (.07)
.294 (.09)
.396 (.07)
.900 (.010)
.894 (.015)
.897 (.013)
Selective
0.5
.179 (.17)
.362 (.13)
.449 (.12)
.237 (.15)
.344 (.13)
.893 (.018)
.82 (.027)
.887 (.023)
Context
0,65
.127 (.22)
.299 (.20)
.391 (118)
.178 (.21)
.287 (.18)
.885 (.025)
.870 (.039)
.877 (,032)
.070 (.28)
.224 (:27)
.311 (.26)
.122 (.26)
.225 (.25)
.874 (.036)
.852 (.057)
.863 (.047)
Table Comparing Selective Context to the Original Context with temperature set to 0.7.
ROUGE
BERTScore
Ratio
BLEU
METEOR
Rouge-1
Rouge-2
Rouge-L
Precision
Recall
F1
Random deletion
0.20
0.437
0.578
0.666
0.503
0.566
0.892
0.909
0.899
0.35
0.360
0.514
0.629
0.423
0.502
0.879
0.895
0.886
0,50
0.443
0.346
0.432
0.867
0.881
0.65
0.210
0.378
0.522
0.279
0.371
0.855
0.868
0.860
0.80
0.156
0.314
0.450
0.219
0.310
0.843
0.853
0.847
Selective Context
0.20
0.527
0.643
0.714
0,585
0,631
0.930
0.932
0,931
0.35
0.446
0.588
0.679
0.508
0.570
0.915
0.916
0.915
0,50
0.528
0.642
0.425
0.501
0.900
0.902
0.900
0.65
0.244
0.418
0.557
0.315
0.404
0.886
0.877
0.881
0.80
0.160
0.328
0.464
0.223
0.319
0.875
0.858
0.866
Table Comparing Selective Context the random deletion baseline when using greedy decoding.
tween answers given selective contexts and original contexts when the reduction ratio is 0.2. Selective Context also yields impressive results when 35% of the content is reduced, with BERT scores around 0.9 and ROUGE-1 scores over 0.5. The drops be- come noticeable as the reduction ratio rises 0.5, where the average BLEU score drops 0. and the average ROUGE-1 drops 0.12. reduction ratio 0.65 and 0.8 tends to be less valuable, as shown
Ratio
#Sorry
Answer len.
Unfaithfulness
Full
160.3
0.2
156.5
.027
0.35
136.0
.050
0.5
140.2
.038
0.65
131.2
.051
0.8
27
103.7
.086
Table 3: Faithfulness test gpt-3, 3. 5-turbo using selective context.
by the 0.18 drop on ROUGE-1 and 0.32 drop on BERTScore-F1.
dom baseline with only 20% content compression.
We then compare Selective Context against the Random compression baseline shown in Table 2. We observe that using Selective Context allows LLMs generate very similar answers to the refer- ence answers (answers given full context) although we significantly reduce the context cost. Selec- tive Context maintains BERTScorr-FI above 0.9 when the compression ratio is 0.5 or lower, which shows a high similarity with the reference answers. ROUGE demonstrates the same trend: ROUGE-1 continues to be above 0.64 and ROUGE-L keeps above 0.5 when the ratio is under 0.5. We also notice that Selective Context is significantly more effective than the random baseline: Selective Con- text with compression ratio 0.5 shows a better overlapping with the reference answer than Ran-
5.2 Faithfulness
To evaluate to what extent selective context affects the faithfulness of the LLMs generated content, we perform manual tests on our question answering results based on the idea of Wang et al. (2020). We evaluate 1000 question-answer pairs (200 for each ratio) with the following procedure: 1) We first extract OpenIE tuples from the answers of selective context, and then 2) manually evaluate whether each tuple is entailed by the reference answer. If the model's answer is "Sorry, don't know", we treat it as "Sorry" cases and do not consider it as unfaithfulness.
As shown in the Table 3, we find that gpt-3 -3.5 tends to generate shorter answers or refuses to an-

---

co
Iecenstn.ction
Mettoor
Roupel
0.7
0.5
0.2
0.3
0.25
2.53
0.50
0.00
0.00
0.20
0.:00
0.30
065
0.80
0.20
0.35
5.96
0..55
0.40
0.20
0.43
0.50
0.65
0.5
RougeL
Bertsoore Procision
Bersscone
Bertscore
6.900
0.500
6.925
0.925
0.000
0.000
5.015
0.875
0.000
8.600
0.850
5.025
0.50
020
0.25
0.50
045
0.00
020
0.25
0.52
0.50
0.05
0.02
Figure 3: Performance of selective context on different tasks. x aaxix represents compression ratios (same helow).
lame-300 vicuna 13b
vicune_13b
BERR
lama 13b
Phrase
Human Evall
0.00
0.9
0.8
0.86
0.7
acceStacee
0.6
0.82
0.78
0.4
0.20
0.25
0.50
0.55
0.3
Figure 5: Effects of lexical units.
0.2
6.20
0.35
0.50
0.65
0.80
original context when the reduction ratio is lower than 0.5, as demonstrated by rather high ROUGE score. Second, we notice that the curves of question answering and summarisation decrease gradually and are continually higher than those of the other two tasks evaluated by BERTScore. We could say Selective Context especially suitable for tasks of summarisation and answer generation.
Figure 4: Acceptance rate of generated summaries.
swer the questions if it fails to identify necessary evidence in the given selective context. With compression ratio of 0.65, gpt-3. 5 refuses an- swer 19 questions (9% of 200), and the answers are 35% shorter than the reference answer (131 tokens in average). However, selective context docsn't 't significantly affect the faithfulness across all com- pression ratios. About 3,8% of all tuples are not entailed by the reference answer when the compres- sion ratio is 0.5, and this number rises slightly to 5.1% the compression ratio increases to 0.65.
5.4 Scaling and Instruct-Tuning
We perform human evaluation to explore the ef- fect of model scales and supervised instruct-tuning on Selective Context. We asked three college stu- dents to evaluate 1150 generated summaries from 1 ama and vicuna (about 55 per model and ra- tio) and record whether they accept the generation as reasonable summary. As shown in Figure 4, we find no specific trends between the scales and generation quality given Selective Context.
5.3 Tasks
In this part, we break down and analyse the perfor- mances of Selective Context in the four different NLP tasks: summarisation, question answering, original context reconstruction, and conversation. The results as shown Fig. 3. First, the re- sults on the Original Context Reconstruction task (RC) show the steepest drop with increasing com- pression ratio, however, Selective Context allows LLMs to preserve most of the key points in the
The vicuna family demonstrates similar sum- marisation capability with 7b and 13b parame- ters. And so does the llama family, larger models do not show stronger robustness towards Selec- tive Context. But instruct-tume tuunu model vicuna demonstrates significantly superior performance

---

Original Context, CUDA Memory 77, 695 MB; Time 110.8 ms ms/token
Please see the original document and summary given full context Appendix B.
Selective Context, Ratio: 0.5, CUDA Memory = 61, 885 MB, Time = 76.3 ms ms/token, Time to construct selective context 46.1
The above paragraph discusses the use of probabilistic methods, safety distance- control methods, and trajectory prediction methods for assisting vehicles in avoiding collisions. It mentions that CNN has made outstanding contributions vision tasks, particularly in road condition inspection, due to its excellent regional feature extraction capabilities. However, the performance CNN-based models for vision tasks is inferior to EfficientNets RepVGG, due to the huge differences between sequential tasks in NLP and image tasks in CV, the difficulty in keeping the original information of inputs after RNN layers, and the computational and memory requirements of switching layers. The paragraph introduces network structure called Sequential Spatial Network (SSN) blocks, which overcomes the limitations of traditional CNN-based models. The SSN block consists of convolutional layers, Upsampling Convolutional Decreasing (UCD) blocks, and Reinforcement Region Unit and Fast MultiHead Self- Attention (FMHSA) to enhance local information and improve normalization capability. The paragraph concludes by stating that the SSN network outperforms existing methods on the Lykit dataset and can be easily transferred for image classification tasks.
Figure 6: Comparing the summary generated by vicuna 13 given original context and selective context.
than 1 ama models given selective context indi- cating mustruct-tuning might help the model to be more robust against context compression. Given se- lective context, 1 alama models often fail to follow instructions and go wild very easily.
and performing lexical unit tokenisation. This en- sures that the initial stage of establishing a selective context incurs very little overhead. Secondly, it shows selective context significantly reduces the memory usage of the GPU and accelerates the gen- cration process. With compression ratio of 0.5, selective context reduces about 36% of the mem- ory cost in inference and makes generation 1.32 times faster (per token). By comparing the content of the two summaries, we see that the summary given selective context missed relevant information about the research background (as denoted by the [1] marker), such as the use of machine learning in autonomous driving technology and instead starts with the different methods directly. This is due to the background parts not being selected and moved as redundancy before feeding vicuna. We tried to ask vi cuna
5.5 Lexical Units
We test the effect of Selective Context based on different lexical units: tokens, phrases, and sen- tences via BERTScore-F1. As shown in Table 5, employing phrase as the basic lexical units in Selec- tive Context is the optimal approach, consistently outperforming the other two variants, followed by toven-level Selective Context. Removing redun- dancy at sentence-level is a rather unstable imple- mentation compared to the token and levese-level. This experiment indicates that a reasonable granu- larity can be crucial in Selective Context.
"what is the background of this study?", given the selective context, and obtained decent answer:
5.6
Case Study
To have a straightforward impression on how well LLMs generate with selective context, we present two summaries given the full and selective context respectively in Figure 6. The original document and processing to obtain selective context are pre- sented in Appendix B.
"the research buckground of this paper is likely to be situated in the domain of autonomous driv- ing technology and the application of artificial in- telligence (AI) for improving vehicle safety and decision-making capabilities.".
We first found that preparing selective context is extremely efficient. takes one-time cost of 46.1 ms to build selective context for the example para- graph, which includes computing self-information
This demonstrates that LLMS are likely be able to infer the deleted parts of background infor- mation in the selective context. Selective context also affects vi cuna't 's decision on what informa-

---

tion should be included in the summary the sec- ond summary includes details about, for example, FMHSA and UCD block (as denoted by the [2] marker) which are not covered in the summary gen- erated with the full context. We find no factual errors in the summary given selective context.
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1377197-1
Séhastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4, arXiv preprint arXiv:2303.1271:
6 Conclusion
We introduced Selective Context improve the context efficiency of LLMs in inference by deleting redundant content measured by self-information. Our extensive experiments on arXiv papers, BBC news articles, and conversation transcripts showed that our proposed method can significantly reduce GPU memory cost, accelerate generation with mi- performance decrease, and potentially enable LLMs handle long documents and extended con- versations without the risk of context truncation.
Razvan Bunescu and Oseremen O Uduehi. 2022. Distribution-base measures of surprise for creative language: Experiments with humor and metaphor. In Proceedings of the 3rd Workshop Figurative Language Processing (FLP), pages 68-78.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. Adapting language models to compress contexts. arXiv preprint arXiv:2305.14788
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, Ion Stoica, and Erie P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
7 Limitations
Selective Context demonstrates promising results, but it is still necessary to note a couple of poten- tial limitations. Firstly, our approach is somewhat influenced by the phrase boundary detection pro- cedure. We employ the noun phrase tokenisation algorithm provided by spacy our experiments. However, we do not consider verb phrases as there is no mature solution for verb phrase tokenisation. We speculate that we can achieve better compres- sion performance with dependency free-based fil- tering procedure which might lead to better bound- ary identification of lexical units. Secondly, in the experiment section, we percentile control the pruning process. However, the optimal com- pression percentile varies based on specific tasks and context. Developing a tool to find the optimal threshold can further enhance the effectiveness of selective context.
Rewon Child. Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904.10500
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv, Abs/18033
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Mcasuring massive multitask language understanding. ArXiv, abs/2009.03300
Yucheng Li. 2023. Unlocking context constraints of 1mss: Enhancing context efficiency of llms with self-information-bee based content filtering. ArXiv, abs/2304.12102.
References
Chin- Yew Lin. 2004. Rouge: A package for automatic
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the aci workshop intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
evaluation of summaries. In Text summarization branches out, pages 74-81.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. arXiv preprint (ah::2304.08466
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The document transformer. arXiv preprint arXiv:2004.055550
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-33 111318.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, liya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
C
The Previous Version of Selective Context
If you're looking for the previous of the paper, please check (Li, 2023).
Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal. 27(3):379-422
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397 139711.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processsin systems, 30.
Alex Wang. Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evalu- ate the factual consistency of summaries. ArXiv, abs/2004.04228.
Ernst-Jan C. Wit Marie Gillette. 1999. What linguistic redundancy. University of Chicago.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 2250002005
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0965.
A
Dataset statistics
Dataset
#Doc
#Sent
#Phrase
#Token
Arxiv
408
28.20
514.55
864.85
ShareGPT
470
27.35
389.42
689.32
BBC
294
25.63
523.96
732.54
Table Statistics of the three datasets. #Sent, #Phrase, #Token are averaged per document.
B Example selective context on long context
Here present example of selective text on rather long context. The original para- graphs is from https: papps://arxte /ttp///avvvv//////////eee//aaaal org/abs/ 2303. 07352. The original paragraphs is shown in Fig. 7. The resulting context shown in Fig. The reference summary is given in Fig. 9.

---

INTRODUCTION In the-past many y researchers have foensed on how to turn vehicles from assisted driving to more intelligent autonomous driving : Due to the iteration ef intelligent hardware -and the improvement -ef chip computing power; -aaaee----rggaaane taggee-monntt of dataa collected by syy-eeor can be quickly converted and fed into models to make decisions In the driving process; the safety factor is the first consideration for users and researchers Therefore, how AV should avoid collisions has beeeme -atopp prtoripri Concepts such -as probabilistic methods [ (eg MHarko--hhinn and Monte-Cart ) safety distance-based control methods -and trajectory prediction methods Mhave-beed designed in reeent years to cope-tith with complex traffic conditions terms -ef vision; CNN hass made outstanding contributions and has been applied a alrrg number of road condition inspection tasks due to its excellent regional feature extraction capabilities : The local feature information obtained by CNN will be used for obstacle detection. Secondly, because the motion trajectory is planned for AV the relationship-bette beetwenn each local feature -ef image obtained by CNN needs to-be be established- Some strategies are based on CNN plus RNN so -hatt tthat-they -an deal with sequential graphs as input number--d of vision tasks their performances are still far inferior to similar-sized convolutional neural networks counterparts -such ass EfficientNets -and RepVGG we believe this is due te the following aspects First the huge differences between the sequential tasks of NLP and the image tasks of CV are ignored For example- + when the local feature information acquired in a te dimenssion! image is compressed inte one-diiensionall time-ssries information how to achieve accurate mapping becomes d-diffeent
+ STDN Although the above strategies have performed -eett in a targe-mmber
Second: ; - is difficult to keep the original information of inputs since after RNN layers- ; wee need to recover the dimension from one to three Besides + due to the several transformations between different dimensions that process becomes even harder especially -iinee our input size -s 224x224x5 Third the computational and memory requirement of switching between layers are extremely heavy tasks which also becomes tricky point for algeriihn to -rim Higher hardware requirements
welt more running time arise when paper, ; -we propose n-new- network structure based -on CNN and ruuming the attention part: In thpee- attention to vision tasks in tnttonemmus driving- The new network structure overcomes problemss by using Sequential Spatial Network (SSN) blocks As shown through the convolution stem for fine- feattrr extrrcttiotttttteitn and are then fed into stack -ef SSN blocks for
in Fig input images first go
The Upsampling Convolutional Decreasing (UCD) blocks are introduced for purposees local information enhancement by deep convolution, -and in SSN block of features generated in first can bhe less loss of image resolution -which is crucial for the subsequent trajectory adjustment task In addition ; we adopt a staged architecture design using five convolutional layers with different kernet sizes and steps gradually decreasing the resolution ( sequence length ) and flexibly increasing the dimensionality Such design helps -te extract local features ee different scales -nd since the first stage retains high resolution
our design -ean effectively reduce #he-reselution eff the output information itt first layer comvosatnonn layer thits redweing tee-cenpttationall effort eff -stbseguent hasteeguent-yyerrs The Reinforcement Region Unit f RRU and the Fast MultiHead Self-Attention (FMHSA y in the SSN block can help obtain globat and tocal structural informatiionn within the intermediate features and improve the normalization capability of the network. Finally, average pooling is used to obtain better trajectory tuning. = Extensive experiments *n the lykit dataset demonstrate the-sperrirriy -ff our SSN network in terms of aeeuraay-- In addition te image classification, SSN block can -he easily transferred to te-other vision tasks and serve-es versatile backbone.
at
Figure 7: Selective context on the introduction of https: pettps://arxiv..or org/ abs/ 2303.07355

---

In researchers how turn vehicles assisted driving more intelligent autonomous driving Due the iteration
intelligent hardware the improvement chip computing power collected quickly converted fed
models In the driving process the safety factor users researchers Therefore how AV should avoid
collisions has Concepts such probabilistic methods { [egg) 1 safety distance-based control methods
trajectory prediction methods designed cope complex traffic conditions In terms vision
CNN made outstanding contributions and applied road condition inspection tasks
due
its excellent regional feature extraction capabilities The local fcature information obtained
CNN
will obstacle detection Secondly because the motion trajectory planned for AV the relationship
each
local feature obtained CNN needs established Some strategies based CNN plus RNN
so
deal sequential graphs as input eg STDN Although the above strategies performed vision tasks
their performances finferior EfficientNets RepVGG We believe this due the following aspects
First the huge differences the sequential tasks NLP the image tasks CV ignored For example when the local feature information acquired compressed how achieve accurate mapping becomes Second difficult keep the original information of inputs since after RNN layers recover the dimension from one three Besides due the several transformations between different dimensions that process becomes especially our input size 224x224x5 Third the computational and memory requirement swwicching layers are extremely heavy tasks also becomes tricky point Higher hardware requirements running arise the attention part In CNN attention to vision tasks The new network structure overcomes Sequential Spatial Network (SSN) blocks As shown Fig input images first go the convolution stem for a stack SSN blocks The Upsampling Convolutional Decreasing (UCD) blocks iintroduced for local information enhancement deep convolution in SSN block of features generated in can less loss of image resolution crucial the subsequent trajectory adjustment task In addition we adopt a staged architecture design using five convolutional layers steps gradually ( (equence length flexibly Such design helps extract local features different scales since the first stage retains high resolution our design effectively reduce the output information at The Reinforcement Region Unit
and the Fast MultiHead Self-Attention (FMHSA in the SSN block can help obtain within the intermediate features and improve the normalization capability Finally average pooling better trajectory tuning Extensive experiments addition image classification SSN block can easily transferred and serve a versatile backbone.
the lykit dataset demonstrate our SSN network In
Figure 8: The resulting context
Given Original Context, CUDA Memory = 77, 695 MB; Time = 110.8 ms ms/tooe ttkken
The paragraph discusses the development of autonomous driving technology and the use of artificial intelligence and machine learning to make decisions in the driving process. The safety of the vehicle is a top priority and methods such probabilistic methods, safety distance-based control methods, and trajectory prediction methods have been developed to cope with complex traffic conditions. The use of convolutional neural networks (CNN) has been successful in image tasks such as obstacle detection, but the performance of these networks is still inferior to similar-sized convolutional neural networks counterparts. This is due to the huge differences between the sequential tasks of NLP and the image tasks of CV, the difficulty of keeping the original information of inputs, and the heavy computational and memory requirement of switching between layers. The authors propose a new network structure based on CNN and attention to vision tasks in autonomous driving, using Sequential Spatial Network (SSN) blocks. The network structure overcomes these problems by using deep convolution and attention mechanisms to extract local and global features, and reducing the computational effort of subsequent layers. The proposed network structure is evaluated on the lykit dataset and shows superior performance terms of accuracy. B.
Figure 9: The reference summary generated by sicuna-133 3b given the full context.


Reviews:
[{"review_id": "JF9vPXHB9t", "reviewer": "Reviewer_PgsA", "summary": "", "questions": "See more in Reasons To Reject sections.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response 1:\nThanks for your thoughtful reading and constructive comments for our paper!\nTo respond your questions:\n\n**“Missing references and discussions”**, we have included popular attempts to accelerate LLMs inferences in the paper such as sparse attention (Child et al., 2019), and soft prompt (Mu et al., 2023; Chevalier et al., 2023). But we do notice other relevant efforts like rectified position embedding (https://github.com/bojone/rerope) and dynamic attention masking (Anagnostidis et al. 2023). We will definitely include more related works in the final version to fully address the reviewer's concern given the extra page. We would also appreciate if the reviewer can share with us what references we might have missed, as indicated in his/her comment.\n\n\n**“the boundary detection is not delicately designed”**, we adopt spaCy, a NLP toolkit, to decide the boundary of phrases in the paper, which is a standard approach. We appreciate the suggestions regarding developing more delicate boundary detection and testing various methods. However,  the primary focus of this work is to demonstrate the potential of context pruning, rather than focus on boundary detection. In addition, we found spaCy was sufficient and did a good job in our experiments. Despite the basic boundary detection, our approach still demonstrated strong performance on accelerating LLMs inference and maintaining good generation quality.\n\n**“Dataset is too short for long context processing”**, that is a great suggestion. We have noticed LongEval recently released by lmsys.org that aims to evaluate long context processing. So we test our method on the line retrieval task with two LLMs. Note that we use a slightly different way to compute self-information due to the long context. As shown in the following table, the results suggest our method is robust while dealing with extremely long context.\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + gpt-3.5-8k| .990 | unknown | unknown |\n|gpt-3.5-16k | .992 | unknown | unknown |\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + vicuna-4k-v1.5-7b| .968 | 68,433 MB |  81.8 ms/token |\n|vicuna-16k-v1.5-7b | .938 | 123,874MB |  130.8 ms/token |\n\nWe will add the LongEval experiments as a complementary part in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The topic of the paper is crucially important for researchers in LM's and AI/AGI with limited context window and large computation cost with Transformer model.\n* This work helps to solve the intrinsic problem (redundancy and repetition) in human language in LLM generation.\n* Compared with existing solutions focusing on architectures or distillations, it proposes a light filter on input without changing or finetuning the model.", "reasons_to_reject": "1. Missing references and discussions about related works. There is already some work but the current paper includes very few discussions about existing similar efforts.\n2. The phrase and sentence level filtering is mostly related to the boundary detection and how tokens are merged.  Inappropriate combinations will lead to the misunderstanding of filtered texts and bad performance in downstream tasks. Methods here are not delicately designed. Besides, no further discussion on the impact when using various methods.\n3. Dataset used in the experiments are too short for long context processing. Please refer to the well acknowledged datasets that are recently proposed for long context evaluations.", "questions_to_authors": "See more in Reasons To Reject sections.\n", "paper_topic_and_main_contributions": "This paper proposes a method of identifying and pruning redundancy in the input context to enhance the inference efficiency and reduce memory cost considering the limited context window of LLM. Evaluations on common long texts dataset and multi-tasks show that it can maintain comparable performance while achieving memory and cost reduction to a great extent.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k5v7MXk0Su", "reviewer": "Reviewer_eKVf", "summary": "", "questions": "Please look at \"Reasons to Reject\" section. Would love to see your opinion on each of the points mentioned there.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response 2:\nThanks for your thoughtful reading and constructive comments for our paper!\nReally appreciate your approval of our method and experiments. \nWe respond to your main questions below:\n\n**“Method may not work when tasks are focusing on text being dropped”**. This is a great question. We have actually addressed this question in the Case Study as well as in Figure 1. These results suggest that the context compression mainly targets the redundancy of input, where the dropped text is either 1) learned by the model in pretraining; or 2) has been mentioned before in the input. In Figure 1, even though the query is focusing on the dropped text, LLMs are still able to generate the correct answer thanks to their inner-knowledge.\n\n**“Computing self-information will lead to significant overhead”**, We should have made ourselves clearer in the paper. Actually, computing self-information is highly efficient. It only requires one step to compute self-information and it is very easy to compute it in parallel. As shown in section 5.6, it only takes 46 ms to finish context compression. Then it leads to a 35 ms reduction for every token the LLM will generate. For the example in Figure 6, where the answer has about 400 tokens, we actually save 14000 ms in total. In summary, self-information computing is highly effective compared to the answer generation.\n\n**“Using NLI benchmarks can allow us to have a better sense on quality loss”**, This is a very interesting idea. But since NLI questions are usually very short (which means there’s less space for further compression) and their answers are usually one single label, the efficiency of generation is not considered as a problem. But your suggestion reminds us that maybe we could try a SQuAD fashion test that uses exact match as the criterion, then we can check the accuracy that better quantifies quality change.\n\n**“Other sophisticated methods”**, many thanks for kindly providing valuable references. I have gone through the two papers. Both of them seem to accelerate LLMs inference via involving extra trainable parameters. We believe our approach complements the studies you mentioned, as it is effective and can be integrated with other methods without conflict.\n\nIn addition, we would like to share an interesting new experiment of our method conducted on the recently released LongEval that assesses model performance on long context processing. The results suggest our method is robust while dealing with extremely long context.\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + gpt-3.5-8k| .990 | unknown | unknown |\n|gpt-3.5-16k | .992 | unknown | unknown |\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + vicuna-4k-v1.5-7b| .968 | 68,433 MB |  81.8 ms/token |\n|vicuna-16k-v1.5-7b | .938 | 123,874MB |  130.8 ms/token |\n\nThank you again for your comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Making efficient inference with long text in context is a very important area of research that can bring down compute used for these tasks significantly.\n2. The method is simple, and works with prominent LLMs, and is very well verified by extensive set of experiments.", "reasons_to_reject": "1. The method may not generalize very well on tasks that specifically focus on text being dropped.\n2. Even though the method reduces the latency of the LLM, there is significant overhead in calculating the tokens that should be dropped.\n3. If we combine the latency from finding the tokens to be dropped, and the latency of the LLM, the overall latency gain isn't very significant while the quality loss might be larger.\n4. Is it possible to quantify the quality loss better on some non-generative task like NLI, so that we have a better sense of how much the quality loss is with metrics like F1.\n5. There are other methods that are more sophisticated and offer better guarantees. For examples see [1] and [2]:\n  - [1] Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers. (https://arxiv.org/pdf/2305.15805)\n  - [2] PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting (https://arxiv.org/abs/2210.08964)", "questions_to_authors": "Please look at \"Reasons to Reject\" section. Would love to see your opinion on each of the points mentioned there.", "paper_topic_and_main_contributions": "The paper proposes a methodology to quantify importance of spans of tokens in a context so that we can remove the least important tokens from context and still maintain reasonable performance while gaining speed up from the reduced length of the context.\n\nThe proposed method uses the generation likelihood of tokens, which translates to the self information of the tokens to quantify the important of the tokens.\n\nThe paper proposes using percentiles (instead of thresholds) to select the top set of tokens to be kept in context.\n\nWith an extensive set of experiments, the authors show that even when dropping significant amount of tokens from context, the model is still able to perform very well on a wide variety of tasks. \n\nSince the method performs best if the input has redundant information, the paper also proposes three datasets with long text and conversations to evaluate performance.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6o6Vm6KxK7", "reviewer": "Reviewer_6QAt", "summary": "", "questions": "As I understand it, currently the choice of lexical units means that Selective Context eliminates either words, phrases OR sentences. Have the authors considered trying to perform eliminations across lexical units?\n\nThere is an implicit assumption that the weaker model used to compute self-attention has the same \"knowledge\" as the more powerful model used to do inference. This is of course not necessarily true. Have the authors considered using the same model for both Selective Context and inference to establish an upper bound of sorts on performance?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response 3:\nThanks for your thoughtful reading and constructive comments for our paper!\nTo respond your questions:\n\n**“It’s unclear whether the time cost of the context compression is included”**, many thanks for raising this question. I should have made that more clear in the paper. Time used to compress context is already included in the paper. Actually, computing self-information is highly efficient. It only requires one step to compute self-information and it is very easy to compute it in parallel. According to section 5.6, it only takes 46 ms to finish context compression. Then it leads to a 35 ms reduction for every token the LLM generates. For the example in Figure 6, where the answer has about 400 tokens, we actually save 14000 ms in total. In summary, the 36% reduction did factor the cost of computing self-information. We will emphasise this in our final version.\n\n**“Why not directly ask a model to summarise the input?”**, this is absolutely a very interesting idea, and we believe this is an interesting baseline to be compared against. However, the limitation of this idea is obvious: the generation of a summary can be very slow (hundreds times slower than self-information computing). As mentioned above, self-information is a super-efficient approach that only requires one-step to calculate. In addition, we have very little control over the generation of summaries. For example, we cannot ask the model to generate a summary that is exactly 20% shorter than the input. On the contrary, our method enables a fine-grained adjustment based on compression ratio. In summary, using a model to summarise the input might be a great idea for text compression, but it would not help accelerating the inference of LLMs.\n\n**“Why are GPT models missed in figure 4?”**. Figure 4 illustrates how model scale and instruct-tuning affects the overall performance. However, as GPT models are closed models, we have little knowledge on their scales and whether or how they were instructed-tuned. Therefore, we cannot add GPT models in Figure 4.\n\n**“Why is only gpt-3.5-turbo analysed in Table 3”**. We chose to conduct a manual evaluation on gpt-3.5-turbo specifically because it is the only large language model for which previous studies have analysed faithfulness. And we use their approach (B Li et al., 2023, H Zhang et al., 2023) in our experiments. This is why our analysis focuses solely on gpt-3.5-turbo. We didn’t perform human evaluation on all baseline models as the annotation procedure can be quite costly, and we believe the results of gpt-3.5-turbo are representative.\n\n**“Tables 1 & 2 should present separate scores for different models”**, first many thanks for your clear understanding of  our intention of using average scores to show our approach is model agnostic. We didn’t present separate scores for all baseline models to avoid the table taking up too much space and ensure the table is readable. However, we do agree that it is very valuable to provide more detailed results, and we will add a new table presenting full breakdown results in the Appendix in the final version.\n\nAt last, we would like to share an interesting new experiment of our method conducted on the recently released LongEval that assesses model performance on long context processing. The results suggest our method is robust while dealing with extremely long context.\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + gpt-3.5-8k| .990 | unknown | unknown |\n|gpt-3.5-16k | .992 | unknown | unknown |\n\n|Method|Accuracy|GPU memory usage|Inference time per step|\n|---|---|---|---|\n|our method (50% reduction) + vicuna-4k-v1.5-7b| .968 | 68,433 MB |  81.8 ms/token |\n|vicuna-16k-v1.5-7b | .938 | 123,874MB |  130.8 ms/token |\n\nThank you again for your comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles a real and important problem with LLMs today. While context sizes continue to grow, there are considerable computational savings that can be achieved by more efficient representation of the input context. The authors have also proposed an interesting and novel solution that has information-theoretic foundations to the problem context compression. Finally, the paper is generally well written and easy to follow.", "reasons_to_reject": "My concerns with the paper are mainly related to the experimental section, and it isn't clear to me that the results support the claims that the authors are making.\n\nFirstly, the headline figure in the abstract states a 32% reduction in inference time. However, it is unclear whether this factors in the cost of actually computing the self-information for every token in the input and subsequently compressing it. Given that this computation is also done by an LLM (albeit a less powerful one), this seems like a non-negligeable factor to report.\n\nFurther, the paper does not compare Selective Context against a baseline approach that simply uses the weaker model to generate a summary (with a prompt that states the compression ratio, for example). As a result, it is unclear whether this complex process of computing self-information for every token yields any meaningful gains over a naive way of compressing the input context.\n\nBeyond this, there are some unusual choices in the presentation of the experimental results that don't make much sense to me. For example, Figure 4 only presents open-source LLMs (why?); conversely Table 3 only shows results on gpt-3.5turbo; gpt-4 is introduced but is never spoken about again. In similar vein, Tables 1 & 2 show averaged scores instead of breaking up scores by model. I understand the overall message here is to show that the authors' approach is model agnostic. However, for the sake of deeper insight about how different models respond to having input elided, a clearer and more principled approach to showing results would have been appreciated.", "questions_to_authors": "As I understand it, currently the choice of lexical units means that Selective Context eliminates either words, phrases OR sentences. Have the authors considered trying to perform eliminations across lexical units?\n\nThere is an implicit assumption that the weaker model used to compute self-attention has the same \"knowledge\" as the more powerful model used to do inference. This is of course not necessarily true. Have the authors considered using the same model for both Selective Context and inference to establish an upper bound of sorts on performance?", "paper_topic_and_main_contributions": "This paper introduces an information-theoretic way to compress the input contexts to LLMs. Specifically, it uses the notion of self-information to elide parts of the input that have low information content with respect to the underlying distribution of a language model - a process the authors call \"Selective Context\". The authors show a number of different experimental results demonstrating the effectiveness of the approach, and argue that Selective Context leads to significant reductions in memory usage and inference time with respect to full inputs, while observing some drops in performance on several downstream applications.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4 mentions vicuna_13b twice. I'm assuming one is vicuna_7b"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an efficient inference method that ranks the spans of tokens in the context, and removes the least important ones. The reviewers appreciate the importance of the problem, the simplicity, novelty and theoretical foundation of the solution, and the writing quality. Nonetheless, there were some concerns about missing references and comparisons, and missing ablations, though many of them did not include a specific missing comparison, so they can be discounted. There were also practical gains given the method overhead. The rebuttal seems to have addressed most of the concerns raised.
Decision: Accept-Main