Compressed Context Memory for Online Language Model Interaction
Authors: Jang-Hyun Kim
Junyoung Yeom
Sangdoo Yun
Hyun Oh Song
Venue: ICLR 2024 poster (2024)
Keywords: context compression
efficient inference
natural language processing
transformer
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose a compressed context KV memory system for memory-efficient online inference of language models

Abstract: This paper presents a context key/value compression method for Transformer language models in online scenarios, where the context continually expands. As the context lengthens, the attention process demands increasing memory and computations, which in turn reduces the throughput of the language model. To address this challenge, we propose a compressed context memory system that continually compresses the accumulating attention key/value pairs into a compact memory space, facilitating language model inference in a limited memory space of computing environments. Our compression process involves integrating a lightweight conditional LoRA into the language model's forward pass during inference, without the need for fine-tuning the model's entire set of weights. We achieve efficient training by modeling the recursive compression process as a single parallelized forward computation. Through evaluations on conversation, personalization, and multi-task learning, we demonstrate that our approach achieves the performance level of a full context model with $5\times$ smaller context memory size. We further demonstrate the applicability of our approach in a streaming setting with an unlimited context length, outperforming the sliding window approach. Codes are available at https://github.com/snu-mllab/context-memory.

Published as a conference paper at ICLR 2024
COMPRESSED CONTEXT MEMORY FOR ONLINE LANGUAGE MODEL INTERACTION
Jang-Hyun Kim Junyoung Yeom Sangdoo Yun" Hyun Oh Song!
Scouu National University. 'Artificial Intelligence Institute (AlIS), 'NAVER Al Lab (janghyun, yeomjy, hyunch @rrunhheemmaa em1 lab. snw.scuuu.. ac..a.c.. sangdoo, wor.ddoeyyuenaaeerrrp.coom
ABSTRACT
This paper presents context key/value compression method for Transformer lan- guage models in online scenarios, where the context continually expands. As the context lengthens, the attention process demands increasing memory and compu- tations, which in turn reduces the throughput of the language model. To address this challenge, we propose compressed context memory system that continually compresses the accumulating attention key/value pairs into a compact memory space, facilitating language model inference in a limited memory space of com- puting environments. Our compression process involves integrating lightweight conditional LoRA into the language model's forward pass during inference, with- out the need for fine-tuning the model's ' entire set of weights. We achieve efficient training by modeling the recursive compression process as single parallelized forward computation. Through evaluations on conversation, personalization, and multi-task learning, we demonstrate that our approach achieves the performance level of a full context model with 5x smaller context memory size. We further demonstrate the applicability of our approach in a streaming setting with an un- limited context length, outperforming the sliding window approach. Codes are available https: / //gittttbb con/ snu-ml lab/ pantext-memos
INTRODUCTION
Transformer language models have exhibited exceptional language processing capabilities, achiev- ing remarkable results in various applications (Vaswani et al., 2017). In particular, the attention mechanism, which encompasses the entire context window, enables the language models to respond with nuanced understanding of context. With this contextual understanding, services like ChatGPT or Bard can generate responses customized to individual users through online interactions (OpenAI, 2023; Manyika, 2023). In this online scenario, the context used for language model inference aceu- mulates over time, raising an important challenge in efficiently handling this growing context.
straightforward approach to deal with previous contexts as a prompt, which leads to a continual increase in inference time and memory usage due to the growing length of contexts. Alternately, caching the attention hidden states of Transformer would be impractical (Dai et al., 2019), as the caching capacity and attention costs increase with the accumulation of contexts. Recent studies pro- pose compressing contextual information into concise sequences of token embeddings or attention keys/values (denoted as KV) (Chevalier et al., 2023; Mu et al., 2023). However, those methods primarily focus on fixed-context scenarios and are not designed for dynamically changing contexts. Thus, they still face inefficiency and redundancy when dealing with accumulating contexts.
In this paper, we propose a novel language model framework incorporating a compressed context memory system for efficient online inference (Figure 1). Our memory system is capable of dynamic updates during online inference with minimal memory and computation overhead. To this end, we optimize a lightweight conditional LoRA (Hu et al., 2022), enabling language models construct compressed attention KV memory of contextual information through the forward computation pass. On the other hand, dynamic memory updates require recursive context compression procedure, which leads to training inefficiencies. To address this challenge, we propose an efficient training
"Corresponding authors

---

Published as a conference paper at ICLR 2024
New context
User
New centext
User
Append
Input
Input
Output
Compressed memory
Ouiput
Update
Attention
Attention
Context history
Compression w/ LoRA
Model
Model (frozen)
Figure Main concept of online inference systems. Left: Conventional online inference ap- proach. Right: The proposed system with compressed context memory. The colored boxes represent attention keys/values (or input tokens) required for Transformer inference. The new context refers to the sequence comprising an input and model output from the preceding interaction.
Table 1: Analysis of inference throughput on the MetalCL dataset (Min et al., 2022) at time step 16 with LLaMA-7B and FP16 precision (Touvron et al., 2023). We measure throughput using batch processing on a single GPU. CCM- CCMM(concat,
refers to our proposed method.
A PCTe 80GB
RTX 3090 24GB
Full context
CCM-concat
CCM-merge
Full context
CCM-concat
CCM-merge
Throughput (sample/sec)
24.4
69,9
3.5
18.6
50.7
Maximum batch Context KV length
60
300
950
10
50
800
128
8
800
128
Performance (Accuracy %)
70.8
70.0
69.6
70.8
70.0
69.6
strategy that unrolls the recursive context compression procedure and processes the recursive proce- dure in parallel. In the inference phase, language models utilize the compressed memory to generate responses to subsequent input queries with reduced attention operations and memory.
Our approach offers several advantages compared to existing efficient context processing meth- ods: 1) Unlike approaches that propose new attention structures such as the Linear Transformer (Katharopoulos et al., 2020), our method simply involves the integration of lightweight adapters to existing Transformer language models, leveraging the weights of pretrained models. 2) Unlike fixed-context compression techniques such Gisting or ICAE (Mu al., 2023; Ge et al., 2023), our approach is able to dynamically compress newly added context with minimal computational overhead. 3) In contrast methods that recurrently compress context into token embeddings, such as RMT or AutoCompressor (Bulatov et al., 2022: Chevalier et al., 2023), our approach focuses on compressing attention keys/values, enabling fully parallelized training process. Notably, our ap- proach achieves a training speed that is 7 x faster than the mentioned approaches (Table 18) and does not require additional forward computations for the compressed token embeddings during inference.
Our online compression framework has a wide range of applications, including conversation, per- sonalization, and multi-task learning. Notably, by compressing continuously provided dialogues, user profiles, and task demonstrations, our approach enables the language model to perform online inference with reduced memory usage and attention costs. To substantiate our claims, we evaluate our system across diverse datasets, including DailyDialog, LaMP, and MetalCL (Li et al., 2017; Salemi et al., 2023; Min al., 2022). Through empirical analyses, we demonstrate that our method excels in both efficiency and performance compared to established context compression baselines. In particular, our method achieves equivalent performance with only 1/5 the context memory required when using the full context (Figure 6). This enhanced memory efficiency translates into substantial improvements language model throughput when using batch processing on memory- constrained GPUs (Table 1). Finally, demonstrate the efficacy of approach in streaming setting with an unlimited context length, outperforming the sliding window method (Figure 8).
2 PRELIMINARY
Target scenario and notation Let T denote a space of texts. We focus the online inference scenario, aiming to predict the output O(t) E T based on the input I(t) e T and the accu- mulated context C(t) [c(1),...,c(1)) for time step & [l....,T] T], where T E N represents
2

---

Published as a conference paper at ICLR 2024
Table 2: Illustrative instances of online inference scenarios.
Application
Dataset
Context C(t)
Input [(t)
Output O(t)
Conversation
DailyDialog (Li et al., 2017)
Dialogue history
User query
Reply
Personalization
LaMP (Salemi et al., 2023)
User profiles
User query
Recommendation
Multi-tast learning
MetalCL (Min et al., 2022)
Task demonstrations
Problem
Answer
the maximum number of time steps. Here, c(t) E TT denotes a newly integrated context at time step which comprises the interaction results from the preceding time step t-1, including ((t-1), O(t-1), and any additional user feedback. In Table 2, we formulate diverse applications accord- ing our target scenario and notations, where each context C(t) contains accumulated information for a specific identity (e.g., a task or user). We represent the dataset with multiple identities as I, [I,.... T}} where Z denotes an index set of identities. We randomly split I into training set Tarain and a test set Tes for experiments.
D = (I), (1), O(())
Context compression Let us consider a pretrained language model fo - which models the probability distribution over the text space T. A typical approach for predicting output (()) involves using the full context C(t) as Ô(t) fo(- ((t), However, this approach requires increasing memory and computation costs over time for maintaining and processing the entire con- text C(t). One employ context compression techniques mitigate this issue, compressing contexts into a shorter sequence of attention key/value pairs or soft prompts (Mu et al., 2023; Ge al., 2023). Given the compression function Scomp- the inference with compressed contexts be- comes Ó(t) ~ fol- existing context compression methods mainly focus on compressing a fixed context C that re- peatedly used prompt (Mu et al., 2023; Ge et al., 2023). The objective of the compression is to generate outputs for given input that are similar to the outputs generated when using the full context: fo(- Jcomp (C), I) 2e fo(. C)
where 19cmmp (C(t)) < |C(t).. is worth noting that
METHODS
In this section, we introduce a novel approach named Compressed Context Memory (CCM), de- signed for efficient online inference of language models. Our system compresses the given current context and dynamically updates the context memory by incorporating the compression result. We further propose a parallelized training strategy to facilitate efficient large-scale optimization.
3.1 COMPRESSED CONTEXT MEMORY
Here, we briefly describe the compression and inference processes at time step t. We denote the compressed context memory t as Mem(t) with initial value of Mem(0) 0. When presented with a context c(t), we condense the information within c(t) into the hidden feature h(t) by using the compression function Jcomp
h(t) =
Scomp (Mem(t-1))
The compressed context memory Mem(() (f) is then updated via an update function Jupdate as
Mem(t) Supdate (MMm(t-1),hi)
Within a limited memory space, Mem(t) (t) stores contextual information up to time t. By encompass- ing only the input (tt and memory Mem(t), we conduct memory
memory-efficient efficient inference as
Ô(I) ~ fo Mem(l),
In the following, we elaborate the compression and update processes.
Compression We compress context information into attention keys/values as in Compressive Transformer (Rae et al., 2020) and Gisting (Mu al., 2023). This compression approach can be applied within each layer of the language model, providing better parallelization than the auto- encoding approach (Ge al., 2023). We introduce specialized compression token (COMP) and
3

---

Published as a conference paper at ICLR 2024
train the language model to compress context information into the at- tention keys/values of the (COMP) token, similar to the Gisting ap- proach.
Transformer layer
Attention
We assume a Transformer language model fo has L layers with a hid- den state dimension of d. To simplify notation, we set compression token length of 1. It is worth noting that the compression token can be extended to arbitrary lengths. Under these conditions, the total size of the attention keys/values of (COMP) token is 2x Lxd. The com- pression process illustrated Figure 2. At each time step t, we ap- pend (COMP) token to the context c(t) and make the (COMP) token to have attention on the keys/values of c(() and the previous memory state Mem(t-1). Utilizing the resulting attention keys/values of the (COMP) token, we obtain the compressed hidden feature htt) 1x2xxx in Equation (1).
Mem((-1)
7i(0)
<(t) (COMP)
Figure 2: The illustration of the compression process at time step l. Each col- ored box symbolizes atten- tion hidden states.
Memory update We propose memory update functions Jupdaae that are differentiable and paral- lelizable during training. In particular. we consider the simplest form of 9updute and verify the effec- tiveness of our compression framework. Considering various application scenarios, we examine two types of memory systems: 1) scalable memory and 2) fixed-size memory, similar to an RNN.
For scalable memory setting, we employ the concatenation function Supdute- Then Mem(() E Rex2xLxd contains the attention key/value pairs associated with (COMP) tokens up to time step t. We denote our system with the concatenation function as CCM-concat.
For fixed-size memory system, we propose a merging function to update information in the memory, Specifically, we update memory by weighted average: Mem(t) 2xLxx Mem(t). (ett))Mem(tt1)+tattee
where a = and ar E [0, 1] for t 2 2. With this recurrence, Mem(t) becomes Mem(t) = Σj= 0j I1 - ax) h(j). In the main experiments, we evaluate an update method based on the arithmetic average of the compressed states with a; = 1/t, i.e. Mem(t) ビ h(j). We denote our method with the merging function as CCM-merge.
in parallel by averaging hidden features h(1)...., h((t simultaneously. In the online inference phase, recurrently update the memory by cumulative average using the prior memory Mem(t-1) and current compression result l(t). In Appendix Table 14, we examine another design choice for the merge function: the exponential mov- ing average. It also worth noting that CCM-concat can be interpreted as a process that dynamically infers coefficients for hidden states h(t) through the attention mechanism.
During training, we compute Mem(I)....,,Men
Parallelized training The direct integration of the compression process of Equation (1) into the training process poses challenge as it requires recursive model executions over j 1,..., t. Such recursive executions prolong training time and amplify back-propagatee errors through the elongated computation graph (Gruslys et al., 2016). To overcome this challenge, propose fully parallelizable training strategy, taking advantage of the Transformer structure.
For training data (C(t), Itt), O(t)) E Durinn insert (COMP) tokens into the accumulated context C(I), forming the sequence (e(1), (COMP) (eomp)....fff c(t), (COMP), [(t)]. We then establish memory update and attention mechanisms, modeling recursive compression processes as parallelized forward com- putations (Figure 3). In detail, within each layer of a Transformer Se. we update Mem(j) for j < I using the attention keys/values of preceding (COMP) tokens. i.e., h(1), ... h(f), as in Figure (a). Following the memory update, we execute the compression procedures for j = 1....,t t in parallel using the masked attention as in Figure (b). As stated Equation (3), we access the context in- formation from previous time steps only through memory during online inference. Therefore, we restrict c(j) to reference only Mem(j-1) for j < and make ((t) exclusively have its attention on Mem(t). Finally, we compute likelihood fa(O(t) | Mem(f), (()) in Equation (3) using the output probability obtained at the last token position of [tt). When the token length of Ott) exceeds 1, we follow the conventional approach by conditioning for the next tokens (Radford et al., 2019). All these steps take place within a single forward pass of fo, and the loss gradients are backpropagated to all tokens across all time steps.
the target label O(t) and calculating the loss
4

---

Published as a conference paper at ICLR 2024
Attention keys/values at each layer
Source
Memt-)
Mem(t-2)
Mcm(t-1)
Mem(t)
6-2 2-1
(C) (C) e(0) (C) 7(0)
(C)
c(t-1)
Update
Target
(C)
c(4)
(C)
M(1-2)
(C)
ctt 1)
(C)
c(t)
(C)
[(t)
(a) Parallel memory update
(b) Attention mask
Figure 3: Illustration of the parallelized training process. In (a), each colored box symbolizes attention keys/values of memory, compression tokens, and normal text tokens. In (b), gray indicates that attention is blocked. In the figures, (C) stands for (COMP). At each layer, after the parallel updates of compressed context memory, the attention operation occurs with the mask in (b). Note the calculation of Mem(t) occurs after e(t) and its subsequent (COMP) token. Reordering the top row of (b) to align with this temporal relation yields an autoregressive mask.
Algorithm Training stage for compression
Input: Language model fe. training set Drain
Initialize conditional LoRA weight A Modify the forward pass of fo to update the compressed context memory repeat
Sample mini-batch B C Dumin and set B" = for (C),)), [,()), O+(()) do
Prepare an input = (1), (COMP), c, (t),,, (COMP), 4()] and target 0,(1) B'= B' {(z,, } end for
Compute loss in eq. (4) on B through a single forward pass using the masked attention Perform gradient descent step w.r.f. AO until convergence Output: AO
Conditional adapter Current compression methods typically rely on fine-tuning language model fe to acquire compression capabilities (Mu et al., 2023). In this approach, the construction of the memory hinges on the adjustment of the language model parameter 6, allowing us to parame- terize the memory for context (t) Mem, f). The objective function for learning compression capability is then formulated as ming EE, log (o(((()) Mem, 0), [(()))]
However, this conventional objective can potentially lead the language model generate swers for input It(t) without considering the memory Mem;(t; 0). Such overfitting to the in- put 1tt) can diminish the importance of compressed context memory during training, which leads to insufficient training of the compression capability. Specifically, when we measure ((t)),, throughout the compression training
the loss without context, E1,-z1- log (t, process with LLaMA-7B on MetalCL, the loss on training set decreases from 2.69 to 1.84, whereas the loss on test set remains 2.59. This observation indicates the presence of overfitting on inputs.
To address this issue, we introduce separate trainable parameters specif- ically for compression. To this end, we propose conditional variant of LoRA (Hu et al., 2022), which operates exclusively on (COMP) tokens. This ensures that the trainable parameters allocated for compression solely influence the model's ' compression capabilities (Figure 4). Let W E dddd denote parameter of feed-forward layer with hidden dimension d, and let AW AT B E dd denote a corresponding LoRA weight with A, B E Rexd and & & d. For input token x and its corresponding hidden
Jw-aw
e(t)
(COMP)
T(X)
Figure 4: Feed for-
ward operations of our conditional LoRA.
5

---

Published as a conference paper at ICLR 2024
t.
4
t 4
4
1-1/1
4
Compress
Compress
e(1)
((t) O(t)
(1)
(5
(C())
(() O(t)
Mem 1) d(8)
Mem(U)
(() O(t)
(a) Full context
(b) Fixed context compression
(c) Compressed context memory
Figure 5: Illustration of the compression and inference processes at time step The arrow indicates the process of referencing the keys/values the left to generate the keys/values the right. Here, le means the expected length of key/value pairs of context c(-), and I denotes the total length of input and output. We assume that each compression outcome has length of Notations at the top of Mem(-) denote the length of key/value pairs corresponding to CCM-concal-mergge
Table 3: Complexity analysis of approaches in online inference scenario at time step Figure presents illustrative explanations for the compression/infernnce processes with respective notations.
Type
Operation
Full context
Fixed-context compression
CCM-concat
CCM-merge
Compression
O
(()+
O(I.)
Memory
Inference
O +
0(4)
0(1+4)
O(4)
Attention FLOPS
Compression
(
O(++lo) O(1i: + +
O(I.)
Inference
O(lld + (2)
O
O(1)
state 24 IRd, we propose the following conditional forward computation:
1/h = W In + m AW The
where m 1(x = (COMP)). We denote all trainable LoRA parameters of a model as A6. The pa- rameter A only affects the formation of compressed context memory, and compression training objective with conditional LoRA is
minimize A8
Er |- log fo(O,(t)
(t; + A0), I(tt))
(4)
We summarize the training procedure of our approach in Algorithm 1.
3.2 COMPLEXITY ANALYSIS
We analyze the complexity of approaches in online inference scenarios in Table 3. In the table, "full context" refers to the method using full context Ctt) during inference, and "fixed-context compression" refers to the method compressing C(t) geomp (C(t)) each time step (Mu et al., 2023). In Figure we visualize these methods and introduce notations used in complexity analysis.
Regarding the full context method, the context length at time step t is tle resulting in inference memory complexity of O + 44) and quadratic attention FLOPS of O(tl,4 + 1?). Fixed-context compression methods offer reduced complexity for inference. However, they process the entire context C(t) for compression, resulting in memory and FLOPS complexities (ll.).
Our method, utilizing compressed context memory for both compression and inference, exhibits reduced complexity. In the case of CCM-merge, compression complexity depends solely on the length of context e(t) (l.)). For CCM-concat, the complexity becomes proportional to the time step & due to growing memory size over time. Nonetheless, the compression complexity reduces from O(tl.) to O(t 1e) when compared to fixed-context compression methods. While CCM- concat exhibits higher complexity than CCM-merge, language model using CCM-concat achieves superior performance, offering trade-off between performance and complexity (Figure 6).
EXPERIMENTS
In this section, we present the empirical validation of our approach in online inference scenario. In Section 4.2, we further substantiate our claims through an ablation study and additional analyses. We provide a detailed training setup in Appendix D.

---

Published as a conference paper at ICLR 2024
No context
Full context
OCM-concar (ours)
CCM-merge (ours)
(a) Accuracy over time step (T)
(b) Attention KV memory
(c) Context compression ratio
71
T16
Pork KV memory (MB)
100
69
T4
T16
14
500
Context length rabio
80
67
400
60
Accuncyy
65
300
40
200
63
T1
20
100
10
52
50
20
21
2
2
4
16
16
Peak KV memory (ratio, log-scale)
Time step demonsttration)
Time step (6 demonstration)
Figure 6: Comparison to full context approach on MetaICL test tasks with LLaMA-7B. Peak KV memory refers to the peak memory space occupied by attention keys/values during compression and inference processes each time step. We provide results for other datasets in Appendix, Figure 10.
Datasets and metrics We conduct evaluations using three datasets: MetaICL (Min et al., 2022), LaMP (Salemi et al., 2023), and DailyDialog (Li et al., 2017). First, MetaICI. is dataset for multi-task in-context learning, aiming at solving tasks unseen during training. We evaluate on the high-to-low resources setting, consisting of 61 training tasks and 26 unseen test tasks. The evalua- tion metric accuracy for multiple-choice questions. Next, LaMP is a dataset for personalization, utilizing user profiles generate personalized recommendations. For evaluation, we measure the accuracy of multi-choice recommendations on new users unseen during training. Lastly, we as- sess performance in conversation scenarios using the DailyDialog dataset, comprising sequences of everyday conversations. We evaluate models by measuring perplexity on actual dialogues. In Appendix C, we provide more detailed information and statistics for each dataset.
Baselines We implement established fixed-context compression techniques with open-source codes. Our primary focus is evaluating the Compressive Transformer (Rae al., 2020) and Gisting (Mu et al., 2023), both designed to compress attention hidden states. To suit online in- ference scenarios, we devise Gisting to compress contexts c(1)...., c(t) separately and evaluate the method using the concatenated compression results for inference. We refer to this approach as Gisting-online. For the recurrent compression approaches. RMT and AutoCompressor (Bulatov et al., 2022; Chevalier et al., 2023), we conduct separate comparison as publicly available trained models are limited to the OPT architecture (Zhang et al., 2022). We also evaluate the performance of language models using full context to quantify the performance degradation due to compression.
4.1 COMPRESSION PERFORMANCE
Comparison to full context method In Figure 6, we analyze the memory efficiency of our method in an online inference scenario. Figure 6-a shows the performance obtained each time step, along with the peak memory required for attention keys/values during the compression and inference pro- cesses illustrated in Figure approach compared to the full context approach. Specifically, CCM-coneat achieves comparable performance by using half the key/value memory space, whereas CCM-merge attains equivalent performance levels with approximately 1 /8 of the key/value memory space. While CCM-concat requires more memory, it outperforms the merge approach as time steps increase. Compared to the No context method, which relies solely on inputs to generate outputs, methods exhibit su- perior performance with a negligible increment context memory size. Remarkably, our method demonstrates an 18% boost in performance compared to the no-context method at time step 16.
The results demonstrate the memory efficiency advantage of our
Comparison to compression baselines Figure 7 compares the test performance of compression methods on various datasets. For fair comparison, we set an identical compression factor for all compression methods, except for CCM-merge, which has a higher compression factor. The figure shows that our compressed context memory approach consistently outperforms established com- pression baselines across all time steps, demonstrating performance that closely parallels the full context approach. Regarding the Gisting approach, which is optimized for compressing fixed
7

---

Published as a conference paper at ICLR 2024
No context
Full context
Gisting-online
Compressive
CCM-concat (ours) CCM-merge (ours)
(a) MctalCL
(b) LaMP
(c) DailyDialog
9.6
9.0
Aggurcy (8)
Perpiesity
Aucumacy
79
7.4
77
75
6.2
69
2
16
12
Time step (# demonsttatioon
Time (# prolile)
Time step (4) dialogue
Figure 7: Test performance of compression methods in online inference scenario with LLaMA-7B. All compression methods have the identical compression factor around 8, except for CCM-merge, which has a higher compression factor. We provide exact values in Appendix, Tables 23 to 25.
8.6
Sink
CCM
Context sliding window
CCM-concal (ours)
Perplexity
8.2
StreamingLLM
Evict oldest
Compress
7.8
7.4
Streaming until reaching the window limit
7.0
0
10k
20k
30k
40%
50k
60k
Figure 9: KV cache during streaming with CCM-concat. The example above assumes a CCM maximum size of 4 and sliding window maximum size of 8.
Context length
Figure 8: Streaming evaluation on PG19 validation set using sliding window with LLaMA-7B.
context in a single iteration, there is no performance improvement as the time step increases. It is worth noting that there is key distinction among the datasets considered. Regarding MetalCL, the task demonstrations c;)).. (t) are mutually complementary, sharing information related to the 7th task. Similarly, LaMP's user profiles share information about specific users. On these datasets, both merge and concatenation approaches yield similar performance, indicating insignificant com- pression loss during the merge operation. On the other hand, in the dialogue dataset, the contexts (1)),...,, (t) conveyed through the 5th conversation have distinct information. In this case, the concatenation approach, which compresses context information into distinct memory spaces, out- performs the merge approach as shown in Figure 7-c. This observation indicates that as diverse information is introduced over time, the loss of information in the merge approach increases.
Unified compression adapter To demonstrate the generalization ability of our method in more general scenarios, we train single compression model and evaluate its performance across various tasks. Specifically,
leverage MetalCL training tasks and conversation dataset, SODA (Kim et al.. 2023), as our training data, and then evaluate on multiple test tasks: MetalCL unseen test tasks, LaMP, and DialoO. In Appendix E, Table 10, we provide evaluation results of the single compression model. We note that the compression performance decreases slightly compared to a compression adapter trained specifically for each application (Figure 7). For example, on the MetalCL test tasks, the compression accuracy gap increases from 0.8% to 1.3%. However, Table 10 shows that our method obtains the best compression ability across all evaluation sets, demonstrating approach's generalization ability on data and scenarios unseen during training.
Streaming with sliding window We incorporate CCM into the sliding window approach with attention sink (Xiao et al., 2023). During streaming, tokens are processed one by one while adhering to the limited KV cache memory size. When the KV cache limit is reached, we compress the oldest tokens in the context window to update the compressed memory (Figure 9). In the case of CCM- concat, we manage the compressed memory size by emitting the oldest compressed key/value pair. Following Xiao et al. (2023), reassign sequential position IDs starting from 0 within the KV
8

---

Published as a conference paper at ICLR 2024
Table An example result using our method with LLaMA-7B on a DailyDialog test sample.
Context:
A: What's the problem, Nada? You look down in the dumps. (COMP)
B: don't know. My life is a big mess. Everything is so complicated. (COMP)
A: Come on, nothing
be that bad. (COMP)
B: But promise me, you'll keep it secret. (COME)
A: Ok, I promise. So what's ' troubling you much? (COMP)
I've fallen love with my boss. (COMP)
A: Really? Is he married? (COMP)
Total 103 tokens. Context compression ratios are 7/103 (CCM-coneat) and 1/103 (CCM-merge).
Input: No, of course not. He is still single.
Output generated w/o context: I'm sorry, I'm not sure what you mean. Output generated by CCM-coneat: So what's the problem?
Output generated by CCM-merge: What's his problem?
Ground truth output: Then what's your problem?
cache in every streaming step. In Figure 8, we compare our approach to StreamingLLM (Xiao et al., 2023), which only stores the most recent keys/values in the sliding window. To ensure a fair comparison, modify the baseline method to have identical KV cache size 500k model Table 11, and conduct evaluation on the PG19 validation set (Rac al., 2020). Specifically, to 160 and the CCM size to 8, while compressing 64 tokens to a size of 2 at each compression step. The results in Figure demonstrate the effectiveness of our compression method in the streaming setting, outperforming the StreamingLLM approach.
approach at
every streaming step. We use the Pretrain+MettlCL
set the maximum KV size
4.2 ANALYSIS
In this section, we provide quantitative and qualitative analyses of our method. In Appendix E, we provide supplementary experimental results on training data sources, compression FLOPS, compression token length, larger model scales, and comparison other baselines, including AutoCompressor and MemoryBank (Chevalier et al., 2023; Zhong et al., 2023).
Effect of conditional LoRA In Equation (4), we compare the compression performance of condi- tional LoRA with the default unconditional LoRA. Ta- ble 4 shows evaluation results obtained using the iden- tical training recipe. The table confirms the consis- tent superiority of our conditional LoRA over the de- fault unconditional LoRA across all methods, including Gisting. In Appendix Table 21, we provide results on LaMP and DailyDialog, demonstrating that our condi- tional LoRA consistently improves the performance.
Table Test accuracy (%) of default LoRA and our conditional LoRA with LLaMA-7B on MetaICL time step 16.
Method
Default
Conditional (ours)
CCM-comeat CCM-merge Gisting
69.4
70.0 (+0.6)
66.3
69.6 (+3.3)
64.6
66.9 (+2.3)
Table illustrates the results of applying our approach to DailyDialog, using (COMP) token length of 1. The table shows that our methods continue a seamless conversation within the given context, while CCM-concat generates a response that better suits the overall context.
Qualitative results
5 CONCLUSION
We present a novel compressed context memory system that dynamically compresses contextual information, thereby enhancing the online inference efficiency of language models. To ensure effi- cient training, we develop a parallelized training strategy and introduce a conditional adapter. Our approach achieves reduced memory and attention FLOPS complexities compared to previous fixed- context compression methods. We validate the practical applicability of our approach through a comprehensive evaluation multi-task learning, personalization, and conversation applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by SNU-NAVER Hyperscale AT Center, Institute of Information & Com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning and NO. 2021-0-01343, Artificial Intelli- gence Graduate School Program (Seoul National University)], and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00274288) Hyun Oh Song and Sangdoo Yun are the corresponding authors.
REFERENCES
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. NeurIPS, 2016.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. NeurIPS, 2022.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint arXiv:2305. /4788, 2023.
Rewon Child, Scott Gray, Alec Radford, and lya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Hyung Won Chung, Le Hou, Shayne Longpre, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.11416, 2022.
Together Computer. Redpajama: open dataset for training large language models. https: 2023.
//github. http:///ggteeerrcmmpppeteeeeeeee
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. ACI,, 2019.
Leo Gao, Stella Biderman, Sid Black, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. arXiv preprint arXiv:2307.06945. 2023.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory-efficient backpropagation through time. NeurlPS, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ICLR, 2022.
DeLesley Hutchins, Imanol Schlag. Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers. NeurlPS, 2022.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. JCML, 2020.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. EMNLP, 2023.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. IJCNLP, 2017.
James Manyika. An overview of bard: an early experiment with generative ai. Technical report, Technical report, Google AI, 2023.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. NAACL, 2022.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305. 16300, 2023.
10

---

Published as a conference paper at ICLR 2024
Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens.
arXiv preprint NXXv:2304.008667 2023.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back-propagation. ICML, 2023.
Alec Radford, Jeftrey Wu, Rewon Child, et al. Language models are unsupervised multitask learn- ers. OpenAl blog, 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.1 1406, 2023.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint 1.X::2209.15189, 2022.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. Infor- mation Theory Workshop, 2015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.1397), 2023.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mitos. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.031700 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Aug- menting language models with long-term memory. NeurIPS, 2023.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. ICLR, 2015.
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. EMNLP, 2022.
Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. ICLR, 2022.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453,755, 2023.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, et al. Big bird: Transformers for longer sequences. NeurIPS, 2020.
Susan Zhang, Stephen Roller, Naman Goyal, et al. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205.01068 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. et Lmsys-chat- Im: A large-scale real-world I1m conversation dataset. arXiv preprint 2xXiv:2309.11992 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang. Memorybank: Enhancing large lan- guage models with long-term memory. arXiv preprint arXiv:2305. 10250, 2023.
11

---

Published as a conference paper at ICLR 2024
APPENDIX
A RELATED WORKS
Context compression Seminal works, such Memory Networks, have introduced novel models and computational approaches to efficiently store contextual information within limited space. en- hancing the inference efficiency language models (Weston al., 2015; Ba al., 2016). Recently, there have been efforts to compress frequently used prompts, aiming to enhance the inference ef- ficiency of large-scale language models. Wingate et al. (2022) advocate condensing prompts into concise soft prompts. Hyper- Tuping (Phang et al., 2023) attempts to convert prompts into model adapters, while Snell et al. (2022) propose distilling prompt information into the model parame- ters. AutoCompressor (Chevalier et al., 2023) and ICAE (Ge et al., 2023) propose auto-encoding approaches for compressing contexts into soft embeddings. Gisting (Mu et al., 2023) introduces learnable tokens designed to compress context information within attention hidden states. These previous methods focus on compressing fixed context to enhance reusability. In this study, we intro- duce task involving context compression during online inference and propose an effective approach for handling dynamically changing contexts.
Long context Transformer In terms of efficient context processing, our approach relates to the long context Transformer. Notably, Dai al. (2019) aims to increase the context length through at- tention hidden state caching, and Rac et al. (2020) proposes a strategy to compress attention hidden states. Efforts have also focused on reducing the complexity of attention operations (Child et al., 2019; Zaheer et al., 2020). These methods, which propose new attention mechanisms, require train- ing large models from scratch, making it challenging to leverage existing pretrained models. The following works propose recurrent meemory-aggmentdd approaches (Bulatov et al., 2022; Hutchins et al., 2022), while Wu et al. (2022) propose k-nearest retrieval of attention key/value pairs to man- age long contexts. These retrieval-basee
approaches, including MemoryBank (Zhong et al., 2023) and LongMem (Wang et al., 2023), primarily focus on the token-level retrieval process, with less emphasis on memory compression. However, LLM's keys and values demand a significant amount of storage, reaching several hundred megabytes even for a context length of 1024. Such high storage requirements can become problematic scenarios such user-level personalization and conversa- tion systems, Recently, notable attempts have been made to extend the context length of LLaMA (Mohtashami & Jaggi, 2023; Tworkowski et al., 2023). While these studies concentrate on handling fixed contexts, our approach aims to dynamically compress expanding contextual information within a compact memory space.
B DISCUSSIONS
Application-specific compression When focusing specific applications, the size of compress- ible contextual information becomes larger than when considering general scenarios (Tishby & Za- slavsky, 2015). This indicates that application-specifi. compression modules can achieve higher compression efficiency compared to their more general counterparts. Similar to fine-tuning foun- dation models for specific applications in various industries, an application-specific compression module can be employed to achieve superior compression capability. It is noteworthy that our method is opplication-agnostie meaning it can be applied effectively a wide range of scenar- ios in a data-driven manner. Obtaining a compression module without requiring application-specifi knowledge or manual adjustments holds practical value. Furthermore, as demonstrated in the gen- eralization test presented in Table 10, our approach shows generalization capabilities across various applications and can be flexibly adapted to different scenarios.
Limitations and future works While our model is capable of generalizing to new tasks or user contexts at test time, training broadly applicable model for arbitrary applications remains impor- tant future direction. Moreover, despite surpassing existing compression baselines in performance, approach still declines in performance compared when utilizing the full context. Developing compression techniques that can ensure a higher level of information preservation remains a crucial direction for future research.
12

---

Published as a conference paper at ICLR 2024
C DATASET DETAILS
Table 6 illustrates training data formats. While all datasets adhere to the same format, the content within each context and input varies. In Table 7, we provide statistics for each dataset. In the case of MetalCL, we employ the high-tooloow-rssource setting consisting of total of 61 training tasks and 26 test tasks, which is the most representative setting (Min et al., 2022). The token length of demonstrations varies depending token length of 256 in both the training and evaluation sets. Taking into account the GPU memory constraints, we set the maximum token length for the entire context to 1024. For LaMP, evaluations in the personalized categorization setting (Salemi et al., 2023). The dataset exhibits relatively lower token length variations than MetaICL. Regarding DailyDialog as more than 90% of the test samples have dialogue turns of 12 or fewer, we set the maximum time step to 12.
the dataset type. We filter out demonstrations exceeding
conduct
Table 6: Illustrative format of each dataset sample.
Dataset
Context (,(+) with (COMP) token
Input 1,(t)
MetalCL
Demonstration for task (COMP) ... Demonstration for task (COMP)
problem task
LaMP
Profile for user i (COMP) ... Profile user (COMP)
query for user
DailyDialog
Turn 1 from dialog (COMP)... Turn from dialog i (COMP)
Tum f+1 from dialog
Table 7: Descriptions for datasets considered.
MetalCI.
LaMP
DailyDialog
Average token length of context cc-) at each time step Maximum token length of context c(-) at each time step Maximum time step T
50
50
15
256
100
128
16
16
12
D EXPERIMENT SETUP
Training setup We begin by fine-tuning LLaMA pretrained models (Touvron et al., 2023) on each training dataset. The performance of these models with full contexts establishes the upper-bound performance of our experiment. We then perform LoRA fine-tuning on these models learn com- pression capabilities. To ensure a fair comparison, we employ identical LoRA configurations and training protocols across all methods considered. All experiments undergo training with a fixed number of data, ranging from 10k to 250k, depending on the datasets. Individual training runs take 3 to 24 hours on a single NVIDIA A100 with 80GB memory. To account for limited GPU memory, we set the maximum token length of each training sample to 1024. Regarding Gisting, utilizing our conditional adapter enhances performance (Table 4). Based on this observation, we report the improved performance achieved by applying our conditional adapter in the main experiment. To confirm the effectiveness of compression, we adjust the length of (COMP) tokens to attain a suffi- ciently large compression factor of approximately 8 for each dataset. For specific training recipes and hyperparameters, please refer to Appendix D.
Training protocol and hyperparameter Our approach first fine-tunes the pretrained LLaMA models on each training dataset, following the training recipe in Table and the LORA configuration in Table 9. The resulting LORA adapters are then merged with the pre-existing model weights. Us- ing this fine-tuned model as foundation, we proceed train the compression capability. To ensure a fair comparison, we optimize both compression baselines and our methods using the same train- ing recipe Table and LORA configuration in Table 9. We jointly optimize the embeddings for (COMP) tokens, where (COMP) tokens at different time steps share the same embedding. All training processes are conducted on a single A A10 PCIe 80GB GPU and take to 24 hours, depending on the dataset.
Evaluation method For MetalCL and LaMP, we measure the accuracy for multi-choice questions by comparing the average log- ligeeiildoood on tokens of each answer choice, following the official evaluation codes provided by MetalCI. (Min et al., 2022).
13

---

Published as a conference paper at ICLR 2024
Table Training recipes of our experiments for LLaMA models.
MetalCL
LaMP
DailyDialog
Training steps
2000
300
1000
Batch size
128
128
128
# training samples
256k
38k
128k
Learning rate
3e-4
3e-4
3e-4
Learning rate scheduling
Cosine
Cosine
Cosine
Mixed precision
FP16
FP16
FP16
(COMP) token length
8
4
2
Table 9: LoRA configurations for LLaMA models. We use this configuration for all experiments.
Argument
Setting
Target modules
q wtt.:k.proj..projooooo
Rank
Alpha
16
Dropout
0.05
E ADDITIONAL EXPERIMENTAL RESULTS
E.I MAIN ANALYSIS
Unified compression adapter We train single compression adapter with LLaMA-7B on the mixture of the MetaICL training tasks and the SODA conversation dataset. We follow the training recipe in Table 8, while we train a model for 4k steps. We train the Gisting and Compressive Transformer baselines using the same dataset and training protocol. We use the (COMP) token length of 2 for CCM-concat and 8 for CCM-merge. Finally, we test the model on the MetalCI. unseen test tasks, LaMP, and DailyDialog at the corresponding maximum time step.
Table 10 demonstrates the generalization ability of our approach on datasets and scenarios unseen during training. Specifically, CCM-concat maintains the best compression performance by a large margin compared to baseline methods. We observe that CCM-merge has increased performance degradation by compression compared to the acenario-specife settings (e.g., the LaMP accuracy degradation by compression increased from 1.2% to 5.1 3)). However, the other compression base- lines have a larger performance gap by compression, demonstrating our approach achieves the best generalization performance among the baselines.
Table 10: Evaluation of a single model trained on MetalCL and SODA training datasets. Memory refers to the peak memory required for attention keys/values during inference.
Test dataset
Metric
No context
Full context
Gisting-online
Compressive
CCM-coneut
CCM-merge
Accuracy (%)
53.6
70.0
59,9
65.0
68.7
67.8
MctalCL
Memory (MB)
50
630
82
82
82
66
Accuracy (%)
37.0
76.4
67.6
58.4
75.2
71.4
LaMP
Memory (MB)
50
82
DailyDialog
Perplexity
11.51
7.02
9.04
9.19
Memory (MB)
32
252
Effect of training data sources To analyze the impact of data used for compression adapter training, we compare the performance of CCM-concat trained with various data sources. Table 11 presents evaluation results using RedPajama-V2 (Computer, 2023) and LmSys-Chat (Zheng et al., 2023) as the base training data. The table shows that the evaluation performance improves when us- ing training data from similar sources. Particularly, when adding a new data source, the performance in the added data source significantly improves with marginal performance decrease in the existing data sources. We believe that different data sources have different compressible information spaces, indicating the importance of constructing training data tailored to the application scenario. Lastly,
14

---

Published as a conference paper at ICLR 2024
worth noting that increasing the amount of training data enhances overall performance (last row in Table 11), underscoring the significance both the quantity and quality of the training data.
Compression performance gap across different data sources used train compression adapter. We measure the perplexity gap compared to the full context method at the maximum time step (accuracy for MetalCL). We use CCM-coneat with (COMP) token length of 2 on LLaMA-7B.
Table 11:
Evaluation dataset
Training dataset
training data
Pretrain
SODA
DailyDialog
MetaICL
Pretrain (= RedPajama + LmSys-Chat)
500k
-0.55
-0.22
-0.74
-4.9%
Pretrain MetalCL
500k
-0.59
-0.26
-0.82
-1.2%
Pretrain MetalCL. SODA
500k
-0.61
-0.10
-0.54
-1.3%
Pretrain MetalCL + SODA
750k
-0.57
-0.09
-0.53
-1.1%
FLOPS analysis
Regarding FLOPS, our approach has two notable effects:
Reduction in attention FLOPS due to the shortened context.
Computation overhead incurred by the compression process.
The reduction in attention FLOPS becomes more pronounced as the number of processed tokens during inference increases. In Table 12, we compute the minimum length of tokens required to be processed during inference, where the benefits from the shortened context outweigh the compression overhead. Our analysis is based on a context token length of 50, according to the dataset statistics in Table With (COMP) token length of 1, our approach reduces the total computation FLOPS when the length of the processed token during inference surpasses 504. We summarize the results on larger (COMP) token lengths in Table 12.
Table 12: Compression FLOPS overhead analysis MetaICL with LLaMa-7B. Threshold refers to the minimum token length required during inference for the reduction in attention FLOPS to outweigh the compression overhead. We assume that the token length of context c(t) is 50, according to the MetalCL and LaMP datasets' statistics (Table 7).
(COMP) token length
2
4
Context compression factor
x50
x25
x13
x6
Threshold (inference token length)
504
1029
2148
4706
In-depth performance analysis We measure the generation performance of our compression ap- proach using the RougeL metric in Table 13. The results verify that our methods deliver the most accurate generation performance compared
other baselines. However, in the case of RougeL, there is pronounced decrease in performance compared to the full context method, whereas, in the case accuracy, the performance drop less than 1%. Upon closer examination of the generated outputs with compressed context, we identify instances where synonyms are generated (e.g.. "Dif- ferent" and "Dissimilar" in the medical medical. questions_paii task) or variations in letter casing are present (e.g., "Hate" and "hate" in the serettevvl_haae task). These observations suggest semantic equiv- alence between the original and generated results, albeit differences in expression. These findings suggest that our approach performs particularly well in situations where prioritizing preferences or nuances outweighs the need for exact phrasing.
Table 13: Evaluation of RougeL and accuracy metrics with LLaMA-7B on MetalCL test tasks.
context
Full context
Gisting-online
Compressive
CCM-coneat
merge
RougeL 12.3
61.4
37.9
47.9
54.7
48.3
Accuracy (%) 51.7
70.8
$7.7
67.8
70.0
69.6
Design choice of merge function In the main experiments, evaluate update method based on the arithmetic average of the compressed states up to the present time, i.e. a, 1/t. An- other natural design choice is exponential moving average (EMA), where o, set constant
15

---

Published as a conference paper at ICLR 2024
value. This strategy weighs higher importance recent information compared to the arithmetic average. Table 14 provides a comparison between the arithmetic average and EMA with at = 0.5, on DailyDialog with LLaMA-7B. The results indicate that both methods yield similar performance. When forming the compression state h(),, our method involves referencing the previous memory Mem(t-1) (Figure 2). We believe this enables the preservation of overall context, with expo- nentially decreasing coefficients for past states by EMA.
Table 14: Comparison of merge function design choices with LLaMA-7B on DailyDialog.
Method Time step
2
8
12
EMA Arithmetic average
7.49
7.06
6.79
6.49
6.38
7.47
7.06
6.87
6.54
6.34
Additional memory-performance graphs In Figure 10, we present graphs illustrating the rela- tionship between attention KV memory and performance across increasing time steps for MetaICL, LaMP, and DailyDialog. The figure comprehensively compares all methods introduced in our main text, including fixed-contes compression method such Gisting. From the figure, we verify that our methods exhibit the best cemorry-prformannt efficiency. Specifically, our methods achieve superior performance while requiring minimal attention KV memory when compared to existing compression baselines.
Full context
Gisting - Gisting-online
Compressive
CCM-coneat (ours) - CCM-merge (ours)
(a) MetalCI.
(b) LaMP
(c) DailyDialog
72
86
8.0
83
Accuracy (%)
Accuracy (
66
Perplxiity
6.8
80
63
6.2
77
60
5.6
100
200 Peak KV memory (MB, log-scale)
120
Poak KV momoryy (MB, log-scale)
Peak KV memory (MB, log-scale)
Figure 10: Test performance of methods with LLaMA-7B 7B over increasing time steps in an online inference scenario. The z-axis refers to the peak memory space occupied by attention keys/values during compression and inference processes at each time step. Here, the time steps span from 1 to 16, except for DailyDialog, which covers a range of 12.
E.2 MODEL AND COMPRESSION TOKEN LENGTH
Different model architecture We evaluate our method with an encoder-decoder structured model, Flan-T5-LL -a-raee (Chung et al., 2022). Since there exists overlap between the training set of Flan- T5 and the MetalCL dataset (Min et al., 2022), we conduct an evaluation using the LaMP dataset. Table 15 presents the evaluation results time step 16. While both Gisting and Compressive Trans- former exhibit a significant drop in accuracy compared to the full context method, our methods achieve the best performance while requiring less key/value memory on the Flan-T5 architecture.
Table 15: Test accuracy and peak key/value memory size with Flan Tan-Te-Large on LaMP time step 16. We evaluate performance across five different random seeds for user profile order.
No context
Full context
Gisting-onling
Compressive
CCM-concat
CCM-mert -ere
Accuracy (%)
71.1 0.0
81.8 + 0.3
78.4 0.3
79.7 +
81.9 + 0.2
82.1 1:
Memory (MB)
20
152
32
32
32
21
16

---

Published as a conference paper at ICLR 2024
Length of compression token In Table 16, analyze the performance of method across varying compression token lengths. In general, increasing the token length leads to a slight im- provement in performance. For MetalCL, experiment shows a I % reduction in perplexity as token length increases. However, when comparing approach to the no-context method, the performance differences attributed to the compression token length are not significant. For example, our method outperforms the no-context approach by approximately 18% the MetaICL experiment. In our main experiment, we set the compres- sion token length according to the average context length of the target dataset, ensuring consistent compression rates across datasets. We provide detailed configuration values in Table 8.
observe 1% accuracy gain, while the DailyDialog
Table 16: Analysis of (COMP) token length with LLaMA-7B at the maximum time step (Table 7). Here, concat refers to CCM-concat, and merge denotes CCM-merge.
(a) MetaICL (Accuracy %). No con- text: 51.6%
(b) LaMP (Accuracy %). No con-
(c) DailyDialog (Perplexity). No
Full context: 70.8%.
text: 69.5%
Full context: 85.1%.
context: 10.3 Full context: 5.85.
(COMP) ) token length
(COMP) token length
(COMP) token length
2
2
4
concal
69.5
69.3
70.0
70.0
concat
84.3
83.9
84.7
concat
6.51
6.37
6.26
merge
68.5
68.1
68.3
69.6
merge
83.4
84.2
83.9
merge
6.67
6.62
6.63
Larger model scale In Table 17, we provide evaluation results with LLaMA-13B on MetalCL. Consistent with 7B models, method exhibits the best performance among the compression base- lines while requiring smaller peak attention KV memory.
Table 17: LLaMA-15 13B test accuracy and peak attention KV memory MetalCL at time step 16.
No context
Full context
Gisting
Gisting-online
Compressive
CCM-concat
CCM-merge
Accuracy (%)
51.4
72.1
66.7
62.5
66.1
70.7
68.6
Memory (MB) 78
984
919
278
278
278
103
E.3 COMPARISON TO BASELINES
Comparison to recurrent compression methods We conduct comparative analysis with RMT and AutoCompressor that recurrently compress contexts into token embeddings (Bulatov et al., 2022: Chevalier al., 2023). These approaches fine-tune OPT pretrained models (Zhang et al., 2022) on the Pile dataset (Gao et al., 2020) to learn compression capabilities. For evaluation, we utilize the fine-tuned models available on the official GitHub repository' We conduct separate ex- periments on each baseline because the released RMT and AutoCompressor models show different performances without compression (AutoCompressor Table 18 and RMT in Appendix Table 22). For a fair comparison, we also provide fine-tuned results of the baseline models on MetalCI. training tasks using identical training steps to ours, denoted as Compreeerr finettue and RMT-finetune. As shown in the tables, compression methods demonstrate superior performance and efficiency. Specifically, RMT and AutoCompressor necessitate recursive model computation at each training step, incurring significant computation time. As shown in Table 18, AutoCompressor requires ap- proximately 7 longer training time per sample than our approach. Meanwhile, our methods exhibit superior performance while using less key/value memory, demonstrating its effectiveness.
Table 18: Comparison with AutoCompressor OPT-2.7B on MetalCL test tasks at time step 16. We measure the training time using identical samples A100 GPU. We evaluate performance across five different random seeds for demonstration order.
No context
Full context
AutoComp:
AutoComp-finetuue
CCM-coneat
CCM-merge
Accuracy (S)
41.4 + 00
54.2 + 0.5
48.1 0.5
50.9 + 0.4
53,5 + 0.5
52.3 0.3
Peak KV memory (MB)
31
394
156
156
111
41
Training time per sample (ms)
1330
1330
195
17

---

Published as a conference paper at ICLR 2024
Comparison to fixed-context compression In Table 19, present evaluation results of Gisting with the fixed-context compression setting described Figure 5-b. While having the same inference complexity CCM-merge, the fixed-context setting incurs significant memory demands during compression. On the other hand, our approach maintains minimal memory requirements for both stages, having low peak memory usage. Moreover, method improves the performance by 3%p compared to Gisting, validating the effectiveness of our training strategy in online scenarios.
Table 19:
Comparison to a fixed-context compression method (Gisting) with LLaMA-7B on MetalCL test tasks time step 16.
Full context
Gisting
CCM-concat
CCM-merge
Accuracy (%) Peak KV memory (MB)
70.8 +
66.9 + 0.2
70.0 + 0.2
69.6 0.1
178
Comparison to text summarization MemoryBank proposes reducing context size through text summarization during language model interaction (Zhong et al., 2023). However, this approach comes with additional computational costs for summarization and the overhead of processing the summarized text for subsequent inference. In contrast, our approach allows for more efficient in- ference without the aforementioned overhead by caching key/value pairs of compression tokens. Following MemoryBank, we conduct experimental comparisons with LLaMA-7B on DailyDialog. Specifically, we use the summarization prompt from MemoryBank to compress context through OpenAl gpt-3.5-turbo API (ChatGPT) and then evaluate models with summarized contexts. Ta- ble 20 shows the test perplexity of methods. The results confirms that our approach achieves supe- rior performance with smaller context memory size, demonstrating the effectivness of our key/value compression approach.
Table 20: Comparison to a text summarization method with LLaMA-70 7B on the DailyDialog test set.
No context
Full context
MemoryBank
CCM-concat
CCM-merge
Perplexity
10.6
5.59
7.06
5.98
6,34
Compressed context length
0
222
60
24
2
18

---

Published as a conference paper at ICLR 2024
Table 21: Evaluation results default LoRA and our conditional LoRA with LLaMA-7B.
(a) LaMP (Accuracy %)
(b) DailyDialog (Perplexity)
Default
Conditional (ours)
Default
Conditional (ours)
CCM-concat
83.9
84.7
CCM-concat CCM-merge
6.01
5.96
CCM-merge 82.6
83,9
6.42
6.33
Comparison with RMT OPT-2.7B MetalCL time step 16. We measure the train- ing time using identical samples on an A100 GPU. We evaluate performance across five different random seeds for demonstration order.
Table 22:
No context
Full context
RMT
RMT-finetune
CCM-concat
CCM-merge
Accuracy (%)
42.1 0.0
54.5 0.4
44.4
0.4
50.0 + 0.3
52.3 + 0.4
52.2 = 0.3
Peak KV memory (MB)
31
394
63
63
111
41
Training time per sample (ms)
1330
1330
195
195
Table 23; Test accuracy (%) on MetalCL with LLaMA-7B.
The test set identical across time steps.
Time step
No context
Full context
Gisting-online
Compressive
CCM-concat
CCM-merge
51.7
62.8
61.7
59.6
63.0
62.7
2
51.7
64.4
62.6
60.7
64.4
65.5
4
51.7
68.7
62.7
66.7
67.8
68.5
8
51.7
69.9
59.7
67.6
68.9
69.3
16
51.7
70.8
57.7
67.8
70.0
69.6
Table 24: Test accuracy (%) on LaMP with LLaMA-7B. The test set is identical across time steps.
Time step
No context
Full context
Gisting-online
Compressive
CCM-concat
CCM-merge
69.5
79,1
78.5
75.0
78.6
79.1
2
69.5
80.6
77.3
77.2
79.5
80.4
69.5
78.5
79.1
82.3
82.2
8
69.5
83.1
79.6
81.6
84.0
83.5
16
69.5
85.1
78.7
82.8
84.7
83.9
Table 25: Test perplexity on DailyDialog with LLaMA-7B.
Time step
No context
Full context
Gisting-online
Compressive
CCM-concat
MeM-ergge
1
8.93
6.97
7.42
7.60
7.38
7.47
2
9.06
6.62
7.47
7.14
6,97
7.04
4
9.33
6.30
7.47
6.91
6.64
6.85
8
9.85
5.84
7.68
6.52
6.17
6.53
12
9.67
5.56
7.44
6.38
5.91
6.27
19


Reviews:
[{"review_id": "5tIU8XBC9Z", "reviewer": "Reviewer_JGpv", "summary": "The paper proposes a novel compressed context memory system for dynamically compressing contextual information during online inference with language models. This allows the model to handle continually expanding contexts efficiently. \n\nThe main contributions are:\n\n- A compressed memory framework that condenses context into a compact representation which is dynamically updated during online inference. This reduces memory usage and computation compared to using the full context.\n\n- A parallel training strategy using masked attention to learn the context compression in a single forward pass.\n\n- A conditional adapter applied only to compression tokens to avoid overfitting to inputs during training.", "questions": "1. **Task-Specific Limitation**: The framework is mentioned to be task-specific and requires retraining for each new task. Could you elaborate on how this limitation affects the scalability of the proposed method, especially in real-world applications where diverse tasks are common? Additionally, are there any plans or potential strategies to make the framework more task-agnostic?\n\n2. **Additional Resources for Training**: Given that the compression module necessitates extra data and computational resources for training on each task, can you provide a quantitative analysis of the additional resources required compared to other existing methods? How does this additional overhead impact the practicality of adopting your framework, particularly in resource-constrained environments?\n\n3. **Inability to Generalize**: The framework's inability to generalize to new tasks could be a significant disadvantage. Have the authors considered hybrid approaches that combine task-specific and task-agnostic components to mitigate this limitation? If so, what were the challenges or outcomes of these considerations?\n\n4. **Performance Gap Analysis**: The paper notes a clear performance gap between the compressed and full context models. Could the authors provide a more in-depth analysis or hypotheses as to why this gap exists? Are there particular types of data or tasks where this performance gap is more pronounced?\n\n5. **Basic Memory Update Functions**: The memory update functions in the proposed framework are described as basic. Could the authors provide examples or discussions on more sophisticated memory update mechanisms that could potentially enhance the framework’s capabilities? What prevented the integration of these more advanced mechanisms in the current version of the framework?\n\n6. **Lack of Comparison to Recurrent Memory Approaches**: The comparison in the paper is mainly drawn with simple adaptations of fixed-context compression methods. Could the authors justify the choice of these particular baselines and discuss the reasons for not including a direct comparison with recurrent memory approaches, such as linear Transformers? How might the results differ with these additional comparisons?\n\n7. **Explaining Effectiveness of Compressed Context**: In relation to the performance gap, could the authors shed light on any specific scenarios or data types where the compressed context performs particularly well or poorly? Understanding these nuances could help in better positioning the framework and identifying areas for improvement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of efficiently handling expanding contexts is highly relevant given the online nature of systems like ChatGPT. The paper addresses an important open challenge.\n\n- The method is flexible and broadly applicable to diverse online inference scenarios like multi-task learning, personalization and conversation.\n\n- Empirical evaluations across three datasets substantiate the memory and computation advantages over baselines. The method achieves slightly lower performance than the full context with 5x smaller memory.\n\n- The parallel training strategy is effective in enabling large model optimization. The conditional adapter improves compression capability.\n\n- The complexity analysis clearly articulates the efficiency benefits, and ablation studies validate the design choices.", "weaknesses": "- The main limitation of the proposed compression framework is that it is task-specific. The compression module must be trained for each task, which requires additional data, computation, and cannot generalize to new tasks. This is a significant drawback in the context of foundation models which are trained on large datasets for general-purpose use.\n- There is still a obvious gap in performance between the compressed and full context models. The paper does not provide a clear explanation for this gap. The authors should provide more analysis into why the compressed context is less effective.\n- While the compression framework is novel, the proposed memory update functions are basic. More sophisticated memory update mechanisms could further enhance capability.\n- The comparison is primarily with simple adaptations of fixed-context compression methods. A direct comparison to recurrent memory approaches, such as linear Transformers, would be more informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "720yspkERI", "reviewer": "Reviewer_JvdE", "summary": "In the LLMs era, during long-term online user-model interactions, a lot of past dialogue history cannot fit the context length of the model and thus have to be compressed into memory to ensure that the model can memorize the past dialogue. It is an emerging and interesting research question. This paper introduces a novel method to compress the long context memory in the online user-machine interactions, named CCM as well as two variants, merge and concat. As LLM is hard to fine-tune to engage such CCM, the author also adopts LoRA adapter to fine-tune LLM in a lightweight setting to engage the model to learn to compress the memory.", "questions": "1. For Table 4 and 5, can you add a new row to present the memory size using number of tokens, which is a more intuitive metric than the disk size in MBs?\n\n2. For CCM-merge, it is a vanilla average of all memory features, which does not make sense. Typically, the latest memory might be more important and contribute more to the current dialogue turn. Is it possible to add a weighted coefficient in terms of time steps for a weighted merge on all memory features, i.e. $$Softmax([1:t_i])\\in R^t$$? \n\n3. Missing references which have been mentioned in weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall sound. The method design is concise, effective, and efficient. Compared with retrieval-based method to re-compute the sentence embedding, the CCM can directly adopt the KV cache of introduced <COMP> token as the memory vector for one utterance and utilize them in further inference. To engage the LLM to utilize such CCM, the parallel training and LoRA adapter are designed well for efficient adaptation.\n\n2. The CCM is efficient in both training and inference. Firstly, there is no need to re-compute the sentence embedding and instead caching the attention keys and values. Secondly, the memory storage cost, the compression ratio, and the algorithmic complexity all demonstrate the efficiency of the method. Thirdly, LoRA based adapter tuning brings a lot of efficiency in memory-engaged adaptation.\n\n2. The evaluation is comprehensive and diverse. Three important benchmarks, MetaICL, LaMP, and DailyDialog are selected for evaluation and MetaICL covers 26 tasks with high-diversity.", "weaknesses": "1. The CCM method is not that novel and has been explored well in some important early milestones before the creation of Transformer, i.e., Memory Networks, Fast Weights to Attend Recent Past. The author should mention and discuss the relation with these methods. Additionally, the Compress Transformer should be briefly introduced as it is not a universally known preliminary for readers.\n\n2. In terms of the baselines, in the main tables, CCM is only compared with “no context\" and \"full context\" baselines on accuracy, which lacks sufficient comparison for demonstrating the effectiveness of the method. As least, the retrieval-based method should be considered as an important baseline and it is now the universally-adopted method for memory compression. If I understand the paper correctly, the token embedding produced by <COMP> token is the same as a sentence embedding. The author can follow MemoryBank for adopting retrieval-based method to compress the long-context during online interactions. Other baselines like LongMem and UnlimitedFormer can be also considered but not necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fR3Rsgvg3g", "reviewer": "Reviewer_UgoG", "summary": "This paper proposes an interesting method to compress context memory. Compared with previous context compression work, the main novelty and contribution claimed in this paper is its ability to handle dynamic contexts. Its idea is similar to RMT, letting the transformer to work in a recurrent memory mechanism. By inserting a compression token in the in-context manner like Gist, Autocompressor and ICAE, the method can process the context recursively into a constant memory cost. The experiments conducted on Meta-ICL, LaMP and DailyDialog dataset show a promising compression rate with full-context performance.", "questions": "See the weakness section", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- A interesting method to compress contexts in the few-shot learning setting.\n\n- The results evaluated in the few-shot learning tasks show the effectiveness and superiority over the conventional approaches like RMT and Gist.", "weaknesses": "While this paper presents a seemingly promising solution to long contexts, I have significant concerns about several limitations.\n\nFirstly, one of the main focuses of this paper is handling dynamic context for interaction. Judging from its experimental design, it mainly conducts experiments with a fine-tuned LLM for few-shot learning scenarios, which are generally simpler tasks, all being multi-choice, or classification tasks. The methods primarily compared in this paper are general context compression or long context handling methods (general and generative tasks). This involves 3 issues. The first issue: for simpler tasks like classification, compression is relatively easy (this is why previous models were easily distilled but GPT was not; I believe it's not because GPT is hard to distill, but because GPT is not for a specific task, but a general model. The authors may better know what I said if reading the paper about information bottleneck: https://arxiv.org/pdf/1503.02406.pdf). Therefore, it's not surprising that this method achieves good compression results (model size compression and context compression are similar, both reducing model capacity), but I believe it's hard for this paper's method and results to scale to general scenarios. At least in this paper, I didn't see any general tests to prove its compression effect. The second issue: for the few-shot learning setting, adding compression tokens after the demonstration makes sense, but for general scenarios, there is no definite boundary to limit compression tokens, making this method hard to generalize. Even if a compression token can be added every K tokens, this approach would lead to inefficient training, as a large amount of sampling is required to ensure the model learns well for each position. The third issue: the main setting of this paper is few-shot learning, and its main claim is online interaction. But for few-shot learning, there doesn't seem to be any online interaction. Users usually provide all demonstrations at once for the model to give an answer, and it's hard for me to imagine a setting where users incrementally provide demonstrations to the model.\n\nSecondly, for dialogue tasks, this is a context compression for a specific task (more preciesely, for a specific dataset). Similar to what I mentioned above, if it's for a specific task/benchmark, there is actually a lot of compression space, which has been discussed in many previous works, such as: https://arxiv.org/pdf/2301.12726.pdf. For context compression of a specific task, even without this method, other methods should also achieve good compression results.\n\nThirdly, I am not entirely convinced by the results presented in the paper. For example, in Table 15, the performance of RMT is almost the same as that of No context, which is hard to believe. Moreover, Table 15 is an experiment conducted on OPT-2.7b. The few-shot learning ability of the 2.7b OPT model, as far as I understand, should be very weak, and changes in the order of sample arrangement will significantly affect the results. For a method like this paper's, which is similar to a recurrent method, it should be very easily influenced by later samples. Unfortunately, I didn't see any discussion about this.\n\nSome questions:\n1. Figure 4 is a little confusing. In (a), what do the blocks in different colors mean?\n2. It seems that this work uses 1 single 80GB A100 for training. Could the authors provide more details about the fine-tuning process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s2cM2Ff25b", "reviewer": "Reviewer_Svka", "summary": "This paper proposes a way to compress LLM context in an online setting, such as conversation, where more information could arrive and needs to be compressed. Here are the contributions:\n1. Extended the gisting technique to multi-context settings, and demonstrated the gains are better than naively adapting gisting to this setting. Further, the authors show that the proposed technique has very little gap with no compression, on three datasets.\n2. The authors also have proposed a parameter efficient method for training this mechanism, which doubles as a regularization. They also proposed an efficient way to train the model, by parallelizing the originally sequential context segments.\n3. Finally there are ablation studies on various design choices, such as the conditional adapter, the model architecture etc.", "questions": "1. In eq (1), $h(t)$ is conditioned on $Mem(t - 1)$. However, in figure 4, and the description of the parallelized training procedure, it's unclear how this could be done, as all the $Mem(\\*)$ are computed after $h(\\*)$, for the same layer. Do you condition $h(t)$ on the previous layer's $Mem(t - 1)$? If so please update the text to make it obvious.\n2. The motivation for the conditional adapter is to regularize against overfitting. The authors only reported the training loss of LLAMA 7B without context. To make it more clear on the overfitting situation, could you also report the loss with context, and the test performance with and without context.\n3. The masking doesn't seem to be autoregressive in figure 4b?\n4. Table 3 only has numbers on MetaICL. What about the other two datasets?\n5. In section 4.2, on different model architecture, have you tried non instruction tuned T5? From what I can tell, instruction following is not necessary for the three datasets used in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The exact method proposed is novel. On the three presented datasets, it seems to work pretty well, judging from the small gap between compressed and uncompressed setups. The problem this paper is trying to address is very important to the field, and I believe this work has significant contributions. Finally, the paper is generally easy to read.", "weaknesses": "This method could be evaluated on more diverse datasets, such as those used for long context, by utilizing a sliding window for example. From my perspective, this work has the potential to be applied more broadly beyond ICL, or dialog. It'll also be nice to have the comparison with RMT and AutoCompressor in the main text, as they are very relevant for this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)