Adapting Language Models to Compress Contexts
Authors: Alexis Chevalier
Alexander Wettig
Anirudh Ajith
Danqi Chen
Venue: EMNLP 2023 Main (2023)
Keywords: language models
transformers
long-range language modeling
retrieval-augmented language modeling
in-context learning
TL;DR: We train language models to compress text into soft prompts.

Abstract: Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments, and summary vectors from all previous segments are used in language modeling. We fine-tune OPT and Llama-2 models on sequences of up to 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompressors on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstrations, increasing accuracy while reducing inference costs.  Finally, we explore the benefits of pre-computing summary vectors for large corpora by applying summary vectors to retrieval-augmented language modeling and a passage re-ranking task.
Overall, AutoCompressors emerge as a simple and inexpensive solution to extend the context window of LMs while speeding up inference over long contexts.

Adapting Language Models to Compress Contexts
Alexis Chevalier Alexander Wettig' Anirudh Ajith
Danqi Chen
Department of Computer Science & Princeton Language and Intelligence Princeton University
{achevalier, wwwranh.rrhc. aji theeijiiteprrintoon edu
fachevalier,aniruuh.ajjjij (awettig, fangic)ecs.princeton edu
Abstract
use for language modeling
summary vectors
Transformer-based language models (LMs) are powerful and applicabe tools, but their usefulness is constrained by finite context window and the expensive computa- tional cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These language models are capable of compressing long contexts into pact summary vectors, which are then acces- sible to the model as soft prompts. Summary vectors trained with unsupervised objec- tive, whereby long documents are processed in segments, and summary vectors from all previ- ous segments are used in language modeling. We fine-tune OPT and Llama-2 models on se- quences of up 30,720 tokens and show that AutoCompressors can utilize long contexts to improve perplexity. We evaluate AutoCompres- on in-context learning by compressing task demonstrations and find that summary vectors are good substitutes for plain-text demonstra- tions, increasing accuracy while reducing infer- ence costs. Finally, we explore the benefits of pre-computing summary vectors for large pora by applying summary vectors to retrieval- augmented language modeling and a passage re-ranking task. Overall, AutoCompressors emerge as simple and inexpensive solution extend the context window of LMs while speeding up inference over long contexts.'
LM
summary vectors
LM
summary
tokens
randomly segmented inpur
Figure 1: AutoCompressors process long documents by recursively generating summary which are passed as soft prompts to all subsequent segments.
2022). With this rise in popularity comes the chal- lenge for researchers to make LMs more efficient, to speed up inference and to deploy LMs at scale, while increasing their versatility, thus allowing users process more data in ways.
With these goals in mind, we propose to teach pre-trained LMs the ability to compress text into summary vectors. Summary vectors short soft prompts (Lester al.. 2021), one or two orders of magnitude shorter than the pre-compressed plain text, that are obtained from the output states of a language model. Summary vectors serve two gen- eral purposes: they can help extend the language model's context window to very long documents with minimal computational overhead, and they help speed up inference text for which summary vectors have been pre-computed and cached.
1 Introduction
Transformer-based (Vaswani et al., 2017) language models (LMs) have recently seen a sharp rise in popularity and are now receiving millions of queries, processing billions of tokens, and generat- ing text for wide variety of applications (Brown al., 2020; Touvron et al., 2023; Zhang et al.,
Our models, which we call AutoCompressors, are trained with simple unsupervised learning objective that encourages the model to store essen- tial information in the summary vectors. Summary vectors are produced segment by segment from long documents and are used to improve language modeling in future segments (Figure 1). Our work
AW contributed equally. This work was done when AC was at the Institute for Advanced Study visited Princeton NLP group.
'Our code and models are publicly available at hhtp://ghhh....coommmmcoooom////

---

builds on the recently proposed RMT architecture (Bulatov al., 2022) with a crucial difference: we introduce summary accumulation, in which sum- mary vectors from all segments are concatenated to produce the summary of the entire document. We also train AutoCompressors with randomly seg- mented inputs they can better compress contexts of variable lengths in downstream tasks. We show that these innovations improve long-range informa- tion retention and enable new ways of reasoning over multiple passages.
key features of AutoCompressors. (2) We show that summary vectors encode useful information for downstream tasks and can be used reduce the inference cost of in-context learning. (3) We demonstrate the benefits of pre-computing sum- mary vectors for large corpora and using Auto- Compressors in conjunction with retrievers.
2 Related Work
Soft prompts Soft prompt tuning is an effective method to adapt pre-trained Transformers without updating existing parameters (Lester et al., 2021; Zhong et al., 2021; Liu et al., 2022). Newly ini- tialized embeddings are prepended to the input se- quence (the "soft prompt"), and optimization is performed with respect to these new parameters while the rest of the model is frozen. It is one of many parameter-efficien fine-tuning methods (Lialin al., 2023) and is related to prefix tuning, where newly initialized parameters are prepended to the attention states instead (Li and Liang, 2021), Prompt compression Wingate et al. (2022) pro- pose to learn a soft prompt a to compress the in- formation contained in context x. Given a pre- trained language model PLM. they draw continu- ations y PLM( xx) based
AutoComprsssors can be initialized with pre- trained LMs to produce powerful and versatile models. We fine-tune AutoCompressors from OPT- 2.7B (Zhang et al., 2022) and Llama-2-7B (Tou- vron et ul., 2023) models on sequences from 6,144 up to 30,720 tokens with single NVIDIA A100 GPU of 80GB memory. We show that summary vectors effective for improving perplexity over long documents and that these compression capa- bilities are robust to domain generalization. Our analysis suggests that AutoCompressors are able to reason over summary vectors, making them useful for diverse set of downstream applications.
We apply AutoCompressors to in-context learn- ing (ICL) by compressing up to 90 in-context demonstrations. We consider 11 classification tasks, including 7 SuperGL.UF tasks (Wang et al., 2019), and we find that summary vectors outper- form few-shot ICL with a comparable number of in-context tokens on out of 11 tasks.
and use dis- tillation objective to align the model's predictions conditioned on the soft prompt PLM(Y () the predictions conditioned on the context PLM æ)). Wingate et al. (2022) find that soft prompts retain high-level information and facilitate controllable generation. However, the approach requires run- ning the optimization for every new context x, with no knowledge transfer between similar contexts. In contrast, our AutoCompressorrs learn to predict their own soft prompts as function of x.
Finally, we explore two applications where AutoCompressors can reduce inference costs by pre-computing summary vectors for large corpora. First, we adopt setting for retrieval-augnen augmmeted language modeling (Shi et al., 2023). We find that for equal sequence lengths, using summary vec- tors achieves 1.5x the perplexity gains compared plain-text passages, and outperforms retrieval- augmented methods for similar computational bud- gets. Secondly, we consider a zero-shot passage re-ranking task (Sachan et al., 2022). We estab- lish that re-ranking passages based on their sum- mary vectors achieves the best trade-off between re-ranking performance and inference throughput.
Context distillation A related line of work (Askell et al., 2021; Snell al., 2022) aims distill in- context information, e.g., instructions, into an un- prompted student model. In concurrent work, Mu et al. (2023) teach models to compress instructions into short key-value attention prefixes. Our ap- proach differs by learning to compress any context information, including long documents, and results in more compact soft prompts.
In summary, main contributions the fol- lowing: (1) We introduce method for extending LMs to long context windows under small-scale computational requirements by learning to generate summary vectors. We propose summary accumula- tion and training with randomized segmenting as
Long-range Transformers A number of archi- tectural modifications have been proposed to scale Transformers to longer context lengths while reduc- ing the high memory costs of full attention. These include restricting and sparsifying the attention window (Dai et al., 2019: 9; Child et al., 2019), ap-

---

proximating the attention (Rae et al., 2020; Zheng et al., 2022; Choromanski et al., 2021), as well as introducing recurrent elements (Ma et al., 2022; Bu- latov et al., 2022), conditional computation (Ainslie et al., 2023), and retrieving previous tokens from the context at the output layer (Zhong et al., 2022). See Tay et al. (2022) for comprehensive survey of efficient long-range architectures.
accumulation, which allows for direct informa- tion pathway between each segment and all seg- ments preceding it: we concatenate the summary vectors ...... Øi-1 to form dci and prepend σzi to Sj- Note that the length of dzi is now (i 1)s, which grows linearly with the document length. Positional embeddings When using a base Trans- former architecture with absolute positional embed- dings, such as the OPT architecture (Zhang et al., 2022), we do not add positional embeddings to the summary tokens <Sum?; nor to the summary vec- tors. This allows us to use all pre-trained position embeddings as context tokens and makes it possi- ble to scale the model to an arbitrary number of compression steps during training. The model still preserves the order of summary tokens due to their separate token embeddings.
Most of these architectures typically require ex- pensive training from scratch, or will deviate sub- stantially from a pre-trained initialization.2 More- over, many language models lack the inductive bias to extrapolate to longer sequences (Press et al., 2022). While AutoCompressors could in principle be trained from scratch, we show that they offer straightforward solution for extending the context window of pre-trained models longer sequences.
If the base Transformer uses relative positional embeddings, such RoPE (Su et al., 2022), we apply the positional embedding to the summary kens and vectors without any further modification.
3 Method
We describe how we adapt a pre-trained language model compress text into summary vectors. An overview of our architecture shown in Figure 1. Summary vectors The AutoCompressor builds on the RMT architecture (Bulatov et al., 2022). We extend the input vocabulary of the base model by K special summary tokens <Sum>; and initialize new input embeddings. When we append the se- quence <Sum>> ... <Sum>, to input, it signals to the model to output special summary vectors of the preceding context. These vectors can then be passed to the next text segment as a soft prompt of length K. Since the embedding spaces of pre- trained language models can span thousands of dimensions, we expect that this mechanism has a high capacity for passing information to subse- quent segments. Furthermore, a soft prompt can interpolate between many token embeddings, and therefore represent more abstract concepts than single discrete token (Wingate et al., 2022).
3.1 Training Summary Vectors
We use simple unsupervised training approach which encourages the model to learn to compress contexts over multiple steps.
Training objective Write "ef....,,/!)) for the segment S, for every S n, where m, is the number of tokens in S,. Conditioning on the concatenated summary vectors dei. we project the Transformer outputs with the language mod- eling head to obtain the next-token probabilities p(a) #t =q:-.-,,74 =|-,,, dc)) We minimize the cross- entropy loss over the entire document:
1 N
L
log p ais....................... 1t Oc))-
where N is the total number of tokens. This ob- jective retains the pre-trained language model's abilities on the first segment S] and incentivizes the model to store useful information in the sum- mary vectors, which future segments can leverage to make better token predictions.
Summary accumulation We split long docu- ments into segments $1....,S, Sn and process them sequentially. Bulatov et al. (2022) incorporate in- formation from previous segments by prepending the compressed summary a;-1 produced from S,-1 to the embedded inputs of S, We propose summary
Unlike Wingate et al. (2022), we do not train with knowledge distillation objective, since the pre-trained LM has a limited context window as teacher, whereas the AutoCompressor student learns to process much longer documents.
"In our pre- pre-liminary experiments, even fine-tuning a pre- trained OPT-2.7b model with Beaafforer--L-sttyye training (Dai al., 2019) caused optimization difficulties and deterior- iated the pre-trained model quality.
Randomized segmenting We randomly vary the lengths m, of the segments S; during training, sub- ject to the condition that each segment fits into
"When fine-tuning OPT models, we observe benefits with initializing the embeddings of the summary tokens with the pre-trained embedding for the ond-of-sequence token 
.

---

We also find that the AutoCompressor benefits from much shorter sequences than seen during training, unlike RMT. See also Figure 2 and Ta- ble 6 for the usefulness of randomized segmenting. the model's context window. This allows Auto- Compressors to compress documents of different lengths and improves performance under evalua- tion with fixed-length segments (see Figure 2).
baselines:
1. We fine-tune an OPT-2.7B baseline on our data. This model is limited to sequences of 2,048 tokens due to pre-training.
BPTT with stop-gradients We employ backprop- agation through time (BPTT) and gradient check- pointing (Chen et al., 2016) for each segment to reduce the size of the computational graph. In addi- tion, we compute and cache summary vectors and stop their gradients after 2 compression steps, simi- lar to caching past attention states in Transformer- XL training (Dai et al., 2019). This assumes that for learning to compress the useful information in S, it is sufficient to predict the tokens in the adja- cent S1+1. In Figure 2, we confirm that this incurs no penalty when predicting long segments, while further reducing GPU memory requirements.
2. Extended full attention: We fine-tune OPT-2.7B sequences of up to 4,096 tokens by extending the model's positional embeddings. We initial- ize the embeddings for positions [2049..4496]
with the embeddings for positions [1..2048]. We are not able to extend the context beyond 4,096 tokens due to GPU memory limitations.
3. RMT-2.7B:: We fine-tune an RMT model on our data with k = 50 summary vectors.
We evaluate on documents of 8,192 tokens, drawn from the training domains or 4 held-out domains. We generate summary vectors for up to 3 segments 2,048 tokens, but also for single segments as short as 128 tokens. For the extended full- attentieno baseline we prepend the previous con- text tokens to the context window.
4 Language Modeling Evaluation
In this section, we train AutoCompressors and eval- uate their long-range language modeling capabil- ities by sampling long sequences which we split into segments of 2,048 tokens. We fix the final segment and compress the previous segments. We track the perplexity of the final segment when conditioning the summary vectors for each We conduct our main experiments and ablations with OPT models (Zhang et al., 2022) of 1.3B or 2.7B parameters, fine-tuned on 2B tokens from the Pile (Gao et al., 2020). In Section 4.1, we evaluate an AutoCompressor on sequences of 8,000 tokens and compare to an equivalent RMT model and an Extended Full Attention baseline. In Section 4.2, we fine-tune an AutoCompressor sequences of 30,000 tokens to demonstrate the feasibility on very long sequences. Finally, in Section 4.3, we scale up AutoCompressors by fine-tuning a Llama-2-7B model on 15B tokens from RedPajama (Togeth- erAl, 2023). Full model hyperparameters and data information can be found in Appendix A.
Results We show the results in Table 1. We find that the AutoCompressor benefits from long con- texts of 6, 144 tokens and consistently outperforms the RMT model.
While extended full attention performs the best on 4,09 4,096-long sequences, we observe a trade-off for shorter contexts where AutoCompressors achieve the best performance. We also stress that the AutoCompressor attends to at most 150 additional soft prompts during evaluation, whereas the full at- tention model is given additional 2,048 tokens. These trends hold for both in-domain and out- of-domain evaluation. However, the gap between the AutoCompressor and the full-artention baseline increases in the out-of-doman setting, suggesting that the summary vectors generalize slightly less than pre-trained attention heads.
4.1 Experiments on 8K-Token Sequences
4.2 Experiments on 30K-Token Sequences Setting We fine-tune OPT-1.3B and OPT-2.7B AutoCompressors on 2B tokens but train on se- quences of 30,720 tokens with 20 compression steps. 4 We use 50 summary tokens, randomized segmenting, and stop-gradients as before. We also
Setting We initialize all models with the 2.7B- parameter OPT model and fine-tune on 2B tokens from 4 domains form the Pile (Gao et al., 2020). Our AutoCompressor
50 summary to-
kens and fine-tuned with summary accumulation over four segments, each ranging from 1,024 to 2,048 tokens. Compressing 2,048 tokens into 50 summary vectors achieves compression rate of 40 tokens per summary vector. We use the following
*Due to the scarcity of very long sequences in the Pile, we only train on data from the Booksä domain, and use the Gutenberg domain as out-of-domain evaluation.

---

In-domain
Out-of-domain
Segments
2
Context tokens
128
512
2048
4096
6144
128
512
2048
4096
6144
Extended FAt
6,331ame
6.15t yars
5.94 psee
8,571 nerw
8.28 soo
7.93" LxBe
RMT
6.42
6.19
6.02
6.02
6.01
8,76
8.44
8.21 u3ar
8.20
8.20
AutoCompressor
6.14
6.04
5.98
5,94 12:0
5.93
8.39 L120
8.26
8.17
8.12
Table I: Held-out perplexity on 2,048 tokens, while varying the length of the preceding context (all the experiments are based on OPT-22.7B models). For RMT and AutoCompressor, we condition on summary vectors. We also report the perplexity gains compared to the fine-tuned OPT baseline without extra context, which achieves 6.28 in-domain and 8.53 out-of-domain (gains shown in colored numbers). Although the extended full attention (Extended FA) achieves similar or slightly better perplexity. it uses up to 2,048 additional tokens and cannot extend further. However, the AutoCompressor uses only 50 3 150 summary vectors to process 6,144 context tokens.
Segments
0
7-
14
CUDA memory
tion effectively captures long-range dependencies in documents. We also report the CUDA memory requirements for fine-tuming each model in Table 2. We train with one NVIDIA A100 GPU with 80GB of memory. Stopping gradients reduces CUDA memory and makes possible fine-tune an Au- toCompressor from OPT-2.7B, while fine-tuning with RMT leads to mut-ooy at that scale.
Context tokens
14336
28672
RMT-1.3B
13.18
12.50
12.50
54GB
AutoCompressor-LB35
13.21
12.49
12,47
38GB
RMT-2.7B AutoCompressor- 2.77B 11.86
00M
11.21
11.18
75GB
Table 2: Evaluation results for AutoCompressors trained on sequences of 30,720 tokens and evaluated on Books3 (in-domain) and Gutenberg (out-of-domain). We train with a single NVIDIA A100 GPU and report the CUDA memory required for fine-tuning using single sequence per batch. AutoCompresson require less memory be- cause we stop gradients after two segments.
4.3 Scaling Up AutoCompressors to Llama-2
Setting We fine-tune a 7B-parameter Llama-2 model an AutoCompressor on single GPU by freezing the model and optimizing only the sum- mary token embeddings and the attention weights via LoRA (Hu et al., 2022). The model is trained on 15B tokens from RedPajama (TogetherAI, 2023), split into sequences of 6, 144 tokens, and we use 50 summary tokens, randomized segmenting, and stop- gradiertsssss. We also fine-tune an Extended Full Attention baseline the same dataset. The context window of the pre-trained model is extended by in- creasing the 0 value in RoPE following (Rozière et al., 2023).
Segments
Context tokens
128
512
2048
4096
6144
Llamu-2
5.52
5,30
5.15
4.98
Extended FA
5.40
5.19
5.06
4.88
4.80
4.76
AutoCompressor 5.40
5.23
5.16
5.11
5.08
5.07
Table 3: Evaluation results for our AutoCompressor trained from Llama-2 2 7B on sequences of 6,144 tokens. For the AutoCompressor, we condition on summary vectors. For Llama-2 and the Extended Full Attention (Extended FA), we condition on plain text tokens.
We compare both models to the pre-trained Llama-2-7B model, which has a context window of 4,096 tokens. All models are evaluated on the final 2,048 tokens of 8,12t-token documents.
fine-tune an RMT model from OPT-1.3B, to use baseline. We are not able to fine-tune a 2.7B- parameter RMT baseline because the RMT method leads to off error.
Results We collect our results in Table 3. The AutoCompressor benefits from the entire context to reduce perplexity: compressing a 4,096-token context into 100 summary vectors achieves simi- lar perplexity to the Extended Full Attention base- line with 512 plain text tokens, and compressing 6, 1444tttken context into 150 summary vectors further improves perplexity slightly. Morcover, we find that summary vectors preserve perplexity when short contexts are compressed.
All models are evaluated on the final 2,048 held- out tokens of documents of size 30,720 tokens by compressing all previous 2,048-token segments.
Results We collect results in Table 2. The evaluation shows that both AutoCompressor mod- els learn utilize the entire 28K tokens reduce perplexity, while the RMT baseline does not benefit from doubling the number of context tokens from 14K to 28K. This shows that summary accumula-
However, Llama-2 and the Extended Full At-

---

improves perplexity over all 2048 token positions. We observe that the extended full attention baseline outperforms the AutoCompressor at the start of the sequence, whereas the AutoCompressor achieves the best performance towards the end of the se- quence. This shows that summary vectors effec- tively capture long-range textual dependencies.
Auto ColoCompeeooo
7.6
AutoCompresoo we summary accunulation
AutoC Cotoesrere w/o randmizeed segmenting
AutoCompessor we stop-gradeens
RMT
7.21
7.0-
In Appendix D, we show examples of sentences and tokens which benefit the most from summary vectors. We find that summary vectors contain salient information, such as names or dates, and that the model can reason over summary vectors. This confirms that summary vectors are useful sum- maries of the compressed text.
1000
2000
30.0
3,00
3000
6000
Number of context tokens
Figure 2: Perplexity on 2048 held-out tokens given dif- ferent numbers of compressed tokens. Compression performed up to 3 segments of 2048 tokens. Abla- tions show that the different components of our fine- uniig strategy help boost performance and that stop- gradients do not impact performance.
5 Compressing Demonstrations for In-Context Learning
In this section, we study the usefulness of summary vectors for performing downstream tasks. We show that in-context demonstrations can reliably be com- pressed down into summary vectors to improve performance while also increasing efficiency on a diverse set of NLP benchmarks.
tention baseline outperform the AutoCompressor when longer contexts are provided. Further re- search is needed to construct summary vectors that preserve all of the context information.
4.4 Analysis
Evaluation We evaluate the in-context learning abilities of the AutoCompressor based on Llama- 2-7B from Section 4.3 on eleven classification and multiple-choice question-anse answering datasets. For each dataset, we evaluate the effect of compressing 1, 3 segments of demonstrations into 50, 100 or 150 summary vectors. For each segment, we include as many demonstrations as possible until we reach 750 tokens. For SST-2, this corresponds to 30 demonstrations per segment on average. We compare this compression approach with the results obtained by prompting the model using 150 and 750 tokens' worth of plain-text demonstrations.
Ablations We train OPT-2.7B models without ran- domized segmenting, summary accumulation, or stop gradients. The results are shown Figure We find that randomized segmenting leads to better compression of short segments, but still improves perplexity when compressing multiple 2048 token segments. As expected, summary accumulation helps improve perplexity beyond one compressed segment. We also confirm that stopping gradients does not impact performance despite reducing GPU memory requirements. In Table 2, we also show that stopping gradients helps reduce GPU memory.
We also train AutoCompressors with x = 20, 50, 70 or 100 summary tokens and report the held- out perplexity results in Table 7 in the Appendix. Surprisingly, find that performance does not increase with longer soft prompts. and x 50 performs the best overall. We hypothesize that learning a larger number of summary vectors may require a larger training budget.
We use contextual calibration (Zhao et al., 2021) and class-balanced sampling when these techniques improve performance on validation set. For each dataset, we report the mean accuracy and standard deviation over 7 random seeds. The detailed set- tings for each dataset can be found in Table 1 11. In Table 12 in the Appendix, we also compare the ICL performance of our OPT-2.7B based Au- toCompressor models against the RMT baseline and pre-trained OPT-2.7B, and include the per- formance of the pre-trained Llama-2-77 model. Results We show evaluation results in Table 4. Results show that summary vectors consistently improve performance over the zero-shot baseline. Furthermore, summary vectors increase accuracy
Token-level analysis We seek to better understand how summary vectors benefit individual token pre- dictions. In Figure 5 in the Appendix, we show perplexity gains each token position for the Au- toCompressor with summary vectors and for the extended full-attention baseline.
We find that conditioning on summary vectors

---

AG News
SST-2
BoolQ
WIC
WSC
RTE
CB
COPA
MuhtRC
MR
Subj
Zero-shot
63.3
67.7
50.8
43.3
68.8
42.9/a.0) 0)
62.5
52.5go.o)
57 57.4
48.3
50 summury vees.
79. 79.6(1.)
()
51.6
47.7
66.3 66.3
46.4
84.5(1
52.0g1.)
91.5p
53.5
100 summary vees.
27.6
92.6
62.5
(2.9/2.)
63.5 63.5)a
64.5
88.9 83.9
(6.1
(00
67.0(p4)
150 summary vees,
85.4
92.3/23)
68.0p18)
$2.8
49.9/24)
65.3ku
54.8/8.8
86.1
54.8
91.1
56.6179)
(C5( tokens)
74.5
92. 92.4
67. 67.4
m2.4
(188
46. 46.4 (21)
80.0
52.5
79.7
57.9/107)
ICI. tokens)
81.2(a1)
93.8
67m7
52.4
40.0g6.3)
(s.)
50.3g..)
82.6 62.6(1.))
47.0gs.2)
91.6(us)
60.7
Table 4: Evaluation of the ICI performance of the Llama-2 7B model. Rach summary is 50 tokens-long and corresponds to segment of 750 tokens' worth of demonstrations. We also report accuracies when prompting the AutoCompressor with 150 and 750 tokens' worth of plaintext demonstrations as baselines. Note that for BoolQ and MultiRC, demonstrations too long to fit into 150 tokens.
compared to 150 tokens worth of plain demonstra- tions on 8/11 tasks. On 8 tasks (AG News, SST-2, BoolQ, WiC, WSC, CB, COPA and MultiRC), sum- mary vectors also out- out-perform ICL with 750 tokens' worth of plain text demonstrations, Summary vec- tors emerge as strong alternative to plain text demonstrations, as they increase accuracy while reducing inference cost.
Pre-process Corpus
Retricval-Augmented LM
Generute...
AutoCempressor
Au AutoCompreaso
pre-compale summary
Petch
top-k
Off-Ibe-shelf Retriever
In Table 12 (Appendix E), we find that the OPT- 2.7B AutoCompressor achieves higher accuracies than the RMT bascline 8 out of 11 tasks and that the RMT model does not benefit from multi- ple compression steps. This shows that summary accumulation an effective mechanism for com- pressing in-context demonstrations. We also ob- serve that our tuned Llama-2 AutoCompressor has substantially worse zero-shot accuracy on some tasks compared to the Llama-2 initialization, and slightly worse ICL performance. We suspect that this is due to domain mismatch in fine-tuning data and the Llama-2 pre-training corpus.
Prompt
Figure 3: Efficient retrieval- augmented language mod- cling with AutoCompressors. Large corpora can be pre-pre pre-processed into compressed summary vectors which can be stored cheaply. Upon retrieval, compressed sum- maries are fused for efficient access to multiple docu- ments in single forward pass.
REPLUG Our case study focuses on REPLUG (Shi et al., 2023), which is a simple method for com- bining a pre-trained language model with an off- the-shelf retriever to improve language modeling performance. Given access to an external corpus C, REPLUG retrieves k passages D {di,..., , based on a segment x to score the next segment y. The overall probability for y is computed by ensem- bling the predictions based on different passages:
6 Compressing Retrieval Corpora for Efficient Inference
We study the usefulness of pre-computing sum- mary vectors for large collections of documents. These can be stored and later retrieved for efficient inference. Since inference is typically more expen- sive than storage, this approach has the potential to achieve good practical trade-offs.
Ply xr D)
Σ X(d p(y CONCAT(d, a)), dED
where X(d, x) are the normalized similarity scores from the retriever and CoNCAT(d,aa x) denotes con- catenation of p and r. This method incurs a sub- stantial overhead, since it requires k forward passes over sequences CONCAT(d, xx, v,y)
6.1 Retrieval-augmentee Language Modeling
Aatrivval-augmenter language models improve to- ken predictions by retrieving information from data store. A number of approaches have been pro- posed infuse external knowledge in the input layer (Guu et al., 2020; Shi al., 2023), intermedi- ate layers (Borgeaud et al., 2022) or at the output layer (Khandelwal et al., 2020; Zhong et al., 2022).
Fused Summaries We introduce a setting for retrieval-augmennte language modeling close to fusion-in-decoder (Izacard and Grave, 2021). We concatenate the summary vectors of retrieved pas- sages D to form the fused summary vectors, dp CONCAT( o.,) ) where dg...., dj are or- dered from least-to-most most relevant. This resembles

---

Perplexity Gain (%)
Throughput (examples/s)
Passages
top-1
top-2
top-5
top-10
top-1
top-2
top-5
top-10
50 tokens
REPLUG
-0.64
0.58
1.68
2.35
51
38
16
9
50 tokens
Fused Passages
0.71
1.01
1.70
2.60
28
27
23
17
512 tokens - 50 sum. vecs.
Fused Summaries
1.04
1.67
2.63
3.74
28
27
23
17
512 tokens
REPLUG
-1.47
2.24
5.25
8.30
18
10
6
3
Table PPL gains (%) from different retrieval-augmentt language modeling settings, over the no-retrieval baseline. We evaluate the OPT-2.7B AutoCompressor and we report throughput on a single NVIDIA A100 GPU for each method without batching examples. Fused Summaries outperforms Fused Passages and REPLUG with 50-token passages. Moreover, Fused Summaries top- outperforms REPLUG top-2 with 512-token passages while also gaining 1.7x throughput increase.
summary accumulation as described in Section 3. We also find it useful to smooth probability scores and re-order the retrieved passages based on their summary vectors (Appendix F). Figure 3 gives an overview our approach.
Moreover, Fused Summaries outperforms RE- PLUG top-2 with 512-token passages and sees a 1.7x throughput increase, which shows that the model benefits from the diversity of compressed documents. However, REPLUG top- 10 outper- forms Fused Summaries. We leave it as future work to explore how to produce higher quality summary vectors to better utilize the compressed passages.
Fused Passages We establish a baseline for fus- ing summary vectors by concatenating the plain- text passages and computing smoothed probabili- ties, see Appendix F. Unlike summary vectors, this method is limited by the model's 's context window. Experiments We evaluate the OPT-2.7B Auto- Compressor introduced in Section 4.1 without any additional fine-tuning. Similar Shi et al. (2023), we retrieve from the Pile. We use Books3, FreeLaw, GitHub, Wikipedia, Gutenberg, ArXiv, HackerNews, and YoutubeSubritles. We index 10B tokens for each domain, which are split into pas- sages of 512 or 50 tokens.
We note that fusing summary vectors effec- tive despite mismatch in training since we draw independent summary vectors from separate docu- ments. Furthermore, our AutoCompressor model is only ever trained to accumulate 3 sets of sum- mary vectors, and yet it benefits from fusing the summary vectors of up to 10 documents.
6.2 Unsupervised Passage Re-ranking
Finally, we consider the case study of passage re- ranking, in which a fast off-the-shelf retriever like M25 retrieves a large set of candidate passages, and a more capable re-ranker refines the ranking to increase the rank of the most relevant passages.
We sample segments of 256 tokens from the Pile validation data, using the first 128 tokens context æ for retrieval and the last 128 tokens y for evalua- tion. We use the Contriever model (Izacard al., 2022) for retrieval, and retrieve the top 10 passages, We also deduplicate our data by removing passages that overlap with & by 64 tokens.
Method Sachan et al. (2022) introduce an effec- tive method for leveraging language models re-rankers with no additional supervision or fine- tuning. Given query q and a set of candidate pas- sages (p pk). the language model scores the likelihood of the query conditioned on the prompt "Passage: (p:). Please write a question based on this passage." " for each passage and re-ranks the passages based on the scores.
Results Results are shown in Table We find that Fused Summaries outperforms Fused Passages and REPLUG when 50-token passages are retrieved. We measure throughput empirically and show that for 10 retrieved documents, Fused Summary Vec- tors remains inexpensive. We note that compress- ing the 1OB token datasets results in disk space of 5TB per domain when stored in pref-icei for- m Therefore Fused Summaries achieves a good trade-off berween storage costs and throughput.
Experiments We consider the task of re-ranking BM25 passages the NQ test set (Balachandran et al., 2021) and compare out-of-the-box AutoCom- pressors with 20 and 50 summary tokens pre- trained OPT models from 125M to 2.7B parameters. We pre-compute summary vectors for 21M pas- sages from a Wikipedia corpus (Karpukhin et al.,
'For comparison, storing the transformer output at every single token (c.g., in an enender-decoder setting) would take 51 TB, and storing all attention states would be 3,276 TB.

---

can be used in retrieval settings. Summary vec- tors can also be pre-computed, cached and re-used. This offers practical efficiency gains by reducing the size of the attention window. Significant future work remains in scaling AutoCompressors to big- ger models and improving the quality of summary vectors to further close the gap with full attention over long-range contexts.
1000
Pareto frunt
75
1000
AuloCoopressorr..0
250
AutoCompressor,..
1000)
OPT-2,78
250
OPT-1,38
Recall 20
72
150
OPT-350M
100
OPT-125M
71
100
50
100
250
I00.
50
100
77
Limitations
69
50
100
60
1. We only apply AutoCompressors to OPT mod- els of up to 2.7B parameters and a Llama model of 7B parameters. Future work needs to estab- lish how AutoCompressors perform for even larger models. As the summary vector dimen- sion grows, there is promise for retaining more information per vector.
50
50
Re-ranking throughput (queriesds)
Figure We compare AutoCompressors (squares) in an unsupervised passage re-ranking setting to pre-trained language models (circles). The number on each data point shows how many passages retrieved by BM25 are re-ranked, and the vertical axis shows the Recall @ 20 performance of the re-ranking system on the NQ test set. We consider the throughput A100 GPU and assume that multiple queries cannot be batched. By leveraging pre-computed summary vectors for passages, AutoCompressors lead to re-ranking solu- tions that lie on the Pareto front of recall vs. compute.
2. Our results suggest that summary vectors ig- nore some useful information that is accessible via full attention. Additionally, models do not always benefit from increasing the number of summary vectors. We suspect that the training signal for learning summary vectors efficiently might be limited by pre-trained models being very good at making predictions from the plain- text tokens in the current segment. Future work is needed to improve this optimization.
single NVIDIA
2020), which requires 2.ITB and 5.4TB disk space in half precision for 20 and 50 summary vectors re- spectively. We measure the quality of the re-ranked results using Recall@20.
3. Summary accumulation still leads to quadratic complexity with increasing number of segments, albeit at much lower rate than full attention. Future work may explore ways combine many summary vectors more efficiently.
Results The results are shown in Figure 4. We measure throughput for individual un-batched queries on single NVIDIA A100 80GB GPU and assume that the latency of loading summary vectors is negligible. Although the passages only 100 words long, resulting in low compres- sion rates, summary vectors substantially speed up the inference, while sacrificing on performance less than smaller models. This leads to a Parcto-optimal trade-off between compute and performance and demonstrates that summary vectors often retain sufficient information from a passage to assess its relevance for particular query.
Acknowledgments
We thank Mengzhou Xia, Howard Chen, Vishvak Murahari, Aatmik Gupta, Zirui Wang, Jiatong Yu, and the members of the Princeton NLP group for helpful discussion and valuable feedback. This re- search supported by an NSF CAREER award (IIS 2239290), a Sloan Research Fellowship, and a Data Science Research Award from Adobe. AC also gratefully acknowledges support from the Min- erva Research Foundation.
7 Conclusion
References
We have introduced training strategy for adapting pre-trained LMs into AutoCompressors, which re- cursively compress contexts into summary vectors. Our experiments indicate that summary vectors re- tain important contextual information, that they can encode in-context demonstrations, and that they
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago On- tañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun- Hsuan Sung, and Sumit Sanghai. 2023. CoLT5: Faster long-range transformers with conditional com- putation.

---

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as laboratory for align- ment. arXiv preprint arXiv:21 2tdi::212..0000 12.00661.
Toutanova, 2019. BoolQ: Exploring the surprising difficulty of natural yes/uo questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume i (Long and Short Papers), pages 2924-2933 Minneapolis, Min- nesota. Association for Computational Linguistics.
Vidhisha Balachandran, Bhuwan Dhingra, Haitian Sun, Michael Collins, and William Cohen. 2021. Inves- tigating the effect of background knowledge on nat- ural questions. In Proceedings of Deep Learning Inside Out (DeeLIO); The 2nd Workshop Knowl- edge Extraction and Integration for Deep Learning Architectures, pages 25-30, Online. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predic- rive Uncertainty Visual Object Classification, and Recognizing Textual Entailment.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer- XL:: Attentive language models beyond fxxe-lengt context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978- 2988, Florence, Italy, Asso- ciation for Computational Linguistics.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Sean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206 2240. PMLR.
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. FlashAutentio: Fast and memory-efficient exact attention with 1O-awareness. In Advances in Neural Information Processing Sys- tems.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learner. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Marie-Cathering de Marneffe, Mandy Simons, and Ju- dith Tonbauser. 2019. The CommitmentBank Inves- tigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 23(2):107-124.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, al. 2020. The Pile: An 800GB dataset of diverse text for lan- guage modeling, arXiv preprint arXiv:2101.0000
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9929-3938. PMLR.
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. In Advances in Neu- ral Information Processing Systems.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06/74
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo. Bernardo Magnini, and Idan Szpektor, 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7, pages 785-794.
Rewon Child, Scott Gray, Alec Radford, and Generating long se-
Πlya Sutskever. 2019. quences with sparse transformers. arXiv preprint 1904 10509.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation large language models. In International Conference on Learning Representations.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Be- langer, Lucy J Colwell, and Adrian Weller. 2021. Rethinking attention with Performers. In Interna- tional Conference on Learning Representations.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Christopher Clark, Kenton Lee, Ming- Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina

---

Gautier Izacard and Edouard Grave, 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguisties.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023, Scaling down to scale up: A guide to parameter-effiieet fine-tuning. arXiv preprint Skiv:2303./5641
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning actoss scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 61 61-68, -6, Dublin, Ireland. Association for Computational Lin- guistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2022. Mega: moving av- erage equipped gated attention. arXiv preprint 17Xi:2209.1065:3
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. arXiv preprial urXiv:2304.88667
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay. and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pu- pers), pages 252-262, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity. In Proceedings of ACL. pages 271-278.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACI. pages 115-124.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA.
Mohammad Taher Pilehvar and Jose Camacho-Collados 2019. WiC: the word-in-context dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Shori Papers), pages 1267-1273, Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficicet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Confer- ence on Learning Representations.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In 13th International Conference on the Principles of Knowl- edge Representation and Reasoning, KR 2012. Pro- ceedings of the International Conference Knowl- edge Representation and Reasoning, pages 552-561. Institute of Electrical and Electronics Engineers Inc. 13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012 Conference date: 20-06-2012 Through 14-06-2012.
Jack W. Rac, Anna Potapenko, Siddhant M. Jayaku- mar. Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-rande range sequence modelling. In International Conference on Learning Representations.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning, In AAAI Spring Symposium: Logical Formal- izations of Commonsense Reasoning.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing \Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan. Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wen- han Xiong, Alexandre Défossez, Jade Copet, Faisal

---

Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier. Thomas Scialom, and Gabriel Synnaeve. 2023. Code Ilama: Open foundation models for code.
David Wingate, Mohammad Shoeybi, and Taylor Sorensen, 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computationnl Linguisties: EMNLP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
Devendra Sachan, Mike Lewis. Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 1739-3, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022, OPT: Open pre-trained transformer language models. arXiv preprint urXiv2205.01066.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke 7ettle- moyer, and Wen-tau tau Yih. 2023. REPLUG: Retrieval- augmented boxck-box language models. arXiv preprint 2ttt:2301.12622..
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28, Curran Associates, Inc.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Muchine Learning Research, pages 12697- PMIRR
Charlie Snell, Dan Klein, and Ruiqi Zhong. 2022. Learning by distilling context. arXiv preprint arXiv:2209115189
language models.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1-162, Seattle, Washington, USA. Association for Computational Linguistics.
Lin Zheng, Chong Wang, and Lingpeng Kong, 2022. Linear complexity randomized self-attention mech- anism. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2701 1-2044.. PMLR.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: En- hanced transformer with rotary position embedding.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021, Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Comput. Surv. 55(6).
TogetherAl. 2023. RedPajama: An open source recipe to reproduce Ilama training dataset.
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 5657-5673, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and ef- ficient foundation language models. arXiv preprint arXiv:2302.13971.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for gemeral-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.

---

2. The AutoCompressor models are fine-tuned from OPT-1.3B and 2.7B on documents of up to 30,720 tokens. Each document is split into A Models and Data
fine-tuned on documents of up to 30,720 tokens. We use a learning rate of 2c-5, a batch size of 130k tokens, 1,000 warm-up steps and the Adam optimizer.
All models are fine-ture tne-tuned from OPT models on the Pile. We conduct our experiments using a sin- gle NVIDIA A100 80GB GPU and we use Flash Attention (Dao et al., 2022) as an efficient imple- mentation of exact attention over long sequences. We also gradient checkpointing between com- pressed segments reduce GPU memory.
1. RMT-1.33 uses K. = 50 summary vectors and is fine-tuned without summary accumulation, ran- domized segmenting, stop-get hoop-gradients. Each document is split into 15 segments of 2,048 to- kens Even with gradient checkpointing, attempt- ing to fine-tune a 2.7B parameter RMT model on this dataset leads to an out-of-memory error.
A.1 OPT Experiments 8K Tokens
We fine-tune our models on 2B tokens from the Pile. We sample 500M tokens from the following Pile subdomains: Books3, FreeLaw, GitHub and Wikipedia.
20 segments such that segment 2i and segment 2 +++1 add up to 3,072 tokens. The length of each segment is randomly sampled between 1,024 and 2,048. We use K 50 summary vectors with summary accumulation and we stop gradi- ents every two compression steps.
The following models use learning rate of 2e-5, a batch size of 130K tokens, 1,00 warm-up steps, and the Adam optimizer (Kingma and Ba, 2015): The fine-tuned OPT-2.7B baseline is fine-tuned on documents of up 2,048 tokens.
The extended full-attention baseline is fine- tuned on documents of up to 4,096 tokens by ex- tending the positional embeddings of OPT-2.7B to 4,096 positions. We initialize the embeddings for positions [2049..1990 1096] with the embeddings for positions [1..2048].
All models are evaluated on documents of 30,720 tokens from the Pile. We 1,000 doc- uments from Books3 (in-domain) and 1,000 docu- ments from Gutenberg -fu-ai--domain.
3. The RMT baseline is fine-tuned documents of up to 8,192 tokens. Each document is seg- mented into four segments of 2,048 tokens. We use x = 50 summary vectors but we do not use summary accumulation, randomized segment- ing, or stop-gradients.
A.3 Llama-2 Experiments on 8K Tokens
We fine-tune our Llama-2 models on 15B tokens from RedPajama. We sample 1B tokens from long documents in ArXiv, Books, C4, GitHub, as well as 10B tokens from CommonCrawl, 800M from Wikipedia and 70M tokens from StackExchange. Both our AutoCompressor and Extended Full Attention baseline are fine-tuned from Llama- 2-7B on sequences of 6, 144 tokens with LoRA (Hu et al., 2022) parameter efficient fine-tuning applied to the attention heads. We use a LoRA dimension of 16 applied to the QKV- and Out-projections. We use a learning rate of 4c-4, batch size of 200K tokens, 5,000 warm-up steps and the Adam opti- mizer. For the AutoCompressor, we also optimize the newly initialized summary token embeddings. We train our AutoCompressor the same way as the OPT-2.7B AutoCompressor, with x = 50, randomly segmenting each sequence into four sem- gents, and stopping gradients every two compres- sion steps. The Extended Full Attention baseline is fine-tuned with RoPE 0 value of 80,000.
4. Our AutoCompressor is fine-tuned on docu- ments of up to 6,144 tokens. Each document is randomly segmented into four segments such that the first two segments add up 3,072 to- kens. The length of each segments ranges from 1,024 to 2,048 tokens. We use & = 50 summary vectors and summary accumulation. We stop gradients every two compression steps.
All models are evaluated on documents sam- pled from the Pile with a fixed length of 8, 8,192 tokens. We sample 610 documents from each of the following domains: Books3, FreeLaww GitHub, Wikipedia (in-domain), and ArXiv, Gutenberg, HackerNews, YoutubeSubritles (out-of-domain), Examples of documents from each of those do- mains can be found in Tables and 10.
We evaluate our models on 500 sequences of 8.192 tokens from each of ArXiv, Books, C4, GitHub, StackExchange, and 5,000 sequences from CommonCrawl.
A.2 OPT Experiments 30K Tokens
We fine-tune our models on 2 billion tokens from the Books3 subdomain of the Pile. All models are

---

B No-context Language Modeling
Results show that the summary vectors help duce perplexity over the entire 2,048-token seg- ment. This shows that summary vectors do not only contain information which helps continue the previous sequence.
Table 6, we verify that our fine-tuning strategy does not significantly affect the language modeling capabilities of the OPT AutoCompressors when no summary tokens are given. We find that the Auto- Compressor performs slightly better than the RMT model and significantly better than the extended full attention model when no additional context is given. Moreover, the AutoCompressor almost matches the OPT02.7B fine-tuned baseline, with perplexity increasing by less than 1 1%.
Figure 5 also shows that the extended full- attention baseline benefits more from the additional 2,048 context tokens than the AutoCompressor at the start of the sequence, but that the AutoCom- pressor achieves stronger gains the end of the sequence. This shows that summary vectors ef- fectively capture long-range textual dependencies and that fine-tuning AutoCompressors produces more robust models than fine-tuning extended full- attention models.
In-domain Out-of-domain
OPT-2.7B
7.53
9.19
OPT-2.7B fine-tuned
6.28
8.53
AutoCompressor-2.78
6.31 tuse
8.60
RMT-2.7B
6,34
8.62 n1R
70-
Extended full attn. (2048 tokens)
AutoCompressor (2048 tokens)
Extended full attention
6.57 164e
8.94
AutoCompressor (no context)
50
Table 6: Held-out perplexity of all models 2048
tokens without summary vectors or additional context.
PPL. gain
30-
20-
C AutoCompressor Ablations
10-
We train OPT AutoCompressor models as in Sec- tion 4.1 while varying к = 20, 50, 70, 100. In Table 7, we report the perplexity evaluation on documents 8192 tokens across all evaluation domains.
0
400
800
1200
1600
2000
Token position
Figure 5: We plot the perplexity gain over OPT-2.7B for our AutoCompressor model and the 4096-extended attention bascline. We track the perplexity each ken position in sequences of 2048 tokens. The Auto- Compressor model almost matches the strong extended- attention baseline at the of sequences and outper- forms it at the end of sequences.
Compressed tokens
K
0
2048
4096
6144
20
7.36
7.05
7.01
7.00
50
7.37
6.99
6.94
6.93
7.41
7.01
6.97
6.95
In Tables 9 and 10, we give hand-picked exam- ples of sequences from each evaluation domain, highlighting which tokens benefit the most from the compressed context. We compress the first 300 tokens in every document from the evaluation set and evaluate on the following 100 tokens. In the notation of Section 3.1, measure the perplexity gain of each token
100
7.48
7.07
7.01
7.00
Table 7: Held-out perplexity across all evaluation do- mains AutoCompressors based on OPT-2.7B trained with different numbers of summary tokens We ob- serve that 50 performs the best overall.
Token-level AutoCompressor Analysis
p(a 2 p I x²_1)
1,01)
In Figure 5, we plot the perplexity gains achieved by the OPT AutoCompressor and the extended full attention baseline from Section 4.1 over the pre-trained OPT-2.7B model. We plot the gains achieved by the AutoCompressor both without any additional context and with the summary vectors obtained from 2048 compressed tokens.
For each example, we record the top 3-5 most im- proved token predictions.
We find that the tokens which benefit the most from the summary vectors are often interpretable. Names of characters, dates, and locations are often

---

copied through the summary vectors (see the ex- amples for Wikipedia, FreeLaw, HackerNews). We also find that the model is able to reason over the summary vectors, as the tokens which benefit the most are sometimes not explicitly present in the compressed context, but are closely associated with the domain of speech (see the examples for Books3, Gutenberg and YoutubeSubritles)) Finally, we find that summary vectors are often useful for continuing the previous sentence (see the GitHub example.)
We provide details and ablations for our pro- posed REPLUG alternative. Inspired by fusion-in- decoder (Izacard and Grave, 2021), we fuse sum- mary vectors or passages in a single forward pass.
Fused Summary Vectors The summary vectors of retrieved passages 'D are concatenated in order of increasing retrieval scores to form fused sum- mary vectors, = Concat(o ..... ddg]- This resembles summary accumulation described in Section 3, but differs in that the retrieved summary vectors were produced independently rather than recursively. Nevertheless, we find that AutoCom- pressors transfer well to this setting.
E In-Context Learning Details
Furthermore, we find it beneficial to smooth the conditioned probabilities with the unconditioned probabilities p(y x),, and compute
We evaluate in-context examples of the follow- ing datasets: AG News (topic classification, Zhang al. (2015)), SST-2 (sentiment analysis, Socher et al. (2013)), BoolQ (Boolean Questions, Clark et al. (2019)), WIC (Word-in-Contcxt, word sense dismabiguation, Pilehvar and Camacho-Collados (2019)), WSC (Winograd Schema Challenge, coref- erence resolution, Levesque et al. (2012)), RTE (Recognizing Textual Entailment, Dagan et al. (2005); Haim et al. (2006); Bentivogli et al. (2009)), CB (Committmen(Bank, de Marneffe al. (2019)). COPA (Choice of Plausible Alternatives, Roem- mele et al. (201 1)), MultiRC (Multi-Senteni) Read- ing Comprehension, Khashabi et al. (2018)), MR (Movie Reviews, Pang and Lee (2005)), Subj (Sub- jectivity, Pang and Lee (2004). We follow the GPT- 3 prompt templates (Brown et al., 2020) and detail evaluation setting for OPT and Llama-2 in Ta- ble 11.
p(y x. D) p(y ConcatloD, æ)) + p(y (
2
We also show that language language-modeling perfor- mance improves when D is re-ordered based the smallest l2 distance between the summary vectors {di)) o (dk)) and oz. This incurs negligible overhead since ds can be constructed during the same forward pass which computes p(v 2)) The ablation for this is shown in Table 8
Fused Passages We establish a baseline for Fusing Summary Vectors by concatenating the correspond- ing plain-text passages D Concat(da. .... d1] and computing
p(y xr, D) p(y Concat D, x)))) + +(y x)
In Table 12, we compile evaluation results for OPT-2.7B, Llama-2-7B, as well as our AutoCom- pressor and RMT models.
Note that this approach is quickly limited by the size of the pre-trained language model's context window, especially when retrieving many long pas- sages.
F Fused Retrieval-augmennted Language Modeling
Perplexity Gain (Sr)
Passages
top-1
top-2
top-5
top-10
Fused Summaries
1.04
1.67
Fused Summaries w/o re-ranking
1.52
2.02
Table 8: PPL gains (%) over the no-retrieval baseline for Fused Summary with and without re-ranking. In re-ranking, order the passages based on the &g norms of their summary vectors before concatenating the sum- mary vectors, whereas w/o re-ranking we use the re- trieval scores from the Contriever model. Re-ranking consistently produces higher perplexities.

---

Domsin
Compressed context
Evaluation sequence
Most
Improved
tokens
Books3
Surveliimm-no for Breton's 's depreciation of 'Red Fromt," but for a seemingly insignificant uside. carly March, before sending the text press, Breton showed Aragon. The latter conseuted to the publications, with one excejtion: a feotnote in which Bicton quoted the PCF official's romark (which Aragon had earlier repoeted to him) about "complicating the simple healthy relationss between man and women" -4 claar illustration, Bieton folt, "jost bow much bad faith mental indigence we were against." Aragon considered internal Party statements to be confidential, and asked that the footnote be removed: according to him, Breton "spontannensll crossed out the nose the galleys with a delete mark that I can still recall... saying that wanted to give the Party so exeuse expelling me." But whenn The Powerty of Poetry_ came off press the next day, the incriminating footinne was still there.
L'Humanité "Our comrado Aragon in- forms us that be has absolutely nothing with publication a pamphlet eneitled The Poserty of Poetry... Hi wishos make it clear that he entirely disavows both the con- tents of this pamphiet and the sitention has drawn to his name, every Communist being duty botyd to condemnn the attacks contained this panpblier incomparible struggle." " This short paragraph was the only notice he ever saw to give of
Poveny
Ar
agon
Human
class
Whether Beeton retained the note test of Aragon's logaliy, whether he deemed this example of PCF stupidity loo good to wastc, whether the peinter simply noglected to make the correction, one has established. But the result was that this single act came repeesent for Amaan every philosophical difference, strieture, humiliation that had ever darkened his long friendship with Breton, On March 10, he responded to the trace via an anony mous note
Wikipodia
<kiSShi Co
event durinng the 2005 Summer Deallympies
Ge
Shi Co G bom 15 December 1985) a Chinese deal female tuble tennis player. She has been reganded
and excelled her dehut Deaflympic event
represented China at Deaflympics four from 2005-2017, as one of the finest athletes to have represented China the Deaflympicc, having won 14 medals at the event since making her debut in 2005 Summer Deaflympics. Biograply
winning gold medals the women's sin- 2005
gles, diaalee and in the mixed doubles. She was also the part
Chinese Tablee ten- Summer
nis team which secured the silver medal in the 2005 Deaflympics,
Shi
Co was bon Yichun, Hollongjüang on 15 December 1985. She was barn with an condition that impuired her hearing which resulted in her deafnnsss and has congenitall malformation
same year, she reeeived the Deaf Sportswoninn of the Year award from the ICSD for remarkuble per- formances at the 2005 Summer Deafhympics. Shi Ce
right car. Her parents decided to consult doctor and took to hospital in the Zhejiang Province in order to cure her ear impairment when she was just five years old, doctor suggested that surgery would cause facial paralysi after Shi Ce's porents demianedd for surger. Shi Co took the sport of Table tonnis and started playing it at age nine.
Carcer
won medals her Deaffympic career as Table tennis player including gold medals. Shi Ce was eligible to compere the National Games of China despire her deafnness, in 2015. the compesition, she socured gold medals in singles, doubles, mixed doubles and in the team events.
2005 Summer Deaflympics Shi Ce made her first appearance an international sports
Gitbub

 import sys import datetime
"type": : "issue", "state": : "apen" J. 1: [ 'texc": "This is general note.". "location": Nane,
location
basictarguments::
import api
"type" "isme""
cnitie apl.crttic Strtessstu Jesting-Trre)
lest
repouitery api repoiitry fotch(eritic, namen" "olitic))
issue
brunch- api www.cc....cc..ccc...... ceichi
repoxitory-repode name srre-ummmsss.eeeeew
review www.reeiew.fesccccieeo beuncb-branch)
alice = agi.user.etchicciic, name='alics fetcheereetccciiiiici, name-"bob"
dave - api user fetch(crilic, name-" "dave"
erin- www.wse..fssccssiitiie. name- 'crin")
all_ comments spi.commet.feet agg.commen..fecchlllciinss fechhAlliccii assert isinstance(all comments, list)
EXPECTED ft: "text" "This is general issue.", "location": None.
Freel.aw
Family Provider, and assets were incressed to $250,000, the minimum required to con- duct business Arizona, where the company was licensed. In January 1976. Doden and Kavanagh met with Halford and Robert ton, president of Old Security. to propose a new agreement between Family Provider for the purpose of obtaining the Fund business. Both Harton Halford considered Family Provider and Farmers Na- tional be "synonymous" believed Kavanagh and Boden
Security
By the end of 1975, Farmers Naticnal was insolvent and under investigation by the Flovida Department Insurance. The Miami new sewspupers published series articles describing the relationship between Hasser and the company. Lawrence Lee, an utterney for an Arizona union group. investigated laammes National connection National proposal. He was told by the Florida insurance depariment that Farmers National was connected with Hauser, that had been injected with questionable assess which were being investiguted by the deportment, and that had been fined $5,000 for failing to disclose contacted Richard Halford,
Old
Family
un Old Security- Furmers
avan
ssscts
Security and
both Hauser's is ownership vice president Old Security in charge of union group insurance, and related the information received. Halfood assured Lee he was aware Hauser's reputation. Hauser was no longer involved with Faamnes National. Halford then called Kavanagh, who told that Hauser had no official capacity with the company. and that financial problems had been cleared up. Halford did not anemge to check the occuracy of Kavanagh's representations with the Florida Department of Insurance.
one of disectors.
that
Hauser controlled second company, Fumily Provider Life Insurance Compuny ("Fumily Provider'). ). In 1975, the company bad no business, no office, and ussets of $30,000, Because of Tarmers Nutional's is insolvency. Hauser decided activate
Table 9: Examples of sequences from in-domain test Pile domains. We highlight the tokens from the evaluation sequence which benefit the most from the summary vectors. In Books3, L 'Humanité is prominent French newspaper associated with Breton and his circle. In GitHub, the summary vectors carry information about the logical and syntactical continuation of the context.

---

Domain
Compressed context
Evalsation sequence
Must
improved
tokens
HackerNews
Hackers steer Toda oncoming traffic placing ihrce stickers on veliu htps://www
those are also going imvolve trade-affs where there are legitimate situations that will stop werking.
Tesla
stickerss the road 2019-1
lest
momen chrisboll From yosenday:
think you could just as casily fitle vidar "Tesla suto-pilor follows road markings even if they really
markings
Ihipsc/tteess http:e/wwwoooooonnoono yoombinnarccmmmeembee htw.m/teeemmmmmmmmddmd--- 19536375)
auto
http:///www.cmmiiocccc.ooom com/ 19536375)
Fdit: The best shot I could get from the viden upset
gregmac
test: these look like the temporary markings of- ten used during construction, just before they come paint the normal lincs using big
makes
While I'm hugely skeptical the current state self-driving cars, you could probably get human drivers to make the same mistake you were to repaint the lines. However, humans also notice oncoming cars there any) avoid getting a head on collision.
The thing missing from this test is thut critical practicl piece: if there was an oonommnng will the Tesia do something avoid the collision! I would assume that not getting head-on oo crash is higher priority than staying the lane markings.
Without cnnvmiing tralfic, all this is tosing what the Testa comsiders valid line markings sure there's room for impeovenent the requiremnnt for how well-defined the lines have to be, etc). but
(such as checking where other lune is, raising
ArXiv
24 = ha (2x) + ti. by -V (0, Rs)
a nonlincur fux-t 1 ( and lis (+), we can adept the Estendal Kalmsn Filler to fit into the framework mothods described this far.
Extended
In the above equations, see that the trancition mutrix Fis-i has boen replancd by the nonlincur vector-valued function fux-s (-), similarly. matrix H., which transforms vector from the stoee space int the messurement space, las bem replaced by the nonlinese vector valued function il (1). The method propesed by the Extended Kalman Filter linearize the nonlinaariies about the current stalc prodiction (vr estimute). That is, we choose Ags-1 the Jacobian of fuj- evaluated 4 f--1p-1- and Nt, as the Jacohian of iu evalunted at aup-- and proceed as in the linear Kalman Filter of Section sce kf.fi8 Numerical accuracy these methods tonds depend heavily on nonlincar functions. E we have lineur comstaints bul
lincur
Nunlinear Equility and Inquulity Constraints
Since equaity and incquality constraints we model are often times nonlinear, it important
make the exsension to nonlinear equality incquality constmined Kalman Fil
Gutenberg
eight cents. Telerran in foreign languages are sent within the empire for sen per word, with a minimum charge of -lie son for live words a fixction thereof, No charge made for delivery within rndias 2/2 miles of the selegrahh offic..
first was limited postal the three lurge cities of lokyo, Kyono, and Os. extended to the whole
limited
akı in 1872 country, with the exception a certain pari of the Hokkaido, which was without maads and almost without population. village er lamolt in the whole land which does not enjoy the convenience of's good postal tem. The mails are sent with prompiness and
postal
There sre no privatc tokegmph corporations The govemment buildh, owns, and operstes the lines just as it does the mails. The postal and 101 ielegraph systems are intimatlly connected, same office does service both.
Tokyo
The first telegraph line in ligun was spenel in 1869 The ventine prowing a suaaass, the following the line extended and general tolggrapiic system for the whole country decided upon. The mpidd construction telegraph lines began 1872, from which year it has gone forward uninteruptedly... At present the lines extend to every comer of the empiro. first lines were surveyed buill, and operated under forsign experts, learned so rapidly that they have been enabled to do away with all foceigu emplooyes.. All of the materials and instruments
with the exception of submarine cables and the delictic electrical metoaring apporstos, are made Japan. MAILS--The Japanese mail system was modeled after the American 1871.
YoutubeSuhrilles
que no vay escucla."
trabsjjls en comediaa American Drenmer, su primer gain trabjjo. Su lamilia pasó
Johnny Gislecki
Parsons
El Lecnard Hoostodeer obtuvo doctorado los 24 años. pero que lo interpeeta wilh lllgei madiso camino de la saundariis. Fn unn entrecists can Time Out Chicago 2009, Johnny Galecki reveló que ahandond la escuela a mediades del octavo gradn luego de de
Long Beach antes de
als
regress Chings y 8 se quiski para conom- tuse en carrera come actor. Jim Pansons
Jim
Dr
voz que divisiones laraas spuraciooon en tercer grado, ihu al hais por 45 nimutos nadic lo notalaa, indas los dins a misma hara del dia, sólo para escapar de ellas." Puede que Galecki tonga un cerebrn matemitico, pero siempre lave inteligancia callejera. "El conocimiento es mejor y más seguro resoro... Vaya, me aburr mi mismo." " los 14 añoss vivió solo en agantamnntn en Burbank, Califoenia, mient
a classs
Le dijo Time
Hl Dr. Shelkon Corgeer foe un niña prodigio.
Comenzó la univensidad cuando senta 11 años
Table 10: Examples of sequences from out-of-domain test Pile domains. We highlight the tokens from the evaluation sequence which benefit the most from the summary vectors. In Gutenberg, 'Tokyo' is not copied over from the compressed context but is inferred from the discussion of Japan. YoutubeSubtitles, 'Jim Parsons' benefits the most from the summary vectors because the context discusses his co-star John Galecki in The Big Bang Theory.

---

OPTT-bused models
Lluma 2-bused mudels
Dataset
Prompt template
Toks. / dem.
Cal. Bal.
Toks. / dem.
Cal.
Ral.
AG News
Article: (text)vnTopic; (label)
65
75
SST-2
Sentence: (sentence//nsentieeet (label)
22
v
25
v
BualQ
(pussag/\nnuestion: (ovestion)/nnanseer:: (label)
165
WiC
(sentence1] [eentencl])nnseentennnn [nt[entteeee))vuuustta Is the word '(wardy' used the sare way the two sentences above?\nansser: (labely
45
/
48
V
WSC
Question:
the sentence "(text)". does the prenoun '[spen2_text)"
61
50
refar to aaaan__txxt)hhoosserr (label)
RTE
(prenise)Xnquestion: (hypothesis) True False?\nanswer: (label)
CB
[prenise)Xinuestionn: hypothests. true, falss nes neithrrrhnanse:r: (label)
98
<
95
COPA
Context: (prentse)\nnnsser: (answer)
v
22
MuhiRC
Context: (ppragraan//uaaassttt)))/ Vnaassve:: (label)
350
350
MR
Review:
(cextt)inSemtinent: (label)
36 4 40
v
Subj
Input:
[caxt)\ntyp:: (label)
/
Table 11: Details of the datasets and prompts used for the ICL evaluation of our OPT-2.7B and Llama-2-7B AutoComprsssors and baselines. "Toks/ / dem." (Tokens per demonstration) denotes how long demonstrations are for the average example. "Cal." (Calibration) denotes whether we use calibration (Sachan et al.. 2022), and "Bal." (Balanced) means whether we enforce class-balanced sampling. We decide the ticks based on which method performs best on a held-out validation set.
AG News
SST-2
BoolQ
WIC
WSC
RTE
CB
COPA
MultiRC
MR
Subj
OPT-2.7B TB AutoCompressor
Zero-shol
78.0
60.2
19.5
00.60g.0)
56.2
43.6
69.0
13.8
60.0gu.)
56.7.u.m
summary vocs.
(2.7
8133
35.8
50.1.
61,3p..
51.8
36.9
71.6p..)
21 11
70.4pom:
63.2,
100 summary vecs.
(1.2
u7.0
57.5pm
50.7
60.2
55.5
54.4
71.9).)
45.6pa.
73. 73.
62.2
150 summary vecs.
(8.2(s.5)
62.6
50.8
518
63.5
55.)
58.3
71.4
46.7g.0
07.0
58.5
ICL (150 tokens)
72.5(2
70.8(12))
20.3
50.4.14)
52.3
57.6
51.gaa
713)
43.8
86.4.2)
61.7
ICL (750 tokens)
67.3/a.4)
87.5
69. 40.1
11.0
62.9
57.4
49.0
72.0
52.0(a.4)
80.7
73.6
O.T RMT
66.9ra.o
72.8
S8.4
50.3
64.4
55.2
42.2
68.8
43.9
62.5
69.8
1 step summary
66.3(g.5)
$6.5
49.6
51.0g1.00
S7.7
51.kua
53.3
67.4(..)
44.9
52,6;2))
63.3
2 step summary vecs.
65.2(
88.6
54.8
50.3
58.0
50.2
49.5
68 68.2
45.5
54.1.m
54.6
3-step summary veca.
63.9rs.a
84.5
41.8p.m
50.6
54.3
50.2
49.5
68 68.0ko.9
45.5
52.8
58.4
(150 wokens)
7..8
75.4gaa)
46.4)
21.7:22)
62.5çaa.)
57.2
46.5
69.3
43.9
89.0;
60.7
(750 tokens)
65.6(4.)
(5.7
51.5 7)
(0.2
57.8
48.2
70.9
51.66
87,5
71.6
OPT 2.7B
traitrained
Zero-shol
6
70.1
19. 19.4.p.) 55.8
(3.0
51 51.2
21.2
66.8
43.7
59 59.0go.)
66.2
(C50 tokens)
7
56.56
55.8
50.6;1
53.30
56.1ga-)
46 (6.2
71.7
43.7
86.7
61.9
ICL (750 tokens)
63.3...
91. 91.9gaa
63.0 63.0g.3)
50.0
63. 63.5
51.
32.1
73.40.0
53.5
89 89.9
61.4po.m
1lama-2-78 2-7B AutoCompnessen
Zero-shot
63.3
67.7
67.4
50.8
43.3
58.8
(2.9().))
52.5ya.
52.5yo.0
57.4gua)
49.3çau
summary
79.6
9422
70.1
51.621
47.7
66:3,z.
46.4
84.5y..)
52.6pu.9
91.5.1m
53.5
100 summary vecs.
87.6
92.6
66.3pa.)
52.5
42.9(2.5)
63.5
64.5
85.9
$6.1
00.7
$7.0
150 summary vecs.
45.4
92.3
68.0
$2.8
40.9
65.3k6.5
54.8
86.1
54.8
91.1
56.6
(C55 tokens)
74.5
92.4
07. 67.4,0.00
52.4:2m)
51.8
69.1g.
46.4g200
80.0q..)
52.5
79.7
87.9
(750 tokens)
$1.2(41:
93.8
67.
52.4g20)
40.0
73.
50.3/2.8)
82.6
47 47.0ga.2)
91.6aaa
60.7
Llama 2-7B Pre trained
Zero shot
68.6
70.0
51.4g0)
65.4
62.8
32.4
75.5xo.0)
54.5yo.0)
84. 84.1
48.9
(C50 tokens)
71.9rast
91.6
70.0
51.0;19
55.4
70.9
39.3(212)
84.2
54.5
90.6
63.6
ICL (750 tokens)
78.2
94.5
70.3
54.9
42.2
71344
51.3g.
85.3)..7
47.0q_.))
92.9/aa)
65.4(14.)
Table 12:
We evaluate the following models on 11 in-context learning tasks: The OPT-2.7B AutoCompresso and RMT model, the Llama-2-7B AutoCompressor, and the respective pre-trained models. For each fine-tuned model, numbers in bold the highest evaluation results using most 150 additional tokens. When using summary vectors, the OPT-2.7B AutoCompressor outperforms the RMT model on 8/11 tasks. Moreover, the OPT-2,7B AutoCompressor benefits from multiple compression steps on most tasks whereas the RMT model performs best without summary vectors on 7/11 tasks and benefits from 3-step summary vectors on none of the above tasks. The Llama-2 AutoCompressor achieves the absolute highest accuracy using summary vectors on 7/1 tasks. It also achieves the highest accuracy with summary vectors on 9/11 tasks using most 150 additional tokens.


Reviews:
[{"review_id": "fqz5SfjXQe", "reviewer": "Reviewer_Db2M", "summary": "", "questions": "1) Line 213-218 (Positional embeddings), the argument is a little unconvincing. Is it the case with all the types of position embeddings such as RoPE?\n2) It seems the position of summary tokens is not preserved (by excluding positional information), it would be interesting to see the impact of positional embedding even if it has limitations such as scalability.\n3) It would be great to see the results for other architectures and maybe other than (causal) decoder-only settings.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and very interesting and pertinent questions! We are pleased to see that you appreciate the simplicity of our method, and we agree that one of the main advantages of AutoCompressors lies in the ability to cache compressed corpora to speed up inference over frequently seen contexts. Please find some comments below addressing your questions about positional embeddings and alternative architectures..\n\n$~$\n> AutoCompressors with other type of positional embeddings such as RoPE\n\nThis is a fascinating question! Indeed, we have done follow-up experiments for LLaMA models with RoPE embeddings, and find that for RoPE embeddings, keeping the normal relative embeddings for summary tokens performs the best. In contrast to absolute position embeddings, this also will not reduce the context window nor introduce newly initialized parameters. We will add the LLaMA results in the final version.\n\n$~$\n> Positional information between different summary tokens is not preserved\n\nThank you for raising this point and acknowledging the limitations this would have on scalability. In small-scale ablation experiments, we find that adding explicit positional embeddings to the summary tokens makes no discernable impact on performance. This is also in agreement with [1], which shows that Transformers with causal masking can easily reconstruct useful positional information.\n\n$~$\n>It would be great to see the results for other architectures and maybe other than (causal) decoder-only settings.\n\nIn follow-up efforts to scale up to larger models and to adapt AutoCompressors to other architectures, we have successfully fine-tuned AutoCompressors from LLaMA-7b models. We will include the results in the final version and release the models publicly. We are pleased to report that AutoCompressors are more effective for LLaMA models than OPT models: fine-tuning on the same corpus and conditioning on 6144 compressed tokens gives perplexity gains of 8.1% in-domain and 10.4% out-of-domain over a fine-tuned LLaMA model (compared to gains of 5.6% and 5.0% for OPT 2.7b-based AutoCompressors reported in Table 1).\nWe are also very interested in applying AutoCompressors to other architectures, such as encoder-decoder, and are looking forward to pursuing this in future work.\n\n$~$\n[1] Kazemnejad, et al., 2023. The Impact of Positional Encoding on Length Generalization in Transformers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are various reasons that make me excited about the approach:\n1) Summary vectors soft prompts are 1-2 orders of magnitude lower than plain text which has not been achieved before.\n2) One can extend the context window significantly with minimal computational overhead.\n3) The idea emerges to be able to reason over summary vectors.\n4) It can help reduce the space occupied by ICL examples (demonstrations) and definitions.\n\nOverall, the paper is well-written and the contributions are strong. ", "reasons_to_reject": "In particular, I don't have any strong reason to reject the work.  I have included my questions in the question for the authors section.", "questions_to_authors": "1) Line 213-218 (Positional embeddings), the argument is a little unconvincing. Is it the case with all the types of position embeddings such as RoPE?\n2) It seems the position of summary tokens is not preserved (by excluding positional information), it would be interesting to see the impact of positional embedding even if it has limitations such as scalability.\n3) It would be great to see the results for other architectures and maybe other than (causal) decoder-only settings.", "paper_topic_and_main_contributions": "The work proposes a new method to tackle the long input-length context problems with language models. It adapts pre-trained LMs into AutoCompressors i.e. compressing long contexts into compact summary vectors as soft prompts. The idea is simple yet effective. In my perspective, the more interesting part is to maintain the low computational overhead while caching the summaries both during training (computational graph) as well as inference (caching).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bqt2fJlX7q", "reviewer": "Reviewer_9A3j", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are glad to see that you find our new applications interesting, and that you appreciate the effort we put into checking every aspect of our method through careful ablations and benchmarking.\n\n\nWe are also happy that you note that we do not fine-tune on any downstream tasks and that our positive results use a single unsupervised fine-tuning objective. As a result, there is substantial follow-up work to investigate how AutoCompressors can be further fine-tuned on specific downstream applications. The unsupervised nature of our work is an important feature which we chose to respect in all our evaluations. It is exciting that AutoCompressors can reason over summary vectors obtained from out-of-domain text and that pre-trained summary vectors support new operations, like Fused Summaries.\n\n$~$\n> Missed a relevant benchmark - Long Range Arena\n\nThank you for pointing out another interesting evaluation setting for efficient long-range transformers. However, we would argue that evaluating on Long Range Arena (LRA) is beyond the scope of the current paper for several reasons.\n\nFirstly, most tasks in LRA do not focus on text data, and are not suitable for adapting pre-trained language models, as they include image classification tasks, the Pathfinder tasks, and the synthetic ListOps tasks. Secondly, due to the diversity of tasks, LRA requires models to be trained for each task individually, and we feel that this is a departure from the unsupervised finetuning approach in many recent long-range LM works. LRA is a good benchmark to understand the architectural implications of AutoCompressors trained from scratch, but in this paper we focus on natural language applications, such as document retrieval and re-ranking.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tWell written.\n2)\tSolid experimental setting.\n3)\tAuthors methodology is well argumented.", "reasons_to_reject": "1)\tI think authors missed a relevant benchmark (Long range arena).", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors attempt to tackle the problem of processing long documents by proposing a novel LLM that can compress long contexts into “summary vectors”. Their method replicates the principle of the Recurrent Memory Transformer architecture, with some differences (like the fact that they accumulate summary vectors from segment to another). Their model processes long documents by fragmenting them to segments, prepend previously accumulated summary vector to each segment before recurrently processing it. At the end, the model outputs the representation of the whole document plus a small (compressed) vector that summarizes all the segments (the whole document). Authors demonstrate how such segments could be used to improve models’ performance in many tasks, such as in in-context-learning, retrieval-based and passage reranking tasks. The AutoCompressor (authors’ model) needs to be initialized by a pretrained model (OPT for instance) and to trained to produce summary vectors (through a language modeling objective). There is no finetuning on downstream tasks. Authors did a remarkable work in heavily experimenting their model, though it is not completely flawless.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VqWixDIjGo", "reviewer": "Reviewer_PB37", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are pleased to see that the reviewer appreciates the thoroughness of our evaluation process and notes the exciting practical implications of our work, which shows that concise summary vectors are more information-dense than plain text for a wide variety of applications. These soft prompt-based applications open up exciting new research avenues, and we discuss below how our AutoCompressors constitute an important new contribution compared to previous works.\n\n$~$\n> Limited innovativeness\n\nWe respectfully disagree with this point. While our approach indeed shares similarities with RMT (which we stated in both the introduction and methodology), our work introduced new techniques in methodology including\n1. Summary accumulation,  which summarizes vectors from all previous segments\n2. Randomized segmenting, which allows better compressing texts of variable lengths in downstream tasks.\n3. Stop-gradients, which reduces computational resources required for fine-tuning\nMore importantly, these innovations improve long-range language modeling performance consistently and enable new applications of summary vectors in downstream tasks (e.g., retrieval-augmented settings), as we demonstrated in our experiments.\n\nCompared to the RMT paper, we demonstrate the possibility of adapting pre-trained large language models (up to 2.7B parameters; we also plan to include results of LLaMA-7B models in our final version; see our response to Db2M if interested), and we show novel applications in downstream tasks, including a) in-context learning with many examples; b) retrieval-augmented language modeling, and c) unsupervised passage re-ranking. By contrast, the RMT paper trained small models from scratch and only evaluated them on language modeling and synthetic tasks.  Hence, we believe that our approach is novel and widely applicable in many scenarios. \n\n$~$\n> RMT not discussed in main context\n\nWe are surprised that the reviewer feels that the connection to the RMT work was not drawn out enough in our main text. RMT is discussed in the introduction (Line 061-072), and we recap the RMT architecture in our main sections 3.1 and 3.2, where we go over the key differences between AutoCompressors and RMT models, including summary accumulation, randomized segmenting, and stop-gradients during fine-tuning. Moreover, we extensively benchmark our AutoCompressors against RMT. We would appreciate more detailed feedback on how we can improve the writing and further highlight the connection to RMT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-The results show that all modules are helpful\n-method is efficient and requires less memory for training\n-This work is clearly written and easy to follow. \n-the applications are pratical", "reasons_to_reject": "-innovativeness isn't high\n-background of RMT is not discussed in main context", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author proposes teaching pre-trained LMs the skill to condense text into summary vectors, which are concise soft prompts. The models they introduce, named AutoCompressors, are developed using an uncomplicated unsupervised training strategy. This strategy motivates the model to retain vital data within the summary vectors. The author introduces a method called summary accumulation, where vectors from all segments come together to form the summary for the whole document. They also train AutoCompressors on inputs split up at random. This ensures the models are more effective in condensing texts of various lengths for different tasks. The analysis implies that summary vectors excel in specific tasks, outdoing few-shot ICL in 8 of 11 evaluated tasks. AutoCompressors were also explored for two applications: retrieval-augmented language modeling, where summary vectors yielded 1.5 times better comprehension than standard texts, and zero-shot passage re-ranking, where they provided the best balance of performance and speed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree on the soundness and excitement of this paper. The reviewers specifically call out the significance of the result by which sequence length in LLMs can be extended to much larger while also speeding up inference. The paper is well written with solid experimental results. The community will benefit from this work being published at EMNLP.
Decision: Accept-Main