RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation
Authors: Fangyuan Xu
Weijia Shi
Eunsol Choi
Venue: ICLR 2024 poster (2024)
Keywords: retrieval augmented language model
language modeling
question answering
summarization
distillation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We train compressor models to shorten retrieved documents passed to language models.

Abstract: Retrieval-augmented language models improve language models (LMs) by retrieving documents and prepending them in-context.
However, these documents, often spanning hundreds of words, make inference substantially less efficient. We propose compressing the retrieved documents into textual summaries prior to in-context integration. This not only reduces the computational costs but also relieve the burden of LMs to identify relevant information in long retrieved documents. We present two compressors -- an extractive compressor which selects useful sentences from retrieved documents  and an abstractive compressor which generates summary by synthesizing information from multiple documents. Both are trained to achieve performance gain in LMs when we prepend the generated summary from the compressor to LMs' input, while minimizing the summary length. When retrieved documents are irrelevant to the input or offer no additional information to LM, our compressors output an empty string, enabling selective augmentation. We evaluate our approach on the language modeling task and open domain question answering task. We achieve a compression rate of as low as 6% with minimal loss in performance for both tasks, significantly outperforming the off-the-shelf summarization models. We show that our compressors trained for one LM can transfer to other LMs on the language modeling task and provide a summary largely faithful to the retrieved documents.

Published as a conference paper at ICLR 2024
RECOMP: IMPROVING RETRIEVAL-AUGMENNE LMs WITH COMPRESSION AND SELECTIVE AUGMENTATION
Fangyuan Xu', Weijia Shi?, Eunsol Choi'
Department of Computer Science
'The University of Texas at Austin, UUniversity of Washington
{fangyuan, eunsol @utexas edu, sw 041 90 cs. washington. edu
ABSTRACT
Retrieving documents and prepending them in-context at inference time improves performance of language model (LMs) on wide range of tasks. However, these documents, often spanning hundreds of words, make inference substantially more expensive. We propose compressing the retrieved documents into textual sum- maries prior to in-context integration. This not only reduces the computational costs but also relieves the burden of LMs to identify relevant information in long trieved documents. We present two compressors - an extractive compressor which selects useful sentences from retrieved documents and abstractive compressor which generates summaries by synthesizing information from multiple documents. Both compressors trained to improve LMs' performance generated summaries are prepended to the LMs' input, while keeping the summary concise. If the retrieved documents information to LM, our compressor can return an empty string, implementing selective augmentation. We evaluate our approach on language modeling task and open domain question answering task. We achieve a compression rate of as low 6% with minimal loss performance for both tasks, significantly outperforming the off-the-shelf summarization models. We show that compressors trained for one LM can transfer to other LMs on the language modeling task and provide summaries largely faithful to the retrieved documents
end tasks when the
irrelevant to the input or offer additional
INTRODUCTION
Retrieval-augmented language models (RALMs) (Khandelwal et al. 2019 Izacard al. 2022 Lewis al.) 2020 Borgeaud et al., 2022) have shown impressive performance knowledge-intene
tasks
(Kwiatkowski et al.) 2019 Petroni et al. 2021). Simply prepending retrieved documents to the input
without updating the language models (LMs) (hi et al. 2023b; Ram et al. 2023 Si et al. 2022
allows retrieval augmentation even for black-box LMs, but such approach comes with limitations. First, it increases computational costs as LMs
encode substantially more tokens. Second, even if
we manage to adapt LMs to incorporate long context (Beltagy et al. 2020 Zaheer et al. 2020), these models struggle
all information in the context, frequently missing information placed in the
middle (Liu et al. 2023). Third, prepending multiple documents in-context can further confuse LMs with irrelevant information, degrading model performances (Mallen al. 2022 Shi al. 2023a).
To overcome such limitations, we propose RECOMP (Retrieve, Compress, Prepend), an inter- mediate step for RALMs which compresses retrieved documents into a textual summary prior to in-context augmentation. Figure iillustrates our approach. The generated summary should be concise to maximize efficiency, be faithful the retrieved evidence documents, and guide RALM to generate desired outputs when prepended to the input. We train compressors with learning objective that encourages both efficiency and effectiveness for a target L.M. Our framework enables selective augmentation, by generating an empty summary when the retrieved documents are irrelevant or unhelpful for target task and the target LM.
We propose two kinds of compressors: (1) Extractive compressor which selects relevant sentences from retrieved document set; (2) Abstractive compressor which generates a summary synthesizing
Our code is available ht

---

Published as a conference paper at ICLR 2024
No retrieval (0 takens)
Input querv.x
2010 X
Retrieved documents
RALM (749 tokens)
when did they stop making the missan xterra?
LM M
2015 X
Seryma Ternsoows, Misssspp. Early models include x and PRD--X wh choice of speed ma manull....
The Niesan Xterra a -irort-enginn, 2-whhel whoel drive, Siaeddoor
2015 0
Compressor
RECOMP (58 takens)
Summary
Retrieve
Compress
Prepend
Figure I: An illustration of RECOMP, which compresses retrieved documents into a texual summary before prepending it as input a language model inference time. The compressed summary guides the LM to generate the correct answer, while reducing the computation costs required to encode the documents. We describe training procedures for the compressors in Section 3
information from multiple retrieved documents. Both compressors implement multi-document query- focused summarization (Xu & Lapata (020), where summarize retrieved evidence document set with respect to the input query. As we aim to enable RALM to generate correct output when summary is prepended to the input query, we design training schemes to optimize the end task performance. Our extractive compressor is trained with a contrastive learning objective to identify sentences that lead to target outputs, and our abstractive compressor is distilled (West et al. 2022) from an extreme-scale LM (e.g. GPT-3), which achieves impressive summarization performance.
improve performance of frozen LMs language model- ing (Merity et al., 2016) and three question answering datasets (Natural Questions (Kwiatkowski et al.) 2017) and HotpotQA (Yang al. (018)), while prepending significantly fewer tokens compared to RALM without compression. We present two oracle compression methods an extractive oracle which selects a sentence in evidence documents that leads the best task performance and an abstractive oracle which chooses between a summary generated by extreme-scale LLM (e.g. GPT-3) and no retrieval augmentation that leads to the best task performance. Both oracle methods achieve compression rate as low as 6% and significantly outperforms prepending full documents. Our trained compressors also show promising results. For language modelling, both trained compressors achieve compression ratio of 25% with minimal performance drop. When applied to QA datasets, our best model compresses the documents to 5 10% of the original tokens with at most less than 10% relative performance drop and 2x inference time speed up. We conclude with careful analyses of our approach that reveal both its strength and weaknesses, thereby building foundation for future work.
Our experiments show that RECOMP
2019), TriviaQA Joshi
2 PROBLEM FORMULATION- RECOMP
Given an input sequence x, a target output sequence y and a set of N retrieved documents D (ld,,d,,- d2....xx] RECOMP compresses retrieved documents D with respect into summary 8 which captures core information in D relevant to x with significantly fewer tokens than D. Our architecture consists of two modules: compressor ea and LM M. In this work, we assume blackbox LM and train the compressor: Given the set of retrieved N documents ((d2, d2 x..)) and the input sequence x. compressor returns a token sequence s. We design our compressor be substantially smaller than LM M, as we aim to reduce computational costs of encoding a set of retrieved documents.
The output from compressor, S, should be: (1) Concise: The summary should be as short as possible to optimize efficiency. If the retrieved documents do not contain relevant information or retrieval augmentation is not necessary, S can be an empty sequence. (2) Efffecive: when is prepended to input sequence x and provided to LM M as prompt, LM should generate the target output sequence y. (3) Faithful: s should be faithful and interpretable summary of the input document set (i.e., must be entailed by the input document set (id1, d..).) We focus training compressors for conciseness and effectiveness. We summarize the key ideas for our two compressors, extractive compressors and abstractive compressor here, and discuss their training schemes formally in Section
Extractive Compressor Given sentences [$1, Sp...Sp in the input document set ([d1, d2, ....))))) D. we train a dual encoder model enca which embeds sentence s and the input sequence x into fixed-
Improving retriever is the focus of this work, so we assume set of retrieved documents provided.
2

---

Published as a conference paper at ICLR 2024
dimensional embeddings respectively. Their inner product represents how helpful it would be for the LM M to prepend Sj to the input x to generate y. The final summary s from the compressor will be a concatenation of top N sentences ranked by their inner product with the input. As this approach is extractive, we assume the faithfulness criteria is mostly satisfied
Abstractive Compressor We train an encoder-decoder model encdeC, to serve as an abstractive compressor, which takes the input sequence x and concatenation of retrieved document set D [d1:d2; .x.]]) and output a summary s. Although we do not have human annotations to train this model, prior work (Goyal et al. 2022; Chen et al., 2023 Potluri et al. 2023) suggests that the extreme- scale LMs can generate good query-focused summaries when prompted carefully. Yet, using an extreme-scale model the compressor is not desirable as we want the compressor to be substantially smaller than the LMs. Thus, we perform distillation (Hinton et al., 2015) of extreme-scale LMs to build lightweight abstractive compressor encdeca. We do not train specifically for faithfulness, but later manually evaluate the faithfulness in Section 6
3 LEARNING THE COMPRESSORS
Our compressor resembles text summarization models that the output should be faithful to the original input, yet the main goal is different. Instead of capturing salient information for humans readers, compressors aim to produce a concise text that are useful for a LM on an end task. this section, describe how to train the extractive compressor ($3.1) and the abstractive compressor (33.2)
leveraging end task signals. Further training details can be found in the Appendix A.2
3.1
EXTRACTIVE COMPRESSION
As we formulate extractive compression as a ranking problem, training extractive compressor re- sembles training reranker for the retrieved document:" with two differences. First, compressor considers a different granularity of input (sentence) compared to the initial retrieval unit (paragraph). Second, the sentence is evaluated based on whether it is useful input for the LM M the downstream task (Shi et al., 2023b Ram et al. 2023).
Model We train a dual-encoder model ence which encodes the input context and the candidate sentence S; separately. We obtain embedding of and 8; by taking the representation of the [CLS] token respectively, and compute their sim- ilarity by their inner product. We initialize our model with the contriever checkpoint (Izacard et al. 2021). This model con- sists of 11M parameters, satisfying the efficiency desideratum of compressor.
Input: Base LM M, Compressor enco. Training data S, where x; is input, S, (sj}]' is of candidate sentences from retrieved documents for x, yi is the target answer, and score threshold c.
Output: An updated extractive compressor encoder
ence
1: T 0
2: for i [L....,") I') do
3:
pi +- argMax ((S
4:
for j (l....,77 n2
5:
L+
6:
if Score(M yi. (s))
+
Score( M, yı, [pr then
presents pseudocode for training an extractive compressor with contrastive loss for the language modeling task. For each input query X;, we iden- tify positive and negative sentences from retrieved documents.
Training Figure 2
7:
C L
if L
> 0 then
9:
N; +
T T [(x,, Ni)) encs(x:)))
10:
11: ence = Finetune(enca,)
For each pair of input sequence Xi and candidate sentences 8j. we
Figure 2: Learning extractive compressor for lan- guage modeling task.
measure Score(M, y1, [s]: lygpM(yl (s:: log likelihood assigned to target output according to LM M when candidate
Recent work /hhag et al. such cases are rare compared to ahstractive approaches which can easily hallucinate.
2022 shows that extractive approach does always preserve faithfulness, but
Raa et al. (023) proposes document reranker based cross-encoder model, which is similar set-up sentence selector, but less compute efficient.
3

---

Published as a conference paper at ICLR 2024
sentence is prepended to the input. We consider the sentence with the highest log likelihood positive example pi (line 3). To construct negative examples N₁ = {nx ( we choose up to five sentences with top contriever score that has the log likelihood lower than the positive sentence for threshold(line 6
Training compressor for QA task works similarly, but scoring will evaluate whether the LM will generate the correct answer with summary prepended (change in line Pseudo code for the QA tasks in Figure/o]the Appendix. We train our encoder with a contrastive loss (Karpukhin al. 2020a)) maximizing the similarity between positive pairs (Xi, pi) and minimize negative pairs (Xi, N2). The (2mm(24)P))
training objective is to minimize -log geim(x;)p))) Enjen, (mm(igg))))
Data For the language modeling task, generate training data using the training split of the Wikitext- 103 dataset, selecting the top 20 sentences from the top BM25 retrieved documents for each input context For the QA tasks, we generate training data using the training split and consider the top 20 sentences from the top 5 contriever-ms. marrrreveer--mmmroooo retrieved documents. We report detailed statisties for the training data in Table 5 in the appendix. For each sentence from the retrieved documents, we prepend the Wikipedia page title to to for decontextualization.
3.2 ABSTRACTIVE COMPRESSION
We distill the query-focused summarization ability of extreme-scale LM by generating training dataset with it, filtering the generated data, and training an encoder-decoder model from the filtered dataset (West et al., 2022). In contrast to prior work (Jung et al., (023) which use intrinsic summariza- tion metric for filtering, we filter with the LM's end task performance with the generated summaries prepended. Fig. presents pseudo algorithm for abstractive compressor training.
3.2.1 CREATING TRAINING DATASET FOR DISTILLATION
Generation From Teacher Model For the language modeling task, we manually construct four prompts to summarize evidence document set ((p:)) 2 Given input X;, retrieved doe- ument set D and a prompt Pj to summarize the document with respect the input, GPT-3.5 generates a summary (line 3).
Input: Teacher LM M. LM M, Summarization prompt set {PI)T. Compressor encdeca, Training data {xi, yr where x; is input, D; is the set of retrieved document for x1. yi is the target answer.
Output: An updated encdec,
1:
T+ 0
Filtering with Critic After generating
2:
for E f1...., T} do
mary for each prompt template, we select the summary which results in the highest end task performance for each example (s&) as the tar- get summary (line 8 Score Vi. Vi. [sj; x;]) is the same as the extractive compressor above. We then compare the end task performance with the target summary prepended and with input Xı only (i.c. no retrieval) on base model M (line 6.. If the end task performance gets worse (e.g., increase in perplexity) when prepending the summary, we set the target summary to empty string (line 7 otherwise we add the tar- get summary to the training (line Dp. This allows for selective augmentation and mitigates the risk of prepending irrelevant documents.
3:
Ur f- -00
4:
for j E {1, n do
5:
Sj Decode (M, [pj x;; D:l)
6:
2y Score yis [5j xi)
7:
if v > e, then St + + U2
8:
9:
va Score( y1, [xi)
10:
if Ur < ua then
11:
T TU ((x, D; 0)} continue
12:
13:
T TU (),)).
14:
encdeca Finetune (eeduune(nncecs,11
Figure 3: Learning an abstractive compressor for language modeling task.
Constructing training datasets for the question answering tasks works similarly, with the following modifications. As summarization for the question answering task more straightforward, we use
http://huggiggffcccccoc ingaginnnncc.... c facebock "The exact prompts can he found in Table 14 in A.1
We use gpt-3. S-turbo in all experiments.
4

---

Published as a conference paper at ICLR 2024
single prompt for each dataset. We filter out examples where prepending the summary does not lead to performance improvement. Pseudo code for the QA tasks is in Figure 7 in the Appendix.
Model & Training We use encoder-decoder LM (775M), initialized from T5-large checkpoint (Raf) fel et al. 2020 This model has been trained with summarization datasets (Hermann et al., 2015).
Data We summarize top 5 retrieved documents for both language modeling and question answering tasks. We generate training examples using subsets of the training set for the Wikitext- 103 dataset, the entire NQ training set and TriviaQA training set. For HotpotQA, we only generate abstractive summaries for the subset of training data (56%) where the gold answer is in the retrieved documents to reduce API costs. We report percentage of data filtered and empty summaries in Table 5 in A.1
4 EXPERIMENTAL SETTINGS
We evaluate our approach on language modeling and open-domain QA following prior work (Shi ct al. 2023b; Ram et al.) 2023). For both tasks, we report the task performance as a measure of effectiveness and the number of tokens provided in context measure of efficiency.
4.1 LANGUAGE MODELING
We evaluate language modeling perplexity on WikiText- 103 (Merity et al. 2016) benchmark on three open-sourced LMs of varying scale: GPT2 (117M), GPT2-XI. (1.5B: Radford al. (2019) and GPT-J (6B; Wang & Komatsuzaki (02IT). ). We train our compressors using GPT2 as the base model and evaluate whether the trained compressor transfer to GPT2-XL and GPT-J. We use the BM25 retriever (Robertson & Zaragoza 2009) to retrieve from the Wikipedia corpus from Dec. 20, 2018 (Karpukhin et al. (020a).. The articles are truncated into non-overlapping documents
100 words.
During retrieval, articles containing the input sequence x is removed from the corpus to prevent data contamination. Following Ram et al. (023), we perform retrieval every 32 tokens.
4.2 OPEN-DOMAIN QA
Datasets We evaluate our model on three benchmark dataset: Natural Questions (NQ) (Kwiatkowski et al. 2019). TriviaQA (Joshi et al. 2017)) and HotpotQA (Yang et al. 2018). We report results on development set of NQ, test set of TriviaQA and randomly sampled 500 examples from HotpotQA development set. We report Exact Match (EM) and token-level F1 of answer strings.
Base Language Models & Retrieval Corpus We use Flan-U1.2 (20B) (CB)(Chung et al. 2022). large scale instruction-tunet LM. We use contriever model trained on MS MARCO dataset (Campos et al.) 2016) retriever on Wikipedia corpus from Dec. 20, 2018 for all three datasets. The articles truncated into non-overlapping documents of 100 words.
Prompt Format We include few-shot in-context examples in the prompt, followed by the retrieved documents and the question. We use five randomly sampled in-context examples from training set, which constitutes 10, 147, and 149 tokens on average for NQ, TQA and HotpotQA respectively. We concatenate retrieved documents in ascending order of retrieval score, with the highest scored one closest to the question (Si et al. 2022). We do not include the retrieved documents for in-context examples it did not improve performance. An example input can be found in Appendix Table I3
4.3 BASELINES AND ORACLES
Baselines We first consider two heuristic token and phrase-level compression methods: BoW, which converts the retrieved documents to a list of ordered unigram and concatenates them together and Named Entities (NE), which extracts a list of ordered named entities from retrieved documents and concatenates them. For the extractive compressor on the language modeling task, we use BM25 and Contriever Izacard al. (2021), which rank the sentences by their similarity the input as baselines. For the QA datasets, we report results using BM25, Contriever finetuned on MS MARCO and DPR (Karpukhin et al. 2020b) fine-tuned on NQ. We also report Random baseline which
5

---

Published as a conference paper at ICLR 2024
Table 1: Results on language modeling task. We report results on GPT-2, GPT2-XL and GPT-J with
compressors trained with GPT-2.
In-Domain GPT2 (117M)
Out-Domain
GPT2-XL (I.5B)
GPTTJ (6B)
In-context Evidence
# tokens
PPL
tokens
PPL
# tokens
PPL
37.84
0
19.89
0
11.44
RALM without compression
Top document
141
32.90
141
17.86
141
10.57
Top documents
512
35.53
Phrase/token level compression
Top document (BoW)
66
36.13
66
18.85
66
10.97
Top document (NE)
34
37.23
33
19.67
11.39
Extractive compression of Top documents (select top sentence)
Oracle
32
30.36
32
16.58
9.92
Oracle (w/ gp(2)
32
30.36
32
16.99
32
10.22
Random
27
36.98
27
19.55
27
11.32
BM25
33
36.63
33
19.02
33
11.08
Contriever
35.54
33
18.98
11.05
Ours (init. w/ Contriever)
31
33.67
31
18.19
31
10.73
Abstractive compression of Top documents
Oracle
68
30.67
66
16.87
65
10.10
Oracle (w/ gpr2)
68
30.67
68
17.23
68
10.37
GPT-3.5
33
34.84
33
18.70
33
10.96
TS
37.80
19.92
11.5
Ours (init. w/ T5)
15
33.64
15
18.09
15
10.66
randomly selects a sentence from the retrieved documents. For abstractive compression, we report the performance of the off-the-shelf T5 (large, 770M) model and that of GPT-3.5 model. As we experimented with multiple prompts for the language modeling task, we report the performance of the summaries generated by GPT-3.5 model with the best single prompt.
Oracle We explore the performance upperbound of compression by considering two oracle ap- proaches. For the extractive approach, construct oracle compressor by considering all sentences & in the evidence document set and choosing the sentence that leads to the best end task performance for each example. For the abstractive approach, we consider summaries generated from different prompts ({s)]ī i in Figure 3) and an empty summary, and choose the one that leads to the best end task performance. As oracle compression is model dependent, results by using GPT-2 as reference LM (Oracle w/ gpf2) to test how well oracle sentences for one model transfer to other models for the language modeling task.
also report modee--ndepennent
5 RESULTS
Language modeling Table I reports the results on language modeling task. All retrieval augmen- tation methods improve perplexity over no retrieval setting across three LMs. Heuristic token / phrase-level compression methods (BoW and NE) are worse than prepending uncompressed docu- ments, potentially due to the disfluency of the prepended text.
Both oracle settings show substantial gain over prepending the entire document set, with only 6-13% of tokens. More tokens are not always better: prepending top document outperforms prepending top 5 documents. This confirms that the naive retrieve-and-pree -pepeed approach has significant room for improvement, as prepending irrelevant documents can hurt performances.
Our trained extractive compressor significantly outperforms other extractive baselines (Contriever and BM25) across all three LMs, while prepending slightly fewer tokens. Compared to prepending one document, achieve compression ratio of 25% at minimum performance drop. Our trained abstractive compressor performs the best across the board, achieving the lowest perplexity and the highest compression ratio. Our abstractive compressor achieves high compression rate through

---

Published as a conference paper at ICLR 2024
Table 2: Open-domain QA results with Flan-UL2 (20B) the LM M. We report number of tokens provided as in-context evidence document, excluding the in-context examples. We train separate compressors (one extractive, abstractive) for each dataset. Extractive compressor selects one sentence for NQ/TQA, and two sentences for HotpotQA.
NQ
TQA EM
HotpotQA
In-Context evidence
# tok
EM
F1
# tok
F1
tok
EM
F1
21.99
29.38
0
49.33
54.85
17.80
26.10
RALM without compression
Top documents
132
33.07
41.45
136
57.84
64.94
138
28.80
40.58
Top 5 documents
660
39.39
48.28
677
62.37
70.09
684
32.80
43.90
Phrase/token level compression
Top 5 documents (NE)
338
23.60
31,02
128
54.96
61.19
157
22.20
31.89
Top 5 documents (BoW) 450
28.48
36.84
259
58.16
65.15
255
25.60
36.00
Extractive compression of top 5 documents
Oracle
34
60.22
64,25
32
79.29
82.06
70
41.80
51.07
Random
32
23.27
31.09
31
50.18
56.24
21.00
29.86
BM25
36
25.82
33.63
37
54.67
61.19
74
26.80
38.02
DPR
39
34.32
43.38
41
56.58
62.96
78
27.40
38.15
Contriever
36
30.06
31.92
40
53.67
60.01
78
28.60
39.48
Ours
37
36.57
44.22
38
58.99
65.26
75
30.40
40.14
Abstractive compression of top 5 documents
Oracle
51
45.68
53.66
37
71.01
76.38
102
35.80
46.25
GPT-3.5
56
37.12
46.35
41
62.03
69.66
107
31.60
42.65
TS
10
25.90
34.63
7
55.18
62.34
7
23.20
33.19
Ours
36
37.04
45.47
32
58.68
66.34
64
28.20
37.91
selective augmentation, prepending summaries to only 33% of examples (length distribution of generated summaries in Fig.
Open-domain QA We report the results QA tasks in Table Similar the language modeling task, all retrieval augmentation methods improve performance over no retrieval setting, across three datasets, consistent with previous study on other LMs (Shi et al. 2023b Mallen et al.) 2022 Si et al. 2022). Unlike language modeling, prepending five documents shows sigmificant gains over prepending single document, motivating the use of compression to incorporate more documents.
We find that extractive oracle outperforms the abstractive one in all datasets. Extractive oracle selects the best one from N candidate sentences, while abstractive oracle selects from two options prepending GPT-3.5 summary prepending nothing. Both oracles show improvements over prepending all information, suggesting that removing irrelevant information benefit the model
Among extractive baselines, DPR performs the best as it has been fine-tuned on high-quality NQ data. IQ, selecting the top DPR ranked sentences from top 5 documents outperforms prepending top 1 document, with much fewer tokens (39 vs. 132). However, its performance degrades in out of domain datasets. Off-the the-shelf summarization model (T5) boasts the highest level of compression, achieving 4-6 points gains in EM while adding mere 7-10 tokens.
The trained compressors, both extractive and abstractive, shows promising performances. On NQ and TQA, the abstractive approach is more effective. On NQ, it achieves compression ratio of 5% tokens while losing 2 EM points compared to prepending full documents. On TQA, we observe similar trends, compression ratio of 5% tokens while losing 3.7 EM points compared to prepending full sets of documents. On HotpotQA which requires multihop understanding of documents, we find extractive approach to be more helpful, achieving 11% compression points compared to prepending full documents. We find that learning an abstractive compressor for more complex tasks, such as HotpotQA, demands further study. While extreme-scale LLM boasts competitive summarization performance for single document, they are not good at synthesizing information from multiple documents (Shaib et (2023); See Section 6 for further analysis).
while losing 2.4 EM
'We provide example where compressed summary yields correct answer while prepending full document does in Table in the appendix.
7

---

Published as a conference paper at ICLR 2024
6 ANALYSIS AND DISCUSSIONS
Transferring Across Different LMs One benefit of textual summary is that they can transfer to other LMs, unlike approaches such as soft prompts (Wingate al. 2022 Chevalier al. 2023 Mu et al. 2023). We evaluate whether our compressors trained to achieve high performance with respect specific LM (GPT2 for language modeling. FlanUL2 for open domain QA) can transfer to other LMs. For language modeling, we find that trained compressor transfers well to other LMs (GPT2-XL and GPT-J), despite they are much larger LMs (Table T For open domain QA, we tested transferring our compressors to LLaMA-13B (Touvron et al. 2023) (Table 11 in the appendix). Overall, the performance is worse than the LM from which compressors are trained on, sometimes unable to outperform other compression baselines (c.g.. ... clear gain from using contriever vs. trained contriever on TQA/Hotpo(QA), leaving considerable gap to the oracle compression for LLaMA itself. Yet, NQ/TQA, our compressor obtains 5% compression ratio with less than 5 EM drop compared to full document setting, showing the robustness of our compeve-compress- corpress-prepeed paradigm.
How do the length of the summaries vary? Can the learned compressor reliably determine when LMs require retrieved documents? As retrieved documents were hurt- ing the model performances for some input queries, 4-24% of training examples for abstractive compressors contain empty summary. Fig. 4 presents the length distribution of abstractive summaries on NQ and Wikitext (histograms for other datasets is in Fig. 8 in the appendix). The input doc- ument lengths do not vary significantly across examples, yet we find the abstractive summary vary significantly in length, suggesting abstractive compressor enables selective retrieval augmentation. We have not experimented selective compression with extractive compressor, fixing the number of prepended sentences for the entire dataset. We leave enabling adaptive augmentation with extractive summarizer to future work.
Wikitext
NQ
6000
1000
Oracle Ours
4000
2000
0
500
100
Figure 4: Histogram of abstractive sum- mary length (# tokens) distribution.
How does model leverage the in-context doc- uments? We evaluate whether retrieval aug- mented LMs tend to copy answers verbatim from in-context evidence documents generate answers not present in the documents. This is an desired behavior only when the gold answer is in the evidence. We first report how frequently gold answer span present in evidence Gold in Evi). As expected, full documents con- tain the answer most frequently, followed by NE and GPT-3 compression. However, having more gold answers the evidence doesn't I equate to better performance, the model cannot always identify the correct answer from the evidence (84 % for NE v.s. 98% for T5(rrs)).
Table 3: Analysis on usage of in-context evidence for NQ. For the last column, report frequency of model copying from the evidence on a subset where gold answer present / when it is not.
Evidence
EM
% Gold in Evi.
% in Evi.
(%
Top
33.1
36
92 / 51
Top 5
39.3
57
96 81
NE
26.0
46
84 / 48
Oracle sent
60.2
Contriever
30,2
25
88/ / 36
Ours
36.6
28
90 / 33
GPT-3.5
37.1
45
98 / 85
T5 Ours
25.9
30
52 / 20
We also observe that model can be easily dis- tracted by irrelevant contexts, copying from document span even when it does not contain gold answer, echoing findings from prior work (Shi
37.0
34
98 / 39
al. 2023ap. Prepending top 5 documents has a higher frequency (81%) of copying incorrectly compared to top document (51%), and GPT-3 compression leads to an even higher incorrect copying frequency (85%), potentially query-focusee summarization generates sentences that seemingly contains the answer. Our compressor successfully reduce such erroneous behavior to 39%.
Are the generated summaries faithful and comprehensive? We (the authors) manually evaluate outputs of the abstractive compressors two axes (Chen et al. 2023): Faithfulness: whether the summary can be entailed by the retrieved documents, Comprehensivenes:: whether the summary contains sufficient information to answer the question, regardless of whether the generated information comes from the retrieved documents. For both, we select one of three labels: Yes, Partially, No, and report the % of Useful summaries which both faithful and comprehensive. Annotation sample
8

---

Published as a conference paper at ICLR 2024
can be found in Table T the appendix. We evaluate the summaries generated by GPT-3.5 and our abstractive compressor. We randomly sample 30 non-empty summaries from the test set.
Table 4 presents annotation results. GPT-3.5, substantially bigger than compressor, generates more use- ful summary across all three datasets. Overall, our abstractive compressors were less faithful compared to the original GPT-3.5, while improving comprchensiveness. The effectiveness of summarization also depends on the datasets summaries from both mod- els were the most faithful for TQA and the least faithful for HotpotQA dataset. terms of comprehensive- ness, find both models easily find the information for NQ, but struggle with HotpotQA. These results partially explain why the performance gain was limited for HotpotQA.
Table 4: Manual analysis abstractive summaries generated for NQ, TQA and HotpotQA (HQA) dataset.
% Faithful
% Compre.
Dataset
Model
% Use.
Y
P
Y
P
N
GPT-3.5
90
0
10
97
0
3
83
NQ
Ours
80
13
100
0
0
80
GPT-3.5
97
90
0
10
TQA
Ours
83
3
14
96
0
4
77
GPT-3.5
74
0
26
78
0
22
50
HQA
Ours
67
0
33
74
0
26
40
RELATED WORK
2021 improves efficiency of RALMs by reducing retrieval cost, such as data store compression and dimensionality reduction for neural retriever. A line of work proposes reduce retrieval frequency (He al., 2021 Mallen et 2022 Martins et al., 2022). In this work,
Efficient RALM He et al.
we improve efficiency of RALM by compressing retrieved documents into concise summary or an empty sequence, facilitating selective retrieval augmentation.
Prompt Compression Recent work (Wingate et al. 2022; Chevalier et al. 2023 Mu et al. 2023
proposes compressing long contexts into soft prompts that can be used by LMs, rather than textual summaries. Such soft prompts serve efficient replacements for plain-text context, minimizing the computational costs during inference. Another related line work proposes context distillation (Snell et al. 2022 Choi al. 2022 Padmanabhan et al.,, (023), which injects the prepended context into the parameters of an LM. Compared to above approaches, our approach yields more interpretable textual summary that can transfer across different LMs, and can be applied to black box LMs without requiring gradient updates. Prior work has studied textual compression for other tasks, such as political fact checking (Chen al. 2023) and instruction learning (Yin et al., 2023). Concurrent works (Li et al.) 2023 [023]Jiang et al. (023) propose token-level prompt compression methods which leverage small model to prune out redundant tokens, while our method preserves readability of the
compressed text. Wang et al. (2023) proposes to filter irrelevant retrieved documents and Yoran et al. (2024) finetunes the language model be able to properly handle irrelevant contexts.
Distillation Goal Oriented Summarization Recent work introduces symbolic knowledge distilla- tion (West et al. 2022), which transfers knowledge from teacher model by training a student model on training dataset generated with the teacher model. For better performance, they introduce critic criteria, which filter undesirable examples from generated training dataset. Such distillation technique has been applied for various applications including summarization (Jung et al. 2023). which aims to generate high quality summaries while we optimize for generating effective summary for downstream LMS. Our setting is similar to Hsu & Tan (2021), which trains an extractive summarization model to optimize for prediction accuracy of sentiment prediction model which uses the summary.
8 CONCLUSION
We introduce RECOMP, a method which compresses retrieved documents into textual summaries before prepending them to improve in-context retrieval augmented language models. We present two compression models an extractive compressor and an abstractive compressor. We design a training scheme which leverages end task signals from blackbox LM to generate useful summaries and enable selective augmentation. Our experiments show that our compressors can improve the efficiency of retrieval augmented LMs significantly with minimal drop in performances.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank the members of the UT and UW NLP community for feedback on the project. We especially thank Alisa Liu, Junyi Jessy Li and Greg Durrett for providing comments on the draft. The project partially funded by NSF grant (I14-2312948).
ETHICS STATEMENT
We commercial language model to generate training data for compressors, which might include factual error. We conduct careful human evaluation on the data generated and present our analysis in the paper.
REPRODUCIBILITY STATEMENT
We release our codes, prompt, and data generated with API access publicly.
REFERENCES
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. ArXiv, flss/2004.051500 2020. URL httph::///aaat www.ntescccoooaa.ooooo wtt.:cchci.a...ooo/oooosssso
215737171
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Ge- offrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. Kama- lika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Reseurch, pp. 2206-2240. PMLR, 17-23 Jul 2022. URL mlr htt.e/ccc............s/////r/raa htt.mmd222a.mmmmmllmm.mmm
Daniel Fernando Campos. Tri Nguyen, Mir Rosenberg. Xia Song, Jianfeng Gao, Saurabh Tiwary, Ran- gan Majumder, Li Deng. and Bhaskar Mitra. Ms marco: human generated machine reading com- prehension dataset. ArXiv, abs/161 1.09268, 2016. Utthttps::/ //ap:::://apii deeeantiiscoa
orq/Corpus ID: 289517
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex claim ver- ification with evidence retrieved in the wild. ArXiv, abs/2305. 11859, 2023. URL https
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint 2rXiv:2305.14788 2023.
Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt injection: Parameterization of fixed inputs. ArXiv, abs/2206.1 1149,, 2022. URL httpss://app...... http://pp...emttt...cccccoo..oor
Corpus:D:2499937e8
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu. Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. finetuned language models. ArXiv, abs/2210.11444 2022.
Scaling instruction instruction-finetuneed
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209 12356, 2022.
10

---

Published as a conference paper at ICLR 2024
Junxian He, Graham Neubig, and Taylor Berg- Kirk--tiiattrc.. Efficient nearest neighbor language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5703-5714, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00.18633/v1/2222 1208::///22.....mmmmmmmaa-a--4 URL https
//aclantho ogy. org 2021 emn p- 461
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. ArXiv, abs/ 0oo//1506.0344)) 2015. btRp::/s: http:/ppp..eemnnt.cccchloooaaa
http:///ee////pppaaaaaaaaaaaaaaa www.:a/.rggggooppoooeeoeoo 6203757
Geoffrey E. Hinton, Oriol Vinyals, and Jeffirey Dean. Distilling the knowledge in a neural network.
ArXiv, abs/1503.02531, 2015.
7200347]
Matthew Honnibal, Ines Montani, Sophie Van Landeghem, and Boyd Adriane. spaCy: Industrial- strength natural language processing in python, 2020.
Chao-Chun Hsu and Chenhao Tan. Decision-focused summarization. Proceedings tf Conference on Empirical Methods in Natural Language Processing, pp. 117-132, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. emm monlp-maain.0
the 2021
10. URL
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021. URL htttts::/axxi..rrgpppt
09118
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. ArXiv, abs/2208 003299, 2022.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Iims in long context scenarios via prompt compression. ArXiv, abs/2310.06839 2023. URL https: api. semanticscholat wttp:/aarr..rooooouuu......
263830692
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 1705.03551, 2017.
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing, arXiv preprint arXiv:2305. 2365,665556.6 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020a. Association for Computational Linguis- tics. doi: 001116663/111220.... htt.:/00mmmmmmmmmi..mmammm URL /ttp::://aclaaattlloog.oooo//// 200
8mn 1 p-ma i 550
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing, 2020b. URL https //api. www.sttttttncooo.cooeteeeccctc
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. ArXiv, abs/191 1.00722 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
11

---

Published as a conference paper at ICLR 2024
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466 2019. doi: a0.11622tacl 00276. URL
1026
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for seowledge-intensiin nlp tasks. ArXiv, abs/2005. 1401, 2020.
Yucheng Li, Bo Dong. Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. In Conference on Empirical Methods in Natural Language Pro- cessing, 2023. URLhttps::/ apii semant
orgg/CorpustD:2333000..
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape. Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112. 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. ArXiv, abs/2212.1051 2022.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. Chunk-based nearest neigh- bor machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4228 4245, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 11863361/20222 emmllp main.284. URL https://ach antho ogy. org/ 2022. emmlp-ma i 284
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ArXiv,
2016.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open open-domain questions. In Conference on Empirical Methods in Natural Language Pro- cessing, 2020. URL Itttp::/ app. wwtpp/sseaaatcscccooosseeeme org/ Crt./oorrr.......2110522222222
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning compress prompts with gist tokens. ArXiv, 00b//334.088467, 2023. URL https://appi
2581 79012
Shankar Padmanabhan, Yasumasa Onoe, Michael J.Q. Zhang, Greg Durrett, and Eunsol Choi. Propagating knowledge updates to lms through distillation. ArXiv, abs/2306.09908 2023. URL
https://ap1 semant iernniiscchn icschoo ar. org/ Corpus ID: 2559163333
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis. Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.118633/112000 wat.222l..aacmmmai... URL https //aclanthology. wwt./org///ooo/ maaaal-mmiin...
Abhilash Potluri, Fangyuan Xu, and Eunsol Choi. Concise answers to complex questions: Sum- marization of long-form answers. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9709-9728, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.188653/11/222..ccoooogg 2023 wwwlo///22....1oooo........
URL
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 214555585-555,, 2020.
12

---

Published as a conference paper at ICLR 2024
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Leytonn-rown, and Yoav Shoham. In-context ettrevallauggenteer language models. ArXiv, abs/2302.00083, 2023.
Nils Reimers and Iryna Gurevych. Sentence-BERT. Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp. 3982 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/w1/D19- 1410. URL http:::/ /tttaa://aclaattoooooo.ooo//////
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3:333-389, 2009.
Chantal Shaib, Millicent Li, Sebastian Joseph, lain Marshall, Junyi Jessy Li, and Byron Wal- lace. Summarizing, simplifying, and synthesizing medical evidence using GPT-3 (with vary- ing success). In Proceedings of the 6/st Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 1387-1407, Toronto, Canada, July 2023, As- sociation for Computational Linguistics. doi: 10. 1/202/3//1/222..a/m act.s02.....--sorrr11999999 URL https //aclanthology ort. 2023 wwwp:/22.............../
acl acl-scort...11..
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Huai hsin Chi, Nathanael Scharli, and Denny Zhou. Large language models can be easily distracted by ir- relevant context. In International Conference on Machine Learning, 2023a. URL https
: 256459776
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models. ArXiv, abs/2301.12652, 2023b.
Chenglei Si, Zhe Gan, Zhengyuan Yang. Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to reliable. ArXiv, abs/2210.09150, 2022.
Charles Burton Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. ArXiv, abs/2209.15189) 2022. URL httph: api. semanticscholar.c www.aaorggo/ooo/ooeeee
252668389
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodrigue7, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288 2023. URL httph:tpss/// /appi. semant icccho ar worg/Cooorsssi// 259950998
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. formb--ttrrrrfffrrr--oro-o--- jax May 2021.
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md. Rizwan Parvez, and Graham Neubig. Learning to filter context for -uugmeeeed generation. ArXiv, abs/2311.083777 2023. URL https
//api.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4602-4625. Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1865/vv/ 2022.naacl- 341. URL wttpttttpp...............t //acl www.eeaantthhooooooooooooo waathelogy.com wwwpgyy...gg///2// naacl
n. 341
13

---

Published as a conference paper at ICLR 2024
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and con- trastive conditioning for controllability and toxicity reduction in language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pp. 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. htgpsss///222222ddnnnn-eemmm/// 4m11.1ge-mmmpp411.. URL htttps:/acclantioooogg.oooo 2022
Fincings-ennh 412
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-ar natural language processing. ArXiv, abs/1910.03771, 2019.
Yumo Xu and Mirella Lapata. Coarse-to-fine query focused multi-document summarization. In
Conference on Empirical Methods in Natural Language Processing, 2020. URL https://app
Semanticscher www.:orrgrooooooeoeee Corpus 22666222
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Fan Yin, Jesse Vig, Philippe Laban, Shafiq R. Joty, Caiming Xiong, and Chien-Sheng Wu. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https://api
semanticschon htt.hhooaa..ree/ooooaaaa01:::03.
org/ Corpporrppsaacc 25:::s3:::29995999
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL http::s//opeeeeeveeeentttt http:://eeeaeeeeeeeeeeeeeeerrnda 2ottt//ooooommddd222222222222222
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, San- tiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. ArXiv, abs/2007. 14062, 2020. URL https
7/api. hwwp:tttttccccc...arooeeeoeeeeee 2D:
Shiyue Zhang, David Wan, and Mohit Bansal. Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization. ArXiv, 000/2229903549 2022. URL
httpps://ape semanticschii ar. wttp//oorpsS1::2221888333
14

---

Published as a conference paper at ICLR 2024
NQ extractive
TQA Extractive
HotpotQA extractive
EM retrieval
0.4
0.4
EM retrieval
0.2
0.3
0.6
0.3
0.0
0.0
0.5
EM
0.0
0.2
0
EM oracle NQ abstractive
EM oracle TQA Abstractive
EM oracle
HotpotQA abstractive
retrieval
0
0.5
0.2
EM retrieval
0.3
0.2
0.6
0.2
0.1
0.1
0.4
EM
0.0
0.1
0
0
oracle
EM oracle
oracle
Figure We report the data distribution on NQ dev set, TriviaQA dev set and HotpotQA dev set comparing the end task performance when prepending the oracle compression method (oracle sentence or GPT-3 summaries) and when not prepending anything for the base model (Flan-UL2).
A
APPENDIX
A.1 COMPRESSOR TRAINING DATA GENERATION
We report the statistics of the data used to train compressors in Table We use SpaCy (Honnibal al. 2020 to extract named entities.
Extractive Data Generation We generate data using the training data for the four datasets we tested (Wikitext, NQ. TQA and HotpotQA). We use the NLTK package to perform sentence splitting. We remove examples without any negatives.
Abstractive Data Generation We report prompt used generate summaries in Table 14 We queried the Open AI API with temperature of 0.7 and top p 1. For the language modeling task, we use ensemble of four prompts and choose the one which leads to the lowest perplexity as the target. If none of the summaries lead to perplexity decrease, we treat an empty summary target. We queried the OpenAl API with temperature of 0.7 and top p = 1. We generate four summaries per example for randomly sampled 2% of the training data (48,013 examples).
A.2 COMPRESSOR TRAINING DETAILS
Extractive Compressor For language modeling, use the contriever checkpoint trained with unsupervised data. For the QA tasks, we use the contriever checkpoint fine-tuned on the MSMARCO task (Campos al. 2016 To following prior work (Si et al. 2022 Shi et al. 2023b). We implement the model using the Transformers (Wolf et al. 2019 and the sentence-transformae transformer library (Reimers) Gurevych] 2019). We train with Adam optimizer (Kingma & Ba 2014). using a batch size of 64, learning rate of 2e-5 and 1000 warmup steps for epochs. We report results on the model with the best reranked perplexity on our validation set for the language modeling task and the best reranked accuracy for the QA tasks.
15

---

Published as a conference paper at ICLR 2024
Table 5: Training data statistics for abstractive and extractive compressors. "% filtered" refers to the percentage of examples not included. For extractive compressor, we filter examples for which prepending sentences with top contriever score leads to the correct answer. For abstractive compressor; we filter examples for which prepending GPT-3.5-turte generated summary and retrieval both produce incorrect answers. [W] refers to the average number negative examples for constrastive training.
Extractive
Abstractive
Dataset
Train
Validation
% filtered
Train
Validation
% filtered
% empty
NQ 42,149
9,769
46
4.44
39,466
4,931
50
TQA
70,032
8,753
56
4.37
48,322
5,887
32
16
HotpotQA Wikitext
24,526
3,068
69
4.33
26,556
2,937
42
1,398,318
1,5483
41
4.04
38,410
9,603
0
24
Table 6: Inference speed of FLAN-UL2 NQ dev set.
Setting
toks
Inference time
Compression time
Total time
example /
No does
4,104s
Os
4,104s
0.88
Top docs
660
10,584s
Os
1,0584s
0.34
Contriever (ours)
37
4,880s
113s
4,993s
0.72
T5 (ours)
36
4,936s
871s
5,807s
0.62
Abstractive Compressor We implement the model using the Transformers (Wolf al. 2019). We train abstractive summarizer with Adam optimizer (Kingma & Ba (014). using a batch size of 16, learning rate of le-5 and 1000 warmup steps for 3 epochs.
A.3 INFERENCE SPEED
We report the number of tokens measure for efficiency in the main paper. Here, report inference speed measured by GPU time as an additional metric for efficiency. Specifically, we report the GPU time for NQ dev set with FLAN-UL2 the base model. We FLAN-UL2 4 A40 GPUs. For compression, we run contriver and T5 on a single A40 GPU 6). Our methods, even when including the time taken for compressor inference, improve throughput compared prepending the full sets documents, with contriever being more efficient, enabling 2x throughput.
We note that unlike the number of tokens, inference speed is contigent implementation and size of the base model. For instance, bigger models will suffer from more latency with more input tokens and thus RECOMP can bring more speed-up.
Table 7: Compressing retrieved documents for contriever and DPR NQ.
DPR
Contriever tokens
Doc
Compression
tokens
EM
EM
No
21.99
0
21.99
Top
133
36.59
130
27.53
Top 5
667
42.44
652
33.19
Top 5
Contriever
26
25.60
31
23.55
Top 5
Extractive (ours)
35.76
35
30.66
Top 5
T5
10
26.84
10
24.32
Top 5
Abstractive (ours)
35
37.40
28
30.55
16

---

Published as a conference paper at ICLR 2024
Table Comparing prepending top documents and prepending GPT-3.5-turbe compressed docu- ments.
GPT 3.5 EM
Top EM
% Data
(no compression)
NQ
TQA
HotpotQA
0
53
31
61
10
0
7
1
30
55
25
A.4 COMPRESSING DOCUMENTS RETRIEVED BY DIFFERENT RETRIEVAL MODELS
We experiment with using our compressor trained with retrieval model to compress texts retrieved by other retrievers. We report results on NQ (Table 7 compressing documents retrieved by contriever and DPR with compressors trained with contriever-ms- ma-maooo.... We see that both our extractive and abstractive compressor generalize robustly to documents retrieved by another retrieval system, with the best compressor outperforming prepending top documents with 4x compression.
A.5 MORE ANALYSIS ON ABSTRACTIVE COMPRESSOR FOR THE QA TASKS
A.5.1 IS THE TEACHER MODEL A GOOD SUMMARIZER?
As shown in Table compressing with the teacher model (GPT-3.5-turbo) results in slight per- formance drop compared to prepending all top 5 documents for all three datasets (NQ, HotporQA and TQA). We look closer into where the performance gap comes from by comparing the end task performance (Exact Match) between prepending GPT-3.5-turbe generated summaries and prepending the top 5 documents (Table R). Compressing with GPT-3.5- leads to performance decreases for 10%, 7% and 8% of data on NQ. TQA and HotpotQA. Notably, it also leads to an increase for a small portion of data (7%, 7% and 6%) on the three datasets.
Manual analysis We randomly sample 20 summaries for which compressing with GPT-3.5-turbo leads to different performance than prepending top 5 documents per datasets to conduct manual analysis. We report manual analysis results in Table Ambiguous questions refer the cases where the question itself is ambiguous (Min et al. (2020) and answers from both systems are valid. Semantic equivalence are cases where both answers are semantically similar. As we can see, these two categories constitute a non-trivial portion of the data analyzed. Insufficient evidence refers to the cases where the retrieved documents cannot sufficiently support the gold answer. This can include cases where (1) GPT-3.5-turbe hallucinates the gold answer or (2) when prepending top 5 documents, the model generates the correct answer, yet it cannot be fully supported by the retrieved documents. This also demonstrates headroom in improving the retrieval system, especially for more challenging dataset such HotpotQA. Lost in the middle refers cases where the model is unable extract the correct answer from the top 5 documents while compression helps. Finally, Incorrect summary refers to genuine errors caused by compressing the retrieved documents.
Table Manual analysis of when compressing with GPT-3.5- turbo leads to different performance compared to prepending top documents. We report % examples belonging to each reason.
Reason
NQ
TQA
HotpotQA
Ambiguous question
17
11
5
Semantic equivalence
44
32
15
Insufficient evidence
11
42
50
Lost in the middle
11
10
15
Incorrect summary
17
5
15
A.5.2 DISTILLATION PERFORMANCE ANALYSIS
We examine whether the abstractive compressors successfully distill the teacher model on the three QA datasets. We compare performances of FLAN-UL2 when compressing with our T5 model and when compressing with the oracle GPT-3.5-turbo model (Table 10 We that models face
17

---

Published as a conference paper at ICLR 2024
Table 10: Analysis on distillation performance, grouped by whether the generated summary is an empty summary. The left two column denotes whether the summaries generated by T5 or oracle GPT-3.5-turbo is empty (E) or not (N). We report the EM for the T5 system and the oracle system (EM(O)) and the percentage of data belong this slice.
NQ
TQA
HotpotQA EM(O)
T5
O
Comment
EM
(M)
EM
EM(()
%
EM
N
0.47
0.49
0.68
0.76
70
0.30
Both generate a sum- mary.
N E
E
0.31
1.0
0.26
1.0
6
0.12
1.0
Incorrect selective
N
0.11
0.25
32
0.32
0.41
21
0.17
0.24
11
augmentation.
E
E
1.0
1.0
1.0
1.0
3
1.0
1.0
1.0
TS correctly generates an emply summary.
challenges in performing selective distillation for all three datasets. Our systems fail at performing selective augmentation for 36%, 27% and 14% of the data on NQ, TQA and HotporQA (it either attempts to generate summary when the teacher model fails to do so, or outputs an empty summary when generating summary will lead to better performance). When both systems generate a summary, our distillation is relatively successful on NQ (4% performance drop) but less effective on TQA (1 (6) and HotpotQA (12%) performance drop.
Selective augmentation We further calculate performance selective augmentation against the oracle method, i.e. treat selective augmentation as binary classification task and evaluate whether our compressor correctly chooses to output an empty summary (or not). The F1 performance 0.20 and 0.10 for NQ, TQA and HotpotQA respectively. The performance of random baseline, which randomly outputs an empty summary at the same frequency of the oracle method, is 0.10, 0.08, and 0.00 for NQ. TQA and HotpotQA. This shows that although our compressors learn to perform selective augmentation to some extent, there is significant headroom to improve selective augmentation performance.
19,
A.5.3 TRANSFERRING LEARNED COMPRESSORS TO ANOTHER MODEL FOR QA TASKS
We report results of transferring compressors to LLaMA-133 Touvron al. 2023 model in Table TI We use exactly the same input (e.g. same in-context example and prepended documents) as the FLAN-UL2 model. We report oracle for the FLAN-UL2 model (Oracle (w/ FLAN)) as well as oracle for LLaMA-2. While we see successful transfer for the language modelling task from GPT2 to GPT2-XL and GPT-J, transferring compressors trained for QA tasks to another model yields rather negative results. First, we see that oracle compression for FLAN-UL2 model significantly lags behind oracle method for LLaMA-2 for both compression methods and all three datasets. As oracle method depends parametric knowledge of the model (i.e. the model relies on parametric when an empty summary prepended), one potential reason for transfer failure is that compression model trained with one model might not transfer another model with parametric knowledge differences. We also note that for summaries generated by GPT-3.5 on HotpotQA, which work almost as well prepending top 5 documents for FLAN-UL.2 lags behind prepending top 5 documents when prepended to LLaMA. This shows that summaries which can be effectively consumed by one model how build compression might be able to transfer another model. Future work should look into
model that work across different models, especially for downstream tasks such as question answering.
18

---

Published as a conference paper at ICLR 2024
Table 11: Open-domain QA results on LLaMA-13B. We report the results of oracle compressions with Flan-UL2, the base model for the compressors, (Orucle w/ FLAN) and the oracle compression results for LLaMA-13B.
NQ
TQA EM
HotpotQA
In-context evidence
tok
EM
tok
tok
EM
F1
30.89
40.73
0
65.00
71.18
24.20
34.50
RALM without comprexsion
Top document
132
33.35
43.13
136
66.62
73.10
138
34.40
44.17
Top 5 documents
660
37.04
47.60
667
70.61
77.51
684
37.00
47.11
Phrase / token level compression
Top 5 documents (BoW)
450
33.05
43.36
259
66.59
73.40
255
30.00
39.13
Top 5 documents (NE)
338
34.60
44,91
128
65.88
72.59
157
29.20
37.93
Extractive compression of top 5 documents
Oracle
31
56.62
68.89
31
84.61
80.46
69
42.20
51.34
Oracle (w/ FLAN)
34
40.89
50.06
32
68.52
74.96
70
35.20
45.13
Random
32
30.33
39.85
31
62.80
69.25
61
27.40
36.27
Contriever
36
32.52
42.01
40
65.88
72.44
34.60
43.99
Ours (init. w/ Contriever)
37
34.38
44.15
38
65.28
71.85
75
33.20
42.88
Abstractive compression of top documents
Oracle
50
45.60
84.87
38
74.37
79.83
98
41,40
51.54
Oracle (w/ FLAN)
51
38.98
49.40
37
69.86
76.46
102
35.40
46.17
GPT-3.5
56
35.71
46.05
41
67.90
74.88
107
34.60
45.40
T5
10
33.38
43.54
63.18
70.92
30.40
40.60
Ours (init, w/ T5)
36
36.32
46.10
32
66.27
73.12
30.80
40.61
Input: Base L. M, Compressor encoder enco, Training data (x1, yt. where x; is input, S {sy} of candidate sentences from the retrieved document for x1, Y; is the target Output: An updated extractive compressor encoder ence
I: +
for i .,, T} do 3: pi < argMax ( Scor,aa
4:
for mrij (1, (1....,71 n do
5:
L- 0
6:
if (SScore (s)i < Score(M, Vi, (p): then
LUsj
[
then
9:
N + arg ((eeToppsee encox)))
T + T U ((x, Nt)}
ency Finetune(ense T)
Figure 6: Learning extractive compressor for QA task. The Score here is the exact match between the decoded answer and the gold answers.
19

---

Published as a conference paper at ICLR 2024
Input: Teacher LM M, Base LM M, Summarization prompt p. Compressor encdeco, Training data
(xi, Dr, yiJi where xı is input, D; the set retrieved document for Xi, Yi is target answer.
Output: An updated encdeca
1: T +- 0
2: for [1,... } do
3:
Decode(M,, xp; Di))
Vs Score (M. Vi, xi])
Vd Score M, Vi. xil)
6: if U2 < Ua then
7: T TU {(x,, Dj, 0)}
8:
continue
T U (x,
10: encdeca = Finetune(encdecr 7)
Figure 7: Learning abstractive compressor for QA task. The Score here is the exact match between the decoded answer and the gold answers.
Wikitext
NQ
Oracle
6000
1000
Ours
4000
500
2000
500
100
Hotpot QA
TQA
200
3000
2000
100
1000
0
200
400
100
200
Figure Histogram of abstractive summary length tokens) distribution for testing data of NQ,TQA HotpotQA and Wikitext.
20

---

Published as a conference paper at ICLR 2024
Table 12: Example abstractive and extractive compression on wikitext-103 103 dev set and NQ.
Wikitext- 103 Inpat present in most the Mediterranean Sea, only missing from the section east
Original Top 1 document
Sea of Crete" its Southem extremity. The sea stretches to the North of the island of Crete, East of the islands of Kythera and Antikythera, South of the Cyclades, and West of the Dodecanese islands of Rhodes, Karpathos and Kassos, The bounding sea to the West is the lonian Sea. To the Northwest the Myrtoan Sea, subdivision of the Mediterranean Sea that lies between the Cyclades and Peloponnese. To the East SE. the rest of the Mediteranean Sea,
Crete The Sea of Crete (. ""Kritkko Pelagos"") or Cretan Sea, a sea, part of Aegean Sea, located
Crete, and along
only the north @-@
west coast of the
Black Sea
Method
Compressed document
Bow
present Mediterranean Sea missing section cast Crete along north west coust Black The Kritiko Pelagos Cretan sea part Negean located Souther extremity stretches North island East islands Kythera Aniikythera South Cyclades West Dodecamese
Rhodes Karpathos Kassos bounding Ionian To Northest Myrtoan subdivision lies Peloponnese SE rest
NE
Kythera Acgean Sea the lonian Sea Crete Southern South Rhodes the Myrtoan Sea Cretan Sea Autikythera Dodocanese
Kassos Karpathos West the Black Sea & Sea of Crete Polopomaese the Mediterrncun Sea Cyclades
Extractive compres
To the Northwess is the Myrtoan Sea, a subdivision of the Medilerranean Seu that lies between the Cycludes and Peloponnese.
sinn
NQ Input
Original Top 5 document
who got the first bel prize in physies
receive diploma. medal and document confirming the prize amount. Nobel Prize in Physics The Nobel Prize in Physics () yearly award given by the Royal Swedish Academy of Sciences for those who have made the most ouistanding contributions for mankind in the field of physics. is one of the five Nobel Prizes established by the will of Alfred Nobel 1895 and awarded since 1901; the others heing the Nobel Prize Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize Physiology Medicine. The first Nobel Prize Physies was
science, Ernest Lawrence won the Nobel Prize in Physies in 1939. Lars Onsager won the 1968 Nobel Prize in Chemistry. Nomnan Borlaug, father the Green Revolution, wan the Nobel Peace Prize 1970. Christian B. Anfinsen won the Nobel Prize for chemistry in 1972. Ivar Giaever physics. In engineering. Clay lon Jacobson lI is credited with the invention of the modern personal watercraf). Ole Singstad a pioneer underwater tunnels. Ole Evinnule invented the first outhoard motor with practical commercial application, recognizable today Nobel Prize in Physics The Nobel Prize in Physics () is a yearly award given by the Royal Swedish Academy Sciences for these who made the most outstanding contributions for mankind the field of physics. one of the live Nobel Prizes established by the will of Alfred Nobel in 1895 and awarted since 1901; the others being the Nobel Prize in Chemistry. Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize Physiology Medicine. The lirst Nobel Prize in Physics awarded physicist Wilhelm R0Ulsntgen recognition of the extraordinary services be was also awarded the Abel prize. In addition, eight fiomalleessuve gone on to receive the Nobel Prize in Physies: Claude Cohen- Tannoudji, Pierre-Gilles de Gennes, Alhert Fort, Alfred Kastler, Gabriel Lippmann, Louis N 00:9cl, Jean Baptiste Perrin and Serge Haroche, while other ENS physicists include such major figures as Paul Langvvin, famous developing Langevin dynamics the Langovin equation. Alumaus Paul Sabatier won the Nobel Prize Chemistry. A ranking of universities worldwide bused on ratios of alunni to Nobel prize winners published in 2016 by American scholars Stephen
the Nobel Prize Physics 1973, Carl Richard Hagen noted for his work
Jonathan Wai placed ENS rendered by the discovery of the remarkable rays (or r-rs)) This award administered widely regarded as the most prestigions award that scientist can receive in physics. It is presented in Stockholl at an annual ceremony on 10 December, the anniversary Nobel's death. Through 2018, total of 209 individuals have been awarded the prize. Only three women (1.4% of laureates) have won the Nobel Prize in Physics: Maric Curic in 1903, Maria Goeppert Mayer in 1963, Donna Strickland in 2018. Alfred Nobel. in his will and testament, stated his
first university worldwide, far
the Nobel Foundation and
Method
Compressed document
T5
Wilhelm Röntgen received the first Nobel Prize in Physics in recognition of his extraordinary services. It is one of the five Nobel Prizes established by Alfred Nobel 1895 and awarded since 1901.
GPT:3.5-tubo
The linst Nobel Prize in Physics was awarded to physicist Wilhelm Röntgen in 1901 for his discovery of the remarkable rays (or x-rays). Since then, 209 individuals have been awarded the prize, with only three women (1 11.4% of lnureates) having won
Table 13: Example input to the Flan Flan-UL2 for NQ with in-context examples and retrieved documents.
Dataset
Prompts
NQ
who won a million on deal or no deal Answer: Tomorrow Rodriguez who is the woman washing the car in cool hand luke Answer: Joy Harmon
who is the actor that plays ragnar on vikings Answer: Travis Fimmel who said 'ss better to have loved and lost Answer: Alfred, Lord Tennyson name the first indian woman to be crowned as miss world Answer: Reita Faria Ret Retvid Docs Question
Answer:
21

---

Published as a conference paper at ICLR 2024
Table 14: Prompts used generated summaries from GPT-3.5-turbo. query and docs represent the actual input query and retrieved documents.
Dataset
Prompts
NQ
Compress the information in the retrieved documents into 2-sentence summary that could be used to answer the question: Question: que ry Retrieved documents: docs Compressed documents:
TQA
Compress the information in the retrieved documents into 2-sentence summary that could be used to answer the question: Question: query Retrieved documents: docs Compressed documents:
HotpotQA
Source documents: docs Question: query Generate reasoning chain to answer the question:
Wikitext
sentences of the given query using the information from the provided documents. \nSSurce Documents: docs \nQuery: query \n
Generate the
Wikitext
Select sentences from the retrieved docs that are most likely be the next sen- tence. |noorce Documents: docs \nQuery: query',n
Wikitext
Generate the next one sentence of the given query using the information from the provided documents's \nSouree Documents: does !nQuery: query \n Summarize the information from the provided documents's (nneeee nnuery: query\n
Wikitext
(nSurrce Documents: docs
22

---

Published as a conference paper at ICLR 2024
Table 15: Case study of how compressing the retrieved documents helps the model to identify the right answer from NQ dev set.
Question: host the late show who was once a correspondent for the daily show. Gold answer: Stephen Colbert Type
In-context documents
Predicted An-
swers
None
Chelsea Han-
dler
Top 5
by Conan OBrien, in 2009. Leno explained that he did not want to soe a repeat of the hard leelings and controversy that occurred when he given the show over David Letterman following Carsoná retirement in 1992. Offriens last "Late Night" episode was taped on February 20, 2009. Former Saturday Night Live alum Jimmy Fallon took
Samantha Boc
host of "Late Night with Jinany Fallon" on March 2, 2009. The Colbert Repurt that aired four days week on Comedy Central from Octoher 17, 2005, was hosted hy Stephen Colhert, one regulars Comedy Centrals The Daily senson as host began with notable interview with former British prime minister Tony Blair. The live interviow occurred the night before book signing Eason's 's which altracted international atlention when Blair was pelied with shoes and eggs and succesefully evaded an attempted citizen's is arrest on charges of war crimes. On I February 2013, Pat Kenny returned bost that night's edition when Tubridy's 's father died. In 2015, Tubridy's 's tone and choice of questions when interviewing Anti-Auster Austerity Alliance TD Paul Murphy relation to the campaign against the implementation much criticised. Opponents of the 'Michigan. interviewing Eminem. Colbert has been given near-full control of the show, with little interference from CBS management in regard to format. Colbert brought most of his staff from "The Colbert Report" with him "The Late Show", as well as outsiders such as Brian Stack, who best known for his work on Conan OBriens programs, and Jon Stewart, former host Colberté previous sister program "The Daily Show", who is credited as executive producer. Calhert no longer uses the character he had portrayed on "The Colbert Report" jokingly remarking to Jeb Bush that "I used to play a Show" has had three regular bosts: Gay Byrne, Pat Kenny and Ryan Tubridy. Frank Hall deputised for Byrme for one semson in the 1961.. There have been at least four occasions on which another presenter has bosted the show. The first when Byme hecame unexpectedly and seriously ill. Frequent panelist Ted Bonner presented instead. The second time was towards the end show about feminism, when Byme ushered a young Marian Finucanc into his seat to present the remainder of the show. On another occasion, radio broadcaster and former news reader Andy OMahony replaced Byme for an interview popular acclaim. Colbert would host the program until he was chosen replace David Letterman bost of CBS$ "Late Show" in 2015. Edd Helms, a former correspondent from 2002 to 2006, also starred NBC "The Office" and was main character in the 2009 hit "The Hangover". After filling in as host during Stewarts two-month absence in the summer of 2013, John Oliver went host his own show on HBO, "Last Week Tonight former correspondent Samanthu Bee launched her own late night talk show "Full Frontal with Samantha Boe". Boed busband Jason
water was
John Oliver". 2016.
GPT-3.5-turbe Former Daily Show correspondent Stephen Colbert was chosen to replace David Letterman host CBS's "Late Show" 2015, while Ed Helms, former correspondent from 2002 to 2006, also stareed on NBC's "The Office" and John Oliver. who lilled in as host during Jon Stewart's abvence in 2013, went on bost his own show HBO.
T5 (ours)
Stephen Colbert was a former correponnent for The Daily Show and later became bost of CBS's "Late Show" in 2015. He has since brought most his staff from "The Colbert Report" with him to "The Late Show", with little interference from CBS management in regard
Stephen bert
Col-
23

---

Published as a conference paper at ICLR 2024
Table 16: Example summaries and their manual analysis labels. See Table 17 for more example.
Dataset
Model
Query, Passages and Summary
Evaluation
NQ
Ours
Question: when will miraculous ladybug scason episode 12 come out
Not faithful. Comprehen-
Passages: 2016 on TVNZ's TV2. In Japan. Disney Channel streamed the episode "Storny Weather" through its mobile application
July 2018. hefore the official premiere 23 July in the same year. The second scason premiere is scheduled for global launch around Segtember-November 2017 in Europe. a pane) announced that, the second season would have its North American release on Netllix in December 2017. with episndes he releaved. KidsClick show in the US starting 30 August 2018, marking the first time that Korea on I Septemher 2015 on EBSI. In the United States, the series debuted on Nickelodeon on Devember. In the United Kingdom and Ireland, the show premiered on 30 January 2016 on Disney Channel. A Christmas special was released in 2016 and the sccond scason premiered in French on TFI in English on Disney Channel UK 2017. Netflix acquired U.S. video-on- demand streaming rights and further seusons are in production. Set in modern day Paris, the series focusses teenagers Marinette Dupain-
sive
Diego Comic-Con 2017. it was
start airing season of this
and Adrien Agreste. When evil arises, Marinette transforms into her superhero persona Ladybag, while Adrien transforms rights in Earope, Eastern Eamope and Scandinavia, and free- to air rights in Spain, Germany, Russia and Turkey. South Korca the first country premiere "Ladybug". with girl group Fiestar to sing its translated theme song. It aired on Septebber 2015 on EBSI, and ran for 13 episodes until November 2015,
repeats through February 2016, and second hall of the season airing from March 2016. SK Breadband, having participated in the production, provided the episodes video on demand exclusively subscribers their IPTV platform B TV, about a half-hour following the South Korcan breadcast of Scason of this show airing on American over the--th-aiir television A third senson is also in development. On 22, January fourth and fift
2018, Zag posted Instagram that the was working August premiered a new series with Miraculous: Tales of Ladybug & Cat Noir which this is in chibi style. film and live action television rights. Kimberly Cooper, blog writer who has contributed to news media such as "The Huffington Post", wrote that the show has inspired teens and adults to create Channel 1 November 2016, In the United Kingdom and Iseland, the show premiered on 30 January 2016 on Disney Channel The first series was also made available to UK. Among free-to-air terrestrial television brondcasters Republic of Ireland, Raidió Teiliii Éireann premiered the show in 2018 on KTE2'' ss teenage Australion Broadcasting Corporation released
On 31
September 2018, Skydance Media has acquired the live action
Netfli subseribers
programming block. TRTÉ. In Australaa, the show on its ABC iview streaming service on January 2016, and premiered the show on lincar channel ABC3 on 22 March 2016. New Zealand, the show premicred on April Summary: Miraculous Ladybug season 2 episode 12 scheduled for release in December 2017 on Netflix in North America. The show premiered in South Korea on September 2015 and has since been aired various countries. including in the United States, the UK and New Zealand.
TQA
Ours
Question: In which novel does Flora Poste move to live with her relatives, the Starkuäders, Sussex?
Faithful, Not comprehensive
Passages: to be writer, decides that the only way for her to live whilst rescarching her writing is stay with relatives. city-based relatives show interest, so she sends letters to her country relatives. There are a few responses, most of them unsuitable, but one is intriguing. Flora decides to stay for while with the Starkadder family on their rundown farm. The Starkadders are an assortment of rustic, uncouth, and truly eccentric characters, each of whom hus hurdle (be it physical, emotional, or spiritual) to overcome befirre reaching his her potential. Flora quickly realises that, relatives the isolated Cold Comfort Farm in the fictional village of Howling in Sussex. The inhabitants of the farm - Aunt Ada Doom, the Starkadders, and their extended family and werkers- foel obliged atone for an unspecilied wrong once done to her father. As is typical in certain genre of romantic 19th-century and early 20th-century literature, each of the farm's 's inhobitants has long festering emotional problem caused by ignorance, hatred, fear, and the farm is hadly run. Flom, being a level headed, urban woman in the dandy tradition, determines that she must Have Always Been Starkadders Cold Comfort Farm"". based on Conferencce at Cold Comfort Farm when Flora is married with several children. was broadcast.
take her to
several years later and
television film was produced which was generally well-received. with critics. Janet Maslin in the '"New York Times" wrote that this screen version "" "gets it exactly right"". The film starred Kate Beckinsale as Flora, Jounna Lumley as her friend and mentor Mary Smiling, Rufus Sewell Seth, lan Mc Kellen Amos Starkadder, Eileen Arkins as Judith, Stephen Fry as Myhug, Miriam Margolyes as Mrs. Beetle, and Angela Thorne dies suddenly
heart attack and Lady Place rented out, with the view that Titus, once grown up. will return to the home and run the business. After twenty years of being
aunt Laura linds hersell feeling increasingly stifled both by obligations to the family and by living in I London. When shopping for flowers on the Miszow Road, I Laura decides she wishes to move to the Chiltern Hills and, buying a guide book and map to the she picks the village Great Mop as her home. Against the wishes of her extended respects to her brother. Agnis is moving to the ancestral family home in Newfoundland, which has been abandoned for 44 years. Realizing that Quoyle she lirst offers to stay a few more days and help him through the crisis, and then persuades him move with While struggling rebuild his life, lix up the derefict house, and care for his daughter, Qunyle meets local resident Wavey Prowse, widow who has pre-teen boy with a learning disabilit. Wavey's son and Quoyle's diughter become friends, while the two adults hecome friends.
total loss through grief,
Summary: Flora Poste moves to live with her relatives, the Starkndders, on their rundown in Sussex. Starkadders are eccentric characters with various obstacles to overcome.
24

---

Published as a conference paper at ICLR 2024
Table 17: Example summaries and their manual analysis labels (continued).
Dataset
Model
Query, Passages and Summary
Evaluation
HolpoiQA
GPT:3,5
Question:
Thee compuser of the music for the ballet "The Seasons" was the director of what 1928?
Not Faithful, Comprehen.
organization
Passages: The Seasons (ballet) The Scasons (, """rremena goda""; also is allegorical ballet in one act, four scenes, by the choreographer Marius Petipa, with music by Alexander Glazunov, his Op. 67. The work composed in 1899 and first performed by the Imperial Ballet in 1900 in St. Petersburg, Russia. The score for Marius Potipa's is '"Los Saisons" originally intended have been composed by the
sive
(""TTh Seasons"") and conducter Riccardo Drigo, who was Glazumewss 's collengue and close friend. Since posts harmonium, guitar and even mandolin). ""The Scasons"" was commenced shortly after the premiere of Tchaikovsky's First Piano Concerio, and continued while he was completing his first baller, ""Swan Lake In 1875, Nikolay Matveyevich Bernard, the editor the St. Petersburg music magazine ""ouvellist"" commissioned Tchaikovsky to write 12 short piano pieces, one for each momth of the year. Bernard suggested subtitle each month's piece. Tchaikoesky accepted the commission and all of Bernard's subtitles, and the December 1875 edition the magazine, readers were promised new Tchaikovsky piece each month throughout 1876. The January and February pieces The Scasons (Cage) The Sensons is a hallet with music by John Cage and choreography hy Merce Cunningham, ' first pieco for orchestra and also first Cage later called the "gammut technique, albeit in an early form. Cage composed the music early
composer
1886, Drigo
music ""chef d'orchestre""
first performed 1947.
what
Interludes" A piano version was first completed, and an orchestral amangement followed. Cage dedicated ""The Seasons" "" to Lincoln Kirstein. The ballet was premicred on May 17, 1947 by the Ballet Society (by which orchestra version. Morton Gould retained the piano
midst of working ""oonatas
the work was commissioned) part for many of the pieces and orchestrated the work throughout, recording it with himself the piano in 1951 for American Columbia. In 1965. Kurt-Heinz Stolze orchestrated number of the pieces as part of the music for John Cranko's bullet ""Onegin"" More recent urchestral versions have been produced by David Matthews (for symphony orchestra), Peter Breiner (for solo violin and symphony orchestra), and Georgii Cherkin (for solo piano and symphony orchestra). French composer Philippe Sarde arranged the Barcarolle as a main theme for the 1988 movie Th"hhe Bear"", Slava Cranko was not invited direct. and Sir John Gielgud was brought in. Prosocuted for homoscxual activity. Cranko left the UK for Sturigart, and 1961 was appointed director the Stuttgart Ballet, where assembled a group of talented performers such as Marcia Haydée, Egon Madsen, Richard Cragun, Birgit Keil Suzanne Hanke. Among following choreographies were ""Romen and Juliet" by William Shakespeare in 1962, set to music by Prokofiev, ""Onegin"" in 1965, an adaptation the verse novel ""Eagene Onegin by Alexander Pushkin, set to music by Tehaikovsky (mainly ""The Seasons ") orchestrated by Kurt Heinz Stolze, ""The Taming
Summary: 1. The composer of the music for the ballet ""The Sensons is Alexander Glazunov.
2. Alexander Glazumav compused the music for the hallet "'The Seasons"" in 1899. ballet "'The Seasons" tm was first performed
Imperial Rallet 1900.
4. From 1905 to 1928, the director of organization was Marius Petipa.
5. Marius Petipa was the choroographer of the ballet "'The Seasons"
6. Therefore, the compuser of music for the ballet ""Thee Seasons" the director of
the Imperial Ballet from 1905 to 1928.
25


Reviews:
[{"review_id": "OXc1E3vJBy", "reviewer": "Reviewer_pQZH", "summary": "In this paper, the authors proposed a method named RECOMP (Retrieve, Compress, Prepend) to compress the retrieved documents used in Retrieval-Augmented language models (RALMs). The proposed method aims to generate concise, effective and faithful summaries to improve the efficiency of RALMs. Two kinds of summarizers are used: an extractive summarization model for selecting informative sentences, and an abstractive summarization model for generating good query-focused summaries. Experiments are conducted on language modeling and question answering tasks, achieving low compression rates and some performance drops.", "questions": "- Q1: As stated in \"Weakness\", the inference speedup should be reported.\n- Q2: How about the training cost? The training of the abstractive compressor uses a teacher LM (GPT-3.5), which is expensive. I'm not sure if the cost saved in the inference stage is worth the cost in the training stage.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The presentation of the core idea is clear. The research topic of effective RAG is important.\n- The proposed method freezes the LMs and only trains the summarization model, which is interesting and beneficial to transfer across different LMs.\n- The analysis in Sec. 6 is sufficient and comprehensive.", "weaknesses": "My major concern is that the paper does not report the inference speedup by their method. Although experimental results show that fewer retrieved tokens are used, the number of tokens does not necessarily reflect the actual efficiency of the system, especially when the context length is not the bottleneck. The tokens of all the retrieved documents are still required to be processed by the compressor (a smaller model though). Therefore, it may be beneficial for authors to report the inference speedup of their method. This could be the core to support their motivations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1HA8hJsaB6", "reviewer": "Reviewer_VoPU", "summary": "The paper proposes RECOMP, an approach to improve retrieval-augmented language models (RALMs) by compressing retrieved documents into summaries before using them as context. The method uses two compression methods: an extractive model selecting relevant sentences, and an abstractive model generating summaries. The compressors are trained to optimize end-task performance when prepending the summary to the input. RECOMP is evaluated on language modeling and open-domain QA tasks.", "questions": "Can you give some results on how the method is compared to traditional RAG with no compression in terms of time? \nCan you provide more details on the input tokens for the abstractive model or is the same number of tokens as the noncompression baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a simple but clever idea and backs it up with a set of well-designed experiments. The baselines are well chosen.", "weaknesses": "Although the method is good, I’m not sure how useful it is in practice. Normal retrieval augmentation without any compression outperformed the compression methods albeit with many more tokens. However, with the growing context lengths of models, it is not clear if the accuracy hit is worth the tokens saved. Secondly, you still have to provide the full tokens to the abstractive model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VCRiyaHGmC", "reviewer": "Reviewer_hWi7", "summary": "In this paper, the authors propose RECOMP (Retrieve, Compress, Prepend) that incorporates extractive and abstractive summarization into the retrieve augmented language model to shorten the token length of retrieved text in prompts. Precisely, these summarization modules receive input text and its retrieved text and then extract or generate the summary for the retrieved text. The authors did not simply join the summarization modules into the language model as separate modules. Instead, they updated them through training to generate summaries relevant to the input text and its retrieved text. The experimental results show that RECOMP can shorten the prompt length while keeping performance in language modeling and open-domain QA tasks.", "questions": "- In the current evaluation, the benefit of using summarization for the retrieve augmented language model needs to be clarified. If the inference speed is improved, the authors should report it.\n- Also, you need to report how much computational cost is decreased by RECOMP. This is related to the first question.\n- Considering the length limitation of the model, we can expect summarization to make the model consider more retrieved texts in its prompt. It may contribute to improving language modeling performance. Did you check such direction in your work?\n\nI will update my score based on the discussion with the authors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors show the effectiveness of both extractive and abstractive summarization. It means we can use insights from commonly used summarization methods for improving retrieve augmented language models.\n- Since the summarization models are updated to fulfill the requirement of retrieving augmented language models, the proposed method is more than model combination and thus novel.\n- The experimental result shows the proposed method RECOMP can save the prompt length while keeping its performance.", "weaknesses": "- The benefit of shortening prompt length is uncertain. If the authors observe a speedup in inference,  they should report it.\n- The performance improvement is limited or nothing in many settings. The authors need to justify it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ASP8rOd5Wr", "reviewer": "Reviewer_WhGn", "summary": "To enhance the efficiency and performance of retrieval-augmented language models, this paper introduces a simple and intuitive method for in-context improvement. Given the retrieved documents and input text, the proposed method first compresses the retrieved documents into a summary. Then, this summary is prepended to the input text, with the subsequent output response generated using a frozen language model. Two types of compressors are proposed: an extractive compressor, corresponding to extractive summarization, and an abstractive compressor, associated with abstractive summarization. Experimental results from both language modeling and question answering tasks demonstrate that the proposed compressors bolster the performance of retrieval-augmented language models.", "questions": "Q1: During the manual evaluation, the authors assess the outputs of the abstractive compressors. What was the level of annotation agreement among the evaluators?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and straightforward. Moreover, it offers an extensive, and well-designed experimental validation of the proposed method and presents thorough analyses.", "weaknesses": "Some concerns regarding this paper include:\n\n- The proposed method utilizes the knowledge of the LMs to train the compressor, while the baselines do not. Some baseline models, such as DRP and contriver, should be finetuned using the dataset employed to train the extractive compressor.\n\n- The paper employs a BM25 retriever, making the results of the proposed method and baselines reliant on the retriever's performance. Therefore, it would be appropriate for the paper to also show performances using an another retriever, such as contriver (not merely as a sentence selector). In addition, presenting the distribution of oracle-retrieved documents, similar to what's done in extractive compression, would be advantageous.\n\n- Relatedly, it is necessary to compare the time taken to generate text by prepending all retrieved documents without compression against the time taken using the proposed method—specifically, compressing the retrieved documents before prepending them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)