Retrieval meets Long Context Large Language Models
Authors: Peng Xu
Wei Ping
Xianchao Wu
Lawrence McAfee
Chen Zhu
Zihan Liu
Sandeep Subramanian
Evelina Bakhturina
Mohammad Shoeybi
Bryan Catanzaro
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Long Context Window
Retrieval
Primary area: generative models

Abstract: Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation versus long context window, which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two state-of-the-art pretrained LLMs, i.e., a proprietary 43B GPT and Llama2-70B. Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation. More importantly, we demonstrate that retrieval can significantly improve the performance of LLMs regardless of their extended context window sizes. Our best model, retrieval-augmented Llama2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on nine long context tasks including question answering, query-based summarization, and in-context few-shot learning tasks. It also outperforms its non-retrieval Llama2-70B-32k baseline by a margin, while being much faster at generation. Our study provides general insights on the choice of retrieval-augmentation versus long context extension of LLM for practitioners.

Published as a conference paper at ICLR 2024
RETRIEVAL MEETS LONG CONTEXT LARGE LANGUAGE MODELS
Peng Xul, Wei Ping', Xianchao Wu, Lawrence McAfee Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina Mohammad Shoeybi, Bryan Catanzaro
NVIDIA
(pengx, wping) (evvviiaa com
ABSTRACT
Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMS with retrieval has existed for years. The natural questions are: which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two sate-oofttheaar pretrained LLMs, i.e., proprietary 43B GPT and Llama2-70B: Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-auumentaaiie rable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation. More importantly, we demonstrate that retrieval can significantly improve the perfor- mance of LLMs regardless of their extended context window sizes. Our best model, aurrened Llama2-70B with 32K context window, outperforms GPT-3.5- turbo-16k and Davinci003 in terms of average score nine long context tasks including question answering, query-based summarization, and in-context few-shot learning tasks. also outperforms its non-retrieval Llama2-70B-32X baseline by a margin, while being much faster at generation. Our study provides general insights on the choice of retrieval-aue auggeettttnn versus long context extension of LLM for practitioners.
versus long context window;
at generation can achieve compa-
INTRODUCTION
The long context large language models (LLM) have recently received a lot of attention in produc- tion (e.g., Anthropic, 2023; OpenAI, 2023b), research community (e.g., Chen et al., 2023; Liu et al., 2023; Tworkowski et al., 2023), and open source community (e.g., Kaiokendev, 2023). Although the approximate attention methods have been studied for years (e.g., Tay al., 2022a) (due the quadratic time and memory complexities of self-attention mechanism in sequence length), the recent advance for long context I.LMs with exact attention is mainly driven by the development of faster GPU with more memory and memory-efficient exact attention (Dao et al., 2022; Dao, 2023).
An alternative and LLMs only read relevant context retrieved from standalone retriever (e.g., Karpukhin al., 2020; Wang et al., 2022; Lin al., 2023), which much easier to scale and runs orders of magnitudes faster than LLMs for selecting relevant context. Conceptually, the retrieval-augie -augeeened decoder-only LLM can be viewed as applying the sparse attention over its long context window, where the sparsity pattern is not predefined Child et al. (2019) but determined by the standalone retriever. In other words, unretrieved context is treated as irrelevant and has zero-valued attention weights.
solution for handling long context is retrieval. Specifically, the
Given the surge of interest in long context LLM research and much more required computation at inference 2, it is still unclear for practitioners whether extending the context window of LLM
'The dense embedding retriever can easily retrieve context from hillions of tokens using the fast similarity search library (Johnson et al., 2019). Forr example, the price of GPT-4 with 32k context length is twice the 8k context model.

---

while it can still be Published as a conference paper at ICLR 2024
provides higher accuracy than the paarieval-augmentttiin for downstream tasks with informative queries. Moreover, it would be compelling we could combine the strength of both methods and achieve even higher accuracies. In this work, attempt to answer the above questions through comprehensive study.
Specifically, we make the following contributions:
1. We perform comprehensive study using two state-of-the-art LLMs, a proprietary 43B pre- trained GPT and Llama2-70B (Touvron al., 2023b) on 9 downstream long context tasks, including single and multi document question answering (QA), query-based summarization, and in context few-shot learning tasks.
2. We demonstrate that retrieval-augmenntiee -ugmentation significantly improves the performance of 4K context I.LMs. Perhaps surprisingly, we find this simple retrievll-ugmmented baseline can perform comparable 16K long context LLMs, i.c.. average score 29.32 vs. 29.45 by using GPT-43B, and 36.02 vs. 36.78 by using Llama2-70B, while using much less computation.
3. Furthermore, we demonstrate that the performance of long context LLM (i.c., 16K or 32K) can still be improved by retrieval, especially for the larger Llama2-70B. As a result, our best model, retrieval augmented Llama2-
with 32K context window (avg. score 43.6), (avg. score 42.8) and Davinci-003 terms of average score.
outperforms EPT-3.5-turbo- It also largely outperforms its non-retrieval 1amma2-70B-325 baseline (avg. score 40.9), while can be much faster at generation (e.g., 4x faster on Narrative(A).
We organize the rest of the paper as follows. We discuss related work in Section 2, and present the experimental setup in Section We report results in Section 4 and conclude the paper in Section 5.
RELATED WORK
In this section, we discuss the related work in long context LLM, efficient attention methods, and retrieval-augmeented language models.
2.1 PARALLEL WORK
When preparing this manuscript, we notice that concurrent work (Bai et al., 2023) (arXived on 28 Aug 2023) also studies the impact of retrieval on long context LLM, including black-box model (Touvron et al., 2023b),
GPT-3.5- GPrro.--Turbo- 16k (OpenAI, 2022), white-box model Llama2-7B-chate and ChatGLMM-6BB333 2B2kkk (Zeng et al., 2022). Different from our findings, they find that retrieval is with 4K context window, but not helpful for long context model,
only helpful for L.lama2-7B-cha-tee i.c., GPT-3.5-Turbo 16k and ChatGLM2-6B-333
We hypothesize the major reasons are: i) it is challenging to do controlled experiments using black-box APIs, the white-box LLMs used in their study are relatively small, thus they have limited zero-shot capability of incorporating context through retrieval. Our conclusions are drawn from much larger LLMs. particular, our best long context model -lama2--70-333 performs well as Davinci003 and further enhanced by retrieval (see Table 3).
2.2 LONG CONTEXT LARGE LANGUAGE MODELS
Over the past few years, pretraining large language models (LLMs) with long context window becomes viable solution thanks to faster GPU with more memory and efficccent exact attention (e.g., Dao al., 2022). For example, the context window for pretrained LLM have been increased from 1024 of GPT-2 (Radford et al., 2019), 2048 of GPT-3 (Brown et al., 2020), 4096 of Ilama 2 (Touvron al., 2023b), to 8192 of GPT-4 (OpenAI, 2023a). However, further extending the context window in pretraining can be challenging, because, pretraining LLM from scratch with long context (c.g., >6K tokens) is very expensive due to the quadratic time and memory complexities of exact attention, and most of documents in pretraining corpus (e.g., Common Crawl) are relatively short.
Most recently, researchers start extend the context window of LLMs with continued training fine-tuning (c.g., Kaiokendev, 2023; Nijkamp et al., 2023; Chen et al., 2023; Tworkowski et al., 2023; Mohtashami & Jaggi, 2023). Tworkowski et al. (2023) introduced Longl. LaMA by fine-tuning the
2

---

Published as a conference paper at ICLR 2024
3B and 7B OpenLLaMA checkpoints with contrastive training 8K context length. Landmark attention (Mohtashami & Jaggi, 2023) extends the context length of LLaMA 7B from 4K to 32K by introducing "landmark tokens" to represent blocks of the context and fine-tuning the attention to use landmark tokens for selecting relevant blocks. Chen et al. (2023) and Kaiokendev (2023) introduced positional interpolation to extend the context window sizes of RoPE-based (Su et al., 2021) pretrained LLMs. In particular, Chen et al. (2023) demonstrates promising results on LLaMA 7B to 65B (Touvron et al., 2023a) with minimal fine-tuning effort (within 1000 steps). ALiBi (Press et al., 2021) extrapolates context window length by removing the positional embeddings while simply biasing the key-query attention scores with linear penalty that proportional their distance, so one does not need finetuning for context window extrapolation. Ratner et al. (2023) chunks long context into multiple sub-windows and re-use the positional embeddings across these windows, thus can handle longer context without any further finetuning. In this work, we apply positional interpolation method to extend the 4K context window of a proprictary 43B pretrained LLM and Llama2-70B (Touvron et al., 2023b) to 16K and 32K, as they both use rotary position embedding at pretraining. In terms of evaluation, we focus on downstream task performance (e.g., Shaham et al., 2023; Bai et al., 2023) after instruction tuning (Wei et al., 2021).
There are other studies showing the interplay between strieval-augmentatte and long context LLM. Liu et al. (2023) performs the black-box evaluation for the long context capability of existing LLM products, including ChatGPT 3.5 (OpenAI, 2022), GPT-4 (OpenAI, 2023a), Claude (Anthropie, 2023), in sirievaal-uumeneted setting, and identify the "lost in the middle" phenomenon in these models.
2.3 EFFICIENT ATTENTION METHODS
In previous study, many approximate attention methods (Tay et al., 2022a) have been introduced for dealing with the quadratic complexity of self-attentio that becomes computational bottleneck for long context. They can be grouped into the following categories: i) Sparse attention mechanisms with predefined sparsity patterns (e.g., Child al., 2019; Parmar et al., 2018; Ho et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020; Zhu et al., 2021), üi) recurrence-based method (Dai et al., 2019; Bulatov et al., 2022), äii) low-rank projection attention (e.g., Wang et al., 2020; Xiong al., 2021; Tay et al., 2021; Zhu et al., 2021), iv) memory-based mechanisms (c.g.. Rae et al., 2020; Liu et al., 2018), v) similarity and clustering based methods (c.g., Kitaev et al., 2020; Tay al., 2020; Roy et al., 2021). These approximate methods introduce inductive bias (e.g.. predefined sparsity) that can fit well for specific domain, but may reduce model quality in general LLM training.
Most recently. FlashAttention (Dao et al., 2022; Dao. 2023) is introduced to speed up the exact atten- tion computation by accounting for reads and writes between levels of GPU memory. FlashAttention is particularly useful for handling longer sequences.
2.4 RETRIEVAL-AUGMENTED LANGUAGE MODELS
Retrieval has been integrated into language models for years to improve perplexity (Borgeaud et al., 2022; Wang al., 2023), factual accuracy (Nakano et al., 2021), downstream task accuracy (Guu et al., 2020; Izacard & Grave, 2021; Izacard et al., 2022; Lewis et al., 2020), and in-context learning capability (Huang et al., 2023). Combined with a standalone retriever (Karpukhin et al., 2020; Wang et al., 2022; Lin et al., 2023), retrieval-augmenner LLM is well established for handling question answering with long document and in open-domain. In previous study, language models have been augmented with retrieval at inference (Khandelwal et al., 2019; Yogatama et al., 2021), fine-tuning (Izacard et al., 2022; Lewis et al., 2020; Guu et al., 2020), and pretraining (Borgeaud al., 2022; Izacard et al., 2022; Wang et al., 2023). There are also methods that try to integrate LLM and retriever in a single model and build the end-to-end solution (c.g.. Jiang et al., 2022; Shi et al., 2023). However, most of previous works mainly study serreval-auumentatioon for LLMS that have around 10 billion parameters, except a few recent ones (e.g.. Shi et al., 2023).
In this work, we focus decoder-ond only LLMs with 43B and 70B parameters trained trillions of tokens, because the LLMs at such scale exhibit strong zhoo capability to incorporate context after instruction tuning (Wei et al., 2021; 2022).
3

---

Published as a conference paper at ICLR 2024
3 EXPERIMENTAL SETUP
In this section, we present the details of our experimental setup.
3.1 LARGE LANGUAGE MODELS
We focus comparing the zero-shot capability of integrating long context information for generative QA or summarization tasks via retrieval or LLM's own self-attention mechanism. In contrast to most existing works that focus on relatively small models (c.g., 3B or 7B) (Kaiokendev, 2023; Nijkamp et al., 2023; Tworkowski et al., 2023; Mohtashami & Jaggi, 2023), we gather the insights by exploring model sizes that are larger than 40B after instruction tuning, as previous study suggests that instruction tuning becomes effective when the decoder-only LLM has around 50B parameters (Wei et al., 2021: 2022).
Specifically, we experimented with two pretrained GPT models, a proprietary Nemo GPT-43B and Llama2-70B. GPT-43B is a 43 billion parameter model that is trained with 1.1T tokens with 70% English corpus and the other 30% for multilingual and code data. For the English pretraining corpus, GPT-43B used Common Crawl web archive (WARC), Wikipedia, Reddit, Books, Gutenberg, ArXiv, StackExchange, PubMed, etc. It contains 48 layers with the hidden dimension of 8,192. It is trained with sequence length of 4,096 and RoPE embeddings (Su et al., 2021). The other Llama2-700 is a public available 70B GPT model trained on 2T tokens using around 90% English data. It contains 80 layers with the hidden dimension of 8, 1919. It also has the context window size of 4,096 and trained with RoPE embeddings.
3.2 DATASETS AND METRICS
In this study, we include seven datasets ranging from single document QA, multi document QA, to query-based summarization for our zero shot evaluations. Specifically, we include four datasets from the validation set of the Scroll benchmark (Shaham et al., 2022).
QMSum (QM) (Zhong et al., 2021) a query-based summarization dataset, consisting of meetings' transcripts and their corresponding summaries from multiple domains such as academic, industrial product. this task, a meeting dialogue transcript is given, and a question summarize certain topic is raised about the dialogue, such "what is agreed between them".
The answer generally contains a few sentences.
Qasper (QASP) (Dasigi et al., 2021) is a question answering dataset over NLP papers filtered from the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020). Qasper contains abstractive, extractive, and yes/no questions, as well as unanswerable ones. In this task, one script is provided together with an information seeking question, such as "which multilingual approaches do they compare with?". A model needs to give a short answer by reasoning over the given context.
NarrativeQA (NQA) (Kočiský et al., 2018) is an established question answering dataset over entire books from Project Gutenberg" and movie scripts from list of websites. this task, the given passage is transcribed from books and is usually noisy. A model is required to generate short phrase by reasoning over the long and noisy text.
QuALITY (QLTY) (Pang et al., 2022) is a question answering dataset over stories and articles collected from several resources, such as Project Gutenberg and the Open American National Corpus". Different from all the other tasks, this is multi-choices dataset and a model is required to select one among four given choices.
We take another three datasets from LongBench (Bai et al., 2023).
HotpotQA (HQA) (Yang et al., 2018) is Wikipedia-basse based question answer dataset. Different from above single hot datasets, HQA is a multi-hop dataset where multiple supporting documents are required to be read for answering and reasoning and the questions are diverse and not constrained to any pre-existing knowledge bases.
hhttps://www, http:////wwwwwgebbbrggggg. www.guutteerorrggg...... hhttss:/anc..r
4

---

Published as a conference paper at ICLR 2024
QM
QASP
NQA
QLTY
MSQ
HQA
MFQA
of samples
200
1,726
2,000
2,000
200
200
150
avg doc length
14,140
4.912
84,770
6,592
16.198
13,319
7.185
avg top-S chunks
2,066
2,071
2,549
2,172
2,352
2,322
2,385
avg top-10 chunks
4.137
3,716
5,125
4,018
4,644
4,554
4,305
avg top-20 chunks
8,160
4,658
10,251
5,890
9,133
8,635
6,570
Table 1: Statistics of seven datasets used for zero-shot evaluation. All lengths are counted by the number of tokens using Llama2-70B tokenizer, and "avg top N chunks" denotes the average number of tokens from the top N retrieved chunks. Figure 2 gives more details.
MuSiQue (MSQ) (Trivedi et al., 2022) another multi-hop hoppi-hopp question answering dataset. Com- pared HQA, MSQ requires connected reasoning by reducing potential reasoning shorteuts, minimizing train-test leakage, and including harder distractor contexts. Thus, MSQ is much harder task than HQA and significantly less cheatable.
MultiField en (MFQA) (Bai al., 2023) was manually curated to better the model's long context understanding ability across diverse fields. The evidences from multiple sources, including legal documents, government reports, encyclopedias, and academic papers, are fairly randomly located in the documents to avoid biases that might occur at the beginning ending of the documents.
The full details of the dataset can be found in Table 1. We can see that our evaluation datasets have a wide range of average document length from 4.9k (QASP) to 84k (NQA). Therefore, for the baseline model without retrieval, we truncate the document accordingly to fit into the input sequence length.
Following the official metrics, we report the geometric mean of ROUGE scores (i.e., ROUGE- 1/2/L) (Lin, 2004) for QM, the exact matching (EM) score for QLTY, and Fl scores for the remaining five datasets QASP, NQA, MSQ. HQA and MFQA.
3.3 CONTEXT WINDOW EXTENSION
We extend the context window length with position interpolation method (Chen al., 2023), it is simple and effective for RoPE embeddings. We extend the 4K context window to 16K for GPT-43B. For Llama2, we extend its 4K context window 32k for Llama2-7B and both 16K and 32K for 1lama2-700B. We follow Chen et al. (2023) and finetune both LLMs on the Pile dataset (Gao et al., 2021) with batch size as 128, constant learning rate of 5e-6 adapt the position embeddings.
3.4 RETRIEVAL.
For the retriever, we experimented with three retrievers: 1) Dragon (Lin et al., 2023) as achieves state-of-the-an results both supervised and zero-shot information retrieval benchmarks (Thakur et al., 2021). Dragon is a dual encoder model that consists of a query encoder and a context encoder. 2) widely used Contriever model (Izacard et al., 2021). Following the MoCo technique (He et al., 2020), Contriever used simple contrastive learning framework to pre-train models for information retrieval. It was trained without supervision and achieved competitive results with BM25 for R@ 100 on the BEIR benchmark (Thakur et al., 2021), and 3) OpenAl embedding". For the OpenAl embedding model,
as recommended by OpenAL. It accepts 8, 191 maximum input tokens for one sequence with an output vector of 1,536 dimensions. The cosine similarities then computed between the questions and the list of contexts for retrieval ranking.
the latest peext-embedding-aaaaooo
To use these retrievers, we first chunk each context document with 300 words, and then we encode both the questions and all chunks independently with corresponding encoders. The most relevant N chunks, ranked by the dot product of the question embedding and chunk embedding, are then concatenated together (following the left to right order from the most relevant to least relevant) as context of the prompt for generation. Table I shows the statistics of the top N retrieved chunks while Figure in the Appendix gives more details of the token length distribution all seven datasets. We
hhttps://plttfor opena wwtp/ccc///ccccccccccccccccccccc guuccs/////tt gui des /ebeeddeediddd ngs
5

---

Published as a conference paper at ICLR 2024
Model
Seq len.
Avg.
QM
QASP
NQA
QLTY
MSQ
HQA
MFQA
GPT-43B
4k
26.44
15.56
23.66
15.64
49.35
11.08
28.91
40,90
+ ret
4k
29.32
16.60
23.45
19.81
51.55
14.95
34.26
44.63
GPT-43B
16k
29.45
16.09
25.75
16.94
50.05
14.74
37.48
45.08
+ rel
16k
29.65
15.69
23.82
21.11
47.90
15.52
36.14
47.39
Llama2-70B
4k
31,61
16,34
27.70
19.07
63.55
15,40
34.64
44.55
+ ret
4k
36.02
17.41
28.74
23,41
70.15
21.39
42.06
48.96
Llama2-70B
16k
36.78
16.72
30.92
22.32
76.10
18.78
43.97
48.63
+ ret
16k
37.23
18.70
29.54
23.12
70.90
23.28
44.81
50.24
Llama2-70B
32k
37.36
15.37
31.88
23.59
73.80
19.07
49.49
48.35
+ ret
32k
39.60
18.34
31.27
24.53
69.55
26.72
53.89
52.91
Llama2-7B
4k
22.65
14,25
22.07
14.38
40.90
8.66
23.13
35.20
+ ret
4k
26.04
16.45
22.97
18.18
43.25
14.68
26.62
40.10
Llama2-7B
32k
28.20
16.09
23.66
19.07
44.50
15.74
31.63
46.71
+ ret
32k
27.63
17.11
23.25
19.12
43.70
15.67
29.55
45.03
Table 2: Comparison of model variants (GPT-43B, Llama2-7B, Llama2-70B) with sequence length ranging from 4k to 32k under seven datasets. "ret" denotes using the best retriever (Dragon or Contriever OpenAl embeddings) and here used top-5 for the retriever.
can see that top-5 chunks can all fit into 4k sequence length (except few outliers) while top-10 and top-20 chunks can fit into 16k sequence length.
3.5 INSTRUCTION TUNING
To train the pretrained LLMs follow instructions for question answering or text summarization, we also performed instruction tuning. We first construct blend of instruction tuning datasets consisting of 102K training samples from the Soda dataset (Kim al., 2022), ELI5 dataset (Fan al., 2019), FLAN dataset (Wei et al., 2021), Open Assistatant dataset (Köpf et al., 2023), Dolly (Conover et al., 2023) and proprietary sourced conversational dataset, to adapt all foundation models follow instructions. In terms of the template, we use "System: [System AnnnUser: (Question) |nnnAsisstant (Answer)" as the format to support multi-turn dialogue training. As of the tasks contain the context information for reasoning over at inference time, add the context before the dialogue, i.c. "System: [System] J Anin [Context) Berrexxtt/nnnnsseee User [Question Couuestion/nnnaaee Assnssistttt::: {Answer]".
We finetune the LLM by taking the loss only on the (Answer) part with batch size 128 and learning rate of 5e-6 for 1000 steps. For the rest of the paper, results are all reported using the instruction tuned chat model on top of the foundational GPT-43B, Llama2-7BB and Llama2-70B.
4 RESULTS
In this section, we report the results and provide detailed analysis.
4.1 MAIN RESULTS
In Table compare different model variants with context lengths ranging from 4K to long as 32K using GPT-43B and Llama2-70B. First, we find that baseline models without retrieval of 4k sequence length achieve the worst results for both GPT-43B and Llama2-70B. This is because the minimum average sequence length of all seven tasks exceeds 4096, the context window of the foundation models and therefore valuable texts get truncated randomly. As a result, retrieval is especially helpful for 4K LLMs e.g., Llama2-70B-4K is improved from 31.61 to 35.73 while GPT-43B-4K is improved from 26.44 to 29.32. Second, we observe that HotpotQA (HQA) especially favors long sequence models as the score improves from 34.64 to 43.97 for Llama2-70B and from 28.91 to 37.48 for GPT-43B when the sequence length increases from 4k to 16k. This is because Hotpot QA is multi-ha multi-hop dataset where the questions not hard answer but all intermediate hops are necessary to get correct answer. Therefore, long context are beneficial to increase the recall of incorporating all intermediate hops.

---

Published as a conference paper at ICLR 2024
EM scores (Llama2 70b)
52
50
45
44
42
Uama2270b-44 Ulma2-708 708--22
Figure 1: Llama2-70B also displays lost-in- in-the--eeeee middle phenomenon
Model
Avg-7
Avg-4*
QM*
QASP=
NQA*
QLTY*
MSQ
HQA
MFQA
Davinci003 (175B)
39.2
40.8
16.9*
52.7*
24.6*
69.0s
22.1
41.2
47.8
GPT-3.5-turbo (4k)
38.4
39.2*
15.6*
49.3*
25.1*
66.6*
21.2
40.9
49.2
+ret
24.4
49.5
49.5
GPT-3.5-turbo 16kk
42.8
42.4
17.6
50.5
28,8
72.6
26.9
51.6
52.3
+ret
30.4
46.6
52.8
Llama2-70B-32k
40.9
42.4
15.6
45.9
28.4
79.6
19.1
49.5
48.4
Alama2--0B-32k--
43.6
43.0
18.5
46.3
31.5
75.6
26.7
53.9
52.9
Table 3: Comparison of our best retrieval-augmenneer and Davinci-003 (175B parameters). For QMSum (QM). Qasper (QASP). Narrative (NQA), QuALITY (QLTY), we used the test set from the ZeroSCROLLS leaderboard as the organizers have prepared the scores of GPT-3.5-turbo (4k) and Davinei-003 (highlighted with *). Avg-7 refers the average score of all 7 datasets, and Avg-4* refers to the average of 4 datasets from ZeroSCROLLS.
1aamaa2-70B32k---aa with GPT-3.5-turbo 16k
is quite interesting that the retrieval-augmeenedd long context LLM (e.g., 16K and 32K) can obtain better results than auggeeal-augmented 4K context LLM, even they are feed with the same top chunks of evidence. We hypothesize this interesting observation is related to the "lost in the middle" phenomenon (Liu al., 2023), where the LLMs has such "U-shaped" performance curve. Specifically, LLMs are better at utilizing relevant information that occurs at the beginning or end of its input context window. To further verify the hypothesis, we conduct the "lost-in-the-mine tihemmiddee study following Liu et al. (2023) for 1lama2-70B-4K and Llama2-70B-333 phenomenon also exists in Llama2-70B with different context lengths. In particular, the comparison of the curves from Llama2-70B 44k and Llama2- 1ommaa----------- 32k suggests that the long context model has better accuracy for incorporating top-5 retrieved context.
As show in Figure I, we confirm that the
Note that, we have very different observation from the conclusion drawn from LongBench work (Bai et al., 2023): "Retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability". Here, we demonstrate retrieval can significantly improve the performance of both GPT-43B and Llama2-70B regardless their context window size. For example, outperforms its baseline w/o retrieval by margin, i.e., 39.60 vs. 37.36. We think the major reason for such different conclusion is that Bai al. (2023) uses much smaller LLM with 6B and 7B parameters. which usually has relatively worse zero-shot capability to incorporate the retrieved chunked context. To further validate the hypothesis, we also report the results using Llama2-7B in Table One can actually draw similar conclusions to Bai et al. (2023). We think the underlying reasons are: i) For LJama2-7B-chat--e its short context length is the bottleneck for long context tasks. Thus, retrieval- and ChaiGLM2-66B333
best retrieval-augmented Llama2-70B-32k-ree
augmentation largely improves the results. ii) For Llama2-7B-cha context length bottleneck has been mostly removed. However, their -aggeeee-e models have limited zero-shot capability of incorporating retrieved chunks of context, due to the smaller size. As a result, retrieval is not helpful for both Llama2-17 7B-32k and ChatGLM2-66 3-332,, which different from large L.LMs like Jlmm2--70-333 in case.
the
7

---

Published as a conference paper at ICLR 2024
Seq len
Setting
Avg.
QM
QASP
NQA
QLTY
MSQ
HQA
MFQA
baseline (w/o ret)
31.61
16.34
27.70
19.07
63.55
15.40
34.64
44.55
Dragon
35.73
18.14
29.20
23.39
70.30
20.09
41.54
47.45
Contriever
36.02
17.41
28.74
23.41
70.15
21.39
42.06
48.96
OpenAl-embeudding
35.79
17.76
28.85
23.57
70.70
19.92
41.76
47.99
32k
bascline (w/o ret)
37.36
15.37
31.88
23.59
73.80
19.07
49.49
48,35
Dragon
39.60
18.34
31.27
24.53
69.55
26.72
53.89
52,91
Contriever
38.85
17.60
31.56
23.88
69.00
26.61
49.65
53.66
OpenAllembeedding
39.34
18.24
32.07
24.36
69.45
24.90
51.64
54.75
Table 4: Comparisons of adding top 5 retrieved chunks from different retrievers to the context under Llama2-70B.
Seq len
Setting
Avg.
QM
QASP
NQA
QLTY
MSQ
HQA
MFQA
4k
base
31.61
16.34
27.70
19.07
63.55
15.40
34.64
44.55
top-5
35.73
18.14
29.20
23.39
70.30
20.09
41.54
47,45
top-10
34.62
16.54
28.67
24.38
68.70
19.00
42.18
42,84
top-20
34.61
16.52
28.67
24.38
68.70
19.00
42.18
42.84
16k
base
36.78
16.72
30.92
22.32
76.10
18.78
43.97
48.63
top-5
37.23
18.70
29.54
23.12
70.90
23.28
44.81
50.24
top-10
38.31
18.41
30,20
25.53
73.60
22.78
47.72
49.91
top-20
36.61
17.26
29.60
25.81
72.30
22.69
41.36
47.23
32k
base
37.36
15.37
31.88
23.59
73.80
19.07
49.49
48.35
top-5
39.60
18.34
31.27
24.53
69.55
26.72
53.89
52.91
top-10
38.98
17.71
30.34
25.94
70.45
22.80
55.73
49.88
top-20
38.38
16.36
30.42
24.42
69.60
24.51
54.67
48.65
Table 5: Comparisons of adding top-5/10/20 retrieved chunks to the context under 4k, 16k, and 32k input sequence lengths using Llama2-70B. More context does not always give better results.
contrast, the larger instruction tuned LLMs like Llama2-70B has much stronger zero-shot capability to incorporate retrieved evidence. This observation is becoming more clear when one compares the gain of cetrevaal-nggmentation between GPT-43B and Llama2-70B, where Llama2-70B enjoys larger benefit of incorporating context through retrieval.
4.2 COMPARING TO OPENAI MODELS
To further understand how good is our best model, i.e., augmenting Llama2-70B-33 with retrieval, we also compare it to (FT-3.5-turbo4kk) GPT-3.5-turbo- and Davinci-003 on those seven datasets." We found that 1llaaa--7B-322---- achieves better results than GGPT-3.5-turb-16 in terms of the average accuracy over seven datasets, while better than Davinci-003 (w/ 175B parameters) on the average over 4 tasks. This indicates Llama2-70B- with retrieval is strong model for these long context tasks, and our conclusion is built on the statc-of-the-an results.
We also report the retrieval augmented results for OPT3.5-turbo MSQ. HQA and MFQA. For GPT3.5-turbo-4. retrieval significantly improves the performance (avg from 37.08 to 41.15). For UPT3.5-turbo-166 the average scores for retrieval (43.27) and non-retrieval (43.60) scores are close to each other which are both lower than our 700---00--22-re- results (44.51). Note that 5PT turbo-16k is blackbox API, don't know how it is implemented, the model size well any preprocessing steps.
'For QMSum (QM), Qasper (QASP), NarrativeQA (NQA), QuALITY (QLTY), we used the test set from the ZemSCROLLS leaderboard as the organizers have prepared the scores of G-rrb.5-turbo (4k) and Davinci-003 there.
8

---

Published as a conference paper at ICLR 2024
Model
Trec
SAMSum
GPT--.5-turbo- 16k
68
41.7
Llama2-70B
Llama2-70B-rel
76
47.3
Table 6: Comparison of Llama2-70B GPT-3.5-turbo- 16k with two few-shot learning tasks from LongBench. Retrieval is helpful for few-shot learning as well.
4.3 ABLATION ON DIFFERENT RETRIEVERS
To investigate the impacts different retrievers on top of Llama2-70B, we compare Dragon, Con- triever, and OpenAl embeddings on top of Mlama2-70B-4e and Llama2-70B-22 The results in Table 4 confirms that our finding, i.e., retrieval can boost the performance of both short context and long context LLMs, is consistent across different retrievers
4.4 INCREASING THE NUMBER OF RETRIEVED CHUNKS
To study the impact of adding more retrieved chunks to the context, we increase the number of retrieved chunks from 5 to 20 using Dragon retriever and the results can be found in Table We observe that for different sequence lengths, the best averaged results are obtained either from top 5 or top Even if 20 chunks can still fit into the 16K and 32K context window (as shown in Figure 2), adding more chunks up 20 is not helpful and will sometime hurt the performance. We believe this related to the "lost in the middle" phenomenon (Liu al., 2023) or the model getting distracted by irrelevant information and therefore needs further research.
4.5 RETRIEVAL FOR FEW-SHOT TASKS
In addition the zero-shot tasks of query-based summarization tasks and question answering tasks mentioned above, we further investigate the benefits of long context models for few-shot tasks using two additional datasets (Trec and SAMSum) from LongBench. We take the question from each dataset as the query and use it to search relevant QA pairs provided in the given few-shot examples. Table 6 shows that our best model 1ama2-70B-32k baseline as well as GPT-335-turbe 6k by a large margin. It again confirms the benefits on using retrieval together with long context models.
outperforms its non-retrieval Llama2-70B-322
CONCLUSION
In this work, we systematically study the perrival-auugmenaatie versus long context extension using the state-of-the-art L.MSs after instruction tuning for various long context QA and query- based summarization tasks. After study, we have the following interesting findings: i) Retrieval largely boosts the performance of both 4K short context LLM and 16K/32K long context LLMs. ii) The 4K context LLMs with simple retrieval-augee augmentatioon can perform comparable to 16K long context LLMs, while being more efficient inference. iii) After context window extension and retrieval-augmentte augmentation, the best model Llama2- 20-22-0---2----- can outperform GPT-3.5-turbo 166 and Davinci003 terms average score on a suit of downstream tasks with informative queries. Our study shed light on the promising direction of combining retrieval and long context techniques together to build better LLM.
6 FUTURE DIRECTIONS
There are many potential research directions that can be extended from this work. One direction is to develop advanced methods (e.g. memory or hierarchical attention) for existing pretrained large language models e.g. Llama2-70B, which is itself non-trivial. Also, further extending the context window to 64k and even longer would be a very interesting study for large 70B parameter models though pre- mre-training longer sequence requires much more computation. Lastly, how to mitigate the ceess-i--the-micee loss (Tay et al., 2022b) could be potential solution.
phenomenon is an open research topic and continue pretraining with UL2

---

Published as a conference paper at ICLR 2024
REFERENCES
Anthropic. Introducing 100k context windows. http:://ww.antttrooic..o.ol// //ndee////
botb-cootext-wwindowe 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 1.Xx::2308.15000 2023.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 0004440555550 55, 2020.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Sean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. NeurIPS, 2022.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306, 15595, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint urXiv: /904. 10509, 2019.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open
instruction-tuned 1lm, 2023. URL https: /www:/wwwwww/////// http://wwwaabbbb.rc.c cks tuned- 11m. 12/dol waredooll---r---ooo-rrr
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell. Quoc V Le. and Ruslan Salakhutdinov. Transformer-X XL: Attentive language models beyond fixed-length context. In ACL, 2019.
Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869/ 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, 2022.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seekinn questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610, Online, June 2021. Association for Computational Linguistics. doi: 18.18653/v 1/202 naacl- m6in.365. URL https: http///aaaaantoooooo/oeo org/ 2021.
365.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv: 1rXiv:1907.0919 1907.09190, 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027. 2021. URL https: /ttp:///axivvvv/ org/ als 2 2101 00027.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. REALM: Retrieval augmented language model pre-training In ICML, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint arXiv: 2019.
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.0799 2023.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In EACL, 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosscini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021. URL https: arxiv. org abs/ 22/221122112 09118.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Y Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022.
Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang, Jamie Callan, and Graham Neubig. Retrieval attention: End-to-end learning of retrieval and reading within single transformer. In EMNLP, 2022.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535- 547, 2019.
Kaiokendev. Things I'm learning while training SuperHOT. https::////.. https://kaiokendee wtw.:ookknnee....t...... 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. EMNLP, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 2019.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, et al. Soda: Million-scabe dialogue distillation with social commonsense contextualization. arXiv preprint 1xXi:22222110666 2022.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Openassistant converations-democratiiig large language model alignment. arXiv preprint arXiv:2304.07337, 2023.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 05 2018. ISSN 2307-387X. doi:
10.1 162/tacl_cccc a_00023. URL https: : doi. org/10 /10. 1111111/aaaee 1122/tacl___0022..
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks. NewrIPS, 2020.
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. org/ WD 1013.
URL https::t////... / /aa/acaaaaa aataaaaaaanntoogooeeeaa
Sheng- Sheng-Chich Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452. 2023.

---

Published as a conference paper at ICLR 2024
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In ICLR, 2018.
Kyle Lo, Lucy Lu Wang. Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 9969-4988 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.1186551122222 a2ainn.ammmmmiin4477777777 URL https: / aalaacaantt anthology org/ 2020. acl-main 447.
main.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXivv2305. 16300, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprini arXiv:21 2021.
Erik Nijkamp, Hiroaki Hayashi, Tian Xie, Congying Xia, Bo Pang. Congying Xia, and et al. Long sequence modeling with XGen: A 7b LLM trained on 8k input sequence length. https:// blog. s.log.aaaeesoorcca.eeeeecc.. wttptrcc..co//ggg//eeec 2023.
OpenAI. Introducing
2022. 2022.
OpenAI. Gpt-4. http:////pppaacccccc////aae////p
2023a.
OpenAl. Function calling and other API updates (longer context). https: http://openaaiiicc com/ 2023b.
blog/ www.ffunnnoonnnoo cal lallling wwwp-aaa--ttttaa---pppeeee
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He. and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, pp. 5336-5358, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.11863/vlll 1ttp://11222222aamma main.391. URL https: ac http///aaaaattoooogyoooooo org/ 2022. aaaacc-main.. 391.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, pp. 4064-4064, 2018.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive Transformers for long-range sequence modelling. In ICLR, 2020.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Browa and Yoav Shoham. Parallel context windows for large language models. In ACL, 2023.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 2021.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: /aw.anttoooogyy.o.ccoooooooooola org/2022. emn lamllpmmaiin in. 823.
12

---

Published as a conference paper at ICLR 2024
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305. 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. RePlug: aetreeeel-uggmented black-box language models. arXiv preprint DrXiv:2301.126520 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprini arXiv: 100.000400000 2021.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In ICML., 2020.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In ICMI, 2021.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: survey. ACM Computing Surveys, 2022a.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, al. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2022b.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092: 09288, 2023b.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022. doi: 10.11662 1.6//1c/ccc__ 00475. URL pttttps:/aclanthhllli org/2022. tacl-1.31
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miloš. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.031700 2023.
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. Shall pretrain autoregressive language models with retrieval? a comprehensive study. arXiv preprint arXiv:2304.066: 067662 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.035533 2022.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.044: 2020.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0163 09.6012, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. arXiv preprint arXiv: 006622 2022.
13

---

Published as a conference paper at ICLR 2024
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystrômformer: A nyström- algorithm for approximating self-attention. AAAI, 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing. pp. 2369-2380, Brussels, Belgium, October-Noventee 2018. Association for Computational Lin- guisties. doi: 10.18653/v1/DI 8- 1259, URL https: a antho log.. hawproooogoooggg///oogo org/D1 8-1299
Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362-373, 2021.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for longer sequences. In NeurIPS, 2020.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 13m-:30b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query- based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 5905-5921, Online, June 2021. Association for Computational Linguis- tics. doi: 1165381112222222 maacl-main.472 baacl-mal main. 472.
URL /tRLststs///elllttooooo.gg org.ol2000y../2111.....
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. NeurIPS, 2021.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
length of full document
length top chunks
80000
6000
70000
60000
4000
50000
40000
3000
30000
20000
1000
10000
QASP
NQA
QLTY
M5Q
HQA
MFQA
QM
QASP
NQA
QLTY
MSQ
HQA
MFQA
length of top chunks
length top chunks
8000
18000
7000
16000
6000
14000
12000
5000
10000
4000
x
8000
3000
6000
2000
1000
2000
QASP
NQA
QLTY
MSQ
HQA
MFQA
Figure 2: Token length distribution of the full document and the top-5, 10, 20 chunks of the seven datasets.
A.1 EXAMPLE
We show an example below where the smaller model Llama2-7B fails incorporate relevant context, while larger models with retrieval could successfully predict the correct answer.
15

---

Published as a conference paper at ICLR 2024
Chunk
On September 18, 2015, the deluxe edition of the album was released containing live and instrumental tracks from the standard edition album, addition to the single "Light" featuring Little Dragon. Critical recep- tion... ... Angelspit has toured with Angel Theory, Ayria, Ikon, KMFDD, Tankt and The Crüxshadows, and have also shared the stage with bands such as The Sisters of Mercy, Nitzer Ebb, Skinny Puppy and Front Line Assembly. They performed with Lords of Acid during a 22-date U.S. tour March 2011 and toured the United States with Blood on the Dance Floor in October 2011. History Karl Learmont (ZooG) and Amelia Tan (Destroyx) met on an online zine forum. They shared an interest in zines and started the distro Vox Populis in 2002.
Chunk
They then started making zines for themselves which became the lyrical inspiration for releases to follow. Angelspit was formed in 2003, and the duo then self-released their debut EP, Nurse Grenade October 2004, video for the remix of "Sleep Now" was released on October 2010. They released their third remix album, Carbon Beauty on 8 March 2011. This new remix album contains 3 new tracks as well as 10 remixes of tracks from the Hideous and Perfect album. video for "Toxic Girl" was released on 13 April 2011, and a video for "Like It?
Chunk
Passage 1: Blood the Dance Floor (band) Blood the Dance Floor was an American electronic music group from Orlando, Florida, formed in 2006. The group's longest standing lineup, from 2009 2016, consisted of Jesus David Torres also known as Dahvie Vanity (born 1984) and Jayy Von Monroe (born 1991). ... The CD was self-released October 2008. Only 300 copies were made. Vanity and Ecstasy recorded the singles "Siq With a Q" and "Suicide Club"
duo 2008, and released three extended plays over the first half of 2009, 1 Scream Scream, OMPG Sneak Peak, and Extended Play.
Chunk
title: source: Lick It!" was released on 27 July 2011. On 15 April 2011, Angelspit announced the addition of three new members: guitarist Valerie Gentile (Black Tape for Blue Girl, The Crüxshadows), drummer Chris Kling (Hanzel und Gretyl, Mortiis) and videojammer The Liar. The new line-up of Angelspit released their fourth studio album, Hello My Name on 11 October 201 Matt James replaced Chris Kling early 2012, and former Crüxshadows guitarist George Bikos filled in for Valerie Gentile on the band's 2012 tour...
Chunk
Vanity denied these allegations nounced that there would be new member, and that Blood on the Dance Floor would be returning on May 5 of that year without Jayy Von Mon- roe. On January 2021, Vanity released new version of Blood on the Dance Floor's "Bewitched" as "Bewitched Reimagined" featuring singer Sammy Beare. As of 2016, Jayy Von Monroe has continued work as drag monster under the name "The Dahli" and Vanity has continued to release music and merchandise by himself under both "The Most Vivid Nightmares" and "Dark Arts Official".
vndeo. April 2017, Vanity an-
Question:
Angelspit toured the United States with an American electronic music duo from Orlando, Florida, that formed in what year?
LLaMA2-7b-32k
Angelspit toured the United States with Blood on the Dance Floor in October 201
LLaMA2--bb32k-ree
Angelspit toured the United States with Blood on the Dance Floor in October 2011.
I_AMAA-700-321
2011
LLaMMA--70-32kkrrtt
2006
Ground truth
2006
Table 7: All models fail to get the correct answer of 2006, except LLaMA2-700-322-re which shows how retrieval together with long context can help derive the correct answer.
16


Reviews:
[{"review_id": "OiS9gFPLpg", "reviewer": "Reviewer_1poR", "summary": "The paper investigates the comparison between retrieval-augmentation and long context extension for LLMs across various downstream tasks. It demonstrates that retrieval-augmentation can improve the performance of LLMs irrespective of context window size. They also reveal that shorter context window LLM with retrieval-augmentation can perform close to longer context LLM finetuned via positional interpolation, while taking much less computation.", "questions": "1. The OpenAI GPT series also offers variants with different context lengths (e.g., GPT-3.5-turbo, GPT-3.5-turbo-16k). Including their results in Table 2 and examining whether the conclusions still hold would be beneficial.\n2. Regarding the results presented in Table 2, it is unclear how the best retriever was determined. Is a better-performing retriever module always associated with improved LLM performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper delves into the significant yet underexplored problem of retrieval-augmentation versus extended context windows, presenting a comprehensive study that spans a variety of popular long-context tasks to compare the two methods.\n2. The findings that a \"shorter context window LLM with simple retrieval-augmentation at inference can perform on par with a longer context LLM fine-tuned through positional interpolation,\" and that \"retrieval can enhance the performance of LLMs regardless of their context window size,\" offer valuable insights for future research endeavors.", "weaknesses": "1. The authors attribute the superior performance of retrieval-augmented long context LLMs (e.g., 16K and 32K) over the 4K context LLM in a top-5 setting to the 'lost-in-the-middle' phenomenon. However, since the 'lost-in-the-middle' conclusions have not been widely recorganized, conducting an ablation study would be instrumental in supporting this hypothesis.\n2. The authors use LLM performance metrics to infer the efficacy of different retrieval modules (Table 4). However, it is questionable whether a retrieval system with better LLM performance necessarily correlates with enhanced retrieval capability. The authors should incorporate some form of automated evaluation or a case study on retrieval performance to substantiate this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "umGHwzUim5", "reviewer": "Reviewer_B6CQ", "summary": "This research paper compares two methods of handling context in large language models (LLMs); extending the context window and retrieval augmentation. The main contributions of this study are as follows;\n\n1. The researchers conducted experiments using two state-of-the-art LLMs, a proprietary 43B GPT and LLaMA2 70B on seven tasks related to long context question answering and summarization.\n2. The results demonstrate that retrieval significantly improves the performance of both 4K) and long (16K/32K) context LLMs. Surprisingly LLMs, with a context size of 4K, when combined with retrieval perform close to LLMs with a context size of 16K without incurring additional overhead.\n3. It is also shown that extending the context window and retrieval augmentation complement each other. The performing model, LLaMA2 70B 32K enhanced by retrieval outperforms ChatGPT 3.5.\n\nTo summarize this paper provides insights into the combination of retrieval techniques and long context strategies for constructing improved language models (LLMs). The key findings indicate that retrieval is beneficial regardless of the context window size and extending the context window alongside retrieval augmentation offers reinforcing solutions, than competing ones.", "questions": "The paper mentions another study called LongBench that reached conclusions regarding the impact of retrieval, on context models. It would be beneficial if the authors could further analyze the differences that led to these opposing observations.\n\nHave you tried training the retriever and LLM from start to finish of just adding augmentation during inference? How does the performance compare in cases?\n\nWhen retrieving passages how do you determine the optimal number to use? Does performance level off. Decline with many passages?\n\nFor context models like those with 64K or 128K tokens do you believe retrieval would still provide benefits?. Does its value decrease after a point?\n\nCould you provide analysis and examples that demonstrate the phenomenon you hypothesize as \"lost in the middle\"?\n\nAre there any approaches besides using long context and retrieval augmentation that you could compare to? For example memory or hierarchical attention.\n\nAlthough the paper focuses on QA and summarization it would be interesting to see if similar conclusions apply to tasks.\n\nHave you considered examining the impacts and limitations of building larger LLMs? A brief discussion, on this topic would be valuable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This research study aims to examine and compare two approaches, for improving Large Language Models (LLMs); extended context utilization, and retrieval mechanisms. The authors utilize two LLMs with 43 billion and 70 billion parameters, which allow for better integration of context. Through experimentation, they assess the performance of these models across challenging datasets related to long context Question Answering and summarization.\n\nThe research provides insights demonstrating that retrieval mechanisms offer performance benefits regardless of the context length. Additionally, the study highlights how these two techniques complement each other. The paper is skillfully written, presenting an explanation of its motivation, background, experimental methodology, and analytical framework. The introduction effectively places the research in the context of work on long-context understanding, efficient attention mechanisms, and retrieval augmentation. This establishes a foundation for the study.\n\nThe authors thoughtfully compare their findings with research, including benchmark models like ChatGPT. This offers a view of the progress made in this field. Furthermore, the paper raises questions about comparing context understanding and retrieval as strategies for enhancing LLMs encouraging further exploration in this area. The extensive original experiments conducted contribute significantly to our understanding of these methods, within the community.\n\nOverall the results of the study are highly significant, for the research community as they shed light on the effects of combining retrieval and long context.", "weaknesses": "Most studies primarily focus on improving models during inference with attention given to examining the impact of training methodologies.\nThere is a lack of analysis when it comes to integrating context and retrieval techniques like determining the ideal number of retrieved passages.\nExperiments have not yet been conducted on contexts, which span 64,000 tokens and this could pose additional challenges.\nThere is a need, for evaluations against advanced methods in long context modeling and further benchmarking is necessary.\nThe experiments section would be more informative if it provided information about characteristics and task formats.\nThe related work section should clearly highlight the contributions of this study compared to work like LongBench.\nThere is a discussion regarding the implications and limitations associated with the techniques being investigated.\nIt's important to note that previous research has explored the concepts of context and retrieval. This study should acknowledge and build upon existing literature.\nThere are opportunities to explore approaches in combining retrieval and context that merit investigation.\nFurther validation is required to assess the suitability of these techniques, for contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "24LIKYnlOB", "reviewer": "Reviewer_MoDw", "summary": "This paper studies tasks that require reading / reasoning over long documents, and considers two commonly used approaches: retrieval augmentation and long-context LLMs. By experimenting on 7 long-context tasks from the Scroll and LongBench benchmarks, this paper first compares the two solutions and finds that a retrieval-augmented LLM with standard context windows can perform close to long-context LLMs. It further shows, contrary to the finding of a contemporary paper, that retrieval helps improves the LLM performance regardless of the context-window size. As a result, their best-performing model, a retrieval-augmented Llama2-70B-32K (Llama 2 finetuned to 32K context window with positional embedding interpolation) outperforms ChatGPT-3.5 on 5 of the 7 tasks.", "questions": "1. What's your retrieval corpus? Do you use the same corpus for all tasks? Or do you use the collection of gold contexts for each dataset as the retrieval corpus for that task? (If so, do you build a single index using contexts from all splits including the training and test set?) Or, do you build a separate index for each example using the chunks from the provided gold context for that example?\nSorry if I missed it, but I wasn't able to find any details on the retrieval corpora you used in your experiments.\n\n2. It seems the evaluation tasks are all \"closed-type\", meaning the gold context is given at inference time. Is this the case? If so, how would retrieval be further helpful if the gold context is already provided? Is it because for certain tasks, the provided gold context can be too long and the relevant parts may have been truncated?\n\n3. In Table 4, it is interesting that using the same top 5 retrieved chunks, DRAGON works better with 32k LLM while Contriever works better with 4k LLM. Any insights into what caused this? Have you tried reversing the order of the retrieved chunks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper investigates a widely-interested problem, long-context modeling, and compares two most popular solutions, long-context LLMs and retrieval-augmented LLMs. Its results and insights are hence of considerable interest to the research community.\n\n- The paper is clearly written, and the experiments are relatively extensive. In particular, it achieved positive results showing retrieval augmentation and long-context LLMs can be complementary to each other, contradicting negative results from an earlier work. Furthermore, it experiments with rather capable SoTA LLMs such as Llama2-70B, instead of much smaller \"toy\" LLMs, making their results more convincing and useful.", "weaknesses": "- While this paper presents a useful empirical study, it does not introduce any new technique. Combining retrieval augmentation and long-context LLM is a straightforward idea, and the two being complementary to each other, though encouraging, is not very surprising.\n\n- If I understand correctly (I find this part of the writing unclear, see Question 1 below and correct me if my assumption is wrong), the retrieval corpus is the collection of the context documents from each evaluation dataset (or maybe even each example). This may be a key reason why retrieval is more helpful in this work. In reality, for most of the tasks, it is impossible to assume to have perfectly relevant retrieval corpora (i.e. gold contexts from the test set), and the helpfulness of retrieval would dramatically decrease if the retrieval corpus mismatches the evaluation data.\nAs a result, the method considered in this paper is more tailored for one type of task, essentially reading comprehension of long documents, because the assumption that gold contexts, albeit long, are provided at test time. It would be more interesting to also explore the \"open\" settings where the gold context is not necessarily given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8BPVu6PFUi", "reviewer": "Reviewer_xWjb", "summary": "The paper presents a detailed comparison of different methods of handling long contexts by 1) using retrieval augmentation 2) increasing the context length of LLMs using positional interpolation. The papers conducts experiments on 7 datasets and show that retrieval can significantly improve the performance of LLMs. The paper also shows results on increasing the number of retrieved context and compares different retrievers.", "questions": "If the chunking is 300 words (tokens ?), why is there so much variance in the top-5, top-10, shouldn't the top-5 context length consistently be around 1500 tokens? If the variability is due to chunking based on words and not tokens, why is that the case ? and why was 300 specifically chosen?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper performs comprehensive experiments on two LLMs with up to 32K context, demonstrating the benefits of retrieval even for long context models. The paper gives good insights about retrieval augmentation confirming a simple but effective alternative to expensive context scaling of huge models. The paper is well-written and easy to understand.", "weaknesses": "The paper has limited novelty. The majority of the paper compares to longer context LLM finetuned via positional interpolation, it may be better to have some results with the other context expansion techniques and with models that are natively trained with larger context lengths to possibly remove that source of error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RIH48SSarm", "reviewer": "Reviewer_FNKh", "summary": "In this paper, the authors systematically study the effectiveness of utilizing retrieval-augmentation and long-context tension in LLMs for downstream tasks. They conduct experiments using two large foundational models and evaluate their performance on seven downstream tasks that require longer context understanding. Compared to a previous study (Bai et al. 2023),  this work employs larger LLMs, leading to a different conclusion: retrieval significantly enhances the performance of LLMs, regardless of the size of their context window. Furthermore, the authors observe that larger models, such as LLaMA2-70B, tend to benefit more from retrieved evidence.", "questions": "- Do you observe different conclusions when using a smaller LLM (i.e., Llama2-7B and ChatGLM2-6B) as in Bai et al. 2023?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The findings from these extensive empirical experiments are intriguing and have the potential to serve as inspiration for future research in this domain.\n- The authors demonstrate that retrieval and extended-context approaches can significantly benefit LLMs, especially when they are of extremely large size.\n- The study incorporates further analyses that clarify the impact of factors such as the number of chunks, context length, and retrievers on the performance of downstream tasks.", "weaknesses": "- While the idea is straightforward, the novelty is limited. \n- The 70B model consistently exhibits improvement when utilizing retrieval or longer context. In contrast, the 43B model shows inconsistent improvement. Consequently, drawing a definitive conclusion regarding when to employ retrieval in relation to specific downstream tasks and model sizes becomes challenging.\n- Given that the conclusion differs from prior work that utilizes smaller models, establishing an apple-to-apple comparison demonstrating instances where the same example could fail in the smaller model but succeed in the larger model would enhance the persuasiveness of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AEMR5tszj3", "reviewer": "Reviewer_L1CH", "summary": "The concept of augmenting language models with retrieval has been widely explored for several years. Recently, there has been a growing interest in expanding the context window of transformers and large language models (LLMs). This research paper focuses on two key research questions related to retrieval augmentation and context window extension:\n\n1. Which approach is more effective for downstream tasks - retrieval augmentation or a longer context window?\n2. Is it possible to combine these methods to achieve better results?\n\nTo address these questions, the authors of this paper examine two pretrained LLMs - a 43B GPT model trained by themselves and LLaMA2-70B. Through extensive experimentation, they find that a shorter context window LLM with simple retrieval augmentation during inference can yield comparable results to a longer context LLM that is fine-tuned using positional interpolation in two specific downstream tasks, namely question answering and query-based summarization. Additionally, the authors note that retrieval can significantly enhance the performance of LLMs, regardless of the size of their context window. The findings of this study offer valuable insights to practitioners, helping them make informed decisions regarding the choice between retrieval augmentation and extending the context window of LLMs.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper has a clear motivation and investigates a significant topic on large language models (retrieval augmentation versus context window extension). Existing research effort on this topic is limited, except [1]. Instead of using black box API in [1] (e.g. GPT-3.5-Turbo), this paper trained their own 43B GPT model for comparisons. \n2. This paper is well-written and the presentation is clear;\n3. Extensive experiments have been conducted on 7 downstream long context tasks, including single and multi-document question answering and query-based summarization. Two different sizes of LLMs and Three different retrievers are examined on eight tasks.\n\n\nReference: \n\n[1] Longbench: A bilingual, multitask benchmark for long context understanding. Arxiv 2023", "weaknesses": "1. The authors of this paper mainly focus on larger LLMs, specifically those with more than 40B parameters. They include two specific LLMs in their study: the 43B GPT and the 70B LLaMA. It is worth noting that most existing efforts in this field only report results on smaller models, such as the 7B model. It would be beneficial if the authors could further investigate the relationship between the size of the LLMs and the tradeoff between retrieval and the context window. The authors propose hypotheses to explain why concurrent work has produced different findings, particularly regarding the usefulness of retrieval for the Llama2-7B-chat-4k model with a 4K context window. It would be better if the authors could provide empirical evidence to support these hypotheses.\n\n2. In terms of context window extension, this paper only utilizes the method mentioned in reference [2] for both LLMs. It would be interesting to explore whether the findings would be influenced by different methods of extending the context window.\n\n3. This paper focuses primarily on question answering and summarization tasks, in contrast to Longbench[1], which encompasses a variety of tasks, including few-shot learning, synthetic tasks, and code completion.\n\n4. The paper lacks a discussion on potential limitations and future directions. For instance, the authors do not address the potential limitations of combining retrieval augmentation and context window extension, such as the possibility of exacerbating the \"lost-in-the-middle\" phenomenon [3]. While one of the key findings of this study is that combining retrieval and long-context techniques improves LLM performance, the authors could suggest avenues for future research on how to better integrate these techniques together.\n\nReference:\n\n[2] Extending context window of large language models via positional interpolation. Arxiv 2023\n[3] Lost in the middle: How language models use long contexts. Arxiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)