RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
Authors: Jie Huang
Wei Ping
Peng Xu
Mohammad Shoeybi
Kevin Chang
Bryan Catanzaro
Venue: COLM (2024)
Keywords: Retrieval-Augmented Language Models, In-Context Learning, Encoder-Decoder Language Models
TL;DR: We investigate the in-context learning ability of retrieval-augmented encoder-decoder language models.

Abstract: In this paper, we investigate the in-context learning ability of retrieval-augmented encoder-decoder language models. We first conduct a comprehensive analysis of existing models and identify their limitations in in-context learning, primarily due to a mismatch between pretraining and inference, as well as a restricted context length. To address these issues, we propose RAVEN, a model that combines retrieval-augmented masked language modeling and prefix language modeling. We further introduce Fusion-in-Context Learning to enhance the few-shot performance by enabling the model to leverage more in-context examples without requiring additional training. Through extensive experiments, we demonstrate that our simple yet effective design significantly improves performance, achieving results comparable to the most advanced language models in certain scenarios, despite having substantially fewer parameters. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning and encourages further research in this direction.

Published as a conference paper at COLM 2024
RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
Jie Huangl.2 1.2, Wei Ping2 Peng Xu²
Mohammad Shoeybi?
Kevin Chen-Chuan Chang!
Bryan Catanzaro²
University of Illinois Urbana- Champaiggn
NVIIDA
jeffhjeillinois.c wpingênvidia.com
Abstract
In this paper, we investigate the in-context learning ability of retrieval- augmented encoder-decoder language models. We first conduct a com- prehensive analysis of existing models and identify their limitations in in-context learning, primarily due to mismatch between pretraining and inference, well as restricted context length. To address these issues, we propose RAVEN, a model that combines retrieval-augmente masked language modeling and prefix language modeling. We further introduce Fusion-in-Conteed Learning to enhance the few-shot performance by enabling the model to leverage more in-context examples without requiring addi- tional training. Through extensive experiments, we demonstrate that our simple yet effective design significantly improves performance, achieving results com comparabl to the most advanced language models in certain scenar- ios, despite having substantially fewer parameters. Our work underscores the potential of retrieval-augmentte encoder-decoder language models for in-context learning and encourages further research in this direction.
Introduction
Recent advancements in natural language processing have been predominantly driven by the development of large language models (LLMs) (Brown et al., 2020; OpenAI, 2022, 2023; Chowdhery et al., 2023; Smith et al., 2022). These models have demonstrated remarkable performance across wide range of tasks (Qin et al., 2023; Bubeck et al., 2023; Huang & Chang, 2023). One the key features that enables these models to excel is their ability to perform in-context learning (Dong et al., 2022). By conditioning on given context, LLMs can adapt to new tasks and domains without the need for specifcii fine-tuning. This enables LLMs to perform well on zero-shot few-shot learning tasks, where only a limited number of examples are available.
While in-context learning has been extensively studied for decoder-only language models like GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2023), research on encoder- decoder language models, which have shown to learn stronger representations (Devlin et al., 2019; Raffel et al., 2020), remains limited. Notably, Patel et al. (2023) tap into the potential of mT5 (Xue et al., 2021), multilingual encoder-decoder LM, by iteratively prompting the model to produce long generations with in-context examples. Chung et al. (2022); Longpre et al. (2023) finetune 15 (Raffel al., 2020) with a large mixture of tasks using instruction tuning (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022) to improve model performance and generalization to unseen tasks in both zero-shot and few-shot settings.
On the other hand, LLMs still face challenges such as hallucination and limitations in representing the long-tail and most recent knowledge (Mallen et al., 2022; Huang et al., 2022; Luu et al., 2022; Jang et al., 2022; Zheng et al., 2023). Retrieval-augmented language models (Izacard et al., 2023; Borgeaud et al., 2022; Wang et al., 2023; Shi et al., 2023) have emerged as powerful approach to address these issues by retrieving relevant knowledge
Work done at NVIDIA. Code available at https: com/ wtt.:jfff/jjjjccooooo///
1

---

Published as a conference paper at COLM 2024
from external corpus, Among these, the encoder-decoder models, such as ATLAS (Izacard et al., 2023), stand out. They benefit from the strong representation ability of a bidirectional encoder, coupled with of the efficacy of Fusion-in-Decoder architecture (Izacard & Grave, 2021), enabling the effective integration of multiple retrieved passages. Despite these advancements, in-context learning with these models remains underexplored.
In this regard, we first conduct a comprehensive analysis of the state-of-the retrieval- augmented encoder-decoder language models by designing and experimenting with differ- ent prompting strategies. We find that these models exhibit a certain in-context learning ability; however, due to a mismatch between pretraining and inference and a limited context length-issues issues that are common to existing encoder-decoder LMs trained with masked language modeling- few-shot performance is not stable and providing more than, e-g. 8-shot, examples does not lead to further improvement.
Based on the analysis, we develop RAVEN" by first mitigating the mismatch between pretraining and inference through a combination of retrievll-augmented masked language modeling and prefix language modeling. Moreover, to enable the model to learn from more in-context examples, we propose Fusion-in-Conteed Learning, a novel approach that allows the model to utilize more in-context examples without modifying the model configuration or requiring additional training. Furthermore, we suggest using the retriever of the model to obtain relevant in-context examples to further enhance few-shot performance. Our empirical results demonstrate that RAVEN significantly outperforms previous retrieval-augmented encoder-decoder LMs in both zero-shot and few-shot settings, even achieving comparable results to decoder-only LLMs in some settings despite having 180 times fewer parameters.
The main contributions of this paper are twofold:
From an analytical standpoint, we provide thorough analysis of the in-context learning ability of retrieval-augmented encoder-decoder language models. We demonstrate the possibilities and offer insights for future development.
From a technological perspective, we introduce RAVEN, coupled with our Fusion- in-Context Learning and In-Context Example Retrieval strategies, building upon the analytical groundwork. These techniques, though simple, are highly effective. They not only enhance the base model's capabilities but also highlight the potential of in-context learning with -arreval-augmennte encoder-decoder LMs.
2 Background and Related Work
aatmeetl- language models are a class of language models designed to enhance their performance by incorporating external knowledge. These models typically employ an information retrieval mechanism to access relevant information from a large corpus, which is then integrated into the model's prediction process. Retrieval-augmmeted LMs can be based on both encoder-decoder (Izacard et al., 2023; Lewis et al., 2020) and decoder- only (Khandelwal et al., 2020; Borgeaud et al., 2022; Shi et al., 2022) architectures. For decoder-only LMs, the computational cost typically increases quadratically with the input length, as well as with the number of retrieval passages. In contrast, for encoder-decoder LMs with a Fusion-in-Decoder architecture, the computation cost grows linearly with the number of retrieved passages, as they only perform self-attention over one passage time (Izacard & Grave, 2021). This concept is also investigated by Ye et al. (2023) for more efficient in-context learning.
While there has been some research on in-context learning with aetrieval-augment decoder-only LMs, which can be straightforwardly implemented by concatenating retrieved passages with the query as the input of the LM (Mallen et al., 2022; Shi et al., 2023; Khattab et al., 2022), in-context learning with retrieval-augmented encoder-decoder LMs remains unexplored to the best of our knowledge. This is despite the fact that encoder-decoder LMs can be more efficient at incorporating multiple (e.g., 40) retrieved passages.
RAVVN, a bird known for its intelligence and adaptability, has the letters "RA" in its name, which represents 'Retrieval-Augmented in our context.
2

---

Published as a conference paper at COLM 2024
Masked Language Modeling (Pretraining)
Prompting Stralegy I
Prompting Stralegy
Input to Encoder:
Input to Eacoder:
Inpur Encoder:
Michine lesening algorithms build a model hised on sample datusestrr, order tocnstra id I> being explicilly programmmd
Quodion: What is the eapital of the Provence-
Question: What is the engital of the Provence-
training data,
Alpes Answer: Marseilles
Azur region l'rance?
Alpes d'Auur region France?
Answer cextra id
to do so. Machine leamning algrrithms are used in medicine.
Quostio:: The Clock word Xero (pronomed sero)
Question: The Grook werd Xero (gronounced vero)
wide variety applications. such email filtering, speech rocogaition, agriculture, and computer visiun sestra_ jd 2- unieusible to develop conventional algurithms to perform thocestra
serography and related terminology Answer: Dry
what!
seregraply related tenninology metss
Answer sextrra _d_1>
Question: in which councry was the fins permanent bungoe jumping sitused?
Question: In which country was the first permanent bungee jumping site situated?
f>>
Passage: ... stachine letening models reqaire high
Presage: fiist permanest costunercial bungee
Passage: first peratanent commercial bungee
quantay of relisble data order for the models...
site, the Kawarau Bridge Bungy the Kowarau
sine. the Kawaran Brdge Bungy at the Kawaran
Gorge Suspension Bridge near Queetstown South Island of New Zealand ...
Gorge Suspension Bridge Queenstown South Island of New Zealand ...
Input to Decuder:
Nome
Input Decoder:
Iapot Decnder:
Output:
<extra_id_cc Co Manselllesceett a Id_ Dry
cextra (bs knowneostra id Is mike predictions
decisions withoutcestr id 2> where it difficult arcestra needed
Ourput: New Zealand
Output:
To Zeoland
Figure 1: Retreeval-uggmented masked language modeling and prompting strategies for in-context learning.
3 Methodology
In this section, we first explore in-context learning with retrieval-augmented encoder- decoder language models in the literature. Building upon the analysis, we develop models with enhanced zero-shot performance and improved in-context learning abilities.
3.1 In-Context Learning with Retrieval-Augmentee Encoder-Decoder LMs
To investigate the in-context learning ability of retreetal-uugmented encoder-decoder language models, we first aim to gain insights from the state-of-the-ard designs in the literature, Among them, the design of ATLAS (Izacard et al., 2023) stands out; it combines reader (i.e., T5 (Raffel et al.,
general-purpose dense retriever with a sequence-to-sequene 2020)) using the Fusion-in-Decodee architecture (Izacard & Grave, 2021). The retriever, encoder and decoder are jointly trained during the pretraining process. In this process, the dense retriever, based on the Contriever model (Izacard et al., 2022), is responsible for selecting relevant passages from external knowledge source, e-g., Wikipedia, based on the given corrupted context. The retrieved passages are then processed along with the context by the encoder, which generates the corresponding output, i.e, the masked spans, at the decoder (Figure 1, left). ATLAS demonstrates exceptional few-shot performance on wowwledge-ntenssin language tasks (Petroni al., 2021), despite having lower parameter count compared to other recent LLMs.
However, in Izacard et al. (2023), the few-shot performance is achieved by finetuning the model with few-shot examples, which requires additional training and may limit its applications, such as dealing with dynamic and diverse real-time user queries like GPT- 3/4 4 (Brown al., 2020; OpenAl, AT, 2023), where in-context learning plays vital role. Therefore, we take the initiative to explore the in-context learning ability of this type of models, using open-domain question answering (Chen et al., 2017) as representative task for some preliminary experiments.
Prompting Strategies. To facilitate in-context learning, an effective prompting strategy is paramount. In contrast to decoder-only LMs, where the input can only be fed to the decoder, encoder-decoder LMs can take input in either the encoder or the decoder. In alignueett with the pretraining objective, we identify two prompting strategies for in-context learning:
Strategy 1. The first strategy involves feeding all example question-answe pairs and the target question to the encoder, without any input to the decoder. The prompt is designed as:?
Here we present format designed for better demonstration. The actual prompt, which follows the template used in pretraining, can be found in Appendix B.4
3

---

Published as a conference paper at COLM 2024
Natural Questions 1-shot
TriviaQA
0-shot
5-shot
8-shot
O-shot
1-shot
5-shot
8-shot
ATLAS ATLAS
11B S1 11B S2
21.3
29.8
31.3
35.5
62.3
63.9
26.7
56.9
21.4
16.3
9.8
49.8
48.4
44.4
Table 1: Results of ATLAS 11B with prompting strategy (S1) and strategy 2 (S2).
Natural Questions
TriviaQA
first
0.7
9.2
random
6.5
19.5
last
29.8
62.3
Table 2: Results of ATLAS 11B (5-shot) with different target question positions.
Enc: Question: 91 Answer: a1 ... Question: 9х Answer: ag Question: 90 Answer:<extra.id id e>
d
where (41, (q,,a),..... (9k, 4x) represent example QA pairs, g0 denotes the target question, <extra.id_00 is sentinel token (Raffel et al., 2020), and d is the relevant passage retrieved with 90- An example in 2-shot setting is illusated in Figure 1 (middle).
Strategy 2. As the decoder of the encoder-decoder model can also accept input, we can feed the answers of in-context examples to the decoder and only feed the questions to the encoder, using multiple sentinel tokens:
Enc: Question: 91 Answerkextra id 0> Question: 9): Answerkexttra id (k 1)> Ques-
tion: 90 inswer:kextra.id
Dec: <extra.id a1... <extra.id 1)> a;
Figure (right) demonstrates an example. The model is expected to learn from in-context examples by examining both the input to the encoder and input to the decoder.
We select two widely-used datasets in the domain open-domain question answering for the preliminary study: Natural Questions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (TQA) (Joshi et al., 2017)3. Table 1 summarizes the results. We find that the model struggles to learn from in-context examples using strategy 2, as the few-shot performance is worse than the zero-shot performance. We hypothesize that this is because the model has difficulty learning the pattern of S2 with masked language modeling during its pretraining, since it is unlikely to obtain several consecutive question-answer pairs (or something similar) in the form of strategy 2 by randomly masking several spans in a sequence.
On the other hand, we observe that with strategy 1, the model does exhibit some in-context learning ability, where the 5-shot and 8-shot performance is significantly better than the zero-shot performance on both NQ and TriviaQA. Therefore, we choose focus on strategy 1 for further study and disregard strategy 2 for the remainder of the paper.
Effect of Position. As the encoder of encoder-decoder language models bidirectional, it can also examine in-context examples that follow the target question to fill in the masked token. This means that we may position the target question the beginning or middle of a sequence, for example:
Question: 4o Answer:<xxtra id 0> Question: @1 Answer: #1
Question: 9k Answer: as d
Question: 91 Answer:
Question: 90 Answer<extra id 0>... Question: 4k Answer: ag d
Table 2 summarizes the results. We denote the target question's position as "first" for the beginning of the sequence, "random"" for a random position, and "last" for the original setting (S1). Interestingly, placing the target question anywhere other than the last position results
Experimental setup detailed in the Appendix B.1.
4

---

Published as a conference paper at COLM 2024
65.0
ATLAS (XXL)
ATLAS 38 (XL) ATLAS 770M (Large)
62.5
Natural Questonns
TrivinOA, EM
60.0
57.5
55.0
52.5
In-Context Examples
IIn-Contett Examples
Figure 2: Results of ATLAS with different numbers of in-context examples.
Questions, EM
30
S-shot
D-shot
60
TriviaQA EM
25-
50
20
45
15
40
5
10
20
30
40
1
5
10
20
30
40
Retrieved Passages
Retrieved Passages
Figure 3: Results of ATLAS 11B with different numbers of retrieved passages.
in a significant performance drop. Upon examining the generated answers, we observe that when the target question is placed at the beginning or in the middle, the model tends to repeat the answer or generate additional text. For example, for the prompt "Question: What number in Bingo sometimes referred as Heinz varieties? Answerkextta id e> Question: ... The generated text is "57 'Heinz varieties' is term used in Bingo to describe". This indicates that the model does not fully understand and follow the style of in-context examples. Therefore, by default, we position the target question after all the in-context examples.
Effect of Number of In-Context Examples. The number of in-context examples is crucial hyperparameter for in-context learning. Generally, we expect better performance from model with more in-context examples, but there is upper limit due to 1) the maximum con- text length setup, e.g., 512 tokens, during the pretraining process, and 2) the point at which the model has received sufficient examples and cannot gain additional information from more examples. The optimal number of in-context examples also varies between models. For instance, Tr, PaLM (Chowdhery et al., 2023) exhibits better 1-shot performance than settings with more examples, while this is not the case for GPT-3 (Brown et al., 2020).
Figure 2 illustrates the impact of varying the number of in-context examples across different model sizes. Interestingly, the 11B model demonstrates poor performance in low-shot settings, e.g., 1-shot, but improves significantly after 4-shot and 5-shot. Upon examining the generated responses, we find that the model tends to produce answers with more tokens in low-shot settings, while the ground truth typically consists of shorter answers with fewer than 5 tokens. By relaxing the criteria for a correct prediction to include instances where the ground-trute answer a substring the model output, we find that the 1-shot performance surpasses that of the 0-shot setting (38.3 vs 32.1 on NQ).
All models perform well in the 5-shot and 8-shot settings, but their performance does not continue to improve with more in-context examples (e.g., 16-shot). We believe this plateau may be attributed to two factors: 1) the sequence length constraints during pretraining, where the maximum input length to the encoder is set to 384 tokens, and the average input sequence length (excluding passages) is around 130 tokens; 2) the model's ability to learn adequately with 5 or examples, making additional examples less beneficial.
Effect of Number of Retrieved Passages. Figure 3 illustrates the impact of the number of retrieved passages model performance. We observe that for both (+shot and 5-shot

---

Published as a conference paper at COLM 2024
settings, the performance of the models increases significantly with the number of retrieved passages. This highlights the effectiveness of the Iusion-in--ecodeer architecture (Izacard & Grave, 2021) for knowledge-intensive tasks like open-domain question answering, and underscores the importance of pretraining language models with retrieval augmentation. Additionally, the 5-shot performance consistently outperforms the 0-shot setting. This observation further emphasizes the value of providing in-context examples to improve the performance of retrieval-augmented encoder-decoder language models.
3.2 RAVEN: Combining Retrieval-Augmented Masked and Prefix Language Modeling
In $3.1, we observe that retrieval-augmented encoder-decoder LMs exhibit certain ability for in- context learning, which has been overlooked in previ- ous studies. However, there are also some limitations such as unstable performance in low-shot settings, and the fact that providing more in-context examples does not consistently improve performance.
Retrieval-Augmented Prefix Language Modeling
Inpot to Encoder:
Machine learning algorithms build model based on sample data, known as training data, in order to make predictions or decisions without heing explicitly programmed to do so. Machine learning, algorithms ure used in a wide variety of applications, such as in medicine, email filtering. speech secognition. agriculluree and computer vision, where difficult orcextra_ Od_0>>
To learn a better retriever and enhance the bidi- rectional understanding ability of the reader, as demonstrated in Izacard et al. (2023), a practical choice is to pretrain the model with the masked language modeling objective, where the input is a corrupted text with several masked spans placed randomly within the sequence (refer to Figure 1 (left) for an example). However, in testing, based on analysis in $3.1, it is most effective to place the target question after all the in-context examples, with masked token (i.e., <extra_id -i_irriiidd)))___ following the ques- tion (Figure 1, middle)). Thus, there exists mismatch
Passage: ... machine learning models require high quantity of reliable data in order the models ...
Input to Decoder:
Nome
Oulput:
<extra, id Do unfeusible to develop conventional
algorithms to perform the needed tasks.
Figure 4: Retreeval-augmented prefix language modeling.
between pretraining and inference.
To solve this issue, we propose combining retrieval-augmenner masked and prefix language modeling. Specifically, in the first stage, following Izacard et al. (2023), the retriever and reader are trained jointly with retrieval-augmenter masked language modeling. The train- ing objective for the retriever is to minimize the KL divergence KL(PREADER PRETRIEVER) between the passage posterior distribution according to the reader and the passage distribution from the retriever over the top-K retrieved passages, i.e., PREADER( (d) expis(da)/T) where s(-) calculates the dot
exp(log (L dg)))
E exp[log ( d;44)) PRETRIEVER(4)
List exp(s(d;q)//T)
product between the query q and passage vectors, and is hyperparameter. The training
objective for the reader is to maximize the likelihood of the masked spans with retrieved
passages: Σ log p(a; 9,
In the second stage, for each sequence, we mask 10% of the tokens on average at the end of the sequence with the <extra-id_0> token. Then, we use the retriever obtained from the first stage to retrieve relevant passages using the prefix and train the reader to recover the suffix of this sequence with the prefix and the passages as input. An example of input and output for retrieval-augmented prefix language modeling is shown in Figure 4. We can observe that the pretraining objective aligns more closely with prompting strategy in Figure 1. We refer to the model trained with this combined objective as RAVEN.
RAVEN benefits from both the petrieval-augmented masked language modeling, which con- tributes to better reader and retriever, and retrieval-augmented prefix language modeling, which mitigates the gap between pretraining and inference. This design is non-trivial. In Appendix C.1, we verify the effectiveness of it by exploring different training strategies.
3.3 Fusion-in-Contex Learning
In $3.1, we observe that the performance does not further improve with more in-context examples after 8-shot. One major reason for this is the limited sequence length during the
6

---

Published as a conference paper at COLM 2024
Standard In- In-Context Learning
9: 9g 92 40 + passage
encode
91 92023 + passage
encode
concat
decode
9:a; 92 9a
passage
encode
Fusion-in-Contexi Learning
9:a;: 92023 9n
passage
encode
93 44 90
passage
encode
concat
decode
4g
9:04; quaxi
passage n
encode
Figure 5: Standard In-Context Learning vs Fusion-in-Context Learning.
pretraining process, which makes it difficult for the model to handle long sequences during inference. Pretraining models with longer contexts would be potential solution, but it would significantly increase computation cost. Additionally, the maximum input length is also constrained by the maximum sequence length of the retriever, i.e, Contriever, which is based on BERT (Devlin et al., 2019) and has maximum length of 512 tokens.
As alternative, we propose an approach to enable models to learn from more in-context examples without requiring additional training. As described in $3.1, the reader is based on the Fusion-in-Decoder architecture (Izacard Grave, 2021), where multiple passages are retrieved, and each passage, concatenated with the in-context examples and target question, is fed the encoder separately (Figure 5, top). To allow the model to process more in-context examples, we can feed different in-context examples to the encoder with each passage (Figure 5, bottom). In this way, the model can incorporate more in-context examples during its inference process. We refer to this strategy Fusion-in-Conlex Learning (FICL).
In implementation, for a k-shot setting, such as 64-shot setting, to effectively utilize the 64 examples, we randomly shuffle these examples and select !! (e.g., 5) examples in order as the input for the encoder each time. all the examples have been used, we shuffle the 64 exam- ples again. We denote the configuration of FiCL as [k-m], which stands for [k-shot, m-fusion].
3.4 In-Context Example Retrieval
Liu et al. (2022); Rubin et al. (2022); Su et al. (2023) demonstrate that a well-chosen selection of in-context examples can enhance in-context learning. Building on this insight, we propose utilizing the retriever of RAVEN to retrieve in-context examples. Specifically, we use RAVEN's retriever to build an index during the preparation step, and then, during testing, when the model receives input, it could efficiently retrieve in-context examples with its retriever.
By integrating RAVEN'S retriever in this manner, we aim to: 1) automate in-context learning, which is particularly practical for model owners who have database of examples. Without this, users would need to manually provide in-context examples; and 2) optimize the selection of in-context examples, thereby improving in-context learning performance.
Experiments
4.1 Experimental Setup
Following the setup in $3.1, we first evaluate two widely-used open-domain question answering datasets: Natural Questions (Kwiatkowski et al., 2019) and Trivi- aQA (Joshi et al., 2017). Additionally, we conduct case study on long-form question answering using the ELI5 dataset (Fan et al., 2019). Furthermore, we assess the models' language understanding ability using the Massively Multitask Language Understanding
Datasets.
7

---

Published as a conference paper at COLM 2024
FICL
66
Natural Questonn.
ATLAS
TriviaDA, EM
64
30
62
60
28
58
26
56
In-Context Examples
# IIn-Context Examples
Figure 6: RAVEN vs ATLAS. We report the best observed performance achieved with more than eight shots for "> 8".
(MMLU) benchmark (Hendrycks et al., 2021). Detailed information regarding the MMLU evaluation is in Appendix B.5. Other evaluation settings are the same as $B.1.
Baselines. Since both RAVEN and ATLAS (Izacard et al., 2023) are trained starting from T5, we choose ATLAS as a primary baseline for comparison. We also compare our model with decoder-only LLMs such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2023), and LLaMA (Touvron et al., 2023) (in closed-book setting). Additionally, for open-domain QA, we evaluate our approach against REPLUG (Shi et al., 2023) and RETRO (Borgeaud et al., 2022), as well as its improved version RETRO++ (Wang et al., 2023). These models decoder-only language models augmented with retrieval. REPLUG is based on Codex (Chen et al., 2021) and Contriever (Izacard et al., 2022), where the passages are retrieved by Contriever (using ensemble and additional adaptation) and fed directly to Codex. RETRO is GPT model (Radford al., 2019) augmented with transformer encoder encode the retrieved passages. RETRO++ is variant of RETRO that feeds the most relevant retrieved passage into the GPT decoder while providing other passages to its encoder. For MMLU, we also include FLAN-T5 (Chung et al., 2022), an enhanced version of T5 that has been trained large mixture of tasks with instruction finetuning.*
4.2 Open-Domain Question Answering
We choose open-domain QA as our primary evaluation task, as it effectively represents knowledge-intensive challenges and is widely employed in real-world applications.
RAVEN vs ATLAS. Figure 6 and Table 3 present the exact match (EM) scores for ATLAS and RAVEN on the NQ and TriviaQA datasets. Both the and 11B RAVEN models significantly outperform ATLAS. For instance, on TriviaQA, RAVEN 11B achieves an improvement of 8.8%, 30.7%, and 2.8% the 0-shot, 1-shot, and few-shot settings respectively, compared to ATLAS 11B. Furthermore, the performance of RAVEN increases steadily with the number of in-context examples, while the performance ATLAS experiences a substantial decline in low-shot setting,, demonstrating the effectiveness of RAVEN across various shot settings.
Fusion-in-Context Learning. We also report the results of models with Fusion-in-Conterd Learning (FiCL) in Table 3. For both ATLAS and RAVEN, FiCL contributes to approximately a 1% improvement, which is not attainable by standard in-context learning, where perfor- mance does not further improve (or even decreases) with more than 8 in-context examples. This demonstrates the superiority of FiCL for enabling models to learn from more examples.
Comparison to Other Models. In Table 3, we further compare RAVEN to other baselines. On NQ, RAVEN'S zero-shot and one-shot performance surpasses all the baselines, including PaLM, even though RAVEN 3B has 180 times fewer parameters than PaLM 540B. The zero-shot performance of RAVEN on TriviaQA is also on par with PaLM 62B, Furthermore, RAVEN'S zero-shot performance significantly exceeds that of both RETRO and RETRO++, which are retrieval-augmenter language models of similar scale.
In the few-shot setting, with FICL, RAVEN achieves performance comparable to GPT-3 175B and PaLM 62B. However, there remains a gap between RAVEN and the larger PaLM 540B
Implementation details described in Appendix B.2.
8

---

Published as a conference paper at COLM 2024
Natural Questions
TriviaQA
O-shot
1-shot
few-shot
0-shot
1-shot
few-shot
GPT-3
13B
7.8
13.7
21.0 (64)
41.8
51.3
57.5 (64)
GPT-3
175B
14.6
23.0
29.9 (64)
64.3
68.0
71.2 (64)
PaLM
8B
8.4
10.6
14.6 (5)
39.5
48.5
47.2 (5)
PaLM
62B
18.1
23.1
27.6
67.3
72.7
70.1 (3)
PaLM
540B
21.2
29.3
39.6 (64)
76.9
81.4
81.4 (1)
Codex
175B
40.6 (16)
73.6 (16)
LLaMA
7B
16.8
18.7
26.1 (64)
50.0
53.4
57.6 (64)
LLaMA
65B
23.8
31.0
39.9 (64)
68.2
71.6
73.0 (61)
Retrieval-Augmented Language Models
Codex Contriever
175B
44.2 (16)
76.0 (16)
Codex REPLUG
175B
44.7 (16)
76.8 (16)
Codex REPLUG LSR
175B
45.5 (16)
77.3
RETRO
9.5B
8.9
36.0
RETRO++
9.5B
25.8
48.3
ATLAS
3B
23.7
25.1
28.4 (5)
54.3
55.5
61.1 (5)
ATLAS FICL
3B
29.6 [64-5]
62.0 [64-5]
ATLAS
11B
26.7
21.3
31.3
56.9
35.5
63.9 (8)
ATLAS FICL
11B
32.0 [64-8]
64.9 [64-8]
RAVEN
3B
29.3
31.7
31.4 (5)
62.4
63.2
62.6 (S)
RAVEN + FICL
3B
32.8 [40-1]
63.6 [40-1]
RAVEN
11B
29.6
31.4
32.7
65.7
66.2
66.7 (5)
RAVEN FiCL
11B
33.5 [64-3]
67.3 [64-5]
Tr, Pal 1-shot performance surpasses other settings. We follow original poper report I-shot result.
For other models, we select the best k-shot (k E (2,3,4,3,8, 16)) performance or report the number in the original paper.
Table 3:
Results on NQ and TriviaQA. Since the performance varies significantly depending on the capability of the base model, the results from models other than ATLAS should only be used for reference to gauge the position. And assume RAVEN can achieve significant performance improvement when based on stronger base model.
Natural Questions. EM
S-shot D-shet
30
TriviQA, EM
60
25
50
15
10
20
30
40
1
10
20
30
40
Passages
Retrieved Passages
Figure 7: Results of RAVEN 11B with different numbers of retrieved passages.
and Codex 175B models. Nevertheless, given the considerably smaller scale of RAVEN in comparison to PaLM and Codex, its performance can be considered impressive. The performance of RAVEN may be further improved if it is built upon a larger model, in which case its few-shot performance is likely to surpass that PaLM and Codex.
Effect of Number of Retrieved Passages. Figure 7 illustrates the effect of the number of retrieved passages. As the number retrieved passages increases, we observe a significant performance improvement of RAVEN 11B in both the 0-shot and 5-shot settings.
In-Context Example Retrieval. suggesss using RAVEN'S retriever for in-context example retrieval. Results in Table 4 show that this approach improves RAVEN'S few-shot results, especially on NQ where a ~00% improvement is observed. This indicates the positive impact of incorporating more relevant in-context examples.
Additional Results. We conduct an ablation study of different training strategies in Appendix C.1 and provide case study on long-form question answering in Appendix C.2.

---

Published as a conference paper at COLM 2024
NQ
TQA
1-shot
5-shot
1-shot
5-shot
3B
+9.1
+11.6
+0.0
+1.6
11B
+9.8
+11.1
-0.5
+1.0
Table 4: Performance improvement of RAVEN with In-Context Example Retrieval.
4.3 MMLU
Table 5 summarizes the results (accuracy) on Massive Multitask Language Under- standing (MMLU). We find that the zero- shot performance of RAVEN is impressive, surpassing the few-shot performance of GPT-3 175B and being slightly worse than PaLM 62B, despite having significantly smaller number of parameters. Further-
0-shot
1-shot
5-shot
GPT-3
13B
26.0
GPT-3
175B
43.9
PaLM
8B
25.3
PaLM
62B
53.7
PaLM
540B
69.3
15
3B
25.7
more, with the same number of parameters, the performance of RAVEN is far superior to T5. Additionally, even without instruction finetuning, RAVEN achieves performance comparable to FLAN-T5, model finetuned on large collection of tasks. We expect fur- ther improvement RAVEN by applying instruction tuning well and leave it for future study.
T5
11B
25.9
FLAN-15
3B
52.4
FLAN-T5
11B
55.1
ATLAS
3B
43.7
36.9
38.5
+ FiCL
3B
42.6 [40-1]
ATLAS
11B
47.4
45.3
44.2
+ FICL
11B
48.0 [0-1]
RAVEN
3B
40.0
40.4
+ FICI
3B
44.5 [64-5]
RAVEN
11B
48.9
49.2
48.7
Interestingly, with standard in-context learning, the few-shot performance of RAVEN is worse than zero-shot, possibly due to the longer questions and answer options in MMLU causing context length issues the 5-shot setting, Also, the one-shot setting, since MMLU is multiple-choice QA task, providing only one example might introduce bias in the model's prediction, favoring specific option. However, with Fusion-in-Contex. Learning, the performance improves significantly, leading to better few-shot performance for the 11B model compared to its zero-shot performance, further demonstrating the effectiveness of FiCL.
FICL
11B
50.5 [40-1]
Table Results on MMLU.
Conclusion
In this study, we have delved into the in-context learning ability of retrieval-augmented encoder-decoder language models. We commenced with comprehensive analysis of the models in the literature and subsequently developed our model based the analysis. Our extensive experimental results demonstrated that our model significantly outperforms previous models and achieves results on par with some of the most advanced language models, even with substantially fewer parameters. These findings highlight the potential of retrieval-augmented encoder-decoder language models the realm of in-context learning.
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saf- fron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela I'aganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Čsaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume
10

---

Published as a conference paper at COLM 2024
162 of Proceedings of Machine Learning Research, pp. 2206-2240, PMLR, 17-23 Jul 2022. URL https://proceediing wwt.rpees//vee/boreeeeee
html.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL htttt:::///roceeeddngg nesrssnnuripp neurips. cc/paper files/ tates/paper/2228/ pdd.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. Xiv preprint ar arXiv:2303.12712, 2023.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meefing of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: (0.18653/v1/P17-11
URL
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar 0a0:2107.033744 2021. URL https://arxivv http:///aakivvorrabb/////// wrgg/bb//2///// 033744.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Isvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri- donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language mod- eling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. URL orgo http//appr//v///2212p 1144. html.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetume language models. arXiv preprint arXix:2210. 11416, 2022. URL hitpps///arxe org/abs/ 11416.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v11 N19-1423, URL https: htattp::/aaclanthollog hww.tthoooooooogg/11114323
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.002.. 2022. URL https: //axxiv wrggabs//3/1...oo.. 00234.
11

---

Published as a conference paper at COLM 2024
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. EL15: Long form question answering. In Proceedings of the 57th Anmual Meet- ing of the Association for Computational Linguistics, pp. 3558-3567, Florence, Italy, July URL 2019. Association for Computational Linguistics. doi: 10.18653/v1/P199 httpa//caannnoooooooorr///-----e
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL http:///ppeeeveeeeeeeeeeeeee d7KB 3GmQ.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computaaionnll Linguistics: ACI. 2023, pp. 1049-1065, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/vl 2023. 1123.iindins--cl..6 URL https: http:///aaaaaallor///// hww.hhhhlllo..oggg//..e org 2023. Findings-acl.67
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang.
Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2038-2047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:/ hatpss////laathlooyyyy wwwpgg.ooooooooooooooooooooooer. 0022. madinggs-mmlpp 148.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 374-880 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/ 202/ mai.. 74. URL
org/2021 wea.lmaa.clmmmm1mm..mmm
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yuu Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, html.
24(251):1-43, 2023. URL http:// http://mmrr.ooooooo org httppppers//22//2222
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. TemporalWiki: A lifelong benchmark for train- ing and evaluating ever-evolving language models. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pp. 6237-6250, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational MocnaalLingune Linguistics. URL momlp-main... 418.
https://aclannhhoogg.oor///
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1 /P17-1114 URL
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https: petpp://oppat http://ppprreeee../eeee/e. http://///eeeeeeee/eeeeerrrrrrr f Hk1BjCEKVH.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search -rredictt: Composing retrieval and language models for knowledgeiintesivee nlp. ar preprint xrXim:2212.14024 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina
12

---

Published as a conference paper at COLM 2024
Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tad //ccl a 00276. URL
htwp:ttthooooooooooo/oooo org/Q1 1026.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmente generation for knowledge-intensive nip tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 9459-9474. Curran Associates, Inc., 2020. URL https::/ hewpeedddiinnnnnniii.seee.e.eeee wwwr.ps.cc/papprrrrreeee http:/pppp_ffleesee pat.le///pper//22e File//bb43220226F78800 10886e6267555778855--55-55
pdf.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning In- side Out (DeelJO 2022): The 3rd Workshop on Knowleegge Extraction and Integration for Deep Learning Architectures, pp. 100-114, Dublin, Ireland and Online, May 2022. As- sociation for Computational Linguistics. doi: (118653/v120222.deee https://aclanthologgy org/2022 deelio-1.10.
URL
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint XfXo:2301113668, 2023. URL
orgg 13688.
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL ctts:://pperrrviiw.eee/ffrrorrm Bkg6RiCqY7.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith.
Time waits for no one! analysis and challenges of temporal misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5944-5958, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653 /22/22222nnaaalaa m33.........35..... URL https://aclanthology www.:/thhholgoooooe/eeeeeeeeee org/ naacl-main 435.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXii:2212.10511, 2022. URL
/rgp/b/////.....ob/ 10511.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-tan Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 www/022.acllloog..g.....o 244. URL https: / http://ccaaantooo...rrrrrr
acl -longlon.... .44.
OpenAl. Chatgpt: Optimizing language models for dialogue. OpenAl, 2022.
OpenAL. Gpt-4 technical report, 2023.
Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Burch. Bidirectional language models are also few-shot learners. In The Eleventh International Conference on Learning Representations, 2023. URL https:// net/ forum))dd=wFF337bzudd
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v vv/ 1/2221 manacl-miin2200 URL ittps://acke hathh/aacaanoooooooooooooooooooe .oryy.rrgggoooo 2rg/20211... macacc-main.. 200.
13

---

Published as a conference paper at COLM 2024
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? arXiv preprint ar 06X::2230.00447, 2023. URL https://arxix: https://arxxw.oor//bb///23/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8),9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., (1), jan 2020. ISSN 1532-4435. URL https: //dl: acm. org/ ////////////
/ /bs// 10. 5555/3455716 3455856.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655-2671, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10.18653/ /t/22222.aaclmmai... URL https:://allannhoolog/ wwwrg//222..oo naacl - 191.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chh- ablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Baw- den, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero- shot task generalization. In International Conference on Learning Representations, 2022. URL wwwttoennnteeeeeeee nett/forummdddd///sooddd.. idd=v/bbD0011...
https://openreet
Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. Nearest neighbor zero-shot inference. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 3254-3266 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL emnlp-main. 214.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. ar Xiv preprint (iXx:2301.12655 2023. URL
http://aaxxi..oor/////////////// 12652.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model. ar Xiv preprint trXie:2201.11990, 2022. URL httpp:///raa org/abs/ 2201. 11990.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Fleventh International Conference on Learning Representations, 2023. URL https::/openrreviieee
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2023. URL 6ruVLB727MC.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXio preprint 2rX.::3302.13971, 2023. URL. https: wwwaariiv.ooggbbbbeee 13971.
14

---

Published as a conference paper at COLM 2024
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. Shall we pretrain autoregressive language models with retrieval? comprehensive study. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 7763-7786, 2023. URL https://aclanthoolog) org/2023. emnlp-main. 482.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL
net/forummidd-gEZrGGooo
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 41. URL https: haap://hoolll org 2221. wwa.c-mmiinnn..mm 41.
Qinyuan Ye, Iz Beltagy, Matthew E Peters, Xiang Ren, and Hannaneh Hajishirzi. Fid-icl: A fusion-in-decoder approach for efficient in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pP. -185, 2023. URL httpt:tt//cclaatttt////////
org/2023
1all-long.455.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does ChatGPT fall short in providing truthful answers? In I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models, 2023. URL https: /opnn//veeeweeeeee///eee net net/forummidwwoae 4LCW9P.
15

---

Published as a conference paper at COLM 2024
A Limitations and Broader Impact
While the performance of RAVEN is impressive considering its scale and training budget, there are also some limitations. One limitation arises from the constrained context length inherent to the base models (i.e., 15 or ATLAS) we employed. This restriction poses challenges to the scalability of in-context learning, especially as the number of in-context examples increases. While our Tusion-in-Conterd mitigative approach to this constraint, an alternative and possibly more optimal solution might involve extending the context length. This would be particularly beneficial for tasks requiring extensive inputs.
Learning (FiCL) strategy does offer
Furthermore, when compared to some of the prevailing decoder-only language models, particularly those exceeding 100B parameters, the models deployed in our research might appear relatively diminutive in scale (in terms of both the number of parameters and the amount of training data). Our endeavor partially seeks to catalyze further investigations into more powerful encoder-decoder models.
Nonetheless, the insights and methods proposed are transferable and have the potential to enhance other models, including those that are domain-specialized such as mT5 (Xue et al., 2021) and UL2 (Tay et al., 2023). Future work focusing on scaling up the model, applying these methods, and further studying its in-context learning ability is encouraged. Drawing on the benefits of scaling up and combining this with our proposed approaches, we believe that there is potential develop even more powerful retrieval- augmented language models in the future. Another promising future direction is exploring how to combine the Fusion-in-Decode architecture with existing decoder-only language models. By doing so, we can harness the advantages of both architectures employing bidirectional architecture to effectively encode retrieved passages for the most powerful decoder-only LLMs.
more powerful,
B
Additional Experimental Details
B.1 Experimental Setup for $3.1
We select two widely-used datasets in the domain of open-domain question answering: Natural Questions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (TQA) (Joshi et al., 2017). To assess the performance, we follow the previous work (Izacard et al., 2023) to employ the standard exact match (EM) metric. For the few-shot settings, we follow Brown et al. (2020) to evaluate each example in the test set by generating in-context examples through randomly sampling instances from the respective task's training set. Following Izacard et al. (2023), we use index composed of December 2018 Wikipedia dump for NQ and an index composed of December 2021 Wikipedia corpora for TriviaQA. We retrieve 40 documents by default. We test the checkpoints released in the official repository of Izacard et al. (2023)5, covering sizes of 11B (XXL), 3B (XI.), and 770M (Large).
B.2 Training Details
We train two versions of RAVEN: 3B and 11B. To isolate the effect of training variance with masked language modeling, we initialize both the retriever and the reader of the models with the weights of ATLAS (3B and 11B) and continue to pretrain the model with prefix language modeling. To isolate the effect of retrieval, we do not update the retriever during the training process for prefix language modeling. We pretrain the reader using the December 2021 Wikipedia corpora preprocessed by Izacard et al. (2023), where the index also constructed using the same corpora. In accordance with Izacard et al. (2023), we retrieve 20 passages for each masked sequence (excluding passages identical to the original sequence). Both the 3B and 11B models are trained for 5,000 steps, using AdamW optimizer (Loshchilov & Hutter, 2019) with a batch size of 64. We employ a learning rate
of 4 x 10 5 for the 3B model and 1 x 10 5 for the 11B model, with linear decay and 100
16

---

Published as a conference paper at COLM 2024
warmup steps. All the models are trained NVIDIA A100 GPUs (80 GB). For the 3B model, we utilize GPUs, whereas for the 11B model, we employ 32 GPUs. The prompt used for prefix language modeling is detailed in Appendix B.3. During testing, we default to retrieving 40 documents for all tasks. The prompts used can be found Appendix B.4 and Appendix B.5.
B.3 Retrieval-Augmentte Prefix Language Modeling
In alignment with the pretraining of ATLAS, design the prompt for prefix language modeling as
(prefix)<extra.idd.> title: [title) context: (text}
where {prefix) represents the prefix of an input sequence. The ftitle) and (text) elements are retrieved by the model's retriever using the prefix as query. Here, (text} signifies the retrieved passage, while (title) denotes the corresponding article and section title of the passage. The model is trained to generate
<extra id 0- (sufffix)
where {suffix} is the suffix (masked by <extra.id 0>) of the input sequence.
B.4 Open-Domain Question Answering
In accordance with pretraining, we use the following prompt for open-domain question answering:
Question: (question] Answer::extra.id.ee title: For example,
{title} context: (text)
Question: In which country was the first permanent bungee jumping site situate? Setwer:<exta...e 0> title: Bungee jumping: Modern sport context: first permanent commercial bungee site, the Kawarau Bridge Bungy at the Kawarau Gorge Suspension Bridge near Queenstown in the South Island of New Zealand. Hackett remains one of the largest commercial operators, with concerns several countries. Several million successful jumps have taken place since 1980. This safety record is attributable to bungee operators rigorously conforming to standards and guidelines governing jumps, such double checking calculations and fittings for every jump. As with any sport, injuries can still occur (see below), and there have been fatalities. relatively common mistake in fatality cases is use cord that
B.5 MMLU
MMLU comprises 57 multiple-choice question answering datasets that span various do- mains, including elementary mathematics, US history, computer science, and more. For the evaluation MMLU, we report the accuracy and use index composed of December 2021 Wikipedia corpora. We follow Izacard et al. (2023) to apply the "de-biased" inference. Specifically, during inference, we execute four forward passes, each corresponding to cyclic permutation of the answer letter-option assignment within the question. For instance, the answer option designated to letter 'A' is shifted to 'B', 'B' 'C', 'C' to 'D', and 'D' to 'A'. The final prediction is obtained by summing up the probabilities from these four forward passes.
We design the prompt in the following format:
Question: {question} Options: {candidate answers) Answer:<extra.id.g title: [title) context: {text}
For example,
17

---

Published as a conference paper at COLM 2024
Natural Questions 1-shot
TriviaQA 1-shot
0-shot
5-shot
O-shot
5-shot
ATLAS
3B (Mask)
23.7
25.1
28.4
54.3
55.3
61.1
ATLAS
3B (Mask, 5k more steps)
22.9
22.5
28.1
50.8
50.1
61.1
RAVEN
3B (Prefix)
24.8
29.1
30.1
55.4
61.4
62.3
RAVEN
3B (Mix)
25.1
28.4
30.9
56.1
61.4
62.2
RAVEN
3B
29.3
31.7
31.4
62.4
63.2
62.6
Table 6: Results of ATLAS and RAVEN trained with different strategies.
Question: Over time, non-volcanic mountains can form due to the interaction of plate boundaries. Which interaction is most likely associated with the formation of non-volcanic mountains? Options: (A) continental plates colliding with continental plates (B) continental plates separating from continental plates
(C) oceanic plates colliding with oceanic plates (D) oceanic plates separating
from oceanic plates knswer:<ettra.id. title: ... context:
Given that many questions in the MMLU benchmark are quite lengthy, concatenating in- context examples (questions and candidate answers) with the target question in few-shot setting is likely to exceed the maximum input length. To mitigate this, we only sample examples with question lengths of fewer than 50 tokens to use as in-context examples.
C
Additional Results
C.1 Ablation Study
We conduct an ablation study by training ATLAS and RAVEN with different pretraining strategies. First, to isolate the effect of more training steps of RAVEN, we also train ATLAS for 5,000 more steps using the masked language modeling objective. Results Table 6 (row 2) show that the performance does not improve, indicating that the performance improvement of RAVEN compared to ATLAS is not simply due to training for more steps.
Second, to verify the effectiveness of RAVEN'S training strategy (i.e., first masked language modeling, and then prefix language modeling), we train two variants of RAVEN, starting from the T5-Im-adapt checkpoint", which is the checkpoint that ATLAS starts from. For the first variant, we use the same prefix language modeling objective of RAVEN. For the second variant, we train the model with mixture of masked and prefix language modeling, Specifically, we construct corrupted texts by both masking 15% spans in the sequence (same as ATL.SS) and replacing the suffix with special mask token <extra_id id_99> (used in testing). We train the model for 10,000 steps and update the retriever and refresh the index during training with the optimal strategy described in Izacard al. (2023). Table 6 (RAVEN" in row 3 and 4) summarizes the results. We find that the performance of these two variants is superior ATLAS, but inferior to RAVEN when trained using the strategy described in $3.2. An explanation for this is that, by training with masked language modeling first, the model can achieve better language understanding ability and is equipped with a more effective retriever (as emperically verified in Izacard et al. (2023)). Subsequently, by training with prefix language modeling, the mismatch between pretraining and inference is mitigated, resulting in improved zero-shot and few-shot performance.
C.2 Long-Form Question Answering
Table presents some example outputs of ATLAS and RAVEN 11B on long-form question an- swering. The questions are sampled from the ELI5 dataset (Fan et al., 2019). An examination of these results reveals that ATLAS typically generates concise answers, while the output from RAVEN generally encompasses more information. This is predictable outcome given
onttps://hugiigtee wwwp:ooogg/oo//teoooooooococccoe (aapglletttx1llee 1t--tt5--1--mmmmmmmm-mm- Idapt
18

---

Published as a conference paper at COLM 2024
that ATLAS is pretrained solely with masked language modeling, where each masked span usually contains only a handful tokens. Besides, while RAVEN's answers are not always entirely accurate, they generally exhibit higher quality compared to ATLAS. Furthermore, the use of Fusion-in-Contex Learning in RAVEN appears to contribute to a more coherent and informative generation.
19

---

Published as a conference paper at COLM 2024
Question
How does a painting like "When Will You Marry" sell for $300 million dollars when it doesn't 't look like anything spectacular
Human
When it comes to a lot of art, context is more important than the actual material. Rarity, historical context etc. make thing more valuable than its actual aesthetic properties. For example, some shitty buggy SNES or N64 games will sell for huge amounts they are well preserved and in the original packaging etc, even when any high school student nowadays could write a better game in week.
ATLAS
1-shot
It's masterpiece.
RAVEN
1-shot
It's not the painting that sells for $300 million dollars, it's the story behind the painting.
RAVEN
FICL
It's not the painting that sells for $300 million dollars, it's the reputation of the artist.
Question
When the society goes fully automatized, what will be the "normal" people needed for?
Human
Call centers... 1 am being totally serious. your Sony TV is being completely fabricated by machines, shipped by machines, sold on amazon and delivered by machines, one of the few value adds companies will be able to deliver is GOOD customer service. So calling that 1-800 number and being instantly connected to knowledgeable, friendly, human being who is going to patiently help you fix whatever issue you have (FYI... you forgot to plug it in). >> FL What will the society go tully automatized? >
ATLAS
1-shot
RAVEN
1-shot
The ones who will program the robots.
RAVEN
(101
The "normal" people will be needed to program the robots.
Question
Why a 2 liter of Coca-cola $1, oz bottle $1.49 and the same size bottle of water $1.99?
Human
Because people will pay. Plain and simple. Capitalism its best.
ATLAS
1-shot
The price of the product is based on the cost of the packaging.
RAVEN
1-shot
The cost of packaging is proportional to the size of the container.
RAVEN
FiCL (401)
The cost of packaging the largest portion the cost of a product. The larger the container, the less packaging is required.
Question
Matter Cannot Be Created or Destroyed... How trees form?
Human
Say you have box of Legos with 500 pieces. With those 500 pieces, you build a bulldozer. Then one day, you decide to rearrange those pieces from a bulldozer to a monster truck. It's still the same 500 pieces, just in different layout. This is how trees are formed. The matter is just rearranged from other objects. Soil, nutrients, water, and even air. This is called "Conservation of Mass." EDIT: added air. Thanks. :)
ATLAS
1-shot 1-shot
Trees are made of carbon and oxygen.
RAVEN
Matter can be created or destroyed by combining two more different types of matter.
RAVEN
FiCL [10-1]
The tree is a living organism. It is made up of matter. The tree is made up of carbon, hydrogen, oxygen, and water.
Question
the flu shot contains a non-active version of the flu, then why are "flu like symptoms" common for the days following?
Human
the non-active proteins trigger an immune system response which can involve some inflammatory system responses. once the immune system is activated for that infection your body will react more readily if the actual infection gets into you.
ATLAS
1-shot
The flu shot contains live virus particles.
RAVEN
1-shot
The flu shot contains live attenuated virus. This means that it is not able to infect the body, but can still cause some side effects.
RAVEN
[ICL
The flu shot contains a weakened version of the virus. This weakened virus is able to trigger an immune response in the body. This immune response can cause flu like symptoms.
Table 7: Example outputs of ATLAS and RAVEN 11B on long-form question answering.
20


Reviews:
[{"review_id": "3WsejPgTLj", "reviewer": "Reviewer_A7yJ", "summary": "This paper studies the problem of retrieval and in-context learning for encoder-decoder languages models. Specifically, they find limitations of prior methods that arise from being not truly few-shot in that they require additional fine-tuning. This work then proposes a new model (RAVEN) that obviates the need for fine-tuning by modifying the prompt/encoding format. The retriever part is trained by a KL objective while the reader is trained using masked language modeling for prefix-based LM. Additionally, this work introduces “Fusion-in-context learning” (FiCL), which distributes a high number of examples (e.g. 64) across multiple embeddings that get encoded. Both RAVEN and FiCL are shown to be effective on Natural Questions, TriviaQA, and MMLU compared to the prior state of the art model. \n\nThe main strengths are that the method appears to work well and also are well-suited solutions to the disadvantages of prior work (fine-tuning); in some sense, this work modernizes past work with more attention to prompts and ICL. The main weakness, which could be a result of my own misunderstanding, is the comparison against baseline, ATLAS, does not seem to correspond with the original paper (more explained below).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the innovation and soundness of our methods, as well as the extensiveness of our experiments.\n\n>Q1: Section 3.1\n\nThank you for your feedback! We believe Section 3.1 is important as it is the ***first*** analysis of the in-context learning ability of Retrieval-Augmented Encoder-Decoder LMs. This analysis inspired the design of RAVEN and FiCL.\n\nYes, Appendix C1 (first/last/random) can also be treated as different prompting strategies. Regarding the design of “putting the retrieved docs into the prompt either”, one reason we don’t apply it is exactly due to the limited context length. Another reason is that this prompting strategy (i.e., q1 + doc1 + q2 + doc2 + … + q0 + doc0 => masked tokens) does not align well with the objective of pretraining (i.e., corrupted text + doc => masked tokens). Designing a pretraining objective for this prompting approach (as well as extend the context length) is a promising future direction.\n\n>Q2: masked spans\n\nIn the first training stage, masked spans are determined similarly to Atlas, with 15% of tokens randomly masked. In the second training stage, we mask 10% of the tokens on average at the end of the sequence, based on preliminary experiments. The p_reader is related to the Likelihood Distillation described in previous literature. We will add more explanations.\n\n>Q3: dynamic retrieval\n\nFollowing ATLAS, we use Contriever (Izacard et al., 2022), where a single encoder embeds both queries and documents (different from DPR). Since we use a single encoder, embeddings for similar/related queries should be close. Therefore, we can simply use the same retriever.\n\n>Q4: prompting vs finetuning\n\nYes, we report the few-shot prompting results, not fine-tuning. Actually, fine-tuning for RAVEN can also yield better performance, but it’s not the focus of this paper since we want to disregard it, considering it’s not the common few-shot usage nowadays.\n\n>Q5: FiCL\n\n- For standard ICL, we perform hyperparameter search in {2,3,4,5,8,16}; for FiCL, we search in {[40,1], [64,5], [64,8]}. We follow previous literature (e.g., GPT-3, PaLM) by evaluating on the dev/test sets directly.\n- According to our results, FiCL [k,m] is (almost) always better than m-shot without FiCL. Settings like k=16 or m=1 also show FiCL outperforms standard ICL.\n\n>Q6: reference\n\nThis is explained in the previous paragraph: “...the computation cost grows linearly with the number of retrieved passages…(Izacard & Grave, 2021).”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method, RAVEN, appears to work well across multiple tasks and addresses the concern that ATLAS needs additional finetuning.\n* The addition method, FiCL, is a novel method to more efficiently combine the notion of in-context learning to encoder-decoder models. The fact that this method works so well is a cool result.\n* The experiments are extensive and cover lots of ablations like model size, prompting strategy, multiple shots, etc. Overall, they are convincing.", "reasons_to_reject": "* The paper presentation is a little bit confusing; the first contribution of “thorough analysis” does not seem too conclusive.\n* Some technical details were confusing, especially as there is a strong reliance on the reader understanding all the technical details and hyper parameters of ATLAS first. \n   * In particular, the ATLAS baselines reported here are lower than the numbers in the ATLAS paper.", "questions_to_authors": "* Section 3.1 feels out of place relative to the rest of the paper. The conclusions of 3.1 are that prompt 1 is better than prompt 2 and that going up to 8-shot is sufficient given the small context sizes (512). Both of these findings could belong in an appendix in favor of clearer explanation and analysis of the main method RAVEN and secondary method contribution FiCL, both of which are, in my opinion, more convincing and stronger results. \n    * To nitpick a little more, strategy 1 and 2 are not the only strategies: C.1. actually outlines 3 (first/last/random) more, and none of these consider putting the retrieved (maybe limited to top-1 title only due to seq. size) docs into the prompt either -- which would be more akin to what RAG-like approaches would do to fully mimic the data point for ICL. \n    * I stared at Figure 1 for a long time at the beginning trying to understand what it had to do with encoder/decoder architecture, expecting it to be a compare/contrast of ATLAS and RAVEN, but later on realized it was unrelated to 3.2.\n\n* This might be a naive question: how are masked spans determined? (i.e. what constitutes a span in this model?) If these decisions follow those of ATLAS, perhaps it should be made even clearer in Sec 3.2 that a lot of the modeling objectives/decisions come from ATLAS. e.g. How does p_reader differ from ATLAS's p_attn?\n\n* The dynamic retrieval confused me a lot. The retriever is retrieving text documents, and the embeddings are embeddings of the documents. The few-show examples are (q, a) pairs, which is a different \"type\" than documents. How, or why, should we use the same retriever? And does this retriever get trained with an auxiliary objective somehow to retrieve good (q, a) pairs?\n\n* ATLAS's numbers in their paper are because they fine-tuned their model further -- is that why the scores reported in this paper are lower than the ones in their paper? Since one of the objectives of this work was to fix some of the shortcomings of ATLAS, it should probably be also compared to the models like ATLAS + finetuning. If it comes close but doesn't beat ATLAS + finetuning yet, then that would be useful to know as practitioners can be informed about which model to use based on their compute budget, etc. If it surpasses ATLAS+finetuning, that would be a stronger result.\n \n* I would have liked to see a little more discussion around FiCL. The tables are picking the best numbers based on the [k, m] hyperparameters\n    * Are [k, m] picked based on a validation set?\n    * What are the tradeoffs between higher [k, m] and no FiCL? Is FiCL always strictly better than few-shot, or if k (or m) is small, would it still be preferable to do few-shot without FiCL?\n\n* \"Encoder-decoder LMs are more efficient at multiple (e.g. 40) retrieved passages\" -- this seems to be missing a citation?", "extra_scores": {}}, {"review_id": "Xa6ESlSm7d", "reviewer": "Reviewer_SfUy", "summary": "The paper investigates to combine retrieval augmentation, in context learning, and encoder-decoder models. After establishing the primary grounding literature (e.g., Atlas), the paper begins by presenting two strategies for in-context learning + retrieval augmentation + encoder-decoder LMs. Provided sufficient in context examples (4 or 5 more depending on QA set), one strategy shows improvement over a zero shot Atlas baseline.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting the solidity of our methods and experiments.\n\n>W1: novel insight\n\nOur methodologies stem from an in-depth understanding of existing limitations and how to overcome them, as discussed in Section 3. The proposed methods are designed carefully based on novel insights and comprehensive analysis (see our response to Q1 below as a simple example).\n\n1. There might be a misunderstanding regarding Section 3.2. Our design differs from previous work by combining retrieval-augmented masked and prefix language modeling to enhance in-context learning. In Section 3.3, we also propose a novel prompting strategy named FiCL that further boosts performance.\n\n2. We don’t understand this comment. Strategy 1 puts all the input into the encoder. Should \"decoder\" in your comment be \"encoder\"?\n\n>W2: error analysis\n\nGreat suggestion! Briefly, for short-form QA, RAVEN achieves higher answer accuracy due to improved language modeling and better utilization of retrieved passages. For long-form QA, RAVEN is able to generate more reasonable and informative responses (examples in Appendix C4). We will provide more error analysis in the final version.\n\n>W3: MMLU\n\nThe gain is more significant for factual-seeking questions, where retrieval is crucial, but the improvement is relatively smaller for tasks such as answering math questions.\n\n\n>Q1: variant of strategy 1\n\nWe designed strategy 1 to align closely with the pretraining objective. The alternative prompting strategies suggested do not match this objective.\n\nIn retrieval-augmented prefix language modeling, the order is (prefix, <extra_id_0>, passages). strategy 1 aligns with this order, i.e., (examples + query, <extra_id_0>, passages).\n\n- If the passage precedes q0 (a typo by the reviewer?), the order is (examples, passages, query, <extra_id_0>).\n\n- If the passage is before all the QA pairs, the order is (passages, examples + query, <extra_id_0>).\n\nBoth alternatives mismatch pretraining, where passages are not after <extra_id_0>, thus are unlikely to perform well.\n\n>Q2: comparison\n\nAs shown in Tables 2 and 4, we also compare our model to other retrieval-augmented LMs such as RePlug, Retro, and Retro++, as well as several popular LMs. Performance varies significantly depending on the base model's capability. But in general, when the capability of the base model is close, RAVEN can significantly outperform the baselines.\n\nThank you for your suggestions. We will take into account all of them in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper’s main strength is demonstrating that with a couple modifications to the train/inference procedure, that it outperforms Atlas by a respectable margin.\n2. The paper proposes a method that does not (at least on the surface) seem overly compute intensive, making it easier to apply.\n3. The paper looks solid in methodology, the methods are clear enough to understand and the experiments seem executed well.", "reasons_to_reject": "1. The paper overall seems too incremental and this isn’t helped by lack of analysis beyond headline performance numbers. While there is a nuanced difference between simple yet powerful methods based on novel insight and incrementally building on existing work, I feel the paper falls into the latter category.\n    1. In 3.2, the main difference between prior work seems to be possibly expanding the number of concatenated examples and instead of repeating the same, shuffling them (compared to Izzacard 2023, it looks like the KL divergence minimization objective is similar/same modulo attention difference).\n    2. Strategy 1 in 3.1 seems like the simplest possible way (ie would be a good baseline) to do retrieval augmented QA with an encoder-decoder model (put QA pairs for context learning in decoder, Q + retrieved passage + blank answer in decoder).\n2. While there are a couple demonstrative examples, the paper does not include much analysis beyond headline performance numbers (or performance as number if IC examples varies). It would have been nice to see something like an error analysis that convincingly argues that method fixes what it aims to fix.\n3. Along similar lines, although there was deeper discussion of two QA benchmarks, the discussion of MMLU was quite short. For example, I would be interested in knowing if the gains are uniform or from specific tasks in MMLU.", "questions_to_authors": "1. Did you try a variant of strategy 1 in Figure 1, where the passage either precede the A0, or the passage is before all the QA pairs?\n2. The tables primarily compare to Atlas, are there other recent models that are worth comparing to?\n\nSuggestions that didn’t influence my scores\n1. I realize that LLMs are relatively new, but aside from datasets, the literature cited is almost exclusively 2020 or after, surely there is relevant literature to draw from before that.\n2. I’m not certain if COLM style is different, but at least for many other venus figures (ie Figure 3, Table 4 should be at the top of the page (unless they are preceded by other tables/figures).\n3. Figure 2: The y-axis scales should show where the red lines for Atlas 11B bottoms out, it looks truncated right now. Same with Figure 5\n4. In 3.1, since you discard strategy 2 past 3.1, to make the paper more focused, I would suggest moving most/all the experiments from the main paper into the appendix and include a textual mention/footnote that notes another strategy was attempted, but it did not work as well as S1, with details in appendix.\n5. In Figure 2, you should indicate the results are for S1 in the caption.\n6. In Table 2, there is a mix between open weights and API only models. I would suggest distinguishing between the two somehow, as I don’t feel confident that it is a fair comparison to open weights models whose training data is known to a larger degree. A similar argument would apply to making it easier to compare small model results to large model results.\n7. The paper spends quite a bit of time discussing details in Atlas, it would be stronger by discussing only the minimal necessary and focusing on the new work being introduced.", "extra_scores": {}}, {"review_id": "XcfpkVlLcw", "reviewer": "Reviewer_gDcV", "summary": "This paper tackles the problem of in-context learning with retrieval-augmented encoder-decoder LMs. There are two main techniques introduced in this paper. One is the pre-training method called RAVEN, which combines retrieval-augmented masked language modeling and prefix language modeling for the training of retriever and generator. The other one the fusion-in-context learning, which enables the model to leverage more in-context examples. The proposed method shows improved performances over Altas and other decoder-only LLMs like GPT-3 and PaLM on NQ and TQA.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the soundness and effectiveness of our methods. \n\n>“It's unclear whether the proposed methods remain effective when they are applied to today's popular LLMs, despite that most of them are based on decoder-only architectures.”\n\nWhile decoder-only LMs are currently more popular, our research aims to explore and highlight the potential of encoder-decoder architectures. From a research standpoint, demonstrating the efficacy of alternative approaches is crucial for driving innovation and may encourage the development of more powerful encoder-decoder LMs. Notably, recent Reka models (https://arxiv.org/abs/2404.12387) are based on encoder-decoder architectures and achieve performance comparable to GPT-4.\n\nAdditionally, it is also interesting to explore the combination of the Fusion-in-Decoder architecture with existing decoder-only LMs. In this way, we can use a bidirectional architecture to effectively encode retrieved passages for powerful decoder-only LLMs.\n\n>“There is no evidence whether in-context learning is still necessary when fine-tuned LLMs are applied for RAG.”\n\nThis comment seems confusing. In-context learning applies to both pretrained and fine-tuned models with or without RAG. It remains important, as shown in leaderboards and technical reports of recent models like Gemini, where many results (of both pretrained or finetuned models) are still reported using in-context learning.\n\n>“Besides, efficiency is another concern of the proposed method, considering that in-context examples in the RAG settings will take a great deal of token consumption.”\n\nThe additional cost of RAG applies to all existing LMs. And encoder-decoder LMs are actually more efficient than decoder-only LMs in RAG settings. As explained in Section 2: “For decoder-only LMs, the computational cost typically increases quadratically with the input length, as well as with the number of retrieval passages. In contrast, for encoder-decoder LMs with a Fusion-in-Decoder architecture, the computation cost grows linearly with the number of retrieved passages, as they only perform self-attention over one passage at a time (Izacard & Grave, 2021).”\n\nThis efficiency also applies to FiCL. If we encode different in-context examples separately with FiCL, the computation cost also grows linearly with the number of in-context examples. Additionally, we don’t retrieve passages for in-context examples, which further improves efficiency and reduces token consumption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is technical sound. The resulted LM improves upon the previous encoder-decoder LM. \n- The effectiveness of the proposed method is empirically verified on NQ and TQA.", "reasons_to_reject": "There are two serious concerns regarding the motivation of the paper. \n- It's unclear whether the proposed methods remain effective when they are applied to today's popular LLMs, despite that most of them are based on decoder-only architectures. \n- There is no evidence whether in-context learning is still necessary when fine-tuned LLMs are applied for RAG. \n\nBesides, efficiency is another concern of the proposed method, considering that in-context examples in the RAG settings will take a great deal of token consumption.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fqL1tUyU6", "reviewer": "Reviewer_UreB", "summary": "This paper aims to improve the in-context learning ability of retrieval-augmented encoder-decoder models---to be more specific, Atlas. The authors first conducted some analysis on the in-context learning behavior of Atlas with different prompting strategies, which shows that Atlas can do in-context learning but is unstable and does not benefit from more in-context examples. Then the authors proposed three ways to improve it: \n\n1. (Training): add a second stage training for Atlas which uses prefix language modeling, to handle the training/inference mismatch (in inference it's almost always a continual generation task). \n\n2. (Inference): instead of using the same group of ICL examples in each fusion-in-decoder context, the authors propose to use different ones so that the model can use more ICL examples without the limit of encoder context window. This is called \"Fusion-in-Context Learning (FiCL)\".\n\n3. Select better ICL examples by using the Atlas retriever following previous work.\n\nThose techniques combined form the new model, Raven. The authors mainly demonstrate its efficacy on open-domain QA tasks (NaturalQuestion, TriviaQA), as well as MMLU. The experiments show that Raven can improve significantly over Atlas. Thorough ablations also demonstrate the efficacy of each of the above techniques.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive feedback. We are happy that the reviewer likes our writing, methods, and experiments.\n\n> W1: evaluation on tasks that may not require retrieval augmentation\n\nThank you for your suggestion! Our primary focus in this paper is on “in-context learning with retrieval-augmented encoder-decoder models”, hence our emphasis on tasks benefiting from retrieval augmentation. We recognize the value of evaluating RAVEN on tasks that do not heavily rely on retrieval augmentation. In the MMLU benchmark, which includes a variety of tasks, some do not require retrieval augmentation. We observe RAVEN demonstrates performance improvements over ATLAS. We also observe performance gains when retrieval is turned off (i.e., retrieval of 0 passages) during evaluation.\n\n> W2: analysis on how ICL/FiCL behaviors (wrt number of demonstrations)\n\nActually, we have already provided some analysis on this in our paper. For example, in Figure 2 and Figure 5, as well as in the results in the tables, we report the performance of RAVEN/ATLAS with different numbers of in-context examples. For decoder-only LMs, we may refer to the results reported in previous papers. The performance varies significantly; for example, on TriviaQA, 1-shot is optimal for PaLM, while GPT-3 uses a 64-shot setting.\n\nIt’s true that the improvement of FiCL may also have a \"ceiling\", but we do not observe a performance drop like standard ICL when continuing to increase the number of examples (as shown in Figure 5).\n\n> W3: comparison to retrieval-augmented decoder-only LMs\n\nWe have compared RAVEN with Replug, Retro, Retro++ in Table 2, which are retrieval-augmented decoder-only LMs. From our experiments, when the capability of the base model is close, RAVEN can significantly outperform retrieval-augmented decoder-only LMs.\n\nFor decoder-only LMs like GPT-4 (stacking all the demonstrations and passages), there is still a gap, but the comparison may not be meaningful since the capability of the base model differs significantly (T5 vs. GPT-4). We notice there is a family of high-performing encoder-decoder LMs recently developed by Reka (https://arxiv.org/abs/2404.12387). It would be very interesting to see the performance of RAVEN built on Reka (though it’s not open source so far). We feel this is a very promising area for future exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, and well motivated, and the experiments are very clear and clean. \n\n2. The analysis on Atlas' in-context learning performance is insightful and well motivated the proposed techniques. The proposed techniques are interesting, effective, and novel. \n\n3. The improvement on the selected tasks is significant. The ablation study is comprehensive.", "reasons_to_reject": "1. This is more of a suggestion than critique: I wish there were more tasks analyzed and evaluated. The authors only explored tasks that significantly benefit from retrieval-augmentation, such as open-domain QA tasks and MMLU. To demonstrate the general ability of the language model, I believe people would also be interested to see how this suite of techniques work on tasks that may not require retrieval augmentation. This would show that FiD/encoder-decoder style LMs not only work for RAG, but also general ICL.\n\n2. It would be great to have some analysis on how ICL behaviors (wrt numbers of demonstrations) differ in FiD/decoder-only models. Even with FiCL, I imagine the improvement has a \"ceiling\" given there is no attention across different encoder context windows. \n\n3. How about comparing this to using retrieval-augmentation in decoder-only LMs too? You can just stack all the demonstrations as well as all the passages in the decoder context as well. Also, what about [Replug](https://arxiv.org/abs/2301.12652)?", "questions_to_authors": "Please see \"reasons to reject\".", "extra_scores": {}}, {"review_id": "luvQeekbZG", "reviewer": "Reviewer_k2PV", "summary": "This paper investigates the in-context learning ability of encoder-decoder models and further proposes an improved version of the studied model. RAVEN, the proposed model, is built on top of ATLAS by further pre-training with the suggested objectives, with an aim to shorten the gap between the masked language modeling during pre-training and in-context learning during inference time. In addition, the few-shots examples are key to the in-context learning performance. The authors further present to use different examples for each retrieved document (Fusion-in-Context Learning) and use the retrieval component to obtain the demonstration examples (In-Context Example Retrieval). Results indicate that RAVEN outperforms ATLAS and even surpasses (or performs on par with) much larger models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for highlighting the writing and extensiveness of our experiments.\n\n> novelty\n\nThe research problem of “in-context learning with retrieval-augmented encoder-decoder LMs” is an important one that has been overlooked in existing studies, making the problem itself novel. Regarding techniques, it is crucial to emphasize that our methodologies stem from an in-depth understanding of the limitations of current models and a careful consideration of how to overcome these issues, as discussed in Sections 3. The design is also non-trivial; it was based on the exploration of different strategies, as detailed in, e.g., Section 3.1 and Appendix C1-3.\n\nIn summary, our work offers contributions from both analytical and technological perspectives:\n- From an analytical standpoint, we provide a thorough analysis of the in-context learning capability of state-of-the-art retrieval-augmented encoder-decoder language models. Our insights also highlight potential avenues for improvement. This analysis serves as the foundation for our proposed RAVEN training strategy and Fusion-in-Context prompting approach.\n- From a technological perspective, we introduce a confluence of retrieval-augmented masked and prefix language modeling, coupled with our Fusion-in-Context Prompting and In-Context Example Retrieval strategies. These techniques not only enhance the base model's capabilities (by mitigating the mismatch between pretraining and testing) but also maximize the potential of in-context learning for these models (either by scaling the number of in-context examples with FiCL or by retrieving better in-context examples).\n\n> interpretation of Table 3\n\nIn this table, we aim to demonstrate how in-context example retrieval can further improve the performance of RAVEN. Yes, the baseline is RAVEN, and the baseline score for RAVEN 3B 1-shot on NQ is 31.7%. With in-context example retrieval, we achieve a further improvement of 9.1%, resulting in an accuracy of 40.8%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and organized.\n* Experiments are thorough, compared with a variety of baseline models.", "reasons_to_reject": "Novelty of the proposed methods is somewhat limited. Essentially, RAVEN is trained by further pre-training ASLAS with objectives that were discussed in the original paper of ASLAS (the prefix LM used in the second stage can be seen as a variant of the standard MLM and standard prefix LM).", "questions_to_authors": "What is the baseline for Table 3 and how to interpret them? If the baseline is RAVEN, does it mean the baseline score, for example, 31.7% for RAVEN 3B 1-shot on NQ can be improved by 9.1%, as reported on Table 3?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the combination of using an encoder-decoder model, with in-context examples, and retrieval augmentation, looking at ATLAS as a main comparison point (without considering finetuning of the model on downstream tasks)

The authors add a pretraining stage that makes train and test mismatch smaller, and also use different examples at test time, and improve prompting and the retrievers. The reviewers mostly agree the paper is written well, experiments are convincing and empirical gains are sizable so this is a good addition to the literature on retrieval-augmented language modeling.
Decision: Accept