RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Authors: Xi Victoria Lin
Xilun Chen
Mingda Chen
Weijia Shi
Maria Lomeli
Richard James
Pedro Rodriguez
Jacob Kahn
Gergely Szilvasy
Mike Lewis
Luke Zettlemoyer
Wen-tau Yih
Venue: ICLR 2024 poster (2024)
Keywords: retrieval-augmented language model
large language model
knowledge intensive NLP
Primary area: generative models
TL;DR: We propose a fine-tuning approach that effectively retrofits any LLM with retrieval capabilities.

Abstract: Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

Published as a conference paper at ICLR 2024
RA-DIT: RETRIEVAL-AUGMENTE DUAL INSTRUC- TION TUNING
Xi Victoria Lin'
Xilun Chen"
Mingda Chen"
Weijia Shi Maria Lomeli
Rich James
Pedro Rodriguez Jacob Kahn
Gergely Szilvasy
Mike Lewis
Luke Zettlemoyer Scott Yih
FAIR Meta
(victorialin, xilun, minddachnn, eetasaaaccceesstttyy.eeeeaaaaaa
ABSTRACT
Retrieval-ant -arievee-- language models (RALMs) improve performance by access- ing long-tail and up-to-date knowledge from external data stores, but are challeng- ing to build. Existing approaches require either expensive retrieval-specific modi- fications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmente Augmented Dual Instruction Tuning (RA-DIT), lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tur tne-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more rel- evant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intense intensive zero- and few-shot learning benchmarks, signif- icantly outperforming existing in-context RALM approaches by up +8.9% in +1.4% 1.4% in 5-shot setting on average.
O-shot setting and +1
INTRODUCTION
Large language models (LLMs) excel as zero- and few-shot learners across various tasks (Brown et al., 2020 Chowdhery et al. 2022 Touvron et al.
2023a b Anil et al. 2023 OpenAl 2023
However, because knowledge is represented only in the model parameters, they struggle to capture
long-tail knowledge (Tirumala et al. 2022 Sun et al. 2023) and require substantial resources to be
kept up-to-date (Miller 2023). Retrieval-Augmennte
Language Modeling (RALM) integrates LLMs
with non-parametric information retrieval to overcome these limitations (Guu al. 2020 [Brgeaudd
ett al. 2022; Izacard et al., 2022b (Sh et al.] 2023b] Ram et al. (023). By explicitly decoupling
knowledge retrieval with the backbone language model, such architectures have exhibited superior
performance on knowledge intensive tasks such as open-domain question answering (Lewis et al.,
2020 Izacard et al. 2022b) and live chat interactions (Liu 2022).
Existing RALM architectures focus on two high-level challenges: (i) enhancing the LLM's capabil- (Z20arz Izacard et al., 2022b) and (ii) refining the retrieval component to return more relevant content (Shi et 2023b Izacard et al. (002b). Previ- ous work have also introduced retrieval capabilities at different stages the model training process.
ity to incorporate retrieved knowledge (Lewis et al.) 2020)
REALM (Guu et al. 2020) and RETRO (Borgeaud et al. 2022) opt for end-to-end end pre-training, in- corporating the retrieval component from the outset. Atlas (Izacard et al. 2022b) builds upon the T5 language model (Raffel et al. 2020), and continuosly pre-trains the framework over unsupervised text. REPLUG (Shi et al. 2023b) and In-Context RALM (Ram et al. 2023) combine off-the-sheif LLMs with general-purpoot through the emergent in-context learning capbabilities of LLMs. However, extensive pre-training of such architectures is expensive, and the off-the-shelf fusion approach also has limitations, particu- larly as the LLMs are not inherently trained to incorporate retrieved content.
retrievers, showing that these two components can be effectively fused
"qqul contribution

---

Published as a conference paper at ICLR 2024
Retriever Fine-tuning
Retrieval Retrieval-augmente augmented Instruction Tuning
max PL
Palei
min KL
Lassune that (he PGA chip bas damage to the substrate lovel, pould be snived temporanly by reflowingg.. ininQ; Why does GPU keep failing with Xid 79 fallen bus?
Background:
not use Y-slitterss to a-pin convarters any
of the PCle power cables supplying the GPU. Maka...
Retriever
Pesale1 (x,9
LLM
max PLu xx)
Paley (x)
min KL.
Background: Microsoft should from the harcheece market Daroause will have a a.0ooalee Imsest on me OEMs that Vir Questio:: Why does my GPU koep failin with Xid 79 fallen off the bus? Answer:
y: Do not use Y-elitters pin to B pin converiars any of the PCle power cacies supplying the GPU. Make...
Why does my keep failing failen off the bus?
Pispley (x,y)
Figure 1: The RA-DIT approach separately fine-tunes the LLM and the retriever. For given exam- ple, the LM-ft component updates the L.M to maximize the likelihood of the correct answer given the retrieval-augmenteed instructions $23): the R-ft component updates the retriever to minimize the KL-Divergence between the retriever score distribution and the LLM preference ($2.4)
In this work, we show lightweight instruction tuning (Chung et al. 2022b [yer et al. 2022 Zhou et al. 2023) alone can significantly boost the performance of RALMs, especially in knowledge intensive scenarios. We propose Retrieval-Augmented Dual Instruction Tuning (RA-DIT), an ap- proach that retrofits any LLM with retrieval capabilities via fine-tuning
set of tasks selected to cultivate knowledge utilization and contextual awareness in the language model predictions. We initialize the framework using pre-trained LLAMA (Tourron al., 2023a) and state-of-the-arr dual- encoder based dense retriever, DRAGON+ (Lin at. (023). Following Shi et al. (2023b), we retrieve relevant text chunks based on the language model prompt. Each retrieved chunk is prepended the prompt, and the predictions from multiple chunks are computed in parallel and ensembled to pro- duce the final output.
We perform instruction -tnningtunggggn in two separate steps. For language model f fine-tuning (LM-ft), we adopt the label-loss objective (Chung et al., [2022b] Iyer et al.) (022) and augment each fine-tuning prompt with a retrieved "background" field prepended to the instructions (Figure D]. We also lever- age the design of existing NI.P tasks and populate this field with the ground truth context for tasks such as reading comprehension and summarization. By incorporating the background text during fine-tuning, we guide the LLM to optimally utilize the retrieved information and ignore distract- ing content (Shi et al., 2023a). For retriever fine-tuning (R-ft), we update the query encoder using a generalized LM -upprvire Retrieval (LSR, Shi et al., 2023b) training objective computed over a combination of supervised tasks and unsupervised text completion. This way enable the retriever to yield more contextually relevant results, aligned with the preferences of the LLM.
We demonstrate that each fine-tuning step offers significant performance gains, and that the fine- tuned LLM and retriever can be combined to achieve further improvements. Our largest model, RA-DIT 65B, attains state-of-the-art performance zero- and few-shot settings on knowledge intensive benchmarks, notably surpassing the un-tuned in-context RALM approach on datasets including MMLU (Hendrycks et al.) 2021a) (+8.2% 0-shot; +0.7% 5-shot) and Natural Ques- tions (Kwiatkowski et al.) 2019) (+22% 0-shot; +3.8% 5-shot). In addition, RA-DIT 65B also substantially outperforms ATLAS 11B on 8 knowledge-intensss
tasks (+7.2% on average in the 64-shot fine-tuning setting). This suggests that language models and retrievers, when optimized in- dependently and then fused through instruction-tuning, can compete effectively with RALMS that have undergone extensive continuous pre-training. We further conduct comprehensive model anal- ysis, showing the effectiveness of our approach across LLMs of varying sizes, as well as evaluating the influence of different fine-tuning strategies and retriever configurations!
We release the scripts for indexing Common Crawl data and generating fine-tuning and inference prompts at: hhttts://ggtthhhee wttpc///ccboooooeeeeerooooo
2

---

Following Shi et al. (2023b). for given language model prompt æ, we retrieve the top-k relevant text chunks C C, context window size limit, each retrieved chunk is prepended to the promp and the language model Published as a conference paper at ICLR 2024
2 METHOD
2.1 ARCHITECTURE
Language Model We focus on retrieval-augmenting pre-trained auto-regressive language mod- els (Brown et al.) 2020). In particular, we use LLAMA (Touvron al. 2023a). family of open- sourced language models pre-trained on trillions of tokens.
Retriever We adopt dual-encoder based retriever architecture, since can be easily fine-tuned and is efficient at the inference stage (Lewis et al. 2020, Izacard et al. 2022b Shi et al. 2023b). Given corpus and query q. the document encoder maps each text chunk c to an embedding Ea(c) and the query encoder maps to an embedding (g) The top-k relevant text chunks for are retrieved based on the query-document embedding similarity, which is often computed via dot product:
8(g, c) = Eg) Ea(c)-
We initialize the retriever using DRAGON+ (Lin et al. 2023 a state-of-the-art dual-encoder model trained with contrastive learning objective and large-scale data augmentation.
Parallel In-Context Retrieval-Augmentation
k. To stay within the
predictions from multiple augmented prompts are computed in parallel. The final output probability a mixture of the probability from each augmented prompt weighted by the chunk relevance score
PLM (ulæ, = Σ
PLM (ulcoz) pa((k)),
GEC'
expe(z,0) where denotes sequence concatenation, and pn(c)z) Edeer exp (2,,e) the retriever scores re-normalized among top-k relevant chunks.
2.2 FINE-TUNING DATASETS
We choose a set of fine-tuning tasks aimed at boosting the language model's s ability to utilize knowl- edge effectively and improving its contextual awareness in generating predictions. As shown in Table our language model fine-tuming datasets (Dt) consists of 20 datasets across distinct cat- egories: dialogue, open-domain QA, reading comprehensiorf) summarization and sfain-of-thhughh reasoning. For retriever fine-tuning datasets DR, we opt for the QA datasets in our collection fea- turing standalone questions, and additionally include two QA datasets, FreebaseQA (Jiang al.) 2019) and MS-MARCO (Nguyen et al. 2016). The examples of each dataset are serialized for in- struction tuning using manually compiled templates (Table [O). For tasks in Dz DR. we use the same template for both fine-tuning steps. In addition, we observe that supplementing the instruction- tuning data with unsupervised text leads to additional performance gains for both language model and retriever fine-tuning, and we detail data mixture used in Appendix
2.3 RETRIEVAL AUGMENTED LANGUAGE MODEL FINE-TUNING
To improve the language model's ability to utilize retrieved information, we fine-tune it on the selected datasets Dr with in-context retrieval augmentation. Formally, we separate each fine-tuning sequence into an instruction segment (x) and an output segment (y). For each example (I,, Bi)
We use pair of start ("Background:") and end (")))) tokens to demarcate the retrieved segment in the augmented prompt. The complete set of our instruction tuning templates are shown in Appendix C
*Our reading comprehension (RC) fine-tuning datasets include SQuAD 2.0 (Rajpurkar et al. 2018). which trains model to determine whether question can be answered using a given passage, and to provide an answer only when the passage is relevant (otherwise the response is set to "I don't know"). As shown in Appendix F fine-tuning on this dataset promotes a desirable behavior: the instruction tuned model tends to respond with "I don't 't know" when the retriever presents an incorrect passage. We leave further exploring this hehavior improve answer generation future work.
3

---

Published as a conference paper at ICLR 2024
Table 1: Our intruction tuning datasets. All datasets downloaded from Hugging Face (hoestt
et al. 2021 with the exception of those marked with 1, which are taken from Iyer et al. 2022
Task
HF identifier
Datasel numn
D2
DR
#Train
Dialogue
basst
OpenAssistant Conversations Datasel KKppf at. 2023
31,598
Commonsensc qu
CommonsecsecOA (Talmor et al. 2019
9,741
math qa
MathQA (Amini et aL. 2019
29,837
Web questions
Web Questions (Berant al 2013
3,778
Open-Domain
QA
Wiki qa
Wiki Question Answering Yang cr 2015
20,360
y Yahno wes.sss
Yahoo! Answers QA
87,362
Treehase Qu
FreehaseQA (Jiang
2019
20,358
MS MARCO Nguyen al. 2016
80.143
coqa
Conversational Queston Answering (Reddy al 2019
108,647
drup
Discrete Reasoning Over Paragraphs
Duu el al. 2019
77,400
marraliveup
NarrativeQA (Kučiský et 2018
32,747
Reading Com-
new
NewsQA (Trischler et al. 2017]
74,160
prehension
Pubmed.qa
PubMedQA (Jm
2019
1,000
quail
QA for Artificill Inteligence (Rugers et al.
2020
10,246
quare
QuaRel (Tafjord et al. 2019
1,941
squad_y2
SQuAD v2 (Kajpurkär al.
2018
130,319
Suminarizätion
cnn dailymai
CNN7 DailyMai (Hermann
287.11
aquarrat
Algebra QA with Rationales (Ling
2017
97,467
Chain-of-
coqal
Explanations for Commonsensa Aggaawal al. 2021
7,598
thought
gsm8k
Grade School Math (Cohbe 2021
7,473
Reasoning
compeition math?
MATH (Hendrycks
202
7,500
stralegyga
Strutegy QA Geva
2,290
We only used the question-and answer pairs the MARCO dataset.
Dr, retrieve the top-k relevant text chunks C. C' based on 2,. Mirroring the inference-ting handling, for each retrieved chunk Ciy E C, we create a separate fine-tuning example by prepending it to the instructions as background field. resulting in i independent fine-tuning instances per original example: {[ay o 24, ji)lj = =1. 1...k]]
We fine-tune the language model using the next-token prediction objective and minimize the loss from tokens in the output segment of each instance (Iyer et al. 2022):
D(Dr)=- = ΣΣ log PLM o x.).
(3)
Integrating in-context retrieval augmentation during fine-tuning gives twofold benefit. First, adapts the LLM to better utilize relevant background knowledge to make prediction. Secondly, even state-of-the-ee retrievers can falter and return inaccurate results. By training the LLM make correct predictions when a wrong retrieved chunk is given, we enable the LLM to ignore misleading retrieval content and lean into parametric knowledge in such cases. The efficacy of this fine- tuning strategy is empirically demonstrated in $..1
2.4 RETRIEVER FINE-TUNING
In addition fine-tuning the language model with retrieval augmentation, also fine-tune the retriever to better align its output with the language model. In particular, we adopt generalized version of LSR (LM-Supervised Retrieval, (Shi et al., (023b) training that leverages the language model itself to provide supervision for retriever fine-tuning.
For training sample (x, y) in the retriever fine-tuning dataset Dn, we define the LSR score for retrieved chunk e as follows:
exp cxp(pLM(ylco=))/7
exp cxp(pLs(slcc
(4) where 7 is a temperature hyperparameter, and C' C C denotes the top-k retrieved chunks for x. A higher LSR score indicates that e more effective at improving the language model's chance of
PLSR(C|E,9) = Evecest exp (PLM x)/r) "dJecr exp (PLM
"The exceptions summarization tasks and RC tasks with context dependent questions (e.g. "when the writer born?"), where we do not perform retrieval create fine-tuning instances using the given background text instcad. For RC tasks with self-contained questions, we use the retrieved chunks in addition to the given hackground text create fine-tuning instances, resulting in À 1 of them per original example.
4

---

Published as a conference paper at ICLR 2024
predicting the correct answer. The goal of LSR training is for the retriever to assign higher scores to chunks that can improve the LLM's likelihood of generating the correct answer. To achieve this, we minimize the KL-divergence between PLss and the retriever scores pn defined in Eq.
L(DR) K |l pLsu(c(x,))
(L(p((()x)
In practice, only update the query encoder of the retriever, as fine-tuning both encoders hurts the performance (:5.1). While previous work (Shi et al., 2023b) relies solely on unlabeled texts incorporate (denoted corpus data) for LSR training, show that LSR be generalized the multi-task instruction data introduced in $2.2 (denoted as MTI data). The MTI data provide direct supervision to the retriever to return relevant information that enhances the language model in various downstream tasks. As shown in $5.1 combining both types of data yields the best results and outperforms using either source alone.
EXPERIMENT SETUP
3.1 RETRIEVER
We initialize the retriever our framework with DRAGON+ (Lin al. 2023 and also use it to study various retriever configurations. To build the retrieval corpus. we combine the text chunks from the Dec. 20, 2021 Wikipedia dump released by Izacard et al. (022b) with additional ones from the 2017-2020 CommonCrawl dumps. We detail the corpus pre-processing and indexing in Appendix A Our final retrieval data store, with the two data sources combined, contain 399M text chunks with maximum length of 200 words. In Appendix E.3] we conduct analysis the impact of using various subsets of the retrieval corpora, as well as different Wikipedia snapshots. We obtain the retrieval queries used for our fine-tuning and evaluation tasks using manuallyP]constun templates (Table 10 and 12
3.2 BASELINES
We focus on comparing our approach to the base LLAMA models (Touvron et al. 2023a and RE- PLUG (Shi al. 2023b), state-of-the-ard approach that integrates off-the-shelf LLMs and retriev- ers, in the zero-shot and in-context few-shot learning settings. We instantiate REPLUG using LLAMA and DRAGON+. In addition, we also compare RA-DIT ATLAS (Izacard et al. 2022b 64-shot fine-tuning setting ($4).
3.3 EVALUATION
We primarily conduct evaluation knowledge-intensive tasks that are not included in our fine- tuning datasets, including MMLU (Hendrycks et al. 2021a), Natural Questions (NQ; Kwiatkowski al. 2019). TriviaQA (TQA; Joshi al.) 2017). and a subse of the tasks in the KILT bench- mark Peerroni et al. 202I). We use the development split of the KILT subset excluding ELI5 to determine fine-tuning hyperparameters (Appendix B. This enables us to report genuine few-shot evaluation results for out of the 10 evaluation tasks. For the remaining tasks, we report few-shot results assuming access in-domain development data. In addition, on commonsense reasoning tasks to measure the impact of the proposed approach on the LLM's parametric knowledge and reasoning capabilities. Details of our evaluation datasets, including the evaluation metrics, template and the scoring functions used, can be found in in Appendix
also evaluate the models
4 MAIN RESULTS
Knowledge- Centeedge-Intensivee Intensive Tasks We report the main results in Table In particular, RA-DIT is com- pared to LLAMA (Touvron et al. 2023a as well as REPLIG (Shi et al. 20235). in both 0-shot and
'We leave automatically generating task-specific retrieval queries to future work.
'The subset consists of seven tasks: HotpotQA (Yang et al. 2018), FEVER (Thome et al.
2018). AIDA
CONILI-YAGO (Hoffart all|2011 101 Zero-Shot RE (Levy et al. of Wikipedia (Dinan et al. 2019) and FI.IS (Fan al. 2019).
2017
T-REx (Elsahar al.
2018 Wizard
5

---

Published as a conference paper at ICLR 2024
Table 2: Main results: Performance on knowledge intensive tasks (test sets).
MMLU NQ
TQA ELI5
HoPo FEV
AIDA zsRE T-REx WoW Avg Avg
O-shot
LLAMA 65B
51.2
5.2
55.8
19.5
12.5
59.3
0.6
6.7
1.3
15.6
32.9
22.8
LLAMA 65B REPLUG
59.7
28.8
72.6
19.1
32.0
73.3
41.8
50.8
36.3
16.1
45.1
43.1
RA-DIT 65B
64.6
35.2
75.4
21.2
39.7
80.7
45.1
73.7
53.1
16.4
49.1
50.5
5-shor in-context
LLAMA 65B
63.4
31.6
71.8
22.1
22.6
81.5
48.2
39.4
52.1
17.4
47.2
45.0
LLAMA 65B REPLUG
64.4
42.3
74.9
22.8
41.1
89.4
46.4
60.4
68.9
16.8
51.1
52.7
RA-DIT 65B
64.9
43.9
75.1
23.2
40.7
90.7
55.8
72.4
68.4
17.3
51.8
55.2
64-shot fune
NQ
TQA
HoPo
FEV
AIDA
zsRE
T-REx
WoW
Avg
ATLAS
42.4
74.5
34.7
87.1
66.5
74.9
58.9
15.5
56.8
RA-DIT 65B
43.5
72.8
36.6
86.9
80.5
78.1
72.8
15.7
60.9
Average MMLU, NQ. TQA, and ELIS.
ATLAS conducts shot tuning each individual task evaluates specific individuall.. RA-DIT, we perform multi-task fine-tuning using 64-shot examples from each task combined, and report the performance of unified model across tasks.
Table 3: Performance on commonsense reasoning tasks (dev sets) without retrieval augmentation.
O-shot
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
OBQA
Avg
LLAMA 65B
85.3
77.0
78.9
56.0
60.2
72.1
RA-DIT 65B
86.7
83.7
57.9
85.1
79.8
83.7
60.5
58.8
74.5
5-shot settings. We first observe that REPLUG works much better than the base LLAMA 65B, con- firming the benefits of RALMs on knowledge-intensive tasks. Furthermore, RA-DIT significantly outperforms REPLUG (+8.9% in 0-shot and +1.4% 5-shot on average over MMLU, NQ. TQA and ELI5) and achieves the best performance bining off-the-shelf LLMS and retrievers sub-optimal, and our dual instruction tuning approach an effective way of retrofitting LLMSs with retrieval capabilities
most datasets. This corroborates our claim that com-
We also compare with ATLAS, state-of-the-an encoder-decodes based RALM that jointly pre- trains the language model and the retriever. Here we adopt 64-shot setting similar to Izacard et al. (002bb with the following differences. While ATLAS conducts 64-shot fine-tuning for each individ- ual task and reports the performance of task-specific models, continuously fine-tune the RA-DIT checkpoint using the 64-shot examples from tasks combined, and report the performance a single model across tasks. As shown in Table despite using single model, RA-DIT outperforms ATLAS by an average of 4.1 points, achieving higher performance on 6 out of the datasets.
Commonsense Reasoning We benchmark RA-DIT 65B to evaluate the impact of retrieval- augmented instruction tuming on the LLM's parametric knowledge and reasoning capabilities. We hence do not perform retrieval augmentation this experiment. shown in Table 3 RA-DIT demonstrates improvements over the base LLAMA models on 7 out of evaluation datasets, indicating that the parametric knowledge and reasoning capabilities of the LLM component are in general preserved. As discussed in Appendix [F] maintaining the parametric knowledge the LLM component is vital as a safety net when the retriever makes mistakes.
set of commonsense reasoning tasks
ANALYSIS
5.1 FINE-TUNING STRATEGIES
Language Model Fine-tuning We compare LLAMA instruction-tunel with retrieval- augmentation (RA-IT 65B) to the base language model, as well as LLAMA that instruction-
We report lower 0-shot performance for LLAMA 65B on NQ and TQA in comparison to Touvron al. 2023a). By examining the model generation, think Touvron al. (023m) reported the ratio of responses that contain the ground truth string the 0-shot setting, while report match.

---

Published as a conference paper at ICLR 2024
Table 4: Ablation of language model fine-tuning strategies. All rows report dev performance.
0/ O-shot
HoPo
FEV
AIDA
zsRE
T-REx
WoW
Avg
LLAMM 65B
12.5 / 23.8
59.6/ 83.7
0.9/64. /4.1
36.0
1.2/
15.7 / 17.4
16.6 46.2
65B
20.0 / 30.0
67.8 /3.2
8.9 58.5
19.0/35.4
/
16.4 / 16.5
24.9
46.2
RA-IT 65B
26.8 / 29.9
/4.8
10.7
3
24.1
16.5/ 16.5
29.0 45.4
top-1 chunk
LLAMA 65B + DRAGON+
25.8 / 39.4
72.8 / 89.8
50.7
31.4/69 /9.6
69.1
15.8 / 17.1
39.0 54.3
IT 65B DRAGON+
33.3 / 38.8
84.0 90.1
43.9/ 50.3
56.8 58.2
44.7
66.4
15.7 15.6
46.4 /3.2
RA-IT 65B DRAGON+
81.0/ 37.6/39.1
90.4
52.3
59.6/57.9
49.6/ 65.8
16.6 16.6
47.7 /3.7
top-3 chunks
LLAMA 65B DRAGON+
29.6/40.8
74.9 90.3
43.1
/
37.2/ 70.8 55.9/62.9
16.0/ 17.2
42.8 55.8
IT 65B DRAGON+
35.2/40.0
85.7 /91.2
49.7
/2.3
4 562/6119
15.6 15.6
48. 1 54.9
RA-IT 65B+ DRAGON+
82.4/911 39.9/40.6
43.2/5344
63.4/61.
67.8/67.6
16.6 16.7
50.1 55.2
top-10 chunks
LLAMA 65B DRAGON+
31.0/41.6
75.4 90.8
44.8/440
58.6/63.7
40.2/71.9
16.0/
44.3 56.6
65B DRAGON+
33.9/40.6
50.5/53.8 99.0
53.9/62.5
49.7
15.6/
47.8/ 55.6
RA-IT 65B DRAGON+
40.0/41.2
82.8 / 92.1
47.2/53.5
65.0/62.5
/4.3/69.0
16.5 16.6
51.0/ 55.8
Table 5: Ablation of retriever fine-tuning strategies. All rows the LLAMA 65B model and report 5-shot performance on the dev sets.
5-shor
MMLU
NQ
TQA
HoPo
FEV
AIDA
zsRE
T-REx
WoW
Avg
Avg
DRAGON+
62.6
41.8
72.9
41.5
90.6
54.1
63.7
72.1
17.5
56.6
57.4
MTL instruction tuning data
61.1
43.6
74.0
36.5
91.4
64.6
56.7
72.1
17.1
56.4
57.5
corpus data (FT both encoders)
61.7
73.8
69.8
57.2
17.5
54.0
55.8
corpus data
62.9
43.0
74.3
41.1
91.6
54.4
63.4
71.8
17.4
56.6
57.8
95% corpus 5% MTL data
63.0
42.1
74.9
41.2
91.6
54.9
65.2
71.6
17.5
57.0
58.0
Average over
development
conventionally (IT 65B) on the same set of tasks. We evaluate all models with in-context retrieval augmentation using the DRAGON+ retriever, adjusting the number of retrieved chunks to 0, I or 10. As shown in Table $ while both instruction tuning methods substantially enhance the O-shot performance, they offers marginal improvements or even hurt the model performance in the 5-shot setting for most tasks except for HotpotQA When in-context retrieval-augeee -allvvaaltggmenaationn is applied, all models show substantial gains in both settings, even when limited to the top-1 chunk. The model performance consistently improves as we include more retrieved chunks. In the 0-shot setting with top-10 10 retrieved chunks, the RA-IT 65B model outperforms the IT 65B model by large margin (51.0% vs. 47.7%). Under this setting, we observe that retrieval-augmente significantly enhances the LLM's ability integrate information from the retrieved text chunks. The model is able to extract the correct answers from relevant chunks with greater confidence, while effectively leaning on its parametric knowledge for prediction when an irrelevant text chunk is present (Appendix F In Appendix E. we also discuss the performance of RA-IT models when applied to smaller LLAMA models (7B and 13B), showing that it offers even larger performance boost in those cases.
instruction tuning
Retriever Fine-tuning In Table study different retriever fine-tuning strategies. As men- tioned in $2.4, we explore two types of retriever fine-tuning data, the multi-task instruction (MTI) data and the corpus data. We observe that fine-tuning the retriever with the corpus data alone im- proves over the base DRAGON+ model by an average of 0.4 points, whereas fine-tuning using only the MTT data improves by a smaller margin 0. points. While fine-tuning with the MTT data yields good performance on certain datasets such as NQ (possibly due to its similarity to the MTI data), fine-tuning with the corpus data appears to generalize better and leads to stronger overall perfor- mance. Furthermore, we experiment with fine-tuning using both the MTI and corpus data. Table 5
Since our instruction tuning datasets include reading comprehension and summarization, the IT models also exposed to problem types that depend on background knowledge. 9This observation aligns with the findings from previous instruction-tuming literature [Iyrr et al. 2022 HotpotQA exception likely hecause is from task category covered in instruction-tuning data.

---

Published as a conference paper at ICLR 2024
shows that fine-tuning with "95% corpus data 5% MTI data" achieves the best accuracy across all models, outperforming the non non-finetuned baseline by 0.6 points on average
Finally, we also compare jointly fine-tuning both the query and document encoders with only fine- tuning the query encoder while freezing the document encoder. Table 5 shows this experiment conducted using the corpus data, where freezing the document encoder produces significantly better performance. As a result, we only fine-tune the query encoder in this work.
5.2 DUAL INSTRUCTION TUNING ABLATION
Table 6: The impact LM and Retriever fine-tuning in our RA-DIT method, comparing the RE-
PLUG baseline, LM-ft only, R-ft only, and RA-DIT. 5-shot dev set performance is reported.
5-shot
MMLU
NQ
TQA
ELIS
HoPo
FEV
AIDA
zsRE
T-REx
WoW
Avg
LLAMA 65B + DRAGON+
61.7
41.7
73.0
22.1
41.6
90.8
54.0
63.7
71.9
17.2
53.8
LLAMA 65B FTed DRAGON+
63.0
42.2
74.9
22.2
41.4
91.6
54.9
65.2
71.4
17.4
54.4
RIT 65B+ + DRAGON+
64.8
42.8
73.1
23.6
41.2
92.1
53,5
62.3
69.0
16.6
53,9
RIT 65B + FTed DRAGON+
64.3
43.8
75.0
23.3
42.0
92.3
52.8
65.2
70.1
17.3
54.6
We isolate the impact of the language model fine-tuning from retriever fine-tuning in RA-DIT method, and illustrate the benefit of each. [1] when used alone, and outperform the REPLUG using LLAMA 65B and the DRAGON+ retriever. On the other hand, the most gain can be achieved when combining LM-ft and R-ft in our RA- DIT method, which outperforms the REPLUG baseline by 0.8 points on average. In our prelimary experiments, we also attempted iterative dual instruction tuning by fine-tuning the retriever using LSR scores from the RA-IT LM or conduct the RA-IT step using passages returned by the fine- tuned retriever, for one or two such iterations, but did not observe further gains. We leave the exploration of multi-step RA-DIT to future work.
According to Table both LM-ft and R-ft are beneficial
5.3 RETRIEVER SETTINGS
Table 7: Retriever settings: report 5-shot dev set performance using LLAMA 65B and various retrievers in the REPLUG setting.
5-shot
MMLU
NQ
TQA
HoPo
FEV
AIDA
zsRE
T-REx
WoW
ELIS
Avg
LLAMA 65B
61.3
30.9
70.6
23.8
83.7
50.2
36.0
52.3
17.4
23.4
45.0
Retriever ablation using LLAAM
65B
and 399M
CC + corpus
Contriever
59.3
41.2
73.0
32.4
88.1
45.0
40.8
56.1
17.2
21.6
47.5
Contriever-msnn msmarco
62.0
42.1
74.1
38.7
89.3
49.3
60.2
62.9
17.4
21.8
51.8
DRAGON+
61.7
41.7
73.0
40.8
90.8
48.8
63.7
71.9
17.8
23.8
53.4
We study the impact of various retriever choices our framework. We use LLAMA 65B as the lan- guage model and combine it with different retrievers. Table 7 first compares DRAGON+ (Lin al.) 2023) with other state-of-the-ar retrievers such as Contriever (Izacard et al. 2022a). All retrieval- augmented models substantially improve the LLAMA baseline, and DRAGON+ significantly outperforms both Contriever and Contrever-MSMMARO. We hence adopt DRAGON+ as our base retriever in all experiments.
RELATED WORK
Retrieval-Angmennted Language Models RALMs augment LMs with paramarmetric memory
facilitate external knowledge access and provide provenance (Guu 2020 Lewis al. 2020
IIn early experiments, also tested other mixtures and found that using 5% 10% MTI data worked the best. (They perform similarly each other.)
MMior performance differences may be observed for the LLAMA 65B + DRAGON+ model in different ablations due to the differences in few-shot example truncation in long prompts. We ensure all rows within each table comparable.
8

---

Published as a conference paper at ICLR 2024
Borgeaud et al. 2022] Shi al. 2023b). Previous work have proposed different ways of fusing the LM and the non parametric component. For example, RETRO (Boggeaud et al. 2022) and FiD (Izacard & Grave 2021: Hofstätter et al. 2022) leverage separate encoder modules to encode the retrieved content, which are integrated with the backhone LM via cross-attention. A more widely adopted approach directly augments the LM input with the retrieved content (Guu et al. 2020 Lewis) itt al. 2220 Shi et al.) (023b). This approach yields competitive results with a moderate inference cost increase, as the LM can effectively contextualize the retrieved content and the original prompt through multi-layer self-attention. RA-DIT is grounded in the in-context RA framework for its simplicity and practicality, Instead of performing extensive pre-training (Guu et al., 2020; [Brgeaud et al., 2022 [zacard et al.) 2022b). we propose a lightweight fine-tuning recipe that primarily utilizes downstream data, and demonstrate improved few-shot generalization of the fine-tuned RALM on knowledge-intensiin language tasks.
Instruction Tuning Instruction tuning has been proposed align pre-trained LLMs follow nat- ural language instructions and avoid extensive prompt engineering (Ouyang et al. [022] Wei et al.
2022 Chung et al., 2022a| Wang et al.] 2022 Iyer et al. (022). We propose retrieval retrieval-ugmmeeted instruction tuning (RA-IT) as part of our dual instruction tuning framework to improve the LM's ability to leverage retrieved information. Concurrent work has also applied instruction tuning to other RALM architectures. Notably, |Wang al. (2023) fine-tunes the backbone LM in the RETRO architecture while freezing the cross-attention module and the memory encoder. In comparison, RA-DIT fine-tunes both the LM and the retriever while decoupling the fine-tuning processes of the two components T2 Asai et al. (2023) fine-tunes an LM to adaptively retrieve passages on de- mand and reflect on the relevancy of the retrieved passages and its generation using special-token
markups. The most related work to ours is SAIL (Luo et al. 2023). ), an approach that fine-tunes the LM with instructions augmented with retrieved content, and examines on public instruction following datasets (Taori et al., 2023] Chiang et al. 2023) using a moderately sized model (7B pa- rameters). In comparison, RA-DIT conducts parallel augmeettttngmnnttiioon for multiple retrieved passages while SAIL concatenates them in the LM context. Furthermore, RA-DIT adopts holistic view of the RALM architecture by employing learnable neural retriever and proposing a dual opti- mization framework. SAIL, in comparison, leans on non-differentiable retrievers such as BM25 and focuses on improving the LM (e.g. it proposes an in-context retrieval selection technique to guide the model focus towards informative content).
Information Retrieval Retrieval methods include sparse retrievers that does matching over a sparse bag-of-word. representation (Robertson & Zaragoza, 2009 Formal et al., 2021), dense re- trievers that embed queries and documents into a fixed-size dense vector for nearest-neighhor search (Karpukhin et al. 2020 Xiong et al.] [002]) and multi-vector retrievers which uses multiple vectors as the representation and more complex search algorithms for increased accuracy (Khat- tab & Zaharia 2020; Li et al. 2023). We adopt ofate-of-the-ane dense retriever, DRAGON (Lin ctt al.]|| 2023), as our base retriever, because of its simplicity, state-of-the-an art accuracy, high retrieval efficiency GPUs, and the ease of further fine-tuning.
7 CONCLUSION
In this paper, we propose RA-DIT, a lightweight Aerrivall-Aggentter Dual Instruction Tuning framework that can effectively retrofit any pre-trained LLM with retrieval capabilities. RA-DIT updates the L.M with retrieval-uugmente edge and ignore irrelevant or distracting information. also fine-tunes the retriever with supervi- sion from the LLM to retrieve texts that can better help the LLM generate correct outputs. RA- DIT achieves state-of-the-ar performance in zero- and few-shot evaluations on knowledge intensive benchmarks, surpassing un-tuned in-context RALM approaches such REPLUG and compete ef- fectively against methods that require extensive pre- pre-trainig such as ATLAS.
instruction tuning to make better use of retrieved knowl-
12 Although the differences in the base LMs, fine-tuning datasets and inference settings make direct compar- isons between the two models challenging, RA-DIT 65B compares favorably to InstructRetro 48B (Wang et al. 2023 in zero-shot setting on the shared evaluation datasets.

---

Published as a conference paper at ICLR 2024
REFERENCES
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for CommonsenseQQ: New dataset and models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, Au- gust 1-6, 2021, pp. 3050-3065. Association for Computational Linguistics, 2021, doi: 18.18/53/ acl- long. 238
www.-lonnl.o....... 238. PRLhttps:: / doi.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Humun I Language Technologies, Volume I (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019, Association for Computational Lin- guisties. doi: 10.18653/v 1/N19-
1245
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellstern Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hycontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov. Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue, Pengeheng Yin, Jiahui Yu, Qiao Zhang. Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 technical report, 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for
Computational Linguistics. URL http:: http:::://caaaahhoooooooooo wwwpaathhoooooooog/oooooo /tt///////00 1160
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence, 2020.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean- Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Ste- fanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. html https: /trpsee/prccediii ngs. ml .pr.spressee... 62/bbrrgaadd2a...
PMLR, 2022. URL
10

---

Published as a conference paper at ICLR 2024
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernrr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL Ittpps://proceeddnggs wtw.:eewedddin......ieeeee neurips wttppss.cccppeeeee httpa//ssseccpppppaeeeee///p/ppe files/paper/2020/ file
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL
https:
// http:///..r///og//020000
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tswyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint wtX.:i2224.00311.. 2022.
Douglas
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cstructioo-finetuune language mod- els. arXiv preprint arXiv: 10. 1416, 2022a.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022b. URL https/////wxwww..//abbbos..
11416
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ V1/N19-1300 URL laahhonnhhoooooo logy. www.lllgyylyoyg1gggg 9-3000
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try are. the A12 reasoning challenge. 2018
arXiv preprint arXiv:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21100.4166 2021. URL
aattps://arxx org/ abs / 2210011111.. 14168
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powert conversational agents. In International Conference on Learning Representations, 2019. httll.tttt/// http://///pppaaeeeeeeee/////a/aa

---

Published as a conference paper at ICLR 2024
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 2368 2378, Minneapolis, Minnesota, June 2019. Association for Computational I Linguistics. doi: 10. 10.18853/11/N19-11
URL /ttpppps//aaaaatoooooooooooo 922/619-126
Hady Elsahar, Pavlos Vougiouklis,, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- tion (ELRA).
wtwp///....///11// 1544
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558-3567, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/P19- 1346...999111.....
URL
orgg
P19- 1366
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and ex- pansion model for first Stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Comput- ing Machinery, 2021. ISBN 978 1754383383.79 doi: 10. 041114457244483344608888 URL https:
//doi. orw..org./0.. 11 1145/ 3404835 3463098
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021. doi: 10. 1621tall a 00370. URL
wrgp/h/logooooogoooooooooo 2021 tacl-......l......
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3929-3938 PMLR, 2020. URL http: pattp::/proceediiigg
mlr. peress/vi19// /ww.aa......../...... html
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. URL. neur ps. ca/ wttp///pp//////11//
haah///eB333ecdddd7337722abbbbb1 wtt.:/aaaaaaaaaaaaaaaaaaatt Abst round2 html
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1701, 2015. URL
neurips. http::/cww........oo.oooooeee//a html
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 792. Edinburgh. Scotland, UK., July 2011. Association for Computa- tional http.///s/////aaaaaaooooooooooo/
12

---

Published as a conference paper at ICLR 2024
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. Multi-task retrieval- augmented text generation with relevance sampling, 2022.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang. Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoy- anov. OPT-IML: scaling language model instruction meta learning through the lens of gen- doi: 10. (0.485500arXXx.2222220000 URL https:
eralization. CoRR, abs/2212. 12017.. 2022.
//doi.org/10: 48550/azzzz www.xxxv..222.2221
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/0021 Laacl- modd-mmin.744 URL hetps:
//aclanthologe wwg.gogg....l..11.. www.lmalanna.ae.eeeeee
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learn-
ing. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. https:
//openzeviww net/ forrmmiddjjKNigaee
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. CoRR, hbb//208.003299 2022b. doi: 10.48550//rxivv2208.. 03299. URL https: //ds:////o.../ http///...45///axii...... 2208 03299
Kelvin Jiang, Dekun Wu, and Hui Jiang. FreebaseQA: A new factoid QA data set matching trivia-style question-answer pairs with Freebase. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies. Volume (Long and Short Papers), pp. 318-323, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N1--1U
URL
org.Nogg//N 9-1028
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP--JCNiPP), pp. 2567-2577, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v/D119-1220 URL
1259
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: htt.tttss:///
10.18653/v1/P17-1111 / wttps///claathhoogggoe
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguis- httpttts:://///ttt aaattppe//acllnntheeooo..g.. org/220
tics. doi:
emm 1 pa-malaaam in.iinn... 550
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextu- bert. In Proceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, pp. 39-48. Association for Computing Machinery, 2020. ISBN 9781450380164, doi: 10.1145/3397271.340111 3401075
alized late interaction
URL
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1 162/tall a 00023.
URL _ hhttps://aat http:///claaahhhtccooo antho www.hhhoooo.oogooooooo--o
13

---

Published as a conference paper at ICLR 2024
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rur Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations-demrrattiii democratizing large language model alignment. arXiv preprint arXtvv2304.073227, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1162/tuclaa00226 URL ottps://aclantiioooy..oro.
1026
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettleoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Lan- guage Learning (CoNLL 2017). pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/11KK17-1000 URL /ttps:://aalaattee ogy.
org/K17-1034
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeenedd generation for knowledge-intensivy NL.P tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 01154445774811515
wt. pttpscccpppp//////aaaaa//// 2020, virtual, 2020. 044433300777000033345567760---00 html
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Can- wen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A commu- nity library for natural language processing. In Proceedings of the 202/ Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- 100.665////00....mmll
tics. doi: 10. 22.18653/v1/2211 http221..emmlppppe.e amnlp-demo.21. URL htt.httpss/acclaahhlloogy// org.2021...
emn p-delo...m..mmm 21
Minghan Li, Sheng- Sheng-Chich Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mchdad, Wen-tau Yih, and Xilun Chen. CITADEL: Conditional token interaction via dynamic lexical routing for efficient and effective multi-vector retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11891-11907, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 18.53/ 1/20211ac122221//// 663. URL paatttps:/aaclaathollooooo http:p////..oaaaaaoooooooor////.
wrtp//23..20000 acn-oogg...........o.. 663
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How train your DRAGON: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07455. 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- eration: Learning solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 168-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.118653/1//117-0133 URL uttp:tppp///elanttooooooooo////p
Jerry Liu. LlamaIndex, 11 2022. URL htttp:::/gittuu.ccom///////
index
Hongyin Luo, Lung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James R. Glass. SAIL: search- instruction learning. CoRR, abs/2305.15225. 2023. doi: 10.48550VaxXx.2230555522 URL https: //o. orq/10.
48550/ 22.....1.30...55
14

---

Published as a conference paper at ICLR 2024
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non non-parametric memories. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.88663//122023alllllooo org/ 2ww.............. -on.e..-ooo.... 546
URL
https://aclantho..
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? a new dataset for open book question answering, In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, Brussels, Belgium, / wwwpo/oooyyorr//--22eee- 10.18663/v1/D18-122
2018. Association for Computational Linguistics. doi:
Katharine Models?, 2023.
Miller.
How
Do
We
Fix and
Update Large
Language
URL
https: wtaaaiisttffrrrd.o edwirddsed///www////
how do fix- ax--and--- update- large language large-language models
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. CoRR, abs/ htw/1611.9920 1.09268, 2016. URL httt::/ http://dblp..oo uni-trier. wwt.d/eedd/////// abb/jouaaals/oor/ corr 61 html #NguyenRSGMMDI1
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744 Curran Associates, Inc., 2022. URL
www.sss.eeurr.p... cc httpappsscccpppeeeeeeeee/eeeeee File/blefee Fileeh/eereeebbee334333115553333 Ceaprrreee-ppppeeeoooeeeeeeeeeee Conference.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 URL
http:////22....ammmmamm.....mm. maaccl-main.200)
org/ www.aaaallmmm
200
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL httt:://jmm= org/
paperss/22120-0775 html
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10//53//1/P112211 URL https: //aclanthor www.aaaooooo.oogoooeeeoo.e
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- and Yoav Shoham. In-context retrieval augmented language models, 2023.
Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333 389, 2009. doi: 10.1561/ 1500000019. URL htRlsttss:.. //to:::///ooooeo http:///oooooooo/../ /tt..ooo//111111111000000000000
15

---

Published as a conference paper at ICLR 2024
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to Al complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020. The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8722-8731, AAAI Press, 2020. URL article/view/633
wwws//nnex.... php /AAI/
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. arXiv preprint arXiv: /907. 10641, 2019.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/w1/D19-1443 1454. URL ht htt.pp::/aa..nntooooyyy..oo wwwp:/oooyoorgiii... 1454.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 23-29 Jul 2023a. URL/https http:tttt///ppoeeeeeeeee http://poocceeenn.eeeee wwt.smmm..pper wtt.mm.......//e222m222m2 wttp:////22//shhh23a2 html
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. REPLUG: retrievallauugmented black-box language models. CoRR, abs/2301.12652, 2023b.
doi: 10-4555WarXix223111222. URL https:://oi..rg////
48550/arXiv.. 2301
12652
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (Ilm)? a.k.a. will iims replace knowledge graphs?, 2023.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. QUAREL: A dataset and models for answering questions about qualitative relationships. In The Thirty-Thine AAAI Conference Artificial Intelligence, AAAI 2019, The Thirty-Firss First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February
1, 2019,9 pp. 7063-7071. AAAI Press, 2019. doi: 10. w00./aaii.33311.333333333 1 1609 aaai. v33101. 330170633
URL https:
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4149-4155 Association for Computational Linguistics. 2019. doi: 10.18653//1/119-1444 1421. URL
https:
//doi. 1ttp://10..1653/1//n1..
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. https: //tttbbb.btbbb...../// thub. com/ coom/atts-- lawssss-aaaaaaaaaa tarb/stafood stanford_e lpaca 2023.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chupter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 809-819, New Orleans, Louisiana, URL
June 2018. Association for Computational Linguistics. doi: 1..18653/1/N118-100
1074
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memo- rization without overfitting: Analyzing the training dynamics of large language models. In
16

---

Published as a conference paper at ICLR 2024
NewrIPS, 2022. 205.:00065587222441111122222--22
haahh// html
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 191-200, Vancouver, Canada, Au- gust 2017. Association for Computational Linguistics. doi: 10. 1865///1//1722222.. URL
org/
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shocybi, and Bryan Catan- zaro. Instructretro: Instruction tuning post retrieval augmented pretraining, 2023.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashii, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mi- rali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Pa- tro, Tanay Dixit, and Xudong Shen. Superr-atturallnsstutttoo
Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109 5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 206533///222222m/ppmmm/ ht httt::://aclantiooooy.ooog matp://pmmm...mppmaaaaaaam iin... 340
URI.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022. URL https: http://ppnneeeeeeieeeeeeeeeeeeep
forum?id-gE2rrGozzag
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnol Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021. URL https: 7/openrevi ew. net./ feet/foormm? dtw.:m/====eeffx.x.
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2013-2018, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10. 10.18653/11/D15-122 1237.
URL
http:://aaaaattooyyyooooo
1155-2377
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
17

---

Published as a conference paper at ICLR 2024
Processing, pp. 2369-2380 Brussels, Belgium, Detober-November 2018. Association for Com- putational Linguistics. doi: 10.18653//1/D18- 1259. URL httpss//aclaathoooogyyy. org/
D18-1233
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4791-4800 Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 1/11165//1/1911/// 1472. URL httpss//aclennhoooogg..ooo
P19-1472
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
18

---

Published as a conference paper at ICLR 2024
A RETRIEVAL CORPUS
We combine the text chunks from the Dec. 20, 2021 Wikipedia dump released by Izacard et al. (202b)) with additional ones from the 2017-2020 CommonCrawl dumps. The Wikipedia dump includes lists and infoboxes in addition to regular articles. The articles are split by section, where long sections are further split into text chunks of equal sizes and contain less than 200 words, leading to a total of 37M text chunks. We randomly sample a subset of articles from the CommonCrawl dumps, and split them into equal-sized text chunks that contain less than 100 white-space-separatedd words, leading to a total of 362M text chunks.
We use GPU-based exact n--narret--eighhbor search index implementation released by Izacard et al. (2022b).
B IMPLEMENTATION DETAILS
Fine-tuning Dataset Selection Prior work (Chung et al. 2022b Iyer et al. 2022) have demon- strated that jointly fine-tuning the language model on diverse collection of citrucction-base datasets leads to improved model generalization for unseen instructions. We adopt a similar strat- egy by combining five categories of fine-tuning tasks to enhance the language model's knowledge utilization (dialogue, open-domain QA, chain-of-thooe of-thhught reasoning) and to improve its contextual awareness for prediction generation (reading comprehension, summarization). These categories were selected due to their representativeness of practical knowledge intengivvtte language tasks.
Retrieval-augmmeneed LM Fine-tuning We use the top-3 retrieved text chunks for a given exam- ple (i.e. k 3) to generate the fine-tuning instances. To improve fine-tuning efficiency, pack multiple examples up to the language model context window limit (2048 tokens). Each example is demacrate by pair of <bos> and <eos> tokens, and we adopt the document attention mask- ing ([yer et al., (022) such that token only attends to the previous tokens in the same example. We use dataset mixture that contains 10% unsupervised text and 5% OASST-1 data. For the remaining datasets, we establish a cap on the number of examples per dataset at 17 = 7500 based on the model performance on our development set We then randomly sample batches in accordance with this adjusted mixture probability.
We fine-tune the 7B, 13B and 65B LLAMA models using 8, 16 and 64 A GPUs, respectively. The fine-tuning hyperparameters are detailed in Table Similar to Zhou et al. (2023), we found that the best generalization performance on the dev set can be achieved using a small number of fine-tuning steps. We evaluate the models every 100 steps, and select the best checkpoint based on the average dev set performance over the 6 development KILT tasks shown Table II (early stopping).
Table 8: Hyperparameters for retrieval-auggeente LM fine-tuning.
early stopping
model parallel
Model
peak
end Ir
Ir scheduler
warm-up
# steps
batch size
seq len
RA-DIT 7B
le-5
lc-7
cosine
200
500
500
64
2048
RA-DIT 13B
le-5
le-7
cosine
200
500
400
128
2048
RA-DIT 65B
le-5
le-7
cosine
200
500
300
128
2048
64-shot Eval Task Fine-tuning Table 9 summarizes our hyperparameters for 64-shot fine-tuning on the 9 KILT eval tasks shown Table except for MMLU. Given the small amount of examples used (64 x 9 = 576), we fine-tune for a significantly less number of steps at this stage without using warm-up. We evaluate the model every 50 steps, and select the best checkpoint based on the average dev set performance over the 6 development KILT tasks shown in Table [1
Retriever Fine-tuning We employ both unsupervised text and downstream tasks for retriever fine- tuning. For the corpus data, we randomly sample 900k text chunks from our retrieval corpus to
httpps: // / thub. com/ facebook wwweeeeheea..... We did thoroughly tune this parameter avoid overlitting the development sets.
19

---

Published as a conference paper at ICLR 2024
Table 9: Hyperparameters for 64-shot fine-tuning on the eval tasks.
carly stopping
model parallel
Model
peak Ir
end Ir
Ir scheduler
warm-up
f steps
batch size
seq len
LLAMA 65B
le-5
le-6
linear
0
100
100
2048
RA-DIT 13B
le-5
le-6
linear
0
100
50
32
2048
RA-DIT 65B
le-5
le-6
linear
0
100
50
32
2048
form a set self-supervised data, using the first 50 tokens of each chunk as the input æ and the last 50 tokens the ground-truth output y. In addition, leverage the multi-task instruction tuning datasets (MTI data) as shown in [ including 10 open-domain question answering and dialog tasks, with total of 286k training examples. As discussed in $5.1 we observe that, when used alone, the corpus data works slightly better than the downstream tasks. However, combining both types of ane-tunnng data yields the best results and outperforms using either source alone. Therefore, we adopt a mixture of 95% corpus data and 5% downstream tasks for retriever fine-tuning in our final model.
We fine-tune the DRAGON+ retriever on 16 A100 GPUs using the dpr-scale codebase [5 The re- triever is fine-tuned using learning rate of le-5 with 1237 warmup steps (DRAGON default), per-GPU batch size of 32, and temperature т = 0.01, for single epoch over a combination of 5% MTI data and 95% corpus data. We adopt the KL-divergence loss as discussed Section 2.4 using the top-10 retrieved chunks for each example. For simplicity and efficiency, we produce the top-10 retrieved chunks and their LSR scores (Equ. 4 using LLAMA 65B and DRAGON+, and do not update them during R-ft. Furthermore, only the query encoder is fine-tuned, there is no need to update the chunk embeddings in the retriever index. Model validation is performed once every 500 steps using the same reciprocal rank (MRR) metric as in the original DRAGON paper (Lin et al. 2023), on a combined validation set from the 10-task MTI data.
Inference Without further specification, we use the top-10 retrieved text chunks for given ex- ample (i.e. k= = 10) and ensemble their predictions during inference. For multi-choice tasks, we compute the weighted average probability of each choice items according to Eq. 2 and select the choice with the highest probability. For generation tasks, we perform decoding using each aug- mented prompt independently, compute the weighted average probability of each unique generated answer, and output the answer with the highest probability When computing probabilities of out- put answers, we use several scoring functions: "nll"", "nll_char", "nll_token", and "nll_ "ompp"" "nll" the sum of negative log likelihood across all tokens in the sequence. "nll char" and "nll token" are "nll" divided by the numbers of characters and subword units in output answers respectively. "nll "omp_" selects answers based on the probability divided by the probability of the answer given
"Answer:": p(yl "
p(p(ar)
C FINE-TUNING DATASET TEMPALTES
Table 10 shows the templates we used to serialize our instruction tuning datasets. Following Chung et al. (2022b) and Tyer et al. (2022), randomize the field markers used during training to avoid overfitting. In pariticular, when serializing a task example, we randomly sample from ("Q:", "Ques- tion: and <**) for <inst_s>, set <inst_e> to "In" and randomly sample from {"A:", "An- swer:") for <answer s>.
D EVALUATION DATASETS AND TEMPLATES
Table II shows the evaluation datasets used in our experiments. For dev set evaluation, we use a maximum of 2500 randomly sampled examples from the respective official dev sets reduce the
https: thub. com/ www.bocceeookkkea wwwpoeccweeeeeceeeeeeeeeeeee.eee
16 A more sophisticated implementation of ensembling for generation tasks involves computing weighted ensemble of the output distribution every step and then sampling from this distribution. However, for the simpler implementation performs reasonahly well and allows exccute inference with fewer GPUs.
20

---

Published as a conference paper at ICLR 2024
Table 10:
Instruction template used for fine-tuning datasets. <inst_s>, <inst_e> and <answer_s> are special markers denoting the start and the end of field.
Category
Instruction Tuning Template
Query Template
Dialogue
Background: (retrieved passage) (assage)](nnan n (turn)) (turn)) Q: {turm) {tuma) (tums)
[tumg]
Open-domain QA
Background: (reirieved
passage [n]nsinntt.ss [quesûin]] qqeesion]
<isst_e> kanswer_ss
{answer)
Reading Compre-
Background: [context]
in [question) <inst.e> {question]
hension
<answer_s> (answer)
Summarization
Background: [contexi] i this article: <inat.e>
<answer_s> [summary)
Chain-of-hhoughi
Background: (retñived passage) (nli<inst.s>> (mniinnttsssssttttiiiinn
(instructions)
{question]
Reasoning
freasoning c <answer_ss (answer)
Table 11: Our evaluation datasets. hyperparameters.
indicates the development datasets we used select fine-tuning
Task
Datasct name
Acronym
Metric
Score
MMLUU (?)
MMLU
Acc.
nll
Natural Questions Kwiatkowski al. 2019
NQ
EM
nll
Open-domain
TriviaQA Joshi el al.
2017
TQA
EM
nll
QA
HHoppotQA Yamg et a
2018
HoPo
EM
nll
EI.15 (Fan ct al. 2019
ELIS
Rouge-T.
nll nllett
Fact Checking
FFEVER (Thorne al.
2018
FRV
Acc.
nll
Entity T Linking
TATDA CONTTI-YACH (Hoffart
2011
AIDA
Acc.
nll
Zero-Shoi (Levy et al. 2017)
zsRE
Acc.
nil
Slot Filling
IT-REx (Elsahar et al. 2018
T-REx
Acc.
nll
Dialogue
Wizard of Wikipedia (Dinan al.
2019
WoW
FI
nll token
BoolQ (Clar al 2019
BoolQ
Ācc.
noCoopppl
PIQA Brik et al.
2020
PIQA
Acc.
nll char
SIQA er el at.
2019
SIQA
Acc.
nll_char
Commonsense
HellaSwag (Zellers
2019
HellaSwag
Acc.
nll
Reasoning
WinoGrande (Sakaguchi el
2019
WinoGrande
nll all_char
ARC-Fasy (Clark al. 2018
ARC-E
nll_char
ARC-
CCC-Chaleenge (Clark 2018
ARC-C
Acc.
nll char
OpenBookQA (Mihaylov lov 2018
OBQA
Acc.
nll_compl
computational cost. For test set evaluation, we use the full set to ensure fair comparison with pre- vious work. The language model instruction templates and retriever queries used in our evaluation are shown in Table We randomly select few-shot examples from the official training splits of the KILT tasks, except for FEV, NQ and TQA, where we use the 64-shot examples released by Izacard et al. (2022b). For these three datasets, we also ensure that the 5-shot examples are subsets of the 64 examples. For retrieval augmented models, we use the top-1 relevant chunk augment the prompt for each in-context few-shot example.
ADDITIONAL EXPERIMENTS
E SCALING LAWS OF RETRIEVAL. AUGMENTED LANGUAGE MODEL. FINE-TUNING
We investigate the impact of the base language model size when retrieval-augmented instruction tuning is applied, and summarize the results in Figure 2 the base DRAGON+ retriever this set of experiments.
We combine the fine-tuned models with
Overall, all models substantially benefit from retrieval augmentation, with smaller models witness- ing even bigger improvements. We further note that retrieval augmentation can be an effective strategy for enhancing the performance of smaller models (hence reducing pre-training and infer- ence costs), given the 7B model leveraging > retrieved chunks surpassed the performance of the vanilla 65B model several tasks. This trend also differs across tasks. For tasks that primarily
21

---

Published as a conference paper at ICLR 2024
Table 12: Language model prompts and retriever query templates used for evaluation datasets. We did not perform retrieval for commonsense reasoning tasks evaluation.
Task
LLM Prompt Template
Query Template
Knowledge-Intene
Tasks Background: fretrieved passage) \n In'nnuesttion {question) J mmA.
MMLU
(question) A.
[choice) / (nB [choice) C,C (choice) \nD {choice) ) InA; (answer)
choice
nBB
chnice
nnC.
cchiice)
nD. {choice)
NO,TTQ,,EL ELIS,
Background: (reinieved passage] passagee (5[ [question] [ñA [answer]
(question)
HoPo, zsRE
AIDA
BacKground: Trerneved passage S Tn Conteece
"the"
{ciniext] "tokens be-
Wikipedia page title of the entity mentioned between [START_ENT and EEN] in the given text fanswer)
tween [START_ENT and [END_ENT] (statement)
FEV
Background:
(retrieved
passage Ial statement true?
statement] (answer)
T-REX
Background: fretrieved passage (nn TSEPT [relation]" {enify_[]
AA: {answer) Background: (rétrieved passage] Tr) [lumn PSA: (lumieFinee [turni] J [turni] [uirna] ...
WOW
[turna] ... ..:::A: (answer)
Commonsensse Reasoning Tasks
ARC-E, ARC-C
Question: (question AnAnswrr (answer)
BoolQ
context]
Question: {question) Annnser (answer)
HellaSwag
context]] [ending]
OpenbookQA
[question] (answer)
PIQA
Question: [question] (nmestioon//Annwwwerr (nAnswer: (answer)
SIQA
[context] Q: {question) A: (answer)
WinoGrande
[prefix] {answer) (suffix}
model_size
HotpotQA
FEVER
Zero-Shot RE
95
40
60
90
35
exact. match
50
30
80
75
25
30
65
20
60
10
10
10
0
3
10
top-k chunks
top-k chunks
top-k chunks
T-REx
CoNLL-YAGO
WoW
16.6
60
16.4
40
50
16.2
40
30
16.0
30
20
20
15.6
15.4
10
top-k chunks
top-k chunks
top-k chunks
Figure 2: RA-IT model performance (combined with DRAGON+) across sizes 7B, 13B and 65B our development tasks. 0-shot performance: dashed lines; 5-shot performance: solid lines.
measure one-hop fact look-up abilities (such as Zero-Shot RE and T-REx), retrieval augmentation provides significant improvements across all model sizes and bring the performance of smaller models closer to that of their larger counterparts. For more complex tasks (such as HotpotQA and WoW), the advantage of using larger L.IM remains prominent.
22

---

Published as a conference paper at ICLR 2024
Table 13: Comparison between parallel rerrieval-angnmentttion and chunk concatenation. The results are obtained using the base DRAGON+ retriever.
O-shot
HoPo
FEV
AIDA
zsRE
T-REx
WoW
Avg
top-3 chunks
RA-IT 65B (parallel)
39.9
82.4
45.2
63.4
52.8
16.6
50.1
RA-IT 65B (concat)
39.5
83.9
52.2
65.2
47.9
16.6
50.9
E.2 COMPARE PARALLEL RETRIEVAL AUGMENTATION TO CHUNK CONCATENATION
We adopt the parallel retrieval-auge -augevattangeetttiionnn approach proposed by Shi et al. (003b) reduce the prompt length, which is necessary in the few-shot settings ($2. Tp. However, this approach is computationally expensive when the individual prompts share long common prefixes (as in the few- shot setting). In addition, by separately encoding the text chunks, this approach is potentially less effective for knowledge synthesis compared to concatenating the retrieved text chunks in a single prompt. To understand the impact of using parallel retrieval-augmentaiee chunk concatenation approach under the setting with top-3 retrieved text chunks. We conduct this experiment using the RA-IT 65B model and O-shot evaluation,
we compare it to the
According to Table 13 the two approaches perform closely on average with chunk concatenation demonstrating small benefit. Specifically, parallel retrieval-augmentation under-performs chunk concatenation on FEVER and Zero-shot Relation Extraction, and perform on par on Wizard of Wikipedia. It also performs slightly better on HotporQA, which is somewhat unexpected, given the dataset specifically designed to necessitate multiple evidence sources for answering a question. We observe wider performance gaps between the two approaches CoNLL-YAGO and T-REx, where concatenation performs much better on the former but worse on the latter.
It is worthnoting that the RA-IT 65B model has been fine-tuned using parallel retrieval augmen- tation, which potentially provides a benefit to using the same configuration during inference. We defer the investigation fine-tuning with chunk concatenation future studies. This direction appears promising, especially considering that state-of-the-art language models are progressively being trained with ever-larger context windows
E.3 RETRIEVAI. CORPORA ABLATION
Table 14: Retriever settings: We report 5-shot dev set performance using LLAMA 65B and various retrievers the REPLUG setting.
5-shot
MMLU
NQ
TQA
HoPo
FEV
AIDA
zsRE
T-REx
WoW
ELI5
Avg
LLAMA 65B
61.3
30.9
70.6
23.8
83.7
50.2
36.0
52.3
17.4
23.4
45.0
Retriever corpus ablation using
LLAMA 65B and the DRAGON+ retriever
CC only
62.8
39.6
72.6
34.4
89.5
54.8
30.3
46.2
17.1
22.9
47.0
Wiki 2021 infobox
62.2
42.0
71.2
41.8
89.8
62.2
65.3
73.1
17.7
22.2
54.8
Wiki 2021
62.2
41.8
71.0
41.7
89.7
62.1
65.2
73.3
17.6
22.2
54.7
Wiki 2018
61.5
42.6
70.7
40.4
90.8
62.1
51.3
59.8
17.6
22.5
51.9
Table 14 shows the impact of varying the retrieval corpora. particular, we consider several subsets of our 399M retrieval corpus, namely CommonCrawl only (362M) and Wikipedia only (with and without infoboxes). We further compare with another Wikipedia snapshot (Wiki 2018) commonly used in the literature (Karpukhin et al. 2020). We observe that retrieving from Wikipedia only is beneficial for a number of KILT tasks such AIDA and zsRE, as Wikipedia was the intended corpus for KILT tasks. We find that Wiki 2018 works better for NQ since the corpus is closer to the date of its data collection, similar to the observations by Izacard et al. (2022b). This indicates that our retrieval-augmeentr LM is faithful to the supplied retrieval corpus, and up-to-date information can be provided by updating the retrieval index at test time.
"https://opena" wtt.e//aa...cmc....ooo///
23

---

Published as a conference paper at ICLR 2024
F EXAMPLES
In this section, we show the task prompts, the corresponding retrieved passages and model pre- dictions generated LLAMA 65B instruction-tuned with retrieval augmentation (RA-IT 65B) and LLAMA 65B instruction-tuned conventionally (IT 65B) on selected task examples.
F.1 HOTPOTQA
We analyze the performance of the two models on the development set of HotpotQA in the zero-shot setting since under this setting RA-IT 65B outperforms IT 65B by large margin. Table 15 show two examples from the HotpotQA development set where RA-IT 65B makes a correct prediction while IT 65B makes wrong prediction. First, observed that the dense retriever struggles to return useful text chunks for the multi-hop questoins in the HotpotQA dataset and most of the returned text chunks contains information that helps the prediction. In this case, the IT 65B model shows a stronger tendency to be misled by distractors within the retrieved text chunk, since has not been trained with noisy passages during fine-tuning. It also tend predict "I don't know" more frequently 18 while the RA-IT 65B can ignore the noisy passages retrieved and predict the correct answer based on its parametric knowledge (Mallen et al., 2023). We also observe that in cases where both models generate wrong predictions because of the distractors (e.g. for the third text chunk in the second example), the generation probability of the wrong answer from RA-IT 65B is much lower; and in cases where both models ignore the noisy passages and rely on the parametric knowledge to make prediction, RA-IT 65B outputs the correct answer with a higher probability (e.g. for the second text chunk in the first example).
As discussed in 2.2 this behavior is induced by fine-tuning SQuAD v2.0 (Rajpurkar et al. 2018 which trains the model predict "I don't know" for passages that does match with the given question.
24

---

Published as a conference paper at ICLR 2024
Table 15: Example predictions in HotpotQA (dev set) in the O-shot setting ensembling 10 retrieved text chunks. The top-3 retrieved chunks and the corresponding model predictions are shown. RA-IT 65B and IT 65B are used to generate these outputs.
Output
Prompt
RA-IT IT
RA-IT IT
Input: Charlotte Hatherley initially came prominence in a band formed in what year? Label: 1992.
RA-IT 65B final prediction: 1992.
IT 65B final prediction; 1997 X
Background: Charlotte Hatherley Born in London, Hatherley was brought up West London and attended Chiswick Community School. Her music career began at the age of 15, when she joined British punk band Nightnurse. Two years later, with Ash looking for guitarist to add to their live sound. Hatherley was hired after frontmun Tim Wheeler saw her play Nightnurse gig. Hatherley's 's Ash debut was at Belfast's I Limelight on 10 August 1997, and the following week the new lineup played the 1997 V Festival in front 50,000 people. Her recording career with the hand hegan later that year the single Á Life Less Ordinaryänd continued the album Nu-Clear Sounds in 1998. Hatherley was a full-time member of Ash for eight years. playing on three studio albums, and wrote handful of the band's songs, most notably Grey Will Fade; on the B-side of the single There's Star, The song
0.27
1992
1997
1.16 1.01
cult favourite among fans, and eventually became the title track of Hatherley's 's debut solo album. On 20 January Hatherley would be leaving
2006 it was announced
in an amicable breakup. (n) Charlotte Hatherley initially came to prominence a band formed in what year?)nA:
Brccground: WM: Charlotte Hatherley only... so CD fans might still have to shell out hig (1.21 bucks for an import. Oh, in case you were wondering who Hatherley is, first heard of her as the gidl guituristin the bund Ash band that I have been fan of since the early 90s when I was getting into all these Britpop-type bands. She nuturally started doing her own solo material and left the band few years ago. The last heard of her was she was in the band new waver Client with Kate Holmes initially came to prominence in a band formed in what (n.:
1992
1992
01.46
0.98
to be confused with the n)) Charlotte Hatherley
Background: Charlotte Hatherley Charlotte Franklin Hatberley (born 20 June 1979) is an En- 0.13 1992 glish singer, songwriter, guitarist and soundtrack composer. She initially came to prominence guitarist and backing vocalist for alternative rock band Ash. Since leaving Ash in 2006, she has pursued solo career and ucted as a touring instrumentalist for Bryan Ferry, KT Tunstall,, for Lashes, Cold Specks, Rosie Lowe and Birdy. Hatherley has also heen touring mem- ber of NZCA Lines and is curreatly musical director for South African artist Nakhane. Aa/aa: Charlotte Hutberley initially cume to prominence in a bund formed in what year?),nA.
I don't 0.54 0.72 know.
Input: Osley Highway ends crastal town that had how many inhabitants in June 2016 Label: 45,698.
RA-IT 65B final prediction: 45,698 /
IT 65B final prediction: don't it know. X
Background: Oxley Electorate: Ipswich Motorway: Dee 2016: House debates (OpenAus- 0.25 10,000 don't 7.27 0.61 tralia. org) Oxley Electorate: Ipswich Motorway The Ipswich Motorway vital link sup- porting the Queensland economy. It forms part of the national land freight network pro- viding connectivity
know.
Acacia Ridge intermodal facility, the major industrial area of Wakool and the Brisbanc markets Rocklea2014in 4in the member torate201 4which are the state's 's lurgest fruit and vegetable markets and mujor centre for produce on the cast coast. The section of the motorway is over capacity with 93,000 vehicles on average each day, including up 12,000 freight vehicles. Numbers increusing euch year at an average of four),))==Q Oxley Highway ends at coastal town that had how many inhabitants in June 2016 P,AA:
industry
Morton's clec-
Background: Post Offices For Sale NSW Lotto Newsagencies Marlow & 45,698 45,698 0.18 0.38 Co South Wales about 390 km north of Sydney, and 570 km south of Brisbane. The town is located on the Tasman Sea coust, at the mouth of the Hastings River, and at the eastern
the Oxley Highway. The town with its suhurhs had population of 45,6998 in June 2016. Port Macquarie is a refreement destination, known for its extensive beaches and waterways. Port Macquarie has a humid sub tropical climate with warm, humid summers and mild winters, with frequent rainfall spread throughout the year. Port 2011 central business district contains two shopping centres, murina, the beginnings of nfnnn Oxley Highway ends at a coastal town that had how many inhabitants in June 2016 P.A:
The Long Paddock THE LONG PADDOCK Long Paddock 4x4, 4WD, 76,000 76,000 caravan, camper trailer, camping products reviews, tests, comparisons by Mark Allen The Long Puddock west, the Oxley Highway is the track you201911 be aiming for and Tamworth the major western town of reference on the map. Once you2019re in the main streets of Port, you20191 911 wonder no more why in excess of 76,000 people now call the area home, As a rough breakdown, the majority of locals are 25 to 44, followed closely by the 45 to 64 year old bracket 2013 just perfect for all you thrill seeking middle aged folk and laid back grey nomads and 1er2019s forget about younger sei now have oodles schooling and after-schooling (nlaa Oxley Highway ends coastal town that had how many inhabitants in June 2016 nnA:
Background:
0.93
25


Reviews:
[{"review_id": "Ov0C50ND8B", "reviewer": "Reviewer_68qv", "summary": "This paper proposes a method to endow LLMs with retrieval-augmented generation capabilities through a lightweight fine-tuning procedure that is in-between costly ad-hoc retrieval-aware pretraining and cheaper but subobtimal post-hoc integration techniques.\nThe proposed approach consists in two fine-tuning steps: one to update a pretrained LLM to make good use of retrieve information and one to fine-tune the retriever itself.\nThe paper shows that each of these procedures improve the performance on tasks that require knowledge utilization and contextual awareness, achieving state-of-the art performance across multiple knowledge-intensive benchmarks.", "questions": "* Connecting back to the point above, it seems that the final version of the algorithm consists in fine-tuning the LM using the pretrained retriever, and then the retriever is fine-tuned using this fine-tuned LM. Is that indeed the case?\n* Either way, what would be the drop in performance in actually performing the two fine-tuning steps independently, as opposed to fine-tuning the retriever with the (as said presumably) already fine-tuned LM?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The method is well-motivated in terms of practical applicability in providing pretrained LLMs with retrieval capabilities\n* Reproducibility is ensured by the detailed documentation of the used instruction tuning datasets and at what stage of fine tuning they are being used\n* Baseline comparisons and ablation studies are thorough and convincing", "weaknesses": "* The proposed components are well motivated and clear, but the presentation could make it easier to get a sense of the paper in a quick glance. For instance, it might be beneficial to expand Figure 1 and its caption to help in that.\n* The method is presented as two independent fine-tuning steps. On the other hand, the two fine-tuned elements depend on each other: LM-fine-tuning uses the retriever, while the retriever fine-tuning needs the LM to score the outputs. It could be beneficial for the understanding of the paper to emphasize this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZiFQ9hVBOl", "reviewer": "Reviewer_5vF7", "summary": "This paper proposes retrieval-augmented instruction tuning for pre-trained LLM. It also finetunes retriever.", "questions": "Detailed comments:\n\n1. “To stay within the context window size limit, each retrieved chunk is prepended individually to the prompt”\n- Llama has 2k context. What’s the length of the retrieved chunk? How many chunks (top-?) are used in the experiment? Did you try to pack top-5 all-together into the prompt?\n\n2. “Secondly, even state-of-the-art retrievers can falter and return inaccurate results. By training the LLM to make correct predictions when a wrong retrieved chunk is given, we enable the LLM to ignore misleading retrieval content and lean into its parametric knowledge in such cases”\n- This process may encourage the LLM to hallucinate if i) answer is not in retrieved context, and ii) the required knowledge for answering the question is not stored in LLM parameters. \n\n3. For baseline Llama 65B, did you apply instruction tuning using the same blended dataset? In Table 4, the improvement from IT 65B vs.RA-IT 65B is pretty marginal with the top-1 chunk. \n\n4. Could the authors also provide the results using top-5 in Table 4?\n\n5. In Table 2 main results, why is Llama 65B so bad on NQ? Do you evaluate it in a close-book manner i.e., non-retrieval setting?\n\n6. In Table 5, fine-tuning retriever seems to only provide very marginal improvements.\n\nConcurrent work:\n\nInstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining.\nIt would be recommended to include relevant discussion in related work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Important topic.\n- Overall good results on knowledge intensive tasks in contrast to previous methods.", "weaknesses": "- Fine-tuning the retrievers seems to be unnecessary given the very marginal improvement. However, this paper is written in a way that encourages the fine-tuning of retrievers.\n- The retrieval-augmented instruction tuning may encourage hallucination. See detailed comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MH7jZV87k0", "reviewer": "Reviewer_HttW", "summary": "The paper proposed the Retriever-Agument dual Instruction tuning, aimed to get the pretrained language model to better understand the retrieved text. The paper constructs a dataset, consisting of different domains (Open book QA, summarization, general conversation, etc.). The dataset improves the performance of the original model by a good margin.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents very solid experiments on the instruction tuning. \n2. The paper is very well presented.\n3. The ablation of the paper is solid.", "weaknesses": "1. The author mentioned the early stopping, training the model for ~500 steps would bring the best performance. How is the dataset size scaling with this? Can we change the cap of 7000 data points per domain to smaller or larger? will the performance change?\n2. Is there a criteria of how to choose the domains? And does jointly training on multiple domains help boost the performance? The choice of different domains seems a bit random.\n3. How sensitive is the model to different type of retrievers? Like can contriever/dpr work in this setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "meJvROCWap", "reviewer": "Reviewer_i49b", "summary": "This paper introduces a two-stage fine-tuning process aimed at enhancing a pre-trained language model's performance. In the first stage, the focus is on improving the LM's ability to effectively utilize retrieved information, while in the second stage, the retriever is fine-tuned to provide more contextually relevant results as desired by the LM. The study reveals that both of these stages contribute significantly to performance enhancements. Furthermore, when both stages are combined, even greater improvements are observed. The proposed model, RA-DIT 65B, achieves state-of-the-art results in various knowledge-intensive zero-/few-shot learning tasks, surpassing existing in-context RALM approaches by a substantial margin.", "questions": "If any of the concerns raised stem from misunderstandings, kindly bring them to my attention. I would greatly appreciate the opportunity to gain a better understanding and, if necessary, reevaluate the content in question", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* One of the standout strengths of this paper is its ability to outperform other presented baseline models. \n\n* The paper's approach offers practical value by demonstrating that the fine-tuning process is lightweight. This means it can be implemented efficiently, making it a more accessible and feasible solution for real-world applications.\n\n* The method presented in the paper is well-motivated and clearly described, enhancing its accessibility and potential for replication. This transparency in methodology ensures that other researchers can easily understand and build upon the work, further advancing the field.", "weaknesses": "* The paper's architecture lacks significant novelty as it primarily relies on existing models and introduces minor modifications. This may limit its impact and originality in the field.\n\n* The paper raises concerns about the accuracy of the ATLAS scores in the 64-shot fine-tuning results presented in Table 2. Discrepancies between the reported scores and those from the ATLAS paper, as observed in Table 10 of the ATLAS paper, undermine the credibility of the findings.\n\n* The paper does not include a comparison with a strong baseline model, such as FID+RS (Hofstätter et al., 2022). If included, it is suggested that FID+RS would outperform the proposed model, based on reported scores from the FID+RS paper. Additionally, FID+RS has the advantage of a significantly smaller model size, which raises questions about the efficiency and resource requirements of the proposed model. (11B vs 65B)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)