Making Retrieval-Augmented Language Models Robust to Irrelevant Context
Authors: Ori Yoran
Tomer Wolfson
Ori Ram
Jonathan Berant
Venue: ICLR 2024 poster (2024)
Keywords: Retrieval Augmented Language Models
Large Language Models
Robustness
Question Answering
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present a thorough analysis of cases where retrieval augmentation hurts performance of large language models, and propose methods that improve their robustness to irrelevant context, thereby increasing their overall performance.

Abstract: Retrieval-augmented language models (RALMs) hold promise to produce language understanding systems that are are factual, efficient, and up-to-date. An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant, and does not harm performance when it is not. This is particularly important in multi-hop reasoning scenarios, where misuse of irrelevant evidence can lead to cascading errors. However, recent work has shown that retrieval augmentation can sometimes have a negative effect on performance. In this work, we present a thorough analysis on five open-domain question answering benchmarks, characterizing cases when retrieval reduces accuracy. We then propose two methods to mitigate this issue. First, a simple baseline that filters out retrieved passages that do not entail question-answer pairs according to a natural language inference (NLI) model. This is effective in preventing performance reduction, but at a cost of also discarding relevant passages. Thus, we propose a method for automatically generating data to fine-tune the language model to properly leverage retrieved passages, using a mix of relevant and irrelevant contexts at training time. We empirically show that even 1,000 examples suffice to train the model to be robust to irrelevant contexts while maintaining high performance on examples with relevant ones.

Published as a conference paper at ICLR 2024
MAKING RETRIEVAL-AUGMENTT LANGUAGE MODELS ROBUST TO IRRELEVANT CONTEXT
Ori Yoran' Tomer Wolfsson 1. Ori Ram' Jonathan Berant
Tel Aviv University, 'Allen Institute for Al {ori. yoran, ori.ram, joberantyecs. tau. tomerwaallenai.ogg
ABSTRACT
Retrieval-augmenttd language models (RALMs) hold promise to produce lan- guage understanding systems that are factual, efficient, and up-to-date. An important desideratum RALMs, is that retrieved information helps model per- formance when it is relevant, and does not harm performance when is not. This is particularly important in multi-hop reasoning scenarios, where misuse of irrele- vant evidence can lead cascading errors. However, recent work has shown that retrieval augmentation can sometimes have a negative effect on performance. In this work, we present thorough analysis ing benchmarks, characterizing cases when retrieval reduces accuracy. We then propose two methods mitigate this issue. First, a simple baseline that filters out retrieved passages that do not entail question-answer pairs according to natural language inference (NI.)) model. This is effective in preventing performance re- duction, but at a cost of also discarding relevant passages. Thus, we propose a method for automatically generating data to fine-tune the language model to prop- erly leverage retrieved passages, including for challenging multi-hop tasks, using a mix of relevant and irrelevant contexts at training time. We empirically show that even 1,000 examples suffice to train the model to be robust to irrelevant contexts while maintaining high performance on examples with relevant ones.
five open-domain question answer-
INTRODUCTION
Large Language Models (LLMM)) (Brown et 2020 Chowdhery et al. 2022 Touvron et al. 2023) are the foundation top of which modern language systems are built. However, open-domain question answering (ODQA; Chen et al. 2017) and other knowledge-intensive tasks (Thorne et al., 2018 Petroni et al. 202T require vast amounts of up-to-date factual knowledge about rare entities that even very large models cannot memorize (Roberts et al.) 2020 Dhingra et al.) 2022). A domi- nant approach for combating this issue has been Retrieval Augmented Language Models (RALMs), which incorporate a retrieval mechanism to reduce the need for storing information in the LLM parameters (Guu et al., 2020 Lewis al. 2020b Izacard al., 2023 Rubin & Berant 2023) Fur- thermore, RALMs have also been shown to improve ODQA performance in an in-context setting (without any training), simply by prepending retrieved sentences
the input question (Ram al. 2023). Nevertheless, retrievers are not perfect and past work has shown that noisy retrieval can negatively affect LLM performance (Petroni al.] 2020 Li et al. 2023). For example, Fig. I when posed with the questions "Who is playing Jason on General Hospital?" a vanilla LLM (left) correctly answers the question while the RALM (right) is "distracted" by irrelevant context about the actor portraying Cooper, not Jason.
In this work, we analyze and improve the robustness of RALMs noisy retrieved contexts. Our def- inition for retrieval-robust LLMs states that: (a) when relevant, the retrieved context should improve model performance; (b) when irrelevant, the retrieved context should not hurt model performance. To this end, we present two methods for retrieval-robustnesss in RALMs 2
First, we consider a setting where we have black-box access to the LLM and cannot train it. Rather than solely relying in-context prompting (Brown et al.) 2020). frame retrieval robustness as a natural language inference (NLI) problem (Dagan et al. 2006 [owmman et all, 2013). Namely, given question and retrieved context, NLI model can predict whether question-answer pair

---

Published as a conference paper at ICLR 2024
Q: Who is the actor playing Jason on general hospital?
Large Language Model (no retrieval)
Retrieval Augmented Language Model
The answer is: Steve Burton
E: Jason Gerhardi (bom April 1874) American actor. known for playing the rola of Cooper Barrett in General Hospital and Zock Kilmer in Mistrassas.
The answer ls: Jason Gerhardt
An example from NQ where retrieval augmentation causes Llama-2-138 to err. Augment- ing with irrelevant retrieved context leads to error (right), although the model able to answer the question without retrieval (left).
Figure 1:
(hypothesis) entailed by the context (premise). Building on the strong performance of recent NLI models (e.g., in detecting model ballucinations (Honovich et al. (022) and attributed question answering (Bohnet al. (023)), we use such models to identify irrelevant contexts. When the context is labeled as irrelevant to the question-answer pair, generate the answer using the LLM without retrieval as a "back-off strategy". Our results show that this natural baseline is highly effective identifying irrelevant contexts, but is too strict and discards relevant ones well ($4).
We then propose method for training RALMs to be retrieval-robust. Intuitively, LLMs are not trained with retrieved passages, and thus brittleness to noisy retrieval is somewhat expected. There- fore, we perform an additional finetuning step that teaches the LLM to be robust to noisy contexts. The core challenge is to generate data for finetuning, and describe procedure for automatically generating such data for both single-hop and multi-hop questions. In the single-hop setting, assum- ing access to gold QA pairs and retriever, we create training examples using retrieved contexts, where we can use low-ranked or random passages as noisy contexts. In the multi-hop setting, train- ing examples need to contain not only retrieved contexts, but also intermediate questions, answers and relevant contexts, which comprise the question decomposition (Fig. 33. shown to be necessary for high performance on multi-hop questions (Wolfson al. 2020 Press et al. 2023). To generate decompositions to train on, we use strong LLM, prompted for decomposition without any re- trieval. Then, we can sample multiple decompositions, and use self-consistency (Wang et al.) (023)
to identify high-quality training examples ($3.233..
To test our methods, we evaluate retrieval robustness on five ODQA benchmarks, four of which contain multi-hop questions, where the retriever is called multiple times (Tiang et al.) 2202). Fig. 2 shows that even with a strong retriever (top-1 Google search) incorporating the retrieved context ac- tually hurts model performance adding randomly -eetrieee-reeereeedddde contexts dramatically decreases accuracy on all five datasets. Our analy- sis ($5) shows that irrelevant context causes wide range of errors, which include copying irrelevant answers from the retrieved sentences and hallucinating incorrect answers and decompositions.
two of the benchmarks (STRATEGY QA and FERMI). Moreover,
Our results demonstrate that finetuning L.Ms to be retrieval-robust enables them ignore irrelevant context while improving their overall accuracy ($4) When using a strong retriever at test time, our finetuned models outperform both models that were finetuned without retrieval, as well as untrained models prompted using in-context learning. To test robustness to noisy context, we evaluate QA accuracy when models are given randomly-retries perform on par with those that were finetuned without retrieval, demonstrating retrieval robustness. In addition, our ablation study shows that training models on mixture of relevant and irrelevant contexts results in models that are much more robust to irrelevant context.
contexts. In this setting, our finetuned models
To summarize, our main contributions are:
We conduct a thorough analysis on the robustness of RALMS to irrelevant retrieved contexts.
We show that small NLI models can be used to identify irrelevant context and improve robustness. without updating the model parameters.
We demonstrate that training LLMs when to use retrieval helps make models robust to irrelevant context and improve their overall performance, including in challenging multi-hop tasks [
'Our code, data, and models are available httttp:://ggthhbb..o////////////
thub. com/
robust
2

---

Published as a conference paper at ICLR 2024
No Retrieval
RALM (Top-1 Retrieval)
RALM (Random Retrieval)
60
62.7
65.6
59.6
53.1
$2.7
47.4
40.5
29.6
26.3
25.6
|2.
13,8
Bamboogle (Explicit Multi-hpp
2WikIMQA (Explicit Multi-hop)
Stratagy (Implicit Multi-hop)
Fermi (Implicit Multi-hpp)
(Single-hop)
Figure 2: Accuracy for Llama-2-1at few-shot prompted on five QA tasks, in three settings: (a) without retrieval, (b) with top-1 retrieval from strong search engine, and (c) with a randomly- retrieved passage. Retrieval augmentation can boost performance, but even strong retrieval hurts performance on StrategyQA and Fermi, and random contexts reduce performance dramatically.
2 MAKING RALMS ROBUST TO IRRELEVANT CONTEXTS
We now present our methods for building RALMs that are robust to irrelevant contexts. We begin by describing the common approach for incorporating evidence into RALMs. Next, we explore a natural baseline for using an NLI model to identify irrelevant contexts. Last, we describe our procedure for finetuning models to be robust to irrelevant context.
In-context RALMs Language models define probability distribution over sequences of to- kens, with auto-regressive models assigning probability via next-token prediction: PLM (x; where Zzi is the sequence of tokens preceding at each step and denotes the parameters of the LM. For RALMs, we follow the definition of in-context RALMs from Ram al. (2023), where context sentences are retrieved from corpus C, and generation is con- ditioned on the retrieved context. Given the retrieval operation Rc- this can be formalized as PRALM I1 (P ( I<s). where [Rc( #<c] denotes the concatenation of the trieved evidence with the generated sequence. Generation in LMs and RALMs can also be condi- tioned additional input, which omit for brevity.
In our setting, we focus on RALMs for ODQA. We follow recent approaches such as Self-Ask and IR-CoT (Press al. 2023 Trivedi al. 2023 Yoran al. 2023). for interleaving retrieval with multi-hop question answering (see Fig. 3). Retrieval is performed for every intermediate question and each context is prepended the question. In the single-hop setting, the model has generate the answer given question and retrieved context. In the multi-hop setting, the model has to generate intermediate questions and answers until arriving at the final answer and the retriever is called for the original question and after each intermediate question. Formally, x in this case is the generated decomposition until an intermediate step and (c are the retrieved contexts for all questions in r.
2.1 IDENTIFYING IRRELEVANT CONTEXTS WITH NLI MODELS.
NLI models DDagan et al. 2006 Bowman et al. 2015 classify whether textual hypothesis is entailed, neutral, or contradicted given textual premise. Recent work successfully used NLI models to automatically identify hallucinations (Honovich et al., 2022) and statement attribution (Bohnet) ctt al. 2023) when presented with a context and generated text. Similarly, a natural baseline is to NLI problem, by using the retrieved context only when
frame irrelevant context identification the hypothesis (i.e., final answer and intermediate question-answer pairs; Fig. 3) are classified as entailed by the premise (i.e., the retrieved context). We simple back-off strategy where generate twice, once with Pr. and once with PRALMS and only use the RALM if the NLI model classified all generated answers (and intermediate questions) entailed by the retrieved evidence.
3

---

Published as a conference paper at ICLR 2024
true that Colonel Walter Phelps served the United States Army for than 30 years?
Decomposition stop #1
Decomposition step #2
Final answer
21: Water Pheko an the Army?
Pheips?
Phelps
United
No.
Retrieval step
Retrieval step
EL Watee (Dot 29, 1852- February 20 Union broughoi Civil officr Eastern Brigade.
E2 Waaer Pholps: ('Wlter Phejps PAllagiance, 'Jiiled Armyl Union]] [Seniccbrmeeh: United Army
1886111957) [Rank;,
Decomposition ans. #1
Decomposition ans. 42
Prejpe Fibbaaeeb
A1: Colonal Water 22 How long did Lolonel Water Phe serve the Unteed States Ammy?
21:Who Calonel Watter Photos?
A1: Colonel Wulter Phelps was an officer the Union Armyy Biroughout the American Civil War.
A2: Colonal Walter Phelps served the United States Army for years.
Figure 3: Interleaving decomposition and retrieval in Self-Ask format (Press et al. (023). The model generates intermediate questions and answers until generating the final answer (model generations are shown pink). Retrieved evidence for intermediate questions is prepended at each step.
For example, in Fig. [ the retrieved evidence "Jason Gerhardt... is an American actor.. known for playing Cooper Barrett..." " serves as the premise while the question and generated answer, "Q: Who is the actor playing Jason on general hospital? A: Steve Burton" are concatenated and serve as our hypothesis. As this context is irrelevant, we expect the NLI model to label the hypothesis as contradicting. Given contradicting or neutral hypothesis, we will use the standard LLM without the (potentially distracting) retrieved context. For multi-hop questions (as Fig. 3 we additionally verify that each ineermdiatee-answe pair is entailed by the retrieved evidence using all retrieved evidence as our premise and the intermediate question-answed pair as the hypothesis. For example, "Q: Who is Colonel Walter Phelps? A: Colonel Walter Phelps was an officer in the Union Army throughout the American Civil War." for the first intermediate question in Fig.
2.2 TRAINING ROBUST RALMS
As in-context RALMs are not trained to use retrieved passages, more effective solution than post- hoe filtering (using NLI) may be train RALMs to ignore irrelevant contexts. We are interested in testing whether training relatively small dataset (several hundreds of examples) would suffice.
Automatically Generating Training Data Our goal is to teach RALMs to be robust to irrelevant context in an ODQA setting. In the single-hop setting, generating training data is straightforward. Given access to dataset of question- answer pairs {(g. a)} (i.e.. without contexts) and retriever Re, we use the retriever to augment questions with retrieved context. To create training examples with relevant contexts, we return the top-1 context from Re, and for irrelevant contexts, either return a low-ranked result from RC(g) or a random context (i.e., Re(q') for another question g') We denote the chosen context by Tq. Then, the training dataset is defined by [(Ir q), a.).
Our main challenge is generating training examples for multi-hop questions. In these questions the model generates a decomposition, consisting of intermediate questions and answers, before arriving at the final answer, while the retriever is called multiple times (Fig. 33. Our goal is to automatically generate arrieval-augmentee decomposition steps, D = {[[r.: x)) })}. where: y is the correct gen- cration for each step (i.c., the correct intermediate question. intermediate answer, or final answer); a consists of the previously generated steps up to y; rz is the retrieved contexts for all steps in x. Our first step to automatically generate decompositions is to prompt strong LLM without access to retrieval and to verify its answers. However, the LLM may arrive at the correct answer using an in- correct decomposition, for example in binary or comparison questions. Hence, we need to ensure the quality of generated decompositions. For multi-hop hopp datasets which provide intermediate answers, simply filter out generated decompositions that do not contain them. When intermediate answer annotations are unavailable, we sample from the LLM that generated the decomposition multiple times and verify self-consistency (Wang et al. 2023). Further details given in $3..33
4

---

Published as a conference paper at ICLR 2024
Dataset
Type
Example
NQ
Single-hop
What episode of law and order is mike tyson in?
2WIKIMQA
Explicit
Where was the place of death of Isabella Of Bourbon's 's father?
BAMBOOGLE
Explicit
What is the maximum airspeed (in km/h) of the third fastest bird?
STRATEGYQA
Implicit
Can Amold Schwarzenegger deadlift an adult Black rhinoceros?
FERMI
Implicit
How many high fives has Lebron James given/received?
Table 1: The QA datasets in our experiments.
Training We use our automatically generated data D to fine-tune models for generating condi- tioned on [raizl with standard maximum likelihood. Since we are mostly interested in the low-data regime, we limit the number of questions in D to 1,000 in the single-hop setting and 500 in the multi-hop setting (splitring multi-hop questions to multiple examples for each step), and use param- eter efficient fine-tuning (Dettmers al. 2023). Thus, training all models takes more than few hours. Additional experimental details is are in $ and $
EXPERIMENTAL SETTING
3.1 DATASETS
We experiment with both single- and multi-hop QA datasets. We list and give an example from each dataset Tab. T Our QA benchmarks can categorized based on their required reasoning skills:
Information-seekinn questions that do not require decomposition. We use the popular Natural Questions (NQ) dataset Kwiatkowski et al. 2019).
Sin:
Explicit Reasoning: Multi-hop questions where reasoning is explicitly expressed in the question. We include 2WIKIMQA (Welbl et al., 2018) and BAMBOOGLE (Press et al. (023).
Implicit Reasoning: Mutli-hop questions where generating reasoning steps requires common- sense (implicit reasoning, Geva et al. (202I)). Such questions may have multiple valid reasoning chains. We evaluate on STRATEGYQA (Geva et al. 202T) and FERMI (Kalyan et al. 202I).
For evaluation, we follow prior work and use EM for NQ and STRATEGYQA: and F1 for 2 2WKKMQQ and BAMBOOGLE. For FERMI, we use the official order-of -faanntd-m evaluation ( Kalyan et al. (22IT) Following prior work (Khattab al. 2022; TTrivei et al., [2033] Yoran et al.
2023). we evaluate on 500 random examples from the development set of each dataset. We provide
additional technical details evaluation in A.2
3.2
MODELS
We next describe our retrievers ( $3.2.1 prompted baselines ($3.2.2), and finetuned models (33.2.3
3.2. RETRIEVERS
Our models use retriever based on GOOGLE SEARCH as well as the open-source COLBERTV2 (Khattab & Zaharia 2020). Since the corpus for our datasets is Wikipedia, we format search queries org q." when accessing GOOGLE SEARCH. For COLBERTV2 V2 our corpus is the 2018 Wikipedia from Karpukhin et al. (2020). To simulate different types of noise, we return either the 1, low-ranked relevant evidence or a random passage that the top-1 evidence for different question intermediate question from the same dataset.
as "en. wikipedi
$
We query Google search via the SerpAPI service: For GOOGLE SEARCH, the lowest returned result from the API, which is at rank 9.3 average.
comm
For BOLBERTV only experiment with top-1 results.
5

---

Published as a conference paper at ICLR 2024
3.2.2 FEW-SHOT PROMPTED BASELINES
Our main baselines are (ama-22-3B models prompted for QA in the Self-Ask format through in- context learning (Brown et al. 2020) with 4-6 exemplars. We also evaluate with Llama-2- on NQ. Our baselines differ based on the retrieved contexts in the exemplars (Full prompts $AA.5):
Self-Ask No Retrieval (SA-NR): Exemplars are gold decompositions without retrieved evidence. We use this prompt to evaluate the performance of models without retrieval, when relying solely on their parametric memory, i.e, the information encoded in the model's parameters. As an additional baseline, we use this non-retrieval prompt, but still apply retrieval during inference.
Self-Ask Retrieval@ @ 1 (SA-R@ 1): Exemplars are gold decomopsitions prepended with the most relevant evidence retrieved from GOOGLE SEARCH for each step.
Self-Ask Retrieval@1 10 (SA-R @ 10): Exemplars are gold decomopsitions prepended with the lowest rank passage from Google (which is rank 10 in most cases).
Self-Ask Random Retrieval (SA-RMix) Exemplars are gold decomopsitions prepended with either the top-1 or lowest-ranked evidence from GOOGLE SEARCH, interchangeably.
NLI-based Models We use BART-Large model (L (Lwwis et al. 2020a) with 407 million param- eters trained on the MNLI dataset (Williams et al. 2018) We consider a question-answer pair as entailed if the probability for the entailment label is 2 0.5. All few-shot prompted baselines have variant with NLI, termed, SA-*-NLI When there no entailment, we use the generation from the SA-NR model, which uses only the parametric memory the back-off strategy.
3.2.3 FINE-TUNED MODELS
We finetune Llama-2- 13B ODQA henchmarks, one single-hop (NQ, 1000 training examples), one explicit (2W III,,MQA, 500 questions, 1,539 examples), and one implicit STRATEGYQA, 414 questions, 1,584 examples). Training hyperparameters are in $A.I
LLM verify questions answerable and generate decompo- sitions F This is done with GPT-3, code-davinci-002 (Brown et al. 2020; Chen et al. 2021) with 175B parameters. We prompt the model to generate decompositions using the SA-NR prompt. 2WIKIMQA contains intermediate answers, and we use those to verify generated decompositions. For the implicit STRATEGYQA we utilize only the final answer, and thus use self-consistency, as explained in 2 We sample 5 decompositions per question (one with greedy decoding and four with temperature 0.7) and only keep the greedily-decoded decomposition when all decompositions lead to the same correct answer. To verify the quality of the generated decompositions, we manu- ally examine 50 decompositions per dataset and find that the generated decompositions are correct in about 90% of the time for STRATEGY YAQ and more than 95% for 2WIKIMQA. As FERMI and BAMBOOGLE contain less than 300 examples, we use them exclusively for evaluation and do not include them in these experiments.
Data Generation We
Incorporating Retrieved Evidence in Training Examples To make sure the model is exposed to relevant and irrelevant context, we use either the top-1, low-ranked, or random evidence with equal probability at each step. We term the trained model SA-RetRobust. We include ablations where training is without retrieved context (SA-NoRet) or only with the top-1 evidence (SA-Ret@1) 1).
4 RESULTS
Fig. presents our main results, evaluating the effect that retrieving top-1 result from GOOGLE SEARCH has on the following RALMs: (a) an In-Context RALM, prompted with the SA-RMix prompt (leftmost yellow), (b) the same model, but using NLI models to identify irrelevant context (center, green), and (c) our proposed SA-RetRobust, a RALM fine-tuned on a mixture of relevant
We use the model from httpt:tpp.... //hugg////ug nng face co/ aaceboooobbatb-aoaaacaaaaa
'To not train our models to hallucinate, we also filter single-hop questions where code-davinei-O0 fails to generate the correct answer. However, do not fully guarantee that the gold answer appears in the retrieved context or encoded in parameters of the model being trained.

---

Published as a conference paper at ICLR 2024
In- Context RALM In-Context RALM NLI Trained RALM (RetRobust)
40
Datasots with training data
Datasets without training data
+11.0
+7.4
+16.1
#21.1
+7.0
+34.9
+0.6 14.55
+15.3
+7.8
-2.1
-10
NQ (29.8)
2WikIMQA (32.0)
StrategyQA (65.6)
Bamboogle
Fermi (27.7)
Figure 4: Results for our models on all evaluation datasets when retrieving top-1 results from GOOGLE SEARCH. Bars show the difference performance from model with no retrieval (whose performance is given in parenthesis for each dataset). Prompting models to use retrieval in-context (leftmost bar) increases performance on single-hop and explicit datasets, but decreases performance implicit ones (STRATEGYQA and FERMI). When using NLI models to identify irrelevant evidence (center bar), retrieval never hurts, cost gains received when retrieval is helpful. Our trained RALMs (rightmost column) outperform all other models when applicable for NQ, 2WIKIMQA, and STRATEGYON YQQ (see $32.3 for more details on data generation).
and irrelevant contexts (rightmost, orange). The bars show the difference in performance from our few-shot prompted model without retrieval (whose performance shown parenthesis for each dataset). For the In-Context RALM, we observe that retrieval helps on NQ, 2WIKIMQA and BAMBOOGLE but reduces performance on the implicit STRATEGYQA and FERMI. Adding NLI to identify irrelevant context ensures that retrieval does not hurt, but gains are limited. Training with retrieval leads to gains across the board. observe similar trends with the COLBERTV2 retriever, albeit an overall decrease in accuracy ($A.3 Tab.
Exploring the Robustness of Models to Irrelevant Context Fig. present results when simu- lating retrieval of irrelevant/noisy context, either by retrieving low-ranked passages (top) or random ones (bottom). When retrieving random passages, the performance of the In-Context RALM drops by more than 10 points on average, a phenomenon that can be mitigated by using NLI models. SA- RetRobust performs best across all settings. To verify that these improvements indeed stem from robustness to irrelevant context rather than task-specific training, we compare SA-RetRobuss to an ablated variant trained and evaluated without retrieval (full results in Tab. 4 (A.3). SA-RetRobust is able to perform similarly to this model (within one standard deviation) when retrieving random contexts. Interestingly, when retrieving low-ranked results, ob-RetRobust outperforms the ablated model by 3.8 and 2.8 points on NQ and 2WIKIMQA, while performing only slightly worse (within a 1.2 point difference) on STRATEGY AA. Overall, our results suggest SA-RetRobust learned to both better utilize retrieval and ignore irrelevant context.
Adding Retrieval to In-context Exemplars Hurt Performance Tab, 2 and Tab, in AA.3 present full results with the GOOGLE SEARCH and COLBERTV2 retrievers. Interestingly, pro- viding exemplars with retrieval performs worse than providing exemplars without retrieval, i.e., the SA-NR prompt leads to better performance even when retrieval is performed at inference time. This SA-NR prompt consistently outperforms the prompts with retrieval (SA-R @ 1, SA-R @ 10, and SA-RMix) when retrieving the top-1 result from COLBERTV2 or random contexts from GOOGLE SEARCH. addition, the SA-R@ model that contains top-1 results in the prompt is not the best performing even when retrieving top-1 results at inference time. losing to SA-NR by more than 2 points on average across datasets. When retrieving noisy contexts at inference time, SA-R @ outperformed by the other models, suggesting that showing examples for retrieval during in-context learning has negative effect that causes over-utilization of irrelevant context. We observe a similar trend with Llama-2-70B in SA.3 Tab.
Effect of NLI When retrieving random contexts evaluating on the implicit STRATEGY QA and FERMI, NLI variants consistently perform best, suggesting small NLI models are sufficient to iden- tify irrelevant evidence (Tab. and Tab. 3 in A.3)). However, they reduce performance in cases

---

Published as a conference paper at ICLR 2024
In-Context RALM
In-Context RALM NLJ Trained RALM (RelRobust)
training data
without training data
5
+1.4
+0.2
+6.3
+0.2
+13.0
+0.2
+3.0
+0.3
0.0
+9.6
0.0
[4:3
0.0
0.0
-0.2
-14.5
22.11
NQ (29.6)
2WikIMAA (32.0)
Strategy (65.6)
Bambooglc (47.4)
Fermi (27.7)
Figure 5: Results with low-rank (top) and random retrieval (bottom). Models are similar to those in 4 Performance significantly decreases for the prompted model in all settings, while it is maintained when using NLI models. Our finetuned SA-RetRobust is best performing in all settings. We show that SA- SA-RetRobust learned to both ignore irrelevant context and better utilize relevant context by comparing to an ablated model without retrieval $4
retrieval is helpful, e.g., on the explicit 2WIKIMQA and BAMBOOGLE. We perform a detailed analysis for our NLI variants in
Results with Finetuned Models Fig. 4 and Fig. show SA-ReiRobust consistently outperforms other models. In AA.3 Tab. 4 present all results for all trained models, showing SA-RetRobust outperforms our ablated baselines. Specifically, it outperforms SA-NoRet (fine-tuned without re- trieval) by 2.7, 2.4, and 2.4 points on average when using the top-1, low-ranked, random context from GOOGLE SEARCH during inference, and SA-@1 1 by 0.2, 0.4, 3.2 points respectively. When retrieving top-1 results from COLBERTV2, -e-ReRobustt outperforms SA-NoRet and SA- @1 by 2.7 and 0.3 points on average, respectively. Our results suggest that training on mixture of relevant and irrelevant contexts necessary for robustness and improved performance. We provide a study on the generalization of our trained models to other settings in A.3
Results with Llama-2-70B We compare SA -A-RetRobust with -lama-2-708 on the NQ dataset to assess whether larger models more robust irrelevant contexts. Without retrieval, the prompted Llama-2-700 outperforms the trained Llama-2-13B by 4.3 points (38.4 vs 34.1). However, when re- trieving the top-1 results from GOOGLE SEARCH, SA-RetRobust outperforms all prompted Llama- 2-70B variants by at least 3.3 points (45.7 vs 42,4), suggesting that increasing model size alone is sufficient make models better utilize retrieval. We provide the full results in {A.3 Tab,
ANALYSIS
When Does Irrelevant Context Cause Errors? To assess emrors caused by irrelevant context, we manually looked at examples from NQ, 2WIKIMQA and STRATEGYQA, where models succeed without retrieval, but fail with it. Specifically, we look examples where the model is prompted with the SA-RMix prompt that includes both top-1 and low-ranked retrieved result and is presented with low-rank or random retrieved evidence during inference. We manually annotated 40 examples in each setting (240 overall), and find that automatic errors indeed correlate with cases in which retrieval augmentation caused the model to err 73% of the cases (65%-854 each setting). We provide additional details and statistical tests in $A.4
We then take a deeper look at the errors. For NQ we find that when using low-ranked context, the wrong generated answer entity appears in the retrieved context in the majority (77%) of the cases, but only in 37% when retrieving random contexts. This suggests that irrelevant context can cause errors even when the generated entities not retrieved, as shown in $A.4 Fig. 6 For multi-hop questions, we test whether irrelevant context leads to errors in question decomposition, or in answer- ing intermediate questions. We find that when retrieving low-ranked passages, most of the emoos
8

---

Published as a conference paper at ICLR 2024
(68%) for the explicit 2 2WIKIMQA are in intermediate answers, contrary to the implicit STRAT- EGYQA were errors are more prevalent in intermediate questions (77% of the cases, we provide an example in $A.4 Fig. 7. Similarly, when retrieving random contexts, most enrors (60%) for 2WIKIMQA are in intermediate questions. This suggests that irrelevant context can cause errors in generating both answering strategy and the answer itself, depending on the task and the retrieved context.
When Do NLI Models Fail? As shown in $4 NLI models are efficient at identifying relevant context, at cost to gains when retrieval is helpful. To better characterize NLI models, we look at the accuracy for our NAII models as a function of the probability that the NLI model assigns to the entailment label. Tab. in $ shows that there are many cases where the probability for entailment is low but retrieval helps for NQ and 2 2WKIIMQA.
To better identify the source for such errors, we manually analysed 25 examples for each dataset where entailment was low, but retrieval augmentation led the SA-RMi: model to generate the correct answer In about half of the cases the NLI model erred and the generated text is indeed entailed from the retrieved contexts. In the remaining examples, for at least a third of the cases the generated answer or decomposition is correct, but the retrieved context does not directly entail the generation.
This can be partially explained by the ability of models to combine retrieval and their parametric knowledge (Talmor et al., 2020; Zhong al. 2023) Cohen et al., 2023). We hopeful that these results can inspire future work to focus on additional aspects of retrieval augmentation, such as the effect augmentation has on generation probability rather than on direct entailment.
RELATED WORK
Recent work has shown that the performance of LLMs can be affected by irrelevant context. Amongst others, Jia & Liang (2017); Petroni et al. (2020); Creswell et al. al.] (2023) show that adding random or irrelevant context can decrease QA performance. This has been shown in many settings,
including but not limited to factual reasoning (Kassner& & Schütze 2020 Pandia & Ettinger] [021]
Misra et al.] [203)) text generation about new entities (Onoe et al. 2022). ). or even code generation (Jones & Steinhardt 2022). In the context of arithmetric reasoning, [Sh et al. (2023) showed that adding irrelevant context to exemplars task specific instructions can help, suggesting the model may be equipped with such skills from pre-training. Other methods try to reduce the number of retrieval calls, by focusing on cases where confidence is low (Jiang et al., 2023) or retrieving in- formation for rare entities (Mallen et al. 2023). Closest to our work is that of Li et al. (2023) that propose LLMs with a "controllable memory" that will enable them to ignore irrelevant context. However, their LLMs are finetuned on over 200K training examples, where our focus is on perfor- mance when training with 1,000 questions or less, and training data is automatically generated. In addition, we focus on multi-hop QA setting, where the retriever is called multiple times ($2).
A similar line of work focuses on when models should use parametric or retrieved knowledge, especially when there are conflicts (Longpre et al.) 2021 Chen et al. 2022). It has been recently proposed to train models to generate from both parametric and retrieved knowledge (Neeman al., 2023) or make better use of in-context exemplars (Zhou et al. 2023
7 CONCLUSION
In this work, we provide a thorough analysis showing current RALMs are not robust to irrelevant retrieved context, causing them to perform worse certain tasks. In cases where training is not possible, a simple NLI baseline is efficient to increase robustness, at a cost of discarding relevant passages. When training is possible, we introduce an automatic data generation framework for single and challenging multi-hop tasks, and show training on as few as 1,000 examples with intentionally varied quality suffice make models robust irrelevant context and improve overall performance. While our focus in this work is on in-domain settings, we are hopeful our work could inspire future research towards general RALM that is robust to irrelevant context.
'There are only 25 such examples for the NQ dataset.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to our colleagues at TAU NLP for their insightful comments. We thank SerpAPI for their support by granting us an academic discount. This research was partially supported by the Yandex Initiative for Machine Learning and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800). This work was completed in partial fulfillment of the Ph.D. of Ori Yoran.
REFERENCES
Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering:
Evaluation and modeling for attributed large language models, 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10.18653/v
URL
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Horbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Touan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https:::/ppoceeeetteeeo neurips........pp. cC /pcpppppe//222/aahh. 1457c0d6bfcb40 (111118abaaa12242a222 AbstteffatA-AAtttttttca 0SST005SCAATAABBBBBaa22aaa------ html
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 1/ URL https:
Lung-Ting Chen, Michael Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2292-2307, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguisties. URL
https: /tttps://aLaanhhology..o wwtp://...//222mmm......2. wwweeeennaaaaeeeeee maiin.14..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh,
10

---

Published as a conference paper at ICLR 2024
Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models, 2023.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL htlLhttps://// http:s/////perevee.... http://wwwwee/fouummii..
3Pf3Wg60-AS A4
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Joaquin Saationero-Candelee Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classi- fication, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelherg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33428-6
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. httptttts//
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022. doi: 10. 10.1 (t02/aacla.000553
URL /ttps:://aaaathhhh www.annhhoogy..ooo logy. org/
wwtp://rg/2222222 2022 tac.-
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguisties, 9:346-361, 2021. doi: 10. 162/tacl a 00370.
htt.:https://ee http:///caaaathho///.o wrg.//orr./oooo.o.........oo.... 2021 10l22..accll. wwt.taalll... -1.cc111.111
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wo Wei Chang. Realm: Retrieval- augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. JMLR.org 2020.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document- grounded Dialogue and Conversational Question Answering, pp. 161-175, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. (8653//1/20222 www.a////.......ooe/// 1.19. URL https: T/aclanthology.oom org/
1.111
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251): 1-43, 2023.
/v.tss//244/2 v24/23 0037 html
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 2021-2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.16653/1/D11711255 URL htRl:httts://eee antho wwttthhoooy....o ogy. ory.orrg/D1111111
11

---

Published as a conference paper at ICLR 2024
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp- 7969-7992 Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10. /t0.://///22..mmlppmmmoo 7/ www.anthlooy..org//ooooooooo wwg./ot..cccc..oooooooooo emnip main. 495
URL https:
Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL htwp:: httpss////ppeeeveee.eeo
Eorum ferumiid-fooe 1ep2iddfoo0s fo09Cg X R
Ashwin Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. How much coffee was consumed during EMNLP 2019? fermi problems: A new reasoning chal- lenge for AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7318-7328, Online and Punta Cana, Dominican Republic, 2021. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2021.
URL hetps:
//aclanthology orq/2021, emn 1p caannp-mmanne 582
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, 2020. Association for Computational Linguis- tics. doi: 10.186533/1/2022.ccm emm 1 p- p- in. 550
UtL http:p/s///aaaaatlleaaaeo org/2020.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7811-7818, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/w1/2020.aall haap://000.0.cmmm..00000000 main.698. URL 2020.ac1-8e main. 698
org/
O. Khattab, Keshav Santhanam, Xiang Lisa Li, David Leo Wright Hall, Percy Liang, Christopher Potts, and Matei A. Zaharia. Demonstrate-Se were--seerrc---eeee Composing retrieval and language models for knowledge intensive nlp. ArXiv preprint, abs/2212. 140224 2022. httpss/:: arxiv.org/abs 221.. 14024
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy Huang. Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25 30, 2020, pp. 39 48. ACM, 2020. doi: 10.1145/3397271 7/doi.org 10.
UtLhttps:
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- King-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10. a 00276. URL htttps://act wwwpe//awww..oo..ooooooooooooe/e
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-scquence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics. pp. 7871-7880, Online, 2020a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020..o www.22000..--mai..70.000 703.. URL https://ee antho weathology.co org 2020 acl ma inaiin.. 703
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NL.P tasks. In
12

---

Published as a conference paper at ICLR 2024
Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Confer- ence Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 2020, virtual, 2020b. URL httppss//proccecingg.eee...
664
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACI. 2023, pp. 1774-1793, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 112116655/12223.nnnlll
URL
wttp:/y...gg/./...//.// www.02...inndi.. findings acl. 112
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7052-7063, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 1..18653/112021.mmmppp www.leemmmi.nnn555.
main. /RLLlttsss//aaaaatollooo////
emm 1aan1p-mac 1---maaim www.a.ni.....m. 565
Alex Mallen, Akari Asai, Victor Zhong. Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July URL.
2023. Association for Computational Linguistics. doi: 1606://///0223..alooooggg https::/aclaatthie .tw.ooooooooooooooo..o..oooooo.o wtt.o///222...1oo....oo 546
Kanishka Misra, Julia Rayz, and Allyson Ettinger. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2928,2949, Dubrovnik, Croatia, 2023. Association for Computational Linguis- 2aw.22223.cclll11. main. 213
tics. URL
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. DisentQA: Disentangling parametric and contextual knowledge with counterfactual question an- swering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10056-10070, Toronto, Canada, July 2023, Asso- ciation for Computational Linguisties. doi: 10. 10.18663////20222aclloloo
URL https:
559
Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. Findings of the Association for Computational Linguistics: NAACL 2022, pp. 693-702, Seattle, United States, July 2022. Association for Computational Lin- guistics. doi: 10. 10.118533/11/0022
findingsiiiiinnnnnn 55. URL https: https::/aclaattoll..oo .rrg//..oor/oooo
2022. findings naacl.5.
Lalchand Pandia and Allyson Ettinger. Sorting through the noise: Testing robustness of information processing pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1583-1596, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 119. htths::: ac www.ooonnnooooooooooooooooooooot org 1ttf221mmnlpmmmamm in..n..11
105/33/1/2021eemm/ main.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. Au- tomated Knowledge Base Construction, 2020. URL pattts://oppnzeete
net/forum
[d=
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp.
Online,
13

---

Published as a conference paper at ICLR 2024
June 2021. Association for Computational Linguistics. doi: http:633/w//2222..aammmm.mmm URL
wttp:orrg/2220 naaci-main. 200
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mea- suring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/11/023. 10n16655/1122222.2nnnnneeemmpp findings-emnlp3375. URL htttss///accla..o
logy. orgg//.rr./0.....
Findings- empip..-mmnp.... 378
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval- language models. Transactions of the Association for Computational Linguistics, 11:1316 1331111311,,,, 2023. doi: 10.1 10.1162/tac...00000) URL https:
tacl -1.75
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into
the parameters of a language model? In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, 2020. Associa- tion for Computational Linguistics. doi: 10. 1t..166//1/2222.eelm-mmmaaaa //aclanthology: wrg.yy/0.gg22000ooooo e:12200mmmlpmmaa.. ma..p-main.437
URL https:
Ohad Rubin and Jonathan Berant. Long-range language modeling with self-retrieval, 2023,
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou, Large language models can be easily distracted by irrelevant context. In Pro- ceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap-of- thought: Teaching pre-trained models to systematically reason over implicit knowledge. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con-
ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httplitttss//pprcedinggg..euts hwtp://wwpppp.pppp....eepamp http://w....rrssoooooppee//////a htc/papper/222//assss.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls,, and Arpit Mittal. FEVER:
large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), pp. 809-819, New Orleans, Louisiana, URL June 2018 Association for Computational Linguistics. doi: 10.18653/v https://aclann
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajwwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut I Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Harsh Trivedi, Niranjan Balasubramaniann, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with chain-of-thougl reasoning for knowledge-intensive multi-step questions. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014-10033 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18.55853/v 1a0653//1122...llloooo.33353335
acl-long.555. URL https: //tp/aaaaaannoooooooaaoaaa .xg/
2023. acl aal.aclllloo.... 557.
14

---

Published as a conference paper at ICLR 2024
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
htttps://spenrevieee Po
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Linguis- lics, 6:287-302, 2018.
doi: 10. 10.11622taela..0021. URL https://aclante: ogy. org/
018-1021
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122, New Orleans, Louisiana, 2018. Association for Com- putational Linguistics. doi: 1111886/1//N1--111 1101. URL
N18 1101
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198, 2020. doi: 10.1162/tacl a 00309, URL
/ httpps///ccaattooooooorr////////
Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. An- swering questions by meta-reasoning
multiple chains of thought. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5942-5966, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/11/023.cooomm m6in.364. URL https:
wrg.ooooooooo../2222/2 //aclanthology. wwg.or//////// 22033. emnlp main. 364
Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pp. 16686-15702. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1166:///1/223..mmmmmmmaaaa
URL
https: ptttp:://aclanthooogg wwwpsyy..rrg//e/. otg/2023..oo.oooooo.2oooo/... emmlp main. 971
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the As-
sociation for Computational Linguistics: EMNLP 2023, pp. 14544-14555 14556. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/20 1/2.3///1222../// http://///222...nnnnme-mmmmm fiiniing--mmlll.eee URL htttts://acclaaei ceattoology.com org. 2023. findi ngs emn 1 968
A APPENDIX
A.1 MODELS
Llama-2 all cases, we use the vanilla variant of the Llama-2 models from https:// hugg i wingnfacc face. wwt.c/////aa....e htmmmmtaaaa 1lama lamm with half precision.
Decomposition Generation Questions in our multi-hop datasets require between 2-4 decomposi- tion steps. Hence we limit the number of generation steps to 5. In Tab. X we show that the number of cases which the model does not arrive at an answer in 5 steps, termed as failures, very small when generating with top-1 results from GOOGLE SEARCHI, at 0.4% for 2WIKIMQA and 1.2% for STRATEGYQA. Failures are much higher when retrieving random contexts, at 37.0% for 2 2WIKIMQA and 34.4% for STRATEGYQA. These are usually cases the model enters infinite loop. Following recent work, (Wang et al. 2023 Yoran et al. 2023) we use greedy decoding when generating decompositions.
15

---

Published as a conference paper at ICLR 2024
Training We fine-tune all models with QLoRA (Dettmers et al. 2023) for parame- ter efficient fine-tuning. We use the default hyperparameters from ht https: /ttpp://ggian /ithhb.. hatm://aiiel-fummnn/aaaaaa httpa///ssssaaaeeooe/a.saaaaaass
gpu/ aouu/_lama-2
guanaco/ sss- aaaa 2- 13b gu ptt.ooooppf.....llll.. We train all our models for epochs, with learning rate of 2e 4 and linear scheduling on single GPU. The training time for each model was no longer than 3.5 hours.
A.2 EVALUATION
In some cases, the models do not arrive final answer ($A.I). In such cases, assign score of 0.5 for TTRATEGYQA and 0 for all other datasets. For FERMI, following past work (Yoran et al.) 2023 all 286 "Real Fermi Problems" for evaluation and provide the gold answers measure units (meters, cubes, litres, etc...) as additional input to our models.
A.3 FULL RESULTS
Tab. 2 and Tab. 3 presents the full results for our prompted models with GOOGLE SEARCH and COLBERTV, respectively. Tab. 4 presents full results for all our trained models, averaged over three seeds. Tab. presents results for Llama-2-70B on NQ with the GOOGLE SEARCH retriever.
Out of Distribution Generalization To test the generalization of our trained models in an out of distribution (OOD) setting, we trained version of our models on mixture of our STRATEGYQA and 2WIKIMQA data and evaluate on BAMBOOGLE and FERMI. Since the evaluation task can differ from the training data (for example in FERMI the model needs to generate an equation before the final answer), we provided the models with one exemplar during inference. We provide the full results for this experiment in Tab. We note that the standard deviation in these experiments larger than in Table 3, probably due to the small support size at 120 for BAMBOOGLE and 286 for FERMI. Still, when comparing between the trained models, SA-RetRobust is either the best performing model or within one standard deviation in all settings. However, we also observe some surprising trends that may be related to failure of the model to generalize to the effect of the in-context exemplar: (a) For BAMBOOGLE, when not using retriever, the model prompted and evaluated without retrieval outperforms the model trained without retrieval (47.4 vs 40.8), and (b) For FERMI, we see slight decrease in accuracy from the model trained and evaluated without retrieval to our trained SA-RetRobust model when evaluating with low-ranked or random retrieval (29.3 vs 27.9 and 27.6 respectively). Overall, we are hopeful that these results will help future research towards general RALM that is robust to irrelevant context.
A.4 ANALYSIS
For our study regarding cases irrelevant context caused SA-RMix to err, we annotate examples with the following categories (a) Valid: the prediction a paraphrase of the correct answer or a plausible answer to an ambiguous question (b) Wrong: the prediction with retrieval is wrong and the prediction without retrieval is correct, (c) Both Wrong: the prediction with retrieval is wrong, but the prediction without retrieval was also wrong (due to bad decomposition that can spuriously lead to correct We verify our
answer in binary or comparison questions). We provide the full result Tab. 7 results are statistical significant by running binomial test for the hypothesis: "Most cases where automatic metrics decrease by the introduction of irrelevant context are not actual errors" which was rejected with 00.0..
Fig. 6 presents an example where irrelevant context causes -lama-2--33 to err when the generated entity does not appear in the retrieved context. Fig. 7 shows example where random retrieval caused the model to generate a bad strategy in STRATEGYQA and Tab, presents the full results for our analysis of NLI models.
A.5 PROMPTS
We provide our SA-NR. SA-R@1. I, and SA-R@ 10 prompts for NQ in Tab. Tab. spectively. For the SA-RMix prompt, we use exemplars form the SA-R @ and SA-R @ 10 prompts,
Tab. 10 re-
16

---

Published as a conference paper at ICLR 2024
Inference Retrieval
NR
NR
@@1
R@1
R@ 10
R@
RMix
RMix
Dataset
-NLI
-NLI
-NLI
-NLI
None
29.6
n/a
n/a
n/a
n/a
n/a
@
41.0
38.4
39.0
36.4
41.0
36.8
40.6
37.0
NQ
@ 10
30.2
29.8
25.6
29,4
30.0
31.0
31.0
29.8
Random
28.2
29.6
17.2
29.4
22.2
29.4
22.0
29.4
None
32.0
n/a
n/a
n/a
n/a
n/a
n/a
n/a
@ I
56.0
39.9
51.6
38.3
51.6
39.2
53.1
39.0
2WIKIMQA
@ 10
33.0
32.2
27.5
32.5
30.9
32.3
29.6
32.2
Random
27.0
32.0
13.7
32.0
32,2
17.5
32.0
None
65.6
n/a
n/a
n/a
n/a
62.1
65.6
63.8
66.7
61.4
65.8
59.6
STRATEGYQA
@ 10
60.4
65.6
61.0
65.6
60.5
65,4
62.1
65.8
Random
58.4
65.6
53.4
65.6
57.0
65.6
52.7
65.6
None
47.4
n/a
n/a
n/a
n/a
n/a
n/a
n/a
@ 1
68.0
55.9
61.2
56.0
68.9
58.0
62.7
55.2
BAMBOOGLE
@ 10
41.4
47.4
32.1
45.9
44.5
45.9
38.1
47.0
Random
39.5
47.4
24.7
47.4
34.8
47.4
26.3
None
27.7
n/a
n/a
n/a
n/a
@
27.4
28.2
25.2
27.6
27.5
27.7
25.6
27.4
FERMI
@ 10
24.0
27.7
27.1
27.6
25.1
27.7
23.6
28.0
Random
22.1
27.7
17.2
27.7
17.4
27.7
13.8
27.7
Table 2: Full results for our prompted Llama-2-2 2--3B models with the GOOGLE SEARCII retriever.
Dataset
Inference Retrieval
NR
NR -NLI
R@1
R@ -NLI
R@10
R@10
RMix
RMix
-NLI
-NII
None
29.6
n/a
n/a
n/a
n/a
NQ
@1
34.6
34.8
31,2
33.2
32.4
33.8
32.8
33.8
None
32.0
n/a
n/a 37.3
n/a 34.9
n/a
n/a
n/a 39.6
n/a
2WIKIMQA
@ 1
42,2
36.2
36.7
35.0
35.3
None
65.6
n/a
n/a
n/a
n/a
n/a
n/a
n/a 64.7
STRATEGYQA
@
61.6
66.0
64,3
65.1
61.1
64.9
61.6
None
47.4
n/a
n/a
n/a
n/a
n/a
n/a
n/a 48.7
BAMBOOGLE
@
50.0
48.6
37.4
46.6
38.1
47.4
38.2
None
27.7
n/a
n/a
n/a
n/a
n/a
n/a
n/a 28.0
FERMI
@
25.9
27.3
23.2
27.8
21.2
28.0
24.4
Table Full results for our prompted Llama-2-138B models with the COLBERTV2 retriever.
interchangeably. We add small instruction for the QA task before the exemplars. Our prompts con- tain 6 exemplars for NQ. 2WIKIMQA, and STRATEGYQA, 5 for FERMI, and for BAMBOOGLE, All our prompts are publicly available, together with our models, data, and code.
17

---

Published as a conference paper at ICLR 2024
Dataset
Retriever
Inference
SA-
SA-
SA- RetRobust
NoRet
Ret@1
None
None
34.12:.8
n/a
n/a
Google
@ I
42.8+0.8
4..3:10.6
45.7:10.6
NQ
Google
10
37.0+1.0
38.2:10.6
37.9+0..5
Google
@ Random
31.1+0.1
31.4++0.5
33.8++0.2
ColBERTV2
@
41.5++0.4
43.5++0.2
43.5++0.6
None
None
42.2:10.6
n/a
n/a
Google
@ 1
64.6+007
66.7+110
66.9++1.0
2WKIMQQ
Google
@ 10
40.8+0.5
43.9+t0.3
45.0++0.4
Google
Random
40.4+0.8
3..5+1.0
41.6++.22
ColBERTV2
@
54.4100.7
57.0.0.5
$7.6
None
None
69.8+0.9
n/a
n/a
Google
@ I
67.1=0.4
69.0+1.2
70.1+1.1
STRATEGYQA
Google
@ 10
66.6111.1
68.1
68.6.10.5
Google
@ Random
66.6.10.7
66.91112
1.
ColBERTV2
65.9
68.4111.4
6..
Table 4: Full results for our trained Llama-2- 2-138 models. Results are averaged over three seeds. For RALMs,
either GOOGLE SEARCH or COLBERTV2 as retrievers during inference.
Dataset
Retriever
Inference
SA-
SA- Ret@1
SA- RetRobust
NoRet
None
None
40.8+:2.0
n/a
Google
@ 1
57.4+2.0
61.3+1.4
6..7+t
BAMBOOGLE
Google
@ 10
33.11:1.9
39.2.12.0
42.0.12.2
Google
@ Random
29.8.11.8
38.4.14.8
43.6.11.6
ColBERTV2
37.1.1..5
48.2.10.7
4..
None
None
29.3.10.4
n/a
n/a
Google
I
31.3.11.2
29.6:10.8
1.6
FERMI
Google
@ 10
28.3+11.5
28.6
27.941
Google
@ Random
25.3+1..
27.9+24
27.6+00.6
ColBERTV2
@
28.340.11
28.910.4
30.0+
Table 5: Full results for our trained Llama-2-331 models in an out of distribution setting. In this
setting, our models are trained on a mixture of STRATEGYQA and 2WIKIMQA and evaluated on BAMBOOGLE and FERMI. Results are averaged over three seeds. For our RALMs, we use either GOOGLE SEARCH or COLBERTV2 as retrievers during inference.
Q: What does the md in md 20/20 stand for?
Large Language retrieval)
Augmented Language Model
The answer is: Mogen David
The first airiine meals were served by Handly London-Paris
ErAifiin meal:
Page Transport, an aidine company founded in 1919, to serve the
Passangers choose from selection sandwichee and fruil. The answer is: Maryland
Figure 6: An example from NQ where retrieval caused Llama-244 2-BBB to err, although the generated entity does not appear in the retrieved context.
18

---

Published as a conference paper at ICLR 2024
Inference
NR
NR
R@1
R@
R@ 10
@@ 10
RMix
RMix
SA- No-Ret
SA-
Retrieval
-NLI
-NLI
-NLI
-NLI
RetBust
#Params
70B
70B
70B
70B
70B
70B
70B
70B
13B
13B
None
38.4
n/a
n/a
n/a
n/a
n/a
n/a
n/a
34.1
n/a
@
41.4
41.8
41.2
42.4
41.6
42.4
41.2
42.0
42.8
45.7
38.8
34.2
33.4
35.4
31.8
35.2
37.0
37.9
Random
33.6
38.2
28.8
36.8
35.2
38.2
31.0
38.0
31.1
33.8
Table 6:
Results for NQ with GOOGLE SEARCH and Llama-2-777
Inference Retricval
Valid
Wrong
Both Wrong
@10 Random
34%
66%
0%
NQ
22%
78%
0%
@ 10 Random
2%
72%
23%
2WIKIMQA
0%
@ 10
3%
65%
32%
STRATEGYQA
Random
70%
30%
Table 7: Full results for our analysis regarding cases where augmenting retrieved contexts caused Llama-2-13B prompted with SA-RMix to err. Classes and additional details are provided in
Q: Would it be safe to have jackfruit thrown at your head?
Large Language Model (no retrieval)
Retrieval Augmented Language Model
Decomposition step #1
Decomposition step #1
Q1: What is the weight of jackfruit?
Q1: What is the weight of jackfruit?
A1: The weight of jackfrult is about 10 kg.
E1: The bald eagle (Hallacctus lourocophalus) bird
eagle, known subspecies and forms species pair with
Decomposition step
A1: The weight of jackfruit is about 10 kg.
Q1: What is the weight of human head?
A1: The weight of a human head is about
Decomposition step #2
Q2: What
of balld cagle?
Decomposition step x3
Romance Three
Guanzhong, 14thh-century historical
01: is 10 kg greater than kg? than
nover attributed towards the end of the Hen
turbullnnt years
Yes,
A2: The weight of bald eagle is about
Final answer 18: No.
Decomposition #3
greater kg?
portraying
appoar
before
Christmas in department stores or shopping malls, or at parties, The practios of this has been credited
A3: 10 is than
Final
So the final answer ls: No.
Figure 7: An example from STRATEGYQA QA irrelevant context causes Llama-2-13B to generate wrong strategy (right). Without retrieval (left), the model succeeds in generating the correct answer.
19

---

Published as a conference paper at ICLR 2024
Inference Retrieval
Failures %
Low-Entailment %
Med-Fnntallmet %
High-Entailment %
A
A
A
@ 0.0%
32.6%
+0.11
12.8%
+0.09
54.6%
+0.11
NQ
@ 10 0.0%
69.4%
+0.01
9.4%
+0.06
21.2%
+0.01
Random
0.0%
97.2%
-0.07
2.2%
-0.2
0.6%
0.0
@ 2.8%
0.4%
83.0%
+0.12
5.6%
+0.34
11.0%
+0.55
2W1KIMQA
10 @
93.8%
-0.02
2.6%
-0.11
0.8%
+0.08
Random
37.0%
63.0%
-0.6
0.0%
0.0
0.0%
0.0
@ 1 @ 10 Random 34.4%
1.2%
96.2%
-0.07
2.4%
+0.17
0.2%
0.0
STRATEGYQA
2.6%
95.8%
-0.0
1,4%
0.0
0.2%
0.0
56.6%
-0.13
0.0%
0.0
0.0%
0.0
Table Results for our NLI analysis. 'Failures' indicates that the decomposition model was not able to arrive the answer (see {A.1). Other examples are split based their entailment probability: low probability is < 3. medium probability is in [3 21 and high probability is > z indicates the improvement in accuracy when using retrieval. For NQ and 2WIKIMQA, many cases where
retrieval is helpful have low entailment probability. For the implicit STRATEGY QA most examples have low entailment, but retrieval helps in the few examples with medium entailment.
Given the following question, answer it by providing follow up questions and intermediate answers. intermediate questions are not necessary, answer the question directly.
Question: how did the big red one get its name Are follow up questions needed here: No. So the final answer is: its shoulder patch
Question: where are the cayman islands on the map Are follow up questions needed here: No.
So the final answer is: western Caribbean Sea
#
Question: who won war between north korea south korea Are follow up questions needed here: No.
So the final answer is: technically still at war
#
Question: when does it's always sunny in philadelphia season 13 start Are follow up questions needed here: No.
So the final answer is: September 5, 2018
#
Question: who sang you got friend in me from toy story follow up questions needed here:
So the final answer is: Randy Newman
#
Question: when was the first person sent space Are follow up questions needed here: No.
So the final answer April 1961
#
Question:
Figure The SA-NR prompt used in our NQ experiments.
20

---

Published as a conference paper at ICLR 2024
Given the following question, answer it by providing follow questions and intermediate answers. If intermediate questions are not necessary, answer the question directly. You are provided with evidence that can help you arrive the answer before the question.
#
Contextl: The Big Red One: Fuller was a World War Il veteran and served with the 1st Infantry Division, which nicknamed "The Big Red One" for the red numeral "I" on the division's shoulder patch. He received the Silver Star, Bronze Star, and Purple Heart during his service. Question: how did the big red one get its name Are follow up questions needed here: No. So the final answer its shoulder patch #
Contextl: Location Map of Cayman Islands: The given Cayman Islands location map shows that the Cay- man Islands located in the western Caribbean Sea. Location Map of Cayman Islands. Where is Cayman
Question: where are cayman islands on the map Are follow up questions needed here: No.
So the final answer is: western Caribbean Sea #
Combatants, Summary, Years, Map Britannica: After more million combat casualties had been suffered on both sides, the fighting ended in July 1953 with Korea still divided into two hostile states. Negotiations in 1954 produced no further agreement, and the front line has been accepted ever since the de facto boundary between North and South Korea. Question: who the between north korea and south korea Are follow up questions needed here: No.
Contextl: Korean
So the final answer is: technically still at war #
Contextl: It's Always Sunny in Philadelphia (season 13): The thirteenth of the American comedy television series Its Always Sunny in Philadelphia premiered on FXX on September 5, 2018. The season consists
Question: when does it's always sunny in philadelphia season 13 start Are follow up questions needed here: No. So the final answer is: September 5, 2018
You're ve Got Friend in Me: "You' ve Got a Friend in Me" is song by Randy Newman. Used as the theme song for the 1995 Disney/Pixar animated film Toy Story, it has since become major Question: who sang you got friend me from toy story follow up questions needed here: No. So the final answer is: Randy Newman
Contextl:
#
Context1: April 1961: Yuri Gagarin from the Soviet Union was the first human in space. His vehicle, Vostok 1 circled Earth at speed of 27,400 kilometers per hour with the flight lasting 108 minutes. Question: when was first person sent to space follow up questions needed here: No.
So the final answer is: 12 April 1961
#
Question:
Figure The SA-R@I prompt used in our NQ experiments.
21

---

Published as a conference paper at ICLR 2024
Given the following question, answer it by providing follow questions and intermediate answers. If intermediate questions not necessary, answer the question directly. You are provided with evidence that can help you arrive the answer before the question.
#
Contextl: 16th Infantry Regiment (United States): As part of the new 1st Expeditionary Division, soon become known as the 'Big Red One', the 16th Infantry, commanded by William Herbert Allaire Jr., sailed Question: how did the big red one get its name Are follow up questions needed here: No. So the final answer is: its shoulder patch
Contextl: Module:Location map/data/Cayman Islands: Module:Location map/data/Caymaa Islands is a location map definition used to overlay markers and labels an equirectangular projection map of Cayman Question: where the cayman islands the map Are follow up questions needed here: No.
So the final answer is: western Caribbean Sea #
Context1: First Battle of Scoul: The First Battle of Scoul, known in North Korean historiography as the Liberation of Scoul, the North Korean capture of the South Korean capital, Scoul,
Question: who won the war between north korea and south korea Are follow up questions needed here: No.
So the final answer is: technically still at war #
Context1: It's Always Sunny in Philadelphia (season 13): The thirteenth of the American comedy
television series Always Sunny in Philadelphia premiered on FXX on September 5, 2018.
Question: when does it's always sunny in philadelphia season 13 start
Are follow up questions needed here: No.
So the final answer is: September 5, 2018
Contextl: Randy Newman You've ve Got a Friend in Me Lyrics: 'You've Got Friend In Me' is the theme song of the Toy Story franchise, recurring throughout the series in different contexts, It's first Question: who sang you got friend in me from toy story Are follow up questions needed here: No. So the final answer Randy Newman
#
Context1: Timeline of space exploration: This is timeline of space exploration which includes notable achievements, first accomplishments and milestones in humanity's exploration of outer space. Question: when was first person sent to space follow up questions needed here: No. So the final answer is: 12 April 1961
#
Question:
Figure 10: The SA-R@ 10 prompt used in our NQ experiments.
22


Reviews:
[{"review_id": "PPYegqYgQX", "reviewer": "Reviewer_CyE8", "summary": "This work presents studies cases where noisy retrieval reduces accuracy in retrieval-augmented LM systems. They propose two methods. In the first one, they use off-the-shelf entailment (NLI) models to fall-back to the LM's internal knowledge when NLI models judge the retrieved context as irrelevant. This method shows some promise but is too aggressive at skipping retrieval.\n\nIn their second method, they fine-tune the LM itself so it's robust to noisy contexts in single- and multi-hop settings. While this is easy to handle for single-hop questions, the authors propose a data generation algorithm that creates fine-tuning data for multi-hop robustness. This method prompts an LLM to generate multiple decompositions of multi-hop questions, and use a self-consistency check to identify high-quality training examples.\n\nThey conduct a rich analysis that shows that irrelevant context \"causes a wide range of errors, which include copying irrelevant answers from the retrieved sentences and hallucinating incorrect answers and decompositions.\" Their evaluation shows gains in practice across several QA benchmarks.", "questions": "1. Llama2 here refers to the vanilla or the chat variant?\n\n2. The choice of NLI model is quite underwhelming. Do you expect this to be different with good prompting of good LMs? Or with better finetuning for NLI?\n\n3. The focus on top-1 passage weakens the search space of the ideas in the work. Have you considered filtering passages (out of several passages) with NLI, or keeping the best-scoring NLI passage?\n\n4. The analysis conducted ends in two rich notes that I expect to yield a lot of value for this paper. I'd be willing to update my score (up or down) depending on that.\n\nQuote 1: \"In addition, the SA-R@1 that contains the top-1 results is not the best performing even when retrieving top-1 results at inference time, and is the worst performing when retrieving noisy contexts at inference time, suggesting that showing examples for retrieval during in-context learning has a negative affect that causes over-utilization of irrelevant context\"\n\nQuote 2: \"for at least 36% of the cases the generated answer or decomposition is correct, but the retrieved context does not directly entail the generation. This can be partially explained by the ability of the model to combine retrieved evidence and its parametric knowledge.\"\n\nHow robust is Quote 1 across LLMs and selection of examples? How can Quote 2 inspire a better way to incorporate NLI?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and very easy to follow. (I give 3/4 for presentation only because of note #2 in weaknesses.)\n\n2. The work is highly systematic, starting from first principles and building multiple rich systems for RALM, with well-conducted experiments sprinkled throughout to support all key claims. The results are solid.\n\n3. The multi-hop data generation approach is novel and interesting.", "weaknesses": "1. If I understand correctly, you use the irrelevant context (e.g., in the single-hop case) to train the LM to answer the question by ignoring the context. Isn't this (almost) the definition of hallucination? The resulting LM will produce information not grounded in any passages. Isn't it better to abstain / request a new query, if the context is irrelevant?\n\n2. More fundamentally, it seems like the take-away message is almost presented as \"you should finetune on some examples with irrelevant/distracting context mixed in\", which however is a very old message incorporated already in multiple mainstream RALM research papers from 2020 on Open-QA (if not indirectly from circa 2018 with HotPotQA, I can't confirm this one).\n\nIt seems that the bigger contribution is: how to apply this for multi-hop tasks (interesting pipeline with code-davinci-002) and the analysis conducted, though I think this demands some changes to the discussion to make it clear what precisely is portrayed as new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNI6IOMqzB", "reviewer": "Reviewer_jc66", "summary": "Retrieval-augmented LM are of major interest in both applied and research contexts. This paper addresses the problem of cases where the retrieved context is not actually relevant and actually degrades task performance. The paper provides error analysis of this phenomenon over benchmark datasets covering variation in problem setting, and proposes two approaches to make RALM \"robust\" to irrelevant context (ie, minimize performance loss). The first approach is a simpler modular \"black box\" solution where a separate (NLI) module evaluates the relevance of the context to reject suspected irrelevant context and prevents it from being supplied to the LM, and the second involves fine-tuning the LM to provide correct answers even when provided irrelevant context. Experiments illustrate improved performance.", "questions": "Will the released dataset contain the Google search results? If not, it may be difficult to fully reproduce the results. This would also handle the fact that these queries were issued against a particular point-in-time snapshot of Wikipedia.\n\nMinor comments / typos:\n* \"parametric memory\" - I understand this is an increasingly common term/phrase, but the first time this term is used in the paper it might be helpful to define it.\n* \"in-context learning has a negative affect\" - should be \"effect\".\n* Figure 5: \"impr oved\" typo, should be \"improved\".\n* \"Overall, this suggest\" - should be \"suggests\".", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core problem of degraded RALM performance due to irrelevant context is very compelling from both practical application and general research perspectives. Also it is useful that, besides providing a \"baseline\" of sorts, the simpler NLI approach is suitable for applications where fine-tuning is not feasible or greater system modularity is desired for architectural reasons.\n\nOverall, the presentation and writing are clear. \n\nThe variation in the benchmark types (single-hop, explicit and implicit multi-hop) provided good coverage of the underlying problem and demonstrated interesting behaviors, as did the various ablations and alternative configurations.\n\nSection 5 (Analysis) has multiple examples of the authors _manually analyzing_ examples that meet certain criteria in order to develop better insights into the behavior - this is fantastic to see and I thought it had good pay offs in terms of deeper / richer understanding. I thought the error analyses here were some of the most interesting and compelling parts of the paper.", "weaknesses": "I would have found it helpful to have other overall findings briefly summarized at a bit higher level for, eg a practitioner trying to build an RALM application. Something like: \"NLI-filtering can increase robustness to noisy IR, but at the cost of leaving IR gains on the table in some cases due to False Negatives. If possible for your setting, fine-tuning the model with intentionally varied IR quality seems to improve robustness without sacrificing performance.\"\"\n\nFigure 4 and Figure 5 conveyed all the results, but I still found it a little confusing or tedious to match up the pairwise analyses from the text to the \"shapes\" of the bar plots. Unfortunately I don't have a specific suggestion in mind here, but it did feel like a lot of cognitive load on the reader to swivel back and forth. \n\nThe claim that the fine-tuning teaches the model _when_ to use the context is not clearly established. That is, the results show that fine-tuning in the presence of both relevant and noisy contexts improves generalization results, but the mechanism by which it accomplishes this is not really known or demonstrated.\n\nResults with Llama-2-70B: \"suggesting it has more parametric knowledge\", I'm not sure this assertion is necessarily supported by the observations either", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJTXy8SUE6", "reviewer": "Reviewer_dWp6", "summary": "This paper makes a systematic study about the robustness of RALM, where it identifies the potential threat resulted from irrelevant retrieved contexts. The paper further introduces two approach to confront this problem. One is to leverage a small-scale NLI model to filter out the irrelevant context. The other one is to fine-tuned the language model with  a mixture of relevant and irrelevant contexts. The paper performs comprehensive experiments on one-hop and multi-hop QA datasets to verify the proposed methods.", "questions": "Q1. How generalizable is the fine-tuned model? If it is applied to other QA datasets, especially with different kind of context noise, what will happen?\n\nQ2. Which part of the experiment can support the statement \"models finet-uned solely with relevant contexts are far less robust than those finetuned with a mixture of relevant and irrelevant contexts\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "s1. This paper presents a through analysis for the robustness of RALM to noisy context, which is fundamentally important to the research of LLM, question answering, and information retrieval.\ns2. Despite simplicity, the two proposed methods are meaningful and empirically positive.", "weaknesses": "w1. The use of a filtering module to mitigate contextual noise and filtering the language model with noisy context are two established approaches found in various related works on open-domain question answering and conversational question answering. While there may be variations in specific implementations, they might not be regarded as technical breakthroughs for this problem. This paper should conduct a more comprehensive investigation of related techniques. \n\nw2. The experimental study can be improved (please refer to my posted questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b00uMQncLO", "reviewer": "Reviewer_Kczf", "summary": "This paper presents an analysis on open-domain question answering benchmarks where retrieval harms the performance. The authors also propose two methods to mitigate such issue: 1. using an NLI model to filter out passages that do not entail question-answer pairs, and 2. automatically generating data to fine-tune the language model to be robust to irrelevant contexts. The authors show that as few as 1000 examples suffice to train the model to be robust to irrelevant context while preserving the performance.", "questions": "- Additional “ranked” in section 3.2.1?\n- For generating training data, it seems that the top-1 passage generated from Rc(q) is considered “relevant”, but for some harder questions, or suboptimal retriever, this is not guarantee, and even google search failed sometime for harder dataset. Although this might not be in the scope, I wonder whether there is any remedy for that?\n- How do you define ambiguous question for section 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper focuses on irrelevant context for open-domain question answering, which is a key and crucial part for RALMs performance. The paper proposes an automatic way to generate decomposed questions for training", "weaknesses": "- I am not fully convinced that the NLI model work / add any significance to the paper presentation, since there is no guarantee on the accuracy if there is no gold passage provided. With results and analysis from section 4 and 5, I don’t believe this proposed NLI model can be claimed as a significant contribution for this paper.\n- section 4 can be better presented, the color coding is a bit confusing…\n- For the analysis in section 5, conclusions drawn from 40 / 25 examples does not show enough statistical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)