Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model
Authors: Parishad BehnamGhader
Santiago Miret
Siva Reddy
Venue: EMNLP 2023 Findings (2023)
Keywords: retriever-augmented language models
reasoning of language models

Abstract: Augmenting pretrained language models with retrievers has shown promise in effectively solving common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmented language models, namely kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever + Flan-T5, in reasoning over retrieved statements across different tasks. 
Our findings indicate that the simple similarity metric employed by retrievers is insufficient for retrieving all the necessary statements for reasoning. Additionally, the language models do not exhibit strong reasoning even when provided with only the required statements. 
Furthermore, when combined with imperfect retrievers, the performance of the language models becomes even worse, e.g., Flan-T5's performance drops by 28.6% when retrieving 5 statements using Contriever. While larger language models improve performance, there is still a substantial room for enhancement. Our further analysis indicates that multihop retrieve-and-read is promising for large language models like GPT-3.5, but does not generalize to other language models like Flan-T5-xxl. The code is available at https://github.com/McGill-NLP/retriever-lm-reasoning.

Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model
Parishad BehnamGhader' Santiago Miret Siva Reddy1.2
McGGill University / Mila FFacebook CIFAR AI Chair IIntel Labs
beenaahhdbbe,,,,gdddrrrr siva. reddy] Jemila.mmmlee quebec santiago mewiiagg.mmeettinee..ccom com
Abstract
Question: Photoos clesse which type of body?
Retrieved Statements
Augmenting pretrained language models with retrievers has shown promise in effectively solv- ing common NLP problems, such as language modeling and question answering. In this paper, we evaluate the strengths and weaknesses of popular retriever-augmeeneed language models, namely &NN-LM, REALM, DPR FiD, Con- triever + ATLAS, and Contriever + Flan-T5, T5, reasoning over retrieved statements across different tasks. Our findings indicate that the simple similarity metric employed by retriev- ers is insufficient for retrieving all the neces- sary statements for reasoning. Additionally, the language models do not exhibit strong reason- ing when provided with only the required statements. Furthermore, when combined with imperfect retrievers, the performance of the lan- guage models becomes even worse, e.g., Flan- T5's performance drops by 28.6% when retriev- ing statements using Contriever. While larger language models improve performance, there is still a substantial room for enhancement. Our further amalysis indicates that multihop retrieve- and-read is promising for large language mod- els like GPT-3.5, but does not generalize to other language models like Flan-T5-xxI.
Knowledge Statements
oratts Photos named
1. Photos orbity Mans.
retriever
2. Mars a kind of planet
god tear and panio Classtfying
3. Moons arbit planets
4. Phobos is named alter the Greek
sclence process.
god of foar and panie.
6. A moon is located in space.
6. Class fying a kind of science
process.
INNLM
orbits Mars.
ideal retriever
REALM
Photos
Retrieved Statements
language modul
Phooos orbits Marsi
ATLAS
orbii planets.
Nars is kind of planeti
Moons orbil planets.
Fann-1
moon
Figure I: Example of retriever and language model failures when reasoning is needed. The correct and incorrect retrievals are highlighted in green and red respectively. This example demonstrates that the re- trievers' similarity metric is insufficient for retrieving required statements, and language models cannot per- form reasoning over the retrieved statements perfectly.
2020; Verga et al., 2021; Zhong et al., 2022). This augmentation helps them acquire new knowledge on-the-fly from external sources rather than relying solely on the implicit knowledge encoded in the model's parameters, thereby making them more robust to domain shifts (Izacard et al., 2022b). Fur- thermore, the retrieved knowledge can provide in- sight into what knowledge the model using.
1 Introduction
Parametric language models, such as decoder-only transformers (c.g. GPT), transformer encoder mod- els (e.g. BERT), and encoder-decoder transformers (e.g. T5), encode all necessary knowledge to solve a given task their parameters and have demon- strated exceptional performance on many natural language tasks (Vaswani et al., 2017; Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020). Non- parametric models improve these models further by augmenting them with knowledge retrievers (Guu et al., 2020; Izacard and Grave, 2021; Izacard et al., 2022b) or memory components (Khandelwal et al.,
While the capabilities of parametric language models have been extensively studied in the lit- erature (Wei et al., 2022; Zelikman et al., 2022), there is no thorough study of the limitations of non-parametric models. For instance, Mallen et al., 2023 examine the performance of non-parametric memories when encountering less popular knowl- edge. In contrast, our work takes systematic approach to study the limitations of retriever- augmented language models for reasoning over re- trieved information. As depicted in Figure 1, these models often fail to solve the tasks that require sequential logical reasoning, such as taxonomic chaining and combining the details.
'The code is availahle at https//githuh.comMMGiiin p

---

In this study, we demonstrate how retriever- augmented language models struggle with logi- cal reasoning when the task involves reasoning over multiple statements. We evaluate these mod- els in language modeling (LM) and question an- swering (QA) tasks using different variations of EntailmentBank (Dalvi et al., 2021) and Strate- gyQA (Geva et al., 2021) datasets, where we have control over the provided supporting statements and reasoning skills. Notably, these datasets do not explicitly indicate the reasoning path within the question itself, making the retrieval process more challenging. other words, knowledge statements and queries may not have surface-level lexical sim- ilarity. For instance, the question in Figure 1 has no lexical similarity with required statements (2) and (3), and without strong reasoning component, models would struggle to retrieve and reason upon such statements.
Photos kind [MASK]
Query
Photos should be classifed
Knowledge Statements
type body?
Photos named
the Gonel
Mara a kind planet
Language Model
Retriever
orbils mara.
query
Moons arbit planets.
Statements
moon space.
The architecture of retrieve-then-ren read retriever-augmenter language models. The language model predicts the answer using the query and the re- triever's selected statements.
Figure 2:
2.1 Retriever-Augmentted Language Models
Major work in eetriever-augmentt language mod- els focused on improving downstream perfor- mance such as question answering and language modeling using retrieve-then-read paradigm (Guu et al., 2020; Izacard and Grave, 2021; Izacard et al., 2022b; Khandelwal et al., 2020). Fig- ure 2 illustrates generic retrieve-then-read chitecture. Some of the popular models that we study are kNN-LM, Retrieval-Augmeented Lan- guage Model (REALM), Fusion-in-Decodee (FiD), and ATLAS (Khandelwal et al., 2020; Guu et al.. 2020; Izacard and Grave, 2021; Izacard et al., 2022b). Each of these leverages retrievers and lan- guage models in a unique manner.
Concretely, we analyze the performance of kNN- LM, REALM, DPR FiD, Contriever ATLAS, and Contriever Flan-T5, As illustrated in Fig- ure these models exhibit shortcomings rooted in both parts of their design: 1) Retrievers struggle to select all the necessary statements for reason- ing when using the similarity between query and knowledge statements (44.3.1);2 2) Language mod- els are imperfect reasoners even when provided with perfect retriever that retrieves all the essen- tial information ($4.3.2); 3) Moreover, the perfor- mance of language models deteriorates further if the retriever imperfect, closer setting real- ity ($4.3.3); 4) Additionally, experimental results indicate that while larger language models yield improvements, even the largest models we stud- ied are imperfect reasoners (84.3.4); 5) Finally, we observe that employing multihop retrieve-and- read enhances GPT-3.5's performance in reasoning tasks, but this improvement does not extend to other models such as Flan- Tan-T5-xxl (84.3.5).
2.2 Role of Retriever
The retriever's role is to retrieve relevant statements for a given query which are then used by the lan- guage model. Dense Passage Retriever (DPR), Contriever, and REALM's retriever fetch the most similar statements based similarity metric (i.e., dense inner product) between query and statements' representations using two independently trained one pre-trained BERT-based encoder (Karpukhin et al., 2020; Izacard et al., 2022a; Guu et al., 2020). In contrast, kNN-LM adopts an L2 similarity met- ric between the representation of the query and partial token sequences (instead of full knowledge statements) to select the most relevant sequences of tokens (Khandelwal et al., 2020). While retrievers typically select statements from large common corpus in the literature, as depicted in Figure 2, we provide speciffe set of statements for cach query to control the supporting information.
2 Background
There has been growing interest in investigat- ing the reasoning abilities of parametrie language models (Wei al., 2022; Chung al., 2022). In contrast, our work focuses on the reasoning abili- ties of retriever- augmented language models. This paper further studies the contributions of each of the models' components.

---

2.3 Role of Language Model (Reader)
Model
# Params
Model
Params
Language Models
The role of the language model to make use of the retrieved knowledge to generate relevant text for the given input query. &NN-LM, decoder- only Transformer, computes the distribution over the next token during generation by interpolating between the transformer's 's next token distribution and the next token from the nearest neighbor mem- ory (Khandelwal et al., 2020). On the other hand. REALM is a masked language model backed by a BERT-based reader that extracts the most promis- ing span from one of the statements as the an- swer (Guu et al., 2020).
REALM
270M
FiD
220M
&NN-LM
250M
ATLAS
250M
Flan-T5-hase
250M
Model Size and Multihop Retrieve-and Read Analysis
80M
Flan-T5-xl
~3B
hlan-T5-hase 250M
Flan-TS-xxl
~1
Flan-T5 la-are ~700
GPT-3.5
~175B
The number of model parameters. We con- trol for model size circumvent the role of size in reasoning abilities. Moreover, we evaluate the impact of model size and multihop retrieve-and-rent using larger models.
Table 1:
FiD and ATLAS are both sequence-to-sequee T5-based neural networks (Izacard and Grave, 2021; Izacard et al., 2022b). ATLAS is specifically designed for jointly finetuning the language model and the retriever, employing various pretext tasks with limited training examples. In these models, the encoder encodes the query and each retrieved statement individually, and the decoder attends to these representations solve the downstream task. While Flan-T5 was not specifically built for retriever-based language modeling, since it is an instruction-tume -unded model, it can be combined with any retriever to complete downstream tasks using the retrieved information (Chung et al., 2022).
and effectively solve the target task through rea- soning over the retrieved knowledge S,. visu- alization of the task is illustrated in Figure 2. We control for model size wherever possible for com- parable results among different models. We also study the effect of scaling the model size. Table presents the number of parameters of the studied models. Appendix A contains additional imple- mentation details. We analyze the performance of these models on two tasks, Language Model- ing (LM) and Question Answering (QA), using datasets (Section 4.1) that are specifically curated for reasoning.
2.4 Multihop Retrieve-and-Read
addition to the previously mentioned widely used retrieve-then-read models, multihop retrieve- and-read iteratively utilizes textual or dense search queries for fixed predefined or variable number of iterations (Xiong et al., 2021; ; Qi et al., 2021; Khot et al.. 2020; Khattab et al., 2022). For instance, Semoontrate--earch-Pee Predict (DSP) can be used in multihop question answering. In each iteration, DSP employs large language models to generate a query by decomposing a complex question into smaller subproblems and summarizes information from retrieved knowledge (Khattab et al., 2022).
Language Modeling (LM). In the language mod- eling setup, we evaluate the effectiveness of the auriever-augmented language models in a targer ranking task. In this task, the model should assign higher likelihood to the correct sentence compared to (up to) four alternative similar but incorrect sen- tences. For instance, in the example illustrated in Figure 2, the models should rank the sentence "Phobos is a kind of moon." " higher than an alterna- tive sentence such as "Phobos is kind of planet." " This task allows comparing masked language mod- els like REALM with autoregressive models. We explain this task in more detail in Appendix C. and present its experimental results in the Appendix D.
3 Problem Definition
In the main experiments, we provide the mod- els with complete set of knowledge statements denoted S {si, $2..... for each sam- ple. In some cases, only subset of these state- ments essential for predicting the answer, re- ferred to as gold statements. The primary objective the models is to 1) retrieve a set of statements S, = fri, r2, r2,....,kk rk] ) C S which find necessary
Question Answering (QA). In the question an- swering setup, the model should answer question given retrieved statements, as illustrated in Figure 2. Except for &NN-LM, all the other models are al- ready exposed to question answering tasks during their training.

---

4 Experimental Setting
DPR
REALM
Contrever
4.1 Datasets
We assess the performance of the models using the following reasoning datasets. These datasets enable
Accuray
evaluate the models' reasoning abilities and the retrievers' performance in LM and QA tasks while controlling for the available supporting information.
30
Langunge Modeling
Queston Anowning
Tasks
EntailmentBank (EB, Dalvi et al., 2021) consists of an input question or hypothesis statement that can be inferred only through multi-step reasoning on the provided statements. For the QA task, we use the question format of the input, and for the LM task (i.e., target ranking), we use the hypothesis statement and alternative statements obtained by replacing the target entity with other alternative entities (see Appendix B). As knowledge statements, we use the provided gold and distracting statements in the dataset. The distracting statements mostly contain entities relations mentioned in the input query which might make them deemed relevant but in actuality they are irrelevant. For instance, for the question "Which rock rype is most useful in studying the history of living organisms?", sentecnes "Nearly all fossils are found in sedimentary rock." and "Limestone is kind of sedimentary rock." are relevant, and sentences "Rock means stone." and " "Oraanisms can be preserved in sedimentary rock." are distracting and irrelevant.
Figure 3: Retrievers' accuracy EB-Hard test set in LM and QA tasks. Results show that retrievers do not select required statements properly, as the best retriever, Contriever, achieves only 47% accuracy in QA task.
Appendix B.
4.2 Evaluation Metrics
In the QA setting, we use the token overlap Fl score between the predicted answer and the gold answer. For the LM task, we evaluate the accuracy of the models in the target ranking problem.
For the retriever, we use accuracy and recall score to indicate the overlap between retrieved statements and the ground-trute statements which are essential to obtain the answer.
4.3 Evaluation and Discussion
In this section, we analyze the limitations of both retrievers and language models in reasoning tasks.
EB consists of multiple splits but we use the EB-2 split since its knowledge store contains both required and distracting statements. We call EB-2 split as EB-Hard in this paper. For each input, there will be up to 25 statements, of which up to 12 statements are required to answer correctly. EB-1 is similar to EB-2 but without any distracting statements. EB also has EB-3 data which includes 25 relevant and irrelevant statements sampled from a large corpus, but we find the trends to be similar to EB-2 (see Appendix D).
4.3.1 The Shorteomings of Retrievers
Current retrievers select k statements based on a relevance score, such as inner product or dis- tance between the query's ss and statements' (or spans of statements) representations (Guu et al., 2020; Karpukhin et al., 2020; Izacard et al., 2022a; Khan- delwal et al., 2020). However, this approach is insufficient for reasoning. Consider the question "Phobos should be classified as which type of body?" in Figure 1, The retriever may select similar state- ments to the query, such as (1) "Phobos orbits Mars." and (4) "Phobos is named after... ... ", but none them contains the answer "moon." " Instead, combin- ing statement (1) with missed statements (2) Mars is a kind of planet." and (3) "Moons orbit plan- ets." would provide the answer to the question. But statements (2) and (3) the ones with the least similarity.
Strategy (Geva et al., 2021) contains yes or no questions accompanied by up to 5 supporting statements from Wikipedia. To evaluate the models in the language modeling setting, we convert each question and answer into a declarative statement, while also generating incorrect statements using alternative entities. The detailed dataset preparation process explained in
We validate our hypothesis that the similarity- based metric for retrieval insufficient for reason- ing using the EB- Hard dataset. Since the EB-Hard

---

Model
Query
Statements
Prediction
In located in warm region, what should be included in the polar bear hibit?
If animal lives certain environment then that animal usually requires that kind of environment. Polar bears live in cold environments.
DPR
+ FiD
Contriever ATLAS
Moons orbit planets. Gravity causes orbits.
What keeps the Moon orbiting Earth?
elliptical
+ As the force of gravity decreases, the weight of the object will decrease.
The robot will weigh less on mars than earth but will have the [MASK].
ENN-LM
The gravitational force of a planet does not change the mass of an object on that planet or celestial body.
mars
Targets: mass vs mars
Table 2: Some examples of models' failures rooted in the retriever. One of the correctly retrieved statements and the one that had to be retrieved in order for the model to solve the task correctly are highlighted in green and red respectively. The sequence of tokens leading to the true answer is marked in bold. Results show that current retrievers struggle to retrieve required statements for solving the tasks.
dataset provides us with both gold and distracting statements to answer a question or infer a hypothe- sis, we evaluate the accuracy of different retrievers when we know k. i.e., the exact number of required statements to retrieve for a given input. We present the results in Figure triever, Contriever, achieves only 57% and 47% ac-
DPR
ANNALM
REALM
Contriever
Language Modeling
100
Resievad statemants rocall score
50
The best-performing re-
5
8
curacy in the LM and QA tasks, respectively. DPR,
Question Answering
the widely used retriever with BERT-based dual en- coder trained on Natural Questions (K (Kwiatkowski et al., 2019), is the worst with an accuracy of only around 1 15%. REALM's retriever, which is DPR trained on self-supervised large-scale salient span masking on Wikipedia and fine-tuned on Natural Questions improves the performance by a large margin but still has only 40% accuracy. ANN-LM's retriever is trained on Wikitext- 103 (Merity et al., 2017) in an auto-regressive fashion to select state- ments that help predict the next word of the input. Due to this, its performance drops 16 points when evaluated for the QA task compared to the LM task, the QA task is out-of-distributtin for &NN-LM.
50
5
ONE
of ratriveed statements
Figure 4: Retreivers' recall score on EB-Hard test set in LM and QA based on the number of retrieved statements (k). Contriever is shown to be superior among the studied retrievers. Results further indicate that ÀNN-LM does cover 100% of the gold state- ments when i = 100 (NNN-LM's recall is ~ 97%).
it still fails to cover 100% of the gold statements. Failures of retrievers illustrated in Table 2 demonstrate that relying solely on query-statement similarity for retrieval may lead to overlooking im- portant statements that are dissimilar to the query but contain information essential for reasoning.
Figure 4 further illustrates the recall score of the retrieved statements for varying k. While all retriev- ers reach a recall of 100% at 25 (i.e., the maximum number of knowledge statements for each sample in EB-Hard), &NN-LM still struggles due to the way it represents the facts in its memory compo- nent. In fact, each statement s = W1 Wn is stored as n key-value pairs in memory. For instance, the key w1 is paired with value w2 (i.c., next token). similarly wi W2 with value wy, and so on. The retriever computes the similarity between the input query and all the keys and selects the top l similar keys (i.c., sequences of tokens) among all. When allowing &NN-LM to retrieve even 100 keys,
4.3.2 The Shorteomings of Language Models The language model has to reason upon the re- trieved statements to answer a given input. In this section, we assume access perfect retriever. To estimate the upper-bound performance of each language model, we use the final single state- ment hypothesis available in EB datasets. This statement can be entailed using an entailment tree

---

REALM
The planets revolve counterelockwise direction. cause of the revolution is mostly due to which force?
Question:
OPR FD
Conmener ATLAS
Conpiever + Fann75
Single Stueenerr
Statement: Gravity causes planets the solar system to orbit the sun.
80
Expected answer: gravitational
ATLAS
gravity
Model:
Response:
40
Flan-T5
gravity
Question:
When compared to the Sun, red dwarf stars
are
Statement: Red dwarf stars cooler than the sun.
Expected answer: cooler
of retrived siatements
Model: ATLAS
cooler than the sun. lot smaller
Response:
Flan-T5
Figure 5: Token overlap Fl score of language models EB-Hard QA test set. The dotted and solid lines re- for to experiments given the single oracle statement and gold statements (when reasoning is required), respec- tively. Results illustrate that language models perform worse when answering the question requires reasoning.
Table 3: The predictions of models when the oracle statement that contains the answer is provided. Fl metric unnecessarily penalizes ATLAS here, although its predictions are correct. Flan-T5's response to the second question is not grounded to the oracle statement and the model is only using its internal knowledge answer the question.
and sufficient to answer the given input. For the example in Figure 2, this statement is "Phobos is a kind of moon", which we refer to as the or- acle statement. The oracle statement is provided to the model the retrieved statement to answer the question "Phobos should be classified which type of body?" (or, for the LM task, to rank the sentence "Phobos is a kind of moon" higher than others which is trivial).?
1a0 cvesso Aut coneeoee
22
Ioseeed tomnentcca)
Figure 5 shows the results of QA task in dot- ted lines. The models ATLAS, FID, and REALM perform the best this task, these models are mainly trained perform QA.
the quedion
Incorrect
prounded ool
Incorrect
10%
4%
Table 3 shows some outputs of various mod- cls. These oracle scores would give an estimate of the upper-bound bound performance of models when the task does not require reasoning over multiple statements. Surprisingly, Flan-T5 performs worse than expected the model struggles to limit it- self the provided knowledge and relies on its parametric memory, a common challenge faced by large language models in non- con-paraettric setting (Mallen et al., 2023). This claim backed by our further analysis on 50 randomly sampled (from among 340 samples) Flan-T5 responses given the oracle statement. The statistics shown in Figure 6 show that 18% of the Flan-T5 responses are not grounded in the oracle statement, even though the oracle statement was relevant to the question and answer. Additionally, 32% the responses yield
Incomecc
Figure 6: Human analysis of Flan-T5's responses given the oracle statement. As demonstrated in Ta- ble 3, 18% of the Flan-T5 responses are not grounded the oracle statement and 32% responses yield low Fl score even though they correct.
low Fl score even though they are correct (due to elaborateness wording mismatch). The scores are lower than expected since the Fl evaluation metric penalizes when there is a word mismatch between true and predicted answer even the pre- dicted answer is semantically the same (Kamal- loo et al., 2023; Chiesurin et al., 2023; Adlakha et al., 2023). Some other examples of Flan-T57 responses also demonstrated in Table 10. Additionally, we experiment with a setting
Forr the LM task, the models achieve high performance since the resulting statement is the same as the target statement among the alternatives.

---

where the models have to reason on multiple re- trieved gold statements. Figure shows the results in solid curves. As we provide the models with an increasing number of gold statements that are essential to answer the input, the performance goes up for Flan-T5, T5, FiD, and ATLAS. However, we do not observe any change in REALM and kNN-LM. Although REALM is QA model, its limitations stem from how it makes use of retrieved documents. Instead of reasoning on all documents jointly, like ATLAS, FiD, and F1an-T5, reasons on each doc- ument separately. In pretraining phase, the model marginalizes the score of each span over all docu- ments. In the inference phase, the model picks the span with the highest retriever and reader score. In our setting, this almost always ends up selecting span from the first few statements, as the state- ments are in the order of importance (see Figure 1). We present additional examples in Table 4. kNN- LM is not designed to perform QA, which why performs worse. Moreover, in our LM experiments, we find that also underperforms other models, so it is unclear when kNN-LM style decoding is preferred (see Appendix D.1).
KNN
RE M (grkk)
RI (e4)
DFR FID (goid)
DPR FID (ol)
Contriever ATLAS (gck))
Controiver 1 ATLAS (a0)
Contreeeer Fan-15 (gord)
Contriever FlanT55 (xi)
Tolen overiap FL score
00
20
20
Vedbe
Figure 7: The negative impact of additional distract- ing information language models' performance in the QA task. The solid bars and the bars with patterns refer the experiments with all the gold statements and all gold and distracting statements, respectively. It can be observed that providing language models with distracting statements hurts their performance.
KNN-MM (poks)
INN-LM (2B)
REAL
AIiAl M
DPR RID
DPR
Contreeer ATLAS
ATLAS
Flan-T5
Token overlap F1 score
40
When we contrast the results of reasoning over multiple retrieved gold statements (solid curves in Figure 5) with only reasoning on the oracle state- ment (dotted), the performance of all models is much lower than the oracle performance even after providing all the essential gold statements. Inter- estingly, although Flan-T5's oracle performance is lower than ATLAS, FID, and REALM, out- performs them when reasoning over multiple state- ments, indicating that it is better reasoner. We conjecture that Flan- T5's multi task training which also includes reasoning tasks like chain-of-thought reasoning, makes it better reasoner than others.
10
retrevved statements
Figure 8: Performance of language models on QA EB-Hard test set coupled with imperfect retrievers. Coupling language models with imperfect retrievers (i.e., when retriever fetches distracting statement) deteriorates the overall performance.
4.3.3
The Blame Game: The Impact of Combining Imperfect Retrievers and Language Models
Furthermore, we investigate the impact of addi- tional distracting information on the performance of language models in the QA task in Figure 7. Flan-T5's performance drops by 8.7% in the pres- ence of distracting statements indicating that irrele- vant information hurts the performance of language models. Although ATLAS looks relatively robust, its performance is lower than Flan-T5, so can- not draw definitive conclusion.
This section explores how the combination of lan- guage models and imperfect retrievers exacerbates the performance of surrevver-then-reee models, a setting closer reality. An incorrect final answer could be blamed on the retriever's s inability to fetch relevant statements or the failure of the language model to reason on the retrieved statements.
Additionally, Figure 8 illustrates the perfor- mance of language models on QA EB--Hard dataset, when coupled with either ideal retrievers (given only gold statements) or imperfect retrievers (given
the LM task, we observe similar trends to the QA task and present the results and analysis in Appendix D.I.

---

Model
Query
Retrieved statements
Prediction
What allows students standing ten feet apart hear each other talk?
Talking is when human produces sound communicate.
Flan-T5
Sound travel through air by vibrating air.
microphone
Andy lives in the southern hemi- sphere. What season does he most likely experience in August?
Andy lives in southern hemisphere. August is during the winter in the southern hemisphere.
southern hemisphere
REALM
Table 4: Some examples of models' failures rooted in the language model. In each example, statements are illustrated. The true answer is marked in bold. We observe that 1) Flan-T3 does not limit itself to the provided knowledge, and 2) REALM extracts the response from the first retrieved statement, disregarding other retrieved statements.
retrieved
Flan-T Ta- (80M)
Flan- TS-base (2500)
DET1 F8D
Contrever ATLAS
Flan Ta large (780M)
- Flan TB-xi
SaalliyyAA
Flan TS
Toxen overtop F1 score
Token overlap soore
80
60
40
40
30
20
20
01 88
10
10 15 20 25
retreved statements
of retrieved statements
Figure 9: Performance of the retrieval-augmented models in QA on test based on the number of retrieved statements. The results demonstrate that al- though Contriever Flan-T5 and Contriever ATLAS superior, the studied models perform poorly soning when answering questions.
Figure 10: Token overlap F1 score Flan-T5 vari- ations EB-Hard QA test set. The results reflect the impressive impact of size on the performance of the models in the tasks where reasoning is required. We use Contriever the retriever in all experiments.
gold and distracting information). The results re- veal significant performance gap, as Flan-T5's performance in the QA experiment drops by 28.6% when retrieving 5 statements using Contriever: We further report the influence of imperfect retriev- ers the studied models in the LM task in Ap- pendix D.1.
ples, it is noteworthy that the retriever misses at least one gold statement in nearly 85% of the cases. Among these, only 19% are generated correctly by the LM. Conversely, when the retriever performs flawlessly (i.e., no missing gold statements), the LM correctly responds in 34% of the cases. In sum- mary, suriever-augmentt be more attributable to the retriever. This is under- scored by the noticeable improvement in the LM's performance (34% compared to 19%) when the triever performs perfectly. emphasizing the pivotal role of the retriever achieving correct generation.
LMs' failures appear to
Moreover, Figure 9 illustrates the performance of the models on QA reasoning datasets. When evaluating REALM and kNN-LM on the Strate- gyQA dataset, append yes/no to each statement. These results highlight the superiority of Contriever Flan-T5 in our experiments that matches our find- ing Section 4.3.2.
4.3.4 The Impact of Model Size
In order to study which component (retriever or LM) is more responsible for the failures of the retriever-augmented LMs, we make hypotheti- cal assumption. We assume that we have prior knowledge of the exact number of statements (k) be retrieved for each input. We then report the number of Contriever + Flan-T5's failure exam- ples (i.c., samples with Fl score less than 0.5) in Appendix D.3. Out a total of 340 data sam-
This subsection examines the influence of model size on the models' performance in reasoning tasks. Specifically, we analyze the performance of differ- ent variations of Flan-T5 coupled with Contriever on the EB-Hard dataset.
The experimental results in Figure 10 demon- strate that larger models achieve better Fl score. However, there still large room for improve- ment as Flan-T5-x T5-xt xxl achieves only 51.8% Fl when

---

GPT-3
DSP (GPT-3)
of multihop retrieve-and-read for large language models like GPT-3.5, but it does not generalize other models.
Flan-T base
DSP (Flan TS Ta-base)
Flan-TS-eal
DSP (Flun-T5-exf)
5 Conclusion
This paper analyzes the reasoning abilities Striever-augmentee language models. We first evaluate popular retrieve-then-read models, includ- ing KNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and Contriever Flan-T5, through lan- guage modeling and question answering tasks.
Token
Datiovett
Figure 11: Token overlap Fl score of GPT-3.5 and Flan-T5 variations using multihop DSP program. All the experiments are done with few-shot examples using Contriever the retriever and 5 retrieved statements in each retrieval step. The experimental results show that while DSP improves GPT-3.5 performance, does not generalize WannTT models in the Fl score.
Our experimental results indicate that retrievers fail to select all essential statements for reasoning when relying on the similarity between the query and statements. Moreover, we observe that lan- guage models also struggle to reason over state- ments even when distracting information is absent.
The performance deteriorates further when coupled with imperfect retrievers, as Flan-T5's perffomman performance
it has to reason over statements but the upperbound performance is 59.7% Fl when provided with the single oracle statement. The performance of the models on other QA datasets is presented in Ap- pendix F.
drops by 28.6% when retrieving 5 statements us- ing Contriever. Furthermore, while larger language models show greater capabilities in reasoning, they still have a large room for improvement. Addition- ally, our experiments on multihop retrieve-and show improvements on GPT-3.5, but these improve- ments do not generalize to other language models such as Flan-T5-xxl.
4.3.5 Impact Multihop Retrieve-and-
To address the limitations of retrieve-then then-read models reasoning datasets, we explore the ef- fectiveness of a strong multihop sarrive--andrees framework known as Aaabonstraat-Saacc--Preeeee (DSP), employing Contriever along with various language models (Khattab et al., 2022). The imple- mentation details of the multihop retricval experi- ments are provided in Appendix G.
These findings present opportunities for future research enhance the performance of retrieve- then-read models by addressing the aforementioned limitations of retrievers or language models. More- over, the development of multihop retrieve-and- read models holds promise for advancing reasoning tasks.
The token overlap Fl scores of the models us- ing the multihop DSP approach are depicted in Figure 11. The results indicate that while DSP im- proves GPT-3.5's performance, it still falls short compared to the serreve-then-reeee Flan-T5-xx xxl. This observation shows a large room for improve- ment in multihop retrieve-and-read methods. Fur- thermore, we observe decline in Flan-T5's per- formance when using multihop retrieval. This can be attributed to Flan-T5's inability to generate ap- propriate subqueries for the given question. Addi- tionally, its generated responses tend to include all retrieved information, leading to high recall but low precision scores. This phenomenon is further exem- plified in the qualitative examples and recall scores of the models, demonstrated in Appendix G. In summary, the results demonstrate the potential
Limitations
This paper examines the reasoning abilities of widely-used retriever-augmennee language mod- els in both LM and QA settings. To ensure fair comparison, we employ models with similar sizes in both LM and QA tasks. Additionally, we in- vestigate the impact of employing larger language models and a recent, strong multihop retrieve-ane and- read approach on the performance of these models. In this paper, while one can finetune the models the specific data, we focus the capabilities of the already pretrained retriever-augmented lan- guage models. Moreover, we analyze the models in two basic NL.P tasks, including language modeling and question answering. However, one of the stud- ied models kNN-LM has been pretrained only for

---

language modeling exclusively, which may have resulted in a subpar performance on the QA task. Surprisingly, kNN-LM also performs the worst at the LM task compared to others.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Finally, unlike the popular retriever-based based meth- ods in the literature which use large corpus of knowledge, we use data-specific set of statements for each data sample. This allowed to have more control over the retrieved statements and the rea- soning behavior of the language models.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. In Proceed- ings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing.
7 Ethics Statements
This is not applicable to this work.
Gautier Izacard. Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022a. Unsupervised dense informa- tion retrieval with contrastive learning. Trunsactions on Machine Learning Research.
References
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023. Eval- uating correctness and faithfulness of instruction- following models for question answering. arXiv: 2307. 16877.
Gautier Izacard and Edouard Grave 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Sabrina Chiesurin, Dimitris Dimakopoulos, Marco An- tonio Sobrevilla Cabezudo, Arash Fishghi Toannis Papaioannou, Verena Rieser, and Toannis Konstas 2023. The dangers of trusting stochastic parrots: Faithfulness and trust in open-domain conversational question answering. In Findings of the Association for Computational Linguistics: ACL2023, 2023, pages 947- 959, Toronto. Canada. Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave 2022b. Atlas: Few-shot learning with retrieval augmented language models. arXiv: 2208.03299.
Hyung Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, and Jason Wei. 2022. Scaling instuccioonfineture language models. arXiv: 2210. 1146.
Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. Evaluating open-domain ques- tion answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5591-5606, Toronto, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, Association for Computational Linguistics.
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighannaa Pipatanangkura, and Peter Clark. 2021. Explaining answers with entail- ment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7358-7370, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-48 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li. David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Composing retrieval and language models for knowledge-intensive nlp. arXiv: 2212. 14024.

---

Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence compo- sition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAA! 2020, New York, NY, USA, February 7-12, 2020, pages 8082-8090. AAAI Press.
transformer. Journal of Machine Learning Research, 21(140):1-67.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021. Adaptable and interpretable neural MemoryOver symbolic knowledge. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages Online. Association for Computational Linguisties.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wi Wing-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24831 Curran Associates, Inc.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9802-988 9822, Toronto, Canada. Association for Computational Linguisties.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguisties.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary Vito,o, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-801 8035. Curran Associates, Inc.
Wenhan Xiong, Xiang Li. Srini Iyer. Jingfei Du, Patrick Lewis, William Yang Wang. Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021. Answering complex open-domain ques- tions with multi-hop dense retrieval. In International Conference on Learning Representations.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems, volume 35. pages 15476-15488. Curran Associates, Inc.
Peng Qi. Haejun Lee, Tg Sido, and Christopher Man- ning. 2021. Answering open-domain questions of varying reasoning steps from text. Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 3599-3614, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 5657-5673, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text

---

A Implementation Details
Model
Target ranking
log 5.1 Dx, p..pc.pppppppp....... [CMSS]] TIQ. 8,) (s,10)
We present the implementation details of the ana- lyzed models in this section. Most of the experi- ments are conducted using Py PyTorch (Paszke et al., 2019) on an RTX8000 GPU with 48GB memory in single run, each taking few minutes to run. We also the experiments with large 30B- 30B-parameter models on an A100 GPU with 80GB memory. Note that we have changed the retriever in each model to retrieve statements from sample-speeific set of statements instead of large common corpus.
REALM
KNN-LM
FiD
log (QQ S-), where Qr the query with [MASK] tokens substitused with T
Plan-T5
ATLAS
xr+1
log (sextra_id_ T!Q,
Table 5: The target scoring function employed by each model. Our language modeling task uses target ranking score to rank multiple sentences with different target candidates filled-in answers. Target entity men- tion is indicated by T = f1/2 1./2...1a ...I.. input query by Q = 9192... 9N. and given retrieved statements by S,- For ATLAS, found that the T5 setup of predicting the mask with rextra_id__ performed slightly better than computing the probability of the entire sentence.
In REALM's experiments, we use the Hug- gingface's transformers implementation for both masked language modeling and question answering (Wolf et al., 2020). We load the check- point as a knowledge encoder for masked language modeling and rea-oena check- point for question answering. For ANN-LM experiments, we use the best checkpoint available in the original papers' GitHub repository, and we find A = 0.65 the best value as the interpolation hyperparameter based on the experiments Entai- mentBank development sets. In FiD experiments, we use nq_reader_base _base checkpoint available in the papers' GitHub repository with using checkpoint of the DPR retriever which is available in their GitHub repository. For experimenting ATLAS, we use the trained htt.daata/mmmeesaaalassssss checkpoint of both the language model and retriever. Also, for the Flan-T5 model, load the flan-t5-base model from Huggingface's transformers to be almost the same size as the other models the main experiments.
Doolmm-c-nees--pretrrieeeee
the yes/no answers into declarative-form sentences. This because StrategyQA samples only include questions as queries, not sentences. We also use hy- pothesis sentences of the EntailmentBank dataset for LM experiments. The next step includes cre- ating alternative sentences for each sample for the target ranking task in our language modeling exper- iments. We keep the data samples that include least one entity mention and mask out the last entity mention in the sentences of StrategyQA and En- tailmen(Bank using Spacy (Honnibal and Montani, 2017). Also, we randomly pick at most four other entities mentioned in the data sample's 's statements as the alternative targets (as described in Section 3) and compare the model's score for each target. Re- garding the question answering experiments, we use datasets' question and answer formats.
the nq, wggberttbbss-eecoder
For the experiments on the EntailmentBank datasets, we run the experiments on the same devel- opment and test sets as the original data. However, in the Strategy dataset, since we do not have access the answers in the test split, we cannot change the samples' formats to declarative form. Therefore, we pick 25% and almost 35% of the train data as the development and test sets, respec- tively.
In order to analyze the impact of the model size, we experiment with various Flan-TS models from Huggingface. Our multihop retrieval experiments employ Contriever addressed before with Ope- nAl's GPT-3.5 text-davinci-002 and different sizes of Flan-T5 model. More concretely, use the same query formatting and templates as provided in the DSP GitHub repository.
B Dataset Preparation Details
C Model and Task-Specific Query Format
In the language modeling experiments, we evalu- ate the models in target ranking task, where the model should assing higher likelihood to the cor- rect sentence compared to some alternative ones. Therefore, we first create an LM reasoning dataset for StrategyQA A by changing the questions and
This section includes the model-specific query for- mats in each task. As stated in Section 3, we aim study the reasoning abilities of retriever-augmented language models in language modeling and ques- tion answering tasks.

---

Model
Altemative Target Scores
D Main Quantitative Results
log |
En cS,, p (DHASK) lithopphere Q a))
REALM
log [85]
Eny ES, p (DMASK) coal 9t) (s,(Q)
This section includes more visualizations and de- tailed results.
ANN-LM FID F1an-T5
log p P(Surface mining alfects the lichosphere and biosphere. (S,) p Surface mining affects coul and biusphere.]55
D.1 Language Modeling
log p (<extra_id_e id_ lithosphere Q.S.)
ATLAS
log p (sextra_id, id_ coal(Q. S.)
First, we report the overall performance of retriever- augmented language models in the reasoning LM task in Figure 12. It can be observed that Flan- T5, an instruction-tuned QA model, performs as the best model. On the other hand, kNN-LM, a pretrained language model, performs as the worst model in our LM task.
Table 6: A sample of the ranking strategies for each model for target ranking in the LM task using re- trieved statements S,, The query (Q) in this example "Surface mining affects the [MASK] and biosphere." with alternative targets "lithosphere" and "coal". For ATLAS experiments, we replace [MASK] with <extra_id, a_id__e______ which is the specific masking token in T5-based mod- els.
Additionally, demonstrate the impact of addi- tional distracting information in Figure 13. Results indicate that similar QA experiments, in LM task, providing the language model with additional distracting information as well as all the required statements generally leads in performance degrada- tion.
C.1 Language Modeling
As explained in Section 3, we evaluate the perfor- mance of the popular retriever-aug auggeetd language models in the language modeling task through the largel ranking problem. In this task, the model should assign higher likelihood to the correct sen- tence compared (up to) four alternative similar but incorrect sentences. These candidate sentences are generated by replacing the masking tokens with alternative entities available in the knowledge state- ments of the data sample. Table 5 depicts the target scoring functions employed by each model for com- puting the score of each candidate sentence. These alternative target scoring functions are further ex- emplified in Table 6 for better understanding. The way models incorporate retrieved statements is explained in Section 2.3. In language modeling setup, the same happens for each model, except for REALM, which LM and QA variants differ. In QA, each statement is assigned a score separately, and the predicted span is the one with the highest retriever and reader score. In LM setup. instead, the score of each alternative target is computed by marginalizing over different retrieved statements, shown in Table 5.
Furthermore, we illustrate the impact of the im- perfect retrievers on language models in Figure 14. Results show that similar to the case with question answering task, the performance of the language models becomes even worse in LM when combined with imperfect retrievers.
Table demonstrates the performance of the best retriever-augmentee models (based on the per- formance on the dev sets) on the test sets in the language modeling task.
D.2 Question Answering
We demonstrate the performance of the models EB-Easy and EB-3 test set in question answering task in Figure 15. It can be observed that and Contriever + Flan-T5 performs the best these datasets, which matches our findings in the main paper. We observe that the models' performance is similar to EB- EB-d dataset with both relevant and distracting statements.
We report the performance of the best retriever- augmented models (based on the performance on the development sets) on the test sets in question answering in Table 8.
C.2 Question Asnwering
D.3 Blame Game Between Retriever and Language Model
In the question answering setting, the other hand, we give the whole question to the model and take the generated output as the answer for Entailment- Bank datasets. In Strategy QA QA experiments, we compute how often models rank the correct yes or no answer higher than the other.
Our analysis of the Contriever Flan-T5's re- sponses in a hypothetical scenario (i.e., the exact number of retrieval is known) is provided in Table 9. Results show that the retriever does not retrieve all

---

KN REALM
DPR FID
Contreevr ATLAS
Contreeer Faan T5
ErakleeerBank-Easy
Entallmen@ankk-ierd
60
40
40
Acqussy
20
EntellmenBerrk33
StratggrAA
100
100
80
80
60
10
25
25
2.
15
20
25
Figure 12: The accuracy of the retriever-auummented language models in the target ranking (LM) problem. The results show that Contriever Flan-T5 and &NN-LM perform as the best and the worst models in our LM experiments.
KNN-LLM (gold)
LN (all)
(gokg)
(ail)
(gold)
REALM (gold)
REALM (al)
DPR (gold)
FID (al)
DPR (gold)
FID
Contriever ATLAS (gold)
Contrever + ATLAS (all)
Contriever ATLAS (gold)
Contriever ATLAS (ail)
Contriever Flan-TS (gold)
Contriever + Flan T5 (all)
Conmiverr Flan-T5 (goic)
Controver Flan-55 (al)
90
Acauacy
Accuracy
80
20
of retrieved
Figure 13: The negative impact of additional distract- ing information on language models' performance in the LM task. The solid bars and the bars with patterns refer to the experiments with all the gold statements and all gold and distracting statements, respectively. be observed that providing language models with distracting statements hurts their performance.
Figure 14: Performance of language models on EB- Hard test set in the LM task coupled with imperfect retrievers. Coupling language models with imperfect retrievers (i.e., retrieving some distracting statements as well as some required ones) deteriorates the overall performance.
the required statements most of the time. Addition- ally, when the retriever performs perfectly, the LM responds to the question correctly more often.
examples of the Flan-T5 failures.
We also demonstrate some failure examples in each of the retrievers and language models in Ta- ble 11 where imperfect retrievers and LMs are com- bined. In this table, few true retrieved statements and the one that had to retrieved order for the model to solve the task correctly are highlighted in green and red, respectively. The true answer (or
E
Main Qualitative Results
Section 4.3.2, we explained that F1 scores of the models given only the oracle statements are lower than expected. Table 10 demonstrates a few more

---

KNN-M
REALM
DPR + FID
Contrivver ATLAS
Conthveer Flan T5
EntalmentBank (Easy)
Entalmentäank3
Token overlup F1 soure
40
10
15
25
2
10
15
25
statements
Figure 15: performance of the retriever-augmeented language models the QA task on EB-Easy and EB-3 datasets. In EB-3, we observe that models perform similarly to EB-Hard including both gold and distracting supporting information. Furthermore, we observe that in EB-Easy models perform similarly to our experimental settings on EB-Hard with only gold statements, as EB-Easy consists of only required statements with additional distracting information.
Language Modeling Accuracy EB-Hard
Token overlap Fl score
Accuracy SQA
EB-Fasy
EB-3
SQA
EB EB-Fasy
EB-Hard
EB-3
ANN-LM
57.92
62.74
61.39
25.06
kNN-LM
12.62
8.13
10.95
23.94
REALM
72.92
71.48
77.22
59.73
REALM
19.43
13.14
16.39
46.76
DPR Fi
DPR FiD
70.83
67.68
71.43
55.48
32.14
27.32
32.27
Contriever + ATLAS 81.25
Contriever + ATLAS
85.55
84.56
57.94
35.95
33.14
37.63
53.69
Contriever Flan-T5
Contriever + Flan-TS
92.92
91.63
87.64
70.92
45.33
39.68
42.29
67.34
Table 7: Experimental results of the best retriever- augmented models in LM on test sets. The two best models are highlighted in green. The results show that Flan-T5 and ATLAS are superior language modeling based the target ranking accuracy.
Table 8: Experimental results of the best retriever- augmented models in QA on test sets. The two best models are highlighted in green. The results show that Flan-TS and ATLAS are the superior models in all of the studied datasets.
Imperfect Retriever
Perfect Retriever
sequence of tokens leading the true answer) for each data sample's 's statements is marked in bold. These examples explain how not retrieving the nec- essary statements for reasoning or not reasoning over true statements can lead to incorrect answers.
LM is Correct LM is Incorrect
54 (19%)
17 (34%)
236 (81%)
33 (66%)
Table 9: An analysis on the blame game between the retriever and LM. This table shows the number of failures of Contriever Flan-T5 on EntailmentBank samples when the number of retrieved statements is known. Results highlight the noticeable improvement in the LM's performance (i.e., 34% compared 19%) when the retriever operates perfectly. Imperfect retriever stands cases where retriever misses at least one state- ment and Incorrect LM refers to cases where the predic- tion of the LM is marked as incorrect (i.e., samples with less than 0.5 Fl score).
Impact of the model size
In this paper, we compare various models from Flan-T5-small to Flan-T5-xxl with 80M to 1IB parameters, respectively. The performance of these Flan-T5 models accompanied with Contriever on EB-Fasy and StrategyQA QA datasets is presented in Figure 16. Experimental results show that larger models perform better in reasoning tasks.

---

Model's response semuntically correct.
(GPT-3) (Flan-T5-bese)
Flan-T-basse
Question: Many animals still being hunted for their fur. Because of this, many of these animals are danger of
Flan-TG-od
DSP (Flan-T5-od)
Statement:
hunting decreases animal population zero, then the animal will be extinct.
Expected answer: extinction
Response: extinct
everlsp
Model's response is incorrect wir the expected answer but is correct wrt commonsense.
Question: Plants use energy directly from sun.
What do they use the energy from the for? Statement:
Plants use energy the sun make
EB Easy
EB Hard
StraegoQA
food.
Expected answer: make food
Datasets
Figure 17: The token overlap recall score GPT-3.5 and Flan-TP T5 models using multihop DSP program. The results show that Flan-T5-xxl achieves high recall score with multihop retrieval which can be due to in- cluding all the retrieved statements in the response. Al- though getting high recall score, not desired behavior from language models.
Response:
to grow
Model's response is completely incorrect.
Question: Which step in the process of photosyn- thesis?
Statement:
Taking carbon dioxide is step in the photosynthesis process.
Expected answer:
plants taking in carhon dioxide
Response: releasing light
G Impact of multihop retrieval
Table 10: The predictions of Flan-Tl T5 when the oracle statement is provided. Statistics of correct and incorrect responses are shown in Figure 6.
This section reports more details about the stud- ied multihop retrieval approach (DSP). In multihop retrieval experiments, we retrieve 5 statements in each retrieval, with the same templates of the orig- inal DSP paper (Khattab et al., 2022). Due the generally longer context size in multihop retrieval setting and the Flan-T5's context window limita- tions, multihop and sertievee-hen--eeee experiments include two and five few-shot demonstrations in the prompt, respectively. Some of the qualitative ex- amples of the multihop retrieval approach of DSP using GPT-3.5 and T5-xl are illustrated in Table 12. Even though Flan-T5-xxl includes the correct answer tokens in its generated response, it can be observed that the subqueries generated by this model are sometimes nothing but paraphras- ing or repetition the original questions, while the goal of the multihop DSP program is to break down the problems into smaller subproblems. Moreover, the Flan-T5-xxI's responses usually include all the retrieved information, which is not desired. The recall score of the models using the multihop DSP approach is illustrated in Figure 17 which shows the relatively high recall score the larger Flan-T5 model due to the problem mentioned above.
(90M)
(250M)
Flan- 15 large (780M)
--- Flan-T5-xl (3B)
Flan-T5-od (118)
EntallmentBank-Easy
Token overlap score
40
2
10
20
StrategyQA
Accuracy
80
70
80
50
2
10
20
# of retrieved statements
Figure 16: Token overlap Fl score of various sizes Flan-T5 on EB-Easy and Strategy QA test sets based on the number of retrieved statements. The results demonstrate that larger models perform better in Fl scores. We use Contriever the retriever in all experiments.

---

Model
Query
Statements
Answer
DPR +
polar bear
Flan-T5
In a zoo located in a warm re- gion, what should be included in the polar bear exhibit?
+ an animal lives a certain environment then that animal usually requires that kind of environment. Polar bears live in cold environments.
DPR FiD
warm
Contriever
What keeps the Moon orbiting Earth?
Moons orbit planets.
elliptical
ATLAS
Gravity causes orhits.
Retriever's Fuilures
The robot will weigh less than earth but will have the [MASK].
As the force gravity decreases, the weight of the object will decrease.
ANN-LM
The gravitational force of planet does change the mass object
mars
Targets: mass vs mars A complete orbit of mercury
planet celestial body.
complete revolution orbit of planct around its star takes 1 one planctary year. One mercury year is about 88 earth days.
around the sun takes [MASK].
between
Targets: around 88 earth days vs between and 365
and 365
REALM
+ A new moon occurred on june 2.
If
occurred June
+ A moon phase occurs 28 days after the last time occurred.
2, when will the occur?
june
plus 28 equals 30,
What allows two students stand- ing ten feet apart to hear each other talk?
+ Talking when a human produces sound to com- municate.
DPR + Flan-T5
micro- phone
+ Sound can travll through air by vibrating air.
Which energy conversion hap-
person kind of animal.
When an animal moves, chemical energy is con-
pens when a person shivers the energy transferred to make the muscles and joints move?
DPR + FiD
verted mechanical energy.
shaking
+ Shivering kind shaking.
+ Shaking is kind of moving.
Wave energy from the ocean can he hamessed to power genera- tors to make electricity, Energy from occan tides can also he used to make electricity. How would you categorize these two sources of energy?
Language Model's Failures
+ Tidal energy means energy from occan tides.
Wave energy
+ Tidal energy a renewable resource.
+ Wave energy is a renewable resource.
Contriever ATLAS
+ Humans changing ecosystems usually has tive impact ecosystem organisms living in ecosystem.
Which changes will most likely have a negative effect on an ecosystem?
Humans changing ccosystems
+ Humans building roads in an ecnsystem causes that ecosystem to change.
The mass of earth causes the
mass a planet causes the pull gravity on that planet.
ANN-LM
pull of gravity on [MASK].
newton
Targets: earth vs newton Andy lives in southem hemi- sphere. What season does he most likely experience gust?
+ Earth a kind of planet.
+ Andy lives in southern hemisphere.
southem hemisphere
REALM
+ August is during the winter in the southern hemi- sphere.
Table II: A complete set of models' failure examples rooted in the retriever or language model modules. the retriever's failures section, few correctly retrieved statements the ones that had be retrieved order for the model to solve the task correctly are highlighted in green and red, respectively. The sequence of tokens leading to the true answer marked in bold.

---

Question
Statements
Answer
+ Skills are learned characteristics.
Which characteristic of cheetah more likely to learned rather than inherited?
+ Inherited characteristics are the opposite learned characteristics.
hunting strategies
+ Hunting is a kind of skill.
GPT-3.5
first subquery
learned characteristics of a choctah
[1] and [2] suggest that skills learned characteristics, while [3] suggests that hunting is kind of skill.
first rationale
second subquery
Is hunting learned inherited characteristic in cheetahs?
First, we need to understand definition each term. [T] and [2] suggest that skills are learned
second rationale
characteristics, while [3] suggests that hunting kind skill. Therefore, the answer must learned characteristic of a cheetah that is not hunting.
response
How to run
Flan-Te T3-xxl
first subquery
What is the most likely characteristic a cheetah be learned rather than inherited?
rationale
What the most likely characteristic a cheetah to learned rather inherited?
second su subquery
What is the most likely characteristic a cheetah to be learned rather than inherited?
second rationale
What is the most likely characteristic of cheetah to be learned rather than inherited?
hunting is kind of skill inherited characteristies opposite of learned characteristies hunting
response
Question
Statements
Answer
Pollination when pollinating animals, wind, or water
carry pollen from one flower another flower.
carry pollen for repro- duction.
A bee depends on cortain flowers for food. The flowers depend on the bee to
bee pollinating animal.
Pollination requires pollinating animals.
Plant reproduction requires pollination.
GPT-3.5
first subquery
What do boes cal?
A bee is pollinating animal. Pollination is when pollinating animals, wind, or water carry pollen from one flower to another flower. Plant reproduction requires pollination. Pollination
first rationale
requires pollinating animals.
second subquery
What is the relationship hetween hees and flowers?
First, we need to what pollination Pollination when an animal carries pollen from flower to another. This important because flowers need to be pollinated order to reproduce. Therefore. bee depends on flowers for food, and the flowers depend on the for pollination. pollinate
second rationale
response
Flan T5-xxl
first suhquery
What do flowers depend on hees for?
first rationale
What do flowers depend bees for?
Pollination when pollinating animals, wind, or water carry pollen from one flower to
second subquery
another flower. What do flowers depend on bees for?
second rationale
What do flowers depend on boes for?
Pollination is when pollinating animals, wind, water carry pollen from one flower to another flower.
response
Question
Statements
Answer inherited
A young boy and his brothers have green eyes. This trait hest described as
Lye color is inhertted characteristic.
Green is a kind of color.
GPT-3.5
subquery
What a trait?
first rationale
Fye color is an inherited characteristic. Green is a kind of color.
second subquery
What is the definition of "inherited characteristic""
We are looking for word that describes how the trait is passed down. This is inherited characteristic, it passed down from parents to children.
second rationale
response
inherited
Flan- Flan-T5-xxl
first subquery
What is the medical term for the color a person's 's eyes?
first rationale
What is the medical term for the color of person's eyes?
Eye color an inherited characteristic. What the medical for the color person's eyes?
second subquery
second rationale
What the medical term for the color a person's eyes?
Eye color is an inherited characteristic. Eye color is kind color. Eye color is inherited.
response
The medical term for the color of person's eyes is eye color. Eye color is an inherited
characteristic. Eye color is kind of color.
Table 12: Some examples of multihop question answering using the DSP approach with Contriever the retriever. In each sample, the generated subqueries. rationales, and final response are presented. The correct and incorrect answers included the generated tokens are highlighted green and red, respectively.


Reviews:
[{"review_id": "xVfdYS6PA7", "reviewer": "Reviewer_WssQ", "summary": "", "questions": "- What would be the most interesting findings from Section 4.3.3 and 4.3.5?\n\n- Is it possible to distinguish the retrieval error from the LM error? and how?\n\n- (Minor) In Table 3, the authors could consider a small-scale case study with human annotators to strengthen the claim.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your positive and thoughtful feedback, as well as your constructive concerns and questions. We are delighted that you found our paper:\n- **well-written** (also highlighted by Reviewer k18u) and **well-organized**,\n- providing **insightful findings** with examples, \n- **clearly demonstrating recent issues** on well-known frameworks of retriever-augmented language models (RALM),\n- a **concrete experimental setup** and **comprehensive analysis**, as also mentioned by Reviewers k18u and cXZR.\n\nWe are also delighted that other reviewers found the paper to be:\n- an interesting idea (Reviewer orzP), \n- a systematic (Reviewer k18u) and controlled (Reviewer cXZR) analysis of the recent and representative models (Reviewer orzP),\n- providing exciting findings (Reviewer cXZR).\n\nWe address your concerns and questions below:\n\n> Lack of novel insights\n \nIn contrast to the literature [1], we are the first to systematically study the shortcomings of several retrievers and LMs in similar experimental settings. We believe such systematicity is highly valuable in the current landscape of NLP where it is becoming hard to disentangle true strengths and weaknesses from confounding factors. While previous work on RALMs focused on QA datasets like Natural Questions and multihop like HotpotQA, we focus on a controlled setting using EntailmentBank where the knowledge statements and queries may not have surface-level lexical similarities. The reasoning connections are not readily obvious from the questions, distinguishing it from previous work. Here is an example (edited for simplicity)\n\n|| EntailmentBank | HotpotQA [2] |\n|---|---|---|\n| Question| What is Phobos? | The football manager who recruited David Beckham managed Manchester United during what timeframe? |\n|Gold knowledge| (a) Phobos orbits Mars, (b) Mars is a planet, (c\\) Moons orbit planets | (a) United manager Sir Alex Ferguson... Beckham was part of a group of young talents Ferguson brought into United in the 1990s.., (b) .. best known for managing Manchester United from 1986 to 2013.|\n\nHere, the question in EntailmentBank has no lexical similarities with required statements (b) and (c\\), and without a strong reasoning component, models would struggle to retrieve and reason upon such statements.\n\nMoreover, our paper had five insights (lines 79--94). Even if some of them are reconfirmations of existing insights, we believe this is a substantial contribution, all in one place. As you pointed in the strengths, these are *`insightful findings`*. Would you reconsider your comment about lack of novel insights in the reasons for rejection? We will add more discussion on the novelty in the paper.\n\n> \"Concerns about overclaims on Blame Game” and \"who bears more responsibility between the retriever and language model\" (also Question 2)\n\nWe thank you for raising this important question. We have provided scenarios that we thought were critical, i.e., retrieval-only failures (Sec 4.3.1), LM-only failures (Sec 4.3.2), and the realistic combination of them (Sec 4.3.3). If you mean pin-pointing precisely whose failure it is for a given example, yes, we provide additional results but some assumptions could be unrealistic. \n\nAssuming we know the exact number of statements (*k*) to be retrieved for a given input, we report the number of results with Contriever as retriever and Flan-T5 as LM (where samples with F1 score less than 0.5 are considered incorrect):\n\n|  |retrieval missing a statement | all required statements retrieved |\n| --- |---|---|\n| **LM is incorrect** | 236 | 33 |\n| **LM is correct** | 54 | 17 |\n\nHowever, assuming the exact number of statements *k* to be already known is a strong assumption. If we relax this assumption, then it becomes a complex problem, which is why in the combination section we provided all the ranges of 'k' (Fig 7).\n\nIn any case, you raise an interesting point. We are happy to include the above table for different RALMs if it is useful. Given this, would this change your mind about overclaim? Thank you!\n\n\n> Interesting findings from Sec 4.3.3 and 4.3.5 (Question 1)\n\nBy comparing the outcomes of Sec 4.3.2 (Fig 5) and Sec 4.3.3 (Fig 7), we illustrate the contributions of LMs and retrievers to the overall shortcomings. Moreover, Fig 7 uncovers additional interesting observations. Notably, the larger gap between the solid and dotted lines for Flan-T5, in contrast to Atlas, demonstrates that providing gold statements to Flan-T5 yields a more significant performance improvement (despite both using the same retriever and retrieved statements). The impact of a subpar retriever is also evident. While FiD and Atlas perform nearly equally well with gold statements, the performance of DPR+FiD notably lags behind Contriever+Atlas when given all (distracting + gold) statements.\n\nPrevious studies [3] show that multi-hop retrieve-and-read makes complex reasoning easier. In Sec 4.3.5, Fig 10 illustrates that DSP detrimentally affects Flan-T5's performance but enhances that of GPT-3. However, even with the improved F1 score for GPT-3 (light blue bar), it still falls short of the retrieve-then-read approach of Flan-T5-xxl (dark pink bar), which underscores the need for further improvement in multi-hop retrieve-and-read techniques.\n\n> A small-scale case study for Table 3 (Question 3)\n\nWe appreciate your constructive comment. Our **further analysis on 50 randomly sampled Flan-T5 responses** shows that 18% of the Flan-T5 responses are not grounded in the oracle statement, even though the oracle statement was relevant to the answer (case 6 and case 7). Additionally, 32% of the responses yield a low F1 score even though they are correct (case 2 and case 3, as also highlighted in [4, 5]). The detailed statistics from our analysis (table below) will be added to the paper to provide clarity to the community.\n\n|Case|Oracle statement is relevant to the answer|Response is grounded in the oracle statement|Correctness of the response|Proportion|\n|---|---|---|---|---|\n|1|+|+| completely (full F1)| 34% |\n|2|+|+| less or more elaborate response | 22% |\n|3|+|+| semantically correct | 10% |\n|4|+|+| incorrect wrt answer but correct wrt general knowledge | 2% |\n|5|+|+| incorrect | 10% |\n|6|+|-| incorrect wrt answer but correct wrt general knowledge | 10% |\n|7|+|-| incorrect | 8% |\n|8|-|-| incorrect | 4% |\n\nWe provide examples for cases 3, 6, and 7 in the following table:\n\n|| Case 3 | Case 6 | Case 7 |\n|--- |---|---|---|\n|Question | Many animals are still being hunted for their fur. Because of this, many of these animals are in danger of | Plants use energy directly from the sun. What do they use the energy from the sun for? | Which is a step in the process of photosynthesis? |\n|Oracle statement|If hunting decreases the animal population to zero, then the animal will be extinct.|Plants use energy from the sun to make food.|Taking in carbon dioxide is a step in the photosynthesis process.|\n|Answer|extinction|to make food|plants taking in carbon dioxide|\n|Response|extinct|to grow|releasing light|\n\nWe again thank you for your insightful comments which will improve the paper further. We intend to incorporate them into the paper. With these revisions, could you reassess your excitement of our work?\n\n### References\n[1] Liang et al, Better retrieval may not lead to better question answering, Arxiv 2022\n\n[2] Yang et al, HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, EMNLP 2018\n\n[3] Khattab et al, DEMONSTRATE–SEARCH–PREDICT: Composing retrieval and language models for knowledge-intensive NLP, Arxiv 2023\n\n[4] Kamalloo et al, Evaluating Open-Domain Question Answering in the Era of Large Language Models, ACL 2023\n\n[5] Adlakha et al, Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and -organized, supporting their claims with proper experimental results.\n\n2. Recent issues in combining retrieval systems with language models are clearly demonstrated in the concrete experimental setup.\n\n3. The authors thoroughly analyze the issues in each component and provide insightful findings with their examples.", "reasons_to_reject": "1. **Lack of novel findings**\nWhile the paper sheds light on the choice of existing well-known frameworks and offers valuable insights into augmenting pre-trained language models with retrievers, it falls short in terms of novel findings. It reconfirms the well-known issues in each component, the failure in retrieval or reasoning over multiple statements. It could have provided novel insights in sections for combining two components or multi-hop QA; however not observed.\n\n2. **Concerns about claims on \"Blame Game\"**\nUsage of the word \"*Blame Game*\" in the title could mislead the readers. It implicates the questions about who bears more responsibility between the retriever and language model, which is not presented in the paper. For claiming it, the quantitative analysis of retrieval and LM error or the presence of unique issues in combining them should be provided.", "questions_to_authors": "- What would be the most interesting findings from Section 4.3.3 and 4.3.5?\n\n- Is it possible to distinguish the retrieval error from the LM error? and how?\n\n- (Minor) In Table 3, the authors could consider a small-scale case study with human annotators to strengthen the claim.", "paper_topic_and_main_contributions": "This paper presents a comprehensive analysis of recent approaches for augmenting pre-trained language models with retrievers such as kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and FLAN-T5. The core investigations highlight three main findings: (1) the existing similarity scores are deemed limited in retrieving adequate evidence for reasoning, (2) language models often fail to perform reasoning even with the golden evidence, and (3) with imperfect retrievers, their performances are degraded. Additionally, the paper includes an extended analysis of multihop retrievers using different language models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In line 855, ‘findings’ was misspelled as ‘fndings’."}, "extra_text": {}}, {"review_id": "N5CJlxdCTb", "reviewer": "Reviewer_orzP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the positive and valuable feedback you have provided.  We are pleased that you find:\n- the idea of this paper to be **interesting**.\n- all the studied retriever-augmented language models (RALMs) to be **recent and representative**.\n- The findings to be **informative for other readers**. \n\nWe are also delighted that other reviewers found the paper:\n- well-written (Reviewers WssQ and k18u),\n- well-organized (Reviewer WssQ), \n- a systematic (Reviewer k18u), comprehensive (Reviewers WssQ, k18u, and cXZR), concrete (Reviewer WssQ), and controlled (Reviewer cXZR) analysis of the recent and well-known models (Reviewer WssQ),\n- providing exciting (Reviewer cXZR) and insightful (Reviewer WssQ) findings.\n\nWe engage with your concerns below:\n\n> Lack of finetuned experiments\n\nWe believe that the community is shifting towards using language models 'as is', in zero-shot or few-shot settings, without further finetuning. Thus, as indicated in the limitations section, our focus is on evaluating the performance of popular retrievers and LMs on datasets they have not encountered during training. To ensure that we do not underestimate the models due to the absence of fine-tuning, we not only compare models with each other but also evaluate each model against its **upper-bound performance** when provided with an oracle statement. \n\n> Some already discussed takeaways\n \nWe appreciate your reference to the papers. We find paper [1] intriguing and will include it in the final version. However, in contrast to [1], we are the first to systematically study the shortcomings of several retrievers and LMs in similar experimental settings. We believe such systematicity is highly valuable in the current landscape of NLP where it is becoming hard to disentangle true strengths and weaknesses from confounding factors. While previous work in RALMs focused on multihop QA datasets like HotpotQA, we focus on a controlled setting using EntailmentBank where the knowledge statements and queries may not have surface level lexical similarities. In our experiments, reasoning connections are not readily obvious from the questions, distinguishing it from previous work. Here is an example (edited for simplicity)\n\n|| EntailmentBank | HotpotQA [2] |\n|---|---|---|\n| Question| What is Phobos? | The football manager who recruited David Beckham managed Manchester United during what timeframe? |\n|Gold knowledge| (a) Phobos orbits Mars, (b) Mars is a planet, (c\\) Moons orbit planets | (a) United manager Sir Alex Ferguson... Beckham was part of a group of young talents Ferguson brought into United in the 1990s.., (b) .. best known for managing Manchester United from 1986 to 2013.|\n\nHere, the question in EntailmentBank has no lexical similarities with required statements (b) and (c\\), and without a strong reasoning component, models would struggle to retrieve and reason upon such statements.\n\nWhile [3] explores model scale across various model types and datasets with chain-of-though prompting,  this paper investigates the impact of model size on performance in a zero-shot setup with distracting information. Our analysis shows while larger models outperform the smaller ones, they still fall short of their upper-bound performance. Surprisingly, our analysis highlights that despite GPT-3.5 being over 10 times larger than Flan-T5-xxl, it fails to achieve a superior token overlap F1 score, whether with or without DSP.\n\nWe thank you for your constructive feedback. In addition to the above clarifications, we also intend to provide additional error analysis for the oracle experiment (upper-bound) and error attribution during the combination (please see response to Reviewer WssQ). Given these revisions, could you reconsider your evaluation of the excitement of our work?\n\n\n### References\n[1] Liang et al, Better retrieval may not lead to better question answering, Arxiv 2022\n\n[2] Yang et al, HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, EMNLP 2018\n\n[3] Wei et al, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - The idea of evaluating the retriever and reader of the language models on reasoning intensive tasks is interesting.\n - The authors evaluate 5 recent and representative retrieval augmented neural models, which is informative to other readers.", "reasons_to_reject": " - One major drawback is that all of the models are only evaluated without any fine-tuning. Although it is good to know the zero-shot performance, knowing their limitation after fine-tuning would be much more helpful. \n - Some takeaways are already discussed in previous literatures, and the results are reported for similar models, which makes this work less exciting. E.g.,\n   * For takeaways (2) and (4) above, similar results are reported in \"Better retrieval may not lead to better question answering\".\n   * For takeaway (5) above, \"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models\"", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper examines a few retrieval augmented language models' performance on reasoning intensive tasks: EntailmentBank and StrategyQA. The performance of both the retrievers and the readers are examined. 5 popular models are evaluated without fine-tuning on the EntailmentBank and StrategyQA datasets: kNN-LM, REALM, DPR + FiD, Contriever + ATLAS and Contriever + Flan-T5. \n\nThe main takeaways from the experiments are: (1) The retrievers of these models tend to fail to perfectly retrieve the necessary statements for the reasoning intensive tasks. (2) Even with the gold evidence, the neural readers could not perfectly answer the question. (3) When multi-step reasoning over the statements is needed to come up with the answer, different models perform differently (e.g., Flan-T5 has a better reasoning ability compared with others). (4) Bad retrieval quality can largely affect QA model's performance. (5) Large model usually has a better QA performance. (6) Multi-step retrieval is promising when combined with large language models. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "enW1xbYqlS", "reviewer": "Reviewer_k18u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback regarding your concerns. We are happy that you found our paper to be:\n- **well-written**, as also highlighted by Reviewer WssQ, \n- a **comprehensive analysis**, \n- with **systematic experiments**. \n\nWe are also thrilled that other reviewers found our paper:\n- well-organized (Reviewer WssQ), \n- with an interesting idea (Reviewer orzP), \n- comprehensive (Reviewer WssQ and cXZR), concrete (Reviewer WssQ), and controlled (Reviewer cXZR) analysis of the recent (Reviewer WssQ and orzP), well-known (Reviewer WssQ), and representative (Reviewer orzP) models,\n- with exciting (Reviewer cXZR) and insightful (Reviewer WssQ) findings.\n\nWe respectfully disagree with the reviewer's criticisms of our work and would encourage them to read the paper carefully and other reviewers' comments. \n\n> Lack of novel findings... evident that the limitations of RALMs stem from either the retriever or the LM\n\nCertainly, we acknowledge that the failures of RALMs are indeed attributed to either of the two modules. However, we are the first to systematically study the shortcomings of several retrievers and LMs in similar experimental settings. We believe such systematicity is highly valuable in the current landscape of NLP where it is becoming hard to disentangle true strengths and weaknesses from confounding factors.\n\nMoreover, our paper had five insights (lines 79–94), a substantial contribution with insightful findings, all in one place.\n\n> No novel concept to overcome the limitations of the multi-hop retrieve-and-read approach, which appears similar to the Demonstrate-Search-Predict (DSP)\n\nOur work precisely states that we use DSP in our experiments. In line 456, we say that “we explore the effectiveness of a strong multi-hop retrieve-and-read framework known as Demonstrate-Search-Predict (DSP)”, not any other similar approach. Through our analysis, we examine the shortcomings of DSP on LLMs and demonstrate that this method heavily relies on the LM's ability to generate subqueries. Surprisingly, even large models like Flan-T5-xxl struggle in this aspect.\n\nAdditionally, we highlight that while DSP enhances GPT-3.5's performance, it still falls behind the retrieve-then-read Flan-T5-xxl. This finding highlights the ongoing need for improvement in multi-hop retrieval methods.\n\n> Typo about ‘gravitational’ in Table 3\n \nNo, this is not a typo. We again encourage you to read the paper attentively. We intentionally highlight this example to demonstrate a scenario where the F1 score penalizes the models' correct response 'gravity.' \n\nWe kindly request you to reassess the work carefully in light of the clarifications we've provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, and the experiments are conducted systematically.", "reasons_to_reject": "[1] While the manuscript is well-written and the experiments are conducted systematically, the novelty of the findings may not be as significant as expected. It appears somewhat evident that the limitations of retriever-augmented language models stem from issues with either the retriever or the language models.\n[2] Additionally, the authors employ a multi-hop retrieve-and-read approach to address the limitations of retrieve-and-read, which appears similar to the Demonstrate-Search-Predict (DSP) method without introducing novel concepts to overcome the limitations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This manuscript provides a comprehensive analysis of the limitations of retriever-augmented language models for reasoning over retrieved information from both the retriever and language model perspectives. The authors demonstrate that the failure of retriever-augmented language models to reason effectively can be attributed to either the retriever's inability to find the necessary statements or the language model's difficulty in reasoning over them.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Regarding Table 3, I noticed that the expected answer 'gravitational' is not shown in the oracle statement."}, "extra_text": {}}, {"review_id": "IhTr9MyeHD", "reviewer": "Reviewer_cXZR", "summary": "", "questions": "A: Do you have a conjecture/hypothesis as to why LMs do not show perfect behavior when they have been provided with the oracle statement in context? I think adding this in the paper will strengthen it.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your comprehensive review and are grateful for your constructive comments. We are thrilled that you believe our paper provides:\n\n* **controlled evaluation** on an **understudied topic quite well**,\n* **thorough and comprehensive with results backing our claims**, as also emphasized by Reviewers WssQ and k18u,\n* **insightful findings**, as also highlighted by Reviewer WssQ.\n\nWe are also delighted that other reviewers found the paper to be:\n * well-written (Reviewers WssQ and k18u),\n * well-organized (Reviewer WssQ),\n * systematic (Reviewer k18u) and concrete (Reviewer WssQ) analysis of the recent (Reviewers WssQ and orzP), well-known (Reviewer WssQ), and representative (Reviewer orzP) models.\n\nWe engage with your questions and suggestions below:\n\n> Move dataset preparation from the Appendix to the main text\n\nThank you for pointing this out. We agree this is important and will move this to the main text in the camera-ready. We will also elaborate on the nature of distracting statements with examples. Most commonly, the distracting statements contain entities or relations mentioned in the input query which might make them deemed relevant but in actuality they are irrelevant.\n\n> Precise difference between the multi-hop method and the setting where ground-truth statements are provided to the model \n\nIn the multi-hop setting, the model has two advantages: 1) it can decompose the problem into smaller problems, allowing it to generate intermediate hypothesis statements that might be useful in the future; and 2) retrieves only relevant facts for each subproblem and incrementally build the answer using retrieved facts and intermediate hypotheses. Previous studies [1] show that intermediate steps make complex reasoning easier. Whereas in the ground-truth setting, the model has access to the gold statements all at once and then has to generate an answer. We will elaborate on the differences and advantages in the camera-ready.\n\n> Reasons LMs do not show perfect behavior with oracle statement\n\nThank you for raising this interesting question. We conducted **additional human analysis on 50 randomly sampled responses of Flan-T5** given the oracle statement (table below). We will add this analysis to the camera-ready since we believe it will be useful for the community and will strengthen our claims. As stated in Sec 4.3.2, upper-bound performance of the models is lower than expected because 1) \"F1 evaluation metric penalizes when there is a word mismatch between true and predicted answer even if the predicted answer is semantically the same\" (case 2 and case 3) which is also observed in [2, 3], and 2) sometimes a model like \"Flan-T5 uses its internal knowledge rather than the provided statement\" (case 6 and case 7).\n\n|Case|Oracle statement is relevant to the answer|Response is grounded in the oracle statement|Correctness of the response|Proportion|\n|---|---|---|---|---|\n|1|+|+| completely (full F1)| 34% |\n|2|+|+| less or more elaborate response | 22% |\n|3|+|+| semantically correct | 10% |\n|4|+|+| incorrect wrt answer but correct wrt general knowledge | 2% |\n|5|+|+| incorrect | 10% |\n|6|+|-| incorrect wrt answer but correct wrt general knowledge | 10% |\n|7|+|-| incorrect | 8% |\n|8|-|-| incorrect | 4% |\n\nWe provide examples for case 3 and case 6.\n|| Case 3 | Case 6 |\n|--- |---|---|\n|Question | Many animals are still being hunted for their fur. Because of this, many of these animals are in danger of | Plants use energy directly from the sun. What do they use the energy from the sun for? |\n|Oracle statement|If hunting decreases the animal population to zero, then the animal will be extinct.|Plants use energy from the sun to make food.|\n|Answer|extinction|to make food|\n|Response|extinct|to grow|\n\n\n> Missing References\n\nThank you for pointing out these papers, we will discuss them in the camera-ready.\n\nWe thank you again for your time and the positive assessment of our work. Given your assessment and these improvements, would you consider championing our paper for publication at this conference? We are happy to engage further.\n\n### References\n[1] Khattab et al, DEMONSTRATE–SEARCH–PREDICT: Composing retrieval and language models for knowledge-intensive NLP, Arxiv 2023\n\n[2] Kamalloo et al, Evaluating Open-Domain Question Answering in the Era of Large Language Models, ACL 2023\n\n[3] Adlakha et al, Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Controlled evaluation of retriever augmented LMs seems to be an understudied topic, and this paper does it quite well.\n- Experiments are thorough and comprehensive, and the results sufficiently support the conclusions.\n- The most exciting finding to me are both the fact that RA-LMs are unable to reason well even with ground truth statements, as well as the fact that the simple similarity based metric employed by retrievers is not well suited for picking out statements that aid in reasoning.\n", "reasons_to_reject": "I do not see any major reasons to reject this paper. \n\nHowever, I do want to highlight the fact that the main text of the paper is not completely sufficient to evaluate it. The method for creating statements with distraction is mentioned in the appendix, but is actually a critical component of the paper---it dictates how the retrievers will behave and has a causal implication for the results. Additionally, the way this method has been presented in the appendix is very condensed and lacks sufficient examples. I would strongly encourage the authors to consider this feedback, move the method to the main text, and present a proper characterization of the distraction statements. Furthermore, I found the description of the multi-hop retrieval to be insufficient--i.e., I am still unclear about the precise difference between this method and the setting where ground truth statements are provided to the model -- wouldn’t this only affect performance on questions that require multi-hop decomposition?", "questions_to_authors": "A: Do you have a conjecture/hypothesis as to why LMs do not show perfect behavior when they have been provided with the oracle statement in context? I think adding this in the paper will strengthen it.", "paper_topic_and_main_contributions": "This paper presents a comprehensive evaluation of reasoning in retriever augmented language models (RA-LMs) -- language models with a separate module to retrieve statements that are relevant and important to solve the reasoning problem. To this end, the authors use existing reasoning datasets that require multiple steps of reasoning (in the form of statements)---EntailmentBank and StrategyQA---and augment them with additional statements that operate as distraction. This way, they are able to perform controlled evaluation of both the retriever and the language modeling component of the RA-LM, and characterize their strengths and weaknesses. Findings from a number of experiments suggests that: (1) similarity-based retrieval mechanisms are fundamentally limited in selecting statements that are useful for precise reasoning; (2) LMs seem to struggle even when they are provided with noise-free, ground truth statements that are sufficient to solve the reasoning problem; (3) there is an interaction effect between the LM and the retriever component, where their overall performance---expectedly---drops when the retrieved statements are insufficient for reasoning. Finally, the authors study a method involving multi-hop retrieval in addition to the standard retrieval process, and find that that while it augments the performance of off-the-shelf non-retriever based LMs like GPT3.5, it is unable to generalize to relatively smaller models such as FLAN-T5-XXL. Overall, findings from this work can be used to develop improved RA-LMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The LM setup is very similar to the one in the COMPS and Surface Form competition papers, and these might be worth citing:\n\n**Surface form competition:** https://aclanthology.org/2021.emnlp-main.564/\n\n**COMPS:** https://aclanthology.org/2023.eacl-main.213/\n\nIn fact the COMPS paper also contains distracting knowledge statements, so it might also be relevant there.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 061: “combining the details” seems a bit vague -- I encourage you to rephrase this.\n\nI would recommend moving the “question” and “statement” parts above the Model and Answer part of the table in Table 3.\n\nLine 463: replace overlapping → overlap\n\nI would recommend converting Fig 6 into a paired line plot that can make the drop in performance when distraction is present more obvious.\n\nThis does not really impact my scores but the limitations do not read like limitations and instead as an additional contribution and discussion section of sorts. \n"}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive analysis of recent approaches for augmenting pre-trained language models with retrievers (kNN-LM, REALM, DPR + FiD, Contriever + ATLAS, and FLAN-T5) and why they fail in reasoning (EntailmentBank and StrategyQA). The main findings are: 
1. the existing similarity scores are inadequate  in retrieving adequate evidence for reasoning, 
2. LLMs often fail to perform reasoning even with the golden evidence, and 
3. with imperfect retrievers, their performances are degraded.
4. Larger LLMs have  better QA and Multi-step retrieval performances.

Strength:
1. The paper is well-written and well-executed.
2. The authors thoroughly analyze the issues in each component and provide insightful findings with their examples.

Weakness:
1. Lack of novel findings 
2. all of the models are only evaluated without any fine-tuning. Although it is good to know the zero-shot performance, knowing their limitation after fine-tuning would be much more helpful.
Decision: Accept-Findings