Internet-augmented language models through few-shot prompting for open-domain question answering
Authors: Angeliki Lazaridou
Elena Gribovskaya
Wojciech Jan Stokowiec
Nikolai Grigorev
Venue: Submitted to ICLR 2023 (2023)
Keywords: language models
few-shot prompting
retrieval-augmented
question answering

Abstract: In this work, we aim to capitalize on the unique few-shot capabilities of large-scale language models (LSLMs) to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric lan4 guage models (LMs), which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-time compute.

Under review as a conference paper at ICLR 2023
INTERNET-AUGMENTE. LANGUAGE MODELS THROUGH FEW-SHOW PROMPTING FOR OPEN-DOMAIN QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we aim to capitalize the unique few-shot capabilities of large-scale language models (LL.Ms) to overcome some of their challenges with respect grounding to factual and up-to-date information. Motivated by semi-parametric lan- guage models (LMs), which ground their decisions in external retrieved evidence, we few-shot prompting to learn condition LMs information returned from the web using Google Search, broad and constantly updated knowledge source. Our approach does not involve fine-tuning making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned the web surpass performance of closed-book models of similar, or even larger, model sizes in open- open-domain question answering. Finally, we find that increasing the inference- timeeeeeeemmeeeee compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings sug- gest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-tin serencce-tmeeeee compute.
learning additional parameters, thus
INTRODUCTION
Undoubtedly, large-scale language models (LLMMs) present breakthrough for language research, particularly for their state-of-the-arr language modeling results (Radford et al.) 2019 Rae et al. 2021) and impressive generative capabilities. Above all, increasing scale has made few-shot learning defining new paradigm for language models (LMs). Due to the versatility of prompting, these models can now be quickly adapted using only a handful of examples to perform tasks ranging from question answering and numeric reasoning to creative writing (Brown et al.) 2020). All these considerations place few-shot LLMs at an excellent position to be used as building blocks for open-ended and "in the wild" user interactions.
Despite these successes, few-shot LLMs still lack a key ingredient; they are susceptible to hallu- cinations (Maynez et al., (220)) caused by incorrect retrieval of knowledge stored in their weights or due to the model having incomplete or outdated knowledge. As for many user interactions we expect factuality play an important role, imperative find ways keep LLMs up-to-date and grounded to factual and new information as it becomes available. As the current trend sees the size of these models to continually grow, mitigating those issues should rely on flexible and robust approaches that can be easily transferred to different domains and tasks.
Here, we aim to capitalize on the unique benefits offered by pre-trained LLMs and propose to overcome some of their limitations by drawing ideas from semi comi-paraettric models (Khandelwal ct al. 2019 Guu et al. 2020 Lewis et al. 2020 Izacard Grave 2020) that ground their decisions in external retrieved evidence to reduce hallucinations and improve factuality (Shuster et al. 2021). Specifically, we use the Internet as a source of up-to-date knowledge, and rely on the powerful few-shot capabilities of these LLMs to learn how to it effectively for answering questions. Taking open-domain question answering as task where factual correctness is vital, we design a system that given question uses retrieval model retrieve relevant documents from the Internet. Then,

---

Under review as a conference paper at ICLR 2023
using few-shot learning we prompt the model to answer the question via conditioning the retrieved documents, without the need to tine-tune or learn extra parameters.
As retrieval system we use search engine- in particular Google Search allowing to treat the whole web as a knowledge source. While Wikipedia has been the dominant knowledge source driving progress on multitude of tasks, given the current progress and the quest towards more complex interactions, there has never been a better time to widen their scope, embracing the opportunities working with the whole web, such considering wider range of topics and views, as well as the many challenges, such as working with more noisy and potentially uncurated and unsafe text in the wild. Indeed, there momentum building up in breaking away from Wikipedia-onle research (Komeili et al.) [021] Nakano et al. 2021 Piktus et al. [2021] Thoppilan et al. (022).
To test the effectiveness of equipping LLMs with Internet search on open-domain question answering. we use a mix of single-hop and multi-hop, language generation and classification tasks. We find that our biggest LLMs benefit from conditioning on the web through few-shot prompting. For the language generation tasks, we see a relative performance increase of 15%-30% over the commonly used closed-book few-shot approach, while also making up performance- wise for LLMs with smaller size. Surprisingly, we find that our method achieves gains, albeit smaller, even on complex multi-hop questions, despite the fact that these questions suffer from higher retrieval errors. While perhaps the mainstream view places scaling models' parameters as the primary way increase their few-shot performance, our results add to the stream of work that emphasizes instead better use of the models' powerful prompting abilities (Rubin et al.) 2021] Liu al. 2021a). As such, our approach presents a lightweight method applicable to virtually any pre-trained LM without the need for fine-tuning or adding extra learnable parameters. Finally, increasing the inference-time compute of models via sampling multiple answers and reranking using scores computed from the same LLMs not only adds further performance gains, but also alleviates generally decreased performance of smaller few-shot LMs, partly closing their performance gap with larger models.
All in all, our findings hint at the possibility of slowing down the race towards the biggest model and instead shifting the attention to more targeted and effective use of models' few-shot capabilities in combination with increasing inference-tine compute, generally more scalable approach.
RELATED WORK
have been recently gaining momentum (Khandelwal et al. 2019 Guu al. 2020 Yogatama et al. 2021 Borgeaud al. 2021 extending monolithic parametric models with information from a knowledge source. This process facilitates overcoming distribution shift (e.g., domain or temporal) in flexible way by simply updating the external knowledge. When applied to question answering tasks (Lewis et al. 2020: Izacard & Gravel 2020; [achan et al. [021]. they surpass performance of parametric-only models they are able to efficiently handle an increasing number of retrieved passages and ground their predictions into additional information, thus reducing hallucinations and improving factuality. However, to be faithful to their input, these models need to be trained (or fine-tuned) to attend to the additional input. In contrast, our work pushes the limits of few-shot prompting as a way to learn to condition on external evidence with no additional training required, thus making it applicable to virtually any pre-trained LM.
Semi- Semi-parametric language models
Web as knowledge source Open-domain question answering traditionally has been studying care- fully constructed benchmarks, where answerability of questions from Wikipedia has been confirmed through annotations. Recently a new trend emerged - using the whole web as knowledge source to support more varied and rich interactions, (Augenstein et al. (2019) and Fan et al. (2020) make use Aateractions,Auugnsteennn
of web data through commercial search engines as part of building more diverse datasets for fact- checking. On the other hand, Piktus et al. (202T) find that considering the web as a retrieval source brings material gains to knowledge intensive tasks, despite any difficulties with building search index from an order of magnitude more (noisy) data than Wikipedia. To avoid similar challenges with building and maintaining a search index, recent work that aims in improving factuality in user interactions adopts the use of commercial search engines as building block for their systems (Komeili et al. 2021 Nakano et al. 2021: Thoppilan et al. [2022] Menick et al., (022). Similar to us, Nakano et al. 2222 analyze benefits of increasing compute at inference time. However, unlike us, they either target open-ended dialogue interactions (Komeili et al. 2021 Thoppilan et al. 2022 or focus on
2

---

Under review as a conference paper at ICLR 2023
optimizing performance using more intensive techniques like fine tin-itniig and reinforcement learn- ing (Nakano et al., 2021 Menick et al. (022). In our work, we take a more light-weight approach without introducing learnable parameters. We push the limits of few-shot prompting and emphasize the need to establish strong baselines, aiming at gaining insights into the strengths and weaknesses of this generally applicable, due to its simplicity, approach.
3 FEW-SHOT PROMPTING FOR INTERNET-AUGMEN
LANGUAGE MODELS
In this section. we describe our approach for improving the performance of pre-trained LMs in the task of open-domain question answering. Specifically, we propose use few-shot prompting as flexible and robust way to condition any pre-trained L.LLM on external evidence, allowing for better grounding to factual and up-to-date information. Our approach consists of 3 steps (see Appendix [A.] for illustration of the method). First, given question we retrieve set of relevant documents from the web using a search engine ($3.I). We then use the retrieved evidence to condition the LM through fhw-shot prompting (.2)). Finally, we generate multiple candidate answers from each evidence and, to select the best answer, rerank them using scores computed using the same LM ($.3))
3.1 RETRIEVE: GOOGLE SEARCH FOR DOCUMENT RETRIEVAL
For question y, need set of relevant documents D which would allow us to extend the model's knowledge to factual and (potentially) new information not already present its weights. Towards more realistic and open-ended user interactions, retrieve documents using off-the-shell search engine, i.c., Google Search. Specifically, we use each question verbatim as query and issuc a call to Google Search via the Google Search API For each question, retrieve the top-20 urls and parse their HTML conten to extract clean text, resulting in a set of documents per question qf]
As documents in D originate from news Wikipedia articles whole books, they tend to be long, with an average length of 2,056 words. As this exceeds the input sequence length of models, condition the model shorter excerpts extracted from the documents in D. Specifically, we first chunk all documents into paragraphs of 6 sentences. We then embed and the paragraphs using TF-IDF and using cosine similarity we produce (ranked) list of evidence paragraphs P. thus only using in the prompt smaller, more relevant parts of the full documents.
Overall, using Google Search allows to tap into the diverse and ever-growing content on the web, instead of being confined on the information found in the static snapshots of the curated content on Wikipedia, commonly used knowledge source question answering tasks. addition, we found that Google Search and the web offers also practical performance gains and is on-par (even sometimes outperforming) current Wikipedia- and comparisons the BEIR benchmark in Appendix A.3) We discuss broader limitations and opportunities using search engines Section
state-of-the art dense retrievers (see Section
3.2 PROMPT: FEW-SHOT PROMPTING FOR CONDITIONING ON EVIDENCE
Given a question q and set of retrieved paragraphs P. we use few-shot prompting to condition pre-trained LMs on the paragraphs. This is done by taking the conventional k-shot prompting for (closed-book) QA, that only considers tuples of (questions, answers), and extending it with an evidence paragraph, resulting a prompt of the form
Evidence:
Question:
Answer:
In all experiments set k = 15. In Section we give details on how we obtain the (evidence, question, answer) triplets to populate the prompi.
https:/ /tt/////vvee/eee gogeerssgooll..om http:/ccccccccooooeeee cust.mm-search search Collecting evidence for 20k questions would incur costs of few hundred dollars.
We access and use for research and publication purposes search corpus comprising text taken from
open web, created in accordance with generally agreed standards for inclusion/exclusion of content.
"T prevent data leakage, filter urls of the official dataset websites from the returned urls.
3

---

Under review as a conference paper at ICLR 2023
While experimenting with prompts, we found that swapping the question with the evidence, thus increasing the distance between questions and answers. yielded consistently lower results across all our datasets. We believe this is another manifestation of LMs' struggles to incorporate information from longer contexts; further performance increase could be achieved by selecting in-context in-contextexamples examples for the prompt retrieved based similarity the question being answered (Liu al. 2021b).
3.3 RERANK: INCREASING INFERENCE TIME COMPUTE VIA ANSWER RERANKING
Increasing compute of models via scaling the number of parameters, hence increasing truining-time compute, has been shown to improve performance of models in various few-shot tasks (Brown) ctt al. 2020; Rac et al, 202I). D. Here, we ask whether similar gains can be obtained when scaling the inference-tine time compute. through sampling multiple answers from the model, which we then rerank using different probabilistic factorizations of the question answering task as scoring functions.
Specifically, P contains retrieved paragraphs ordered via their similarity to question q. select the top n = 50, use each one separately to prompt the model and produce multiple candidate answers for each paragraph (for classification tasks we produce distribution over all class labels for each paragraph). Overall, this allows us to consider a larger number of possible answers candidates while overcoming potential retrieval emrors considering more paragraphs increases retrieval recall.
Given an answer a; for a question q conditioned on each of the n retrieved paragraph ps, we consider the following ways for estimating the answer probability: (i) direct inference, picks an answer that frar (pxl 4).p(a; 9, pz), referred to as RAG here and inspired from
maximizes p(a q)
the model of Lewis et al. (2020) (ii) Noisy channel inference, picks an answer that maximizes
p(a,,0 q pp)) pla o.. ppaa, Ps) Echihabi & Marcu 2003 Lewis & Fan 2019 (iii) Product-of-
pc)
Experts (PoE), which combines all probabilities used above, in addition to p(p:
Pre-trained LSLMs scorers All conditional probabilities computed using LMs that k-shot prompt for producing the respective distributions (A- = 15) (see Appendix [A.7 / for example prompts). In this way, we turn LMs into models of arbitrary probabilities, which the best of our knowledge is something not explored before. Exception to this is Pe (p: ()) that is computed as the normalized cosine similarities between the TF-IDF passage and question representations as passages can be long, getting reliable estimates of p(p; q) using prompted LMs was challenging.
4 EXPERIMENTAL SETUP
Datasets We use 3 question answering datasets: the single-hop NQ (Kwiatkowski et al. 2019). and the multi-hop HOTPOTQA (Yang al. 2018) and STRATEGYQA (Geva et al. [021): and fact-checking multi-hop dataset FEVER (Thome et al. 2018). We select these datasets as they allow us to consider mixture of language generation (NQ. HOTPOTQA) and classification (2-way for STRATEGYQA, 3-way for FEVER) tasks, as well as single- and multi-hop questions. We represent all items in the datasets tuples (4, A, G), where q is the question, A is the the set of possible answers (or a single class label for the classification datasets), and G is the set of gold evidence documents provided by the dataset. For the sew- learning. we use the prompt format presented in Section 3.22 and create dataset-specific 15-shot prompts (for each of the datasets, we use the same set of questions for the few-shot examples), for total of 4 prompts (for computing different scoring probabilities), populating them with the necessary (evidence, question, answer) triplets from each dataset As evidence, we use the gold evidence documents G.
Evaluation metries To evaluate the performance on these tasks, we report exact match for gen- cration tasks and accuracy for classification tasks. Moreover, to better understand the interaction between retrieval and subsequent QA performance, we introduce a retrieval score. For generation tasks, we calculate answer recall in the conditioning evidence paragraphs P. For classification tasks, since the answer takes the form of a class label, we instead compute the normalized word overlap (excluding stopwords) between the gold paragraph G and each of the conditioning paragraphs in P. and report the maximum word overlap achieved among the paragraphs P.
"The interpolation weights of probabilities are optimized on held-out set of 10% of data.
"See Appendix for the prompts we used for each dataset.
4

---

Under review as a conference paper at ICLR 2023
Language Models All experiments in this work use the GOPHER LM of 280 billion parameters (Rae) et al. 2.2I). Alongside this model, and in order and to answer questions regarding scaling, we also smaller models of the same family abbreviated by their number of parameters. i.e., 44M, 1 117M, 400M, 1B and 7B. Besides the number of parameters, the models also differ in the input sequence length: 2048 for 7B and 280R, and 1024 for 44M, 117M, 400M, and B.. All models were trained on MassiveTexi for 300 billion tokens. For generation tasks, we use nucleus decoding, with parameters 0.8 for the probability cut-off and temperature of
Open-domain question answering models Here, we describe the open-domain question answering systems we build, all based on LMs presented above. We first describe our open-book models (referred to as OB), which condition the provided evidence to generate answer. OB Google will use the Google retrieved paragraphs. For each question g we will generate 4 candidate answers a' for each of the 50 paragraphs p. for a total of 200 answers. We will then select the best answer a" as arg m f(a', ....).)), where f is pre-defined scoring function. In its basic form, OB will use scoring function only the probability returned by the question answering model. Models B Channel ,OB
Google will use the noisy channel, direct inference and PoE factorizations, introduced in Section 3.3 Moreover, better assess the performance of few-shot prompting, we design a model that assumes an oracle retrieval system - BB conditions on gold evidence passages for each question provided by the datasets.
Google and OB
Finally, as a baseline model, we will use a pre-trained LM without evidence, prompting it with (question, answer) tuples. This the conventional way found in the literature of few-shot language models (Brown et al., (020) for performing open open-domain question answering. These models are usually referred as closed-book, as they do not use any knowledge source (unlike open-book), but solely rely on the knowledge encoded in their weights. We refer to these models as CB.
To fairly compare the different models, we need account for the fact that G searches for the right answer in big pool of candidates generated by conditioning on multiple paragraphs; for both CB and OB Gold we sample 200 answers, and select the one with the highest probability. 6
Gold
5 RESULTS
5.1
CONDITIONING A LARGE- SCALE LANGUAGE MODEL ON GOOGLE SEARCH RESULTS
#sampled answers: 1
#sampled answers: 200
Retrieval Pefformance500
Dataset
SOTA
CB
OB
to remnking Google
CB
OB
Google
PrE OB
NQ
55.9*
21.7
23.1
25,8
61.7
32.7
38.4
85.0
HOTPOTQA
65.2
20.7
24.5
21.2
54.8
26.3
30.3
55.5
FEVER
73.2"
44.5
52.2
44.5
66.6"
52.0
57.2
43.3
STRATEGYQA
63.6
61.0
61.1
61.0
80.4
64.6
66.2
34.9
Fajoik
Note that this work use protuhilitios of class lihels directly, wheress in these protubiliiies were used is features for classification model hased on multi-class logistic regression.
b00|
YYnng ct 2018
Kruengtrai ct
b021
Clevs et at
[002
ma
202
Table 1: Results on 4 question answering datasets using the GOPHER-2800 model.
We assess the effectiveness of method for improving the performance of pre-trained LMs open- domain question answering. We start with Gopher-280B, the best LM among the ones considered in this work; we test whether using few-shot prompting evidence (i.e., O improves the performance of Gopher-280B over its closed- book version (i.e., CB), which uses knowledge solely in its weights. Table presents the results on our datasets.
way condition on external retrieved
Overall. we find that, indeed, conditioning Gopher-280B improved performance on all datasets, despite the fact that both Gopher-2800 open- and closed- models use the same underlying LM. We see stronger performance over the closed-book version for the generation tasks, with the relative improvements reaching 30% on the NQ dataset. For NQ and HOTPOTQA, stronger performance is driven by higher retrieval recall and strong extractive
the Google Search results leads to
PPrrformance platcaued at 50 samples- - sampling many answers with the same prompt decreases diversity.
5

---

Under review as a conference paper at ICLR 2023
behaviour of the model, very frequently producing answer present in the conditioning evidence, a welcomed feature for extractive tasks (i.e., generated answer is present in evidence in 89.4% and 70.5% cases, for NQ and HOTPOTQA respectively). We also see gains, albeit smaller in scale, for the more reasoning classification tasks, indicating that few-shot prompting with retrieved evidence is beneficial over and above its extractive power. Appendix [A.8 presents a manual error analysis.
Retrieval performance Turning to the retrieval performance achieved by Google Search (see that the effectiveness of our approach of using
Retrieval performance column in Table D the question verbatim as search query heavily depends on the complexity of the questions in each dataset. Compare for example a typical question in the single-hop NQ "How many episodes in season 2 breaking bad?", where recall @ 50 reaches 85%, with one from HOTPOTQA "What is the relationship of Yeshahework Yilma's 's mother to the man who was Ethiopia's 's emperor from 1930 to 1974", where recall performance drops to 55.5%. Indeed, performance on multi-hop datasets (HOTPOTQA and STRATEGY
generally worse than in the single-hop ones (NQ and FEVER). In fact, STRATEGY QA sees the smallest relative increase in performance with respect to the closed-boot version. Questions in this dataset, albeit somewhat artificial, are inventive and involve linking non- trivial, normally unrelated facts (e.g., "Could a sloth hypothetically watch an entire episode of Scrubs underwater?"). stress- -esting performance of retrieval models and calling for more sophisticated learning to search approaches (Adolphs et al. 2021; Komeili et al.) 2021 Nakano et al. 2021).
Compared to current Wikipedia- baseed state-of- of the-art models, which require training on the specific datasets, we find that the generic Google Search retrieval outperforms DPR (Karpukhin et al. 2020) on NQ (DPR recall@50 50 84%), while is only marginally outperformed by MDR (Xiong et al. 2020 on HOTPOTQA (MDR recall @ 20 52.1 % zero-shot retrieval, capable of generalizing on different tasks and knowledge corpus. Finally, our probabilistic reranking further improves performance: across all datasets, OB outperforms
recall @ 20 50.1 1%). Google Search acts here
consistently the simpler Google widening the gap between closed-book and Google-conditionee model. We provide more in-depth discussion of reranking results in Section 5.2
5.2 ABLATIONS
Effect of Reranking In Table (see columns #sampled answers:)), we ablate the use of ranking- even conditioning on a single Google evidence can bring gains over the closed-book model, evident by the higher performance achieved by O reranking compared to CB.
Google
Effect of different scoring functions Figure
70
presents a comparison of the 4 different scor-
p(alq.p)
ing functions introduced in Section 3.3 Note
60
RAG
Noisy Channel
that, besides p(alg. p). which is computed us- ing G8p, the remaining probabilities used to compute the 3 scores, i.c., PoE, RAG
perfomace
50-
PoE
40
and Noisy Channel, are obtained by few-shot prompting the smaller 7B model. We find that that reranking with factorizations that consider scores beyond just the answer probability im- prove performance across all datasets, despite the fact that we do not train specialized models for deriving the extra scores but rather use an order of magnitdee smaller prompted LMs.
30-
20
HotPotOA
FEVER
StrategyOA
Figure 1: Question answering performance on all 4 datasets using Gopher-280B BB model in combination with different answer scoring func- tions for reranking the answers generated for each of the top 50 Google retrieved paragraphs.
Looking at the individual scores, we find p(a 4.P%) and p(g Pt; a) to be most informa- tive - this last probability captures how well the model can "explain" the question given retrieved evidence and an answer. In fact, as also show in Appendix [.2]] this score could be reliably used for reranking passages that are more predictive of the final answer. On the other hand, p(p: 9, a). while being correlated to p(q pr. a), has a higher variance (likely due to varying length of the passages) and is much less informative overall. Among the three factorizations that consider, the lowest performance is consistently observed for the RAG-inspired factorization. We attribute this to the way the passage relevance is derived: do not use p(p; lq, a) from the model, we instead rely

---

Under review as a conference paper at ICLR 2023
on normalized cosine similarities between TF-IDF vectors which, as show in Appendix A.2 more reliable than the passage probabilities from the LM, but less accurate than p(g Pi. a).
While more elaborate scores able to reduce the gap with the in-domain fine-tuned models (compare column SOTA with column Google in Table I). line with previous work (Wei et al. 2021 we find that few-shot prompting still generally lags behind the specialist models. However, this should be considered in perspective of how generalizeable our setup is diverse set of questions, from single-hop to multi-hop to fact-checking, turning any LM to a retrieval-augmentee LMM
Oracle retrieval The results of G using gold evidence, hence assuming oracle retrieval, can be treated as upper-bound. Results suggest that there is room for improvement relying more on the conditioning evidence. We envision several directions: if we remain within the few-shot paradigm. more accurate prompt optimization, like in-context example selection, can further boost results (Liu et al. 1b2 while, constrained decoding (Post& & Vilar [0188] Lu et al., [021] can be way to condition the model at inference-tima timee by explicitly grounding an answer the provided evidence.
5.3 SCALING ANALYSIS OF OPEN- AND CLOSED-BOL BOOK MODELS
So far, we observed that conditioning 280 billion LM on retrieved evidence from Google Search resulted in improved performance over the closed-boot version of the same LM. But are these gains only confined on the (very) large- scale regime or could this method be used to also boost performance of smaller models? To
07
0.40
HotFotGA
NetFatgg-Googe
0.75
NO
ME--Google
0.50
Exact Match
0.25
Accuracy
answer this, we compute the open- and closed- book performance of additional models con- taining 44m, 1 117M. 400M, 1B and 7B parame- ters. We follow the same approach as before; we use k-shot learning prompt for the open-book models using Google Search results and k-shot learning prompt for (question, answer) for the closed-book models, with k = 15. In Fig- ure 2 we present results function of models' parameters in millions, for open-book (in solid lines) and closed-book models (dashed lines) models. We report accuracy for the classifica- tion tasks (left figure) and exact match for the generation tasks (right figure).
0.20
0.15
TEVTR
TEVER-GGoggle
0.3
Random way
StratgyAA
StrategyOA Google
2-way
10 10 10th 10
10" 15" 100" 10
Million Parameters
Million Parameters
Figure Scaling analysis of open-book (solid lines) and closed-book (dashed lines) question an- swering models using LMs of varying sizes rang- ing from 44M to 280B parameters.
Conditioning smaller models on retrieved evidence particularly beneficial for the generation tasks, which given reliable evidence can be more extractive in nature. In fact, making use of external evidence does not simply improve the factuality of the models through grounding to external sources, but in many cases results in smaller open-book models outperforming bigger closed for both NQ (in green lines) and HOTPOTQA (in purple lines) the open-book 7B model is on par with the several times larger closed-book book 280B model and for NQ the 1B is only 4% worse than the the 200x larger closed-bovd -ook 280B model. Despite the simplicity of the approach and the potential errors introduced by the retrieval method, conditioning through few-shot prompting can, some extent, make up for a smaller model size.
models:
we also see that for the more reasoning tasks, i.e., FEVER (in black lines) and STRATEGYQA (in blue lines). the gains manifest mostly in the 7B models, though are smaller. For the (relatively) smaller models, their closed-book, performance fluctuates around the random baseline. As such, grounding to factual information is perhaps secondary concern in those cases; retrieval can boost reasoning capabilities for models with pre-existing good reasoning skills (as in the case of 280B), but cannot compensate for general lack of reasoning abilities.
However,
7

---

Under review as a conference paper at ICLR 2023
5.4 INCREASING INFERENCE- TIME COMPUTE
The main driver of performance improvements of few-shot LMs has been scaling their model size. Increasing training-time compute is thus spent in injecting (and potentially memorizing) Internet-scale data. Here, we put forward different hypothesis; we posit that we can improve performance of smaller LMs by giving them direct access to the Internet, thus freeing training-time compute which could then be spent to increasing their inference- timee compute While there are many ways to achieve that, here we choose to increase models' compute via using multiple Google retrieved paragraphs to sample candidate answers for each and then reranking those answers using the functions introduced in Section[..3 We focus on the biggest models used in this study, comparing the open-book version of IB and 7B with the closed-book version of Gopher-280B. We conduct this study on NQ
Figure Heft presents exact match performance on NQ as function of number of paragraphs (and hence candidate answers we are sampling and reranking). We see that considering more than the top-1 paragraph improves performance of models. Moreover, as evident by the slight upward slops, considering gradually more para- graphs benefits those scoring functions that in- corporate some notion of paragraph "goodness", ie., RAG (in purple lines) and PoE (in green lines) and Noisy Channel (in orange lines). In contrary, reranking only using the probability p(alg,p) of the answer under the model (in blue lines) results in decreased performance after a certain point. This is potentially due to some pathological behaviour of the model, e.g., as- signing high probability to candidate answers that exist in the conditioning evidence but are otherwise wrong. This is consistent with find- ings in Machine Translation which find that reranking only using the forward probability gives worse results as the number of candidates increases beyond a certain threshold (Stahlberg & Byrne/ 2019).
32.5
30.0
27.5
Exact Match
25.0
22.5
70
p(alcus) 18
20.0
((NO.8) 7B
Nozy Channel
Nosy Channel 78
15.0
CR 2008
25 35 45 Number Paragraphs
13 15
1042
1042
1014
FLOPs
Figure 3: Scaling analysis on the NQ dataset: ex- act match as a function of number of conditioned paragraphs (left) and FLOPs (right).
More interestingly, we find that using as little as 5 paragraphs from Google Search, in conjunction with the 7B model (in dashed lines for the different scoring functions) surpasses the performance of closed- book Gopher-280B model (horizontal black line), suggesting that searching the Internet is worth more than 273 billion parameters. Finally, Figure right presents a similar plot, but as a function of FLOPs; this accounts for additional factors, top of the number of Google retrieved paragraphs, like models' size and compute spent to calculate the scores for each of the scoring functions. Closed-book Gopher-280B spends compute for implicitly combining retrieving the facts that are memorized in its weights, and then reason over them. By explicitly outsourcing the memorization of facts and their retrieval Google Search, the same inference-time compute can now be spent more effectively in reranking more samples with a smaller model. As such, for the same or less inference-time compute, reranking models achieve better performance, as indicated by all the lines being to the left of the blue dot in the horizontal dashed line, i.c., the FLOPs of the closed-boot Gopher-280B.
5.5 KREPING QA MODEL UP-TO-DATE
We now test whether using a commercial engine as a source of up- to-date information about the world can help stale models answer questions about new events. Since Gopher-280B was trained with data up to (and including) November 2020, questions about facts beyond that date would not have been seen in its training data. To derive such questions, we use the SituatedQA dataset (Zhang & Choil 202T) which contains questions grounded different dates. Table presents the exact match results on the complete development set of
Questions
All
Post-2020
CB
26.3
15.8
OB
OB
Google
28.1
22.4
Table 2: Exact Match perfor-
mance on SituatedQA.
Khandelwal et al. 2019) also find that training smaller models with large datastores surpasses perplexity performance of bigger language models.
8

---

Under review as a conference paper at ICLR 2023
questions (all) also create smaller subset of 80 questions about facts in 2021 and beyond to test adaptation to new information (post-2020).
The higher performance of OB Google indicates that using few-shot prompting as a way to condition on evidence is an effective way of incorporating truly new information into the QA system. However, despite having access updated information, the performance of OB on post-2020 questions is substantially lower than the performance the complete set of questions, suggesting that conflicting parametric (i.e., in language model) and contextual (i.e., in retrieved evidence) knowledge poses challenges for retrieval-based models (Longpre et al. 2021
5.6 REPRODUUCIBILIIY
We take steps to reproduce the proposed technique using publicly available material; Appendix A.44 presents results using our prompts and data in conjunction with the OPT-6B model (Zhang et al. 2022) and Appendix A.3 presents results using our language model and prompts in conjunction to Wikipedia- DPR evidence (Izacard & Grave 2020).
6 DISCUSSION
Towards more open-ended and "in the wild" user interactions, in this work we presented straightfor- ward method targeted to alleviate some of the challenges faced by LSLMs with respect to grounding to factual and up-to-date information. The core of our method lies in combining the powerful few-shot abilities of LSLLSs with a state-of-the-an retrieval model, i.e., Google Search, for access to broad and constantly updated knowledge source, the whole web. We applied our method on open-domain question answering and found that, despite its simplicity, using few-shot prompting to condition models on the web provides an effective approach for increasing models' factuality. Improvements were not just confined to the largest LMs: we saw increases in performance across the board of model sizes, with smaller open-book models often surpassing performance of bigger few-shot closed- models. Further gains were achieved when increasing inference-ting compute via using multiple retrieved evidences to generate multiple answers followed by reranking stage that uses scores computed by the same LMs. Our approach offers a simple way to turn virtually any pre-trained LM retrieval-augmented LM model without the need for fine-tuning or adding learnable parameters.
Mainstream view places scaling LMs' parameters as the main way to improve their few-shot per- formance. However, significant gains can be achieved by inference-type interventions, i.e., more targeted use of few-shot abilities and increase of inference-time compute. This may slow down the race towards the biggest LM, shifting attention towards finding more effective ways to use models.
Limitations
Despite their progress, LLMs are still sometimes outperformed by fine-tuned. and even smaller, models (Wei et all] (021). While our targeted interventions were successful in closing this gap on open- open-domain question answering, we are still behind in-domain fine-tuned models. For reasoning tasks, retrieval was able to improve only the largest amongst the considered models. Moreover, while we have considered a wide-range of diverse question answering datasets, our current experiments only capture a small fraction of (simple) user interactions where factuality plays a crucial role.
Finally, in an attempt to work towards open-ended interactions and improve our system's 's performance, we used commercial search engine as a retrieval model, allowing us to work with the whole web as a knowledge source. Since we are not confined to working only with the curated and "sanitized" Wikipedia, we anticipate number of safety issues arise, including misinformation. harmful content. While we have relied on the underlying safety nets of the search engines we use, more work should be put in place scoping out and better understanding the risks and, most importantly, providing effective ways to mitigate those. As working with the whole Web gains more momentum, we expect see more work that surfaces and tackles these points. Another potential concern reproducibility of the research results given that we do not have as tight control over retrieval results as in the case of offfine retrieval. This might, indeed, create some discrepancies over longer time horizons, not only due to changes in the underlying search engine logic. but also because new published documents might provide more direct answers. Overall we believe that potential benefits of understanding how to use Google Search with LLMs overweight the potential downsides, if done responsibly.
Publishhd test set results for fine-tuned open- and closed-book are 23.0% and 18.3% (hhag & Choi 2021

---

Under review as a conference paper at ICLR 2023
REFERENCES
Leonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimiliano Ciaramita, Lasse Espeholt, Thomas Hofmann, and Yannic Kilcher, Boosting search engines with interactive agents. arXiv preprint arXiv:2109.00522 2021.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang. Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Ba San-Baatiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprint 0rXiv:211200444 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Abdessamad Echihabi and Daniel Marcu, chisy-channel approach question answering. Technical report, University of Southern Californi, Maria Del Rey, Information Sciences Institute, 2003.
Martin Fajcik, Martin Docekal, Karel Ondrej, and Pavel Smrz. R2-d2: A modular baseline for open-domain question answering. arXiv preprint arXiv:2109.03502 2021.
Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vla- chos, Antoine Bordes, and Sebastian Riedel. Generating fact checking briefs. arXiv preprint arXiv:2011.05448, 2020.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 2021.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01222, 2020.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yit Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave, Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.0322 2022.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint urXiv:2004.04906 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00177 2019.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2 2021.
Canasai Kruengkrai, Junichi Yamagishi, and Xin Wang. A multi-level attention model for evidence- based fact checking. arXiv preprint (oXix:2106.00900 2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
10

---

Under review as a conference paper at ICLR 2023
Mike Lewis and Angela Fan. Generative question answering: Learning answer the whole question. In ICLR, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems; 2020.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematie survey of prompting methods in natural language processing. arXiv preprint arXiv:2107. 13586, 2021b.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. arXiv preprint wrXiv:210900055: 2021.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang. Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, et al. Neurologic a esque decoding: Constrained text generation with lookahead heuristics. arXiv preprint 1.X.x:2112.08722: 2021.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.0066 2020.
Jacob Menick, Maja Trebacz. Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Camphell- Gillingham, Geoffrey Irving, and Nat McAleese. Teaching language models to support answers with verified quotes. arXiv submission, 2022.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.. 2021.
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-tau Yih, et al. The web is your oyster- knowledge-intensivy nlp against a very large web corpus. arXiv preprint 2021.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmaan, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint ar arXiv:21 /2./ |446, 2021.
11

---

Under review as a conference paper at ICLR 2023
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. arXiv preprint arXiv:2112.0863: 2021.
Devendra Sachan, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34, 2021.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.0756,, 2021.
Felix Stahlherg and Bill Byme. On nmt search errors and model errors: Cat got your tongue? arXiv preprint arXiv:I908. 100,0000000000 2019.
Nandan Thakur, Nils Reimers, Andreas Rücklé Abhishek Srivastava, and Iryna Gurevych. Beir: heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Reee-Tzee Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.0880 200002290.,,,,, 2022.
James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long Papers), 2018.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint Stti:220090065555 2021.
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint 20Xvv:2009.22766, 2020.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 2021.
Michael J.Q. Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.010 01068, 2022.
A APPENDIX
A.1 ILLUSTRATION OF METHOD
Figure presents an illustration of our method.
A.2
USING L.L--DERIVER DERIVED SCORES FOR PASSAGE RERANKING
Figure presents results when using dereivee scores rerank passages.
12

---

Under review as a conference paper at ICLR 2023
Question: Who won the NPL football coach of the year?
Query Google Search
Chunk documents into passages, rescore them
Evidence puragraph)
Most relevant
paragraph
Evidunce paragraph 48
Evidence paragraph 49
Fvidence paragraph 50
k-shot prompt
Generate answers using LM
Generate scores
Rerank
Doastioo:
BoE
Answert 1I
Evidence paragraph
0.5
Questiion: Wos
coach
seary
Evidenoe paragraph 50
Baker
Figure 4: Schematic representation of the method presented in Section
80
70
Answer Recall
60
50
pnaassage/questoon
passagee)
20
TF-IDF
DPR
10
10
20
30
40
50
Number of Passages
Figure 5: Comparing NQ answer recall when reranking Google Search passages using L.LMs-derived scores against DPR retriever on Wikipedia.
A.3 RETRIEVAL PERFORMANCE ON A SUBSET OF BEIR BENCHMARK
Over and above access to updated information, search engines are also optimized for good retrieval performance across variety sources. domains and queries. The discourse the community is slowly moving away from in-domain performance of retrievers when trained with abundance of in-domain data, towards more realistic scenarios of out-of -itto--distribution generalization using zero-shot evaluations. The recent BEIR benchmark (Thakur et al.) (021) provides such an evaluation and so used this to compare the zero-shot retrieval performance Google against DPR (we use the DPR model used in the BEIR benchmark pre-trained on NQ, TriviaQA, WebQuestions, and CuratedTREC), As the search spaces between DPR and Google are different (Google uses the whole web, DPR is using the search space provided for each domain by BEIR), evaluated models using word overlap metric between the gold and retrieved passages. We only report metrics for the subsets of BEIR that contain more than 500k documents search space.
13

---

Under review as a conference paper at ICLR 2023
average
scidocs
scifact
nfcorpus
arguana
fiqa
trec.covid
webis touche2020
DPR@1 @
21.1
33.05
15.99
31.54
31.54
21.91
16,44
10.59
Google@
43.7
56.85
32.48
60.59
60.59
57.07
37.37
35.50
DPR @ 100
50.3
33.66
78.44
28.79
85.57
55,41
24.14
46.44
Google@ 100
67.7
68.67
76.00
55.17
86.18
78.97
54.03
55.16
Table 3: Word overlap results between retrieved and gold evidence on subsets of the BEIR benchmark for different retrieval models.
Looking into the domain -averags performance in Table that Google@ is +22% better than DPR across all domains considered here. We note that these domains do not include the Wikipedia based ones (e.g., NQ and HotpotQA) in which know DPR scores high. On looking at top100, we see that for certain domains (c.g., nncorpus, arguana) DPR approaches Google. The difference between Google and DPR for top100 smaller than top1, but DPR still outperformed by google by 17%.
A.4 REPRODUCING RESULTS WITH OPT-6B MODEL
We performed an experiment using the OPT model of 6.7 billion parameters. We measured the performance of OPT-CB and OPT-OB, following the same experimental setup we followed for the experiments using Gopher-280Bi.e. same number of evidences used, same number of samples taken and the same prompts). We found the performance of the OTB-CB to be 15% (compared the similar-sized 7B-CB 11% from Figure 2 and the OTB-OB to be 24% (compared to our 7B-OB 26% from Figure 2.
of out-of out-of-distribure distribbutiion generalization using zero-shot evaluations. The recent BEIR bench- mark (Thakur et al., 2021) provides such an evaluation and so we used this to compare the zero-shot retrieval performance of Google against DPR (we use the DPR model used in the BEIR benchmark pre-trained on NQ, TriviaQA, WehQuestions, and CuratedTREC). As the search spaces between DPR and Google are different (Google uses the whole web, DPR is using the search space provided for each domain by BEIR), we evaluated models using a word overlap metric between the gold and retrieved passages. We only report metries for the subsets of BEIR that contain documents as search space.
than 500k
A.5 REPRODUCING RESULTS USING WIKIPEDIA-BANE BASED DPR EVIDENCE
find that GPR+er performance (32.9% is almost indistinguishable from the Google+Gopher (32.7%), which is expected as the retrieval performance for both is almost the same. Since NQ has been created using Wikipedia articles, DPR has been trained on Wikipedia and the search space is the exact Wikipedia snapshot that used to create the dataset, this experiment is the best possible case scenario for DPR. For example, we see that many questions in NQ are time- sensitive (c.g., When album
does thomas rhett's the world, is at disadvantage when compared to DPR with. The recent Atlas work (Izacard et al.) 2022) report similar findings with respect to the effects of temporal alignment between the Wikipedia dumps used to create NQ and retrieval evidence.
out?) and Google, which is using updated snapshot of
A.6 QA PROMPTS
Bellow we provide the prompts used for each of the datasets to create the open-book models. The prompts for building the closed-book model is derived by omitting the Evidence part of the prompt.
14

---

Under review as a conference paper at ICLR 2023
A.6.1 NQ
Svidence: Top 20 rankings a5 at 15 October 2017 Rank Charge Tean Pointe Bernany 1631 Brasil 1619 Portugal 1446 Argentina 1445 5
4 Relgium 1353 6 Poland 1323 3 Irance 1996 8 Spain 1218 9 Chille 1175 10 Pero 1160 11 Switserland 11:34 12 Regg and 1116 15 Collombi
4 1095 14 Relwa 1072 15 Italy 1065 16 Kexico 1050 17 Druguny 1034 18 Cooatia 2013 19 7 Denmask 10C1 20 Nelberlanda 931 Change
14 Septenber 2017 compleze rankings FIPA.com
Ooest ionz who has been ranked no, in the latest football rankings announced by Fs
Answer: : Germany
"Your Lave" is song by the Engl ish rook band the futll, From thear debut albun Play Deep (19851. "he song + penned by the band'e quitarist John Spinks.
Rvidence:
Ouestion: who sings i just went to use your love tonight
Aravers Rnglish rock band the Outifield
Rvidences Prinaipel photogrephy began on May, 20, 2016, in Melch, West Virginia. Queat whece vea the novie the glass cestle filned
Answer: in Melch, West Virginia
Evidenec: No. Hame Field Mrtlliaticn Date of Appointnen: Date of Retirement Roope Ganguly Art Bacratiyz Janato Farty 34-0ct-2016 4 03-06t-2022 Samahaji Rage Secial work Baeratiys Janata Farty 0r--uu-20166 03-Mey-2022 Suresh Gogt Art Bhezatiya Janata Farty
25-4pr-2016 24-Apr-2022
Subranani Sweny Rognemics Rharat danata Party A5-Apr-2016 24-Apr-2622 Harendra Jechav Rconenies
Sport Nominated 25-Ap=-2016 24-29r-2022 sxapan Desgepta, Journalion Heminated 25-1pr-2016 24-Apt-2022 R.I.S. Tulei Law Rominated 20-Feo-20: a 24-Feb-2020 9 K. Parasagan Las Noniasted 09-3un-2012 2012
Koninated 23-250-2016
26-Apr-2022 Mary
28-Jun-2018 10 Rekhe Art Voninated 27 21-Apr-2612 26-Ap--2018 11 Bachin Tendol kan Social service Nonirated 27 27-Apr-2012 A6-Apr-0018
12 Anu Aga Puriness Neminated 27-1ps-2012 26-1pr-2019
Question: who the tiest Lady nominaced menzer the radya saaha
Answer: Mary Kon
chiocen sanowich sold the internationel tast-tood chair. HcDonald's, The sandrich consists ot toasted wheat. run, breaded chicken patty, saradded lcttuze, and maycanaise. Quastion: shat 15 en a mechicken sandwich fron medonalds Ansver: preaded chicken patty
Svidence:
The McChicken
Evidence: Lite et is Canadian tantasy adventure novel by Yann Martel published in 2001. The protagenist is Piscine Moliter rpi" Facel, an Indian boy fron Pondi cherry who explores issues of spirituality and practicality fron an early ses, He survives 227 4 days after chipwrock while stranded on lifcbost in the Pazific occan with a Bergal tiger nancd Richard Farker. question: what is the tigers Ansver: Rishard Pasker
4
in lite
Evidence:
Malxare, short for malicious scttwere, is ar. umbrella term used to reter to a variety of torns ot harntul or intrucive inciuding computer viruses, It can take the forn exccutablc codo, scripts, activo content, and other sattwero. Nalvare is detined by its malicicus incent, acting asainat the sequisements of the conputer user -- and se does not include softvare thas causes unintentional harn due to seme deficiency.
horses,
apyware, adwaze,
and other malicious prograns.
quastion: the general
sattwere that is designed damags dicablc stesl data
Anaver: Nalvace
Evidenee:
Man Geare Sitcon Created by Stetan tolaszeweki Writton by Stetan belaszawek: Directed by Richard Laxton Stetan Gelaszeweki Stazring Lesley Manville Peter Mullan san avairabury Lisa NeGrillla Opening theme by talu and Lampshades Country
origin United Kingdon. original language ls: Eaclish No. of peries No. opisades ito March 2018) Production Running tine 29 minutes Production company (si Big Talk Productions Distrisutor 11V Stutios Ralease Original network BBC Two (2016-presen) o B2C Two ED (2016 Picture format 16: 9 10801 Audio fornat stereo Original release 13 May 2015 (2015-05-13) prezent boc2
thene Anawers Lulu and the Langahatie
quastion: who
Evidenee:
The Chess World tup 2017 was 128-p1zyer single-slimnnttin chess tournament, held in Thilici, Ceorgia, fron 2 to 27 Septenbes 2017,
Acnenian grandnaste: Levon Asonian. This the second Lime had the Chass World Cups 12 years aftar his first win 2003.
quastion: shere wEs the world chass tournament 2013 held Anaver: Thilisi, Georgia
Rridences T.J. Miller as Randy Revin Michael Rtchaaddon as Baxin, others Bevid Rowchnes as Robert "Bob Page* Pogrohvich, Prank o obese, chainancking bosa. Revin Parley as Babe, Cart, others daty cole as Rodgez Dunbarcon, the cutar and founder of the airlinas share Frank and his co-verkers work. Zoo Buck as Lou Gagliardi, others John DiHaagio as scoop Dunbarton, Regez nocunic nephes. Allison Janney Benristla Van Borns Miller Bandy Michael Mill ians Baoky Queations who voices randy in E is foz family
burbertun's
Ansver: 2.J. Miller
15

---

Under review as a conference paper at ICLR 2023
Evidences Las Vegas Stadiun is the working name for a domad acadian under construction Pazadise, Nevada for the Las Vegas Raideza of the Natioon Pootball Teague 1891) and the oNL Rebels Foothall tean from the tir University of Nevada, bay Veges (UNT2). is Becisnda and Polacis Brive, just west at Interstate 15, construction of the 91.9 billion stadium began in September 2019 and 15 expected to be a completed in time for the 2020 NPI, seeson.
looated
Nandalay
Bussell
menaiions whece Chey building the taiders stadiun
Rvidence:
AL the Line of ita initial publin offeing (IPO) oa the abouk nacket in dune 1992, BLarbacka had 140 out leis, with US 9 13.5 million, up from US 91.3 million in 1987, The company's market value us 5271 milliio this time. The port of the corpany
company, facillitat doubl runber of stores over the nexz tuo years, By September 1992, Starbucks chare peice had risen by 105 to over 100 times the earnings per share of the previous year,
525
Queat ions when did starbucks becone publicly traded congany
Answer: June
Rvidences At the and of becember 31, 2015, its erployen strength wea 170, 664, Abid Als Neenuchwala was appointed as Wipro's 130 atter T.K,
in 2016,
wipro comрany 2016
Answer: Abid All Recnuchxslo
Tvidencee WINER::
Genf Zakarian Ppizode 5 Commente Zakamlan WTN TN TN oo WIN MTN The Next Tron Chef Fa kner TX Elim: Pressure chiarello IN IN IN 1N MEN OUT Elim: Biss Semeelseon co IN OUT ELim: Sterytelling Macii: Lian HIN IN co OUT ELim: Improvication Hughes sohn IN IN our El in: Trgensity Trvine IN our in: Thans Formati fon Mendel sohn GBT Rham: Resourcefil ness ness chets
oUT Passion
Bursell
Questions wine the izon chat Zakerien
Svidenee:
Support for and the clevaticn et leaves, flowers and truits. The stens keep the leaves in the 1ight and previce place Stosage natrioncs Production
for plant
truits. Transport at fluids litespan of planz cells tarce years. Stens have colls
the shoots in the and phloen
living tiscue. called necistens that incually genorate new living tisoue. Ouestion: what are the main functions of the atem Prodection isving sissue
16

---

Under review as a conference paper at ICLR 2023
A.6.2 HOTPOTQA
Svidence: Seesex in a musical with book by Michael Bennect, music by by Coleman, and lyrics by Dorathy Fields, Michael Bennett 4 (April 8, 1943 i duly 2, 1987) was an Anerican musical sheatie di director, writer, choreogragher, and danter, Pe von seven Tony o Avarda for his chureugraphy and direct ian of Broadway ahuwa and was noninated for an addit additiinal. wleven. the wester of Seeeav born?
Question: When
Arsver: April 8, 1943
Svidence:
Heinrich Auguet Merschner 116 August 1795 1.14 Decenser (861) the ixportant congoser of Gernan opere petveen 19 Novenber 1786 June 1826) German composer, corductor,
Reber Negner, Carl Maria Priedrich Renst von Raber pianist, quiterict and critic, and was one of the tiest significont composere of the Ronantic school, Question: Heiinichh Nerschner wae corposer who performed in the time frome after ore of the first sigriticant corpozers in what + school of work?
Answer: Romantic
Rvidence: The Elliett Donoldson House is historin in mansion in okalone, Missizaippi, 0.S., It was built in 1850, a desede prier to the Anericen civil Nar of 2861-1865. By the end of the in 1855, Contederste States Army General Nathan Bedtord Forrest house: rest. 't has been listed the Nat Register Ristoric Places Septenber 18, 1980. Nathan Bodford Forrest IJaly 13, 1822 I dctcber 29, 1877), colled Bedterd Forrest in his lifctime, was a licuterant generol in the 4 Concederate Amy during the American Civil Mar. Quest that leeutenant general stayed in the Answer: Nathan Bedtord Forzest
stayed
ldson House?
Rridence: laca Parmitano [born 27 Septenber 1976 in Paterno, sielly) is on ian engineer and ratwet in the Roragean rstant Corps for the European Space Agency (ESA), The astronauts work micsions at the International Space Station. He was selected as an ESA sstreneut May 2009, Frot. Dr, Uleica Hans Walter Iborn February 5, 19591 is a German phyeiots:/enggnnen and a former DEVLR astrensut.
Quastion: Who in younger Ulrich Wolter or Luca Pornitan?? Ansver: Luse Parmitano
Svidence: & Boitznann nachine is a type et stoctastic recurvent neural network (and Markoy Randon Field) , Lutvig Eduard Boltzmann 4 (February 20, 1864 Septemte: S, 19063 was an Austmsan physicist and pat potlopppeer whose grestest achievenen: was in the the properties atoms (ouch nass, charge,
seveicsment of statisticsl machanics, which explains predicts structure) deternine the paysical propertics of matter (outh as viscosity, tharmal conductivity, and dittusion). Quastion: that nachina has the sans name as another nachine created by Leaviq Boltzmanc? Answer: Boltzmena machine
Evidenos: Grahen Linehan. I born 22 May 1968) is Irish telavisicn conady writar and director who, ofien in partnership with Artaer Mathove, has written or co-viitten a number of popuier television conedies. Be is most ncted for the sizcons "Fathor Ted', "Black Books* and "The IT Groud", " Amongst otaets, be has 5.50 worked on "Big Train", *Count Arthur Strong", *Brass Bye and *Iae British sitcon Channel 4, written by Grahan Linaban, produced by Atalla and starring Chris O'Dawd, Richard hyoade, Kathorine Parkiasen, and Mast Berry.
Show". The 2I Crowd
Queat Grahan. Linehan vaa the creator of the Aat: paalda-reddeedd sitcon for what UK channel?
Answer: Channel
Evidencs: Andrea Chéniez veriamo opara four acta by the composer Unbarto Glordano, sut to an Italian libretto by luigi Illica. It was first performad on 28 Harch 1896 at la scala, Milan, The opera's scory is based icosely on the life of the French poet Andrd Caenier (1762(17541, who was exccuted during the French Revolution. The character Carlo Gérord is partly based on Joan-Lanbert fallien, leading fiqure 1896 spera *Andrsa Chénisg" by Unkerto Giardanc.
sevelution. "la
(thay kille ny moshar) ls aria from ast the
quastion: that is the name of the third act in a play where a charector named Carlo Gerard is partly based on revolutionary + fiqure?
La nanna nerta
Evidanca: Richard Kasur (born Noverbez 20, 1949) is an Amarican actor who has appeazed in noze than B0 novies. From 1995 to 1999, he served two terms as president et the Screcn Actors Guild (5MG). Hasur curreatly sits on the corperate Board of the Motion adventuze filn weitien Nell Tulkin and dixected by Beunan in his feature filn dizactorial debut. It stars Cosey Bain, Carey Foldman, Beathar Grahan, Carol Xano, 3ichard Nasuz, Michael Manascari and Kina sienaszko.
Ficture Pelevision Pund, License Drive
1998
Queations Licenae to Drive fealuzed what future president of the Screen Accors Guild?
Ansver: Richard
2vidences The British Breadeasting Cosporat ion (B3C) is a British public service broadcaster with its beadquartars at Broadcasting House in London. The SBC is the world's oldest national broadnasting erganination and the largest brozdcastar in the world by 950 staft total, 16,672 of when in public sector broaduast ing. The Lotal number of staff is 35, 402 when part- flexible, and fixed contrast staff are included. Denko was television drans, co-pzoduced by the BBC and the ABC.
number eagloywes.
erploys
Queat lons Whece is the beaduartees for the service browdcastes the show Tenko is on?
Ansvers Broadiasting House London
Zvidences Thons Halthew "Ton" Chapgell (boen 1943) is an Amerian businesanan and namffactuzes and po founder of Tunt' of Naine in 1970. Ton's of Haina is brandnane and nanu Cantzerr of naccral- Lngg-ggdddnnt--eed parsonal case products, na jorisy-omnee
subsidiary of Calgete Palmel since 2306. The conpany's products intent fora noatly made shtt ingred enta that chenically derived, have a negalive env inoomnataal inpact, or are Lested on aninala, While nost of it producta are vegan, sone contain propolis and/oz baessax sourced fron bees.
Quest Thomes Hatthew "Ton" Chapgell 1 oo Founded compaany in 1970 that nanufactures what products? natural ingredients only personal
products
Rvidence: Franciy y Banon, lat Viscmun So Alban, 1111 6, 121; ", 13%; ", 141; *) (. 22 January 15619 April 1626) wes an Engi ish 4 philossohies, statesman, scsencist, jurist, ocalor, and authur. He aerowd both Allosney Genweal and Lord Chancw liee of works, especially
England. After his death, renained extrenely ly infi iefluention through practitiones of the scients nethed during the scientifig revolot lon, Janes Spedding (28 June 1908 ( 9 Manch 1881) was w English authos, chiefly known as the editor of the works of Prancia Bacon.
pai poipotoptionl advocate and
Questions Janes Spedding was chiefly know the editor of the works of an aothor who served both Attorney General and as what?
Lord Chanseller Rrgland
17

---

Under review as a conference paper at ICLR 2023
Evidence: Ronao Rontaque (Itallan: "Ronee Mentecchi" is the protagonist of Millian Blakespaaze'a cracedy *Bonso and Jullet", The san of Nontague and his wife, he secretly loows and marries Ju iet, nember the rivel House Cepulet. Foroed into exile Tybelt,
comils suicide apon leeaang tictionel character Shacespeare's drama *Roneo and duliet", leee is Nortaque's nephex and Roneo's cousin, Bervolio serves as an u unsucoesefol peacenster in the play, atterpting nn to prevent vio ence bet ween the Capu et and Montague fani lies. Queations Whigh chazacler does thas prosagoniat, who secsecly luves and narsies neaber of the rival Louse, of Willian slay?
after slaying Jullet's
Jaliel's dealb, Benvalio
Shakeggeare's tragedy Araver: Tyoelt
Francesca Schlavone born 23 June 1980 in Milan) Italian teanis the 2010 French Ogen singles sitle, beconing cunner-up at the 2011 French open, Ber career high ranking is vorid Ne. & achieves on 31 January 2011. To date, Schiavone i5 + the last one handed-bacthane plager to win Grand Slan title on the wonen's tour. Carly Gullickkor Iborn November 26, 19861 is Forner Anerican professional tennis player,
Svidence:
torned professional in 1998, She singles. the
Italian
Question: that ecoupation have Carly Sallickson and Francesca Arsver: professional ternis pleyer.
Svidence:
Otelic ie penulsinate opere, Gisseppe Verds Italian libretzo Arrigo Boito, based Shakerpeare' play February After Aida (original citlc: "Verdi's is Nossiah"| is 1985 play-vith-mesic by Julian Nitchell. It about diuseppe Verdi, and the pressuze pot upon him atter his attenpt to retire trom congosing, Continued instacent prodring tron ais triends eventually his
in
performed
results
mesterpieces, the opera, Mhich was subject or Atter A165, tiest portornod?
"Grello", which preniered or 1887,
Quastion: theze
Arsver: Teatro alla Scala
Evidenee:
The Commedore
Commodors Internatiensl with 8:02-compatible 7501 8501 CPU, released in only
interdes to an entry-level computer replace VIC-20, cost-reduces version, Commodore Earope, Ia the siddle of 1984 Braxillar corgany called Prolágics, which made its own veraions of 8 bits conpatera, brought to tae Brazilian market new equigment ror its personal computer series called "CP" (shorten ot Personal Computer in Fortaguesel.
Quastion: Moza Commodore Ansver: no
CE-1CO
country?
18

---

Under review as a conference paper at ICLR 2023
A.6.3 STRATEGYQA
Svidences The Albenien Declaration of Independer.ce is written A.banian, Sheg, Toek, and Ottoman Torkish, The Arvanite Greek's are Cs a najor Tosk speaking group of southern Albania. Queations Can Acvanite Greek understand some of the Albanian Declaration of Independence?
Rvidences An ansious person may benefit Irom sedication oe thecapy. The Risard of 0z cannot give bousage Le anyone. Quest fonz Sould an anxious person benefit from receiving courage From the Mixard Ox?
Rridence: Sili ioon is a key naterial for the product for of serri conductrr chips. s-ligon shortage sould mean Fexer sen' conductor + chips could be produced. A business that produwee fewer products than nornel w:1l swcelve lover than narnal rRONNUM.
Question: Would Bilicon shortage be tor Intel's sales? Aravers true
Svidence:
National Football The National Footbell is sports football enjoys najority of popal popularity in the United States bengel Found exclusively on the Indian auboon inent.
The Sugerbowl
champienahip
Feasball
Ouestion: Is a bengal tox likely so see the Superbox1?
Araver: Falise
Svidence: The letter is the second lecter in the Latin Alphazet, There was one total. lunar eclipse in 2008,
Quest hoey the letter B''s place in alphabet exceed number of 2008 total Tonar eel ppaes?es?
Answer: true
Rvidence: The Battle of Raghead wes the U.S. invasion of Reghdad in the year 2003, Just Bieber's 'x elbun Beliave was released 4 2012.
Question: 3id U.S. soldiers Histen to Justin Bieber's Believe albun during the Bettle of Baghdad? Answer:
Svidence:
The itn fan Rensiseance was a period of History from the 13th century to 1600, A chepcracy is type of rule in which soligious leadere power, Friar Cirolamo Sevorarola November I May 1498. Quest Naa Plarence Theecracy during Ttalian Rensistance?
ruler Florsace, after driving the Medict fanily,
Answer: true
zukushina pallchi nuclear рeиez plant meltdown Nuclear molzdowna lead release soadly levols et rodiation Godrilla draws power tron resiation and is not hurt by it Osestion: Could Giszills have been kill by the Tohoku eartaquake?
Evidcnes: The Tehoku earthquake
Svidence: Robert Moses Stove woe baseball player nicknamed letty Grove, Pablo Escobar had several nicknames including: Dan Pablo, 4 E1 Fadrino, and El Fatzón.
Question: 3id Pabic Escober's nicknane collection ouzchine Robert Heses Grove's? Ansver: true
Svidence: Anaheim is the biggest city in Orange County, calitornia Ansheim was counced by risty Gernan tamilies People trom Gernany speak
quastion: 2id founders of the biggest city in osange Councy, colifernia speak Italian?
Answer: talse
Anerscan band tast has active nembers. The 2020 Mitsubishs Outlander has tiextalc seating allove tor seven seat capacity. Quastion: can hcrommith dit in a 2020 Mitsubishi outlandar?
Svidcnes:
Acrosmith
Answer: true
Vietnan (1945-45) lasted around menchs. The gestation pariod fox llana is meaths. Quastion: could 11ama Eivth twice during Mar in Vietnan (1945-161?
Evidenes: The
Anaver: false
Territie vas nicknamed terrible because st his harch ruis. Ivan Terrible's tether, Vosild III Ivanovich, was bo nicknaned Vasili the Adoquate. Ivan the Terrible's gsandfather, Ivan III Vasilyevich, was nicknaned Ivan the Great. Quastion: 2id Ivan the Terrible's father and grandfacher have nicer aicknancs?
Evidenee:
Evidenes: The Beatles vere activa from 1960 until 1959, Disco beeen to appear sround 1972.
Queat ionz bid the Beatles wrile any music in the Diaco gense?
false
Zvidences Ganynede is noon of Jupices. Dupiter is the largeat planet in oue solar system, The soles systen is pack of the Milky lo Ray calaxy. Ganymede in the Milky Nay galaxy?
Anaver: Lruw
19

---

Under review as a conference paper at ICLR 2023
A.6.4 FEVER
Evidence: Begazza served as Militazy Aide to the Kilicaxy Goveznoz of Puezto Rico Thaodore Roosevelt Jr. and daring Hoeld Baz II comanded the 65th Infantry Reginent Bases (fila) Teatures Pakistani
in lead cule.
Arswers ercor
Rvidences With an estimaled 92.7 million inbabitanta it is Lhe wurld "s 14th noss pupuloux counleyy and the ninth must populous ssian country
Quest lonz Vietnam not the mirth most populous Asiar country.
Ansver: falas
enbrai led in series of roversiess including using alcernation facls making seference Lo Booling Grven massccre that never copursed y claining Nichael 71yar. the 4 full contidence of the president hours before be was diemi and publicly andorsing commercial products associated with the president la daugheer Tvankca Trump
Tvirdence:
Trump inauguracion e Conwey
Question: Kellyaane Conway been embroiled in series controversies. Answer: true
Evidence:
The vi sllge around 2000 tamilies This village People
the cnlebrations Batukamaa testival celebrated during
by lages here play Ratukanna Ocext lonz untun o'Neel had three children juvenile archritis.
Answer: ercor
storyboard artist on Care -1RS- -RRB- and Cors -13B- 2011 Ouestion: Cars isn't the third Care movie Brian Fee has worked on, Anguer: False
Evidenes: It acing directed Brian
Evidence: * Up
song by English Seime artist Stornzy Duestioon: Shot Up song.
Answer: true
Evidcnes: Fatzice Loke Fronch formez fsothsller
Quastion: The dranatic film The Sood Corman was directed by Steven Soderburgh. Ansver: ercor
Evidence:
Foseidon grossed 181, 576, B17 at the worldvide ottice budget 160 million Ouestion: don lost $181, 617,
Answer: ralse
fictional character 1999 film Star Wars : Episode The Phanton Nensce Question: Qui-Gen Jinn is character in the Star Ware tranchise.
Evidenes: Qui-Gca
Kazs franchise portzayad by Lian Nesson as the main protagonist
Bridence: It is only known tron a partial skull and several vertearae but conparieone with other species of nonitor Lizard put its + size batween 60 and in length
quastion: The Mernen population
cignificintly since 1980.
Anaver: 01201
Python teatures cynamic type system and automatic menory nanagement and supports multiple programming paradigns including ject-eriented inperative faretionsl programming procedural atyles quastion: Python lacks dynamic sype systen.
Evidonee:
Answer: talse
Svidenes: The scries nominated four Anato Awards Charactor Animation i Character Design and starynoarding an Animated BBoodddatt Production Tronthe an Americar. compteeraniinted fantasy tolavisicn series created for Neeflix by Gaillerno del Tozo and produced by DreanMorks Animation and Double Darc Yeu
winning three categories Ousstanding achievencat
Queations Trolihanera aninated.
Ansver: true
Byidences Guilleenc Kuschel LLS- born 1918 - -33- bliie-born untunologiat living in Now Zealand Maxinilian Kuschel -1RB- 1851 oraitholocist and solegis: quastion: Sham Carlson American and German.
1909 -REB-
Anaver: 0i:0z
Rvidennee Kher subsequently appeared in
conedies including
LaB 2305 138 Like 100
2005 -B8B- What Bappens Vegas Americar. romantic canedy fiim directed by Ivan Reitnan and written by Elizabeth Moriwether Queat lonz Ashitom Kutchee was not, disected by Ivan Beitman.
2008 -266 and Sizings Actached 2011 -R3B- acrings Altached 2011
false
Rridences It dea formerly celled Iolan Jaya to Guinea
12B beform that Boat Irlan
or Irian Baset RRB and cumpriaed all of Indunessian ian New
fons Pepue formerly called
Anavers Lrum
A.7 PROMPTS FOR CALCULATING SCORES
Here, we provide the prompts used to obtain the different scores considered the different factoriza- tions. We provide the prompts derived for NQ.
20

---

Under review as a conference paper at ICLR 2023
A.7.1 CALCULATING p(q ats Ps)
Svidence: FIFA World Rankings, Top 20 rankings on of 16 October 20:7 Rank Change Team Foints Gernany 1631 Brasil 1619 Portugel 1 1446
Argentire ina 1445 5 Belgiun 1335 Paland 1823 France 1226 3 Spain 1218 9 Chill 1373 10 Peru 1158 11 Sxitzerland 1134 12 England 1116 13 colanbia 1085 14 Males 1072 15 Italy 1065 16 Nexico 1960 Bruquay 1334 18 Croalia 1013 Dennack 1001
Kecherlands 951 Change from 14 September 2013 Complete rankings at PIPA.con
Ansver: Germany
Questions who been ranked
latest tootnall zankings announced by
Rvidences Your Lave tTh Out field songh. "Your love" is a wong by the English rock band the Out filld Laken from their debut album 4 Flay Deep 119851, The song was pennes by the band's guitarist dean Spinks,
Arsuers Regii rock band the Butfiell
Queat lons who sings just want use your love tonigh:
Rvidence: The Blass last (filn). Principal I photogrephy began on May 20, 2016, in Me Melch, West Virginia. Answes: in Helch, Real Virginia
Ouestion: where wee movse the glass cestle tilmed
Evidence: List et noninated nambers oz Babya Sabha, No. Kane Field Artillation Date ar Appointment Daze of Retivenent Rcopa Ganguly
Art Bharstiya Janata Party 04-Get-2016 03-0ct-2022 Sambneds Raje Social vork Sharatiya Jansta Party 02-Jun-2018 03-May-2022
Suhrananian Swany Reanonics lhhartiia danata Party 25 Apr- 2016 26-1pr-2022 Karendra Jashay Econonics Nominated 23-Ap=-2016 24-2pr-2022 Mary Kom Sport Nominated 25-2pr-2015 24-Ape-2022
furesh Gepi Rherat danata Party 25 Apr- 14-Apr-200
Swapan Dasgupta dournelien Koninated A5-Ap-22016 6 24-2pr-2022 8 K.T.S. Tulai law Noninated 25-Feb-2014 24-7eb-2020 9 K. Parszaran
lav Nominated 09 Ju---222112 28-Jun-200 2018 10 Bekha Art Nominated 27 Apr 2012 26 3pr-2018 11 Sachin fendulkar Social service
Noninated 27-2pr-2012 26-Apr-2018 Anu Ags ausinese Nominated 21-ApS-2012 26-2pr-2018
Ansver: Many Kom
quastion: who was the rirst lady neminated nember of the radya satha
chicken sardwich sold by internstional fast-food chain Meflonal''s. The sandwich consists + of a taasted whest bun, a broaded chicken patty, chrodded lettuce, and maysara162. Answer: greaded caicken patty Quest what is nechicker oken sandeich from nedonalds
Rvidence: Mothi cken,
Mochicken
Svidence: Lite et Pi. Lite ot Pi in a Canadian tantasy adventuze novel by Yann Kartel published in 2001. The protagonist is Piscine 4 Moliter "gi" Patel, an Indian boy fren Fondi cherry who explores issues of spiritualicy and practiczlity from an early ago, Be survives 227 after shipwrock shile stranded litebast the Pacific Ocean with Bengal tiger named Richard Parker, Answer: Rishard Parker
Quastion: shat is the tigers nana in lif: of pi
Malxare. Neivare, short malicious sottware, is unbrella used to reter to variety torns hermtul intrusive seftware, incleding computer virusas, worns, Trojan herses, ransomvare, срyxars, adwars, searewora, and other malicious prograns. It can tase the form of execstable codo, scripts, active content, and other softuare, Halware is dctined by 4 its nalicious intent, acting against the zequizanencs of the computer user -- and so does not inclede software that causes unintentional due
Evidence:
deficioncy.
Answer: Halvace
Question: the csneral corn for saftware that is desigrad to damage disable or steal data is
Svidence: Man series).
Genre sitcon Created by Stetan Golaszowskt Meitten Ststan Golassovest Directed Richard Saxton Stefan Gelaszowski Stazring Losley Hanville Petar Hullan san Svainabuzy Lisa MeGrillia Opening theme Cups by hain and
Lampshades country of origin United Kingdon Original language is) English No. of series No. of opicodes 12 (to 27 March 2018: Production Running time 10 ninutes Production company (s) Big Talk Productions Districotor ITV Studios Release Originsl network
Two HD (2016-pzeseat) Ficture fornat 18: 10301 Audia forma Stareo original release 13 May (2016-05-13) present
B2C (2016-present)
Anaver: Lulu and the Langshadea
quastion: who sines the thene tune to mun on boc2
Chess Morld Cup 2017. Cheas Horld Cap 2017 was 128-player sincle-elinita: chess tournament, held in Tallisi, Goorgia, frem 2 to 27 Septenber 2017. It was won by Arnenian grandraster Levon Arorian. This was the second time he had Nen the 4 Chess World cup, 12 years after his sirst win in 2005.
Bvidences
4
7bill9i, Georgia
Quastion: sheze wes the world chass tournament 2017 held
Evidance: F Is for Fanily. T.J. Miller as Randy Kevin Michael Richardscn as Rosie, others David Koachnez as Robart *Bob Pogo" Kevin Farley Babe, Carl, othars Gary Colo Rodger Dunbarton, Foundes the airlines wheze Frank and his ou woskeza wark. Jue Buck as loa Geglardi,, othera Coin DiKagylo Souop Bunbasion, Rooar Duniartton's racist and neronic nephew. Allison Janney as Henzietra Van Bozne T.J. Millr as Raady Michael K. Willians as Smoky
Fogzohvich, Frank's obase, chainsmoking
and
Anawees 7.J. Milles
Queation: sho voices randy in E in foz family
Rvidences Las Vegas Stadiun. Las Vegas Stadiun is the working nane for doned at at un undez conat conteucciion iea in Pasadias, Nevada for the Vacas Raldera the National Football Leaque (NFL) and the UNLV Rebols football
cron the Univarsity Havada, is located on about 62 acres west of Mandalay Bay at Bussell Road and Raciande Avenue and betueen Polaris o Avence and Dean Martin brive, just west of Incerstate 15. Conatzustion of the 51.9 billion at un began in Septenbur 2017 and oy is expactad to be completad in time for the 2020 REL season.
Vegas (UNIV).
Answers Paradise, Nevada
Quaations aheze are they building the new raidera stadiun
with Starlmoks, At the Line of its initial pabl in offeeing (IPO) on the stoock markes in June 1992, BLarbucks had 140 out lutll Us 8 73.5 million, up from 81.3 million in 1587. The company's markst value US 9271 million by this tine. The 124 port fon of the company that was sold raised around US 225 willion for the company, which Fect litated a doubl ing of o the murber of stures over the next Lao years. by Septsmber 1992, Starburik shace icw had sisea by 709 Lo over 100 Linex the up carnings pez share of the previous year.
Rvidencee
Arswers June 1992
Oueationn when did starbucks becone a publicly Leaded cumgany
21

---

Under review as a conference paper at ICLR 2023
Rvidences Nappo. At the and of December 21, 2015, ita enployau stzength was 176, Abid AL1 Neenachvala was appointed as Wipzo's S cit after HtK, stepped down in early 2016.
Answees Abid ALL Neemudiiaaa Ocestion: who become the ceo ot it wipro company ir 2015
Svidences
The Next Tran Checc WINNER: GwafFewy Zakacian Egisodee 5 5 7. 8 Commenta Rekarian WIX IN IN co IN co WIN MIN The Bext Tran Blim: Fressuce Chiare
Falkner IN MIN IN IN IN our Burrell. 1N 1N TN RIN OuT Elimz Bisk Sanuel waon 00 TN IN IN cDr Rham Storytell ing MacMillan WEN TX DO oun Rlim: o impcovisalies Haghes IN IN 0U? EL Riine Ingenally Ievine IN 00? Rling Trans Counat Mendelsoha oU2 21in: Resouroefuloes
po
OUT Elim: Peesion Gusenaschell U1
Ansvers Rakarien
Quest lons who wins the Fron chef suger chefs
Tvidence: Plant stren, Sugport For and the Klwvat for of leaves, Flowers and fruits, The acens keep the leavex in the light and provide a place the plant to keep its clovers and truits, "raneport at cluies between the roeze and the shoots the xylen and phloen Storage of nut nntrrence Product ion of new living tissue, The normal 1 Fespan of plant celle ie one to three years, Stens + have celly cal called meristens that annuel ly generate new 1 living tissue. living sissue aten
Answer: Prodection
what the functionn
A.7.2 CALCULATING p Ip:)
Same as above, but omitting the Answer field of the prompt.
A.7.3 CALCULATING p(p; 14)
Queations who has been sanked no. in the latest football zankings announced by ELEa
Rridences PTPA World Bankings, Top 20 rankings es of 16 October 20"7 Rank Change Team Points Gernany 1631 Braxi 1619 Portugel 1 1446 4 Argentine iia 1445 5 Belgiun 1335 Paland 1323 France 1226 3 Spain 1218 9 Chile 1173 10 Paru 1150 11 Seitzerland 1134 12 England 13 colombia 1095 Wells 1072 i5 Italy
16 cexico 1960 Uruguay 1334 10 Drostis 1013 Dennack 1001 Necherlands 951 Change from 14 September 2013 Complete rankings at FIPA.com
Questions who sings i just want to your love tonight twidences Lave iTh fitli song). "Your love" is zong by English rock band the Out from their debut alhmn c Play Doup (1985). The sang was geoned by Uhe band's guitarist doin Spicks.
Queat fonz where the movie the glass caxtle filned
Svidence: The Glacs Castle (tiim). . Principal paotography began on May 20, 2016, in Woich, West Virginin.
Queat ionz who way the first Lody nont nated nember of the rajya sasha
nemen
Svidence: List et nominated nembers Batya Sabha, No. Nane Field Artiltation Date at Appointment Date of Retivenent Reopa Ganguly
Party 07-Jun-2016 03-May-2022 Apr 2022 Suhremanian Swany Reononi es tharatiya danata Party 25 Apr 2016 28-Apr-2022 Karenara Jadhay Econonics Mominated 25-Ap2-2016 6 24-xpr-2022 Mary Rom Sport Sominazed 25-2gr-2015 24-Apz-2022 y
Bharatiye Janata Party 04-cot-2016 03-0ct-2022 Sambnedi Rade Social work Sharstiya
furesh Gopi Art Bherat danata Party Ap1-
Ewapen Dasguata dournalien Poninated 25-Apr-2016 6 24-2gr-2028 K.T.S. Tulal lav Noninated 25-Feb-2014 24-Feb-2020 9 K. Parazaran
Law Hominated 09-Jun-2012 28-Jun-2018 Rekha Art Nominated 27-Aps-2012 26-29r-2018 11 Sachin Teadulkar Social servics
Koninated 27-Apr-2012 26-Apr-2018 12 Anu Ags Business Nominated 21-Apr-2012 26-29r-2018
quastion: what is nechicken sandwich medonalds Svidence: Methicken,
Mcchicken is a chicken sandwich sold ay the internationel tast-tood chain McDonald's. The sandwich coneists of a toested wheat bun, breased chicken patty, shredded lettuce, and nayonnstee.
Question: what is the tigers 2vidcnes: Life of Fi. Lide of P1 is Canadian fantasy adventure novel by Yann Martel published in 2001. The protagonist is Piscina Molitor "p1" Patol, an Indian boy fron caertyy who explores iesses of spiestuolity and practicrlity from an carly ago, Be 4 survives 227 days atter shipwreck wh strarded on licebset in the Pacitic ocean with a Bengal tiger named Richard Parker,
in lite
question: the general corn for zottware that is designed to damage dicable steal data is Svidence: Melxare. Meivare, short for malicious sottware, is an unbrella tem used to reter to a variety ot torns at herntul or 4 inzzuzive seftxare, including computer viruses, worns, Trojan herces, ransomvrre, spyxare, adware, sczrexars, and other scripts, active contert, and its malicious intent, acting aqainst the zequiremence of the conouter user and so does not inclede softvare that causea unintentional horm. due to some deficiency.
malicious prograns.
the
executable
sottware. Halvare detined by
quastion: who sines the thane
boe2
Svidincs: Man IIV series). . Rim Ganza sitson Created by Stofan Golaszowski Mritten by Stotan Colaszevski Directed by Richard Saxton Steton Goleszowski Starring tesley Hanville Peter Mollan San Swainsbery Lisa MeGelllis opening theme Cups by iasu and the Lampshades country of origin United Kingdon Original lanquage 19) English No. of series No. of episodes 12 (to 27 Harch 2018) Froduction Running time 30 ninutes Production company Big Talk Preductions Distributer 1TV Studios Release Originsl network Two HD (2016-preceat) Ficture tornat 16: 10901 Audia tormat Stereo Original release 13 May 2016
BBC Twa (2016-presenti (20:6-05-12) -- present
Queations wheze the world cheas
2017 held
Evidence: Chess Morld Cup 2017. The Chess World cap 2017 was a 128-player sincte-alimiazzinn chess teurnanent, hold in Yollisi, Goorgia, zrem 2 to 27 September 2017, It was von by Armanian graadnaster Levon Acorian, This was the second time he had wen the Chess World cup, 12 years after his first win in 2005.
quastion: who voices randy in t is for family
2vidance: F Ia for Farily. 7.J. Miller as Randy Kevin Michael Richardsen as Rosie, others David Roachnez as Robart *Bab Pogo*
Fogrohvich, Frank's ebese, chainsnoking bess. Ravin Farley as Babe, carl, othars Gary cols as Rodgez Dunbarton, the owner and
foundes the airlines where Frank his waskess wark. Buck deglaardi, otlers coin. DiKagyio Sauog Bunbarton,
Rogar Dunbarton's ls racist and neronic nechex. Allison Janney as Henzietta Van Borne I.J. Miller as Randy Michael K. Willians as Smoky
question: xheze they building the raidera stadiun
Svidenee: Las Veges stadiun. Las Vegas stadium is the wozking nane for doned stadiun under construction Paradise, Nevada for
the Las Vegas Raldera of the National Poutball League (NFL) and the Dny Rebela Football Lean Crom the Universily of Havada, has
Vegas (UN19). It is losated on about 62 acres sest of Mandalay Bay at Russell Road and Hacienda Avenus and between Polaris Avenue and Bean in brive, just west Inserstate 15. Const rustion
$1.9 billion adi began in enber 2017
is expected Lo be completed in Lime for the 2020 NEL awason.
22

---

Under review as a conference paper at ICLR 2023
Queat ions ahen did starbucka becone a publiely traded company
tvidence: Starbucks, At the time of its initial publ offering (TPO) the stock narket in Jone 1992, Starbucks had "40 out lets, US $ 13.5 millien, up from 81.3 million. it 1987. The cungany's narket value time, The 124 portion of the company that was sold raised sround US #25 million for the company, which ract racilitated a doubling or the number of stores over the next two years, by Septenber 1 1992, Starbucks share price had risen by 70% to over "00 tines the earnings ger share of the previous year.
with
$271 million by this
Queat ionz who becone the aed of is wipro company 2016 2vidence: Nipcu, AL the and of December 21, 2015, iLa employee strenggh was 170, 684, Abid Ali Nesnadiiaaa was appuinted as Mipeo's CEO after T.X, stepped dow in early 2016.
Question: who wins the next izon chet super chets
Svidence: The text Tran Chef. WUNNER: Geeffrey Zacarian Egi 5.6 7 8 Comments Rekarian MIX TN TN on IN co WEN MIN The Hext Tran 4 Che Palkner TN IN WIN TN CD WIN 6o oUr ilin: Fressure chiarello lo IX Do IN WIN 18 1N DUT Eling Pession Gueenexchel 15 IN MIX IN IN IN
our Blami Storyselling MacMi Improvisation Raghes IN TN O07 RI im: rygeniity Trvine IN OU7 K. ims Trane format on Mendel sohn OUT 3lims Resouroefulness
IN our Burrell
OUT Elim:
Samuelsson co
co our Blim:
Question: what the tuncticns
tvidence: Plant sten. Support For
elevatior leaves, flovers froits. scens keep leaves in the light provide a place for the plant to keep its clowers and truits. transport of fluids between the rocte and the shoots in the xylen and philem Storage or nutesence Production of new living tissue, The normal litespan at plant celle is one to three yeare, Stems have celly cal led neriatens that annual generate new living tissue.
A.8 ERROR ANALYSIS
In addition to discussion of SITUATEDQA in Section 5.5, which can be considered a tail of NQ dataset, we also perform error analysis of HOtPOTQA and STRATEGYQA; see examples in Table 4
Questions in HOTPOTQA fall into two categories: comparison (c.g., "Which was fought earlier in our nation's ' history, the Seven Days Battles or the Battle of Manila?") and bridge entity questions (e.g.. "Which American professional poker player also starred in the 2015 movie "Extraction" While performance of G is comparable for both types, the bridge questions appear particularly difficult for both CB and Google configurations: for Dlla Google we observe EM of 21% for bridge vs 47% for comparison questions. We hypothesize that this is because comparison questions always have access to the correct answers in the prompt through the question itself, even if retrieval fails. Having a correct answer in the prompt makes the task easier for the model. In contrast, for bridge questions the answer needs to be generated "from scratch" if it is not present in retrieved evidence.
Manual error analysis of questions where Google retrieval, OB and CB fail simultaneously suggests that there are several sources of error. One is Google Search's 's recency bias, some questions require older documents; next are failures of Exact Match as an accuracy metric (e.g.. 'rookie of year' vs 'national basketball associations rookie of year', 'nandrolone' vs "anabolic-androgenic steroid aas"); finally, we also observe data annotations errors (multiple gold answers are possible instead of a single one provided or the gold answers are wrong). Perhaps surprisingly, on small subset of questions that we manually checked, we observe the latter two to be the prevalent causes of errors. Therefore, the measured accuracy of 21% bridge questions is likely lower-bound of OB Google performance. In fact, we believe that the comparison questions present a more difficult challenge to the language models, as they often require numeric reasoning.
The corner cases of STRATEGYQA QA include questions where Google Search either entirely fails to retrieve evidence (e.g.. "Could a sloth hypothetically watch an entire episode of Scrubs underwater?") or it retrieves relevant evidence that contradicts reasoning provided by annotators (c.g., "Can vitamin C rich fruits be bad for health?"") (note also that some of the questions are subjective, with different points of view being equally valid). Manual analysis of the results also suggests that most of the questions in the dataset require iterative retrieval and reasoning, where the uub-questions for the individual steps cannot be obtained from the original question through simple modifications, but require reformulation. Hence, whereas retrieved evidence is generally on topic, it is not accurate enough to support fine-grained reasoning required to answer the questions.
23

---

Under review as a conference paper at ICLR 2023
Example I HOTPOTQQ: OR is correct
Q: Which American poofessional poker player also starred in the 2015 movie" "Extraction"?
OB dan bilzerian
CR: chiis faruson
Gold: dan bileeriaa
Type:
E Dan Hilzerion
Worth, Height Births
Deathe Dan Bilzerian Bio, Net Worth, Height ORIGIN Dan Bilzerioa in
American professional poker player
Exidence Dan Blizerina Net Worth (As of Oet. 2021) Colehinskdrzom.
He carnel most money by playing poker. ...
Blogmaphy Poker Playing Playhay-
Poker-Playing Playbay
Din Biizerins is your rypical playboy. He surronnds himself with heautiful fast just anything heart desires.
HOTPORQA: Google
Q: Who is the American singer Tingr--ungwriing riter, who won an award for Best Female Video the 2009 MTV Video Music Awands.
song
Soundtrack?
Gangle
colibic caillar
CB: taylor switl
Gold:
lnylor
Type:
Eridence@1 Cotbie Caillat
Wikipedin. August 2009 released Breakthrough.
She was also part the group
that Album the Year the 2010 Grammy Awards
for her festured vocals and writing on Taylor Swift Fearless album...
Exidence Colbie Caillar
Wikipedin. She also recorded French trunsiated version of this song.
Evidence 6/3 The Sougs Taylor Swift
Switzeriand. Then there is the absolute sadness.
The sadness of losing this person, losing all the memories, and the hopes you had for the fature...
Example 3 HOTPOTQA: Grogle
fails
Q: Which was fought earlier in our nation's history, the Seven Days Bottles or the Hattle of Manila?
OB
battle of manila
CB:
seven days battles
Gold: seven days batiles
Type componison
Evidence81 The Seven Days Battles
American Bottlefield Trist. back almost your.
By late afteracon May 31, 1862...
Exidenee
Munila (1898) Wikipedia. Army private
shot
at Filipino revolutionry soldier and Filipino revolutionary forces returned fire.
Exidence 63 Hattle of Manila (1945) Wikipedin. The city's cagture was marked General Douglas
MocAnthur's key to victory in compaign of receqqustt..
Examppe 4 HorPorQA: OR CB
Q: Whit North Carolina native did Danja produce foe?
justin timberlake
CR: beineyy
Gold:
Type bridge
Exidence Producer Danja Exploins Why Timberlake
"Hiityy" Docsui ...
I cané that "Fiithy" was descrihed in the triker...
Exidemee Producer Danja Exploins Why Justin Timherlike "FFithy" Daesof
This what we
were some those purmetters that he gave you? Exidenced33 Danja prouucionn disoography Wikipedia. "Yesh Yeah You Would' (feat, Grace Jones)- 06. "Hale You Now" - 14. "Hello Good Mumiin" (feat. T.L.) No Boys Alkowelfedi)
"Toy Soldier" (co-writen Keri)
Exumple STRATEGYQA: OB
Q: Does Santa Claus work during summer?
OB Tiue
CB: False
Gold:
Evidence Does Sinta Clus
Summer? Mystic Christmas Blog Mrs.
During the Summer Santa and his Elves also continue their workdwide churity work
along with fighting forees
Exidence What Does Santa Class la The Summer? Mystic Christmas Blog.
What Does Sonta Closs Da In The Summer?.
However
Christaas July
by Cluss.
He hegins preparations
Christmas
What really
professional Santa Class Business Insider.
Table 4: Examples of questions, CB and OB answers together with retrieved evidence from HOTPOTQA and STRATEGY QA datasets.
24


Reviews:
[{"review_id": "nGOJB9_5Gec", "reviewer": "Reviewer_rQxP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the method of few-shot prompting for internet-augmented language models consisting of three steps: 1) retrieval, which searches top 50 paragraphs using Google search, 2) prompting, where four prompts are applied on each of top 50 retrieved paragraphs to generate 200 candidate answers in total, 3) reranking, which reranks these candiate answers. In particular, authors present various methods for the “reranking” step – RAG-style inference, noisy channel inference, and PoE. Experiment results on NQ, HotpotQA, FEVER, and StrategyQA show that the proposed few-shot internet-augmented LMs make further improvements, comparing to the closed book model, Gopher 280B, under both no-reranking and reranking settings. When evaluating the performance with respect to inference time, the proposed few-shot prompting produces better performances than the closed book model, when using similar inference time (i.e. FLOPS). \n\nThe key contribution of the paper is the simple but novel few-shot prompting for using internet search to improve the large language model under the autoregressive Transformer architecture, thereby confirming that that retrieval-augmented models are more promising and effective than the huge closed book model (i.e., Gopher 280B). \n", "summary_of_the_review": "Overall, this paper is well-written and easy to follow. The proposed idea of few-shot prompting for internet-augmented language models is novel, interesting, and timely proper in the literature of the autoregressive language model. Experiment results are solid, showing improvements over the baseline huge Gopher 280B, and Gopher’s performance can be achieved by much smaller model. The concluded direction of reconsidering towards huge pretrained models reasonably run from the experiments in the paper.\n\nFurther comments: \n\n- In the right of Figure 3, CB 280B is only plotted. But, it would be better or required to plot the results of other smaller CB models.\n- In Table 1, the results may rely on the performance of the retrieval. In addition to using other resources like Wikipedia, can we control the retrieval performance by randomly putting noise irrelevant passages and check the degree of the performance from which the final QA performance start to decrease?  \n\n", "strength_and_weaknesses": "Strengths\n\n- This paper is well written and easy to follow the major idea and content. \n- The experiment results are solid, showing consistent improvements comparing to the huge closed book model. \n- The proposed methods are simple but valuable to be explored, particularly presenting a new approach for the few-shot “in-context” learning in retrieval augmented language model under the autoregressive transformer. \n\nWeaknesses\n\n- Only google search results are used for retrieval. It is not clear whether the improvement come from the high quality of google’s retrieved contents or from the retrieval-augmented method. Other collection sources for retrieval such as Wikipedia or the training corpus of Gopher 280B also need to be used for evaluation, in addition to Google’s results on internet. In other words, the authors further need to whether the improvements over CB (in Table mainly come from by keeping or perfectly memorizing the corpus used for pretraining CB, or whether it comes from the use of new but well-matched corpus that is NOT seen when pretraining CB. \n- The proposed method is applied to the autoregressive transformer using the “in-context” learning. But, it is not clear how the method is extended to the encoder-decoder model like T5. \n- Evaluation datasets are limited. KILT or other knowledge-intensive datasets need to be further used for evaluation. \n- What is the advantage comparing to other few-shot learning of retrieval-augmented models, such as:\nGautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, Edouard Grave, Few-shot Learning with Retrieval Augmented Language Modelsm 2022:\n\n   How the proposed method is complementary applied to the above few-shot learning method? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JVSQ1E-IK0", "reviewer": "Reviewer_YZVR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly tackles open-domain question answering. It uses Google search to obtain web information to prompt large-scale language model for answer generation.", "summary_of_the_review": "Although this paper has weaknesses on insufficient experiments, it has contributions in pushing the research about large-scale LM inference and its combination with web-scale knowledge. Thus my recommendation is marginally above the acceptance threshold. ", "strength_and_weaknesses": "Strength:\n1. This paper utilizes web-scale information through google search for downstream tasks like open-domain QA.\n2. It proposes 4 methods to rerank the generated answers through LM produced probability without fine-tuning.\n3. The proposed methods achieve effective results on 4 downstream datasets. This paper also provides analysis on the model scale and inference time, and the ability to answer questions grounded in different dates.\n\nWeakness:\n1. The method is only tested on the Gopher model. It would make this paper much stronger if it's also tested on other large-scale LMs such as OPT and GPT-3.\n2. More experiments should be done: How does the number of shots k affect the model performance? Can we improve the results by simply increasing k? Another baseline is to simply combine multiple paragraphs into one paragraph and pass into the model instead of generate different answers using each paragraph separately. In this case we hope the LM can produce more accurate answers by joint inference over multiple paragraphs. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GD-pXcWYA1", "reviewer": "Reviewer_H18i", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple semiparametric few-shot prompting model for open-domain question answering. For the retriever component of the model, the model uses google search instead of Wikipedia. To account for the long length of documents, each doc is broken into a list of paragraphs where each paragraph is a sequence of 6 sentences. Next, the paragraphs are re-ranked based on Tf-idf similarity within the document. \n\nThe prompts for each task are static (i.e. does not change for each query) and consist of 15 (evidence, question, answer) tuples. No parameter training is required. \n\nAnother contribution of the paper is to demonstrate that performance of these models can be improved by increasing computing at inference time, i.e., by sampling multiple answers from the model and re-ranking them by the probabilities computed by the model. The paper explores three kinds of ranking - (i) directly estimating p(a|q) by marginalizing an evidence latent variable, (ii) estimating the joint probability of answer and question given the evidence, and (iii) Product-of-Experts which combines (multiplies?) different probabilities from the model. All the probabilities are estimated directly from the model except for p(evidence | question) which is estimated by normalizing the tf-IDF score. The paper should definitely write more to motivate each of the re-ranker, currently, the writing of this part is too short, and especially this is one of the salient contributions of the paper.\n\nThe underlying LM used is Gopher LM with varying number of model parameters (44M, 117M, 400M, 1B, 7B, 280B).\n\nThe model is tested on 3 open-domain QA datasets (NQ, HotpotQA, StrategyQA)  and a fact-verification dataset (Fever). The proposed semiparametric model outperforms its corresponding closed-book counterparts. However, the best performance of the models is far behind the state-of-the-art (which is surprising) and I thought the paper did a bad job of explaining this difference in performance and what could be done to fix it.", "summary_of_the_review": "Given the lack of novelty, and the wide gap in performance w.r.t SoTA and also the lack of discussion addressing that result, I am unfortunately tending towards rejecting the paper. ", "strength_and_weaknesses": "**Strengths**\n\n- The proposed approach of the paper is simple. I also agree with the conclusion that instead of racing towards building the largest model, we should carefully explore the few-shot capabilities of mid-sized models.\n- Apart from the short motivation of the re-ranking methods, the paper is well-written and readable\n\n**Weaknesses**\n\n- My main concern about the paper is its very limited technical novelty. It uses Google search for retrieval and a set of static prompts (per dataset) to query the LLMs. It proposes three re-ranking methods (without much motivation). Using Google search for retrieval and static prompting has been explored before. There also is a lot of work that explores re-ranking. Although the paper claims that obtaining probabilities/scores from LMs have not been explored, there exist works such as Izacard and Grave 2021, ART (Sachan et al 2022) that use scores from LMs.\n- The paper also does not explore the easy-to-try extension of prompting. For example, retrieving similar questions w.r.t a given question. This is easy to try and might work better and I would argue is a more practical prompting strategy than the current strategy of designing dataset-specific prompts.\n- Another concern regarding the paper that I have is that most of the results are pretty well-known - e.g. performance of open-book models are better than parametric models, and Google search is better. One interesting result was Gopher 7B + retrieval was better than Gopher 280B without retrieval for NQ and HotpotQA. However, this result does not transfer to reasoning benchmarks, so I am not sure if this result is widely applicable.\n- Can you explain the product-of-experts and noisy-channel re-ranking in more detail? It would be also good to know the motivation behind applying this.\n- Even though open-book models perform better than closed-book models the overall results when compared to SoTA performance is quite lacking. For example (55.9 v/s 38.4 in NQ, 65.2 v/s 30.3 in HotpotQA). Although it is not required for a paper to achieve state-of-the-art for publication, but I couldn't find any discussion of this wide gap in results and what could be done to fix it. I think the paper would benefit a lot if a section is added which discusses how can this gap be possibly closed.\n- Table 1 shows the retrieval performance of Google search and the text in section 5.1 compares that to DPR and MDR. Can you also report the retrieval performance wrt more recent (and better) retrievers such as contriever, etc.\n- [Minor] The abbreviation LSLM is first introduced in Section 3 and used in a few places later, but it is unclear what it stands for. Large Semiparametric Language Models?\n- [Minor] The second paragraph of Sec A.4 seems to be out of place.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CkN8TsjoIP8", "reviewer": "Reviewer_z3mQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve the use LLMs on open-domain QA tasks, via searching up-to-date information from internet, few-shot prompting, and inference-time computing with the searched-out information. Specifically,\n\n(1) Retrieve information from Google Search.\n\n(2) Input a few existing examples as the prefix to prompt LLM.\n\n(3) In inference time, re-rank the answers with different retrieved paragraphs as evidence.", "summary_of_the_review": "Although the paper is well-written and with plenty of experiments and discussions, the proposed approach is not well-motivated and doesn't have significant originality and looks just like training a model towards the benchmark by tuning some modules in a retrieval-based LM and using a very large LM as the backbone.", "strength_and_weaknesses": "Strengths:\n* The paper is well-written and easy to understand.\n* Plenty of empirical discussion and study.\n\nWeakness:\n* The motivation of this work is unclear. It reads like the most significant contribution is using Google search as the information retriever compared with previous work using other search engines on Wikipedia, etc.\n* The technical contribution is incremental, without significant novelty. The 3 proposed methods can all be summarized as hyperparameter/model tuning on the retrieval-based LMs, i.e., tuning the search engine, input prompting, and ensemble strategy.\n* Although the authors have a lot of discussions on their empirical results, they don't finally deliver impressive insights or conclusions, besides using Google search and prompting performs better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject