Teaching Smaller Language Models To Generalise To Unseen Compositional Questions
Authors: Tim Hartill
Neset TAN
Michael Witbrock
Patricia J. Riddle
Venue: Accepted by TMLR ()

Abstract: We equip a smaller Language Model to generalise to answering challenging compositional questions that have not been seen in training. To do so we propose a combination of multitask supervised pretraining on up to 93 tasks designed to instill diverse reasoning abilities, and a dense retrieval system that aims to retrieve a set of evidential paragraph fragments. Recent progress in question-answering  has been achieved either through prompting methods against very large pretrained Language Models in zero or few-shot fashion, or by fine-tuning smaller models, sometimes in conjunction with information retrieval. We focus on the less explored question of the extent to which zero-shot generalisation can be enabled in smaller models with retrieval against a corpus within which sufficient information to answer a particular question may not exist. We establish strong baselines in this setting for diverse evaluation datasets (StrategyQA, CommonsenseQA, IIRC, DROP, Musique and ARC-DA), and show that performance can be significantly improved by adding retrieval-augmented training datasets which are designed to expose our models to a variety of heuristic reasoning strategies such as weighing partial evidence or ignoring an irrelevant context.

Published in Transactions on Machine Learning Research (08/2023)
Teaching Smaller Language Models To Generalise To Unseen Compositional Questions
Tim Hartill School of Computer Science University of Auckland
thar01 1 Bouuckanddknllnccooc.r
Nesct TAN
atand@07@0auckkanduuii.c.ooo
School of Computer Science University of Auckland
Michael Witbrock
m.uilUroockBouckkaaddccccc
School of Computer Science University of Auckland
Patricia J. Riddle
p. ridddee ac..nz
School of Computer Science University of Auckland
Reviewed on OpenReview: https: // openrevie.. net/ ffrmm?? id= d4 Vr6600jjm
Abstract
We equip smaller Language Model to generalise to answering challenging compositional questions that have not been seen in training, To do propose combination of multi- task supervised pretraining up to 93 tasks designed to instill diverse reasoning abilities, and dense retrieval system that aims to retrieve a set of evidential paragraph fragments. Recent progress question-as wwnssiionnnnwrrinng has been achieved either through prompting methods against very large pretrained Language Models in zero or few-shot fashion, by fine-tuning smaller models, sometimes in conjunction with information retrieval. We focus on the less explored question of the extent to which zero-shot generalisation can be enabled in smaller models with retrieval against corpus within which sufficient information to answer par- ticular question may not exist. We establish strong baselines in this setting for diverse evaluation datasets (StrategyQA, Commonsense(A, IIRC, DROP, Musique and ARC-DA), and show that performance can be significantly improved by adding retrioval-angmented training datasets which are designed to expose our models variety of heuristic reasoning strategies such weighing partial evidence or ignoring an irrelevant context.
1 Introduction
We are inspired by recent progress with pretrained large Language Models (LLM), which when promptod with task demonstrations (Brown al. 2020), instructions (Sanh al. 2022 Wei et al. 2021 Ouyang et al.) 2022) or reasoning chains (Wei et al. 2022), show an ability to answer questions unlikely to have been encountered during training. However diversity of potential applications exist in the broad domain of reasoning systems and considerations such as latency, cost, energy efficiency, physical compute size and internet connectivity requirements are relevant. in determining the most. appropriate approach for given situation.
Rather than encoding all knowledge in the parameters of LLM, an alternative approach has been transform the original question-answering problem into reading comprehension (RC) problem by retrieving

---

Published in Transactions on Machine Learning Research (08/2023)
relevant information for answering particular query from an external corpus, and training smaller amallermmodell (QA Model) to reason over the concatenation of the query and retrieved information to derive an answer 2 e.g. Chen et al. (2117). In this paper we extend retrieval methods as described in section 1.1 in conjunction with a supervised multitask pretraining regime for the QA Model involving 79 tasks for our bascline and 93 tasks for the improved model.
The viability of this approach outside of fine-tuned settings is currently subject to limitations, both in the retrieval component, discussed below, and with respect to the inabilities of smaller language models to perform the reasoning function well as larger models. We aim to quantify performance limitations and evaluate mitigations for some of them.
There are at least two significant challenges in retrieval to be overcome. Firstly, no malter how large the corpus is, there will always be missing information, particularly so in our setting where neither datasets nor corpus have been normalised such that sufficient information is in the corpus to make cach question answerable through deductively valid means. Secondly, as long as humans ask questions with ambiguous references e.g. "Who is the spouse of the Green performer?" (Trivedi et al. 2022a),, retrieval will necessarily be imperfect even where sufficient knowledge exists in the corpus and the retrieval method is otherwise perfect.
We evaluate a method for addressing these issues. Specifically, we measure the effect of adding datasets to our QA Model training regime that are designed to impart heuristic strategies for reasoning to a plausible rather than an entailed answer. We construct these datasets by building contexts for training questions using our retrieval system against a fixed corpus of English Wikipedia paragraphs. The resulting samples ("rotrievallangmentee training datascts", abbreviated to RATD) are included in training our QA Model irrespective of whether they contain partial, full, no evidence. Our approach carries the advantage that a diversity of reasoning strategies may be imparted. Such strategies include ignoring an irrelevant context completely or weighing partially evidential facts; c.g. reasoning toward answering "Do tecnagers always rebel against their parents?" (Talmor et al. 2021) can be aided by the retrieval of knowledge that "Adolescents who have a good relationship with their parents are less likely to engage in various risk behaviours", even though there is no entailment implied.
Generally our method is most applicable to question-answering tasks where the desired answer is short i.c. from a word to a short sentence, and the question itself does not come already supplied with fully evidential context. We also assume that it is possible to retrieve sufficient information from our corpus so as to make a question answerable within a modest sequence length (we limit ourselves to a 512 token maximum) c.g. we are unlikely to be able to answer a question such as "How many songs have a person's name in the title?" even through retrieving every instance is theoretically possible. We focus our study on a set of unseen evaluation datasets that meet the following criteria: (1) Datasets collectively involve diverse textual and numerical reasoning strategies. (2) Questions are generally readily answerable by humans with access (3) Questions tend to be compositional.
(4) Relevant comparison with prior work exists. In regards to defining compositionality, others c.g. (Dankers et al. 2022: Hupkes et al. 2020) have noted challenges in singularly defining compositionality as il relates to NLP; for our purposes we pragmatically define a question as compositional if it is unlikely to be answerable by our QA Model with a memorised answer from a similar training example, and requires reasoning over a context by utilising at least one reasoning operation (e.g. conjunction) involving more than one textual fact, and/or at least one numerical operation involving more than one number.
to web browser and without specialised knowledge.
This criteria leads us to select six evaluation datasets: StrategyQA (Geva et al. 2021 contains commonsense samples requiring diverse multi-hop reasoning strategies. On average samples require content from 2.33 separate paragraphs to answer when considering retrieval from Wikipedia. Musique (Trivedi et al. 2022a is a multi-hop dataset focused on factual questions that require retrieval of content from two to four paragraphs. IRRC Fergusso et al.) (020)) contains questions where an initial paragraph is given and answers depend upon reasoning over this plus one to over four additional paragraphs that must. be retrieved. ARC-DA
We define smaller language models generative Transformer models with 400 million billion parameters, i.c those that are large enough be effective reasoncrs whilst being able perform training and inference with reasonable latency, cost and energy efficiency. 2oode, models and datasets will be released at
2

---

Published in Transactions on Machine Learning Research (08/2023)
Bhakthavatsalam et al. 2021 is question-only subset of ARC (Clark et al. 2018 The Worldtree database provides explanatory fact sets for ARC samples which average six facts per sample (Xie et al.) 2020). DROP (uu et al. 2019) is a RC dataset wherein answering each question requires numerical or temporal reasoning over a provided context to reach an often abstractive answer c.g. "How many field goals were scored in the first quarter? The Lions scored wirst...with a 23-yard field goal...he Buccaneers tied it up with 38-yard field ghen took the lead...The Lions responded with a 28-yard field goal..." " The answer is 3 which isn't explicit in the context. CommonsenseQA (Talmor al. 2019) contains samples that are often unlikely to be answerable by finding singular fact e.g. "I'm crossing the river, my feet are wet but my body is dry, where am I? (A) waterfall (B) bridge (C) valley (D) bank (E) island" is answered by considering knowledge related to each option. These datasets are discussed more fully in section 3.1
In addition to the possibility of answer leakage from directly memorised samples, it has been shown that models are able utilise more subtle cues such as the writing style particular annotator who contributed to both train and test splits for better results than are achievable where the test split truly independent of the training split (Geva et al. 2019). To minimise such issues well to facilitate comparison in similar setting as other zero/fery shot studies which have varying definitions of "seen-ness", unscen question as one from an evaluation dataset that is disjoint from our training datasets. Against this definition, two of our evaluation datasets, ARC-DA and Musique, are "partially seen" as discussed further below.
simply define
In summary the major contributions of this paper are: (A) We offer what is to our knowledge the most comprehensive set of baselines evaluating smaller Language Model zero-shot reasoning abilities published to date. (B) We show that augmenting the training regime with RATD datasets signíficantly improves performance from the baselines. (C) We demonstrate that training for numerical literacy and unanswerability is brittle in the unseen setting in the absence of sufficienlly similarly formatted training examples. (D) We propose effective extensions to the retrieval approach as described below.
Query Query
Top-z paras,
Retriever
Query, para,
Query Encoder
Top-k paras,
Reranker
Query + top-x sentences
No
Encoded Paragraphs
Para Relevance Scores; Sentence Relevance Scores,
Evidence Set Scorer
t= m
Evidence Set Score;
Sentence Relevance Scores
Yes
Queryyo
Query top-x Para Fragments,r
Multitask-trained QA Model
Answer
The Iterator (green boxes) and QA Model. An initial query for hop t=0 is input into the Retriever. The Reranker scores each of the retrieved paragraphs and constituent sentences. Top-z sentences (Evidence Set.gr) are selected from top-ranked sentences from the Reranker and from the prior hop Evidence Setzee The query Evidence Setge are input into the Evidence Set Scorer which computes an overall Evidence Set Relevance Score and individual sentence relevance scores. Paragraphs associated with the top five sentences of Evidence Setgr are appended to the query and the process repeats tmax times, Finally, paragraph fragments recovered from the Evidence Set for hop t-arg max(e) are concatenated with the original query and input into the QA Model for answer generation.
Figure 1: Major system components:
1.1 System Components and Related Work
Retrieval. Chen et al. 2017) first used sparse retricval, namely TF-IDF (Spärkk Jones 1972 , against Wikipedia in the context. of open domain question-answering, In dense retrieval, query and corpus documents are embedded into the same vector space and retrieval is typically performed through maximum inner product scarch (MIPS) over the resulting dense vectors. Several such approaches c.g. Karpukhin et al. (2020) focus on retrieving the single most relevant document sufficient for answering single-hop query. Xiong et al. 2222) introduce multi-hop dense retrieval (MDR), to retrieve mulliple documents necessary to answer
3

---

Published in Transactions on Machine Learning Research (08/2023)
complex multi-hop question. They focus on the two-hop situation where a maximum of two documents are sufficient to answer a question. In this situation training samples are input to a shared question and document encoder as: (1) Input (q)) with an objective of minimizing distance to the vector representing d (hercafter denoted (qs) + dio), where qi and d, are the i-th question and the t-th supporting document of g: to retrieve respectively. (2) Input (4:, (4,)) - d We extend the MDR training regime and loss computation to enable retricval of an arbitrary maximum number of documents i.c. (q: di,o,..,,d... + d
Wang et al. (0188) introduced the concept of Reranker that refines retrieved results. IRRR (Qi et al,, 2021 combined sparse retrieval and reranking into an iterative single model that can also answer multi-hop questions that have extractive answers. Baleen (Khattab et al.) 2021), is also iterative but uses a dense retrieval system based upon encoding a dense vector per input token. Their two-stage condenser system comprises a Reranker that scores the relevance of each sentence for each retrieved document followed by an additional module that scores relevance of each sentence from the top-scoring sentences selected over multiple documents from the first stage. It then generates a compressed context of relevant sentences, to be utilised by a separate QA Model. We lake inspiration from Baleen's 's two-stage approach but other than differ most notably in that we introduce an Evidence Set Score into the second
using our own retriever, stage with the goal of quantifying the sufficiency of the entire set of selected sentences for answering a query, in addition to scoring the relevance of individual sentences.
Sparse retrieval offers the advantage that it can perform well in zero-shot settings where lexical overlap is sufficient to identify relevant documents. Several studies evaluate methods that improve the performance of dense retrieval models in zero-shot settings. A number of these use diverse unsupervised techniques involving
creating queries and positive passages from unlabelled text c.g. (Lee et al. 2019: Ram al.] 2022 Izacard
et al. 2022). In a different approach, Chen et al. (0221) I) trained a dense retriever to imitate lexical- bassed model with good results. TTakur et al. (2021) created the BEIR benchmark to further the study of retrieval in the zero-shot setting and a number of more recent papers report results against this benchmark. We are unable to do so as some of our retriever training datascts are BEIR components, however we note as a future direction that our retriever training might benefit further from applying techniques that have been effective on BEIR.
Multitask Pretraining. Raffel et al. (2020) showed that when trained using self-supervised pretraining followed by supervised multitask training, a single lext-lo-texi Transformer model without task-specific architectural modification was capable of performing well on all the diverse tasks it had been trained upon.
Since then, a number of studies have shown the efficacy of supervised multitask training in facilitating
generalisation in question-answvering tasks (Khashabi et al.] 2020b Sanh et al. 2021 Wei et al. 2021
Khashabi et al. p022). Orthogonal to our approach, many studies e.g. [anh et al. (2021) ) Wei et al.
(2021
Ouyang et al. (2022) make use of instruction-based tuning to facilitate generalisation. In order to focus on
evaluation of differing training data regimes, we make use of similar fixed prompting format to Khashabi
et al.) ct al. 2022b combines multitask training over multi-choice datasets with external retricval which they use to augment the training set. However their implementation diverges from ours in that they use sparse retrieval and then a fusion- based method similar to Izacard & Grave (2021) wherein multiple retrieved document vectors are used with gated cross-attention to focus on salient information. Their evaluation datasets are disjoint with ours and don't cover broader reasoning skills like numeracy, so comparison must be left to future work.
2020b 2022) and utilise many of their converted QA datasets. Perhaps most similar to our work, Wang
Longpre et al. 2022 1 created a synthetic dataset by substituting entity names in existing dataset contexts and updating corresponding labels to produce new unfactual but logically consistent samples. They show that training on the new dataset plus the original causes their model to rely on reasoning over the context more, and less on knowledge encoded in parameters. Recently, Li et al. (002)) extended this approach to a fine-tuning framework for LLMs wherein the model is trained on relevant, irrelevant, and counterfactual but logically consistent contexts. Their approach differs from ours in that our RATD datasets are constructed so as to encourage reasoning to a plausible conclusion whereas theirs are constructed with logical entailment in mind i.e. to ignore contexts where deductively valid reasoning is not possible in favor of knowledge stored in the LLM parameters.
4

---

Published in Transactions on Machine Learning Research (08/2023)
Numerical Literacy. Yoran et al. 2022). Pi et al. (2022) and Geva et al. (2020) all develop numeracy- focused pretraining datasets that we adapt and utilise. Generally these approaches have concentrated on finetuned settings and to our knowledge we are the first to study their performance against a diversity of unseen evaluation datasets. Recently Trivedi et al. (202bb) released numeracy-focused pre-training datascts constructed using QDMR (Wolison et al.] 2020) decompositions. These were released too late for us to include in our pretraining regime but we report comparisons Table 2
2
Method
We develop and train the Retrieval, Reranking, Evidence Set Scoring (collectively the "Iterator"), and QA model components separately as visualised in Figure | Comparisons with retrieval systems in our setting are limited since gold paragraph annotation does not exist. Moreover, excepting Khashabi et al. (2020b; 2022 papers tend not to report zero-shot results for smaller language models such as the BART [Lewis et al. 2020 QA model we use. Therefore we initially evaluate the performance of components on in-domain settings with comparisons to strong prior work, and report results this section. In subsequent sections we move the major focus of our study, namely to evaluate our method of adding RATD datasets improve reasoning in the sctting where questions are unseen, sufficient evidence to deductively answer query may not be retrievable, and the model is too small to effectively answer open domain questions without a context to reason over.
2.1 Retrieval
For the retrieval component of the Iterator, as discussed above we extend MDR from a two hop maximum to enable training on samples with an arbitrary maximum number of hops (fmux). Training is over a mixture of datasets with questions involving one to four hops to answer; HotpolQA (Yang el al.; 2018). Hover (Jiang et al., 2020 : Natural Questions (Kwiatkowski et al., 2019), and Musique (Trivedi et al. 2022a Hence in practice we set fmax = 4. Multi-hop questions contain multiple possible reasoning paths through the labelled gold paragraphs, some of which the encoder is able to learn to generalise from (*learnable") and some not (Xiong et al.) 2021). For example, given a set of supporting documents for a 4-hop as {di,o, di,1, d1,2, d semantic overlaps between qi and the documents might enable learnable reasoning paths of (qi, di,,, di,1, d1,2, d1.)) or (g,, d,,1, d1,0, dr3, d1,2) but not (qi, d1,2, di,,, d,,, d1,3) or others. Our training regime samples learnable reasoning path and builds training samples for subsets; e.g. from (0,,,,1,4 d,,, d,,,,,,,,, d12) we would build four single-hop samples (q:) + dr,, (gr, dr,1) - di,; (gi, d,,1, dr.)) di,3 and (gi, di,1, di,o, d1,3) + di,2. We based document sequencing for learnable reasoning paths for Musique on the decomposed reasoning steps provided with that dataset. For HotpotQA and Hover we used the ordering that was used in Xiong et al. 2021 and Khattab et al. (2021) respectively, while Natural Questions is treated as single-hop.
For each training sample, positive documents from other training examples in the batch are used as negatives, to which are added two adversarially negative paragraphs specific to that question. Where adversarial negative documents were not otherwise available we created them from our Wikipedia corpus by taking used this
the first paragraph of directly hyperlinked documents from each gold paragraph. Specifically, strategy to create negative documents for Hover as well as to create additional negatives for Musique. We used adversarial negatives for HotpotQA and Natural Questions supplied from Xiong et al. (2021) and Karpukhin et al. (2020) respectively.
Our objective function similar to others e.g. [ioong et al. 2021 Karpukhin et al. 2020). For hop of the th training sample it models the probability of each next document given a query as:
exp(dvecuu+1 .uec,,.)
P(dvec,.=+190eca.t) Edveog D. exp(dvee- queci,))
Where quee,, = encl(gi, d. d, dvec,r+1 = (dc((4).+))), enc the shared encoder, queq,, is the encoded query vector, dvec,,+1 1 is the encoded next document vector, D2 is the set of positive and negative document. vectors for zi and denotes the inner product operation.
5

---

Published in Transactions on Machine Learning Research (08/2023)
2.2 Reranking and Evidence Set Scoring
To refine retrieved documents we implement a two-stage system comprising Reranker and Evidence Set Scor- ing modcls. Both models were trained using a mixture of datascts that come with sentence-level annotations, namely HotpotQA, Hover and FEVER (Thorne et al. 2018).
Training samples for the Reranker are built from learnable reasoning paths. For two-hop paths, samples are randomly built to one or two hops i.c. (i,) to score dı, relevance, (g: dio, to score dı, To remediate imbalance in hop distribution three and four hop samples are always built to the respective maximum hop count. Each query is paired with both positive paragraph to be scored, and a substituted negative paragraph. The sampling function implements a form of shared normalization (Clark & Gardner 2018) such that pairs are positioned in the same training batch.
In the Reranker, a paragraph relevance score (p) in addition to individual sentence relevance scores (sp) are learned. The objective function for each is binary cross-entropy with the overall loss being an unweighted summation. Intuitively, high-scoring sentence in relevant paragraph is more likely to be evidential than high scoring sentence in an irrelevant paragraph. We manually observed that p is often more accurate than Sp and hence experimented with tuning a weight, w, in a sentence scoring function = wp + (1 w) For in-domain datasets such as HotpolQA we found non-zero values of w that improved both sentence and paragraph recall by over 2%, and Fl score by over 6%, confirming our observation. However the optimal value of w varied between 0.0 and 0.9 over in-domain datasets and tuning w for any of our unsoen datasets using their gold annotations would compromise our experimental setup. Hence we simply score each sentence in our main experiments as s = 0.5p + 0.58p.
For the second stage Evidence Set Scorer, at each hop t the Evidence Setgt is selected from top-ranked sentences from the Reranker and from the prior Evidence Set if any. The query and Evidence Setgt are input into the Evidence Set Scorer which scores evidence set relevance (e), and sentence relevance (8e) in the context of the evidence set. The sentences for the ++ 1 evidence set are selected by ranking according to 0.5p + 0.5s, and then taking a maximum of five sentences that score over a threshold. The 0.5 coefficients were chosen after a similar evaluation as was done for the Reranker scoring function described above. We observed instances where the evidence set weakened as well where it strengthened with additional hops, so we then take the evidence set from hop = arg max(e) rather than assuming that kmax always selects the best.
We observed that a high-scoring sentence is sometimes contextualized by adjacont sentences and collectively they create a stronger rationale. Hence final context for each query, both for RATD dataset creation and for creating context for unseen evaluation samples, is created by recovering a paragraph fragment for each selected sentence by prepending/ prepending/apppedding appending the preceding and subsequent sentence from the associated full paragraph where these exist, and then concatenating the document title with the resulting fragment. Ordering of paragraph fragments is by 0.5p + 0.58max where 8max is the maximum Evidence Set Scorer sentence relevance score per paragraph. Using these paragraph fragments it is possible to fit contexts of approximately 6-7 paragraph fragments within a 512-token maximum sequence length. In the case of datasets such as IIRC (Ferguson et al. 2020) that provide an initial paragraph in addition to the question, the initial paragraph is prepended to the context.
The Evidence Set Scoring model is trained with Evidence Sets built as combinations of positive and nega- tive sentences, including replacing positive sentences with negative sentences from positive paragraphs and negative sentences from negative paragraphs. Each question is paired with both fully evidential set of sentences and a partially evidential (or non-evidential) set of sentences sampled such that pairs are in the same training batch. The objective functions for both e and 8, are binary cross-ent and as with the Reranker the final loss is an unweighted summation. The label for e is 1.0 if a subset of the Evidence Set is fully evidential, 0.0 otherwise.
Further details of the Iterator components are in Appendix D
6

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: In-domain Retrieval and Reranking Evaluation on Hover development set with & 25. Baleen is finetuned Hover, MDR is trained HotpotQA, and our retriever is trained on mixture of HotpotQA, Hover, Musique and Natural Questions.
Sentence EM
Sentence F1
Model / # of Hops->
3
All
2
3
4
All
Balcen 4-hop + FLIPR retriever
47.3
37.7
33.3
39.2
81.2
82.5
80.0
81.5
Tterator + MDR retriever
64.6
39.3
14.8
40.1
81.7
72.1
59.0
71.4
Iterator + our retriever
66.7
45.4
27.5
46.8
82.5
75.7
68.7
75.8
2.3 Iterator In-domain Evaluation
We initially evaluate performance of the Tterator an in-domain setting using the Hover development set against the HotpotQA Wikipodia Abstracts Corpus (Yang et al. 2018), since Hover contains samples with up to four hops and it possible to compare against. the published Baleen performance. Here the number of paragraphs retrieved each hop (k) is 25, Results (Table [ indicate that our Iterator is competitive with Baleen in this setting with our two-hop performance better but their four-hop performance dominating, A reason we are stronger overall than Baleen on EM while the reverse is true for F1 is due to our choice of ranking function Baleen ranks sentences entirely using 8e whereas we utilise a linear combination of our Reranker paragraph score and se. Unsurprisingly our retriever performance is progressively better than MDR as the number of hops increases.
Our main experiments below corpus consisting of English Wikipedia paragraphs from the August 1 2020 dump. Details are in Appendix C
2.4 QA Models
A number of studies have shown the efficacy of supervised multitask training facilitating generalisation
question-answert tasks (Khashabi al. 2020b] Sanh al.) 2021 Wei et al. 2021 Khashabi et al., 2022
We adopt this approach for training our QA models.
To facilitate numerical computation we adapt the QA model tokenizer for digit tokenisation Wallace et al. 2019
Geva et al. 2020) in all experiments.
Noting that some the numerical pretraining tasks take much longer to train to a reasonable degree of proficiency than our textual question- aantiionnnnwwringgg tasks, we contimue training our QA models from their original pretraining checkpoint with two additional stages of multitask pretraining.
2.4.1 Stage 1 Pretraining
In Stage train using tasks that are aimed at imparting by abstraction diversity of foundational reasoning skills, with a bias towards simple numerical literacy. Specifically we utilise existing tasks from Yoran et al. (002)), Pi et al. (2022) and Geva et al. (2020) as well as some we create ourselves. Stage 1 training is on a total of 33 tasks. One these a version of the original self-supervised masked language modelling task which is sampled with probability x 0.35. The remaining tasks are sampled using an error-basod sampling regime (Gottumukkala et al. 2020 whereby tasks with low accuracy in the previous validation step are oversampled in the subsequent training steps and vice-versa.
2.4.2 Stage 2 Pretraining
In Stage 2, we add five open domain (i.c. question-only) question-answering tasks to the above foundational Stage lasks (for 38 lasks in lotal, denoted Group 1). We add the open domain lasks with the primary aim of teaching the model about the expected form of answer for a given question type e.g. yes or no for "Could an Aardvark use a knife and fork?" noting that has been shown that smaller models cannot learn such open domain tasks well (Lewis et al.] (021). To avoid the possibility of catastrophic forgetting, we continue to train Group in conjunction with new set of tasks, Group 2, which is sampled with À

---

Published in Transactions on Machine Learning Research (08/2023)
0.8. Group 2, described further below, contains tasks aimed at teaching more question-answeriin specific reasoning skills, with a bias towards RC datasets.
Our purpose in having two groups is to enable us to implement differing sampling strategies within a single training regime. For Group 1 we utilise uniform sampling over all tasks and for Group 2 we use error-based sampling. This combination represents our solution to the issue noted in Yoran al. (2022), namely that excessive oversampling will occur for tasks that the model cannot learn well. In addition we find uniform sampling useful for regulating the sampling of the tasks that the model has already learned in Stage 1.
2.4.3 Base and R Models
We now discuss two resulting models, both continuing training from the best Stage checkpoint and using the same Group 1 tasks but different in Group 2 tasks.
The first, our Base model, uses 41 tasks in Group 2 for an overall total of 79 tasks (38 Group 1 + 41 Group 2). Group 2 consists of a diverse range of question--nswering datasets. Of note, to facilitate an ability to identify relevant information and perform deductively valid reasoning, for HotpotQA, Hover, FEVER, Musique, Natural Questions, CREAK (Onoc ct al. 2021) and TriviaQA (Joshi et al. 2017), we construct fully evidential contexts with many irrelevant distractors using a combination of gold and distractor paragraph fragments such that we truncating sentences. Since some evaluation samples have label of "unanswerable", of HotpotQA, Hover, FEVER and Musique by similar construction to the fully evidential samples but with key gold sentences or paragraphs removed. These are assigned an "unanswerable" label.
close to our maximum sequence length of 512 tokens as possible without
also create versions
For our second model, Group 2 consists of the tasks in the above Base Group 2 plus an additional 14 RATD datasets for a total of 55 tasks. Our resulting Base+RATD model is thus trained on a total of 93 tasks (38 Group I + 55 Group 2). As described above, the RATD dataset contexts are constructed using our Iterator against the full Wikipedia corpus. Recalling that none of our original datascts against the version of Wikipedia we use, the resulting contexts are noisy, often containing partial or no relevant evidence and many distractors. We hypothesise that the utility of these is to impart variety of heuristic strategies using a context form similar to that which our downstream unseen evaluation datasets will have. Thus our Base+RATD model may be equipped for reasoning to a plausible answer from partial information as well as the deductively valid answer derivable for the majority of dalasets used to train the Base model.
normalisod
Details of all datasets utilised in QA Model training are in Appendix
2.4.4 QA Model In-domain Evaluation
Table 2:
Comparison of our QA Model performance to related pretraining methods finetuned setting
DROP dev set and IRCC test set (F1). Pi et al. (2022); Yoran al. (2022) trained without digit
tokenisation; from Trivedi et al. 2022b) wherein PReasM retrained with digit tokenisation; d Trivedi
et al. 2022bb
Pretraining Regime
DROP
IRCC
IRCR
POET-SQL (BART)*
82.2
PReasM (T5-large)*
72.3
75.0
45.1
PReasM w w/digit tok. (T5-large)
80.0
73.3
40.9
PReasM + Teabreac (T5-large)
83.2
77.9
47.6
Teabreac (T5-3B)
86.7
79.5
51.0
Ours: Base (BART)
79.2
"80.2
53.6
Ours: Base+ Base+RATD RATD (BART)
79.6
80.1
52.8
For comparison with related approaches, we fine-tune our models on DROP (Dua et al. 2019 and soparately on IRR, and IRRR (Ferguson et al.) 2020). IIRG; an oracle setting, with context consisting of gold sentences and surrounding text. IRRCR has retrieved context using respective retrieval methods from each
8

---

Published in Transactions on Machine Learning Research (08/2023)
study. As shown in Table 2 competitive with other approaches in this in-domain setting: We are slightly behind on DROP compared to POET (Pi et al. 2022) and Teabreac (Trivedi et al. 2022b), however we are state of the art on LIRC and IIRCR.
3 Experiments
Our experiments aimed at answering three main research questions:
R1. What is the impact of adding RATD datasets to the QA Model Base training regime?
R2. How effective is pretraining for numerical literacy in the unseen setting for smaller language models?
R3. What is the performance differential between our QA model with differing evaluation dataset context configurations and high-performing comparisons in a similar unseen setting?
For each evaluation dataset, where possible we report our results against other zero/ zero/lew-shol /ew-shot work. If known, also report the current state of the art. As applicable for each dataset report results without retrieval, with our retrieval (denoted DatasetR), and with gold context (denoted Datasete or similar).
To facilitate comparison against prior work on DROP (Dua el al., 2019) and IRCC (Ferguson et al. 2020 we use the numeracy-focused F1 calculation introduced in Dua et al. (2019) whereby if the gold label is a number, the predicted answer must contain that number irrespective of other token overlap. For consistency we retain this method for reporting F1 for other datasets noting this is equivalent the gold answer is not number and disad vantageoss to our results where the gold answer is number. For datasets with binary labels adopt the calculation used in Khashabi et al. (2020b) where to count a match the predicted answer must appear in the gold label and the opposing answer must not. For multi-choice evaluation, we take the option with the highest overlap with the predicted answer and then score as exact match.
standard F1 where
Where comparing performance of our Base against Base+ RATD models we use the paired bootstrap test [Efron& & Tibshirani 1993) to test for statistical significance (p < 0.05).
We report results against the dataset splits commonly reported by relevant prior work.
3.1 Unseen Evaluation Datasets
StrategyQA (Geva el al. 2021b contains binary-labeled commonsense samples requiring a diversity of multi- hop reasoning strategies. The form of questions is generally implicit, meaning they do not leak information as to how they could be docomposed (c.g. "Did Aristotle use a laptop?" versus "Was Aristotle alive at the time that laptops were invented?"). Many samples involve reasoning to a plausible rather than an entailed conclusion even where gold paragraphs are provided (Tiang et al. 2022) e.g. "Is greed the most prevalent of the Seven Deadly Sins?". To facilitate comparison with other zero-shot approaches we use the full training set for evaluation per BIG-bench (Srivastava et al.) 2022 (denoted SQA for question-only and SQAR for question plus our retrieval). We also report results with two forms of gold context; using the provided summary notes which have a short paragraph, rationale-like form (SQAcr), and using the full paragraphs from each of three annotators (SQAcP)- for brevity we report the mean score over the three gold paragraph sets.
CommonsenscQA Talmor et al. 2019 (CSQA) a 5-way multi-choice dataset of commonsense questions derived from Conceptnet (Speer et al. 2017). Many of the questions involve commonsense knowledge that is a river can you hold cup upright to catch water
unlikely to be retrievable from generic corpus ("Where on a sunny day"). However retrieving specific related examples such as "At the river, I filled my cup at a waterfall" may sometimes be possible (Piktus et al.) 2021 CSQA augmented with our retrieval is denoted CSQAR-
DROP (uua et al. 2019) is a RC datasct wherein answering each question requires simple numerical or temporal reasoning. Questions only make sense with the provided gold paragraph so we do not perform retrieval. Answers may be numbers, dates text spans.
9

---

Published in Transactions on Machine Learning Research (08/2023)
IIRC (Ferguson et al. 2020) contains questions where initial paragraph is given and answers depend upon this plus additional paragraphs that must. be retrieved. Each sample is provided with links to all supporting documents, and prior work leverages these to restrict the number of documents to be retrieved from. We instead use our Iterator to augment samples from the full Wikipedia corpus using the concatenation of question and initial paragraph as the query, without reference to the given links (IRCCR). We also report comparison against an oracle context (ITRCc). Answers may be numbers, binary, text spans or labeled unanswerable. For IIRC unanswerable samples, we construct contexts using the initial paragraph fragment plus 1-2 random distractor paragraphs.
ARC-DA (Bhakthavatsalam et al., 2021 is a question-only subset of ARC (Clark et al. 2018 where questions have been re-worded to make sense in an open domain context. The original multichoice versions of ARC are part of our training regime, hence compositionality doubtful and samples are only partially unscen in the sense that the question format is different (and we use the test split). Nonetheless we report results in the interests of exploring diversity. We experiment with Iterator-augmentod (ARCDAR) versions as well as with gold context that we construct from Worldtree (Xie et al. 2020) (ARCDAG).
Musique (Trivedi et al. 2022ah is multihop dataset constructed by combining single-hop questions from existing datasets including SQuAD (Rajpurkar et al. 2016) which is also part of our training regime. More- over we utilise the training split of Musique in both our retriever and QA model training. However the provided development split has been constructed such that for all samples no single hop question, answer, or associated paragraph is common to the corresponding element of any training sample. Therefore we construct a new development set from the training set and experiment with the original Musique develop- ment split as "partially seen", this time where the form of questions is "seen" " but the exact questions are not. Prior work generally uses specialised retrieval for Musique where selection is from the set of gold and distractor paragraphs provided for each sample. We experiment with our retrieval (MusiqueR), and with a gold context constructed from gold paragraphs (Musiquea).
3.2 Models
The Retriever component of the Iterator is built upon RoBERTa- base (Liu et al. 2019) and both the Reranker and Evidence Set Scorer use ELECTRA- reported against the same two final QA Models which based on BART (Lewis et al. 2020). All models use the the Huggingface (Wolf et al. 2020 implementations.
|Clark et al. 2020). Unless noted otherwise, all results are
3.3 Experimental Results
3.3.1 StrategyQA and CommonsenseQQ
Rase+RATD significantly outperforms Base on Strategy QA (Table 3 On SQAR (which uses our retrieved contexts) our much smaller RRATD model slightly exceeds performance of the two 11 billion parameter models and is comparable with OPT 175B (Zhang et al. 2022).
Our Base model fails to improve with SQAer (which has contexts of gold paragraphs) versus the question- only SQA version. The improvement on SQAer with the addition of RATD draws attention to the fact that outside of our RATD datasets the majority of our multihop training samples are aimed at imparting deductively valid forms of reasoning which, as noted above, are often inapplicable for SQAGP.
As described in section 3.1 the contexts of SQAGF are of a condensed, rationalc-like form, distinct from the standard verbose paragraph form of SQAGP. Model performance on SQAGF hugely outperforms our other configurations. This shows that with a context of form the model has learned to reason with, it is possible to solve challenging implicit questions. As to where our models may have learned to reason with this short context form we note that some of the training datasets contain similar short form contexts e.g. BoolQ CCark et al. 2019b), which like StrategyQA has binary labels. Our Base model has 84.9% development set. accuracy on BoolQ.
As Table 4 shows, augmenting CommonsenscQA samples with retrieval (CSQAR) yields mixed results. Others e.g. Piktus et al. (2021) have observed that the best zero/ few shot performance on this type of
10

---

Published in Transactions on Machine Learning Research (08/2023)
Table 3: StrategyQA QA performance comparison (Accuracy). StrategyQA contains binary-labelled, multi-hop commonsense questions. Bold figures denote the better of our two models. All Base versus Base+ RATD differences statistically significant. *Wann al. (2022a); Tay al. (2022); c *Chowdhery al. (2022): from Taylor et al. (2022): Sanh et al. (2021]: Khashabi et al. (0022) Below-randond performance
our Base model with Q +ererricval is due to the model predicting text other than yes no. Prepending "Yes
or no _ to each question improves the score from 48.4 to 54.9. The corresponding Base+RATD figure is
58.8 which retains statistical significance.
Model
Params
Base
Base- RATD
Random PaLM COT-Salf-cons.""
50.0
50.0
540B
81.6
U-PaLM - shot
540B
78.3
PaLM shot
540B
73.9
OPT shot
175B
58.5
T0++
11B
54.4
UnifiedQA v2
11B
57.9
PaLM 5 shot
8B
55.4
UnifiedQA v2
770M
51.6
Ours: SQA
440M
51.6
53.9
Ours: SQAR (Our retrieval)
440M
48.49
58.9
Ours: SQAGF (Gold facts)
440M
72.8
71.2
Ours: SQAgr (Gold paras)
440M
51.6
55.8
Table 4: CommonsenseQA development set performance comparison (Accuracy). CommonsenseeQA contains
multi-choice commonsense questions. Bold figures denote the better of two models. Base+RATD
improvement is statistically significant for CSQA but not for CSQAR (adding retrieved context improves
Base but not + RATD).
Xu et al.
2021
Chowdhery el al. 2022
Brown el al. 2020
Khashabi
ct al. 2020b
Model
Params
Base
Base Base+RAAD RATD
Random
20.0
20.0
Prior work (finetune))
418M
91.2
PaLM 0/5 shot
540B
69.2/81.5
GPT3 0/few shot
175B
81.5/85.0
UnifiedQA v1d
11B
76.2
PaLM 0/5 shot
8B
66.0/77.6
GPT3 0/few shot
760M
61.8/62.7
UnifiedQA vl
770M
60.9
Ours: CSQA
440M
61.1
64.0
Ours: CSQAR (Our retrieval)
440M
62.4
63.6
dataset has been achieved with much larger models rather than external retrieval and our analysis bears this out.
The addition of extra reasoning strategies via the RATD datasets more successful; as with Strategy performance on Commonsense(A is improved with the Base+RAme RATD model.
3.3.2 DROP and IIRC
As with PaLM, our Base and Base+RATD models are trained using digit tokenization. On DROP both our models outperform all models not trained using this method including GPT3 175B and InstructGPT 175B Ouyang al.) 2022) (Table 5). Performance of our models approaches that of PaLM 8B and PaLM 540B in the zero shot setting but both superior to with 5-shot prompt.
11

---

Published in Transactions on Machine Learning Research (08/2023)
Table 5: DROP development set performance comparison (F1). DROP primarily tests numeracy in reading
comprehension. Reduced performance on Base RATD versus Base statistically significant. (Chowdhery
al.
2022;;
Brown et al. (2020); OOuyang al. (2022); Khashabi al. (2020b)
Model
Params
Base
PaLM - 0/5 shot."
540B
43.7/70.8
GPT3 - 0/few shot
175B
23.6/36.5
InstructGPT PPO+ptx 0/few shot
175B
15.2/33.3
UnifiedQA v1d
11B
32.5
PaLM 0/5 shot
8B
45.1/69.4
UnifiedQA vl
770M
24.6
GPT3 0/few shot
760M
14.4/24.0
Ours
440M
40.7
40.0
Ablative experiments on our training regime components (Table indicate that digit tokenization, numerical literacy training datasets and two stage training are all important achieving the best DROP performance in our setting.
Table 6: DROP development set (F1). Ablative results on our QA models trained using Base+ RATD datasets trained one or two stages, with/ without digit tokenization (+/-DT), and with/without numerical literacy training datasets (+/-NumLit). Note that the -NumLit setting is only relevant for single-stage
training.
Model
All Ans. Types
Numeric Ans.
Only
Two Stage: +DT +iumLit
40.0
25.4
One Stage: +DT +NumLit
38.2
22.9
Two Stage: -DT +NumLit
34.7
16.6
One Stage: +DT -NumLit
29.0
11.2
Table 7: IIRC test set evaluation (F1). IRC tests diverse rcasoning requiring retrieval. Both Base to Base+RATD comparisons are statistically significant. a *Ferguson et al. (2022) use a finetuned QA model and specialised retrieval with corpus restricted to documents linked from each initial paragraph. best of our knowledge our Base model finetuned on IRRRR and separately on IIRC, are both SOTA at the time of writing so we report these given unavailability of unscen comparisons.
the
Model
Params Basc
Basc+RATD RATD
Prior work: IIRCR" "
123M
51.6
Ours: Finetuned IRRC (Our retrieval)
440M
53.6
Ours: ARRO (Our retrieval)
440M
23.5
25.5
Ours: Finetuned IRCa (Gold context)
440M
80.2
Ours: TIR (Gold context)
440M
59.6
58.1
Table shows performance IICC. A first glance suggests that poor retrieval is the major cause of low performance IIRCR, however inspection of retrieved items suggests that retrieval is often fully evidential. The breakdown by answer types in Table g indicates that major cause of failure is that in contrast to DROP, almost all numeric answers are predicted incorrectly for both IIRC; (gold contexts) and IIRCR (retrieved contexts). Finetuning alleviates the issue, confirming that the model is capable of performing the necessary computation when trained with sufficiently similar examples.
Our RATD model generally correctly predicts mnnnsweraillity for IIRCs but almost never does for MRCn. The IRCR context. frequently contains eit enough information to make the question answerable.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Table 8: Breakdown by answer type on DROP development set and ITC test set (F1). Sample counts are in brackets. Finetuned models are trained from the Base + RATD chockpoint.
Datasct Ans. Type
Base RATD Finetuned
DROP
Span (2962)
67.4
82.3
Multi-span (567)
42.0
72.2
Num (5850)
25.4
79.0
Date (157)
62.4
74.0
All (9536)
40.0
79.6
URCG
Span (5444)
59.8
74.3
Binary (66)
57.1
64.7
Num (277)
2.9
67.4
No answer (414)
92.8
98.8
All (1301)
58.1
80.1
IRCC
Span (544)
48.9
44.8
Binary (66)
68.2
57.6
Num (277)
3.8
41.5
No answer (414)
2.6
69.9
All (1301)
25.5
52.8
or more frequently such relevant information as to make it appear answerable. Similar to the numerical com- putation issue, adding sufficiently similar training examples via finetuning enables the model to distinguish unanswerable samples. Appendix G illustrates failure cases for numeric and unanswerable types.
3.3.3 ARC-DA and Musique
Table 9: ARC-DA (Lest accuracy) and Musique (development. F1) comparisons. ARC-DA is science question answering and Musique involves multi-hop question answering. All Base to Base+RATD differences are statistically significant. Musique performance degradation Base+RATD caused by adding Musique RATD in training; results for an ablative model trained with all dalasets excepl for Musique RATD is shown
in brackets. Bhakthavatsalam et al. (2021) b: Training includes ARC-DA. à Trivedi et al. 2022a): EX(SA)
uses specialised retrieval from each Musique sample's 's gold and distractor paragraphs.
Model
Params
Base Base+RATH
UnifiedQA ARCDAA// with IR
11B
61.4
Ours: ARCDAR (Our retrieval)
440M
28.8
31.6
Ours: ARCDAG (Gold context)
440M
56.8
59.1
Musique EX(SA)
102M
49.8
Ours: Musiquen (Our retrieva]]
440M
24.3
22.2
(28.2)
Ours:
Musiqueg (Gold context)
440M
60.8
43.8 (62.4)
Table shows model performance on our "partially seen" datasets, ARC-DA and Musique. On ARC-DA, adding RATD datasets significantly improves results in both retrieved and gold settings. By contrast, Musique performance significantly degrades with BaSe+RATD. Noting that Musique is the only evaluation dataset for which we create RATD datascts, we hypothesise that in the case of highly similar training examples lo particular evaluation samples, the model prediction is the memorised answer of a similar training example. We confirm this by examining the predicted answers of the 1,670 Musique evaluation samples that scored Fl against Mase+RAATh Of these the predicted answers of 716 samples are an exact match Musique training sample gold answer (e.g. "Who is the spouse of the Green performer?" is incorrectly answered "anna gordy gaye" because this is the label to number of training questions of "Who is the
13

---

Published in Transactions on Machine Learning Research (08/2023)
spouse of ..." form). An ablative experiment, wherein trained a version of Base+RATD without the Musique RATD datasets, results in improved performance versus Bose and the original Base+RATH on Musique (Table 9 without material impact to other evaluation dataset results.
The Musique training split has 19,938 samples but only 2,057 unique labels, and questions with the same answer tend to be of similar form, such as the above "Who is the spouse of..." example. Therefore we consider the question of whether the poor performance of Base+RATD here is a general weakness of our method whether it is specific to the particular bias of Musique. We trained another B model, this time with the Musique RATD training dataset substituted with a filtered variation that only contains samples with unique labels. Similar to the above Musique RATD ablation, this version also improves against the original RoSD (+3.0 F1 for MusiqueR and -00.6 F1 for Musiquea) without impact to other results. Hence, assuming appropriate consideration of existing dataset bias when selecting RATD training samples, we affirm the robustness of our method.
4 Conclusion
We have argued that an ability reason over imperfect and incomplete information is a critical skill with which question-answering models must be endowed. To facilitate such ability create RATD datasets that are designed to impart heuristic reasoning strategies with context of form similar to that which retrieved contexts for downstream tasks will have. We show that training on RATD datasets improves performance on all unseen evaluation datasets with retrieved contexts. This sometimes comes at a small cost in situations where questions come with gold contexts that are in a form that our model is already good at utilizing (SQAGF, DROP, and IlRCa) although we suggest. that practice such gold contexts are the less common case. (R1)
We also show that even with our large and diverse pre-training regime, questions involving numerical com- pulation and those labelled unanswerable remain sensitive to the similarily of training samples. (R2)
Our results demonstrate that generic retrieval without normalisation can outperform specialised methods (c.g. we are state of the art on fine-tuned IIRCR) and that our overall method can yicld performance on par or better than that of much larger models without fine-tuning (e.g. SQAR, DROP). (R3)
Broader Impact Statement
In common with the well known issues with larger models, our system is capable of generating hallucinated, false and/or potentially offensive answers. We consider its usage at this stage of development to be most appropriate research environments.
Conversely, latency, physical compute size, cost and energy efficiency are important considerations where smaller models offer material benefits. As noted we suggest that diversity of applications exist in the broad domain of reasoning systems and that due weight be assigned to all factors in determining the most appropriate approach for a given situation.
Acknowledgments
We are grateful to the authors of Pi et al. (2022) for providing us with their POET-SQL dataset and to Omar Khattab for providing us with Hover paragraph sequencing data.
References
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the Al: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662 678, November 2020.
Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer ques-
14

---

Published in Transactions on Machine Learning Research (08/2023)
tion answering? try ARC-DA, the direct-answer A12 reasoning challenge. arXiv preprint arXiv:2102.08915, 2021.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. PIQA:
Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34(05), pp. 7432 7439. Association for the Advancement of Artificial Intelligence, 2020.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Adilya Ramesh, Daniel M Ziegler, Jellrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are Few-Shot learners. In Advances in Neural Information Processing Systems 33, pp. 1877-1901, 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, Reading Wikipedia to answer Open-Domal questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, 2017. Association for Computational Lin- guistics.
Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-Tau Yih. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? arXiv preprint orXiv:2110.0601 October 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Ailor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bremnan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meicr- Hellstenn,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. arXiv preprint arXte:2204.02311 April 2022.
Christopher Clark and Mail Gardner. Simple and effective Multi-Paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 845 845, Mclbourne, Australia, July 2018. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Was Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural Yes/No questions. In Proceedings of the 2019 Conference of the North American Chapler of the Associalion for Compulational Linguislios: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924 2936. Association for Com- putational Linguistics, 2019a.
Kevin Clark, Minh-Thang Thang Luong, Quoc V Le, and Christopher D Manning, ELECTRA: Pre-training text en- coders as discriminators rather than generators. In International Conference on Learning Representations, March 2020.
Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. Combining retrieval, statistics, and inference answer elementary science questions. In A AAAT Conference on Artificial Intelligence, volume 30. Association for the Advancement of Artificial Intelligence, 2016.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafford. Think you have solved question answering? try ARC, the AI2 reasoning challenge. orXiv preprint arXiv:1803.055 2018.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groen- eveld, Michal Guerquin, and Michael Schmitz. From If' to 'a' on the n.y. regents science exams: An overview of the aristo project. arXiv:1909. 01938e3 fcs. CLJ, 2019b.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4154 4175, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-scckinng questions and answers anchored in rescarch papers. In Proceedings of the 2021 Con- ference of the North American Chapler of the Associalion for Compulational Linguisties: Human Language Technologies, pp. 1599--6600 Stroudsburg, PA, USA, 2021. Association for Computational Linguisties.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguisties: Human Language Technologies, Vol- ume 1 (Long and Short Papers), Stroudsburg, PA, USA, 2019. Association for Computational Linguisties.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2368-2378, Minncapolis, Minncsota, June 2019. Association for Computational Linguistics.
Bradley Efron and Robert Tibshirani. An Introduction to the Bootstrup. Monographs on Statistics and Applied Probability, 57. Chapman and Hall, New York, NY, 1993.
James Ferguson, Matt Gardner, Hannanch Hajishirzi, Tushar Khot, and Pradeep Dasigi. IIRC: dataset of incomplete information reading comprehension questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1137-1147 Stroudsburg, PA, USA, November 2020. Association for Computational Linguistics.
James Ferguson, Hannanch Hajishirzi, Pradeep Dasigi, and Tushar Khot. Retrieval data augmentation informed by downstream question answering performance. In Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER), pp. 1-5, 2022.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the lask or the annotator? an investiga- tion of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- pp. 1161-1166, Stroudsburg, PA, USA, November 2019.
ural Language Processing Association for Computational Linguisties.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Compuuational Linguistics, pp. 946 958, Online, 2020. Association for Computational Linguistics.
Mor Geva, Daniel Khashabi, Flad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategics. Transactions of the Association for Compulational Linguistics, 9:346 361, 2021.
Ananth Gottumukkala, Dheeru Dua, Sameer Singh, and Matt Gardner. Dynamic sampling strategies for multi-task reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 920-924. Stroudsburg, PA, USA, July 2020. Association for Computational Tinguistics.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model Pre-Training. In Hal Daumé Ti and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3929 3938. PMLR, 2020.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Reseorch, 67:757-795, 2020.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 874-880, Online, 2021. Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, August 2022.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for Many-Hop fact extraction and claim verification. In Findings of the Association for Computa- lional Lònguistics: EMNLP 2020, pp. 3441 1461.. Association for Computational Linguistics, 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, Stroudsburg, PA, USA, November 2019. Association for Computa- tional Linguistics.
Jeff Johnson, Matthijs Douze, and Herve Jegou. Billion-scale similarity search with GPUs. IEEE transactions on big data, 7(3):535-541) 547, 2019.
Mandar Joshi, Funsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Papers), pp. 1601 1611, Stroudsburg, PA, USA, July 2017. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-Tau Yih. Dense passage retrieval for Open-Domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 252-262. Association for Computational Lingustics, 2018.
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. More bang for your buck: Natural perturbation for robust question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 163 170, Online, November 2020a. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, 2020b. Association for Computational Linguistics.
Daniel Khashabi, Yeganch Kordi, and Hannanch Hajishirzi. UnifiedQA-v2: Stronger generalization via broader cross-format. training. Xix preprint arXiv: 2202. 12359, February 2022.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Omar Khattab, Christopher Potts, and Matci Zaharia. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. In Advances in Newral Information. Processing Systems, 34, pp. 17670-27653 2021.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for ques- tion answering via sentence composition. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34()5), pp. 8082 8090. Association for the Advancement of Artificial Intelligence, 2020.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Compulational Linguistics, T:453-466, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding com- prehension dataset from examinations. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA, 2017. Association for Computational Linguistics.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086 6096. Association for Computational Linguistics, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamod, Omer Lovy, Veselin Stoyanov, and Luke Zeltlemoyer. BART: Denoising Soqueecc--o-Sqqueee
pre-training for natural lan- guage generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, 2020. Association for Computational Linguistics.
Patrick Lewis, Pontus Stenctorp, and Scbastian Riedel. Question and answer Test-Train overlap in Open- Domain question answering datasets. In Proceedings of the 16th Conference of the European Chapler of the Association for Computational Linguisties: Main Volume, pp. 1000-1008, Online, 2021. Association for Computational Linguistics.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and San- jiv Kunar. Large language models with controllable working memory. arXiv preprint or arXiv:2211.000 11.2211111110 November 2022.
Zhengzhong Liang, Tushar Khot, Steven Bethard, Mihai Surdeanu, and Ashish Sabharwal. Better retrieval may not lead to better question answering. arXiv preprint ar Xrxi::2205. 03665,, May 2022.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. Tn Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 58 62. Association for Computational Linguistics, 2019.
Yinhan Lău, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprinl 1iv: 1699,, July 2019.
Shayne Longpre, Kartik Perisetla, Authony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- Based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7052 7063, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Yu A Malkov and D Yashunin. Efficient and robust approximate nearest neighbor scarch using hicrarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4): 824-830 044-836,47 2018.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 2391, Brussels, Belgium, 2018. Association for Computational Linguistics.
Yasumasa Onoe, Michael J Q Zhang, Eunsol Choi, and Greg Durrett. CREAK: A dataset for commonsense reasoning over entity knowledge. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), November 2021.
Long Ouyang, Jefl Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 35, pp. 27730-2773 27744, 2022.
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, and Weizhu Chen. Reasoning like program executors. aixic preprint 1117:2200..1173,, 2022.
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Tzac- ard, Patrick Lewis, Barlas Oguz, Edouard Grave, Wen-Tau Yih, and Sebastian Riedel. The web your oyster knowledge-intensive NLP against very large web corpus. arXiv preprint arXiv:21 12. 12.09924, December 2021.
Peng Qi, Haejun Lee, Tg Sido, and Christopher Manning, Answering Open-Domain questions of varying rea- soning steps from text. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 3599 3614, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified Text-to-Text transformer. Journal of Machine Learning Research, 21:1-67, 67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 2392. Association for Computational Lingustics, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers), pp. 784-789. Association for Computational Linguistics, 2018.
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. Learning to retrieve passages without supervision. In Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 0687-2700, Seattle, United States, July 2022. Association for Computational Linguistics.
Matthew Richardson, Christopher J C Burges, and Erin Renshaw. Metest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 193 203. Association for Computational Linguisties, 2013.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to Al complete question answering: A set of prerequisite real tasks. In AAAI Conference on Artificial Intelligence (AAAI-20), volume 34, pp. 8722 8731. Association for the Advancement of Artificial Intelligence, April 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732 8740. Association for the Advancement of Artificial Intelligence, 2020.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations. 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Stroudsburg, PA, USA, November 2019. Association for Computa- tional Linguistios.
Karen Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11 21, 1972.
Robyn Speer, Joshua Chin, and Catherine Havasi. ConceptNet 5.5: An open multilingual graph of general knowledge. In Thirly-Firss First AAAI Conference on Artificial Intelligence, pp. 4444 0451. Association for the Advancement of Artificial Intelligence, 2017.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Sarriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S Tyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Ben- jamin Inden, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D Man- ning, Christopher Potts, Cindy Ramirez, Clara E Rivera, Clemencia Siro, Colin Ralfel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Free- man, Daniel Khashabi, Daniel Levy, Daniel Mosoguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellic Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gon- zalo Jaimovitch-Iópez, Gregor Betz, Guy Gur-Ari, Hana Galijasevie, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boudeman, Joseph Jones, Joshua B Tenenbaum, Joshua S Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D Dhole,
20

---

Published in Transactions on Machine Learning Research (08/2023)
Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contrera--Ohhando,
Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt,, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkichn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swødrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Varma T Mukund, Nanyun Peng, Nathan Chi, Nayeon Lee, Nela Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S Iyer, Noah Constant, Noah Fiodel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Del- gado, Raphaäl Millière, Rhythm Garg, Richard Barnes, Rif A Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R Bowman, Samuel S Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A Rous, Sarik Ghaz- arian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sncha Priscilla Makini, Soo-Hwan Lec, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T Piantadosi, Stuart M Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, To-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadch, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xix preprint or Xri::22006. 04615, June 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeQA: A question an- swering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Pupers), pp. 4149 4158. Association for Computational Linguistios, 2019.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. CommonsenseeQA 2.0: Exposing the limits of AT through gamification. In Thirly-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), November 2021.
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoe V Le, and Mostafa Dehghani. Transcending scaling laws with 0.1% extra compute. arXiv preprint ar arXi::2210.112 11399, October 2022.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. arXiv preprint DXNi:2211.09088 November 2022.
21

---

Published in Transactions on Machine Learning Research (08/2023)
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirly-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), October 2021.
James Thorne, Andreas Vlachos, Christos Christoddoulopoulss, and Arpit Mittal. FEVER: A large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lánguistics: Human Language Technologies, Volume (Long Papers), Stroudsburg, PA, USA, 2018. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Ka- heer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 191 200. Association for Computational Linguistics, 2017.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop ques- tions via hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022a.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Teaching broad reasoning skills for Multi-Step QA by generating hard contexts. arXio preprint ar urx::22005. 12496, May 2022b.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know mum- bers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 5307 5315, Hong Kong, China, November 2019. Association for Computational Linguistics.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang- R Reinforced Ranker-Reader for open-domain question answering. Tn Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. Association for the Advancement of Artificial Intelligence, April 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency improves chain of thought reasoning in language models. arXie preprint arXiv:2208.11174 March 2022a.
Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, and Heng Ji. Zemi: Learning Zero-Shot Semi-Parametric language models from multiple tasks. arXiv preprint or arXi::2210.001 October 2022b.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In siith Conference on Neural Information Processing Systems (NeurIPS 2022), January 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Stroudsburg, PA, USA, 2020. Association for Computational Linguisties.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break down: question understanding benchmark. Transactions of the Association for Com- pulational Linguisties, 8:183 198, 2020.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Chien-Sheng Wu, Andrea Madotto, Wenhao Liu, Pascale Fung, and Caiming Xiong. QAConv: Question answering on informative conversations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5389 5411, Stroudsburg, PA, USA, 2022. Association for Computational Linguistics.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 3733-3742. IEEE, June 2018.
Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter Jansen. WorldTree v2: A corpus of science-domain structured explanations and inference patterns sup- porting multi-hop inference. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 5456 5473, Marseille, France, 2020. European Language Resources Association.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. TWEETQA: social media focused question answering dataset. Proceedings of the 57th Annual Meeting of the Association for Computational Lònguistics, pp. 5020 5031, Florence, Italy, July 2019. Association for Computational Linguistics.
Wenhan Xiong, Xiang Li, Srini Tyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering complex Open-Domain questions with Multi-Hop dense retrieval. In International Conference on Learning Representations, 2021.
Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengeheng He, Michael Zeng, and Xuedong Huang, Human parity on CommonsenseeQA: Augmenting Self- Self-Attention with external attention. arXiv preprint arXiv: 0125,, December 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Pro- ceedingg of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-226900 2369-2380. Association for Computational Linguistics, 2018.
Ori Yoran, Alon Talmor, and Jonathan Berant. Turning lables: Generating examples from semi-structured tables for endowing language models with reasoning skills. In Proceedings of the 60th Annuall Meeting of the Association for Computational Linguistics (Volume 1: Long Popers), pp. 6016 6031, Dublin, Ireland, 2022. Association for Computational Linguistics.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. ReClor: reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations, February 2020.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD; Bridging the gap between human and machine commonsense reading comprchension. arXiv preprint ar1iv:1810. 12885, October 2018.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zeutlemoyer. OPT: Open pre-trained transformer language models. arXiv preprint ar Xrxi:2200..0 01066, May 2022.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. TAT-QA: A question answering benchmark on hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2277,3287, Stroudsburg, PA, USA, August 2021. Association for Computational Linguistics.
23

---

Published in Transactions on Machine Learning Research (08/2023)
A Hyperparameters
All models are trained on one GPU (either an Nvidia RTX8000 or A100) except. for the Retriever models which are trained on six 80GB A100 GPUs. All models are trained using mixed precision using a linear learning rate decay schodule. Initial learning rates and other hyperparameters are shown in Table [0 The optimiser used for the Retriever, Reranker and Evidence Set Scorer is Adam. All other models use AdamW. A maximum sequence length of 512 tokens was used for all models.
Table 10: Hyperparameters used for each model. Each training step is one batch input ie the number of All final models are selected as the best
optimization steps TrepssngSteps/ model on the development set (s) over the specified number of training steps.
watpa/teepe/eaaeettttooooo ArradientAccmmulationn.ttt
Initial
Batch
Grad. Accum
Train Steps
Model
LR
Size
Retriever
2e-5
150
99K
Retriever memory bank
1e-5
250
59K
Paragraph Reranker
5e-5
8
140K
Evidence Set Scorer
5c-5
8
140K
QA Model Stage
2e-5
32
4
1M
QA Model Stage 2 Base
2e-5
32
4
1M
QA Model Stage 2 Base+ RATD
20-5
32
4
1M
DROP finetuned
2e-5
32
4
260K
NRCe finetuned
20-5
32
40K
IRRCR finetuned
2e-5
32
40K
B QA Model Input Format
We employed simple and fixed input format based on that used in UnifiedQA KKashabi et al. 2020b with minor extensions follows:
Open domain form: question? Ann
Reading comprehension (RC) form: question? Aln (sragraphs))
Multiple choice form: question? An (A) option text a (B) option text
Multiple choice with RC form:
question? Ann (A) option text a (B) option text ... An paragraphs))
We standardised the formatting of any paragraphs paragraph fragments that had associated doc- ument titles follows. Further detail on how such contexts were constructed is in Appendix B
Title 1: Sentence 1. Sentence 2. Title 2: Sentence 1. Sentence 2.
C
Wikipedia Corpora
For experiments aimed at evaluating the Iterator components in an in-domain setting (Table D used the corpus of Wikipedia abstracts from October 2017 that HotpotQA and Hover are based upon.
24

---

Published in Transactions on Machine Learning Research (08/2023)
For main experiments and for various peripheral tasks such as identifying negative paragraphs for retrieval training we start with the August 2020 Wikipedia dump preprocessed by (Qi et al. 2021 We retain all paragraphs with more than seven words, and extract hyperlinks and calculate sentence offsets from each. There are a total slightly over 35 million paragraphs. We note that all results this paper use the original HotpooQA question set rather than the version used in (Qi el al. 2021 that has been normalised against this Wikipedia version.
D Iterator Training Details
D.1 Retrieval Model Additional Details
Our final Retrieval model was trained similarly to Xiong et al. (2021) in that following the initial stage of training, additional training with large memory bank (Wu et al. 2018 of negative paragraph embedding vectors was applied.
For retrieval paragraphs for RATD datasets, the number of paragraphs retrieved at each hop (k) was set 60 as to complete reasonable time. In building unseen evaluation dataset contexts was arbitrarily set 150 to maintain reasonable performance on queries that are very different to those used in retrieval training.
We used FAISS (Johnson et al. 2019) for the search over paragraph embedding vectors. Generally we used an approximate search mechanism, HNSW (Malkov & Yashunin, 2018), except for the Hover experiment (Table D
where an exact inner product search was employed.
D.2 Reranker Model
The Reranker has an input format as follows:
[CLS] query [SEP] yes no [INSUFF] [SEP] title [SM] sentence [SEP]
[SM] sentence 1.
The query component is encoded as:
question [QSEP] title 1 sentence 1. sentence 2. [QSEP] title sentence
Special tokens are utilised as follows:
[CLS]: Trained using onc-layer head to be the Paragraph relevance score with binary cross-entropy objective.
INSUFF]: Insufficient Evidence token, used by the start and end token span predictors that are imple- mented as per Devlin et al.] (2019). Although we utilise separate abstractive QA model, we use the span predictors as a debugging tool and retain this component in the final loss function.
[SM]: Sentence Marker(s). Used to score sentence relevance, Trained using a one-layer head with a binary cross-cntropy objective. [QSEP]: query components separator.
The final training objective is the unweighted summation of the paragraph relevance loss, sentence relevance loss and span loss.
D.3 Evidence Set Scoring Model
This model has an input format as follows:
[CLS] question [SEP] yes no [INSUFF] [SEP] [SM] title
sentence
[SM] title
sentence [SM] title sentence ... [SEP]
25

---

Published in Transactions on Machine Learning Research (08/2023)
Special tokens are utilised as follows:
[CT.S]: Evidence Set score. Trained using a one-layer head with binary cross-entropy. The label is 1.0 if all of the gold sentences from all gold paragraphs are present and zero otherwise. Insufficient Evidence token, as per the Reranker model. Sentence Marker, as per the Reranker model.
INSUFF]:
[SM]:
The final training objective is the unweighted summation of the evidence set loss, sentence relevance loss and span loss.
Following Khattab et al. (2021), the maximum number of sentences in an evidence set was set to nine in all experiments. To select the sentences for constructing the retriever query and evidence set for the next hop a maximum of five sentences over a threshold are selected, also following Khattab et al. (2021). The minimum threshold used to select sentences 0.1 unless fewer than 2 sentences qualify which case the
two lop-scoring sentences are laken.
E QA Model Multitask Training Details
We trained both the first and the second stage for million steps (batches) with the best model defined as that with highest mean exact match accuracy over all development sets. To ensure reasonable elapsed time for each validation step we used reduced development sets where development sets of more than 1250 samples were reduced to approximately 1250 by taking every nth sample with n = round(c/1250) where c is the sample count. validation step occurs every 10,000 training steps.
Table I1 enumerates datasets used in Stage 1 and in Stage Group (those above the dotted line were added for Stage namely CREAK (Onoe et al. 2021), CommonsenseeQA 2.0 (Talmor et al. 2021), TriviaQA
(Joshi et al.) 2017). Natural Questions (Kwiatkowski et al. 2019) and Twenty Question: During Stage 1 training, error-based sampling for these datasets was employed and in Stage 2, uniform sampling.
Datasets names containing the term "opendomain" only use the question text as input and are added with the primary aim of teaching the model about the expected form of answer for a given question type (e.g. yes or no for "Could an Aardvark use a knife and fork?".
Datasets preceded by "preasm" are as provided by Yoran et al. (2022) with reformatting into our standard form. Datasets preceded by "poetsql" are the POET-SQL dataset kindly directly provided to us by the authors Pi et al. (2022). We split POET-SQL into separate datasets based on the type of SQL statement and converted into our standard form.
For the hetic _ datasets we extended the original code provided by Geva et al. 2020 to output
in the variablised form proposed in (Pi et al.) 2022) (e.g. "I + 3" becomes "x + y Vnn x= 1; y=3; z=0; where z is a distractor). Additionally we added two datasets with questions of the form "Is
< between
y [and x)?" for numbers and dates respectively. We generated one million samples for each of the resulting eight datasets. The "synthetic textual" task is as provided by Geva et al. (2000) aside from reformatting into our standard format.
Finally, we created a self-supervised task (enwiki 20200801 selfsvised), by sequentially concatenating para- graphs from documents in our Wikipedia dump until a sequence length of approximately 512 tokens was reached. During training, spans were masked from each sample input based on their being named entities (Guu et al. 2020) or noun phrases with X 0.65, or randomly with À = 1 0.65. The training objective was to prediet just the masked spans as with T5 (Raffel et al., 20200) rather than the original BART {Lewis et al., 2020) objective of predicting the entire unmasked input sequence. A small development set was randomly selected to enable this task to be included with other tasks in model selection.
Table enumerates datasets contained in Group 2 for Stage 2 training. We converted TAT-QA ZZhu et al. 2021 to our format by lincarising the constituent tables. A number of the other datasets (i.e. those whose names do not contain key terms described below) are provided by (Khashabi et al. 2020b 2022). These are:
26

---

Published in Transactions on Machine Learning Research (08/2023)
Table 11: QA model Stage 1 and Stage Group 1 training datasets. All figures are Exact Match on reduced development sets from the single overall best model without per-dataset finctuning. Datasets above the dotted line were added for Stage
Dataset
Base
Base+RATD
creak _onk_oopendoain
76.6
76.1
49.4
51.9
triviaqa open opendomain
8.0
7.4
naturalquestions_pgg open open_opendomaal
5.4
8.7
twentyquestions _oaassomaennnnnnnnnn
88.8
87.9
preasm arithmetic addition
99.6
99.8
preasm auithmetic_superraaee
97.9
97.9
preasm composition
93.4
93.7
preasm composition hop
93.5
93.7
preasm conjunction
80.2
81.0
preasm counting
96.6
96.5
preasm_everr__ee qvert_quantiffer
99.8
99.6
preasm most quantifier
99.8
99.7
preasm cumerriccommarrionn boolean
99.9
99.8
preasm numeric numeric_sspperlative,
98.1
97.9
preasm baassm_only_quniile
99.4
99.4
preasm temporal comparison
93.7
93.0
preasm preasm_tempora comparison boolean
99.8
99.7
preasm tensp_at difference
94.3
95.1
preasm _aaptitttmmpa_a_____ (emporal_superlativee superlatives
97.5
97.1
poetsql _ulti
36.2
34.5
poetsql select abs
84.2
94.0
poetsql select select__aithh arith
89.7
85.1
poetsql select. count
80.8
80.2
poctsql select max
79.6
75.7
poetsql select min
82.5
81.3
poetsql_select sum sum poetsql_single
50.6
52.7
79.4
79.0
arg min max synthetic num date diff synthetic_mum date min max synthetic_mun __im max__ve synthetic num percent
synthetic num
100.0
100.0
82.6
82.7
93.2
95.7
69.3
68.8
99.0
98.2
synthetic_mun sii___iin______eee arith
76.1
78.6
synthetic_ num yn dates pauthtic_num__ve sain__y__ummss nums synthetic textual
99.8
99.8
100.0
100.0
92.4
92.4
enwiki 20200801 selfsvised
22.5
24.1
AdversarialQA Bartolo et al. 2020 ARC CCark et al.
2016
2018
BoolQ (Clark et al.
2019a
BoolQ
NP (Khashabi et al. 2020a). MCTest (Richardson al.
2013), the yes
yes/no /no subset of MultiRC
Khashabi
ct al.
2018), NarrativeQA Kočiský et al. 2018). NewsQA (Trischler et al. 2017). OpenbookQJA
Mihaylov
el al.
2018). PhysicalIQA (Bik et al.; 2020),
PubmedQA (Jin et al.] 2019 1 QAConv (Wu et al.
2022
QASC
(Khot et al. 2020), Quail (Rogers et al.
2020), Quoref (Dasigi et al. 2021 RACE (Li et al.
2017
Reclor
Yu et al. (2020), Record (Zhang et al.
2018). Ropes (Lin et al. 2019). SocialIQA (Sap al.
2019
SQuAD 1.1 (Rajpurkar et al.
2016), SQuAD (Rajpurkar al.] 2018 TweetQA (Xiong et al.) (019)) and
Winogrande (Sakaguchi et al.
2020 For readability, omit citations for other datasets already referenced.
27

---

Published in Transactions on Machine Learning Research (08/2023)
Table 12: QA model Group training datasets. All figures are Exact Match on reduced development sets from the single overall best model without per-dataset finctuning.
Dataset
Base
Base+ RATD
adversarialqs all
46.0
47.6
ai2 science seience__idddee middle
67.2
63.2
ai2 ahtence_elementaee
67.5
69.9
are hard
56.5
54.2
hard with ir are arc_easy
59.5
59.5
68.3
70.4
are eas_easy_oo with ir
77.5
79.3
boolq
84.7
84.2
boolq_np creak goldphuddistratoos creak ratd
82.0
81.7
85.2
83.8
85.9
creak readrraddeeeeee mard__raspaaass
85.6
esqa2_radd ratd
56.7
esqa2_ _aud__axxparaus
57.7
fever golaphssdistractors
85.9
89.2
hotpotqa__te _ harer__oov____nsseerrr
noanswer
83.5
76.9
hotpotqa_gokee goduuuuu_goollussssstttttssss
65.9
66.7
hotpotqa_rate
53.0
hotpotqa _ato max/paras
52.5
hover_ggolppllee
goldplusdistractors
84.0
82.2
hover_rate
78.5
hover rater_ratd_a max4paras
77.2
metest
91.3
90.0
multire
100.0
100.0
musiquee_oooee solpuuusslllllsssstttttoosss
88.0
87.2
musique noanswer
96.6
95.6
musique_ratt _atd
74.4
musique_ ratd max4paras
75.1
narrativeqa
30.0
29.1
naturalquestions goldplusdistractors
56.5
58.8
naluralquestions. catpillooooppoo_ttt__eeeeee paad
40.9
naluralquestions__e _epesspppp_________ rald max4paras
39.9
newsqa
44.3
44.4
openbookqa
67.2
69.2
openbookqa with ir
68.4
70.6
physical iqa
66.9
67.0
pubmedqa pqal short ans qaconv qasc
100.0
54.3
54.8
53.4
55.5
gasc with ir
72.0
70.8
gasc ratd
61.9
qasc ratd max4paras
62.6
quail
78,1
76.1
quoref
71.5
70.2
race
76.4
74.8
reclor
43.0
41.4
record
53.1
53.1
ropes
77.6
81.8
social iqq
75.1
74.0
squadl
66.5
64.9
squad2
66.2
67.4
tatqa
41.6
40.8
triviaga__oldphhaeee goddassssssdsssssstttttsss
63.9
65.3
tweetqa
34.5
33.6
winogrande. xl
69.7
69.1
28

---

Published in Transactions on Machine Learning Research (08/2023)
Dataset names containing "ratd" are those created by us by concatenating the original question with the retrieved context from our Iterator described in the main text.
Dataset names additionally containing the term "max4paras" use these same contexts but are truncated to the top 4 retrieved paragraph fragments. We found that sometimes longer and sometimes shorter contexts provided better results and hence we added both forms to provide diversity length.
Dataset names containing the phrase "with ir" have retrieved contexts provided by Khashabi al. (222bb) which we use unmodified.
Contexts for dataset names incorporating the term "goldplusdistractoos" are constructed using the positive and negative paragraphs from corresponding retrieval training datasets. In both cases the document title was randomly withheld (A = 0.1). For positive paragraphs we included the gold sentences plus random other sentences sentence-level annotation was available, otherwise the full paragraph text. For negatives we similarly included either random sentences or full text such that the length distribution of positive and negative paragraphs was similar.
Squad 2 provides some unanswerable training samples. We supplemented these by creating unanswerable samples from HotpotQA, Hover and FEVER positives in similar manner to the "goldphusdistractors" datasets except here we randomly drop gold sentence(s) and/or full gold paragraphs such that there is guaranteed to at least one missing gold sentence. We performed the same activity for Musique at the paragraph level. All unanswerable samples have the label string " "NNo Answer>".
F
Paired Bootstrap P-values
P-values for all Base to RATD model comparisons under the Paired Bootstrap test are in Table 13
Table 13: Paired Bootstrap p-values. SQAgPx denotes gold paragraphs from each of three annotators.
Dataset
P-value
SQA
0.008
SQAR
0.000
SQAR w/ Yes or no prefix
0.000
SQAGF
0.031
SQAGr1
0.000
SQAGP2
0.000
SQAGP3
0.000
CSQA
0,006
CSQAR
0.155
DROP
0.017
IRCR
0.017
IRCG
0.049
ARCDAR
0.001
ARCDAG
0.013
MusiqueR
0.001
Musiquca w/o Musique RATD
0.000
Musiqueg
0,000
Musiqueg w/o Musique RATD
0.009
29

---

Published in Transactions on Machine Learning Research (08/2023)
G
Example Failure cases
Table 14
contains examples of failure cases for samples with numeric and "umanswerable" labels from the
IRRR Lest split.
Table 14: Example failure cases for IRRR samples on the Base+RATD model. The top two rows have numeric labels, the bottom two are labelled umanswerable. Bolded context text highlights information that could be used deriving an answer.
Question / Answer
Retrieved Context (condensed)
How old the Grand Olympic Audito- rium at the time of New Regime playing a landmark concert there? Gold answer: 60. Predicted Answer: 1924
New Regime (American band]: ... That landmark concert was held at the Grand Olympic Auditorium on April 13, 1984 Grand Olympic Auditorium: ... The venue was built in 1924
How old Messe when the First World War started? Gold Answer 30. Pre- dicted answer: 28.
Giovanni Messe: Messe born
10 December 1883. 20th- century events: The First World War started in 1914 and ended in 1918.. Military career of Adolf Hitler: He was 25 years old in August 1914, when Austria-Hungary and the German Empire entered the First World War.
What albums were ranked higher than "It Takes Nation of Millions to Hold Us Back' Rolling Stone's 500 Great- est Albums of All Time? Gold answer:
Takes Nation of Millions to Hold Back: ... 2003, Rolling Stone ranked the album number 48 on its list of the 500 Greatest Albums of Time... maintaining
rating in a 2012 revised list. Rolling Stone's 500 Greatest Albums of All Time: topped by the Beatles' 1967 album "Sgt. Pepper's Lonely Hearts Club Band", with top 10 that featured four entries from the Beatles (Nos. 1, and 10), two from Bob Dylan (No. 4 and 9), and one each from the Beach Boys (No. 2), Marvin Gaye (No. 6), the Rolling Stones (No. 7) and the Clash (No. 8).
<no answer>. Predicted answer: the beatles.
In what direction does the Goulburn River flow Sugarloaf Creek? Gold an- swer: <no answer>. Predicted an- swer: oorth west.
Charles Bonney: was the first to overland sheep, bringing some 10,000 ... to Sugarloaf Creek, Victoria station a tributary of the Goulburn River... Goulburn River: The river flows generally north, then west, then north, then west...
30


Reviews:
[{"review_id": "Us5cD3iBuA", "reviewer": "Reviewer_xHAu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. The paper addresses a significant problem by focusing on the reasoning capabilities of smaller language models while taking practical limitations, such as latency, cost, energy efficiency, and physical compute size, into account.\n2. The introduction of Retrieval-Augmented Training Datasets (RATD) and the subsequent evaluation of their impact on improving the reasoning performances of small language models is an innovative contribution.\n3. The paper features a comprehensive evaluation and comparison with prior research, encompassing both fine-tuned and zero-shot settings.\n\n### Weaknesses:\n\n1. The paper lacks sufficient detail regarding the construction of RATD datasets and how these datasets provide superior reasoning capabilities compared to existing datasets. An illustration example would help clarify how the pipeline works.\n2. The definition of compositionality is not well-explained in the paper. Clarification is needed concerning the compositionality of questions within evaluation datasets, and the justification for the number of hops needed per example. Furthermore, a clearer definition of \"seen\" and \"unseen\" questions and their relation to the evaluation strategy would be beneficial.\n3. The authors do not provide sufficient explanation about the choice of evaluation datasets, the configurations for each component depicted in Figure 1, and the selection of hyperparameters (such as z and t_max). An evaluation of each component's performance contribution would also be useful.\n4. The paper's novelty is somewhat limited. Retrieval, reranking, and iterative learning for generalization is a common pipeline for modern QA systems, and the proposed method's specific advantages compared to existing pipelines are not clearly articulated.\n5. The paper's presentation could be improved (see suggested changes below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\n1. The authors proposed a method for improving the reasoning abilities of small language models by training them on Retrieval-Augmented Training Datasets (RATD).\n2. A comprehensive set of baselines is provided for evaluating the zero-shot reasoning abilities of small language models.\n3. The paper demonstrates how training on RATD datasets can improve the performance of small language models on unseen evaluation datasets such as StrategyQA, CommonsenseQA, etc.\n4. The authors identified the brittleness of training for numerical literacy and unanswerability in the unseen setting and proposed effective extensions to the retrieval approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Presentation: Use more illustrative examples to clarify the pipeline and highlight the motivation for each component. Including clearer explanations and justifications for each design choice would help readers better understand the authors' decisions.\n- Problem Formulation: The paper should better define the problem and the concept of compositionality.\n- Evaluation: Authors could improve the experimental section by stating research questions first and then presenting detailed analyses and results. The current structure is somewhat difficult to follow. Additionally, the inclusion of more retrieval and reranking methods as baseline approaches would strengthen the evaluation.\n- Related Work Discussion: Authors should consider adding more discussion about related work in the same research direction. ", "broader_impact_concerns": "It has a section for this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dlzV7OWKCe", "reviewer": "Reviewer_bda2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- S1: The overall framing seems likely to be useful for future work, as it has interesting thoughts about designing systems that prioritize different factors (e.g., compute efficiency vs. accuracy) and also about being able to reason from imperfect or incomplete information.\n- S2: The proposed approach is well-motivated (both the RATD component and the other components). Though there are several components to the approach, the authors have given clear reasons for why each one is set up the way it is.\n- S3: Empirically, the proposed approach is effective in many cases, and the authors are open about cases where it is not so helpful.\n- S4: The experiments are extensive, covering a large range of datasets and a large number of baselines and comparisons to prior work.\n\nWeaknesses: I did not identify any serious weaknesses. The main shortcomings were the following, but there is nothing wrong with these issues - they are just points that might limit the impact of the work, rather than problems that need to be fixed:\n- W1: There is a lot going on in the paper, and sometimes I lost track of what each term referred to or what was being tested. See “Requested changes” below for some thoughts on quick things that could help with this.\n- W2: The benefits provided by RATD are inconsistent: That is, in some cases it does not help, and in other cases it helps by only a small margin. However, I do not view this as a serious weakness: First, the idea is an interesting one, so it is useful to see what its effect is, whether positive or negative. Second, it is hard to get any improvement when working with messy large-scale datasets such as these, so the fact that any improvements were observed is impressive.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper works in the context of question answering augmented by retrieval methods (i.e., retrieving relevant information in order to help answer the question, rather than storing all knowledge in LM parameters). The authors propose a new method that involves adding “retrieval augmented training data” (RATD) during training - that is, adding training datasets augmented with retrieved passages. One advantage that is proposed for this approach is that it can teach models to use heuristics to find plausible answers in cases where there is no answer that is perfectly entailed by the available material. The authors then perform an extensive set of experiments testing this approach; it is extensive both in the number of datasets that they consider and in the thoroughness with which they compare against baselines and prior literature. The RATD augmentation is in general a useful modification: in some cases it allows them to get a new SOTA or to create models that are not SOTA but are competitive with much larger models from prior literature. Finally, the authors also include some useful analyses of 2 major areas of weakness for models, namely numerical computation (where they show that additional finetuning is very helpful) and recognizing unanswerability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None of these changes are crucial for securing my recommendation; they are just things that I think would help the reader to keep track of important points in the discussion.\n- RC1: In the table captions, give a brief one-phrase reminder of what the dataset is like (e.g, “Results on StrategyQA, a binary dataset of multihop commonsense reasoning questions”)\n- RC2: In both the text and the tables, whenever using a subscript abbreviation, it would be helpful to remind what the subscript stands for, as I found it hard to keep track of this - e.g., SQA_GF, IIRC_G. I know that this might be a lot, so there’s definitely no need to include it everywhere, but the most helpful places to add a reminder would be where you discuss some comparison that depends crucially on the subscript - e.g., “Our Base model fails to improve with SQA_GP (when the question is augmented with paragraphs from annotators) versus the question-only SQA version”.\n- RC3: To help contextualize the numbers in the tables, it might be helpful to list a chance baseline (50% for StrategyQA, 20% for CommonsenseQA, etc.)\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AW1pkKKnKe", "reviewer": "Reviewer_HiAz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Personally I have learned a lot from the related work section. \n\nThe few-shot performances of big LMs are necessary and it’s great that the authors have included them. \n\nThe motivation of having smaller models that specialize in question answering is great, and the performance is quite impressive given the comparison with large LMs (sometimes even much better than large LMs in the few-shot setting). \n\n---\n\nIt would be prudent to qualify the tasks which the authors’ approach/model would excel in. \n- Would the authors’ model work on any question answering task? How about long-context question answering tasks (e.g., answering questions about novels)? \n- Is the approach tailored to open-domain question answering tasks or other types of tasks?\n\nExplain paragraph relevance score and sentence relevance scores. \n- Would the tradeoff in coefficients impact final reranker performance? Are the two scores on the same scale, and do they need to be tuned? \n- The paper claims that the paragraph relevance score is often more accurate than the sentence relevance scores; is there empirical evidence for this claim?\n- Similar issue for the evidence set scorer coefficients in the following paragraph.  \n\nConfusion on the number of tasks: I’m confused about how many tasks are in Group 1 and Group 2. Section 2.4.2 says there are 38 tasks in Group 2. Section 2.4.3 says there are 41 tasks in Group 2 (and 41+14 tasks for the second model). How did you obtain “total of 93 tasks” – is it 38+41+14? I think the authors could be more clear on what each cluster of tasks represent, and why the tasks are separated this way.\n\nBelow contains some concerns on experiments:\n\nOn StrategyQA experiments: “Our SQA-GF setting hugely outperforms our other configurations. This shows that with a context of a form the model has learned to reason with, it is possible to solve challenging implicit questions.”\n- What specifically is the “form the model has learned to reason with” and are there any summary statistics on the “form” of learning? \n- What’s an “implicit question” – definitions and examples would be helpful. \n\nIt’s unclear what the source of improvement is, for the DROP task. Is it because of digit tokenization, or any particular stage in the training pipeline? \n\nIt’s still unclear to me when RATD would be harmful. If the task data is too similar to RATD data, then would there be significantly worse performance as in Section 3.3.3?\n\nMinor (feel free to ignore if it would take lots of effort): The GPT-3 performances are from the original GPT-3 paper, not updated GPT-3 models (with instruction tuning, etc.) in the past three years. If it’s easy to get those results, I would include them in Table 5. If not, then it’s fine. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate zero-shot generalization in question-answering but for smaller models, in particular, using retrieval from a corpus that doesn’t necessarily contain sufficient information to answer the question. \n\nThe paper offers baselines on smaller LMs reasoning abilities in the zero-shot setting. The paper shows RATD augmentation would lead to improved performance; numerical literacy and unanswerability question answering tasks are difficult to train for without similar training examples; improved retrieval approaches lead to improved performance. \n\nMore specifically, for retrieval, the authors expand multi-hop dense retrieval (MDR) from 2-hop max to 4-hop max. A chain is built e.g., q -> d1, (q, d1) -> d0, (q, d1, d0) -> d3 (where q stands for a question and d_i stands for the i-th document). The authors use clever tricks (partly based on Wikipedia) to obtain negatives for retriever training. The retriever is trained on four datasets: HotpotQA, Hover, Natural Questions, Musique. \n\nThen, the reranking and evidence set scorers are trained. The reranker combines a paragraph relevance score and the individual sentence relevance score. The evidence set scorer combines paragraph relevance score and sentence relevance in the context of the evidence set. In-domain retrieval and reranking evaluation are done on the Hover dev set (number of retrieved paragraphs = 25). Performance is much better using the authors’ retriever compared to MDR retriever. \n\nQA models are further pretrained using two stages. Stage 1 (33 tasks) deals with improving numerical literacy. Group 2 (38 tasks including the prior 33 tasks) is co-trained with Group 1; Group 2 aims to teach models about the expected form of answer for a given question type. Another difference is that Group 1 training use uniform sampling of tasks, and Group 2 uses error-based sampling of tasks. \n\nRetriever is based on RoBERTa-base; reranker and evidence set scorer are based on ELECTRA-large; QA models are based on BART. \n\nA few results are as follows. Base+RATD (the augmented data) > Base on StrategyQA (zero-shot), and the performance is better or comparable with some of the 11B models and OPT-175B, which seems impressive. SQA-GF (explained at the beginning of Section 3.1) outperforms other configurations of contexts in StrategyQA. The authors’ model’s DROP performance is better than many baselines, despite the model being much smaller and the setting is zero-shot (see Table 5). The model also achieves good performance on IIRC, ARC-DA, but shows significantly decreased performance on Musique and the authors have provided an explanation. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have made several changes already taking the reviewer suggestions into account. Upon further discussion, the reviewers all agree that this is a good paper with solid technical insights and experiments. However, a few minor concerns still remain around the writing and clarity of presentation. In particular, reviewers feel that the paper still lacks adequate motivation for the problem and that the writing can be improved in places.

Therefore, I'm recommending acceptance, and strongly urge the authors to make these minor revisions (as suggested by reviewer xHAu) for the camera ready version:
1. Begin the paper with more compelling examples rather than relying on numerous abstract sentences and ideas. For example, a single running example in the introduction can help the readers understand the problems and challenges more easily.
2. Better motivate and explain the key components of the framework. Again, an example or two may help the reader better understand the purpose of the different parts in the framework. 
Decision: Accept (poster)