To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
Year: 2023

Abstract:
Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on general-purpose domains like Wikipedia. While some work has been investigating how well ODQA models perform when tested for out-of-domain (OOD) generalization, these studies have been conducted only under conservative shifts in data distribution and typically focus on a single component (i.e., retriever or reader) rather than an end-to-end system. This work proposes a more realistic endto-end domain shift evaluation setting covering five diverse domains. We not only find that endto-end models fail to generalize but that high retrieval scores often still yield poor answer prediction accuracy. To address these failures, we investigate several interventions, in the form of data augmentations, for improving model adaption and use our evaluation set to elucidate the relationship between the efficacy of an intervention scheme and the particular type of dataset shifts we consider. We propose a generalizability test that estimates the type of shift in a target dataset without training a model in the target domain and that the type of shift is predictive of which data augmentation schemes will be effective for domain adaption. Overall, we find that these interventions increase end-to-end performance by up to ∼24 points.

heading: 1 Introduction
text: General-purpose open-domain question answering (ODQA; Chen et al. (2017); Lee et al. (2019); Izacard et al. (2022)) is an important task that automates reading and understanding a large corpus of documents to answer a given question succinctly. It is especially crucial in fields such as biomedicine, legal, news, etc., where more documents are added daily, outpacing the speed at which a user can process the information. Current state-of-the-art ODQA systems perform a two-stage pipeline process (Izacard et al., 2022): 1) Given a question
∗*This work was done while authors were at Google.
and document corpus, a retriever (Karpukhin et al., 2020; Izacard et al., 2021; Raffel et al., 2020) selects relevant passages and 2) a question answering model, also known as a reader (Izacard and Grave, 2021) answers the given question based on the retrieved passages. This decoupling allows for independent advances in domain adaptation of general-purpose retrievers (Thakur et al., 2021) and question-answering (Fisch et al., 2019) models.
To enable practical application, an ODQA system should assist humans in keeping up with new knowledge without requiring annotations for every new domain or concept. For this, the system should be resilient to changes in the document, question,
14429
and answer distributions. Unfortunately, the current work in ODQA focus solely on Wikipedia corpus and do not study effectiveness of a model trained on such a general-purpose domain when applied to an unseen domain. To gauge how likely it is for a source domain model to succeed on an unseen domain we need to understand its ability to work out-of-the-box or even adapt to a new target domain, under varying types and degrees of dataset shifts. (Quinonero-Candela et al., 2008).
In this work, we study the challenges and interventions for generalizing ODQA models to new domains via four contributions. First, to understand how well the state-of-the-art ODQA system (trained on the general-purpose domain) performs on a variety of target distributions, we define a collection of datasets for evaluating domain generalization. We aggregate a set of seven ODQA datasets spanning five different domains (§2). We observe that the source ODQA model does not generalize well (Fig.1, Top) on this collection (§4). Second, to automatically determine the type of data shift with only a small number of labeled target domain examples, we propose a generalizability test. This test assesses the type and degree of shift a new domain suffers with respect to the source domain (§3). Third, to understand the adaptability of the source model to a target domain, we analyze the performance of various intervention schemes, including existing zero-shot in-domain question generation and a novel few-shot language modelaided generation. These schemes create data akin to the target domain which is augmented with the source domain to learn an adapted version of the source model. Overall, we observe improvement in performance across all the target datasets (Fig. 1). The degree of improvement depends on the intervention scheme and underlying dataset shift (§5). Finally, we propose a simple and effective few-shot method that improves the performance by up to 24% in F1. This method prompts a large language model with 8 examples to generate examples for further adaptation.
Putting it all together, we use the generalizability test to gauge the type and degree of dataset shift in a target dataset. Then, we empirically show that certain types of dataset shifts respond well to specific intervention schemes (§5, Fig. 1). This helps ascertain whether we can adapt a source model to unseen domain with minimal supervision. The resources used in this work are released at https:
//github.com/dDua/adapt-or-annotate

heading: 2 Background and Evaluation Setup
text: An ODQA model learns interactions among three random variables: Question (Q), answer (A) and context corpus (C). For a given q ∈ Q, first the retriever R returns a set of passages, Cq = R(q,C). These passages are then sent to a reader modelM to obtain the final answer, â←M(a|q,Cq).
Following prior work, we evaluate retriever performance with the Acc@k metric, which computes if the oracle answer is found in the top-k retrieved passages1. We set k=100 in all of our experiments. For reader performance, we compute token-level F1 between the oracle and predicted answer.

heading: 2.1 Datasets
text: We test the generalization capabilities of a model trained on a source domain when applied to seven datasets in five very different target domains. Source Domain: For source domain we use documents from English Wikipedia and QA pairs for supervision from NaturalQuestions (NQ) (Kwiatkowski et al., 2019) and BoolQ (Clark et al., 2019). We treat this domain as our source as it is used for the vast majority of current work in ODQA (and many other areas of language research). In addition to the supervised training data from NQ and BoolQ, we also consider clozestyle questions derived from the QA pairs in NQ. For each QA pair, we retrieve a sentence from Wikipedia with the highest BM25 similarity score. We convert the retrieved sentence into a cloze-style question by replacing the answer string in the sentence with sentinel markers (Raffel et al., 2020)2. Target Domains: For our target corpora, we re-purpose seven open-domain QA and/or reading comprehension datasets spanning five different domains (Stack Overflow, Reddit, Pubmed, Japanese Statute Law codes, CNN/DailyMail, and Wikipedia). The datasets Quasar-S (Dhingra et al., 2017), Quasar-T (Dhingra et al., 2017), SearchQA (Dunn et al., 2017) and BioASQ (Balikas et al., 2015) were introduced as ODQA
1The only exception is COLIEE dataset which primarily contains boolean (yes/no) answers so we instead use oracle passage to compute Acc@100.
2We use cloze augmentation for training reader models because some target datasets contain cloze-style questions, keeping the question distribution consistent across different experimental setups. We do not perform this augmentation for retrievers because we observed a performance drop in initial experiments.
datasets over Stackoverflow, Reddit, Wikipedia, and Pubmed corpus respectively. Additionally, we re-purpose reading comprehension datasets, NewsQA (Trischler et al., 2017) and CliCR (Šuster and Daelemans, 2018) as ODQA datasets, by retrieving a set of passages for each QA pair from Pubmed and CNN/Dailymail corpus. For COLIEE (Rabelo et al., 2022), we convert the original entailment questions into boolean questions and retrieve passages from legal code statutes provided with the task. We confirm that these reading comprehension datasets can be reasonably re-purposed for our ODQA setup by achieving a reasonable endto-end performance of ODQA models trained on gold target domain QA pairs with BM25 retrievals from the target corpus (UB-Ret, Fig. 3).

heading: 2.2 Models
text: We compare four retrievers: (1) BM25 (Robertson and Spärck Jones, 1994) (sparse and unsupervised), (2) Contriever, semi-supervised with MSMARCO (Izacard et al., 2021), (3) Dense Passage Retriever (DPR) (Karpukhin et al., 2020), and (4) the state-of-the-art source domain model Spider (Ram et al., 2022). DPR and Spider are dense and supervised. As for reader, we use the stateof-art fusion-in-decoder (FiD) model (Izacard and Grave, 2021) that uses the top 100 documents to generate the final answer.

heading: 3 Generalizability Test
text: There are many aspects that determine in what ways and to what extent one data distribution differs from another. It is often challenging to quantify the degree of generalizability or diverseness for a new domain without collecting enough samples to train a model in the new domain. To address this issue, we propose a method to assess the type and degree of diversity by utilizing only a few examples from the target domain as an evaluation set.

heading: 3.1 Types of dataset shift
text: Different types of dataset shifts (QuinoneroCandela et al., 2008) have been proposed in the literature but they are often studied in a classification setup. For our application, we consider concept and covariate shifts which are more amenable to our pipelined ODQA setup — with input as a joint distribution over question and contexts and output as a distribution over answers given question and contexts as input.
No shift occurs when the input and output distributions match across the source and target domains.
Concept shift (Widmer and Kubát, 2004) occurs when the input distribution of the source and target domains match, i.e., ps(x) = pt(x) while the output distribution between source and target domain does not match, ps(y|x) , pt(y|x). Covariate shift (Zadrozny, 2004) occurs when the source and target input distributions do not match, i.e. ps(x) , pt(x) while the output distributions match ps(y|x) = pt(y|x). Full shift occurs when both the source and target input and output distributions do not match.

heading: 3.2 Calculating shift for ODQA
text: We characterize the shift in ODQA as a two-step process. First, we compute the input distribution, i.e, the joint question and context distribution using un-normalized (energy) scores from a dense retriever (Karpukhin et al., 2020) that quantifies the compatibility between a given question, q and a context, c via R(q, c). Then, we normalize the scores from the retriever over a set of contexts. Ideally, the set of contexts should be the entire target domain document corpus, however, that can be prohibitively computationally expensive and also results in a high entropy distribution. Instead, we use a subset of contexts, C, from the entire corpus C. We ignore the prior over questions since it remains constant when calculating the context distribution for a specific question. Instead, we approximate the joint with conditional distribution over contexts given question.
p(q, c) ∝ R(q, c)∑ ck∈C R(q, ck) (1)
In the second step, we test whether the output distributions match by computing the likelihood of generating the oracle answer given a question, q, and the relevant contexts, Cq. In an ideal scenario, we can do this by performing global normalization (Goyal et al., 2019) over all possible answer spans in the corpus which is intractable. Instead, we use a sub-sample of answers,A, to compute the output distribution as shown below.
p(a|q,Cq) = ∏
tM(at|a<t, q,Cq)∑ ak∈A ∏ tM(atk|a<tk , q,Cq) (2)

heading: 3.3 Predicting type of dataset shift
text: To compute the type of shift (§3.1), we need a model trained on the target domain (pt) which requires a large number of examples. However, our goal is to determine if a source model can be adapted to the target dataset with only a few examples for target evaluation. To do this, we conceptualize adapting or fine-tuning a pre-trained source model as a Bayesian framework. In this framework, the source model acts as a prior which when exposed to interventional data (for adapting) and target data (for fine-tuning), results in an adapted or fine-tuned posterior distribution. If the prior (source model) contains an informative signal with respect to the target dataset then we do not require much supervision to learn an effective posterior. However, if the prior is non-informative we need a lot of supervision to learn the posterior. Towards this end, we devise a generalizability test, where we use a small set of evaluation examples sampled from each target dataset to compute input and output distribution using the source domain model. Then, we compare these distribution with the a non-
informative prior like uniform distribution and informative prior like the oracle distribution to gauge if the source model is closer to uniform or oracle distribution. This helps us assess the effectiveness of the source model towards the target dataset without having to train a model in the target domain.
Input/Retriever Distribution: To determine if the input distribution contains informative signal with respect to target evaluation set, we need to compute the distance of the input distribution from uniform and oracle distribution. To do this, we follow Eq. 1 and compute the input distribution, with passages from across examples in the entire target evaluation set as the subset for normalizer computation. Then, for a each question, we compute the Wasserstein distance, wtu, (Kantorovich, 1960) between the input distribution and the uniform distribution and average these values over all the examples in the target evaluation set. Similarly, we also compute the distance between the gold or oracle distribution and the input distribution as wtg. If wtu > w g u, we conclude that the target distribution is far from the uniform distribution and closer to the gold distribution, indicating that the source model is compatible with the target distribution (Fig. 2).
Output/Reader Distribution: In similar vein as input distribution, we need to compare the output distribution with corresponding uniform and oracle distribution over answers. To do this, we follow Eq. 2 and compute the output distribution, with set of answer spans from across all the examples in the target evaluation set for normalizer computation. Then, we compute the Wasserstein distance between the uniform and output distribution aver-
aged over the target evaluation set as vtu. In an ideal scenario, we would compare the distance between and oracle and output distribution with vtu, similar to input distribution. However, empirically we find that output distribution is always closer to uniform than oracle, even when evaluated on source domain. We believe this is because of two reasons. First, the conditional answer generation model (M) is not trained with a contrastive loss like the retriever, resulting in a high entropy answer likelihood distribution. Second, the support set of answers used for normalization contains only grammatically correct answer spans making the likelihood scores attenuated. To address these issues, we use a reference answer conditional distribution to de-bias the likelihood scores with a threshold. To obtain this threshold, we consider the source distribution as a reference and compute the distance between output distribution evaluated on examples from source evaluation set and the uniform distribution as vru. Since the reference based output distribution is in-domain, it should be far from the uniform distribution and closer to oracle distribution. As a result, if vru − vtu is close to 0, we assess that the target is far from uniform and that source model is compatible with the target dataset.
In Figure 2, we put this altogether as a decision tree to identify the type of dataset shift. We observe that SearchQA falls under the No shift category as it is close to the source domain, hence, we conjecture that it will observe minimal improvements under most data intervention schemes as the source model already captures the target distribution (§5). We also conjecture that datasets falling under Concept shift and Covariate shift are more amenable to zero-shot data interventions, while, Full shift would benefit more from few-shot or in-domain annotations from the target domain. We consider few shot augmentations as a proxy for annotating examples in the target domain because they are generated with supervision from target dataset.

heading: 4 How Well do Models Generalize?
text: We test the OOD performance of the source model on target datasets and analyze the failures.

heading: 4.1 Reader Generalization
text: In Fig. 3, we test the end-to-end performance of three model variants: Source: a reader trained with source dataset and contexts retrieved by BM25, demonstrating zero-
shot generalization performance. Upperbound-Reader (UB-READ): a reader trained on the target dataset with contexts retrieved by BM25 – the overall strongest retriever. Upperbound-Retriever (UB-RET): a reader trained on the target dataset with gold contexts to approximate upper-bound performance.
We observe large performance drops when evaluating the source model on target domains (Fig. 3), especially when the target corpus differs from Wikipedia, such as in Quasar-S (Stack Overflow) and CliCR (PubMed), even though the model requires similar reading capabilities to those needed in the source domain. Interestingly, even though BM25 retriever accuracy is relatively high on the target datasets (Fig. 4, ∼83% Acc@100 on QuasarS), that accuracy does not translate to strong reader performance (Fig. 3, ∼11% F1 on Quasar-S).
To understand this performance gap, we manually sample 50 predictions from each target dataset where retrieved passages contain the oracle answer but the reader produced an incorrect prediction. We observe that in ∼65% cases, the Acc@100 metric yields a false positive, where the passage contains an exact string match of the correct answer, but the context does not actually answer the given question. In other cases, the reader is unable to understand the context. For example, for the question: What is the name of the office used by the president in the white house? and answer: oval, the retrieved passage: A tunnel was dug into the White House connecting the Oval Office to a location in the East Wing.... is credited (incorrectly) as context answering the question.

heading: 4.2 Retriever Generalization
text: We compare the zero-shot generalization of four retrieval models in Fig. 4. Spider, which is the best performing model on the source domain, exhibits improvement on SearchQA (∼1%) (which is similar to source distribution), but shows large drops in performance when applied to the target datasets: ∼40% on NewsQA, ∼28% on Quasar-T and, Quasar-S. To understand the drop, we manually analyze 50 random incorrect predictions from Spider. We observe two major failure modes. First, we find that dense models are sensitive to changes in the length of contexts. When exposed to documents with heterogeneous lengths, models tend to over-retrieve shorter contexts. To
quantify the sensitivity to changes in lengths on source domains itself, we pool passages from all target corpus into a combined index. We observe that the performance of Spider when exposed to this combined index reduces by ∼15% and restricting the minimum length of contexts to 50 words alleviates the problem and recovers the original performance. The second common failure mode occurs due to changes in distribution of entity types from source to target. For example, words like plant in Which is produced in plants of narora kakrapar tarapur refers to power plant in Wikipedia, while in case of PubMed it often refers to living organic matter (Sciavolino et al., 2021). Overall, BM25, being an unsupervised method, has the best performance across all domains.

heading: 5 Interventions for Improving Adaptation
text: Domain Adaptation is shown to be a causal intervention (Jin et al., 2021) mechanism to effectively understand impact of an augmentation technique without much concern about spurious correlations.

heading: 5.1 Zero-shot adaptation methods
text: We perform a set of zero-shot data intervention methods, where we consider the effect of change in distribution of each random variable: Question, answer and context one at a time, while keeping the other two fixed.
Varying context distribution To test the effect of change in context distribution, we pool passages
from all corpora into a combined index. We observe that supervised models like Spider are sensitive to out-of-domain distractors, unlike BM25, especially when the target dataset uses same corpus as source (Wikipedia). For example, SearchQA suffers a performance drop of ∼15%. On average we see a performance improvement of ∼2% (w/o COLIEE) when the target index is changed to the combined index. BM25 still out-performs Spider on average by 19.1% with the combined index. However, we observe a drop in performance of the FiD reader of up to ∼5% in F1 for NewsQA with the combined index. More details are in the appendix (Figs. 5 and 6.)
Varying answer distribution Many works (Gururangan et al., 2018; Dua et al., 2020; Jiang and Bansal, 2019) have shown that bias in the answer prior distribution can introduce spurious correlations in model learning. This effectively improves the model performance at the cost pf OOD generation. To test whether we can improve the performance of adapted source model by varying the answer distribution, we experiment with a variety of answer distributions over plausible set of answer spans. To obtain the set of answer spans, we extract and annotate coarse-grained entity types from the
Dataset Retriever Reader
Source ClozeQA QGen Source ClozeQA QGen
BioASQ 50.41 48.0 45.4 45.3 49.4 46.4 CliCR 23.8 24.9 23.9 6.12 7.34 10.5 Quasar-S 50.3 66.8 68.2 10.2 21.7 17.4 Quasar-T 54.7 53.9 55.5 34.9 41.9 44.7 NewsQA 12.5 18.7 15.2 18.5 21.2 12.7 SearchQA 63.0 52.9 54.7 34.6 38.8 37.2 COLIEE 61.4 60.5 57.8 46.7 54.1 62.3
Table 3: Zero-shot: Comparing retriever (DPR) and reader (FiD with BM25 retrievals) performance on two types of question formats for augmentation.
target corpus using spaCy3. We use this coarsegrained entity type information as a set of classes from which to choose 50k entities with four different sampling strategies: Most frequent, uniform, randomly sampled based on entity type categories, and sampling in proportion to entity type distribution of answers in the target training set.
The source model has reasonable end-to-end performance on BioASQ, even with passages from the source corpus (Wikipedia), suggesting that it contains sufficient information for answering many BioASQ questions. Consequently, we select BioASQ for these controlled experiments (Appendix Fig. 6). This allows us to use the Wikipedia corpus alone for retrieval, which makes it easier to fix the passage distribution. In Table 2, we show that uniform sampling boosts retriever performance compared to random sampling, allowing the model to learn from all types of answers and generalize better to unseen answer distributions. On the other hand, the best reader model performance is when we know the correct answer distribution of the target dataset up front, showing that the answer priors influence reader performance. However, in a zero-shot setup, we do not have access to this distribution, so we adopt the second-best technique, uniform sampling from across the entity type categories, in the following experiments.
Varying question distribution We vary the question distribution by augmenting the source domain with QA pairs generated using two different methods. Our first approach (QGen) uses a question generation model (Subramanian et al., 2017) trained on the source domain to generate a question given a passage and an answer. This question generation model is applied to a new target passage and a plausible answer span from the passage (Shakeri
3https://spacy.io/
et al., 2020; Krishna and Iyyer, 2019; Song et al., 2018; Klein and Nabi, 2019). The second approach (Cloze QA), which has been less explored previously, converts a sentence in the target corpus to a fill-in-the-blank style cloze question (Taylor, 1953) by masking a plausible answer span (entity mention) in the sentence. We sample answer spans uniformly based on an entity type distribution from the target corpus and then query our combined index to create a dataset containing cloze style questions aligned with relevant documents. We use these same sampled answers to generate standard QGen QA pairs as well. We combine these data interventions with our initial source domain data to train a DPR retriever and a FiD reader (Table 3). We observe similar average performance across both intervention types in retriever and reader models. However, cloze QA pairs are computationally much more efficient to generate as they do not require additional question generation models.
Discussion on generalizability test In §3, we hypothesized that datasets with less severe shift (Quasar-S, Quasar-T, and BioASQ) would show more performance improvements with zero-shot adaptation as compared to datasets with severe shift (CliCR and NewsQA). Indeed, we observe an avg. improvement of about 8.5% F1 on datasets having Concept and Covariate shift while only 3.5% F1 on datasets with Full shift in Table 3. Moreover, in Fig. 1, we see that target datasets with No shift, do not show much improvement with any intervention as the source model already captures the distribution. Datasets with Full shift need few-shot examples for better adaptation while datasets with Concept and Covariate shift are able to adapt with zero-shot data interventions.

heading: 5.2 Few-shot Generalizability and Adapatability
text: Zero-shot adaptation does not work well when the target distribution is far from the source. For these cases, we experiment with few-shot adaptation.
Few-shot data generation Zero-shot interventions like QGen are trained on the source and do not produce generations that are fully compatible with the target domain and thereby do not provide much useful signal. An alternative approach would be to train a question generation model with a few examples from the target domain. However, it is difficult to adapt or fine-tune a question genera-
tion and answering model (for validating QA pair correctness) with very few examples.
To capture target distribution without a lot of supervision, we propose a few-shot technique (DataGen) that prompts a large language model (LLM; Chowdhery et al. (2022)) to generate a sentence given a passage. We use eight seed examples from the target domain to generate additional training data to help bootstrap adaptation in the target domain. We observe that it is easier for large language models to condition on a single variable (context) and compress (Goyal et al., 2022) multiple facts from the passage into a single sentence, as compared to conditioning on a context and answer span together. Moreover, in section 5.1 we observed that augmentation with cloze-style QA pairs yields similar performance to using question-formatted QA pairs, offering evidence that the precise format is not as important as the content itself.
We prompt the model in the following format: After reading the article, «context» the doctor said «sentence» for PubMed articles. For other target corpora we replace doctor with engineer, journalist, and poster for Stack Overflow, DailyMail, and Reddit respectively. To filter out invalid sentences, we remove any generation that: 1) includes a number, 2) does not repeat part of the passage verbatim, and 3) has less than 75% word set overlap with the passage (after removing stopwords). To gauge the precision of our generations, we sampled 20 generated sentences for each dataset and found that they are correct more than 70% of the time. To test retriever performance, we train a DPR model with source domain data and ∼8k examples containing pairs of original passage and generated sentence for each target dataset. We observe performance improvements of
∼18% on NewsQA, ∼13% on CliCR, and ∼24% on Quasar-S (Table 4). Moreover, LLMs contain substantial factual knowledge in their parameters and we observe that they do particularly well in a closed-book setting on datasets with trivia-based factual questions, like SearchQA and Quasar-T, but do not perform well in other cases. Following our conjecture in §3, datasets with Full shift on average show an improvement of 12.1% with few-shot interventions, compared to 3.5% with zero-shot, which is also evident in Fig. 1. We show qualitative examples in Appendix (Fig. 8).

heading: 6 Related Work
text: Domain generalization in readers The most popular work in generalization in reading comprehension was introduced as part of the MRQA (Fisch et al., 2019) challenge, which focuses on transfer learning from multiple source datasets to unseen target datasets (Gottumukkala et al., 2020). This multi-task learning setup requires the model to perform complex reasoning at test time that may be unseen at training. However, in this work, we focus on the generalization capabilities of an end-to-end ODQA setup that is able to understand passages in the new domain and not the ability to perform unseen reasoning.
Domain generalization in retrievers A recent line of work that tests domain generalization of retrievers (Petroni et al., 2021; Ram et al., 2022; Izacard et al., 2022) focuses on conservative changes to the source domain, for instance, testing generalization of a model trained on Natural Questions applied to WebQuestions (Berant et al., 2013) or TriviaQA (Joshi et al., 2017), all of which use the same Wikipedia corpus. BEIR is a recent retrieval benchmark, (Thakur et al., 2021) tests the generalizasbility of only the retriever in isolation and not end-to-end ODQA performance, which is a brittle metric.
Domain adaptation work in retrievers (Dai et al., 2022) generate passages using few shots but do not require the answer to be correct. Ma et al. (2021) performs a zero-shot adaptation using noisy labels for retrievers. Siriwardhana et al. (2022) utilizes examples from the target domain in a transfer learning setup while we work in a zero to a few shot setting.
Domain generalization in other tasks Incidental supervision signals in (He et al., 2021) deter-
mine which dataset has a useful signal for a target classification task. Similar to (Fisch et al., 2019), in machine translation, various works (Dua et al., 2022; Fedus et al., 2022) learn to balance positive and negative feature transfer from multiple source domains to a target domain.

heading: 7 Conclusion
text: We investigate failures of ODQA models under non-conservative dataset shift. We also propose a way to test compatibility of source model with new domains without much supervision. We establish how different dataset shift behave under a variety of intervention schemes. We hope future research will adopt these target datasets for evaluation.

heading: 8 Limitations
text: This work focuses on English QA datasets only. Similar techniques should apply in other languages as well; however, we did not evaluate them. The augmentations generated are difficult to validate for yes/no questions for the few-shot method. Moreover, it can be challenging to generate these augmentations if access to large LM is unavailable. However, under those scenarios, data in the target domain should be annotated, which ideally would perform better than the few-shot setting. Our models also suffer from similar problems as LLMs, like hallucinations, misinformation, etc.

heading: 9 Ethics Statement
text: This works focuses on testing the generalization and adaptability of general-purpose models to various domains. It uses existing training data and conventional methods of testing model performance. This work does not deal with any social impacts or biases in natural language processing systems.

heading: Acknowledgements
text: We would like to thank William Cohen, Haitian Sun, Tom Kwiatkowski and the anonymous reviewers for their feedback. This work was partly supported by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research.

heading: A Experimental details
text: We used JAX on TPUs for training reader models and PyTorch on GPU for training retriever models. We used open-source github implementations for DPR 4, Contriever 5 and Spider 6. For retrieving top-100 passages for reader input, we used ScaNN7 library. We use T5-base model for reader and BERT-base for retriever. We fine-tune the retriever and reader with learning rate 1e-5 and 5e-5 respectively.

heading: B How are evaluation sets curated?
text: We consider validation sets from each of the target dataset, BioASQ, CliCR, Quasar-S, Quasar-T, NewsQA, SearchQA, COLIEE as part of our evaluation set. SearchQA, Quasar-S and Quasar-T were already published as ODQA datasets so we used them as it is while we had re-purpose some of the other datasets that were not originally ODQA dataset by processing them as described below.
COLIEE: The COLIEE Shared Task 48 provides a list of Japanese legal codes in English language. To convert these legal codes from a flat list into paragraphs, first we split them into specific article sections with regex string "Article [0-9]+ ". We further split each article into passages containing a maximum of 256 words.
NewsQA: We created an index on CNN/Dailymail documents by splitting them into passages of 256 words and pooled them together to create a corpus.
CliCR and BioASQ: We used PubMed corpus published as part of BEIR (Thakur et al., 2021) benchmark. We split the pubmed abstracts in this corpus into passages of size 256 words.
C Varying context distribution
As described in §5.1, we test retriever (Fig. 5) and reader performance (Fig. 6) when exposed to different set of passage. Fig. 6 shows reader performance with passages retrieved with BM25 on source (i.e. wikipedia), target (i.e. respective target corpus) and combined (i.e. all corpora pooled together). Fig. 5 compared performance of Spider and BM25
4https://github.com/facebookresearch/DPR.git 5https://github.com/facebookresearch/contriever.git 6https://github.com/oriram/spider 7https://ai.googleblog.com/2020/07/announcing-scann-
efficient-vector.html 8https://sites.ualberta.ca/ rabelo/COLIEE2022/
with Target (i.e. dataset specific target corpus) and Comb (i.e. all corpora pooled together)
0 20 40 60 80
COLIEE
SearchQA
NewsQA
Quasar-T
Quasar-S
CliCR
BioASQ
NQ
62.38
71.88
9.62
42.33
54.73
24.77
41.92
83.37
57.66
12.05
56.63
62.75
24.67
43.41
68.19
82.56
70.76
51.25
70.53
83.05
45.05
62.08
78.11
61.47
72.61
37.47
69.73
84.74
44.59
62.64
77.48
Spider-Target Spider-Comb BM25-Target BM25-Comb
(a) Spider-Comb has 0% Acc@k on this COLIEE due to a large numbre of distractors.
0 20 40 60 80
COLIEE
SearchQA
NewsQA
Quasar-T
Quasar-S
CliCR
BioASQ
NQ
73.39
63.03
12.54
54.76
50.36
23.87
50.41
77.1
52.19
20.03
59.1
63.74
23.88
52.37
74.2
82.6
74.8
26.5
72.8
52.1
40.9
34.8
81.7
47.7
77.5
29.9
74.7
71.2
42.5
56.6
78.7
DPR-Target DPR-Comb Cont-Target Cont-Comb
(b) DPR-Comb has 0% Acc@k on this dataset due to a large numbre of distractors.
Figure 5: Retriever Performance (Acc@100): Varying context distribution by creating a combined document index. For COLIEE, we use oracle passages for performance computation.
D Varying answer distribution and pre-training corpus
Following §5.1 we try to understand the impact of pre-training and fine-tuning corpus on answer distribution. We do this by comparing the performance of the FiD reader initialized from T5 pretrained on common-crawl dataset(C4) compared to one that was pre-trained on PubMed articles (Table 5). After pre-training, both models are then fine-tuned on our source domain data. In this case,
we observe that fine-tuning on a domain that differs from that used in pre-training results in deterioration of model performance.

heading: E Degree of domain shift
text: In Table 1, we showed only differences that governed which side of decision tree the shift types were categorized into, while in Table 6 we show all the raw distance values.

heading: F Statistical Significance
text: The number of examples in all datasets except COLIEE are in the order of thousands, making the performance improvements significant. In the case of COLIEE, which has a boolean output space (i.e. answers are yes/no), we performed a binomial test to test the significance of few-shot reader performance in Table 4. The number of samples n = 116 (number of test examples), p0=0.468 and pt=0.616. We will reject the null hypothesis that baseline and few-shot distribution are equivalent, when P(X >= pt ∗ n) <= 0.05, where X is drawn
from a binomial distribution, i.e., X ∼ B(n, p0) (Berg-Kirkpatrick et al., 2012) and we can compute the L.H.S to be, P(X >= 0.616 ∗ 116) = 0.00006 making it significant.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
7 A2. Did you discuss any potential risks of your work? This is a study of existing work, we do not use a lot of compute resources for running these experiments so this point is not applicable.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1(introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 4 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, 5 and Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We hope future research will adopt these target datasets for evaluation