MoQA: Benchmarking Multi-Type Open-Domain Question Answering
Year: 2023

Abstract:
Previous research on open-domain question answering (QA) focuses mainly on shortanswered questions. However, informationseeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer. In this paper, we present MOQA1, a benchmark for opendomain QA that requires building one system that can provide short, medium, long, and yes/no answers to different questions accordingly. MOQA builds upon Natural Questions (Kwiatkowski et al., 2019) with multiple types of questions and additional crowdsourcing efforts to ensure high data quality. We adapt state-of-the-art models, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, training one retriever on all types achieves the overall best performance, but it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) State-of-theart large language models such as the largest GPT-3 models (Brown et al., 2020; Ouyang et al., 2022) also lag behind open-book QA models. Our benchmark and analysis call for more effort to build versatile open-domain QA models in the future.2

heading: 1 Introduction
text: Open-domain question answering (QA) leverages a large knowledge source such as Wikipedia to answer open-domain questions (Voorhees and Tice, 2000; Chen et al., 2017). Such a task mimics humans’ information-seeking process—finding relevant documents and composing answers based on them—and has potential to become a fundamental
1MoQA = Multi-type Open-domain Question Answering. It is pronounced as mocha.
2https://github.com/princeton-nlp/MoQA
component of next generation chatbots and search engines. An ideal open-domain QA system should be able to answer all types of questions human may ask, and provide answers in proper formats depending on the nature of questions.
However, for the ease of evaluation, most existing research on open-domain QA focuses on questions with short answers (Karpukhin et al., 2020; Lee et al., 2021; Izacard and Grave, 2020), originating from earlier work by Lee et al. (2019)3. This trend leaves a large portion of the open-domain QA task unattended: for example, Why questions often require sentence-level answers as you cannot explain “why is the sky blue” in a few words; yes-no questions like “is there a Prison Break 5 episode 10?” are also prevalent but largely neglected.
There has been recent efforts to study other forms of answers in open-domain QA (Fan et al., 2019; Stelmakh et al., 2022). In particular, GooAQ (Khashabi et al., 2021) is a benchmark consisting of different types of questions mined from Google autocomplete system and Google’s answers boxes without human annotation. Nonetheless, all
3Lee et al. (2019) only considered answers ≤ 5 tokens in Natural Questions and exact match as the metric. This setting has been adopted by numerous follow-up work.
8
of these works consider each type of questions separately. The closest work to ours is KILT (Petroni et al., 2021), which also considers different types of open-domain questions and builds one system to answer them. However, KILT is a collection of different tasks such as QA, fact checking, entity linking, etc., each with its own unique collection pipeline, which can introduce superficial cues that help the model determine the format of the answer.
Building one system for different types of questions has been studied in the reading comprehension setting (Kwiatkowski et al., 2019; Khashabi et al., 2020), where the supporting evidence is given as input. On the other hand, open-domain QA requires retrieving the evidence from a large text corpus or recalling specific facts from a model’s parametric knowledge, both of which poses significantly different challenges.
Different from previous work, we bring different types of questions drawn from the same distribution together, and aim to build a single open-domain system (either open-book or closed-book QA) that can handle various types of questions with finegrained annotations, based on a single knowledge source i.e., English Wikipedia. We further discuss the differences with past works in Section 6.
In this paper, we propose MOQA, a benchmark that requires an open-domain QA system to answer multiple types of questions (Figure 1). We build our benchmark by extending Natural Questions (NQ) (Kwiatkowski et al., 2019) into the open-domain setup, aligning each answer to a fixedlength Wikipedia passage, while keeping questions of four types: short, medium, long, and yes/no. We also use additional human annotations on the test set to filter out ill-defined long-answer questions, ensuring the high quality of MOQA evaluation.
We conduct a comprehensive set of experiments on MOQA with state-of-the-art QA models (Karpukhin et al., 2020; Izacard and Grave, 2021; Roberts et al., 2020; Brown et al., 2020; Ouyang et al., 2022), and reveal multiple interesting findings: • Retrieving text passages for different types of questions exhibits a similar difficulty, and simply training a dense retriever (Karpukhin et al., 2020) on all types achieves overall best performance.
• However, building a single reader remains challenging. Joint training of a reader model on all types performs poorly, and it is also difficult to train a classifier to identify question types before
applying individual reader models of each type.
• Finally, closed-book QA models trained on all questions as well as few-shot large language models (LLMs) without explicit retrieval significantly lag behind their open-book counterparts. Our findings suggest building such a versatile opendomain system still has a series of unsolved challenges and call for more effort in the direction.

heading: 2 The MOQA Benchmark
text: In open-domain QA, models are given a set of K passages D = {d1, . . . , dK} for a question q. In MOQA, the correct answer a to the question q could be a span ranging from a couple words to a paragraph, , YES, or NO. Unlike existing opendomain benchmarks that assume an answer span to be short (usually ≤ 5 tokens), MOQA does not assume a length limit and requires models to output different formats of answers considering the nature of each question.

heading: 2.1 Dataset Collection
text: We first build our dataset by adapting the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019) into the open-domain setting while keeping all types of questions presented below4. We use the 2018-12-20 Wikipedia snapshot as the knowledge source and slice it to 100-word passages following Karpukhin et al. (2020). We then align each question-answer pair to a supporting passage and slightly modify the gold answers so that they could be found in the passage. Table 1 shows statistics of our dataset, and about 33% of the dataset is not short-answer questions. For more details of the dataset, and preprocessing steps, see §A and §C. Short. The short-answer questions are defined as those having at least one short answer up to 5 tokens. It is exactly the same as Lee et al. (2019),
4We do not include unanswerable questions in the original NQ dataset, since many of them become answerable in the open-domain setting.
followed by most open-domain QA research. We keep our short-type questions consistent with the train/dev/test sets from Min et al. (2019).
Medium. The medium-answer questions can still be answered by span of text but contain more than 5 tokens. Most of them are noun or verb phrases or text around the length of one to two sentences, other than proper nouns, dates or numbers. For example, the question “what does the class on a sd card mean?” can be answered by “a minimum rate at which data can be written to the card”, which cannot be reduced to just a couple words.
Long. The long-answer questions have paragraph-level answers and cannot be answered by a short string. We focus on long-answers that originates from paragraph as these questions tend to be inherently complex and therefore require long natural language explanations, and filter out those originating from tables and lists.
Yes/No. The yes/no-type questions can only be answered by either YES or NO. The original NQ annotations consider them as short-type questions, but we separate them since yes/no-type questions often require a different reasoning ability (Clark et al., 2019) and model design.

heading: 2.2 Mechanical Turk for Data Filtering
text: From preliminary qualitative analysis, we found that the original NQ long-answer annotations are noisy (many questions are marked as ‘long answer’ even though they can be answered by a few words), we carry out a manual filtering for the long-answer questions in the test set to ensure that these questions truly require long explanations to answer. To this end, we use Mechanical Turk (MTurk) for further data filtering, where each question is annotated by at least five MTurk workers. Each worker is shown both the question and the original long answers, and decides if the question can only be answered by the long answer, and if not, what is the shortest answer.
The Fleiss kappa between the human annotators is 0.143 when judging if a question requires short or long answer. The low agreement suggests that the length of the answer could be subjective for some questions — the annotator’s familiarity with the subject may be a factor. Thus, we filter out any question that is marked as short by at least one worker. This provides a high guarantee that all human annotators agree that our long-type questions truly require long-form answers.

heading: 2.3 Evaluation
text: We use the original NQ development set as the test set and split the original NQ training sets into training and development sets with a 9 : 1 ratio.
For retrieval, we report A@k (accuracy at topk passages) and MRR@k (mean reciprocal rank). We consider a passage to be correct if it contains the long answer of the question. For answer prediction, we report exact match (EM) and F1 for short and medium answers, EM and ROUGE-L for long answer, and accuracy for yes/no answers. We also report the macro-averaged EM as the overall result.
Furthermore, we report the BERTScore (Zhang et al., 2019) for short, medium, and long-type answers in Table 8, and found that these metrics correlate with the F1 score and the ROUGE-L score. We also conduct human evaluation for selected models to study the difficulty of our dataset in Section 5. For more details about the metric, please refer to Appendix D.

heading: 3 Adapting QA models for MoQA
text: We establish multiple baselines for MOQA, by adapting state-of-the-art open-domain QA systems, including both supervised open-book QA models (Chen et al., 2017; Lee et al., 2019), supervised closed-book QA models (Roberts et al., 2020), and LLMs (Brown et al., 2020; Ouyang et al., 2022).

heading: 3.1 Open-book QA Models
text: Open-book QA models first retrieve supporting passages from a large corpus such as Wikipedia, and then apply a more expensive reader to predict answers using the passages. We carefully consider both components in MOQA by training both the retriever and the reader on different portions of the dataset and examine their trade-offs.
Retrievers. Retrievers return the most relevant passages from a large text corpus for a given query. We investigate the trade-offs of training one retriever on all questions together versus training a separate retriever for each question types.
We use DPR (Karpukhin et al., 2020) for the retriever component. (1) For the jointly trained model, we include all questions, and use all annotated gold passages as positive passages and passages retrieved by BM25 that are not the gold passages as hard negatives. The resulting model is DPRall. (2) We also add a reference baseline, DPR†, which assumes we know the question type, and we
train a DPR model on only the corresponding type of questions. See §E for more details and baselines.
Readers. Readers leverages attention across both the query and the retrieved passages to predict an answer. Similar to retriever, we are interested in the trade-offs between training a single reader model for all question types versus training separate readers for each question type.
Furthermore, we use two common types of readers: (1) An extractive reader based on RoBERTa (Liu et al., 2019), which predicts the rank of a passage and the start and the end positions of the answer span given the concatenation of the question and the retrieved passage5. (2) the generative reader model Fusion-in-Decoder (FiD; Izacard and Grave, 2021) based on T5-base (Raffel et al., 2019), which achieves the state-of-the-art performance on short-form answers since it is able to attend to multiple passages during decoding. We then analyze the performance of readers with different architectures on different types of questions.
For each reader model, we consider training them on all questions combined, as well as training them on each individual question type (there will be four readers in this case). For the latter, we need to train an extra question classifier in the pipeline to decide which individual reader to be applied. The question classifier is implemented by feeding the question into a RoBERTa model (Liu et al., 2019) and training a classifier on top of the [CLS] token (see §G for details).

heading: 3.2 Closed-book QA Models
text: Unlike open-book QA models, closed-book QA models entirely rely on their parameters to generate the answers. We consider both fine-tuning approach and few-shot in-context learning approach. Furthermore, we examine how the more powerful LLMs behave under the settings where the question type is either given or not given.
We follow Roberts et al. (2020) and fine-tune a T5-large model (Raffel et al., 2019) on all the questions in the training set, since the fine-tuned T5 model achieve impressive results on short-form QA and closed the gap to open-book models. See Appendix H for training details.
We evaluate the largest GPT-3 models: davinci (Brown et al., 2020) and text-davinci-003 (Ouyang et al., 2022), due to their abilities to re-
5To support yes/no answers, we prepend every passage with two special tokens: [YES] and [NO].
call factual knowledge and performance on opendomain QA benchmarks. For each LLM, we use in-context learning with eight-shot ICL consisting of two demonstrations from each question type. We randomly sample one question from each type to construct sets of four demonstration, and balance the answer labels for yes/no-type questions. We also consider an oracle setting where the question type is given, and we only include two demonstrations from that question type for comparison.
Additionally, we use GENREAD (Yu et al., 2023) and prompt InstructGPT to first generate a supporting passage before outputting the final answer. Previous works showed that generating a supporting evidence improves the LLM’s performance on short-form QA, and we investigate the method’s robustness to different question types. For details on the prompting and examples, see Appendix H.2.

heading: 4 Results
text: Retriever: different types of questions have similar difficulty. We first demonstrate the retrieval results in Table 2. We can see that overall using all question type training data outperforms using the corresponding training data for each type, and the performance across different types does not differ much. We assume this comes from the nature of retrieval – because retrieval is mostly about topic, answer types do not matter much and training data can generalize to all types of questions.

heading: Retriever Reader Short Medium Long Yes/No Avg.

heading: Oracle Question Types

heading: Open-book QA Models

heading: Closed-book QA Models
text: Reader: challenging to train one versatile reader that handles all questions. Table 3 shows the main QA results. Different from the retrieval component, we see that a single reader trained on all types of questions shows very imbalanced performance on different subsets. The single reader also significantly lags behind the oracle (marked by †) that assumes knowing the question type and applies corresponding readers. Between
the extractive RoBERTa and the generative FiD, we find that FiD performs better on short-answer and yes/no questions but RoBERTa is better on medium and long-answer questions, even though in previous work FiD always prevails.
Though the oracle model is much better than the single reader, we see that training question classifier and 4 readers underperforms training jointly. This comes down to the difficulty to train the question classifier – Figure 2 shows the confusion matrix for question type prediction. It is noticeable that medium and long-answer questions are the most difficult to predict. For example, it is not trivial to understand that the question “who is covered under payment of gratuity act 1972” requires a long answer. Improving the classifier will be a straightforward to advance MOQA performance, as the current systems still have a huge gap to the corresponding type oracles. Please refer to Appendix J for qualitative examples.

heading: Closed-book QA models significantly lag behind.
text: Though competitive on short-answer questions as shown in previous work, closed-book QA models perform poorly on other types, especially on longanswer questions. Both the trained T5-large model
as well as the LLMs lag behind open-book QA models. Our result suggests that it is still challenging to solve MOQA relying purely on parameters.
One possible explanation is the fact that more complex questions and extensive explanations require more memorization than short-form answers from the closed-book QA models, so it is harder to rely on knowledge stored in the model’s parameters for this task. Furthermore, these explanations are less likely to appear in the pre-training corpus than popular entities such as people’s names, famous places, and significant dates. Even for models such as GPT-3 that has been trained on an extremely large corpus, these answers still appear to be harder to recall than shorter answers.
Generating supporting passages improves performance. By first generating a supporting passage, InstructGPT can improve its performance on all question types except for long questions. Our findings are consistent with previous works that LLM can improve downstream performance with intermediate generation steps (Yu et al., 2023; Wei et al., 2022; Kojima et al., 2022). From qualitative analysis, we found that the vanilla prompting strategy results in text-davinci-003 generating long and excessive answers, hence the low performance on short and medium-type questions. The additional generation step allows the model to output more concise and refined answers.
However, text-davinci-003 performs worse with GENREAD as its outputs becomes shorter for long-type questions. We will further analyze this behavior in Section 5.

heading: 5 Human Evaluation

heading: 5.1 Annotation Process
text: To further analyze the difficulty of MOQAand gain insights into how and why current models fail, we randomly sample and annotate 25 outputs from each question type on the test set for selected models: We assign two scores to each output: a correctness score and a sufficiency score, and show the results in Figure 3.
Correctness score. We first score the output on the factuality and relevance of the output: correct means the output is factually correct with respect to the question; relevant means the output is relevant to the query but not factually correct; wrong means the output is factually incorrect and irrelevant to the query.
Sufficiency score. We then determine if the output provides the appropriate amount of context: lacking means the output lacks the amount of explanation and context expected for the query; good means the output contains the perfect amount of explanation; extra means the output supply excessive explanation.
For more details on the human evaluation and examples of the annotations, please see §I.

heading: 5.2 Analysis
text: Extractive reader can adapt to short, long, and yes/no type questions. Outputs from the RoBERTa model are often both correct and contain the right amount of information. We also noticed that the model will always return either YES or NO for yes/no type questions, which supports our finding that boolean questions are relatively easy to identify. Since the outputs are always a span from the top 24 retrieved passages, they are almost always relevant to the query.
Generative models tend to generate shorter responses. Consider the examples shown in Table 4, where the question is inherently complex as it is asking about the historical origin of the phrase “go commando”. In this example, the RoBERTa reader outputs the ground truth answer, while the other models fail in various ways. DPRall + FiDall simply generates “unknown”, which is irrelevant to the question and does not answer it at all. davinci elaborates on the word “commando” but does not consider the phrase “go commando". GENREAD+ text-davinci-003 only offers one explanation, while the complexity of the query necessitates more context given its historical background.
All models typically output relevant information. Even when the model does not output the correct answer, they often return relevant information. For instance, for the question “who plays the woodsman in Over the Garden Wall”, the correct answer is “Christopher Lloyd”. davinci returned “Tom Kenny”, who is a voice actor involved in many animated series similar to Over the Garden Wall.
Furthermore, one of the main challenges of longtype questions is the complexity of the query. Generative models output incorrect answers more often on long-type questions, which suggests that composing an answer with detailed explanations and historical contexts means more room for the model to make mistakes.
All models struggle with the boundaries of medium-type questions. All model sometimes output insufficient or excessive explanations for medium-type questions, which suggests that the boundary between medium-type questions and the other question types are not quite clear. Indeed, this can be the case since medium-type answers may contain characteristics of both short and long-type answers. For example, the answer “Harry Potter and the Deathly Hallows” is more than 5 tokens long, but it is also a proper noun typically found in short-type answers. On the other hand, some answers such as “a transformative change of heart; especially: a spiritual conversion” is more descriptive and similar to long-type questions.

heading: 5.3 Output Length Distributions
text: To illustrate the issue of providing insufficient and excessive information, we show the difference be-
tween the length of the output generation and the length of the ground truth answer in Figure 4.
Specifically, let Ai = {a1, a2, . . . , an} be the set of n possible answers to the question qi and len(s) be the function that returns the number of tokens in the string s after whitespace tokenization. We define the difference in length ∆li between output oi with Ai:
∆li = (lo − Lmin)1lo<Lmin + (lo − Lmax)1lo>Lmax
where lo = len(oi), Lmax and Lmin are the maximum and minimum of the answers Ai, respectively.
Extractive model is better at identifying the question type. Even though the RoBERTa model often returns outputs shorter than the ground truth of long-type question, its ∆li distribution is centered around 0 for all question types. In contrast, all generative models’ outputs often lack explanation for long-type questions. Interestingly, GENREAD causes text-davinci-003 to generate shorter outputs on average as the density of the ∆li distribution shifts left. One possible explanation for this is that text-davinci-003 treats the final output as a summary of the generated passage. As a result, it ends up generating sequences shorter than the passage.

heading: 6 Related Work
text: Long-form QA. ELI5 (Fan et al., 2019) mines questions and answers from the subreddit Explain Like I’m Five6, the answers are abstractive and are not grounded in a knowledge source. Similar to our work, ASQA (Stelmakh et al., 2022) also considers long-form answers in open-domain QA and its questions originates from NQ (Kwiatkowski et al., 2019). However, it only focuses on the questions that have long answers due to ambiguity. GooAQ (Khashabi et al., 2021) propose a dataset consisted of different question types, but these questions are mined from Google’s answer boxes and therefore noisy. Furthermore, ELI5, ASQA, and GooAQ all study each question type independent of each other.
Extension of Natural Questions. Recent works leverage the original NQ beyond those with only short answers. For example, AquaMUSE (Kulkarni et al., 2020) use the long answer only questions from the original NQ for the task of query-based multi-document summarization. They consider NQ
6https://www.reddit.com/r/explainlikeimfive/
questions that only have long answers as these question “result in open-ended and complex topic answers". BoolQ (Clark et al., 2019) focuses on boolean questions that can be answered by true or false, and expands on the yes/no questions from the original NQ by collecting additional questions using the same pipeline. The authors found the task challenging and require robust reasoning ability. The unanswerable questions in NQ were explored by Asai and Choi (2020), but such questions are difficult to study in the open-domain setting and require extensive manually annotations. In contrast to these works, we study draw a diverse set of questions with different answers from the same query distribution and study these types together.

heading: Multi-type Question Answering UnifiedQA
text: (Khashabi et al., 2020) builds a single QA system to answer different types of questions. However, their setting is limited to the closed-domain setup, and only analyzes fine-tuned generative models.
LLMs achieves impressive performance on many knowledge-intensive tasks due to the knowledge packed in its parameters (Brown et al., 2020). They do not require any additional fine-tuning and only rely on in-context examples to adapt to different tasks. A recent line of work proposes to further improve LLMs capabilities by prompting them to generate a series of intermediate reasoning steps (Wei et al., 2022). GENREAD (Yu et al., 2023) applies a similar idea to open-domain QA, where the LLM first generates a supporting passage instead of relying on external retriever models, and then output the answer.

heading: 7 Conclusion
text: We propose MOQA, an open-domain QA benchmark with multi-type questions, and evaluate a range of baseline models. Our findings suggest that the main difficulty lies in the reader’s task, and building a versatile reader or an accurate classifier is challenging. Competitive closed-book QA models also degrade on the new benchmark. Improving the classifier or mining more data to build a generalizable reader might be promising directions towards truly all-round systems.

heading: Limitations
text: Though our ultimate goal is to build a versatile QA system that can handle all types of questions, our benchmark mainly focuses on extractive questions – those can be explicitly answered by copying from a document in the knowledge source. We start from extractive QA because they cover a wide range of real-world questions and are easier to be automatically evaluated.
Although we addressed the issue of long-form QA evaluation with human evaluation and a range of automatic evaluation metrics, there is still much room for improvements in terms of evaluation of long-form text — human evaluation can be expensive and non-reproducible while current automatic metrics are not without faults. We encourage future work exploring various evaluation strategies of long-form QA.
Furthermore, all questions are in English and possibly collected from English-speaking users. We also use the English Wikipedia as our knowledge source. Thus, our models and dataset may under-represent the non-English speakers.

heading: Ethical Statement
text: Training language models can use significant amount of energy as the process is very computationally expensive, this can come at an environmental cost. In our work, we attempt to minimize this effects by using pre-trained models like RoBERTa (Liu et al., 2019) and only doing the necessary fine-tuning to minimize the computation cost. Furthermore, to promote reproducible and accessible academic research, we will publicly release all of our dataset and code. Natural language datasets can contain biases like gender and racial stereotypes. Although this issue is not as prominent in QA datasets compared to large copra used for pre-training language models, we encourage the community to build robust QA models that are more resistant to these biases.

heading: A Dataset Statistics
text: We show detailed statistics about the MOQA dataset in Table 5. We also show the majority label for our yes/no questions in Table 6.

heading: B Mechanical Turk
text: The annotation results of our MTurk campaign on the test set long-type questions is shown in Table 7.

heading: C Pre-processing Steps

heading: C.1 Original Natural Questions
text: Natural Question is a question answering dataset collected from the Google search engine. Real Google queries are first inputted into the Google search engine to obtain the top ranked Wikipedia page. Then, both the question and the Wikipedia article are given to annotators. The annotators first decide if a question is of good quality, in which case the annotator will attempt to find an answer to the question within the Wikipedia page. The annotators may not find a valid answer or select an HTML bounding box–typically a natural paragraph or a table–as the long answer. If a long answer is selected, then the annotator can choose to select a substring as the short answer or annotate the answer as simply "YES" or "NO". For more details about the original Natural Questions dataset, we refer readers to Kwiatkowski et al. (2019). If the annotator does not find an answer or deem the question of bad quality, then the annotator would denote the answer as "NULL".
We first obtain all the questions that are found in NQ-Open (Lee et al., 2019) and denote them as short-type questions. Specifically, these questions are all the questions that have at least one annotated answer that is 5 words or shorter.
For all other questions, we first filter out all questions where the top Wikipedia page title contains "(disambiguation)", "(disambiguation page)", "(List of .+)", "(Index of .+)", or "(Outline of .+)". Then, we use the Python library BeautifulSoup7 to clean the HTML tags and obtain the text of each long answer(s) and short answer(s) associated with the question (if they exist).
We denote all questions with at least one short answer as medium-type questions. We denote all remaining questions with at least one YES/NO answers
7https://www.crummy.com/software/ BeautifulSoup/bs4/doc/
as yes/no-type questions. We denote all remaining questions with at least one long answer as long-type questions except for those questions where the long answer is originally from the HTML boxes "<table>", "<td>", "<tr>", or "<ol>" or if the length is shorter than 10 tokens. We do this because answers from tables and lists are often low quality after being converted to paragraph-like texts, when their structure is lost, or too short to fit the criteria of a long answer, and we discard these questions. We do not use any questions that were not annotated with a long answer.
Finally, we use the original development set as our test set, as the original test is hidden. Following the original Natural Questions, we filter out any questions in the test set that do not have at least 2 non-NULL annotations.

heading: C.2 Long Answers
text: Since Natural Questions were collected at different timestamps, each Wikipedia page and therefore the long answers and short answers are from different times. This typically don’t affect the short answers, because short spans of text can be easily found in different snapshots of Wikipedia.
However, long answers can be affected by minor grammar changes and the exact paragraph often cannot be found in different versions of Wikipedia. This poses a challenge for extractive models because some long answers might be unanswerable purely due to the difference in Wikipedia snapshots. Therefore, we replace every long answer with the corresponding string from the December 20, 2018, dump of Wikipedia, which is a popular version of Wikipedia dump used by many retrievers (Karpukhin et al., 2020, Lee et al., 2021). Specifically, we use the version where paragraphs are split into 100 token passages from Karpukhin et al. (2020) to minimize the variance in passage length.
To do this, we follow the matching strategy from KILT (Petroni et al., 2021). For every long answer, we first find the same Wikipedia article in our snapshot using the page title. Then, for each passage in the page, we find the span with the highest BLEU score to the long answer. We denote the passage containing the span with the highest BLEU as the gold passage, and the span as the long answer.
For long-type questions, we do an additional filtering where the best span must have a BLEU score of at least 0.5 to ensure that the matching long answer does not deviate significantly from the
original long answer, and we discard any long-type questions where we fail to find such span.
For yes/no-type questions, we only add the gold passage and matching long answer annotation if the best span’s BLEU score is 0.5, but we do not discard the questions where no matching long answer is found.
For short-type and medium-type question, we restrict the gold passage selection to only the passages where the short or medium answer is found as an exact substring, but use no BLEU score threshold when matching the long answer, because if the
short or medium answer is found within the same Wikipedia page, then the long answer is likely to be found in one of the paragraphs. We also do not discard the questions where no matching long answer is found.
Therefore, every long-type question is guaranteed to have at least one matching long answer, while the other types are not guaranteed to have any matching long answer.

heading: C.3 Unanswerable Questions
text: Although the original Natural Questions dataset (Kwiatkowski et al., 2019) includes questions that were not annotated with any answers, we chose not to include them in MOQA. This is because the nature of the dataset collection process does not guarantee the question to be unanswerable or truly have NULL as an answer.
Specifically, annotators are only shown the top Wikipedia result returned by the used search engine and they can only choose an extractive answer based on that Wikipedia article. However, it is not always true that the gold answer in the in top Wikipedia page returned by the search engine: they can exist in other pages.
Therefore, even though unanswerable questions are important to study (Rajpurkar et al., 2018), the questions not annotated with any answers in the original Natural Question cannot be taken as true
unanswerable questions without significantly more annotations.

heading: D Evaluation
text: For retrieval, we regard a retrieved passage is correct if the passage contains the matching long answer of the question (we exclude the questions without matching long answers in retrieval evaluation). We report accuracy@k that checks if the top k retrieved passages contains a correct passage. We also report mean reciprocal rank (MRR)@k, which averages the reciprocal rank of the top correct passage.
For the final answer, we report the exact match (EM) and the F1 score for short-type and mediumtype questions, EM and ROUGE-L for long answers, and accuracy for yes/no-type questions. Although EM is often the primary metric used in opendomain QA research, we believe that the F1 and ROUGE-L scores are more suitable for medium and long answers following past work on longform QA (Fan et al., 2019; Stelmakh et al., 2022; Khashabi et al., 2021). We also report the macroaveraged EM, which averages the EM scores across the 4 question types.
Furthermore, we also report the BERTScore(Zhang et al., 2019) on short, medium, and long-type questions. BERTScore is a model-based metrics that is better at capturing semantic similarities between long-form texts than traditional metrics. This is especially important for the medium and long type questions, since there are many possible ways of answering the question without using the exact same words. We use the authors’ implementation8 and the set-up with the best human correlation score at the time of writing: model is microsoft/deberta-xlarge-mnli9 and the layer is 4010. We found that the BERTScore correlates with F1 scores and ROUGE-L scores on short, medium, and long type answers. This suggests that the best performing models RoBERTaall and FiDall also output the most semantically similar texts to the ground truth answers.
8https://github.com/Tiiiger/bert_score 9https://huggingface.co/microsoft/
deberta-xlarge-mnli 10hash is microsoft/deberta-xlarge-mnli_L40 _no-idf_version=0.3.12(hug_trans=4.23.0)

heading: E Retrieval
text: We use the code repo11 from Karpukhin et al. (2020) for our DPR experiments. We also follow all of their hyperparameter settings; specifically, we use a learning rate of 2× 10−5, and a per GPU batch size of 16 and a gradient accumulation of 2 on 4 80GB A100 GPUs to achieve a total batch size of 128. We train for 40 epochs, except for DPRyesno-gold and DPRyesno-DS where we train for 400 epochs due to its small train set. We then use the model with the highest validation rank loss as our final model.
Similar to the original DPR, we also explore how the final retrieval model perform when using the annotated gold passage and when using a distantsupervision annotation approach that selects the positive passage based on the answer alone. These two strategies are particularly interesting to investigate in Natural Questions because it’s unique from other open-domain QA dataset such that all answers are grounded in one gold passage.
In this section, we describe our two approaches to training DPR, even though we only present DPR trained using gold passages in the main results.

heading: E.1 Distantly-Supervised DPR
text: In distant supervision, we first retrieve the top 100 passages from the corpus using an BM25 index. We use the Pyserini12 implementation of BM25. The input to the BM25 search is the question in each example, and we use the default BM25 hyperparameters also used by DPR (Karpukhin et al., 2020). Specifically, the parameters we use are: b = 0.4 (document length normalization) and k1 = 0.9 (term frequency scaling). Then, for each questionanswer pair, we choose the positive passages by checking if the answer exists within the retrieved passage.
Another interesting aspect to consider is which answer we use for selecting the positive passage. Namely, we can use either short or long answers for short questions, and medium or long answers for medium questions. From preliminary experiments, we found that using the long answers for choosing the positive passages for short questions performed better on the development set while using the medium answers for choosing the positive passages for medium questions. This is likely due to long answer filtering reduces the number of false
11https://github.com/facebookresearch/DPR 12https://github.com/castorini/pyserini
positives, and it is often likely to get the short answer string in irrelevant passages, and even though the training set size is reduced significantly, the higher quality of data makes up for the smaller dataset. However, using medium answers for filtering already obtain high enough quality of positive passages such that the harsher filtering with long answers will cut down on the training set size enough that it hurts retrieval performance.
When using long answer for filtering, we use a fuzzy matching where the positive passages are those that obtain a BLEU score of at least 0.5 with the long answer. When using short/medium answers for filtering, we check for exact match within the passage. All passages that were filtered out are used as hard negative passages.
For DPRall-DS, we use the strategy that worked the best for each individual types. Specifically, we use long answer filtering for short, long, and yes/no
questions and medium answer filtering for medium questions.

heading: E.2 Gold Passage Supervised DPR
text: Our main results use the gold passage supervised DPR. We follow the BM25 retrieval steps previously described, but we choose our positive passage differently. That is, we denote the gold passage as the positive passage for every question, and we choose all other retrieved passages as the hard negative passages, except for the gold passage if it were retrieved. Our final model DPRall uses all questions and their gold passages.

heading: F Reader

heading: F.1 Extractive Reader
text: For the reader model, we employ a similar architecture to the reader model from Karpukhin et al. (2020). However, one additional change we make
is prepending every passage with two special tokens: [YES] and [NO]. This is a necessary addition in order to always give the reader model the choice of outputting YES/NO. We separate these two special tokens from the rest of the passage with a separator token 
.
Before training the reader, we first run inference on the retriever and obtain the top 100 passages for each question. We train the reader model on 8 RTX-3090 GPU. We train RoBERTaall for 5 epochs, RoBERTashort and RoBERTamedium for 10 epochs, RoBERTalong for 20 epochs, RoBERTayesno for 40 epochs. We use a learning rate of 2× 10−5 and a batch size of 32 questions with 24 passages each; specifically, we use a batch size of 1 question with 24 passages per GPU and a gradient accumulation of 4. However, we use a total batch size of 16 for RoBERTayesno due to its small training set by reducing the gradient accumulation to 2.
We first did a hyperparameter sweep over batch size = {16, 32} and learning rate = {1×10−5, 2× 10−5} for each question type, and select our final model based on the development set performance.
For each question, we use 1 positive passage and 23 negative passages. The positive passages for short-type, medium-type, and long-type questions are passages that contain the gold answer, and the negative passages are any passages that do not contain the gold answer. We annotate the positive passage with the starting and ending position of all answers in the passage. The positive passages for yes/no-type questions are the retrieved passages with the highest F1 score with the long answer, and all other passages can serve as negative passages. In practice, we use the passages with the lowest F1 score with the long answer as negative passages. We annotate these positive passages with the correct answer span as the special tokens [YES] or [NO] accordingly to the answer.
For each question, we maximize the marginal log-likelihood of all correct spans in the positive passage combined with the log-likelihood of the correct passage being selected. We refer reader to Karpukhin et al. (2020) for more details.
During evaluation, we input the top 50 retrieved passages and select the span with the highest combined span score and passage rank score. We follow the computation in Karpukhin et al. (2020). We chose to include the top 50 passages by first sweeping over the hyperparameter k = {10, 25, 50, 100}, and chose the best k on the development set.

heading: F.2 Fusion-in-Decoder
text: We follow (Izacard and Grave, 2021) and use their code repo13 for training our FiD models, which uses T5-base. We follow the hyperparameters described in the original paper: the top 100 retrieved passages and their title are included in the input during training and evaluation, learning rate is 1 × 10−4, 10k total steps with 1k warm up steps, and a batch size of 64. We train on 8 RTX-3090 GPUs and a gradient accumulation of 8. We evaluate the model every 500 steps and chose the model with the best development set performance for our final model.
G Classifier
We train our classifier similar to sentence classification tasks. Specifically, we use the pre-trained RoBERTa-base (Liu et al., 2019) as our base model, and finetune it for 10 epochs over our entire dataset. The input is the question, and we use the last hidden state of the [CLS] token followed by a linear layer and softmax to predict the probabilities of four question types. We use Huggingface14’s implementation of RoBERTaForSequenceClassification to train our classifier; we use Cross Entropy Loss to maximum the log probability of the correct question type.
For our hyperparameters, we use a batch size of 8 and a learning rate of 1×10−5. We performed hyperparameter search over batch size = {8, 16, 32} and learning rate = {1×10−5, 2×10−5, 5×10−5}, and chose the model with the highest accuracy on the development set. We also evaluate the model after every epoch and choose the model with the best development accuracy.

heading: H Closed-Book QA

heading: H.1 Supervised models
text: For supervised closed-book QA, we use T5-large (Raffel et al., 2019) as our base model. T5-largeall is trained on all question types together. We follow (Roberts et al., 2020) for training with a constant learning rate of 0.001. We use a batch size per GPU of 8 and 8 32GB GPUs for a total batch size of 64. We train for 20k steps, and evaluate using beam search with a beam size of 4. We evaluate our model on the development set every 1k steps
13https://github.com/facebookresearch/FiD 14https://huggingface.co
and choose the model with the highest performance on the development set for our final model.

heading: H.2 Unsupervised Large Language Models
text: We use the following large language models in our experiments: GPT-3 (davinci; (Brown et al., 2020)), Codex (code-davinci-002; (Chen et al., 2021)), and InstructGPT (text-davinci-003; Ouyang et al., 2022).
The prompt used for the two-shot experiments with only two demonstration from one question type is shown in 5. We follow the same prompting strategy for the experiments with two demonstrations from each question type. Furthermore, we balance the in-context examples such that the first four demonstrations contain one example from each question type and that the last four demonstrations also contain one example from each question type. We also use one yes/no question with YES as the answer and another question with NO as the answer with the yes/no-type questions are used in the demonstrations.
For the experiments with GENREAD, we use the two-step prompting strategy shown in 6, but other experimental setups are identical to the experiments without GENREAD.

heading: I Human Evaluation
text: We describe our procedure for assigning the correctness score and the explanation in further detail here. We also show examples of the annotation in 11.
We use the following guidelines when assigning the correctness score:
1. We assign an output oi with correct(oi) = correct if it is factually correct and supported by Ai even if the word/phrasing is not exact. For instance, the question "Who is the 44th President of the United States" has multiple valid outputs: "Obama", "Barack Obama", and "Barack Hussein Obama" even though only "Barack Obama" is annotated as the old answer.
2. We assign an output oi with correct(oi) = relevant if it does not correctly answer the question qi according to Ai, but include some relevant information. For instance, an example of such output to the question "Who is the 44th President of the United States" include "Donald Trump" (who is the 45th President

heading: Completion → A:
text: Context → Q: What purpose did seasonal monsoon winds have on trade? Completion 1 → Passage: The seasonal monsoon winds were an important factor in the
development of trade in the Indian Ocean region. The monsoons were used to transport goods and people from the Arabian Peninsula to India, and from India to East Africa and Southeast Asia. The monsoons also allowed traders to travel from the Persian Gulf to the Bay of Bengal and the Malay Peninsula. The monsoons were also used to transport goods from India to China, and from China to the Middle East. The monsoons allowed traders to travel from the Persian Gulf to the Bay of Bengal and the Malay Peninsula.
Completion 2 → A: The seasonal monsoon winds were an important factor in the development of trade in the Indian Ocean region, allowing traders to travel from the Persian Gulf to the Bay of Bengal and the Malay Peninsula.
Figure 6: Example of zero-shot in-context learning with GENREAD used for large pretrained language models. We first prompt the model with Context+ Passage: to generate Completion 1. Then, we prompt the model again with the Context+ Completion 1+ A: to generate the answer. We take the generation from the second step as the final output.
of the United States) and "Joe Biden" (who was Obama’s Vice President as well as the 46th President of the United States). In practice, this is the most common mistake that models make, where if the ground truth answer is a politician, artist, or date, the model would typically output something in the same category. Furthermore, outputs are also put into this category if only parts of the output is factually correct but also contain unsupported information (e.g. "Obama and Biden" in this example).
3. We assign an output oi with correct(oi) = wrong if it does not correctly answer the question qi according to Ai and does not include any relevant information. For instance, an example of such output to the question "Who is the 44th President of the United States" could be "turtle eggs". In practice, this mistake rarely occurs, and we only assign this score when there is no obvious connection between the output and the question topic.
In general, the procedure for determining the explanation score involves first reading the question
and the ground truth answers to decide the minimal amount of information that is needed to answer the entire question. For instance, if the question asks "what are Poseidon’s symbols and what do they mean?", then an appropriate answer should include all of his symbols in Greek mythology (e.g. the trident, the horse, the bull, etc.) as well as the meaning of each symbol. If the output only include one symbol or multiple symbols without their meaning, then we would consider the output to be not sufficient, yielding a score of lacking. If the output includes things beyond the symbols but also the stories from Greek mythology about Poseidon, then we will consider the output to be too long, yielding a score of extra.

heading: J Prediction Samples
text: We show example output for selected models for each question type in Figures 12, 13, 15, and 14. It is interesting to note that while both models are capable of outputting the correct answers and the appropriate granularity for some question, they still have much room for improvement. For example, both models can generate a short answer where a long answer would be more appropriate, while
Question Question Model Output Correctness SufficencyType Score Score
Short who plays the woodsman inover the garden wall FiDall Christopher Lloyd Correct Good GPT-3all Tom Kenny Relevant Good Medium what is the meaning of thegreek word metanoia RoBERTaall a transformative change of heart; especially: a spiritual conversion Correct Good
GPT-3all metanoia is a Greek word meaning "change of mind" or "change of heart". It is used in the New Testament to describe a change of mind or heart in regards to one’s relationship with God.
Correct Extra
they can also generate some short answer where a yes/no answer would be correct.
It is also worth noting that questions that may appear to require short answers may actually require a much more length explanation. In other words, even though ”who”, ”when”, and ”where” questions may seem to only require short answers, they can actually sometimes require much longer explanation. For example, both ”Who is responsible for redrawing congressional and legislative districts?” and "Who is covered under payment of gratuity act 1972?" may appear to be simple and straight forward at first, but there is actually greater complexity behind the question that cannot be answered by just a few words, and it only becomes clear that they require length explanations given extensive knowledge on the topic. It is worth noting that our classifier model classifies both of those questions as short questions, which motivates future work into question classification.
Other questions that require long answers even though they appear to be short questions include:
• ”When is a system said to be reproducible?”
• ”Where does the word christ come from and what does it mean?”
• ”What makes miranda a central character in the tempest?”
• ”Where does the term spelling bee come from?”
In summary, question type classification is not as simple and straight forward as it may appear, and it actually requires extensive knowledge on the related background.

heading: Future Work
text: It is worth noting that our classifier model classifies both of those questions as short questions, which motivates future work into question classification. Other questions that require long answers even though they appear to be short questions include:
• ”When is a system said to be reproducible?”
• ”Where does the word christ come from and what does it mean?”
• ”What makes miranda a central character in the tempest?”
• ”Where does the term spelling bee come from?”
In summary, question type classification is not as simple and straight forward as it may appear, and it actually requires extensive knowledge on the related background