SlowGuess/NAACL_2022_OCR huggingface dataset:
Linqing Liu† Patrick Lewis‡† Sebastian Riedel‡† Pontus Stenetorp†
†University College London ‡Facebook AI Research
{linqing.liu,patrick. Lewis, s.riedel, p.stenetorp}@cs.ucl.ac.uk
Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions. However, it is unclear which aspects of novel questions make them challenging. Drawing upon studies on systematic generalization, we introduce and annotate questions according to three categories that measure different levels and kinds of generalization: training set overlap, compositional generalization (comp-gen), and novel-entity generalization (novel-entity). When evaluating six popular parametric and non-parametric models, we find that for the established Natural Questions and TriviaQA datasets, even the strongest model performance for comp-gen/novel-entity is  $13.1 / 5.4\%$  and  $9.6 / 1.5\%$  lower compared to that for the full test set – indicating the challenge posed by these types of questions. Furthermore, we show that whilst non-parametric models can handle questions containing novel entities relatively well, they struggle with those requiring compositional generalization. Lastly, we find that key question difficulty factors are: cascading errors from the retrieval component, frequency of question pattern, and frequency of the entity.
Over the last few years we have seen model innovations improving on standard natural language processing (NLP) benchmarks across the board (Devlin et al., 2019; Raffel et al., 2020; Lewis et al., 2020a). However, it is still clear that we are yet to obtain generalizable language understanding, as recent work has found that adversarial (Jia and Liang, 2017; Mudrakarta et al., 2018; Belinkov and Bisk, 2018) and out-of-distribution samples (Talmor and Berant, 2019; Elsahar and Galle, 2019; McCoy et al., 2020) remain challenging for existing models across numerous tasks.
Figure 1: Questions categorized according to their relation to the training set: 1) Overlap: there exists a paraphrase of the question in the training set. 2) Compositional: all individual facts and the structure of the question has been observed across several questions in the training set – but not the given composition. 3) Novel entity: the question contains at least one entity (marked here with yellow) not present in the training set.
Open-domain question answering (ODQA), which aims to answer factoid questions without any given context, is a task that has been receiving increasing attention in the community (Chen et al., 2017; Lee et al., 2019; Karpukhin et al., 2020; Izacard and Grave, 2021; Min et al., 2021). However, recent work has shown that there is a large discrepancy in model performance between questions and answers observed at train time and novel questions and answers - even if they are derived from the same distribution (Lewis et al., 2021a). This raises the question: "What are the aspects of these novel questions that make generalization challenging?" which we seek to explore in this paper.
In work on systematic generalization (Bahdanau et al., 2018; Lake and Baroni, 2018; Ruis et al., 2020), it is argued that even though a model has only observed a very small subset of all possible combinations of facts during training time, a good model should be able to generalize to all possible combinations of facts at test time. We draw upon these ideas to study generalization for ODQA and define the following three categories to support our investigation: training set overlap, composi
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2014 - 2029 July 10-15, 2022 ©2022 Association for Computational Linguistics
tional generalization, and novel-entity generalization. See Figure 1 for definitions and examples. Our categorization breakdown is motivated by how they capture different levels of generalization: overlap requiring no generalization beyond recognizing paraphrases, comp-gen requiring generalization to novel compositions of previously observed entities and structures, and novel-entity requiring generalization to entities not present in the training set. It is worth noting that we explicitly study in-distribution generalization rather than out-of-distribution generalization (such as cross-domain generalization (Fisch et al., 2019)), as we will later demonstrate that even in-distribution generalization poses a major challenge for existing approaches.
We decompose and manually annotate three previously introduced ODQA datasets (Natural Questions (Lee et al., 2019), TriviaQA (Joshi et al., 2017), and WebQuestions (Berant et al., 2013)). Following this, we evaluate six recently proposed non-parametric and parametric ODQA models and analyze their performance, using both aggregate metrics and a breakdown according to our proposed categories. Non-parametric and parametric models differ in their access to information: the former has no access to any external context or knowledge, whereas the latter is provided relevant information alongside the question (Roberts et al., 2020).
One potential source of difficulty could be the question structure itself and as a byproduct of our decomposition approach we are able to derive a high-level question pattern for each question. We find a strong positive correlation between the pattern frequency in the training set and test accuracy. We then study how non-parametric models handle the comp-gen and novel-entity subsets respectively, since the performance on them is significantly worse than on the overlap subset. For comp-gen questions, perhaps surprisingly, we find that the frequency of entities mentioned in a question is strongly negatively correlated with test accuracy. For novel-entity questions, when we replace novel entities in the question and its support passages with entities seen in the training set the performance remains largely unchanged; we thus hypothesize that specific unseen entities are not the main bottleneck for model performance but rather a failure of the model to generalise compositionally. Aside from questions, we further analyze the retrieved passages and find the retrieval accuracy is equally lacking for the comp-gen and novel-entity
subsets, at  $\sim 75\%$  for top-20 accuracy. We also observe that many of the passages that do contain the correct answer lack sufficiently informative contexts for the question anchor words for the reader model to be able to locate it, indicating a need to either improve the reader models ability reason over multiple passages or the retriever model to provide passages with richer contexts.
To conclude, our key contributions are as follows: 1) We provide the first detailed study on generalization for ODQA, based on categories that measure different levels and kinds of generalization, that we use to annotate three previously proposed ODQA datasets<sup>1</sup>. 2) We show that for novel questions, non-parametric models handle novel question entities comparatively well, while they struggle to perform compositional generalization. 3) We demonstrate and quantify key factors that impact model generalization performance, which we believe will show the direction for future research towards more robust and generalizable ODQA models.
2 Dataset Construction
In this section, we describe how we process and annotate ODQA datasets to enable us to investigate generalization.
2.1 Question Decomposition
To study the compositional and novel-entity generalization of questions, we follow Keysers et al. (2019) and propose to view each question as being composed of primitive elements (atoms). Consider the question "Who got the first Nobel Prize in Physics?". The atoms intuitively correspond to the modifier or adjunct of the predicate "who", predicate "got" and the entity "first noble prize in physics". The combination of these atoms cover the main semantics of the question.
The way we measure generalization necessarily depends on how we break down the questions into atoms. Following manual analysis of questions from three popular ODQA datasets, we developed the following decomposition strategy to obtain atoms which cover all the desired question semantics. These are: question words, verbs, Wikipedia named entities (wiki Entities), and finally, other arguments (other_args) which correspond to other relevant aspects of the question. We explicitly ex
1Our data and code are available at https://github. com/likicode/QA-generalize
Figure 2: Example decomposition for the question "Who is the main character in Green eggs and ham?"
Group	Natural Questions	WebQ	TriviaQA
Overlap	837	501	458
Comp-gen	1,105	512	475
Novel-entity	597	640	456
Table 1: Number of questions for each generalization subset for the three datasets' test sets
tract wiki Entities since they leverage crucial semantics in factoid questions and other_args define essential details surrounding wiki Entities.
In order to automatically decompose questions, we first use an off-the-shelf semantic role labeling (SRL) model (Shi and Lin, 2019) to produce predicate-argument structures for each question. This provides us with the verb (i.e. the predicate), and semantic arguments. The question word is trivially obtained by identifying WH-words. We apply an off-the-shelf entity linking model (Li et al., 2020) to obtain the wiki Entities in the question. Finally, other_args are the SRL arguments which remain after we filter out arguments corresponding to wiki Entities. An example question decomposition is illustrated in Figure 2. More details are included in Appendix A.1.
2.2 Generalization Category Definitions
Based on the question decomposition, we define three generalization categories for ODQA datasets. We denote  $S_{q}$  as the set of the decomposed atoms of question  $q$  and  $C_{Q}$  as the complete set of decomposed atoms for all the questions in dataset  $Q$ . Our category subsets are then defined as:
-  $Q_{\text{overlap}} \triangleq \{ q \in Q_{\text{test}} \mid \exists q' \in Q_{\text{train}}, S_q \subseteq S_{q'} \}$
-  $Q_{\mathrm{comp\_gen}} \triangleq \{ q \in Q_{\mathrm{test}} \mid \exists q_1', q_2', \dots, q_k' \in Q_{\mathrm{train}}, S_q \subseteq \bigcup_{i=1}^{k} S_{q_i'}, S_q \not\subsetneq S_{q_i'} \}$
-  $Q_{\text{novel\_entity}} \triangleq \{ q \in Q_{\text{test}} \mid \exists s \in S_q, s \notin C_{\text{train}} \}$
For overlap test question, there exists a training question where they have the same decomposed atoms or are subset of them; for comp_gen test
question, its decomposed atoms are fully covered by the training set (a subset of the union of multiple training questions atoms), but not in one particular training question; and for novel-entity test question, there exist wiki Entities not present in the training set.
2.3 Question Categorization and Human Verification
With the decomposed atoms for all questions, we first categorize the test questions into overlap, comp-gen, and novel-entity categories based on the definitions of each generalization category. We optimize the selection criteria to cover as many eligible candidates for each category as possible. Further details can be found in Appendix A.2.
As our test set subsets are obtained automatically, we need to perform manual human verification to ensure that they are of high enough quality to draw empirical conclusions. To do this, we employ four expert annotators and use the following annotation process for each of the respective categories. Overlap: Annotators are shown  $q_{\mathrm{test}}$  and the training questions with the highest degree of character-level overlap. If any of these questions are a paraphrase of  $q$ , the annotator will mark  $q_{\mathrm{test}}$  as an overlap question. Comp-gen:  $q_{\mathrm{test}}$  is presented to the annotators along with the training questions with the highest degree of word overlap. Annotators then verify that the test question is truly a compositional generalization and not a paraphrase of any of the given training questions. Novel-entity: Annotators need to: 1) Verify that the wiki Entities identified by the entity-linking model are indeed wiki entities. 2) Verify that the entities in  $q_{\mathrm{test}}$  are not present among a set of questions from the training set whose entities have a high degree of character-level overlap with the entities in  $q_{\mathrm{test}}$ . Statistics for the annotated category subsets are summarized in Table 1, examples are shown in Table 2, and additional details are covered in Appendix A.3.
We analyse three widely used ODQA datasets, each one is briefly introduced as follows:
Open Natural Questions (NQ) is an open-domain variant of Natural Questions (Kwiatkowski et al., 2019) introduced by Lee et al. (2019). This dataset consists of questions mined from Google
Group	Test question	Paired training question for annotator	Label
Overlap	who got the first Nobel prize in physics	who won the first Nobel prize in physics	T
whens the last time the patriots played the eagles	when did the philadelphia eagles last win the super bowl	F
Comp-gen	when is the next scandal episode coming out	when is next fairy tail episode coming out	T
what is the corporate tax rate in great britain	what is the rate of corporation tax in uk	F
Novel-entity	who wrote the song the glory of love	who sang guilty of love in the first degree	T
who sings too much time on my hands lyrics	who sings i’ve got too much time on my hands	F
Table 2: Example of questions from Natural Questions (see Appendix A.8 for examples from the other two datasets) for human verification and their respective annotated labels (T for True and F for False).
search logs, with answers annotated as short spans of text in Wikipedia articles by crowd-workers. The NQ questions are generally simple, short, and information-seeking, as the questioner is unlikely to have known the question's answer when they formulated it. It consists of 79,168 train, 8,757 dev, and 3,610 test question answer pairs.
TriviaQA (Joshi et al., 2017) consists of questions and answers which were obtained by scraping trivia websites. TriviaQA questions are generally less information-seeking than those in NQ, and exhibit substantial syntactic and lexical variability. We use the open domain splits which contains 78,785 train, 8,837 dev, and 11,313 test question answer pairs (Lee et al., 2019). Answers in TriviaQA are Wikipedia entities, and any alias of the answer entity is considered a correct answer. We randomly sampled and annotated 2,000 questions from the test set for our analyses.
WebQuestions (Berant et al., 2013) consists of questions that were collected by performing a breadth-first search using the Google Suggest API. The questions in WebQuestions resemble those in NQ, but are generally shorter and simpler and demonstrate less variability. WebQuestions' answers are Freebase (Bollacker et al., 2008) entities, annotated by crowdworkers. It contains 3,778 train and 2,032 test questions.
Non-parametric models mostly adopt a retrieve-and-read framework, retrieving relevant Wikipedia documents for the given question, and then produce the final answer conditioned on these documents. We consider two generative reader models: Retrieval-Augmented Generation (RAG, Lewis et al., 2020b), and Fusion-In-Decoder (FiD, Izacard and Grave, 2021). RAG combines a DPR (Karpukhin et al., 2020) dense retriever with a BART (Lewis et al., 2020a) generator, which are jointly fine-tuned end to end. FiD is a
pipeline approach which uses DPR to retrieve a set of documents, and the decoder attends over all encoded document representations to generate the final answer. As an extractive reader model we use the reader component from DPR (Karpukhin et al., 2020). It extracts answer span from the highest-scoring document ranked from a passage selection model. We also include RePAQ (Lewis et al., 2021b), a QA-pair retriever which does not follow the retrieve-and-read paradigm. It retrieves QA-pairs from PAQ, a large resource of 65M automatically-generated QA-pairs, returning the answer of the most relevant QA-pair.
Parametric models are directly trained with QA pairs without access to an external corpus and thus store the required knowledge in its entirety in the model parameters. For our analyses, we include a BART-large model (Lewis et al., 2020a) and a more powerful T5-11B model (Roberts et al., 2020). They are both trained with questions as input and output question-answer pairs.
3.3 Model Category Analysis
Table 3 shows the Exact Match scores for models on our test set splits.
Non-parametric models on novel-entity questions For the non-parametric models, EM scores on novel-entity questions are relatively close to their overall total scores, with an average drop by  $6.5\%$  and  $3.1\%$  on NQ and TriviaQA respectively, with the exception of WebQuestions. The questions in WebQuestions only contain a single entity, which also tend to be high frequency entities. However, due to the very small size of the WebQuestions training set, many of these questions are considered to be in the novel-entity subset, despite containing relatively frequent entities, which, with a larger training set, would likely be classified as comp-gen questions, querying various relations regarding known entities.
Model	Natural Questions	TriviaQA	WebQuestions
Total	Overlap	Comp -gen	Novel -entity	Total	Overlap	Comp -gen	Novel -entity	Total	Overlap	Comp -gen	Novel -entity
Non-parametric	RAG	44.49	75.75	30.41	37.69	56.83	87.12	47.58	47.81	45.52	80.64	33.40	31.88
FiD	53.13	78.85	40.00	47.74	67.69	90.39	58.10	66.23	-	-	-	-
DPR	41.27	71.33	25.88	33.84	57.91	82.31	46.11	58.99	42.42	73.45	31.05	31.25
RePAQ	47.26	78.61	34.21	36.85	52.06	89.08	42.95	38.38	-	-	-	-
Parametric	T5-11B+SSM	36.59	81.48	17.47	12.56	-	-	-	-	44.69	81.24	35.35	25.78
BART	26.54	76.34	5.88	3.35	26.78	78.38	11.37	10.09	27.41	70.46	13.28	8.75
Non-parametric models on comp-gen questions
Surprisingly, the performance of all non-parametric models degrades significantly on the comp-gen subset (drop by  $14.2\%$  on NQ,  $10.2\%$  on TriviaQA and  $11.7\%$  on WebQuestions). This finding suggests that non-parametric models struggle to perform compositional generalization, whereas they handle novel question entities comparatively well. We investigate this finding in greater detail in Section 4.
Parametric models on novel-entity and comp-gen questions parametric model performance drops significantly on both comp-gen and novel-entity subsets, but they achieve relatively higher EM scores on comp-gen questions. This indicates that novel-entity questions are more challenging for parametric models. This makes intuitive sense, since, for entities not seen during training, parametric models will struggle to "know" enough about the entity to generate a correct answer. In such cases, we find evidence that parametric models often resort to generating answers from superficially similar training questions, with  $63.2\%$  and  $53.3\%$  of answer predictions also occurring in the training data for T5-11B+SSM on NQ for comp-gen and novel-entity questions respectively.
Implications for modeling Among the nonparametric models, FiD achieves the highest EM scores for both comp-gen and novel-entity questions. FiD aggregates multiple passages together when generating answers. In contrast, the extractive DPR reader only uses the highest-scoring passage to extract the final answer. Based on observations from the experiment in Appendix A.4, we hypothesize that the NQ FiD model adopts a strategy similar to a reranker, and extracts an answer from the highest latently-relevant document.
Although without access to external knowledge but only automatically-generated QA-pairs in advance when answering questions, RePAQ still
Table 3: Exact Match scores for each model. "Total" refers to the overall performance on the full test set. "Overlap", "Comp-gen", and "Novel-entity" refers to the model performance on the respective subset.
NQ	Total	Overlap	Comp-gen	Novel-entity
Top-20	80.1	89.5	74.7	75.4
Top-100	86.1	92.0	82.4	83.1
Table 4: Top 20 and Top 100 retrieval accuracy on NQ test set for the DPR retriever.
achieves higher or comparable performance as retrieve-and-read model RAG and DPR. It indicates that generating, storing and retrieving questions is a valid path in terms of model generalization.
Parametric models perform significantly worse compared to non-parametric models. BART struggles to answer any novel questions correctly, while T5-11B+SSM performs better due to much larger capacity. Petroni et al. (2019) demonstrate that language models are able to recall factual knowledge without any fine-tuning and can somewhat function as an unsupervised ODQA system. However, our experiments suggest that, large-scale language models (when fine-tuned to directly answer questions using a set of training QA pairs) struggle to answer questions about low frequency entities and relations, similar to the findings of Kassner et al. (2020) and Dufter et al. (2021).
Additional observations All models perform significantly higher on overlap questions, consistent with the findings of Lewis et al. (2021a). Parametric models with more parameters are the most effective at rote-memorizing training questions, and T5-11B+SSM even outperforms the non-parametric models on NQ and WebQuestions.
4 How Do Non-parametric Models Generalize?
Experimental results show that the performance of non-parametric models degrades significantly on the comp-gen subsets across all datasets. In this section, we would like to examine what the under
Figure 3: Influence of question pattern frequency, where test questions are binned based on the frequency of their question pattern in the training set.
lying challenge is for these questions. We focus on the NQ dataset as it has the largest annotated test set among three datasets.
Table 4 shows the top-  $k$  retrieval accuracy - which is the number of questions for which at least one passage of the top-  $k$  retrieved passages contains the gold answer. The difference in retrieval accuracy between comp-gen and novel-entity splits is relatively small (< 1%), but is significantly lower than the overlap subset results. This indicates that the retriever performance is a confounding factor for the overall performance of comp-gen and novel-entity questions. Solely improving the retriever would benefit the model greatly for the subsets requiring generalization. Allowing us to study the reader model in isolation, for the remainder of our analysis we will only use the subset of questions for which there is at least one support passage that contains the gold answer.
4.1 Effects of Question Pattern Frequency
One might ask questions such as "Who plays the doctor in Sons of Anarchy?" and "Who plays Stacey's mum in Gavin and Stacey?". Although semantically different, they share the structure "who plays [entity] in [entity]", which we refer to as a question pattern. To study if the frequency of
these patterns affect model performance, we collect question patterns by replacing all wiki Entities in a question with the token [entity], unifying the prepositions, and stemming each word.
We group test questions for each category by the frequency of their patterns in the training set. In Figure 3, we analyze FiD as an example since it achieves the highest EM score on unseen questions (results for other models can be found in Figure 6 in the Appendix). In the upper figure, the EM scores show that the model is more likely to make correct predictions for more common patterns. Given this observation, we would like to investigate if the significant performance edge of the overlap category is due to a larger percentage of more frequent patterns. According to the lower figure, which shows the proportion of questions for each frequency bin, the frequency distribution for each category is largely similar. Therefore the performance gap between overlap and the other two categories can not simply be explained by a difference in pattern distribution.