SlowGuess/NAACL_2022_OCR huggingface dataset:
In Figure 3, we also note that as the pattern frequency increases, the performance between comp-gen and novel-entity diverges (for concrete question pattern examples see Figure 5). This gap has a significant effect on overall model performance, since common patterns make up a majority of the test set. Based on error analysis (see Appendix A.5 for details), we hypothesise that in the retrieved passages for comp-gen questions, answers do not always co-locate with the question anchor words. This indicates future research should encourage the retriever to fetch passages that cover all aspects of the question in order for it to be answerable. Under the assumption that the model could answer all patterns of questions equally well, regardless of frequency, the overall performance would be improved by  $\sim 11\%$ .
4.2 How do Non-parametric Models Handle Comp-gen Questions?
We use the decomposed atoms as the basis for our analyses on comp-gen questions. Following the previous subsection 2.1, we know that wiki Entities leverage crucial semantics for factoid questions and Wikipedia is the most widely used source of knowledge in current ODQA datasets (Yang et al., 2015; Hewlett et al., 2016; Rogers et al., 2021). Therefore, we would like to carefully study if the training set wiki Entities frequency affects model perfor
Figure 4: Plot showing the influence of the wiki Entities frequency in the question. The x-axis represents the wiki Entities frequency in the training set and we use the most frequent wiki Entities in each comp_gen question.
mance. Figure 4 plots the EM score as a function of how often a test question's wiki entity appears in a training question. We see that test accuracy is anticorrelated with the training-set frequency of test questions' entities. At first glance, this result seems surprising, and inconsistent with the well-known difficulty of modeling long-tail phenomena. However, the following interpretation helps to explain this apparent contradiction.
We manually inspect the questions with the most frequent wiki Entities, and find most of them are questions about countries, which is a frequent question topic in the NQ training set. For example, for the question "How many farmers are there in the USA", almost all the retrieved passages are highly relevant. The gold answer is "3.2 million" with the context "There were 3.2 million farmers". The model, however, generates the answer "2.2 million", taken from the context "There were 2.2 million farms...". Both passages come from an article titled "Agriculture in the United States", and the model is failing to draw a distinction between farms and farmers. While it is easier to retrieve relevant documents for questions with more frequent wiki Entities (Chen et al., 2021), the passages retrieved for high-frequency entities are much more likely to contain type-consistent close-negatives and distractors, making it more difficult for the model to select the correct answer. In other cases, questions are highly ambiguous, such as, "What is the average salary for a US congressman?", the gold answer $174,000 applies for the year 2012, while predicted answer $169,300 applies for the year 2008. For NQ, the existence of high-frequency entities could be indicative of an ambiguous ques
tion. If we conduct an analysis using the NQ dev set annotations provided by Min et al. (2020), we note that  $50\%$  of questions with the entity "US" and  $64\%$  of questions with the entity "NBA" are ambiguous. To quantify the impact, using FiD as an example, we note that if we match the performance of comp-gen questions with common wiki Entities to those with the unpopular wiki Entities, the accuracy could be improved with  $\sim 4\%$  points.
Besides wiki Entities, it's prudent to consider the remaining atoms as well. The results are illustrated in Figure 7 and some findings are observed in the following: 1) For question word, all models achieve better performance for questions asking about WHO and WHICH, while performing worse on questions without any question word. Although EM scores drop significantly for WHY questions, it is hard to draw conclusions as there are only limited number of them in the test set. 2) There is no clear correlation between model performance and verb frequency. Some of the "best performing" verbs are: sing, sang, wrote, and play, which closely correlate with the most frequent question patterns such as "who sing song [ent]". 3) Since there is no clear correlation between model performance and other_args frequency either, we group test questions based on the number of other_args in each of the questions. It shows that models achieve higher EM scores on questions with fewer other_args. Interestingly, the most performing other_args are closely related to WHO and WHICH questions, such as "s wife", "main character", and "tv show", while the "worse performed" other_args are mostly the comparative and superlative adjectives such as "biggest house" and "second largest" (also observed in Dua et al., 2019).
To summarize, the remaining atoms are codependent on each other, especially for limited-length factoid questions. They should preferably be treated as a single unit (e.g. question pattern) to arrive at the meaning of the question. In essence, their compositionality cannot be ensured and isolated (Dankers et al., 2021). Wiki Entities on the other hands are independent of the context. The question is meaning-preserving even under wiki Entities substitution. The subpart for ODQA compositionality should focus on wiki Entities and question patterns. As discussed above, their individual frequency have different impacts on the various components of ODQA models.
4.3 How do Non-parametric Models Handle Novel-entity Questions?
Although we explicitly categorize unseen questions into comp-gen and novel-entity, broadly speaking, questions with novel entities also require the model to generalize to novel compositions and thus could be considered to belong to the comp-gen category. We seek to understand if the novel entities are the main bottleneck for ODQA models, or the model can handle them well enough to process the questions appropriately. To explore this issue further, we run an ablation study, where, at inference time, we replace the novel entities in the question and the support passages with an entity that has been seen from the training set. Our experimental setup is working under the following constraints: 1) There can be only one wiki-entity mentioned in the test question, so that replacing it will not risk altering the semantics of the original question. 2) The replacement entity must not be present in the original test question or its retrieved passages.
We run the inference for FiD model on 100 eligible questions, and find the model rarely changes its predicted answers, despite the modification, with  $73\%$  of the predicted answers remaining unchanged. We manually verified the remaining questions and observe that some differences are due to inherent limitations of our entity-swapping process, such as errors in entity-linking (see Appendix A.6 for examples). Interestingly, we find that three altered questions give the right answers, despite originally generating incorrect ones. Given these observations, we suggest that the model learns relatively good contextual embeddings for the novel entities by exploiting the context provided by the passages. Thus, specific unseen entities are not the main bottleneck for the model to locate the desired answers.
5.1 Open Domain Question Answering
Early systems relied on surface text pattern matching methods to detect answers (Ravichandran and Hovy, 2002; Soubbotin and Soubbotin, 2001). For traditional ODQA systems, linguistic experts first identify a set of question types and expected answer types using rule-based mapping methods for each type of questions (Allam and Haggag, 2012). The input question needs to be classified into a certain type or taxonomy in order to be answered (Li and Roth, 2002; Suzuki et al., 2003). This approach is
sub-optimal for most realistic use-cases, as it is not possible to enumerate all possible question types.
With the introduction of deep neural networks, recent ODQA system mostly adopt a "Retrieve-and-Read" architecture, popularized by Chen et al. (2017), retrieving relevant documents for a given question and inferring an answer from these documents. Recent retriever models learn to encode questions and documents into dense vectors to score their similarity (Lee et al., 2019; Karpukhin et al., 2020; Khattab et al., 2021). Reader models can be categorized into extractive models that predict an answer span within the document (Das et al., 2018; Lin et al., 2018; Yang et al., 2019) and generative that generate answers condition on the question and the retrieved passages (Lewis et al., 2020b; Izacard and Grave, 2021). Recent ODQA models provide substantial improvements over traditional systems (Zhu et al., 2021), but as shown in Section 4.1, they still struggle with complex and infrequent questions.
5.2 ODQA Model Analysis
Retrieving relevant passages is an essential component for open-book ODQA models. A broad spectrum of recent work apply transformer (Vaswani et al., 2017) models such as BERT (Devlin et al., 2019) for information retrieval (Yates et al., 2021). Following the success of using pretrained language models (Craswell et al., 2020), studies have been made regarding their properties. Luan et al. (2021) compare the lexical-matching abilities of these models to traditional methods such as BM25. Ma et al. (2021) and Wang et al. (2021) study reproducibility, and demonstrate improvements by combining lexical-matching and dense retrievers. Thakur et al. (2021) introduce the BEIR benchmark to study zero-shot generalization for multiple neural retrieval approaches. Their conclusion is consistent with our findings that there is considerable room for improving the generalization of dense-retrieval models.
To infer answers from retrieved documents, models generally use a reader component implemented as a neural Machine Reading Comprehension (MRC) model. Previous work has analyzed the MRC model by crafting adversarial attacks (Jia and Liang, 2017; Mudrakarta et al., 2018), studying the difficulty of popular benchmarks (Kaushik and Lipton, 2018), and demonstrating annotation bias (Gururangan et al., 2018; Sugawara et al., 2018; Chen
and Durrett, 2019). Despite the success for various datasets, there is little work analyzing the whole pipeline of question answering systems. Lewis et al. (2021a) showed that models perform substantially worse on questions that cannot be memorized from training sets. Krishna et al. (2021) found that long-form question answering (LFQA) systems do not ground their answers in the retrieved passages. In contrast, for ODQA, we observe that when we replace retrieved passages with randomly-sampled passages at inference time, the model FiD (Izacard and Grave, 2021) largely fails to correctly answer any questions (see Appendix A.7 for experimental details). Gu et al. (2021) define similar generalization levels based on schemas for Knowledge Base Question Answering. However, our setting works without a schema and our generalization categories are derived from question decomposition atoms.
We study ODQA model generalization and categorize unseen questions into three subsets: overlap, comp-gen, novel-entity. Treating questions as being compositional, we decompose them into atomic elements based on their semantics. We believe that this decomposition strategy can help future work related to question structure and unification. We evaluated several recent ODQA models on these three subsets for three popular datasets. Our experimental findings both pinpoint the specific problems when handling different categories of novel questions and shed light on how to compositionally approach the factoid questions in ODQA task.
Ali Mohamed Nabil Allam and Mohamed Hassan Haggag. 2012. The question answering systems: A survey. Science, 2(3).
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2018. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013
conference on empirical methods in natural language processing, pages 1533-1544.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250.
Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021. Evaluating entity disambiguation and the role of popularity in retrieval-based nlp. arXiv preprint arXiv:2106.06830.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879.
Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4026-4032.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2021. The paradox of the compositionality of natural language: a neural machine translation case study. arXiv preprint arXiv:2108.05885.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2018. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Philipp Dufter, Nora Kassner, and Hinrich Schütze. 2021. Static embeddings as efficient knowledge bases? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2353-2363.
Hady Elsahar and Matthias Galle. 2019. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163-2173.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477-3488.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112.
Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1545.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. Are pretrained language models symbolic reasoners over knowledge? In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552-564.
Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015.
Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. 2019. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations.
Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Relevance-guided supervision for openqa with colbert. Transactions of the Association for Computational Linguistics, 9:929-944.
Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940-4957.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Brenden M Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In ICML.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020a. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021a. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000-1008.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021b. Paq: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-1115.
Belinda Z Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433-6441.
Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1736-1745.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329-345.
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A replication study of dense passage retriever. arXiv preprint arXiv:2104.05740.
R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third Blackbox NLP Workshop on Analyzing and Interpreting Neural Networks for NLP, Online. Association for Computational Linguistics.