SlowGuess/NAACL_2022_OCR huggingface dataset:
Sewon Min, Jordan L Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, et al. 2021. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783-5797.
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1896-1906.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Deepak Ravichandran and Eduard Hovy. 2002. Learning surface text patterns for a question answering system. In Proceedings of the 40th Annual meeting of the association for Computational Linguistics, pages 41-47.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426.
Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2021. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. arXiv preprint arXiv:2107.12708.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. 2020. A benchmark for systematic generalization in grounded language understanding. Advances in Neural Information Processing Systems, 33.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Martin M Soubbotin and Sergei M Soubbotin. 2001. Patterns of potential answer expressions as clues to the right answers. In TREC.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208-4219.
Jun Suzuki, Hirotoshi Taira, Yutaka Sasaki, and Eisaku Maeda. 2003. Question classification using hdag kernel. In Proceedings of the ACL 2003 workshop on Multilingual summarization and question answering, pages 61-68.
Alon Talmor and Jonathan Berant. 2019. Multiqa: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911-4921.
Nandan Thakur, Nils Reimers, Andreas Rückle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021. Bert-based dense retrievers require interpolation with bm25 for effective passage retrieval. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, pages 317-324.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72-77.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013-2018.
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials, pages 1-4, Online. Association for Computational Linguistics.
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.
A.1 Question Decomposition
Below is a random selection of question decomposition examples from the NQ dataset. In each question,  $\underline{\mathbf{x}}_{qw}$  denotes the question_word,  $\underline{\mathbf{y}}_{verb}$  denotes the verb, and the spans of other_args and wiki ents spans are denoted by brackets. Note that these structure slots are not always fully present in the question (e.g., Q3, Q4, Q6, Q7, Q10).
As we rely on automated systems as a part of our decomposition process, this leads to the following limitations. At times, the ELQ model fails to label wiki_ents, such as for Q8 where every light in the house is marked as other_args. Furthermore, as seen in Q9 there is the possibility of multiple question words being present although our approach only extracts a single question_word. Limitations such as these is one motivation for why we elected to perform manual verification for each question (Section 2.3).
1.  $\underline{\mathrm{Who}}_{qw}$  is verb the [other_args: owner] of [wiki Entities: Reading Football Club]?
2.  $\underline{\mathrm{Who}}_{qw}$  died verb in the [other_args: plane crash] [wiki Entities: Grey's Anatomy]?
3. [other_args: Cast] of [wiki Entities: Law & Order Special Victim Unit]?
4.  $\underline{\mathrm{When}}_{qw}$  did [wiki Entities: United States] enter verb [wiki Entities: World War I]?
5. Where  $q_{w}$  are most [wiki Entities: nutrients] absorbed in the [wiki Entities: human digestive tract]?
6. When  $q_{w}$  did the [other_args: government] change verb the [other_args: retirement age]?
7.  $\underline{\text{What}}_{qw}$  is\_verb the [other_args: name] of the [other_args: gap] between [other_args: two front teeth]?
8.  $\underline{\mathrm{Who}}_{qw}$  sings verb [other_args: every light in the house is on]?
9. Where  $_{qw}$  are  $_{verb}$  the [wiki Entities: Winter Olympics] and when do they start?
10. [wiki Entities: Swan Lake] [wiki Entities: the Sleeping Beauty] and [wiki Entities: the Nutcracker] are verb [other_args: three famous ballets] by?
A.2 Question Collection for Human Verification
We use the following selection criteria to collect candidate questions for human verification. For the overlap subset, as a first step, each  $q$  is paired with each train question that shares the same answer or have answers which are a sub-sequence of  $q$ 's answer. As a second step, we then require that the train question's similarity measurement score to  $q$  is over a pre-defined threshold and that they have the same wiki Entities as  $q$ . For the remaining test questions, we consider  $q$  as a candidate for compgen if all of its parsed elements are covered by the collection of all parsed elements in the training set. Lastly, if there exists any novel wiki Entities in  $q$  which are not present in the training set,  $q$  is considered as a novel-entity candidate.
A.3 Generalization Subset Details
As guidelines for the human annotators, we provide the following to resolve ambiguous or potentially problematic cases: 1) For overlap, we only consider questions that are superficial paraphrases and exclude those that require more complex forms of reasoning (e.g. Who played Mark on the show The Rifleman? / Who played the boy on the show The Rifleman?). 2) For comp-gen, all other_args in the test question must be covered in the collection of training set entities and all question_word atoms alongside with the verb must be present in the training set. However, there are questions where other_args are not covered in the training set (e.g. Animation Resort) or are highly specific due to the decomposition processing and thus not covered (e.g. fourth movie compared to movie or three different types compared to types) and are thus excluded from comp-gen. 3) For novel-entity, there are cases when ELQ fails to extract wikients in questions because of words variation, such as Who sang It Going to Take a Miracle? compared to the correct wikients It's Gonna Take a Miracle. 4) There are also intrinsic problems in the datasets, some test questions are exactly the same as train questions but paired with different answers: (Where did Dolly Parton grow up? with the answer Tennessee and Where did Dolly Parton grew up with the answer Sevierville). Following this manual verification, for Natural Questions, WebQuestions, and TriviaQA,  $70.3\%$ ,  $81.3\%$ , and  $69.5\%$  of their test questions are covered in the generalization subsets respectively.
Figure 5: Examples of question patterns and EM scores for their corresponding questions. For each question pattern, we sample the same number of comp-gen and novel-entity questions. The two uppermost patterns are the most frequent (thousands of occurrences), the following two are of medium frequency (hundreds of occurrences), and the last is a novel pattern.
A.4 FiD Performance Analysis
Among the non-parametric models, FiD achieves the highest EM scores for both comp-gen and novel-entity questions. We are interested in understanding if FiD's improved performance is due to leveraging a greater amount of contextual evidence provided by multiple passages, or whether it simply generates the most frequently-mentioned plausible answer. We perform a simple experiment, by first collecting 544 questions answered incorrectly by FiD, where the gold answers occur less frequently than FiD's predicted answer in the retrieved passages. We then adjust the retrieved passages so that the original predicted answer and gold answer are mentioned an equal number of times, by masking out some of the original prediction mentions. After adjusting the frequencies, we regenerate the answer predictions, and observe that FiD only produces 44 correct answers out of 544. This suggests that answer mention frequency is not the governing feature for FiD when generating answers on NQ. It suggests the NQ FiD model adopts a strategy similar to a reranker, and extracts an answer from the highest latently-relevant document.
A.5 Additional Question Pattern Analyses
We sample the same number of comp-gen and novel-entity questions for each example pattern, and display the results in Figure 5. We checked several instances for the pattern "who play [ent] on [ent]", and find that the model fails more on comp-gen questions partially because the retrieved pas
sages do not provide enough information to locate the answer. For example, for the question "Who played Mary in Christmas with the Kranks?" none of the retrieved passages contain both Mary and the movie name. The model produces the answer Julie Gonzalo from the passage Julie Gonzalo Julieta [...] is an [...] actress. [She] is also known for her roles "Christmas with the Kranks", whereas the gold answer is Felicity Huffman from the passage She also starred in [...] "Christmas with the Kranks". Since "Mary" is not mentioned in either passage, it is impossible to infer that the correct answer is Felicity Huffman. The support passages for novel-entity questions, on the contrary, more often cover both of the anchor entities (e.g. context Little Boy Blue is an ITV drama series ... Stephen Graham was cast as Detective ... for the question "Who played the detective in Little Boy Blue").
A.6 Additional Non-parametric Generalization Analysis
When analyzing the performance impact of the frequency of wiki Entities in questions, one will have to account for the fact that there might be more than one entity present in the same question. In our analysis in Section 4.2 we consciously only considered the most frequent entity in a question. Note that we also experiment with the least frequent entity and they show the same negative correlation between entity frequency and performance.
As we noted in Section 4.3, at times the novel entities in the original question may not match the corresponding mentions in the passage due to errors from the entity linking step. For instance, for the question Who sings So Come and Dance with Me Jai Ho? we swap the entity span "So Come and Dance with Me Jai Ho", however, this span is too wide as an entity as the correct entity would be "Jai Ho". Therefore the model is unable to match the correct song name in the passage; thus giving a different answer. Other error cases can be attributed to the granularity of the predicted answer: e.g. "624 CE" and "13 March 624 CE". We do however note that for the great majority of cases our entity-swapping procedure works as intended.
A.7 Answer Grounding in Retrieved Passages
We noted in Section 4 that we find evidence the FiD (Izacard and Grave, 2021) ODQA model does ground its answers in the retrieved passages. This observation can be contrasted to that of Krishna
Figure 6: Influence of question pattern frequency. Each figure is associated with one non-parametric model, which is DPR, RAG and RePAQ from left to right. The test questions are binned based on the frequency of their question pattern in the training set. The y-axis shows the Exact Match score on the NQ test set.
Figure 7: Influence of question word, verb, and other_args in the question (from left to right). For the two left-most figures, the test questions are binned based on the individual atom frequency in the training set, “-” indicates test questions whose question word or verb is not covered in the training set. For the right-most figure, the x-axis shows the number of other_args in each test question. All models are evaluated on the NQ test set.
Passage Processing	Total	Overlap	Comp-gen	Novel-entity
Original retrieved	53.1	78.9	40.0	47.7
50% random	53.2	78.3	39.9	48.3
99% random	55.5	74.3	46.1	54.0
100% random	3.6	5.1	2.0	3.0
Table 5: Comparison of FiD's predictions for the NQ test set, conditioned on the originally retrieved passages and a gradually increasing number of randomly chosen passages.  $\mathrm{x}\%$  means the percentage of retrieved passages are replaced with random ones. For  ${99}\%$  random,the rest passage is gold passage which contains the gold answer span.
et al. (2021), who found that answers to long-form questions were not grounded in the passage, in that models would provide the same answer regardless of the context provided. A complete picture of the results from our experiment can be seen in Table 5. We note that when the models is fed solely random passages it fails to answer nearly all questions  $(3.6\%)$ . However, but provided with half gold and half random passages, it performs on par with its original performance. Lastly, we note that when presented with a single gold passage and otherwise only random passages, the model is still able to
determine which passage is the gold passage and answer the question correctly – in fact, the performance even improves upon the original performance with more than more than  $5\%$  for comp-gen and novel-entity questions.
A.8 Additional Examples for three generalization subsets
Additional examples from Natural Questions are provided in Table 6, WebQuestions in Table 7, and TriviaQA datasets in Table 8.
Group	Test question	Train question
Overlap	Where does patience is a virtue come fromWho was the killer in the movie I Know What You did Last SummerWhen was the last time Arsenal win Premier LeagueWhere does blood go when it leaves the pulmonary artery	Where did the saying patience is a virtue come fromWho was the murderer in I Know What You did Last SummerWhen was the last time Arsenal won the Premier League titleWhere does blood go after the pulmonary artery
Comp-gen	What is the most popular religion in SwedenWhat are the main functions of the stemWho is in charge of ratifying treaties in the USCast of the Have and Have Nots play	What is the most popular religion in UkraineWhat are the main functions of the control busWho is in charge if president is impeachedThe last episode of the Haves and Have Nots
Novel-entity	Where does wild caught sockeey salmon come fromThe probability of making a Type I Error when retaining ... isWho was the Pinkerton Detective Agency's first female detectiveWhere was the world economic forum held this year	When was Sony walkman first sold in storesWhen was tower of terror built in DisneylandWho played detective Green on Law & OrderWho holds the world record for 100 meters
Table 6: Example questions from NQ test set.
Group	Test question	Train question
Overlap	Which is the highest waterfall in the worldIn the cartoon series, what kind of dog is Scooby DooWho directed the film “Gladiator”, starring Russell CroweWhich is the largest island in Canada	What is the tallest waterfall in the worldWhat breed of dog is Scooby-DooWho directed the film GladiatorWhat is Canada’s largest island
Comp-gen	- What nationality was the painter Vincent Van Gogh- What post was held by Winston Churchill during the 1926 general strike in the UK- By population, which is the second biggest city in France- In humans, the medical condition prepatellar bursitis affects which part of the body	- What nationality was painter Piet Mondrian- What role was played by Arthur CookIn the general strike of 1926- In terms of population, which is the second largest city in Finland 1926- The medical condition aerotitis affects which part of the human body
Novel-entity	- In ‘follow that camel’, the fourteenth carry on film, sid james was replaced by which us actor- Who has recently overtaken brian o’driscollto become ireland’s most capped player- Shining Tor is the highest point in which county- Who had a Too Legit to Quit tour	- What was the cause of death of carmen in the opera of that name- In the 2005 remake of king kong, who played the writer jack driscoll- Shinto is the main religion in which country- Which sweets were advertised as the Too Good to Hurry Mints
Table 7: Example questions from TriviaQA test set.
Group	Test question	Train question
Overlap	What is the currency of Puerto Rico called	What type of currency is used in Puerto Rico
Which countries speak German officially	What countries speak German as a first language
What language is spoken in Haiti today	What language do Haitian speak
What team is Hank Baskett on 2010	What team is Hank Baskett playing for in 2010
Comp-gen	What year was George W Bush elected	What is George W Bush's middle name
What year did the Seahawks win the Superbowl	In what Super Bowl did the Seahawks face the Steelers
Where did Queensland get its name from	From where did the Guillotine get its name
Where was Theodore Roosevelt buried	Where is George v1 buried
Novel-entity	Where did Andy Murray started playing tennis	When did Sean Murray first appear on NCIS
What time in Hilo Hawaii	Who was Phil Harris married to
Where did Bristol Palin go to school	What team is Chris Paul on
What time does American Horror Story air	Who made the American Red Cross
Table 8: Example questions from WebQ test set.