SlowGuess/NAACL_2022_OCR huggingface dataset:
jointly indexed. For the  $Text + tables + KB$  model, the KB relations are indexed separately. As described in §3.2, we use DPR to retrieve individual KB relations for each question, and the top-scoring KB relations are concatenated into 100-token passages to be fed to the reader. These passages are then merged with the passages retrieved from the  $Text + tables$  index using a fixed quota for KB relations. This quota is determined by maximizing retrieval recall on the development set (see §7.3). We also experiment with combining multiple KBs by using DPR to jointly retrieve from all relations of both KBs, which is straightforward to implement with our approach despite differences in the KB structure.
5.3 A multi-dataset model
In a realistic setting, the best knowledge source to answer a given question is unknown a priori to the system, but most open-domain QA datasets are collected with respect to a specific information source (e.g., Wikipedia for NQ and Freebase for WebQ). To better simulate the real-world scenario, we also experiment with a setting where we train a single model on the combination of all 4 datasets and evaluate without any input to the model as to the source of questions. We refer to this as the multi-dataset setting. This setting was previously investigated in several works (Karpukhin et al., 2020; Maillard et al., 2021; Qi et al., 2021), but not in the multi-source context. We train multi-dataset models for all 5 variants described above. The smaller datasets, WebQ and TREC, are upsampled 5 and 8 times respectively while training.
Main results are presented in Table 2. In the first set of experiments, we train a reader model independently for each dataset, as typically done in previous work. We use Freebase as knowledge base for WebQuestions as intended, and use Wikidata for all others. The multi-dataset model uses Wikidata.
The results highlight the limitation of current state-of-the-art open-domain QA models which use texts as the only information source. On WebQ, for instance, the KB-only model performs  $5\%$  better than the text-only one, and previous state of the art is also achieved by the KBQA model. Moreover,
4We normalize the questions by removing question marks and by presenting them in lowercase.
Model	NQ	WebQ	Trivia	TREC	Avg.
SoTA	51.41	55.13	67.61	55.32	57.3
Retrieval-free	28.54	30.64	28.74	-	-
Per-dataset models
Text	49.0	50.6	64.0	54.3	54.5
Tables	36.0	41.0	34.5	32.7	36.1
KB	27.9	55.6	35.4	32.4	37.8
Text + tables	54.1	50.2	65.1	53.9	55.8
Text + tables + KB	54.0	57.8	64.1	55.3	57.8
Multi-dataset model
Text	50.3	45.0	62.6	45.7	50.9
Tables	34.2	38.4	33.7	31.1	34.4
KB	25.9	43.3	34.2	38.0	35.4
Text + tables	54.6	44.3	64.0	48.7	52.9
Text + tables + KB	53.7	55.5	63.4	51.3	56.0
Table 2: Exact match results on the test set. SoTA numbers are from (Izacard and Grave,  ${2021}{)}^{1}$  ,(Iyer et al.,  ${2021}{)}^{2}$  which are TextQA approaches,and (Jain,  ${2016}{)}^{3}$  ,which is a KBQA method. (Jain, 2016) reports another metric; however, their predictions are available from which we calculated the EM score. Retrieval-free numbers refer to closed-book results from Roberts et al.  ${\left( {2020}\right) }^{4}$  with the same T5 model.
Source(s)	NQ	WebQ	Trivia	TREC
KB-only (1 KB)	27.9	55.6	35.4	32.4
KB-only (2 KBs)	30.9	56.7	41.5	36.0
All (1 KB)	54.0	57.8	64.1	55.3
All (2 KBs)	54.9	57.7	65.5	54.0
Table 3: Results for combining Freebase and Wikidata.
adding structured information sources significantly improves the performance over text-only models on all datasets, obtaining state-of-the-art results for NQ, WebQ and TREC. This indicates that KBs and tables contain valuable knowledge which is either absent in the unstructured texts or harder to extract from them (see also §6).
In the multi-dataset setting, we also observe clear improvements from combining sources, with the Text + tables + KB model outperforming the Text-only baseline by 5.4 points on average. The performance is generally lower than the per-dataset models, especially for the small datasets (WebQ and TREC), which may be due to the fact that each of these datasets was collected on a single information source and the multi-dataset model is less likely to exploit this prior knowledge.
Multiple KBs We also experiment with combining both Wikidata and Freebase. We see substantial improvements on all datasets in the KB-only setting over using a single KB, as well as significant
gains over our best numbers for NQ and TriviaQA in the Text+tables+KB setting (Table 3).
Having demonstrated that combining information sources does improve answer accuracy, we now provide more analysis on how this is achieved by inspecting both retriever and reader closely.
Retriever One natural assumption is that adding more data increases the coverage of relevant contexts that can be used to answer the input questions, thereby improving the end-to-end performance. We verify this by examining the retrieval results of different models using the NQ development set, where a context is considered relevant if it contains the correct answer string. When more knowledge sources are added, our system is able to improve retrieval recall (Table 4, top half), which may correlate with the end-to-end answer accuracy shown in Table 2.
Reader Although including additional information sources improves the chance of retrieving relevant contexts, it is not guaranteed that the reader can leverage those contexts and output the correct answers. For instance, reader model training may benefit from diverse sources of contexts, and the end-to-end improvement of answer accuracy may simply be attributed to a reader model that per
Model	R@20	R@100
Text-only	80.0	85.9
w/ lists	82.7	89.6
w/ tables	83.1	91.0
w/ lists + tables	85.0	92.2
w/ lists + tables + KB	83.4	92.8
Tables-only
simple linearization	86.3	94.3
template linearization	60.8	69.4
Table 4: Retrieval recall on the NQ dev set with different settings. Tables only results are for the NQ dev subset which has answers in tables.
forms better on contexts from regular text. Due to the nature of the FiD generative reader, however, it is non-trivial to ascertain which input context(s) contribute the answer. As a proxy, we look at the correlation between the source of positive contexts (those which contain a correct answer string) feeding into the reader model and the performance change in the outcome.
Suppose we are comparing two reader models  $M_u$  and  $M_t$ , where  $M_u$  uses additional sources of information compared to  $M_t$  (e.g.,  $M_t$  uses text only and  $M_u$  uses text and KB). Let  $Q$  be all the questions in our development set,  $Q_u \subseteq Q$  and  $Q_t \subseteq Q$  the subsets of questions answered correctly by  $M_u$  and  $M_t$ , respectively. The improvement set  $Q' = Q_u - Q_t$  is thus the questions that  $M_u$  manages to improve upon  $M_t$ . Examining the source of the positive contexts for the questions in  $Q'$  can help shed some light on how  $M_u$  performs better. For example, if more positive contexts are from KB rather than text, then the improvement is more likely due to additional information present at inference time. Figure 3 plots the percentages of positive contexts originating from the additional sources for the questions in the full development set ( $Q$ ) vs those in the improvement set ( $Q'$ ) in two cases. The first one compares a baseline text-only model to a model with lists and tables added on NQ, and the second compares a text+tables model with text+tables+KB on WebQ. In both cases, answers retrieved from the additional source correlate with a better outcome.
To examine the effects of other indirect factors, such as the change of overall model quality due to the inclusion of varied sources or more training samples from the tables, we evaluate the text
Figure 3: Percentage of questions with answers in additional sources. For NQ the additional sources are list and tables. For WebQ the additional source is KB.
+ tables model with text-only input. We find that this achieves similar performance (48.7 EM) on the NQ test set compared to a text-only model on the same input, suggesting that these other factors are not a major contributor and that the improved performance is primarily due to the added knowledge from structured sources.
7 Implementation Details
The code, data, and trained model checkpoints of UniK-QA are available at: https://github.com/facebookresearch/UniK-QA.
7.1 DPR Training
Our DPR model is trained on the entire Wikipedia dump, including lists and tables, as described in §3.3. Specifically, lists are treated as normal texts and included in standard text passages, while tables are converted to their own "passages" using our linearization approach. We combine all these passages from the text, lists and tables into the Wikipedia passage collection, and train DPR using the standard setup (Karpukhin et al., 2020): We use BERT-base (Devlin et al., 2019) encoders, 100-token text passages, and a single negative document per question. Negatives are mined with BM25 in the first iteration, and from the first iteration model for the second iteration. We train for 40 epochs with a linear warmup of 500 steps, batch size of 128 and learning rate  $10^{-5}$ .
As mentioned in §3.2, we do not retrain DPR for retrieving KB relations. The public DPR checkpoint for open-domain question answering is used in our WebQSP experiment (§4), while we use our own DPR model trained on text, lists and tables for retrieving KB relations in our multi-source QA experiments (§5).
KB Quota	NQ	WebQ	TREC	Trivia
Wikipedia + Wikidata	10	30	10	10
Wikipedia + Freebase	10	40	10	20
Wikipedia + Wikidata & Freebase	10	30	10	20
Table 5: The quota of "passages" converted from KB relations in each experiment.
7.2 FiD Training
We adopt the FiD model with T5-large (Raffel et al., 2020) and 100 context documents and use the original hyper-parameters of FiD (Izacard and Grave, 2021) whenever possible. In particular, the Adam (Kingma and Ba, 2015) optimizer is used with a constant learning rate of 0.0001. The model is trained for  $10k$  steps, with a batch size of 64, using 64 V100 GPUs. We did not perform any hyper-parameter search.
7.3 Merging KB and Text
As mentioned in §5.2, we tune the quota for KB relations by maximizing retrieval recall on the development set. Table 5 shows the number of KB "passages" (out of 100 total context passages) selected in our final model. The text and KB passages are interleaved in the final context passages.
For each dataset, the KB quota (which can also be interpreted as the helpfulness of the KB) is relatively stable across different choices of KBs. WebQuestions has the highest KB quota, which is expected given that it was originally collected as a KBQA dataset. Experimental results in Table 2 also confirm that using KB brings the most gains on WebQuestions.
We demonstrated a powerful new approach, UniK-QA, for unifying structured and unstructured information sources for open-domain question answering. We adopt the simple and general retriever-reader framework and show not only that it works for structured sources, but improves over traditional KBQA approaches by a wide margin. By combining sources in this way, we achieved new state-of-the-art results for two popular open-domain QA benchmarks.
However, our model also has several shortcomings in its current form. As a result of flattening all sources into text, we lose some desirable features of structured knowledge bases: the ability to return all answers corresponding to a query, and the
ability to infer multi-hop paths to answer more complex questions. In this work we have side-stepped the first issue by focusing on the exact match metric (equivalent to Hits@1), which is standard in the open-domain QA literature, but largely ignores multiple answers. We were also able to ignore the second issue, since the datasets we evaluated on, while standard, are composed mostly of simple, natural user questions which can be answered from a single piece of information.
We do believe these are important details and they can be addressed within the framework described here. For instance, outgoing edges of an entity with the same relation can easily be merged, thus encoding all answer entities into a single text representation. It is also possible to simply generate multiple answer candidates from the reader's decoder. For multi-hop question answering, there is recent work (Xiong et al., 2021b) successfully extending dense retrieval to the multi-hop setting (Yang et al., 2018; Welbl et al., 2018), which could naturally be applied within our framework. It remains to be seen how these approaches would compare to more traditional structured methods.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554-3565, Online. Association for Computational Linguistics.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The Semantic Web, pages 722-735, Berlin, Heidelberg. Springer Berlin Heidelberg.
Petr Baudis and Jan Šedivý. 2015. Modeling of the question answering task in the yodaqa system. In Proceedings of the 6th International Conference on Experimental IR Meets Multilinguality, Multimodality, and Interaction - Volume 9283, CLEF'15, page 222-228, Berlin, Heidelberg. Springer-Verlag.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In International Conference on Learning Representations.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020a. Tabfact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026-1036, Online. Association for Computational Linguistics.
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowledge bases and text using universal schema and memory networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 358-365, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA - August 24 - 27, 2014, pages 1156-1165. ACM.
David A. Ferrucci. 2012. Introduction to "this is watson". IBM Journal of Research and Development, 56(3.4):1:1-1:15.
Bin Fu, Yunqi Qiu, Chengguang Tang, Yang Li, Haiyang Yu, and Jian Sun. 2020. A survey on complex question answering over knowledge base: Recent advances and challenges. arXiv preprint, arXiv:2007.13069.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Srinivasan Iyer, Sewon Min, Yashar Mehdad, and Wen-tau Yih. 2021. RECONSIDER: Improved reranking using span-focused cross-attention for open domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1280–1287, Online. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computational Linguistics.
Sarthak Jain. 2016. Question answering over knowledge base using factual memory networks. In Proceedings of the NAACL Student Research Workshop, pages 109-115, San Diego, California. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020a. Efficient one-pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433-6441, Online. Association for Computational Linguistics.
Xintong Li, Aleksandre Maskharashvili, Symon Jory Stevens-Guille, and Michael White. 2020b. Leveraging large pretrained models for WebNLG 2020. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 117-124, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Vanessa Lopez, Victoria Uren, Marta Sabou, and Enrico Motta. 2011. Is question answering fit for the semantic web? a survey. Semantic Web, 2(2):125-155.
Xiaolu Lu, Soumajit Pramanik, Rishiraj Saha Roy, Abdalghani Abujabal, Yafang Wang, and Gerhard Weikum. 2019. Answering complex questions by joining multi-document evidence with quasi knowledge graphs. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 105-114. ACM.
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. 2021. Multi-task retrieval for
knowledge-intensive tasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1098-1111, Online. Association for Computational Linguistics.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.
Peng Qi, Haejun Lee, Tg Sido, and Christopher Manning. 2021. Answering open-domain questions of varying reasoning steps from text. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3599-3614, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, Atlanta, Georgia. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. 2020. Faithful embeddings for knowledge base queries. In Advances in Neural Information Processing Systems, volume 33, pages 22505-22516. Curran Associates, Inc.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical