SlowGuess/NAACL_2022_OCR huggingface dataset:
Barlas Oğuz<sup>1*</sup>, Xilun Chen<sup>1*</sup>, Vladimir Karpukhin<sup>1</sup>, Stan Peshterliev<sup>1</sup>, Dmytro Okhonko<sup>1</sup>, Michael Schlichtkrull<sup>2,3†</sup>, Sonal Gupta<sup>1</sup>, Yashar Mehdad<sup>1</sup>, Wen-tau Yih<sup>1</sup>
$^{1}$ Meta AI,  $^{2}$ University of Amsterdam,  $^{3}$ University of Edinburgh,
{barlaso,xilun,stanvp,sonalgupta,mehdad,scottyih}@fb.com vlad.karpuhin@gmail.com, d.okhonko@gmail.com, m.s.schlichtkrull@uva.nl
We study open-domain question answering with structured, unstructured and semistructured knowledge sources, including text, tables, lists and knowledge bases. Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only. Our approach greatly improves the results on knowledge-base QA tasks by 11 points, compared to latest graph-based methods. More importantly, we demonstrate that our unified knowledge (UniK-QA $^1$ ) model is a simple and yet effective way to combine heterogeneous sources of knowledge, advancing the state-of-the-art results on two popular question answering benchmarks, NaturalQuestions and WebQuestions, by 3.5 and 2.6 points, respectively.
Answering factual questions has long been an inspirational challenge to information retrieval and artificial intelligence researchers (Voorhees and Tice, 2000; Lopez et al., 2011). In its most general form, users can ask about any topic and the answer may be found in any information source. Defined as such, the challenge of open domain question answering is extremely broad and complex. Though there have been successful undertakings which embrace this complexity (notably Ferrucci, 2012), most recent works make simplifying assumptions as to the source of answers, which fall largely in two categories: structured data and unstructured text.
A long line of research aims to answer user questions using a structured knowledge base (KB) (Berant et al., 2013; Yih et al., 2015), known as KBQA.
Q: Who was the drummer for the Beatles?
Text passages
Linearized KB triplets
Fusion-in-decoder Reader
Linearized tables
A: Ringo Starr
Dense Retriever
Figure 1: Illustration of UniK-QA's workflow for unified-knowledge question answering: Heterogeneous information sources are linearized into text. A dense retriever retrieves passages from a mix of sources, which are jointly processed by the reader to produce the answer.
Typically, a KB can be viewed as a knowledge graph consisting of entities, properties, and a predefined set of relations between them. A question can be answered, provided that it can be expressed within the language of relations and objects present in the knowledge graph. With a high-quality, carefully curated KB, answers can be extracted with fairly high precision. KBQA, however, struggles with low answer coverage due to the cost of curating an extensive KB, as well as the fact that many questions simply cannot be answered using a KB if the answers are not entities.
A second line of work targets a large collection of unstructured text (such as Wikipedia) (Chen et al., 2017) as the source of answers. Thanks to the latest advances in machine reading comprehension and text retrieval, substantial progress has been made for open-domain question answering from text (TextQA) in just the past couple years (Yang
$\dagger$  Work done while interning with Meta AI.
1The code of UniK-QA is available at: https://
github.com/facebookresearch/UniK-QA.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1535 - 1546
et al., 2019; Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Izacard and Grave, 2021). On the other hand, semi-structured tables and structured KBs can be valuable knowledge sources, yet TextQA methods are restricted in taking only unstructured text as input, missing the opportunity of using these complementary sources of information to answer more questions.
When it comes to answering questions using both structured and unstructured information, a straightforward solution is combining specialized TextQA and KBQA systems. The input question is sent to multiple sub-systems, and one of them is selected to output the final answer. While this approach may take advantage of the state-of-the-art models designed for different information sources, the whole end-to-end system becomes fairly complex. It is also difficult to handle questions that require reasoning with information from multiple sources.
Having a more integrated system design that covers heterogeneous information sources has proven to be difficult. One main reason is that techniques used for KBQA and TextQA are drastically different. The former exploits the graph structure and/or semantic parsing to convert the question into a structured query, while TextQA has mostly settled on the retriever-reader architecture powered by pre-trained transformers. Recent work on multisource QA has tried to incorporate free text into graph nodes (Sun et al., 2018; Lu et al., 2019) to make texts amenable to KBQA methods, but the performance remains unconvincing.
In this work, we propose a novel unified knowledge representation (UniK-QA) approach for open-domain question answering with heterogeneous information sources. Instead of having multiple specialized sub-systems or incorporating text into knowledge graphs, we flatten the structured data and apply TextQA methods. Our main motivation for doing so is to make the powerful machinery of pre-trained transformers available for structured QA. In addition, this approach opens the door to a simple and unified architecture. We can easily support semi-structured sources such as lists and tables, as well as fully structured knowledge bases. Moreover, there is no need to specially handle the schema or ontology that defines the structure of the KB, making it straightforward to support multiple KBs. Our UniK-QA model incorporates some 27 million passages composed of text and lists,
455,907 Wikipedia tables, and 3 billion relations from two knowledge bases (Freebase and Wikidata) in a single, unified open-domain QA model.
We first validate our approach by modeling KBQA as a pure TextQA task. We represent all relations in the KB with their textual surface form, and train a retriever-reader model on them as if they were text documents. This simple approach works incredibly well, improving the exact match score on the WebQSP dataset by  $11\%$  over previous state of the art. This result further justifies our choice of unifying multi-source QA under the TextQA framework as it can improve KBQA performance per se.
For our multi-source QA experiments, we consider lists, tables, and knowledge bases as sources of structured information, and convert each of them to text using simple heuristics. We model various combinations of structured sources with text, and evaluate on four popular open-domain QA datasets, ranging from entity-heavy KBQA benchmarks to those targeting free-form text sources. Our results indicate that our multi-source UniK-QA approach, unlike existing efforts on combining KBQA and TextQA, consistently improves over strong TextQA baselines in all cases. We obtain new state-of-the-art results for two datasets, advancing the published art on NaturalQuestions by 3.5 points and on WebQuestions by 2.6 points.
In addition, we consider the realistic setting in which the source of questions is not known a priori, as would be the case for a practical system. We train a single multi-dataset model on a combined dataset from several benchmarks, and show that it outperforms all single-source baselines across this diverse set of questions.
2.1 Knowledge-base question answering (KBQA)
A knowledge base (KB) considered in this work is a collection of facts, represented as a set of subject-predicate-object triples. Each triple  $(e_1, p, e_2)$  denotes a binary relationship between the subject entity  $e_1$  and the object  $e_2$  (e.g., places, persons, dates or numbers), as well as their relation type, or predicate  $p$  (e.g., capital_of, married_to, etc.).
Modern large-scale KBs, such as Freebase (Bollacker et al., 2008), DBPedia (Auer et al., 2007) and Wikidata (Vrandecic and Krötzsch, 2014) can contain billions of triples that describe relations
between millions of entities, making them great sources of answers to open-domain questions. The prevailing approach for knowledge-base question answering (KBQA) is semantic parsing (Berant et al., 2013; Yih et al., 2015), where a natural language question is converted into a logical form that can be used to query the knowledge base. Such methods are tailored to the specific graph structure of the KB and are usually not directly applicable to other knowledge sources.
2.2 Open-domain question answering from text (TextQA)
KBQA is ultimately limited in its coverage of facts and the types of questions it can answer. On the other hand, large collections of text such as Wikipedia or CommonCrawl promise to be a richer source of knowledge for truly open domain question answering systems. This line of work (which we will refer to as TextQA) has been popularized by the TREC QA tracks (Voorhees and Tice, 2000), and has seen explosive growth with the advent of neural machine reading (MRC) (Rajpurkar et al., 2016) models. In the neural era, Chen et al. (2017) were the first to combine MRC with retrieval for end-to-end QA. Subsequent work cemented this retriever-reader paradigm, with improved reader models (Yang et al., 2019; Izacard and Grave, 2021) and neural retrievers (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020). Despite impressive advances, TextQA systems still underperform KBQA, especially on benchmarks originally created for KBs such as WebQuestions. Furthermore, they also fall short of universal coverage, due to the exclusion of other (semi-)structured information sources such as tables.
2.3 Question answering from tables
Large amounts of authoritative data such as national statistics are often available in the form of tables. While KBQA and TextQA have enjoyed increasing popularity, tables as a source of information has surprisingly escaped the attention of the community save for a few recent works.
Working with web tables can be challenging, due to the lack of formal schema, inconsistent formatting and ambiguous cell values (e.g., entity names). In contrast to relational databases and KBs, tables can at best be described as semi-structured information. Sun et al. (2016) considered open domain QA from web tables, however made no use of unstructured text. Some recent work investigated MRC
with tables without a retrieval component (Pasupat and Liang, 2015; Yin et al., 2020; Chen et al., 2020a). In addition, Chen et al. (2021, 2020b) investigated open domain QA using tables and text. While they are in a similar direction, these works focus on complex, crowd-sourced questions requiring more specialized methods, while we target the case of simple, natural questions and investigate if popular TextQA and KBQA benchmarks can be further improved with the addition of tables.
2.4 Fusion of text and knowledge-base
As discussed, KBQA and TextQA are intuitively complementary, and several attempts have been made to merge them to get the benefits of both. An early example is (Ferrucci, 2012), which combines multiple expert systems and re-ranks them to produce the answer. More recent work attempts to enrich the KB by extracting structure from text. One way to accomplish this is using OpenIE triplets (Fader et al., 2014; Xu et al., 2016), thus staying completely within the semantic parsing paradigm. Somewhat closer to our approach are UniversalSchemas (Riedel et al., 2013; Das et al., 2017), which embed KB relations and textual relations in a common space. Yet, UniversalSchemas are also constrained to an entity-relation structure. The latest in this line are the works of (Sun et al., 2018, 2019), which augments the knowledge graph with text nodes and applies graph methods to identify candidate answers.
By retaining structure, previous work was able to take advantage of KBQA methods, but also failed to capture the full richness of TextQA. We depart radically in our approach, by foregoing all structure, and directly applying TextQA methods based on the more general retriever-reader architecture. We also evaluate on a more diverse benchmark set composed of natural open domain datasets, as well as those originally meant for KBQA, and demonstrate strong improvements in this truly open-domain setting. Concurrent work (Agarwal et al., 2021) proposed a similar idea for language model pretraining and also evaluated on open-domain QA. Our work differs in that (1) we have a more comprehensive treatment of sources (including tables, lists and multiple KBs) and ODQA datasets, (2) we compare against and improve on much stronger state-of-the-art baselines, and (3) we also evaluate in a more realistic multi-dataset setting with all datasets handled by a single model.
3.1 UniK-QA architecture
We use a retriever-reader architecture, with dense passage retriever (DPR) (Karpukhin et al., 2020) as retriever and fusion-in-decoder (FiD) (Izacard and Grave, 2021) as our reader. Structured knowledge such as tables, lists and KB relations are converted to text with simple heuristics ( $\S 3.2$ ,  $\S 3.3$ ), and we generalize DPR to retrieve from these heterogeneous documents as well as regular text passages. Each retrieved document is concatenated with the question, then independently encoded by the reader encoder. Fusion of information happens in the decoder, which computes full attention over the entire concatenated input representations. The overall architecture is illustrated in Figure 1.
Retriever The DPR retriever consists of a dense document encoder and a question encoder, trained such that positive documents have embeddings closer to the question embedding in dot product space. We follow the original DPR implementation and hyperparameters (see §7). We further include tables, lists and KB relations in the index. The details of how these are processed into documents and merged are in the subsequent sections.
One improvement we make to the training process is iterative training, where better hard negatives are mined at each step using the model at the previous step, similar to (Xiong et al., 2021a). All models including our text-only baselines benefit from this change. We find 2 iterations sufficient.
Reader The FiD reader has demonstrated strong performance in the text-only setting and effective in fusing information from a large number of documents (Izacard and Grave, 2021). We thus find it a natural candidate for fusing knowledge from various sources. We use the FiD model with T5-large (Raffel et al., 2020), 100 context documents, and the original hyper-parameters for all experiments. See §7 for more implementation details.
3.2 Unified representations for KBs
In order to apply our retriever-reader model, we first convert KB relations into text using simple heuristics. For a relation triple  $\langle \text{subj}, \text{pred}, \text{obj} \rangle$ , where  $\text{subj}$ ,  $\text{pred}$  and  $\text{obj}$  are the subject, predicate and object of the relation respectively, we serialize it by concatenating the text surface forms of  $\text{subj}$ ,  $\text{pred}$  and  $\text{obj}$ .
More complex ( $n$ -ary) relations involve multiple predicates and objects, such as Natalie Portman
Freebase Relation (with CVT entities):
Converted Text:
Natalie Portman performance film Star Wars Episode I, and performance character Padmé Amidala.
Wikidata Relation (with qualifiers):
Converted Text:
Figure 2: Converting Freebase and Wikidata relations to text.
Star Wars Episode I cast member Natalie Portman, and character role Padmé Amidala.
played the character Padmé Amidala in the movie Star Wars, and can be expressed differently depending on the KB. In particular, Freebase uses compound value types (CVTs) to convert an  $n$ -ary relation into multiple standard triples, while Wiki-data allows a predicate to have qualifiers to express additional properties (Tanon et al., 2016). In this work, we convert an  $n$ -ary relation into a single sentence by forming a comma-separated clause for each predicate (Figure 2).<sup>2</sup>
Besides our heuristic-based linearization of KB relations, there are alternatives such as template-based or model-based methods. Since KBs such as Freebase and Wikidata have hundreds of thousands of different types of relations, it is prohibitive to come up with templates for each relation type. On the other hand, model-based linearization achieves worse retrieval recall than our simple heuristics despite being much more expensive. In particular, we experiment with a top-ranked KB-to-text model (Li et al., 2020b) from the WebNLG 2020 challenge (Castro Ferreira et al., 2020), which is based on T5-large. Preliminary results on KBQA show that the WebNLG model achieves a  $87.9\%$  retrieval recall @100 on the dev set of WebQSP (Yih et al., 2016), while our simple heuristics performs better at  $94.7\%$ . We hence stick with our simple heuristics in all experiments.
Once converted to text, relations can be indexed and retrieved using DPR. We use existing TextQA DPR checkpoints for retrieving KB relations with
${}^{2}$  A side benefit of this approach is that these complex relations are now represented as a single piece of text, whereas they would normally be considered multi-hop and require more complex methods (Fu et al., 2020) if using traditional graph-based KBQA models.
out any retraining. We index individual relations to best leverage the power of DPR for retrieving the most relevant relations for a given question<sup>3</sup>. Unlike most existing KBQA works, our approach can also seamlessly incorporate multiple KBs by storing all relations into a joint index and retrieving from it (see §5.4).
Directly indexing billions of relations in the entire KB can bring additional engineering challenges. To avoid these, we implement retrieval of relations in two steps, where an entity linking system is used in the first step to narrow down the search to a high-recall 2-hop neighborhood of the retrieved entities for each question (We use STAGG (Yih et al., 2015) in the case of Freebase and ELQ (Li et al., 2020a) for Wikidata). We then use DPR to retrieve relations from this reduced set. As the relation representations are usually short sentences, we combine retrieved relations into passages of at most 100 tokens, after which they are fed to the FiD reader in the same way as text paragraphs.
3.3 Unified representations for lists & tables
Karpukhin et al. (2020) excludes lists and tables from their passage collection. For lists, we simply retain them as part of the text documents without special preprocessing, which improves retrieval recall in our experiments (see Table 4 in §6). We now discuss about our treatment of tables.
English Wikipedia contains more than 3 million tables ('classical' tables embedded in text as well as specialized tables like info-boxes), which are a huge source of factual knowledge by themselves and can substantially increase the coverage of open-domain QA systems. For instance, the answer to approximately a quarter of the questions in the NaturalQuestions (NQ) dataset can be found in Wikipedia tables (Kwiatkowski et al., 2019).
We start from a large subset of Wikipedia tables extracted and released as part of the NaturalQuestions dataset. We include all candidate documents which are part of the training set, extract nested tables into independent units, and filter out single-row tables as well as 'service' tables. This results in a corpus of 455,907 tables, which are used in our experiments.
As with KB relations, semi-structured content in tables need to be 'linearized' into text for the
Model	Hits@1
GraftNet (Sun et al., 2018)	67.8
PullNet (Sun et al., 2019)	68.1
EmQL (Sun et al., 2020)	75.5*
Our KBQA (T5-base)	76.7
Our KBQA (T5-large)	79.1
Table 1: Hits@1 on WebQSP dataset using Freebase.  $(^{*})\mathrm{EmQL}$  uses oracle entities, hence is not directly comparable with the others.
retriever-reader model to work. There are many ways to do such linearization (see Yin et al., 2020; Chen et al., 2020a). We tried two types of tables linearization: 'template'-like encoding used in recent literature (Chen et al., 2020a) and a simpler one which we find works the best in our experiments (see Table 4, bottom half). In particular, we concatenate cell values on the same row, separated by commas, to form the text representation, and multiple rows are then combined into longer documents delimited by newlines. As with TextQA, we divide linearized tables into 100-token chunks for indexing and retrieval. We take the first non-empty table row as the header and include it in every table chunk. This heuristic to select the first non-empty row as header is crucial and adds 4-6 points to top-20 passage accuracy.
4 KBQA as TextQA: A Motivating Experiment
In this section, we present a motivating experiment showing that our UniK-QA approach not only provides a natural pathway to multi-source open-domain QA, but also improves KBQA per se. In particular, we evaluate our approach on a widely-used KBQA dataset, WebQSP (Yih et al., 2016), in the single-source setting.
We use Freebase as the knowledge source, and re-use pre-computed STAGG entity linking results and 2-hop neighborhoods as provided by Sun et al. (2018) for fair comparisons. We convert KB relations in the 2-hop neighborhood into text, retrieve the most relevant ones using DPR to form 100 context passages, and feed them into the T5 FiD reader as described in Section 3.2. We use the original DPR checkpoint from Karpukhin et al. (2020) for retrieval, and train FiD using the training questions in WebQSP and the DPR-retrieved contexts with default hyperparameters (see §7). The
3Indexing at a coarser granularity (such as creating a document for each entity) also has practical challenges because certain entities (e.g., United States) may have hundreds of thousands of relations, resulting in extremely long documents.
results are shown in Table 1, where the numbers represent Hits@1, or the percentage of the model's top-predicted answer being a "hit" (exact match) against one of the gold-standard answers.
We see that our KBQA method outperforms previous state-of-the-art methods by a wide margin, improving exact match accuracy to  $79.1\%$ . Since we adopt the exact same KB setup and pre-processing procedure from previous work, this improvement can be attributed purely to our UniK-QA model. We take this result as strong evidence for our claim that powerful TextQA methods generalize well to structured data, and offer a natural new framework for unifying structured and unstructured information sources.
5 Multi-Source QA Experiments
We now present our main experiments on unified multi-source question answering.
For our main experiments, we use the same datasets that have recently become somewhat standard for evaluating open-domain QA (Lee et al., 2019):
NaturalQuestions (NQ) (Kwiatkowski et al., 2019) consists of questions mined from real Google search queries and Wikipedia articles with answer spans annotated. While the answer spans are usually on the regular, free-form text, some span annotations are in tables.
WebQuestions (WebQ) (Berant et al., 2013) targets Freebase as the source of answers, with questions coming from Google Suggest API.
TriviaQA (Trivia) (Joshi et al., 2017) contains a set of trivia questions with answers originally scraped from the Web.
CuratedTREC (TREC) (Baudiš and Šedivý, 2015) is a collection of questions from TREC QA tracks and various Web sources, intended to benchmark open-domain QA on unstructured text.
5.2 Combinations of sources
We compare 5 variations of our model, each with a different combination of information sources. We have Text-only, Tables-only and KB-only variants as single-source baselines. Next, the Text + tables model makes use of the entire Wikipedia dump, including lists and tables. Finally we add the KBs resulting in the Text + tables + KB model.
The Text + tables model uses a unified dense index, where text passages and table chunks are