Open Question Answering over Tables and Text

1 INTRODUCTION . Open question answering considers the problem of retrieving documents from a fixed corpus with a retriever , and then analyzes retrieved evidence to provide answers to a given question with a reader . Prior open question answering systems focused only on retrieving and reading free-form passages or documents . However , a significant amount of real-world information is stored in other forms , such as semi-structured web tables due to its compact representation to aggregate related information . For example , tables are often used to hold large quantities of related facts , especially numeric facts , such as ‘ Career Statistics for Lebron James ’ . This type of detailed information is found much less frequently in unstructured text . Tables are also commonly used for collections of homogeneous entities or recurring events , like ‘ List of Periodic Comets ’ or ‘ List of Champions League Winners since 1966 ’ . Hence tabular information serves as an excellent complement to textual data , especially in the open setting . Despite these advantages , no previous studies have exploited the millions of web tables to augment their open QA system . In this paper , we describe the first study to jointly exploit tables and text for open-domain question answering . For this purpose , we construct a new dataset , Open Table-and-Text Question Answering ( OTT-QA ) . OTT-QA is built on the HybridQA dataset ( Chen et al. , 2020 ) , and like HybridQA , OTTQA questions are multi-hop questions which require aggregating information from both tables and text to answer . However , unlike HybridQA , OTT-QA requires the system to retrieve relevant tables and text — in contrast , in HybridQA , the ground truth tables and textual passages required for each question are given . To produce OTT-QA ’ s questions , we begin by re-annotating the questions from HybridQA to ‘ decontextualize ’ them—i.e. , we make questions suitable for the open-domain setting ∗Part of this work was done during an internship at Google . 1Data was released in https : //github.com/wenhuchen/OTT-QA by UCSB NLP Group so that unique answers can be determined from the question alone , without needing context from the provided text and tables . We then add new questions to remove potential biases . After these steps , OTT-QA contains 45K human-annotated questions that require retrieving and aggregating information over tables and text from the whole Wikipedia . Examples from OTT-QA are depicted in Figure 1 . Note the table and passages contain non-overlapping information , and both of them must be understood to answer the question . For example , the question has a low lexical overlap with the passage about the ‘ Lakers ’ , and it needs the table as the bridge to retrieve this passage . Such cross-modality multi-hop retrieval features OTT-QA . More examples are displayed in Appendix . OTT-QA is distinguished from the existing QA datasets in two aspects . Existing table-based QA datasets ( Pasupat & Liang , 2015 ; Yu et al. , 2018 ; Chen et al. , 2020 ) operates in the closed setting without requiring any retrieval , whereas most existing open QA datasets ( Joshi et al. , 2017 ; Yang et al. , 2018 ) require only text retrieval , not table retrieval . One dataset , Natural Questions ( NQ ) ( Kwiatkowski et al. , 2019 ) includes some tabular information in its corpus , but the tables are nearly always of a restricted type ( infobox tables with only a single row ) . In contrast , OTT-QA models require retrieving both tabular data and text , and unlike the NQ dataset , requires information fusion from text and tables in non-trivial ways . OTT-QA poses novel and realistic challenges to both the retriever and reader in open QA though the questions are less natural than the real queries from NQ ( Kwiatkowski et al. , 2019 ) . Retrievers for OTT-QA need to consider two information formats , making the search space larger . Even worse , as questions in OTT-QA often require multi-hop inference , one round of retrieval is often not enough . Readers for OTT-QA also need to aggregate a significant amount of knowledge-intensive information , compared to other reader models : a single table in OTT-QA has an average length of over 300 words . Moreover , readers are often expected to process multiple retrieved units due to the uncertainty in retrieval , which makes it difficult to design strong reader models ( Devlin et al. , 2019 ; Liu et al. , 2019 ) with a length limit of 512 tokens . The baseline system that we propose to address these challenges uses an iterative retriever ( Sun et al. , 2019 ; Qi et al. , 2019 ; Min et al. , 2019 ; Ding et al. , 2019 ; Asai et al. , 2019 ) and a BERT reader ( Devlin et al. , 2019 ) . The iterative retriever explores multiple evidence documents iteratively , interacting with the candidate pool to gradually reformulate the query . Beam search is used to find multiple subsets of documents that may contain all the required evidence , and each subset is then fed to the BERT reader to predict the answer span . The highest-scored prediction is chosen as the answer . The iterative retriever needs to re-encode the query with a big transformer and re-search over the candidate pool , such a procedure ( especially dense ) can be computationally expensive . Furthermore , the BERT reader fails to capture a global overview of the retrieved documents , which leads to bad local optimum in the model prediction . We propose a more sophisticated system that addresses these challenges with two novel strategies : namely fusion retrieval and cross-block reading . The fusion retriever first pre-aligns the table segments to their highly related passages , using entity linking . Then , the aligned table segments and passages are grouped as a fused block , which contains aggregated information from two modalities ; hence , compared to the previous documents , it contains richer context to benefit the following retrieval . We view the fused block as the basic unit to be retrieved , and instead of performing multiple runs of retrieval iteratively , the fusion retriever is used once to retrieve the top K fused blocks ; however , due to errors in fusion and retrieval , the retrieved top-1 fused block might not contain the necessary information . We thus also propose a cross-block reader based on a sparse-attention based transformer architecture ( Ainslie et al. , 2020 ; Zaheer et al. , 2020 ) , which can process extremely long sequences efficiently . We use the cross-block reader to read all the top-K retrieved fused blocks jointly . Both strategies have proven effective compared to the baseline system : the best model combining the two strategies improves the accuracy of the baseline system by a huge margin . 2 BACKGROUND . The aim of an open QA system is to extract an answer to a question q from a given large corpus . Most open QA models are retriever-reader models , which extract answers in two steps : retrieval and reading . In the retrieval step , a retrieval model f is used to retrieve a set of passages from the text corpus . In the reading step , the reader is then used to extract the answer from them . Retrieval Function There are two commonly-used types of retrieval function f : sparse retrievers and dense retrievers . Our sparse retriever uses a unigram-based BM-25 score to retrieve an evidence unit b from the candidate pool B . Our dense retrieval function is a dual-encoder model ( Bromley et al. , 1994 ) , and we follow ( Lee et al. , 2019 ; Guu et al. , 2020 ) for the dual encoder design . The query and the passage are encoded with separate Transformers . As in ( Devlin et al. , 2019 ) , the vector corresponding to the first token , [ CLS ] , is used as a “ pooled ” representation of the sequence . The dense retrieval function is the dot product between hq = BERTQ ( q ) [ CLS ] and hb = BERTB ( b ) [ CLS ] for each evidence block b in the candidate corpus—i.e. , the scoring function is f ( q , b ) = hTq hb , which can viewed as finding the nearest neighbor in vector space . In the multi-hop open QA setting ( Yang et al. , 2018 ) , an iterative retrieval function ( Sun et al. , 2019 ; Min et al. , 2019 ; Ding et al. , 2019 ) is proposed , which defines the retrieval process as an auto-regressive formula . Our iterative retriever function is denoted as f ( [ q , b1 , · · · , bj−1 ] , bj ) , which appends the previous j− 1 rounds of retrieval to the original q in in the j-th round of retrieval . Beam search is used in test time . Single-Block Reader Due to the uncertainty in retrieval , the top-1 document might not always contain the answer . Existing models normally retrieve the top-k documents and feed them to the reader for span selection . The standard reader ( Chen et al. , 2017 ; Joshi et al. , 2017 ) aims to extract a span from each of the retrieved blocks bi and assign a confidence f ( q , bi ) fread ( a|q , bi ) to it , with f ( q , bi ) indicating the retrieval probability and fread ( a|q , bi ) denoting the span selection probability by reader . Multiple answers { a1 , · · · , ak } are ranked with this confidence score and the highest scored answer span â is the final answer . Note that the reader needs to run k times , once for each of the top-k retrievals . We refer to this model as the single-block reader and use it as our baseline . HybridQA HybridQA ( Chen et al. , 2020 ) , a closed-domain QA dataset , is the most related to ours . During the annotation of HybridQA , a table T and its relevant passages { P1 , · · · , PN } ( surrounding text and hyperlinked passage ) are given to a crowd worker to write questions which necessarily require both the passage and table information to answer . The original dataset contains 72K multi-hop questions paired with 13K tables with their paired passages . During training/testing time , the ground-truth tables and passages are given to a model , HYBRIDER , to find the final answer . HYBRIDER also serves as an important baseline in our paper . 3 TASK AND DATASET . In OTT-QA , the retrieval corpus consists of a set of table candidates BT and a set of passage candidates BP . The task is to answer question q by extracting answer strings from blocks b ∈ BT ∪ BP , where b can be either textual and tabular data . We adopt the standard exact match ( EM ) and F1 scores ( Yang et al. , 2018 ) for evaluation . Different from HybridQA , OTT-QA ’ s table candidates are web tables without hyperlinks provided . This decision was made to make the problem setting more general , as otherwise systems that solve OTT-QA could only be applied to high-quality data in Wikipedia . However , in OTT-QA , we provide hyperlinks in the training subset , but not dev/test set . Removing hyperlinks in tables makes the overall task much more challenging , but makes the final systems applicable to more general domains . Thus , an OTT-QA model needs to jointly retrieve both tables and text , without abusing gold hyperlinks , and then aggregate them to find the answer . Candidate Pool For our table collection BT , we extracted all Wikipedia regular tables with their metadata including page title , page section title , and section text . The metadata , denoted TM , is essential for de-contextualization . We obtain a table corpus containing over 400k high-quality tables with an average length of 320 words including metadata . For the text passage collection BP , we crawl English Wikipedia dump pages and filter out noisy pages . We follow HybridQA ( Chen et al. , 2020 ) and only keep a maximum of 12 sentences in the introduction section as the passage . We obtain a corpus containing over 5 million passages , with an average of 94 words . Notation We define each table as a matrix T , which consists of cells Ti , j with i specifying the row , and j specifying the column . Each cell Ti , j could be a number , date , phrase or even sentence due to its semi-structured nature . However , a single complete table with structured representation ( Herzig et al. , 2020 ) can easily exceed the 512-token limit , which poses great challenges to the downstream reader to process top-K retrieval . Hence we propose to decompose each table T into multiple rows Ri , which are combined with the headers , metadata , and global max/min information from the original table as a table segment . The table segment is used as the basic retrieval block in our paper . This decomposition procedure increases candidate BT from 400k to 5 million , making the retrieval problems even more fine-grained and more challenging . Our table segment representation is described in Appendix subsection B.1 . In summary , we build a candidate pool of 5 million table segments BT and a pool of 5 million passages BP . We denote as B as our full candidate pool , which our model needs to find the block b ( a table segment or a passage ) containing the answer span .

Summary: In this paper, the authors introduce the task of open domain QA using table and text. Unlike recent tasks on table and text retrieval, where the table associated with the text is known, in this task both tables and text need to be retrieved from a large collection. Similar to the recent HybridQA task, questions may require fusing information both tables and text to answer questions. The dataset has been constructed by reusing the Hybrid QA dataset as well as crawling additional tables and passages from Wikipedia. Since the questions in the dataset may be non-contextual (for instance, not explicitly mentioning the name of a country when the data is only for that country), the authors used crowd-sourced workers to insert words or phrases from the section title, caption etc and make the question self-contained. However, this can cause the questions to become unnatural and excessively long  - the authors manually select the 25% worst questions and get them refined by a second set of annotation tasks. Additionally, to include new samples beyond Hybrid QA, they collect 2200 new QA pairs using the additional tables crawled and include them as part of the dev and test set. The exact answer in the table is annotated using distant supervision and has an extraction accuracy of 85%.  The overall dataset contains 41.5K questions in train and approximately 2K questions each in the dev and test set.