Open Question Answering over Tables and Text

1 INTRODUCTION . Open question answering considers the problem of retrieving documents from a fixed corpus with a retriever , and then analyzes retrieved evidence to provide answers to a given question with a reader . Prior open question answering systems focused only on retrieving and reading free-form passages or documents . However , a significant amount of real-world information is stored in other forms , such as semi-structured web tables due to its compact representation to aggregate related information . For example , tables are often used to hold large quantities of related facts , especially numeric facts , such as ‘ Career Statistics for Lebron James ’ . This type of detailed information is found much less frequently in unstructured text . Tables are also commonly used for collections of homogeneous entities or recurring events , like ‘ List of Periodic Comets ’ or ‘ List of Champions League Winners since 1966 ’ . Hence tabular information serves as an excellent complement to textual data , especially in the open setting . Despite these advantages , no previous studies have exploited the millions of web tables to augment their open QA system . In this paper , we describe the first study to jointly exploit tables and text for open-domain question answering . For this purpose , we construct a new dataset , Open Table-and-Text Question Answering ( OTT-QA ) . OTT-QA is built on the HybridQA dataset ( Chen et al. , 2020 ) , and like HybridQA , OTTQA questions are multi-hop questions which require aggregating information from both tables and text to answer . However , unlike HybridQA , OTT-QA requires the system to retrieve relevant tables and text — in contrast , in HybridQA , the ground truth tables and textual passages required for each question are given . To produce OTT-QA ’ s questions , we begin by re-annotating the questions from HybridQA to ‘ decontextualize ’ them—i.e. , we make questions suitable for the open-domain setting ∗Part of this work was done during an internship at Google . 1Data was released in https : //github.com/wenhuchen/OTT-QA by UCSB NLP Group so that unique answers can be determined from the question alone , without needing context from the provided text and tables . We then add new questions to remove potential biases . After these steps , OTT-QA contains 45K human-annotated questions that require retrieving and aggregating information over tables and text from the whole Wikipedia . Examples from OTT-QA are depicted in Figure 1 . Note the table and passages contain non-overlapping information , and both of them must be understood to answer the question . For example , the question has a low lexical overlap with the passage about the ‘ Lakers ’ , and it needs the table as the bridge to retrieve this passage . Such cross-modality multi-hop retrieval features OTT-QA . More examples are displayed in Appendix . OTT-QA is distinguished from the existing QA datasets in two aspects . Existing table-based QA datasets ( Pasupat & Liang , 2015 ; Yu et al. , 2018 ; Chen et al. , 2020 ) operates in the closed setting without requiring any retrieval , whereas most existing open QA datasets ( Joshi et al. , 2017 ; Yang et al. , 2018 ) require only text retrieval , not table retrieval . One dataset , Natural Questions ( NQ ) ( Kwiatkowski et al. , 2019 ) includes some tabular information in its corpus , but the tables are nearly always of a restricted type ( infobox tables with only a single row ) . In contrast , OTT-QA models require retrieving both tabular data and text , and unlike the NQ dataset , requires information fusion from text and tables in non-trivial ways . OTT-QA poses novel and realistic challenges to both the retriever and reader in open QA though the questions are less natural than the real queries from NQ ( Kwiatkowski et al. , 2019 ) . Retrievers for OTT-QA need to consider two information formats , making the search space larger . Even worse , as questions in OTT-QA often require multi-hop inference , one round of retrieval is often not enough . Readers for OTT-QA also need to aggregate a significant amount of knowledge-intensive information , compared to other reader models : a single table in OTT-QA has an average length of over 300 words . Moreover , readers are often expected to process multiple retrieved units due to the uncertainty in retrieval , which makes it difficult to design strong reader models ( Devlin et al. , 2019 ; Liu et al. , 2019 ) with a length limit of 512 tokens . The baseline system that we propose to address these challenges uses an iterative retriever ( Sun et al. , 2019 ; Qi et al. , 2019 ; Min et al. , 2019 ; Ding et al. , 2019 ; Asai et al. , 2019 ) and a BERT reader ( Devlin et al. , 2019 ) . The iterative retriever explores multiple evidence documents iteratively , interacting with the candidate pool to gradually reformulate the query . Beam search is used to find multiple subsets of documents that may contain all the required evidence , and each subset is then fed to the BERT reader to predict the answer span . The highest-scored prediction is chosen as the answer . The iterative retriever needs to re-encode the query with a big transformer and re-search over the candidate pool , such a procedure ( especially dense ) can be computationally expensive . Furthermore , the BERT reader fails to capture a global overview of the retrieved documents , which leads to bad local optimum in the model prediction . We propose a more sophisticated system that addresses these challenges with two novel strategies : namely fusion retrieval and cross-block reading . The fusion retriever first pre-aligns the table segments to their highly related passages , using entity linking . Then , the aligned table segments and passages are grouped as a fused block , which contains aggregated information from two modalities ; hence , compared to the previous documents , it contains richer context to benefit the following retrieval . We view the fused block as the basic unit to be retrieved , and instead of performing multiple runs of retrieval iteratively , the fusion retriever is used once to retrieve the top K fused blocks ; however , due to errors in fusion and retrieval , the retrieved top-1 fused block might not contain the necessary information . We thus also propose a cross-block reader based on a sparse-attention based transformer architecture ( Ainslie et al. , 2020 ; Zaheer et al. , 2020 ) , which can process extremely long sequences efficiently . We use the cross-block reader to read all the top-K retrieved fused blocks jointly . Both strategies have proven effective compared to the baseline system : the best model combining the two strategies improves the accuracy of the baseline system by a huge margin . 2 BACKGROUND . The aim of an open QA system is to extract an answer to a question q from a given large corpus . Most open QA models are retriever-reader models , which extract answers in two steps : retrieval and reading . In the retrieval step , a retrieval model f is used to retrieve a set of passages from the text corpus . In the reading step , the reader is then used to extract the answer from them . Retrieval Function There are two commonly-used types of retrieval function f : sparse retrievers and dense retrievers . Our sparse retriever uses a unigram-based BM-25 score to retrieve an evidence unit b from the candidate pool B . Our dense retrieval function is a dual-encoder model ( Bromley et al. , 1994 ) , and we follow ( Lee et al. , 2019 ; Guu et al. , 2020 ) for the dual encoder design . The query and the passage are encoded with separate Transformers . As in ( Devlin et al. , 2019 ) , the vector corresponding to the first token , [ CLS ] , is used as a “ pooled ” representation of the sequence . The dense retrieval function is the dot product between hq = BERTQ ( q ) [ CLS ] and hb = BERTB ( b ) [ CLS ] for each evidence block b in the candidate corpus—i.e. , the scoring function is f ( q , b ) = hTq hb , which can viewed as finding the nearest neighbor in vector space . In the multi-hop open QA setting ( Yang et al. , 2018 ) , an iterative retrieval function ( Sun et al. , 2019 ; Min et al. , 2019 ; Ding et al. , 2019 ) is proposed , which defines the retrieval process as an auto-regressive formula . Our iterative retriever function is denoted as f ( [ q , b1 , · · · , bj−1 ] , bj ) , which appends the previous j− 1 rounds of retrieval to the original q in in the j-th round of retrieval . Beam search is used in test time . Single-Block Reader Due to the uncertainty in retrieval , the top-1 document might not always contain the answer . Existing models normally retrieve the top-k documents and feed them to the reader for span selection . The standard reader ( Chen et al. , 2017 ; Joshi et al. , 2017 ) aims to extract a span from each of the retrieved blocks bi and assign a confidence f ( q , bi ) fread ( a|q , bi ) to it , with f ( q , bi ) indicating the retrieval probability and fread ( a|q , bi ) denoting the span selection probability by reader . Multiple answers { a1 , · · · , ak } are ranked with this confidence score and the highest scored answer span â is the final answer . Note that the reader needs to run k times , once for each of the top-k retrievals . We refer to this model as the single-block reader and use it as our baseline . HybridQA HybridQA ( Chen et al. , 2020 ) , a closed-domain QA dataset , is the most related to ours . During the annotation of HybridQA , a table T and its relevant passages { P1 , · · · , PN } ( surrounding text and hyperlinked passage ) are given to a crowd worker to write questions which necessarily require both the passage and table information to answer . The original dataset contains 72K multi-hop questions paired with 13K tables with their paired passages . During training/testing time , the ground-truth tables and passages are given to a model , HYBRIDER , to find the final answer . HYBRIDER also serves as an important baseline in our paper . 3 TASK AND DATASET . In OTT-QA , the retrieval corpus consists of a set of table candidates BT and a set of passage candidates BP . The task is to answer question q by extracting answer strings from blocks b ∈ BT ∪ BP , where b can be either textual and tabular data . We adopt the standard exact match ( EM ) and F1 scores ( Yang et al. , 2018 ) for evaluation . Different from HybridQA , OTT-QA ’ s table candidates are web tables without hyperlinks provided . This decision was made to make the problem setting more general , as otherwise systems that solve OTT-QA could only be applied to high-quality data in Wikipedia . However , in OTT-QA , we provide hyperlinks in the training subset , but not dev/test set . Removing hyperlinks in tables makes the overall task much more challenging , but makes the final systems applicable to more general domains . Thus , an OTT-QA model needs to jointly retrieve both tables and text , without abusing gold hyperlinks , and then aggregate them to find the answer . Candidate Pool For our table collection BT , we extracted all Wikipedia regular tables with their metadata including page title , page section title , and section text . The metadata , denoted TM , is essential for de-contextualization . We obtain a table corpus containing over 400k high-quality tables with an average length of 320 words including metadata . For the text passage collection BP , we crawl English Wikipedia dump pages and filter out noisy pages . We follow HybridQA ( Chen et al. , 2020 ) and only keep a maximum of 12 sentences in the introduction section as the passage . We obtain a corpus containing over 5 million passages , with an average of 94 words . Notation We define each table as a matrix T , which consists of cells Ti , j with i specifying the row , and j specifying the column . Each cell Ti , j could be a number , date , phrase or even sentence due to its semi-structured nature . However , a single complete table with structured representation ( Herzig et al. , 2020 ) can easily exceed the 512-token limit , which poses great challenges to the downstream reader to process top-K retrieval . Hence we propose to decompose each table T into multiple rows Ri , which are combined with the headers , metadata , and global max/min information from the original table as a table segment . The table segment is used as the basic retrieval block in our paper . This decomposition procedure increases candidate BT from 400k to 5 million , making the retrieval problems even more fine-grained and more challenging . Our table segment representation is described in Appendix subsection B.1 . In summary , we build a candidate pool of 5 million table segments BT and a pool of 5 million passages BP . We denote as B as our full candidate pool , which our model needs to find the block b ( a table segment or a passage ) containing the answer span .

Summary: The paper provides a interesting direction in open question answering. In particular, it proposes an open QA problem over both tabular and textual data, and present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task. Two techniques are introduced to address the challenge of retrieving and aggregating evidence for OTT-QA. Results show that the newly introduced techs bring improvements.