Open Question Answering over Tables and Text
Authors: Wenhu Chen
Ming-Wei Chang
Eva Schlinger
William Yang Wang
William W. Cohen
Venue: ICLR 2021 Poster (2021)
Keywords: Question Answering
Tabular Data
Open-domain
Retrieval

Abstract: In open question answering (QA), the answer to a question is produced by retrieving and then analyzing documents that might contain answers to the question.  Most open QA systems have considered only retrieving information from unstructured text.  Here we consider for the first time open QA over {\em both} tabular and textual data and present a new large-scale dataset \emph{Open Table-and-Text Question Answering} (OTT-QA) to evaluate performance on this task. Most questions in OTT-QA require multi-hop inference across tabular data and unstructured text, and the evidence required to answer a question can be distributed in different ways over these two types of input, making evidence retrieval challenging---our baseline model using an iterative retriever and BERT-based reader achieves an exact match score less than 10\%. We then propose two novel techniques to address the challenge of retrieving and aggregating evidence for OTT-QA. The first technique is to use ``early fusion'' to group multiple highly relevant tabular and textual units into a fused block, which provides more context for the retriever to search for.  The second technique is to use a cross-block reader to model the cross-dependency between multiple retrieved evidence with global-local sparse attention. Combining these two techniques improves the score significantly, to above 27\%.

Published as a conference paper at ICLR 2021
OPEN QUESTION ANSWERING OVER TABLES AND TEXT
Wenhu Chen't Ming-Weii Chang", Eva Schlinger", William Wang', William W. Cohen? Universityy of California, Santa Barbara
Google Research
(wenhuchen, wi william} ucsb. edu
(mingweichang, eschling, wcohen]8google.. com
ABSTRACT
In open question answering (QA), the answer to a question is produced by re- trieving and then analyzing documents that might contain answers the ques- tion. Most open QA systems have considered only retrieving information from unstructured text. Here we consider for the first time open QA over both tabular and textual data and present new large-scale dataset Open Table-and-Text Ques- tion Answering (OTT-QA) to evaluate performance on this task Most questions in OTT-QA require multi--hpp inference across tabular data and unstructured text, and the evidence required to answer a question can be distributed different ways over these two types of input, making evidence retrieval challenging our base- line model using an iterative retriever and BERT-basee reader achieves an exact match score less than 10%. We then propose two novel techniques to address the challenge retrieving and aggregating evidence for OTT-QA. The first technique is to "early fusion" to group multiple highly relevant tabular and textual units into fused block, which provides more context for the retriever to search for. The second technique is to use cross-block reader model the cross-dependency be- tween multiple retrieved evidence with global-local sparse attention. Combining these two techniques improves the score significantly, to above 27%.
INTRODUCTION
Open question answering considers the problem of retrieving documents from fixed corpus with retriever, and then analyzes retrieved evidence to provide answers to given question with reader. Prior open question answering systems focused only on retrieving and reading free-form passages or documents. However, a significant amount of real-world information stored in other forms, such as semi-structured web tables due to its compact representation to aggregate related informa- tion. For example, tables are often used to hold large quantities of related facts, especially numeric facts, such as 'Career Statistics for Lebron James'. This type of detailed information found much less frequently in unstructured text. Tables are also commonly used for collections of homogencous entities or recurring events, like 'List of Periodic Comets' or 'List of Champi ons League Winners since 1966'. Hence tabular information serves as an excellent complement to textual data, especially in the open setting. Despite these advantages, previous studies have exploited the millions of web tables to augment their open QA system.
In this paper, we describe the first study to jointly exploit tables and text for open-domain question answering. For this purpose, we construct new dataset, Open Table-and-Text Question Answering (OTT-QA). OTT-QA is built on the HybridQA dataset (Chen et al. 2020), and like HybridQA, OTT- QA questions are multi-hop questions which require aggregating information from both tables and text answer. However, unlike HybridQA, OTT-QA requires the system to retrieve relevant tables and text - in contrast, in HybridQA, the ground truth tables and textual passages required for each question are given. To produce OTT-QA's 's questions, we begin by re-annotating the questions from HybridQA to "decontextualize" them-i.e.. we make questions suitable for the open-domain setting
Part of this work was done during an internship at Google. by UCSB NLP Group
Data was released in http:///gttubb.oom//wuuueeeetttt

---

Published as a conference paper at ICLR 2021
Q: How many points per game did Lebron James get the NBA Season suspended by COVID?
A: 25.3
@ Vi0-19 Fundentecy
28 20 NBA Best Ployeors
Lobson Jomes
28-20
COU1D-19
Lebron Juiees Conde
Stobooics of NBA
HALA
COMD--12
of teame
Career gratitic
2019-2020
Hoterageneous Retriever + Reader
Lebron James Careor Stabitica (Regular Season)
Year
Team
Blocks Per Game
Points Per Game
Acsists Per Game
Point
Rebound Per Game
17:18
Cleveland
0.9
27.5
9.1
367
8.6
19-20
LA. Lakers
0.5
25.3
10.2
348
7.8
2017-18 season the scason National
2019-20 season the sesson of the National Baskertball Association (NBA). The season was suspended COVID-19.
The 2018-19 Los Angeles Lakers season was the franchise's 71st season, its 70th season in the National Bosketball Association (MBA)
Basketbail Association (NAA). The regular season began on October 17, 2017. ended on June 3 with the Gol State Warriors defeating the Cleveland Cavollers the 2019 Final.
The problem setting: A OTT-QA model needs to retrieve from two candidate pools and then perform multi-hop reasoning to find answers.
Figure 1:
so that unique answers can be determined from the question alone, without needing context from the provided text and tables. We then add new questions to remove potential biases. After these questions that require retrieving and aggregating information over tables and text from the whole Wikipedia. Examples from OTT-QA are depicted in Figure Note the table and passages contain non-overlapping information, and both of them must be understood to answer the question. For example, the question has a low lexical overlap with the passage about the 'Lakers', and it needs the table as the bridge to retrieve this passage. Such cross-modality multi-hop retrieval features OTT-QA. More examples are displayed in Appendix.
steps, OTT-QA contains 45K numan-annotatie
OTT-QA is distinguished from the existing QA datasets in two aspects. Existing table-based QA datasets (Pasupat & Liang 2015] Yu al. 2018, Chen et al., 2020) operates in the closed setting without requiring any retrieval, whereas most existing open QA datasets (Joshi et al. 2017] Yang
ct al. 2018 require only text retrieval, not table retrieval. One dataset, Natural Questions (NQ) (Kwiatkowski et al. 2019) includes some tabular information in its corpus, but the tables are nearly always of restricted type (infobox tables with only single row). In contrast, OTT-QA models require retrieving both tabular data and text, and unlike the NQ dataset, requires information fusion from text and tables in non-trivial ways. OTT-QA poses novel and realistic challenges both the retriever and reader in open QA though the questions are less natural than the real queries from NQ (Kwiatkowski et al. 2019 Retrievers for OTT-QA need to consider two information formats, making the search space larger. Even worse, as questions in OTT-QA often require multi-hop in- ference, one round of retrieval is often not enough. Readers for OTT-QA also need to aggregate a information, compared to other reader models: a single
significant amount of snowledge-intensin table in OTT-QA has an average length of over 300 words. Moreover, readers are often expected to process multiple retrieved units due to the uncertainty in retrieval, which makes it difficult to design strong reader models (Devlin et al. 2019 Liu et al. 2019) with length limit of 5 512 tokens.
The baseline system that propose to address these challenges uses an iterative retriever (Sun) cet al., 2019:] Qi et al., 2019; Min et al.) 2019 Ding et al.) 2019 Asai et al. 2019) and BERT reader (Devlin et al. (019). The iterative retriever explores multiple evidence documents iteratively, interacting with the candidate pool to gradually reformulate the query. Beam search is used to find multiple subsets of documents that may contain all the required evidence, and each subset is then fed to the BERT reader to predict the answer span. The highest-scored prediction is chosen as the answer. The iterative retriever needs to re-encode the query with a big transformer and re-search over the candidate pool, such a procedure (especially dense) can be computationally expensive. Furthermore, the BERT reader fails to capture global overview of the retrieved documents, which leads to bad local optimum in the model prediction.
We propose more sophisticated system that addresses these challenges with two novel strategies: namely fusion retrieval and cross-block reading. The fusion retriever first pre-aligns the table seg- ments to their highly related passages, using entity linking. Then, the aligned table segments and passages are grouped as a fused block, which contains aggregated information from two modali- ties: hence, compared to the previous documents, it contains richer context to benefit the following retrieval. We view the fused block as the basic unit to be retrieved, and instead of performing mul- tiple runs of retrieval iteratively, the fusion retriever is used once to retrieve the top K fused blocks; however, due errors in fusion and retrieval, the retrieved top-1 fused block might not contain
2

---

Published as a conference paper at ICLR 2021
the necessary information. We thus also propose cross-block reader based on sparse-attention based transformer architecture (Ainslie et al., 2020 Zaheer et al., (020). which can process ex- tremely long sequences efficiently. We use the cross-block reader to read all the top-K retrieved fused blocks jointly. Both strategies have proven effective compared to the baseline system: the best model combining the two strategies improves the accuracy of the baseline system by huge margin.
BACKGROUND
The aim of an open QA system is extract an answer to a question g from a given large corpus. Most open QA models are retriever-reader models, which extract answers in two steps: retrieval and reading. In the retrieval step, a retrieval model f is used to retrieve a set of passages from the text corpus. In the reading step, the reader is then used to extract the answer from them.
Retrieval Function There are two commonly-used types of retrieval function f: sparse retrievers and dense retrievers. Our sparse retriever uses a unigram- based BM-25 score to retrieve an evidence unit b from the candidate pool E. Our dense retrieval function is dual-encoder model (Bromley) al. 1994), and we follow (Lee et al. 2019 Guu et al., 2020) for the dual encoder design. The query and the passage are encoded with separate Transformers. As in (Devlin et al.]] 2019), the vector corresponding to the first token, [CLS]] is used as a "pooled" representation of the sequence. The dense retrieval function is the dot product between h. = (g) and lss = BERT (b)[CLS] for each evidence block b in the candidate corpus-i.e., the scoring function is f'q, b) h les. which can viewed as finding the nearest neighbor vector space. In the multi-hop open QA set- ting Yang et al. 2018). an iterative retrieval function (Sun al. 2019 Min al.) 2019 Ding al.) 2019) is proposed, which defines the retrieval process as an auto- regr-regressive formula. Our iterative retrieeer function is denoted as ((g. 61 ... by 1).1,,bb) b,). which appends the previous
rounds of retrieval to the original in in the j-th round of retrieval. Beam search is used in test time.
Single-Block Reader Due to the uncertainty in retrieval, the top-1 document might not always contain the answer. Existing models normally retrieve the top-k documents and feed them to the reader for span selection. The standard reader (Chen et al. [0177] Joshi et al.) 2017) aims to extract span from each of the retrieved blocks b; and assign confidence f'l., bx) fread (alg. by) to it, with (q. by) indicating the retrieval probability and fread qg b.) denoting the span selection probability by reader. Multiple answers {ai,... .xx) are ranked with this confidence score and the highest scored answer span à is the final answer. Note that the reader needs to run k times, once for each of the top-k retrievals. We refer to this model the single-block reader and use our baseline.
HybridQA HybridQA (Chen et al. 2020 closed -omeinn QA dataset, is the most related to ours. During the annotation of Hybrid QAb table and its relevant passages {PI..... ........ (sur- rounding text and hyperlinked passage) essarily require both the passage and table information to answer. The original dataset contains 72K multi-hop questions paired with 13K tables with their paired passages. During training/testing time, the ground-truth tables and passages are given to a model, HYBRIDER, to find the final answer. HYBRIDER also serves as an important baseline in our paper.
given crowd worker to write questions which nec-
TASK AND DATASET
In OTT-QA, the retrieval corpus consists of a set of table candidates By and a set of passage candi- dates Rp, The task is to answer question g by extracting answer strings from blocks b E By U BBp, where b can be either textual and tabular data. We adopt the standard exact match (EM) and F1 scores (Yang et al., 2018) for evaluation. Different from HybridQA, OTT-QA's table candidates are web tables without hyperlinks provided. This decision was made to make the problem setting more general, as otherwise systems that solve OTT-QA could only be applied to high-quality data in Wikipedia. However, in OTT-QA, we provide hyperlinks in the training subset, but not dev/test set. Removing hyperlinks in tables makes the overall task much more challenging, but makes the final systems applicable to more general domains. Thus, an OTT-QA model needs to jointly retrieve both tables and text, without abusing gold hyperlinks, and then aggregate them to find the answer.
Candidate Pool For our table collection By, we extracted all Wikipedia regular tables with their metadata including page title, page section title, and section text. The metadata, denoted TMr is essential for de-contextualization. We obtain a table corpus containing over 400k high-quality tables
3

---

Published as a conference paper at ICLR 2021
with average length of 320 words including metadata. For the text passage collection Bp, crawl English Wikipedia dump pages and filter out noisy pages. We follow HybridQA (Chen et al. 2020) and only keep maximum of 12 sentences in the introduction section as the passage. We obtain a corpus containing over million passages, with an average of 94 words.
Notation We define each table as a matrix T, which consists of cells Ti, with i specifying the row, and j specifying the column. Each cell Ti could be number, date, phrase or even sentence due to its semi-structured nature. However, a single complete table with structured representation (Herzig al. 2020) can easily exceed the 512-token limit, which poses great challenges to the downstream reader to process top- K retrieval. Hence we propose to decompose each table T into multiple rows R2 which are combined with the headers, metadata, and global max/min information from the original table as table segment. The table segment is used the basic retrieval block in our paper. This decomposition procedure increases candidate By from 400k to 5 million, making the retrieval problems described in Appendix (subsection B.I] In summary, we build candidate pool of 5 million table segments By and a pool of million passages By. We denote B as our full candidate pool, which our model needs to find the block b (a table segment or a passage) containing the answer span.
more fine-gree aeneigr and more challenging. Our table segment representation
3.1 QUESTION AND ANSWER ANNOTATIONS
Our question and answer pairs are built upon the existing HybridQA (Chen et al. 2020) dataset, with several significant changes. First, crowd workers 'decontextualize' the questions so that they are not under-specified or context-dependent, and thus suitable for the open setting. Second, we add more questions to the development/test set to remove possible annotation bias. During annotation, we adopt strict quality controfand more details are described in Appendix (ppendii(section section A.1
Decontextualization Most questions in HybridQA are contextualzeed with given table and several passages, with corresponding questions written by crowd workers. Often, the crowd- sourced questions assume the context. For example, the questions might contain the words "the players" because the given table is about "Netherlande players". We thus needed 'de- contextualize" (Parikh al. 2020) the original context-dependent questions, so they could serve as standalone questions, specific enough to imply a unique answer relative to the corpus. To dis- enforce two-step annotation procedure, as depicted in Figure 2] In the first phase, the worker is only allowed to insert minimum words or phrases (or replace pronouns) into the questions based on the information presented by Wikipedia Title, Section Title, and Section Text to make the question have a unique answer. After this step, we often poten- tially obtain overly-complicated questions that are artificial and unnatural. Therefore, we manually selected the worst 25% questions and sent them back to make them more concise and natural.
courage excessive unwanted modification,
Page Tille: Netherlands al the European Track Championships
Section Title: European Track Championships (elitt) 2010-current
schema
Medal
Championship
Name
Event
Ranking
Silver
Pruszków
Veldt
Men's omnium
2nd
content
Bronze
2011 Apeldoorn
Kirsten Wild
Women's is omniuum
3rd
OTT-QA Annotation
Original
Which city does player winning the silver medal Men's Omnium from?
Insertion
Which city Netherlands player winning the Men's Omnium silver medal ETC after 2010 from?
Naturalize
Which city Netherlands Men's Omnium meda list after 2010
from?
Figure 2: The dée-contettualization annotation phase of OTT-QA. In the first step, the annotator is restricted add phrases from the context. the second step, the annotator is specifically requested to make the sentence more concise and natural.
Additional Evaluation Examples As all the questions from HybridQA are based on the 13k tables from the HybridQA set, no questions are asked about the newly crawled 400k tables. This
"The collection is conducted on an established crowdsourcing platform with annotators from countries with English as the native language. The annotators were required to meet the requirements: 1) native speaker in English-speaking country having approval rate of over 95% and having at least 500 approved jobs.
4

---

Published as a conference paper at ICLR 2021
potentially generates unwanted statistical biases artifacts for the model to exploit, and potentially biases the final evaluation results. Therefore, we randomly sampled another 1100 tables from the newly crawled tables, and follow the original annotation process used by HybridQA to re-collect 2200 new questions. These new questions were mainly used in the dev/test set. Below we refer to the subset of tables used by original HybridQA the in-domain tables.
Distant Supervision Signals For the in-domain tables (<z 8k), the cell-wise hyperlinks are pro- vided in OTT-QA as potential signal for supervision. We use H., = (b1, b2.... E [Bp} } to denote the hyperlinks in cell T.j. Since in HybridQA the oracle fine-grainee answer span is not explicitly annotated, we approximate this by traversing the table and hyperlinked pasasages to find all exact matches. This process contains some noise-a manual study reveals that it roughly contains 15% error. We use this "weakly-supervieed" fine-ge fine-grained information to train our models. We denote the "approximate" block of the answer span for answer ba. and use it to train model.
3.2 DATASET STATISTICS
After annotation, we sampled roughly 2K questions from the in-domain HybridQA dataset, and then mix them with the newly collected out-domain questions to construct our dev and test sets. Finally, we have 41,469 questions in the training set, 2,214 questions in the dev set, and 2,158 questions in the test set. We conduct more in-detailed analysis over the reasoning types and show them in the Appendix A.3 remarkable difference from original HybridQA is that a proportion of questions actually have multiple plausible inference chains in the open-domain setting.
MODEL
Our model for OTT-QA retriever-reader model with new designs for both retriever and reader. As discussed briefly above, we propose to use a fusion retriever instead of using standard iterative retrieve, and we also propose to use cross-block readers to replace a standard single-block reader.
b3
Linking
b
Q
by
Fusion
b9
Figure 3: Left:
Iterative 3-step retrieval individual blocks (baseline). Right: Fusion 1-step
retrieval over fused groups, which greatly lowers the cost of iterative encoding and retrieving.
4.1 FUSION RETRIEVER
Iterative retrieval (Figure 3 Left) has the following issues. First, iterative retrieval training often requires having supervision signals for every retrieval step reach good performance, which is not available in OTT-QA. The iterative retrieval also suffers from the problem of error propagation, as early mistakes can propagate later retrieval stages. Finally, the computation dual-encoder for iterative retrieval is very high, for every stage, the query embedding has to be re-encoded to include the entire retrieval history.
for applying
We propose an alternative strategy to replace multi-step retrieval, namely fusion retrieval (Figure Right). In the fusion retriever, we first use an 'early fusion' strategy to group relevant heterogeneous data before retrieval. The fusion procedure groups several highly-relevant blocks from different modalities as a self-contained group (fused block), which provides more clues for the retriever to utilize. Early fusion is very important for retrieving table segments, which often have incomplete context by themselves. The early fusion process aims to fuse a table segment and relevant pas- sages into a group. Here propose fuse entities mentioned in table segment to the appropri- ate passages for those entities; this is similar to document expansion based on a traditional entity linking step. The problem is challenging due to the mismatch between the lexical forms from the
5

---

Published as a conference paper at ICLR 2021
How many points per game did Lebron James get in the NBA Season suspended by COVID?
512
Ans: 25.3 Conf: 94%
BERT
Attention
512
Ans: 30.1
BERT
Conf: 95%
511
Ans: 0.9
C
BERT
Conf: 96%
Length >4K
ETC
Ans: 25.3
Figure 4: Left: Single-block reader with input shorter than 512 tokens (baseline). Right: Cross- block reader with length over 4K tokens, and denotes the global state assigned to local block A. The single-block reader is stuck at local optimum, while cross-block reader outputs global optimum.
table (which for brevity are often abbreviated) and the relevant passage titles. For example, a cell the table of "NCAA Division 1 Men's Football l'urrnament' contains the term "penn State". Directly matching "Penn State" against the passage corpus will lead to "Penn State University" rather than the ground-truth hyperlinked entity, named "Penn State Nittany propose additional augmentation step, which takes in table
Lions football", Therefore, segment block by and generates a sequence of augmented queries Q1, f2..... In token by token to make the queries more similar to the passage title. The augmented queries for nearest neighbors the passage corpus Ep using BM25 as the final entity linking step, which is depicted in Appendix. The query augmentation is implemented with GPT-2 model (Radford al., 2019), fine-tuned on the supervised pairs of (table segment, hyperlink) from the in-domain tables. Each by is fed to find its companions b........ .b), they are collectively called bF.
then used to search
We follow the standard dual-encoder setting (section 2) and the only difference is that we replace the input of the block encoder with h4 BERT; [br, bp.... .b))... which captures the cross-altention between the table and the text within a block. The fused embedding contains richer context from both modalities to complement each other. The retriever only needs to retrieve once from the candidate pool, which dramatically decreases the complexity compared to the existing iterative retrievers.
To enhance the neural retrieval system to retrieve fused blocks, we apply the Inverse Cloze Task (ICT) (Lee et al.] 2019) pretraining task on the corpus of fused blocks. ICT is a way to generate pseudo-training data for dense retrieval. Unlike standard document-wise ICT, our fused block con- tains both table segments and multiple passages. Given a fused block by, generate the pscudo- query in the following way: I) we first corrupt the table segment by randomly dropping half of the words from the table metadata and cells to obtain a partial table segment by. 2) We then randomly sample a sentence bp from the fused passage. We combine by and 6p as a pseudo query g and pair with the original fused block by as pre-training data. The pre-training data is applied enhance the dual encoder's ability to select lexically matched documents. After pre-training, the retriever is fine-tuned on OTT-QA. Finally, at inference time, the retriever is used retrieve the top K fused blocks for a question, which are then passed to the reader for answer prediction.
4.2 PROSs-BLOCK READER
The reader typically needs to process the top-k retrieved blocks returned by the retriever extract the best answer, as the top-1 block might not contain enough evidence to answer the question. As demonstrated in ingiiueee 4| the cross-block reader aims to address this issue by using cross attention between different blocks to model their dependencies. To obtain the cross-block reader, we take the pre-trained long-range sparse attention transformer (ETC) (Ainslie al. 2020), which can accept up to 4096 tokens input, and then fine-tune the model on the distant supervision data. During training, the ground truth (fused) blocks are mixed with hard negative blocks from the retriever. We take the top-k retrieval results to fill the 4096 token space (roughly 15 fused blocks).
Cross- attenttonn between blocks allows a much more powerful way to aggregate information across the l retrieved blocks compared to the single-bloc reader, especially when the blocks fused. This is feasible because of the design of the sparse attention structure in ETC, which can constrain the attention of each token its neighboring tokens within local radius in its local block. Such

---

Published as a conference paper at ICLR 2021
Retriever
Dev-Sparse
Dev-Dense
Test-Best
Model
EM
F1
EM
F1
EM
F1
HYBRIDER (Top-1) (Chen el al.
2020)
8.7
10.9
8.9
11.3
8.4
10.6
HYBRIDER (best Top-K) (Chen al. 2020
9.9
12.2
10.3
13.0
9.7
12.8
llerativeReetrieve Single-Block Reader
9.8
13.3
7.9
11.1
9.6
13.1
Musion-Rettieve + Single-Blooe Block Reader
14.3
17.8
13.8
17.2
13.4
16.9
SeerativeReetievall + Cross- Block Reader
17.1
20.7
14.4
18.5
16.9
20.9
Retrrevvl Cross-Block Reader
27.7
31.8
28.1
32.5
27.2
31.5
Table-only Retrieval + Cross-Block Reader
4.6
6.9
4.9
7.2
4,4
7.0
Text-only Retrieval Cross-Block Reader
8.2
12.4
8.9
12.8
8.8
12.1
Oracle Link + Fusion-Retrieval + Cross-Block Reader
35.8
40.1
35.2
39.9
35.0
39.5
Oracle Table + Link (w/o Retrieval) + HYBRIDER
44.1
50.8
44.1
50.8
43.0
49.8
Table 1: Main Results. We conduct experiments with both sparse and dense retrievers using the dev set, and then select the best setting report the test results (as indicated by the word "Best"). Fusion- Retriver and Cross-Bloct Reader are combined to obtain the highest score. i are ablations.
sparse attention can decrease the attention computation complexity from quadratic O(N2) to linear O(N] R(), where R is the local radius (where N 4096 and R| 84 in our experiments). To allow cross-block interaction, ETC assigns a global state for each local block in the long sequence, and blocks attention to each other through multiple layers of such global-local structures.
EXPERIMENTS
All of our code is based on Tensorflow (Abadi et al. 2016). For the retriever part, the sparse retriever built on top of DrQA (Chen et al.] (017)) with unigram features, and the dense retriever built with BERT. The single-block retriever based on BERT-uncased, and the cross-block reader is based on ETC (Ainslie et adl])200))) Both of them consist of 12 layers with a hidden size 768, the minor differences in the relative positional embedding used in ETC. All the models trained with a learning rate of le-5 optimized by Adam (Loshchilov & Hutter 2019). We use in-batch negatives (Lee al., 2019) to train dense retrievers. A more detailed implementation of the baseline iterative retriever is described in Appendix |subsection B.2 In fusion retriever, we use the 'fused' block containing the "approximate' answer block b, as the positive instance. In iterative retriever, since the auto-regressive model [b,ygg b.. ... b requires fine- tineggrinedd inference chain for step-wise supervision, which is not given OTT-QA. We apply lexical match based heuristics to synthesize inference chains as weakly supervised training data (described in Appendix). For all the dense retrievers, we pre-train with 10K steps using the generated pseudo query and then fine-tune them another 10K step using batch size of 2048. For the cross-block reader, we fine-tune with batch size of 64, Both are using 16 cloud TPUs.
Main Results In experiments, experiment with different types of retriever and reader mod- els under both sparse and dense setting, the details are described as follows:
this model, designed for closed domain HybridQA questions, is one baseline. Since this model requires ground truth table with its hyperlinks to do modularized reasoning, we use BM25 to retrieve the most relevant table and passages to reconstruct an 'approximated" input for this model. We experiment with top-1.2,3,4 cases where we use the answer with the highest confi- dence as the final result. We also directly feed the ground-truth table and hyperlinks to HYBRIDER, which roughly estimates an upper limit of this task.
HYBRIDER:
Perative-RReriever (Sparse): We use a 2-step iterative retriever: in the first step, we apply the question to retrieve the top- 10 table segments and top-10 10 passages. In the second step, we use each retrieved table segment to retrieve its related top-5 passages and concatenate each retrieved passage title with the original question to retrieve the top-5 table segments. We merge and calculate the re- trieval score of each unique block and rank them by their score. For the single-bloct reader, we split the retrieved blocks into 512-token chunks and feed them to the BERT reader. For the cross-block reader, we truncate the top 4096 subword tokens and only feed these tokens to reader.
Iterative-Be Retriveer (Dense): We use a 3-step iterative retriever. In the first step. we encode the question and retrieve the top-8 blocks (either table segment passage); in the second step,

---

Published as a conference paper at ICLR 2021
concatenate the previous retrieved block and the question to re-encode the query vector to further retrieve top-4 blocks; similarly, the last step retrieves top-2 blocks.
Fusion Retriever (Iterative): We use a sparse retriever to directly retrieve the top- fused blocks based bag-of-we words BM25 score, and then split it into individual table segments and passage blocks. Since passage could be associated with multiple fused blocks, we merge duplicate blocks and use their summed score. Finally, rank each block based on its merged retrieval score and truncate the first 4096 subword tokens for the next step.
Fusion Retriever (Iterative): We use a dual-encoder dense retriever to directly retrieve the top- fused blocks, and then follow the same procedure as above. Without specifying the dense retriever uses ICT for pre-training by default.
Fusion Retriever w/o ICT and w/o GPT-2: these two ablation studies are aimed to show the effec- tiveness of our proposed ICT pre-trainn pre-training and query augmentation.
The main results are presented in Table First, we can observe that best HYBRIDER top-2 can only achieve comprised exact match of 9.9% while the oracle HYBRIDER can obtain score of 44%, which reflects the difficulties of the hybrid retrieval in our dataset. We restrain the retriever to only retrieve table and text to answer the questions and report their results in (Table 1] even with the strong cross-block reader, the model only obtains 10% EM. These experiments demonstrate the necessity to integrate information from both forms OTT-QA.
By combining the standard iterative retriever and single-block reader, we can slightly improve the score can to roughly 10%. By replacing the iterative retriever with the proposed sparse fusion retriever, the EM score reach 14%, 4.5% absolute improvement. By replacing the single-block with the proposed cross-blost block reader, the EM score can reach 17% a 7% absolute improvement. However, by combining the two strategies, the final EM score reach 28%, with 18% absolute improvement, which is greater than the sum individual improvements. The observation suggests the two components can affect each other in a positive way. We conjecture that the fusion retriever blocks in group, which makes the multi-hop reasoning
more likely to retrieve mutually-supporiae across different blocks easier for the following cross-block reader. In comparison, the iterative retriever retrieves isolated table segments and passages separately, which can easily miss out on the bridging evidence for building the complete reasoning chain. Thus, the cross-blocs reader cannot maximize its advantage in reasoning across blocks.
By removing the ICT pre-training and query augmentation, we observe that the Dev-EM score drops to 24.6%. By removing the GPT-2 query augmentation, the Dev-EM performance drops to 22.1% 16. These two results indicate the effectiveness of the proposed two strategies. By replacing the predicted hyperlinks with the oracle links, the fusion model performance can increase by 7% EM. This indicates that there is still plenty of room to improve for the table-passage fusion model.
Linker/Retriever Results To understand the results more, we evaluate the standalone table- passage entity linking accuracy and retriever recall.
60
60 BBB2
Dual-Enn U GPT2
sparse II dense
52.4
45.1
50.7
50.1
50.4
50
42.7
44.214.1
43.3
37.2
35.8
40
35.9
24. 24.222.6 8
26.2
30.3
30
20
precision
recall
F1
Iter-1
Iter-2
Iter-3
Fusion
Figure 5: Entity linker performance (F1).
Figure 6: Retriever performance (HITS @4K).
We consider the following linking models: a) BM25 model, which directly uses the cell value to retrieve passages based on their titles without query augmentation, b) Dual-Encodes model, which encodes the cell value and meta information into query vector to compute dot dot-product over all the passage candidate to retrieve, c) a GPT-2 model, which first augments the cell value by the context and then uses BM25. We demonstrate findings in Figure 5) and evaluate with table-segment--ise
8

---

Published as a conference paper at ICLR 2021
Fl score. We observe that directly using BM25 leads to compromised precision of 30.3%, which is mainly due to the lack of context information. By using a dual-encoder retriever, the precision can be improved to 42%. However, many table segments have either zero or multiple linked passages and can be better modeled by an auto-regressive retrieval process.
We use HITS@4K is used measure the retriever performance, which indicates the chance of ground truth block existing in the retrieved 4096 subword tokens. The results are reported in Fig- ure We vary the steps of iterative retrievers to show the necessity of multi-hop retrieval in OTT- QA. We observe that the 1stepp retrieval has the lowest recall because the answer block in OTT-QA normally has a lower lexical overlap with the query. Adding the second retrieval step can greatly improve the recall, but adding the third retrieval hop has very little impact. In contrast to the iterative retriever, the fusion retriever can consistently improve the performance over the iterative setting for both sparse and dense setting. The sparse setting can rise from 35.8% to 48.1% indicating the advan- tage of 'early' fusion. The dense retriever's improvement is more dramatic (from 27.2% to 52.4%). We believe this is because the iterative retriever heavily relies on noisy synthetic inference chain data, while the fusion retriever does not require such a fine-grained supervision signal, thus less prone to noise. To better understand the retriever, we conduct detailed error analysis in Appenddxx C
6 RELATED WORK
Table Retrieval: Tables are pervasive on the Web, there have been some studies on mining web tables to answer open-domain questions (Sun et al., 2016 Chakrabarti et al. 2020). In [Sun et al. (2016), the authors have proposed a pipeline framework to first detect the topic entity and then generate candidate chain, finally ranking chains to predict the answer cell. In Chakrabarti et al. 2020), the authors investigate different similarity matching features to retrieve tables from the web. Our paper is significantly different from these two studies in two aspects: 1) the previous papers use private small-scale datasets while we collect a large-scale dataset and release it for public use, 2) the previous studies are restricted to only using tables as evidence, while our paper considers a more realistic and challenging setting with both table and text corpus. Tables have been ubiquitous infor- mation representation form to express sttisstructured information. There has been a effort to utilize tables in natural language processing applications (Pasupat & Liang] 2015 Zhong jet al., 2017: Yu et al., 20188 Parikh et al. [2020] Chen et al., 2019). However, these existing tasks are restricted to in-domain cases without requiring any retrieval, and our paper is the first to investigate retrieving web tables for downstream tasks. Another pair of related works are TAPAS (Herzig et al.) 2020) and TABERT YYnn et al. 2020), which investigate joint pre-training over textual and tabular data. Our method draws inspiration from these models, and also uses special tokens and embeddings to encode spatial and logical operations inside tables.
Recently, many transformer variants resolve the (n attention cost have been proposed including Sparse Attention (Child et al. 2019)), Reformer (Kitaev et al., 2020). Routing Transformer (Roy et al., 2020). Longformer (Beltagy al. 2020 and ETC (Ainslie al.) 2020). These different transformer models apply hierarchical architecture, local-sensitive hashing, global-local state to decrease the attention complexity to nearly linear. Our cross-block reader is based on ETC (Ainslie et al., (200)). but unlike prior works that process one long document for QA, our task requires reading multille blocks containing both structured and unstructured data. To handle the long sequence of retrieved documents in open-domain question answering, Fusion- in-Decoder (Izacard & Gravel (220)) has been proposed to replace the extractive model with an encoder-decoder generative model. The long sequence of passages dently to decrease the computation complexity, but the decoder still uses full attention over the tens of thousands of encoded vectors to generate the answer token by token. Such full-attention can de- crease the decoding speed by an order of magnitude, while our aateee-attettinbaasee cross-blocd reader still maintain the same speed as the standard BERT model.
Long Range Transformer:
split and encoded indepen-
7 CONCLUSION
We focus on the problem of performing open question answering over tables and text in this paper. By proposing the fusion retriever and sparse reader, manage the increase the model's effective- ness and efficiency by a large margin. One interesting question we would like to ask in the future is: can extend open question answering system more modalities like images or audios, etc?
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In /2th {USENIX} symposium on operating systems design and imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Etc: Encoding long and structured data in transformers. Proceedings of EMNLP 2020, 2020.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learn- ing to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations, 2019.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 2xxiv22004055 150, 2020.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network, Advances in neural information processing systems, pp. 737-744, 1994.
Kaushik Chakrabarti, Zhimin Chen, Siamak Shakeri, and Guihong Cao. Open domain question answering using web tables. arXiv preprint arXiv:2001.0322 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1870-1879, 2017.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, 2019.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang. and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Proceedings of Findings of EMNLP 2020, 2020.
Rewon Child, Scott Gray, Alec Radford, and Uya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1at../9444//000 10509, 2019.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00900 009044 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W Cohen. Differentiable reasoning over a virtual knowledge base. In International Conference on Learning Representations, 2019.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2694-2703, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training. Proceedings of ICMI. 2020, 2020.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. ACL 2020, 2020.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering arXiv preprint urXiv:2007.0/2822 2020.
10

---

Published as a conference paper at ICLR 2021
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen- tau Yih. Dense passage retrieval for open-domain question answering. EMNLP 2020, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Kenton Lee, Ming-We Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086-6096, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint urXiv:1907, 11692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR 2019.
Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. Knowledge guided text retrieval and reading for open domain question answering. arXiv preprint arXiv: 1tXiv:/9/1.0386/ 1.03868, 2019.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: controlled table-to-text generation dataset. arXiv preprint arXiv:2004. 14373, 2020.
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 1470-1480, 2015.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. Kilt: benchmark for knowl- edge intensive language tasks. arXiv preprint arXiv:2009.02255: 2020.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher Manning. Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- on Natural Language Processing (EMNLP-IJCNLP)), pp. 2590-2602,, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog. 188)99 2019.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance relevanceframework. BM25 and beyond. Now Publishers Inc, 2009.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997 2020.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pp. 4231-4245 Brussels, Belgium. October-Novemtnn 2018. Association for Computational Linguistics. doi: 10.18653/v1/DD 14555 URL U https: www. aclweb. owg. org/anthology/
D1 D18-1455)

---

Published as a conference paper at ICLR 2021
Haitian Sun, Tania Bedrax- Weiss, and William Cohen. PullNet: Open domain question answer- ing with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNI,P-JCNLPP) pp. 2380-2390, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19--1244 URL
https: www wttp://wwwwe...g/aaaaa... http://ww....ge...oooooo///./-// /ttp/////////.o..oooooooooooooor
Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. Table cell search for question answering. Proceedings of the 25th International Conference on World Wide Web, pp- 771-782, 2016.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380 2018.
Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. Learning discriminative pro- jections for text similarity measures. In Proceedings of the fifteenth conference on computational natural language learning, pp. 247-256, 2011.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. ACL 2020, 2020.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3911-3921, 2018.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007. /4062,, 2020.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00000 2017.
12

---

Published as a conference paper at ICLR 2021
A DATASET COLLECTION
A.1 DATASET ANNOTATION
Filtering The original HybridQA dataset contains over 72k questions paired with 13k hyperlinked tables. We adopt two filtering heuristics to make the decontextualization easier. First, we filter out tables without enough meta-information we filter out overly-long questions, i.c., questions longer than 30 words. These two filtering heuris- ties result in cleaner subset of 46k questions paired with 9%: in-domain tables.
containing too much non-textual information Second,
Quality Control During annotation, we conduct strict manual quality evaluation over the de- contextualized questions, with the following criteria: the annotated question retains the same semantics and answer as before, 2) the annotated question still requires multi-hop reasoning over both table and passages, and the annotated question concise and fluent. The manual quality checking was performed over batches distributed to the same annotator. Each batch consists of six questions, one of which will be sampled to decide the acceptance/rejection of the whole batch. The overall acceptance rate for the crowd-sourcin job is 7 1%, and rejected job was re-distributed until was accepted.
A.2 DATASET EXAMPLES
We demonstrate more examples in Figure/ 7 which includes more diverse inference chains, like table + text; text > table - text; text + text - comparative + table. Our model able to perform these reasoning types quite well by jointly matching query against a fused table-text block.
ColVDD Pandemics
Best Players
Lebron Jomes
15.20
COVI-11 socome
Lebron lomeer caree
Starstics of 19 NBA
COMID-12
of deamns
Career Statotics
2019-2020
Wikipedia Tables
Retrieva
Wikipedia Passages
Lubron James Career Statistica
2010-J0 20 season the season the National Basietball Association (MBA)). The unason was susponded COVID-19,
2015-15 Angeles
Year
Team
Blocks For Game
Points Per Game
sesson was the franchise's 71at season, its 70th season in the National
17:18
Cleveland
27.5
Beskwtbblll Association (MBA), and
19:20
Lakeess
25.3
The Cleveland Cave iers often referred to as the Cavs, are an American profesttral basketball tezm based in Cleveland The Cavaliers compete National Basketboll Association
Basketball Association (NBA), The regular season began on Galden State Warriors defeating the Cleveland Cavallers in the 2018 Final.
October 17, 2017. ended
of J.B.
Diveton,
Q1: many points per game
James get COVID-19 19 NBA Season?
A1: COVID-19 19-20 Season 25.3
Lebron played achieve highest
A2: 27.5 Cleveland J. B. Bickerstaff
score in his career?
Q3:What suspends the NBA season during which Lebron James has an average points game of 25.37
A3: 25.3 19-20 Season -> COVID-19
Q4:For the season suspended by COVID-19 and the season defested by Warriors in Final, which season has Lebron obtained more paints?
A4: 21.3 27.5 17-18
Figure 7: More examples from OTT-QA
A.3 QUESTION TYPES
We randomly sampled 100 questions from the dataset to manually analyze the kinds of inference chains seen in OTT-QA and divide the major types into the following categories:
Single hop questions (13%) require reading one table or one passage to answer.
2. Two hop questions (57%) require reading one passage and one table answer. These be subclassified as 'table bridge' - 'answer text (4) or 'text bridge" + 'answer table'.
3 Like longitude, latitude, mathematical formulae, etc
'L.c,, table forms bridge hetween question and textual passage, which is read in the first hop, and the extracted from the
13

---

Published as a conference paper at ICLR 2021
3. Multi-hop questions (30%) require reading two passages and table to answer. These mainly following the reasoning chain of 'text bridge + 'table bridge' - "answer text'.
4. Questions with multiple reasoning paths: Due to information redundancy in Wikipedia, similar information can appear both tables and answerable by reading one text passage, 18% of questions are answerable by reading two text passages and 4% of questions are answerable by reading tables.
We find that 9% of questions are
B MODEL DETAILS
B. RETRIEVAL BLOCK REPRESENTATION
The table decomposition is visualized Figure 8 The title/section title are prefixed to the table segment. We add the row position token "Ist' and max/min special token over the column to infuse global table information into the segmented unit. The column embedding is added as another vector the representation. The table segment representation is relatively small and easy deal with in the following reader model. After the table-passage alignment, we group the highly related
Labron James Career Statistics
Year
Blocks
2017
Cleveland
2018
Lakers
2019
Lakers
Decomposition
Token
[TAB]
[TITLE]
Lebron
James
Career
Statistics
[ROW]
2st
[MIN]
Year
Token
2017
Team
Cleveland
[MAX]
Blocks
Is
0.90
Column
Statistics
Cleveland Cava Bers
NBA
Lebron James
Cleveland
2017-5eason
Fusion
Taken
[TAB]
[SEP]
Type
D
3
Figure 8: The decomposition of the original table into segments.
units together and represent them the lower part demonstrated in Figure 8 We add [SEP] tokens to separate different passages and set their type id to 0. Such a flattened representation for fused block by will be used throughout experiments for both sparse/dense retriever and ETC reader.
B.2 ITERATIVE RETRIEVER
Iterative retrieval has been used in recent graph-based multi-hop retrieval models gradually re- trieve documents to find the correct supporting evidence. Specifically, the retriever conditions the i-th round retrieval on the previous round of retrieval results.
Sparse Retriever The sparse retriever uses uni-gram lexical feature to compute the BM-25 score between the g, b1. over b, B to get the top candidates. Here we describe two-step retrieval procedure, called here the LxM procedure. between question over all the candidates in B to select top L/2 table segments br, and L/2 passages bp. In the second step, the question is concatenated with the retrieved table segment to form L/2 new queries (q; by] which are used to retrieve LM/2 passages from B. The question is also concatenated with the retrieved passage titles to form another K/2 queries (q: bp] to retrieve LM/2 table segments. The retrieval procedure results in at most LM+L. unique blocks. Each unique block aggregates its
the first step, the model calculates the BM25 score
14

---

Published as a conference paper at ICLR 2021
score from two rounds, denoted as fq, b), which is used to rank the top-K candidates for the next step. We truncate the top-K candidate by thresholding their combined length.
Dual-Encoder Retriever The dual encoder uses a BERT-based encoder to compress each ques- tion, table segment, and passage into fixed-length vector and then computes the dot product be- tween fixed vectors to obtain the highest scored candidates from pool IB. However, since the dataset does not provide an explicit supervision signal for the iterative retrieval, we heuristically synthe- size some noisy retrieval chains using lexical matching. The retrieval inference chain is depicted by -> b2 bx, which is used train the model [bx (g, b1_.-1) in supervised manner. At inference time, the dual encoder retriever will encode query into fixed vector and retrieve the first blocks from The blocks are appended query q form L new queries [g: bu], which re-encoded and search for LM new neighbors. We experiment with a maximum of 3-step retrieval of LxMxN to obtain maximum of L+LxM+LMN unique blocks. Similarly, each unique block aggregates its score from different rounds to select the top-K candidates for the next step.
B.3 SPARSE FUSED RETRIEVER
The sparse fused retriever uses the uni-gram lexical feature to compute the BM-25 score between over bF E BF. The uni-gram feature by is based on representation depicted Subsection B. Note that this BM25 feature will be much more abundant than the BM25 feature in iterative sparse retriever because it encloses more uni-grams. Instead doing multiple rounds of retrieval, the fused retrieval once retrieve once the candidate pool and treat all the units inside the block as the same retrieval score. Finally, We truncate the top-K candidate by thresholding their combined length.
B.4 QUERY AUGMENTATION
The query augmentation procedure depicted in Figure 9
Lebron James Career Statistics
NBA 17-18 Cleveland NBA
NBA 17-18 Season Cleveland Cavallers
Year
Team
Blocks
GPT-2
17:18
Augment
Search
Figure Fusion: 1) GPT-2 query augmentation, 2) nearest neighbor search over passages.
B.5 DENSE RETRIEVAL/IN- NEGATIVE
Recently, different dense-retrieval methods (Lee al. 2019 Guu et al. 2020 Karpukhin et al. 2020) based on dual-encoders (Bromley et al. 1994) have been shown to surpass traditional sparse retrieval in open-QA models. The query and the passage both encoded using Transformer, which produces a vector for every token. As in (Devlin et al.] 2019). the vector corresponding the first token, [CLS], which is used BERTaIs). The dense retrieval function can be represented as the dot product between BERT (g) and (ERTeLs(p) for each document the text collection, much like TF-IDF (Chen al. 2017) and BM25 (Robertson & Zaragoza 2009) on some Open QA datasets. To train the -ual-enncodrr, the in-batch negative trick (Yih al. 2011 Karpukhin et al.) (2020) plays an important role, which uses B training instances in each batch and views the other B-1 instances inside the batch as the negatives. In this way, the model reuses computation and effectively trains on B² question/document pairs in each batch.
"pooled" representation of the sequence (denoted
C PERFORMANCE ANALYSIS
C.1 QUESTION TYPE BREAKDOWN PERFORMANCE
We measure our best model's 's performance (dense fusion retriever cross-block reader) and baseline model (dense Iterative-Retriever to show the breakdown statistics in Figure and Figure 11, As observe, the gap between
Retriever single-bloct reader) on different question types dsubsection A.3)
15

---

Published as a conference paper at ICLR 2021
model vs. baseline in 1-hop question is less significant 2-hop and 3-hop questions. The iterative retriever's performance is sensitive to the number hops in the question, which is the largely due to the eπrоr propagation in the beam search stage. If the retriever fails include the golden block in the earlier stage beam, the retrieval in later stage cannot recover from such failure. In contrast, our fusion retriever group the related information prior retrieval to retrieve all the blocks at once, which makes the model less prone to the eгror propagation issue. Another reason due the cross-block reader, which can reason over different blocks in the latent space, such implicit reasoning can also decrease the error propagation issue. To sum up, our model is more powerful to deal with complex multi-hop open questions with much less performance drop.
IDEM IFFI
EM
IFI
40
40
34.3
30.7
28.1
23
23.4
20
20
20
8.7
3.3
0
1-hop
2-hop
3-hop
1-hop
2-hop
3-hop
Figure 10: Breakdown for iterative retriever Figure 11: Breakdown for fusion retriever + cross-block reader.
sing-block reader.
C.2 RETRIEVER ERROR ANALYSIS
We conduct error analysis to see what are the major issues with the retriever and conclude the fol- lowing types in Figure 12 The major issues causing the system retrieve unrelated evidence low lexical overlap, fusion errors, numerical reasoning and distracting passages or tables. In the
Reason
Example
Groundtruth Block
Overlap
University
now serves
Science
Error Fusion
Who the coach of Pittsburgh team in NCAA division foothall team 1980?
Table: of NCAA Division I
Cell: Pittsburgh
Fail link to the passage: Pittsburgh Panthers
Reasoning
achieves highest score Prix 1980s Men's
Table:
1980s Men's
The time is not regular format, SQL operation
cannat select the max/min row.
Distraction
Where of 1990 Grammy winner for come from in?
Grammy Lifetime Achievement Award Many tables from are about general "Grammy Awards", the groundtruth from the others little bit.
only differentiate
Retriever Error Breakdown
Breakdown of Errors
30%
20%
10%
Numerical
Fusion
Distractoin
Reader
Figure 12: The main error types in the retriever.
Low-Lexical-Overiee case, the enrors are mainly coming from the abbreviation, rephrasing of the table metadata, for example, 'New York University' is shortened as 'NYU', etc. In the Fusion-Erre the issue is mainly because the entity-linking model fails to fuse all the hyperlinked passages, the error (F1=50%) is quantitatively reflected in the entity-linker performance figure. Numerical- Reasoning error is mainly related to the failure to find mav/min/earlies//laeet in the table. The
16

---

Published as a conference paper at ICLR 2021
distraction error is mainly caused by some distracting passages tables having very similar in- formation. We sample error samples from the dev-set and attribute their errors to the above categories. As shown in the left part, found that the numerical reasoning enror is not severe as the other three types because the proportion of questions requiring is relatively small. Besides the low-lexical overlap error, which is general across other open QA datasets like NQ and HoptpotQA, we found the fusion and distraction errors quite specific in our dataset.
(Fusion) Questions which ask about tables which are linked to too many linked passages. For example. a question over table "Team Record" its / //n. wikipedia org/ wiki/Sevens Grand Prix Series is hard because some table rows associate with over 10 passages, it's hard to link them and fuse all of them into a fused block.
(Distraction) Questions which ask about topics which are contained by too many sim- ilar tables, it's hard to differentiate the true tables in https::// / //n. wiki wikipedia. wwtiiaa..r//iiio For example, there over ten RMII_MMiieesR Univessity. people, these similar tables easily distract the attention of the retriever to select the wrong one from the same page.
From our quantitative results, we can attribute the errors retriever and reader, among all the examples, 49% of examples cannot find the correct supporting block. For the rest 51% % examples with correct block retrieved, the reader fails to select the correct span for 23% of them.
C.3 LENGTH SENSITIVITY ANALYSIS RETRIEVAL/READER
We perform sensitivity analysis for both retriever and reader in Figure 13
We gradually increase
Retrieval Recall vs. Retrieval Length Limit
60
Iterative - Fusion
40
Recall
20
0
665
1834
2655
3060
3372
3682
4096
Retrieval Length Limit (top-K tokens)
Reader EM vs. Input Length Limit
30
Single-Block
Cross-Block
Exact Match
20
10
462
665
1834
2655
3060
3372
3682
4096
Input Length Limit (top-K tokens)
Figure 13: Analyzing retriever performance.
the length limit of retrieved evidence from 400 to 4096 to first visualize its impact on the sparse iterative and dense fusion retriever. For both fusion and iterative retriever, we can observe that both of their recall K significantly improves as the length limit increases. With low budget of token limit, their performance is gap is smaller because its performance is dominated by the single- hop questions the dataset. As the length limit increases, the improvement for fusion retriever is steeper than iterative retriever because the contextualized fusion block becomes easier retrieve than standalone table segment or passage.
We also visualize the input length's impact the single-block The performance of single cross- block reader. With a low budget of token limit, both single-block and cross-block readers are com- parable. However, the limit increases 4000, the cross-block reader can digest long input with
17

---

Published as a conference paper at ICLR 2021
its sparse attention mechanism to achieve better scores, while the single-block reader needs to trun- cate the information to read independently, which leads to even lower EM score due to introduced noise. This observation reveals the importance of modeling cross-attention between different trieved evidence units to reach a consistent answer. In single-block reader, dealing with different blocks independently can lead suboptimal prediction in our dataset.
D CONNECTION TO EXISTING WORK
The problem combining structured and unstructured data has been studied in ques- tion answering. The previous approaches are mainly divided into two categories: 1) FusionNet and PullNet (Sun et al. 2018 2019) simulate a KB-incomplete setting by masking out some triples from knowledge graph and use textual information to complete the masked KB triples; these experi- ments are conducted on KB-based QA datasets. DrKIT (Dhingra al. 2019) and Knowledge- Guided Retrieval (Min et al. 2019) propose to use entity mentions and relations to guide the retrieval from the web. However, the KB mainly used as an assisting tool, rather than necessary informa- tion source. In OTT-QA, the structured data is used as necessary information in realistic setting. The two information forms are combined in non-trivial way, which makes the problem much harder than the other structure-unstruccureed QA settings.
KB and Text
Entity Linking Our generative entity linker is related to knowledge-enhanced language understanding (Petroni al. 2020), which proposes seq2seq model to deal with different knowledge-intensive tasks like slot filling, entity linking, etc. There is a concurrent related work on auto-regressive entity linking (De Cao al. 2020), which also demonstrates the advantages of using an autoregressive generation model for entity retrieval.
18


Reviews:
[{"review_id": "peL9FAtu89r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new setting of open-domain question answering. Usually, we only retrieve question-related text from the web or Wikipedia for answering questions. The authors build up a new dataset which need to retrieve both text and the corresponding table to answer open-domain questions. This setting is more close to a real world setting where structured information is also essential. Moreover, the authors propose a pipeline of fused retrieval and cross-block reader to solve the problem. This baseline is very strong and consists of many SOTA methods such as ICT, ETC. And I like the idea of fused retrieval which is very important to build a connection between table and text. Although the idea is close to the entity linking or hyperlinks for multi-hop QA, it is new under this open QA setting. Overall, I would like this paper accepted.\n\nPros:\n1. Release a new task and dataset for answering open-domain questions with text and table. The dataset is carefully annotated by two steps by making use of decontextualization method.\n2. Set up very strong baselines, and propose fused retrieval which is important to achieve strong performance for this task.\n3. This paper is well-written and easy to read.\n\nCons:\n1. I would like to see more analysis on the dataset. For example, what's the distribution of different question types? What kinds of questions are hard to solve? Can the proposed model solve reasoning problems from the dataset?\n2. Missing baselines of using text only and table only to answer the question. It's unclear whether we do need both text and table. \n\n###update###\nI have read the other reviews and the author feedbacks. I would like to keep my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kFngTXTHjD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides a interesting direction in open question answering. In particular, it proposes an open QA problem over both tabular and textual data, and present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task. Two techniques are introduced to address the challenge of retrieving and aggregating evidence for OTT-QA. Results show that the newly introduced techs bring improvements.\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I vote for accepting. I like the idea of open question answering with various types of evidence. The major contribution of this work, in my personal opinion, is the the creation of the dataset which would foster the research on open question answering over text and table. The techs introduced are sound but the novelty in terms of methodology is limited. \n\n \n##########################################################################Pros: \nComments:\n \n1. The paper formulate an interesting problem of open QA problem over both tabular and textual data. \n\n \n2. The creation of the dataset (OTT-QA) is a great contribution to the community. The authors claim to release the data to public. Would the test set make blind so that make it a challenge like SQuAD?\n\n \n3. The method is sound. Experiment study is convincing. Two introduced techs bring improvements. \n\n \n##########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "EEw0skkUGpV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(This review is a collaboration between a junior and a senior reviewer as part of the training to the junior reviewer. Both of them read the paper in detail.)\n\nSummary\n\nThis work extends the task of answering questions over tables and text to open-domain. They construct a new dataset - OTT-QA - on top of a closed domain multi-hop question-answering dataset, which requires reasoning over tables and text. Adding a retriever step poses a challenge for the system to retrieve relevant tables and text, given a question. The authors propose two strategies for retrieval - 1) preprocessing the dataset offline to group tables and text into blocks, which are later retrieved by the retriever, 2) using long-range sparse attention transformers to read multiple retrieved blocks at once. Applying both strategies gives significant gains over other baselines. \n\nStrengths\n\nNew task and a dataset for this task by converting HybridQA to an open-setting.\n\nNew baselines for this task.\n\nWeaknesses\n\nThe proposed dataset is a makeshift dataset that is wrapped on top of HybridQA. This makes the task artificial. HybridQA is built by first selecting a table and then related anchoring documents. Whereas in a true open-setting, there may not be such dependence. A question may be answered on document1, which could lead to document 2 and then a table so on (i.e., the table and document1 need not be related). \n\nApart from the retrieval, no qualitative differences between HybridQA and OTT-QA are presented. \n\nThe reviewer is left with distaste making them wonder what are the scientific takeaways from this work. Currently, the paper feels like here is a dataset and here are a bunch of models. What are the unique linguistic phenomena present in this task? Is it mainly the retrieval that is hardest or the reasoning (reading) once retrieved? \n\nExperiments:\n\n1. An oracle baseline where the gold table is given but not the documents (and vice versa) should be presented to understand the impact of HybridQA annotation procedure on the naturality of this task. Due to the annotation biases, the retrieval task could boil down merely to retrieving the correct table (and its associated documents).\n\n2. Pre-training is a computationally expensive operation. As a practitioner who wants to apply or extend the methods proposed in this work, it would be helpful to know how much performance gains are stemming from pre-training of the neural retrieval system.\n\n3. Results using just tables and just documents are absent.\n\n4. The supervised pairs of (table segment, hyperlink) are essential for forming the fused blocks, however, this information cannot be used by iterative retrievers. What will be the performance of the fusion retriever in the absence of hyperlinks? i.e. without using GPT-2 for query augmentation, or using it without fine-tuning. This will quantify the importance of hyperlinks as a supervision signal.\n\nQuestions for the Authors:\n\n1. How do you define over-complicated, artificial and unnatural decontextualized questions? \n\n2. The authors say one round of retrieval is not enough for OTT-QA because questions require multihop. This is just an artifact because they split the table into multiple blocks whereas in the actual HybridQA setting, all you require is one table and an associated document(s). Doesn’t this mean only one round of retrieval is just enough if you use full-table and associated documents?\n\nSuggested improvements in paper presentation\n\nAlthough the paper mentions that a dual-encoder design is used, the current notation can give a wrong impression that a single encoder is generating the representation of the query and the block. I would request the authors to consider using the notation for dual-encoder as used in [1].\nModify the Lebron James Career Statistics table in Figure 1 and Appendix subsection A.5 to denote “L.A. Lakers” instead of “Lakers” as given in the table on Lebron James’s Wikipedia page.\nTable segment representation is not very clear from Figure 8 of Appendix subsection A.5. I believe the last 4 tokens should be “[MAX] Blocks is 0.90” instead of “[MAX] Blocks is Cleveland”. It would be helpful to have an example in which the minimum and maximum entries are from rows different from the row being encoded.\nIt will be helpful to the reader if there is a paragraph in the Appendix containing details of the formulation of the sparse Fusion Retriever.\nInclude specific Appendix subsection in Section 1 (“ More examples are displayed in Appendix”) and Section 5 (“weakly supervised training data (described in Appendix)”). Remove “discuss in Appendix” from the Dual-Encoder Retriever paragraph in Appendix subsection A.6.\n\nTypos\nTherefore, We -> Therefore, we\nTable 1 caption  - “both brings significant improvement.” -> “both bring significant improvement.”\nAppendix subsection A.1 - “Secone, we filter out” -> “Second, we filter out”\n\n[1] Latent retrieval for weakly supervised open domain question answering, Lee et al., ACL 2019\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3HCymMP9uf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-----------------------------\nSummary\n-----------------------------\n\nIn this paper, the authors introduce the task of open domain QA using table and text. Unlike recent tasks on table and text retrieval, where the table associated with the text is known, in this task both tables and text need to be retrieved from a large collection. Similar to the recent HybridQA task, questions may require fusing information both tables and text to answer questions. The dataset has been constructed by reusing the Hybrid QA dataset as well as crawling additional tables and passages from Wikipedia. Since the questions in the dataset may be non-contextual (for instance, not explicitly mentioning the name of a country when the data is only for that country), the authors used crowd-sourced workers to insert words or phrases from the section title, caption etc and make the question self-contained. However, this can cause the questions to become unnatural and excessively long  - the authors manually select the 25% worst questions and get them refined by a second set of annotation tasks. Additionally, to include new samples beyond Hybrid QA, they collect 2200 new QA pairs using the additional tables crawled and include them as part of the dev and test set. The exact answer in the table is annotated using distant supervision and has an extraction accuracy of 85%.  The overall dataset contains 41.5K questions in train and approximately 2K questions each in the dev and test set.\n\nThe authors also present a method for for this task - It consists of a \"fusion reader\" which that aligns a passage and table segments ( Table row+ metadata= Table segment). These are then considered as one unit for retrieval. To identify the passage-segment alignments, it takes a table segment and generates augmented queries to fetch candidate passages.  The augmented queries are generated token-by-token and are used to transform the queries  into the passage titles. This augmented query generator is based on GPT2 and has been fine tuned using the training data. BM25 retrieval is used to fetch passages given the generated queries. Given a table segment and its related (retrieved passages), BERT is used to jointly encode the two sequentially and the CLS token is used to create a representation of the \"fused\" block.  \n\nThe retriever uses a question, a fused block and computes the dot product between the BERT CLS embeddings to score blocks (as described in Section 2 but the results section suggests its ORCA -- See Q3). Inverse Cloze pre-training is done on the fused blocks to improve the retriever. Answer generation is done by using the pre-trained long-range sparse attention transformer (ETC) to encode questions and the retrieved blocks to return the answer span. \n\nExperiments have been presented using the HYBRIDER baseline from the HybridQA task as well as an iterative retriever reasoner baseline in place of the fused retriever. Experiments indicate that the use of fusion retrieval as well as cross block attention (using ETC) individually and jointly help improve performance over baselines. It is also interesting to note that the retrieval of fused blocks works better with BM25 (Sparse) as opposed to using BERT. \n\n\n\n---------------------------------------------------------\nStrengths and Weaknesses \n--------------------------------------------------------\n\nStrengths\n+  New Task and Model for a challenging problem\n+ Reuses existing datasets as well as baselines\n\nWeaknesses\n\n- Paper is a little hard to follow and required multiple readings.  For instance,  the use and introduction of the words such as \"block\", \"segment\", \"cell\", \"row\" in writing make the paper hard to follow -- perhaps it would be good to define them in the beginning with examples so that it is easy to reference when reading \n- Limited qualitative insights and almost no error analysis -- It would have been useful to include an error analysis along with some qualitative insights though the motivation for fused reasoning over iterative reasoning are clear. What could someone do next on this task? The task is far from solved as the results in Table 1 indicate and the information in Figure 9 in the appendix is not adequate.\n- Both the iterative reader as well as fused retriever would be sensitive to the value of \"top-k\" - No study or details have been provided. This is important -- for example, in case of the iterative reader, 50% of the results included are from passages while 50% of the results are from the tables (as per the information in the appendix). Some recall statistics could be helpful (Also see Q2)\n\n---------------------\nQuestions\n----------------------\n\nQ1 What is the performance of models broken down by question types -- the dataset analysis in the appendix suggests two-hop questions dominate the dataset? \n\n\nQ2 The performance of the HYBRIDER (a model designed for multi-hop inference on text-table data) appears to be lower than the Iterative retriever and single block reader? Could that not be because the BM25 R@1 is likely to be very low -- can the model even do multi-hop reasoning across passages in this case? Perhaps a method that aggregated scores after running HYBRIDER on the top-K retrieved blocks, to return a final ranking may have been a better baseline? \n\n\nQ3 Unless I have missed it, I'm unclear about the sparse/dense retrieval In Table 1 -- is that BM25 applied on the blocks B_F in raw text i.e table segment + passage? If the dense retrieval is based ORCA what is the text in the second paragraph for? Is that not used for retrieval along with the \"retrieval function\" in Section 2?\n\n\nOverall this is a good interesting paper but it has some important questions that were not adequately answered. \n\n\n------------------------------------------------\nUpdated after author response\n------------------------------------------------\nI have updated my overall rating in view of the author response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new dataset for open domain QA where the evidence required for answering a question is gathered from both structured data as well as unstructured data. The authors first show that a standard iterative retriever with a BERT based reader performs poorly on this task. They then propose fused retrieval (grouping relevant tabular and textual elements) followed by a cross-block reader which improves performance.  

R4 has raised strong objections about the artificiality of the dataset. I agree with that and it is unfortunate that the authors did not adequately address the reviewer's concern but instead digressed a bit. As suggested by R4, the authors should tone down their claims about the nature of the dataset. The authors should also simplify the presentation of the dataset as suggested by R2 and not make it unnecessarily complex for the reader. 

However, overall, based on reviewer feedback, the authors have made significant changes to the paper. In particular they have added more baselines, ablation studies and error analysis which makes the paper much more informative. 

I am okay with this paper getting accepted with the assumption that the authors will make the changes suggested above. 


Decision: Accept (Poster)