Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: (This review is a collaboration between a junior and a senior reviewer as part of the training to the junior reviewer. Both of them read the paper in detail.)

Summary

This work extends the task of answering questions over tables and text to open-domain. They construct a new dataset - OTT-QA - on top of a closed domain multi-hop question-answering dataset, which requires reasoning over tables and text. Adding a retriever step poses a challenge for the system to retrieve relevant tables and text, given a question. The authors propose two strategies for retrieval - 1) preprocessing the dataset offline to group tables and text into blocks, which are later retrieved by the retriever, 2) using long-range sparse attention transformers to read multiple retrieved blocks at once. Applying both strategies gives significant gains over other baselines. 

Strengths

New task and a dataset for this task by converting HybridQA to an open-setting.

New baselines for this task.

Weaknesses

The proposed dataset is a makeshift dataset that is wrapped on top of HybridQA. This makes the task artificial. HybridQA is built by first selecting a table and then related anchoring documents. Whereas in a true open-setting, there may not be such dependence. A question may be answered on document1, which could lead to document 2 and then a table so on (i.e., the table and document1 need not be related). 

Apart from the retrieval, no qualitative differences between HybridQA and OTT-QA are presented. 

The reviewer is left with distaste making them wonder what are the scientific takeaways from this work. Currently, the paper feels like here is a dataset and here are a bunch of models. What are the unique linguistic phenomena present in this task? Is it mainly the retrieval that is hardest or the reasoning (reading) once retrieved? 

Experiments:

1. An oracle baseline where the gold table is given but not the documents (and vice versa) should be presented to understand the impact of HybridQA annotation procedure on the naturality of this task. Due to the annotation biases, the retrieval task could boil down merely to retrieving the correct table (and its associated documents).

2. Pre-training is a computationally expensive operation. As a practitioner who wants to apply or extend the methods proposed in this work, it would be helpful to know how much performance gains are stemming from pre-training of the neural retrieval system.

3. Results using just tables and just documents are absent.

4. The supervised pairs of (table segment, hyperlink) are essential for forming the fused blocks, however, this information cannot be used by iterative retrievers. What will be the performance of the fusion retriever in the absence of hyperlinks? i.e. without using GPT-2 for query augmentation, or using it without fine-tuning. This will quantify the importance of hyperlinks as a supervision signal.

Questions for the Authors:

1. How do you define over-complicated, artificial and unnatural decontextualized questions? 

2. The authors say one round of retrieval is not enough for OTT-QA because questions require multihop. This is just an artifact because they split the table into multiple blocks whereas in the actual HybridQA setting, all you require is one table and an associated document(s). Doesn’t this mean only one round of retrieval is just enough if you use full-table and associated documents?

Suggested improvements in paper presentation

Although the paper mentions that a dual-encoder design is used, the current notation can give a wrong impression that a single encoder is generating the representation of the query and the block. I would request the authors to consider using the notation for dual-encoder as used in [1].
Modify the Lebron James Career Statistics table in Figure 1 and Appendix subsection A.5 to denote “L.A. Lakers” instead of “Lakers” as given in the table on Lebron James’s Wikipedia page.
Table segment representation is not very clear from Figure 8 of Appendix subsection A.5. I believe the last 4 tokens should be “[MAX] Blocks is 0.90” instead of “[MAX] Blocks is Cleveland”. It would be helpful to have an example in which the minimum and maximum entries are from rows different from the row being encoded.
It will be helpful to the reader if there is a paragraph in the Appendix containing details of the formulation of the sparse Fusion Retriever.
Include specific Appendix subsection in Section 1 (“ More examples are displayed in Appendix”) and Section 5 (“weakly supervised training data (described in Appendix)”). Remove “discuss in Appendix” from the Dual-Encoder Retriever paragraph in Appendix subsection A.6.

Typos
Therefore, We -> Therefore, we
Table 1 caption  - “both brings significant improvement.” -> “both bring significant improvement.”
Appendix subsection A.1 - “Secone, we filter out” -> “Second, we filter out”

[1] Latent retrieval for weakly supervised open domain question answering, Lee et al., ACL 2019

review_title: New task, a makeshift dataset and baselines. Less scientific and more engineering.

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

 
The paper provides a interesting direction in open question answering. In particular, it proposes an open QA problem over both tabular and textual data, and present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task. Two techniques are introduced to address the challenge of retrieving and aggregating evidence for OTT-QA. Results show that the newly introduced techs bring improvements.

##########################################################################

Reasons for score: 

 
Overall, I vote for accepting. I like the idea of open question answering with various types of evidence. The major contribution of this work, in my personal opinion, is the the creation of the dataset which would foster the research on open question answering over text and table. The techs introduced are sound but the novelty in terms of methodology is limited. 

 
##########################################################################Pros: 
Comments:
 
1. The paper formulate an interesting problem of open QA problem over both tabular and textual data. 

 
2. The creation of the dataset (OTT-QA) is a great contribution to the community. The authors claim to release the data to public. Would the test set make blind so that make it a challenge like SQuAD?

 
3. The method is sound. Experiment study is convincing. Two introduced techs bring improvements. 

 
##########################################################################
review_title: new dataset to support research on open QA over text and table; two techs to retrieve and aggregate evidence; good empirical results

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new setting of open-domain question answering. Usually, we only retrieve question-related text from the web or Wikipedia for answering questions. The authors build up a new dataset which need to retrieve both text and the corresponding table to answer open-domain questions. This setting is more close to a real world setting where structured information is also essential. Moreover, the authors propose a pipeline of fused retrieval and cross-block reader to solve the problem. This baseline is very strong and consists of many SOTA methods such as ICT, ETC. And I like the idea of fused retrieval which is very important to build a connection between table and text. Although the idea is close to the entity linking or hyperlinks for multi-hop QA, it is new under this open QA setting. Overall, I would like this paper accepted.

Pros:
1. Release a new task and dataset for answering open-domain questions with text and table. The dataset is carefully annotated by two steps by making use of decontextualization method.
2. Set up very strong baselines, and propose fused retrieval which is important to achieve strong performance for this task.
3. This paper is well-written and easy to read.

Cons:
1. I would like to see more analysis on the dataset. For example, what's the distribution of different question types? What kinds of questions are hard to solve? Can the proposed model solve reasoning problems from the dataset?
2. Missing baselines of using text only and table only to answer the question. It's unclear whether we do need both text and table. 

###update###
I have read the other reviews and the author feedbacks. I would like to keep my rating.
review_title: Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: -----------------------------
Summary
-----------------------------

In this paper, the authors introduce the task of open domain QA using table and text. Unlike recent tasks on table and text retrieval, where the table associated with the text is known, in this task both tables and text need to be retrieved from a large collection. Similar to the recent HybridQA task, questions may require fusing information both tables and text to answer questions. The dataset has been constructed by reusing the Hybrid QA dataset as well as crawling additional tables and passages from Wikipedia. Since the questions in the dataset may be non-contextual (for instance, not explicitly mentioning the name of a country when the data is only for that country), the authors used crowd-sourced workers to insert words or phrases from the section title, caption etc and make the question self-contained. However, this can cause the questions to become unnatural and excessively long  - the authors manually select the 25% worst questions and get them refined by a second set of annotation tasks. Additionally, to include new samples beyond Hybrid QA, they collect 2200 new QA pairs using the additional tables crawled and include them as part of the dev and test set. The exact answer in the table is annotated using distant supervision and has an extraction accuracy of 85%.  The overall dataset contains 41.5K questions in train and approximately 2K questions each in the dev and test set.

The authors also present a method for for this task - It consists of a "fusion reader" which that aligns a passage and table segments ( Table row+ metadata= Table segment). These are then considered as one unit for retrieval. To identify the passage-segment alignments, it takes a table segment and generates augmented queries to fetch candidate passages.  The augmented queries are generated token-by-token and are used to transform the queries  into the passage titles. This augmented query generator is based on GPT2 and has been fine tuned using the training data. BM25 retrieval is used to fetch passages given the generated queries. Given a table segment and its related (retrieved passages), BERT is used to jointly encode the two sequentially and the CLS token is used to create a representation of the "fused" block.  

The retriever uses a question, a fused block and computes the dot product between the BERT CLS embeddings to score blocks (as described in Section 2 but the results section suggests its ORCA -- See Q3). Inverse Cloze pre-training is done on the fused blocks to improve the retriever. Answer generation is done by using the pre-trained long-range sparse attention transformer (ETC) to encode questions and the retrieved blocks to return the answer span. 

Experiments have been presented using the HYBRIDER baseline from the HybridQA task as well as an iterative retriever reasoner baseline in place of the fused retriever. Experiments indicate that the use of fusion retrieval as well as cross block attention (using ETC) individually and jointly help improve performance over baselines. It is also interesting to note that the retrieval of fused blocks works better with BM25 (Sparse) as opposed to using BERT. 



---------------------------------------------------------
Strengths and Weaknesses 
--------------------------------------------------------

Strengths
+  New Task and Model for a challenging problem
+ Reuses existing datasets as well as baselines

Weaknesses

- Paper is a little hard to follow and required multiple readings.  For instance,  the use and introduction of the words such as "block", "segment", "cell", "row" in writing make the paper hard to follow -- perhaps it would be good to define them in the beginning with examples so that it is easy to reference when reading 
- Limited qualitative insights and almost no error analysis -- It would have been useful to include an error analysis along with some qualitative insights though the motivation for fused reasoning over iterative reasoning are clear. What could someone do next on this task? The task is far from solved as the results in Table 1 indicate and the information in Figure 9 in the appendix is not adequate.
- Both the iterative reader as well as fused retriever would be sensitive to the value of "top-k" - No study or details have been provided. This is important -- for example, in case of the iterative reader, 50% of the results included are from passages while 50% of the results are from the tables (as per the information in the appendix). Some recall statistics could be helpful (Also see Q2)

---------------------
Questions
----------------------

Q1 What is the performance of models broken down by question types -- the dataset analysis in the appendix suggests two-hop questions dominate the dataset? 


Q2 The performance of the HYBRIDER (a model designed for multi-hop inference on text-table data) appears to be lower than the Iterative retriever and single block reader? Could that not be because the BM25 R@1 is likely to be very low -- can the model even do multi-hop reasoning across passages in this case? Perhaps a method that aggregated scores after running HYBRIDER on the top-K retrieved blocks, to return a final ranking may have been a better baseline? 


Q3 Unless I have missed it, I'm unclear about the sparse/dense retrieval In Table 1 -- is that BM25 applied on the blocks B_F in raw text i.e table segment + passage? If the dense retrieval is based ORCA what is the text in the second paragraph for? Is that not used for retrieval along with the "retrieval function" in Section 2?


Overall this is a good interesting paper but it has some important questions that were not adequately answered. 


------------------------------------------------
Updated after author response
------------------------------------------------
I have updated my overall rating in view of the author response.
review_title: Good interesting paper but it has some important questions that were not adequately answered

Meta-review: This paper presents a new dataset for open domain QA where the evidence required for answering a question is gathered from both structured data as well as unstructured data. The authors first show that a standard iterative retriever with a BERT based reader performs poorly on this task. They then propose fused retrieval (grouping relevant tabular and textual elements) followed by a cross-block reader which improves performance.  

R4 has raised strong objections about the artificiality of the dataset. I agree with that and it is unfortunate that the authors did not adequately address the reviewer's concern but instead digressed a bit. As suggested by R4, the authors should tone down their claims about the nature of the dataset. The authors should also simplify the presentation of the dataset as suggested by R2 and not make it unnecessarily complex for the reader. 

However, overall, based on reviewer feedback, the authors have made significant changes to the paper. In particular they have added more baselines, ablation studies and error analysis which makes the paper much more informative. 

I am okay with this paper getting accepted with the assumption that the authors will make the changes suggested above. 


Decision: Accept (Poster)