Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces DocHopper, a new model for complex question answering over long documents (e.g., multi-hop QA over multiple paragraphs, conversational QA, reasoning over scientific documents). DocHopper is based on ETC (Ainslie et al., 2020) and DocHopper extends the existing hierarchical attentions from ETC with a new approach to update query representations in latent space. Their model does not jointly encode a question and context and does not require re-encoding of queries as in prior work, which leads to their effectiveness at inference time. They evaluate DocHopper on four different datasets: ShARC, QASPER, HotpotQA, and HybridQA. The proposed method achieves strong performance on those datasets, reducing the computational cost at the inference time. 
main_review: ### Strengths
I think overall the proposed approache is technically sound well motivated by the challenges in complex reasoning over a long context. They also test their approaches on diverse datasets including conversational QA and show the applicability to diverse QA data. 
Their proposed method largely outperforms its base model, ETC, showing the effectiveness of newly introduced modules. Also, removing the necessity of jointly encoding query-document and reusing the encoded document embedding is useful to improve the run time efficiency.  

### Weaknesses
I have two main concerns listed below. 
1. **Limited novelties and technical contributions**: although the method seems to be technically sound, the main component is based on ETC, and from my understanding, their technical contribution mainly lies in iterative attention to update the query representations in embedding space. Updating query representations in embedding spaces instead of actually updating query text for multi-step retrieval&reasoning have been studied in prior work such as Feldman and El-Yaniv (2019) or Das et al. (2019), but those papers are not cited, and I would like to see comparison or discussions on the differences between those studies.  

2. **Comparisons with the prior work**: Although they claim the proposed approach achieves state-of-the-art results on three of the benchmarks, the experimental settings are often slightly different from the original settings (e.g., only evaluate on the subset of the questions, use more passages than the default settings such as HotpotQA-Long). Claiming state-of-the-art results on those variant settings may not be fair, especially when the baselines to be compared are not specifically designed for the new setting. 

In addition, I sometimes have difficulties following the paper due to some missing details in the method section. For example, I think there should be more details on how "attention is supervised" in the loss function paragraph in Section 3.4, instead of talking about them in the implementation details. On the other hand, how to embed queries for each task is discussed in detail in Section 3.2 before talking about DocHopper's technical contribution (Section 3.4. Iterative attention). This makes it hard for me to understand what is the key technical contribution of DocHopper, and which parts are generally appreciable to different tasks, and which parts should be customized for each task. 

### Missing reference 
- Feldman and El-Yaniv. 2020. Multi-Hop Paragraph Retrieval for Open-Domain Question Answering. In Proc. ACL. 
- Das et al. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In Proc. ICLR. 
summary_of_the_review: In summary, I think the proposed approach is technically sound and a nice extension of ETC, but I don't agree that it provides significant technical contributions. Experimental results are strong, but as the experimental settings are different from the original setting, I do not think the paper should claim DOCHOPPER achieves state-of-the-art results on three datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an iterative approach for multi-hop question answering. At high-level the proposed model breaks a question into multiple sub-questions and then adds information relevant to each sub-question to the query vector for the next step retrieval. At each iteration an ETC encoder is used to encode the document and a sub-question; the vector corresponding to sub-question is then updated iteratively by contextualizing it over sentences and paragraphs in the document and is used to extract the final answer using a subsequent BERT reader. Evaluation results show improvements on 3 of 4 datasets. 
main_review: The proposed approach is interesting and more efficient compared with cross encoding approaches at inference time. 
Most prior results in the literature suggest that separate encoding approaches work inferior to cross encoders, thus results in this paper are encouraging. Despite encouraging results, there are some shortcomings in the execution.

- One critical result from last year's research was that Transformers can capture multihop relationships in long documents without the need for directly modeling multi-hop behavior and hierarchical structures (see BigBird, ETC, Longformer papers). This paper somewhat goes against that direction and suggests that explicitly encoding multi-hop information can improve results. However, it remains difficult to fully accept this claim as some of the key comparisons are not fully controlled. More details below.

- As mentioned in the paper, the ETC baseline is hampered by the 4096 context size limit. For the baseline, the authors encode the context paragraph by paragraph to sidestep the issue. However, this makes this baseline significantly weaker. Why not just evaluate the proposed model on the original HotpotQA data so that results are directly comparable? In addition Longformer can handle sequences of up to 16K tokens. I would have liked to see a Longformer based encoder as retriever + a BERT based reader as another more fair baseline on all the datasets.

- While motivation for longer documents is important, most of the datasets used are not the original versions. E.g., Hotpot and ShARC are modified to make them longer and for Qasper only a subset of extractive answers are used. This makes direct comparisons with SOTA not possible.

- I would have liked to see a comparison of number of parameters of the proposed model compared with baselines. For example, the LED baseline for QASPER is a single stage model where as the proposed approach uses two models, one for extracting evidence + a BERT-based reader for extracting the final answer. In this regard, the comparison is not entirely fair.

- The exact method of constructing the paragraph embeddings by a weighted average of sentences based on query relevance is a bit under motivated. It would have been nice to see an ablation on importance of this design (e.g., compared with simple or learned weighted average pooling).

- One of the stated benefits of the proposed approach is that it is more efficient by reducing the need for cross encoding the query and context at inference time. However, I would have liked to see some discussion about the added storage cost compared with the cross encoding methods. Pre-computing and storing vectors of all sentences and paragraphs in a document can be incur significant storage costs. 

- It would have been nice to see some error analysis, case study, or some discussion looking at what types of information the proposed model is able to capture compared with the baselines.

Questions for authors:
- What is finally used as text of q_null? 
- Which ETC model size is used? Which model size is used for the BERT-based reader?

Terminology:
- I'm not sure "hierarchical attention" is standard terminology for the model architecture of ETC. It might be confused by multi-layer models that are designed so that lower layers encode smaller pieces of information (e.g, words) and subsequent layers ingest these fine-grained representations and produce coarser level representations (Yang et al 2016). I would either use another terminology or add an explicit definition to make it clear that by "hierarchical" you mean that the model provides representation of individual blocks of text within the full sequence, as ETC does not have an explicit hierarchical structure. 
summary_of_the_review: Overall, the paper proposes an interesting approach to answer questions over long documents efficiently. However, there are some issues with the execution, presented results, and analyses making it difficult to fully accept the main claims in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple attention-based model for conversational and multi-hop QA tasks. The model use BERT-like pre-trained LM ETC separately encodes questions and paragraph (i.e., a collection of sentences). Besides the encodings on sentence-level, the final context encodings also contain extra paragraph embeddings, which are a weighted sum of sentences’ encodings using a simple dot product attention. For the QA interaction, the models use a hard attention mechanism to select an entry representing either a sentence or a paragraph. 
The experiments on two extractive QA datasets HYBRIDQA and QASPER show the model performs worse than the MATE model on HYBRIDQA but marginally better than other baselines on QASPER. On multi-hop QA and conversational QA tasks, the model performs marginally better than baselines on *expanded dataset*, but authors do not provide results on original datasets.

main_review: 1. If it is claimed as "hierarchical", I would expect something like a "general and narrow down" strategy. Unfortunately, what the model actually does is just simply flatten every paragraph (see the definition of content embedding C and Eqs. 4, 5, and 6). The only "hierarchical" stuff is a simple add-on embedding that is a simple weighted sum of all sentences. Also, the "iterative" comes from the problem definition itself while having nothing to do with the method. In this sense, the proposed attention model is neither "iterative" nor "hierarchical". I do think it makes novel technical contributions.

2. The process when attending to a paragraph vector (i.e., unpack the paragraph by reusing dot-product attention, then pack it again with the extra learnable attention) seems tricky and cumbersome but the intuition behind is not clearly illustrated. 

3. The experiments on the expanded datasets are unfair to other baselines. As their deeply contextual QA interaction either falls in token-level or is subject to the ability of the backbones (e.g., BERT’s 512 length limit). I would recommend authors provide results on the original dataset.

4. The performance of the proposed model lags far behind MATE on HybridQA dataset. It is curious that how MATE would be performed if applied to QASPER dataset.

Minor comment:

Strictly, the softmax term in Eq.3 (and argmax term q_t c_m) should be s^i_j q_t^T, as q_t and s^i_j are both 1*d matrices according to the notation at the bottom of page 3.



summary_of_the_review: Trivial method, contributing little scientific knowledge.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a novel MRC model (DocHopper) for multi-hop QA over long structured documents. In multi-hop QA, the evidence necessary to answer a user's question is spread across different parts of the long document. Previous approaches find the evidence by iteratively updating the user's query. The problems in these previous approaches are 1) computational efficiency and 2) ineffective modeling strategy for figuring out the relations of the evidence. DocHopper resolves these problems with a hierarchical attention mechanism. Hierarchical attention mechanism provides two types of embedding vectors for a single paragraph: 1) local sentence vectors and 2) global context vector of the paragraph. DocHopper computes the similarities between the query vectors and these sentence/paragraph vectors and selects the proper evidence. Since the sentence/paragraph vectors can be pre-computed, the only inference time required for this method is the time for question embedding, and this brings drastic improvement in the computational efficiency. This paper uses four types of datasets for evaluation: 1) conversational QA (ShARC), 2) TableQA (HybridQA), 3) QA on academic paper (QASPER), and 4) multi-hop factual QA (HotpotQA), and shows the QA performance and computational efficiency of their model.
main_review: Strengths:

1. This paper proposes a machine reading comprehension model, which is computationally efficient and also can handle long-document in multiple granularities.
2. The computational efficiency of recent QA models has been a problem in building a real-time QA system. This paper tries to solve the computational efficiency of recent MRC models. Their idea is somewhat similar to the idea used in phrase-indexed QA in open-domain QA. Thus, the topic of this paper is well-aligned with the recent approaches in the QA community.

Weakness:

1. The authors converted some QA datasets (ShARC and HybridQA) to long-document QA forms and evaluated DocHopper on these datasets. However, the authors did not show evidence that their modification on ShARC and HybridQA datasets is valid.
2. The runtime efficiency of DocHopper is one of the main contributions that the authors claim in their paper. However, this paper does not provide enough information on measuring the runtime. For example, does the runtimes of the baselines and DocHopper measured in a fair environment? Please provide more detailed settings used in measuring runtime efficiency.
3. IRRR is a multi-hop QA model in the full-wiki setting, and HotpotQA-long is in the distractor setting. Therefore, comparing IRRR and DocHopper is not a fair comparison. Although the authors provide the performance of HGN (in distractor setting), the performance of DocHopper underperforms. However, this is not a weak point since one of the research problems of this paper is alleviating the runtime problem while maintaining the MRC performance. However, since it is not clear how the authors conduct experiments to compute the runtime of each model, the HotpotQA-long results become insufficient evidence.
summary_of_the_review: The research problem of this paper is the inefficient computation time of existing QA models. This paper proposes a novel machine reading comprehension model for a long document. This paper provides evaluation results of their model on four types of datasets, and some experimental results show that DocHopper outperforms other baselines models. However, some evaluations results (results on ShARC and HybridQA) are insufficient to show the validity of their idea since these datasets are not designed for long-document QA. Also, it is unclear which experimental setting the authors used to compute the runtime of their model and baselines.

Since my concerns are from the unclear description of this paper, I'm willing to increase the overall recommendation score if the authors provide a more clear description.
review_title: null

Meta-review: Strength
* The paper is relatively clearly written.
* The proposed method appears to be sound.

Weakness
* The novelty of the work seems to be limited.  
* The experiment part needs significant improvements.  The comparison with existing methods may not be fair.  Evaluation of efficiency should be given. There are also detailed investigations that need to be conducted, as indicated by the reviewers.
* There are technical issues that need to be addressed.
Decision: Reject