Iterative Hierarchical Attention for Answering Complex Questions over Long Documents
Authors: Haitian Sun
William W. Cohen
Ruslan Salakhutdinov
Venue: ICLR 2022 Submitted (2022)
Keywords: Question Answering
Natural Language Processing
Attention Methods

Abstract: We propose a new model, DocHopper, that iteratively attends to different parts of long, hierarchically structured documents to answer complex questions. Similar to multi-hop question-answering (QA) systems, at each step, DocHopper uses a query q to attend to information from a document, combines this “retrieved” information with q to produce the next query. However, in contrast to most previous multi-hop QA systems,  DocHopper is able to “retrieve” either short passages or long sections of the document, thus emulating a multi-step process of “navigating” through a long document to answer a question. To enable this novel behavior, DocHopper does not combine document information with q by concatenating text to the text of q, but by combining a compact neural representation of q with a compact neural representation of a hierarchical part of the document -- potentially a large part.  We experiment with DocHopper on four different QA tasks that require reading long and complex documents to answer multi-hop questions, and show that DocHopper outperforms all baseline models and achieves state-of-the-art results on all datasets. Additionally, DocHopper is efficient at inference time, being 3 - 10 times faster than the baselines.

Under review as a conference paper at ICLR 2022
ITERATIVE HIERARCHICAL ATTENTION FOR ANSWER- ING COMPLEX QUESTIONS OVER LONG DOCUMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new model, DocHOPPER, that iteratively attends different parts of long, heirarchically structured documents to answer complex questions. Similar to multi-hop question-answering (QA) systems, at each step, DoCHoPPER uses a query q to attend to information from a document, combines this "retrieved" in- formation with q to produce the next query. However, in contrast to most previous multi-hop QA systems, DoCHoPPER is able to "retrieve" either short passages or long sections of the document, thus emulating a multi-step process of "navigat- ing" through long document to answer question. To enable this novel behavior, DocHoppeR does not combine document information with q by concatenating text to the text of q. but by combining compact neural representation of with a compact neural representation of a hierarchical part of the document, which can potentially be quite large. We experiment with DocHOPPER on four different QA tasks that require reading long and complex documents to answer multi-hop questions, and show that DocHoPPER achieves state-of the results on three of the datasets. Additionally, DocHoPPER is efficient at inference time, being 10 times faster than the baselines
INTRODUCTION
In this work we focus on the problem of answering complex questions over long structured docu- ments. A long document typically contains coherent information certain topic, and the contents are grouped by sections or other structures. To answer complex, multi-hop questions over long doc- uments often requires navigating through different parts of the documents to find different pieces of information relevant to question. This navigation, in turn, requires understanding high-level infor- mation about the structure of the document. For example, to answer the question "What modules in DocHoppER will be finetuned in all the experiments?", one might first turn to the section title "Model" to identify the different modules in DoCHOPPER, and then read the "Experiments" with these modules in mind, potentially further attending to specific subsections (such as the ones titled "Implementation Details"). As in academic papers (Dasigi et al., 202T). similar tasks are common for questions about government policies (Saeidi et al., 2018)) or legal documents. This type of QA tests not only the ability to understand short passages of text, but also the ability to understand the goals of the question and the structure of documents in domain.
common approach of solving multi-hop questions is to iteratively find evidence for one hop, and then use that evidence to update the query used in the next hop of the QA process. The update can be performed by either explicitly predicting the intermediate answers [Talmor & Berant 2018 Sun let al.
2019) or directly appending previous evidences to the questions (Zhao et al. 202 Qi ct al.
Li et al. 2020 Xiong et al. 2021). While appending retrieved evidence to a query works well on many factual QA tasks, where it is possible to answer questions with evidences that are short pieces of text, this approach is expensive if one wishes to retrieve larger pieces of text as evidences (e.g., the "experiments" section of paper). Another disadvantage is that appending together many small fragments of text intuitively fails to capture the relationships between them, and the structure of the document from which they were extracted.
2021
To capture high-level structural information in a document as well as detailed information from short passages, hierarchical attention mechanisms have been proposed, which learn neural representations
We will open-source our code and data.

---

Under review as a conference paper at ICLR 2022
at different levels that are then mixed to make final predictions for simple questions (Wang al. 2018 Chang et al. 2019). Hierarchical attention has also been adopted in pretrained language mod- els, c.g. ETC (Ainslie et al. 2020), which introduced global-local attention mechanism where embeddings of special global tokens are used to encode high-level information. Our DOCHoPPER system incorporates ETC as document encoder. However, while ETC has previously performed well on multi-hop QA tasks like HotporQA and WikiHop (Yang et al. 2018; Welbl et al.) 2018)) which require combining information from small number of short passages, it has not been pre- viously evaluated on tasks of the sort considered here. Our experiments show that DoCHoPPER outperforms past approaches to using ETC for multi-hop questions.
DocHoPPER extends the existing bierarchical attention methods with a novel approach to updating queries in a multi-hop setting. DoCHoPPER iteratively attends to different parts of the document, either at fine-grained level or at higher level. This process can be viewed as either retrieving short passage, or navigating to a part of a document. In each iteration, the query vector is updated in the embedding space rather than by re-encoding sequence of concatenated tokens. This updating step is end-to-end differentiable and efficient. In our experiments, we also show it is effective on four different benchmarks involving complex queries over long structured documents.
In particular, we evaluate DoCHoPPER on four different tasks: conversational QA for discourse entailment reasoning, using the ShARC (Seiiii et al. 2018) benchmark factual QA with table and text, using HybridQA (Chen et al., 2020); information seeking QA on academic papers. using QASPER (Dasigi et al.] 2021); and multi-hop factual QA, using HotpotQA (Yang al., 2018)- Since the outputs of the four tasks are different, additional layers or simple downstream models are appended to DocHoPPER to get the final answers. DocHopPER achieves state-of-the ooftthe-art art results on three of datasets, outperforming the baseline models by 3%-5%. Additionally, DocHoPPER runs 3- 10 faster than the baseline models, because the neural representations of documents is computed, which significantly reduces computation cost at inference time.
RELATED WORKS
Graph-based models have been widely used for answering multi-hop questions in factual QA (Min) let al. 2020: Sun et al. 2018 2019 Qiu et al. 2019 Fang et al. 2019). However, most of the graph- based models are grounded to entities, i.e., evidences (from knowledge bases or text corpus) are connected by entities in the graph. The graph construction step also heavily relies on many discrete features such as hyperlinks or entities predicted with external entity linkers. It's not clear how to apply these models to more general tasks if context is not entity-centric, such as questions about academic papers or government documents. Similar problems also exist in memory-augmente many factual QA tasks (Guu et al.) 2020)] Lewis et al. 2021 Verga et al., 2020; Dhingra et al. 2020 Sun et al. 202I).
language models that achieved the state-of-the-art
Alternatively, one can adopt the "retrieve and read" pipeline to answer multi-hop questions over long documents. Recent works proposed to extend the dense retrieval methods (Karpukhin et al.] (2000) to multi-hop questions (Zhao et al. 1 Qi al. 2021 Li et al. 2020). However, such models retrieve one small piece of evidence at a time, lacking the ability of navigating between different parts of the documents find relevant information both higher and lower levels of the document- structure hierarchy. Another disadvantage of these iterative models is that they are not end-to-end differentiable. Updating the questions for the next hop requires re-encoding the concatenated tokens from the questions and previously retrieved evidences. It also makes the model inefficient because re-encoding tokens with large Transformer models very expensive.
Besides question answering tasks, hicrarchical attention has been successfully used in tasks such as document classification (Yang et al.) 2016 Chang et al. (201)). summarization (Gidiotis & Tsoumakas] 2020; Xiao & Carenini] 2019 Zhang et al. 2019), sentiment analysis (Ruder et al.] 2016), text segmentation (Koshorek et al., 2018). etc. It is worth mentioning that ETC (Ainslie et al. 2020) was also used on a key-phrase extraction task on web pages using the structured DOM tree. However, none of these models can be easily adapted to answering complex questions over long documents.
We modify the original dataset and replace the oracle snippet with the entire web page as input. Please see section $4. for more details.
2

---

Under review as a conference paper at ICLR 2022
Updatto
Porogroph Emb
Broadcast Qusnr Emb
4
q1
Swntercw Query Errb
Guery lipaaate
Mixing
Attention Step
40 Cwc
Query Encoder
Context Embeddings
Iniial Lausofar Carl gel this bendi?? Edlizemnn11:
Fdlicesin ArR you allen? Yea!
Figure DocHoPPER Overview for structured document consisting of sentences and paragraphs. The query encoder first computes embeddings 9o..... 9., for a sequence of sub-questions. During the iterative attention process, DocHOOPPER attends to paragraph or sentence from the context embedding table that contains both paragraph embeddings and sentence embeddings. Information in the attended sentence para- graph will be mixed with the query vector compute query update. the query attends to paragraph, e.g. the first query 9, in the figure, the query vector will be broadcast to the associated sentences. If the query attends sentence, e-g. the second query 4₁- only the sentence sf will be used. DocHoPPER then updates the query vector q, for the next round of attention. The selected sentences will be used make final predictions.
MODEL
In this section, we first introduce how to compute neural representations for questions and context at different levels. Then, we present the iterative process that attends to information in the document and updates the query vector. Additional layers or simple downstream models required for different downstream tasks will be explained in the next section ($4)
3.1 BACKGROUND
Input long document usually contains multiple levels of hierarchy, e.g. sections, sub-sections, paragraphs, sentences, etc. For simplicity, we only consider two levels of hierarchy in this paper: paragraph-level and sentence-level. sentence the lowest granularity that can be retrieved, while paragraph is an abstraction of a collection of sentences, which can be used to represent sections or other levels in the hierarchy, depending on the application. Formally, let = {po..... P|a } E D be document in the corpus D that contains a sequence of paragraphs pi, and let paragraph P. = { sp.. contain a sequence of sentences si.. A sentence si will be encoded into fixed length vector st Rdd.
Pretrained ETC We use ETC (Ainslie et al. 2020) as our query and context encoder as it is pre- trained to produce both token-level and sentence-level embeddings. ETC is pretrained Language Model (MLM). Different from BERT (Devlin et al.] 2019). it employs an additional global-local attention mechanism. ETC assigns each sentence special global token that only attends to local tokens in the sentence, and its embedding is trained to summarize the information of local tokens in the sentence. A global token also attends the global tokens of other sentences in the input. ETC additionally adopts Contrastive Predictive Coding (CPC) (Oord et al.) 2018)) to train the embedding of global tokens to make them aware of other sentences the context. We use embeddings of global tokens in ETC as encodings for sentences.
Mask
Specifically, ETC takes a list of sentences p; - s(o.)) } as input, and returns list of vectors ....... s/p.1. where each st E Rd represents the embedding of sentence s).
ETC( ... ...13)) Rle.lxd
3

---

Under review as a conference paper at ICLR 2022
3.2 QUERY EMBEDDINGS
We discuss the strategy of computing query vectors for two types of input, conversational QA and multi-hop QA. We use ETC (Ainslie et al.) (0200 as our query encoder for the reasons discussed above. Other pretrained language models for similar purposes should work as well.
Conversational QA A conversational question contains multiple rounds of interaction between the machine and human. As the conversation goes on, the topic of the conversation will shift, but questions asked in the next turn are related the history of the conversation. We consider this multi-turn process as a multi-hop QA task. In our specific task, the conversation starts with an initial question go- The model will answer the question if there's enough information. Otherwise, the model will ask a followup question fi and expect the user's 's answer a1. The model will keep asking followup questions for a few more iterations. The task is to predict the answer if there is enough information after reading the full conversation history, mark the question not answerable.
We denote a, as the answer to the followup question f. received from the users at the i'th iteration. We consider the full conversation history the question P, {go. (fi,a1)..... (frs an)) with the initial question denoted Qo and the i-th followup question-answe a, to the followup question fü is concatenated to the end of the questions and represented sentence. We call Py a question paragraph as it contains a sequence of question question-answer answer pairs. We use ETC compute the query embeddings q,- one for each sentence in the query paragraph Py- The query embeddings q, will be used to perform iterative attention over the document.
pair denoted (fi, a:). The answer
single
4o
In = ETC, (fi,01).... (n)))
(I)
Multi-hop QA A multi-hop question, e.g. "Which gulf is north of the Somalian city with 550,000 residents" requires the model to first find the answer of the first "hop" of the question, and then per- form second round to find the final answer. Assume for now there from conversational QA, questions in multi-hop QA do not have a clear split between the two hops of questions, making it impossible to explicitly split a question into two sub-questions. Instead, we add dummy question quuil to the question paragraph 9p = {qu. lmmi }. The global-to-loces attention mask of ETČ modified to allow the global token of the dummy question to attend to tokens in the question 4o- With this modification, the query embeddings 4, and 91 for go and Imall can to attend any part of the question, but each can also attend to different parts of the question. One could append additional dummy questions to the question paragraph Py if the number of hops is larger. Thus we let
exactly two hops. Different
40: 91 = -ETC, (100 )mml))
(2)
Note that we assume the true number of hops is known in multi-hop QA tasks, which is two or four for the experiments in this paper. However, one can train models to decide when to stop if questions have various numbers of hops. We leave this as topic for future work.
3.3 NEURAL REPRESENTATION OF PARAGRAPHS
We compute embeddings s for the sentence sÿ with the context encoder. We use ETC as our context encoder, but similar pretrained language models will also work. Recall that paragraph p. (sp....,.$pp contains a sequence of sentences sy. The sentence embeddings s) in the paragraph D; are simply computed by applying ETC on the paragraph input.
p.l = ETC,(( 2 sbp.)))
The paragraph embeddings are directly derived from sentence embeddings si and dependent on the queries qr. the embedding of the f'th hop of the question. The paragraph embedding is a weighted sum of sentence embeddings sy in the paragraph P., where a, is the attention weights of the query vector 4, to the sentence embedding sy. The paragraph embeddings P; are thus dependent on the query, but do not require jointly encoding tokens from queries and context, as in many BERT-style reading comprehension models. Computing paragraph embeddings with Eq 3 hence very efficient.
p; > aj S aj = softmax s²)
(3)
4

---

Under review as a conference paper at ICLR 2022
3.4 ITERATIVE ATTENTION
We put the sentence embeddings and paragraph embeddings of document d into a combined context embedding table, so the model has the flexibility decide which sentence or paragraph attend to. Different update rules will be applied according to whether sentences or paragraphs are attended to.
To construct the embedding table, we iterate through all paragraphs a document and apply the ETC encoder on each paragraph to compute the paragraph and sentences embeddings. Sen- tence and paragraph embeddings from all paragraphs are then merged together to form the com- bined embedding table. We denote the combined embedding table for document d as Ca (po.s....., . P1 So...... spp. . }. Let em be the embedding of the m 'tth entry from Ca: we emphasize that Cm can represent either a sentence or paragraph embedding.
Attention Step At each iteration, DoCHoPPER computes the inner product scores between the query vector q, and embeddings Cm in Ca. and returns the entry è with the largest score, which is usually referred as hard attention. (As will è is not directly used for computation, but is helpful in explaining the attention method).
argmax.... (9 Cm)
Mixing Step DoCHOPPER then mixes the embedding of the attended entry with the query vector q, to find the missing information. Since the combined embedding table Ca contains both sentence and paragraph embeddings, the selected entry è can represent either sentence or a paragraph. The two cases are separately considered. If è is a sentence, i.e. č = s'j. DoCHoPPER computes the mixed embeddings as
q. W [9:
(4)
where [q is the concatenation of two vectors and sj. Assume is a paragraph, i.e., è = DocHoppER first looks up sentences in pi, i.e. the list { p.1)). The following process is then used to compute the mixed embedding 9, (1) DocHoPPER computes the attention weights of the query vector to the embeddings of associated sentences (sh.... ,,p.)) that measures the relevance scores between the query vector and the sentences. This attention weight is the same as the weight aj in 3q that used to compute the paragraph embeddings. so we reuse aj in this equation. In the implementation, we also re-use the value of o; if it has been computed for the query-dependent paragraph embeddings. (2) The query vector q, is compared with every sentences in paragraph to extract missing information. 9, is broadcast to the sentence embedding s with the weight ay, i.c. the query vector more important if more relevant. The comparison is performed as a linear projection of the concatenated query and sentence embeddings, q. and sy.
k, = WT gris]]
(5)
Then (3) the concatenated vectors ky are summed with the weight 3y. where 8j is the attention weight of learned vector v to the concatenated vector kj. The learned vector coordinates the importance of sentences from the attended paragraph after comparing them with the query vector and decides what information pass the next step of retrieval.
Σ By kj: By = softmax(y"kk;
q
It is not hard to see that computing the mixed embedding in Eq. 6 for the case that a paragraph is retrieved essentially the same as Eq. 4f the retrieved paragraph pt only contains one sentence, i.e. 0j and B3 1 if [pal = 1; hence the same logic can be used regardless of whether è is sentence or paragraph.
Update Step The mixed embedding q, is then used to update the query vector 4 for the next step. Intuitively, q, is the residual from the previous step. Adding the residual embedding encourages the model to attend to information that is not fully satisfied from previous steps.
9.+1 9.+1 q.
5

---

Under review as a conference paper at ICLR 2022
Loss Function Attention is supervised if (distantly) supervised labels are available in the dataset. q. Cm is the inner product score between the query vector q, and a context embedding Cm- Il... an indicator function that equals to 1 iff the label of Cm is positive.
4 = cross_entrrpy(ssftmxx(( Sm,)Hem..
The loss function is computed at the final step, and possibly at intermediate steps if labels are avail- able. Supervision labels are sometimes distantly constructed. For example, in the extractive QA task, a positive candidate is the sentence paragraph that contains the answer span (see $
3.5 RUNTIME EFFICIENCY
DoCHoPPER is very efficient at runtime thanks to the precomputed context embeddings ($3.3) at in- ference time. Different from previous reading comprehension models that jointly encode questions and context (Beltagy et al. 2020: Ainslie et al.] 2020), DoCHOPPER encode question embeddings and context embeddings independently. tend to the precomputed context embeddings, significantly reducing the computation cost compared cross-attention models that jointly encode questions and context.
inference time, DocHoPPER lets questions directly
4 EXPERIMENTS
We evaluate DOCHOPPER four different datasets: ShARC (Saeidi al.
2018 HybridQA (Chen
et al. 2020). QASPER (Dasigi et al. 2021). and HotpotQA (Yang et al. 2018
3 Since the down-
stream tasks of the four datasets Transformer- baser extractive QA model for HybridQA, QASPER and HotpotQA to extract the final answers. We will discuss the setup for each case separately.
different, we apply an additional layer for ShARC, and
4.1 EXTRACTIVE QA: HYBRIDQA AND QASPER
Dataset HybridQA is a dataset that requires jointly using information from tables and hyperlinked text from cells to find the answers. Please see the Appendix for more information. This dataset requires the model to first locate the correct row in the table, and then find the answer from cells (or their hyperlinked text). To apply DoCHoPPER on the HybridQA dataset, we first convert a table with hyperlinked text into long document. Each row in the table is considered paragraph by concatenating the column header, cell text, and hyperlinked text if any. The average length of the documents is 9345.5 tokens.
Dataset QASPER Dasigi et al., 2021 is a QA dataset constructed from NLP papers. The dataset contains mixture of extractive, abstractive, and yes/no questions. We experiment with the subset of extractive questions (51.8% the datasets) in this paper. Some questions in the dataset are answerable with single-hop of retrieval. However, described in the original paper, 55.5% of the questions have multi-paragran evidence, and thus aggregating multiple pieces of information should improve the accuracy. Answers in the QASPER dataset have average of 14.4 tokens. We treat each subsection as paragraph and prepend the section title and subsection title to the beginning of the subsection.
Implementation Details Question embeddings 4o 41 are initialized from Eq. 2 and updated as Eq. 7 For the two datasets, we perform 2-hop attention: the first hop is trained to attend paragraph, and the second hop to attend to a sentence. Note that we do not require that sentences attended in the second hop must come from the previously attended paragraphs. The attention scores of paragraphs and sentences are linearly combined to find the best sentence that contains the answer, which will then be read by a BERT-based reader to extract the final answer. The final attention score of sentence sf p; is
score
9. +
4 q.P. A2 sparse P:)
We modify the original ShARC and HotpotQA datasets to evaluate them in the long-document setting. Please the dataset descriptions for details.

---

Under review as a conference paper at ICLR 2022
HybridQA
QASPER (Extractive)
Dev
Test
Dev
Test
runtime
Retrieval ETC
43.5
34.1 40.3
8.3 18.7
9.6 19.1
8.3/s
Sequential (ETC)
39.4/ 44.8
47.0/43.0
11.2/24. -/
12.4/27.0 -1-
0.6 1/s
Hybrider
44.0/ -/-
13.8/50.0 -/-
5.1/s
LED
1/26.1
-/31.0
0.5/s
DoCHOPPER
45.7
53.3
14.0/
19.5 36.4
74,6/s
(w/o sparse)
54.4/51.2
-/-
(14.0/29.6)
(w/o query update)
44.2/50.9
12.2
(sentence-only)
36.7 (5.7/43.7
11.4/27.2
(single-hop)
27.8/34.1
11.8/77.
(w/ cell)
53.1/614
-/-
MATE
63.4/71.0
62.8 / 70.2
Table 1: EM/FI performance on HybridQA and QASPER. Runtime is measured by reruning their open- sourced codes (failed rerun MATE). Numbers for QASPER are reported on the subset of extractive questions. See Results and Analysis for discussion on MATE's performance.
where qf and q P; are the attention scores at sentence and paragraph levels, and sparse Ps) is the similarity score using sparse features A and X2 are hyper-parameters tuned on dev data. We set A 1.5 and X2 3.0 for HybridQA, and A 0.5 and X 0.0 for QASPER.
Baselines We compare DoCHOPPER with the previous state-of-the-art models, MATE (Eisenschlos) al, ) and LED (Dasigi et al. 202T), and several other competitive baselines to show the efficacy of DoCHoPPER. MATE (Eisenschlos et al. (021) is pretrained Transformer model with sparse attention between cells that specifically designed for tabular data. HYBRIDER (Chen et al. 2020) is pipeline system that (1) links cells, (2) reranks linked cells, (3) hops from one cell to another, and (4) reads text to find answers. All four stages are trained separately. LED an encoder- decoder model that builds on Longformer (Beltagy et al. 2020). We also experiment with directly reader ETC (Ainslic et al. 2020): though it can't fit the entire document into its input, it is still one of the best models for reading long sequences (up to 4096 tokens). To handle longer documents, we adopt the sequential reading strategy: the model reads the document paragraph by paragraph, and picks the most confident prediction the answer. We also report the numbers of a "retrieve and read" pipeline with BM25 retriever and a finetuned ETC reader. The numbers are shown in Table T Runtime measured as examples per second with a batch size of 1.
reading the documents with a Transformer-ban
Results and Analysis On QASPER, DoCHOPPER outperforms the previous state-of-the-att models by 3-5%, and runs more than 10 times faster. DocHOPPER performs worse than MATE (Eisensch- los et al. T on HybridQA. This is due three reasons. (1) MATE is specifically designed and pretrained to understand tabular data, while DoCHoPPER is applied to general documents. (2) To convert tables to DoCHoPPER' input, DocHoPPER serializes the tables by rows and thus loses in- formation from other table structures, e.g. cells and columns, that are commonly used to understand tabular data. In an ablated experiment with the assumption that sentences are grouped by cells, we let DocHoPpER return cells that contain the selected sentences and pass the cells (instead of single sentences) to the underlying extractive QA model (labeled w/ cell). This improves the performance by 5.4 points. (3) MATE restricts the length of text in cells 5 sentences and enforces it by trieving top-k sentences from the hyperlinked text. It also restricts the total length of tables to 2048 tokens. DoCHoPPER does not put any restrictions the length of cell or the length of tables, and can thus be easily applied to more general tasks.
We additionally performed more ablated experiments with DoCHOPPER. The query update (see the row w/o query update) in Eq. iis also important, causing 3.5% and 1.8% difference in performance on both datasets. We also ablated the model by using one step of attention select the most relevant sentence from the document (single-hop), and note again that performance drops noticeably. Adding one more step of attention, while only attending to sentences (sentence-only in the table), leads to
"Similar to the baseline model Chen et al. (020)), we use paragraph-leve sparse features to improve accu- racy. In HybridQA, sparse(go. p.) computes the length of longest common substrings in the question go and the paragraph pt. Sparse features only used the end of retrieval, not any intermediate steps.
7

---

Under review as a conference paper at ICLR 2022
HotpotQA-Lonn
ShARC-Long
Answer
Support
Easy
Striet
runtime
IRRR (direct)
53.6 / 64.8
14.7//3.22
ETC
61.1
11.2/s
IRRR (rerank)
62.1 /
5 81.3
Retrioval + DISCERN
65.2
50.7
42.0/s
Sequential (DISCERN)
63.7
54.2
8.8/s
DocHoPPER (direct)
59.4
45.1/73.8
DocHorPER (rerank)
66.5/79.7
61.4/
DoCHOPPER
72.3
60.2
164.3/s
(w/o query update)
72.4
59.9
ETC (distractor)
-181.7
19.4
(sentence only)
62.2
43.2
HGN (distractor)
8/3
-/ 89.2
(single-hop)
68.0
52.7
Table 2: EM/FI performance on answer and sup- porting evidence on HotpotQA (dev set) with full Wikipedia pages context. HGN [ang al. (2019) and ETC Fang al.] (2019) are evaluated on the dis- tractor setting.
Table Classification accuracy on ShARC-Long dataset. The Easy setting only checks the predicted la- bels, while the Strict setting additionally checks if all required evidences retrieved. DISCERN is with their open- open- sourced codes.
some improvement, but is still worse than attending at both paragraph and sentence levels. The performance of sentence selection (with ablated numbers) is presented in the Appendix.
4.2 HOTPOTQA-LONG
Dataset HotpotQA requires multi-hop reasoning to answer two types of questions, bridge and com- parison. The original dataset provides the first paragraphs of 10 Wikipedia pages as its context. To test the model's ability to extract information from longer context, 10 Wikipedia pages, and concatenate the 10 pages into a single document. This increases the average length of context from 897 to 9970 tokens. We call this variant of the dataset HotporQA-Lowe
use first 2048 tokens of the
Implementation Details The experiment setup for HotpotQAA-Lng is similar to HybridQA and QASPER. Attention for this dataset is supervised, using the information about which evidences support each answer provided in the dataset. Here, we refer to a Wikipedia page (with a maximum of 2048 tokens) as a paragraph. The iterative attention is performed similarly to $4.1 but repeated for 4 hops: paragraph, sentence, paragraph, and sentence. The first two hops are supervised by the first supporting evidence, and the last two hops are supervised by the second supporting evidence. The final score for each evidence is computed as in Eq. g with x1 = 0.7 for the first evidence and A1 = 0.0 for the second, and A2 for both. As suggested by Xiong et al. (2021), reranking can improve the overall performance, so we take the top 4 sentences for each step and rerank the 16 combinations
Baselines We compare DoCHoPPER with IRRR (Xiong et al. 2021) which is an multi-hop retrieval model that iteratively retrieves small piece of evidence and re-encodes query vectors at each step by concatenating tokens from the retrieved passages at the previous step We compare IRRR both with or without the reranking step. Since questions in HotpotQA require combining evidences from multiple paragraphs, it's not clear how to run the sequential reading bascline on HotpoiQA- Long, but as a reference, we show the performance of ETC and HGN in the distractor setting (with the original context, which is 10x smaller) in Table 2
Results DoCHORPER outperforms the baseline models by 4-6 points on the accuracy of both answer and supporting evidence. More surprisingly, the performance of DocHoPPER in the long document setting is only ~4 points lower than the state-of-the-ard on the distractor setting, even though the context is more than 10 times longer. Please see the Appendix for ablated experiments and analysis.
4.3 CONVERSATIONAL. QA: SHARC-LONG
Dataset ShARC Saeidi et al. (2018) a conversational QA dataset for discourse entailment reason- ing. Each example consists of a document that describes a government policy and conversation history about the document between the machine and the user. The conversation starts with an ini- tial questions asked by the user and follows by sequence of clarification questions and answers collected from the interaction between the machine and the user. The model takes the conversation
Pleasse refer to Xiong ct al. 2021 for more details on the reranking step.
'TRR is the best model (ranked #9 on leaderboard) on HotpotQA (fullwiki) with open-sourced codes.
8

---

Under review as a conference paper at ICLR 2022
history imput and predicts the answer to the initial question. An answer has one of the four labels: "Yes", "No", "Irrelevant", or "Inquire". "Irrelevant" means the question is not related to the pro- vided context and "Inquire" means there's not enough information to answer the question. Similar to HotporQA-Long we expand the original context to test the long document setting. The expanded context contains 737.1 tokens on average, 13.5 times longer than the original context. Please the Appendix for more details and examples.
Implementation Details The query embeddings 96..... "o....:9, .4, are initialized from Eq. where n is the number of followup questions in the conversation. The query vector q, at the step / will be updated with the residuals following the update rule in Eq/7 The iterative attention process is distantly supervised at intermediate steps. See the Appendix for information on distant supervision.
We add simple classification layer on DOCHOPPER to make the final prediction. As in- put to this layer, we reuse the concatenated vector kj in Eq. 5 Let k5) be the concate- nated vector of the sentence attended the l'th retrieval step, either sentence directly at- tended or associated from attended paragraph. Concatenated embeddings at all attention steps (k ....., kk()) ..... kg"") ....., ,k()) are linearly combined into vector k that will be used to make the final prediction, using weights 7/1 computed across the attended sentences from all steps 1. m E B holds the logits of the 4 classes that is used to compute the softmax cross entropy with the one-hor hot encoding of the positive class labels.
k = ΣΣ
7i softmax(u" (aam)ax("))))) (0)),, m W KER
Baselines We compare our model with ETC and a "retrieve and read" bascline. In using ETC, we concatenate the full context and the conversation into single input, and prepend global [CLS] token. The embedding of the global [CLS] token will be used to make the final prediction. For the "retrieve and read"' pipeline, we adopt the previous state-of-the-art model DISCERN the reader, and pair it with a learned retriever"] We also run a sequential reading baseline with DISCERN where documents chunked every 128 tokens with a stride of 32 tokens and then read with DISCERN, predicting the class with the highest probability among all chunked inputs 8 Runtime is measured as examples per second.
Results The evaluation is performed in two settings: Easy and Striet. The Easy setting only eval- uates the accuracy of classification. In the Strict setting, we additionally require that all evidences (provided in the original dataset) are retrieved. We report the (micro) accuracy as the evaluation metric. DoCHoPPER outperforms all baseline models by more than points in both easy and strict settings, while being more than 3 times faster than all baseline models. We performed the previ- ous ablated experiments and observed similar changes in performance. Please see the Appendix for more numbers evidence selection.
CONCLUSION
We consider on the problem of answering complex questions over long structured documents. Like multi-hop open QA tasks, this problem requires not only conventional "machine reading" abilities, but the ability to retrieve relevant information and refine queries based on retrieved information. Additionally, it requires the ability to navigate through a document, by understanding the relation- ship between sections of the document and parts of the question. In our framework, navigation is modeling similarly to retrieval in multi-hop models: the model attends to document section, and uses compact neural encoding of the section to update the query. Unlike most prior multi- hop QA models, however, queries are updated in embedding space, rather than by appending discrete representation of question text. This approach is end-to-end differentiable and very fast. Experiments also demonstrate that this use of hierarchical attention can significantly improve the performance on QA tasks: in fact, the DoCHOPPER model achieves the start-of-the-art results on four challenging QA datasets, outperforming the baseline models by 3-5,, while also being 3-10 times faster.
The retriever is ablated DocHoppeR model that attends the paragraph level only.
'The model is often extremely confident on the "Irrelevant" class because most chunked inputs are obviously irrelevant. We tuned a paprr-paraeeter& = 0.99 so the model predicts "Irrelevant" if the probabilities of all chunked inputs larger than

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Pretrained language models (LM) are effective in many tasks but are criticized due to their high training cost and environmental concerns. Recently more attention has been brought into devel- opment efficient models. The DOCHoPPER model proposed in this paper follows research in this direction. DoCHoPPER exploits the existing language models released by other research institutes. Without any additional pretraining, DocHoPPER achieved competitive state-of-the the-trr results in several challenging QA datasets. All experiments in this paper were conducted with a single GPU.
Besides being efficient in training, DoCHoPPER is also extremely efficient during inference. Ex- periments show that DocHoppeR runs 3-10 times faster compared to existing QA models and thus could be deployed using limited computation resources.
7 REPRODUCIBILITY STATEMENT
Experiments in this paper are conducted on publicly available datasets. Codes and processed data will be open-sourced upon the acceptance of this paper.
REFERENCES
Joshua Ainslic, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 268-260820 284, 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 05000 2020.
Ming-Wei Chang, Kristina Toutanova, Kenton Lee, and Jacob Devlin. Language model pre-training for hierarchical document representations. arXiv preprint arXiv: 1901.09128, 2019.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. Findings of EMNLP 2020, 2020.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information--sekking questions and answers anchored in research papers. arXiv preprint arXiv:2105.030/1, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pp. 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18633/v1/119-111 1423. URL https:
7/www. wwwwwb.www......... orw.orggaaaooo http://////.aoooo///o//.//// 1443
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W Cohen. Differentiable reasoning over virtual knowledge base. arXiv preprint arXiv:2002.10640. 2020.
Julian Martin Eisenschlos, Maharshi Gor, Thomas Müller, and William W. Cohen. Mate: Multi-view attention for table transformer efficiency, 2021.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. Hierarchical graph network for mop question answering. arXiv preprint 19Xiv:1971.0885 1.0363,,, 2019.
Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty. Steven CH Hoi, Caiming Xiong, Irwin King, and Michael R Lyu. Discern: Discourse-awar entailment reasoning network for conversational machine reading. arXiv preprint arXiv:2010.01833 2020.
10

---

Under review as a conference paper at ICLR 2022
Alexios Gidiotis and Grigorios Tsoumakas. A Alvid--and-conuue approach to the summarization of long documents. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 5029-3040, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wes Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint urXiv:2002.88099 2020.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. Text segmentation as supervised learning task. arXiv preprint arXiv: /803. 09337, 2018.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenttdd generation for knowledge-intensivee nlp tasks, 2021.
Shaobo Li, Xiaoguang Li, Lifeng Shang, Xin Jiang. Qun Liu, Chengjie Sun, Zhenzhou Ji, and Bingquan Liu. Hopretriever: Retrieve hops over wikipedia to answer complex questions, 2020.
Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. Knowledge guided text retrieval and reading for open domain question answering. 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037// 2018
Peng Qi, Haejun Lee, Oghenetegiri "TG" Sido, and Christopher D. Manning. Retrieve, read, rerank, then iterate: Answering open-domain questions of varying reasoning steps from text, 2021.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6140-6150, 2019.
Sebastian Ruder, Parsa Ghaffari, and John G Breslin. A hierarchical model of reviews for aspect- based sentiment analysis. arXiv preprint 1rXivv1609.02444 2016.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guil- laume Bouchard, and Sebastian Riedel. Interpretation of natural language rules in conversational machine reading. arXiv preprint IrXiv::809.0000 809.01494, 2018.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. Open domain question answering using early fusion of knowledge bases and text. arXiv preprint
2018.
Haitian Sun, Tania Bedrax-Wert and William W. Cohen. Pullnet: Open domain question answering with iterative retrieval knowledge bases and text, 2019.
Haitian Sun, Pat Verga, Bhuwan Dhingra, Ruslan Salakhutdinov, and William W Cohen. Reasoning over virtual knowledge bases with open predicate relations. arXiv preprint arXiv:2102.07043, 2021.
A. Talmor and J. Berant. The web knowledge- for answering complex questions. In North American Association for Computational Linguistics (NAACL), 2018.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.00849, 2020.
Wei Wang. Ming Yan, and Chen Wu. Multi-granulaan hicrarchical attention fusion networks for reading comprehension and question answering. arXiv preprint 1.. 2018.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. Transactions of the Association for Computational Lin- guistics, 6:287-302, 2018.
11

---

Under review as a conference paper at ICLR 2022
Wen Xiao and Giuseppe Carenini. Extractive summarization of long documents by combining global and local context. arXiv preprint arXiv:1909.0800: 2019.
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer. Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oğuz. Answering complex open- open-domain questions with multi-hop dense retrieval, 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1489, 2016.
Xingxing Zhang, Furu Wei, and Ming Zhou Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization, 2019.
Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daumé III au2. Multi-step reasoning over unstructured text with beam dense retrieval, 2021.
A APPENDIX
A.1 DATASET DETAILS
HybridQA (Chen et al. 2020 is dataset that requires jointly using information from tables and text hyperlinked from table cells to find the answers of multi-hop questions. A row in the table describes attributes of instance, for example, person an event. Attributes are organized by columns. For example, the table of Medalist of Sweden in 1932 g contains a row "[Medal:] Gold; [Name:] Rudolf Svensson; [Sport:] Wrestling (Greco-Roman): [Event:] Men's Heavyweight". Text in the square brackets are the headers of the table. The medal winner "Rudolf Svensson" and the event "Wrestling (Greco-Romanj"" are hyperlinked to the first paragraph of their Wikipedia pages. A question asks "What was the nickname of the gold medal winner in the men 's heavyweight greco- roman wrestling event of the 1932 Summer Olympics?" requires the model first locate the correct row in the table, and find the answer from other cells in the row or their hyperlinked text.
To apply our model on the HybridQA dataset, we first convert a table with hyperlinked text into a long document. Fach row in the table is considered paragraph by concatenating the column header, cell text, and hyperlinked text any. The column name and cell text are each treated as one sentence. Hyperlinked text is also split into sentences. In the example above, the row becomes "Medal. Gold. Name. Rudolf Svensson. Johan Rudolf Svensson (27 March 1899 4 December 1978) was Swedish wrestler. He competed ...", The average length of the documents is 9345.5.
QASPER (Dasiii et al.] 202I) is a QA dataset constructed from NLP papers. They hired graduate students to read the papers and ask questions. A different group of students are hired to answer the questions. For example, a question asks "What are the baseline models used in this paper?". The answers are ("BERT"" "RoBERTa"". The dataset contains mixture of extractive, abstractive, and yes/no questions. We focus on the subset of extractive questions (51.8% of the datasets) in this paper. Some questions in the dataset are answerable with a single-hop. However, as suggested in the original paper, 55.5% of the questions have multi-paragraph evidence, and thus aggregating multiple pieces of information should improve the accuracy. Answers in the QASPER dataset are longer, with an average of 14.4 tokens. We treat each subsection as paragraph and prepend the section title and subsection title to the beginning of the subsection.
ShARC Saeidi et al. 2018) is conversational QA dataset for discourse entailment reasoning. Questions in ShARC are about government policy crawled from government websites. Users engage with a machine to check if they qualify for some benefits. A question in the dataset starts with
https:: http:///eeewviiieeaaa.oorooo htt.:/pelia..og////i////eeeeeeee tteet__t________ [332__ummer___ _01mmpics
12

---

Under review as a conference paper at ICLR 2022
HybridQA
QASPER
(Extractive)
DoCHOPPER
56.5
39.1
(w/o sparse)
53.3
(39.1)
(w/o query update)
51.8
37.2
(sentence-only)
46.4
36.1
(single-hop)
34.2
36.8
Table 4: Hits@1 accuracy on selecting sentences that actually contains the answer (on dev set).
initial question, c.g. "Can I get standard deduction for my federal tax return?", with a user scenario, e.g. "/ lived in the US for years with student visa", and few followup questions and answers through the interaction between the machine and users, e.g. "Bot: Are you resident alien for tax purpose? User: No".. The model reviews the conversation and predicts one of the three labels: "Yes", "No", or "Irrelevant". If the model think there's not enough information to make the prediction, should predict fourth label "Inquire".
Besides the conversation, each example in the ShARC dataset provides a snippet that the conver- sation is originated from. A snippet is short paragraph that the conversation created from, e.g. "Certain taxpayers aren't 't entitled to the standard deduction: (1) A married individual filing as mar- ried... (2) An individual ... " Since the snippets are usually short, with an average of 54.7 tokens, previous models, e.g. DISCERN Gao et al. (2020), concatenate the snippet and the conversation, and jointly encode them with transformer-based models, e.g. BERT or RoBERTa. Here we con- sider instead more challeging long-document setting, in which the snippet is not known, and the model must also locate the snippet from the document. We crawl the web pages with the provided URL. The pages contain 737.1 tokens average, 13.5 times longer than the original snippets, and the longest page contains 3927 tokens. We name this new variant ShARC-Long.
A.2 IMPLEMENTATION DETAILS FOR SHARC-LONG
Changes to Context Representations Instead of computing the paragraph embeddings as a weighted sum of sentence embeddings, directly obtain the paragraph embeddings from ETC output for this dataset. Recall that a paragraph p; = {ss....,, sp.)) contains a sequence of sen-
tences st. We prepend dummy sentence Smull to the beginning of the paragraph, and again, modify the global-to-local attention mask to allow the global token of the dummy sentence to attend all tokens in the paragraph pi. Let p, E Ed be the embedding of paragraph p.. The embeddings for a paragraph and its contained sentences are:
P So....
s sp.. = ETC(fsoull, 86
(..1))
Distant Supervision The iterative attention process is distantly supervised with supervision at in- termediate steps. At each step, the model is trained to attend to both the correct paragraph and the correct sentences if they exists. Since the embedding table C. consists of both paragraph and sentence embeddings, we only need to compute the attention scores once at each step, but consider both the correct paragraph and the correct sentence positive. The positive paragraph is one of
the paragraphs from the crawled web page with the highest BLEU score 10 We notice that some web pages the provided URLs have been changed significantly, datasets may not exist any more, hence we discard the associated data if the highest BLEU scores of the paragraphs is less than 0.7. We follow the heuristics used by baseline models (Gao et al. 2020 to get positive sentence candidates by finding the sentence with the minimum edit distance.
the snippets provided in the
A.3 ADDITIONAL RESULTS
We report the performance of eventually selecting the correct evidences in Table and
Comments HotpotQA-Loong We also observe that ablated experiment evidence selection (w/o query update) is only 7.8 points lower than the full model. To understand the underlying
We drop the brevity penalty term in BLEU score.
13

---

Under review as a conference paper at ICLR 2022
HotporQA-LLon
IRRR
56.8
DocHOPPER
64.7
(w/o query update)
56.5
(sentence-only)
61.8
(single-hop)
37.4
Table 5: Accuracy of correctly predicting supporting facts for both hops HotpotQA--Lng (with- out reranking).
ShARC-Long
DocHoPPER
82.2
81.8 (sentence-only)
(w/o query update)
63.0 (single-hop)
72.4
Table 6: Accuracy of selecting all required evidences on ShARC-Long.
reason, we train the model to perform a one-step attention only for supporting facts of the second hop (for bridge questions). The accuracy is 71.7, only 3.6 points lower than the accuracy of the full multi-hop process. This is likely due to the high surface form overlap between the questions and their context.
14


Reviews:
[{"review_id": "UT3L6gN8qZi", "reviewer": "Reviewer_qv3L", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. If it is claimed as \"hierarchical\", I would expect something like a \"general and narrow down\" strategy. Unfortunately, what the model actually does is just simply flatten every paragraph (see the definition of content embedding C and Eqs. 4, 5, and 6). The only \"hierarchical\" stuff is a simple add-on embedding that is a simple weighted sum of all sentences. Also, the \"iterative\" comes from the problem definition itself while having nothing to do with the method. In this sense, the proposed attention model is neither \"iterative\" nor \"hierarchical\". I do think it makes novel technical contributions.\n\n2. The process when attending to a paragraph vector (i.e., unpack the paragraph by reusing dot-product attention, then pack it again with the extra learnable attention) seems tricky and cumbersome but the intuition behind is not clearly illustrated. \n\n3. The experiments on the expanded datasets are unfair to other baselines. As their deeply contextual QA interaction either falls in token-level or is subject to the ability of the backbones (e.g., BERT’s 512 length limit). I would recommend authors provide results on the original dataset.\n\n4. The performance of the proposed model lags far behind MATE on HybridQA dataset. It is curious that how MATE would be performed if applied to QASPER dataset.\n\nMinor comment:\n\nStrictly, the softmax term in Eq.3 (and argmax term q_t c_m) should be s^i_j q_t^T, as q_t and s^i_j are both 1*d matrices according to the notation at the bottom of page 3.\n\n\n", "summary_of_the_paper": "The paper proposes a simple attention-based model for conversational and multi-hop QA tasks. The model use BERT-like pre-trained LM ETC separately encodes questions and paragraph (i.e., a collection of sentences). Besides the encodings on sentence-level, the final context encodings also contain extra paragraph embeddings, which are a weighted sum of sentences’ encodings using a simple dot product attention. For the QA interaction, the models use a hard attention mechanism to select an entry representing either a sentence or a paragraph. \nThe experiments on two extractive QA datasets HYBRIDQA and QASPER show the model performs worse than the MATE model on HYBRIDQA but marginally better than other baselines on QASPER. On multi-hop QA and conversational QA tasks, the model performs marginally better than baselines on *expanded dataset*, but authors do not provide results on original datasets.\n", "summary_of_the_review": "Trivial method, contributing little scientific knowledge. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SoCwBSVmL5S", "reviewer": "Reviewer_XkRC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper proposes a machine reading comprehension model, which is computationally efficient and also can handle long-document in multiple granularities.\n2. The computational efficiency of recent QA models has been a problem in building a real-time QA system. This paper tries to solve the computational efficiency of recent MRC models. Their idea is somewhat similar to the idea used in phrase-indexed QA in open-domain QA. Thus, the topic of this paper is well-aligned with the recent approaches in the QA community.\n\nWeakness:\n\n1. The authors converted some QA datasets (ShARC and HybridQA) to long-document QA forms and evaluated DocHopper on these datasets. However, the authors did not show evidence that their modification on ShARC and HybridQA datasets is valid.\n2. The runtime efficiency of DocHopper is one of the main contributions that the authors claim in their paper. However, this paper does not provide enough information on measuring the runtime. For example, does the runtimes of the baselines and DocHopper measured in a fair environment? Please provide more detailed settings used in measuring runtime efficiency.\n3. IRRR is a multi-hop QA model in the full-wiki setting, and HotpotQA-long is in the distractor setting. Therefore, comparing IRRR and DocHopper is not a fair comparison. Although the authors provide the performance of HGN (in distractor setting), the performance of DocHopper underperforms. However, this is not a weak point since one of the research problems of this paper is alleviating the runtime problem while maintaining the MRC performance. However, since it is not clear how the authors conduct experiments to compute the runtime of each model, the HotpotQA-long results become insufficient evidence.", "summary_of_the_paper": "This paper provides a novel MRC model (DocHopper) for multi-hop QA over long structured documents. In multi-hop QA, the evidence necessary to answer a user's question is spread across different parts of the long document. Previous approaches find the evidence by iteratively updating the user's query. The problems in these previous approaches are 1) computational efficiency and 2) ineffective modeling strategy for figuring out the relations of the evidence. DocHopper resolves these problems with a hierarchical attention mechanism. Hierarchical attention mechanism provides two types of embedding vectors for a single paragraph: 1) local sentence vectors and 2) global context vector of the paragraph. DocHopper computes the similarities between the query vectors and these sentence/paragraph vectors and selects the proper evidence. Since the sentence/paragraph vectors can be pre-computed, the only inference time required for this method is the time for question embedding, and this brings drastic improvement in the computational efficiency. This paper uses four types of datasets for evaluation: 1) conversational QA (ShARC), 2) TableQA (HybridQA), 3) QA on academic paper (QASPER), and 4) multi-hop factual QA (HotpotQA), and shows the QA performance and computational efficiency of their model.", "summary_of_the_review": "The research problem of this paper is the inefficient computation time of existing QA models. This paper proposes a novel machine reading comprehension model for a long document. This paper provides evaluation results of their model on four types of datasets, and some experimental results show that DocHopper outperforms other baselines models. However, some evaluations results (results on ShARC and HybridQA) are insufficient to show the validity of their idea since these datasets are not designed for long-document QA. Also, it is unclear which experimental setting the authors used to compute the runtime of their model and baselines.\n\nSince my concerns are from the unclear description of this paper, I'm willing to increase the overall recommendation score if the authors provide a more clear description.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7BV8cWdLJuZ", "reviewer": "Reviewer_Ld3M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nI think overall the proposed approache is technically sound well motivated by the challenges in complex reasoning over a long context. They also test their approaches on diverse datasets including conversational QA and show the applicability to diverse QA data. \nTheir proposed method largely outperforms its base model, ETC, showing the effectiveness of newly introduced modules. Also, removing the necessity of jointly encoding query-document and reusing the encoded document embedding is useful to improve the run time efficiency.  \n\n### Weaknesses\nI have two main concerns listed below. \n1. **Limited novelties and technical contributions**: although the method seems to be technically sound, the main component is based on ETC, and from my understanding, their technical contribution mainly lies in iterative attention to update the query representations in embedding space. Updating query representations in embedding spaces instead of actually updating query text for multi-step retrieval&reasoning have been studied in prior work such as Feldman and El-Yaniv (2019) or Das et al. (2019), but those papers are not cited, and I would like to see comparison or discussions on the differences between those studies.  \n\n2. **Comparisons with the prior work**: Although they claim the proposed approach achieves state-of-the-art results on three of the benchmarks, the experimental settings are often slightly different from the original settings (e.g., only evaluate on the subset of the questions, use more passages than the default settings such as HotpotQA-Long). Claiming state-of-the-art results on those variant settings may not be fair, especially when the baselines to be compared are not specifically designed for the new setting. \n\nIn addition, I sometimes have difficulties following the paper due to some missing details in the method section. For example, I think there should be more details on how \"attention is supervised\" in the loss function paragraph in Section 3.4, instead of talking about them in the implementation details. On the other hand, how to embed queries for each task is discussed in detail in Section 3.2 before talking about DocHopper's technical contribution (Section 3.4. Iterative attention). This makes it hard for me to understand what is the key technical contribution of DocHopper, and which parts are generally appreciable to different tasks, and which parts should be customized for each task. \n\n### Missing reference \n- Feldman and El-Yaniv. 2020. Multi-Hop Paragraph Retrieval for Open-Domain Question Answering. In Proc. ACL. \n- Das et al. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. In Proc. ICLR. ", "summary_of_the_paper": "This paper introduces DocHopper, a new model for complex question answering over long documents (e.g., multi-hop QA over multiple paragraphs, conversational QA, reasoning over scientific documents). DocHopper is based on ETC (Ainslie et al., 2020) and DocHopper extends the existing hierarchical attentions from ETC with a new approach to update query representations in latent space. Their model does not jointly encode a question and context and does not require re-encoding of queries as in prior work, which leads to their effectiveness at inference time. They evaluate DocHopper on four different datasets: ShARC, QASPER, HotpotQA, and HybridQA. The proposed method achieves strong performance on those datasets, reducing the computational cost at the inference time. ", "summary_of_the_review": "In summary, I think the proposed approach is technically sound and a nice extension of ETC, but I don't agree that it provides significant technical contributions. Experimental results are strong, but as the experimental settings are different from the original setting, I do not think the paper should claim DOCHOPPER achieves state-of-the-art results on three datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5C-Ckh7i7GD", "reviewer": "Reviewer_zXR3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is interesting and more efficient compared with cross encoding approaches at inference time. \nMost prior results in the literature suggest that separate encoding approaches work inferior to cross encoders, thus results in this paper are encouraging. Despite encouraging results, there are some shortcomings in the execution.\n\n- One critical result from last year's research was that Transformers can capture multihop relationships in long documents without the need for directly modeling multi-hop behavior and hierarchical structures (see BigBird, ETC, Longformer papers). This paper somewhat goes against that direction and suggests that explicitly encoding multi-hop information can improve results. However, it remains difficult to fully accept this claim as some of the key comparisons are not fully controlled. More details below.\n\n- As mentioned in the paper, the ETC baseline is hampered by the 4096 context size limit. For the baseline, the authors encode the context paragraph by paragraph to sidestep the issue. However, this makes this baseline significantly weaker. Why not just evaluate the proposed model on the original HotpotQA data so that results are directly comparable? In addition Longformer can handle sequences of up to 16K tokens. I would have liked to see a Longformer based encoder as retriever + a BERT based reader as another more fair baseline on all the datasets.\n\n- While motivation for longer documents is important, most of the datasets used are not the original versions. E.g., Hotpot and ShARC are modified to make them longer and for Qasper only a subset of extractive answers are used. This makes direct comparisons with SOTA not possible.\n\n- I would have liked to see a comparison of number of parameters of the proposed model compared with baselines. For example, the LED baseline for QASPER is a single stage model where as the proposed approach uses two models, one for extracting evidence + a BERT-based reader for extracting the final answer. In this regard, the comparison is not entirely fair.\n\n- The exact method of constructing the paragraph embeddings by a weighted average of sentences based on query relevance is a bit under motivated. It would have been nice to see an ablation on importance of this design (e.g., compared with simple or learned weighted average pooling).\n\n- One of the stated benefits of the proposed approach is that it is more efficient by reducing the need for cross encoding the query and context at inference time. However, I would have liked to see some discussion about the added storage cost compared with the cross encoding methods. Pre-computing and storing vectors of all sentences and paragraphs in a document can be incur significant storage costs. \n\n- It would have been nice to see some error analysis, case study, or some discussion looking at what types of information the proposed model is able to capture compared with the baselines.\n\nQuestions for authors:\n- What is finally used as text of q_null? \n- Which ETC model size is used? Which model size is used for the BERT-based reader?\n\nTerminology:\n- I'm not sure \"hierarchical attention\" is standard terminology for the model architecture of ETC. It might be confused by multi-layer models that are designed so that lower layers encode smaller pieces of information (e.g, words) and subsequent layers ingest these fine-grained representations and produce coarser level representations (Yang et al 2016). I would either use another terminology or add an explicit definition to make it clear that by \"hierarchical\" you mean that the model provides representation of individual blocks of text within the full sequence, as ETC does not have an explicit hierarchical structure. ", "summary_of_the_paper": "The paper proposes an iterative approach for multi-hop question answering. At high-level the proposed model breaks a question into multiple sub-questions and then adds information relevant to each sub-question to the query vector for the next step retrieval. At each iteration an ETC encoder is used to encode the document and a sub-question; the vector corresponding to sub-question is then updated iteratively by contextualizing it over sentences and paragraphs in the document and is used to extract the final answer using a subsequent BERT reader. Evaluation results show improvements on 3 of 4 datasets. ", "summary_of_the_review": "Overall, the paper proposes an interesting approach to answer questions over long documents efficiently. However, there are some issues with the execution, presented results, and analyses making it difficult to fully accept the main claims in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strength
* The paper is relatively clearly written.
* The proposed method appears to be sound.

Weakness
* The novelty of the work seems to be limited.  
* The experiment part needs significant improvements.  The comparison with existing methods may not be fair.  Evaluation of efficiency should be given. There are also detailed investigations that need to be conducted, as indicated by the reviewers.
* There are technical issues that need to be addressed.
Decision: Reject