Iterative Hierarchical Attention for Answering Complex Questions over Long Documents

1 INTRODUCTION . In this work we focus on the problem of answering complex questions over long structured documents . A long document typically contains coherent information on a certain topic , and the contents are grouped by sections or other structures . To answer complex , multi-hop questions over long documents often requires navigating through different parts of the documents to find different pieces of information relevant to a question . This navigation , in turn , requires understanding high-level information about the structure of the document . For example , to answer the question “ What modules in DOCHOPPER will be finetuned in all the experiments ? ” , one might first turn to the section title “ Model ” to identify the different modules in DOCHOPPER , and then read the “ Experiments ” with these modules in mind , potentially further attending to specific subsections ( such as the ones titled “ Implementation Details ” ) . As in academic papers ( Dasigi et al. , 2021 ) , similar tasks are common for questions about government policies ( Saeidi et al. , 2018 ) or legal documents . This type of QA tests not only the ability to understand short passages of text , but also the ability to understand the goals of the question and the structure of documents in a domain . A common approach of solving multi-hop questions is to iteratively find evidence for one hop , and then use that evidence to update the query used in the next hop of the QA process . The update can be performed by either explicitly predicting the intermediate answers ( Talmor & Berant , 2018 ; Sun et al. , 2019 ) or directly appending previous evidences to the questions ( Zhao et al. , 2021 ; Qi et al. , 2021 ; Li et al. , 2020 ; Xiong et al. , 2021 ) . While appending retrieved evidence to a query works well on many factual QA tasks , where it is possible to answer questions with evidences that are short pieces of text , this approach is expensive if one wishes to retrieve larger pieces of text as evidences ( e.g. , the “ experiments ” section of a paper ) . Another disadvantage is that appending together many small fragments of text intuitively fails to capture the relationships between them , and the structure of the document from which they were extracted . To capture high-level structural information in a document as well as detailed information from short passages , hierarchical attention mechanisms have been proposed , which learn neural representations 1We will open-source our code and data . at different levels that are then mixed to make final predictions for simple questions ( Wang et al. , 2018 ; Chang et al. , 2019 ) . Hierarchical attention has also been adopted in pretrained language models , e.g . ETC ( Ainslie et al. , 2020 ) , which introduced a global-local attention mechanism where embeddings of special global tokens are used to encode high-level information . Our DOCHOPPER system incorporates ETC as a document encoder . However , while ETC has previously performed well on multi-hop QA tasks like HotpotQA and WikiHop ( Yang et al. , 2018 ; Welbl et al. , 2018 ) which require combining information from a small number of short passages , it has not been previously evaluated on tasks of the sort considered here . Our experiments show that DOCHOPPER outperforms past approaches to using ETC for multi-hop questions . DOCHOPPER extends the existing hierarchical attention methods with a novel approach to updating queries in a multi-hop setting . DOCHOPPER iteratively attends to different parts of the document , either at fine-grained level or at higher level . This process can be viewed as either retrieving a short passage , or navigating to a part of a document . In each iteration , the query vector is updated in the embedding space rather than by re-encoding a sequence of concatenated tokens . This updating step is end-to-end differentiable and efficient . In our experiments , we also show it is effective on four different benchmarks involving complex queries over long structured documents . In particular , we evaluate DOCHOPPER on four different tasks : conversational QA for discourse entailment reasoning , using the ShARC ( Saeidi et al. , 2018 ) benchmark2 ; factual QA with table and text , using HybridQA ( Chen et al. , 2020 ) ; information seeking QA on academic papers , using QASPER ( Dasigi et al. , 2021 ) ; and multi-hop factual QA , using HotpotQA ( Yang et al. , 2018 ) . Since the outputs of the four tasks are different , additional layers or simple downstream models are appended to DOCHOPPER to get the final answers . DOCHOPPER achieves state-of-the-art results on three of datasets , outperforming the baseline models by 3 % -5 % . Additionally , DOCHOPPER runs 3– 10 faster than the baseline models , because the neural representations of documents is pre-computed , which significantly reduces computation cost at inference time . 2 RELATED WORKS . Graph-based models have been widely used for answering multi-hop questions in factual QA ( Min et al. , 2020 ; Sun et al. , 2018 ; 2019 ; Qiu et al. , 2019 ; Fang et al. , 2019 ) . However , most of the graphbased models are grounded to entities , i.e. , evidences ( from knowledge bases or text corpus ) are connected by entities in the graph . The graph construction step also heavily relies on many discrete features such as hyperlinks or entities predicted with external entity linkers . It ’ s not clear how to apply these models to more general tasks if context is not entity-centric , such as questions about academic papers or government documents . Similar problems also exist in memory-augmented language models that achieved the state-of-the-art on many factual QA tasks ( Guu et al. , 2020 ; Lewis et al. , 2021 ; Verga et al. , 2020 ; Dhingra et al. , 2020 ; Sun et al. , 2021 ) . Alternatively , one can adopt the “ retrieve and read ” pipeline to answer multi-hop questions over long documents . Recent works proposed to extend the dense retrieval methods ( Karpukhin et al. , 2020 ) to multi-hop questions ( Zhao et al. , 2021 ; Qi et al. , 2021 ; Li et al. , 2020 ) . However , such models retrieve one small piece of evidence at a time , lacking the ability of navigating between different parts of the documents to find relevant information at both higher and lower levels of the documentstructure hierarchy . Another disadvantage of these iterative models is that they are not end-to-end differentiable . Updating the questions for the next hop requires re-encoding the concatenated tokens from the questions and previously retrieved evidences . It also makes the model inefficient because re-encoding tokens with large Transformer models is very expensive . Besides question answering tasks , hierarchical attention has been successfully used in tasks such as document classification ( Yang et al. , 2016 ; Chang et al. , 2019 ) , summarization ( Gidiotis & Tsoumakas , 2020 ; Xiao & Carenini , 2019 ; Zhang et al. , 2019 ) , sentiment analysis ( Ruder et al. , 2016 ) , text segmentation ( Koshorek et al. , 2018 ) , etc . It is worth mentioning that ETC ( Ainslie et al. , 2020 ) was also used on a key-phrase extraction task on web pages using the structured DOM tree . However , none of these models can be easily adapted to answering complex questions over long documents . 2We modify the original dataset and replace the oracle snippet with the entire web page as input . Please see section §4.1 for more details . 3 MODEL . In this section , we first introduce how to compute neural representations for questions and context at different levels . Then , we present the iterative process that attends to information in the document and updates the query vector . Additional layers or simple downstream models required for different downstream tasks will be explained in the next section ( §4 ) . 3.1 BACKGROUND . Input A long document usually contains multiple levels of hierarchy , e.g . sections , sub-sections , paragraphs , sentences , etc . For simplicity , we only consider two levels of hierarchy in this paper : paragraph-level and sentence-level . A sentence is the lowest granularity that can be retrieved , while a paragraph is an abstraction of a collection of sentences , which can be used to represent sections or other levels in the hierarchy , depending on the application . Formally , let d = { p0 , . . . , p|d| } ∈ D be a document in the corpus D that contains a sequence of paragraphs pi , and let a paragraph pi = { si0 , . . . , si|pi| } contain a sequence of sentences s i 0 . A sentence s i j will be encoded into a fixed length vector sij ∈ Rd . Pretrained ETC We use ETC ( Ainslie et al. , 2020 ) as our query and context encoder as it is pretrained to produce both token-level and sentence-level embeddings . ETC is pretrained as a Mask Language Model ( MLM ) . Different from BERT ( Devlin et al. , 2019 ) , it employs an additional global-local attention mechanism . ETC assigns to each sentence a special global token that only attends to local tokens in the sentence , and its embedding is trained to summarize the information of local tokens in the sentence . A global token also attends the global tokens of other sentences in the input . ETC additionally adopts Contrastive Predictive Coding ( CPC ) ( Oord et al. , 2018 ) to train the embedding of global tokens to make them aware of other sentences in the context . We use the embeddings of global tokens in ETC as encodings for sentences . Specifically , ETC takes a list of sentences pi = { si0 , . . . , si|pi| } as input , and returns a list of vectors si0 , . . . , si|pi| , where each s i j ∈ Rd represents the embedding of a sentence sij . si0 , . . . , s i |pi| = ETC ( { s i 0 , . . . , s i |pi| } ) ∈ R |pi|×d 3.2 QUERY EMBEDDINGS . We discuss the strategy of computing query vectors for two types of input , conversational QA and multi-hop QA . We use ETC ( Ainslie et al. , 2020 ) as our query encoder for the reasons discussed above . Other pretrained language models for similar purposes should work as well . Conversational QA A conversational question contains multiple rounds of interaction between the machine and human . As the conversation goes on , the topic of the conversation will shift , but questions asked in the next turn are related to the history of the conversation . We consider this multi-turn process as a multi-hop QA task . In our specific task , the conversation starts with an initial question q0 . The model will answer the question if there ’ s enough information . Otherwise , the model will ask a followup question f1 and expect the user ’ s answer a1 . The model will keep asking followup questions for a few more iterations . The task is to predict the answer if there is enough information after reading the full conversation history , or mark the question as not answerable . We denote ai as the answer to the followup question fi received from the users at the i ’ th iteration . We consider the full conversation history as the question pq = { q0 , ( f1 , a1 ) , . . . , ( fn , an ) } with the initial question denoted q0 and the i-th followup question-answer pair denoted ( fi , ai ) . The answer ai to the followup question fi is concatenated to the end of the questions and represented as a single sentence . We call pq a question paragraph as it contains a sequence of question-answer pairs . We use ETC to compute the query embeddings qi , one for each sentence in the query paragraph pq . The query embeddings qi will be used to perform iterative attention over the document . q0 , . . . , qn = ETCq ( { q0 , ( f1 , a1 ) , . . . , ( fn , an ) } ) ( 1 ) Multi-hop QA A multi-hop question , e.g . “ Which gulf is north of the Somalian city with 550,000 residents ” , requires the model to first find the answer of the first “ hop ” of the question , and then perform a second round to find the final answer . Assume for now there are exactly two hops . Different from conversational QA , questions in multi-hop QA do not have a clear split between the two hops of questions , making it impossible to explicitly split a question into two sub-questions . Instead , we add a dummy question qnull to the question paragraph qp = { q0 , qnull } . The global-to-local attention mask of ETC is modified to allow the global token of the dummy question to attend to tokens in the question q0 . With this modification , the query embeddings q0 and q1 for q0 and qnull can to attend any part of the question , but each can also attend to different parts of the question . One could append additional dummy questions to the question paragraph pq if the number of hops is larger . Thus we let q0 , q1 = ETCq ( { q0 , qnull } ) ( 2 ) Note that we assume the true number of hops is known in multi-hop QA tasks , which is two or four for the experiments in this paper . However , one can train models to decide when to stop if questions have various numbers of hops . We leave this as a topic for future work .

Summary: This paper introduces DocHopper, a new model for complex question answering over long documents (e.g., multi-hop QA over multiple paragraphs, conversational QA, reasoning over scientific documents). DocHopper is based on ETC (Ainslie et al., 2020) and DocHopper extends the existing hierarchical attentions from ETC with a new approach to update query representations in latent space. Their model does not jointly encode a question and context and does not require re-encoding of queries as in prior work, which leads to their effectiveness at inference time. They evaluate DocHopper on four different datasets: ShARC, QASPER, HotpotQA, and HybridQA. The proposed method achieves strong performance on those datasets, reducing the computational cost at the inference time.