Iterative Hierarchical Attention for Answering Complex Questions over Long Documents

1 INTRODUCTION . In this work we focus on the problem of answering complex questions over long structured documents . A long document typically contains coherent information on a certain topic , and the contents are grouped by sections or other structures . To answer complex , multi-hop questions over long documents often requires navigating through different parts of the documents to find different pieces of information relevant to a question . This navigation , in turn , requires understanding high-level information about the structure of the document . For example , to answer the question “ What modules in DOCHOPPER will be finetuned in all the experiments ? ” , one might first turn to the section title “ Model ” to identify the different modules in DOCHOPPER , and then read the “ Experiments ” with these modules in mind , potentially further attending to specific subsections ( such as the ones titled “ Implementation Details ” ) . As in academic papers ( Dasigi et al. , 2021 ) , similar tasks are common for questions about government policies ( Saeidi et al. , 2018 ) or legal documents . This type of QA tests not only the ability to understand short passages of text , but also the ability to understand the goals of the question and the structure of documents in a domain . A common approach of solving multi-hop questions is to iteratively find evidence for one hop , and then use that evidence to update the query used in the next hop of the QA process . The update can be performed by either explicitly predicting the intermediate answers ( Talmor & Berant , 2018 ; Sun et al. , 2019 ) or directly appending previous evidences to the questions ( Zhao et al. , 2021 ; Qi et al. , 2021 ; Li et al. , 2020 ; Xiong et al. , 2021 ) . While appending retrieved evidence to a query works well on many factual QA tasks , where it is possible to answer questions with evidences that are short pieces of text , this approach is expensive if one wishes to retrieve larger pieces of text as evidences ( e.g. , the “ experiments ” section of a paper ) . Another disadvantage is that appending together many small fragments of text intuitively fails to capture the relationships between them , and the structure of the document from which they were extracted . To capture high-level structural information in a document as well as detailed information from short passages , hierarchical attention mechanisms have been proposed , which learn neural representations 1We will open-source our code and data . at different levels that are then mixed to make final predictions for simple questions ( Wang et al. , 2018 ; Chang et al. , 2019 ) . Hierarchical attention has also been adopted in pretrained language models , e.g . ETC ( Ainslie et al. , 2020 ) , which introduced a global-local attention mechanism where embeddings of special global tokens are used to encode high-level information . Our DOCHOPPER system incorporates ETC as a document encoder . However , while ETC has previously performed well on multi-hop QA tasks like HotpotQA and WikiHop ( Yang et al. , 2018 ; Welbl et al. , 2018 ) which require combining information from a small number of short passages , it has not been previously evaluated on tasks of the sort considered here . Our experiments show that DOCHOPPER outperforms past approaches to using ETC for multi-hop questions . DOCHOPPER extends the existing hierarchical attention methods with a novel approach to updating queries in a multi-hop setting . DOCHOPPER iteratively attends to different parts of the document , either at fine-grained level or at higher level . This process can be viewed as either retrieving a short passage , or navigating to a part of a document . In each iteration , the query vector is updated in the embedding space rather than by re-encoding a sequence of concatenated tokens . This updating step is end-to-end differentiable and efficient . In our experiments , we also show it is effective on four different benchmarks involving complex queries over long structured documents . In particular , we evaluate DOCHOPPER on four different tasks : conversational QA for discourse entailment reasoning , using the ShARC ( Saeidi et al. , 2018 ) benchmark2 ; factual QA with table and text , using HybridQA ( Chen et al. , 2020 ) ; information seeking QA on academic papers , using QASPER ( Dasigi et al. , 2021 ) ; and multi-hop factual QA , using HotpotQA ( Yang et al. , 2018 ) . Since the outputs of the four tasks are different , additional layers or simple downstream models are appended to DOCHOPPER to get the final answers . DOCHOPPER achieves state-of-the-art results on three of datasets , outperforming the baseline models by 3 % -5 % . Additionally , DOCHOPPER runs 3– 10 faster than the baseline models , because the neural representations of documents is pre-computed , which significantly reduces computation cost at inference time . 2 RELATED WORKS . Graph-based models have been widely used for answering multi-hop questions in factual QA ( Min et al. , 2020 ; Sun et al. , 2018 ; 2019 ; Qiu et al. , 2019 ; Fang et al. , 2019 ) . However , most of the graphbased models are grounded to entities , i.e. , evidences ( from knowledge bases or text corpus ) are connected by entities in the graph . The graph construction step also heavily relies on many discrete features such as hyperlinks or entities predicted with external entity linkers . It ’ s not clear how to apply these models to more general tasks if context is not entity-centric , such as questions about academic papers or government documents . Similar problems also exist in memory-augmented language models that achieved the state-of-the-art on many factual QA tasks ( Guu et al. , 2020 ; Lewis et al. , 2021 ; Verga et al. , 2020 ; Dhingra et al. , 2020 ; Sun et al. , 2021 ) . Alternatively , one can adopt the “ retrieve and read ” pipeline to answer multi-hop questions over long documents . Recent works proposed to extend the dense retrieval methods ( Karpukhin et al. , 2020 ) to multi-hop questions ( Zhao et al. , 2021 ; Qi et al. , 2021 ; Li et al. , 2020 ) . However , such models retrieve one small piece of evidence at a time , lacking the ability of navigating between different parts of the documents to find relevant information at both higher and lower levels of the documentstructure hierarchy . Another disadvantage of these iterative models is that they are not end-to-end differentiable . Updating the questions for the next hop requires re-encoding the concatenated tokens from the questions and previously retrieved evidences . It also makes the model inefficient because re-encoding tokens with large Transformer models is very expensive . Besides question answering tasks , hierarchical attention has been successfully used in tasks such as document classification ( Yang et al. , 2016 ; Chang et al. , 2019 ) , summarization ( Gidiotis & Tsoumakas , 2020 ; Xiao & Carenini , 2019 ; Zhang et al. , 2019 ) , sentiment analysis ( Ruder et al. , 2016 ) , text segmentation ( Koshorek et al. , 2018 ) , etc . It is worth mentioning that ETC ( Ainslie et al. , 2020 ) was also used on a key-phrase extraction task on web pages using the structured DOM tree . However , none of these models can be easily adapted to answering complex questions over long documents . 2We modify the original dataset and replace the oracle snippet with the entire web page as input . Please see section §4.1 for more details . 3 MODEL . In this section , we first introduce how to compute neural representations for questions and context at different levels . Then , we present the iterative process that attends to information in the document and updates the query vector . Additional layers or simple downstream models required for different downstream tasks will be explained in the next section ( §4 ) . 3.1 BACKGROUND . Input A long document usually contains multiple levels of hierarchy , e.g . sections , sub-sections , paragraphs , sentences , etc . For simplicity , we only consider two levels of hierarchy in this paper : paragraph-level and sentence-level . A sentence is the lowest granularity that can be retrieved , while a paragraph is an abstraction of a collection of sentences , which can be used to represent sections or other levels in the hierarchy , depending on the application . Formally , let d = { p0 , . . . , p|d| } ∈ D be a document in the corpus D that contains a sequence of paragraphs pi , and let a paragraph pi = { si0 , . . . , si|pi| } contain a sequence of sentences s i 0 . A sentence s i j will be encoded into a fixed length vector sij ∈ Rd . Pretrained ETC We use ETC ( Ainslie et al. , 2020 ) as our query and context encoder as it is pretrained to produce both token-level and sentence-level embeddings . ETC is pretrained as a Mask Language Model ( MLM ) . Different from BERT ( Devlin et al. , 2019 ) , it employs an additional global-local attention mechanism . ETC assigns to each sentence a special global token that only attends to local tokens in the sentence , and its embedding is trained to summarize the information of local tokens in the sentence . A global token also attends the global tokens of other sentences in the input . ETC additionally adopts Contrastive Predictive Coding ( CPC ) ( Oord et al. , 2018 ) to train the embedding of global tokens to make them aware of other sentences in the context . We use the embeddings of global tokens in ETC as encodings for sentences . Specifically , ETC takes a list of sentences pi = { si0 , . . . , si|pi| } as input , and returns a list of vectors si0 , . . . , si|pi| , where each s i j ∈ Rd represents the embedding of a sentence sij . si0 , . . . , s i |pi| = ETC ( { s i 0 , . . . , s i |pi| } ) ∈ R |pi|×d 3.2 QUERY EMBEDDINGS . We discuss the strategy of computing query vectors for two types of input , conversational QA and multi-hop QA . We use ETC ( Ainslie et al. , 2020 ) as our query encoder for the reasons discussed above . Other pretrained language models for similar purposes should work as well . Conversational QA A conversational question contains multiple rounds of interaction between the machine and human . As the conversation goes on , the topic of the conversation will shift , but questions asked in the next turn are related to the history of the conversation . We consider this multi-turn process as a multi-hop QA task . In our specific task , the conversation starts with an initial question q0 . The model will answer the question if there ’ s enough information . Otherwise , the model will ask a followup question f1 and expect the user ’ s answer a1 . The model will keep asking followup questions for a few more iterations . The task is to predict the answer if there is enough information after reading the full conversation history , or mark the question as not answerable . We denote ai as the answer to the followup question fi received from the users at the i ’ th iteration . We consider the full conversation history as the question pq = { q0 , ( f1 , a1 ) , . . . , ( fn , an ) } with the initial question denoted q0 and the i-th followup question-answer pair denoted ( fi , ai ) . The answer ai to the followup question fi is concatenated to the end of the questions and represented as a single sentence . We call pq a question paragraph as it contains a sequence of question-answer pairs . We use ETC to compute the query embeddings qi , one for each sentence in the query paragraph pq . The query embeddings qi will be used to perform iterative attention over the document . q0 , . . . , qn = ETCq ( { q0 , ( f1 , a1 ) , . . . , ( fn , an ) } ) ( 1 ) Multi-hop QA A multi-hop question , e.g . “ Which gulf is north of the Somalian city with 550,000 residents ” , requires the model to first find the answer of the first “ hop ” of the question , and then perform a second round to find the final answer . Assume for now there are exactly two hops . Different from conversational QA , questions in multi-hop QA do not have a clear split between the two hops of questions , making it impossible to explicitly split a question into two sub-questions . Instead , we add a dummy question qnull to the question paragraph qp = { q0 , qnull } . The global-to-local attention mask of ETC is modified to allow the global token of the dummy question to attend to tokens in the question q0 . With this modification , the query embeddings q0 and q1 for q0 and qnull can to attend any part of the question , but each can also attend to different parts of the question . One could append additional dummy questions to the question paragraph pq if the number of hops is larger . Thus we let q0 , q1 = ETCq ( { q0 , qnull } ) ( 2 ) Note that we assume the true number of hops is known in multi-hop QA tasks , which is two or four for the experiments in this paper . However , one can train models to decide when to stop if questions have various numbers of hops . We leave this as a topic for future work .

Summary: The paper proposes a simple attention-based model for conversational and multi-hop QA tasks. The model use BERT-like pre-trained LM ETC separately encodes questions and paragraph (i.e., a collection of sentences). Besides the encodings on sentence-level, the final context encodings also contain extra paragraph embeddings, which are a weighted sum of sentences’ encodings using a simple dot product attention. For the QA interaction, the models use a hard attention mechanism to select an entry representing either a sentence or a paragraph.  The experiments on two extractive QA datasets HYBRIDQA and QASPER show the model performs worse than the MATE model on HYBRIDQA but marginally better than other baselines on QASPER. On multi-hop QA and conversational QA tasks, the model performs marginally better than baselines on *expanded dataset*, but authors do not provide results on original datasets.