Long Document Summarization with Top-Down and Bottom-Up Representation Inference

1 INTRODUCTION . Text summarization involves compressing a document and preserving key content and meaning . It can be done in either an extractive or abstractive manner . While an extractive summarization model extracts salient fragments ( e.g. , words , sentences ) from the source document to form a summary , an abstractive summarization system aims to generate a semantically coherent and linguistically fluent summary by conditioning on the document . The abstractive approach aligns better with how a human does summarization and generally performs better than extractive models in recent works ( Pilault et al. , 2020 ; Zhang et al. , 2020 ) . We thus focuses on abstractive summarization . The dominant approach for abstractive summarization is to use a Seq2Seq model ( Sutskever et al. , 2014 ) with an encoder-decoder architecture instantiated with either RNNs ( Hochreiter & Schmidhuber , 1997 ) or , more recently , transformers ( Vaswani et al. , 2017 ) . In such a model , an encoder infers the latent representations of observed tokens ( words or subwords ) in the document , conditioning on which a decoder generates a summary . This paper studies the problem of how to infer good latent representations , which in turn would improve summarization . We propose a framework which ( 1 ) assumes a multi-scale latent structure of a document and ( 2 ) synergizes bottom-up inference with top-down inference . In a multi-scale structure , high-level variables ( like those representing sentences , segments ) model the document at a coarser time-scale and abstract away details , and are suitable for capturing long range dependency of the document ; in contrast , low-level variables ( like those representing tokens ) preserves details , and prevent the summary from losing key details . In our framework , the summary is generated by conditioning on token representations ( low-level vari- ables ) , similar to recent abstractive summarization models ( Zhang et al. , 2020 ; Zaheer et al. , 2020 ; Beltagy et al. , 2020 ) . There is however a critical difference . In our framework , token representations are first bottom-up inferred and then top-down updated with high level representations , hence rendering low-level representations aware of long range information . We hypothesize that the proposed inference approach would improve summarization . Multi-level models have been widely studied in modeling for images ( Sønderby et al. , 2016 ) , speech ( Mehri et al. , 2016 ) , and language ( Chung et al. , 2016 ) . Prior summarization works ( Cheng & Lapata , 2016 ; Nallapati et al. , 2016 ; Zhang et al. , 2019 ; Xu et al. , 2020 ) have also explored hierarchical models . But they mostly focus on extractive summarization and follow a bottom-up inference approach . They pool information in words or sub-words to form sentence representations , based on which a classification is done to make an extraction decision . In comparison , our framework combines bottom-up and top-down inference . This draws direct inspiration from a line of work which examines variational inference for hierarchical top-down generative models ( Sønderby et al. , 2016 ; Maaløe et al. , 2019 ; Child , 2020 ) . In these models , in the bottom-up path distribution parameters of higher level stochastic variables are computed as a function of lower level stochastic variables , while in the top-down path distribution parameters of lower level variables are corrected for as a function of higher level variables . Although we do not assume stochasticity of the document latent representations , our encoder or inference model follows the same idea to infer token representations . The proposed framework is agnostic to model architecture . Due to the dominance of transformer models in NLP ( Chen et al. , 2018 ; Zhang et al. , 2020 ; Sun et al. , 2019 ; Martin et al. , 2020 ) and to leverage pre-trained language models ( Liu et al. , 2019 ; Lewis et al. , 2020 ) , we instantiate our framework with a transformer-based model . There is a bottleneck of applying transformers to long documents , because its computational and memory cost has a quadratic dependency on the sequence length . This issue is especially critical for summarization since we are more interested in summarizing long documents since short ones can be quickly read through by humans . To address this issue , a large amount of prior works have been devoted to developing efficient transformers with sub-quadratic complexity . They approach this problem with kernel-based methods ( Katharopoulos et al. , 2020 ; Choromanski et al. , 2020 ) , by low-rank approximation to the attention matrix ( Wang et al. , 2020 ) , by synthesizing the attention weights ( Tay et al. , 2021 ) , or by designing contentindependent ( Child et al. , 2019 ; Beltagy et al. , 2020 ; Ainslie et al. , 2020 ; Zaheer et al. , 2020 ) or content-dependent sparse attention mechanisms ( Kitaev et al. , 2020 ; Roy et al. , 2021 ; Wang et al. , 2021 ) . Our framework provides a natural way to diminish this quadratic complexity issue . In the bottomup inference , we use local self-attention where each token only attends tokens within a local fixedlength window , and thus the complexity does not grow as a function of the input sequence length . The top-down correction for the token representations enable them to capture long-range context , reducing the limitation of local attention . Furthermore , in contrast to most prior efficient transformers that are incompatible with pre-training language models , our framework is flexible for leveraging any pre-trained encoder-decoder models such as BART ( Lewis et al. , 2020 ) , T5 ( Raffel et al. , 2020 ) . We call the transformer-based model following the proposed framework as top-down transformer , to emphasize the importance of the top-down inference . We evaluate the top-down transformer on a set of distinct summarization benchmarks . These benchmarks cover documents from a variety of domains , including news articles and scientific , conversational , and narrative documents , and of various lengths ranging from hundreds of words ( e.g. , a news article ) , several thousands to over ten thousands of words ( e.g. , a scientific paper , a book chapter ) , to even over hundred thousands of words ( e.g. , an entire book ) . On short documents , models following our framework achieves on-par or better summarization performance than models with full self-attention , and are more computeand memory-efficient . Across all long document datasets , our models achieve state-of-the-art performance . In the end , we show that our model is able to summarize a whole book . Compared to a concurrent work ( Wu et al. , 2021 ) using GPT-3 and requiring humans to extensively label data , our model achieves competitive performance with 380 times less parameters and a small amount of publicly available data . The diverse and strong empirical results support the effectiveness and wide applicability of the proposed model . 2 METHODS . Figure 1 gives a graphical overview of the top-down transformer , instantiating the proposed framework . We introduce its details in this section . Suppose a document has N tokens , t = { ti } Ni=1 . In our method , token representations are inferred by combining top-down and bottom-up inference . This leads to effective and efficient inference for token representations . They are then attended by a decoder to generate a summary , as in a regular encoder-decoder transformer . 2.1 BOTTOM-UP INFERENCE . In the bottom-up inference , contextual embeddings of the tokens , { ei | ei ∈ Rd } Ni=1 , are computed with N1 layers of local self-attention . In particular , each token ti only attends to nearby tokens within a window of size of w. The complexity is hence O ( Nw ) , in contrast to O ( N2 ) for a full self-attention models . Please see Supplementary B.1 for more details . 2.2 TOP-DOWN INFERENCE . The efficiency with local self-attention in the bottom-up inference nevertheless comes with a limitation , that is , each ei only captures the context within a local window instead of that of the whole document . To mitigate this issue , we propose a top-down inference for token representations . Consider a two-level multi-scale latent structure for a document . The low level consists of token representations , { ei } Ni=1 , computed by the bottom-up inference . The top level consists of units at a coarser level . It is affordable to apply full self-attention at the top level due to its coarser granularity , allowing these top-level units to capture global document context . In our work , the self-attention mechanism for the top-level representations is simply the original multi-head self-attention proposed in Vaswani et al . ( 2017 ) . Readers are referred to Vaswani et al . ( 2017 ) for details . Denote the top level representations after self-attention update as { sj | sj ∈ Rd } Mj=1 ( see Section 2.3 for details on top-level representation initialization methods ) . We can then update the bottom-upinferred token representations with the top-level representations . This is achieved withN3 top-down inference layers , as illustrated by the middle panel in Figure 1 . Each layer contains three transformations on { ei } : ( 1 ) token self-attention , ( 2 ) token-segment cross-attention , ( 3 ) feed-forward . ( 1 ) and ( 3 ) are the same as those in the bottom-up inference layers or regular self-attention layer with local attention . ( 2 ) implementing the cross-attention between the top and bottom levels is the critical operation . In particular , each ei is updated with cross-attention , ẽi = ei + LayerNorm ( M∑ j=1 αijfv ( sj ) ) , αij = exp ( fq ( ei ) T fk ( sj ) ) √ d ∑M l=1 exp ( fq ( ei ) T fk ( sl ) ) ( 1 ) where fq , fk , and fv indicate query , key , and value linear mappings , respectively . For notional clarity , Equation 1 only illustrates the case with a single attention head . In practice , we use multiheads . The cross-attention operation injects global contextual information into bottom-up-inferred token representations , ei , and yields global-context-aware token representations , ẽi , conditioning on which a summary can be generated by a decoder . To instantiate the top-down inference , we need to make two choices : ( 1 ) the number of top-levels above the token level and ( 2 ) the unit representation for each top-level . We choose to use one top level since it is sufficiently coarser to apply full self-attention for a wide range of long document benchmarks we experimented on . A natural choice for top level units is sentence , paragraph , and chapter , depending on the number top level considered . Such a choice however might lead to complicated implementations and non-scalability due to the varying length of these units . We hence choose a simpler approach , where the top level consists of fixed-length segments of the documents . While we use a single top level , multiple top levels can be simply achieved with segments with increasingly coarser granularity . In the top-down inference , segment-level self-attention has a complexity of O ( M2 ) , and tokensegment cross-attention has a complexity of O ( NM ) . Thus , together with bottom-up inference , the complexity is O ( Nw +M2 + NM ) . In practice , we use relatively small w ( window size ) and M ( number of segments ) .

Summary: Modeling long documents is a challenging problem. This paper works on this problem with top-down and bottom-up structures. The hierarchical structure proposed in this paper adopts local and top-down correction to make the model learn local and long-range dependency. The experiment on long document summarization benchmarks shows the effectiveness of their proposed method.