Long Document Summarization with Top-Down and Bottom-Up Representation Inference

1 INTRODUCTION . Text summarization involves compressing a document and preserving key content and meaning . It can be done in either an extractive or abstractive manner . While an extractive summarization model extracts salient fragments ( e.g. , words , sentences ) from the source document to form a summary , an abstractive summarization system aims to generate a semantically coherent and linguistically fluent summary by conditioning on the document . The abstractive approach aligns better with how a human does summarization and generally performs better than extractive models in recent works ( Pilault et al. , 2020 ; Zhang et al. , 2020 ) . We thus focuses on abstractive summarization . The dominant approach for abstractive summarization is to use a Seq2Seq model ( Sutskever et al. , 2014 ) with an encoder-decoder architecture instantiated with either RNNs ( Hochreiter & Schmidhuber , 1997 ) or , more recently , transformers ( Vaswani et al. , 2017 ) . In such a model , an encoder infers the latent representations of observed tokens ( words or subwords ) in the document , conditioning on which a decoder generates a summary . This paper studies the problem of how to infer good latent representations , which in turn would improve summarization . We propose a framework which ( 1 ) assumes a multi-scale latent structure of a document and ( 2 ) synergizes bottom-up inference with top-down inference . In a multi-scale structure , high-level variables ( like those representing sentences , segments ) model the document at a coarser time-scale and abstract away details , and are suitable for capturing long range dependency of the document ; in contrast , low-level variables ( like those representing tokens ) preserves details , and prevent the summary from losing key details . In our framework , the summary is generated by conditioning on token representations ( low-level vari- ables ) , similar to recent abstractive summarization models ( Zhang et al. , 2020 ; Zaheer et al. , 2020 ; Beltagy et al. , 2020 ) . There is however a critical difference . In our framework , token representations are first bottom-up inferred and then top-down updated with high level representations , hence rendering low-level representations aware of long range information . We hypothesize that the proposed inference approach would improve summarization . Multi-level models have been widely studied in modeling for images ( Sønderby et al. , 2016 ) , speech ( Mehri et al. , 2016 ) , and language ( Chung et al. , 2016 ) . Prior summarization works ( Cheng & Lapata , 2016 ; Nallapati et al. , 2016 ; Zhang et al. , 2019 ; Xu et al. , 2020 ) have also explored hierarchical models . But they mostly focus on extractive summarization and follow a bottom-up inference approach . They pool information in words or sub-words to form sentence representations , based on which a classification is done to make an extraction decision . In comparison , our framework combines bottom-up and top-down inference . This draws direct inspiration from a line of work which examines variational inference for hierarchical top-down generative models ( Sønderby et al. , 2016 ; Maaløe et al. , 2019 ; Child , 2020 ) . In these models , in the bottom-up path distribution parameters of higher level stochastic variables are computed as a function of lower level stochastic variables , while in the top-down path distribution parameters of lower level variables are corrected for as a function of higher level variables . Although we do not assume stochasticity of the document latent representations , our encoder or inference model follows the same idea to infer token representations . The proposed framework is agnostic to model architecture . Due to the dominance of transformer models in NLP ( Chen et al. , 2018 ; Zhang et al. , 2020 ; Sun et al. , 2019 ; Martin et al. , 2020 ) and to leverage pre-trained language models ( Liu et al. , 2019 ; Lewis et al. , 2020 ) , we instantiate our framework with a transformer-based model . There is a bottleneck of applying transformers to long documents , because its computational and memory cost has a quadratic dependency on the sequence length . This issue is especially critical for summarization since we are more interested in summarizing long documents since short ones can be quickly read through by humans . To address this issue , a large amount of prior works have been devoted to developing efficient transformers with sub-quadratic complexity . They approach this problem with kernel-based methods ( Katharopoulos et al. , 2020 ; Choromanski et al. , 2020 ) , by low-rank approximation to the attention matrix ( Wang et al. , 2020 ) , by synthesizing the attention weights ( Tay et al. , 2021 ) , or by designing contentindependent ( Child et al. , 2019 ; Beltagy et al. , 2020 ; Ainslie et al. , 2020 ; Zaheer et al. , 2020 ) or content-dependent sparse attention mechanisms ( Kitaev et al. , 2020 ; Roy et al. , 2021 ; Wang et al. , 2021 ) . Our framework provides a natural way to diminish this quadratic complexity issue . In the bottomup inference , we use local self-attention where each token only attends tokens within a local fixedlength window , and thus the complexity does not grow as a function of the input sequence length . The top-down correction for the token representations enable them to capture long-range context , reducing the limitation of local attention . Furthermore , in contrast to most prior efficient transformers that are incompatible with pre-training language models , our framework is flexible for leveraging any pre-trained encoder-decoder models such as BART ( Lewis et al. , 2020 ) , T5 ( Raffel et al. , 2020 ) . We call the transformer-based model following the proposed framework as top-down transformer , to emphasize the importance of the top-down inference . We evaluate the top-down transformer on a set of distinct summarization benchmarks . These benchmarks cover documents from a variety of domains , including news articles and scientific , conversational , and narrative documents , and of various lengths ranging from hundreds of words ( e.g. , a news article ) , several thousands to over ten thousands of words ( e.g. , a scientific paper , a book chapter ) , to even over hundred thousands of words ( e.g. , an entire book ) . On short documents , models following our framework achieves on-par or better summarization performance than models with full self-attention , and are more computeand memory-efficient . Across all long document datasets , our models achieve state-of-the-art performance . In the end , we show that our model is able to summarize a whole book . Compared to a concurrent work ( Wu et al. , 2021 ) using GPT-3 and requiring humans to extensively label data , our model achieves competitive performance with 380 times less parameters and a small amount of publicly available data . The diverse and strong empirical results support the effectiveness and wide applicability of the proposed model . 2 METHODS . Figure 1 gives a graphical overview of the top-down transformer , instantiating the proposed framework . We introduce its details in this section . Suppose a document has N tokens , t = { ti } Ni=1 . In our method , token representations are inferred by combining top-down and bottom-up inference . This leads to effective and efficient inference for token representations . They are then attended by a decoder to generate a summary , as in a regular encoder-decoder transformer . 2.1 BOTTOM-UP INFERENCE . In the bottom-up inference , contextual embeddings of the tokens , { ei | ei ∈ Rd } Ni=1 , are computed with N1 layers of local self-attention . In particular , each token ti only attends to nearby tokens within a window of size of w. The complexity is hence O ( Nw ) , in contrast to O ( N2 ) for a full self-attention models . Please see Supplementary B.1 for more details . 2.2 TOP-DOWN INFERENCE . The efficiency with local self-attention in the bottom-up inference nevertheless comes with a limitation , that is , each ei only captures the context within a local window instead of that of the whole document . To mitigate this issue , we propose a top-down inference for token representations . Consider a two-level multi-scale latent structure for a document . The low level consists of token representations , { ei } Ni=1 , computed by the bottom-up inference . The top level consists of units at a coarser level . It is affordable to apply full self-attention at the top level due to its coarser granularity , allowing these top-level units to capture global document context . In our work , the self-attention mechanism for the top-level representations is simply the original multi-head self-attention proposed in Vaswani et al . ( 2017 ) . Readers are referred to Vaswani et al . ( 2017 ) for details . Denote the top level representations after self-attention update as { sj | sj ∈ Rd } Mj=1 ( see Section 2.3 for details on top-level representation initialization methods ) . We can then update the bottom-upinferred token representations with the top-level representations . This is achieved withN3 top-down inference layers , as illustrated by the middle panel in Figure 1 . Each layer contains three transformations on { ei } : ( 1 ) token self-attention , ( 2 ) token-segment cross-attention , ( 3 ) feed-forward . ( 1 ) and ( 3 ) are the same as those in the bottom-up inference layers or regular self-attention layer with local attention . ( 2 ) implementing the cross-attention between the top and bottom levels is the critical operation . In particular , each ei is updated with cross-attention , ẽi = ei + LayerNorm ( M∑ j=1 αijfv ( sj ) ) , αij = exp ( fq ( ei ) T fk ( sj ) ) √ d ∑M l=1 exp ( fq ( ei ) T fk ( sl ) ) ( 1 ) where fq , fk , and fv indicate query , key , and value linear mappings , respectively . For notional clarity , Equation 1 only illustrates the case with a single attention head . In practice , we use multiheads . The cross-attention operation injects global contextual information into bottom-up-inferred token representations , ei , and yields global-context-aware token representations , ẽi , conditioning on which a summary can be generated by a decoder . To instantiate the top-down inference , we need to make two choices : ( 1 ) the number of top-levels above the token level and ( 2 ) the unit representation for each top-level . We choose to use one top level since it is sufficiently coarser to apply full self-attention for a wide range of long document benchmarks we experimented on . A natural choice for top level units is sentence , paragraph , and chapter , depending on the number top level considered . Such a choice however might lead to complicated implementations and non-scalability due to the varying length of these units . We hence choose a simpler approach , where the top level consists of fixed-length segments of the documents . While we use a single top level , multiple top levels can be simply achieved with segments with increasingly coarser granularity . In the top-down inference , segment-level self-attention has a complexity of O ( M2 ) , and tokensegment cross-attention has a complexity of O ( NM ) . Thus , together with bottom-up inference , the complexity is O ( Nw +M2 + NM ) . In practice , we use relatively small w ( window size ) and M ( number of segments ) .

Summary: The paper proposes a new model architecture for abstractive text summarization. The framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency at a coarser time scale, and the bottom token level preserves the detail. The proposed model shows some advantage in both capability in capturing global and local context  and being computationally efficient.  The model is validated on many datasets and it beats almost all previous model.