Long Document Summarization with Top-Down and Bottom-Up Representation Inference
Authors: Bo Pang
Erik Nijkamp
Wojciech Maciej Kryscinski
Silvio Savarese
Yingbo Zhou
Caiming Xiong
Venue: ICLR 2022 Submitted (2022)
Keywords: top-down inference
bottom-up inference
long document summarization

Abstract: Text summarization aims to condense long documents and retain key information. Critical to the success of a summarization model is the faithful inference of latent representations of words or tokens in the source documents. Most recent models infer the latent representations with a transformer encoder, which is purely bottom-up. Also, self-attention-based inference models face the challenge of quadratic complexity with respect to sequence length. We propose a principled inference framework to improve summarization models on these two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency at a coarser time scale and the bottom token level preserves the details. Critically, this hierarchical structure enables token representations to be updated in both a bottom-up and top-down manner. In the bottom-up pass, token representations are inferred with local self-attention to leverage its efficiency. Top-down correction is then applied to allow tokens to capture long-range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets, including narrative, conversational, scientific documents and news. Our model achieves (1) competitive or better performance on short documents with higher memory and compute efficiency, compared to full attention transformers, and (2) state-of--the-art performance on a wide range of long document summarization benchmarks, compared to recent efficient transformers. We also show that our model can summarize an entire book and achieve competitive performance using $0.27\%$ parameters (464M vs. 175B) and much less training data, compared to a recent GPT-3-based model. These results indicate the general applicability and benefits of the proposed framework. 

Under review as a conference paper at ICLR 2022
LONG DOCUMENT SUMMARIZATION WITH TOP-DOWN AND BOTTOM-UP INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Text summarization aims condense long documents and retain key informa- tion. Critical to the success of summarization model is the faithful inference of latent representations of words or tokens in the source documents. Most re- cent models infer the latent representations with transformer encoder, which is purely bottom-up. Also, self-attention-baase inference models face the challenge of quadratic complexity with respect to sequence length. We propose principled inference framework to improve summarization models on these two aspects. Our framework assumes hierarchical latent structure of document where the top- level captures the long range dependency at a coarser time scale and the bottom token level preserves the details. Critically, this hierarchical structure enables to- ken representations to be updated in both bottom-up and top-down manner. In the bottom-up pass, token representations are inferred with local self-attention to leverage its efficiency. Top-down correction is then applied to allow tokens to capture long-range dependency. We demonstrate the effectiveness of the pro- posed framework on a diverse set of summarization datasets, including narrative, conversational, scientific documents and news. Our model achieves (1) competi- tive or better performance on short documents with higher memory and compute efficiency, compared to full attention transformers, and (2) state-of-the-ar perfor- mance on wide range of long document summarization benchmarks, compared to recent efficient transformers. We also show that our model can summarize an entire book and achieve competitive performance using 0.27% parameters (464M vs. 175B) and much less training data, compared to a recent GPT-3-based model. These results indicate the general applicability and benefits of the proposed frame- work.
INTRODUCTION
Text summarization involves compressing document and preserving key content and meaning. can be done in either extractive abstractive manner. While extractive summarization model extracts salient fragments (c.g., words, sentences) from the source document to form summary, an abstractive summarization system aims to generate a semantically coherent and linguistically flu- ent summary by conditioning on the document. The abstractive approach aligns better with how a human does summarization and generally performs better than extractive models in recent works (Pi- lault et al. 2020 Zhang et al.) 2020). We thus focuses on abstractive summarization.
The dominant approach for abstractive summarization to a Seq2Seq model (Sutskever al. 2014) with an encoder-decoder architecture instantiated with either RNNs (Hochreiter & Schmidhu- ber 1997) or, more recently, transformers (Vaswani al., 2017). In such model, an encoder infers the latent representations of observed tokens (words or subwords) in the document, conditioning on which a decoder generates summary. This paper studies the problem of how infer good latent representations, which in turn would improve summarization. We propose framework which (1) assumes a multi-scale latent structure of a document and (2) synergizes bottom-up inference with top-down inference. In multi-scale structure, high-level variables (like those representing sen- tences, segments) model the document at a coarser time-scale and abstract away details, and are suitable for capturing long range dependency of the document; in contrast, low-level variables (like those representing tokens) preserves details, and prevent the summary from losing key details. In framework, the summary is generated by conditioning on token representations (low-level vari-

---

Under review as a conference paper at ICLR 2022
BOTTOM-UP INFERENCE
TOP-DOWN INFERENCE
Top Level
oseetaation
Final Tokan Representation
XN4
xN2
xN3
Full Self- Seff-Attention
Feedforward
Top-Level Representation Intialization
Token-Segment Cross-Attention
Source- Target,
Gross-Attention
Decoder
Pooling
K
Bottom-Up-infered Token esentiiinn
Token Local Self-Attention
Local Self-Attention
Bottom--Up-intereee Token Representotion
Outputs
Inputs
(Shifted Right)
Copy
An overview of the lop-down transformer. Suppose a document with tokens the inputs to the model, as shown the bottom left. The bottom-up inference is achieved with local self-attention (Ni layers) as shown in the left panel. To initialize the top-level representations, tations with either equal weights or adaptive weights (see Section 2.3 details). Top-level representations are then updated with full self-attention (N2 layers) to capture global context. They are then used to update bottom- up-inferred token representations, accounting for the top-down update for token representations, as shown in the middle panel. The final token representations attended by the deceder to generate summary. Note that inference is used in the of statistical inference for latent variables and does imply training.
Figure 1:
pool bottom-up cnoommup-ifferred inferred token represen-
ables), similar to recent abstractive summarization models (Zhang et al. 2020 Zaheer et al. 2020 Beltagy al. 2020). There however critical difference. In our framework, token representations are first bottom-up inferred and then top-down updated with high level representations, hence ren- dering low-level representations aware of long range information. We hypothesize that the proposed inference approach would improve summarization.
Multi-level models have been widely studied in modeling for images (Sønderby et al., 2016). speech (Mehri et al., (216). and language (Chung et al. 2016). Prior summarization works (Cheng Laapata 2016 Nallapati et al. 2016; Zhang et al. 2019 Xu et al. 2020) have also explored hierar- chical models. But they mostly focus extractive summarization and follow a bottom-up inference approach. They pool information in words or sub-words to form sentence representations, based on which classification is done to make extraction decision.
In comparison, our framework combines bottom-up and top-down inference. This draws direct inspiration from line of work which examines variational inference for hierarchical top-down gen- erative models (Sønderby et al.) 2016), Maalçe et al.) 2019 Child 2020). these models, in the bottom-up path distribution parameters of higher level stochastic variables computed as a func- tion of lower level stochastic variables, while in the top-down path distribution parameters of lower level variables are corrected for function of higher level variables. Although do not stochasticity of the document latent representations, our encoder or inference model follows the same idea infer token representations.
The proposed framework is agnostic to model architecture. Due to the dominance of transformer models in NLP (Chen et al. 2018, Zhang et al. 2020, Sun et al., 2019, Martin et al.) 2020) and to leverage pre-trained language models (Liuu et al.) 2019 Lewis et al. 2020), we instantiate our Lewis
framework with transformer-based model. There is bottleneck of applying transformers to long documents, because its computational and memory cost has a quadratic dependency length. This issue is especially critical for summarization since we are more interested in summa- rizing long documents since short ones be quickly read through by humans. To address this issue, large amount of prior works have been devoted to developing efficient transformers with sub-quadratic complexity. They approach this problem with kernel-based methods (Katharopoulos
the sequence
2

---

Under review as a conference paper at ICLR 2022
et al.
2020
Choromanski al. 2020), by low-rank approximation to the attention matrix (Wang
et al. 2020). by synthesizing the attention weights (Tay et al. (021), or by designing content-
independent (Child et al., 2019 Beltagy et al. 2020] Ainslie et al. 2020; Zaheer et al. (0200 or
content-dependent sparse attention mechanisms (Kitaev et al. 2020 Roy et al. 2021 Wang et al. 2021].
Our framework provides a natural way to diminish this quadratic complexity issue. In the bottom- up inference, we use local self-attention where each token only attends tokens within local fixed- length window, and thus the complexity does not grow as a function of the input sequence length. The top-down correction for the token representations enable them to capture long-range context, re- ducing the limitation of local attention. Furthermore, in contrast to most prior efficient transformers that are incompatible with pre-training language models, our framework flexible for leveraging any pre-trained encoder-decoder models such as BART (Lewis et al. 2020), T5 (Raffel et al., 2020).
We call the transformer-based model following the proposed framework as top-down transformer, to emphasize the importance of the top-down inference. We evaluate the top-down transformer on a set of distinct summarization benchmarks. These benchmarks cover documents from a variety of domains, including news articles and scientífic, conversational, and narrative documents, and of various lengths ranging from hundreds of words (e.g., a news article), several thousands to over ten thousands of words (e.g., scientific paper, book chapter), to even over hundred thousands of words (e.g., an entire book). On short documents, models following our framework achieves on-par better summarization performance than models with full self-attention, and more compute- and memory-efficient. Across all long document datasets, our models achieve state-of-the-ar per- formance. In the end, we show that our model is able summarize a whole book. Compared to a concurrent work (Wu et al. (02T) using GPT-3 and requiring humans to extensively label data, our model achieves competitive performance with 380 times less parameters and a small amount of publicly available data. The diverse and strong empirical results support the effectiveness and wide applicability of the proposed model.
METHODS
Figure[give gives graphical overview of the top-down transformer, instantiating the proposed frame- work. We introduce its details in this section. Suppose a document has N tokens, t = {ti] In our method, token representations are inferred by combining top-down and bottom-up inference. This leads to effective and efficient inference for token representations. They are then attended by decoder to generate a summary, as in regular encoder-decoder transformer.
2.1 BOTTOM- UP INFERENCE
In the bottom-up inference, contextual embeddings of the tokens, {e; l E R are computed with N₁ layers of local self-attention. In particular, each token t only attends to nearby tokens within window of size of w. The complexity is hence O(Nw), in contrast to O(N2) for full self-attention models. Please see Supplementary B. for more details.
2.2 TOP-DowN INFERENCE
The efficiency with local self-attention in the bottom-up inference nevertheless comes with limi- tation, that is, each e; only captures the context within a local window instead of that of the whole document. To mitigate this issue, we propose a top-down inference token representations.
Consider two-level multi-scale latent structure for a document. The low level consists of token representations, (e.) computed by the bottom-up inference. The top level consists units at a coarser level. It is affordable to apply full self-attention at the top level due to its coarser granularity, allowing these top-level units to capture global document context. In our work, the self-attention mechanism for the top-level representations reppeeeatt is simply the original multi-head self-attention proposed in Vaswani et al. (2017). Readers are referred to Vaswani et al. (2017) for details.
Denote the top level representations after self-attention update {s, sy R R (see Section 2.3 for details on top-level representation initialization methods). We can then update the bottom-up- inferred token representations with the top-level representations. This is achieved with Ng top-down
M
3

---

Under review as a conference paper at ICLR 2022
inference layers, as illustrated by the middle panel in Figure Each layer contains three transfor- mations on }er}: (1) token self-attention, (2) segment cross-attention (3) weed-forarrd. (1) and (3) are the same those in the bottom-up inference layers or regular self-attention layer with local attention. (2) implementing the cross-attention between the top and bottom levels is the critical operation. In partieular, each e; is updated with cross-attention,
exp (fg(e)) é, = LayerNorm E asy (o)g)) aij VdΣ 1==1 exp (u((e))
(1)
where fg. fk. and fo indicate query, key, and value linear mappings, respectively. For notional clarity, Equation only illustrates the case with a single attention head. In practice, we use multi- heads. The cross-attention operation injects global contextual information into bottom-up-innerred token representations, e;, and yields global-context-avaan token representations, é₁ conditioning on which a summary can be generated by decoder.
To instantiate the top-down inference, we need to make two choices: (1) the number of top-levels above the token level and (2) the unit representation for each top-level. We choose to one top level since it is sufficiently coarser to apply full self-attention for a wide range of long document benchmarks we experimented on. A natural choice for top level units is sentence, paragraph, and chapter, depending on the number top level considered. Such choice however might lead to com- plicated implementations and non-scalability due the varying length of these units. We hence choose a simpler approach, where the top level consists of fixed-length segments of the documents. While we use a single top level, multiple top levels can be simply achieved with segments with increasingly coarser granularity.
In the top-down inference, segment-level self-attention has a complexity of O(M²), and token- segment coss-attention has complexity of O(N Thus, together with bottom-up inference, the complexity is O(Nw + M² + NM.. In practice, we use relatively small w (window size) and M (number of segments).
2.3 POOLING METHODS
As aforementioned, we use a single top level, consisting of fixed-length segments, in the current work. The segment representations are initialized by pooling token representations. Following the notation above, suppose a document is divided into M segments, and the embedding of the jth segment is initialized as,
Σ Palixdtn
where k is the kernel size and d is the stride. Pn is the weight for the nth token. We introduce two approaches to compute the weights. The first method is average pooling (AvgPool) and hence p.. = T, which is simple and convenient. define the importance of each token assign adaptive weights (AdaPool). Particularly, we learn an importance tagger with labels constructed with the reference summaries, which involves three steps:
the second approach, we leverage the reference summary to
1. construct training labels for the importance tagger: (1) word lemmatization for document and reference words; (2) label document word important if it appears the reference word list and is non-stopword
2. train top-down transformer encoder with constructed labels as the importance tagger
3. train the summarization model with oracle weights (i.e., constructed labels from Step 1.) and test with the adaptive importance weight assigned by the learned tagger
In our experiments, we also used OracleAdaPool where the weights are obtained from Step I with the reference summaries. Note that if {p. Ji does not form valid probability distribution, sy can be computed with a normalized weight distribution within each pooling window as follows,
*
exp(pn)ejxdas Σm=1 exp(pn)
(0)
{5) J==1 are updated with self attentotennon, yielding {s, 1 which are then used in top-down infer-
(0) J.M
1M
ence for token representations, as discussed in Section 2.2
4

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
We thoroughly evaluate the proposed framework on distinct summarization datasets. See Table for a summary of datasets used in the current work. Our model is first evaluated on two standard long document summarization benchmarks, PubMed and arXiv (Cohan et al., 2018). It outperforms various efficient transformers and other approaches and achieves state-of-the-art performance. Al- though we focus on long document summarization, models under our framework is also applicable shorter documents, We test our model on CNN- Mii-millymail (See et al. 2017). the most widely used short summarization dataset. Compared to a full self-attention model, our model achieves competi- tive or better performance but more memory- and compute-efficien. Recently, more challenging benchmark, SummScreen (Chen et al.) 2021). D. is proposed, where summarization systems need to summarize TV show scripts. These documents convey plot events often indirectly and implicitly in dialogues, in contrast to news and scientific articles where statements follow logical order and facts are offered explicitly. Moreover, a typical episode contains multiple subplots that proceed in paral- lel. Solving this benchmark thus requires system to draw information from utterances spreading out through the entirety of the input and inte them to a concise description. Our model outper- forms strong baselines this challenging benchmark by significant margin. Another challenging dataset, BookSum (Kryšciñski et al. 2021). is also recently released. It covers books from the lit- erature domain, including stories, plays, and novels. Similar ScreenSum, requires integrating plot events from indirectly expressed descriptions. A further challenge is to process long-form texts up to hundreds of pages or over 100,000 words. A model under our framework does well on this challenge, achieving competitive or superior performance compared to a concurrent work (Wu et al., 202T) using GPT-3, While the GPT-3-based model has 175 billion parameters and requires human labelers to extensively write summaries and provide reward information, our model with 464 mil- lion parameters is 380 times smaller and merely requires training on relatively minimal data. These results suggest our framework is generally effectively for documents of various lengths, domains.
Dataset
# Docs.
# Input Words
# Summary Words
Domain
PubMed
I33K
3,224
214
Scientific
arXiv
215K
6,913
292
Scientific
TVMegaSite
22.5K
6,420
380
Conversational
ForeverDreaming
4.3K
7,605
113
Conversational
BookSum- Bartsuu---hhht---Le Lhavter-Level
12K
5,102
505
Narrative
436
112,885
1,167
Narrative
CNN-DM
311K
906
News
Table 1:
Summarization Datasets. It shows the total number documents, the average number of input words, the average number summary words, and the domain for each daiaset.
We use the same encoder-decoder architecture for all datasets. The encoder has 8 bottom-up in- ference layers and 4 top-down inference layers for tokens, and 2 self-attention layers for segments. The decoder has 12 layers. The encoder layers for tokens (12 layers) and the decoder layers all initialized from BART (Lewis et al.) (020) except the parameters for token-segment cross-attention in the top-down inference layers, which are randomly initialized. The self-attention parameters for segments are also randomly initialized, The window size is 1024 unless otherwise specified. Our settings closely follow Longformer (Beltagy al. 2020) which has 12 layers for the encoder and decoder, is initialized from BART, and uses local window size of 1024. Thus, comparison with Longformer is test of the effect of top-down correction for token representations. PubMed, arXiv, and CNN-DaillMail are obtained from Huggingface Datasets [ SummScreen and BookSum are provided by the authors. Standard train/validation/tesl splits, provided by either Huggingface or the dataset authors, are used for all datasets. Model performance is evaluated with ROUGE scores (Lin) 2004). Reported performance is based on the checkpoint with the best validation R-2 score. Sum- mary samples for each dataset generated by models provided in the appendix.
3.1 SCIENTIFIC DOCUMENTS
We first test the effectiveness of our framework on two widely used datasets based on scientific documents, PubMed and arXiv. They consists of long documents of length ranging from several
https://tuggingfaccccoaassssss
5

---

Under review as a conference paper at ICLR 2022
thousands of words to over ten thousands words. Each document in PubMed is scientific article, collected from PubMed.com, and the reference summary is the associated abstract. Documents in arXiv are collected from arxiv.org. Three variants of our model with various pooling weights presented. AvgPool, AdaPool, and Adaole in Tablee2indican iindiatee average pooling, pooling with adaptive weights, pooling with adaptive weights determined by references, respectively.
PubMed
arXiv
R-1
R-2
R-L
R-1
R-2
R-L
Pegasus (568M)
44.21
16.95
38.83
44.21
16.95
38.83
Dancer
46.34
19.97
42.42
45.01
17.60
40.56
TLM-I+E
42.13
16.27
39.21
41.62
14.69
38.03
SSN-DM
46.73
21.00
34.10
44.90
19.06
32.77
BigBird (577M)
46.32
20.65
42.33
46.63
19.02
41.77
Longformer (460M)
46.97
20.23
42.88
46.63
19.62
41.83
LSH
48.12
21.06
42.72
Top Down Transformer (AvgPool) (464M)
48.34
21.40
44.22
48.67
20.70
43,91
Ours
Top Down Transformer (AdaPool) (464M)
51.05
23.26
46.47
50.95
21.93
45.61
Top Down Transtormer (OracleAdaPool)
55.15
26.55
50.25
64.16
33.39
56.88
Table 2: Results Scientitic Articles. performance (not relying on oracle) bold. second best undertined.
The experiment results are displayed in Table 2 Pegasus (Zhang et al., 2020) is pretrained on a large-scale of dataset with a pretraining objective specifically designed for summarization. uses full self-attention encoder and thus has truncate the source document due the quadratic memory complexity. The summarization aotirariiaaiioooieeeeeedd large-scale pre-training makes it a strong baseline. Dancer (Gidiotis & Tsoumakas (2020) takes divide-and-conquer approach in which the summary is divided into sections and each section is paired to the appropriate section of the document and the model is trained on short sequences and has low memory requirement. This is straightforward approach achieving strong performance.
TLM-I+E (Pilault et al. 2020) first extracts salient sentences and then uses a GPT-style model to generate a summary by conditioning on the introduction section and extracted sentences (instead of the whole document), thus reducing memory requirement. SSN-DM (Cui & Hu 2021) 1) is an extrac- tive model and uses sliding encoder to process segments of a document and a memory module to capture autoregressive dependency between segments. These two models bear similarities to our model in that they use multi-scale structure. The extracted only salient sentences in TLM-I+E can be considered a representation of the document at a coarser granularity since salient information is retained. Instead of keeping the coarser representations in the latent space, TLM-I+E reads out them to the observed word space. In SSN-DM, the fixed-size memory module pooling information from each segments also be considered a high level representation of the document. Despite these similarities, our model, following a principled framework to synergize bottom-up and top-down inference, clearly outperforms these prior models.
BigBird (Zaheer et al. 2020). Longformer (Beltagy et al. 2020 and LSH (Kitaev et al. 2020 Huang et al.) 2021) efficient transformers. BigBird based Pegasus pre-training combines lo- cal attention, random attention tokens, and global attention tokens. LSH uses content-dependen sparse attention based on local sensitivity hashing. Longformer is closely related our models. uses the same local attention as in our bottom-uy inference except it has an extra [CLS] token which global attention token. Longformer is also initialized from BART, same as ours. The only dif- ference is that our models infer token representations with both top-down and bottom-up inference, contrary to pure bottom-up inference Longformer. The clear performance improvement over Longformer and other efficient transformers indicates the effectiveness of the synergy of bottom-up and top-down inference.
3.2 SHORT DOCUMENTS
To demonstrate the general applicability of the proposed framework, we show its efficiency and effectiveness on short document summarization and compare to full self-attention inference model. We hypothesize that although the bottom-up inference uses local self-attentin (for efficiency), the top-down correction would enable the effectiveness our inference and hence lead to competitive or better summarization performance.

---

Under review as a conference paper at ICLR 2022
CNN-DailyMail
R-1
R-2
R-L
BART (Reported)
44.15
21.28
40.90
BART (Re-eval)
43.93
20.81
40.79
Top Down Transformer (AvgPool)
44.32
21.03
41.40
Ours
Top Down Transformer (AdaPool)
44.85
21.31
41.15
Top Down Transtormer (OracleAdaPool)
63.87
38.42
39.10
Table 3: Results on CNN Mail Best performunce (nol relying on oracle) in bold, and the secund best is undertined.
Our model parameters are initialized from BART. Hence, BART with full self-attention forms a natural baseline, allowing for direct comparison. In the bottom-up inference, the local attention window size is 256. As shown in Table 3 models under our framework achieve slightly better performance, especially in terms of R-1 and R-L. than BART. It confirms our hypothesis that synergy of bottom-up inference with local attention and top-down inference with global attention is effective and achieves on-par or better performance as full self-attention.
3.3 SUMMSCREEN
TVMegaSite
Forever Dreaming
R-2
R-L
R-1
R-2
R-L
Extractive Oracle
49.0
11.6
46.9
38.8
11.5
33.9
Longformer
42.9
11.9
41.6
25.9
4.2
23.8
Hybrid (BART Content Selection)
38.8
10.2
36.9
25.3
3.9
23.1
Hybrid (BART + Oracle Content Selection)
42.1
11.9
40.9
26.4
Top Down Transtormer (AvgPool)
49.30
14.35
47.45
35.84
8.86
30.62
Ours
Top Down Transformer (AdaPool)
51.02
14.66
49.01
36.84
9.19
31.12
Top Down Transformer (OracleAdaPool)
53.55
15.63
51.29
39.54
10.08
33.59
Table 4:
Results SummScreen. Best perfommance relying oracle) is hold, and the second best is underlined.
Scientific and news articles often require that facts are offered explicitly and statements follow a logical order, which might allow summarization models to exploit layout and stylistic biases. We next test the proposed framework on a more challenging dataset, SummScreen, which requires model draw and integrate information from indirect expressions across wide range of the docu- ment. SummScreen (Chen et al.) (222)) provides two datasets, TVMegaSite and ForeverDreaming, collecting from two different TV show transcript websites. Each document is the transcript of TV show episode and the summary is an associated recap.
Table summarizes the results. Extractive oracle an extractive method by extracting nearest neighbors based on Rouge scores. Longformer is an abstractive method and takes the whole docu- ment as input. Hybrid models first select salient sentences and then input them to BART. Our models outperform these strong baselines and even achieves comparable or superior performance than those having access to oracle information.
3.4 BOOKSUM
BookSum (Krššcinski et al. 2021) is another challenging dataset, consisting of books from the liter- ature domain including stories, plays and novels. includes examples on three levels of granularity with increasing difficulty: (1) paragraph-level with inputs with hundreds of words, (2) chapter- level, with inputs with several thousands or over ten thousands of words, (3) book-level. with inputs span- ning up to hundreds of pages and over hundred thousands of words. The chapter-level examples have comparable lengths to other popular long-form summarization datasets such as PubMed, arXiv. We first test our models the chapter level. The book-level summarization is extremely challenging. First, the number of examples (313 books) limited. Second, a book too long to fit in current models. We train model in curriculum and recursive way to address the two issues.

---

Under review as a conference paper at ICLR 2022
BookSum Chapter Level
R-1
R-2
R-L
Extractive Oracle
42.68
9.66
21.33
BART B4RT(400M)
37.09
8.23
15.37
(738M)
37.38
8.42
16.77
Pegasus (568M)
36.17
7.79
16.09
Longformer (460M)
32.84
7.45
14.59
BigBird (577M)
31.78
6.50
14.17
Top Down Transformer (AvgPool) (464M)
37.99
9.10
18.02
Ours
Top Down Transformer (AdaPool) (464M)
38.34
9.19
18.08
Top Down Transtormer (OraclcAdaPool)
41.10
9.49
19.19
Table
Results on BookSum Chapter Level. Best performance (not relying on oraclo) is bold, and the sccond best is underlined.
3.4.1 CHAPTER LEVEL
displays the results. Kryšciński et al. (021) takes a divide-and approach to summa- rize chapters. They finetune BART, 15, and Pegasus on the paragraph level data and the chapter sum- mary is obtained by concatenating the paragraph summary. This might miss the intra-paragraph con- text. Our models directly summarize the whole chapters and outperform these divide-and-conuee models. Efficient transformers, Longformer and BigBird, also able to take in the whole chapters as inputs. But these bottom-up up approaches clearly underperform our models.
3.4.2 BOOK LEVEL.
We first train top-down transformer on the chapter-level data and then fine-tune it on the book-level data. The inputs to the book-level model are (1) the concatenated chapter reference summaries in training or (2) the concatenated chapter summaries generated by the chapter-level model in testing- The chapter-to-book curriculum training is to mitigate the scarcity of book-level data. The recursive summarization of chapters and then books can be considered abstractive content selection applied to book data, and is used to address the extremely long length of books.
BookSum Book Level
R-1
R-2
R-L
Extractive Oracle
46.62
9.17
18.31
BART
29.97
6.02
10.97
TS
39.46
7.69
13.77
Pegasus
35.29
6.79
12.71
175B full tree RI.
41.51
10.46
16,88
175B first subtree RL
43.19
10.63
17.10
6B full tree RL
36.79
7.22
14.84
Top Down Transformer (464M)
44.19
10.89
16.13
Table 6: Results on BookSum Book Best performance (not relying on oracle) bold, second underlined.
T the book-level results. The middle section shows the performance for the models with the dlwide-and-congeee approach (Kryšcinski al. (021). same those for the chapter-level data. A concurrent work (Wu et al.) 2021) based on GPT-3 with reinforcement learning (RL) also attempts summarize books. Their method shares similarity with
in that they decompose books into shorter sequences and train the model and summarize the text segments recursively. There are four major differences between our approach and theirs. First, our model has only 464 million parameters and is 380 times smaller than GPT-3 with 175 billion parameters. Second, we train our model with the limited and publicly available data from BookSum, while Wu al. (202T) requires human labelers to write summaries and give preference, which is highly costly. Third, our model has lower complexity, allowing to takes longer input. Thus, we only need decompose the book one time (into chapters), in contrast to multiple recursive decomposition steps. Multiple recursive summarization steps is prone to accumulating error.. Forth, GPT-3 uses bottom-up inference to infer token representations, in contrast to the synergy of bottom-up and top-down inference in approach, which we believe leads to better representation inference. The last two differences might account for our competitive performance using much smaller model and less data.
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK
Summarization Models Prior works have proposed extractive models (Nallapati et al. 2017 Cui
& Hu 2021), abstractive models (Nallapati et al. 2016; Zhang al. 2020). and hybrid models combining extractive and abstractive methods (Gehrmann et al. 2018: Pilault et al. 2020). for text summarization. Although our model mostly follows the abstractive approach, it also has connections to the hybrid models. These models usually first extract salient sentences from the source document and then summarize the extracted sentences with an abstractive model. Extracted sentences can be viewed high level representation of the document, although it is the observed space but not in the latent space as in our framework. A continuous representations in the latent space facilities end-to-end learning. Moreover, assigning importance weight with the importance tagger in our method resembles an extractive step in a hybrid model, and thus top down transformer with learned importance tagger can be considered a hybrid model.
Inference for Hierarchical Latent Variable Model Our work draws inspiration from the latent variable inference for hierarchical top-down generative models. To faithfully infer multi-layer la- tent variables needs to account for the dependency between them. MCMC approaches Nijkamp et al. (2020) naturally accounts for such dependency. Amortized inference Sønderby et al. (2016); Maaløe et al. (2019); Child (2020) makes a special design to capture the multi-layer dependency. In partic- ular, in bottom-up path, the parameters of the distribution of higher level variables are computed as a function of the lower level variables; in top-down path, the parameters of the distribution of lower level variables are corrected as function of the higher level variables.
Siference/Sonderby
Efficient Transformers Despite the effectiiveness of transformers on a variety of tasks, its quadratic complexity with respect to the sequence length has limited its application to problems with long sequences. A large amount of works have attempted address this limitation. A major line of work focuses on designing various sparse attention mechanisms. These works can be roughly categorized into two groups, depending on whether the sparsity pattern is content-dependent (Ki- taev et al., 2020; Roy et al.) 2021 Wang et al., (021)) or content-independent (Child et al.) 2019 Beltagy et al. [2000] Ainslie et al.) 2020) Zaheer et al.) 2020). Our work is mostly related to content- independent sparse attetntion. A main assumption of content-independent sparse attention is that the context temporally and/or spatially proximate to the query token is more important, which is intuitively sensible and supported by empirical attention analysis (Child et al., 2019). Thus, com- mon and basic sparse attention pattern is local attention, where each query token only attends to neighborhood within fixed temporal and/or spatial window. While this reduces the complexity to be linear, a model with only local attention cannot model long-range dependency. Prior works combine local attention with other attention patterns with wider global receptive field such as dilated attention, random attention tokens, and global attention tokens (Beltagy et al. [0200] Zaheer et al. 2020). Our models also use local attention for its efficiency and leverage top-down inference to enable global-context awareness.
5 CONCLUSION
In this work, we propose a principled inference framework to improve latent representation infer- ence for summarization models. It assumes hierarchical latent structure of document where the top-level captures the long range dependency at coarser granularity and the bottom token level preserves the details. We leverage this hierarchical structure and synergize bottom-up inference with top-down inference to improve token representation inference. In the bottom-up pass, token representations are inferred with local self-attention to exploit its efficiency. Top-down correction is then applied to allow tokens to capture long-range dependency. We demonstrate the effectiveness of the proposed framework on wide range of summarization datasets, including narrative, con- versational, scientific documents and news. Our model achieves (I) comparable or superior perfor- mance on short documents with higher memory and compute efficiency, compared to full attention transformers, (2) state-of-the-art performance
wide range of long document summarization benchmarks, compared to recent efficient transformers, and (3) competitive performance on sum- marizing whole books using 0.27% parameters and much less training data, compared to recent GPT-3-based model. These results indicate the general applicability and benefits of the proposed framework.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Ainslic, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. ETC: Encoding long and struc-
tured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pp. 268-284, Online, November 2020. Associa- tion for Computational Linguistics. doi: 10. 10.18653/112222.mmmmpp. //aclanthology htwpollooyy.oo//eeeeeo wttpo/////// emn 1a..ppmmmin1999
URL https:
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint #xXiv2004.005550 2020.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Fos- ter, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszko- reit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 76-86, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. N0.18653/v1/P18 1008.. URL htRLhhtps://aatt http:p/://cclannooooo..ooroog
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. Summscreen: A dataset for abstrac- tive screenplay summarization. arXiv preprint (rXiv:2104007091, 2021.
Jianpeng Cheng and Mirella Lapata. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 484 494, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/P16- 1046. URL
org/
P16 046
R. Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. ArXiv, abs/1904. 10509, 2019.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2020.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural net- works. arXiv preprint 1rXivv1609.000 704, 2016.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-awe awaaee attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp- 615-621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/NI/ 2097. URL htttst/::/// ac aattho logy. wtggN/8-2C9999
Peng Cui and Le Hu. Sliding selector network with dynamic memory for extractive summarization of long documents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5881-5891, On- line, June 2021. Association for Computational Linguistics. doi: 10. 101118663vv11 1/2021 .aaalllallaaaaa main. 470. URL https: : www.anntolloyyy...ocoooooooooolt org 2021 maaacl-maann 470
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4098-4109, Brussels, Belgium, October-November 2018, Association for Computational Lin- guisties. doi: 10. 1/D1://///11/I/11 1443. URL /RLL/ittps://aaaaitiooot org/ 1443
Alexios Gidiotis and Grigorios Tsoumakas. A divide-and approach to the summarization of long documents. JEEE/ACM Transactions on Audio. Speech, and Language Processing, 28: 3029-3040, 2020.
10

---

Under review as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.naacl-main 12. URL http://:///claantoooooo.ooo -tt.://///.....aaaaaaaaaa.....a nwacl-aaclaaaaama.a
A. Katharopoulos, A. Vyas, N. Pappas, and Fleuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference Machine Learning (ICML), 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Represeniations, 2020.
met/forun Net/Forum?iddIkeee EigKKKHHtE
Wojeiech Kryšciński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguisties. doi: 10. 11111881122200.00 acl-main.70. URL Mttpps://ccaatthloogy/
acl maain.ain... 703
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL httllttts///alatttttttte http:///aaaaattoooeeoee
www.ergg/eec.. 1013
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1662, 2019.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther, Biva: A very deep hierarchy of latent variables for generative modeling. Advances Neural Information Processing Systems, 32:6551-6562. 2019.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de Clergerie, Djamé Seddah, and Benoît Sagot. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 7203-7219, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/vvl 0020.acl-main.66 URL
wwcchhoooyy..oom ogy wwwpoogg/00000 aww.mmaln.a.......... 645
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani. Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv: 2016.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization 060023, 2016.
using eqquence-to-sequence rnns and beyond. arXiv preprint
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-Firsi AAAI Conference on Artificial Intelligence, 2017.
Erik Nijkamp, Bo Pang, Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning multi-layer latent variable model via variational optimization short run meme for approximate inference. In European Conference Computer Vision, pp. 361-378. Springer, 2020.

---

Under review as a conference paper at ICLR 2022
Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. On extractive and abstractive
neural document summarization with transformer language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9308-9319, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.1 1 1 1/0220. attpttpss//aa...nhtoooggp//
emnlp-main.744 748.
emn 1 p n 748
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Lin- guistics, 9:53-68, 2021. doi: 10.1162/tacLa.0033 00353, URL
org/
2021.tacl=
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 1/1188/9////11111999 URL https: wttp:////...//////1999999
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29:3738-3746, 2016.
Chi Sun, Luyao Huang, and Xipeng Qiu. Utilizing BERT for aspect-based sentiment analy- sis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pp. 380-385, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18633/11/1191000 1035. https::: 7/ haclaathoooooy.orr//I1---0000
Uya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-31 3--1112222 12, 2014.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Re- thinking alff-attentoon for transformer models. In International Conference on Machine Learning, pp. 10183-10192. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-6008 2017.
Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. Cluster-former: Clustering- sparse transformer for question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3958- 3968, Online, August 2021. Association for Computational Linguisties. doi: 10. 10.18653/v1/2011. findings-acl.3466
orgpooo0..or//2211..1.1 findings acl. 346
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 0476688000604768, 2020.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 2021.
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, and Ming Zhou. Unsupervised extractive summarization by pre-training hierarchical transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1795, Online, November 2020. Associa- tion for Computational Linguistics. doi: 10. 10.118/3/v1// 10/6333/1/222..11m 12220mmiii/2....miniiinng-g- findi ems-
URL https:
7/acl antaa
emn 1p. 161
12

---

Under review as a conference paper at ICLR 2022
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp. 11328-11339, PMLR, 2020.
Xingxing Zhang, Furu Wei, and Ming Zhou. HIBERT: Document level pre-training of hier- archical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5059-5069, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1499. URL ottess///ecaaattooooo..oo////2/3
13

---

Under review as a conference paper at ICLR 2022
A QUALITATIVE EXAMPLES
PubMed Exumple #1: Reference
a new class of water- soluble cli transfecting agants has been prepared using hirschbingel chemisty and assessed for their ability to act as gene - delivery vectors in vitns in an effont to clucidate the relationship hetween the hydiophhhiiit of the fullerese core, the hydrophhillity the water- wilbbitizing groups, and the eveall clarge
gene delivery expression, several different c6il derivatives charged, noutral chemscal fusctionaliiiss under physooloical conditions, these fullerene deriiztivs were then tested thir ability transfect cells grown milture with dna camyyigg the prozn fhorescent protcin ( glp reporter gene. sttisticallly significant expresiion of glp was aiserved for all forms of the efi derivatives when used as dnn vestors and compared to the diility of niked dea plane tranifect cells. however, efficiett in vitro trnnffcction was only achieved with the two derivatized cld and dodeca-
the cl0
synthesized yield either positively charged. negutively
decivatized c60 in dosc- dependent nanner. incressed levels cellular toxicity were ohserved posiiiely charged verioes relative vectors, structural analyses using dynamtic light semering and optical microscopy offered fasther insights into possible conceltions between the varions derivatized c60 compounds, the cli vector cna complexes, their physccal atmiuute agaregation. charge and their transfection efficiencies. recently. similar ediec00- - based imaging
paunively charged
decivatives, numely,
clu negatively charged and nestril
showed
texicity
compoinds have demoestratedd patential advanced controst agonts magnetic uptake, intracellular transport, and gene expression from dna using eli vectors suggests the possbility developing analogoos gdoecc0 based vectors to serve simultnnonsly as both therapoutic and dingnostic apents.
thus, the sucesssful demonstration intracellular daa
PubMed Exumple #1: Model Output
new class water soluble mranfetccng agents with positivelyy chargat, negatively charged, neutral cheascaa fauctionalities under physiological conditions was prepared using hirschhingel chemistry, transfcction, cyintoiciity and biophy structure / autivity studies were perforsed in an effont to clucsdate the relationship berween the hydrophohicity of the fullerene core. bydroqhilicity and the oveall charge state of the cS0 vectors in delivery/ expression. only two posirively showed efficiest
charged
dna delivery cells, incressed levels of collolte todicity were observed for positively chorged compounds relative to the negatively chargal nestma vectors, and aggregation was the mujor facter that negatively affected the cytntoxic profiles of these c60 vector / dra complexes the uncesssful demansimtion of intracellular dna uptikc and gene expression from dna dna conjugates torme from eS0 compounds suggests the possibilaty of developing analogous gdoec60 vectors to serve as both tiempenii and diagnostic agems.
namely,
amino dedivatzdd c60 dodeca amine darived c60
PubMed Exumple #2: Reference
context: isolated involvement of the appendix in crohn's disease reported be 0.2% 18 S, and usually associated with ileocurcal crchn's 's discase in 25 and 50% % caccal discase. while appendiciis a polient was previously diagnose to have flooceecal and ileocaecal resection, sqrencicactomyy slone when performed for appendicitis a patient with mnnugpected Hlonssocal crohe's diserse could lead postoperative complication including emerrouuiaevas fistulacase report: a young female puieent who underwent appendieectomyy elsew here for acute appendiciiss presented to with persistent enterecuianoou fistula of weeks durstion. she had complained general ill health and occasional alterad bowel habit for 6 months prior to acuie appendiciis presentation, our investigations, inchading a scan, suggested the possibility of ileocasçal crchn's 's disease. she underwent exciston entercnnannous fistula and ileocaecal resection, and histopahooogyy of the resecied specimen confirmed crohn's disease. in the pestopenatvve period she received masaslazine, when list seen 2 years later during her regular follow up. she was fourd to be in good health conillionnusion : the possibility of illocssccll crehn's 's disease should be considered patinnis presenting with unexplained postoperative enterveutancous fistula following appendieectomy, high index clinscal suspicton required to make prompt diagnocis institue appropriate further treatment form of ileocaecal resection.
maraged apprndicactomy
PuhMed Exumple #2: Model Output
introduction: Isolated involvement of the appendix crohn's 's discase reported be 0.2 % 18 5, and usually ussociated with ileocaecal crohn's 'ss diseuse previously diagnosed have ileoceaal ilencecal crohn's may managed
% and 50 of cuecal disease. with appendizetoonis, appendiorciomy alone when performed for acule appendicllectoyy could lead to postoperative complication including entercctinemns fisuula.uuse report young female patient who underwent appendiclectomy elsewhere presented to us with persissent enterococcal fistula of 5 weeks duration. further questionng. she informed that she had presented this hospital with 2 week history abduminal poin, comlined to right lower ahdomen. she prior
uppendicitis in a putient
had complsined genend health and cocasional altered howel habits a computerized tomography stan, suggested the possibility of crohns diseuse. she underwent excision of the enterococuus fistula and resection of the caecum, and histoaahology of the resected spacimen confirmed the diagnosis in the postoperatvve period she reveived mesasalaziee when last seen 2 years later during her regular good health concuiicnccuiionn: apgendiccction alone safe procedure patients with suspected crohn ileocoecal involvement, while
appendectenty prosentation.
investigations including
follow up, she found appendiceal resection will suffice those who present subucutely are diagnosed preoperatively by clinical signs, high clinical suspicion required make a prompt dingnosis.
Table 7: Summary Samples for PubMed
14

---

Under review as a conference paper at ICLR 2022
arXiv Example #1: Reference
we study experimentally the thermaal flactuations of energy impul and dissipation in a harmonic uscillator driven out of equliirium and search for floctuation relations, study trunscent evolution from the equilibcium state. together with non equiitrrum steady stutes. Buctnations relations are obtained expenimenaalyy for both the work und the heut, for the stationzry und transient evolutions. stationzry state iluctuttion thearem verified for the transient iluciuation theurem satisfied
prescriptons the turque, dissipated by the system the of forting. experimental
given the system but not observations on the statistical and dynamical properties of the ticctuation of the angle, we derive anzlytical expeessoes for the probability density function of the work and the heat. we obtain for the fins time unalytic expression of the probubility density function of the heat. agreement between experiments and our modeling excellen.
arXiv Example #1: Model Outgut
we study the ilccuations of energy and dissipation a thermostated harmonic uscillator diven out equilibrium by in exeemal determinisci forving. this oscillater is very well desembed by second onder langevin equation. using the fluctuation dissipuiin theurem, obsai fluciuation relations the work steady states, from experimental
the system
by the extemnal tocque and for the heat dissipazed by the syseem by writing results, we derive unalytical expressions for the prubability deasity functions of the work und the dissiputed heat, and for their finite time vorrections both for a lorcing linear in time and a forcing sinusoidal in sime. from these expressions, we present some analyticul derications of flactuation theurems based on experimental observation.
priccppec of thermodynanics between
arXiv Example Reference
paiting hamiltocian constitutes important approximation many- body systems, exactly soluble and quantum integrable. other hand, continum single purticle level density ( cspld contains information about continuum energy spectrum. the question pairing streggh for correlations in the coninmm is still unanwwered. in this poper we generalize the dibardson etact solution for the pairing hamiltnnza including cocrelations in the contincum. the resonant and non resonact continum incladed through the cspid. the resonant correations are made explicit by using the low lying
can use the hamilonian constant
cauchy
with seniority coustant pairing in costi summ using the cspld. found
two caliolated for
carhon isatopes. conclude that energy levels can indoed he cakclated continuam
micleus @ d'nattill unbouod. real and camplex energy representationn developed and their differences are shown, the trajectary the pair enerpies in the continum for the meleus d'mmathle is shown.
arXiv Example #2: Model Output
exact solution the richardion pairing hamiltonia obtained by including the resontat and resonant continuum through the contimma single particle deusity I espid) ). the gamoo states, which appear in the complex energy repeesenizzinn provide the main committtion from the cominnam. the inclusion of the continumm his been ned study the unhonnd incope Promalie and beyord. it was found that the continuam pairs piir energies with pasitive real composents par the unconelated pair esergy they appear complex conjagate partners. result the sotal energy the system may complex. from exact solutions of the pairing and pairing like bamitoniaas the csmld used investigate many- body correlations.
converge
Table 8: Summary Samples for arXiv
Mail Example #1: Reference
Michelle MacLaren is no longer sel direct the first Wonder Woman "heatrical movie. for 2017.
MacLaren left the project over " ereative differeneee MMvvie is currenily
CNN-Daily Mail Example #1: Model Output
CNN confimms that directer Michelle MaeLaren leoving the" Wonder Woman "movie. The movie, starring Gal Gadot, sill set for release on June 23, 2017.
CNN-Daily Mail Example 92: Reference
Andrew Mogni, 20, from Gien Elly,, Illinois, had only just arrived for semester prugram when the incidens huppened in January. Lie was flown back to Chicago via tir on March 20 but he died on Sunday, Initial police repects indicated the fall was oocadent but authoritaes are invsttgating the possibility that Magai was rabbed. attcked and shown 40 from bridge.
His cousin claims
CNN-DailyMall Exampic 42: Model Output
Andrew Magni, 20, from Gien Ellyn, Mlnooo. had only just anrived for semester program Italy when the incident happesed in January. He was flown back Chicago air ambulance March he died Sunday after falling off bridge Rome suspected pobbety atack Rome. police reports indicated was accident but authorities are investigating the possibility be was robbed.
Table 9: Summary Samples for CNN-DailyMail
15

---

Under review as a conference paper at ICLR 2022
MVgegaiite Example #1: Reference
lake meets Tad ConFassin where Tal is enjuying a salad. Tad doeun't believe Duvid's story as to why he is in Gluaeeter, Liza joins them and serves Tall with a restaining ondor to stay away from the hir in Glousester Amanda takes Trevar for on exam at the hoopiial and joins Angie David also joins them. Frica sits slone in her hotel when Opal
Emmu Ryan going to ask if she's ducumentary. In Glouceser, Gayle tomes
the documentary
Pine Valley. Ryan
at blank television when Emma comes downstairs with Corinaa. check Creenlee, Creenlee tells Gzyle
to get home. Greenlee
clinches her fist as imagines how it would be if were home. Frica punics as to what Ryan might have said the documentary Frica reusives mnother copy the documentary that Hayley did of Pine Valley. David asks Angie if read the gift that he gave her. Amanda tells David that she knew that he had lied to her about having patients see the hospital and instenl had
to Glouceseer without telling Amamda. lake gets Gloncester Tad reprimands lake for wanting answers, "I Long enough" Madison
back week and Madison's 's mum. Erica finally gives in and agrees to watch the DVD. Opul is thrilled. but Erica insists that she wants to watch alone. Ryan visits with DVD hand and suggests that ihey watch il together. David mests with Dr. Clayton and tells him about Greenlee. Greemder dreums of her family and friends back in Pine Vailey. Ercca Ryan watch the DVD, Liza lets Madison know that her futher had gotten juil
how long
been standing
Angie
but would be out juil within year. Amands and Jake discuss David her persisss sccusing David then doesn't how much longer they can gu on. Greenlee meets with Dr. Clayton about her surgery. Ryan and Erics kiss. Lixa and Tad kiss in his apariment.
Jake doess't believe that he really sick. Amanda tells lake
TV MVMegaSite Example #1: Model Output
Tod and are Krystal's, Jake Ted know that David going Glourester. Liza walks and gives Tad restraining order against him. the bospital, Angle tells Amonda ihat she had soen her on Hayley'ss 'sy' documentary. David walks up and listens to their conversation. At home, Opul questions Erica she had waiched the documentary on ielevasion about Pine Valley. Erica tells Opal that she doesn't I want to see the documentary. Ryan gets ready for Emma and Corinna's 's sleepover, Ryun less Emma know that hain't been able to watch the documentary that Iaallyy had shown television. Greenlee dreams that she back in Pine Valley with her much had to fall in love again. Opul asks her if she insecure about Ryan and what going on with him. Erica gees a visit from man, who gives her a DVD of the documentary from Dayley. David and Angle argue over the fact that he isn't as sick he claims to be. Amanda accuses David of lying to her about w where he had been the night that she stalbed. Tad tries calk Liza of breaking into David's har Gloncester. but she refuses listen him. Madison
family. Erica awakess from coma and finds
Jackson
Groenlee
her.
lets Opul
would like
into the hoopital and tells Angie know her father being sentenced today and was going go, but Angle encourages
father's sentencing. Madison her to go. lake and Tad try to talk her out going to the heuring. but Liza insists on going. lake thie to gel Thd to promise that he will not gu intu Gloucester without Lim'ss permission. Erica and Opal watch the video of Ryan's confession. Ryan comes tn visit Trica and toks her to watch littl TV Dowid meets with Dr. Clayton about Greenlee's 's condition. David introduces Greendee Clayton. Tad visits Liza the and apologizes to her for putiing her this position. Tad and Liza begin argue over his interference in life. David calls Greenlee and her that they are going to take on a the medical facilities Glouzesershire.
Angie that can go
MogesiSite Example #2: Reference
and Ryan are her office Pusion kissing when Greasiee walls up to the and stars tum the docckooh. David claspe hand over mouth keep her from screaming At ConPisionn Liza and Tid kiss before they start to dance. Krystal watches then asks Rob to take her back to his place. Jake and Amanda spend quiet evening home when there an incessant knocking en the door. Jake opens the door and Opal less them know that she read her tea leaves and knows that samanne headed back
lives. She fears that David. lake Amanda don't
concemed
anviety
leaves.
David reminds Greentee to the conclusion that it is Kendall. At ConlFasion, Tad sees David rushing out to his car. Ryan and Enca come clean to the press that they are involved. Liza visies Amanda and tris to soothe her fears that David back in town. Tid and Jake visit David at Wildwind. like promises David that he will he watching him. Ryun and penthouse and finds things completely
Colemaa städ she could face moother surgery. In talking to David, Greeniee realizes that Ryan with moother woman comes
linca couse home and soes him and Frica making lowe.
order. kinea and Kyan make love froat the firaplace. Greenlee herself Kyan's place
TVMeguSite Example Model Output
At the hospital, Liza kisses lad. Krystal walks and sees them kissing, Liza adks Thd she stcal him. At Wildwind, Jake and Amanda bed with the baby knows David coming know where Daviid is. At Fusion, Greenlee questions David is to what he doing here. Greesie demands to see Ryan. but David refuses to let her see Ryas. David triss to Greeniee calm down and let him examine her, but Greenlee insists relationship. Jake Deal Wildwind heing sold
when there knock on door. answers
it Opal. Opal tells
lown. assures her
doesn't
up the roof talk to Kyan. At Kyun's is home, krica tells Ryan that she agent showing
is used things going smoothly head from David is while. Astanda comes dowistairs and tells Jake about Opal's leaves giving her strong feeling that something
the public. like lets
Ogal know that he from their past is coming to town and that David could be there alrendy. lake asks Amanda if she is all right. Amanda less lake know that David had disappeared At the Confusion har, Tad tells liza that he thinks that there is something going on between him and Krystal, but Liza denies it. David etamines Greenlee and lets her the
David offers take Greeniee fire the tireplace. Ryan Erica arrive home Greenlze that he is taking her back to Glancester for another surgery. David less Greznize know that Zach and Kendall had left town with Spike and hid taken Spike with them. lake and Tad bust into David's is botel room and find him hanging upside down in the ottic. Jake orders David to leave town. David insists that he will leave opens door town.
Ryan with another
his place, but she insists knowing picture the moon the table. David
before
her mind.
Enica Ryan to take
home for romantic dinner
Lake him they Amanda and tells that they have file a lowsnit against David bocause
waiching him. David leaves hotel but
he finds that Gieennee gone. Lake
Table 10: Summary Samples for TVMegaSite
16

---

Under review as a conference paper at ICLR 2022
ForeverDreaming Example frl: Reference
Sydney races to find pure for Vaughn, but order to find the antidose, Sydney must make deal with Sark that could endanger Sloune's is life. Meanwhile, Will vontinues his research for Vaughn and discovers some disturbing inconsistencies invelving 20 20-year old stadardiizd IQ lests. Sydney finds out that Naughn has girlfriend.
FrraveeDreaming Example #l: Model Output
Sydney must lind a way save Vaughn's life while trying to track down Sark's buse of operations in order to obtain the antidote to a deadlly virus thas. Sloune has boen infected with. Miemwhille, Vaughn critical condition after being exposed to Rambaldi's toxin, and Sydney is forced make dangerous deall with Sark standardized tesis were ever
exchunge
getting the s, and discovers that one ten thoused fins graders could pass the lest.
professor
any
American
ForeweDreaming Dreaming Example #2:
Reference
Hall's universe from Earth, the Doctor and Mariha answer distress call from cargo ship SS Pentallion, which will impact a siar in 42 minutes. The ship scuoped part of the which alive, for cheap fuel. The star uses crew members Korwin and Ashton host bodies, and begins taking Ashton. The Doctor remagnetises the
board. Mantaa space engines, getting rid sun particlss fuel. This canses
member Riley, while trying reach the
ship, over by star.
Martha Riley back. the engines to stazt woeking again, and frees the ship froes the star's pull.
begins being
Forever Fereve-Deammngg Example Model Outpet
Torajii syseem, crew trying prevens ship from colliding Doctor uses the sonie serewdriver on Martha's mobile phone to activure Universal Koaming Activation which allows him to troel anywhere space and time without interfirence from the shap's control centre. However, the device malfunctions and the shap begins to fall towards the sun, and the Ductor is forced to use the emergency crewmembers, destroyed by impact, but Doctor manages
trupped on a space
escape escape. pod, which
cy stop stip hitting Doctor, Mutha does
the
Table 11: Summary Samples for FrraminDge
17

---

Under review as a conference paper at ICLR 2022
BookSum Look-Level Example 41: Reference
At the opening of Act I, is cloudy sutumn day Russian country estate In the ganlen. the old Marina stands the samovar and offers Doctur Astro refuses He complains
diffieully his job, Tolegin, impoverished local landowner, with them. Voyniskyy known Vamya comes out the house and joins them. He aloord fifty and werry and initaille He complains about his heother-i--lww Serchryakow, Serchryskoo''s yonng socond wife, Illlen, and about how their visil has turned the place upside down. Serebryakov, Heien, and Serebryakuy's isss daughter, Sunya, join them for moment. Afler they dejurt. Vanya sighs about Helen's besuty and then compliins about how he his toiled his whole life on this estate for the professor and it has come naught. After Vanya's sister's deach. he Sonya worked here su professur could continue his studies and writings, but Varya has come see that work foolish and irclevant. When Astrue suggesis that Vanya is jealouss Vanga luugs that be obviously is, especially as the old, gout and rhhumutiinm-iiieee man scems to stiract beautiful women. Helln ventures ounside and tells Astrow his services are not needed for her hosband. Mrs. Vaynitsly Vanya's mother and Sonya's grandmuther, that all they do read pamphless she becomes distressed claims he hates factory; doctoe bitterly depuro. discuss how he is very interssted in forestry work Sonya speaks up choerfully about how Astroy is trying to sove the old forest from destruction bocause forests make
something eat
tells them about
pumphlec written by friend in Kharkov. When Vanya frustraedd laborcr arrives tells Astrow he wamtdd
Vanya merely says he
before they
people happier. Astrov speaks of how Russias have tom down the forests and destruyed the wildliee: they no longer create. but rather destray Aller Sonya walks She muses about how Sonya clearly soems love the doctor but he does not love her back. Helen Screbryakov complaizs to
Astroy Vanya tries seduce Helee. but she pushes him
of how old and no one respects him. quenilous behavior unl annuys lellen. who begs him to stop it. Serebryakov igmures her and bemoons how his life schularship seems to be noching now. Sonya joins them and tells them Serebrgakkow must see Astroy now; she wants her father to stop behaving like a child. The olderly murse Marina comforts Serebryauuy and leads him out. Helen tells Vanya, who entered the
sighs
simply boced and life
for her In
is anmoyed and muves to leave. that her husband weatees Vanya only lament that everything over for him and his life was wasted trivial things. Holen has way.
accusses him of being drunk, and he admits to it. After Helln sweeps out the reom. Vanya ruminates what a fool was not to fall in love with her when she was younger. he once admined the profisoer but now he does not When Astruc returns, he mocks Vinya for having feelings for Helen, but Vanya will not admit it Astroy leaves to get drink; Sonya pulls him aside and makes him promise country life makes
uncle drunk, Hle agrees. They vontinue talk for moment. He comments that lelen beautiful but jill and useless. promise to stop drinking and stup getting her
like that, and he educated people ridiculuus Hi only likes foresis, Sonya compliments him and tries to cheer him up. As he prepares to leave, she usks how be might feel if he were to out that a friend of hers has fcelings for him, and he droily says he cunnot love anyone Afler he leaves, Sonya feels a surge of huppiness though she not sure why. In Act III, Sonya confesses to Helen that she loves doctar loves Senva Sanya gives
despises it has beaten down sees light the end for himself. The peasents are
same,
Astrow, and Helen suggests that she say something fcrestry maps. He discourses volubly on pattems deforestation until sees that Helen is uninterested. Helen insists she interested but says they should talk about something else. She poin: blank asks he likes Sonya, and he says no He then moves in seduce Helen. but she wanss nome of it. As he mies to kiss her, Vanya enters the room with Blowers Helen is borrifed by the situation and begs Vanya to tell her huchand that they must leave today. A moment Elleen
permission for Helen
this, Astrow Helen
ustensibly
and Serehryskoy announces that his idea begins to complain violenily about how Sencbryakov is feaud, is uninspired, thamkless, labored for Serebryakow his whole life and for no reason. He iasues this is Sonya's is estate. He runs out of the room. Serebryakov is startled by Vanya's outburst. He insists he cunnot stay here and morer.. Sanya implures him talk to her uncle and he agrees. He depurs. and those in the room hear gunsbot, then amother. Helen and Vanya struggle
laser. Screbryakow and the others announcement angers Vanya tremendoully
the
because
afford place the This
how he, Vanya. has
revolver Vauya screeches that he missed and Helen's 's planned departure
Telegan Marina discuss Serebeyakov day, then cut room. Vanya and Astray come in. Astrov mocks Vanya his behavior asks
professor. Cursing, sinks anto chair.
stole. Vanya maintains that he is innocent of thefl. Astrov laments how this purochial existance crushes people: it stultifying und meless. When Sonya enters Astroy tells her to tell her uncle to return the bottle of morphia he stote. Sonya tums to her uncle and tearfully asks him for the bottle. He complics. She takes him to make up He
retum
goodbye Serebryakov and llelet, who leave estate. Sonya seduce again in ruherr lackluster fushion; she kisses him and bids him farewell. Everyone bids Vanya return work. Marina suggests Astrow stay for dinner; refunes and says he must off. Astroy laaves, and even though Sonya knows he did not love her, she is sad, Vasya, though, claims he is extremely depressed. With both in tears, Souya comforts her uncle, She tells him that life may be difficul for them now, but the afterlife will be full of peace. love, and rest.
with her futher. Helen
to Astrov
BookSam Book-LLeve Example #1: Model Outpot
The seene opens a country house in the Kussian countryside. Ivnn, a young man, is sitting with his mother, who is knitiing him a Christaa stocking. He tells her that he does not want to have tea, but she insists that he drink anyway. She asks him how long they have knows each other, and be says that it has been ten years since he first met her. Hl his not had a day of since he known her, and has toiled withoul repose typhoid fever. then sat down and clased his eyes for a while
day's freedom ever since. Ile had typhoid epidemic
in the third week of Last, and when retumedd home. had to take care a man and wondered if he had killed the man. He wonders the future generations will forget him, but he sure that they will not. Ivan tells them that the house a mess, because the Professar has heen up all night writing and doesn't t have time to cat or dritk, He naks them to wake him at two in the momnnng but when they do not They all
had
wake him, he thinks that the Prafessor how they are all living peace and comeetment. happiness. The work of preserving the forest is theilling. he says, and Dr. Astroff watches over the old woods and sets out new plantations every year. He waits to show them to Helena and Sonia, and Helena asks him it be finds interesting. Helena sitting next to him, and teils him to look for Banustka's is works in the library the next day. He worried that getting angina from his old age. Hellna
from teir walk.
they come they talk about how happy they looks her, heatt swells
be together, and unbridled
struck by the sight of beautiful woman, and says
him make peace with himself, and stop being so angry with everyone else. Hut he says he will not listen her, becouse he spent too such time thinking about the past, and now he cannot fear death. Helesa gets up and walks away from him, sitting down at distance. She tris to get him to tell her way he doesn't It love anyone. He says he is not an idior. that he can understind her, but that he cannot understand her because ale young and healthy and beauiifal and konging for life and he on old docard, almost dead understand him and that should able understund her Now, she says, the the estate, so they have to some way guarantee
watch the success of otherss and
She tells his that
old and
continue live in the country.
and she has a young daughter and a young wife to care that they cun live in town on the money genersted ench year. They decide tn sell the wonds, bocmse they don't have anough money to do so every year, and they can't it afferd tn buy the woods every year either. They aeed find a way to make sure they can affoed a cestain amcunt of money buy cottage in Panland. They also live Eoousnd coubles, and the fusilyy ties are what gat estate to be worth much money.
doesn't it to on the income generated
have figure what do with the of the property. The estule of twenty-five. Semyon Lakedemoonof's beother banght the estate from the Lakeedmnons' msele. Everyone agrees that the estate for Socia's 's good, and that site should get to keep it. But now that be his gotten old and ill, the time has come for him to dispase of his property in regard the interests of his VOITSKI, ASTROFF SEREBR KOOFF Vanya, and SONTA arrive the house wimessed, and decide go not staying in village any longer. Excegtt for Vanya and Alexander, who stay to say good-bye his wife and son-in-low The Profossor kisses them all theee times, and then goes out to see thom off He gives them one lasi kiss to each of them before he leaves. They say they will always cemenber each other with pleasure, that they are interesting and original, and original. They shall rest
bought from his uncle. who had
goodbye to Tommo and Marina. city
They it they can find place to live there. They
Kharkow live with the professoe and his wife. They have been fiightsed
what they have
Table 12: Summary Samples for BookSum Beok-Level
18

---

Under review as a conference paper at ICLR 2022
BookSam Rook-LLvel Example #2: Reforenec
In his Lundon studio, artist Basil Haallard puts the finishing touches un his lated portrait, that of young Although Lurd Uleury, who visiting with Basil. asks about the young man's idearat, Hasil declines to naswer, noting his preference secrecy. Hasil never intends to extubit the painting, hecause did, would the despest feelings in his soul. However, Basil lets slip that the subject of the partrait is Derian Gray. who shortly thereafirr pays the two men house call. Lord Henry immedistell begins to influence Darian, suggesting that he should treasure and geard his youth and benury while he his them, hecamse they will soon fide. Testified aging, Dorian wastes could trade his portrit thus begins to take on
young he looks the portnait.
while later, he to bur the portratt likeaed "munder" and
that he could young while
image the painting aged. prefers the perfemm of the portnnit to the real Darian. Doviss falls love with young actress, Sibyl Vanc. a woman he harely knows, She plays a different womas or each night's perforinance, carning the label of "penius" from Dorian, who as smittes with her actingg more than with her personality. They become engaged, much wholesome Sibyl discusses engagement with her family. Because her mother indebted the theatre manages. Mr. baates, for fifty pounds, she against the marriage wnless Decian is wealthy: they do not know that he is. Sibyl's angyy brotter, lames, leaving for Australia, but he vows to kill Lociin if he wrongs his sister in any way. James also contreats his mother about gosspp he has heard - that his momerr and decensed futher never married, which Mis. Vine admits is time Dovian attends perfermance of Sibyl's with Ladd Henry and Ravil, but the performance torvilee Sihyl tolk Devian she what drew
like existence; in fact, Basul's is
surptise of Lord Henry and Basil. The
heaniful reality. Dorian disgusted by her poor acting, because her perfonmances not show on Donan's iss face. begins suspect that his wish coming true, in he vows in he good so that both he and the postrait con romain young. He. therefore, intends to spologize to Sihyl the next day and makes marry her after all. However, he is too late: Sibyl commits suicide at the theatre that night Decian first feds responsibility for her death, but then views it both wenderful entertanment
loeger act, bocause he showa
her,
dismisses
retunts home. To suprise,
portrait shows marks of crueity around the moutt. lines
selfish act her part. Lord Herry thes keep Doria''s events of the previons day. Devisn, howeves, complesely unconcemted about
of scandal. Donian and Lord Herry spend the evening
open. The next morning, Rasil arrives and expresies concern Dorian, given Sibyl or her family; he wasts to talk only of happy subyeess The next day. he covers his portratt and moves it to the attic. to which Dorian has the only key. He then pass, and Dorian lives a hedonivtic life necording agly. Doeian remains young. beautfull and innncent. People- consequence to him. Finally, when thiry-- thirt-eighht
vertles read a vellow book kent by Lnd Henry: the book becomes Dorian's i bluqpiit for life. Several the pastrit has
guidelines establishted Lord Henry talk about Doriat's "madness pleasure" and his dreadful infieence the people around him, years old, Devian shows the portnit to Rasill who begs Dovisn to repent of his sin and mit that the wish be mooked. Instaad. Dosian kills Bavil and hides his hody. Black kiniklinn his old friend Alan Camphell, Darian able to dispose of Rasil' body. An hour later, Derimn sriends a pasty, but bated and distrnctd. He then heads an opinm den out the street, meets Sibyl's younger benther, who has twenty-gene-olld and cannot the man lames looking for. A woman the street reveals, Doviae "sold himseif to the devil for a pretty face," so lames again puranes Docian. At his county estate one week lates, Derim entetttins guests hut helieves James in hansing him. Derina soon leams, however, that man accidentally killed in heating accident is James, and so he feels safe. The novel conclades six months later. reflects
the yellow book. While
waiting for oggectuiity
Docian for nearly
years. Docial makes
a case for mistalen identity when claims have the
Dovian and Lord Henry dise, and
serinss- Derian talks Rasil, and Loal park. Loed Henry also inquires about socret of Derian's youth, which Darian dismissess Dorian asks Lord Heny never to give the yellow
he heard the presicus Sunday while walking the
anyose clse. That evening. while Dorian examines the portnai, he decides to destroy it with the knife used to mirder Basil, Soon after Dorian's 's servints and is police offieer find old, only man lying dead on the ground in front of portrait of young and inmocent Dovine.
BookSum Book-Level Example 42: Model Output
Dociao Gray is sitting the gurden of his unele's 's house in London, playnng the giano. in the center of the rocms a portmait of young man of extrooedinary beauty. and in front of is Rasil Hallward's portrait of Darian Glasy The postoil helongs to Hollwand, and be reflsess to show st the Groveener Acalomy, becamse it containe Dorian and
mary him. Darian sitting the piano and finishes a painting that he has boen working on. He looks at the partriil and realizos that it is of him. He realizes that he will never able look again, but cannor stop looking at hocanse he wants to remember the beanty of his own frce. He asks his uncle foe some information about Doris,, and his unck tells him that Dorian has benariful mother who was marriad to a poor man who was killed in duel. She left him a son, who is very good-Sooking and who has inhecried all of her day, Darian
much himself in it. He tells Dorian's mneee. Lord Heary Watton, that he
alvice, and Deviom agroes. Honry's ys friend, Mr. Eiskinc Treadley, and and Heary wants to her. He also wants to get hes our of the hands of the Jew who has her bound to him for there years and eight months. He proposes to her, but she refuses him. She says that of his rank. He is disapyeinted, but does not esy anything to his mother about it The and
property. Lord Henry tells him write him rich family
Sibyl Vane. beautiful young weman who weeks
is a govemess
Fast Fnd of London, She the emplay of
she does not think he is good enough for her, and she will never lowe the Ducchess Monmouth, who tells him he should find wife
him have He agrees, but when be tells her that he does not love her, she laughs al him und refuses to call him by his new nome. Prince Chamming He goes to see the play, and is homified to see that the fuce on the camms is that of the portrait of Romen and Jolict. He cannor believe that he could hove done such a torrible thing to Bullct and that she could still be his wife. He lesves the theater and wanders the streets of London until he finds himself in Covent Ganden. He finds some women waiting for her by his nickname. "Prince Charming"" by a man with gan pointed his haad. It is Jimes Vane Vane Preations peace beflre he kills him. When Dorian gets to the street, he finds that the man he was trying to kill not the same man he thought he was. It tams out that Vanc is nventy-eighi:
day, he
mary her, She
spend his money frivolously.
him, and one them laughs when
curses him runs away. Derian docen't 't make peace with God. He gives Dorisn one minute to make his
into darks alley and suddenly gnilbedd
younger than Dorimm. The woman who look his money tells him not talk to her again. She off, and when Dorian looks back, the Alin Camphell, telling him that there dead man sining on table in his house, and that he must destroy the body so that no one will ever know who he is. He then gues his bedroom and finds a small bus of lnqquer, which he takes out and puts inside. He puts the bus back, gets into hoese-draen cartang, and gives the driver in address. The driver takes him to the aidress, and is he leaving the house, he sees the dead body o man on the tbble. When Camphell refurns, he tolls Alan not to disturh the hodly, but to ooeme back distuubed. lle netumss home and
has dicipperedd When he wales up neat mornnng he
hnd nightmune.
two
seven o'clock
evening. When the man arrives, throws the picture over cable. but Dorian does not believe that has finds that Camphall has brought back the chemicals and irons, and the other things that meeds do the job. He opens the cabinel where had hiddeen Basil's is cost and hag. and finds the green paste. At midnight, he gats a hansom and leaves the house with the instrertions to moot him at 7 clock the next day. He sins in the back of the wakes up the middle the night find that the portrait hus not clunged.
as the driver drives him through the streets. Il wonders if is possible to cure the soul means of the senses and the body by way of the soul.
Table 13: Summary Samples for BookSum Book-Level
19

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL EXPERIMENT DETAILS
Due to the space limit, additional experiment details are reported here. As we discussed in the main text, all of our models are initialized from BART Large with 12 layers (Lewis et al. 2020). In the bottom-up inference module (the left panel in Figure 1.. the local self-attention in our models has layers and all parameters are initialized from the first 8 layers of BART (including parameters for layer normalization). We use 4 layers for top-down inference (the middle panel in Figure I segment cross- pooss-attention, and (3) feed- forward. (1) and (3) are initialized from the last 4 layers of BART (including parameters for layer normalization). All other parameters are randomly initialized. The segment-pooling has kernel size of 32 and stride size of 24. The maximum document lengths for PubMed, arXiv, CNN-DM, TVMegaSite, DreavirDreaming, BookSum are 8192, 16384, 1024, 12288, 12288, 12288, respec- tively.
Each layer consists of (1) token local self-attention, (2) token- token-segment
B.1 LOCAL SELF-ATTENTION
The local self-attention used in our work is widely used in prior works on sparse attention (Beltagy al. 2020 Zaheer et al. 2020). is illustrated Figure It shows local self-attention of tokens with window size 4. Each token attends 2 tokens on the left and 2 tokens on the right, long as there are sufficient right and left neighbors. The attended nearby tokens are in light green. Each token also attends itself, indicated by dark green. White color in Figure 2 indicates absence of attention.
It is also called sliding window attention (Beltagy et al. 2020). We call it local self-attention to make a direct contrast with the full self-attention used for the segment-level representations. Despite its efficiency, it misses information outside of the local attention window. Thus, it is often used to- gether with other attention mechanisms. In Longformer, sliding window attention is combined with dilated sliding window attention and global token attention. In BigBird, it combined with random attention and global token attention. In our work, we use segment-level tokens to collect long range information which is then used enrich token-level representations through token-segment cross attention (see tow-down inference in Figure
Figure 2: An illustration of local self-attention. It illustrates local self attf-attention of 9 tokens with window size Each token attends tokens on the and tokens on the right, as long as there are sufficient right and neighbors. The attended nearby tokens are light green. Each token also attends itself, as indicated by dark green. White color indicates absence of attention.
20

---

Under review as a conference paper at ICLR 2022
C ABLATION STUDIES
We present results for a series of ablation studies this section. The experiments are performed with PubMed. The results are summarized Table [4] The first row shows the performance of the lop-down transformer with top-down update via cross-attention and window size 1024, which is our final model (Please see Figure I for an illustration).
The second row shows the performance for a variant of top-down update. In this variant, to update the bottom-up inferred token representations, we concatenate the token representations with the corresponding top-level segment representations, in contrast to the cross-attention approach used in the final model. We can see a clear performance degradation, indicating the importance of the cross-attention-ben based top-down update.
The third row displays the results without top-down update, and the decoder attends the bottom-up- inferred token representations to generate summaries. Compared to our final model, the performance also degraded, suggesting the effectiveness of the top-down update.
The bottom panel of Table 14 presents ablation results on the window size of local self-attention (see Figure for an illustration). These results are also plotted Figure 3 They show an effect of window size. That is, as the window size increases, the performance effect quite large when the window size is increased from 32 to 256. The effect becomes smaller after 256, but the model performance still benefit from larger window size.
all metrics enhances. The
R-1
R-2
R-L
Tow-Down Translormer
with top-down update cross-attention with top-down update via concat without top-down update
window size 1024
48.34
21.40
44.22
window size 1024
47.04
20.36
43.03
window size 1024
46.97
20.23
42.88
with top-down update cross- coss-attention
window size
46.30
19.55
42.21
with top-down update via cross-attention
window size
47.25
20.37
43.12
with top-down update via cross-attention
window size 128
47.44
20.56
43.35
with top-down update via cross-attention
window size 256
47.89
21.06
43.77
with top-down update via ccoss-attentoon
window size 512
48.08
21.16
44.05
Table 14: Ablation studies of Top-Down Transformer with PubMed.
48.00
22.25
41.50
25 50
40.00
20.25
46.30
-8- R-1
44.00
43.50
43.00
42.50
200
400
500
200
1000
Figure 3: Ablation attention window size with PubMed. The window sizes tested 32, 64, 256, 256, 512,
1024.
21


Reviews:
[{"review_id": "rqOTtfYMqZA", "reviewer": "Reviewer_Kugi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n- The hierarchical structure proposed in this paper is reasonable. By modeling the input document in a bottom-up and top-down manner, the model can model long documents in coarse and fine granularity levels.\n- The performance on benchmark datasets looks good.\n\nWeakness:\n- In their paper, the author mentioned that the local attention window size is 1024, which is even larger than 512. I think such a large window size might not be local. However, in Table 1, most datasets are not that long and even shorter than 1000. Did the authors consider different window sizes? (except the window size in Table 3.). Also, how to slide the window across the inputs? Is there any overlap between local windows?\n- The combination of top-down and bottom-up is new. But what's the performance without cross-attention? Also, it is not clear how the copy operation is implemented. \n- The author mentioned that they use BART as the initialization. Did they consider initialization with vanilla BERT or just random? I think this might better show the effectiveness of the proposed method by removing the BART contribution.\n- In the experiment part, what is the parameter for those baseline methods (BigBird, Longformer, etc.) Instead of only comparing with GPT-3 on model size, they should also report the model size for other models.\n- The writing of this paper needs to be improved. Especially for the experiment part, what's the define and implementation of OracleAdaPool?  What is RL denote?", "summary_of_the_paper": "Modeling long documents is a challenging problem. This paper works on this problem with top-down and bottom-up structures. The hierarchical structure proposed in this paper adopts local and top-down correction to make the model learn local and long-range dependency. The experiment on long document summarization benchmarks shows the effectiveness of their proposed method.", "summary_of_the_review": "The top-down and bottom-up framework proposed in this paper achieves good performance on long document summarization tasks. However, despite the results, the novelty of this design is somehow limited since the bottom-up and top-down idea is not new. The experiment part of this paper is also not clear, some details are missing, and some issues need to resolve (please see the weakness part).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "px2fwK0r6Ko", "reviewer": "Reviewer_VzGN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\nEven a little surprisingly, the performance is super good on all datasets. It beats all state-of-the-art models, including Longformer, LSH, BigBird, etc, by a decent margin (usually more than +1 R-1). The model is just initialized from BART. I feel it’s almost magic that fine-tuning a BART parameter based model could be that effective. A lot of prior work shares the same conceptual design principles (hierarchical, coarse-and-fine grained, local-global, etc.), but this model works very well indeed.\n\nWeakness\nI can’t really conclude the difference or principle which contributes to the performance gain. The authors developed a new way to assemble these blocks to build a model, and it empirically works well. However, are there any distilled piece we can take from this paper? What’s the killing component in this model? It’s not clear for me. Some ablation study would be helpful. \n\nThe paper is a very straightforward technical report, but it could be more thought-provoking or intellectual. I don’t mean to criticize the paper because it’s easy to follow. I sometimes prefer papers with more insights or thoughts, although I totally understand they are usually not accompanied with empirical success. \n\nMinor: \nI am not a huge fan of the usage of “inference” in this paper. All these “inference” modules are trained and fine-tuned. It is a little confusing when I read the first page. \nThe caption and presentation of Figure 1 could be improved. The caption contains too many terms which do not show in the Figure. Please consider unify the model name between the figure and caption. ", "summary_of_the_paper": "The paper proposes a new model architecture for abstractive text summarization. The framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency at a coarser time scale, and the bottom token level preserves the detail. The proposed model shows some advantage in both capability in capturing global and local context  and being computationally efficient. \nThe model is validated on many datasets and it beats almost all previous model. \n", "summary_of_the_review": "It's a model architecture paper for neural abstractive text summarization. It works super well on many widely used datasets. Despite the empirical success, it lacks more principled insights and justification about what works behind it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5s43467tDhe", "reviewer": "Reviewer_fnCT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n1、This paper conducted a variety of experiments on long document data sets of different levels, and proved that the top-down structure can effectively improve the performance of long document summaries.\n2、Compared with the model of gpt-3, the model proposed can summarize a complete book, and use 0.27% parameters and less training data to achieve competitive performance.\n3、The experimental result has proved that the proposed model can effectively improve the performance of long text summaries, which has been greatly improved comparing with other baseline models.\nWeaknesses \n1、The method proposed in this paper is not innovative enough. Similar methods have been extensively studied in images and speech, and the model structure does not fully reflect the problem of information redundancy faced by long document summaries.\n2、The detailed description of the model in this article is not clear enough, for example, there is no detailed description of the overall operation process of the local attention and the top-down model.\n3、In this paper, the experimental verification for calculation efficiency and memory usage of model is not sufficient. It only compares the parameters of the model, and does not conduct quantitative analysis through experiments.", "summary_of_the_paper": "Aiming at the task of long text summarization, this paper proposes a top-down and bottom-up reasoning method to improve the traditional bottom-up converter encoder structure, which has higher memory and computational efficiency. The model captures remote dependencies on a coarser time scale at the top, and the token level at the bottom retains details. It uses local self-attention to improve computational efficiency. At the same time, it has achieved good results on a large number of long text summary data sets. However, the method proposed in this article has been extensively studied in many fields, and there is no detailed analysis of the problems of long texts, and the theoretical innovation is insufficient.", "summary_of_the_review": "1、It is recommended to explain the specific calculation process of the model in detail, especially for the top-down inference process;\n2、It is recommended to further verify the computational efficiency and memory usage of this method through experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with the task of long text summarization. Inspired by earlier work on top-down and bottom-up architectures, this work focuses on improving the traditional bottom-up converter encoder structure, and the fine resolution representations. 

Pros:
- Their model can model longer documents in coarse and fine granularity levels.
- The performance on benchmark datasets looks pretty good compared to strong baselines
- Computationally efficient. 

Cons: The reviewers have raised several concerns including:
- the experimental verification for calculation efficiency and memory usage of model is not sufficient.
- the novelty of this design is somehow limited since the bottom-up and top-down idea is not new. 
- several details about the figures and especially the experiments were missing.

The authors have addressed several of the suggestions, added new experiments results addressing the issues raised by the reviewers. During the rebuttal period, the authors further conducted empirical investigations showing that the top-down update for token representations, especially with good top-level representations, leads to good summarization because of enriched token-level representations by the top-down. Despite positive results, some reviewers raised concerns that with only using BART as a backbone, it is surprising to achieve this great performance boost with the top-down/bottom-up models on long document summarization when they compared to the state-of-the-art transformer models (BigBird, Longformer and T5) that have been shown to encode longer sequences and beat several summarization models.
Decision: Reject