言語処理学会 第30回年次大会 発表論文集（2024年3月）

Recurrent Memory Transformer for Incremental Summarisationof extremely long text
Kaifan Li
Shuntaro Yada
Eiji Aramaki
Nara Institute of Science and Technology{li.kaifan.ln9,s-yada,aramaki}@is.naist.jp

Abstract
Pupular transformer-based language models are often
constrained with a limited number of input length, resulting in subpar performance on the task of long-text summarisation. Prior work has attempted to alter the model’sinternal architecture to accommodate longer inputs. Evenif a model supports longer input texts, limited RAMs inuniversity laboratories and edge devices prohibit us fromunleashing that input length. We, thus, explore the taskof long-text summarisation based on Recurrent Memory
Transformer (RMT) which provides an external memory
for the transformer-based models without modifying theinternal structure, and further proposed RMT-Summ. To
demonstrate the validity of RMT-Summ, we introduce
an incremental summarisation task, and built a dedicateddataset from PubMed medical articles containing structured abstracts. Our experimental results show that anRMT-Summ powered BART model performed better thanthe baseline original BART by 1.24 points in ROUGE-1.

1Introduction

Large language Models (LLMs) play a pivotal role inNLP. Self-attention, as the foundation of LLMs, suﬀers
from limitation of the input length due to the quadraticcomputational complexity. Pupular transformer-based language models are often constrained with a limited numberof input tokens, such as around 1K to 4K. Since this constraint harms long-text processing tasks (e.g. summarisation and complex dialogue), previous work has attempted
modiﬁcations to the self-attention mechanism to accommodate longer inputs. Despite the model’s support for
longer text inputs, the increased length places a burden onRAM, making the utilization of the model less feasible fortypical university laboratories and edge devices.

Figure 1

Overview of Incremental summarisation task: The
source text is ﬁrstly segmented based on its discourse structure.
Subsequently, each segment is fed into the summarizer to generatea summary that encapsulates all previously mentioned content.

Figure 2 Overview of RMT-Summ structure for Incremental
Summarisation task.

To address the former constraint, Recurrent Memory
Transformer (RMT) [1, 2] was proposed, which is simple
yet eﬀective external memory mechanism for the models
suﬀering from limited input length, by directly incorporating randomly initialized special tokens into the preﬁxof the input embeddings. Despite the excellent performance of RMT in Natural Language Understanding tasks
such as enwik8 and WikiText-103, its eﬀectiveness in Natural Language Generation remains to be validated. Hence,
we apply the idea of RMT into an encoder-decoder architecture, aligning better with the sequence-to-sequencetask. Furthermore, we reﬁne the training objective in thiscontext, referring to this variant as RMT-Summ. Speciﬁclly, for an extremely long text, we initially segment it andthen process each segment in turn. During this process,― 2412 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

the memory tokens of RMT-Summ are inserted into theto randomly access several tokens from other positions.

embeddings of the input to preserve the memory of previ-

The fusion of BigBird and Pegasus [6] at that time re-ous segment through the depth propagation, and facilitate

sulted in a signiﬁcant improvement and achieved the state-memory transmission between segments. RMT and RMT-of-the-art (SOTA) performance across long-text summari-Summ does not alter the internal structure of the model;

sation datasets. LongT5 [7], upon the local attention, in-therefore, in a plug-and-play manner, RMT-Summ can ef-

troduced Transient Global Attention, which avoids token

fectively integrates with various transformer-based models

selection or adding extra tokens for global attention. In-and other additional technologies.

stead, it dynamically constructs global tokens before self-

To validate the eﬀectiveness of RMT-Summ, we intro-

attention at each step.

duce a novel incremental summarisation task shown in

However, since these models reconstructed self-Figure 1: a long text is inputted per segment; the model

attention, retraining on large-scale corpora is required,must output summarisations of the current segment along

which introduced additional computational overhead. Ourwith all the past segments so far. Notably, this summari-RMT-Summ, as a plug-and-play approach, eﬃciently lever-sation task deviates from just summation or cumulationof segmented summaries because it should occasionallyages pre-trained models and requires only limited computational resources for ﬁne-tuning.

modify (edit) the past summary, where the model must
memorise the past segments.

2.2

Divide & Conquer

Based on this deﬁnition, we created a sub dataset from

Traditional text summarisation tasks often involve gen-the PubMed that contains structured abstracts for experi-erating a single summary for an entire text. However,ments. The experimental results showed a substantial en-with the increase in text length, such as in scientiﬁc papers

hancement in the performance of the RMT-Summ powered

datasets [8], long texts are often structured by sections orBART compared to the baseline original BART.

chapters.

2

Multiple prior works attempted to decompose the task

Related Workof long text summarisation into a multi-step process.

The most straightforward approach to address the long-DANCER [9] is a representative work, which initially di-text summarisation is by directly truncating the input text,

vides long text into multiple non-overlapping segments

often achieved by directly extracting the initial segment up

based on semantic structures. Subsequently, each seg-

to the maximum input length of the PLMs. However, thisment is individually modeled and processed. Finally, themethod is signiﬁcantly impacted by lead bias [3].
To minimize the information loss, it’s essential to extend

generated summaries for each segment are concatenated
together. SUMM 𝑁 [10] adopts a similar approach by

the length constraints of the PLMs while controlling com-initially segmenting documents into multiple sections. It

putational resource consumption. Previous research has

generates coarse-grained summaries for each segment andprimarily attempted breakthroughs from two perspectives:

subsequently integrates these coarse-grained summaries toEﬀective Attention (Sec. 2.1) and the Divide & Conquer

produce ﬁner-grained summaries.

strategy (Sec. 2.2).

2.1

Unlike prior research, we introduce an extra memory
preﬁx that pass salient information throught a segment-Eﬀective Attention

level recurrence. This memory preﬁx eﬃciently transfers

A popular method is to replace full-attention mechanismwith ﬁxed-patterns to achieve more eﬀecient model inference. Longformer [4], as a seminal work, introduced ahybrid attention mechanism that utilizes a form of local
attention known as slidng window attention. This alteration allowed Longformer to extend its maximum sequencelength up to 16,384 tokens. BigBird [5] introduced random attention on top of Longformer, allowing each token

crucial information between diﬀerent sections, enhancing
the generation of more precise summaries.

3Methods

3.1

Recurrent Memory Transformer

Since our proposed method RMT-Summ (Sec.

3.2)

is highly based on the seminal work, Rucurrent Memory― 2413 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Transformer (RMT) [1, 2], we introduce its foundation ﬁrst.

for summary generation. Speciﬁcally, in the context of anCompare to a prior long-text model called Transformer-XL

encoder-decoder architecture, the update of memory and[11], RMT utilizes a set of global tokens as memory units

the propagation of our RMT-Summ occur as follows:

instead of directly caching the hidden states of the entire𝐻˜ 0𝜏 = [𝐻 𝜏𝑚𝑒𝑚 ◦ 𝐻 0𝜏 ]segment. RMT adopted a decoder-only architecture in𝐻¯ 𝜏𝑁 = 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 ( 𝐻˜ 0𝜏 )their ﬁrst work [1]. Due to the unidirectional nature of
the decoder, within RMT, the memory unit is divided intoread and write modules. The read cell in the preﬁx part[ 𝐻¯ 𝜏𝑚𝑒𝑚 ◦ 𝐻 0𝜏 ] := 𝐻¯ 𝜏𝑁serves as the continuation of memory from the previous𝑇˜𝜏 = 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 ( 𝐻¯ 𝜏𝑁 , 𝑇𝜏 )segment, while the write cell acts as the storage for the𝑚𝑒𝑚𝑚𝑒𝑚◦ 𝐻 0𝜏+1 ]:= 𝐻¯ 𝜏𝑚𝑒𝑚 , 𝐻˜ 0𝜏+1 = [𝐻 𝜏+1𝐻 𝜏+1

memory of the current segment. Speciﬁcally, for time step
𝜏 and segment hidden representations 𝐻 0𝜏 , the segmentlevel recurrence implementation in RMT is realized by thefollowing equations:Notably, in contrast to the traditional summarizationtask, our optimization objective is tailored for the incremental summarization task, as elaborated in Section 4.

𝐻˜ 0𝜏 = [𝐻 𝜏𝑚𝑒𝑚 ◦ 𝐻 0𝜏 ◦ 𝐻 𝜏𝑚𝑒𝑚 ]4𝐻¯ 𝜏𝑁 = 𝑇𝑟𝑎𝑛𝑠 𝑓 𝑜𝑟𝑚𝑒𝑟 ( 𝐻˜ 0𝜏 )Incremental Summarisation
To align RMT-Summ with summarisation task, in this[𝐻 𝑟𝜏𝑒𝑎𝑑 ◦ 𝐻 0𝜏 ◦ 𝐻 𝜏𝑤𝑟𝑖𝑡𝑒 ] := 𝐻¯ 𝜏𝑁work, we propose the concept of incremental summarisa-

Here 𝑁 represents the number of transformer layers, while◦ represente the concatenate operation.
In the model’s embedding layer, the preﬁx and postﬁx memory cells are concatenated with the input text’stion. Speciﬁcally, given an input document 𝐷 comprisingmultiple segments 𝑆 = {𝑆1 , 𝑆2 , ..., 𝑆 𝑁 }, the corresponding summary 𝑦 can also be segmented into multiple parts𝑦 = {𝑦 1 , 𝑦 2 , ..., 𝑦 𝑁 }. For each input segment 𝑆 𝜏 , our train-embeddings. Subsequently, they undergo a vertical pas-

ing target should be 𝑇𝜏 = 𝑒𝑑𝑖𝑡 (𝑐𝑜𝑛𝑐𝑎𝑡 (𝑦 1 , 𝑦 2 , ..., 𝑦 𝜏 )). It issage through the transformer. During this process, theimportant to emphasize here that the function edit implies

ory of the current segment as the network passes throughthat in practical scenarios, our summaries may not simply
be a straightforward concatenation of segment summaries

to achieve the depth-level propagation. Subsequently, tobut a reintegration process as shown in Figure 1.

achieve segment-level recurrence, the write tokens of the5Experiment Setup

5.1

Data

write tokens from the postﬁx progressively retain the mem-current segment are passed on to the next segment:
𝑚𝑒𝑚𝑚𝑒𝑚𝑚𝑒𝑚𝐻 𝜏+1:= 𝐻 𝜏𝑤𝑟𝑖𝑡𝑒 , 𝐻˜ 0𝜏+1 = [𝐻 𝜏+1◦ 𝐻 0𝜏+1 ◦ 𝐻 𝜏+1]

3.2

PubMed [13] is a large scale long scientiﬁc dataset col-RMT-Summlected from PubMed.com, which encompassing a wealth of

Text summarisation, as a sequence-to-sequence task, isbiomedical and life science literature and journals. More-more suitable for application to a sequence-to-sequence

over, due to the presence of structured abstracts withinmodel rather than encoder-only or decoder-only structures

certain papers in PubMed, it is well-suited for validating[12]. In an encoder-decoder architecture, the sentence rep-

our incremental summarisation task. We applied speciﬁcresentation is encoded in the encoder and the target sen-preprocessing to this dataset and ﬁltered out a subset, which

tence is generated in the decoder. Due to the bidirectional
nature of the encoder, a single memory cell can simulta-

we named PubMed-IncreSumm.
Speciﬁcally, following the methodology outlined in thisneously serve both read and write functions. Based onwork [14], we extracted the main sections from the bodythe aforementioned concept, we only insert memory pre-text and segmented it into four parts based on the section

ﬁx into the encoder following the latest work of RMT [2],headings. Additionally, for some literature, the abstract

to retain the memory of the current segment and circulate

sections could be explicitly categorized into “Introduc-

between segments, while we utilize an addtion decodertion”, “Methods”, “Results”, and “Conclusion” through― 2414 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Section

Keyword#Tokens for memory

Introduction introduction, case, objective, purpose, . . .

Rouge1 Rouge2 RougeL

Baseline BART

49.35

19.38

48.05

result, experiments, observations, . . .

20

48.94

18.76

47.66

conclusion, discussion, summary, concluding, . . .

32

49.44

19.16

48.14

Keywords used to match the corresponding sections.
If a section heading is not found in this keyword list, we disregard
it. Additionally, we performed the same grouping operation for
the abstract sections.

64

49.14

18.79

47.85

10049.8319.66

48.52

128

50.04

19.75

48.72

150

50.15

19.91

48.85

keyword matching, as speciﬁed in Table 1. We ﬁltered out

200

50.59

20.24

49.31

such articles accordingly.

256

50.25

20.03

48.92

Methodmethods, materials, methodology, . . .

Results
Conclusion
Table 1

From PubMed’s train, test, and valid sets, we acquired

Table 2

kenizer to conduct text length statistics for all datasets.

ROUGE scores on PubMed-IncreSumm. The numberof tokens represents the length of the memory preﬁx for RMT–
Summ. We denote the optimal results with bold font and thesecond-best results with an underline.

The average length of the source text was 2,736.26, while

tial overlaps between various sections, which we believe

the ﬁnal abstracts had an average length of 298.68.

might contribute to the model’s sustained strong perfor-

5.2

mance. Furthermore, we observed that an increase in the24,843, 1,399, and 1,431 data entries, respectively as
PubMed-IncreSumm. We employed the NLTK [15] to-Model & Implementation settings

number of memory tokens exhibits a certain positive correWe opted to employ BART [16] (i.e. ‘facebook/bart-lation with the ROUGE scores. The performance reaches

base’ available on Hugging Face) as the backbone modelits peak at around 200 tokens, with the ROUGE-1 scorebecause of its excellent performance in short text sum-reaching 50.59, surpassing the baseline by 1.24 points.

marisation tasks. The sample counts for the training set,

Subsequently, after further increasing the token count tovalidation set, and test set were 24,843, 1,399, and 1,431,256, the rouge scores remained almost unchanged. How-

respectively. We set the learning rate to 5.0 × 10−5 , beam

ever, the number of memory tokens around 200 appears tosearch width to 4, and the maximum text length for eachbe excessive for a practical situation, suggesting room for

section’s main text to 512, with a maximum generation

improvement of our approach.

length of 300. We utilized the F1-score of ROUGE-1,ROUGE-2, and ROUGE-L as evaluation metrics, invoking

7

Conclusions

the evaluation package provided by HuggingFace. Here,For long document summarisation, we proposed a vari-

we computed the results for the generated output of eachant of plug-and-play RMT termed RMT-Summ, which in-segment, rather than solely focusing on the output of theﬁnal segment. We utilized two Quadro RTX 8000 GPUs,

corporates memory preﬁxes into the encoder-decoder ar-each with 48GiB memory, alongside one Quadro P6000

we introduced a new optimisation objective, incremen-

GPU with 24GiB memory.

tal summarisation, and created a dedicated dataset from6

PubMed articles, i.e. PubMed-IncreSumm datasets. TheResults and Discussion

chitecture. To verify the eﬀectiveness of the RMT-Summ,

experimental results show that when the token length of

Our experiments primarily aimed to validate the impact

the memory preﬁx is around 200, the metric of ROUGE

of diﬀerent sizes of memory preﬁx in contrast to base-scores outperforms the baseline. Since carefully designingline results. As shown in Table 2, under the settings ofa task and training dataset could enable models to gener-

incremental summarisation, the BART model performs

ate memory-encoded vectors that are useful for long-textstrongly even without any additional mechanism in theprocessing, we will continue exploring our approach byevaluating it on diverse models and tasks.

PubMed-Incremental dataset. This is to some extent reliant on BART’s powerful language modeling capabilities.
On the other hand, scientiﬁc papers often have substan-― 2415 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgement
This work was supported by JST CREST (Grant Number: JPMJCR22N1), Japan.

References[1] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent Memory Transformer. Advances in Neural Information Processing Systems, Vol. 35, pp. 11079–11091, 2022.
[2] Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling transformer to 1m tokens and beyond with RMT. arXiv
preprint arXiv:2304.11062, 2023.
[3] Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao.
A survey on long text modeling with transformers. arXiv
preprint arXiv:2302.14502, 2023.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint
arXiv:2004.05150, 2020.
[5] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey,
Joshua Ainslie, Chris Alberti, Santiago Ontanon, PhilipPham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big
bird: Transformers for longer sequences. Advances inneural information processing systems, Vol. 33, pp.
17283–17297, 2020.
[6] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter
Liu. Pegasus: Pre-training with extracted gap-sentences forabstractive summarization. In International Conferenceon Machine Learning, pp. 11328–11339. PMLR, 2020.
[7] Mandy Guo, Joshua Ainslie, David C Uthus, SantiagoOntanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang.
Longt5: Eﬃcient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724–736, 2022.
[8] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung
Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive
summarization of long documents. Proceedings of thexl: Attentive language models beyond a ﬁxed-length context. In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics, pp.
2978–2988, 2019.
[12] Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan.
An empirical survey on long document summarization:
Datasets, models, and metrics. ACM computing surveys, Vol. 55, No. 8, pp. 1–35, 2022.
[13] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung
Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian.
A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the2018 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621, 2018.

[14] Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji,and Lu Wang. Eﬃcient attentions for long document summarization. In Proceedings of the 2021 Conference of[15]the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pp. 1419–1436, 2021.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with
the natural language toolkit. ” O’Reilly Media, Inc.”,2009.
[16] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan
Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language
generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pp. 7871–7880, 2020.

2018 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018.

[9] Alexios Gidiotis and Grigorios Tsoumakas. A divide-andconquer approach to the summarization of long documents.
IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 28, pp. 3029–3040, 2020.

[10] Yusen Zhang, Ansong Ni, Ziming Mao, Chen HenryWu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah,
Dragomir Radev, and Rui Zhang. Summn: A multi-stage
summarization framework for long input dialogues anddocuments: A multi-stage summarization framework forlong input dialogues and documents. In Proceedings ofthe 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.

1592–1604, 2022.
[11] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-― 2416 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).