Compressive Transformers for Long-Range Sequence Modelling
Authors: Jack W. Rae
Anna Potapenko
Siddhant M. Jayakumar
Chloe Hillier
Timothy P. Lillicrap
Venue:  (2020)
Keywords: memory
language modeling
transformer
compression

Abstract: We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

COMPRESSIVE TRANSFORMERS FOR LONG-RANGE SEQUENCE MODELLING
Jack W. Rae* t
Anna Potapenko*1
Siddhant M. Jayakumart
Chloe Hillierf
Timothy P. Lillicrapit
ABSTRACT
We present the Compressive Transformer, attentive sequence model which compresses past memories for long-range sequence learning. We find the Com- pressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respec- tively. We also find can model high-frequency speech effectively and can be used as memory mechanism for RI., demonstrated on object matching task. To promote the domain of long-range sequence learning, we propose a new open- vocabulary language modelling benchmark derived from books, PG-19.
INTRODUCTION
Humans have remarkable ability to remember information over long time horizons. When reading a book, we build up a compressed representation of the past narrative, such as the characters and events that have built up the story far. We can do this even if they are separated by thousands of words from the current text, or long stretches of time between readings. During daily life, we make of memories varying time-scales: from locating the car keys, placed in the morning, to recalling the name of an old friend from decades ago. These feats of memorisation are not achieved by storing every sensory glimpse throughout one's lifetime, but via lossy compression. We aggressively select, filter, or integrate input stimuli based on factors of surprise, perceived danger, or repetition amongst other signals (Richards and Frankland 2017).
Memory systems in artificial neural networks began with very compact representations of the past. Recurrent neural networks (RNNs, |Rumelhart et al. (1986) learn to represent the history of obser- vations in compressed state vector. The state is compressed because it uses far less space than the history of observations - the model only preserving information that pertinent to the optimization of the loss. The LSTM (Hochreiter and Schmidhuber 1997) is perhaps the most ubiquitous RNN variant; it uses learned gates on its state vector to determine what information stored or forgotten from memory.
However since the LSTM, there has been great benefit discovered in not bottlenecking histori- cal information in the state, but instead in keeping past activations around in external memory and attending to them. The Transformer (Vaswani et al. 2017) is a sequence model which stores the hidden activation of every time-step, and integrates this information using attention operator (Bahdanau et al., 2014). The Transformer will thus represent the past with a tensor (depth x mem- ory size x dimension) of past observations that is, in practice, order of magnitude larger than an LSTM's hidden state. With this granular memory. the Transformer has brought about step-change in state-of-the-art performance, within machine translation (Vaswani al. 2017). language mod- elling (Dai et al.) 2019 Shoeybi et al., 2019). video captioning (Zhou et al. 2018), and multitude of language understanding benchmarks (Devlin al. 2018 Yang et al. 2019) amongst others.
One drawback in storing everything is the computational cost of attending to every time-step and the storage cost of preserving this large memory. Several works have focused on reducing the computational cost of attention with sparse access mechanisms (Rae et al. (016)) Child et al. 2019
Authorr contributed equally, i DeepMind, London, UK. $ CoMPLEX, Computer Science, University Col- lege London, UK. Please direct correspondence apotapekkk) google..com.

---

However sparse attention does not solve the storage problem, and often requires custom sparse kernels for efficient implementation. Instead we look back to the notion of compactly representing the past. We show this can be built with simple dense linear-algebra components, such as convolutions, and can reduce both the space and compute cost of our models.
Sukhbaatar et al. 2019 Lample al. 2019).
We propose the Compressive Transformer, a simple extension to the Transformer which maps past hidden activations (memories) to a smaller set of compressed representations (compressed memo- ries). The Compressive Transformer uses the same attention mechanism over its set of memories and compressed memories, learning to query both its short-term granular memory and longer- coarse memory. We observe this improves the modelling of text, achieving state-of-the of-the--rttttttt results in character-based language modelling - 0.97 bpe on Enwik8 from the Hutter Prize (Hutter) 2012) and word-level language modelling 17.1 perplexity on WikiText- (Merity et al., 2016). Specifically, we see the Compressive Transformer improves the modelling of rare words.
We show the Compressive Transformer works not only for language, but can also model the waveform of high-frequency speech with a trend lower likelihood than the TransformerXL. and Wavenet (Oord et al. (016) when trained over 400,000 steps. We also show the Compressive Trans- former can be used as a memory component within an RL agent, IMPALA (Espeholt et al. 2018), and can successfully compress and make use of past observations.
Furthermore we present a new book-level language-modelligg benchmark PG-19, extracted from texts in Project Gutenberg!] to further promote the direction of long-context sequence modelling. This is over double the size of existing LM benchmarks and contains text with much longer contexts.
2 RELATED WORK
There have been a variety of recent attempts to extend the range of attention, particularly in the Transformer, or to replace the attention operation with something less expensive. [Wu et al. (2019) show that convolution-like operator that runs linear time can actually exceed the performance of the quadratic-timg self-attention layer in the Transformer at -oreence----eeeeccee translation and sentence-level language modelling. However such a mechanism inhibits the flow of information across a large number of time-steps for a given layer, and has not shown to be beneficial for long- range sequence modelling.
Dai et al. (2019) propose the TransformerXL, which keeps past activations around in memory. They also propose novel relative positional embedding scheme which they see outperforms the Trans- former's original absolute positional system. Our model incorporates both of these ideas, the use of a memory to preserve prior activations and their relative positional embedding scheme.
The Sparse Transformer (Child et al. 2019) uses fixed sparse attention masks to attend to roughly vn locations in memory. This approach still requires keeping all memories around during training, however with careful re-materialization of activations and custom kernels, the authors train the model with a reasonable budget of memory and compute. When run on Enwik8, the much larger attention window of 8, 000 improves model performance, but overall it does not significantly outperform simpler TransformerXL with a much smaller attention window.
able to
The use of dynamic attention spans is explored in Sukhbaatar al. (2019). Different attention heads can learn to have shorter or longer spans of attention - and they observe this achieves state-of- the-art in bharacter-based language modelling. This idea could easily be combined with our contri- bution - compressive memory. However an efficient implementation is not possible on current conse-iinea-aaleebr accelerators, such as Google's 's TPUs, due to the need for dynamic and sparse computation. Our approach builds on simple dense linear algebra components, such as convolutions.
3 MODEL
We present the Compressive Transformer, long-range sequence model which compacts past acti- vations into a compressed memory ? The Compressive Transformer is variant of the Transformer
Project Gutenberg: https: /wwww www.ggberg.orgg////... A TF implementation he found in Sonnet: https: thub. com/ deem/deemm nd/sonnet
2

---

Compressed Memory
Memory
Sequence
F.0
Figure I: The Compressive Transformer keeps a fine- goee-graind memory of past activations, which are then compressed into coarser compressed memories. The above model has three layers, sequence length n, = 3, memory size 7m 6, compressed memory size Rem = The highlighted memories compacted, with compression function fa per layer, single compressed memory instead of being discarded at the next sequence. In this example, the rate of compression c 3.
Vaswani et al. 2017). deep residual network which only uses attention to propagate information over time (namely multi-head attention). We build on the ideas of the TransformerXL (Dai al. 2019) which maintains a memory of past activations at each layer to preserve longer history of con- text. The TransformerXI. discards past activations when they become sufficiently old (controlled by the size of the memory). The key principle of the Compressive Transformer is to compress these old memories, instead of discarding them, and store them an additional compressed memory.
3.1 DESCRIPTION
We define nm and Nea to be the number of respective memory and compressive memory slots in the model per layer. The overall input sequence 21, ,2, .2,..... 3|a represents input observations (e.g. tokens from book). These are split into fixed-size windows size n, for the model to process in parallel. The model observes x .t..... Ittn, at time t, which we refer to as the sequence (e.g. Figure D. As the model moves the next sequence, its n, hidden activations are pushed into fxed-sized memory (like the TransformerXL) of size nem The oldest n, activations in memory are evicted, but unlike the TransformerXL we do not discard them. Instead we apply a compression operation, fe m which we then store in secondary FIFO compressed memory of size 70mm d denotes the hidden size of activations and refers to the compression rate, a higher value indicates more coarse-grained compressed memories. The overall temporal range of the model becomes | x (n, + 7m + C Tee where is the number layers as discussed in Supplementary Section The full architecture is
Rl ":" jxd mapping the n. oldest memories to [i compressed memories
described in Algorithm
Algorithm Compressive Transformer
At time zero
1: mn +-
Initialize memory to zeros (1 x nm d) // Initialize compressed memory to zeros (1 x Nem d)
2: cmo <- O
At time
3: h()) <- xW emb
|/ Embed input sequence(n, d)
4: for layer = 1, 1, do (1) (1)
5: mem <- concat(cm," .m
I/ (Reem + nm)) x d) MHA over both mem types (n, x d)
6: ā()) multihead_attentionn (h
(h mem @
7: a()) + layer_norm( + (3)
// Regular skip + layernorm (nem d)
8: old_ment (1) m
(i) nz]
I/ Oldest memories to be forgotten (n. d)
9: new cm) (i) <
(()
(el (1)
|/ Compress oldest memories by factor e <[ [ x d)
m concat(m) (m. ii hh)
(i)
Update memory (nm d)
11: cm, concat(cm," new cm [-) :]
(i)
// Update compressed memory (Nem d) // Mixing MLP (n, d)
12:
hi-1) + layer more (a(")) + (1))
3

---

Algorithm 2 Attention-Recconstruciioo Loss
1: L Lattn f-0 0
2: for layer = 2, .., do hoop_gaadien((h"))
I/ Stop compression grads from passing... I/ ...into transformer network. 5: Q K, V < wetss-tpppdpcce cooo--gradenn(attention params at layer i) Re-use attention weight matrices.
3: h(4)
4: old_mem
stop siop-gradient(( (ld_ Leem
6: def a m) ((hQ) (mV)
I/ Use content-based attention (no relative). Compression network (to be optimized). Lettn f- Lattn + ||attn(h () old mem (1) attn(h()) new new_cm(") cm!))]la
7: new_cm (i) < f2 (old_mem'
8:
3.2 COMPRESSION FUNCTIONS AND LOSSES
For choices of compression functions fe consider (1) max/mean pooling, where the kernel and stride is set to the compression rate (2) 1D convolution also with kernel & stride set to (3) dilated convolutions; (4) most-used where the memories are sorted by their average attention (usage) and the most-used are preserved. The pooling is used fast and simple baseline. The most- used compression scheme inspired from the garbage collection mechanism in the Differentiable Neural Computer (Graves et al. 2016) where low-usage memories are erased. The convolutional compression functions contain parameters which require training.
One can train the compression network using gradients from the loss; however for very old memories this requires backpropagating-thoon whrpoiignggtouu--iiieeee (BPTT) over long unrolls. As such we also consider some local auxiliary compression losses. We consider auto-encoding loss where reconstruct the original memories from the compressed memories Lee [lold_mem g(new__enn ||2, where g - xd + Rn. is learned. This is a lossless compression objective it attempts to retain all information in memory. We also consider an attention-reconstuuttiin loss described in Algorithm which reconstructs the content-based attention over memory, with content-basee attention over the compressed memories. This is a lossy objective, as information that is no longer attended can be discarded, and we found this worked best. We stop compression loss gradients from passing into the main network as this prevents learning. Instead the Transformer optimizes the task objective and the compression network optimizes the compression objective conditioned on task-relevant representations; there is no need to mix the losses with tuning constant.
4 PG-19 BENCHMARK
As models begin to incorporate longer-range memories, it is important to train and benchmark them on data containing larger contexts. Natural language in the form of text provides us with vast repository of data containing long-range dependencies, that easily accessible. We propose new language modelling benchmark, PG-19, using text from books extracted from Project Gutenberg F We select Project Gutenberg books which were published over 100 years old, i.e. before 1919 (hence the name PG-19) to avoid complications with international copyright, and remove short texts.
The dataset contains 28, 752 books, or 11GB of text which makes it over double the size of BookCorpus and Billion Word Benchmark.
4.1 RELATED DATASETS
The two benchmarked word-level language modelling datasets either stress the modelling of stand-alone sentences (Billion Word Benchmark from Chelba et al. (2013)) or the modelling of a small selection of short news articles (Penn Treebank processed by Mikolov al. (20T0)). MMrity ct al. (2016) proposed the WikiText-103 dataset, which contains text from high quality subset of English-langeen languge wikipedia articles. These articles are on average 3, 600 words long. This dataset has been a popular recent LM benchmark due to the potential to exploit longer-range dependencies (Grave et al. 2016 Rae al. 2018 Bai et al. 2018b). However recent Transformer models, such
PPG-19 is available httpps /ttttts:://ithiie wttp:thww...cchhhteeteeeeee hnt
4

---

Table 1: Comparison to existing popular language modelling benchmarks.
Avg. length (words)
Train Size
Vocab
Type
IB Word
27
4. 4.BGGB
793K
News (sentences)
Penn Treebank
355
5.IMB
10K
News (articles)
WikiText-103
3.6K
515MB
267K
Wikipedia (articles)
PG-19
69K
10.9GB
(open)
Books
as the TransformerXL. (Dai et al. 2019) appear to be able to exploit temporal dependencies on the order of several thousand words. This motivates a larger dataset with longer contexts.
Books are a natural choice of long-form text, and provide us with stylistically rich and varied natural language. Texts extracted from books have been used for prior NLP benchmarks; such as the Chil- dren's Book Test (Hill et al., 2015) and LAMBADA (Paperno et al. 2016). These benchmarks use text from Project Gutenberg,
online repository of books with expired US copyright, and Book- Corpus (Zhu et al.) 2015). a prior dataset of 11K unpublished (at time authorship) books. CBT and LAMBADA contain extracts from books, with specific task of predicting held-out words. In the case of LAMBADA the held-out word is specifically designed to be predictable for humans with access to the full textual context but difficult guess with only local context.
CBT and LAMBADA are useful for probing the linguistic intelligence of models, but are not ideal for training long-range language models from scratch they truncate text extracts at most couple of paragraphs, and discard lot of the books' text. There has been prior work on training models on book data using BookCorpus directly (e.g. BERT from Devlin al. (2018)) however BookCorpus is no longer distributed due to licensing issues, and the source of data is dynamically changing - which makes exact benchmarking difficult over time.
The NarrativeQA Book Comprehension Task (Kočiský et al., 2018) uses Project Gutenberg texts paired with Wikipedia articles, which can be used as summaries. Due the requirement of needing a corresponding summary. NarrativeQA contains a smaller selection of books: 1,527 versus the 28,752 books in PG-19. However reasonable that PG-19 may be useful for pre-training book summarisation models.
4.2 STATISTICS
A brief comparison of PG-19 to other LM datasets can be found in Table | We intentionally do not limit the vocabulary by unk-ing rare words, and release the dataset as an open-vocabulary bench- mark. To compare models we propose to continue measuring the word-level perplexity. This can still be computed for any chosen character- based, byte- byseebaseddao or subword-basee one calculates the total cross-entropy loss L E. log(Pt(P<e) over the given validation subset using a chosen tokenization scheme, and then one normalizes this value by the number of words: L/Rlwords where wworrs is the total number of words in the given subset, taken from Table 2 The word-level perplexity is thus el//toorrds For sake of model comparisons, it is important to use the exact number of words computed in Table 2 as the normalisation constant.
scheme. To do this, test
Alongside quantitative analyses, we build an LDA topic model (Blei et al. 2003) for qualitative inspection of the text. We present key words for several topics in the Supplementary Table These topics include art, education, naval exploration, geographical description, war, ancient civilisations, and more poctic topics concerning the human condition - love, society, religion, virtue etc. This contrasts to the more objective domains of Wikipedia and news corpora.
EXPERIMENTS
We optimised all models with Adam (Kingma and Ba 2014). We used a learning rate schedule with a linear warmup from le-6 to 3e-4 and a cosine decay back down to le-n6. For character- based LM we used 4, 000 warmup steps with 100, 000 decay steps, and for word-based LM we used 16, 000 warmup steps with 500, 000 decay steps. We found that decreasing the optimisation update frequency helped (see Section (.3ti. namely we only applied parameter updates every steps after 60, 000 iterations. However found the models would optimise well for a range of warmup/warm-
5

---

Table 2: PG-19 statistics split by subsets.
Table 3: Eval. perplexities PG-19.
Train
Valid.
Test
Valid.
Test
books
28,602 1,973,136,207
100
36L TransformrrX
45.5
36.3
# words
3,007,061
6,966,499
36L Compressive Transf.
43,4
33.6
Table 4: State-of-the results Enwik8.
Table 5: Compression approaches Enwik8.
Model
BPC
Compression Compression loss
BPC
7L LSTM (Graves
2013
1.67
Conv
BPTT
0.996
LN HyperNetworks, Ha et al.
2016
1.34
Max Pooling
N/A
0.986
LN HM-LSTM Chung et al.
2016
1.32
Conv
Auto-encoding
0.984
ByteNet KKlcchrrenne et
2016
1.31
Mean Pooling
N/A
0.982
RHN Zilly et al. (2017)
1.27
Most-used
N/A
0.980
ML.STMKrauu et al. 6
1.24
Dilated conv
Attention
0.977
641. Transf. Al-Rtou al.
2019)
1.06
Conv
Attention
0.973
24L.TXL (Dai
2019
0.99
Sparse Transl. (Chid
2019)
0.991
Adaptive Transi. (Sukhbaatar et al.
2019)
0.98
24L (ours)
0.98
24L Compressive Transformer
0.97
down values. We clipped the gradients to have a norm of at most 0.1, which was crucial to successful optimisation.
5.1 PG-19
We benchmark the Compressive Transformer against the TransformerXI. on the newly proposed PG- 19 books dataset. Because it is open-vocabulary, train subword vocabulary of size 32000 with SubwordTextfincoddr from the tfds package in TensorFlow and use the dataset statisties to compute word-level perplexity, described Section 4.2 We train layer Compressive Transformer with a window size of 512, both memory and compressed memory size of 512, and compression rate C' = We compare this to 36 layer TransformerXL trained with window size 512 and attention window 1024. The model was trained on 256 TPUv3 cores with a total batch size of 512 and converged after processing around 100 billion subword tokens. We display the results in Table 3 where we see Compressive Transformer obtains a test perplexity of 33.6 versus the TransformerXLL's 36.3. Despite the dataset size, clearly a challenging domain. This can suit as a first baseline on the proposed long-range language modelling benchmark. We show samples from this model in Supplementary Section F The model is able to generate long-form narrative of varying styles: from character dialogue, first person diary entries, to descriptive third-person text.
5.2 ENWIK8
We compare the TransformerXL and the Compressive Transformer on the standard character-level language modelling benchmark Enwiki8 taken from the Hutter Prize (Hutter) 2012). which contains 100M bytes of unprocessed Wikipedia text. We select the first 90MB for training, 5MB for valida- tion, and the latter 5MB for testing as per convention. We train 24-layer models with a sequence window size of 768. During training, we set the TransformerXL's memory size to 2304, and for the Compressive Transformer we use memory of size 768 and compressed memory of size 1152 with compression rate 3. During evaluation, we increased the TransformerXI. memory size to 4096 and the compressed memory in model to 3072 (after sweeping over the validation set), obtaining the numbers reported in Table 4 We show the effect scaling the compressed memory size and evaluation performance in Supplementary Section C The proposed model achieves the new ttate-of-the-a on this dataset with 0.97 bits bits-per-character.
We sweep compression rates 2, 3. and 4 and report results with the best performing value for each row. BPTT signifies that auxiliary compression loss was used to train the network other than the
We compare compression functions and the of auxiliary losses in Table 5

---

Table 6: Validation and test perplexities WikiTexx-
Valid.
Test
LSTM (Graves et al.
2014)
48.7
Temporal CNN (Bai et al.
8a
45.2
GCNN-14 Dauphin el al.
2016)
37.2
Quasi-RNN Bradbury
20166
32
33
RMC (Santoro el al. 2018)
30.8
31.9
LSbM+Hebb. (Rae et al. 2018
29.0
29.2
Transformer (Baevski and Auli
2019)
18.7
18I. TransformerXI, M=384 (Dai al. 2019)
18.3
18L TransformerXL. M- (ours)
18.1
18L Compressive Transformer, M=1024
16.0
17.1
overall training loss. To feed gradients into the compression function we unrolled the model over double the sequence length and halved the batch size fit the larger unroll into memory.
5.3 WIKITEXT-103
We train an eighteen-layered Compressive Transformer on the vocabulaay word-level lan- guage modelling benchmark WikiText-103, which contains articles from Wikipedia. We train the model with compressed memory size, memory size, and sequence window size all equal to 512. We trained the model over 64 Tensor Processing Units (TPU) v3 with batch size of 2 per core making for total batch size of 128. The model converged in a little over 12 hours. We found the single-layer convolution worked best, with compression rate of c 4. This model obtained 17.6 perplexity on the test set. By tuning the memory size over the validation set - setting the memory size to 500, and compressed memory size 1, 500
obtain 17.1 perplexity. This is 1.2 per- plexity points over prior state of the art, and means the model places a ze 5% higher probability on the correct word over the prior SotA TransformerXL.
is worth noting that in Table 6 we do not list methods that use additional training data, or that make of test-time labels to continue training the model the test set (known as dynamic evaluation Graves)] 2013)). If we incorporate a very naive dynamic evaluation approach of loading model checkpoint and continuing training over one epoch of the test set, then we obtain a test perplexity of 16.1. This is slightly better than the published 16.4 from Krause et al. 2019 which uses more sophisticated dynamic evaluation approach on top of the TransformerX However in most settings, one does not have access to test-time labels and thus we do not focus on this setting. Furthermore there has been great progress in showing that more data equates much better lan- guage modelling; (Shoeybi et al. (2019) find large transformer 8B- SB-parameter transformer trained on 170GB text obtains 10.7 word-level perplexity on WikiText-
However is not clear to what extent the WikiText- 103 test set may be leaked inside these larger training corpora. For clarity of model comparisons, we compare to published results trained on the WikiText- 103 training set.
We break perplexity down by word frequency in Table 7 and see the Compressive Trans- former makes only a small modelling improvement for frequent words (2.6% over the Trans- formerXI. baseline) but obtains much larger improvement of 20% for infrequent words. Fur- thermore, we see 10X improvement in modelling rare words over the prior state-of-the-art LSTM language model published in 2018 which demonstrates the rate of progress in this area.
5.4 COMPRESSIBILITY OF LAYERS
We can use compression to better understand the model's mode of operation. We inspect how compressible Transformer's activations are as they progress through higher layers in the network. One may expect representations become more difficult to compress at higher layers, if more semantic information is represented there. We monitor the compression loss at each layer of our pessorerrfoming Compressive Transformer models trained on Enwik8 and WikiText- 103 and display these in Supplementary Section[B] B Figure We note that the compression loss is about one order of magnitude higher for word-level language modelling (WikiText-100 103) over character-level langauge

---

Table 7: WikiText- 103 test perplexity broken down by word frequency buckets. The most frequent bucket is words which appear in the training set more than 10, 000 times, displayed on the left. For reference, a uniform model would have perplexity [V] 2.6e5 for all frequency buckets. *LSTM
comparison from Rae et al. (2018)
10K
1K-10K
100
1K
100
All
L
12.1
219
1.197
9,725
36.4
TransformerXL (ours)
7.8
61.2
188
1.123
18.1
Compressive Transformer
7.6
55.9
158
937
17.1
Relative gain over TXL
2.6%
9.5%
21%
19.9%
5.8%
0.14
Avergge sttetton weight
12-
0.12
0.10
1.1
Change learning rate
training
0.00
Leamning Rate
0.0
3e undate period-2
0.00
10
5000
10000
15000
20000
25000
Compressed
Merory
Training iterations
Figure Attention weight on Enwik8. Av- erage attention weight from the sequence over the compressed memory (oldest), memory, and sequence (newest) respectively. The sequence self-attention is causally masked, so more at- tention is placed on earlier elements in the se- quence. There an increase in attention at the transition from memory to compressed memory.
Figure 3: Learning rate analysis. Reducing the learning rate (e.g. to zero) during training (on Enwik8) harms training performance. Reduc- ing the frequency of optimisation updates (ef- fectively increasing the batch size) is preferable.
modelling (Enwik8). Furthermore the first layer of the Transformer is highly compressible. However there is not clear trend of compression cost increasing with layer depth.
5.5 ATTENTION
We inspect where the network is attending to on average, to determine whether it is using its com- pressed memory. We average the attention weight over a sample of 20, 000 sequences from a trained model on Enwik8. We aggregate the attention into eighteen buckets, six for each of the compressed memory, memory, and sequence respectively. We the size of the sequence, memory and com- pressed memory all to be 768. We plot this average attention weight per bucket in Figure 2 with a la standard eиror. We see most of the attention placed on the current sequence; with greater weight placed on earlier elements of the sequence due to the causal self-attention mechanism which masks future attention weights. We also observe there an increase in attention from the oldest activations stored in the regular memory, to the activations stored in the compressed memory. This goes against the trend of older memories being accessed less frequently - and gives evidence that the network is learning to preserve salient information.
5.5.1
OPTIMISATION SCHEDULE
We make observation about an interesting but undesirable meta-learning phenomenon during long-context training. When the learning rate tuned to be much smaller (or set to zero) during training. performance degrades drastically both for the TransformerXL and the Compressive Trans- former. This is displayed in Figure
Usually we consider distributional shift from the training data to the test data, but we can also observe a shift in the model when transferring from a training to evaluation mode (even when the model is evaluated on the training data). In this case, this is due to the online updating of parameters whilst processing long contiguous articles. We would like the model to generalise well to scenarios where it is not continuously optimised. Updating the parameters only at article boundaries (and then
8

---

resetting the state) could be one solution for long-range memory models, but this would slow down learning significantly.
Instead, propose reducing the frequency of optimisation updates during training. We find this allows for the best of both worlds - fast initial learning with frequent updates, and better gen- eralisation near the end of training with less frequent updates (e.g. every 4 steps). Reducing the optimisation frequency increases the effective batch size, which has also been shown to be prefer- able to learning rate decay in image modelling (Smith et al. 2018). We observed final performance improvement in our TransformerXI baseline on Enwik8, from 0.995 - which approximately repli- 0.984 which matches the most recent SotA architecture. We note,
cates the published result- the additional space and compute cost of accumulating gradients is negligible across iterations, so there was no performance regression in using this scheme.
5.6 SPEECH
We train the Compressive Transformer on the waveform of speech to assess its performance on different modalities. Speech is interesting because it is sampled at an incredibly high frequency, but we know it contains a lot of information on the level of phonemes and entire phrases.
To encourage long-term reasoning, we refrain from conditioning the model speaker identity or text features, but focus on unconditional speech modelling. We train the model on 24.6 hours of 24kHz North American speech data. We chunk the sequences into windows of size 3840, roughly 80ms of audio, and compare a 20-layer Compressive Transformer to a 20-layer TransformerXL and a 30-layer WaveNet model (Oord al., 2016) used to serve production speech synthesis applications at Google (Oord et al., 2018). All networks have approximately 40M parameters, WaveNet is more parameter-efficien per layer. We train each network with 32 V100 GPUs, and a batch size of 1 per core (total batch size of 32) using synchronous training.
state-of-the-art audio generative model
WaveNet processes an entire chunk in parallel, however the TransformerXL and Compressive Trans- former are trained with window size of 768 and a total memory size of 568 (for the Compres- sive Transformer we use 768 memory + 768 compressed). We thus unroll the model over the se- quence. Despite this sequential unroll, the attention-based models train at only half the speed of WaveNet. We see the test-set negative-log- lok--iilldooood in Figure E and observe that a Compressive Transformer with a compression rate of able to outperform the TransformerXL and maintain slim advantage over WaveNet. However we only trained models for at most one week (with 32GPUs) and would be advantageous to continue training until full convergence before defini- tive conclusions are made.
5.7 REINFORCHMENT LEARNING
Compression is good fit for video input sequences because subsequent frames have high mutual information. Here we do not test out the Compressive Transformer on video, but progress straight to reinforcement learning agent task that receives video stream of visual observations but must ultimately learn to use its memory to reason over a policy.
We test the Compressive Transformer (Espeholt et al.) 2018). Otherwise, we use the same training framework and agent architecture as described in the original work with fixed learning rate of 1.5e-5 and entropy cost coefficient of 2e-3. We test the Compressive Transformer on a challenging memory task within the DMLab-30 (Beattie et al.) 2016) domain, rooms select nonmatching object. This requires the agent explore room in visually rich 3D environment and remember the object present. The agent can then advance to second room where it must select the object not present in the original room. This necessitates that the agent both remember events far in the past, and also learn to efficiently reason about them.
drop-in replacement an LSTM the IMPALA setup
We fix both the memory and compressed memory sizes to 64. In Figure 5 we present results for range of compression rates, averaged over 3 seeds. We see that the best performing agents endowed able to solve the task to human-level. We note that the model
with the Compressive Transformer with compression rate 1 is unable to learn the task to the same proficiency. The speed of learning and stability seem to increase proportionally with higher rates of compression (up limit) i.e.
9

---

Compresiive Transfomer Emd
1.8
TransfomeniL
62-
NLL
1.86
60-
1.84
Compresson Rate
182
Iumman 400000
180
200030 250000 Training
129
Figure 4: Speech Modelling. We see the Com- pressive Transformer is able obtain competi- tive results against the state-of-the-art WaveNet in the modelling of raw speech sampled at 24kHz.
Figure 5: Vision and RL. We see the Com- pressive Transformer integrates visual informa- tion across time within an IMPALA RI. agent, trained on an object matching task.
the effective memory window of the agent and we find compression rate 4 to once again be the best performing. We see this promising sign that the architecture is able efficiently learn, and suitably use, compressed representations of its visual input and hope to test this more widely in future work.
CONCLUSION
In this paper we explore the notion of compression as a means of extending the temporal receptive field of Transformer-bar
sequence models. We see benefit to this approach in the domain of text, with the Compressive Transformer outperforming existing architectures modelling. To continue innovation in this area, we also propose new book-level LM benchmark, PG-19. This may be used to compare long-range language models, or to pre-train on other long- range reasoning language tasks, such NarrativeQA (Kočiský al. 2018).
long-range language
We see the idea of compressive memories is applicable not only to the modality of text, but also audio, in the form of modelling the waveform of speech, and vision, within reinforcement-lerrinng agent trained on a maze-like memory task. In both cases, we compare to very strong baselines (Wavenet (Oord et al. 2016) and IMPALA (Espeholt al. (018)).
The main limitation of this work is additional complexity, if the task one wishes to solve does not contain long-range reasoning then the Compressive Transformer is unlikely to provide additional benefit. However as means of scaling memory and attention, we do think compression is a simpler approach dynamic sparse attention which often requires custom kernels to make efficient. One can build effective compression modules from simple neural network components, such as convolutions. The compression components are immediately efficient to run on GPUs and TPUs.
Memory systems for neural networks began as compressed state representations within RNNs. The recent wave of progress using attention-base models with deep and granular memories shows us that it is beneficial to refrain from immediately compressing the past. However we hypothesise that more powerful models will contain mixture of granular recent memories and coarser compressed memories. Future directions could include the investigation of adaptive compression rates by layer, the use of long-range shallow memory layers together with deep short-range memory, and even the use of RNNs compressors. Compressive memories should not be forgotten about just yet.
ACKNOWLEDGEMENT
We thank Chris Dyer, Felix Gimeno, and Koray Kavukcuoglu for reviewing the manuscript. We thank Peter Dayan, Adam Santoro, Jacob Menick, Emilio Parisotto, Hyunjik Kim, Simon Osin- dero, Sergey Bartunov, David Raposo, and Daan Wierstra for ideas regarding model design. We thank Yazhe and Aaron Van de Oord for their help and advice in instrumenting speech modelling experiments. Finally, thank wider DeepMind colleagues for supporting this project with stimulating discussions, engineering infrastructure, and positive reinforcement signals.
10

---

REFERENCES
R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3159-3166, 2019.
A. Baevski and M. Auli. Adaptive input representations for neural language modeling, arXiv preprint 1809. 10853, 2019.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477: 2014.
S. Bai, J. Z. Kolter, and V. Koltun. Convolutional sequence modeling revisited, 2018a. URL https: http::://ppeeeeiiiee ew net wow.oooddmmurrtrri.
S. Bai, J. Z. Kolter, and V. Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:/810.00 11..//1..0000.... 880006682, 2018b.
C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, Legg. and S. Petersen. Deepmind lab. CoRR, abs/1612.03801. 2016. URL
http:// htpp://arxiv. org/ abs 1 1612.1111 03801
D. M. Blei, A. Y. Ng. and M. Jordan. Latent dirichlet allocation. Mach. Learn. Res., 3:993-1022, Mar. 2003. ISSN 1532-4435.
J. Bradbury, S. Merity, C. Xiong, and R. Socher. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576 2016.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One bil- lion word benchmark for measuring progress in statistical language modeling. arXiv preprint
arXiv:1312.3005, 2013.
R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv: (0t:::660.0000 01 704, 2016.
Z. Dai, Yang. Y. Yang. W. W. Cohen, Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer- xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 100::090100000 022660 2019.
Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. arXiv preprint arXiv: 1612.0661.00808. 2016.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 11Xiv 2018.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner ar- chitectures. In International Conference Machine Learning, pages 1406-1415 2018.
E. Grave, A. Joulin, and N. Usunier. Improving neural language models with continuous cache. arXiv preprint
2016.
A. Graves. Generating sequences with recurrent neural networks. arXiv preprint 108.00000. 2013.
A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Bart Barwinssk, S. G. Col- menarejo, E. Grefenstette, T. Ramalho, J. Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626):411, 2016.
D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprint 16Xiv:1609.0 0900,, 2016.
Hill, A. Bordes, S. Chopra, and J. Weston. The goldilocks principle: Reading children's books with explicit memory representations. arXiv preprini 15 2015.

---

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1335-1780 1997.
A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.00 09751, 2019.
M. Hutter. The human knowledge compression contest. URL. hup://prizee hutterl. net, 6, 2012.
N. Kalchbrenner, L. Espcholt, K. Simonyan, A. v. d. Oord, A. Graves, and K. Kavukcuoglu. Neural
machine translation in linear time. arXiv preprint arXiv: /610. 2016.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
T. Kočiskš, J. Schwarz, P. Blunsom, C. Dyer. K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
B. Krause, L. Lu, I. Murray, and S. Renals. Multiplicative Istm for sequence modelling. arXiv preprint arXiv: 1609,0799, 2016.
B. Krause, E. Kahembwe, I. Murray, and S. Renals. Dynamic evaluation of transformer language models. CoRR, hts/1904.08378. 2019. URL
08378
G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, and H. Jégou. Large memory layers with product keys. arXiv preprint arXiv:/907 /90../997.2..,,. 2019.
S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 07843, 2016.
T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010.
A. Oord, L.,, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. Cobo, F. Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, pages 3915-3923, 2018.
A. v. d. Oord, Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.034999 2016.
D. Paperno, G. Kruszewski, A. Lazaridou, Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, R. Fernández, K. Erk, et al. The lambada dataset: Word prediction requiring broad discourse context. Association for Computational Linguistics, 2016.
J. Rac, J. J. Hunt, I. Danihelka, T. Harley, A. W. Senior, G. Wayne, A. Graves, and T. Lillicrap. Scaling memory-augmente
neural networks with sparse reads and writes. Advances Neural Information Processing Systems, pages 3621-3620 2016.
J. W. Rae, C. Dyer, P. Dayan, and T. P. Lillicrap. Fast parametric learning with activation memo- rization. arXiv preprint arXiv: /803. 10049, 2018.
B. A. Richards and P. W. Frankland. The persistence and transience of memory. Neuron, 94(6); 1071-1084, 2017.
D. E. Rumelhart, G. E. Hinton, and R. Williams. Learning representations by back-propagating enror.. Nature, 323(688)):533, 1986.
A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap. Relational recurrent neural networks. In Advances in Neural Informa- tion Processing Systems, pages 7299-7310, 2018.
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, Casper, and Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2019.
S. Smith, P. jan Kindermans, C. Ying, and Q Le. Don't decay the learning rate, increase the batch size. 2018. /tt.ateee//ppeeeeeesseeee/////ep
12

---

S. Sukhbaatar, E. Grave, P. Bojanowski, and A. Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.077990 2019.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10450, 2019.
Z. Yang, Z. Dai, Y. Yang. J. Carbonell, R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autore- gressive pretraining for language understanding. arXiv preprint arXiv: (rXx:19906008237,, 08237, 2019.
L. Zhou, Y. Zhou, masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 9739-8748, 2018.
Corso, Socher, and C. Xiong. End-to-end dense video captioning with
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasum, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 4189- 4198. JMLR. org, 2017.
13

---

SUPPLEMENTARY MATERIALS
A TEMPORAL RANGE OF THE COMPRESSIVE TRANSFORMER
The TransformerXL. with a memory of size n has a maximum temporal range of x n with an attention cost of O + ntn)) (see Dai et al. (2019) for a detailed discussion). The Compressive Transformer now has a maximum temporal range of x (nig fm + C* (tem) with an attention cost of O + (1 Rem )). For example, setting Rem 7'm n/2 and 3 obtain maximum temporal range that is two times greater than the TransformerXL with an identical attention cost. Thus if learn in the significantly increased.
compressed setting, the temporal range of the model can be
B COMPRESSION ACROSS LAYERS
We inspect the compression loss broken down by the layer index, to investigate whether there is trend in network depth with how compressible the representations are. The compression loss here refers to the attentioon-reconsttuction attention loss. We plot this for a 24 layer trained model on Enwik8, and an 18 layer model trained WikiText-103. The compression loss for character- based language modelling is about one order of magnitude lower than that of word-level language modelling. The first layer's ss representations are highly compressible, however from then there no fixed trend. Some non-contiguous layers have a very similar compression loss (e.g. 4 & 6, 5 & 7) which suggests information being routed from these layer pairs via the skip connection.
Frwika
WikiTex-1002
1.6e-
1.6e-2
Compression Loss
1.2e3
1.2e22
8e-4
8e-3
463
12002 22224
10
15
17 18
Figure 6: Model analysis. Compression loss broken down by layer.
C COMPARISON OF COMPRESSED MEMORY SIZES
We compare the best test perplexity obtained for the Compressive Transformer trained on WikiTeext- 103 and Enwik8 across a range of compressed memory sizes. For both models, the best model used a ID convolution compression network with compression rate of 3. The Enwik8 model was trained with an embedding size of 1024, attention heads, 24 layers, an mlp hidden size 3072, a sequence window size of 768, and memory size of 768. We see the best compressed memory size is 3. 072 in this sweep, facilitating a total attention window 3840. The WikiText-109 model was trained with embedding size of 1024, adaptive inputs using the
parameters (Sukhbaatar et al. 2019). 16 attention heads, 18 layers, an mlp hidden size of 4096, a sequence window of size 512 and memory of size 512. The best compressed memory size is 1536 resulting in total attention window of c. 2048.
Compressed Memory Size Enwik8 BPC
512
1024
2048
3072
4096
1.01
0.99
0.98
0.97
1.00
Table 8: Compressed memory size test performance for Enwik8
14

---

Compressed Memory Size
256
512
1024
1536
2048
WikiText-103 Perplexity
18.2
17.9
17.6
17.1
17.7
Table 9: Compressed memory size vs test performance for WikiText-103
D PG-19 GREPROCESSING
The raw texts from the Gutenberg project were minimally pre-processed by removing boilerplate license text. We then also replaced discriminatory words with unique (DWx) token using the Ofcom list of discriminatory words f
E PG-19 TOPICS
We present top-words for some of the topics on the PG-19 corpus. These were generated with LDA topic model (Blei et al. 2003).
Table 10: Examples of top topies PG-19 corpus.
Geography
War
Civilisations
Human Condition
Naval
Education
Art
water people river emperor rome
roman
love
island
work
poet
religion religious
ship
school music
feet
greek
life
miles army city
life
men children
poctry
north death gods
moral
captain may
work
south battle
king
human coast social
literature
mountains city first
society
land child
art
sea
soldiers
caesar
man
great
education
great
lake power
great
virtue
found conditions
poem
rock thousand
romans
nature islands well written
mountain arms
athens
marriage
shore study english
country
empire
greece
women
voyage
best author
valley
upon
temple
christian
vessels
years
play
ice
country
age
time
possible
genius
west
time
egypt
law
english class style
F PG-19 SAMPLES
show a few different samples from the Compressive Transformer trained on PG-19, We use Nucleus Sampling with p 0.98 (Holtzman al. 2019). We choose extracts of books from the test set as prefixes. We see the model is able to continue in the style the text, creating artificial dialogue descriptive text, and remembering the names of characters
hundreds of words.
As the Compressive Transformer is trained without state resetting, it is actually slightly out of sample when provided with the (relatively) short contexts. This is because its memory and compressed memory may be still empty (whereas they are always full during training). However we see a trend of the samples usually improving towards the end.
https::/www http://wwwwwwwccctt/o/ooo www.kk//kkkkkk..kkkkkko data/assets/pll fi / wwtp://e1e111//1/11 01002391662 (com@RG-AOG.. pdf
15

---

Context from Baby Mine by Maragaret Mayo:
CHAPTER
Even in college Alfred Hardy was young man of fixed ideas and high ideals and proud of it.
His friend, Jimmy Jinks, had few ideas and no ideals, and was glad of it, and before half of their first college term had passed, Jimmy had ridded himself of all such worries as making up his own mind or directing his own morals. Alfred did all these things so much better, argued Jimmy, furthermore, Alfred LIKED to do them-Jimmy owed it to his friend give him that pleasure.
Compressive Transformer:
He did not like doing them in the most elegant of all possible marble walks-where freshmen and sophomores stand and talk and laugh and chatter till evening falls and look cool and still and cool and beautifully white in spite of the hot sun, but had to go there for a month before going home. happened one day that he and Jimmy were strolling together in the green shade of St. Martin's. ss and Jimmy naturally asked why he was so silent. He was very talkative as most boys are who have once got the habit of on shore, so the need for answering was soon supplied.
"I am wondering," he said, "why my old friend and chum, Jimmy Jinks, is living single life and so has no boys. Have you not noticed that many of his old friends had their own boys to follow in their footsteps and help, and often helped successfully too?"
"Oh, yes," said Alfred, "it plain enough then that he liked to live by himself and got none of his present pleasures or entertainments because of the other fellow's 's giving it to him."
"Then it IS sad," said Jimmy, "that an old friend like the friend of Big Lights, Jimmy Reynolds, should be living the single life and having no boys."
" "And it is pretty true, too," said Alfred, "yet I believe it is the best thing that could happen even now."
"And so do I," said Jimmy, "and also believe that it is time someone did something to make the things worse for the other's partner."
"And so do I," said Alfred, "and so do I, and so do I."
"What can we do?" asked Jimmy.
"I don't know," said Alfred, "but if someone will come down to the valley and go through a regular mill with the old forces work, doing the work of the two, and doing better than any other Miller he ever saw or ever hoped to see, think there will be good deal of change in the things and things be cared for for and for driven and counter-driven."
"I will come and do that mill," said Jimmy.
16

---

Context from The Diary of Samuel Pepys:
June 1st. Having taken our leaves of Sir W. Batten and my Lady, who are gone this mor-- ing to keep their Whitsuntide, Sir W. Pen and and Mr. Gauden by water to Woolwich, and there went from ship to ship to give order for and take notice of their forwardness to forth, and then Deptford and did the like, having dined Woolwich with Captain Poole at the tavern there. From Deptford we walked to Redriffe, calling at the half-way house, and there come into room where there was infinite of new cakes placed that are made against Whit- suntide, and there we were very merry. By water home, and there did businesses of the office. Among others got my Lord's imprest of L1000 and Mr. Creed's of L10,000 against this voyage their bills signed. Having wrote letters into the country and read some things went to bed.
Compressive Transformer:
So rose and did some business the office, and then to dinner, where great abundance of good things of the kitchen I only being partaker in the feast. After dinner went to look at the yard and found
very bad condition, things should make shift to get in my way and so I will leave off.
went home and there finding delight in seeing
So I took leave of my great friend, and the next day being Sunday, my wife and I and great number of friends went by water Deptford, and there took the oaths of allegiance and supremacy and promised such obedience to the Lord as should hereafter turn to the advantage of our State. Being come back we found the shops and warchouses in a pitiable condition, so my wife and and all the rest of the friends went home and there did the like and by Saturday day being alone in my office found myself in very bad situation, for my Lord, and Mr. Creed, and Pett were all standing ready receive and receive the oaths of allegiance and right to make use of the ten days' licence had allowed for this purpose by virtue the law that was such a help to the servant. Having looked the ships, I told my wife what had seen and that I was greatly troubled at my situation and, hope that will mend it.
So that is the end of a long and patient seeking after those things which in my opinion I deserve and may get if can only by fair means get them.
17

---

Context from The Patrol of the Sun Dance Trail by Ralph Connor:
High up on the hillside in the midst of a rugged group of jack pines the Union Jack shook out its folds gallantly in the breeze that swept down the Kicking Horse Pass. That gallant flag marked the headquarters of Superintendent Strong, of the North West Mounted Police, whose special duty was to preserve law and order along the construction line of the Canadian Pacific Railway Company, now pushed west some scores of miles.
Along the tote-road, which ran parallel to the steel, a man, dark of skin, slight but wiry, came running. his hard panting, his streaming face, his open mouth proclaiming his ex- haustion. At little trail that led to the left he paused, noted its course toward the flaunt- ing flag, turned into it, then struggled up the rocky hillside till he came to the wooden shack, with deep porch rumning round it, and surrounded by rustic fence which en- closed a garden whose neatness illustrated a characteristic of the British soldier. The run- ner passed in through the gate and up the little gravel walk and began to ascend the steps.
Compressive Transformer:
As though by mutual consent, all the three members of the Superintendent Force were standing on the veranda watching his progress, man came strolling slowly down the path, stopping often to gaze curiously about him, then stopping once more when he saw the four standing on the veranda. His face was dark, his eyes prominent, and the lines that furrowed his forehead showed marks of many woes, yet his step was alert and he began to mount the steps with steady strides.
The man's attention was diverted by the unexpected appearance of the two policemen; third man came slipping up, grumbling under his breath, and stopped long enough to look doubtfully around him, then slipped away again. In the meantime, the four on the veranda continued to watch the approach of the new arrival. In spite of himself, the Superintendent Force became conscious of curious ceeling-uneasiesss that was almost fear-in the presence of that strange man. He would not allow himself to admit the fact, yet he felt it in his bones. But to the watchers, the plain, everyday features of that stranger and his coming, seemed only just what the Seven White Shee owed him-their weight, their hurry, their blast.
Had chariot been made for the good of the three horses, and had the driver been given orders that he should speed them that he might win, they would have been heartening things in the sight of the veteran and the victor. To you they would have been unintelligible to the root of your understanding. When you gaze up in the faces of those four gray horses, you can see clearly through the clouds of dust that rise from their hoofs, and discern plainly where the banker is and where the hobo. Then you will understand why you shall not press the bitter grapes and why you shall not spurn the generous doctrines. You will understand why you shall not praise the lash or the spur, for you will know where the true would be and where the false would be. Then you will understand why you, man with reason and heart, need not tear your hair over- bitter and why you need not laugh over the blunders of an ignorant man.
About nine o' clock that morning, two buggies, drawn by powerful horses, crossed the Rubicon and turned the railroad from Sandhurst into the Hollow of the Mountains. And though the char- ioteers stood at their horses' heads, and their drivers cried at their loudest, there was not a man in the four teams who did not feel that his day was worth all the toil and all the peril that he had undergone. And if there were a man in them who did not know that-who did not feel that the road through the Hollow of the Mountains is made easy by the arrival of travelers and by the coming of government, there was who did not that moment care whether his day's work were worth all the toil and all the danger that he had had to endure or whether it were not worth more than all.
18

---

AUTHOR CONTRIBUTIONS
Model and Experiment design: JR, TL, AP, SJ
Dataset creation: AP, JR, CH
Text experiments: JR, AP
RL experiments: SJ
Speech experiments: JR
19


Reviews:
[{"review_id": "rkloDrohKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Updated review\n\nI have read the rebuttal. First I'd like to thank the authors for the detailled rebuttal. \nThe latest version of the paper adressed all my concerns, hence I change my rating to Accept.\n\n## Original review\n\nThis paper presents a new variation of the Transformer model, named Compressive Transformer. The key novelty of this model is to preserve long range memory in a compressed form, instead of discarding them as previous models have done. This improves the long-range dependencies modelling capabilities of the approach. The model is evaluated on two common language modelling benchmarks and yields state of the art results in both of them. The paper also introduces a new benchmark for long-range dependencies modelling composed of thousands of books. The paper finally presents an analysis of the compressed memory and provide some insights, including the fact that the attention model uses the compressed memory. The model is also evaluated on two other tasks: speech generation and reinforcement learning on videos.\n\nI think this paper should be accepted, mainly because:\n- The proposed model is novel as far as I can tell. \n- The presented approach is significant, as modelling long-range dependencies is an important milestone in sequence modelling.\n- The new benchmark is a good addition.\n- The comparison with the relevant literature is thorough and well done.\n- The experiments are convincing and demonstrate the viability of the approach, although some aspects can be improved (see below).\n\nDetailed comments:\n- About the character-level language modelling on Enwik8, the improvement is very small, it seems that the task doesn't benefit from have long-range memory, could it be because character-level modelling is less dependent on the long-range past? can the authors comment on that? It would also been interesting to evaluate the gain of the memory, for instance by varying the size of the compressed memory from 0 to 1152. \n- The WikiText-103 evaluation is interesting, specially Table 6, which shows the advantages of the model. However when comparing with the literature, it's not clear if the performance gain is due to the compressed memory or to the network capacity. A study with different lengths of the compressed memory (starting at 0) would bring some insights about that.\n- In Section 5.6.2: can the authors justify why the attention weights were split in only 6 bins? creating a trended curve on only 6 points could be problematic, and I don't see why more bins couldn't be used.\n- The speech analysis section (5.7) is not very insightful. It shows that the proposed model is on par with WaveNet on unconstrained speech generation, which is not very useful and feels a bit half-finished. I think that the authors should either commit to this study by constraining the model with linguistic features like in (Oord et al. 2018) and evaluate it in a TTS framework with subjective evaluation or discard this section entirely. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hylu-DRpYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates a so-called \"compressive transformer\" approach. The idea is to compress distant past memories into a coarse-grained representation while keeping a fine-grained representation for close past memories.  A variety of compression techniques and training strategies have been investigated in the paper and verified using tasks from multiple domains including language modeling, speech synthesis and reinforcement learning. Particularly, the authors propose a new benchmark PG-19 for long-term sequence modeling.  \n\nOverall, I found the work interesting and experiments are thorough and strong.   It is always great to see a new benchmark released to the community.  That being said, I have concerns regarding the paper.  The authors put huge amount of effort into the experiments but only describe the proposed technique in a very rough and abstract way, lacking necessary technical details to formulate the technique. What is the mathematical formulation of the problem?  How exactly the compression is carried out on various network architectures is not clear after reading the paper.  Also, I guess many readers including me do not have a perfect understanding of Fig. 1 although it shows something intuitively. (What is the difference between different colors? What is the difference between sequence, memory, and compressed memory?  What do the arrows mean? There is no explanation whatsoever either in the figure or in the caption).  This is the major concern I have regarding the paper.  Despite of the strong experimental presentation, lacking the technical details has significantly hurt the quality of the paper.  \n\nP.S.  Thanks for the rebuttal.  I have lifted my score. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxPYrrhYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a way to compress past hidden states for modeling long sequences. Attention is used to query the compressed representation. The authors introduce several methods for compression such as convolution, pooling etc. The outcome is a versatile model that enables long-range sequence modeling, achieving strong results on not only language model tasks but also RL and speech. For testing and evaluating the modeling of really long context sequence modeling, the authors introduce PG-19, a new benchmark based on Project Gutenberg narratives. \n\nThe idea is a simple and straightforward one. The choices of compression functions are intuitive and natural. The probably more interesting part of this paper is the training schemes designed to train the memory compression network. \n\nResults are very strong and there is a pretty diverse set of experiments. That said,  it seems like a huge amount of resources were spent on this work alone. It also seems like these models are not trivial to train (or get them to work). It would be interesting to find out how much resources were spent (in terms of preliminary experiments) to getting these models to start working decently. There are also no reports of parameter counts, which might make the experiments unfair. \n\nAchieving SOTA is one thing, which could be attributed to large resource pools and maybe larger parameter sizes of models.\n\nOverall, I am voting for a weak accept. While this paper is more incremental and novelty may be slightly lacking, I think the breadth of experiments and competitive results warrants an acceptance. \n\nSeveral issues and questions for the authors:\n\n1) Why are the results on PG-19 not reported in a Table format? Why are there no results of the base Transformer on PG-19? I think this is really necessary and should be reported.\n2) The authors mention that this memory compression architecture enables long sequence modeling. However, is there an intended way of use for long-text that is not necessarily framed as a LM problem? For instance, results on NarrativeQA benchmark would be nice. \n\nUPDATE: I have read the author response and other reviewer's comments. I am happy with the efforts made by the authors and I am raising my score to 8 (accept). \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a "compressive transformer", an extension of the transformer, that keeps a compressed long term memory in addition to the fixed sized memory.  Both memories can be queried using attention weights.  Unlike TransfomerXL that discards the oldest memories, the authors propose to "compress" those memories.  The main contribution of this work is that that it introduces a model that can handle extremely long sequences. The authors also introduces a new language modeling dataset based on text from Project Gutenberg that has much longer sequences of words than existing datasets.  They provide comprehensive experiments comparing against different compression strategies and compares against previous methods, showing that this method is able to result in lower word-level perplexity. In addition, the authors also present evaluations on speech, and image sequences for RL.

Initially the paper received weak positive responses from the reviewers. The reviewers pointed out some clarity issues with details of the method and figures and some questions about design decisions. After rebuttal, all of the reviewers expressed that they were very satisfied with the authors responses and increased their scores (for a final of 2 accepts and 1 weak accept).

The authors have provided a thorough and well-written paper, with comprehensive and convincing experiments. In addition, the ability to model long-range sequences and dependencies is an important problem and the AC agrees that this paper makes a solid contribution in tackling that problem.  Thus, acceptance is recommended.
Decision: Accept (Poster)