Adding Recurrence to Pretrained Transformers
Authors: Davis Yoshida
Allyson Ettinger
Kevin Gimpel
Venue:  (2021)
Keywords: Language modeling
Transformers
Recurrence
Gradient checkpointing
Pretraining

Abstract: Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years. While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures. We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time. An additional benefit is that our method removes the fixed context size constraint that most transformer models have, allowing for more flexible use. When applied to the GPT-2 language model, we find that our method attains better perplexity than an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.

Under review as a conference paper at ICLR 2021
ADDING RECURRENCE TO PRETRAINED TRANSFORM- ERS
Anonymous authors Paper under double-blind review
ABSTRACT
Fine-tuning a pretrained transformer for downstream task has become a stan- dard method in NLP the last few years. While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures. We present novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time. An additional benefit is that our method removes the fixed context size constraint that most transformer models have, allowing for more flexible use. When applied to the GPT-2 language model, we find that our method attains better perplexity than unmodified GPT-2 model on the PG-19 and WikiText- 103 corpora, for given amount of computation or memory.
INTRODUCTION
Recent progress in NLP has been dominated by large pretrained transformer neural net- works (Vaswani et al, 2017), such as BERT (Devlin et al. 2019), and GPT-2 (Radford et al.] 2019). However, these models have memory footprint that is quadratic in input sequence length. Al- though architectural innovations such as those of Kitaev et al. (2019) and Rae et al. (2019) mitigate this and the issue of predetermined maximum context size, large pretrained models applying these techniques are not available at this time. Even if large pretrained models of this kind are released in the future, they will likely not cover the wide range of domains that BERT-family models have been published for. For example, there have been BERT-based models trained for other languages such as French [Le al. 2020 Martin et al. 2020), Italian (Polignano et al. 2019), and many other languages (see Nozza et al. (020) for an overview) as well as specific domains such scientific papers (Beltagy et al. 2019), biomedical papers (Lee et al. 2020), and health records (Rasmy al.) 2020). Individuals working with these models may not have the resources to train new models from scratch using the latest tricks, as the computation requirements for pretraining are extremely high. As such, identifying ways that already existing models can be improved could be widely impactful.
Another drawback this family of models is that they have an a priori fixed maximum context size (typically 512 or 1024 tokens for the currently available pretrained models). A typical application of pretrained language models is producing contextual embeddings for a document. If the document simply chunked into disjoint segments of 512 tokens, tokens the boundary of window will have less contextual information than tokens in the center of a window. This can be mitigated by striding the evaluation of the model, and only keeping the embedding for a token which has the largest context- but this adds quite bit of wasted computation.
In this paper, we propose method for augmenting and fine-tuning pretrained transformer language models to use context without directly attending to it. Our method simultancously allows for in- creasing the context size transformer processes, while allowing controllable trade-off between computation and perplexity. We accomplish this by adding small recurrence module that com- putes a fixed size representation from the transformer hidden states in a window of text. Then, the representation for that window is used during processing of the next window. Shrinking the window size then a way to reduce the memory footprint of the model. with less loss of performance than would occur with standard transformer. Our experiments add recurrence GPT-2 language mod- els, and fine-tune them on the PG-19 (Rae et al.] (019) and WikiText- 103 corpora (Merity et al.) 2016), and require only the same amount of memory used for standard fine-tuning of pretrained

---

Under review as a conference paper at ICLR 2021
language model. We demonstrate improvements in perplexity compared baseline model using the same amount of computation. Qualitative analysis shows that our recurrent module propagates certain information from previous windows of text, which can facilitate handling of long-distance dependencies with fixed-size input windows.
RELATED WORK
Many methods have been proposed lower the memory footprint or computation time of trans- former language models, or allow them to be used on larger contexts. The Tran (Dai et al. 2019) allows a position within an attention window to attend to tokens from the previous windows by introducing relative position embeddings. While that mechanism, like ours, allows in- formation to flow between windows, existing BERT and GPT-2 models do not use relative position embeddings, so training from scratch would be necessary to take advantage of this architecture. Ad- ditionally, each layer in the Transformer-X attends to the previous layer in the previous window, so the maximum attention horizon is finite. Our recurrent method could theoretically pass informa- tion across an arbitrary distance, although one would not expect it to exceed the Transformer-XL's horizon without much larger scale of data than we experiment with.
We list here some other modifications of the transformer architecture, somewhat imprecisely group- ing them for brevity. For more detailed discussion, see Tay et al. (2020b). Child al. (2019), Qiu et al. (2019). Kitaev et al. (2019), Sukhbaatar et al. 2019), and Roy et al. (2020) introduce sparsity self-attention in various forms, reducing its memory cost. Rae al. (2019) and Bellagy et al. (2020) dynamically and statically respectively add extra tokens to attend to which allow for global passing of information. (Ta al. (2020a) and [W et al. (2019) replace dynamically com- puted self-attention with cheaper alternatives. While the above methods all allow for reduction computation, they also all require training from scratch. Our goal to allow more efficient and powerful use of the wide array of existing pre-trained models that cover many domains.
Cao et al. (2200) propose the DeFormer, which also modifies the execution of a pretrained trans- former. However, unlike our method, they decompose single window into multiple windows by re- moving the attention interactions between these windows. This is largely orthogonal to our method, one could both decompose windows of text, and additionally use method allow information to be passed between neighboring windows. Similarly, distilled versions of pre-trained models such DistilBERT (Sanh al., 2019) provide more computational efficiency, but could be combined with our method to apply them to longer contexts, or reduce the quadratic cost of self-attention.
Hao et al. (2019) apply pre-trained transformers recurrently for machine translation, but do so by using an attention network to embed the document, applying a recurrent encoder to those embed- dings, and using the recurrent encoder alongside a typical transformer encoder. This differs from our method as we are fine-tuning language models, which are transformer decoders, and directly modifying the transformer's computation with a recurrent connection, rather than running an RNN on top of embeddings produced by a transformer.
METHOD
The main idea our method is to take a transformer that was pretrained in a fixed context size setting and add recurrence at the level of T-token windows of text. For example, instead of executing the model on one 1000 token window of text, we could instead execute our model with 10 windows of 100 tokens. The first window is processed by the transformer model normal, but for subsequent windows we add a supplementary embedding, which is generated using the hidden states from the preceding window (see Figure []. The recurrence module is extremely small compared the size of transformer language model, so the additional computation required is negligible.
3.1 ADDING RECURRENCE TO PRETRAINED TRANSFORMERS
Starting by defining terms, we will consider pretrained transformer with L layers, a hidden state size of k, and a maximum context size of tokens. Let h 9 e R be the output of the e-th layer of the pretrained model, position i. To produce fixed-size representation of tokens 41, 22....2 Lr,
2

---

Under review as a conference paper at ICLR 2021
P ((aaera)))
P (rpeneree)
Pretrained Transformer
Protrained Transformer
Agrev
Ag
(ite
hy Mimi
MLP
Cine -1
Cire
Augmenting a pretrained trans- former with recurrence module, allowing reduction of attention computation as well as simpler processing of longer contexts.
Figure 1:
Figure 2: Agrev is added as additional key and value to one self-attention layer. Arrows show which positions can pass information to which other positions.
the embeddings produced by the pretrained transformer are mean-pooled as follows:
T L
21
weh
(1)
T 1=
where wy are weights softmax-normalized from learned parameters dg:
exv
Weg
ea
The fixed-size representation, 21, is passed through a feedforward network to produce an embedding 1prev.1 which represents the tokens processed so far, (1. Next, instead of evaluating the pretrained transformer without modification on positions T layer (denoted lis) of the pretrained model, as an additional embedding that may be used in the computation of attention, as shown in Figure [ To keep the number embeddings per layer fixed, this embedding only used key and value, but a query, in the self-attention layer. That is, for a window size of 300 tokens, there are 301 inputs to layer fims but still only 300 outputs. The embeddings for positions + to 2T are then pooled the same way as Equation [ to produce 22 and passed through the feedforward network, outputting fpp.v.2 hprev, is used to modify the execution of the pretrained language model on tokens 27 + 1 through 37, and so on. Because the model is now being applied recurrently, it is trained end-to-end with backpropagation through time.
through 27', hpeev1 inserted a single
One could consider more complex recurrence modules, other methods for pooling the previous win- dow's sw's embeddings, or for inserting fr into the computation for the next window. We experimented with modifications such as max pooling instead of mean pooling, inserting multiple embeddings into the next window, inserting an embedding at all layers of the transformer for the next window, and using fixed key attention as the pooling function. However during our preliminary experiments, were not successful in finding significantly higher performing architecture than the one given above, so it is the one we present results for.
3.2 GRADIENT CHECKPOINTING IN NETWORKS WITH BOTTLENECKS
While our method can reduce the quadratic cost of attention by splitting the input into windows, we can also easily apply it to much longer contexts by use of gradient checkpointing (Chen et al.) 2016).
method for lowering the peak memory requirement of training large neural networks. This is accomplished by storing only subset of activations during the forward pass, and recomputing forward from those cached states during the backwards pass. For example, in 100 layer feedforward network with uniformly wide layers, every 10th layer. Then, during the backward pass, in order to compute the gradients for the 95th layer, would compue layers 91 through 99 using the stored 90th layer activations. The overall memory cost is reduced to / T. the cost of a single additional forward pass.
Gradient checkpointing
could store the output of only
3

---

Under review as a conference paper at ICLR 2021
(c) Intermediate degree of over- lap. Except the first prediction, all predictions attend to at least tokens of context.
(a) Disjoint exccution. Predic- tions have context ranging be- tween and 3 tokens.
(b) Maximum overlap. All pre- dictions except the first two have maximal context.
Figure 3: Varying degree of overlap while evaluating transformer with a window size of 3. The green (top) circles are outputs, and the blue (bottom) circles are inputs.
In a network with variable width, the memory reduction can be even larger. When gradient check- pointing is applied to transformers, the outputs of each layer are usually stored (k x L. x T values), so that at most one set of self-attention activations in memory at once. In the case of our recurrent models, we have an even narrower bottleneck: the z 's and Apee,,''. Storing only these values means that the maximum number of activations present memory while training on sequences tokens in length is M + 2 N 1. where M is the number of activations stored when training the transformer on an individual window of length T. Because & is extremely small compared to M, our model can be applied to very long contexts on any GPU which the pretrained model can be fine-tuned.
REVISITING THE EVALUATION OF TRANSFORMER LANGUAGE MODELS
Before describing the empirical evaluation of our method, we discuss how transformer language models are evaluated in related work. The standard way of measuring perplexity uses extra com- putation in order to make as much context available for each token prediction. This yields low perplexities, but does not reflect how practitioners use transformer language models in applications. In this section, we describe the situation in detail and propose practical solutions that achieve rela- tively low perplexities while being closer to how transformers used in practice.
4.1 POTENTIAL MISALIGNMENT BETWEEN LM EVALUATION AND APPLICATION
Transformers are often described as having quadratic time complexity in comparison to RNNs which have linear time complexity. However, this can be somewhat misleading when comes to evaluation of perplexity. Given test set of length N, RNN requires O(N) time to evaluate- but reaching the best perplexity for a transformer requires ONT ). where T is its maximum context size. (These time complexities exclude hidden state size, number of layers, and batch size.) This much higher time complexity due to the fact that a transformer may be run with its full context size once for each token in the test set, so that the maximum context available for each prediction. Re-execution of the whole model for each token is required for models with absolute position embeddings, since hidden state reuse is only possible up the maximum context size of the network. Note that is possible to achieve smaller wall-clock time by splitting evaluation of test set over multiple GPUs, but this is not applicable to the generation setting where outputs depend on prior ones.
To illustrate why re-computation is necessary, consider executing GPT-2 (which has 1024 position embeddings) on a test set. Each of the first 1024 tokens of a test set will have been passed into the network using a distinct position embedding. Having exhausted the position embeddings, one option to start again with the 1025th token being treated as position 1-we will refer to this as disjoint execution, illustrated in Figure 3a The issue with disjoint execution is that it requires predicting the tokens at the beginning of a window from a very small amount of context.
The alternative, which is used for standard test set evaluation, is overlapped execution, shown in Figure 3b) The position embeddings are advanced by one position for each prediction, meaning that T- 1 tokens are repeated between consecutive evaluations of the transformer, requiring much more computation. The benefit of this method is that it allows a model with position embeddings to have T tokens of context for each prediction, opposed
variable amount between and T.
Stepping a transformer decoder forward one token at a time measures the best that such a model could perform, but it reflects generative story that does not align with how the models may be used
4

---

Under review as a conference paper at ICLR 2021
in practice. A perplexity that only measures the ability of GPT-2 to generate the 1024th token given a context of 1023 tokens is not necessarily indicative of the model's 's performance when generating from smaller context. For example, the popular website Talk To Transforme"ggeneeee generates samples from GPT-2, but only provides 150 tokens of output. The evaluation of GPT-2 by stepping forward one token at time provides little information about the quality of such generations.
An example where the discrepancy is length instead of brevity is the GPT backed text adventure game Al Dungeon P In this setting, the number of tokens can easily reach and exceed the full context size GPT-2 was trained on. Using overlapped execution as described above, generating each token would be 1024 times slower than with disjoint execution, so perplexity calculated by overlapped execution does not match this use case either.
While lower perplexity seems to correspond to better generation with shorter contexts in practice (perhaps due to parameter sharing between all sequence positions), there is no reason that this need be the case in principle. To demonstrate an extreme case of the concern being discussed, let F be a transformer model with vocabulary V, which uses the previous 1023 tokens as context, and consider the following generative story for generating token 4;:
Uniform(V)
i S 1023 otherwise
t
((((-1023)(0-1)
Clearly the above generative model would not be of any practical use for generation or otherwise. However, because perplexity is calculated per token, increasing the size of the test set will lead to measured perplexity that approaches that of a standard evaluation of the model F. This example is not representative of the models that trained in practice, as even generations much shorter than the maximum context size from a GPT-2 model are quite impressive. However, it does demonstrate that the criteria that we use compare models, or to select the best model during early stopping, place very high weight on the ability of the model to produce text given full context, and poten- tially vanishingly small amount on its ability to generate text using shorter contexts.
4.2 VARYING OVERLAP FOR EVALUATION
As we are interested in increasing computational efficiency at evaluation time for pretrained models, we investigate their performance using overlapped execution, but with a reduced degree of overlap between windows. Varying the overlap lets us investigate the connection between degree of overlap and perplexity. The overlap used in evaluation will be defined to be the number of tokens from each input window that are repeated in the next window (see Figure 3}. For example, consider a window size = 10 and an overlap of 3. The windows that the transformer will be executed are then f1:10- 18:17, f15:24. .... 11+7n:1077 where n indexes the window. These input windows are used to predict the spans of tokens 12.11 1 112 19.25 ........ ..... 15 7m:17 7n:11+ 4n Figure Be aciillustrtes an intermediate overlap setting with T = 3 and an overlap of 1. The perplexity-minimizngg evaluation setting is then the extreme with overlap T 1, and an overlap of 0 corresponds to disjoint execution.
While a transformer can be evaluated with any degree of overlap, our augmentation method produces the embedding fuprev. which is used during training to help predict the first token of window. If we change the overlap at test time, the alignment of the text represented by hprev and the current window will be different than the model was trained for, and so performance will degrade. To address this, we use the same overlap that will be used at test time during training for the recurrent models!"
EXPERIMENTS
We now present experiments comparing our proposed technique to the default usage of transformer language models. We describe experiments on the WikiText- 103 corpus and a subset of the PG-19
hatps//tllkkorrasfoomer..ommm
butps:/aiiunneenn.i.. Note that AlDungeon now uses the OpenAl GPT-3 API, but a similar project without OpenAl API access would still have use GPT-2.
Evaauaating recurrent models trained with no overlap between adjacent windows on different level of over- lap is possible by changing which positions are pooled. We found that it led to a slight increase in perplexity, so report results with training and evaluation matching.
5

---

Under review as a conference paper at ICLR 2021
corpus, using the GPT-2-small language model as the pretrained transformer in models. We also provide proof-of-concept experiments using RoBERTa (Liu et al. 2019) on the HotpotQA (Yang et al., 2018) question answering dataset, indicating that our method can improve encoder perfor- mance for tasks other than language modeling. All of our experiments are based on the Hugging Face Transformers library (Wolf al. 2019).
WikiText- 103 is standard language modeling corpus composed of approximately 29,000 docu- ments from English Wikipedia, containing 103 million words. We use the WikiText-101 103 "raw" corpus, which does not have rare words replaced by "UNK". While GPT-2 uses BPE tokenization, we compute perplexity using the number of words rather than the number of BPE tokens for clarity.
Although WikiText-103 does test long term dependencies, many of the documents are still shorter than the context size of the models we test. Therefore, we also use PG-19, which consists of books from the Project Gutenberg corpus. The average length of a WikiText- 103 document is 3.6K words, while PG-19 documents (i.e. books) average 69K words, which far exceeds the context size of the models we test. However, the full PG-19 dataset is over 20 times larger than WikiText- 100,, so we use only subset of it for training due to computational constraints. Specifically, we use only the first (alphabetically by filename) 1250 books of the PG-19 corpus, and use only the first 15000 tokens of each of the books in the validation set for early stopping. We make no modifications to the test set.
In all our experiments we use the HuggingFace implementation of the pretrained GPT-2 small model (12 layers, 768-dimensional hidden state). For both the recurrent and baseline models, the GPT-2 model was fine-tuned, not left frozen. We selected learning rates for both models and the baseline separately, by evaluating on WikiText-103 for the same set of candidate learning rates. We used the same learning rates for the PG-19 experiments without further hyperparameter search. We fine-tune all models for 2 epochs, measuring the validation loss every 2 million tokens. All models were trained with Adam (Kingma Ba 2014), warming the learning rate up linearly from its final value over 100 steps. The feedforward network used to produce hprev.d from window i - 1 consisted of hidden layers with dimension 200. We fixed lins to be 2
Recall from Section 4 that we are interested in evaluating the models in a setting similar how they would be used in practice. To that end, we report separate perplexities for different degrees of overlap between adjacent windows of text, as described in Section 4.2 For our models, we train with the same overlap that we test with, as unlike the baseline models, they cannot be trained with no overlap between adjacent windows and then tested with an overlap. This because the embedding of the previous window of text is expected to represent all tokens up until the first token of the current window, but with an overlap of 30 for example, that embedding would be representing all tokens up until the 30th token of the current window.
5.1 RESULTS
We first show that with the same amount of fine-tuning, method achieves lower perplexity than a baseline GPT-2 model when evaluated using the same window size and degree of overlap between adjacent windows of text.
important to emphasize that the perplexities we report are based on pretrained models, and so should not be compared to models trained from scratch these datasets. The GPT-2 models were trained on text from a web crawl from which all Wikipedia documents are removed, but this still leaves open the possibility of quotes from Wikipedia having been encountered, or text from PG-19.
Table the perplexity of our models and the non-recurrent GPT-2 models on the WikiText-103 dataset. The models compared here all use windows of 300 tokens, with varying degrees of over- lap. The baseline models can only access information from the previous window of text through the overlapping tokens, while the recurrent models have fixed size representation of the longer context. Our addition of recurrence increases the performance of the GPT-2 models in this setting, but by relatively small amount. Increasing the overlap between each window of text decreases the perplex- ities of the baseline model as expected, but also decreases the perplexity of the recurrent models 5
*During our preliminary experiments, we found that setting lin be one of the final layer in the network gave slightly worse results, but we did not re-tune this hyperparameter for PG-19 or our final architecture. 'ee did not attempt to train recurrent models with extremely high overlaps, as that would greatly increase the required training time.

---

Under review as a conference paper at ICLR 2021
Table 1: Results on WikiText-103
Model
Overlap
Validation Perplexity
Test Perplexity
FLOPs/token
0
29.00
30.47
1.75 10
5
27.99
29.36
1.78 10a
GPT-2 (small), 300
10
27.58
28.88
1.81 10
token window
30
26.72
27.96
1.94 10
50
26.17
27.31
2.10 x 10
0
27.70
29.01
1.75 10
Recurrent, 20
5
26.88
28.12
1.78 x 10
windows of 300
10
26.51
27.77
1.81 10
tokens (Ours)
30
25.90
27.12
1.94 10
50
25.53
26.73
2.10 x 10
Table 2: Results PG-19
Model
Overlap
Validation Perplexity
Test Perplexity
FLOPs/token
0
172.25
147.71
1,75 x 1
165.93
142.30
1.78 10
10
162.66
139.49
1.81 10
GPT-2 (small), 300
30
156.21
134.30
1.94 10
50
152.64
131.25
2.10 10
token window
75
149.54
128.46
2.33 10
100
147.05
126.51
2.62 10
150
143.62
123.53
3.50 10
200
141.14
121.40
5.25 10
155.27
133.02
1.75 10
Recurrent, 20
5
150.00
128.78
1.78 108
windows of 300
10
147.53
127.05
1.81 10
tokens (Ours)
30
142.35
122.22
1.94 10
50
140.10
119.93
2.10 10
This indicates that there is room to increase the capacity of the recurrence mechanism (potentially requiring more training data), as an ideal recurrence mechanism would render these overlapping tokens redundant. On the other hand, some useful information beyond what is contained in the local context is being propagated, as otherwise the baseline model should catch up in perplexity at higher overlaps. To investigate this further, also experiment with the PG-19 19 dataset.
The results for the PG-19 experiments are shown in Table 2 While we find only small increases PG-19, confirming
in performances the WikiText- 103 dataset, our prediction that the gains would be larger on a dataset that has a larger context available for each recurrence module leads to model that gives
larger improvements
prediction average. We find that adding good perplexity with no overlap between adjacent windows as an unmodified model does when evaluated with an overlap of 30 out of 300 tokens in each window. Training the recurrent model with a 5 token overlap gives perplexity lower than the baseline perplexity with an overlap of 50 or even 75. In terms of FLOPs, adding our recurrence module and overlapping adjacent windows of tokens by 50 is less than half as costly as using non-recurrent model with an overlap of 200.
5.2 EFFECT OF WINDOW SIZE
As one our motivations is to retain performance while decreasing compute requirements, we experiment with varying the window size used by model and unmodified GPT-2 model. At smaller window sizes the recurrent model has access to much more information than GPT-2, which only attend to the current window. Because of this, expect augmentation to cause the

---

Under review as a conference paper at ICLR 2021
(cverippe,,
200
Validation Perplexity (PG-19)
200
No recurrecce (Baselina)
Valddation Parpeedy 18
190
190
170
170
160
160
150
140
130
300
600
Window size
2.0 per
Figure 4: Effect of window size on perfor-
Figure 5: Relationship between FLOPs and perplexity. Curves range over window sizes from 200 to 600.
mance on PG-19 validation set.
performance to fall off less rapidly with decreasing window size. The results, shown in Figure 4 confirm this prediction, as the performance gap widens with smaller windows. Figure contains the same points (and additional baseline curves for various overlaps), but in terms of FLOPs rather than window size. All of the results of the recurrent models lie on the Pareto frontier, meaning that improve perplexity or computational cost, one must worsen the other. The non-monotonicity of the overlap 30 and 50 curves is due to the fact that at smaller window sizes, an overlap represents a higher fraction of the computation being used for positions that predictions were already produced for. Also note that while the baseline with overlap 50 curve has the lowest absolute perplexity in the recurrent models trained with overlaps shown in Table perform better.
Figure []]
5.3 WHAT INFORMATION IS BEING PROPAGATED BETWEEN WINDOWS?
We now discuss some features that our models display in greedily decoded continuations from con- texts in the PG-19 validation set, which illustrate types of information that the recurrent module passes (or fails to pass) forward. Samples are included in Tables 5 the appendix.
The most common phenomenon we identify in these samples is successful propagation of topical information between adjacent windows. For instance, we see in Table 4 context discussing ge- ography and rivers, followed by a continuation maintaining the same topic, and we see a context discussing the topic of payment, leading to mention of money in the continuation. We give more rigorous quantitative support of this claim in Section 5.3.1 Beyond passing of topical information, another success in the generations is passing of certain information about characters between windows- Table 5 we see that pronouns in the continuations often reflect characters mentioned in the context, and we see example in which the continuation includes "the two women", after context mentioning "the aunts". This behavior was likely learned due to the fact that PG-19 consists of narratives, correctly passing character information between windows is quite beneficial.
However, these examples also contain discontinuities between the context and the continuation, in terms of local syntax or facts of the narrative. We see that some sentences are not completed in the expected form (for instance, "There are lots of ways of being" is continued with a new quote rather than completion of the thought), and new characters are sometimes invented rather than continuing to reference those described in the context. One sample has a closing quotation mark, predicted from the previous window, being interpreted as an opening quotation mark. These are the types issues that an overlap between adjacent windows easily addresses fact that likely accounts in part for the gap between the recurrent model with disjoint and overlapped execution in Table 2 A higher capacity recurrent module might fix these issues in exchange for additional computation.
5.3.1 QUANTITATIVE EVALUATION OF TOPIC PROPAGATION
To verify the trend we identified of topic propagation in continuations generated by our recurrent models, we fit an LDA topic model (Blei et al. 2003) with 20 topics 5000 books from the PG-19
8

---

Under review as a conference paper at ICLR 2021
Table 3: Results on HotpotQA distractor setting development set, using 30,000 randomly selected training examples. Scores are answer only (no supporting fact prediction).
Method
F1
Exact Match
Disjoint RoBERTa
48.17
43.55
Recurrent RoBERTa
49.12
44.55
training Given a bag of words, this topic model will assign distribution over topics, so can use a statistical distance as a metric for the similarity between the topics of two segments of text.
We sampled 8000 contexts of 300 tokens from the PG-19 validation set, and computed argmax decoded continuations of 30 tokens from the same models used to generate Table 4 We then computed the Jensen-Shannon divergence (ISD) between the topic distribution of each context and the corresponding continuations. This procedure finds that continuations from the recurrent model have an average topic JSD of 0.5331. while those from the baseline model have an average topic JSD of 0.5951. For a given context, the continuation given by the recurrent model is likely to have a lower JSD at least 60% of the time (p < 0.00001).
5.4 QUESTION ANSWERING EXPERIMENTS
To investigate whether our recurrence method would helpful in tasks other than language model- ing, we ran a small experiment on the HotpotQA extractive question answering task, in the distractor setting. this setting, 10 paragraphs of context are given which must be used to answer the given question. HotpotQA's inputs can greatly exceed one 512 token window in length, making it an ideal test of our method. The questions are a mix of span-based and yes/no questions. In order to be able to reduce training time, we use subset of 30000 randomly sampled questions from the training set.
We use the RoBERTa-bees base model for both the bascline and the recurrently augmented model. To evaluate whether recurrence improves encoder performance this task, we directly finetune the models to predict answer span start and end tokens, as done for question answering by Devlin et al. (2019) and max pool the embedding of the [CLS] ] token across windows and use the result for three way classification between "span", "yes", and "no". Because the non-recurrent baseline cannot process entire example once, we begin each input window with the question, separated from the text by [SEP] token. We use this input format for both the baseline and recurrent models.
For both models, we use a learning rate of 2c-5 and train for 4 epochs. For the recurrent model, we mean- pool the final RoBERTa layer for the previous window, and use a 2 layer, 768- -08-dimensional hidden layer MLP to produce an embedding which is inserted at the second layer of the next window (i.e., Lins 2).
Table shows Fl and exact match scores for both models on the HotpotQA dev set. Adding the recurrence module improves both scores by about point, indicating that our method of propagating information between windows can be beneficial for question answering modeling. It should be noted that these values are not directly comparable to scores on the HotpotQA leaderboard, as we only used a subset of the training set, in addition to evaluating on the dev set rather than the private test set Nonetheless, we find these initial experiments to be highly promising, especially given the lack of hyperparameter tuning.
addition to language
6 CONCLUSION AND FUTURE WORK
We showed that augmenting a pretrained language model with recurrence module during fine- tuning can allow increased performance given a fixed computational budget. Our method can be similarly applied to improve the computational efficiency of pretrained models that already exist for many languages and domains, as well as for future models that will be developed. can also allow their application longer contexts than they were trained for, increasing their flexibility.
The baseline receives one token context to begin generating from
Thheee initial results thus represent preliminary experiments that completed prior to the revision deadline; the next version of the paper will have more thorough results including when training on the entire training set.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In EMNIP//JCNLLP. 2019.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv, abs/2004.051555 1500,, 2020.
David Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022 2003.
Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. DeFormer: Decomposing pre-trained transformers for faster question answering. arXiv, abs/2005.00697, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv, hoss/604.061777 2016.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1999940000900 2019.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting the Association for Computational Linguisties, pp. 2978 2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v 111553//11119122/2 URL https:/ / www www.waccll....................c. www.wwlww.bbb................... worooor//aaaaoo LogetthhoooyyPP1
ogy/P1 9-155
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-111 14213.9-122..... URL http:: 7/ // /ww. ac www.wweebb..o wtt.roorggntooogo//ll----------.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1198-1207, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2019.
Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexan- dre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. FlauBERT: Unsupervised language model pre-training for French. In LREC, 2020.
Jinhyuk Lee, Wonjin Yoon. Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jac- Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 19X.:19977.1192.... 2019.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, 'Eric de la Clergerie, Djamé Seddah, and Benoît Sagot. CamemBERT: a tasty French language model. arXiv, abs/1911.03894 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609v 07843, 2016.
10

---

Under review as a conference paper at ICLR 2021
Debora Nozza, Federico Bianchi, and Dirk Hovy. What the [MASK]] making sense of language- specific BERT models. arXiv, abs/2003.02912, 2020.
Marco Polignano, Pierpaolo Basile, Marco Degemmis, Giovanni Semeraro, and Valerio Basile. Al- BERTo: Italian BERT language understanding model for NLP challenging tasks based on tweets. In CLiC-ir, 2019.
Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self-attention for long document understanding. arXiv preprint 19t.:/99100297 2019.
Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. Better language models and their implications. OpenAl Blog https://opeaa. 2019.
con/bleebbetee--lngggg------e
Jack W Rac, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL
htttps://arxiv. www./r//asss...... abs/ 1tt///991..551111
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-BERT: pre-trained contextual- ized embeddings large-scale structured electronic health records for disease prediction. arXiv, abs/2005. 12833, 2020.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.00 108, 2019.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguisties, doi: 10.1865//1/P119-100 URLhttps: /wwp///wwwww../.// http://wwwwwwbb..occocc aclweb. http://www.rg/ttthooogggooo 919-1033
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv, abs/2005.00743,, 2020a.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint XXXx:22009.06332. 2020b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, pp. arXiv-1910, 2019.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv, abtp///01.1043300 2019.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
A APPENDIX
Here we provide some example continuations for contexts from the PG-19 validation set. The sam- ples were generated with greedy argmax decoding, which leads to a large amount of repetition, however we were more concerned with reducing variance and identifying the most likely continua- tion than optimizing for sample quality.
11

---

Under review as a conference paper at ICLR 2021
Table 4: Continuations of PG-19 validation set passages using recurrent model (T' = 300, disjoint) which display approximate continuation of topic. Contexts are shortened for space, 300 tokens of context were used for each.
Context
Continuation
Topic
drains a territory of 300,000 square miles, and, traced back to the rise of its principal source, is 2,000 miles long. At two points, Nee- dles and Yuma the California boundary, it is crossed by a railroad. Elsewhere its course lies far from Caucasian settlements and far from the routes of common travel, in the heart of vast region fenced on the one hand by arid plains or deep forests and on the other by formidable mountains
The first the two great rivers, the Mississippi, was the first to be navi- gated by the Americans, and the first to be navigated by the Indians. The other was the first to be navigated by the Americans, and the first to be navigated by the Indians. The first of the great streams, the Missis- sippi, was navigable by the Indians, and the first by the English...
Rivers
"Mr. Hammerby will have to wait for his money." "Will he?" "I don't 't't what else he can do. But, aside from that, three dollars and half won't keep us month. I'll have
get the money, and he'll be glad to Money see you again." "I'm glad to see you again, Tom. I'm glad to see you again. I'm glad
you again. I'm glad to see you again. I'm glad to see you again
Long's his sister, Thankful, lived, we couldn't say nothing, of course. If she wanted to slave and work so's her brother could live in idleness and sloth, why, that was her business. There ain't any law against body's making a fool of herself, more's 's the pity. But she's been dead a year, and he's done nothing since but live those that'll II trust him, and ask help from the town. He ain't sick-except sick of work. [...] to-mor
rer and his wife, and the whole fam- ily. He was good man, and he was good man. He was a good man, and he was good man. He was a good man, and he was a good man. He was good man, and he was a good man.
Family, Descrip- tion of
man
He
and any form of thought or any institution tending to suppress education destroy intel- ligence strikes at the very essence of the gov- ernment, and constitutes treason which no law can meet, and for which no punishment ade- quate. Education, then, as universally diffused
is the only way to preserve the peace Government of the world. The only way pre-
serve the peace of the world is to pre-
serve the liberty of the people. The
first step to taken is to abolish the power of the government. The
as the elements of God, is the life-blood of our
first step to be taken is to abolish the
body politic. The intelligence of the people is
power of the people. The first step
the one great fact of our civilization and our prosperity.it
12

---

Under review as a conference paper at ICLR 2021
Table 5: Continuations of PG-19 validation set passages using recurrent model (I' 300, disjoint) which display continuation of correct pronouns or references to characters from context. Contexts are left-truncated, 300 tokens of context were given for each continuation.
Context
Continuation
Elaine, however, did not gratify their hopes; she re- ferred to the ball with the detachment she would have shown in describing a drawing-room show of cottage in- dustries. It was not difficult to discern in her descrip- tion of the affair the confession that she had been slightly bored. From Courtenay, later in the day, the aunts re- ceived a much livelier impression of
the house, and the two women were standing in the doorway. "I am afraid I have not been very well," she said, as she looked at the two men, "I have been very much worried about you, and I have been very much worried about you. have been very much wor- ried about you, and I have been very much worried about you. have been very much pleased with you, and have been very much
"Oh, he's not at all bookworm," said Suzette quickly, "though he's tremendously well-read. He's quite the man of action." "Does he hunt?" asked Elaine. "No, he doesn't get much time or opportunity for rid- ing." "What pity," commented Elaine; "I don't think I could marry a man who wasn
" "I don't know," said the Sergeant, "but have heard that you are very good friend." "I am not," said the girl, "I am not."' "I am not," said the Sergeant, "I am not." "I am not," said the girl, "I am not.
Gregg was still the soft-stepping scoundrel who in- vited the toe of honesty, and Mornway, as he entered, was conscious of sharp revulsion of feeling, But it was im- possible evade the interview, and he sat silent while the man stated his case. Mrs. Mornway had represented the stenographer as being in desperate straits, and ready to accept any job that could be found, but though his ap- pearance might have seemed to corroborate her account, he evidently took a less hopeless view of his case, and the Governor found with surprise that he had fixed his eye on elerkship
that he had been in the habit of doing, and that he had been in the habit of doing. "I'm not going to be a fool," he said, "and I'm not going be fool....
Many transactions too, from a desire to avoid worry- ing me, were carried through without my knowledge, al- though formerly, as a matter of course, they would have been submitted to me. Strangers, when they called, asked to see Johnson or Marsh. directed the messenger that they were to be shown into my room if was disengaged. This was a failure, for, when they came, was obliged to ask for help, which Sometimes sent for the papers, but it took long time to read them, and my visitors became impatient. Dur- ing one of these interviews, remember that was sorely perplexed, but had managed to
get the money, and I was glad to get it. The next day I went to the Court of the Emperor, and found him standing in the courtyard of the Palace, with large number of soldiers. He was dressed in a very dirty uniform, and wore very dirty hat. He was dressed in a very dirty uniform, and wore very dirty hat. He was dressed in a very dirty hat, and wore very dirty hat
not given very generously.
know which crossings to look out-I know what I'm going to see in the shop-windows. It saves a lot of wear and tear to know what's coming. For good many years I never did know, from one minute to another, and now I like to think that everything's 's cut-and-dried, and nothing unexpected can jump out at me like tramp from ditch." " He paused calmly to knock the ashes from his cigar, and Garnett said with smile: "Doesn't such plan of life cut off nearly all the possibilities?" The old gen- tleman made contemptuous motion. "Possibilities of what? Of being multifariously miserable? There are lots of ways of being
"I'm afraid m not going to be able to do that," he said. "I'm m going to have to go to the station. I'm going to have to go to the station. I want to see the sta- tion. want to see the station. want see the station. I want to see the sta- tion. want to see the station. want the station. I want to see the sta- tion. want to see
13


Reviews:
[{"review_id": "qQNEkzJ2OvR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this work is to enable existing pre-trained transformers (e.g. GPT-2) to operate over long input contexts. This is achieved by breaking the input sequence into segments and processing each segment through the transformers while allowing tokens in the current segment to attend over a summary vector of the tokens in the previous segment. The summary vector is created as a weighted combination of the tokens in the summarized segment. Thus the summary vector introduces recurrence where each segment can use information from the previous segment. These modifications yield a better language model for long input texts. \n\nThe main benefits of this approach are as follows: (1) The modifications yield a better language model for long input texts, especially when compared to a tiling based approach (2) Potential for reducing memory footprint in these models by shrinking the amount of text that is to be processed in one-go. \n\nMy main concern with the paper is that unfortunately the evaluation is only limited to perplexity numbers on a couple of datasets. While this is a useful metric for evaluation, this alone is inadequate to demonstrate the quality of the model as a text generating system or as a language model that will be fine-tuned for target tasks or to understand how much impact the model will have in these applications.\n\n-- For the model to be considered as a text generation system, there needs to be some human evaluation of the generated outputs. There are a small number of examples in the paper but that is not enough for a quantitative assessment. To clearly establish the benefits of the proposed modification it would be even better to consider generation tasks where conditioning on long inputs is essential. \n\n-- How will model fare when used in a target task defined over long input contexts? The related work section includes some papers that evaluate on such tasks. For example on target task could be HotpotQA, which requires QA over ten paragraphs which easily exceed the 512 token limits. It is important to know how the proposed recurrence model compares to tiling GPT (disjoint version) or other simpler approaches on these long input tasks.\n\n-- Another key strength of the model is that it potentially allows for processing the input in smaller segments. While perplexity gains are helpful, here again there is a missed opportunity in terms of human evaluation of the generated outputs over shorter segments, and the impact of these choices in different applications. \n\n\nTo reiterate, this paper presents a very nice idea to a well-motivated problem. The executed experiments show that this idea is likely to work but the gaps in experimentation leave much room for speculation about the potential impact of this approach in end applications. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nd8-ViHCEzW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes recurrent connections between two adjacent Transformers, which transfers the previous context to the next step. This is a practically useful technique, improving the performance (perplexity in the experiments), and worth publishing. However, I have some comments and questions about the article.\n\nSection 5.3 is an interesting question. The authors argue that topical information or so between adjacent windows is propagated. Although it is a plausible argument, it seems like it is hardly supported by table 3.\n\nThe authors said more complex recurrence modules do not make any significant difference. Then, the authors need to explain why the variations do not matter. For example, the authors fixed l_ins to be 2, without an explanation.  \n\nIt is interesting to see the relationship between the overlap length and improvement using the recurrent connection. It would be better to have further discussion about the relation and different roles. \n\nThe Transformer model is also fine-tuned with the recurrent connection. So, I was wondering if the fine-tuning improves the Transformer model too. It would be interesting to compare the updated Transformer to the previous one. \n\nIn Eq. (1), is there 1/T? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9bPwIXKKFYE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposed to add a recurrent component to pretrained transformers. The component pools the hidden states of a context window and passes it to the next context window as an additional input to the self-attention layer. The component reduces the memory usage at both training and inference time, and enables the Transformer model to work on a longer sequence. The component is evaluated on two language modeling datasets and outperforms baseline models.\n\nReasons for score: \n\nI vote for accepting the paper. The paper proposed a nice and simple way to make use of the existing pretrained Transformers with reduced memory usage and extended sequence length. This should benefit practitioners who want to apply these language models on a more diverse set of downstream tasks where the sequence length doesn’t fit the one from the original pretrained model. The results presented in the paper are significant. The paper is well-written and easy to follow.\n\nComments:\n\n1. It would be better to include the failure results stated in the end of section 3.1. I’m surprised that a key-value pair can boost the performance that much. \n2. The paper should add more content on differentiating with Transformer-XL. I believe the difference is more than relative embeddings. For example, each Transformer-XL layer attends to an earlier layer of previous timestep, this convolutional operation making the structure no longer “recurrent”.\n\nTypos: \n- Third line in section 3.1: “at position t” -> “at position i”\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose to add recurrence to pre-trained language models such as GPT-2 or BERT. The idea is similar to the compressive transformer paper: a small module is added to the network, and used to compress the representations from the previous chunk of data from the sequence to a single vector. Then, this vector is added to the keys and values of the self-attention module when processing the next chunk. The main contribution of the paper is to show that this technique can be added to pre-trained models at fine-tuning time.  The main concerns regarding the paper are technical novelty and limited empirical results. The idea of adding recurrence to transformers was previously explored in compressive transformer, and many previous work have considered adding modules with small number of parameters at fine-tuning time. Moreover, I do not believe that the empirical section is strong enough to justify the acceptance of the paper, as the method is only evaluated on two language modeling tasks (and one early experiment on HotpotQA). The baselines are weak, and thus, the results are not convincing. For these reasons, I weakly recommend to reject the paper, and encourage the authors to make the empirical section stronger.
Decision: Reject