Title: TRANSFORMER-XL: LANGUAGE MODELING WITH LONGER-TERM DEPENDENCY

Abstract: We propose a novel neural architecture, Transformer-XL, for modeling longerterm dependency. To address the limitation of fixed-length contexts, we introduce a notion of recurrence by reusing the representations from the history. Empirically, we show state-of-the-art (SoTA) results on both word-level and character-level language modeling datasets, including WikiText-103, One Billion Word, Penn Treebank, and enwiki8. Notably, we improve the SoTA results from 1.06 to 0.99 in bpc on enwiki8, from 33.0 to 18.9 in perplexity on WikiText-103, and from 28.0 to 23.5 in perplexity on One Billion Word. Performance improves when the attention length increases during evaluation, and our best model attends to up to 1,600 words and 3,800 characters. To quantify the effective length of dependency, we devise a new metric and show that on WikiText-103 Transformer-XL manages to model dependency that is about 80% longer than recurrent networks and 450% longer than Transformer. Moreover, Transformer-XL is up to 1,800+ times faster than vanilla Transformer during evaluation.

1. INTRODUCTION:

Language modeling is among the important problems that require modeling long-term dependency. However, it has been a challenge to equip neural networks with the capability to model long-term dependency in sequential data. Recurrent neural networks (RNNs), in particular Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) , have been a standard solution for language modeling and obtained strong results on multiple benchmarks. However, RNNs are difficult to optimize due to gradient vanishing and explosion (Hochreiter et al., 2001) , and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013; Pascanu et al., 2012) might not be sufficient to fully address this issue. It was shown that LSTM language models use 200 context words on average (Khandelwal et al., 2018) , indicating room for further improvement.

On the other hand, the direct connections between long-distance word pairs baked in attention mechanisms might ease optimization and enable the learning of long-term dependency (Bahdanau et al., 2014; Vaswani et al., 2017) . Recently, Al-Rfou et al. (2018) designed a set of auxiliary losses to train deep Transformer networks for character-level language modeling, which outperform LSTMs by a large margin. However, the context length is fixed to hundreds of characters and thus it is not possible to model longer-term dependency. Moreover, it is not clear how the model performs on word-level language modeling data, as the granularity changes.

To address the limitation of fixed-length contexts, we propose a new architecture called Transformer-XL (meaning extra long). We introduce the notion of recurrence into our deep self-attention network. In particular, instead of computing the hidden states from scratch for each new segment, we reuse the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. Importantly, to reuse the hidden states, it is necessary to use relative positional encodings rather than absolute ones. As an additional technical contribution, we introduce a simple and effective relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.

Transformer-XL obtained strong results on four datasets, varying from word-level to character-level language modeling. Transformer-XL improves the previous state-of-the-art (SoTA) results from 33.0 to 18.9 in perplexity on WikiText-103, from 1.06 to 0.99 in bpc on enwiki8, and from 28.0 to 23.5 in perplexity on One Billion Word. On small data, Transformer-XL also achieves a perplexity of 54.5 on Penn Treebank, which is SoTA when comparable settings are considered.

We use two methods to quantitatively study the effective lengths of Transformer-XL and the baselines. Similar to Khandelwal et al. (2018) , we gradually increase the attention length at test time until no further noticeable improvement (∼0.1% relative gains) can be observed. Our best model in this settings use attention lengths of 1,600 and 3,800 on WikiText-103 and enwiki8 respectively. In addition, since the effective context length of Transformer-XL can be longer than the attention length due to our recurrent formulation, we devise a metric called Relative Effective Context Length (RECL) that aims to perform a fair comparison of the gains brought by increasing the context lengths for different models. In this setting, Transformer-XL learns a RECL of 900 words on WikiText-103, while the numbers for recurrent networks and Transformer are only 500 and 128.

2. RELATED WORK:

In the last few years, the field of language modeling has witnessed many significant advances, including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et al., 2016; Chung et al., 2016; Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018) , improving regularization and optimization algorithms Zaremba et al. (2014) ; Inan et al. (2016) ; Press & Wolf (2016) ; Merity et al. (2017) ; Gal & Ghahramani (2016) , speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016) , and enriching the output distribution family (Yang et al., 2017; Kanai et al., 2018) .

To capture the long-range context in language modeling, a line of work directly feeds a representation of the wider context into the network as an additional input. Existing works range from ones where context representations are manually defined (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017) .

More broadly, in generic sequence modeling, how to capture long-term dependency has been a long-standing research problem. From this perspective, since the ubiquitous adaption of LSTM, many efforts have been spent on relieving the vanishing gradient problem, including better initialization (Le et al., 2015) , additional loss signal (Trinh et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014) ; Koutnik et al. (2014) ; Wu et al. (2016) ; Li et al. (2018) . Different from them, our work is based on the Transformer architecture and shows that language modeling as a real-world task benefits from the ability to learn longer-term dependency.

3. MODEL:

Given a corpus of tokens x = (x 1 , . . . , x T ), the task of language modeling is to estimate the joint probability P (x), which is often auto-regressively factorized as P (x) = t P (x t | x <t ). With the factorization, the problem reduces to estimating each conditional factor. In this work, we stick to the standard neural approach to modeling the conditional probability. Specifically, a trainable neural network is used to encode the context x <t into a fixed size hidden state, which is multiplied with the word embeddings to obtain the logits. The logits are then fed into the Softmax function, yielding a categorical probability distribution over the next token.

3.1. VANILLA TRANSFORMER LANGUAGE MODELS:

In order to apply Transformer or self-attention to language modeling, the central problem is how to train a Transformer to effectively encode an arbitrarily long context into a fixed size representation. Given infinite memory and computation, a simple solution would be to process the entire context sequence using an unconditional Transformer decoder, similar to a feed-forward neural network. However, this is usually infeasible with the limited resource in practice. One feasible but crude approximation is to split the entire corpus into shorter segments of manageable sizes, and only train the model within each segment, ignoring all contextual information from previous segments. This is the idea adopted by Al-Rfou et al. (2018) . We call it the vanilla model and visualize it in Fig. 1a . Under this training paradigm, information never flows across segments in either the forward or backward pass. Hence, the largest possible dependency length is upper bounded by the segment length, which is a few hundred on character-level language modeling (Al-Rfou et al., 2018) . Therefore, although the self-attention mechanism is less affected by the vanishing gradient problem compared to RNNs, the vanilla model is not able to fully exploit this optimization advantage.

During evaluation, at each step, the vanilla model also consumes a segment of the same length as in training, but only makes one prediction at the last position. Then, at the next step, the segment is shifted to the right by only one position, and the new segment has to be processed all from scratch. As shown in Fig. 1b , this procedure ensures that each prediction utilizes the longest possible context exposed during training. To address the limitation of using fixed-length contexts, we propose to introduce a recurrence mechanism to the Transformer architecture. During training, the hidden state sequence computed for the previous segment is fixed and cached to be reused as an extended context when the model processes the next new segment, as shown in Fig. 2a . Although the gradient still remains within a segment, this additional input allows the network to exploit information in the history, leading to an ability of modeling longer-term dependency. Formally, let the two consecutive segments of length L be s

3.2. SEGMENT-LEVEL RECURRENCE WITH STATE REUSE:

τ = [x τ,1 , • • • , x τ,L ] and s τ +1 = [x τ +1,1 , • • • , x τ +1,L

] respectively. Denoting the n-th layer hidden state sequence produced for the τ -th segment s τ by h n τ ∈ R L×d , where d is the hidden dimension. Then, the n-th layer hidden state for segment s τ +1 is produced as follows,

h n-1 τ +1 = [SG(h n-1 τ ) • h n-1 τ +1 ], (extended context) q n τ +1 , k n τ +1 , v n τ +1 = h n-1 τ +1 W q , h n-1 τ +1 W k , h n-1 τ +1 W v ,

(query, key, value vectors)

h n τ +1 = Transformer-Layer q n τ +1 , k n τ +1 , v n τ +1 . (self-attention + feed-forward)

where the function SG(•) stands for stop-gradient, the notation [h u • h v ] indicates the concatenation of two hidden sequences along the length dimension, and W • denotes model parameters. Compared to the standard Transformer, the critical difference lies in that the key k n τ +1 and value v n τ +1 are conditioned on the extended context h n-1 τ +1 and hence h n-1 τ cached from the previous segment. We emphasize this particular design by the green paths in Fig. 2a .

With this recurrence mechanism applied to every two consecutive segments of a corpus, it essentially creates a segment-level recurrence in the hidden states. As a result, the effective context being utilized can go way beyond just two segments. However, notice that the recurrent dependency between h n τ +1 and h n-1 τ shifts one layer downwards per-segment, which differs from the samelayer recurrence in conventional RNN-LM. Consequently, the largest possible dependency length grows linearly w.r.t. the number of layers as well as the segment length, as visualized by the shaded area in Fig. 2b . This is analogous to truncated BPTT (Mikolov et al., 2010) , a technique developed for training RNN-LM. However, different from truncated BPTT, our method caches a sequence of hidden states instead of the last one, and should be applied together with the relative positional encoding technique described in Section 3.3.

Besides the extra long context, another benefit that comes with the recurrence scheme is significantly faster evaluation. Specifically, during evaluation, the representations from the previous segments can be reused instead of being computed from scratch as in the case of the vanilla model. In our experiments on enwiki8, Transformer-XL is up to 1,800+ times faster than the vanilla model during evaluation (see Appendix 4.4 for more comparisons).

Finally, notice that the recurrence scheme does not need to be restricted to only the previous segment. In theory, we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment. In practice, we cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory m n τ ∈ R M ×d , due to a clear connection to the memory augmented neural networks (Graves et al., 2014; Weston et al., 2014) . In our experiments, we set M equal to the segment length during training, and increase it by multiple times during evaluation.

3.3. RELATIVE POSITIONAL ENCODINGS:

While we found the idea presented in the previous subsection very appealing, there is a crucial technical challenge we haven't solved in order to reuse the hidden states. That is, how can we keep the positional information coherent when we reuse the states? Recall that, in the standard Transformer, the information of sequence order is provided by a set of positional encodings, denoted as U ∈ R Lmax×d , where the i-th row U i corresponds to the i-th absolute position within a segment and L max prescribes the maximum possible length to be modeled. Then, the actual input to the Transformer is the element-wise addition of the word embeddings and the positional encodings. If we simply adapt this positional encoding to our recurrence mechanism introduced above, the hidden state sequence would be computed schematically by

h τ +1 = f (h τ , E sτ+1 + U 1:L ) and h τ = f (h τ -1 , E sτ + U 1:L ),

where E sτ ∈ R L×d is the word embedding sequence of s τ , and f represents a transformation function. Notice that, both E sτ and E sτ+1 are associated with the same positional encoding U 1:L . As a result, the model has no information to distinguish the positional difference between x τ,j and x τ +1,j for any j = 1, . . . , L, resulting in a sheer performance loss.

In order to avoid this failure mode, the fundamental idea is to only encode the relative positional information in the hidden states. Intuitively, when a query vector q τ,i attends on the key vectors k τ,≤i from the i-th position, it does not need to know the absolute position of each key vector to identify the order of the segment. Instead, it suffices to know the relative distance between each key vector k τ,j and itself q τ,i , i.e. i -j. 1 Practically, one can create a set of relative positional encodings R ∈ R Lmax×d , where the i-th row R i indicates a relative distance of i between two positions. By injecting this distance information into the attention, the sequence order can be completely captured without any loss.

Previously, the idea of relative positional encodings has been explored in the context of machine translation (Shaw et al., 2018) and music generation (Huang et al., 2018) . Here, we offer a different derivation, arriving at a new form of relative positional encodings, which not only has a one-to-one correspondence to its absolute counterpart but also enjoys much better generalization empirically (see Section 4). Firstly, in the standard Transformer (Vaswani et al., 2017) , the attention score between query q i and key vector k j within the same segment can be decomposed as

A abs i,j = q i k j = E xi W q W k E xj (a) + E xi W q W k U j (b) + U i W q W k E xj (c) + U i W q W k U j (d)

.

With the goal of removing all information about absolute positions, we propose to re-parameterize the four terms as follows

EQUATION

.

• The first change we make is to replace all appearances of the absolute positional embedding U j for computing key vectors in term (b) and (d) with its relative counterpart R i-j . This essentially reflects the prior that only the relative distance matters for where to attend. Note that R is a sinusoid encoding matrix (Vaswani et al., 2017) without learnable parameters. • Secondly, we introduce a trainable parameter u ∈ R d to replace the query U i W q in term (c).

In this case, since the query vector is the same for all query positions, it suggests that the attentive bias towards different words should remain the same regardless of the query position. With a similar reasoning, a trainable parameter v ∈ R d is added to substitute U i W q in term (d). (Vaswani et al., 2017) . In contrast, our relative positional embedding R adapts the sinusoid formulation. As a benefit of the inductive bias, a model trained on a memory of some certain length can automatically generalize to a memory several times longer during evaluation.

Equipping the recurrence mechanism with our proposed relative positional embedding, we finally arrive at the Transformer-XL architecture. For completeness, we summarize the computational procedure for a N -layer Transformer-XL with a single attention head below:

For n = 1, . . . , N :

h n-1 τ = SG(m n-1 τ ) • h n-1 τ q n τ , k n τ , v n τ = h n-1 τ W n q , h n-1 τ W n k,E , h n-1 τ W n v A n τ,i,j = q n τ,i k n τ,j + q n τ,i W n k,R R i-j + u k τ,j + v W n k,R R i-j a n τ = Masked-Softmax(A n τ )v n τ o n τ = LayerNorm(a n τ + h n-1 τ ) h n τ = Positionwise-Feed-Forward(o n τ

) with h 0 τ := E sτ defined as the word embedding sequence. In addition, it is worth mentioning that a naive way to compute A requires computing W n k,R R i-j for all pairs (i, j), whose cost is quadratic w.r.t. the sequence length. However, noticing that the value of i -j only ranges from zero to the sequence length, we show a simple computation procedure in Appendix B, which reduces the cost to be linear w.r.t. the sequence length.

4. EXPERIMENTS:



4.1. MAIN RESULTS:

We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et 2016), enwiki8 (LLC, 2009), One Billion Word (Chelba et al., 2013) , and Penn Treebank (Mikolov & Zweig, 2012) .

WikiText-103 is the largest available word-level language modeling benchmark with long-term dependency. It contains 103M training tokens from 28K articles, with an average length of 3.6K tokens per article, which allows testing the ability of long-term dependency modeling. We set the attention length to 384 during training and 1600 during evaluation. As shown in Table 1 , Transformer-XL reduces the previous SoTA perplexity from 33.0 to 18.9, which demonstrates that our new architecture is substantially better than previous models on word-level datasets that require modeling long-term dependency.

The dataset enwiki8 contains 100M bytes of unprocessed Wikipedia text. We compare our architecture with the previous SoTA results in 

4.2. ABLATION STUDY:

We conduct an ablation study on WikiText-103 to examine the effects of two proposed techniques used in Transformer-XL: the recurrence mechanism and the new positional encoding scheme. The results are reported in Table 5 . Among the compared encoding schemes, Shaw et al. ( 2018) is relative, while Vaswani et al. (2017) and Al-Rfou et al. ( 2018) are absolute. "Full" and "half" losses refer to applying a cross entropy loss to all or the recent half positions in the segment. We found that absolute encodings only work well with half losses because half losses exclude positions with very short attention lengths during training for better generalization. Table 5 shows that both the recurrence mechanism and our encoding scheme are necessary to achieve the best performance, as well as generalizing to longer attention sequences during evaluation time. Although the backpropagation length during training is only 128, with the two techniques the attention length can be increased to 640 at test time. In the standard setting with 151M parameters, the perplexity decreases as the attention length increases. This is consistent with our motivation and discussion in Section 3.

Since the recurrence mechanism costs additional memory usage, we also compare Transformer-XL with baselines under the same GPU memory constraints. As shown in 2018), except that two auxiliary losses are not implemented in our experiments. "PPL init" refers to using the same length as training. "PPL best" indicates the perplexity obtained by using the optimal length. "Attn Len" is the shortest possible attention length during evaluation to achieve the corresponding result (PPL best). Increasing the attention length during evaluation improves performance only when our positional encoding is used. The "Transformer-XL (151M)" setting uses a standard parameter budget as previous work Merity et al. (2018) , where we observe a similar effect when increasing the attention length during evaluation.

Model r = 0.1 r = 0.5 r = 1.0 

B EFFICIENT COMPUTATION OF THE ATTENTION WITH RELATIVE POSITIONAL EMBEDDING:

As we discussed in section 3.3, the naive way of computing the W k,R R i-j for all pairs (i, j) is subject to a quadratic cost. Here, we present a simple method with only a linear cost. Firstly, notice that the relative distance i -j can only be integer from 0 to M + L -1, where M and L are the memory length and segment length respectively. Hence, the rows of the matrix

Q :=        R M +L-1 R M +L-2 . . . R 1 R 0        W k,R =         [W k,R R M +L-1 ] [W k,R R M +L-2 ] . . . [W k,R R 1 ] [W k,R R 0 ]         ∈ R (M +L)×d

consist of all possible vector outputs of W k,R R i-j for any (i, j). Note that we have defined Q in a reversed order, i.e., Q k = W k,R R M +L-1-k , to make further discussion easier.

Next, we collect the term (b) for all possible i, j into the following L × (M + L) matrix,

B =      q 0 W k,R R M • • • q 0 W k,R R 0 0 • • • 0 q 1 W k,R R M +1 • • • q 1 W k,R R 1 q 1 W k,R R 0 • • • 0 . . . . . . . . . . . . . . . . . . q L-1 W k,R R M +L-1 • • • q L-1 W k,R R M +L-1 q L-1 W k,R R L-1 • • • q L-1 W k,R R 0      =      q 0 Q L-1 • • • q 0 Q M +L-1 0 • • • 0 q 1 Q L-2 • • • q 1 Q M +L-2 q 1 Q M +L-1 • • • 0 . . . . . . . . . . . . . . . . . . q L-1 Q 0 • • • q L-1 Q M q L-1 Q M +1 • • • q L-1 Q M +L-1     

Then, we further define

B = qQ =      q 0 Q 0 • • • q 0 Q M q 0 Q M +1 • • • q 0 Q M +L-1 q 1 Q 0 • • • q 1 Q M q 1 Q M +1 • • • q 1 Q M +L-1 . . . . . . . . . . . . . . . . . . q L-1 Q 0 • • • q L-1 Q M q L-1 Q M +1 • • • q L-1 Q M +L-1     

. Now, it is easy to see an immediate relationship between B and B, where the i-th row of B is simply a left-shifted version of i-th row of B. Hence, the computation of B only requires a matrix multiplication qQ to compute B and then a set of left-shifts.

Similarly, we can collect all term (d) for all possible i, j into another L × (M + L) matrix D,

D =      v Q L-1 • • • v Q M +L-1 0 • • • 0 v Q L-2 • • • v Q M +L-2 v Q M +L-1 • • • 0 . . . . . . . . . . . . . . . . . . v Q 0 • • • v Q M v Q M +1 • • • v Q M +L-1     

.

Then, we can follow the same procedure to define

d = [Qv] = v Q 0 • • • v Q M v Q M +1 • • • v Q M +L-1 .

Again, each row of D is simply a left-shift version of d. Hence, the main computation cost comes from the matrix-vector multiplication d = [Qv] , which is not expensive any more.

C DETAILS ABOUT RECL 

i (c, t) = -log P mi (x t |x t-1 , • • • , x t-c )

where P mi is the probability distribution given by model m i , and x t is the t-th token in the corpus. The relative loss of m i w.r.t. the model group M is written as

f i (c, c ) = 1 |T | t∈T min (b(c, t), l i (c , t))

The above equation uses the minimum loss of all models on the short length c as a baseline, and only losses smaller than the baseline will be effectively counted towards the relative loss. This enables fair comparison between multiple models because all models with a long context length c need to improve over the same baseline. Sometimes we only care about those positions where the baseline performs poorly (which means short-term dependency with context length c is not sufficient), so given a ratio parameter r, we define the set T is the above equation as T = top-r positions t with largest b(c, t)

The relative gain is subsequently defined as the relative perplexity reduction:

g i (c, c ) = exp f i (c, c) -exp f i (c, c ) exp f i (c, c)

Given a step size ∆, we then use an algorithm to find the RECL by thresholding the relative gain: In Figure 3 , we visualize the unnormalized relative perplexity gains (exp

f i (c, c) -exp f i (c, c ))

with various pairs of (c, c ) when r = 0.1. It is clear that Transformer-XL has a longer RECL compared to RNNs and other baselines because the relative gains are substantially larger.

For reference, we plot the perplexities with varying context lengths in Figure 4 . The y-axis denotes the "normal" perplexity (not calibrated by baselines).

D ATTENTION VISUALIZATION:

In this section, we provide some visualization of the attention learned by the SoTA model on the WikiText-103 validation set. Recall that, this model has 16 10-head transformer layers and relies on a memory of length 640. The first visualization aims at revealing the overall trend of where the model is attending. Specifically, for each attention head of each layer, we average the attention distributions of all tokens in the validation set. This is shown in Fig. 5 . As we can see, the overall trend is to focus more on the nearby tokens than the faraway ones. However, it is also very clear that some attention heads have a wider attention distribution over the entire memory span, notably the head 8 from layer 1, head 78 from layer 8, and the head 158 from layer 16.

Since we are focused on learning long-range dependency, we are especially interested in these heads with a wider attention span. Thus, in the second set of visualization, we pick the three notable heads mentioned above, and visualize their attention behavior for a randomly chosen position, as shown in Fig. 6 . Here, we see three different patterns of wider attention:

• For the head 8 in the 1st layer, we see an almost uniform attention over the entire memory span. This is quite intuitive, as lower-level layers needs to screen the entire memory span to decide where to focus for higher-level layers • For the head 78 in the 8th layer (a middle-level layer), we see a very sparse attention pattern scattered in all ranges of the memory. Again, this well fits our intuition that as information accumulates, the network may focus on some particular position with special interests. • For the head 158 in the 16th layer (i.e. the last layer), each target location (corresponding to each row) has its own distinct sparse focus, differing from head 78 where target locations largely share the same attentive location in memory. Meanwhile, the pattern is also different from the case of head 8, where a few locations are clearly attended more than others.

Finally, as we have discussed in section 3.3, the attention score can be decomposed into four intuitive terms. Here, we want to further investigate how these four terms contribute to the overall attention trend in Fig. 5 . Since the term (c) represents the global content bias, i.e., the prior importance of each word regardless of the context, we will leave it out and focus on the terms (a), (b) and (d). So, for each term, we take the Softmax w.r.t. the memory span and average the resulted distribution of all tokens in the validation set. The results are visualized in Fig. 7 :

• Since term (a) is fully content-based addressing, when averaging over all target words, the result is essentially uniform over the entire context, except for a few very close words, which are likely to be semantically similar to the target word. • The overall trend of term (b) highly resembles that of the entire attention distribution in Fig. 5 .

It suggests that the global trend of focusing on the nearby context is largely contributed by this content-dependent positional bias. • The overall trend of term (d) is also focusing more on nearby words. However, compared to the trend of term (b), it is clearly flatter and biases towards a longer context.

:

In this work, as we focus on language modeling, the attention is also causal, i.e., from right to left. Hence, the relative distance, defined as the right index minus the left index, is always non-negative. But it is straightforward to extend this idea to the non-causal case.

Figures and Tables:

Figure 1: Illustration of the vanilla model with a segment length 4.

Figure 2: Illustration of the Transformer-XL model with a segment length 4.

• Finally, we deliberately separate the two weight matrices W k,E and W k,R for producing the content-based key vectors and location-based key vectors respectively. Under the new parameterization, each term has an intuitive meaning: term (a) represents contentbased addressing, term (b) captures a content-dependent positional bias, term (c) governs a global content bias, and (d) encodes a global positional bias. In comparison, the formulation in Shaw et al. (2018) only has terms (a) and (b), dropping the two bias terms (c) and (d). Moreover, Shaw et al. (2018) merge the multiplication W k R into a single trainable matrix R, which abandons the inductive bias built into the original sinusoid positional encoding

Figure 3: Visualizing unnormalized relative perplexity gains with r = 0.1.

Figure 4: Perplexity vs context length.

Given a short context length c and a long context length c such that c ≥ c, we can further define a baseline for each position t, b(c, t) = N min i=1 l i (c, t)

Set initial short context length c, and long context length c = c + ∆ 2. Compute g i (c, c ). If g i (c, c ) < 0.01, return RECL = c. If g i (c, c ) ≥ 0.01, set c = c , c = c + ∆ and go to step 1.

Figure 5: Average attention over the previous 640 tokens, where each row corresponds to a attention head and each column corresponds to a relative location. There are totally 160 attention heads, and every 10 heads come from a single layer. Darker colors indicate higher values.

Figure 6: Visualization of the three heads with a wide attention range. Each row corresponds to a target location/token and each column corresponds to a context location/token. Tokens in the memory that have top 20% attention values are highlighted in red.

al., Comparison with state-of-the-art results on WikiText-103.

Table content: <table><tr><td>Model</td><td colspan="3">#Params Validation PPL Test PPL</td></tr><tr><td>Grave et al. (2016b) -LSTM</td><td>-</td><td>-</td><td>48.7</td></tr><tr><td>Bai et al. (2018) -TCN</td><td>-</td><td>-</td><td>45.2</td></tr><tr><td>Dauphin et al. (2016) -GCNN-8</td><td>-</td><td>-</td><td>44.9</td></tr><tr><td>Grave et al. (2016b) -LSTM + neural cache</td><td>-</td><td>-</td><td>40.8</td></tr><tr><td>Dauphin et al. (2016) -GCNN-14</td><td>-</td><td>-</td><td>37.2</td></tr><tr><td>Merity et al. (2018) -4-layer QRNN</td><td>151M</td><td>32.0</td><td>33.0</td></tr><tr><td>Ours -Transformer-XL Standard</td><td>151M</td><td>23.1</td><td>24.0</td></tr><tr><td>Ours -Transformer-XL Large</td><td>257M</td><td>18.2</td><td>18.9</td></tr><tr><td>Model</td><td/><td colspan="2">#Params Test bpc</td></tr><tr><td>Ha et al. (2016) -LN HyperNetworks</td><td/><td>27M</td><td>1.34</td></tr><tr><td>Chung et al. (2016) -LN HM-LSTM</td><td/><td>35M</td><td>1.32</td></tr><tr><td colspan="2">Zilly et al. (2016) -Recurrent highway networks</td><td>46M</td><td>1.27</td></tr><tr><td colspan="2">Mujika et al. (2017) -Large FS-LSTM-4</td><td>47M</td><td>1.25</td></tr><tr><td>Krause et al. (2016) -Large mLSTM</td><td/><td>46M</td><td>1.24</td></tr><tr><td>Knol (2017) -cmix v13</td><td/><td>-</td><td>1.23</td></tr><tr><td colspan="2">Al-Rfou et al. (2018) -12-layer Transformer</td><td>44M</td><td>1.11</td></tr><tr><td>Ours -12-layer Transformer-XL</td><td/><td>41M</td><td>1.06</td></tr><tr><td colspan="2">Al-Rfou et al. (2018) -64-layer Transformer</td><td>235M</td><td>1.06</td></tr><tr><td>Ours -18-layer Transformer-XL</td><td/><td>88M</td><td>1.03</td></tr><tr><td>Ours -24-layer Transformer-XL</td><td/><td>277M</td><td>0.99</td></tr></table>

Comparison with state-of-the-art results on enwiki8.

Table content: <table/>



Table content: <table><tr><td>. The 12-layer Transformer-XL achieves a new</td></tr></table>

Comparison with state-of-the-art results on Penn Treebank word-level language modeling. † indicates using two-step finetuning.

Table content: <table><tr><td>Model</td><td colspan="2">#Params PPL</td></tr><tr><td>Shazeer et al. (2014) -Sparse Non-Negative</td><td>33B</td><td>52.9</td></tr><tr><td>Chelba et al. (2013) -RNN-1024 + 9 Gram</td><td>20B</td><td>51.3</td></tr><tr><td>Jozefowicz et al. (2016) -LSTM-2048-512</td><td>0.83B</td><td>43.7</td></tr><tr><td>Kuchaiev &amp; Ginsburg (2017) -BIG G-LSTM-2</td><td>-</td><td>36.0</td></tr><tr><td>Dauphin et al. (2016) -GCNN-14 bottleneck</td><td>-</td><td>31.9</td></tr><tr><td>Jozefowicz et al. (2016) -LSTM-8192-1024</td><td>1.8B</td><td>30.6</td></tr><tr><td>Jozefowicz et al. (2016) -LSTM-8192-1024</td><td>1.04B</td><td>30.0</td></tr><tr><td>Shazeer et al. (2017) -Low-Budget MoE</td><td>∼5B</td><td>34.1</td></tr><tr><td>Shazeer et al. (2017) -High-Budget MoE</td><td>∼5B</td><td>28.0</td></tr><tr><td>Ours -Transformer-XL</td><td>0.46B</td><td>23.5</td></tr></table>

Comparison with state-of-the-art results on One Billion Word.

Table content: <table/>



Table content: <table><tr><td>Remark</td><td>Recurrence</td><td>Encoding</td><td colspan="4">Loss PPL init PPL best Attn Len</td></tr><tr><td>Transformer-XL (128M)</td><td/><td>Ours</td><td>Full</td><td>27.02</td><td>26.77</td><td>500</td></tr><tr><td>-</td><td/><td>Shaw et al. (2018)</td><td>Full</td><td>27.94</td><td>27.94</td><td>256</td></tr><tr><td>-</td><td/><td>Ours</td><td>Half</td><td>28.69</td><td>28.33</td><td>460</td></tr><tr><td>-</td><td/><td>Ours</td><td>Full</td><td>29.59</td><td>29.02</td><td>260</td></tr><tr><td>-</td><td/><td>Ours</td><td>Half</td><td>30.10</td><td>30.10</td><td>120</td></tr><tr><td>-</td><td/><td>Shaw et al. (2018)</td><td>Full</td><td>29.75</td><td>29.75</td><td>120</td></tr><tr><td>-</td><td/><td>Shaw et al. (2018)</td><td>Half</td><td>30.50</td><td>30.50</td><td>120</td></tr><tr><td>-</td><td/><td colspan="2">Vaswani et al. (2017) Half</td><td>30.97</td><td>30.97</td><td>120</td></tr><tr><td>Transformer (128M)  †</td><td/><td colspan="2">Al-Rfou et al. (2018) Half</td><td>31.16</td><td>31.16</td><td>120</td></tr><tr><td/><td/><td/><td/><td/><td>23.09</td><td>640</td></tr><tr><td>Transformer-XL (151M)</td><td/><td>Ours</td><td>Full</td><td>23.43</td><td>23.16</td><td>450</td></tr><tr><td/><td/><td/><td/><td/><td>23.35</td><td>300</td></tr></table>

Ablation study on WikiText-103. For the first two blocks, we use a slightly smaller model (128M parameters). † indicates that the corresponding row is reduced to the same setting as the Transformer network in Al-Rfou et al. (

Table content: <table/>

Relative effective context length (RECL) comparison. See text for the definition of RECL and r. The first three models and the last four models are compared as two model groups when we calculate RECL (RECL is computed on a model group rather than a single model). Each group has the same parameter budget. more than a threshold. However, ECL ignores the fact that it is harder to get improvement when a model already achieves a lower perplexity using only a shorter context, and thus it is not suitable for fair comparison among multiple models. We instead propose a new metric called Relative Effective Context Length (RECL). RECL is defined on a model group instead of a single model, and the gain of a long context is measure by the relative improvement over the best short context model. As such, the model group shares the same baseline to enable fair comparison. RECL also has a parameter r, which means constraining the comparison on top-r hard examples. See Appedix C for more details about RECL. As shown in Table6, Transformer-XL manages to model dependency of 900 words long on average with r = 0.1. The RECL of Transformer-XL is 80% and 450% longer than recurrent networks and Transformer respectively. Both the recurrence mechanism and our positional encodings contribute to a longer RECL. This further substantiates our argument that Transformer-XL is able to model longer-term dependency.

Table content: <table><tr><td>Transformer-XL 151M</td><td>900</td><td>800</td><td>700</td></tr><tr><td>QRNN</td><td>500</td><td>400</td><td>300</td></tr><tr><td>LSTM</td><td>400</td><td>300</td><td>200</td></tr><tr><td>Transformer-XL 128M</td><td>700</td><td>600</td><td>500</td></tr><tr><td>-use Shaw et al. (2018) encoding</td><td>400</td><td>400</td><td>300</td></tr><tr><td>-remove recurrence</td><td>300</td><td>300</td><td>300</td></tr><tr><td>Transformer</td><td>128</td><td>128</td><td>128</td></tr></table>

Slowdown in terms of computational time during evaluation. Evaluation is based on per-token time

Table content: <table><tr><td>on one GPU.</td></tr></table>

Ablation study on WikiText-103 with the same GPU memory constraints.

Table content: <table/>

Transformer-XL with baseline under the same memory budget. Transformer-XL still outperforms the baseline even with a shorter backprop length.

Table content: <table/>

Peer review (iclr2019):
 This paper proposes a Transformer based RNN structure "Transformer-XL" to capture long-range contextual relations and targets on language model task. The idea is straightforward: it splits the input sequence into equal and fixed length segments, and recurrently apply the Transformer over the sequence of segments, in which the hidden states for the previous segment are treated as a memory to attend for the next segment. This paper is well-organized and well-written, and easy to follow. The empirical results also demonstrate the proposed model can achieve SoTA performance on several word- and character-based language model benchmarks. Pros: 1. The model is designed based on a careful engineering: 1) taking into account the history hidden states for long-term dependency modeling and 2) alignment scores calculated from multiple perspectives for relative position modeling and global significance capturing. In addition, in contrast to the previous Transformer-based language model, benefiting from the recurrent architecture, both training and decoding can be accelerated. 2. The experimental results show that the proposed Transformer-XL can surpass the baseline model and achieve new state-of-the-art perplexity or bpc on word- or char-based language model task. And, based on the proposed new metric, RECL, the analysis for context length modeling verifies the proposed model can make the best of long-range dependencies. Cons: 1. The proposed model is ad-hoc and is only compatible with language model task. Is it possible to extend the proposed model to more general and practical tasks (e.g., seq2seq tasks)? 2. The absence of a popular language model benchmark, WikiText-2, which has been evaluated in most previous papers. 3. It is notable that there are no ubiquitous decoding techniques for the language used in both the proposed model and baselines, such as dynamical evaluation and continuous cache pointer. However, these techniques are essential for the RNN-LM baselines to achieve state-of-the-art performance, and has been standardly used in most previous works. Therefore, the comparison seems unfair. Minor comments: In Figure 1 and 2, it is better to include a legend explaining the meaning of different colors for different nodes.