Transformer-QL: A Step Towards Making Transformer Network Quadratically Large
Authors: Suvadeep Hajra
Venue:  (2021)
Keywords: deep learning
language model
transformer network
multi-scale transformer network
natural language processing
transformer-xl

Abstract: Transformer networks have shown outstanding performance on many natural language processing tasks. However the context length (the number of previous tokens on which the output states depend) of a Transformer network grows at best linearly with the memory and computational power used. This limitation prevents a transformer network to have very long context in a resource limited application. In this work, we propose a class of transformer networks, namely Transformer-QL (\bf{Q}uadratically \bf{L}arge), in which, the context length can grow at best quadratically with the memory and computational power used. We have empirically evaluated a Transformer-QL model in three long range language modeling datasets. The results show that Transformer-QL can provide significant improvements over other state of the art networks.

Under review as a conference paper at ICLR 2021
TRANSFORMER-QL: A STEP TOWARDS MAKING TRANSFORMER NETWORK QUADRATICALLY LARGE
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer networks have shown outstanding performance many natural lan- guage processing tasks. However the context length (the number of previous to- kens which the output states depend) of Transformer network grows at best linearly with the memory and computational power used. This limitation prevents transformer network to have very long context in resource limited application. In this work, we propose a class transformer networks, namely Transformer - Quadratically Large), in which, the context length grow at best quadratically with the memory and computational power used. We have empirically evaluated Transformer- QL model in three long range language modeling datasets. The re- sults show that Transformer-QU can provide significant improvements over other state of the networks.
INTRODUCTION
Since its introduction in Vaswani et al. (2017), Transformer networks have overtaken its predecessor Recurrent Neural Networks (RNN) almost every natural language processing task. However, one limitation of Transformer network is its high requirement of memory and computational power. In vanilla Transformer network, the memory and computational requirement grows quadratically with the sequence length, and thus with context length.
In an effort overcome the above limitation, Transformer-X XL (Dai et al. 2019 and Compressive Transformer (Rae et al.) 2020) have been recently proposed. However, in both the network, the context length can grow at best linearly with the memory and computation usage. An alternative strategy have been explored in Li et al. (2019h: ; Ye et al. (2019): Child et al. (2019); /aaheer et al.) Belltay et al. 2020); Wang et al. (2020b); Kitaev et al. (2020); Katharopoulos et al. (2020); Choromanski al. (2020); Wang et al. (2020a). All these works have proposed to replace the vanilla self-attention network by a different one with linear or log-lineat memory and computation complex- ity leading to novel transformer architectures with overall linear or log-linear cost. Although, they provide an improvement over the quadratic cost of the vanilla transformer network, the achieved cost is still, at best, linear. Besides, since those techniques based on either sparsification or compression of the self attention mechanism, they struggle to accumulate long distance information (Gupta& & Berant 2020).
Several works such as Burtsev & Sapunov 2000)); Ainslie et al. (2020); Gupta & Berant (2020)
have proposed to increase the context length by introducing a global attention which attains every input token, thus capable of capturing long distance dependency. However, capturing long distance dependency using those approaches involves extreme compression of state space by the global tention mechanism. Moreover, even though, they perform well on several tasks, their performance on language modeling task have not been tested. Another line of work (Zhang et al., 2019 Pappa- gari al. 2019) have suggested to use hierarchical arrangement of transformer network to capture document- wide dependency. However, applicability of those networks requires hicrarchical struc- ture in the data itself. Moreover, those techniques have been proposed for document compression rather than language modeling.
In this paper, propose class of transformer architectures, namely Transformer-O QL (Quadratically Large), to alleviate the problem of capturing long distance dependency. Similar to multi-scale transformer networks (Donahue et al. 2019 Subramanian al. 2020 Zhho al. 2020

---

Under review as a conference paper at ICLR 2021
Dai et al. 2020), Transformer- QL captures the contextual information in multiple temporal scales finer scales to capture recent past information and coarser scales to capture distance past information. Additionally like Transformer-XLL it keeps the hidden states of past segment in memory and use to process future segments causing the context length to grow beyond the current segment. Overall, the context length in Transformer-QL. can grow up quadratically with the memory/computational usage. The contributions of the work are as follows:
We have proposed a novel class of transformer architectures, namely Transformer-QL. in which, the context length can be made to grow linearly with memory and computation cost. Further, employing a linear cost self attention layer like Wang et al. (022b);; Katharopoulos et al. 2020)) the context length of Transformer-QL. can be made grow quadratically in both memory and computational cost.
We have empirically evaluated a Transformer-QU. model on three long range language modeling datasets. The results show significant improvement in perplexity score over Transformer-XI. and Compressive Transformer.
The organization of the paper is as follows. In section 2 the proposed Transformer-O architec- ture along with its background has been introduced. Section 3 provides empirical evaluation of Transformer-QL. The section also studies the sensitivity of Transformer-QL to several hyperparam- eters. Finally, in Section 4. the conclusion has been drawn and future directions of the work have been suggested.
METHOD
2.1 TERMINOLOGY AND NOTATIONS
In transformer network, the input sequence are partitioned into smaller segments of fixed length. Each segment is processed independently of other segments. We refer to the number of tokens in each segment as segment length. In a transformer network with recurrent memory like Transformer- XL, the hidden states of the recent past segments are preserved in a fixed length memory. We refer the number of tokens in each layer of the memory unit as memory length. For an output state (i.e. the output states of the last layer), we use the term context length to refer to the number of past tokens on which the output state depends. In transformer network, different output states might have different context length. We respectively refer the minimum and maximum of the context lengths of all the output states in a network as minimum context length and maximum context length of the network. We refer the sum of segment length and the memory length using the term window length.
We denote the segment length, memory length, window length and model dimension by ng 7' Tl and dm respectively. Thus, we have nw n, + 7mm We also use the notations s/ and m to denote the output and memory of [-th layer at time step t for = 1, ... L where L is the total number of Layers. The output and memory of embedding layer time step have been denoted by sy and mf respectively. The number of heads in the self attention layers has been denoted by H.
2.2 BACKGROUND
Transformer A transformer network consists of stacked collection of multiple transformer layers. Each transformer layer contains a multi-head self-attention layer followed by a position- wise feed forward layer. Though the memory and computational cost of position-wise feed forward layer is linear in the length of input sequence, the multi-head self attention layer has quadratic cost. The transformer network tackles the quadratic memory and computational cost by dividing the input se- quence into smaller segments and applying the transformer network on each segment independently. However, such method limits the context lengths to the segment length. Dai al. (2019) has named this problem as context fragmentation problem.
Transformer-Xi XL Dai et al. 2019) has proposed Transformer-X to solve the context fragmenta- tion problem. In Transformer-X XL, instead of discarding the hidden states after the computation of segment, they are saved in memory (please refer to Figure During the computation of the follow- ing segments, the self attention is applied over the hidden states of both the current segment and the
2

---

Under review as a conference paper at ICLR 2021
memory, thus has an increased context length without quadratic increase in the memory and compu- tational cost. fact, the memory/conputational cost of the self-attention layer of Transformer XL grows only quadratically only with the segment size n, and linearly with the memory length n On the other hand, the context lengths get increased by a length 7mm per layer. By keeping nt small and making 7m large enough, the memory and computational cost of Transformer-Xi can be made close to linear with respect to the context lengths. Rae et al. (2020) has proposed to improved the memorylcomputational
of Transformer-XI further by keeping the part of the memory states in compressed form. However, even with this improvement, the memory and computational cost can be at best linear in the context length.
2.3
THE MODEL
Travforer-- XL, Layer
Trassoner-Xl Layer
Trusfomer- XL Layet
Cassal Aucunuatiin Layer
Tranforner- Layer
Aulonegrossie Producion
Congressin
Compessio
Scale
Output Layers
Figure I: High Level Visualization of Proposed Model. The model processes the input tokens in multiple temporal scales. Each scale has several transformer layers with recurrent memory. The output of the last layer of one scale is compressed to form the input of the next scale. As the segment length gets reduced because of compression, the memory length is increased to make total length (i.e. segment length + memory length) of all the layers same. In the figure, the blue boxes represent hidden states the current time step where as the red boxes represent the memory states.
Overview In this paper, we explore to increase the context length by compressing the hidden states hierarchically. The high level view of our architecture is shown in Figure As shown in the figure, the model processes the input sequence in several scales temporal granularity. Each scale consists of several transformer layers with recurrent memory. The output of the last layer of one scale is compressed causing the temporal granularity as well as the segment length to reduce. As the segment length reduces, we also simultaneously increase the memory length keep the total length (i.e. segment length + memory length) of the layer constant. Then the new segment and memory is fed as the input the first layer of the next scale. The resulting architecture similar the multi- scale transformer architectures (Donahue et al., 2019; (Subramanian et al.) 2020 Zhao et al., 2020). Additionally, Transformer- QI. keeps recurrent memory store hidden states of previous segments. Therefore, in Transformer-QL. the layers belonging to finer scale process contextual information fine-graine grained manner, but have smaller context length. On the other hand, a layer belonging to coarser scale process information in coarse-grained manner, but have a longer context length (please refer to Figure 5 for a detailed illustration the context lengths of Transformer- QL layers). To get the final output of the network, we causally combine the (possibly over-sampled) outputs of the last layer of each scale and pass those through several transformer layers (following Subramanian et al. 2020) Zhao al. 2020 to learn deep representation of the output.
3

---

Under review as a conference paper at ICLR 2021
Function Compute(s, m,
Function Shift(s, m)
begin
begin
s' MultiHeadSAAtt (s. m)
7m len(m)
LayerNorm(s + +*)
m f- concat (m,s)
8" LayerNorm(s" + PPssiselF''s" (s"))
m" m' :]
return s"
return slop.gradient(mm))
(b) One Transformer-Xi layer over hidden states s and memory m
(d) Shift the current hidden state into memory
m
At the beginning begin
m.:
m
1-1
<- 0
11 initialize all the memory to zero
1 (-1
// initialize time step to 1
while there more data to process
sf + X. W cmb
// embed input segment // shift hidden states into memory // initialize shift Length n, 11 list to hold all the inputs of accumulation layer
m. t (hift(s; m
shift_len n acc acc_inputs
10
1g +
// initialize global layer index to 1
for i +- to no scales do
Ls of layers scale for 1 to L;
L; the number of layers at scale
13
s," +- Compute(s," u m .y) m, <- Shift(s/" [: shift sen] m 4 1 +1
// run one Transformer-Xi Layer // shift hidden states into memory // increase the global layer index
15
17
acc imputs (ppend m
1/ put output scale in acc // not the last scale, add a compression layer s fe(concat(m] 1-11 4-11 "[n"" --==))))=)== // reduce the shift_len for the new scale 1a Shift(s!" (: shift len), m(,) // shift hidden states into memory m,
18
if < of then
// compress the states
29
shift_len + shift_len/c
22
lo 1 +1
// increase the global layer index
23
shift len n
set the shift ft length for the output layers // combine the states in acc inputs // shift hidden states into memory // increase the global layer index 11 set Lo to the number of output layers
do + Accumulate(ace inputs)
24
la Shift(s!" len), m/z,)
26
la + +-++1 11+1
27
Lo + no of output layers
for 1 + to Lo do
do + Compute(s) m [b) m, la + Shift(s/"1 sen], m( + 1 +1
// one Transformer-xi. layer shift hidden states into memory // increase the global layer index increase the time step
30
31
32
(c) Forward pass of Transformer QL. st and m² represents the hidden states and memory states of layer I at time step ni, and nin represent the segment length and memory length of 1-th layer.
Figure 2: Transformer-0 QL Algorithm.
The Compression Function For compression, we use one of average pooling and max pooling with pool size and stride both equal to where c is the rate by which we compress the states while transitioning from one scale to the next. Let s and m the output and memory states (-th layer with length n', and ni,, respectively. We apply the compression function on the concatenation of m and sf to get the output s++ of length n.+1 (n., n/ (for simplicity assume that n², + ni,, is divisible by e). If n+1 ny. take the last n¹; elements of s,+1 form the output of the
4

---

Under review as a conference paper at ICLR 2021
compression layer. Finally, we keep recurrent memory m; 1+1 of length n' + n/n m making n2, + nm = nf + n to hold.
The Memory Updates In Transformer-XI. with segment length n, and memory length ams the segment of length n, is get shifted into the memory. In other words, the memory for the next time concat(m', s)[_ :] for all layer l. However, in Transformer-QU.
step is computed m the granularity of layers belonging to different scales are different. More precisely, a segment of length nº belonging scale 1 is compressed into segment of length ng//²-1 at layer belonging to scale i. Thus, in Transformer- QL, we update the memory of a layer belonging to scale i as m 1 concat memory length at scale i for = 0, 1, ... respectively. The complete algorithm of Transformer-QU. is shown in Figure 2
[*))--n", :] where ni n and n'n are the shift length and the
Droppath Regularization Since the output of the last layer of every scale is summed in the ac- cumulation layer, the path through higher scale forms to deeper network while the path through a lower scale forms to a shallower network. Consequently, layers in the higher scales might remain under-fitted due lack of gradient flow through the deeper network while the layers in the lower scales get over-fitted. To alleviate this problem, we have introduced droppath regularization. In the accumulation layer, let each output be computed as 8 Σi= st where s represents the (possibly over-sampled) output of scale and is the total number of scales. droppath regular- ization with droppath probability p, we drop the output of all the scales below with probability p/(1 (1- - 1) for j = 2,3, ........ , from the accumulated output. More precisely, we generate a random
number u. from uniform probability distribution and compute the output as s" [-J4T Ei=, st if
(1-2pp 6-11p For u 2 p, no droppath applied. 1-1 1-1
E
2.4 THE COMPLEXITY
The memory/computationa complexity of Transformer-Xl. network (Dai et al. 2019) with seg- ment length Th memory length nm and L layer is (((((m) Tua) + L) where a(...) is the complexity of self-attention layer. The context length n. of the network is () Since
2019 Child et al. 2019 Zaheer et al. 2020; am n2. Q(n, + ni.) (Li et al. 2019 Ye et al.
Beltagy al. 2020 Wang al. 2020b Kitaev al. 2020 Katharopoulos et al. [2020] Choroman
ski et al. 2020). the memory and computational complexity of Transformer- XL in terms of context
length is ) ). Similarly, the memory and computational complexity of Compressive Transformer
(Rae et al. 2020) in term of context length is ((ne/c) where is the compression rate. Therefore, the memory/computational complexity of both Transformer-XI network and Compressive Transformer network in term of the context length is at least linear. Consequently, increasing the context length in both the networks requires least linear increase in the amount of both memory and computational requirements.
On the other hand, a Transformer-Qu QL network with L Transformer-Xi layers and compression layers, the context length n, becomes )...)) O(c"g,"> (n, + nmm) )- = (n,,(n, n.,) where ni = n and nm ni are the segment and memory length in scale 1 of the network. Note that, since at most = log."ls compression layer can be used in Transformer-QI. we have o' Ocd"g.".)
O(n.). If we set Tim O(n.). we have ne O(n.))). However, the time and memory complexity of a Transformer-QL network is O(a(m,, P + (m, + nm)i)) (a nm + i)). Since a(n,, nm) Q(n, + nm) and we set n' = O(ng). the mem- ory/computational complexity of Transformer- QL becomes [(n,(L + i)). Therefore, the mem- orylcomputational complexity of Transformer- QL. terms of context length is ((yT.(T. + f))
(L+ + log. Thus, the complexity of Transformer-QL. can be at best sub-linear. More- over, if we set the compression rate to nt,, the memory and computational complexity can be at best (vvrz) or, in other words, the context length can be at best quadratic in the memory and com- putational cost. In Appendix B we provide an algorithm compute tight estimation of the context length of a Transformer- QL network. In the appendix, we have also provided a detailed illustration of the dependency structure of the hidden states of Transformer-QU QI. network the past tokens.
Ω (V
5

---

Under review as a conference paper at ICLR 2021
Dataset
Model
Test
Average test Ne
Test
n2/ mm/ Bem
Perplexity
Transformer-Xi
01/12/
16
98
20.15
Comp- Tranffornnormm
04/06/06
16
146
19.67
SimpleBooks-2
Transformer-QI.
04/12/-
16
138
18.
Transformer-X XL
08/24/-
32
196
19.61
Comp-Transformer
08/12/12
32
292
19.15
Transformer-U
08/24/-
32
276
18.56
Transformer-Xe
04/12/-
16
98
12.93
Comp-Transformer
04/06/06
16
146
12.49
SimpleBooks-92
Transformer-QL
04/12/-
16
138
12.17
Transformer-X -
08/24/-
32
196
12.35
Comp-Treee Transformer
08/12/12
32
292
12.01
Transformer-QI.
08/24/-
32
276
11.88
Transformer-X XL
04/12/-
16
98
31.19
Comp-Transfoo Tran-orrasfrrmer
01/06/06
16
146
31.91
WikiText-103
Transformer-00 QL
04/12/-
16
138
29.13
Transformer-XI
08/24/-
32
196
27.52
Comp-Transformer
08/12/12
32
292
27.19
Transformer-QL
08/24/-
32
276
26.63
Table 1: Perplexity scores (lower is better) of the three networks: Transformer-Ql. Transformer- XI. and Compressive Transformer (Comp-Transforee length (n,), memory length (nm) and compressed memory length (nem) of the test model. The forth column shows the window length new of the test model. Note that, for Transformer-Xe and Transformer-QL. the window length is n, + 7m and for Compressive Transformer the window length n, + 7m + Teem- The average test context length n, has been shown the fifth column. The average test context length has been computed by taking the average of the minimum and max- imum context length of the test model where the minimum and maximum context length have been calculated using the algorithm of Appendix
The third column shows segment
EMPIRICAL EVALUATION
In this section, empirically evaluate the efficacy of Traansoormer-QQ QL long range language mod- eling task. Towards that goal, we compare the results of Transformer-QL. with that of Transformer- XL (Dai al. 2019) and Compressive Transformer (Rae al. 2020). Then we evaluate the sensi- tivity of Transformer- QL to several hyper- pareerr-raeterss....
3.1 COMPARISON WITH STATE OF THE ART METHODS
State the Art Methods compare Transformer-QL network with the following two networks:
Transformer-XL (Dai et al., 2019) Transformer-XI. is similar to vanilla Transformer with two modifications. It uses recurrent memory to store and access the hidden states of the past time steps. The recurrent memory enables to increase the minimum context length up to n.n I. where n.m is the memory length and I, is the number of layers. also uses relative positional embedding of token instead of absolute positional embedding.
Compressive Transformer (Rae et al. 2020) Like Transformer-XI.. Compressive Transformer also uses recurrent memory. However, Compressive Transformer keeps part of the recurrent memory in compressed format, thus has increased context length over Transformer-XL
We compare Transformer-Ou QL against the above two networks on three long range lan- guage modeling datasets: SimpleBooks-2 (Nguyen, 2019), SimpleBooks-9P (Nguyen 2019) and WikiText- (Merity et al., (017). SimpleBooks-2 and SimpleBooks-9 922 are created from Guten- berg book corpus (www.gutenberg.org while WikiText- 103 created from Wikipedia articles. All
Datasets

---

Under review as a conference paper at ICLR 2021
Model dimension
Train
Test
Trans-QI.
Trans-XL.
Improvement
Relative
T/. Pm/
Improvement
512
16/16
16/16
33.32
32.68
-0.64
-1.66%
1024
16/16
16/16
27.70
28.42
0.72
+2.53%
1536
16/16
16/16
27.00
28.22
1.22
+44.8%%
Table Improvement in test perplexity score (lower is better) of Transformer--Q over Transformer- XL for three different model dimensions. The forth and fifth columns show the test perplexity obtained by Transformer- and Transformer-XI.
the three datasets preserve paragraph and section structures of their sources making those suitable for long range language modeling task. The statistics of the three datasets are shown in Table of Appendix
Experimental Details For the experiments of Transformer-Xe and Compressive Transformer, we have used 8-layer network. And for the experiments of Transformer-QU we have used network with 3 layers in scale 1, 3 layers in scale 2 and 2 layers in the output block. Thus, the Transformer- QL has total of eight layers in Transformer-XL and Compressive Transformer. We the compression rate of Compressive Transformer to 2. In Transformer-QL, we have used max-pooling layer with pool size 2 and stride 2 as the compression layer. Thus, both the Transformer-O!! and Compressive Transformer have compression rate of 2. For the experiments on the SimpleBooks- 92 and WikiText-103, we have set the model dimension 1536 and used an initial learning rate of 1 x 10-1 On the other hand, for the experiments on SimpleBooks-2, we have set the model dimension to 256 and learning rate 2.5 x 10-4. All the models have been trained using Adam optimizer. We set the droppath probability of Transformer-QL to 0.3. The details of other hyper- parameters can be found in Appendix
The results of the comparison are shown in Table [ The results are grouped by the window length uu of test model as the lower bound of memory and computation requirement directly depends on it. In all the datasets and settings, Tra performs worst among all three. Worst performance of Transformer-Xe XL is not surprising as it has smallest average con- text length (shown in the fifth column) for given Rw. However, Compressive Transformer has a slightly larger average context length than Transformer-QL. Yet, Transformer- QL has performed similarly or significantly better than Compressive Transformer in all the setting which indicates that Transformer- QL can exploits the contextual information more effectively than Compressive Trans- former.
Results
3.2 EFFECT OF MODEL DIMENSION
In this section, we investigate the effect of model dimension on the performance of Transformer- QL. Towards that goal, we have performed experiments on WikiText-103 dataset with varying model dimension. For each experiment, we compared the test perplexity of Transformer-0 that of Transformer-Xit XL. The results are shown in Table The improvement in test perplexity of Transformer--0 over Transformer-Xi has been computed by subtracting the test perplexity of Transformer-O0 QL from that of Transformer-XL. The relative improvement is computed by
with
Improvement x 100
Relative improvement
Test perplexity of Transformer-X.
As shown the table, Transformer-QL. performs relatively worse for small model dimension like 512 and relative improvement increases as the model dimension increases. We speculate that the relatively worse performance of Transformer-QL1 for smaller model dimension caused by the difficulty in compressing hidden states during switching from one scale the next. To alleviate the problem, Donahue et al. (2019) have proposed to increase the model dimension as the model transits from lower scale to higher one. On the other hand, Dai al. (2020) have suggested novel sneer-oonly-owee to solve the problem. We take it as a future work to try those approaches in -ra

---

Under review as a conference paper at ICLR 2021
3.3 EFFECT OF CONTEXT LENGTH
In this section, we study the relative improvement in perplexity scores obtained by Transformer-Ql. over Transformer-XI for varying context length. The results are shown in Table 3 From the table,
can be noticed that relative improvement obtained by Transformer-QU. is more when the context length of the Transformer-XL networks are smaller the first place. For example, for test n, = 08 and P'm = 08, the relative improvement is as high as 8.80% On the other hand, for the test n, = 02 and Pum = 30, the relative improvement is only 2.76%. This can be explained by the fact that for the segment and memory length 02 and 30, the average context length of Transformer-XX XL is already large enough (241) to provide good result. By extending the average context length from 241 to 332, Transformer-QL. provides only a small improvement following the law of diminishing return (Hestness et al. 2017).
Model Dimension
Test
Avg.
test
Perplexity
/a//um
Trans-XL
Trans-QL.
Trans-XI.
Trans- QL
Rel Impry
08/08
68
100
33.08
30.17
+8.80%
1536
16/16
136
200
28.22
27.00
+ +4.32%
02/30
211
332
27.19
26.44
+...6%
Table 3: Relative improvements in perplexity scores (lower better) obtained by Transformer-QU over Transformer-XL on WikiText-100 dataset. The second column shows the test segment (n,) and memory (nm) length. The third and forth column respectively show the average test context length (ne) of Transformer-Xe and Transformer- QL network.
CONCLUSION AND FUTURE WORK
In the work, we have proposed a class of transformer networks namely Transformer- QL in which the context length can grow quadratically in memory and computational usage. Our empirical eval- uation shows that Transformer-QU -L can perform significantly better than other long range language modeling networks like Transformer- XL and Multi-scale Transformer by exploiting longer context length. Further more, it can perform significantly better than Compressive Transformer by exploiting the contextual information more effectively.
In our empirical evaluation, we have evaluated Transformer-QU network with only compres- sion layer. In future, we want to evaluate a network with more then one compression layers. Also, have empirically found that the performance of Transformer-QL. network can be worse than that of Transformer-Xi when the model dimension is small. As our future work, we want explore different methods for removing this limitation.
REFERENCES
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. ETC: encoding long and structured data in transformers. CoRR, abs/2004.08483, 2020.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, 00s/2004.00150, 2020.
Mikhail S. Burtsev and Grigory V. Sapunov. Memory transformer. CoRR, abs/2006. 11522000.11527,. 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse 10509, 2019.
transformers. CoRR,
Krzysztof Choromanski. Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamás Sarlós, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for pro- teins via linearly scalable long-context transformers. CoRR, abs/2006.03555 03555, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoe Viet Le, and Ruslan Salakhutdi- nov. Transforme-Xi XL: Attentive language models beyond fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pp. 2978- 2988. Association for Computational Linguistics, 2019.
8

---

Under review as a conference paper at ICLR 2021
Zihang Dai, Guokun Lai, Yiming Yang. and Quoe V. Le. Fumnel-Transforen Filtering out sequen- tial redundancy for efficient language processing. CoRR, abs/2006.03236, 2020.
David Donahue, Vladislav Lialin, and Anna Rumshisky, Injecting hierarchy with U-Net transform- ers. CoRR, abs/1910. 10488, 2019.
Ankit Gupta and Jonathan Berant. GMAT: global memory augmentation for transformers. CoRR, abs/2006.03274, 2020. URL
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang, and Yangi Zhou. Deep learning scaling predictable, empirically. CoRR, abs/1712.00400 2017.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. CoRR, abs/2006.16236, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020.April 26-30, 2020. OpenRe- view.net, 2020.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. In Advances in Neural Information Processing Systems 32: NeurIPS 2019, pp. 5244- 5254, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, April 24-26, 2017, Conference Track Proceedings. Openkeview.net, 2017.
Huyen Nguyen. SimpleBooks: Long-term dependency book dataset with simplified English vocab- ulary for word-level language modeling. CoRR, abs/1911.12399 2019.
Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. Hierar- chical transformers for long document classification. IEEE Automatic Speech Recognition and Understanding Workshop. ASRU 2019, Singapore, December 14-18, 2019, pp. 838 844. IEEE, 2019.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
Sandeep Subramanian, Ronan Collobert, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multi-scale transformer language models. CORR, abs/2005.00581. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30, pp. 5998-6008, 2017.
Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. Cluster-former. Clustering-bant sparse transformer for longerange dependency encoding. CoRR, ab6/2009.060977, 2020a.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, (bs/2006.04768, 2020b.
Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. BP-Transformer: Modelling long-range context via binary partitioning. CoRR, abs/1911, 1.1111, 2019.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. Big bird: Transformers for longer sequences. CoRR, abs/2007. 14062, 2020.

---

Under review as a conference paper at ICLR 2021
Xingxing Zhang, Furu Wei, and Ming Zhou. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Volume 1: Long Papers, pp- 5059- 5069. Association for Computational Linguistics, 2019.
Yucheng Zhao, Chong Luo, Zheng-Jun Zha, and Wenjun Zeng. Multi-scale group transformer for long sequence modeling in speech separation. In Proceedings of the /wentt-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3251-3257, ijeai.org, 2020.
A TRANSFORMER -XL ALGORITHM
The algorithm for Transformer-XL, is shown in Figure
At the beginning
begin
m.
1L-1 L-11
+- 0
// initialize memory to zero
$ 1
// initialize time step to 1
while there more data process do
s < X, W cmb
// embed input segment shirt aidden states into memory
m. < Shi . m 1)
for +- 1 to L do
Compute(s,) 1-1 m 1-11111 m Shift(s/, m
// run one Transformer-Xii layer // shift hidden states into memory // increase the time step
11
Figure 3: Forward pass of Transformer-Xe XL. The functions Compute and Shift are given in Figure 2b and 2d respectively.
cur + nh
// set current segment length to ns // set current memory length // initialize minimum context length to o
cur_mlen
min_clen
cur carerayrr.cler <- Thm
// set current layer context length to nm
for 4- 1 to of do
L, + no of layers at scale
// set L, to the number of layers at scale L min.clen + min Jayerryrn // increase min.llen by cur.layer.clen
for ( 1 to L, do
if < no of scales then
// there is a compression layer
// update min.clen, cur.mlen, cur_slen and cur cur
// appropristely
10
min.clen +- min.clen min(cur.nneeeeeeeeee /cur
11
cur_mlen + (nm + n.) min( (n,, + n.) n,)/2, n.)
cur.slen + cur_slen/c
13
car.eayer.clen + curnmlen na/cur_sle
14 bayer.clen-- L, of output layers
11 reset layer.ccn for output layers set L. to the number of output layers
16 lorl=11 to L., do
17 L min.clen 4- min.clen + cur layer.ccnn
// increase min.clen by cur.layer.clen
return min_clen
Computation of minimum context length a Transformer-QU model. The n., nm respec- tively represent the segment and memory length of the scale of the network.
Figure 4:
10

---

Under review as a conference paper at ICLR 2021
B CONTEXT LENGTH OF TRANSFORMER-ON QL
A tight estimate of the minimum context length of a Transformer-QL. network can be computed using algorithm of Figure 4 For simplicity, we have assumed that all the division operations result into integer output. We have also assumed that there is at least one layer in every scale. The maximum context length can be obtained by adding n, to the minimum context length.
Autoregressive Prediction
S 514-5
6
477-
Transformer XI.I Layer
-7
47.c
474-5
Pt 47
aid -2
Accumulation
Layer Layer
SB
Transformer XL Layer
33 25
1---555-- 20
a1--11 10
1 27:3-11
1 200--8
19ct-4
y 19tr
Compression
1-277-11
20c--0
23
19
19ct-2
10rt
Transformer XLLneer
10
10:4
15tt 10
154-8
11:-6
1=f-44
7
Compression
S 11
TransformerXI Layer
7
St 6---- 64-6
4
33--3
(-24 26
Figure 5: Dependency of hidden states to the past tokens Transformer-QU network.
Additionally, in Figure we have shown the detailed computation of minimun/maximum context length with example. In the figure, the notation S St used to denote a hidden states of 1-th layer and the state depends on the 41-th to ly-th tokens of the input sequence. In the example of the figure, each output state depends on at least 44 previous tokens. In other words, minimum context length of the network is 44. On the other hand, in Transformer-XX XL network of same segment length, memory length and number of layers, the minimum context length would have been 4 x 16.
C STATISTICS OF DATASETS
The statistics of the datasets shown in Table 4

---

Under review as a conference paper at ICLR 2021
Dataset
Number Tokens
Vocahulary Size
Average
Frequency
SimpleBooks-2
2.2M
11,
195.43
SimpleBooks-92
91.5M
98, 304
931.40
WikiText-
103M
267, 735
385.56
Table 4: Statistics of the datasets used in the experiments.
D COMPARISON WITH MULTI-SCALE TRANSFORMER
In this section, we empirically compare Transformer-Qu with Multi-scale Transformer (Subrama- nian et al., 2020). Our implementation of Multi-scale Transformer is same Transformer- QI. without any recurrent memory. The resultant Multi-scale Transformer is similar to the button-up model of Subramanian et al. (2020). We have used hyperparameter settings same as Transformer- QL to train the Multi-scale Transformer. The result is shown Table 5 From the table, we can see that Multi-scale Transformer has been widely bitten by Transformer- QL even when the Multi-scale Transformer has been trained and tested with a larger window length.
Dataset
Model
Train
Test
Average test
Test Perplexity
71m
71m
MS-Translormer
16
16
16
8
25.13
SimpleBooks-2
Transformer-QL.
08
08
08
08
100
18.92
MS-Trarslormer
16
21.89
Transformer-QU
08
08
16
16
32
200
18.57
MS- Torr
64
16
28.37
SimpleBooks-92
Transformer- QL
08
08
08
08
16
100
12.42
MS-Transforrer
64
64
64
32
14.30
Transformer- QL
08
08
16
200
11.92
MS-Transtormer
128
32
16
44.05
WikiTexx-
Transformer-0 QI.
16
16
08
08
100
30.17
MS- MSS-Transformer
128
128
128
64
29.15
Transformer-O1 QL
16
16
16
16
32
200
27.00
Table 5: Comparison of Transformer-QL. with Multi-scale Transformer (MS-Transformer), The third and forth column respectively show the segment length (n.) and the memory length (nm) used during training. The fifth, sixth and seventh columns respectively show the segment, memory and the window (n n, Rim) length used to compute the text perplexities. The eighth column shows the average context length (n.) of the test models.
E HYPERPARAMETER SETTING
We used the following values for hyperparameter for the experiments SimpleBooks-22 datasets:
12

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Transtormer-XL.
Compressive Transformer
Transtormer-OL.
Multi-scale Transformer
d model
256
256
256
256
d embed
256
256
256
256
div_val
1
1
untie_r
False True
False True
False True
False True
proj.same_dim
head
4
dehead
64
64
64
64
d d_inner
1024
1024
1024
1024
train batch
128x8
128x8
128x8
128x8
Rx 08
train fis
08
04
08
train 7ym
08
06
08
train Nem
06
pre pro_lnorm
True
True
True
True
warmup_ster
0
0
train_steps
600 000
60, 000
60, 000
60. 000
learning rate
0.00025
0.00025
0.00025
0.00025
min Ir ratio
0.004
0.004
0.004
0.004
clip
0.25
0.25
0.25
0.25
dropout
0.1
0.1
0.1
0.1
dropatt
0.1
0.1
0.1
0.1
droppath
0.3
0.3
init_std
0.02
0.02
0.02
0.02
proj_init init_
0.01
0.01
0.01
0.01
recons loss. weight
0.01
For the SimpleBooks-92 datasets and model dimension 1536, the following values of hyperparame- ters are used:
Hyperparameter
Transformer-X XL
Compressive Transformer
Transformer-G QL
Multi-scale Transformer
d_model d embed div_val
1536
1536
1536
1536
1536
1536
1536
1536
4
4
untie_r
False
False True
False
False
proj_same_ding
True
True
True
n_head
16
16
16
16
d head
96
96
96
96
inner
6144
6144
6144
6144
train batch size
512x8
512x8
512x8
512x8
train 71.
08
04
0s
08
train 7mm
08
06
08
08
train Rem
06
pre Inorm
True
True
True
True
warmup_steps
0
0
train_steps
250, 000 0.0001
250, 000
250, 000 0.0001
250, 000
learning
0.0001
0.0001
min ratio
0.004
0.004
0.004
0.004
clip
0.1
0.1
0.1
0.1
dropout
0.15
0.15
0.15
0.15
dropatt
0.15
0.15
0.15
0.15
droppath
0.3
0.3
0.3
init_std
0.02
0.02
0.02
0.02
proj-init_std
0.01
0.01
0.01
0.01
recons loss weight
0.01
13

---

Under review as a conference paper at ICLR 2021
For the Wiki'exx- 103 datasets and model dimension 1536, the following values of hyperparameters are used:
Hyperparameter
Transformer-XX
Compressive Transformer
Transformer-QL.
Multi-scale Transformer
d_model
1536
1536
1536
1536
d_embed div val untie.r
1536
1536
1536
1536
4
4
4
False
False
False
False
proj_same_dimm
True
True
True
True
n_head d d_head d_inner
16
16
16
16
96
96
96
96
6144
6144
6144
6144
batchsseth
512x16
512x16
512x16
512x16
th
na 08
n. 16
train 7i
16
16
train 74m
16
12
train 7e
12
pre_Inorm
True
True
True
True
warmup steps
0
0
0
train steps
350, 000 0.0001
350, 000 0.0001
350, 000
350, 000
learning_rate
0.0001
0.0001
min
0.004
0.004
0.004
0.004
clip
0.1
0.1
0.1
0.1
dropout
0.15
0.15
0.15
0.15
dropatt
0.15
0.15
0.15
0.15
droppath
0.3
0.3
init_
0.02
0.02
0.02
0.02
proj_init_std
0.01
0.01
0.01
0.01
compression rate
2
recons loss weight
0.01
For training models of model dimensions 512 and 1024, we have used initial learning rate of 0.0005 and 0.00025 respectively keeping the rest of the hyper-parameters same.
14


Reviews:
[{"review_id": "hbKcZiXjhad", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nUsing multi-scale hierarchical and compressive techniques, this paper examines a way to increase the context length of transformers.  Although the individual components are similar to previous work, they are combined in a novel way that shows a path toward longer and more efficient context lengths.\n\nPositives\n---------\nIncreasing the context length of transformers is an interesting and relevant topic, and the proposed solution can have real impact in moving the state of the art forward.\n\nThe results demonstrated in the experiments show an improvement over previous models.\n\nThe paper is clear and detailed, and well situated in the literature.  The algorithms were clear and the comments were useful for understanding the proposed idea.\n\nNegatives\n---------\nThe experiments do not compare to many other approaches, even though those approaches are cited throughout the paper.  In the conclusion, the paper mentions comparison with Multi-Scale approaches, but that is not present in the experiments.  Since the architecture (ignoring the compression) is similar to multi-scale approaches, it would be good to compare against empirically.\n\nAlthough the complexity analysis is thorough, I'd like to see empirical results of memory/compute requirements as a function of the context length.  If I understand correctly, the sub-linear results depend on particular settings of the memory length and compression rate.  How do the settings used in the experiments compare to those used for the analysis?  Is there a perplexity/efficiency tradeoff, and can you characterize that experimentally?\n\nReasons for Score\n-----------------\nThe idea proposed in the paper is novel and exciting, but I have some concerns about whether the gains promised by the theoretical analysis can be realized while maintaining modeling quality.\n\nMinor issues that did not affect score\n------------------\nFigure 1 has some scaling/resolution issues that make it hard to read.  It's also sideways, which is inconvenient.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tkjgb9P_xsx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Transformer-QL, an improvement over Transformer-XL architecture which allows to use longer context at reduced cost. The paper is well-written and its good experimental results are strengthened by theoretical complexity estimation.\n\nThe text of the paper, though well-structured and concise, requires some polishing and misspellings correction (for example, I am doubtful that the proposed architecture is endowed with the ability to bite while section D of the appendix claims that \"Multi-scale Transformer has been widely bitten by Transformer-QL\").", "extra_scores": {}, "extra_text": {}}, {"review_id": "GGMOZ9g1qog", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work introduces the Transformer-QL, a transformer-based model that aims to capture long distance dependencies in the input. The network processes the information defining multiple temporal scales, with finer scales for nearby elements, and coarser scales for distant information. It also includes the recurrent memory extension from Transformer-XL from Dai et al. The model is tested in a long range language modeling task.\n \n==================\n\nThe presented Transformer-QL model improves over similar previous methods on language modeling. A thorough complexity analysis is included. The presentation can be improved, all the definitions are hard to follow. The layers are described in a textual fashion, barely any math (and extended in the pseudo-code). Further, the performance improvements are nice, though not impressive. \n\n==================\n\nThe paper is hard to follow for moments. For example, a Figure to define what are $n_s, n_c, n_m$ would help to understand the paper with a nice visual benefit. Introducing the layer and networks in a simple way would help clarify the implementation and other notation.\n\nThe paper describes the idea of multiple temporal scales. In the experimental sections, it seems that only two scales are used. What would happen if the number of scales is increased to capture  longer contexts? Would the method gain in performance? Further, how does the network perform when a longer context is obtained *maintaining the same number of parameters* as a network with less temporal scales? \n\nThe motivation behind Transformer-QL is to increase the context length processed beyond what other methods can. I agree with the authors that extending the context is important. How effective is the method to capture farther long-term dependencies compared to previous methods? The experimental results mainly address similar networks with similar context lengths. \nThis could be tested by ablating elements from the input or ablating the recurrent memory vectors (setting them to zero during inference).\n\nWhat would happen to the network if compression is removed? \n\n==================\n\nMy score is based on the clarity improvements needed and the lack of evaluation of the effectiveness on the main objective of the method (longer contexts). I would appreciate if the authors can reply to my questions above.\n\n==================\n\nMinor issues:\n\n-Section 2.1: “input sequence are” -> “input sequences are”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-4DW1m-CoGz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Transformer-QL to capture the contextual information in multiple temporal scales - finer scales to capture recent past information and coarser scales to capture distance past information. The results show significant improvement in the perplexity score over Transformer-XL and Compressive Transformer. I agree that making use of multiple temporal scales can capturing more context information. However, the proposed method is like to merge the key ideas from Transformer-XL and Compressive Transformer. And my major concern regards the experiments. The performance of Transformer-QL is still far from the work \"ADAPTIVE INPUT REPRESENTATIONS FOR NEURAL LANGUAGE MODELING\" which does not target capturing long context. Then why do we need Transformer-QL? The authors should have done more experiments on other datasets, maybe refer to the paper, Big Bird: Transformers for Longer Sequences.\n\nPros:\n1. The idea of using multiple temporal scales to capture more context information is convincing.\n2. The experiments show that Transformer-QL can outperformance Transformer-XL and Compressive Transformer.\n\nCons:\n1. The authors only test the model on one language modeling task and far from SOTA model on this task. More datasets should be used to evaluate the model.\n2. The authors should have some comparison to some recent models on processing long sequences, such as the models from \"Efficient Transformers: A Survey\"\n\n#######\nAs no author response, I will keep my rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces Transformer-QL, a new variant of transformer networks that can process long sequences more efficiently. This is an important research problem, which has been widely studied recently. Unfortunately, this paper does not compare to such previous works (eg. see "Efficient transformers: A survey"), the only considered baselines being Transformer-XL and Compressive transformer. Moreover, the reviewers found the experimental section to be lacking, as the results are weak compared to existing work, and important ablation studies are missing. The authors did not provide a rebuttal. For these reasons, I recommend to reject the paper.
Decision: Reject