R-TRANSFORMER: RECURRENT NEURAL NETWORK ENHANCED TRANSFORMER
Authors: Zhiwei Wang
Yao Ma
Zitao Liu
Jiliang Tang
Venue:  (2020)
Keywords: Sequence Modeling
Multi-head Attention
RNNs

Abstract: Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that are built on convolution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effectiveness in capturing long-term dependencies in a variety of sequence modeling tasks. Despite their success, however, these models lack necessary components to model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks. The proposed model can effectively capture both local structures and global long-term dependencies in sequences without any use of position embeddings. We evaluate R-Transformer through extensive experiments with data from a wide range of domains and the empirical results show that R-Transformer outperforms the state-of-the-art methods by a large margin in most of the tasks.

Under review as a conference paper at ICLR 2020
R- R-RRANSFORM RECURRENT NEURAL NETWORK ENHANCED TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
Recurrent Neural Networks have long been the dominating choice for sequence modeling. However, it severely suffers from two issues: impotent in capturing very long-term dependencies and unable to parallelize the sequential computation procedure. Therefore, many non-recurrent sequence models that built con- volution and attention operations have been proposed recently. Notably, models with multi-head attention such as Transformer have demonstrated extreme effec- tiveness in capturing long-term dependencies in variety of sequence modeling tasks. Despite their success, however, these models lack necessary components model local structures in sequences and heavily rely on position embeddings that have limited effects and require a considerable amount of design efforts. In this paper, we propose the R-Transformer which enjoys the advantages of both RNNs and the multi-head attention mechanism while avoids their respective drawbacks, The proposed model can effectively capture both local structures and global long- term dependencies in sequences without any use of position embeddings. We eval- uate R- R-Transformer through extensive experiments with data from wide range of domains and the empirical results show that R- R-Transformer outperforms the state-of-the-ard methods by large margin in most of the tasks. We have made the code and data publicly available
INTRODUCTION
Recurrent Neural Networks (RNNs) especially its variants such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have achieved great success in wide range of sequence learning tasks including language modeling, speech recognition, recommendation, etc (Mikolov) et al. 2010, Sundermeyer et al. 2012: Graves & Jaitly 2014; Hinton et al. 2012: Hidasi et al., 2015). Despite their success, however, the recurrent structure is often troubled by two notorious issues. First, it easily suffers from gradient vanishing and exploding problems, which largely limits their ability to learn very long-term dependencies (Pascanu et al., (001)). Second, the sequential nature of both forward and backward passes makes it extremely difficult, if not impossible, to par- allelize the computation, which dramatically increases the time complexity in both training and testing procedure. Therefore, many recently developed sequence learning models have completely jettisoned the recurrent structure and only rely on convolution operation or attention mechanism that easy parallelize and allow the information flow an arbitrary length. Two representative mod- els that have drawn great attention are Temporal Convolution Networks(TCN) (Bai et al. (2018) and Transformer (Vaswani et al. (017). variety of sequence learning tasks, they have demonstrated comparable or even better performance than that of RNNs (Gehring et al., 2017: Bai et al. 2018 Devlin et al. 2018).
The remarkable performance achieved by such models largely comes from their ability to capture long-term dependencies sequences. particular, the multi-head attention mechanism in Trans- former allows every position to be directly connected to any other positions in a sequence. Thus, the information can flow across positions without any intermediate loss. Nevertheless, there are two issues that can harm the effecctiveness of multi-head attention mechanism for sequence learn- ing. The first comes from the loss of sequential information of positions as it treats every position identically. To mitigate this problem, Transformer introduces position embeddings, whose effects,

---

Under review as a conference paper at ICLR 2020
FeedForward:
Multi-head Attention:
M
M
M
M
Local RNN:
R
R
R
And & Horai
Figure 1: The illustration of one layer of R- Transformer. There are three different networks that are arranged hierarchically. particular, lower-level locaIRNNs that process positions local window sequentially (This figure shows an example of local window of size 3); The middle-level multi-head attention networks which capture the global long-term dependencies; The upper-level is Position-wis feedforward networks that conduct non-linear feature transformation. These three networks are connected by residual and layer normalization operation. The circles with dash line the paddings of the input sequence
however, have been shown to be limited (Dehghani et al. 2018: Al-Rfou et al., 2018). In addition, requires considerable amount of efforts to design more effective position embeddings or different ways to incorporate them in the learning process (Dai et al. 2019). Second, while multi-head atten- tion mechanism is able to learn the global dependencies, we argue that it ignores the local structures that are inherently important in sequences such as natural languages. Even with the help of position embeddings, the signals at local positions can still be very weak as the number of other positions is significantly more.
To address the aforementioned limitations of the standard Transformer, in this paper, we propose novel sequence learning model, termed R-Transformer. It is multi-layer architecture built on RNNs and the standard Transformer, and enjoys the advantages of both worlds while naturally avoids their respective drawbacks. More specifically, before computing global dependencies of po- sitions with the multi-head attention mechanism, we firstly refine the representation of each position such that the sequential and local information within its neighborhood can be compressed in the representation. To do this, we introduce local recurrent neural network, referred to as LocalRNN, to process signals within a local window ending at given position. In addition, the LocaIRNN operates on local windows of all the positions identically and independently and produces latent representation for each of them. In this way, the locality the sequence explicitly captured. In addition, as the local window is sliding along the sequence position by position, the global sequential information is also incorporated. More importantly, because the localRNN is only ap- plied to local windows, the aforementioned two drawbacks of R.NSs can be naturally mitigated. We evaluate the effectiveness of R- Traanfformer with a various of sequence learning tasks from different domains and the empirical results demonstrate that R- R-Transformer achieves much stronger perfor- mance than both TCN and standard Transformer as well as other stte-offthhe-a sequence models.
The rest of the paper is organized follows: Section 2 discusses the sequence modeling problem we aim to solve: The proposed R- R-Transformer model is presented in Section 3. In Section 4, we describe the experimental details and discuss the results. The related work briefly reviewed in Section 5. Section 6 concludes this work.
SEQUENCE MODELING PROBLEM
Before introducing the proposed R- T-Tranfformer model, we formally describe the sequence modeling problem. Given sequence of length N: 21, 22, UN, aim learn function that maps the
2

---

Under review as a conference paper at ICLR 2020
input sequence into a label space y: (f: XN - v).. Formally,
y 22, ,2N)
where y E y' is the label of the input sequence. Depending on the definition of label y. many tasks can be formatted as the sequence modeling problem defined above. For example, in language modeling task, 2y is the character/word in a textual sentence and y is the character/word next position (Mikolov et al.) 2010); in session- based recommendation, in session and y is the future item that users will interact with (Hidasi et al. 2015); when I; is a nucleotide in DNA sequence and y is its function, this problem becomes a DNA function prediction task (Quang & Xie 2016). Note that, in this paper, we do not consider the sequence-to ceennne-to-sqqueee -o-eueeneee learning problems. However, the proposed model can be easily extended to solve these problems and we will leave it as one future work.
is the user-item interaction
3 THE R- R-TRANSFORMER MODEL
The proposed R-Transformer consists of a stack of identical layers. Each layer has 3 components that are organized hierarchically and the architecture of the layer structure is shown in Figure D As shown the figure, the lower level is the local recurrent neural networks that are designed to model local structures in sequence; the middle level is a multi-head attention that is able to capture global long-term dependencies; and the upper level is position-wist feedforward networks which conducts non-linear feature transformation. Next, we describe each level in detail.
3.1 LOCALRNN:: MODELING LOCAL STRUCTURES
Sequential data such as natural language inherently exhibits strong local structures. Thus, it is desir- able and necessary to design components to model such locality. In this subsection, we propose to take the advantage of RNNs to achieve this. Unlike previous works where RNNs are often applied to the whole sequence, we instead reorganize the original long sequence into many short sequences which only contain local information and are processed by shared RNN independently and identi- cally. In particular, we construct a local window of size M for each target position such that the local window includes M consecutive positions and ends at the target position. Thus, positions in each local window form local short sequence, from which the shared RNN will learn a latent represen- tation. In this way. the local structure information of each local region of the sequence is explicitly incorporated in the learned latent representations. We refer to the shared RNN as LocalRNN. Com- paring to original RNN operation, LocaIRNN only focuses on local short-term dependencies without considering any long-term dependencies. Figure 2 LocaIRNN operations. Concretely, given the positions zt- M+1, It M42, ... 3y. of a local short sequence of length M, the LocalRNN processes them sequentially and outputs M hidden states, the last of which is used as the representation of the local short sequences:
shows the different between original RNN and
h LocaIRNN(Z
.2e)
(2)
where RNN denotes any RNN cell such as Vanilla RNN cell, LSTM, GRU, etc. To enable the model to process the sequence in an auto-regressive manner and take care that no future information is available when processing one position, pad the input sequence by (M - 1) positions before the start of a sequence. Thus, from sequence perspective, the LocaIRNN takes an input sequence and outputs sequence of hidden representations that incorporate information of local regions:
h1 h1, ,,y Local N (21, 22, (NN)
(3)
The localRNN is analogous 1-D Convolution Neural Networks where each local window is pro- cessed by convolution operations. However, the convolution operation completely ignores the se- quential information of positions within the local window. Although the position embeddings have been proposed to mitigate this problem, a major deficiency of this approach is that the effective- ness of the position embedding could be limited; thus it requires considerable amount of extra ef- forts (Gehring et al. 2017). On the other hand, the LocalRNN is able fully capture the sequential information within each window. In addition, the by--by-one sliding operation also naturally incor- porates the global sequential information.
3

---

Under review as a conference paper at ICLR 2020
R R
R
R
(a) Original RNN.
(b) Local RNN.
Figure 2: An illustration of the original and local RNN. contrast to orignal RNN which maintains a hidden state at each position summarizing all the information seen so far, LocalRNN only operates on positions within local window. At each position, LocalRNN will produce hidden state that represents the information in the local window ending at that position.
Discussion: RNNs have long been dominating choice for sequence modeling but it severely suffers from two problems The first one its limited ability to capture the long-term dependencies and the second one is the time complexity, which is linear the sequence length. However, in LocalRNN, these problems are naturally mitigated. Because the LocalRNN is applied to a short sequence within local window of fixed size, where no long-term dependency is needed capture. In addition, the computation procedures for processing the short sequences are independent of each other. Therefore, very straightforward for the parallel implementation (e.g., using GPUs), which can greatly improve the computation efficiency.
3.2 CAPTURING THE GLOBAL LONG-TERN DEPENDENCIES WITH MULTI-HEAD ATTENTION
The RNNs the lower level introduced the previous subsection will refine representation each positions such that it incorporates its local information. In this subsection, we build sub-layer on top of the LocalRNN capture the global long-term dependencies. We term as pooling sub- layer because it functions similarly to the pooling operation in CNNs. Recent works have shown that the multi-head attention mechanism is extremely effective to learn the long-term dependencies, as it allows direct connection between every pair of positions. More specifically, in the multi- head attention mechanism, each position will attend to all the positions in the past and obtains set of attention scores that used to refine its representation. Mathematically. given current representations h1 h2, .. h,, the refined new representations u, are calculated as:
.h4) W1 Concatenation(head (ha), headhhaaahhh ... heady W
where heady (h,) is the result of 4t attention pooling and T is linear projection matrix. Con- sidering both efficiency and effectiveness, the scaled dot product is used as the attention func- tion (Vaswani et al. 2017). Specifically, head:(h,) is the weighted of all value vectors and
4

---

Under review as a conference paper at ICLR 2020
the weights are calculated by applying attention function all the query, key pairs:
{o1, 02, ... an Softmar(( < 9 > < <q,
< q, hin >
(5)
V(ds) v(dx)
v(dx)
head:(hi) Σ a5 j=1
where q, k;, and v2 the query, key, and value vectors and dx is the dimension of kg More- over, q. k, and ie are obtained by projecting the input vectors into query, key and value spaces, respectively (Vaswani al. 2017). They are formally defined as:
q, kg 2 Whh,, W Whh
(6)
where W W and W are the projection matrices and each attention pooling head, has its own projection matrices. As shown in Eq. (5) each head, is obtained by letting hú attending to all the "past" positions, thus any long-term dependencies between ht and hı can be captured. In addition, different heads will focus on dependencies in different aspects. After obtaining the refined rep- resentation of each position by the multi-head attention mechanism, we add a position-wise fully connected feed-forward network sub-layer, which is applied to each position independently and identically. This feedforward network transforms the features non- non-linearly and is defined as follows:
FeedF Freadorwwrddmm) = maz(0, u, W1 + b1) + 62
(7)
Following (Vaswani et al. 2017), We add residual (He al. 2016 and layernorm (Ba al. connection between all the sub-layers.
2016)
3.3 OVERALL ARCHITECTURE OF R- R-TRANSFORMER
With all the aforementioned model components, we can now give a formal description of the overall architecture of an N l-layer R-Transformer. For the ith layer (i 21,2, ... ::::::::
Mi h2 M hg.
h4 RNN o2 ..,)))
(8)
Ri LayerNorm(hh + ot, h1 + x22 (hi, hy. (/))
11 + xt)
u mz.
ui Layer + híg u/ + K2g ... u/ + lir))
mi, my
my = FeedForrd orward( uz (iT)
x1 222
++1 4+1
Layer Norm(( + of m² + ūi..... m (fr))
where T is the length of the input sequence and x² is the input position of the layer at time step
Comparing with TCN: R- Traanffommer is partly motivated by the hierarchical structure in TCN Bai al. (018). thus, make detailed comparison here. In TCN, the locality in sequences in captured by convolution filters. However, the sequential information within each receptive field ignored by convolution operations. In contrast, the LocaIRNN structure in R-Transformer can fully incorporate it by the sequential nature of RNNs. For modeling global long-term dependencies, TCN achieves with dilated convolutions that operate on nonconsecutive positions. Although such operation leads to larger receptive fields in lower-level layers, it misses considerable amount of information from a large portion of positions in each layer. On the other hand, the multi-head attention pooling in R-Transformer considers every past positions and takes much more information into consideration than TCN.
The proposed R-Transformer and standard Transformer enjoys similar long-term memorization capacities thanks to the multi-head attention mechanism (Vaswani et al. 2017). Nevertheless, two important features distinguish R-Transformer from the standard Transformer. First, R- Transformer explicitly and effectively captures the locality in sequences with the novel LocalRNN structure while standard Transformer models it very vaguely with multi-head attention that operates on all of the positions. Second, R- T-Trnsfforme does not rely on any position embeddings Transformer does. In fact, the benefits of simple position embeddings very
Comparing with Transformer:
5

---

Under review as a conference paper at ICLR 2020
Table 1: MNIST classification task results. Italic numbers denote that the results directly copied from other papers that have the same settings.
Model
# of layers / hidden size
Test Accuracy(%)
RNN
Bai ct al.
2018
21.5
GRU
Bai et al. 2018
96.2
LSTM
Bai et al. 2018
11 130
87.2
TCN
Bai et al.
2018)
8 /25
99.0
Transormer
8/32
98.2
R- Transformer
8/32
99.1
limited (AI-Rfou et al., 2018) and it requires considerable amount of efforts to design effective position embeddings as well as proper ways to incorporate them (Dai et al.] (019).. In the next section, we will empirically demonstrate the advantages of R-Transformer over both TCN and the standard Transformer.
EXPERIMENT
Since the R- R-Transformer is a general sequential learning framework, we evaluate it with sequential data from various domains including images, audios and natural languages. We mainly compare with canonical recurrent architectures (Vanilla RNN, GRU, LSTM) and two of the most popular generic sequence models that do not have any recurrent structures, namely, TCN and Transformer. However, since the majority of existing efforts to enhance Transformer are for natural languages, in the natural language evaluation, also include recent advanced Transformer, i.e., Transformer- XL. For all the tasks, Transformer and R- Transformer were implemented with Pytorch and the re- sults for canonical recurrent architectures and TCN were directly copied from Bai et al. (2018) as we follow the same experimental settings. In addition, to make the comparison fair, we use the same of hyperparameters (i.e, hidden size, number of layers, number of heads) for R-Transformer and Transformer. Moreover, unless specified otherwise, for training, all models are trained with same optimizer and learning rate is chosen from the same of values according to validation perfor- mance. In addition, the learning rate annealed such that it is reduced when validation performance reaches plateau.
4.1 PIXEL-BY-PIXEL MNIST: SEQUENCE CLASSIFICATION
This task is designed to test model ability to memorize long-term dependencies. It was firstly pro- posed by Le et al. (2015) and has been used by many previous works (Wisdom et al. 2016 Chang et al. 2017] Zhang et al. 2016; Krueger et al. 2016). Following previous settings, we rescale each 28 x 28 image in MNIST dataset
al. 19998 into 784 X sequence, which will be clas- sified into ten categories (each image corresponds to one of the digits from 0 to 9) by the sequence models. Since the rescaling could make pixels that are connected in the origin images far apart from each other, it requires the sequence models to learn very long-term dependencies to understand the content of each sequence. The dataset is split into training and testing sets as same as the default ones in Pytorch(version 1.0.0) 2 The model hyperparameters and classification accuracy are re- ported in Table I From the table, it can be observed that firstly, RNNs based methods generally perform worse than others. This is because the input sequences exhibit very long-term dependen- cies and is extremely difficult for RNNs to memorize them. On the other hand, methods that build direct connections among positions, i.e., Transformer, TCN, achieve much better results. It is also stan- interesting to see that TCN slightly better than Transformer, we argue that this is because dard Transformer cannot model the locality very well. However, proposed R- Transformer that leverages LocalRNN to incorporate local information, has achieved better performance than TCN.
hhupss//pyorcch.org

---

Under review as a conference paper at ICLR 2020
Table 2: Polyphonic music modeling. Italic numbers denote that the results directly copied from other papers that have the same settings.
Model
# of layers/ hidden size
NLL
RNN
Bai ct al.
2018
4.05
GRU
Bai et al.
2018
3.46
LSTM
Bai et al.
2018
3.29
TCN
Bai al.
2018)
4/150
3.07
Transtormer
3/160
3.34
R- Transformer
3/160
2.37
Table 3: Character-level language modeling. Italic numbers denote that the results are directly copied from other papers that have the same settings.
Model
of layers hidden size
bpc
RNN Bai et al.
2018
1.48
GRU
Bai et al.
2018
1.37
LSTM
Bai al.
2018
2 1600
1.36
TCN
Bar et al.
2018)
1450
1.31
Transtormer
3/512
1.34
R- R-Transformer
3/512
1.24
4.2 NOTTINGHAM: POLYPHONIC MUSIC MODELING
Next, we evaluate R-Transformer on the task of polyphonic music modeling with Nottingham dataset (Boullnger-Lewwaddoww)
al. 2012). This dataset collects British and American folk tunes and has been commonly used in previous works to investigate the model's 's ability for polyphonic mu- et al., 2212) Chung al. 2014 Bai al. 2018). Following the same setting in Bai al. (2018), we split the data into training, validation, and testing sets which contains 694, 173 and 170 tunes, respectively. The learning rate is chosen from (,e-4, 5e-5, 5,, -5 and dropout with probability of 0.1 is used to avoid overfitting. Moreover, gradient clipping is used during the training process. We choose negative log-likelihood (NLL) as the evaluation metrics and lower value indicates better performance. The experimental results are shown in Table Both LTSM and TCN outperform Transformer in this task. We suspect this because these music tunes exhibit strong local structures. While Transformer is equipped with multi-head attention mechanism that is effective to capture long-term dependencies, it fails to capture local structures in sequences that could provide strong signals. On the other hand, R-Transformer enhanced by LocalRNN has achieved much better results than Transformer, In addition. it also outperforms TCN by a large mar- gin. This is expected because TCN tends ignore the sequential information in the local structure, which can play an important role as suggested by (Gehring et al. 2017).
sic modeling Booulange-LLwandowwe
4.3 PENNTREEBANK: LANGUAGE MODELING
In this subsection, we further evaluate R- R-Transformer's ability on both character-level and word- level language modeling tasks. The dataset we use is PennTreebank(PPTB (Marcus et al. 1993) that contains million words and has been extensively used by previous works to investigate sequence models (Chen & Goodman, 1999) Chelba & Jelinek 2000 Kim et al. 2016 Tran et al. 2016). For character-level language modeling task, the model is required to predict the next character given a context. Following the experimental settings in Bai et al. (2018), split the dataset into train-

---

Under review as a conference paper at ICLR 2020
ing, validation and testing sets that contains 5059K, 396K and 446K characters, respectively. For Transformer and R- R-Transformer, the learning rate is chosen from (1, 2, 3} and dropout rate is 0.15. Gradient clipping is also used during the training process. The bpe is used to measure the predicting performance.
For word-level language modeling, the models are required predict the next word given the textual words. Similarly, we follow previous works and split PTB into training, validation, and testing sets with 888K, 70K and 79K words, respectively. The vocabulary size of PTB 10K. As with character-level language modeling,the the learning rate is chosen from {1, 2, 3} for Transformer and R-Transformer and dropout rate 0.35. In this task, we also add Transformer-X XL (Dai et al, 2019) as one baseline, which has been particularly designed for language modeling tasks and has achieved state-of-the-art performance. Note that to make the comparison fair, we apply the same model configuration, i.e., number of layers, to Transformer-Xl. All other settings such as optimizer are the same as its original ones. The learning rate chosen from (0.01, 0.001, 0.0001} } and its best validation performance is achieved with 0.001. Note that, except dropout, other regularization tricks such as variational dropout and weight dropout are applied. The prediction performance is evaluated with perplexity, the lower value of which denotes better performance.
The experimental results of character-level and word-level language modeling tasks are shown in TableBaan and Table respectively. Several observations can be made from the Table First, Trans- former performs only slightly better than RNNs while much worse than other models. 'he reason for this observation is similar the case of polyphonic music modeling task that language hibits strong local structures and standard Transformer can not fully capture them. Second, TCN achieves better results than all of the RNNs, which is attributed to its ability capture both local structures and long-term dependencies in languages. Notably, for both local structures and long- term dependencies, R- R-Transformer has more powerful components than TCN, i.e., LocaIRNN and Multi head attention. Therefore, it is not surprising to see that R- Transformer achieves significantly better results. Table 3 presents the results for word-level language modeling. Similar trends are observed, with the only exception that L.TTM achieves the best results among all the methods. In addition, the result of Transformer-XL. is only slightly better than R-transformer. Considering the fact that Transformer-Xee is specifically designed for language modeling and employs the recurrent connection of segments (Dai et al. 2019). this result suggests the limited contribution of engineered positional embeddings.
4.4 DISCUSSIONS AND EVALUATION LIMITATIONS
In summary, experimental results have shown that the standard Transformer can achieve better sults than RNNs when sequences exhibit very long-term dependencies, i.e., sequential MNIST while its performance can drop dramatically when strong locality exists in sequences, i.e., polyphonic mu- sic and language. Meanwhile, TCN is a very strong sequence model that can effectively learn both local structures and Long-term dependencies and has very stable performance in different tasks. More importantly. the proposed R- T-Transformer that combines a lower level LocalRNN and higher level multi-head attention, outperforms both TCN and Transformer by large margin consistently in most of the tasks. The experiments are conducted on various sequential learning tasks with datasets from different domains. Moreover, all experimental settings are fair to all baselines. Thus, the observa- tions from the experiments are reliable with the current experimental settings. However, due to the computational limitations, we are currently restricted our evaluation settings to moderate model and dataset sizes. Thus, more evaluations on big models and large datasets can make the results more convincing. We would like to leave this as one future work.
5 RELATED WORK
Recurrent Neural Networks including its variants such LSTM (Hochreiter & Schmidhuber] 1997) and GRU (Cho et al.) 20144) have long been the default choices for generic sequence modeling. A RNN sequentially processes each position in a sequence and maintains internal hidden state to compresses information of positions that have been seen. While its design is appealing and has been successfully applied in various tasks, several problems caused by its recursive structures including low computation efficiency and gradient exploding or vanishing make it ineffective when learning long sequences. Therefore, in recent years, lot of efforts has been made develop models
8

---

Under review as a conference paper at ICLR 2020
Table 4: Word-level language modeling. Italic numbers denote that the results are directly copied from other papers that have the same settings.
Model
# of layers / hidden size
Perplexity
RNN Bai ct al.
2018
114.50
GRU
Bai et al.
2018
92.48
LSTM
Bai et al.
2018
3 1700
78.93
TCN
Bai et al.
2018)
4 /600
88.68
Transtormer
3/128
122.37
Transformer XL R- R-Transformer
3/128
83,38
3/128
84.38
without recursive structures and they be roughly divided into two categories depending whether they rely on convolutions operations or not.
The first category includes models that mainly built convolution operations. For example, van den Oord et al. have designed an autoregressive WaveNet that is based on causal filters and dilated convolution capture both global and local information in raw audios (Van Den Oord al. 2016). Ghring et al. has successfully replace traditional RNN based encoder and decoder with convolu- tional ones and outperforms LSTM setup in neural machine translation tasks (Gehring et al. 2017 2016). Moreover, researchers introduced gate mechanism into convolutions structures model se- quential dependencies languages (Dauphin al., 2017). Most recently, a generic architecture for sequence modeling, termed Temporal Convolutional Networks (TCN), that combines compo- nents from previous works has been proposed in (Bai et al., 2018). Authors in (Bai et al. 2018) have systematically compared TCN with canonical recurrent networks in a wide range of tasks and TCN able achieve better performance in most cases. Our R-transformer is motivated by works this group in sense that firstly models local information and then focus global ones.
The most popular works in second category are those based on multi- head attention mechanism. The multi-head attention mechanism firstly proposed in Vaswani performance in machine translation task has been achieved with Transformer. It was then frequently used in other sequence learning models (Devlin et al.] 2018 Dehghani al.) 2018) Dai al. 2019). The success of multi-head attention largely comes from its ability to learn long-term dependencies through direct connections between any pair of positions. However, it heavily relies on position embeddings that have limited effects and require fair amount of effort to design effective ones. In addition, our empirical results shown that the local information could easily be ignored by multi-head attention even with the existence of position embeddings. Unlike previously proposed Transformer-lik models, R-Transformer in this work leverages the strength RNN and is able model the local structures effectively without the need of any position embeddings.
(2017), where impressive
CONCLUSION
In this paper, we propose a novel generic sequence model that enjoys the advantages of both RNN and the multi-head attention while mitigating their disadvantages. Specifically, it consists of a Lo- caIRNN that learns the local structures without suffering from any of the weaknesses of RNN and a multi-head attention pooling that effectively captures long-term dependencies without any help of position embeddings. In addition, the model can be easily implemented with full parallelization over the positions in a sequence. The empirical results on sequence modeling tasks from a wide range of domains have demonstrated the remarkable advantages of R- Transformer over sate-offthe-as non- recurrent sequence models such as TCN and standard Transformer as well as canonical recurrent architectures.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level lan- guage modeling with deeper self-attention. arXiv preprint arXiv: 04444, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint /83.012/0 2018.
Nicolas Boollanger-ewandowwke Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. XrXiv preprint arXiv: 6392, 2012.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 2017.
Ciprian Chelba and Frederick Jelinek. Structured language modeling. Computer Speech & Lan- guage, 14(4):283-332, 2000.
Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359-394, 1999.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint / 000260, 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, In Proceedings of the 34th International Conference on Machine Eearning-Volumme 70, pp. 933-941. JMLR. org, 2017.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv: 2018
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805 2018.
Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. A convolutional encoder model for neural machine translation. arXiv preprint
1.02344, 2016.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Vooie Volume 70, pp. 1252- JMLR. org, 2017.
Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. International conference on machine learning, pp. 1764-1772, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Cession-based rec- ommendations with recurrent neural networks. arXiv preprint arXiv: 2015.
10

---

Under review as a conference paper at ICLR 2020
Geoffrey Hinton, Li Deng. Dong Yu, George Dahl, Abdel-rahman Mohamed. Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
David Krueger, Tegan Maharaj. János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing mns by randomly preserving hidden activations. arXiv preprint
2016.
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks 2015.
of rectified linear units. arXiv preprint
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (((11)2727-2324 1998.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocky, and Sanjeev Khudanpur. Recurrent neural network based language model. Eleventh annual conference of the international speech communication association, 2010.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning. pp. 1310-1318, 2013.
Daniel Quang and Xiaohui Xie. Danq: a hybrid convolutional and recurrent deep neural network for quantifying the function dna sequences. Nucleic acids research, 44(11):e17-e117 2016.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language mod- eling. In Thirteenth annual conference of the international speech communication association, 2012.
Ke Tran, Arianna Bisazza, and Christof Monz. Recurrent memory networks for language modeling. arXiv preprint 1rXiv:1601.0122. 2016.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 125, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances Neural Information Processing Systems, pp- 4880-4888, 2016.
Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan R Salakhutdinov, and Yoshua Bengio. Architectural complexity measures of recurrent neural networks. In Advances in neural information processing systems, pp. 1822-188 1830, 2016.
11


Reviews:
[{"review_id": "r1gU10tB_H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a new architecture, R-Transformer, that blends the Transformer networks and the recurrent networks, so as to better capture both the long- and short-term features. By injecting a local RNN layer at every level of the network, the authors hoped to enhance the Transformer's ability to model locality structure. To demonstrate the modeling power of R-Trasnformer, the paper evaluates the effectiveness of R-Transformer on 4 different sequence tasks (seqMNIST, polyphonic music, character- and word-level PTB).\n\nContribution: The authors propose an architecture that combines the practices of recurrent and feed-forward sequence models. However, I have major concerns regarding the novelty this paper, the various claims it makes, as well as its experiment setting.\n\n----------------------------------------------------------------------------\n\nMajor issues/questions:\n\n1. The techniques proposed by this paper lack novelty. For instance, the entire section 3.2 is simply the original design of the multi-head self-attention by Vaswani et al. The major difference between R-Transformer and the original Transformer is the replacement of positional embedding with an RNN layer, but (in my opinion) the authors did not demonstrate sufficiently its effectiveness via ablative studies (see below). Moreover, some prior works have already exploited the locality structure in Transformers. For instance, [1] showed that a sparse, local Transformer can work extremely well and be very efficient (they achieved SOTA on large-scale char-level language modeling tasks).\n\n2. The experiments do not entirely convince me. \n    i) The authors use the \"same hidden size for R-Transformer and Transformer.\" But in fact, as the R-Transformer has one extra RNN/LSTM/GRU layer at every level of the network, the tests were carried out (in effect) using a larger model than the baselines. I think the authors should instead control the # of model parameters, especially since you are running only on small tasks with small-sized models.\n    ii) It is nice that the authors tested R-Transformer on a variety of tasks--- this is important. However, in no way do these number achieve the levels of the \"state-of-the-art\", which the authors claim at the end of Section 1 (e.g., [2] has better number on seqMNIST and character-level PTB, and the Transformer-XL actually achieves <55 perplexity on word-level PTB). Therefore, the numbers don't look particularly appealing to me. \n    iii) Lack of more challenging, or large-scale experiments. Sequential MNIST is known to be a relatively simple task, and alternatives such as sequential CIFAR-10 or permuted sequential MNIST are more valuable \"small sequence tasks.\" (e.g., prior works studying long-term sequence model dependencies [3]) I also think that benchmarking R-Transformer on large-scale tasks like WikiText-103 or 1Bword (which prior works like QRNN, TrellisNet, RMC and Transformer-XL all explored) would be much more indicative of its usefulness (and should not be left out in such a paper). For instance, it would be useful to compare on small models with controlled model size, even if larger models that require TPUs are not available.\n    iv) Lack of ablative study. Table 4 seems to suggest that Transformer-XL is better than R-Transformer. Is it because of their usage of the relative positional embedding? How does the \"finite window size\" affect the performance of R-Transformer? Why don't you use Transformer-XL for the other 3 tasks? Lots of interesting questions are unanswered here. More details in (3) below.\n\n3. Regarding the motivation to insert a local RNN at the start of each layer. The authors claim (by citing Al-Rfou et al.) that the positional embedding (PE) have only limited effect. But in fact, Al-Rfou only says that the PE features (which is added only at the first layer, by the design of Vaswani et al.) can get lost once the transformer gets very deep--- which is totally expected. Therefore, Al-Rfou et al. propose to use a learnable embedding for each layer. This in no way suggests that positional embedding has \"limited effect\" (Note that in their ablative study, when they turned off the learnable PE, they also added back the original PE). Moreover, the authors didn't use convolution to capture local data, because it \"completely ignores the sequential information of positions within the local window\". That is true. However, **stacking** convolutional layers does capture sequential information. Prior works like [2] showed exactly how temporal convolutions are related to finite-window RNNs. I would suggest the authors to at least compare these different options with ablative studies. I would at least expect a comparison of i) temporal convolution; 2) fixed-window RNN (LocalRNN); 3) unlimited-window RNN; 4) positional embedding; 5) relative PE (which Transformer-XL uses; is that why it's better in Table 4?).\n\n4. The authors claim that the finite-window RNN captures local features. But doesn't that claim only applies to the first layer? Once the first layer multi-head attention mixes all input elements across the sequence, the \"local\" features fed into the second layer RNN will be, actually, **global** features? Doesn't that \"defeat\" the purpose of using a local RNN though?\n\n============================\n\nMinor issues that have mild or zero impact on the score:\n\n5. Inconsistent notations. Equation (2) and (3) both describe the LocalRNN(...) function, but clearly have a different input-output signature. Instead of using italics, it's better to have well-defined notations. Another case is Equation (7): you have 'FeedForward(mt)', but 'mt' is not on the right-hand side of the equation at all. Moreover, the symbols used in Eq. (7) are currently inconsistent with Eq. (8). \n\n6. Show the # of parameters in the model in Table 1-4.\n\n7. According to the code released in the dropbox URL, you precompute the index in the finite window and later called 'torch.index_select' to produce a tensor that is 'ksize' larger than the original sequence (cf. Line 130-143 in models/RTransformers.py in the dropbox folder). How does R-Transformer compare to Transformers in terms of speed and memory? Since you convert the batch dimension to (batch_size * seq_len) at every level of the network, I imagine that could slow down the process, especially in high-dimensional/large-scale experiments?\n\n8. There are some typos (e.g., Sec. 4.2, LTSM) and grammatical mistakes in the paper (Sec. 3.3). \n\n9. In Section 4.2, the authors claim that both LSTM and TCN performed better than Transformers on the polyphonic music dataset because \"these music tunes exhibit strong local structures\". However, the difference between Transformer and LSTM is actually very small, and it's even better than GRU/vanilla RNN. Is that too big a claim to make? In polyphonic music datasets like JSB or Nottingham, there are still longer sequences with longer dependencies...\n\n============================\n\nOverall, I feel that this paper has a good motivation to combine different sequence model families (Transformers, TCNs, RNNs) to improve their modeling power. But at the same time, I feel that the experiments can be a lot stronger, and the paper has limited novelty when compared to prior works. I'm happy to consider adjusting my score if my concerns above are addressed. \n\n[1] \"Generating Long Sequences with Sparse Transformers\", https://arxiv.org/abs/1904.10509\n[2] \"Trellis Networks for Sequence Modeling\", https://arxiv.org/abs/1810.06682\n[3] \"Learning Longer-term Dependencies in RNNs with Auxiliary Losses\", https://arxiv.org/abs/1803.00144", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1lr66X6KB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the R-Transformer architecture which adds a local RNN layer before each attention layer in Transformer. The authors claim state-of-the-art performance but only test on tiny tasks where Transformer models have not been heavily optimized and omit the main problem with RNNs - namely their speed. It is an interesting paper still and the locality is a nice way to remedy the speed problem, but the paper lacks a true study and ablations on this main limitation. In summary: the main new idea of the paper is to make RNNs local in Transformer (trying to add RNN layers has been explored before). This idea could be a good tradeoff between full RNN (slow) and no RNN (lack of context), but the following is missing: (1) ablations on speed vs results by locality window, (2) experiments on more widely reported and larger data-sets and models, at least including some language modeling task (wiki or lm1b) and some translation task (like en-de). Without these results, we cannot recommend to accept this paper.\n\nI'm grateful to the authors for their reply. Presently, a very good LM1B or WMT model can be trained for free in Google colab in under a day, so I do not believe it's computationally infeasable to run the experiments I asked for. Even if it took much longer, I'd believe that the time should be invested before acceptance, so I stand by my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkliykIptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel transformer model called R-Transformer. \nBased on the observation that positional embeddings require a \na lot of design efforts in vanilla Transformer, the authors propose to use local RNNs to\nencode local information in replace of positional embeddings.\n\nThe paper is well presented and the proposed algorithm is explained in detail. \nHowever, it is not clear how the proposed model obtains positional information of input nodes. \nIn vanilla Transformer, positional embeddings contains global positional information. \nIn the R-Transformer, however, the multi-head attention layer will not be able to obtain positional information from local RNN outputs. Will such loss in positional information affect the performance?\n\nThe empirical study shows that R-Transformer can outperform vanilla Transformers and recurrent architectures, which is promising. Still, it would be more convincing if the authors could provide comparisons on NMT tasks or larger language modeling datasets such as WikText. Also, I am interested in the training efficiency of these models. How much overhead does the local RNN introduce?\n\nOverall I think this is an interesting paper but experiments could be improved. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a variant of a Transformer architecture that does not use positional embeddings to model local structural patterns but instead adds a recurrent layer before each attention layer to maintain local context. The approach is empirically verified on a number of domains.

The reviewers had concerns with the paper, most notably that the architectural modification is not sufficiently novel or significant to warrant publication, that appropriate ablations and baselines were not done to convincingly show the benefit of the approach, that the speed tradeoff was not adequately discussed, and that the results were not compared to actual SOTA results.

For these reasons, the recommendation is to reject the paper.
Decision: Reject