R-TRANSFORMER: RECURRENT NEURAL NETWORK ENHANCED TRANSFORMER

1 INTRODUCTION . Recurrent Neural Networks ( RNNs ) especially its variants such as Long Short-Term Memory ( LSTM ) and Gated Recurrent Unit ( GRU ) have achieved great success in a wide range of sequence learning tasks including language modeling , speech recognition , recommendation , etc ( Mikolov et al. , 2010 ; Sundermeyer et al. , 2012 ; Graves & Jaitly , 2014 ; Hinton et al. , 2012 ; Hidasi et al. , 2015 ) . Despite their success , however , the recurrent structure is often troubled by two notorious issues . First , it easily suffers from gradient vanishing and exploding problems , which largely limits their ability to learn very long-term dependencies ( Pascanu et al. , 2013 ) . Second , the sequential nature of both forward and backward passes makes it extremely difficult , if not impossible , to parallelize the computation , which dramatically increases the time complexity in both training and testing procedure . Therefore , many recently developed sequence learning models have completely jettisoned the recurrent structure and only rely on convolution operation or attention mechanism that are easy to parallelize and allow the information flow at an arbitrary length . Two representative models that have drawn great attention are Temporal Convolution Networks ( TCN ) ( Bai et al. , 2018 ) and Transformer ( Vaswani et al. , 2017 ) . In a variety of sequence learning tasks , they have demonstrated comparable or even better performance than that of RNNs ( Gehring et al. , 2017 ; Bai et al. , 2018 ; Devlin et al. , 2018 ) . The remarkable performance achieved by such models largely comes from their ability to capture long-term dependencies in sequences . In particular , the multi-head attention mechanism in Transformer allows every position to be directly connected to any other positions in a sequence . Thus , the information can flow across positions without any intermediate loss . Nevertheless , there are two issues that can harm the effectiveness of multi-head attention mechanism for sequence learning . The first comes from the loss of sequential information of positions as it treats every position identically . To mitigate this problem , Transformer introduces position embeddings , whose effects , 1https : //www.dropbox.com/sh/u35qgqnmjpywcqn/AAAITcId7DRPOD9KRooQW7i2a ? dl=0 however , have been shown to be limited ( Dehghani et al. , 2018 ; Al-Rfou et al. , 2018 ) . In addition , it requires considerable amount of efforts to design more effective position embeddings or different ways to incorporate them in the learning process ( Dai et al. , 2019 ) . Second , while multi-head attention mechanism is able to learn the global dependencies , we argue that it ignores the local structures that are inherently important in sequences such as natural languages . Even with the help of position embeddings , the signals at local positions can still be very weak as the number of other positions is significantly more . To address the aforementioned limitations of the standard Transformer , in this paper , we propose a novel sequence learning model , termed as R-Transformer . It is a multi-layer architecture built on RNNs and the standard Transformer , and enjoys the advantages of both worlds while naturally avoids their respective drawbacks . More specifically , before computing global dependencies of positions with the multi-head attention mechanism , we firstly refine the representation of each position such that the sequential and local information within its neighborhood can be compressed in the representation . To do this , we introduce a local recurrent neural network , referred to as LocalRNN , to process signals within a local window ending at a given position . In addition , the LocalRNN operates on local windows of all the positions identically and independently and produces a latent representation for each of them . In this way , the locality in the sequence is explicitly captured . In addition , as the local window is sliding along the sequence one position by one position , the global sequential information is also incorporated . More importantly , because the localRNN is only applied to local windows , the aforementioned two drawbacks of RNNs can be naturally mitigated . We evaluate the effectiveness of R-Transformer with a various of sequence learning tasks from different domains and the empirical results demonstrate that R-Transformer achieves much stronger performance than both TCN and standard Transformer as well as other state-of-the-art sequence models . The rest of the paper is organized as follows : Section 2 discusses the sequence modeling problem we aim to solve ; The proposed R-Transformer model is presented in Section 3 . In Section 4 , we describe the experimental details and discuss the results . The related work is briefly reviewed in Section 5 . Section 6 concludes this work . 2 SEQUENCE MODELING PROBLEM . Before introducing the proposed R-Transformer model , we formally describe the sequence modeling problem . Given a sequence of length N : x1 , x2 , · · · , xN , we aim to learn a function that maps the input sequence into a label space Y : ( f : XN → Y ) . Formally , y = f ( x1 , x2 , · · · , xN ) ( 1 ) where y ∈ Y is the label of the input sequence . Depending on the definition of label y , many tasks can be formatted as the sequence modeling problem defined above . For example , in language modeling task , xt is the character/word in a textual sentence and y is the character/word at next position ( Mikolov et al. , 2010 ) ; in session-based recommendation , xt is the user-item interaction in a session and y is the future item that users will interact with ( Hidasi et al. , 2015 ) ; when xt is a nucleotide in a DNA sequence and y is its function , this problem becomes a DNA function prediction task ( Quang & Xie , 2016 ) . Note that , in this paper , we do not consider the sequence-to-sequence learning problems . However , the proposed model can be easily extended to solve these problems and we will leave it as one future work . 3 THE R-TRANSFORMER MODEL . The proposed R-Transformer consists of a stack of identical layers . Each layer has 3 components that are organized hierarchically and the architecture of the layer structure is shown in Figure 1 . As shown in the figure , the lower level is the local recurrent neural networks that are designed to model local structures in a sequence ; the middle level is a multi-head attention that is able to capture global long-term dependencies ; and the upper level is a position-wise feedforward networks which conducts a non-linear feature transformation . Next , we describe each level in detail . 3.1 LOCALRNN : MODELING LOCAL STRUCTURES . Sequential data such as natural language inherently exhibits strong local structures . Thus , it is desirable and necessary to design components to model such locality . In this subsection , we propose to take the advantage of RNNs to achieve this . Unlike previous works where RNNs are often applied to the whole sequence , we instead reorganize the original long sequence into many short sequences which only contain local information and are processed by a shared RNN independently and identically . In particular , we construct a local window of sizeM for each target position such that the local window includes M consecutive positions and ends at the target position . Thus , positions in each local window form a local short sequence , from which the shared RNN will learn a latent representation . In this way , the local structure information of each local region of the sequence is explicitly incorporated in the learned latent representations . We refer to the shared RNN as LocalRNN . Comparing to original RNN operation , LocalRNN only focuses on local short-term dependencies without considering any long-term dependencies . Figure 2 shows the different between original RNN and LocalRNN operations . Concretely , given the positions xt−M+1 , xt−M+2 , · · · , xt of a local short sequence of length M , the LocalRNN processes them sequentially and outputs M hidden states , the last of which is used as the representation of the local short sequences : ht = LocalRNN ( xt−M+1 , xt−M+2 , · · · , xt ) ( 2 ) where RNN denotes any RNN cell such as Vanilla RNN cell , LSTM , GRU , etc . To enable the model to process the sequence in an auto-regressive manner and take care that no future information is available when processing one position , we pad the input sequence by ( M − 1 ) positions before the start of a sequence . Thus , from sequence perspective , the LocalRNN takes an input sequence and outputs a sequence of hidden representations that incorporate information of local regions : h1 , h2 , · · · , hN = LocalRNN ( x1 , x2 , · · · , xN ) ( 3 ) The localRNN is analogous to 1-D Convolution Neural Networks where each local window is processed by convolution operations . However , the convolution operation completely ignores the sequential information of positions within the local window . Although the position embeddings have been proposed to mitigate this problem , a major deficiency of this approach is that the effectiveness of the position embedding could be limited ; thus it requires considerable amount of extra efforts ( Gehring et al. , 2017 ) . On the other hand , the LocalRNN is able to fully capture the sequential information within each window . In addition , the one-by-one sliding operation also naturally incorporates the global sequential information . Discussion : RNNs have long been a dominating choice for sequence modeling but it severely suffers from two problems – The first one is its limited ability to capture the long-term dependencies and the second one is the time complexity , which is linear to the sequence length . However , in LocalRNN , these problems are naturally mitigated . Because the LocalRNN is applied to a short sequence within a local window of fixed size , where no long-term dependency is needed to capture . In addition , the computation procedures for processing the short sequences are independent of each other . Therefore , it is very straightforward for the parallel implementation ( e.g. , using GPUs ) , which can greatly improve the computation efficiency . 3.2 CAPTURING THE GLOBAL LONG-TERM DEPENDENCIES WITH MULTI-HEAD ATTENTION . The RNNs at the lower level introduced in the previous subsection will refine representation of each positions such that it incorporates its local information . In this subsection , we build a sub-layer on top of the LocalRNN to capture the global long-term dependencies . We term it as pooling sublayer because it functions similarly to the pooling operation in CNNs . Recent works have shown that the multi-head attention mechanism is extremely effective to learn the long-term dependencies , as it allows a direct connection between every pair of positions . More specifically , in the multihead attention mechanism , each position will attend to all the positions in the past and obtains a set of attention scores that are used to refine its representation . Mathematically , given current representations h1 , h2 , · · · , ht , the refined new representations ut are calculated as : ut =MultiHeadAttention ( h1 , h2 , · · · , ht ) ( 4 ) = Concatenation ( head1 ( ht ) , head2 ( ht ) , · · · , headk ( ht ) ) W o where headk ( ht ) is the result of kth attention pooling and W o is a linear projection matrix . Considering both efficiency and effectiveness , the scaled dot product is used as the attention function ( Vaswani et al. , 2017 ) . Specifically , headi ( ht ) is the weighted sum of all value vectors and the weights are calculated by applying attention function to all the query , key pairs : { α1 , α2 , · · ·αn } = Softmax ( { < q , k1 > √ ( dk ) , < q , k2 > √ ( dk ) , · · · , < q , kn > √ ( dk ) } ) ( 5 ) headi ( ht ) = n∑ j=1 αjvj where q , ki , and vi are the query , key , and value vectors and dk is the dimension of ki . Moreover , q , ki , and vi are obtained by projecting the input vectors into query , key and value spaces , respectively ( Vaswani et al. , 2017 ) . They are formally defined as : q , ki , vi =W qht , W khi , W vhi ( 6 ) where W q , W k and W v are the projection matrices and each attention pooling headi has its own projection matrices . As shown in Eq . ( 5 ) , each headi is obtained by letting ht attending to all the “ past ” positions , thus any long-term dependencies between ht and hi can be captured . In addition , different heads will focus on dependencies in different aspects . After obtaining the refined representation of each position by the multi-head attention mechanism , we add a position-wise fully connected feed-forward network sub-layer , which is applied to each position independently and identically . This feedforward network transforms the features non-linearly and is defined as follows : FeedForward ( mt ) = max ( 0 , utW1 + b1 ) W2 + b2 ( 7 ) Following ( Vaswani et al. , 2017 ) , We add a residual ( He et al. , 2016 ) and layernorm ( Ba et al. , 2016 ) connection between all the sub-layers .

Summary: The paper introduces the R-Transformer architecture which adds a local RNN layer before each attention layer in Transformer. The authors claim state-of-the-art performance but only test on tiny tasks where Transformer models have not been heavily optimized and omit the main problem with RNNs - namely their speed. It is an interesting paper still and the locality is a nice way to remedy the speed problem, but the paper lacks a true study and ablations on this main limitation. In summary: the main new idea of the paper is to make RNNs local in Transformer (trying to add RNN layers has been explored before). This idea could be a good tradeoff between full RNN (slow) and no RNN (lack of context), but the following is missing: (1) ablations on speed vs results by locality window, (2) experiments on more widely reported and larger data-sets and models, at least including some language modeling task (wiki or lm1b) and some translation task (like en-de). Without these results, we cannot recommend to accept this paper.