Transforming Recurrent Neural Networks with Attention and Fixed-point Equations

1 INTRODUCTION . Recurrent Neural Network , known as RNN , has been widely applied to various tasks in the last decade , such as Neural Machine Translation ( Kalchbrenner & Blunsom , 2013 ; Sutskever et al. , 2014 ) , Text Classification ( Zhou et al. , 2016 ) , Name Entity Recognition ( Zhang & Yang , 2018 ; Chiu & Nichols , 2016 ) , Machine Reading Comprehension ( Hermann et al. , 2015 ; Kadlec et al. , 2016 ) and Natural Language Inference ( Chen et al. , 2017 ; Wang et al. , 2017 ) . Models applied to these tasks are not the vanilla RNNs but two of their famous variants , Gated Recurrent Unit ( Cho et al. , 2014 ) , known as GRU , and Long Short Term Memory ( Hochreiter & Schmidhuber , 1997 ) , known as LSTM , in which gates play an important role . RNNs are hard to be computed parallelly . They are not bidirectional either , meaning that a word can not utilize the information of words coming after it . A general way to alleviate this problem is to reverse the input sequence and combine results given by two different RNN encoders with operations like concatenation and addition . However , Transformer ( Vaswani et al. , 2017 ) has provided a better solution . It is based on purely attention mechanism , which has been widely used in Neural Machine Translation since Bahdanau et al . ( 2014 ) . Models based on self-attention mechanism are mostly Transformer and its variants , such as Transformer-XL ( Dai et al. , 2019 ) , Universal Transformer ( Dehghani et al. , 2019 ) and Star-Transformer ( Guo et al. , 2019 ) . Compared with recurrent units such as GRU and LSTM , self-attention-style models can be computed parallelly , which means they suit better large-scale training . But each of these Transformers has an FFN layer with a very high vector dimension , which still is the bottleneck to improve the computation efficency . In this paper , we present a new framework based on Banach Fixed-point Theorem to transform the vanilla RNN and its variants with self-attention mechanism . StarSaber , one of such transformed models , outperforms both the vanilla Transformer and ReZero ( Bachlechner et al. , 2020 ) in our experiments with less parameters and thus less computational power . To start with , we need a different view of attention . Attention is a way to build a relation graph between words , and the vanilla RNN is nothing but a model with a relation graph as a chain . This graph is in fact represented with an adjacent matrix , which is computed by mapping each pair of positions to a positive real number and normalizing the numbers related to each position , which are just those in the same row of the adjacent matrix , so that they sum up to one . The vanilla RNN updates hidden states through a chain , that is , the hidden state for each position only depends on that in the previous position . However , if we have this relation graph , the hidden state for each position depends on hidden states for all other positions in a sequence . This is where we obtain equations . In our opinion , a bidirectional RNN is defined by some equations and Banach Fixed-point Theorem inspires us to iterate according to them . When we fix the number of iterations and specify distinct weights for each of them , a self-attention-style model is then constructed . In Transformer , Position Embedding ( PE ) as a way to capture word order information in language by adding a matrix to the input , is indispensable . But in StarSaber , position encoding is done in the aggregation step after the construction of a relation graph . For each position , we sum up linear transformations of hidden states in all positions with the corresponding weights in the relation matrix in order to get an attention vector . In the calculation of such a vector , we specify different linear transformation weights for the ” future ” and the ” past ” . Then the hidden vector for a position is computed with the corresponding attention vector and an input vector , which turns into a direct path from the input layer to each hidden layer . And we directly drop the FFN layer in Transformer achieving still competive and even better results with much less parameters on three datasets provided by CLUE ( Xu et al. , 2020 ) : the AFQMC dataset of Sentence Similarity , the TNEWS dataset of Text Classification and the CMNLI dataset of Natural Language Inference . More importantly , our derivation of StarSaber shows a universal way to transform different RNNs , such as LSTM and GRU discussed in the following content , providing possibilities other than Transformers for self-attention models . 2 RELATED WORK . Gates were first introduced into recurrent networks in LSTM and were rediscovered and simplified in GRU . Gate mechanism is an operation that multiplies an output by a single sigmoid layer of the input and is often seen as an approach to address the gradient vanishing issue . But if only so , other approaches which addresses this problem should achieve similar results to LSTM and GRU . In this paper , we show by experiments that in StarSaber which doesn ’ t have such a problem , gates can also help improve the performance . Attention in sequence modeling is a weighted sum of the output in each position of a sequence , which simulates the way a man distributes his attention to all its parts . Weights in this sum are given by a certain function of some inputs . And self-attention is an approach computing both the weighted sum and weights on the same sequence without any other inputs . There are different types of attention like multi-head attention and scaled dot product attention in Transformer , attention based on addition in Bahdanau et al . ( 2014 ) , and bilinear attention in Luong et al . ( 2015 ) . Our model applies the bilinear attention in the construction of a word relation graph . Residual Connection was proposed by He et al . ( 2015 ) . It alleviates the problem of training deep neural networks . In Natural Language Processing , Residual Connection alleviates both the gradient vanishing problem and the degration problem of deep networks . Our model uses a weighted residual connection ( Bachlechner et al. , 2020 ) which further alleviates the degration problem . Another similar idea is the highway connection ( Srivastava et al. , 2015 ) . In this paper , we inspect the gate mechanism in our self-attention-style model . Note that the highway connection can also fit into our framework , which is a fixed-point generalization of GRU . Pretraining has proved to be extremely useful since Embeddings from Language Models ( ELMO ) ( Peters et al. , 2018 ) . Many works that follow such as BERT ( Devlin et al. , 2018 ) , ALBERT ( Lan et al. , 2020 ) , XLNET ( Yang et al. , 2019 ) have outperformed humans . Pretraining is a training pattern which trains a language model , usually extremely large , on an enormous dataset with one or more unsupervised tasks and fine-tunes it on other datasets and tasks . There are two types of language models in general , known as auto-regressive models ( e.g. , XLNET ) and auto-encoder ones ( e.g. , BERT ) . However , pretraining on a large dataset requires resources . We show in this paper that only pretraining on a dadtaset formed by collecting the training , development and test inputs together can as well improve the performance , revealing the significance of pretraining tasks . MLM is the unsupervised task utilized by BERT to pretrain . It randomly masks some pieces of a sentence and train the model to predict what has been masked . In this way , knowledge is gained and the model is initialized for downstream tasks . However , experiments show that even not pretrained on a large dataset , using MLM to pretrain on a dataset formed by collecting the training , development and test inputs can still improve the performance , inspiring us that a more flexible and task-related pretraining method is beneficial . 3 MODEL ARCHITECTURE . 3.1 RECURRENT NEURAL NETWORKS AND BIDIRECTIONAL EQUATIONS WITH SELF-ATTENTION . This section follows the intuition we have discussed before . The vanilla RNN formulas are listed below : hn = tanh ( Uhn−1 +Wxn ) ( 1 ) In self-attention , we don ’ t just utilize the hidden state from the previous position but hidden states from all positions , to compute a hidden vector for position n. To encode information of relative order , we specify distinct linear transformation weights . For simplicity , we ignore all bias terms in the following derivation . Following the bilinear self-attention mechanism , we have : hn = tanh ( An + V xn ) An = ∑ i < n GniU lefthi + ∑ i≥n GniU righthi 1 Gni = softmax ( g , −1 ) = gni∑ j gnj gni = exp ( hTnWhi√ d ) 2 ( 2 ) What we have done here is replacing hn−1 with an attention vector An . Notice a fact that in the first equality , h appears on both the left-hand side and the right-hand side ( we use h to compute An ) , turning it into an equation . This means a bidirectional model is defined by a set of equations , because the word relation graph constructed by attention is not free of loops . Moreover , introducing equations can be seen as a constraint to obtain stable representation of a sentence . Intuitively , if we view the non-linear function on the right hand side as an updating operation and the hidden vector we obtain in each position as a semantic representation , it simply means that when the model ” reads ” the whole sentence again based on the current understanding , it should produce the same representation , meaning that it has ” fully understands ” the whole sentence and makes no changes on the hidden vectors . 3.2 GENERALIZE EQUATIONS WITH FIXED-POINT . Now we have an equation to solve , which is extremely complex and difficult . But Banach Fixedpoint theorem shows us a way . Theorem 3.1 ( Banach Fixed-point Theorem ) For any real-valued function f ( x ) , if | dfdx | < 1 , then iteration process xn+1 = f ( xn ) converges and lim n→+∞ xn = x ∗ , where x∗ = f ( x∗ ) . 1If not stated , a sum without a limit is to sum over all possible values . 2All these U , V , Ws are matrices that satisfy rules of the matrix-vector product . The hyperparameter d here is the hidden size . The attention here is scaled for faster convergence . The equation above is an equation of an iterative pattern , and this Theorem just tells us that as long as we keep iterating , we will obtain a root of the equation if its jacobian matrix satisfies some conditions . The iterative pattern is given as follows : hl+1n = tanh ( A l n + V xn ) Aln = ∑ i < n GlniU lefthli + ∑ i≥n GlniU righthli Glni = softmax ( g l , −1 ) = g l ni∑ j glnj glni = exp ( ( hln ) TWhli√ d ) ( 3 ) We can then iterate till it converges . Similar ideas are in Bai et al . ( 2019 ) , where the authors solve the fixed-point directly with very high computational cost . Sometimes it can not even converge to a fixedpoint , since the convergence condition is quite strict . A sufficient condition for convergence is that all parameter matrices are strictly orthogonal , making the optimization problem hard . Therefore , if we want to obtain a faster and more stable model , we can approximate it with a Multi-layer Perceptron ( MLP ) and relax the condition of convergence . In addition , we allow our model to assign different weights for different layers . The reason why we don ’ t reuse parameters in each layer is that iterating with the same set of parameters without a constraint of orthogonality often diverges . Even if we fix the number of iterations , it is still hardly possible to converge to the correct fixed-point . In this case , specifying different weights for each layer allows our model to learn a better fit for the whole iteration process . Therefore , we have hl+1n = tanh ( A l n + V lxn ) Aln = ∑ i < n GlniU lhli + ∑ i≥n GlniQ lhli Glni = softmax ( g l , −1 ) = g l ni∑ j glnj glni = exp ( ( hln ) TW lhli√ d ) ( 4 ) Here we also need an initial point to start the iteration . In our model , we choose the input sequence itself to be the initial value , that is to set h0i = xi . In more general cases , the initial value may be a linear transformation of the input or just some fixed vector like a zero one .

Summary: This paper aims to incorporate the attention mechanism into recurrent neural networks by using fixed point equations. In particular, the authors define a bidirectional RNN with attention by a fixed point equation and then transform it to a variant of the Transformer block. The proposed model StarSaber is shown to be more parameter efficient than the Transformer model and achieve competitive performance on three CLUE datasets.