Layer Flexible Adaptive Computation Time for Recurrent Neural Networks

1 INTRODUCTION . Recurrent neural networks ( RNN ) are widely used in supervised machine learning tasks for their superior performance in sequence data , such as machine translation Auli et al . ( 2013 ) ; Liu et al . ( 2014 ) , speech recognition Graves et al . ( 2013 ) ; Hannun et al . ( 2014 ) , image description generation Karpathy & Fei-Fei ( 2015 ) ; Mao et al . ( 2015 ) , and music generation Boulanger-Lewandowski et al . ( 2012 ) . The design of the underlying network is always a daunting task requiring substantial computational resources and experimentation . Many recent breakthroughs hinge on multilayer neural networks ability to increase model accuracy , Hinton et al . ( 2012 ) ; Mohamed et al . ( 2012 ) ; Srivastava et al . ( 2015 ) , leading to the important decision in RNNs of the number of computational steps . First , the right choice requires running several very expensive training processes to try many different computational steps . Even if a reinforcement learning algorithm is used to determine a good computational steps , Baker et al . ( 2017 ) ; Zoph & Le ( 2017 ) , it still requires a substantial training effort . The second issue with the fixed structure RNNs is the fact that the same computational steps is applied to each input in a sequence . It is conceivable that some inputs are harder to classify than others and thus such harder inputs should employ more computational steps . A similar argument holds for steps , e.g. , certain steps in a sample can bear less predictive power and thus should use fewer computational steps in order to decrease the computational burden . The goal of our work is to introduce a network that automatically determines the computational steps - and together with this the number of hidden vectors to use - in training and inference which is dynamic with respect to samples and step number . To resolve the inherent problems of fixed structure neural networks , Graves Graves ( 2016 ) addresses this by providing an Adaptive Computation Time ( ACT ) model for RNN . In Graves ’ model , a sigmoidal halting unit is utilized to calculate a halting probability for each intermediate round within a step , and a computation stops when the accumulated halting probability reaches or exceeds a threshold . ACT can utilize multiple computation rounds within each individual step and it can dynamically adapt to different samples and steps . The model is appealing due to its modeling flexibility and its advantage in increasing model accuracy Dehghani et al . ( 2018 ) . With the ACT mechanism , when a step of computation is halted , all intermediate states and outputs are used to calculate one meanfield state and output . The mean-field state and output have drawbacks . The outputs of the deepest computational step are the most informative , and should be the final outputs . The output from early computational steps may cause errors in the mean-field result which calls for using the last output only . However in such a case all computational steps should benefit from transmissions from the previous time step . This is not offered by ACT since its design is based on mean-field states and is a key feature of the proposed model . To distinguish the roles among different computational steps , each one should obtain its computation ability and receive its state individually from the previous time step . Thus a more natural design should be a multilayer RNN with a flexible number of layers which is exactly what our proposed model offers . Our experimental results show that ACT has marginal benefits over basic RNN or sequence to sequence ( seq2seq ) models , indicating that ACT , with a single hidden vector , can not always work well . This also motivates us to develop the layer flexible RNN model with adaptive computation time . The novelty of our work is that the number of layers in our model is flexible , so that it can both achieve adaptive computation time and maintain the individual roles among different layers . Similar to Graves ’ work , we also utilize a unit to determine the action of each computational step within a time step by calculating their halting probabilities . To obtain the optimal computation ability , each layer should learn from the previous time step individually , and there should be concepts to decide how much to learn from each layer in the previous time step . We face the challenge that the number of layers is different between two consecutive time steps , so that we can not set specific constant rules of how to transmit the states . In our model , each time step produces multiple hidden states ( one state per computational time within the step ) . These multiple hidden states are then combined into a different number of hidden states for the next step using attention ideas Bahdanau et al . ( 2014 ) ; Luong et al . ( 2015 ) ( the number of new hidden states equals to the number of computational steps in the next step ) . The network can thus have a flexible number of layers with dynamic number of transmission states . In this paper , we proposed a layer flexible adaptive computation time ( LFACT ) model for RNNs . Each layer indicates a computational step , produces a hidden state and receives its own transmission state from the previous time step . We also extend the model to the seq2seq framework . Our experimental results show that LFACT offers significant improvements over ACT and RNN on different data sets and frameworks . With LFACT , there is no need to decide the specific structure of an RNN model through extensive experimentation , since LFACT can automatically make decisions of computational steps based on its inputs . LFACT is designed with a different logic in mind from ACT , and at the same time overcomes the problems of ACT , e.g . poor performance on certain data sets . Our model increases the accuracy of 7 % to 8 % on a financial data set and 12 % on Wikipedia language modeling , which attests to its robustness . The rest of the manuscript is structured as follows . In Section 2 we review the literature . In Section 3 , the flexible layer adaptive computation time RNN model is presented , including all of the alternative options . In Section 4 we introduce the data sets and discuss all the experimental results . 2 LITERATURE REVIEW . A deep learning model and algorithm have many hyperparameters . In an RNN , one of the problems is deciding the computation amount of a certain input sequence . A simple solution is comparing different depths of networks and manually selecting the best option , but a series of expensive training processes is required to make the right decision . Hyperparameter optimization Bergstra et al . ( 2011 ; 2013 ) and Bayesian optimization Snoek et al . ( 2012 ) ; Mendoza et al . ( 2016 ) ; Saxena & Verbeek ( 2016 ) have been proposed to select an efficient architecture of a network . Based on these concepts , Zoph Zoph & Le ( 2017 ) and Baker Baker et al . ( 2017 ) propose mechanisms for network configuration using reinforcement learning . However , massive training efforts are still present . Another problem of such approaches is the assumption of a fixed structure of the network , irrespective of the underlying sample and step . The difficulty of classification varies in each data set and sample , and it is comprehensible that harder samples would require more computation . Therefore , applying networks with the same computational steps is inflexible and it can not achieve the goal of flexible computation time among different samples . Conditional computation provides general ideas for alleviating the weaknesses of a fixed-structure deep network by establishing a learning policy Dahl et al . ( 2012 ) ; Bengio et al . ( 2015 ) . A halt neuron is designed and used as an activation threshold in self-delimiting neural networks Schmidhuber ( 2012 ) ; Srivastava et al . ( 2013 ) to stop an ongoing computation whenever it reaches or exceeds the halting threshold . Work Ying & Fragkiadaki ( 2017 ) shows that conditional computation helps the networks obtain adaptive depth and thus yield higher accuracy than fixed depth structures . Graves Graves ( 2016 ) introduces an Adaptive Computation Time ( ACT ) mechanism for RNN to dynamically calculate each input step computation time and determine their halting condition . These series of work focus on formulating the policies of halting conditions and use a single hidden vector in each cell ; none of them contribute to designing flexible multilayer networks or study learning the rules of state transmission . The ACT mechanism Graves ( 2016 ) is proved to improve performances and is applied in a few different problems . Universal Transformers Dehghani et al . ( 2018 ) apply ACT on a self-attentive RNN to automatically halt computation . A dynamic time model for visual attention Li et al . ( 2017 ) is proposed to accelerate the processing time by adding a binary action at each step to determine whether to continue or stop . Figurnov et al . Figurnov et al . ( 2017 ) prove that applying ACT on Residual Networks can dynamically choose the number of evaluated computational steps and propose spatially adaptive computation time for Residual Networks for image processing to adapt the computation amount between spatial positions . Similarly , Neumann et al . Neumann et al . ( 2016 ) extend ACT to a recognizing textual entailment task . In addition , ACT is also applied to reduce computation cost and calculate computation time in speech recognition Li & Liu ( 2018 ) , image classification Leroux et al . ( 2018 ) , natural language processing Yu et al . ( 2018 ) , and highway networks Park & Yoo ( 2017 ) . These models simply apply the ACT mechanism on other models to achieve the abilities of adaptive halting computations . They focus on solving their specific problems but do not make any change to the structure of ACT cells . However , our work concentrates in the inner design of a layer flexible ACT cell for its ability of automatically and dynamically adapting the number of layers . 3 MODEL . We start with an explanation of RNN and ACT . A standard RNN contains three layers : the input layer , the hidden layer , and the output layer . The input layer receives input sequences x and transmits them to the hidden layer to compute the hidden states u . The output layer calculates the output y based on the updated state of each step . The equations are as follows : ut = f ( xt , ut−1 ) , yt = σ ( Wout + bo ) . In step t , input xt from the input sequence x is delivered to the network . A cell in the hidden layer uses the input xt and the state ut−1 from the previous step to update the hidden state ut in the current step . Long Short-Term Memory ( LSTM ) Hochreiter & Schmidhuber ( 1997 ) and Gated Recurrent Unit ( GRU ) Cho et al . ( 2014 ) are frequently applied in the hidden layer cell f , which contain the dynamic computation information and the activation of the hidden cells . The output yt is computed utilizing an output weight Wo , an output bias bo , and an activation function σ . ACT extends the standard RNN . The hidden layer contains several rounds of computation and each round produces an intermediate state and output . The representation of intermediate states unt and intermediate outputs ont are as follows : unt = { f ( x0t , ut−1 ) , n = 0 f ( xnt , u n−1 t ) n > 0 , xnt = ( δn , xt ) , o n t = σ ( Wou n t + bo ) . The first hidden cell , in step t , receives the state ut−1 from the previous step t − 1 and computes the first intermediate state . All the following rounds of computation use the previous intermediate output un−1t and produce an updated state u n t . To distinguish different rounds of computation , a flag δ0 is augmented to the input xt for the first round and another flag δn is added for all others . Each intermediate output ont is computed based on the intermediate state u n t in the same round . To determine the halting condition of a series of rounds of computation , units hnt are introduced in each computation round n as hnt = σ ( Whu n t + bh ) . Here Wh is the halting weight and bh is the halting bias . The total computation time Nt in a step is decided by the halting units and the maximum threshold L. Whenever the accumulated halting units ’ value in a step t is over 1 or the computation time reaches L , the computation halts . The definition of total computation time Nt is as follows : Nt = min { min { n| n∑ i=1 hit ≥ 1− } , L } , ( 1 ) where is a hyperparameter . ACT uses all the intermediate states and outputs to calculate one mean-field state ut and output yt ( as represented in ( 2 ) and ( 3 ) below ) for each step . A probability pnt produced by halting unit h n t is introduced into ACT for calculating the mean-field state and output according to the contribution of each intermediate computation round in a step . The updated mean-field state ut is transmitted to the next input step and the output ot is delivered to the output layer as the current step ’ s output . pnt = { hnt , n < Nt 1− ∑Nt−1 i=1 h i t n = Nt ut = Nt∑ i=1 pitu i t ( 2 ) yt = Nt∑ i=1 pito i t ( 3 ) Given an input sequence x , the ACT model tends to compute as much as possible in each step to avoid making erroneous predictions and incurring errors . This can cause an extra computational expense and impede achieving the goal to adapt the computation time . Therefore , training the model to decrease the amount of computation becomes necessary . ACT introduces ponder cost P ( x ) as P ( x ) = Nt + pNtt to represent the total computation time during the input sequence . The loss function L ( x , gt ) with gt being the ground truth is modified to encourage the network to also minimize P ( x ) : L̂ ( x , gt ) = L ( y ( x ) , gt ) + τP ( x ) ( 4 ) where τ is a hyperparameter time penalty that balances the ponder cost and prediction errors .

Summary: This paper proposes a layer-flexible adaptive computation time model which enables learning with a different number of layers at each time step.  It proposed a set of mechanisms to make the variable layer possible. It uses attention to re-arrange the hidden states in different layers into a different number of hidden states, thus allowing the hidden layers to be variable between different time steps. It also augmented the RNN to have a transmission state that transmits layer information to the next time step.