Layer Flexible Adaptive Computation Time for Recurrent Neural Networks
Authors: Lida Zhang
Diego Klabjan
Venue:  (2020)

Abstract: Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of a sequence. We propose a layer flexible recurrent neural network with adaptive computation time, and expand it to a sequence to sequence model. Different from the adaptive computation time model, our model has a dynamic number of transmission states which vary by step and sequence. We evaluate the model on a financial data set and Wikipedia language modeling. Experimental results show the performance improvement of 7% to 12% and indicate the model's ability to dynamically change the number of layers along with the computational steps.

Under review as a conference paper at ICLR 2020
LAYER FLEXIBLE ADAPTIVE COMPUTATION TIME FOR RECURRENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of sequence. We propose layer flexible recurrent neural network with adap- tive computation time, and expand to a sequence to sequence model. Different from the adaptive computation time model, our model has dynamic number of transmission states which vary by step and sequence. We evaluate the model on financial data set and Wikipedia language modeling. Experimental results show the performance improvement of 7% to 12% and indicate the model's ability to dynamically change the number of layers along with the computational steps.
INTRODUCTION
Recurrent neural networks (RNN) are widely used in supervised machine learning tasks for their superior performance in sequence data, such as machine translation AAuli et al. (2013); Liu et al. (2014), speech recognition Graves et al. [ Hannun et al. (2014), image description generation Karpathy & Fei-Fei (2015): Mao et (215)). and music generation [Buulanger-Lewaddowsss Lewandowski et al. (2012). The design of the underlying network is always daunting task requiring substantial com- putational resources and experimentation. Many recent breakthroughs hinge on multilayer neural networks ability to increase model accuracy, [Hinton al. (2012;: Mohamed al. (2012); [rrivas] lava et al. (2013). leading to the important decision in RNNs of the number of computational steps. First, the right choice requires running several very expensive training processes try many dif- ferent computational steps. Even if reinforcement learning algorithm is used to determine good computational steps, Baker et al. (0017): Zoph & Le (2017). it still requires a substantial training effort. The second issue with the fixed structure RNNs is the fact that the same computational steps applied to each input in sequence. It is conceivable that some inputs are harder to classify than others and thus such harder inputs should employ more computational steps. A similar argument holds for steps, e.g., certain steps a sample can bear less predictive power and thus should use fewer computational steps in order to decrease the computational burden. The goal of our work is to introduce a network that automatically determines the computational steps and together with this the number of hidden vectors use in training and inference which is dynamic with respect to samples and step number.
To resolve the inherent problems of fixed structure neural networks, Graves Graves (206)) addresses this by providing an Adaptive Computation Time (ACT) model for RNN. In Graves" model, a sig- moidal halting unit is utilized calculate halting probability for each intermediate round within step, and a computation stops when the accumulated halting probability reaches or exceeds a thresh- old. ACT can utilize multiple computation rounds within each individual step and it can dynamically adapt to different samples and steps. The model is appealing due to its modeling flexibility and its advantage in increasing model accuracy Dehghani et al. (2018). With the ACT mechanism, when step of computation is halted, all intermediate states and outputs are used to calculate one mean- field state and output. The mean-field state and output have drawbacks. The outputs of the deepest computational step are the most informative, and should be the final outputs. The output from early computational steps may cause errors in the mean-field result which calls for using the last output only. However in such case all computational steps should benefit from transmissions from the previous time step. This is not offered by ACT since its design is based on mean-fieh states and is key feature of the proposed model. To distinguish the roles among different computational steps,

---

Under review as a conference paper at ICLR 2020
each one should obtain its computation ability and receive its state individually from the previous time step. Thus a more natural design should be a multilayer RNN with a flexible number of lay- which is exactly what our proposed model offers. Our experimental results show that ACT has marginal benefits over basic RNN or sequence to sequence (seq2seq) models, indicating that ACT, with a single hidden vector, cannot always work well. This also motivates us to develop the layer flexible RNN model with adaptive computation time.
The novelty of our work is that the number of layers in our model is flexible, so that it can both achieve adaptive computation time and maintain the individual roles among different layers. Similar to Graves' work, we also utilize a unit determine the action of each computational step within a time step by calculating their halting probabilities. To obtain the optimal computation ability, each layer should learn from the previous time step individually, and there should be concepts to decide how much to learn from each layer in the previous time step. We face the challenge that the number of layers is different between two consecutive time steps, so that we cannot set specific constant rules of how to transmit the states. In our model, each time step produces multiple hidden states (one state per computational time within the step). These multiple hidden states are then combined into a different number of hidden states for the next step using attention ideas Bahddanuu et al., (2014); Luong et al. (2015) (the number of new hidden states equals to the number of computational steps in the next step). The network can thus have flexible number of layers with dynamic number of transmission states.
In this paper, we proposed layer flexible adaptive computation time (LFACT) model for RNNs. Each layer indicates a computational step, produces a hidden state and receives its own transmis- sion state from the previous time step. We also extend the model the seq2seq framework. Our experimental results show that LFACT offers significant improvements over ACT and RNN on dif- ferent data sets and frameworks. With LFACT, there no need to decide the specific structure of an RNN model through extensive experimentation, since LFACT can automatically make decisions of computational steps based on its inputs. L.ACT is designed with a different logic mind from ACT, and at the same time overcomes the problems of ACT, e.g. poor performance on certain data sets. Our model increases the accuracy of 7% to 8% on a financial data set and 12% on Wikipedia language modeling, which attests to its robustness.
The rest of the manuscript is structured as follows. In Section we review the literature. In Section 3, the flexible layer adaptive computation time RNN model is presented, including all of the alternative options. In Section we introduce the data sets and discuss all the experimental results.
LITERATURE REVIEW
A deep learning model and algorithm have many hyperparameters. an RNN, one of the problems
is deciding the computation amount of a certain input sequence. A simple solution is comparing dif-
ferent depths of networks and manually selecting the best option, but series of expensive training processes is required to make the right decision. Hyperparameter optimization Bergstra et al. (201T)
2013) and Baycsian optimization [Snock et al. (2012); Mendoza et al. (2016); Saxena Verbeek (2016) have been proposed to select an efficient architecture of network. Based these concepts, Zoph Zoph & Le (2017) and Baker Baker et al. (2017) propose mechanisms for network config- uration using reinforcement learning. However, massive training efforts are still present. Another problem of such approaches is the assumption of a fixed structure of the network, irrespective of the underlying sample and step. The difficulty of classification varies in each data set and sample, and it is comprehensible that harder samples would require more computation. Therefore, applying networks with the same computational steps is inflexible and it cannot achieve the goal of flexible computation time among different samples. Conditional computation provides general ideas for al- leviating the weaknesses of a fixed-structure deep network by establishing a learning policy [Dall et al. (2012); |Bengio et al.) (2015). A halt neuron is designed and used as an activation threshold in sel self-delimitig neural networks Schmidhuber (2012); Srivastava et al. (2013) to stop an ongoing computation whenever it reaches or exceeds the halting threshold. Work Ying & Fragkiadaki (2017) shows that conditional computation helps the networks obtain adaptive depth and thus yield higher accuracy than fixed depth structures. Graves Graves (2016) introduces an Adaptive Computation Time (ACT) mechanism for RNN to dynamically calculate each input step computation time and determine their halting condition. These series of work focus on formulating the policies of halting
2

---

Under review as a conference paper at ICLR 2020
conditions and use single hidden vector in each cell; none of them contribute to designing flexible multilayer networks or study learning the rules of state transmission.
The ACT mechanism Graves (016)) is proved to improve performances and is applied in few dif- ferent problems. Universal Transtormers Dehghani et al. (2188) apply ACT on a self self-attentive RNN to automatically halt computation. A dynamic time model for visual attention Li et al. (2017) is pro- posed to accelerate the processing time by adding a binary action at each step to determine whether continue or stop. Figurnov al. Figurnov al. (2017) prove that applying ACT on Residual Net- works can dynamically choose the number of evaluated computational steps and propose spatially adaptive computation time for Residual Networks for image processing to adapt the computation amount between spatial positions. Similarly, Neumann et al. Neumann et al. (2016) extend ACT to recognizing textual entailment task. In addition, ACT is also applied reduce computation cost and calculate computation time in speech recognition Li & Liu (2018), image classification Ler-- oux et al. 2018), natural language processing Yu al. (2018), and highway networks Park & Yoo (2017). These models simply apply the ACT mechanism on other models to achieve the abilities of adaptive halting computations. They focus on solving their specific problems but do not make any change to the structure of ACT cells. However, work concentrates in the inner design of layer flexible ACT cell for its ability of automatically and dynamically adapting the number layers.
3 MODEL
We start with an explanation of RNN and ACT, standard RNN contains three layers: the input layer, the hidden layer, and the output layer. The input layer receives input sequences x and transmits them to the hidden layer to compute the hidden states u. The output layer calculates the output based on the updated state of each step. The equations are as follows:
= f 11)
yt = oWW + bo).
In step f, input कy from the input sequence x is delivered to the network. A cell in the hidden layer uses the input 24 and the state u 1 from the previous step to update the hidden state uy in the current step. Long Short-Tern Term Memory (LSTM) Hochreiter & Schmidhuber (1997) and Gated Recurrent Unit (GRU) Cho et al.] (2014) are frequently applied in the hidden layer cell f. which contain the dynamic computation information and the activation of the hidden cells. The output y1 is computed utilizing an output weight Wo, an output bias by and an activation function o.
ACT extends the standard RNN. The hidden layer contains several rounds of computation and each round produces an intermediate state and output. The representation of intermediate states u. and intermediate outputs of
follows:
"C. {
[al (. u1-1), n == f(x) ut 1
xx (6),
of = oW. + bo).
n n>0
The first hidden cell, in step t. receives the state u from the previous step t - and computes the first intermediate state. All the following rounds of computation the previous intermediate output wf n-1 and produce an updated state u²". To distinguish different rounds of computation, a flag dn is augmented the input xy for the first round and another flag 5, is added for all others. Each intermediate output of is computed based on the intermediate state u,"" in the same round.
To determine the halting condition each computation round n as hi = oWW, + bR). Here WA. is the halting weight and by is the halting bias.
series of rounds of computation, units hi are introduced in
The total computation time N, in a step is decided by the halting units and the maximum threshold L. Whenever the accumulated halting units' value in a step t is over or the computation time reaches L, the computation halts. The definition of total computation time N, is as follows:
N: = min (min
Σ
2 1=,,).
where is hyperparameter.
ACT uses all the intermediate states and outputs to calculate one mean field state u, and output y (as represented in (2) and (3) below) for each step. A probability Pi produced by halting unit he is
3

---

Under review as a conference paper at ICLR 2020
R.
Rp-apt
1
Pow
Figure 1: LFACT model an example of three consecutive steps. Step has three layers, step has two layers, and step + 1 includes four layers.
introduced into ACT for calculating the mean-field state and output according to the contribution of each intermediate computation round in a step. The updated mean-field state u, is transmitted the next input step and the output is delivered to the output layer as the current step's output.
{
< N hi n No 11-2
h
N
N
U4
(2)
V
plo
Given an imput sequence x, the ACT model tends to compute as much as possible each step to avoid making erroneous predictions and incurring errors. This can cause extra computational ex- pense and impede achieving the goal to adapt the computation time. Therefore, training the model to decrease the amount of computation becomes necessary. ACT introduces ponder cost P(x)) as P(x) N₂+ + pi to represent the total computation time during the input sequence, The loss func- tion [(x, gt) with gt being the ground truth is modified to encourage the network to also minimize P(x):
( gl) = L(v(x), g() + +P(x:)
(4)
where is hyperparameter time penalty that balances the ponder cost and prediction errors.
3.1 LAYER FLEXIBLE ADAPTIVE COMPUTATION TIME RECURRENT NEURAL NETWORK
In this section, our Layer Flexible Adaptive Computation Time (LFACT) model is introduced. The main idea of LFACT is dynamically adjusting the number of layers according to the imminent char- acteristic of different inputs and efficiently transmitting each layer's information to the same layer in the next step. Differing from ACT where only the mean-field state u, in (2) is transmitted to the next step, which
be viewed as single layer network, LFACT is designed for transmitting each layer's 's state individually between every consecutive step. In LEACT we compute N₂ and N as in ACT. Each cell (layer n) in step takes 2; and i'" 1 input and creates u, for 1, 1,.., N₁. Vector uF is computed from the output u," of the previous cell and the hidden state ū;-1 from the previous step and same layer n. The problem is that at step we produce už for = 1, ... N₂ but for step + we need up for I,..,, Ne+1. The key of our model is to use the attention principle to create ui, ..., un N.+1 from ut, ub, ..., u Figure depicts the model.
The representation of the LFACT model is as follows:
Ne-1 9 u g( 1 u n-1
2/-1 -
n =
uf = f(ze, ((11,07_1) 2 =o=o(W.u bo).
n n>0
The LFACT model contains two types of states. One state u² is the primary output of each hidden cell, which is the same as the states in standard RNN. The other state is the transmission state up that is used for transmitting layer information the next step. The primary state from previous layer
4

---

Under review as a conference paper at ICLR 2020
"i" ui" and the transmission state 1i-1 from the same layer in the previous time step combined together through function g. The combined state is delivered to the current cell. Possible options for g are multi-layer fully connected neural network,
affine transformation of (æ, y) followed by an activation function. our experiments, we use g y) = oWir + W2y + b).
In step f, the hidden layer cell uses the input and the combined state from function g compute and update the primary state uy. The primary states are used to compute the transmission state uy for the next step. To avoid possible errors caused by the previous layer, input 2; is directly delivered to each layer as an input. For n I N141, the equations governing the relationship between two transmission states read
eDtin
Ortin
Ey= e
Bein = VT oWWoui+ + Voui + bQ) i c.
(5)
To compute the transmission states up attention unit introduced to represent the relationship between the primary states uy in certain layer and the primary states in other layers. We propose two choices to select c;:
min N), N
(a) (b)
Option (a) only considers the relationship between the state u. of the current layer and the states ui from the lower layers (i.e. S n). called limited (LTD). Alternative (b) utilizes all computed transmission states (i.e. i S Nº), called ALL. When strategy LTD applied and No+1 S N,, all primary states of in deeper layers (i.e. i >> N₂) cannot be used. Strategy ALL aims to include the computed information of all the layers. To distinguish different layers, extra weights V, are utilized to compute a. Weights Vn, Wq and Vo in (5) to compute are vectors.
We use the same method as ACT to compute N, (as represented in (T), the computation time of each step. But unlike ACT, the halting unit is computed based on the output and transmission state of each layer as hī oWW, V, + bn). In addition, instead of computing a mean-field output, we directly take the output of the deepest layer one step output y or
When applying loss function (4) to LFACT. the shallow layers have limited involvement in calculat- ing gradients. Therefore, to get the prediction of each layer accurate possible, we introduce all of the intermediate outputs in the loss function, as
N
Ō(x, gt) = _(x, + Σ [l)(()).
(6)
In the experiments we use - = L.
3.2 SEQUENCE TO SEQUENCE MODEL WITH LFACT
In order to deal with sequence tasks, propose combination model using seq2seq (encoder- decoder) model and our LFACT model, as Figure[ijn Appendix A. shows. In the seq2seq model, cell in each step is replaced with our LFACT model to form deep and flexible network. The seq2seq encoder part accepts a sequence input, and in the decoder part, we use the last ground truth as input.
4 COMPUTATIONAL EXPERIMENTS
All the models trained starting with random weights, i.e. pretraining. Training the LFACT model takes 20% to 30% more time than a typical ACT model. Most experiments are based on a single seed, but in Section 4.2 conclude that the variance is low if the seed is varied.
5

---

Under review as a conference paper at ICLR 2020
POS WING2 WINC3
2.40
2.35
IYpoWemeE OHE
0.144
0335
2.22
0.29
3:00
2.00
0.01
2.65
0.50
0.32
0.50
(a) Model performance improvement
(b) Hyperparameter comparison
0.30
1.20
4
033
0.00
Step
(c) F1 distribution
(d) N₂ distribution
Figure 2: Results of RNN based models. (a) F1 score improvements over RNN on financial data set three instances (INS1, INS2 and INS3). (b) Average Fl improvements over RNN for different values on all three instances. (c) Average Fl score at each step on INSI. (d) Average computation time (N,) for LFACT INS1.
4.1 FINANCIAL DATA SET
We test LFACT models financial data set from Harmon & Klabjan (2018). The data set consists of the tick prices of twenty-two ETFs at five minute intervals. The data is labeled into five classes to represent the significance of the price changes, e.g., one class corresponds the price being within one standard deviation. We have 22 softmax classification layers in each step. We have three test instances, and in each one train our model on 50 weeks of returns (45,950 samples), use the next week (905 samples) as validation data to save the best performing weights, and test the model based on the saved weights using the following week (905 samples). Sequences have lenght 20. The financial data set is tested on both RNN and seq2seq frameworks.
RNN Based Models: RNN based models predict the next step price changes in each time step. The LFACT model utilizes option affine transformation for g (g(x, y) = aWW, + W2y + b)) and strategy ALL for computing transmission state ū (cf = N2). We test plain ACT and RNN, which have been tuned with respect all hyperparameters as our baseline models, and compare them with the RNN based LFACT model. We apply 0.001 as our ponder time penalty (T = 0.001) for LFACT and ACT (the value is obtained by the general optimal value of the experiments from GraverGraves 2016)) ) and use the Adam optimizer with 0.0005 learning rate to train the models. The maximum number of layers L is and GRU cells with hidden vectors of size 128 utilized in all the models.
Figure 2a shows the F1 score improvements of LFACT and ACT over RNN. We test all models on three diffferent instances INSI, INS2, and INS3. Each bar indicates the average Fl score for all prediction steps in an instance. The results of LFACT are based on applying 0. to μ in loss (6) The Fl score of RNN 0.475, 0.461, 0.447 for INSI, INS2, INS3, respectively. From Figure 2a LFACT improves 14.1 14.1% % over RNN on average, and ACT improves 6.3%. We introduce the new order directly update the weights of each layer from the intermediate outputs. Figure 2b provides the performance comparison for different μ. The results are the average F1 score improvement over RNN for three instances. The best range for μ (6) 0.01 to 0, and is better than the original one in (4) by 1.2%. The application of different values shows that our new loss function yields improvements.
loss function
Figure 2c provides the Fl score distribution of steps to 20 INSI. LFACT consistently performs better than ACT, indicating that multiple layers of hidden vectors bring better effectiveness than single one. The difficulty of sequential prediction task is higher in early steps than in late ones, because the early steps have limited information from the input. LFACT and ACT both are stable in all prediction steps, but RNN acts poorly in early predictions. This benefit of LFACT and

---

Under review as a conference paper at ICLR 2020
#/533 0.83
0.03
0:011
0.34
30
15
0:8
(c) Training set Ne distribution
Mormum 300 Fesansle
(a) Model performance improve-
ment
135
ITAT awite
Step
Index
832
(d) Validation set N₂ distribution
Wilddmm Percurda
3.0
135
5
7
Decoder Stp I1dea
(b) distribution
(e) Test set N₂ distribution
Figure 3: Results of seq2seq based models. (a) F1 score improvements over seq2seq on financial data set three instances (INS1, INS2 and INS3). (b) Average F1 score at each step on INS1. (c) (d) (e) are computation time (Nv) distributions based on optimized LFACT weights on INSI: X-axis is the step index; 1 to 20 indicate encoder; 21 to 30 are from the decoder part.
ACT implies that adaptive computation can contribute to hard tasks. Figure 2d gives the average computation time (N₂) of each step on the test set of INSI. Higher average N, of early steps proves LFACT's s ability of deeply computing on hard tasks, and further explains why LFACT is so effective on early predictions.
Seq2seq Based Models (10 Prediction Steps): In addition to the RNN framework, we also use the seq2seq version of models predict the following ten steps. The raw sequence data with input length of 20 is delivered into seq2seq models as the inputs of the encoder part. All hyperparameters are the same as in the RNN based experiments, and the same strategies for g and e as in the RNN based LFACT are applied to the seq2seq framework. Considering that the encoder part does not have outputs, we apply loss function (4) in this task.
In Figure Bh, we present the F1 scores relative changes over seq2seq alone for each instance. The F1 scores of seq2seq are 0.439, 0.481, 0.447. The ACT model worse than seq2seq on INS3, so the improvement here is negative. From the results, the seq2seq based LFACT improves Fl 7.4% over seq2seq, and ACT acts similar to seq2seq. In Figure 33, we provide the Fl scores for the ten prediction steps in the decoder individually on INS1. All three models decrease over time, but LFACT and ACT are more stable than seq2seq. In seq2seq based models, the decoder part has con- stant input of last ground truth, and cause information deterioration time passes. Thus, the benefits of LFACT on late predictions over seq2seq alone imply better abilities of LFACT on infor- mation transmission and memorization. Surprisingly, the first prediction of seq2seq better than LFACT, which conflicts the results from RNN. This may be caused by LFACT requiring delay when transforming from input to predictions since it has more trainable weights than seq2seq. However, the whole point of the seq2seq framework is multiple steps of predictions, and LFACT catches up very fast at the second prediction, so the disadvantage of LFACT should not be concerning.
Figure 3 also presents the computation time (N₂) results for INSI: Figures Be and 3d are the results the training and validation process based on the optimized weights, and Figure 3e is for test. The result shows the change of N, among the different steps, indicating that the LFACT model has the ability of adapting computation time dynamically according to its input. Because of the same input in the decoder, N₂ values are the same from step 21 to 30 within each set. In addition, the low N₂ values in test set imply that LFACT has low computation request in the decoder part. Thus, the multiple computation ability of LFACT is not the reason for the good performance in the seq2seq setting, as it is in the early predictions in the RNN setting. Comparing to seq2seq alone which contains only one computation time well in the decoder, the significant benefits in
7

---

Under review as a conference paper at ICLR 2020
6.14
13237
6.330
-139
0.11
185
123
0.30
3.81
0.35
0.30
3.30
1.0
Trar Indre
(a) Models performance
(b) Hyperparameter comparison
(c) N₂ change
Figure 4: Results of Wikipedia language modeling task. (a) Models performances. Numbers above bars are BPC values and the percentage inside of hars are the relative changes over RNN. (b) Aver-
age F1 improvements over RNN for different μ values. (c) Computation time (N₂) change during LFACT model training process.
late predictions for LFACT further confirm the conclusion that LFACT has the excellent abilities for information transmission and memorization.
We also conduct similar experiments by making 5 predictions. These are shown in Appendix A.2. The observations are very similar.
4.2 WIKIPEDIA LANGUAGE MODELING
This task focuses predicting characters from the Hutter Prize Wikipedia data set, which is also used in Graves' ACT paper Graves (016)) The original unicode text used without any prepro- cessing. Every character represented one-hot, and presents one time step. In our experiment, 10,240 sequences including 512,000 characters in total are randomly selected as the training set, and 1,280 sequences with 64,000 characters in total are chosen as validation and test sets without repetition. Each sequence includes 50 consecutive characters, and the next character is predicted at each time step this task (RNN setting). GRU cells with 128 hidden size are used to struc- ture all models. The maximum number of layers I. is set to 3, and softmax layer with size 256 added to each step the decoder. We apply the optimized ponder time penalty (7) 0.06 from Graves' experiments Graves (2016) for this task. The models are evaluated using bit per character BPC = E Σ. - log, Pr(z++1 (b)) Lower BPC values reflect better performances. All results are based on option affine transformation for g (g(x, y) ( z W2y + b)) and strategy ALL (c"= (N)=,,,
In Figure 4a we present the experimental results of LFACT and the two baseline models ACT and RNN on the language modeling task. The reported BPC values for LFACT are from different settings of hyperparameter ju in loss (6). Three different random seeds are applied for ACT and RNN to test the stability of the models. Maximum, minimum, and average BPC values are provided. The bars in Figure repressent F average BPC values, and error bars indicate maximum and minimum BPC. From the experiment, ACT does not have significant benefit over RNN, but LFACT improves 11.9% over ACT and 12.6% over standard RNN. From the error bars, LFACT has the smallest variance and ACT varies the most. Strong stability for LFACT reflects its better ability deal with complex situations. To test the influence of the hyperparameter in loss function (6), we compare the different settings of μ Figure 4b When μ 0, the loss function equal to the original one in (4). From Figure 44b the best range for μ is from 0.01 to 0.1. However, when μ is set to be larger value (p > 0.3), the new loss function does not bring any performance improvement over the original loss function. To assess scalability of LFACT, we also test it on different sizes of training data, as Figure shows Appendix A.4. The results indicate that LFACT performs well on different sizes of training data ranging from 100,000 characters to 10 million.
In addition, we test the fully connected network option for g(x, y) and strategy LTD (c) min(N, n)). The fully connected network for provides 1.074 BPC, and LTD gives 1.678. Neither of them are better than our experimental settings. Therefore, the affine transformation for g and ALL better strategies for LFACT.
In Figure 4c
we provide the average maximum and average of each step computation time (N₂) during training of the Wikipedia language modeling task. We observe clear decrease during the early training epochs, which eventually stabilizes. Note that during epochs 5 to 10, the maximum N₂ increases but the average N, still decreases. We postulate that the L.ACT model has already
8

---

Under review as a conference paper at ICLR 2020
obtained the ability to predict most samples during this period, and is putting more effort the difficult samples. Figure X in Appendix A.3 shows the Maximum N, distributions of training, validation, and test based on the optimized weights. We only present the last 25 steps; the first 25 steps are all The distributions show that the LFACT model is able to keep the computation time as low as possible, but also has the ability of deep computation for certain samples. With the optimized weights, only 0.03% of the sequences in the training set have more than one computation time, and validation and test sets have 0.24% and 0.16% of the sequences with multiple computation. This difference happens because the model is trained based on the training set, and the model should have learned the most efficient way to predict characters in the training set.
5 CONCLUSION
Deciding the structure of recurrent neural networks has been a problem in deep learning applica- tions, in particular the number of computational steps. A halting unit is applied previous work to adapt the computation time to inputs. but single hidden vector structure leads to information transmission weaknesses. We propose LFACT which utilizes an attention strategy in designing an information transmission policy which leads to flexible multilayer recurrent neural network with adaptive computation time. LFACT can automatically adjust computation time according to the computing complexity of imputs and has outstanding dynamic information transmission abilities be- tween consecutive time steps. We apply LFACT in an RNN and a seq2seq setting and evaluate the model on financial data set and Wikipedia language modeling. The experimental results show significant improvement of LFACT over RNN and seq2seq and ACT on both data sets. The dif- ferent number of layers in practice indicates LFACT's ability of adapting computation time and information transmission.
REFERENCES
Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. Joint language and translation modeling with recurrent neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: MeXXx:1:990.. /409.0473, 2014.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network archi- tectures using reinforcement learning. International Conference on Learning Representations, 2017.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. International Conference on Learning Representations, 2015.
J. Bergstra, D. Yamins, and D. D. Cox. Making science of model search: Hyperparameter opti- mization in hundreds of dimensions for vision architectures. In Proceedings of the 30th Interna- tional Conference on International Conference Machine Learning, 2013.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Advances in Neural Information Processing Systems, 2011.
Nicolas
Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. Proceedings of the 29th International Conference on Machine Learning, 2012.
Kyunghyun Cho. Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014.
George E Dahl. Dong Yu, Li Deng, and Alex Acero. Context-dependen pre traitedd deep neural networks for lorge-voyabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30-44 2012.

---

Under review as a conference paper at ICLR 2020
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.0381 2018.
Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry P Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2017.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint
1XXvv:603.089833 2016.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep reeur- rent neural networks. In 2013 IEEE International Conference Acoustics, Speech and Signal Processing, 2013.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint
2014.
Mark Harmon and Diego Klabjan. Dynamic prediction length for time series with sequence to sequence networks. arXiv preprint 1rXiv:1807.00425, 2018.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):22-97 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2015.
Sam Leroux, Pavlo Molchanov, Pieter Simoens, Bart Dhoedt, Thomas Breuel, and Jan Kautz. Iamnn: Iterative and adaptive mobile neural network for efficient image classification. Workshop on International Conference Learning Representations, 2018
Mohan Li and Min Liu. End-to-end speech recognition with adaptive computation steps. arXiv preprint arXiv: 10088, 2018.
Zhichao Li, Yi Yang, Xiao Liu, Feng Zhou, Shilei Wen, and Wei Xu. Dynamic computational time for visual attention. IEEE International Conference on Computer Vision Workshop. 2017.
Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. A recursive recurrent neural network for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tutional Linguistics, 2014.
Thang Luong. Hieu Pham, and Christopher D. Manning. Effective approaches to attention- neural machine translation. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing. 2015.
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-RNN). International Conference on Learning Representations, 2015.
Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. To- wards automatically-tum neural networks. In Workshop on Automatic Machine Learning, 2016.
Abdel-rahman Mohamed, George Dahl, Geoffrey Hinton, et al. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14-22, 2012.
Mark Neumann, Pontus Stenetorp, and Sebastian Riedel. Learning to reason with adaptive compu- tation. NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems, 2016.
10

---

Under review as a conference paper at ICLR 2020
Hyunsin Park and Chang D Yoo. Early improving recurrent elastic highway network. arXiv preprint arXiv:1708. 140.1.0166,, 2017.
Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In Advances in Neural Informa- lion Processing Systems, 2016.
Jürgen Schmidhuber. Self-delimiting neural networks. arXiv preprint arXiv: 2012.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, 2012.
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in Neural Information Processing Systems, 2015.
Rupesh Kumar Srivastava, Bas R Steunebrink, and Jürgen Schmidhuber. First experiments with powerplay. The 2nd Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, 2013.
Chris Ying and Katerina Fragkiadaki. Depth-adaptive computational policies for efficient visual tracking. In International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 109-122. Springer, 2017.
Adams Wei Yu, Hongrae Lee, and Quoc Le. Learning to skim text. The 55th Annual Meeting of the Association for Computational Linguistics, 2018
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017.
A APPENDIX
A. SEQ2SEQ MODEL WITH LFACT
LEACT walel
LEACY
LPART
AFACT
EPACT
LEART
Figure 5: Seq2seq with LFACT model: the first four steps represent the encoder and the last four steps indicate the decoder. Ii ZT are inputs the encoder, 21 to 2p are inputs in the decoder, and y1 to ye are i' steps of predictions.
A.2 SEQ2SEQ BASED MODELS (5 PREDICTION STEPS)
To examine the stability of the LFACT model, we further test the seq2seq based models with pre- diction steps. The setting is the same as in the 10-prediction case except we have only predictions. F Figure shows the relative Fl scores for LFACT and ACT based seq2seq alone. The Fl scores for seq2seq on the three instances are 0.492, 0.534, and 0.498. The seq2seq based LFACT performs better than both ACT and seq2seq in the 5-prediction task, and the benefit is significant over ACT.

---

Under review as a conference paper at ICLR 2020
However, the improvement of LFACT over seg2seq is not as pronounced as in the 10 10-prediction task, and ACT is even worse than seq2seq. Figure iss is the Fl score distributions for the three models on INSI. The results match the 10-prediction task, and show that the advantage of LFACT is more likely to affect late predictions in the seq2seq framework.
0.04
m1522 1953
0.34
PPACT
2,023
0.52
0.02
0:03
0.66
4.23
0.64
0.53
3.005
0.22
0.54
0.40
0.043
Desodar Step Incuu
Figure 6: Performances of seq2seq based models (5 prediction steps): Fl relative changes over seq2seq on the financial data three instances (INS1, INS2, INS3).
F1 distributions (5 prediction steps): average Fl score at each prediction step for seq2seq based models on INS1.
Figure 7:
A.3 COMPUTATIONAL TIME FOR WIKIPEDIA LANGUAGE MODELING BASED ON OPTIMAL WEIGHTS
training
mvalication test
Maximum Mo
27
30
31
34
35
39
43
46
47
Step Index
Figure 8
A.4 PERFORMANCE ON DIFFERENT TRAINING SIZES FOR WIKIPEDIA LANGUAGE MODELING.
We test LFACT on different training sizes ranging from 100,000 characters in total to 10 million, as Figure 9 shows. As the training set size increases, our model achieves better performance and eventually gets around 0.99, which indicates scalability. We conclude that LFACT consistently has over 7% improvement on all of the training sizes over ACT and RNN. Due to the computational resource limitations, all the results in Section 4.2] including hyperparameter comparison, are based on 512,000 characters and 10,240 sequences training size, and 64,000 characters, 1,280 sequences test size.
1:30
1.90
1.00
0.00
Training Set Sie
Figure 9: Performance on different training size
12


Reviews:
[{"review_id": "ryxhOoWN9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a layer-flexible adaptive computation time model which enables learning with a different number of layers at each time step.  It proposed a set of mechanisms to make the variable layer possible. It uses attention to re-arrange the hidden states in different layers into a different number of hidden states, thus allowing the hidden layers to be variable between different time steps. It also augmented the RNN to have a transmission state that transmits layer information to the next time step. \n\nIn the LFACT model equation after Eq.4 on Page 4, the lower layer hidden states are incorporated with the previous time step hidden states through the function g, before they were sent into the RNN cell f. This makes it different than a more straight forward setting in stacked RNN where the lower layer outputs are sent to the upper layer cell directly. I am wondering why this indirect way of stacking? will it work well if the layers are stacked as the normal stacked RNNs do (i.e., u_t^n takes the lower layer hidden state u_t^{n-1} as input)?\n\nIn the experiments, it seems that the N_t is very stable through time steps. Most of them are 1 or 2 layers. Although the LFACT outperforms the RNN and ACT significantly, that could be a factor of hyperparameter tuning or model structure advantage. Have the authors tried to fix N_t as a constant (let’s say 2), and then perform the whole thing on the same setting again? I highly doubt that it will yield worse results. \n\nFor the N_t, is there any implication on what kinds of time steps should have more pondering steps and how does the model’s choice match with the expected pondering time steps? Or  at least, if we are in an unsupervised setting so that we don’t really have the “expected pondering time steps” to check the quality of adaptive layers, we should run multiple replicas of the model with different parameter initializations and show that the distribution of N_t on each of the time steps are not uniform. If the “pondering time” (N_t’s) are playing a role in processing the data, they should at least show some patterns that are related to the corresponding input, either decipherable or not. \n\nLearning through a different number of layers between different time steps is not a novel idea. For example, the TARDIS model (https://arxiv.org/abs/1701.08718) has set the number of hidden layers variable. \n\nIn general, I think the authors have provided an interesting idea and the experiments are well performed. I’d expect the authors to isolate more factors from the model in order to show the effectiveness of the adaptive layer mechanism.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1li9bECtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes to use adaptive computation time (ACT) for the number of layers (instead of the number of timesteps in the original paper) for a RNN. The intuition is that some inputs are more complex and may need more processing compare to other inputs. \n\nComments on the paper,\n\n-  I am slightly confused over figure 2. For example, fig 2.c is the model performance over different steps. Does the number of steps here mean the number of time-steps per sequence? It seems that the performance of the proposed model and the RNN baseline matches at step 13. It seems to be natural to run RNNs over a sequence length of 13. it is not clear to me what is the advantage of using the proposed model in this case. \n\n- It is also unclear to me how many layers does the RNN baseline have and how the results changes wrt the number of the layers for the baselines, i would imagine that this would get better as the number of layers increases. Can the authors compare the proposed model to RNN baselines trained with different number of layers?\n\n\n- It is unclear to me what RNN-based models are, it seems that they are GRU models reading from section 4.1, but it seems to be implicit. Is there a reason to use GRUs compared to LSTMs? Do they achieve similar results?\n\n- I am not sure the proposed datasets have been picked. Is there a reason why each dataset is picked, what would the authors expect to see (hypothesis to test) from each dataset?\n\n- Some analytical experiments to help better understand the model would be nice. For example, some datasets would need more computational resources for some steps, but not others. Would the model be able to learn and pick that up? A really simple example if the copy task, would the model to learn to just have minimal layers for the 0's that does not contain information and would it use more layers for the steps that needs more processings (the non-zeros for example).\n\n\nMinor comments:\n\n1. some typos. P6, section 4.1, \"length\" was incorrectly spelled.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xYOcu0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose Layer Flexible Adaptive Computation Time, an RNN-esque sequence model with varying depth at each time step. The idea is that the model can adaptively choose how much computational effort to spend on each example. The authors evaluate the model empirically on a financial dataset and Wikipedia language modeling, and find that it outperforms a vanilla RNN and the original adaptive computation time (ACT) model.\n\nUnfortunately, the presentation of the idea is unclear, the idea itself is not very novel, and the experimental evaluation is lacking. These weaknesses lead me to vote for a weak reject.\n\nI address specific clarity points below.\n\nIn regards to the novelty claim, there have been several developments of depth-based (as opposed to time-based) adaptive computation time in the literature, for example:\n\n[1] McGill et al 2017 \"Deciding How to Decide: Dynamic Routing in Artificial Neural Networks\"\n[2] Bolukbasi et al 2017 \"Adaptive Neural Networks for Efficient Inference\"\n[3] Figurnov et al 2017 \"Spatially Adaptive Computation Time for Residual Networks\" (this paper is cited by the authors)\n\nThese papers do not present sequence models, but the ideas in them readily apply to sequence models. Thus, the main novelty in the authors' paper is handling the different number of hidden states at each timestep via their 'attention mechanism'. While this is definitely a contribution, the unclear presentation and lacking experimental evaluation combine to decrease the value of the paper.\n\nExperimentally, the authors evaluate on a financial time-series dataset and Wikipedia language modeling. They compare to Adaptive computation time and a standard RNN. While the experiments demonstrate a modest improvement over ACT and an RNN, they do not compare on larger, more standard datasets such as the WMT datasets, etc... Additionally, they do not compare with other, more powerful models. Both are required to thoroughly demonstrate their model's effectiveness.\n\nTo change my mind the authors would have to (in order of importance):\n1) Add experiments on WMT or other bigger datasets and compare with current SOTA models.\n2) Thoroughly edit their paper for clarity (specific points below).\n\nThe authors may want also add explorations of how much computation time can be saved using their model versus others, as this is very common in the ACT-esque literature.\n\nSpecific points:\n* While the original ACT paper does use 'mean-field' to denote the convex combination of states at a current timestep, that term has a specific technical meaning different from how it is used here. I would suggest using a different word.\n* The introduction is too long and repeats itself in several places.\n* There are repeated citations in the literature review.\n* In figure 1 sometimes nodes denote functions and sometimes nodes denote outputs. Sometimes the nodes are round and sometimes they are rectangular. Sometimes arrows denote inputs to a function, and sometimes they denote multiplication. These inconsistencies make the figure very hard to decipher.\n* The text description of your model is confusing. Specifically, distinguishing between the functions of u_t, \\hat{u}_t and \\overline{u}_t was difficult.\n* In your experiments section the plots are very difficult to interpret because they are phrase as 'improvement over x'. The standard presentation is a table of absolute results. Furthermore, bar charts can be misleading because the scale can make improvements seem bigger than they actually are. The figures should stand alone without having to read the text.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers assessed this paper as a weak reject.
The AC recommends rejection.
Decision: Reject