GATO: Gates Are Not the Only Option
Authors: Mark Goldstein*
Xintian Han*
Rajesh Ranganath
Venue:  (2020)
Keywords: Sequence Models
Vanishing Gradients
Recurrent neural networks
Long-term dependence

Abstract: Recurrent Neural Networks (RNNs) facilitate prediction and generation of structured temporal data such as text and sound. However, training RNNs is hard. Vanishing gradients cause difficulties for learning long-range dependencies. Hidden states can explode for long sequences and send unbounded gradients to model parameters, even when hidden-to-hidden Jacobians are bounded. Models like the LSTM and GRU use gates to bound their hidden state, but most choices of gating functions lead to saturating gradients that contribute to, instead of alleviate, vanishing gradients. Moreover, performance of these models is not robust across random initializations. In this work, we specify desiderata for sequence models. We develop one model that satisfies them and that is capable of learning long-term dependencies, called GATO. GATO is constructed so that part of its hidden state does not have vanishing gradients, regardless of sequence length. We study GATO on copying and arithmetic tasks with long dependencies and on modeling intensive care unit and language data. Training GATO is more stable across random seeds and learning rates than GRUs and LSTMs. GATO solves these tasks using an order of magnitude fewer parameters.

Under review as a conference paper at ICLR 2020
GATO: GATES ARE NOT THE ONLY OPTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Recurrent Neural Networks (RNNS) facilitate prediction and generation of struc- tured temporal data such as text and sound. However, training RNNN hard. Vanishing gradients cause difficulties for learning long-range dependencies. Hid- den states can explode for long sequences and send unbounded gradients to model parameters, even when hidden-to-hidden Jacobians are bounded. Models like the [STM] and GRU use gates to bound their hidden state, but most choices gating functions lead to saturating gradients that contribute to, instead of alleviate, vanishing gradients. Moreover, performance of these models is not robust across random initializations. In this work, specify desiderata for sequence models.
We develop one model that satisfies them and that is capable of learning long-term dependencies, called GATO GATO is constructed so that part of its hidden state does not have vanishing gradients. regardless of sequence length, and so that its state does not explode. We study GATO on copying and arithmetic tasks with long dependencies and on modeling intensive care unit and language data. Training GATO is more stable across random seeds and learning rates than GRUS and LSTMS
INTRODUCTION
RNNS allow model structured data like text, image, and sound from domains such as art, healthcare, and environmental sciences. These models and their deep variants form the basis for tasks such generation and prediction (Pascanu al. (013). Modeling joint distributions of sequences facilitates the generation of images (Oord et al. 2016). videos (Denton & Fergus, 2018), drawings (Ha & Bck] (017), and jazz. music (Huang et al. 2019). Modeling conditional distributions enables applications like predicting patient outcomes in medical centers with electronic health records (Che) (et al. 2018). In all of these cases, the data generation process may involve dependencies that range over hundreds of time steps (e.g. number of video frames health over a lifetime) where early events may determine the distribution of later events (Kamiya et al. 2015 Denton & Fergus 2018).
The basic RNN (Elan 1990) suffers from vanishing and exploding gradient problems (Bengio et al. 1994) that cause difficulties for learning long-range dependencies. The problems stem from extreme eigenvalues of parameter matrices in the RNN Such models can also suffer from unbounded growth or shrinkage of the hidden state over long sequence lengths. This can happen even when the hidden-to-hine to-hidden Jacobian does not vanish or explode and contributes additional difficulties for learning recurrent and decoder parameters. Careful parameterization and initialization of RNNN is crucial for their success in learning long-range dependencies using gradient- based optimization.
The majority of proposed solutions make use of RNNS with gated units, notably LSTMS (Hochreiter & Schmidhuber 1997). ). GRUS (Cho et al.) 2014) and Recurrent Highway Networks (RHNs) (Zilly all]][201 2017). These models and deep variants have been the basis for many state-of-the-ar empirical results. Gated architectures address gradient issues by bounding the hidden state and designing the recurrence copy the state forward in time. The GRU and RHN update their state with weighted average of bounded updates and the current state, where the weighted average in GRU is a convex combination. The LSTM uses an auxiliary cell state with linear recursive updates to copy the hidden state forward. However, gated models are difficult to optimize in practice, in part due to the saturating gradients of gating non-linearities.
An alternative to gated models is to restrict hidden transformations to be unitary (Arjovsky et al.) 2016 Wisdom et al. 2016 Jing et al., [017 Mhammedi et al.) (017). Unitary matrices can be safely raised to powers without vanishing gradients since they have modulus 1 eigenvalues. Besides unitary transformations, Zhang et al. (2018) controls the singular values of the transformation using SVD avoid vanishing and exploding gradient problems. But, these approaches often impose computational

---

Under review as a conference paper at ICLR 2020
overhead to ensure the constraint and require non-trivial parameterizations. These approaches only address hidden-to- to--hddeeenn gradients. In this work we discuss other properties, such as exploding hidden states in forward propagation when the input is constant for long periods.
We propose a sequence model called GATO The key insight in GATO is that only part of the hidden state must have non-vanishing gradients for states in the far past to influence the current state. GATO splits its hidden state into two channels. At each time step, each channel is updated by a function of the input and current hidden state. To ensure that one of the channels will not have vanishing gradients. GATO limits the functions for both channels to only depend on one of the channels. This results in an identity-block structure in the hidden- tiiddenn Jacobian. GATO then uses non saturating functions, such as the cosine, to bound the hidden state upon input to the decoder to stabilize gradients for model parameters. We use a simple version GATO where the dimensions within the channels do not interact and show that it can model arbitrary conditional distributions through memorization. Partitioned hidden states have been mentioned in (Oliva et al. 2017) and (Zhang et al., 2018). Oliva et al.
2017) can guarantee the identity Jacobian when = but in this case the state stays constant.
We study CATOOcopying and adding on long sequences. GATO solves the copying task while the others do not Figure T). On adding, the GRU experiences difficulty as sequence length increases while GATO does not (Figure 2). We also test GATO on health and language data. Using a critical care database collected in Boston (Johnson et al. 2016)) GATO predicts patient length-of-stay in the intensive care unit more accurately than GRUS L across seeds and learning rates. [ATO achieves lower held-out perplexity on language modeling using the Penn Treebank dataset. We explore this in a setting with no methods such as dropout or gradient clipping. GATO is stable to train across random initializations compared to GRUS LSTMS and other recently proposed (RNNs such as
the SRU Oliva et al. 2017). EURNN Jing et al. 2017). and RIIN (Zilly et al. 2017).
2 BACKGROUND ON SEQUENCE MODELS
Sequence modeling targets learning the sequence of conditional distributions that constitute the joint. Let (x1, X..... X)) be sequence of T observations with joint distribution Q(x). We consider the standard probabilistic model po (x) with parameter 8, where each sequence element depends on all previous elements. Let Le at timestep & be the negative conditional log probability of the next sequence element according to the model:
T-1
Pe (x) = (x1)
II t=1
pa px (x
(0) log po (X++1 (x<:);
[(0) - -
To minimize L(6) on samples from Q, we need expressive pe to capture complicated conditionals.
2.1 SEQUENCE MODELING WITH NEURAL. NETWORKS
Sequence models often represent conditionals using a hidden state that evolves as a sequence of data is observed. Using a function D to map from states to distribution parameters, we define the conditional probability p(x++1 (x{x) with parameters D(h.). To represent the true conditionals of Q h must summarize the information xj...X; x; useful for predicting x>t- Typically, D is chosen parameters, W be data-to-hidden
to be feed-forward neural network. Let U be hidden-to-hiiee parameters, and a be an element- wise non-linearity. RNNN proceed by parameterizing the dynamics of the hidden state via the following discrete-time recurrence:
h o (Uh, + Wx++1).
Though RNNS can represent broad class of sequences (Schäfer & Zimmermann 2006 Siegelmann & Sontag 1995), they face problems in modeling long sequences where an early part of the sequence influences a later part. On one hand, modeling such data requires the use of powerful functions, but on the other, gradient-based optimization of such functions is unstable and requires manual tuning.
2.2 RNNs HAVE PROBLEMS
Vanishing and Exploding Gradients. Consider states h, and hy with times t < T. Explod- ing gradients occur when by changes dramatically for small changes in h4, while vanishing gradients occur when the influence of h2 hr decreases exponentially with gap T -t.
Consider the RNN as in Equation (1) and recall that o is an element-wise wise non-linearity Let Ey be the loss at time T. Let D. diag(o (alggo(Uhh,- + Wx.)) be the Jacobian for Suppose that À is
2

---

Under review as a conference paper at ICLR 2020
lower-bound on the Insolute-valuee of the Jacobian D's entries. Then if the smallest singular value of U is larger than 1/A, 0hr/ /0, blows up exponentially as time gap T - -t increases. Suppose instead that À is an upper-bound on the absolute-value of D's entries. Then if the largest singular value of U smaller than 1 1/x, @hrr/hh decays exponentially with time gap T t. The Jacobian Ohr//hr involved in LT/0U is
OCT 0U
OCr ahr Th, Σ Ohr dh, 00
and can stop losses Lr from backpropagating to U. These issues cause short-term dependencies to be favored over long-term ones (Bengio et al. T994. Chung et al. 2014).
Exploding Forward Computation. The hidden state can also grow without bound or vanish for long sequences. Consider the following recurrence where a denotes sigmoid:
z 11 o (W X +
z(²) o( X1-1 U.):
h tanh W X
U hi); h = zz()) h, + (2) @ h.
The state h can grow without bound. Two examples of systems with good hiiden gradients but that have unbounded forward propagation are RRH Ziilyy et al. 2017) and (Efficient) Unitary RNN (Arjovsky et al. 2016 [ing et al., 2017). They are described in Appendix Gland Appendix H
These models NAN during forward propagation in our experiments.
2.3
LONG SHORT-TERN MEMORY AND GATED RECURRENT UNITS
To improve gradients, Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014) augment RNNS with additional computation that helps bound the state as well as reduce gradient vanishing. The LSTM retains a cell state that copies the hidden state forward in time (Appendix E The GRU uses a convex combination of the current state and a bounded proposal. Its state is bounded when initialized correctly (Appendix F).
Grejuees
This variety of neural sequence models has produced state-of-the-art performance in machine transla- tion (Sutskever et al. 2014 Luong et al.) 2015) and image captioning (Xu et al. (2015). However, these models are difficult to optimize. Practitioners must experiment with optimization techinques including gradient clipping, randomly dropping-out hidden state updates (Krueger et al.) 2016), and even training diseriminators to keep generated and data sequences close (Lamb et al. 2016).
Empirical investigation into the behavior of the LSTM gates has also shown that forget and input gate values tend toward 0.5, contradicting explicit interpretations as gates (Li et al.) 2018b). Recent analysis has shown that LSTMS still have gradient issues that cause them to favor learning short-term dependencies in data (Kanuparthi et al. 2019). See Appendix J for a brief discussion.
DESIDERATA FOR SEQUENCE MODELS
We now develop criteria for building sequence models and consider several important gradients.
Hidden-to- Hiddenn gradients. For sequence models to learn long-term dependencies, part of the hidden state at time needs to have non-decaying influence on states at later times T. Therefore only some, not all, of the hidden units need avoid vanishing gradients. Our first criterion is:
Part (a block) of the Jacobian /2h, does not depend on time gap T -t
Decoder Gradients. Let be non-linear function and consider loss time t defined through linear multi-output decoder (h;)) = W>(h.. Then the gradient of the loss with respect to W is:
0L./ ew (BC./8D)(0D/ow) (OC,/)D)>(hh)
To ensure that ||ac,/o/ [o|| does not become too large, one property that y needs have is:
2. y is bounded to ensure that decoder gradients 0D/ /äW do not grow with time.
Loss-to-Hidden Gradients. aC,/0h, for one time step & also depends on the choice of through its gradient 7/ It is defined through the decoder by
ac,/Oh, (0C./0D)(0//0h) (a00////00)-/(((////0000 (WT(OL//DD) ) (bi))
This means that:
3

---

Under review as a conference paper at ICLR 2020
3. ((e))) should not converge to 0 as ||h/|| grows (if h, does grow).
4. The magnitude ||'|| evaluated h, is bounded even when [hr] increases.
These criteria require bounded, differentiable functions whose gradients do not go to 0 as their input increases. There a relationship between the particular form of h's update and the choice of 7. For example, if the update contracts to a constant c from both the positive and negative direction, then should be well-behaved at с (e.g. non-zero gradients). These criteria rule out traditional choices of non-linear functions that saturate, such sigmoio id and t tanh, and functions that grow such as Softh us and ReLU. Periodic functions such as cos and s sin meet these criteria.
Forward Computation. In tasks such as language modeling, the hidden state can be processed for thousands of steps through the corpus. Let r2 denote the part h, involved the recurrence to compute h Then if ||r is large at time t during training, || will be large and can cause additional issues in learning recurrent parameters
5.
||r||| in general should not become large as /. increases.
4
GATO
We propose new sequence model that meets the desiderata GaTO GATO is capable of learning long-range dependencies and is stable to train, exhibiting little variance across learning rates and initializations. [ATO requires fewer parameters to perform well on tasks such as copying, arithimetic, classification, and language modeling. We describe the model and analyze its gradients.
4.1 GATO MODEL
Building on the ideas in Section GATO partitions the the hidden state into two parts, one which is guaranteed to have good long-tern gradients by construction. The updates of the hidden state impose structure on the hidden-to-hidder Jacobian wherein a sub-matrix is the identity I. This means that parts of /hr/Oh, do not dependent T / and therefore do not vanish with time.
Partitioned Hidden State. The hidden state h, has two channels r; and s,. At each time step, both channels are updated by non-linear function of the previous state. To ensure that one part of the state, 8,, has time-independent and ge be non-linear functions with parameters 6 and s/. GATO
gradients, the updates do not depend on s. Let So hidden state evolves as:
1 (r,, Xe+1);
S St Go (re, Xx+1);
h
[1141,8:4
This is like an RNN where the hidden state update only function of part of the state, and where the updates of the other part are defined by residual connections (He et al. 2016 Wang & Tian 2016). In our experiments, r and s are the same size. In Apppendix A we experiment with s smaller than r.
4.2 GATO HAS GOOD GRADIENTS
Hidden-to-Hinne Hidden Gradients. As shown in Section 2.2 RNNS suffer from vanishing gradients.
suffer
We show that the GATO hidden state recurrence does not have vanishing gradients for part of the hidden state. First consider ØST /ōs, and recall the evolution of and s:
rT =
X++1))
(rk, ST = s, 9 Xx+1)
The right hand side for rr and the second term of the right hand side for Sy do not depend on St, so Øst/ /0s is the identity matrix I. Next, drr/Os, 0 because is not included among the arguments to the update fo (r,, x.). The whole Jacobian matrix of hidden states hy with respect to h, is:
ahr Oh
OST 0s.
dr.
I
Og ( Org ΣT= - (fs(re
Ory.
Grr
Grr
Dr:
drs
Since 0sr/Os, = I does not depend on time difference T -t t, part of the hidden state, Sy, retains its influence from st (criterion []. Though Øsr /Or, and Orr/Or, complicated, dsy/Os, is guaranteed not to vanish so gradients can be propagated back through time. In Inpppendix A] we explore the effect of removing this feature and find that model performance suffers.
4

---

input in g(f) when the count rlf)) equals j. A powerful decoder can then read the memorized sequence from the hidden state and compute a conditional Under review as a conference paper at ICLR 2020
Decoder and Loss-to-Hidden Gradients. We have just shown that the hidden state channel s; gradient does not vanish. As discussed in Secction 3] there are additional optimization issues that can be introduced by the hidden state. We focus 8t- In the of loss defined through decoder ((s,) Ws(S)): : 7 must be chosen to ensure that [0C,//W] and [@Lr/Osi] do not shrink or grow, regardless of ||s]]] In the experiments, GATO ensures this by using (8x) cos(ss). We experiment with sin in Appendix B.2
4.3 NON-INTER INTERACTAC HIDDEN UNITS
Consider a version of GATO where interactions are limited to pairs of units. Let j index pairs of hidden units and let the hidden state size be 2J. Then the updates for each pair (rO), ; s(1) are:
( r++1 1 xx+1);;
6 S11 -
(3)
(i) 94 (re Xx+1).
Non-interacting units in RNN hidden states have been explored previously Li et al. 2018a). We show that this version of GATO can represent arbitrary conditionals by memorization. Consider sequence model of scalar data with conditionals defined via hidden states p( ((gge) = p( (ss,, rt)). For this model to represent arbitrary conditionals, the hidden state needs to store Xgr. One way for GATO in Equation (3) to memorize its inputs x would for each y() to learn how to count and each function Ug to store its one-dimension
(2)
distribution. This construction requires the size J to be at least the length of the sequence T.
4.4 TWO EXAMPLES OF NON-INTERACTING GATO
Let RD denote the input time I. Let h, [rr, St. denote the hidden state with & S, R. Let A, B, C E R and a, b, c E Rd be parameters. Let a be nonlinear function whose maximum absolute value is 1. Let X E E.
B,C
One layer GATO
St+1 = S; SoftPLus(Ax + Oa)r:
(4)
r++1 = o o + bbrr) Or tanh + +cOr).
Let F(r, x) denote a neural network with inputs r, x such that each output dimension Fj is hidden layer neural network with ReLU activation and hidden size that only depends on ry and x. Then two-layer GATO defined as:
Two layer GATO
8 S; (F (x.)..)))): r++1 = X x (Ax141 + aa Or tanh + +brr).
(5)
We call the (-) term the regularizer. both models, setting À to number less than one helps
meet criterion 5 in /eeciion 3 To see this, assume 0.7 and that takes its maximum value 1:
Fr+1 0.7 r tanh (aanhBBx+111 + b rz).
Now assume that the tanh takes on either of its extremes, for example 1. Then the update is r = - +1 1 and any dimension of r that is greater than 10/3 will be shrunk towards 10/3. We set the sigmoid in the main experiments (Section 5) and to tanh infppeedix B.I
EXPERIMENTS
We study GATO on two standard sequence tasks, copying and performing arithmetic on long sequences (Hochreiter & Schmidhuber 1997 Arjovsky et al., (016). We compare against several kinds of NNSS recently proposed alternatives. In these tasks, the models train samples from the data distribution each batch. There is notion of overfitting these experiments. These experiments show where common sequence models fail.
ranging from well-established directly
We then use GATO for hospital length of stay prediction using electronic health records and on language modeling. For these tasks, we report held-out metrics (accuracy and perplexity), though do not focus regularizing sequence models in this work. We explore this in setting with gradient clipping (which could stabilize training) or dropout (which could reduce overfitting). We run experiments across three random seeds and choices of learning rates.
5

---

Under review as a conference paper at ICLR 2020
5.1 MODEL AND BASELINE SPECIFICATIONS
For baselines, we use PyTorch modules rin GRU and nn LSTM (Paszke et al. 2017). Since GATO has fewer parameters than these models for given hidden size, we consider two alternative baselines. We use diagonal weight matrices for LSTM and GRU
GATO weight matrices are also diagonal). When we set them to have the same number of parameters as GATO rather than the same hidden size, we call them GRU-DIAG (same param) and LSTT--IAG (same param). We also compare against [RR] (Appendix 1). RHN (Appendix G). and EURNN [Appendix H.. For GATO| we choose in the update regularization term to be the s rrigmoic id and choose À 0.7. In our experiments, we find À 0.5 performed similar to 0.7.
We call these models GRU-DIAG and LSTM- LIST--IAGI
We give exact numbers of RNN parameters for each experiment. Embedding and decoder parameters are constant across models and excluded in counts. More details including initialization are described in Appendix C All models trained across three random seeds for all tasks. The error bands in all plots represent the minimum and maximum across seeds with a bold line representing the mean.
5.2 EXPERIMENT 1: LONG COPY TASK
Task Description. We model sequences of the form ABA, where A is a sequence of length 20 with tokens uniformly drawn from {1...., 10} and B is 100 constant BLANK tokens. The model must predict the second A as a copy of the first. The hidden state must memorize the first 20 tokens and
keep them in memory while processing the 100 BLANK's. This a harder variant of the copy task in (Arjovsky et al. 2016). We do not use a special token to mark the beginning of the second A region.
Experiment Setup. All models use an input embedding size of 4, recurrent hidden size of 1024, and conne-co decoder with one hidden layer of size 256. We use two layer GATO GATO has 138, 752 parameters,whin the GRU and LSTM havve 3, 118, 080 and 4, 222, 976, respectively. We draw new sequences from the data distribution for each step of training. We evaluate on a held-out set of 1000 sequences. We let each model see million training points.
the probability of the tokens to be copied is near chance (0.10) across the three seeds. GATO learns the task with at least .50 probability averaged across the held-out set for all tokens to be copied. GRU-DIAG beats other baselines when excluding EURNN
Results. We show the results in Figure For the LSTM Lnd GRU
EURNN NAN' ed one seed (not shown) and was unstable for the other two.
GATO
LSTM
LSTM-DIAG
LSTM-DIAG (same param)
SRU
GRU
GRU-DIAG
GRU-DIAG (same param]
EURNN
RHN
Probability
0.6
0.4
0.2
0.0
0
200
400
600
800
1000
Number of Training Examples Seen (Thousands)
Figure 1: GATO learns to copy long sequences with an order of magnitude fewer parameters than the GRU and LSTM We plot average probability copied tokens on a held-out set. Error bands correspond to random seeds. The The/GRU and LSTM output probabilities near chance for the copied tokens. GRU-DIAG with same number of parameters as GATO beats other baselines when excluding EURNN EURNN NANed on one seed (not shown) and was unstable for the other two.
5.3 EXPERIMENT 2: ADDING PROBLEM
Task Description. This experiment follows the adding task described in (Hochreiter & Schmidhu ber 1997). In this task, the input consists of two sequences of length T. Each element of the first sequence is sampled from the uniform distribution 2/ 1]. The second sequence plays the role of an indicator. It is all 0's except for two 1 's.. The location of the first 1 is sampled uniformly from the first half of the sequence, while the location of the second is sampled uniformly from the second half of

---

Under review as a conference paper at ICLR 2020
the sequence. The task is to predict the sum of the two numbers in the first sequence whose locations correspond to the locations of the I's in the second sequence. The mean squared error of predicting the expectation, 1, is 0.167. This forms the baseline.
two-layer GATO Loss is measured by mean squared error. We do not use embeddings for this task, so the input size at each timestep is (uniform variate and indicator). All models use recurrent hidden size 512 and a decoder with one hidden layer of size 256. For recurrent parameters, [ATO has 43,264 parameters while the GRU and LSTM have 792,576 and 1,056,768 respectively. To make a prediction. we apply the decoder to the RNN hidden state after processing the sequence. We sample new data points at each step of training. We evaluate on a held-out set of 1000 examples. We run experiment with sequence lengths T = 100, 200, 400, 750.
Experiment Setup. We compare all models with the baseline and
Results. Our results are shown in Figure After training with 200,000 examples, the mean squared error of the LSTM and both ASST-DIAGE was still around the baseline. [GRU and the GRU-DIAGG learned for T=100 and 200. On T=400, GRU had NAN error one seed (denoted by infinite loss). The GRUJ did not learn the task on 75750, but GRU-DIAG did. Neither [SRU nor RHN could learn the task, with missingness in the plot resulting from infinite and NAN losses. GATO (magenta line)
learned this task at all lengths and was stable across random seeds, also learning with fewer examples.
Baseline
LSTM GRU-DIAG LSTM-DIAG
GRU-DIAG (same param)
RHN
GATO
LSTM-DIAG (same param)
SRU
GRU
EURNN
Sequence Length T 100
Sequence Length T 200
0.5
0.5
Mean Squared Error
0.4
Mean Squared Error
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Sequence Length T 400
Sequence Length T 750
0.5
0.5
Mean Squared Error
0.4
Mean Squared Error
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0
50 100
150
200
200 Number of Training Examples Seen (Thousands)
0
50
100
150
Number of Training Examples Seen (Thousands)
Figure 2: GATO is more robust across different seeds and sequence lengths. We show the mean squared errors of the adding problem for sequence lengths T 100, 200, 400, 750 over number of training examples. LSTM failed to learn the task. For length T 400, the GRU numerically failed on one seed (NAN error). We denote this by infinite error. GRU-DIAG
learn the task with more
samples than GATO SRU EURNN and/RHN could not learn this task (NAN and/or worse than chance).
NANS and errors greater than 0.5 (significantly worse than chance) are not shown (c.g. EURNN T=750).
7

---

Under review as a conference paper at ICLR 2020
5.4 EXPERIMENT 3: LENGTH OF STAY THE INTENSIVE CARE UNIT (ICU)
Task Description and Experimental Setup. This experiment uses MIMIC-III publicly-availabee Boston-based critical care database. The task is to predict if a patient stays more than I week in the ICU unit using the first 48 hours of their vitals. There are 53,211 patients in total, 27,088 of whom stay more than one week. We use the vitals extracted for Interpolation- iiterpolation-Preditttoo Networks (Shukla) & Marlin 2019). See Appendix for more details. We split the data into a training set of size 43, 50,000 validation set of size 5,106, and test set of 5,105. We use cross entropy loss. To make prediction, we apply a decoder to the RNN hidden state after processing the sequence of vital signs. We use a recurrent hidden size 512 and decoder hidden size 1024 for all models. We use one-layer[GATO
We report average test accuracy across seeds Table GATO has higher accuracy than LSTM and their diagonal variants on this task across two learning rates. RHN performs in between the GRU and LSTM variants. SRU performs best on this task, but has more parameters for the same hidden size. The ranges performance for GATO and SRU SRU overlap.
Results.
GRU
Table GATO performs well on test accuracy with both learning rates 4e-3 and 1e-4. We show the accuracy of length of stay prediction averaged across seeds, [SRU performs best on this task. The ranges in performance for GATO and SRU SRUURRROOsr overlap. EEURNN had an NAN error on all seeds shortly following the reported accuracies.
Model
# Parameters
Test Accuracy, LR 4c-3
Test Accuracy, LR le-4
GATO
20,736
0.693
0.646
GRU
827,904
0.519
0,569
LSTM
1,103,872
0.650
0.569
41.472
0.505
0.506
LSTM-DIAG
55,296
0.633
0.506
GRU-DIAG (same params)
21,504
0.570
0.505
LSTM-DIAG
(same params)
22,400
0.569
0.506
SRU
315,282
0.700
0.683
EURNN
14,848
0.658
0.610*
RHN
827,904
0.619
0.570
5.5 EXPERIMENT 4: LANGUAGE MODELING ON PENN TREEBANK
Description and Results. A language model predicts the probability distribution of the next word sentence based on the previous words. We test GATO on the Penn Treebank dataset preprocessed by Mikolov et al. (2010). We choose hidden size 1300, embedding size 650 and a linear decoder for all the models for this task. We use one-layer GATO in this experiment. Table shows model
perplexity. GATO achieves similar perplexity to GRU RHN gets infinite loss because of its unbounded hidden state and the long sequence lengths on this task. Excluding the infinite loss, SRU and EURNN
have the highest test perplexity. By applying principles from GATO we bound RHN hidden state and call BRHN
BRHN performs similarly to GATO and GRU
CONCLUSION AND FUTURE WORK
In this work, we study criteria for building sequence models that can capture long-term dependencies. We introduce one such model, GATO GATO uses partitioned hidden states. One part of the hidden state modified, but never used to update future states. This ensures that part of the hidden state does not have vanishing gradients. We explore a variant GATO with non-interacting hidden units and show it is sufficient in our experiments. We also introduce criteria for avoiding unbounded forward propagation in sequence models. Empirically, such explosion occurs in our baseline models. GATO performs as well as GRUS and LSTMS in four typical sequence modeling problems: copying, arithmetic tasks with long-term dependencies, classification, and language modeling. We made similar comparisons against GRU and LSTM by matching the number of parameters instead of the hidden size. All alternative models (SRU RHN EURNN), demonstrate significant instabilities. In RHN] we identify that its hidden state is unbounded and use principles from this work to improve its performance.
8

---

Under review as a conference paper at ICLR 2020
Table
GATO outperforms GRU and LSTM We calculate the perplexity averaged over three seeds on Penn Treebank language modeling in the unregularized setting. GATO achieves similar perplexity RHN] gets infinite loss. Excluding the the|RHN SRU and EURNN have the highest test perplexity. *By applying principles from GATO we bound RIIN hidden state make it perform similarly to GATO and GRU (called BRIIN).
GRU
Model
Parameters
Test Perplexity Score
GATO
1,273,350
112.85
GRU
7,612,800
115.78
LSTTM
10,150,400
122.86
GRU-DIAG
2,546,700
124.44
LSTM-DIA
3,395,600
119.09
GRU-DIAG
(same params)
1,273,350
124.65
LSTM-DIAG
(same params)
1,306,000
122.01
SRU
2,198,560
149.97
EURNN
850,200
768.96
RTN
7,612,800
BRHN
5,075,200
116.11
Future directions include understanding when forward propagation is stable and how the principles used to develop GATO can be used to design feed-forward neural networks. Since several years of
research have been devoted to investigating regularization of GRUS and LSTMS important direction
to see whether the same techniques apply to alternate models such as GATO others. Studying combinations of GATO and other (deep) variants of RNNS such Phase LSTM
SRU
FURNN and
(Neil)
iet al. (2016). Dilated RNN (Chang et al. (201)), Skip RNN (Campos et al. 2018), Fast-slow RNN
(Mujika al. 2017) and Deep RNN Pascaan et 2013) is another interesting direction.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pp. 1120-1128, 2016.
Yoshua Bengio, Patrice Simard, Paolo Frasconi, et al. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Víctor Campos, Brendan Jou, Xavier Giró Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: Learn- ing to skip state updates in recurrent neural networks. In International Conference Learning Representations, 2018.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetricRNN: A dynamical system view recurrent neural networks. In International Conference Learning Representations, 2019.
net/ www.:nneeeorrmmmleeeeeeeooooomee
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprini arXiv: 1406. 1078, 2014.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint
arXiv: 177017807..77, 2018
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 1990.
David Ha and Douglas Eck. A neural representation of sketch drawings. arXiv preprint arXiv:1004.03477 2017.
2017
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu. and Douglas Eck. Music transformer. In International Conference on Learning Representations, 2019. httpp/
//openrevien eww net for.foormm///mm deeeeiddrreeeeheeeeee
Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljacic. Tunable efficient unitary neural networks (eunn) and their application to rnns. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3: 3:160035, 2016.
Kenji Kamiya, Kotaro Ozasa, Suminori Akiba, Ohstura Niwa, Kazunori Kodama, Noboru Takamura, Elena K Zaharieva, Yuko Kimura, and Richard Wakeford. Long-term effects of radiation exposure on health. The lancet, 386(9992):4499477 2015.
Bhargav Kanuparthi, Devansh Arpit, Giancarlo Kerg, Nan Rosemary Ke, Ioannis Mitliagkas, and Yoshua Bengio. h-detach: Modifying the Istm gradient toward better optimization. In International Conference on Learning Representations, 2019.
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing mns by randomly preserving hidden activations. arXiv preprint arXiv: 01305, 2016.
10

---

Under review as a conference paper at ICLR 2020
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang. Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp. 4601-4609,
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indron): Building a longer and deeper rnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5457-5466, 2018a.
Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-yan Liu. Towards binary- valued gates for robust Istm training. In Proceedings of 35th International Conference on Machine Learning, 2018b.
Minh-Thang Luong, Hicu Pham, and Christopher D Manning. Effective approaches to attention-base neural machine translation. arXiv preprint arXiv: 04025, 2015.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTT
language models. In International Conference on Learning Representations, 2018. URL https
//openreview.oo net/ faee/forre ftrumoirmmm SyyGPPOTZ
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2401-22122000 2409.. JMLR. org. 2017.
Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systemx, pp. 5915-5924, 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in neural information processing systems, pp. 2882-3890, 2016.
Junier B Oliva, Barnabás Póczos, and Jeff Schneider. The statistical recurrent unit. In Proceedings of the 34th International Conference on Machine Learning-Volas Volume 70, pp. 2671-2680, JMLR. org, 2017.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 161.0796, 2016.
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6000 2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In s1 Conference on Neural Information Processing Systems, Workshop paper, 2017.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In International Conference on Artificial Neural Nenworks, pp. 632-640. Springer, 2006.
Satya Narayan Shukla and Benjamin M. Marlin. Interpolation networks for irregularly sampled time series. In International Conference on Learning Representations, pp. 18800-4888 2019.
Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. Journal of computer and system sciences, 50(1): 132-1500, 1995.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Yiren Wang and Fei Tian. Recurrent residual learning for sequence classification. In Conference on Empirical Methods in Natural Language Processing, 2016.
11

---

Under review as a conference paper at ICLR 2020
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances Neural Information Processing Systems, pp- 4880 4880- 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXhv:1502.03044, 2015.
Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning, pp. 5801-5809, 2018.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189 4198. JMLR. org, 2017.
12

---

Under review as a conference paper at ICLR 2020
A
GATO CANNOT LEARN ADDING WITHOUT LONG-TERM GRADIENT
Recall that GATO partitions its hidden state as h, = (et, re] and that S, and r2 are updated by:
Tt+1 fo(x
S = St Iy (x
We experiment with two ablations of GATO for the adding task and show that the long-term gradients through 08 are necessary for its performance.
1. Set the update for s to O. This makes GATO close to a standard RNN for half of its state.
2. Set the update for Pe- This is standard RNN with half of its state but where the decoder can use go to compute an additional function of the state r.
Both of these ablations lose GATOSS identity block matrix in the Jacobian 0hr /h, In these experiments, GATO does not learn the adding task for sequence length 750 above chance for either ablation, across three seeds. We perform a third ablation by keeping s, but reduce its size:
3. Set be 1/4 of the hidden state instead of 1/2.
We find that 1/4 is less stable for the adding task (Figure 3) and performs worse for Penn TreeBank Table 3). This suggests that having larger fraction of the state devoted to long-tems longtterm gradient propagation is important.
Sequence Length 100
Sequence Length 200
0.5
0.5
Baseline
Mean Squared Error
0.4
Mean Squared Error
0.4
50% state (original)
25% of state
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Sequence Length T= 400
Sequence Length 750
0.5
0.5
Mean Squared Efror
0.4
Mean Squared Enor
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Figure 3: Both sizes of work well for the add task. 50% is more stable, suggesting long-term gradient propagation important for this task. This is consistent with the findings in Appendix where 0% failed on this task.
13

---

Under review as a conference paper at ICLR 2020
50% of state (original)
0.75
Probability
25% of
0.50
0.25
0.00
200
400
800
1000
Number of Training Examples Seen (Thousands)
Figure 4: Both sizes of work well for the copy task.
Table 3: Size of can matter. ForMMIMCC using 1/4 of the hidden state for s works slightly better than using 1/2 in the main experiments. For Penn Treebank, 1/2 better than 1/4, suggesting long-term gradient propagation is important for this task.
Model
MIMIC Test Acc., LR 4c-3
MIMIC Test Acc., LR le-4
Penn Test PPL
1/2 state (original)
0.693
0.646
112.85 122.293
1/4 state
0.696
0.651
B ABLATION STUDY: GATO'S NON-LINEARITIES
B.I NON-LINEARITY IN UPDATE FOR R
Though is guaranteed to have good gradients, care must still be taken when choosing the update for r as any recurrent model. Recall the update for r:
r++1 o (Bx,11 + b rt) @ rr + (Cx41 + C c
We experiment by setting the a in the regularizing term to be tanh instead of sigmoid. Performance similar s simmoid on Copy (Figure 5), Add (Figure 6). MIMIC TTble 4), and Penn TreeBank (Tblle 4)
B.2
DECODER NON-LINEARITY FOR
The criteria in /eccionn 3 suggest bounding the hidden state with periodic functions before decoding. Our experiments used cos. We believe that these criteria are necessary, but not necessarily sufficient. We also try sin and observe no change in performance on Copy (Figure 7]. Add (Figure 8) MIMIC TTblle 5). and Penn TreeBank (Table 5). It is unclear which other functions meet these criteria. A next step would explore finite sums of cos and sin.
0.75
regularizer sigmoid (original)
Probability
regularizer tanh
0.50
0.25
0.00
200
400
800
1000
Number Training Examples Seen (Thousands)
Figure 5:
Tanh performs similarly to sigmoid for regularizing non-linearity on copy.
Table 4: Tanh performs similarly to sigmoid for regularizing non-linearity on MIMIC and Penn TreeBank.
Model
MIMIC Test Acc., LR 4e-3
MIMIC Test Acc., LR le-4
Penn Test PPL
si gmoid d (original) tanh
0.693
0.646
112.85
0.693
0.646
118.955
14

---

Under review as a conference paper at ICLR 2020
Sequence Length T= 100
Sequence Length T= 200
0.5
0.5
Baseline
Mean Squared Error
0.4
Mean Squared Error
0.4
regularizer sigmoid (original)
regularizer tanh
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Sequence Length T= 400
Sequence Length T 750
0.5
0.5
Mean Squared Error
Mean Squared Error
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.0
0
50
100
150
200
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Figure 6: Tanh performs similarly sigmoid for regularizing non-linearity on add.
decoder cos (original)
0.75
Probability
decoder sin
0.50
0.25
0.00
200
400
600
800
1000
Number of Training Examples Seen (Thousands)
Figure 7: sin for decoder non-linearity performs similarly to on copy task.
Table 5: sin for decoder non-linearity performs similarly to cos on MIMIC and Penn TreeBank
Model
MIMIC Test Acc., LR 4e-3
MIMIC Test Acc., LR le-4
Penn Test PPL
cos (original)
0.693
0.646
112.85
0.692
0.648
112.89
15

---

Under review as a conference paper at ICLR 2020
Sequence Length T= 100
Sequence Length T= 200
0.5
0.5
Baseline
Mean Squared Error
0.4
Mean Squared Error
0.4
decoder cos (original)
decoder sin
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Sequence Length T= 400
Sequence Length T= 750
0.5
0.5
Mean Squared Error
0.4
Mean Squared Error
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.0
0
50
100
150
200
0
50
100
150
200
Number of Training Examples Seen (Thousands)
Number of Training Examples Seen (Thousands)
Figure sin for decoder non-linearity performs similarly cos on adding task.
C EXPERIMENTAL DETAILS
C.I MIMIC-III
Features. To predict hospital length of stay the MIMIC-III dataset, we use the same features extracted for Carpolaatio-Prrdiitin Networks (Shukla Marlin 2019). The features include blood oxygen level, heart rate, respiratory rate, systolic blood pressure, diastolic blood pressure, temperature, oxygen saturation, and others. Not features are recorded at each observation time. We use 25 features including 12 vitals, 12 additional missingness indicators for those features, and measurement time.
Continuous Embeddings. All models apply a linear transformation followed by an element-wise tanh function to the continuous input before it is passed to the RNN The linear transformation is learned alongside other model parameters.
C.2 INITIALIZATION
Recall that experiment 1 is copying, experiment 2 is adding, experiment 3 is hospital stay prediction, and experiment 4 is language modeling on Penn Treebank. We use different initializations for GATO in experiments and 2 than do for experiments 3 and 4. In experiments 1 and 2, use initialization 1,, 0.1] for all RNN parameters in GATO In experiments 3 and 4, we use initialization ((1-0.1, 0.1] for all RNN parameters besides the weight matrices in the linear layers used to project x. For those. we use the default Py ToTch initializations. For embeddings and decoders for all models, as well as all GRU and LSTM] parameters. we use default PyTorch initializations. We use initialization 14,-0.1, 0. 0. for parameters in RHN PyTorch initializations for the linear layers in SRU and the linear layer for inputs in EURNN For the representation of unitary matrices parameters in EURNN we use standard normal initializations.
C.3 LEARNING RATE AND OPTIMIZER
We use Adam optimizer for all experiments. We use learning rate 4e-3 for experiments and 2, and le-4 for experiment 4, We check both learning rates 4e-3 and 1e-4 for experiment 3. In experiment
16

---

Under review as a conference paper at ICLR 2020
start with learning rate 0.004 and check the training loss after seeing every 10,000 examples. If the training loss is larger than for the previous 10,000 examples, we divide the learning rate by 2.
C.4 VALIDATION AND TEST SCORES
On MIMIC-III and Penn Treebank, we report test set accuracy corresponding to each model's predic- tions when they achieved their best validation accuracy. Perplexity on the language modeling task is defined as in Merity et al. (2018).
C.5 BATCH SIZE
We use batch size 32, 64, 256, and 20 in experiment 2, 3, and 4 respectively.
C.6 WEIGHT DECAY
We use 1.2e-6 weight decay (the default number in Merity et al. (2018) in experiments 3 and Since there is generalization problem in experiments and (because data samples are drawn from the true generating distribution), we do not use weight decay in those experiments.
C.7 LANGUAGE MODEL TRAINING
We follow the training strategy in (Merity et al. 2018). With probability 0.95, we take a batch with sentence length sampled from / (70,, 5") (then rounded to an integer) and with probability 0.05 take a batch with sentence length sampled from 335,, 5²) (then rounded to an integer).
C.8 OTHER TRICKS
We do not use dropout, gradient clipping and other tricks in all the experiments.
D VANISHING AND EXPLODING GRADIENTS
Consider the RNN as in Equation (1) and recall that a is an element-wise non- non-linearity:
h = (Uh, + Wxe+)..
Let LT be the loss at time T. Let D, = diag(o (Uh/ooUUUh-11 + Wx:)) be the Jacobian for a.
Then the gradient of the loss Cr with respect to U is:
QLI dU = Σ = DCr Ohy Oh,
Eour ohr dh, dU
Σ
[ DR
OCT
h
Oh7
Suppose that a nower-bound on the absolute-valle of the Jacobian D 's entries. Then the smallest singular value of U is larger than 1/X, Öhy/Oh, blows up exponentially with time gap T l. This called the exploding gradient problem. Suppose instead that an upper-bound on the absolute-value of D 's entries. Then if the largest singular value of U is smaller than 1/X, Ohr/Oh decays exponentially with time gap T - tt This is called the vanishing gradient problem. The term Öhr/Oh, /h, in the gradient calculation for ²Cr OO is therefore problematic for models like the RNN] Such issues can stop losses Lr at later time steps T from backpropagating to U to adjust the computation of earlier states h, for < T.
E LSTM
The LSTM (Hochreiter & Schmidhuber 1997 maintains an auxilary cell state that helps copy the hidden state forward in time as a sequence is processed. C₁ is called the cell state, and h, is called the hidden state. Letting fe, i,, g: and o, be non-linear functions of h and x; and letting O denote elsment-wise product, the LSTM hidden state computation for and h is defined by
fg ④ Ct
gg
h, = o, @ tanh(c.).
In the update of the cell state, the forget gate f, determines the influence of previous cell states on the next. The inpul gate determines the influence of new information on the cell. f, i,, O, and ge each defined via usual RNN recurrences:
f. = (W on h + W + by), g: = tanh W gh h + W + bby, i, a WWahhi W bi),
o, a ak h W W bo).
17

---

Under review as a conference paper at ICLR 2020
F
GATED RECURRENT UNITS
The GRU
Cho et al. 2014) is similar to the LSTM but with fewer gates and parameters. First, two RNN-like functions and of inputs x; and h are computed. They are called the reset and update gates, respectively. Then, a third RNN-like function with inputs X,, h and r computes proposed state h. Finally, the new state is convex combination of the previous and proposed states, Let be the s sigmoic and ø be any non- con-liiear like the tanh. Then the GRU computation is:
r a (W,xc x + 1,
2 = (W.x; + -Uhh-1)
h e ((Wx; +U (r h,-1)))
h - 2 h1 + (1 -1- @
The expression for h, is convex combination of h and proposed update h. Therefore, chosen to be bounded by 1 and if h starts with all coordinates in (0, 1), the state will stay bounded.
G RECURRENT HIGHWAY NETWORK (RHN)
One-layer RHNS Zilly et al. 2017 another alternative gated recurrent networks. Define h X be hidden states and inputs at time t. The update in the RHNS computation is:
(W, X + U,
(Wax, xx U.h--1)
tanh (Wx, + Uh,1)
h. ② h, + t h
The RHN does not bound its hidden state. Based on the criterion in Section we bound the hidden state using a similar structure as the GRU We call the new structure Bounded Recurrent Highway Network (BRHN) The BRHN updates as follows:
(W,x) + U,h:-1)
h tanh (Wx, Uh;-1
h r h + (1 -1-11 r) о
BRHN performs similarly to GATO and GRU on Penn TreeBank, while RHN numerically fails.
H EFFICIENT UNITARY RECURRENT NEURAL NETWORK (EURNN)
Efficient Unitary Recurrent Neural Network (RURNN) Jing et al. 2017) is fast and tunable version
of the Unitary Recurrent Network (Arjovsky et al. 20166). The forward propagation in EURNN the RNN framework:
h, SWx, + Uh,-)
Define D be diagonal matrix and Fi...., Fz to be sequence of rotation matrices. parameterization to make U unitary:
EURNN uses
U = DF....
equals to the number of rows of U. it can capture the full space of the unitary matrices. Following Jing al. (2017), we choose L have faster computation and the same nonlinearity
f(z))
ReLU((22] b.).
where b; is trainable parameter. Though EURNN has bounded hidden-to-hidae gradients, we know from [Section that these are not the only gradients that matter. The forward propagation of the hidden state blow up, which may still cause gradient problems. It is good research question whether is possible to have both unitary hidden-to- hidden transformations and a bounded hidden state.
18

---

Under review as a conference paper at ICLR 2020
I STATISTICAL RECURRENT UNIT (SRU)
Statistical Recurrent Unit (SRU) Oliva et al. 2017) uses a moving average update for hidden state Pe In the update of each time step SRU first summarizes the information from the previous hidden state M4-1 into and then calculates the candidate hidden state Φi based on and the inputs x,. The final update is a weighted average of previous hidden state μ4-1 and candidate Φe. SRU has separate hidden states {u, }. a. Following Oliva et al. 2017). we choose = {0, 0.25, 0.5, 0.9, 0.99}
our experiments. The SRU updates as follows:
F, ReLU (W,
Φ; ReLU W WTr + WxX+)
H4 (a) apl (1 a) Va a.
one a equals one, the corresponding hidden state ft has the identity hidden-to-hidden gradient matrix property. However, when = L, |4 1 will never be updated. To capture iong-term dependencies sequence modeling, we want changes the hidden state earlier time affect the later hidden states. If fut (1) never changes, then we cannot use it to model long-ten men--eemm dependencies. In GATO the update of s, does not depend on S but is updated based on rę-1 and x,. Therefore, GATO not only has the identity hidden- to- hidden gradient matrix property for but also can use s, to model long-term dependencies.
(a) For H4 a f 1, the update of ut (a)) depends on 14-11 Therefore, when a does not equal to 1, SRU
(a)
does not have the same hidden-to-hideen identity matrix gradient property as GATO
LSTMs AL.SO HAVE GRADIENT PROBLEMS
By the form of the update c; = f -, +
ge Equation (7) OCt/Oc, has two gradient paths: one through each term in the sum. The first path only linear in the cell state. This has led to claims that the LSTM avoids vanishing gradients by allowing information to flow along cell states over long time intervals. However, further analysis reveals that the second path depends weight matrices W "hh, W f/e Wik, oh, exponentiating their magnitude to power T t (Kanuparthi al., 2019). The magnitude of the second gradient path suppress the first that weight updates are made in favor of short-term dependencies.
the entries of the
19


Reviews:
[{"review_id": "SygdaoLEtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new RNN architecture designed to overcome vanishing/exploding gradient problems and to improve long-term memory for sequence modelling. The main ideas are (i) to split the hidden state into two parts, one of which does not influence the recurrence relation, and can therefore not blow up or contract by self-feedback; and (ii) to use periodic functions, in particular the cosine, as non-linearity in the decoder, so that the output is bounded but does not saturate.\n\nThe paper puts forward a fairly systematic analysis of the gradients in RNNs. The analysis appears correct, and is in fact quite similar to considerations in earlier RNN work (which is correctly cited), and forms the basis for the proposed GATO unit. There are two loose ends in this part:\n1) the cosine non-linearity results from a purely negative selection - the function should be bounded, but not saturating. The paper does not even ask the question which periodic function might be a good choice.\n2) While the method is presented as a grand theory, with the only constraint that a part of the hidden state does not influence the recurrence function; the actual implementation and experiments are limited to the narrow special case of \"non-interacting\" GATO, where the \"passive\" variables make up exactly half of the hidden vector, and the update of each individual hidden variable is influenced only by a single variable from the previous state. So there are in fact no empirical results, not even on toy data, for the general case that the paper claims to introduce.\n\nIn the experiments, there are two artificial problems (copying, adding) for sequences of symbols. These are illustrative and sensible to verify and analyse  the behaviour of GATO in a controlled setting, but rather far from most real sequence modelling tasks. In a third experiment the task is to classify whether or not patients will stay in intensive care for >1 week, based on a (seemingly private) database of time series of vital parameters. Unfortunately, the results for that experiment do not go beyond the usual \"ours is better\". While the numbers clearly support GATO, it would have been nice to look a bit closer and pinpoint what makes the difference. Also, the comparison is a bit loose. It would have been better to add additional baselines where also LSTM and GRU are restricted to \"non-interacting\", element-wise recurrence (as far as technically feasible). As it stands, it is unfair to claim \"we can do it with much fewer parameters\" - perhaps LSTM / GRU could, too. In fact, it could even be that the task is just simple, so that more restricted model with fewer parameters generally perform better - I do not claim this is the case, but the experiments do not rule it out and, hence, do not confirm that the clever GATO recurrence makes the difference.\n\nA small gap is also that even the \"real\" experiment might not be completely realistic. Nowadays it is a popular strategy to use deep LSTMS / GRUs, i.e., stack multiple levels of recurrence, possibly with temporal sub-sampling, to better capture long-term relations. While this is potentially even more brittle, because of the additional gradient flow across layers, it does seem to work. But deep RNNs are not tested (in fact, not even mentioned) in the paper.\n\nOverall, in spite of a few loose ends, I find both the design and the practical performance of the proposed GATO unit very interesting, and potentially valuable for the ICLR crowd. This is one of the more convincing RNN papers I have recently read.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJxmtISTFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel recurrent architecture called GATO. \nSpecifically, the authors focused on sequence modeling tasks and developed criteria for RNN models on such tasks. \nTha GATO model can resolve the vanishing/exploding gradient issue and is robust to initializations. \nEmpirical results show GATO can outperform LSTM and RNN in both synthetic datasets and real datasets.\n\nThe key insight of the proposed model is that only part of the hidden states is recurrently updated. \nThe GATO achieves this by adding the skip connection channel (or residual connection) along the temporal dimension.\nGATO summarizes the hidden states r_t by recurrently adding (transformed) r_t to s_t.\nThis idea also appears in many previous RNN models, such as highway RNN/LSTM, Statistical/Fourier Recurrent Units [1][2].\nSpecifically, the proposed GATO is a special case of SRU ( alpha=1). This limits the novelty of the paper and thus make the contribution marginal.\n\nAs for the experimental studies, the authors only provide comparisons with LSTM and GRU. There are a lot of advanced RNN architectures to address vanishing/exploding gradient issues, such as uRNN[3], oRNN[4], Spectral-RNN[5] and SRU/FRU [1][2]. It would be more convincing if the \nauthors could include these models into comparison.\n\nOverall I think this paper should be further improved before being accepted.  \n\n\n\n[1] Oliva, J.B., Póczos, B. and Schneider, J., The statistical recurrent unit. \nIn ICML 2017 (pp. 2671-2680).\n\n[2] Zhang, J., Lin, Y., Song, Z. and Dhillon, I., Learning Long Term Dependencies via Fourier Recurrent Units. \nIn ICML 2018 (pp. 5810-5818).\n\n[3] Arjovsky, M., Shah, A. and Bengio, Y., Unitary evolution recurrent neural networks. \nIn ICML 2016 (pp. 1120-1128).\n\n[4] Mhammedi, Z., Hellicar, A., Rahman, A. and Bailey, J., Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. \nIn ICML 2017 (pp. 2401-2409).\n\n[5] Zhang, J., Lei, Q. and Dhillon, I., Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization. \nIn ICML 2018 (pp. 5801-5809).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxeizHRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a modification of RNN that does not suffer from vanishing and exploding gradient problems. The proposed model, GATO partitions the RNN hidden state into two channels, and both are updated by the previous state. This model ensures that the state in one of the parts is time-independent by using residual connections. The experiments on the long copy and adding tasks, as well as language modeling on the Penn TreeBank dataset show the performance improvement against the basic LSTM and GRU. \n\nThe paper tackles an interesting and challenging problem with a novel approach in sequence modeling. The idea is clear and the paper is well-written. The mathematical insights are well reasoned. \n\nThe proposed method outperforms LSTM and RNN with much fewer number of parameters. However, there is no regularization is used for such big LSTM/GRU models. There is a chance that such a big LSTM/GRU model increased the chance of overfitting, and therefore the performance is low. I would like to see the comparison after adding any common regularization that prevents overfitting across the recurrent connections. \n\nThere are many advanced RNN/LSTMs proposed in recent years [1-5] addressing the vanishing gradient problem. It is hard to judge the quality of the proposed method due to the lack of evaluation/comparisons. This paper needs more intensive evaluations with recent RNN-based methods. For instance based on [6], AWD-LSTM [6] and RHN [4] achieved 52.8 and 65.4 test perplexity scores on the Penn TreeBank dataset respectively. The the best score in this paper is 112.85. \n\n[1] \"Phased LSTM: Accelerating recurrent network training for long or event-based sequences.\" 2016.\n[2] \"Fast-slow recurrent neural networks.\" 2017.\n[3] \"Skip RNN: Learning to skip state updates in recurrent neural networks.\" 2017.\n[4] \"Recurrent highway networks.\" 2017.\n[5] \"Dilated recurrent neural networks.\" 2017.\n[6] \"Regularizing and optimizing LSTM language models.\" 2017\n\nAll experiments are performed with 1 or 2 layers. Hierarchical RNN/LSTM performs much better in sequence learning. Is there any reason authors only showed 1 or 2 layers? How does GATO change with more than 2 layers?\n\nHow do other choices of non-linear functions affect the performance in practice?   \n\n\nTypo\nr -> r_t in Eq. 6\n\n--- \nAfter rebuttal:\nOne of my main concerns, weak baselines and unfair comparisons, was partially answered in the updated paper.  I am not fully convinced by their new comparisons.\nFor instance, authors mentioned that 'RHN and EURNN performed poorly because they have unbounded forward propagation'. To overcome this, they introduced 'Bounded RHN' in Append G and it performs similarly to GATO and GRU. However, this 'Bounded RHN' is the one used in the original RHN paper. Overall, it is hard to trust their additional comparisons. \nAlthough, I believe that this paper is well-structured and justified. Also it has high potential for the community. However, the paper itself is not ready to be published.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a modification of RNN that does not suffer from vanishing and exploding gradient problems. The proposed model, GATO partitions the RNN hidden state into two channels, and both are updated by the previous state. This model ensures that the state in one of the parts is time-independent by using residual connections. 

The reviews are mixed for this paper, but the general consensus was that the experiments could be better (baseline comparisons could have been fairer). The reviewers have low confidence in the revised/updated results. Moreover, it remains unclear what the critical components are that make things work. It would be great to read a paper and understand why something works and not that something works. 

Overall: Nice idea, but the paper is not quite ready yet.


Decision: Reject