Improving the Gating Mechanism of Recurrent Neural Networks
Authors: Albert Gua
Caglar Gulcehre
Tom le Paine
Razvan Pascanu
Matt Hoffman
Venue:  (2020)
Keywords: recurrent neural networks
LSTM
GRUs
gating mechanisms
deep learning
reinforcement learning

Abstract: In this work, we revisit the gating mechanisms widely used in various recurrent and feedforward networks such as LSTMs, GRUs, or highway networks. These gates are meant to control information flow, allowing gradients to better propagate back in time for recurrent models. However, to propagate gradients over very long temporal windows, they need to operate close to their saturation regime. We propose two independent and synergistic modifications to the standard gating mechanism that are easy to implement, introduce no additional hyper-parameters, and are aimed at improving learnability of the gates when they are close to saturation. Our proposals are theoretically justified, and we show a generic framework that encompasses other recently proposed gating mechanisms such as chrono-initialization and master gates . We perform systematic analyses and ablation studies on the proposed improvements and evaluate our method on a wide range of applications including synthetic memorization tasks, sequential image classification, language modeling, and reinforcement learning.  Empirically, our proposed gating mechanisms robustly increase the performance of recurrent models such as LSTMs, especially on tasks requiring long temporal dependencies.

Under review as conference paper at ICLR 2020
IMPROVING THE GATING MECHANISM OF RECURRENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Gating mechanisms are widely used in neural network models, where they allow gradients backpropagate more easily through depth or time. However, their satura- tion property introduces problems of its own. For example, in recurrent models these gates need have outputs near to propagate information over long time-delays, which requires them to operate in their saturation regime and hinders gradient-based learning of the gate mechanism. We address this problem by deriving two synergistic modifications to the standard gating mechanism that are easy to implement, introduce no additional hyperparameters, and improve learnability the gates when they are close to saturation. We show how these changes are related to and improve on alternative recently proposed gating mechanisms such as iotooniiitializttionn and Ordered Neurons. Empirically, our simple gating mechanisms robustly improve the performance of recurrent models on a range of applications, including synthetic memorization tasks, sequential image classification, language modeling, and reinforcement learning, particularly when long-term dependencies are involved.
INTRODUCTION
Recurrent neural networks (RNNs) have become standard machine learning tool for learning from sequential data. However, RNNs are prone to the vanishing gradient problem, which occurs when the gradients of the recurrent weights become vanishingly small as they get backpropagated through time (Hochreiter al. 200T). A common approach to alleviate the vanishing gradient problem is to use gating mechanisms, leading to models such as the long short term memory (Hochreiter & Schmidhuber 1997 LSTM) and gated recurrent units (Chung al. 2014 GRUs). These gated RNNs have been very suc- cessful in several different application areas such as in reinforcement learning (Kapurowski et al. 2018 Espeholt al. 2018) and natural language processing (Bahdanau al.) 2014 Kočiský et al. 20188).
At every time step, gated recurrent models form a weighted combination of the history summarized by the previous state, and (a function of) the incoming inputs, create the next state. The values of the gates, which are the coefficients of the combination, control the length of temporal dependencies that can be addressed. This weighted update can be seen an additive residual connection on the recurrent state, which helps signals propagate through time without vanishing. However, the gates themselves are prone to saturating property which can also hamper gradient-based learning. This is particularly troublesome for RNNs, where carrying information for very long time delays requires gates to be very close to their saturated states.
We formulate and address two particular problems that arise with the standard gating mechanism of re- current models. First, typical initialization of the gates is relatively concentrated. This restricts the range of timescales the model can address, as the timescale of a particular unit is dictated by its gates. Our first proposal, which we call uniform gate initialization (Section 2.2). addresses this by directly initializing the activations of these gates from distribution that captures wider spread of dependency lengths.
Second, learning when gates are in their saturation regime is difficult because of vanishing gradients through the gates. We derive modification that uses auxiliary refine gate to modulate main gate, which allows it to have a wider range of activations without gradients vanishing as quickly.
Combining these two independent modifications yields our main proposal, which call the UR- gating mechanism. These changes can be applied to any gate (i.e. bounded parametrized function) and have minimal no overhead in terms of speed, memory, code complexity, and (hyper-)pprrmmee

---

Under review as conference paper at ICLR 2020
We apply them to the forget gate of recurrent models, and evaluate on many benchmarks including synthetic long-term dependency tasks, sequential pixel-level image classification, language modeling, program execution, and reinforcement learning. Finally, connect our methods to other proposed gating modifications, introduce a framework that allows each component to be replaced with similar ones, and perform theoretical analysis and extensive ablations of our method. Empirically, the UR- gating mechanism robustly improves on the standard forget and input gates of gated recurrent models. When applied to the LSTM, these simple modifications solve synthetic memory tasks that are pathologically difficult for the standard LSTM, achieve state-of- tfate-of-the-ar the-artt results on sequential MNIST and CIFAR-10, and show consistent improvements in language modeling on the WikiText- 103 dataset (Merity et al.) 2016) and reinforcement learning tasks (Hung et al. 2018
2 GATED RECURRENT NEURAL NETWORKS
Broadly speaking, RNNs are used to sweep over a sequence of input data z; to produce a sequence of recurrent states h2 R summarizing information seen so far. At high level, an RNN is just parametrized function in which each sequential application of the network computes a state update u: (-:, 1) + hy. Gating mechanisms were introduced address the vanishing gradient problem (Bengio et al.) 1994, Hochreiter et al.) (001). and have proven crucial to the success of RNNs. This mechanism essentially smooths out the update using the following equation,
h4 (e
(1)
where the forgel gate J2 and input gate i, are (0.1)-vvllued functions that control how fast information forgotten or allowed into the memory state. When the gates tied, i.e. fa+ti 1 in GRUs, they behave as low-pass pass filter, deciding the time-scale on which the unit will respond (Tallec & Ollivier 2018). For example, large forget gate activations close to ft are necessary for recurrent models to address long-term dependencies
We will introduce our improvements the gating mechanism primarily in the context of the LSTM, which is the most popular recurrent model. However, these techniques can be used in any model that makes similar use of gates.
A typical LST (equations 2 is an RNN whose state is rep- resented by tuple (h+,ct) consisting of a "hidden" state and "cell" state. The basic gate equation (1) is used to create the next cell state et 5 Note that the gate and update activations are a function of
ft = o (C hr-1))
(2)
i, o (Lelwr, 11-1))
(3)
ur tanh (Cu (Fe, ht-1))
(4)
the previous hidden state hi- instead of ct-1. Here, L. stands for a parameterized linear function of its inputs with bias b., e.g.
C. = f100--1 440
(5)
Or a (at,
(6)
(x,
y. W Whhh-1+bb
h o,tanh(c;)
(7)
and (-) refers to the standard sigmoid activation function which we will assume is used for defining [0,1]-valued activations in the rest of this paper. The gates of the LSTM were initially motivated as binary mechanism, switching on or off allow information and gradients to pass through. However, in reality this fails to happen due to a combination of initialization and saturation. This can be problematic, such as when very long dependencies are present.
2.1 THE UR-LSTM
We present two solutions which work in tandem to address the previously described issues. The first ensures a diverse range of gate values the start of training by sampling the gate's biases so that the activations will be approximately uniformly distributed at initialization. We call this Uniform Gate Initialization (UGI). The second allows better gradient flow by reparameterizing the gate using auxiliary "refine" gate.
b (2(0.1)
(9)
f h-1))
(10)
re re= (C, 11-))
(11)
ge (1-
+(1 (1-r)) f?
(12)
c: - + (1 - ).
(13)
As our main application is for recurrent models, we present the full UR-LSTM model in equations (1) However, we note that
q (F, h h==ottan((c))
(14)
(15)
In this work, we use "gate" to alternatively refer to [0, 1-valud function or the value ("activation" of that function.
2

---

Under review as conference paper at ICLR 2020
these methods can be used to modify any gate (or more generally, bounded function) in any model. In this context the UR-LSTM is simply defined by applying UGI and refine gate r on the original forget gate f to create an effective forget gate (equation (12) ). This effective gate is then used in the cell
state update (13). Empirically, these small modifications to an LSTM are enough to allow it to achieve
nearly binary activations and solve difficult memory problems (Figure In the rest of Section 2, we provide theoretical justifications for UGI and refine gates.
2.2 UNIFORM GATE INITIALIZATION
Standard initialization schemes for the gates prevent the learning of long-term temporal correla- tions (Tallee& & Ollivier 2018). 8). For example, supposing that a unit in the cell state has constant forget gate value ft. then the contribution of an input x in l time steps will decay by (ft)*. This gives the unit an effective decay period or characteristic timescale of O (() Standard initialization of linear layers L sets the bias term to 0, which causes the forget gate values (2) to concentrate around 1/2. A common trick of setting the forget gate bias to by = 1.0 (Jozefowicz et al. 2015) does increase the value of the decay period to [1. 3.7. However, this is still relatively small, and moreover fixed, hindering the model from easily learning dependencies at varying timescales.
We instead propose to directly control the distribution of forget gates, and hence the corresponding distribution of decay periods. In particular, we propose to simply initialize the value of the forget gate activations fi according a uniform distribution (0,1), as described in Section 2.1 An important 2018) inftializations is that negative forget biases allowed. The effect of UGI is that all timescales are covered, from units with very high forget activations remembering information (nearly) indefinitely, to those with low activations focusing solely on the incoming input. Additionally, introduces no additional parameters; it even can have less hyperparameters than the standard gate initialization, which sometimes tunes the forget bias by. Appendix B.2 and B.3 further discuss the theoretical effects of UGI on timescales.
difference between UGI and standard or other (e.g. Tallec & Ollivier
2.3 THE REFINE GATE
Given gate f=o(C,(x) E 1.. the refine gate is an independent gate r= (z)), and modulates to produce value E [0,1] that will be used in place of f downstream. Is motivated by considering how to modify the output of a gate f in a way that promotes gradient-based learning, derived below.
An additive modification The root of the saturation problem is that the gradient V of gate, which can be written solely a function of the activation value as f(l- decays rapidly approaches 0 or 1. Thus when the activation f is past a certain upper or lower threshold, learning effectively stops. This problem cannot be fully addressed only by modifying the input the sigmoid, in UGI and other techniques, as the gradient will still vanish by backpropagating through the activation function.
Therefore to better control activations near the saturating regime, instead of changing the input the sigmoid in f = (C(x)) we consider modifying the output. In particular, we consider adjusting f with an input-dependent update ((fxx) for some function 6, to create an effective gate g= f f o( .x) that will be used in place of downstream such as in the main state update (I]. This sort of additive ("residual") connection common technique to increase gradient flow, and indeed was the motivation of the LSTM additive gated update [I) itself (Hochreiter & Schmidhuber 1997).
Choosing the adjustment function Although many choices seem plausible for selecting the additive update o, we reason backwards from necessary properties of the effective activation to deduce a principled function c. The refine gate will appear as result.
First, note that ft might need to be increased or decreased, regardless of what its value is. For example, given a large activation near saturation, it may need to be even higher to address long-term dependencies in recurrent models; alternatively. if it is too high by initialization or needs to unlearn previous behavior, it may need decrease. Therefore, the additive update to ʃ should create an effective activation 9t in the range ta for some a. Note that the allowed adjustment range a= af))) needs be a function of f in order keep g e [0.1].
Thii corresponds the number of timesteps it takes decay by 1 1/e.
3

---

Under review as conference paper at ICLR 2020
In particular, the additive adjustment range af) should satisfy the following natural properties:
Validiry: a(f) S min(f,1-5 f), to ensure g (f) E [0.1].
Symmetry: Since 0 and 1 are completely symmetrical in the gating framework, =af)==(l-f)) Differentiaaility: aff) will be used in backpropagation, requiring a E (IR).
Figure 2a iillustaates the general appearance of (f()) based on these properties. In particular, Validity implies that that its derivative satisfies d'(0) S and a1(1) 2 -1, Symmetry implies a'())
-a'(1 f), and Differentiability implies a' is continuous. The simplest such function satisfying these is the linear a' (1)==113 - 2f, yielding ( ((-1))
Given such af)), recall that the goal is produce effective activation g=f+ such that g gef+ (Figure 2b Our final observation is that the simplest such function á satisfying this is ) for some = Using the standard method for defining |-1,1| --luud-vllued -valued functions via a tanh non-linearity leads to ((fxx) (2r for another gate (Crr)
The full update is given in Equation [16
g -1--
(16)
Equation (16) has the clegant interpretation that the gate linearly interpolates between the lower band (1-f)² f)² (Figure(2b.. In other words, the original gate f is the coarse- grriieed determinant of the effective gate g. while the gate r "refines" This allows the effective gate g to reach much higher and lower activations than the constituent gates and r (Figure 2c) 2 bypassing the saturating gradient problem. For example, this allows the effective forget gate to reach g g= 0.99 when the forget gate is only 0.9..
f-a(f)= f and the symmetric upper band f+a(f)=
2.4 REFINING RECURRENT MODELS
Formally, the full mechanism of the refine gate as applied to gated recurrent models is defined in equations [ Note that it is an isolated change where the forget gate (TO) is modified before applying the the standard update (I. Figure illlustatts the refine gate in an LSTM cell. Figure illustrates how the refine gate re defined and how it changes the effective gate fa produce an
effective gate 9t.
LSTM
refine mechanism
g
C,
tanh
1-x
9
1-x
U,
refine
tanh
refine gate
neural network layer
pointwise unitary opetation
pointwise binary operation
X
Figure 1: LSTM with refine gate The refine gate r; modifies another gate, such as the forget gate Se for recurrent models. It interpolates between upperbound Uy, and lowerbound Ly. functions of the forget gate. The resulting effective forget gate St is then used in place of fe in the state update (5)
Finally, to simplify comparisons and ensure that we always use the same number of parameters the standard gates, when using the refine gate we the input gate to the effective forget gate, i, = St- However, we emphasize that these techniques are extremely simple and broad, and can be applied to any gate (or more broadly, any bounded function) to improve initialization distribution and help optimization. For example, our methods can be combined in different ways in recurrent models, e.g. an independent input gate can be modified with its own refine gate. Alternatively, the refine gate can also be initialized uniformly, which we do in experiments whenever both UGI and refine gates used.
2.5 RELATED GATING MECHANISMS
We highlight a few recent works that also propose small gate changes to address problems of long-term variable-length dependencies. Like ours, they be applied any gated update equation.
4

---

Under review as conference paper at ICLR 2020
94
0.-
Oiscciv pata
(a)
(b)
(d)
Figure 2: Refine gate in action: (a) [Solid] function (ffe) satisfying natural properties chosen to define band within which the forget gate is refined. (b) The forget gate (x is conventionally defined with the sigmoid function (black). The refine gate interpolates around the original gate ft to yield an effective gate 9r within the upper and lower curves, 9r ft (ff)). as function the forget and refine gates fe, Ft. High effective activations can achieved with more modest Jayt values. (d) The gradient Vge a function of effective gate activation 9t- [Black, blue]: Lower and upper bounds on the ratio of the gradient when using a refine gate vs. without.
(c) Contours of the effective gate g
Tallec & Ollivier 2018) suggest an initialization strategy to capture long-term dependencies on the order of maz. by sampling the gate biases from by ~og (1. (1.Tmar 1). Although similar UGI in definition, chrono initialization (CI) has key differences in the timescales captured, for example by using an explicit timescale parameter and having no negative biases. Due its relation to UGI, we provide more detailed comparison in Appendix B.3 As mentioned in Section 2.3 techniques such as these that only modify the input to a sigmoid gate do not fully address the saturation problem.
The Ordered Neuron LSTM introduced by (Shen et al. (2018) aims to induce ordering over the units in the hidden states such that "higher-level" neurons retain information for longer and capture higher-level information. We highlight this work due to its recent success in NLP, and also because its novelties can be factored into introducing two mechanisms which only affect the forget and input gates, namely (i) the cumax: cumsumosoftmas activation function which creates monotonically increasing vector in [0,1]. J.. and (ii) a pair of "master gates" which are ordered by cumax and fine-tuned with another pair of gates.
In fact, we observe that these are related to our techniques that one controls the distribution of a gate activation, and the other is auxiliary gate with modulating behavior. Despite its important novelties, we find that the ON-LSTM has drawbacks including speed/stability issues and theoretical flaws in the scaling of its gates. We provide the formal definition and detailed analysis of the ON-LSTM in Appendix B.4 In particular we flesh out deeper relation between the master and refine gates and show how they can be interchanged for each other.
We include more thorough overview of other related works on RNNs in Appendix B.1 These methods are largely orthogonal the isolated gate changes considered here and are not analyzed. We note that an important drawback common to all other approaches is the introduction of substantial hyperparameters the form of constants, training protocol, and significant architectural changes. For example, even for chrono initialization, one of the less intrusive proposals, we experimentally find it to be particularly sensitive to the hyperparameter T Tmaz (Section
2.6
GATE ABLATIONS
Our insights about previous work with related gate components allow us to perform extensive ablations of our contributions. We observe two independent axes of variation, namely, activation function/ini- tialization (cumax, constant bias sigmoid, CI, UGI) and auxiliary modulating gates (master, refine), where different components can be replaced with each other. Therefore we propose several other gate combinations to isolate the effects of different gating mechanisms. We summarize few ablations here; precise details are given in Appendix B.5] O-: Ordered gates. A natural simplification of the main idea of ON-LSTM, while keeping the hierarchical bias the forget activations, simply drop the auxiliary master gates and define fr - using the cumax activation function. UM- UGI master gates. This variant of the ON-LSTMMs gates ablates the cumax operation the master gates,
5

---

Under review as conference paper at ICLR 2020
replacing it with sigmoid activation and UGI which maintains the same initial distribution on the activation values. OR- Refine instead of master. A final variant in between the UR- gates and the ON-LSTM's 's gates combines cumax with refine gates. In this formulation, as in UR- gates, the refine gate modifies the forget gate and the input gate is tied to the effective forget gate. The forget gate is ordered using cumax.
Table I
summarizes the gating modifications we consider and their naming conventions. Note that we also denote the ON-LSTM method "OM-LSTM" (M for master) for mnemonic ease. Finally, we remark that all methods here are controlled with the same number of parameters as the standard LSTM, aside from the OM-LSTM and UM-LSTM which use an additional 20 -fraction parameters where C is the downsize factor on the master gates (Appendix(B.) [B44).
Table 1:
Summary of gating mechanisms considered in this work as applied to the main forget/input gates of recurrent models. To preserve parameters. refine gate methods use tied input gate, and master gate methods use a downsize factor C'> 1. (Left) Existing approaches and our main method. (right) Ablations of our gates with different components.
Name
Gate Mechanism
U-
Uniform gate initialization. no auxiliary gate
Standard gate initialization [
R-
Refine gate with standard gate initialization
C-
Chrono initialization, no auxiliary gate Ordered main gates, auxiliary master gates
O-
cumax activation on forget/input gates
OM-
UM-
UGI main gates, auxiliary master gates
UR-
UGI main gate, auxiliary refine gate
OR-
Ordered main gate, auxiliary refine gate
EXPERIMENTS
We first perform full ablations of the gating variants (Section2.0) on benchmark synthetic memorization and pixel-by-pixel image classification tasks. We then evaluate our main method on important applications for recurrent models including language modeling and reinforcement learning, comparing against baseline methods where appropriate.
The vanilla LSTM uses forget bias 1.0 (Section 2.2). When explicitly tuned, we set Taa to be proportional to the hidden size. This heuristic uses the intuition that if dependencies of length T exist, then so should dependencies of all lengths << T, Moreover, the amount of information that can be remembered is proportional to the number of hidden units.
is used and not
All of our benchmarks have prior work with recurrent baselines, from which we used the same models, protocol, and hyperparameters whenever possible, changing only the gating mechanism. Since our simple gate changes are compatible with other recurrent cores, evaluate them in tandem with recurrent models such as the GRU, Reconstructive Memory Agent (RMA; [Hung et al.) 2018), and Relational Memory Core (RMC: (aattoo al. (2018) whenever they were used on these tasks. Full protocols and details for all experiments are given in Appendix D
(RMC:[Santrro
3.1 SYNTHETIC TASKS
Our first of experiments is on synthetic memory tasks (Hochreiter & Schmidhuber 1997 Arjovsky et al. 2016) that are known to be hard for standard LSTMs to solve.
Copy task. The input is a sequence of N + 20 digits where the first 10 tokens (ap.01.....ag) are randomly chosen from { {1, the middle N tokens are set to 0, and the last ten tokens are 9. The goal of the recurrent model is to output (ag) order on the last 10 time steps, whenever the cue token 9 is presented. We trained our models using cross-entropy with baseline loss log(8) (Appendix D.Ip
Adding task. The input consists of two sequences: 1. N numbers (4a....,0, -1-) sampled independently from 24 [0.1] target output is a4, + +a, and models are evaluated by the mean squared error with baseline loss 1/6.
2. an index in E (2. and i [N/2.N ), together encoded a two-hot sequence. The
Figure 3 shows the loss of various methods on the Copy and Adding tasks. The only gate combinations capable of solving Copy completely are OR-, UR- O-, and C-LSTM. This confirms the mechanism of their gates: these are the only methods capable of producing high enough forget gate values either through the cumax non-linearity, the refine gate, or extremely high forget biases. The U-LSTM is the only other method able to make progress, but converges slower it suffers from gate saturation

---

Under review as conference paper at ICLR 2020
Mathod
-3
Mathod
C-LSTMM
C-LSTM O-LSTM OM-LSTM OR-LSTM
-5
O-LSTM
Log Locs
OM-LSTM
OR-LSTM
-10
R-LSTM
UM-LSTM
UM-LSTM UR-LSTM
-15
UR-LSTM
Step
Step
Figure 3: (Left) Copy task length 500 (Right) Adding task length 2000. Every method besides the LSTM solves the Adding task. The only methods capable of solving copy OR-,UR-,--, and C-LSTM models, with all other models aside from U-LSTM stuck at baseline. Refine gates are fastest.
138
I Lobll .. Lite I
(a) UR-LSTM
(b) U-LSTM
(c) LSTM
(d) C-LSTM
Figure 4: Histograms of forget gate fi activations (averaged over time and batch) before (Top) and after (Bottom) training Copy (y-axis independently scaled). C-LSTM initializes with extremal activations which barely change during training. Standard LSTM initialization cannot learn large enough fa and makes no progress on the task. U-LSTM makes progress by encouraging range of forget gate values, but this distribution does not change significantly during training due to saturation. UR-LSTM starts with the same distribution, but is able learn extremal gate values. Complementary to here when learning large activations is necessary. Appendix E.1 shows reverse task where the UR-LSTM is able un-learn from saturated regime.
without the refine gate. The vanilla LSTM makes no progress. The OM-LSTM and UM-LSTM also get stuck the baseline loss, despite the OM-LSTM's cumax activation, which we hypothesize is due to the suboptimal magnitudes of the gates at initialization (Appendix B.4). On the Adding task, every method besides the basic LSTM is able to eventually solve with all refine gate variants fastest.
Figure 4 shows the distributions of forget gate activations of sigmoid aactviitativationn methods, before and after training on the Copy task. It shows that activations near 1.0 are important for a model's ability to make progress or solve this task, and that adding the refine gate makes this significantly easier.
3.2 PIXEL-BYS BY-PIL IMAGE CLASSIFICATION
These tasks involve feeding a recurrent model the pixels of an image a scanline order before producing classification label. We test the sequential MNIST (sMNIST), permuted MNIST (pMNIST) (Le ct 2015), and sequential CIFAR- (sCIFAR) tasks. Each LSTM method was ran with learning rate sweep with 3 seeds each. The best validation score found over any run is reported in the first. two rows of Table We find general that all methods are able to improve over the vanilla L.TTM. However, the differences become even more pronounced when stability is considered. Although Table 2 reports the best validation accuracies found on any run, we found that many methods were quite unstable. Asterisks are marked next to score denoting how many of the 3 seeds diverged, for the leamiing rate that score was found at.
MNIST is not included here it too easy, making difficult draw conclusions.
7

---

Under review conference paper at ICLR 2020
Validation Accuraci
0.96
0.70
-LTMM
C-ETTM
CASTM
0.94
0M-LSTM
0.60
OM-LSTH
0.92
8.LSTM
U-LSTM
0.55
B4STMM
U-LSTM
UR-LSTM
UR-LSTM
25
50
75
100
25
50
75
100
Epoch
Epoch
(a) Permuted MNIST
Sequential CIFAR-10
Figure 5: Learning curves with deviations on pixel image classification, at the best stable learning rate.
Conversely, FigureBishows the accuracy curves of each method at their best stable learning rate. The basic LSTM is noticeably worse than all of the others. This suggests that any of the gate modifications, whether better initialization, cumax non-linearity, or master or refine gates, are better than standard gates especially when long-term dependencies are present. Additionally, the uniform gate initialization methods are generally better than the ordered and chrono initialization, and the refine gate performs better than the master gate. We additionally consider applying other techniques developed for recurrent models that are independent of the gating mechanism. Table 2 gating mechanisms are applied to the GRU model instead of the LSTM, where similar trends hold across the gating variants. In particular, UR-GRU is the only method that is able to stably attain good performance. As another example, the addition of a generic regularization technique-we chose Zoneout (Krueger al. 2016) with default typeraarameters =2.=0.5. 0.5, 2 = 0.05) continued improving the UR-LSTM/GRU, outperforming even non non-recurrent models on sequential MNIST and CIFAR-10. Table comppres the test accuracy of our main model against other models.
also reports scores when the same
Table 2: Validation accuracies on pixel image classification. Asterisks denote divergent the learning rate the best validation score was found at.
Gating Method
R-
OM-
OR-
UM-
UR-
PMNIST
94.77**
94.69
96.17
96.05
95.84"
95.98
96.40
95.50
96.43
SCIFAR
63.24
65.60
67.78
67.63
71.85"
67.73*
70.41
67.29*
71.05
SCIFAR (GRU)
71.30*
64.61
69.81*
70.10
70.74*
70.20*
71.40*
69.17"
71.04
Table 3: ace.
acc. on pixel-by-pixel image classification benchmarks. Top: Recurrent baselines and variants. Middle: Non-recurrent sequence models with global receptive field. Bottom: Our methods.
Model
SMNIST
PMNIST
SCIFAR
LSTM (ours)
98.9
95.11
63.01
Dilated GRU Chang et al.
2017
99.0
94.6
IndRNN et 2018a)
99.0
96.0
--MTTM LZae with Auxiliary Loss)
Trinh et al. 2018)
98.4
95.2
72.2
Transformer (Trinh al. 2018)
98.9
97.9
62.2
Temporal convolution network Bai et al.
2018a
99.0
97.2
TrellisNet (Bai et al. 2018b)
99.20
98.13
73.42
UR-LSTM
99.28
96.96
71.00
UR-LSTM Zoneout Krueger et al.
2016
99.21
97.58
74.34
UR-GRU + Zoneout
99.27
96.51
74.4
From Sections 3.1 and 3.2 we draw few conclusions about the comparative performance of different
gate modifications. First, the refine gate is consistently better than comparable master gates. solves the synthetic memory tasks but is worse than any other variant outside of those. We find ordered (cumax) gates to be effective, but speed issues prevent us from using them in more complicated tasks. UR- gates consistently among the best performing and most stable.
8

---

Under review as conference paper at ICLR 2020
Model
Valid.
Test
Methed
CASIM
LSTM
34.3
35.8
Ca2-L5TH
C-LSTM
35.0
36.4
1STM
C-LSTM (T (maa== =8)
34.3
36.1
C-LSTM (T maa =11] 11)
34.6
35.8
(RLLTTM
OM-LSTM
34.0
34.7
U-LSTM
33.8
34.9
UR-LSTM
33.6
34.6
333800
200006
300000
400063
S00000
Blenation
(a) Perplexities on the WikiText- 103 dataset.
(b) Validation learning curves, illustrating training speed and generalization (i.c. overfitting) behavior.
3.3 LANGUAGE MODELING
We consider word-level language modeling on the WikiText-100 103 dataset, where (i) the dependency lengths are much shorter than in the synthetic tasks, (ii) language has an implicit hierarchical structure and timescales of varying lengths. We evaluate our gate modifications against the exact hyperparrrmet of SOTA SSTM-based baseline (Rae et al. (2018) without additional tuning (Appendix DD.. Addi- tionally, we compare against ON-LSTM, which was designed for this domain (Shen et al. 2018), and chrono initialization, which addresses dependencies of a particular timescale opposed to timescale- agnostic UGI methods. In addition to our default hyperparameter-Fie frre initialization, we tested models with the chrono hyperparameter T max manually set to 8 and 11, values previously used for language modeling to mimic fixed biases of about 1.0 and 2.0 respectively (Tallec & Ollivier 2018).
Table/6ashow Validation and Test set perplexities for various models. We find that the OM-LSTM, U-LSTTM and UR-LSTM improve over the standard LSTM with no additional tuning. However, although the OM-LSTM was designed capture the hierarchical nature of language with the cumax activation, it does not perform better than the U-LSTM and UR-LSTM. The chrono initialization with our default initialization strategy far too large. While manually tweaking the Tmaz hyperparameter helps, it is still far from any "GI-based methods. We attribute these observations to the nature of language having dependencies on multiple widely-varying timescales, and that UGI is enough to capture these without resorting to strictly enforced hierarchies such as in OM-LSTM.
3.4 REINFORCEMENT LEARNING
In many partially observable reinforcement learning (RL) tasks, the agent can observe only part of the environment at time and thus requires a memory model to summarize what it has seen previously. However, designing memory architectures for reinforcement learning problems has been challenging task (Oh et al. 2016: Wayne et al., 2018). These are usually based on an LSTM core to summarize what agent has seen into state.
We investigated if changing the gates of these recurrent cores can improve the performance of RL agents, especially on difficult tasks involving memory and long-term credit assignment. We chose the Passive and Active Image Match tasks from Hung et al. (2018) using A3C agents (Mnih et al. 2016). In these tasks, agents are either initially shown colored indicator (Passive) or must search for (Active), before being teleported to a room in which they must press a switch with matching color to receive reward. In between these two phases is an intermediate phase where they can acquire distractor rewards, but the true objective reported is the final reward in the last phase. Episodes last 450 600 steps, so these tasks require memorization and credit assignment across long sequences.
Hung et (2188) evaluated agents with different recurrent cores: the basic LSTM, the DNC (an LSTM with memory), and the RMA (which also uses an LSTM core). We modified each of these with our gates. Figure shows the results of different models the Passive Matching and Active Matching tasks without distractors. These are the most similar to the synthetic tasks (Sec. 3.I., and we found that those trends largely transferred to the RL setting even with several additional confounders present such as agents learning via RL algorithms, being required to learn relevant features from pixels rather than being given the relevant tokens, and being required to explore in the Active Match case.

---

Under review conference paper at ICLR 2020
CEESTM -ME
-URLIM
Ralwnee Diyn
2,53
(a) Passive Match without Distractor Rewards
(b) Active Match without Distractor Rewards
Figure 7: We evaluated the image matching tasks from Hung et al. (018)). which test memorization and credit assignment, using A3C agent (Mnih et al., (016) with LSTM policy core. We observe that general trends from the synthetic tasks (Section (31)) transfer to this reinforcement learning setting.
(a) Active Match with Distractor Rewards LSTM
(b) Active Match with Distractor Rewards RMA
Figure 8: The addition of distractor rewards changes the task and relative performance of different gating mechanisms. For both LSTM and RMA recurrent cores, the UR- gates still perform best.
We found that the UR- gates substantially improved the performance the basic LSTM core on both Passive Match and Active Match tasks, with without distractor rewards. On the difficult Active Match task, it was the only method to achieve better than random behavior. Figure Bhowoss performance of LSTM and RMA cores the harder Active Match task with distractors. Here the UR- gates again learn the fastest and reach the highest reward. In particular, although the RMA is a memory architecture with explicit memory bank designed for long-term credit assignment, its performance also improved.
3.5 ADDITIONAL RESULTS AND EXPERIMENTAL CONCLUSIONS
Appendix (E.1) shows additional synthetic experiment investigating the effect of refine gates saturation. Appendix EE.3 has results on a program execution task, which interesting for having explicit long and variable-length dependencies and hierarchical structure. It additionally shows another very different gated recurrent model where the UR- gates provide consistent improvement.
Finally, we would like to comment on the longevity of the LSTM, which for example was frequently found to outperform newer competitors when tuned better (Melis et al., 2017). Although many improvements have been suggested over the years, none have been proven to be robust the LSTM across an enormously diverse range of sequence modeling tasks. By experimentally starting from well-tuned LSTM baselines, we believe our simple isolated gate modifications actually be robust improvements. In Appendix BB.3)and B.4 we offer a few conclusions for the practitioner about the other gate components considered based on our experimental experience.
DISCUSSION
In this work, we introduce, analyze, and evaluate several modifications to the ubiquitous gating mechanism that appears recurrent neural networks. We describe theoretically-justnn methods that improve on the standard gating method by alleviating problems with initialization and optimization. The mechanisms considered include changes on independent axes, namely initialization method and auxiliary gates, and we perform extensive ablations our improvements with previously considered modifications. Our main gate model robustly improves on standard gates across many different tasks and recurrent cores, while requiring less tuning Finally, emphasize that these improvements completely independent of the large body of research on neural network architectures that use gates, and hope that these insights can be applied to improve machine learning models at large.
10

---

Under review as conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0477 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:/803.0/271 201
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint urXivv/810..00 06682, 201 8b.
Yoshua Bengio, Patrice Simard, Paolo Frasconi, et al. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-16,, 1994.
Sarath Chandar, Chinnadhurai Sankar, Eugene Vorontsov, Samira Ebrahimi Kahou, and Yoshua Bengio. Towards non saturating recurrent units for modelling long-term dependencies. arXiv preprint arXivv:902.06700 2019.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 77-87, 2017.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:14066 1078, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiiv:442..355 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv: 1609.01 704, 2016.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint 111.1000000008600000 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1406-1415, 2018.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv: 1400,5401 2014.
Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odysscy. IEEE transactions on neural networks and learning systems, 28(10): 2016.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In International conference machine learning, pp. 3059-3068. 2016.
Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory augmented neural networks with wormhole connections. arXiv preprint arXiv: 011.77071..78... 2017.
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. arXiv preprint arXiv:1602. 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.

---

Under review as conference paper at ICLR 2020
Chia-Chun Hung, Timothy Lillicrap. Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting 2018.
value. arXiv preprint
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 161Xiv:661..01 144, 2016.
Rafal Jozefowicz, Wojciech Zaremba, and llya Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning, pp. 2342-2350, 2015.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experi- ence replay in distributed reinforcement learning. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1r12. 6980, 2014.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguisties, 6:317-328, 2018.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. clockwork rnn. arXiv preprint 2014.
David Krueger, Tegan Maharaj. János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXiv: 1reXivv:606.00 01000,035550 2016.
Quoc V Le. Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint 0rXiv:1504.00941 2015.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmnn): Building a longer and deeper mn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5457-5466, 2018a.
Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-Yan Liu. Towards binary-valued gates for robust lstm training. arXiv preprint arXivv:8066 02988, 2018b.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: J6/ 1.00712, 2016.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv: 05589 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.07843, 2016.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1933 2016.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak I Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv: arXiv: 00..000..9.... 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188), 2019.
Jack W Rae, Chris Dyer, Peter Dayan, and Timothy P Lillicrap. Fast parametric learning with activation memorization. arXiv preprint arXiv: 100.: 1803. 10049, 2018
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski. Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 7299-7310, 2018.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv: hat::/8//00 1810.0930.. 2018.
12

---

Under review as conference paper at ICLR 2020
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-195) 2014.
Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv: 2018.
Trieu H Trinh, Andrew Dai, Minh-Thang Luong, and Quoe V Le. Learning dengerr-eemmddeeeeeeeeeee in mns with auxiliary losses. arXiv preprint arXiv:/803.001443 2018.
Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint arXiv: 04849, 2018.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, et al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint 1rXiv:180..1070000 2018.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv: 1410.3916, 2014.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv: 1.13::1400.4615,200000
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189-4198. JMLR. org, 2017.
A PSEUDOCODE
We show how the gated update in a typical LSTM implementation can be easily replaced by UR- gates. The following snippets show pseudocode for the the gated state updates for vanilla LSTM model (top) and UR-LSTM (bottom).
forget bias 1.0 hyperparameter
f, i, Linear (x, prev_ hidden)
f_ sigmoid forget_bias)
i sigmoid (1)
next_cell f_ prev_cell 1_ tanb (u)
next_hiddes = o tanh (next_cell)
Listing 1: LSTM
Initialization
u= random. uniform (Loioornnoo=:///ddddde..... size, high=1-1/hidden size, hidden_ size)
forget_bias = Log
Recurrent update
f, r, u, 0 Linear (x, prev_hidden)
f_ -sigmoid (f forget_bias)
sigmoid
forget_bias
g 2+r_*f_ ff (1-2vz_)<f_x22
next next_cell g prev_cell + (1-g) x tanh (u)
next hidden o tanh (next_coll)
Listing 2: UR-LSTM
13

---

Under review as conference paper at ICLR 2020
B FURTHER DISCUSSION ON RELATED METHODS
Section 2.5 briefly introduced chrono initialization Tallec & Ollivier 2018) and the ON-LSTM (Shen et al. 2018), closely related methods that modify the gating mechanism of LSTMs. We provide more detailed discussion on these Sections B.3 and B.4 respectively. Section B. I has a more thorough overview of related work on recurrent neural networks that address long-term dependencies or saturating gates.
B.1 RELATED WORK
Several methods exist for addressing gate saturation or allowing more binary activations. Gulcehre piece-wise linear functions with noise in order to allow the gates operate in saturated regimes. [i et al. (2018b) instead use the Gumbel trick (Maddison et al. (2016] 2016), a technique for learning discrete variables within a neural network, to train LSTM models with discrete gates. These stochastic approaches can suffer from issues such as gradient estimation bias, unstable training, and limited expressivity from discrete instead of continuous gates. Additionally they require more involved training protocols with an additional temperature hyperparameter that needs to be tuned explicitly.
at. ) proposed
(0016JJang Jang et al.
Alternatively, gates can be removed entirely if strong constraints are imposed on other parts of the model. (Li et al., 2018a) use diagonal weight matrices and require stacked RNN layers to combine information between hidden units. A long line of work has investigated the use of identity or orthogonal initializations and constraints on the recurrent weights to control multiplicative gradients unrolled through time (Le et al. 2015 Arjovsky al., 2016 Henaff et al. 2016). Chandar et al. (2019) proposed another RNN architecture using additive state updates and non non-saturating activation functions instead of gates. However, although these gate-less techniques be used to alleviate the vanishing gradient problem with RNNs, unbounded activation functions can cause less stable learning dynamics and exploding gradients.
As mentioned, a particular consequence of the inability of gates to approach extrema is that gated recurrent models struggle to capture very long dependencies. These problems have traditionally been addressed by introducing new components to the basic RNN setup. Some techniques include stacking layers in hierarchy (Chung et al. (0166). adding skip connections and dilations (Koutnik et al.) 20144] Chang et al. (017)) using an external memory (Graves et al. [2014] Weston et al. [2014] Wayne et al. G018ehel et al. 2017). auxiliary uuii-uupeviiioo (Trinh et al. a 2018), and more. However, these approaches have not been widely adopted over the standard LSTM as they are often specialized for certain tasks, are not as robust, and introduce additional complexity. Recently the transformer model has been successful in many applications areas such as NLP (Radford al. [019 Dai et al. 2019). However, recurrent neural networks are still important and commonly used due their faster inference without the need to maintain the entire sequence in memory. We emphasize that the vast majority of proposed RNN changes are completely orthogonal to the simple gate improvements this work, and we do not focus on them.
A few other recurrent cores that use the basic gated update (1) but use more sophisticated update functions include the GRU, Reconstructive Memory Agent (RMA; et. al. (0018), and Relational Memory Core (RMC: Santoro et al.) 2018), which we consider in our experiments.
B.2 EFFECT OF PROPOSED METHODS ON TIMESCALES
We briefly review the connection between our methods and the effective timescales that gated RNNs capture. Recall that Section 2.2 defines the characteristic timescale of a neuron with forget activation fi as 1/(1-f)) fi), which would be the number of timesteps takes to decay that neuron by a constant.
The fundamental principle of gated RNNs is that the activations of the gates affects the timescales that the model can address; for example, forget gate activations near 1.0 are necessary to capture long-term dependencies.
Thus, although our methods were defined in terms of activations gt. it is illustrative to reason with their characteristic timescales 1/ 1/(1- (1-10))) gi) instead, whence both UGI and refine gate also have clean interpretations.
14

---

Under review as conference paper at ICLR 2020
First, UGI is equivalent initializing the decay period from particular heavy-tailed distribution, in contrast to standard initialization with a fixed decay period (1)o(by))
Proposition 1. UGI is equivalent to sampling the decay period D D=11/ 11 ft) from a distribution with density proportional to P(D =)) xx de -1-))/)>==" i.e. Pareto(ar= distribution.
On the other hand, for any forget gate activation ft with timescale D= 1/11111111 the refine gate fine-tunes it between D D=1/(1-f2 1 Proposition 2. Given forget gate activation with timescale D, the refine gate creates an effective forget gate with timescale in (D/2, D2) D²).
and 1/(1- ft)².
B.3 CHRONO INITIALIZ.ATINN
The chrono initialization
by 1 (og(o maz 1J))
(17)
b; b-=
(18)
was the first to explicitly attempt to initialize the activation of gates across distributional range. It was motivated by matching the gate activations to the desired timescales.
They also elucidate the benefits of tying the input and forget gates, leading the simple trick 18 for approximating tying the gates at initialization, which we borrow for UGL. (We remark that perfect tied initialization can be accomplished by fully tying the linear maps Cj,Ci, but (18) is a good approximation.)
However, the main drawback of CI is that the initialization distribution too heavily biased toward large terms. This leads to empirical consequences such as difficult tuning (due to most units starting in the saturation regime, requiring different learning rates) and high sensitivity to the hyperparameter T Tmaz that represents the maximum potential length of dependencies. For example, Tallec & Ollivier (2018) set this parameter according to different protocol for every task, with values ranging from to 2000. Our experiments used hyperparameterr-fre method to initialize Tmaz (Section 33. and we found that chrono initialization generally severely werreemphsiiess long-term dependencies if Tmaz is not carefully controlled.
different workaround suggested by Tallec & Ollivier 20188) is to sample from kk) x Klog (k+ and setting by by= log(").). Note that such an initialization would be almost equivalent to sampling the decay period from the distribution with density P(D z)) (xlog"z) (since the decay period is (1- = 1 +e ))) This parameter-free initialization is thus similar in spirit to the uniform gate initialization (Proposition I but from much heavier-tailed distribution that emphasizes very long-term dependencies.
These interpretations suggest that it plausible to define family of Pareto-like distributions from which to draw the initial decay periods from, with this distribution treated as hyperparameter. However, with no additional prior information on the task, we believe the uniform gate initialization be the best candidate, as it 1. is simple distribution with easy implementation, 2. has characteristic timescale distributed as an intermediate balance between the heavy-tailed chrono initialization and sharply decaying standard initialization, and 3. is similar to the ON-LSTM' 's cumax activation, in particular matching the initialization distribution of the cumax activation.
Table 4 summarizes the decay period distributions at initialization using different activations and initialization strategies.
In general, our experimental recommendation for CI is that it can be better than standard initialization or UGI when certain conditions are met (tasks with long dependencies and nearly fixed-length sequences as in Sections 3.1 3.4) and/or when it can be explicitly tuned (both the hyperparameter Tmaz well as the learning rate to compensate for almost all units starting in saturation). Otherwise, we recommend UGI or standard initialization. We found scenarios where outperformed UR- gates.
B.4 ON-LSTM
In this section we elaborate on the connection between the mechanism of Shen et al. (2018) and our methods. We define the full ON-LSTM and show how its gating mechanisms can be improved. For
15

---

Under review as conference paper at ICLR 2020
Table 4: Distribution of the decay period D D= =1 (1-f)-1 using different initialization strategies.
Initialization method
Timescale distribution
Constant bias =b
=b
P(D) = x 1 [ 1e=e)
Chrono initialization (known timescale maxx)
P(D= x {x [2,4 manl]
Chrono initialization (unknown timescale)
P(D=z) X zlog²x
Uniform gate initialization cumax activation
P(D = X
P(D x)) X
example, there is a remarkable connection between its master gates and our refine gates 1 independently of the derivation of refine gates in Section 2.3] we show how specific way of fixing the normalization of master gates becomes equivalent to a single refine gate.
First, formally define the full ON-LSTM. The master gates cumax-activation gate
ft cumax ( (E
(19)
? = cumax (x,,,
(20)
These combine with independent pair of forget and input gates fi, meant to control fin-ggrained behavior, create effective forget/input gate fe, which are used update the state (equation or (5)
W: = f Г. - foows + (f+ -ia)
(21)
(22)
0, = iow (ze-wr). wt).
(23)
As mentioned in Section B. this model modifies the standard forget/input gates in two main ways, namely ordering the gates via the cumax activation, and supplying an auxiliary set of gates controlling fine-grained behavior. Both of these are important novelties and together allow recurrent models to better capture tree structures.
However, the UGI and refine gate can be viewed as improvements over each of these, respectively, demonstrated both theoretically (below) and empirically (Sections and E. even on tasks involving hierarchical sequences.
Despite having the same parameter count and asymptotic efficiency as standard sigmoid gates, cumax gates seem noticeably slower and less stable in practice for large hidden sizes. Additionally, using auxiliary master gates creates additional parameters compared to the basic LSTM. Shen et al.) 2018) 8) alleviated both of these problems by defining a downsize operation, whereby neurons are grouped chunks of size C, each of which share the same master gate values. However, this also creates additional hyperparameter.
Ordered gates
The speed and stability issues can be fixed by just using the sigmoid non-linearity instead of cumax. To recover the most important properties of the coinax-aativationn at multiple timescales the equivalent sigmoid gate can be initialized so as to match the distribution of cumax gates at initialization. This is just uniform gate initialization (equation (9).
However, we believe that the cumax activation is still valuable in many situations if speed and instability not issues. These include when the hidden size is small, when extremal gate activations are desired, or when ordering needs to be strictly enforced to induce explicit hierarchical structure. For example, Section 3.. shows that they can solve hard memory tasks by themselves.
Master gates interpretation of gated recurrent models shows that they are discretization of continuous differential equation. This leads to the leaky RNN model h = (1 - a h2 au,, where u, is the update to the model such as tanh(W, Whbh+b) Learning function of the current time step leads the
We observe that the magnitudes master gates are suboptimally normalized. A nice
16

---

Under review as conference paper at ICLR 2020
simplest gated recurrent mode
ft = o (L " = ( h = - (1-
Tallee & Ollivier show that this exactly corresponds to the discretization that is invariant to time warpings and time rescalings. In the context of the LSTM, this interpretation requires the values of the forget and input gates to be tied so that ft + i = 1. This weight-tying is often enforced, for example in the most popular LSTT variant, the GRU (Cho et al. 2014),
differential equation
UR- gates. In a large-scale LSTM architecture search, it was found that removing the input gate was not significantly detrimental (Greft et al. 2016).
However, the ON-LSTM does not satisfy this conventional wisdom that the input and forget gates should sum to close
Proposition 3. At initialization, the expected value the the average effective forget gate activation fe is 5/6.
Let us consider the sum the effective forget and input gates at initialization. Adding equations (2
and 23 yields
= (f
2wx)
= f (Je+ - ow)
Note that the master gates 19
200 sum 1 in expectation at initialization, as do original forget and
input gates. Looking at individual units in the ordered master gates, we have E fu) 2 E, = Thus the above simplifies
Blfc+Lg] = Eur
E - (1
n
E E p00)
+2(0)
xdx+ S( 22
x1
5
6
The gate normalization can be fixed by re-scaling equations and 23 It turns out that tying the master gates and re-scaling is exactly equivalent to the mechanism refine gate. In this equivalence, the role of the master and forget gates of the ON-LSTM are played by our forget and refine gate respectively.
Proposition 4. Suppose the master gates fr are and the equations 22 23) defining the effective gates fi are rescaled such as to ensure B[fc+ at initialization. The resulting gate mechanism is exactly equivalent to that of the refine gate.
Consider the following set of equations where the master gates are tied (f++ = 1.ft and 23 are modified with an extra coefficient (rescaling in bold):
7 1
(24)
fe
(25)
i
Se- we + (Je (we)
(26)
i ie we + (hx (wr)
(27)
Now we have
fr + + -
=1+2(f1 2( + +44-)
'In the literature, this is called the JANET van der Westhuizen & Lasenby 2018). which is also equivalent to the GRU without reset gate (Chung et al. 2014 4) or recurrent highway network with depth I.= 1 (Zilly al. 2017).
17

---

Under review as conference paper at ICLR 2020
which has the correct scaling, i.c. Eife + id] at initialization assuming that Elfe + tc] at initialization.
But (26) can be rewritten as follows:
(f
-2ʃ-1-21-
-f22f-f-P1.-P--
-100-1-77
This is equivalent to the refine gate, where the master gate plays the role of the forget gate and the forget gate plays the role of the refine gate. It can be shown that in this case, the effective input gate à (7 is also defined through refine gate mechanism, where i, = 1- is refined by iz:
à= i (1-(1-))) (1-
Based experimental findings, in general would recommend the refine gate in place of the master gate.
B.5
GATE ABLATION DETAILS
For clarity, we formally define the gate ablations considered which mix and match different gate components.
We remark that other combinations are possible, for example combining CI with either auxiliary gate type, which would lead to CR- or CM- gates. Alternatively, the master or refine gates could be defined using different activation and initialization strategies. We chose not to consider these methods due to lack of interpretation and theoretical soundness.
O- This ablation uses the cumax activation to order the forget/input gates and has no auxiliary gates.
ft = (umax(()(xt,hhee
(28)
i = 1 - cumax (x),
(29)
We note that one difficulty with this in practice is the reliance on the expensive cumax, and hypothesize that this is perhaps the ON-LSTM's original motivation for the second set of gates combined with downsizing.
UM-
This variant of the ON-LSTM ablates the cumax operation on the master gates, replacing with sigmoid activation initialized with UGL Equations 19). (20) are replaced with
(0,1))
(30)
b b
(31)
= (1)
(32)
(33)
Equations
21
23
are then used to define effective gates fe. which are used in the gated update
or
OR- This ablation combines ordered main gates with auxilliary refine gate.
fo = (
(34)
re = or (Cr (-11)
(35)
9t 11 ft)
(36)
1=1-9h
(37)
ge used the effective forget and input gates.
18

---

Under review conference paper at ICLR 2020
C
ANALYSIS DETAILS
The gradient analysis in Figure effective gates
was constructed as follows. Let f.r.g be the forget, refine, and
--2rf+(1-2r)2
Then
V X42-77111HH441120000000011110---
| -
Substituting the relation
9-f²
2f(1-f)
this reduces to
Voll2= (
(9-
-
(0-F
21(1-f)
(25(1-)))
g-f² 9
=
2f(1-f)
Given the constraint f2 9
this function can be minimized and maximized in terms of g to produce the upper and lower bounds in Figure This was performed numerically.
2)
D EXPERIMENTAL DETAILS
To normalize the number of parameters used for models using master gates, i.e. the OM- and UM- gating mechanisms, we used a downsize factor on the main gates (see Se.tion[B.4). This was set to C= 16 for the synthetic and image classification tasks, and C 32 for the language modeling and program execution tasks which used larger hidden sizes.
D.1 SYNTHETIC TASKS
All models consisted of single layer LSTMs with 256 hidden units, trained with the Adam opti- mizer Kingma & Ba (2014) with learning rate 1e-3, Gradients were clipped at 1.0.
The training data consisted of randomly generated sequences for every minibatch rather than iterating through fixed dataset. Each method ran 3 seeds, with the same training data for every method.
Our version of the Copy task is a very minor variant other versions reported in the literature, with main difference being that the loss is considered only over the last 10 output tokens which need to be memorized. This normalizes the loss so that losses approaching indicate true progress. In contrast, this task is usually defined with the model being required to output dummy token at the first N + 10 steps, meaning it can be hard to evaluate performance since low average losses simply indicate that the model learns output the dummy token.
For Figure 3 the log loss curves show the median of 3 seeds, and the error bars indicate 60% confidence.
For Figure 4 each histogram represents the distribution of forget gate values of the hidden units (of which there are 256). The values are created by averaging units over time and samples, i.e., reducing minibatch of forget gate activations of shape (batch size size, sequence length, hi hidden size) over the first two diensions, to produce the average activation value for every unit.
D.2 IMAGE CLASSIFICATION
All models used a single hidden layer recurrent network (LSTM or Inputs x to the model were given batches sequence of shape (sequence length, channel s) (e.g. (1024,3)
19

---

Under review as conference paper at ICLR 2020
for CIFAR- 10), by flattening the input image left-to-right top-to-bottom. The outputs of the model of shape (sequence length, hidden size) were processed independently with a single ReLU hidden layer of size 256 before the final fully-connected layer outputting softmax logits. All training was performed with the Adam optimizer, batch size 50, and gradients clipped at 1.0. MNIST trained for 150 epochs, CIFAR-I 10 used 100 epochs over the training set.
Table 2 All models (LSTM and GRU) used hidden state size 512. Learning rate swept in {2e 4,5c
1.1e 3.2e-33 3} with three seeds each.
Table
2 reports the highest validation score found. The GRU model swept over learning rates {2e 4,5e ) all methods were unstable higher learning rates.
Figure 5 shows the median validation accuracy with quartiles (25/75% confidence intervals) over the seeds, for the best-performing stable learning rate (i.e. the one with highest average validation the final epoch).
Table 3
The UR-LSTM and UR-GRU used 1024 hidden units for the sequential and permuted MNIST task, and 2048 hidden units for the sequential CIFAR task. The vanilla LSTM baseline used 512 hidden units for MNIST and 1024 for CIFAR. Larger hidden sizes were found be unstable.
Zoneout parameters were fixed to reasonable default settings based on Krueger et al. (2016), which are 2, = 0.5.2 0.05 for LSTM and =0.1 for GRU. When zoneout used, standard Dropout (Srivas tava et al. 2014) with probability 0.5 was also applied to the output classification hidden layer.
D.3 LANGUAGE MODELING
Hyperparameters taken from Rae et al. (2018) tuned for the vanilla LSTM, which consist of (chosen parameter bolded out of sweep): {1,2] LSTM layer, (0.0,0.1,0.2 00.2,..3 embedding dropout, [yes,no} layer norm, and (shared,noo not shared] input/output embedding parameters. Our only divergence is using a hidden size of 3072 instead of 2048, which we found improved the performance of the vanilla LSTM. Training was performed with Adam learning rate le-3, gradients clipped to 0.1, sequence length 128, and batch size 128 on TPU. The LSTM state was reset between article boundaries. Figure oo smoothed validation perplexity curves showing the 95% confidence intervals over the last 1% of data.
Reinforcement Learning The Active Match and Passive Match tasks were borrowed from Hung al. (0118) with the same settings. For Figures and TO the discount factor in the environment was set to .96. For Figure 8 the discount factor was = .998. Figure [Ocorresponds to the full Active Match task in Hung et al. (2018), while Figure their version with small distractor rewards where the apples in the distractor phase give instead of reward.
D.4 PROGRAM EVALUATION
Protocol was taken from Santoro al. (2018) with minor changes the hyperparameter search. All models were trained with the Adam optimizer, the Mix curriculum strategy from Zaremba & Sutskever (0144), and batch size 128.
RMC: The RMC models used a fixed memory slot size of 512 and swept over (2,4) memories and {2,4] attention heads for total memory size 1024 or 2048. They were trained for 2e5 iterations.
LSTM: Instead of two-layer LSTMs with sweeps over skip connections and output concatenation, single-layer LSTMs of size 1024 or 2048 were used. Learning rate was swept (5e-4, le-3), and models were trained for 5e5 iterations. Note that training was still faster than the RMC models despite the greater number of iterations.
D.5 ADDITIONAL DETAILS
Implementation Details The inverse sigmoid function g can be unstable if the input is too close to (0,1}. Uniform gate initialization was instead implemented by sampling from the distribution u(1/d,1- 1/d] instead of (/(0,1), where d is the hidden size, to avoid any potential numerical edge cases. This choice is justified by the fact that with perfect uniform sampling, the expected smallest and largest samples would be 1/(d+ 1) and 1 (dd+1))
20

---

Under review as conference paper at ICLR 2020
For distributional initialization strategies, trainable bias vector sampled independently from the chosen distribution (i.e. equation (17) or and added/subtracted to the forget and input gate (2 before the non-linearity. Additionally, each linear model such as W - +oyg:ttWWhhll W had its own trainable bias vector, effectively doubling the learning rate on the pre-activation bias terms on the forget and input gates. This an artifact of implementation and not intended to affect performance.
The refine gate update equation
12 can instead be implemented as
9. - (1-
=2r (1-r)))
Permuted image classification In effort to standardize the permutation used in the Permuted MNIST benchmark, we use a particular deterministic permutation rather than a random one. After flattening the input image into one- one-dimensione sequence, This permutation sends the index to the index j such that j'' binary representation is the reverse of i's binary representation. The intuition is that if two indices i, are close, they must differ in their lower-order bits. Then the bit-reversed indices will be far apart. Therefore the bit-reversal permutation destroys spatial and temporal locality, which is desirable for these sequence classification tasks meant to test long- range dependencies rather than local structure.
apply the bit reversal permutation.
def bitreversal po2 (n)
m int (math. log (n) / math. log (2) perm = np, arange (n) reshape (n, 1) for i in range (m)
n1 perm. shape //
perm np.hstack (perm [:nil, perm[]]:])
return perm.squeeze (0)
def bitreversal www.errsa_pppmmuaaiiioooooo (n)
int (math. ceil www.t...tooo..oo.. (n) math. (2))
1 m perm bitreversal_pox (N)
return np.extract (perm < perm)
E ADDITIONAL EXPERIMENTS
E. SYNTHETIC FORGETTING
Figure the Copy task demonstrates that extremal gate activations necessary solve the task, and initializing the activations near 1.0 is helpful.
This raises the question: what happens if the initialization distribution does not match the task at hand; could the gates learn back to more moderate regime? We point out that such a phenomenon could occur con-pathologically on more complex setups, such a scenario where model trains remember on a Copy-like task and then needs to "unlearn" as part of a meta-learning or continual learning setup.
Here, we consider such synthetic scenario and experimentally show that the addition of a refine gate helps models train much faster while in saturated regime with extremal activations. We also point to the poor performance of C- outside synthetic memory tasks when using our high hyperparameter- free initialization more evidence that it is very difficult for standard gates to unlearn undesired saturated behavior.
For this experiment, we initialize the biases of the gates extremely high (effective forget activation Rs (6). We then consider the Adding task (Section 3. of length 500, hidden size 64, learning rate le-4. The R-L.TM is able to solve the task, while the L.STM is stuck after le4 iterations.
E.2 REIFFORCEMENT LEARNING
an Figures 7 and evaluated our gating methods with the LSTM and RMA models on the Passive Match and Active Match tasks, with and without distractors. We additionally the agents even harder
21

---

Under review as conference paper at ICLR 2020
12.
20 J
10
4-
|II
0--
0.34
0.96
0.98
1.00
0.355 0.360 0.365 0.570 0.375 0.380 0.985 0.990 0.395 1.000
(a) R-LSTM
(b) LSTM
Figure Distribution of forget gate activations after extremal initialization, and training on the Adding task. The UR-LSTM is able to learn much faster in this saturated gate regime while the LSTM does not solve the task. The smallest forget unit for the UR-LSTM after training has characteristic timescale over an order of magnitude smaller than that of the LSTM.
Stope
Figure 10: The full Active Match task with large distractor rewards, using agents with LSTM or DNC recurrent cores.
version of the Active Match task with larger distractor rewards (the full Active Match from Hong et al.) (2018)) Learning curves are shown in Figure I0] Similarly to the other results, the UR- gated core is noticeably better than the others. For the DNC model, it is the only one that performs better than random chance.
E.3 PROGRAM EXECUTION
The Learning to Execute (Zarebba & Sutskever 2014) dataset consists of algorithmic snippets from a programming language of pseudo-code. An input is a program from this language presented one character at time, and the target output is numeric sequence of characters representing the execution output of the program. There are three categories of tasks: Addition, Control, and Program, with distinctive types of input programs. We use the most difficult setting from Zaremba & Sutskever (2014), which uses the parameters nesting=4, control structure and base length of numeric literals, respectively. Examples of input programs shown in previous works Zaremba & Sutskever) 2014 Santoro et al. 2018).
length=9, referring to the nesting depth of
We are interested in this task for several reasons. First, are interested in comparing against the C- and OM- gate methods, because
The maximum sequence length is fairly long (several hundred tokens), meaning our T Tmaz heuristic for C- gates is within the right order of magnitude of dependency lengths.
The task has highly variable sequence lengths, wherein the standard training procedure randomly samples inputs of varying lengths (called the "Mix" curriculum in Zaremba & Sutskever 22014)). Additionally, the Control and Program tasks contain complex control flow and nested structure. They are thus a measure of a sequence model's ' ability to model dependencies of differing lengths, well as hierarchical information. Thus we interested in comparing the effects of UGI methods, as well as the full OM- gates which are designed for hierarchical structures (Shen al. 2018).
22

---

Under review as conference paper at ICLR 2020
Finally, this task has prior work using different type of recurrent core, the Relational Memory Core (RMC), that we also use as a baseline to evaluate our gates on different models Santoro et al. 2018). Both the LSTM and RMC were found to outperform other recurrent baselines such as the Differential Neural Computer (DNC) and EntNet.
Training curves shown in Figure 11 which plots the median accuracy with confidence intervals. We point out a few observations. First, despite having a Tmaz value on the right order magnitude, the C- gated methods have very poor performance across the board, reaffirming the chrono initialization's high sensitivity to this hyperparameter.
Second, the U-LSTM and U-RMC are the best methods the Addition task. Additionally, the UR-RMC vs. RMC on Addition is one of the very few tasks we have found where generic substitution of the UR- gate does not improve on the basic gate, We have not investigated what property of this task caused these phenomena.
Aside from the U-LSTM on addition, the UR-LSTM outperforms all other LSTM cores. The UR-RMC is also the best core on both Control and Program, the tasks involving hierarchical inputs and longer dependencies. For the most part, the improved mechanisms of the UR- gates seem to transfer to this recurrent core as well. We highlight that this is not true of similar gating mechanisms. In particular, the OM-LSTM, which is supposed to model hierarchies, has good performance on Control and Program as expected (although not better than the UR-LSTM), However, the OM- gates' performance plummets when transferred to the RMC core.
Interestingly, the -LSTM cores are consistently better than the -RMC versions, contrary to previous findings on easier versions this task using similar protocol and hyperparameters Santoro et al. (2018). We did not explore different hyperparameter regimes on this more difficult setting.
23

---

Under review as conference paper at ICLR 2020
Addition
Control
Program
0.5
0.5
0.75
Method
Accuracy
0.4
LSTM
C
0.50
0.3
U-LTMM
URASTH
0.25
2.5e5
5.0e5
7.5e5
1.0u6
2.5e5 5.0e5
7.5e5
1.066
0
2.5e5
5.0e5
7.545
1.0e6
Iteration
(a) LSTM Learning to Execute (nesting=4, length=9)
Program
0.4
Method
Accuracy
0.5
AMC
0.3
C-RMC
OM-RMC
UR-RMC
0.3
1.0e5 2.0e5 4.0w5
3.0e5 4.0w5 Iteration
5.0e5
(b) RMC Learning to Execute (nesting=4, length=9)
Figure 11: Program Execution evaluation accuracies.
24


Reviews:
[{"review_id": "rkl3__O0tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve the learnability of the gating mechanism in RNN by two modifications on the standard RNN structure, uniform gate initialization and refine gate. The authors give some propositions to show that the refine gate can maintain an effective forget effect within a larger range of timescale. The authors conduct experiments on four different tasks and compare the proposed modification with baseline methods.\n\nStrong points:\n1. The authors propose a new refine structure that seems to have a longer \"memory\".\n2. The authors designed a good synthetic experiment to demonstrate whether the proposed refine structure can help to remember information in longer sequence.\n\nWeak points:\n1. There are several parts in the experiment that are not very convincing.\n     a. In Figure 3(a), where are the other baselines? Are they performing too badly so that they can not show up in the figure? It needs more explanation.\n     b.  In Figure 3(b), actually a lot of methods are performing similar, while some methods converge similarly. What is the reason?\n2. It is not defined why the uniform gate initialization works.\n3. The proposed results actually not always perform the best. For instance, in Table 3, purely using the UR-LSTM only achieve good results on sMNIST. What is the reason? The proposed method seems not very general.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxC9rZ2FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces two novel techniques to help long term signal propagation in RNNs. One is an initialization strategy which uses inverse sigmoid function to avoid the decay of the contribution of the input earlier in time and another is a new design of a refine gate which pushes the value of the gate closer to 0 or 1.  The authors conduct exhaustive ablation and empirical studies on copy task, sequential MNIST, language modeling and reinforcement learning. \n\nThough the experiment section is solid, I still vote for rejection for the following reasons:\n\n1. The writing in the description of the UGI and refine fate is not clear.\na. The authors compares UGI to standard initialization but where is the standard initialization? I do not see \"standard initialization\" clearly defined in the paper.\nb. I am not convinced how the UGI gate help avoid decaying of the input. There is a proposition 2 trying to explain some part of the mechanism of UGI. But the proposition is never proved anywhere and I am not sure why this proposition is important. More explanations are needed. Also this proposition is far away from the place the authors introduce the UGI. The authors may want to refer it in the place introducing UGI.\nc. Similar to proposition 2, proposition 3 is not explained and proved in the paper. It is hard for me to analyze the importance of these two propositions. Overall, propositions 2 and 3 look isolated in the section.\nd. Proposition 1 looks like a definition. Not sure why the authors name it as a proposition.\n\n2. Even though the title of the paper is \"improving the gating mechanism of recurrent neural networks\", the authors try to solve signal propagation problems. It is unclear why \"gate\" is important. Maybe other designs of the recurrent neural network can satisfy better the desiderata the authors want. Based on my limited knowledge, the initialization the authors mention (saturation) is exactly from the need of using a sigmoid gate. The importance of using \"gate\" should be discussed.\n\n3. The authors shrink the space before and after headings in the paper. I think this is not allowed in ICLR. It would be better that the authors correct the spacing in the revised version.\n\n\nMinors:\n1. page 1 second paragraph: repeated “proven”\n2. page 1 second last paragraph: “due” -> “due to” \n3. page 2 second last paragraph: repeated “be”\n4. page 2 Equation (4) and (5): using some symbols like \\odot for element wise multiplication will be good for the readers.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1llO2HCKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper introduces and studies modifications to gating mechanisms in RNNs.\nThe first modification is uniform gate initialization. Here the biases in the forget and input gate bias are sampled such that after the application of the sigmoid the values are in the range (1/d, 1-1/d) where d is the dimensionality of the hidden space for the bias. The second modification is the introduction of a refine gating mechanism with a view to allow for gradients to flow when the forget gates f_t in the LSTM updates are near saturation. The idea is to create an effective gate g = f_t +/- phi(f_t, x_t). The paper proposes using phi (f_t, x_t) = f_t(1-f_t) * (2*r_t-1) where (r_t is between 0 or 1). The effect of such a change is that g_t can reach values of 0.99 when the value of f_t is 0.9 allowing gradients to flow more freely through the parameters that constitute the forget gate. Overall the change corresponds to improving gradient flow for the forget gate by interpolating between f_t^2 and (1-f_t)^2. i.e. the authors note that the result of these changes is that it corresponds to sampling biases from a heavier tailed distribution while the refine gate (by allowing the forget gate to reach values close to 0 and 1), allows for capturing information on a much longer time scale.\n\n\nThe paper studies various combinations of the two changes proposed to gating architectures. Other baselines include a vanilla LSTM, a Chrono initialized LSTM, and an ordered Neuron LSTM. The models are trained on several synthetic and real world tasks. On the copy and add tasks, the LSTMs that contain the refine gate converge the fastest. A similar story is observed on the task of pixel by pixel image classification. The refine gate was also adapted to memory architectures such as the DNC and RMA where it was found to improve performance on two different tasks.\n\nOverall, the paper is written well, I like the (second) idea of the refine gate and the contributions are explained in an accessible manner. While I'm not entirely convinced about the proposed initialization scheme but across the many different tasks tried, the use of the refine gate does appear to give performance improvements that lead me to conclude that this aspect of the work is a solid contribution to the literature.\n\nQuestions and comments:\n* This manuscript already quite long and has several formatting issues. Several of the figures are unreadable when printed. For example, every piece of text on Figure 2(d) is unreadable on paper. Figure 3 and 5 are difficult to read; they contain too many alternatives with a colour scheme that makes it difficult to distinguish between them -- consider displaying a subset of the options via a plot and using a table to display (# steps to convergence) as a metric instead. It also appears as if the caption for Table 6 is deleted?\n* I think that for this approach to work, two conditions need to be satisfied (a) there must be foreseeable improvements in the use of a forget gate that can reach values close to 0/1 for the task at hand and (b) r_t needs to function well despite not being too close to 0 or 1 (lest its parameters suffer from gradient flow issues).\n  * Was there any visualizations done on whether (a) happened? i.e. for the URLSTMs that performed well, were the values of the forget gate closer to 0/1 than the baselines?\n  * What were typical values of r_t, did the models need the refine gate to reach values close to 0 or 1 for the overall approach to work?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a new gating mechanism to improve gradient information propagation during back-propagation when training recurrent neural networks.

Strengths:
-The problem is interesting and important.
-The proposed method is novel.

Weaknesses:
-The justification and motivation of the UGI mechanism was not clear and/or convincing.
-The experimental validation is sometimes hard to interpret and the proposed improvements of the gating mechanism are not well-reflected in the quantitative results.
-The submission was hard to read and some images were initially illegible.

The authors improved several of the weaknesses but not to the desired level.

AC agrees with the majority recommendation to reject.
Decision: Reject