user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Overcoming the vanishing gradient problem in plain recurrent networks

Paper content:
Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs.
This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs.
We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.
We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks.
The RINs demonstrate competitive performance and converge faster in all tasks.
Notably, small RIN models produce 12%–67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.
Numerous methods have been proposed for mitigating the vanishing gradient problem including the use of second-order optimization methods (e.g., Hessian-free optimization BID15 ), specific training schedules (e.g., Greedy Layer-wise training BID20 BID7 BID24 ), and special weight initialization methods when training on both plain FFNs and RNNs BID3 BID16 BID13 BID10 BID26 BID11 .Gated
Neural Networks (GNNs) also help to mitigate this problem by introducing "gates" to control information flow through the network over layers or sequences. Notable
examples include recurrent networks such as Long-short Term Memory (LSTM) BID8 , Gated Recurrent Unit (GRU) BID1 , and feedforward networks such as Highway Networks (HNs) BID21 , and Residual Networks (ResNets) BID5 . One can
successfully train very deep models by employing these models, e.g., ResNets can be trained with over 1,000 layers. It has
been demonstrated that removing (lesioning) or reordering (re-shuffling) random layers in deep feedforward GNNs does not noticeable affect the performance of the network BID23 Noticeably, one interpretation for this effect as given by BID4 is that the functional blocks in HNs or ResNets engage in an Unrolled Iterative Estimate (UIE) of representations and that layers in this block of HNs or ResNets iteratively refine a single set of representations.In this paper, we investigate if the view of Iterative Estimation (IE) can also be applied towards recurrent GNNs (Section 2.1). We present
a formal analysis for GNNs by examining a dual gate design common in LSTM and GRU (Section 2.2). The analysis
suggests that the use of gates in GNNs encourages the network to learn an identity mapping which can be beneficial in training deep architectures BID6 BID4 .We propose a
new formulation of a plain RNN, called a Recurrent Identity Network (RIN) , that is encouraged to learn an identity mapping without the use of gates (Section 2). This network
uses ReLU as the activation function and contains a set of non-trainable parameters. This simple
yet effective method helps the plain recurrent network to overcome the vanishing gradient problem while it is still able to model long-range dependencies. This network
is compared against two competing networks, the IRNN (Le et al., 2015) and LSTM, on several long sequence modeling tasks including the adding problem (Section 3.1), Sequential and Permuted MNIST classification tasks (Section 3.2), and bAbI question answering tasks (Section 3.3). RINs show faster
convergence than IRNNs and LSTMs in the early stage of the training phase and reach competitive performance in all benchmarks. Note that the use
of ReLU in RNNs usually leads to training instability, and therefore the network is sensitive to training hyperparameters. Our proposed RIN
network demonstrates that a plain RNN does not suffer from this problem even with the use of ReLUs as shown in Section 3. We discuss further
implications of this network and related work in Section 4.
In this paper, we discussed the iterative representation refinement in RNNs and how this viewpoint could help in learning identity mapping.
Under this observation, we demonstrated that the contribution of each recurrent step a GNN can be jointly determined by the representation that is formed up to the current step, and the openness of the carry gate in later recurrent updates.
Note in Eq. 9, the element-wise multiplication of C t s selects the encoded representation that could arrive at the output of the layer.
Thus, it is possible to embed a special function in C t s so that they are sensitive to certain pattern of interests.
For example, in Phased LSTM, the time gate is inherently interested in temporal frequency selection BID17 .Motivated
by the analysis presented in Section 2, we propose a novel plain recurrent network variant, the Recurrent Identity Network (RIN), that can model long-range dependencies without the use of gates. Compared
to the conventional formulation of plain RNNs, the formulation of RINs only adds a set of non-trainable weights to represent a "surrogate memory" component so that the learned representation can be maintained across two recurrent steps.Experimental results in Section 3 show that RINs are competitive against other network models such as IRNNs and LSTMs. Particularly
, small RINs produce 12%-67% higher accuracy in the Sequential and Permuted MNIST. Furthermore
, RINs demonstrated much faster convergence speed in early phase of training, which is a desirable advantage for platforms with limited computing resources. RINs work
well without advanced methods of weight initializations and are relatively insensitive to hyperparameters such as learning rate, batch size, and selection of optimizer. This property
can be very helpful when the time available for choosing hyperparameters is limited. Note that we
do not claim that RINs outperform LSTMs in general because LSTMs may achieve comparable performance with finely-tuned hyperparameters.The use of ReLU in RNNs might be counterintuitive at first sight because the repeated application of this activation is more likely causing gradient explosion than conventional choices of activation function, such as hyperbolic tangent (tanh) function or sigmoid function. Although the
proposed IRNN BID13 reduces the problem by the identity initialization, in our experiments, we usually found that IRNN is more sensitive to training parameters and more unstable than RINs and LSTMs. On the contrary
, feedforward models that use ReLU usually produce better results and converge faster than FFNs that use the tanh or sigmoid activation function. In this paper,
we provide a promising method of using ReLU in RNNs so that the network is less sensitive to the training conditions. The experimental
results also support the argument that the use of ReLU significantly speeds up the convergence.During the development of this paper, a recent independent work BID27 presented a similar network formulation with a focus on training of deep plain FFNs without skip connections. DiracNet uses the
idea of ResNets where it assumes that the identity initialization can replace the role of the skip-connection in ResNets. DiracNet employed
a particular kind of activation function -negative concatenated ReLU (NCReLU), and this activation function allows the layer output to approximate the layer input when the expectation of the weights are close to zero. In this paper, we
showed that an RNN can be trained without the use of gates or special activation functions, which complements the findings and provides theoretical basis in BID27 .We hope to see more
empirical and theoretical insights that explains the effectiveness of the RIN by simply embedding a non-trainable identity matrix. In future, we will
investigate the reasons for the faster convergence speed of the RIN during training. Furthermore, we will
investigate why RIN can be trained stably with the repeated application of ReLU and why it is less sensitive to training parameters than the two other models.A ALGEBRA OF EQS. 8-9Popular GNNs such
as LSTM, GRU; and recent variants such as the Phased-LSTM BID17 , and Intersection RNN BID2 , share the same dual gate design described as follows: DISPLAYFORM0 where t ∈ [1, T ], H t = σ(x t , h t−1 ) represents the hidden transformation, T t = τ (x t , h t−1 ) is the transform gate, and C t = φ(x t , h t−1 ) is the carry gate. σ, τ and φ are recurrent
layers that have their trainable parameters and activation functions. represents element-wise
product operator. Note that h t may not be
the output activation at the recurrent step t. For example in LSTM, h t
represents the memory cell state. Typically, the elements
of transform gate T t,k and carry gate C t,k are between 0 (close) and 1 (open), the value indicates the openness of the gate at the kth neuron. Hence, a plain recurrent
network is a subcase of Eq. 14 when T t = 1 and C t = 0.Note that conventionally, the initial hidden activation h 0 is 0 to represent a "void state" at the start of computation. For h 0 to fit into Eq.
4's framework, we define an auxiliary state h −1 as the previous state of h 0 , and T 0 = 1, C 0 = 0. We also define another
auxiliary state h T +1 = h T , T T +1 = 0, and C T +1 = 1 as the succeeding state of h T .Based on the recursive
definition in Eq. 4, we can write the final layer output h T as follows: DISPLAYFORM1 where we use to represent element-wise multiplication over a series of terms.According to Eq. 3, and supposing that Eq. 5 fulfills the Eq. 1, we can use a zero-mean residual t for describing the difference between the outputs of recurrent steps: DISPLAYFORM2 Then we can rewrite Eq. 16 as: DISPLAYFORM3 Substituting Eq. 18 into Eq. 15: DISPLAYFORM4 We can rearrange Eqn. 20 to DISPLAYFORM5 The
term λ in Eq. 23 can be reorganized to, DISPLAYFORM6 B DETAILS IN THE ADDING PROBLEM EXPERIMENTS Average Estimation Error RIN 2-100 1st IRNN 2-100 1st LSTM 2-100 1st 0 100 200 300 400 500 600 700 800 layer 2 step index RIN 2-100 2nd IRNN 2-100 2nd LSTM 2-100 2nd DISPLAYFORM7 DISPLAYFORM8

assistant: text: We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates.