Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper investigates the iterative estimation view on gated recurrent networks (GNN). Authors observe that the average estimation error between a given hidden state and the last hidden state  gradually decreases toward zeros. This suggest that GNN are bias toward an identity mapping and learn to preserve the activation through time.
Given this observation, authors then propose RIN, a new RNN parametrization where the hidden to hidden matrix is decomposed as a learnable weight matrix plus the identity matrix.
Authors evaluate their RIN on the adding, sequential MNIST and the baby tasks and show that their IRNN outperforms the IRNN and LSTM models.

Questions:
- Section 2 suggests that use of the gate  in GNNs encourages to learn an identity mapping. Does the average iteration error behaves differently in case of a tanh-RNN ?
- It seems from Figure 4 (a) that the average estimation error is higher for RIN than IRNN and LSTM and only decrease toward zero at the very end. What could explain this phenomenon?
- While the LSTM baseline matches the results of Le et al., later work such as Recurrent Batch Normalization or Unitary Evolution RNN have demonstrated much better performance with a vanilla LSTM on those tasks (outperforming both IRNN and RIN). What could explain this difference in the performances?
- Unless I am mistaken, Gated Orthogonal Recurrent Units: On Learning to Forget from Jing et al. also reports better performances for the LSTM (and GRU) baselines that outperform RIN on the baby tasks with mean performances of 58.2 and 56.0 for GRU and LSTM respectively?

- Quality/Clarity:
The paper is well written and pleasant to read

- Originality:
Looking at RNN from an iterative refinement point of view seems novel.

- Significance:
While looking at RNN from an iterative estimation is interesting, the experimental part does not really show what are the advantages of the propose RIN. In particular, the LSTM baseline seems to weak compared to other works.
review_title: Iterative Estimation point of view on recurrent GNN

final_score: rating: 2: Strong rejection
review_content: Here are my main critics of the papers:

1. Equation (1), (2), (3) are those expectations w.r.t. the data distribution (otherwise I can't think of any other stochasticity)? If so your phrase "is zero given a sequence of inputs X1, ...,T" is misleading. 
2. Lack of motivation for IE or UIE. Where is your background material? I do not understand why we would like to assume (1), (2), (3). Why the same intuition of UIE can be applied to RNNs? 
3. The paper proposed the new architecture RIN, but it is not much different than a simple RNN with identity initialization. Not much novelty.
4. The experimental results are not convincing. It's not compared against any previous published results. E.g. the addition tasks and sMNIST tasks are not as good as those reported in [1]. Also it only has been tested on very simple datasets.


[1] Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations. Behnam Neyshabur, Yuhuai Wu, Ruslan Salakhutdinov, Nathan Srebro.
review_title: confusing analysis, little novelty

final_score: rating: 7: Good paper, accept
review_content: Summary: 
The authors present a simple variation of vanilla recurrent neural networks, which use ReLU hiddens and a fixed identity matrix that is added to the hidden-to-hidden weight matrix. This identity connection acts as a “surrogate memory” component, preserving hidden activations over time steps. 
The experiments demonstrate that this architecture reliably solves the addition task for up to 400 input frames. It also achieves a very good performance on sequential and permuted MNIST and achieves SOTA performance on bAbI.
The authors observe that the proposed recurrent identity network (RIN) is relatively robust to hyperparameter choices. After Le et al. (2015), the paper presents another convincing case for the application of ReLUs in RNNs.

Review: 
I very much like the paper. The motivation and architecture is presented very clearly and I am happy to also see explorations of simpler recurrent architectures in parallel to research of gated architectures!
I have a few comments and questions:
1) Clarification: In Section 2.2, do you really mean bit-wise multiplication or element-wise? If bit-wise, can you elaborate why? I might have missed something.
2) Why does the learning curve of the IRNN stop around epoch 270 in Figure 2c? Also some curves in the appendix stop abruptly without visible explosions. Were these experiments run until completion? If so, would it be possible to plot the complete curves?
3) I think for a fair comparison with LSTMs and IRNNs a limited hyperparameter search should be performed separately on all three architectures at least for the addition task. Optimal hyperparameters are usually model-specific. Admittedly, the authors mention that they do not intend to make claims about superior performance to LSTMs, however the competitive performance of small RINs is mentioned a couple of times in the manuscript.
Le et al. (2015) for instance perform a coarse grid search for each model.
4) I wouldn't say that ResNets are Gated Neural Networks, as the branches are just summed up. There is no (multiplicative) gating as in Highway Networks.
5) I think what enables the training of very deep networks or LSTMs on long sequences is the presence of a (close-to-)identity component in forward/backward propagation, not the gating. The use of ReLU activations in IRNNs (with identity initialization of the hidden-to-hidden weights) and RINs (effectively initialized with identity plus some noise) makes the recurrence more linear than with squashing activation functions.
6) Regarding the absence of gating in RINs: What is your intuition on how the model would perform in tasks for which conditional forgetting is useful. Consider for example a task with long sequences, outputs at every time step and hidden activations not necessarily being encouraged to estimate last step hidden activations. Would RINs readily learn to reset parts of the hidden state?
7) Henaff et al. (2016) might be related, as they are also looking into the addition task with long sequences.

Overall, the presented idea is novel to the best of my knowledge and the manuscript is well-written. I would recommend it for acceptance, but would like to see the above points addressed (especially 1-3 and some comments on 4-6). After a revision I would consider to increase the score.

References:
Henaff, Mikael, Arthur Szlam, and Yann LeCun. "Recurrent orthogonal networks and long-memory tasks." In International Conference on Machine Learning, pp. 2034-2042. 2016.
Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015).
review_title: Simple trick to improve the training of simple RNNs. Novel idea and well presented, but the experimental evaluation could be improved

Meta-review: The authors propose to use identity + some weights in the recurrent connections to prevent vanishing gradients. The reviewers found the experiments to have weak baselines, weakening the claims of the paper.
Decision: Reject