Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Motivated by the sub-optimality of using orthogonal recurrent  matrix in RNNs with nonlinearity and noise, the authors look into non-normal alternatives, in particular matrices with chain-like structure in preserving memory in RNNs. The authors compare normal and non-normal RNNs on several sequential benchmark datasets, and show that non-normal RNNs perform better than their normal counterpart. 

The paper is easy to follow. The novelty of the work is limited though. The chain structure was introduced in Ganguli et al. (2008). The work studies the benefit of initializing recurrent weights in nonlinear RNNs with these chain-like structures.

Chen et. al. (2018) already pointed out the limitation of orthogonal initialization alone for nonlinear RNNs, and proposed closed-form initialization for RNNs with different activation functions. It would be worthwhile to include a comparison to that method.

In experiment section 2.3.1, it would be helpful to include comparison of performance of chain with feedback using different beta values to confirm the intuition that stronger feedback strength would negatively impact the memory. 

Results in section 2.3.2 Table 1 are not exactly align with the story. Do the authors have any intuition on why the chain with feedback perform better than the chain variant. 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The focus of this paper is on exploring non-normal initializations for training vanilla RNN for sequential tasks. They show on 3 different tasks, and a real-world LM task that  non-normal initializations of vanilla RNNs outperform their orthogonal counter-parts when particular forms of initialization are considered. 


Although the results for sequence task do not outperform the gated counterparts, the authors present an interesting exploration of initializing non-normal RNNs that outperform the orthogonal counterparts. It is good to see this line of work being explored as an alternative to exploring more complex architectures with many more parameters than necessary for the task. 

Strengths:
    1. The paper explores non-normal RNNs and demonstrates on  3 synthetic tasks - copy, addition and pMNIST - how with careful initialization the proposed approach outperforms their orthogonal initialization counterpart. This line of experimentation is interesting as it potentially opens the door for more expressive modeling for sequential tasks by expanding the solution space of the weight matrices being learnt i.e orthogonal matrices are a special case.
    2. The authors do a great job in motivating the paper, and the explanation is clear and easily understandable. The toy simulations in Section2.2 really helps drive the reasoning behind why chain initialization improves over orthogonal initialization.
    3. Based on the insight from trained RNNs where the trained  weights exhibit a chain like structure, the authors attempt to modify the LSTM gate initializations well. However, they do not see any specific gain by doing so, and moreover they show analysis that demonstrate that the LSTM gates do not learn these chain like structures. However, they do have insight into the regularities of these learnt weights which could potentially open the door for more interesting initialization methods for training such gated architectures.


Issues to be addressed in the paper:
1. The plots are quite small and hard to follow. Can the authors enlarge these so they span the entire page? Also, for pMNIST it would be good to provide accuracy scores as well as a function of the training epochs.Finally, it would be good to include a comparison against LSTMs (and even Transformer networks) so it is easier for the reader to see where these approaches stack against architecture changes.
2. The authors are missing a reference to this  work - http://proceedings.mlr.press/v48/henaff16.pdf  - which provides empirical analysis for the 3 synthetic tasks to test the ability of vanilla RNNs for solving long span sequential tasks. 
3. What about stability of these non-normal RNNs? For example, if we perturb the inputs to the training for the LM task how much variance do we see in the performance of these models? 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Contributions:
 This paper proposes to explore nonnormal matrix initialization in RNNs. Authors demonstrate on various tasks (Copy/Addition, Permuted-SMNIST, PTB, enwik8) that chain-like nonnormal matrix initializations can outperform orthogonal or identity initialization in vanilla RNNs. However, nonnormal RNNs underperform their gated counterpart such as LSTM. Authors also show results where they use their initialization scheme in update gate of a LSTM.

Comments:
The paper is well written and pleasant to read. The paper structure could be a bit improved. For instance, section 2 is named “Results” while 2.1 which take significant part of the section is about some prior results in (Ganguli et al. 2018). It would be better to have it under an explicit prior work section. 

The description of the experiments reported in Figure 2. is a bit vague: what is the training/evaluation data?, do you train all the model parameters or only the linear layer?, what is the type of noise used? It is unclear to me how robust is the observation made in Figure-2. Do you see similar behavior with different noise-scale and other non-linearity such as tanh?

The experimental section provides convincing data showing that non-normal initialization schemes outperform orthogonal and identity initialization in vanilla RNN. However, it would be nice to add some comparisons with prior works. It is unclear how the current method compare with nn-RNN of (Kerg et al. 2019) and the unitary-RNNs. 

 Why the score reported for the 3-LSTM in Table 3.  is underperforming 3-layer LSTM baseline used in (Merity et al., 2018), reported in Table 1.?  In addition, did you try saturating non-linearities for the RNN experiments?

Overall, I think the method is promising, but comparison with prior work is missing. I would encourage the authors to compare their approach with unitary-RNN, and nn-RNN to better demonstrate the significance of their works.

Additional remarks:
-	SNR could be defined more precisely in the introduction. In particular, the introduction states that the stochasticity of SGD is a source of noise which is true. But the model presented in section 2 seems to focus mostly on input noise? 

review_title: Official Blind Review #2

Meta-review: This paper proposes to explore nonnormal matrix initialization in RNNs.  Two reviewers recommended acceptance and one recommended rejection.  The reviewers recommending acceptance highlighted the utility of the approach, its potential to inspire future work, and the clarity and quality of writing and accompanying experiments.  One reviewer recommending weak acceptance expressed appreciation of the quality of the rebuttal and that their concerns were largely addressed.  The reviewer recommending rejection was primarily concerned with the novelty of the method.  Their review suggested the inclusion of an additional citation, which was included in a revised version for the rebuttal but not with a direct comparison of results.  On the balance, the paper has a relatively high degree of support from the reviewers, and presents an interesting and potentially useful initialization in a clear and well-motivated way.
Decision: Accept (Poster)