Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose an RNN architecture named SGORNN. The high-level idea is to add a residual connection between hidden states, and to parametrize the weight matrix by the product of a series of rotation and permutation matrices so that the weight matrix is orthogonal. The proposed model can be seen as an extension of the FastRNN model, the theoretical analyses of which also apply as a result. The proposed model is evaluated on a few tasks including the addition problem, HAR-2 classification, and PTB.
main_review: The paper is clearly written and easy to follow. However, the proposed method seems to be a simple extension to the FastRNN model, and has limited novelty. The only addition to the FastRNN model is the use of orthogonal weight matrices, which is also a well-established line of research in the field. The experimental results could also be improved by adding more baseline methods, including RNNs with orthogonal weight matrices, as listed in the introduction section. 
summary_of_the_review: The proposed method seems to be a simple extension of an existing method (FastRNN) and thus has limited novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors demonstrate a new RNN-based architecture that avoids exploding gradients via careful selection of primitives and hyperparameters. The authors motivate their selections with gradient magnitude bounds and show proofs for them. Finally, the authors demonstrate their model's superior performance against several RNN-based baselines directly relevant to their architecture.
main_review: I thought this was a good submission. The model architecture is relatively novel yet simple to implement, the bounds are easier to satisfy compared to FastRNN or VPRNN, but most importantly the model's performance is far superior to the baselines. The experiments were well done in that a relatively large hyperparameter sweep was done in order to induce gradient explosions. 

The one issue I have is that perhaps slightly less relevant, harder baselines should have been chosen so as to show how much work there is to do to get a SGORNN-like architecture to converge to performance in more typical models. 

All in all, I think this paper provides an interesting theoretical contribution that the community would appreciate.
summary_of_the_review: An interesting model with nice properties that does relatively well empirically, but more baselines (eg, orthogonal RNNs) should be included before the paper is accepted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a novel recurrent network architecture called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). The model is a combination of (1) orthogonal transition matrix and (2) fixed residual connection as in FastRNN (Kusupati et al., 2019).
The authors theoretically studied the generation bound and gradient exploding condition of the model. 
Lastly, experiment results show that the proposed model outperforms standard FastRNN and a version of vanilla orthogonal RNN on three tasks: (1) synthetic adding problem, (2) HAR-2 classification, (3) Penn Treebank word-level language modeling.  
main_review: 1. The proposed model has very limited novelty. It simply put the orthogonal constraint on a standard FastRNN. Combining orthogonal transition matrices with other mechanisms in RNNs has been studied in the literature (arXiv:1706.02761), but the authors failed to mention any of them.

2. The orthogonal constraint method used in the paper is from VPRNN (Taylor-Melanson et al., 2021), which seems problematic. It is not proven to cover the entire orthogonal space and appears to have heavily redundant operations. Since the paper's main contribution is to combine orthogonal constraint with FastRNN (Kusupati et al., 2019), it is necessary to show that the combination works for ANY orthogonal parametrization methods (arXiv:1511.06464, arXiv:1612.00188, arXiv:1612.05231, arXiv:1901.08428).

3. The theory part is very impressive. It starts with FastRNN's original theorems and derives a convincing conclusion based on the orthogonal condition. 

4. Experiments only compared the proposed model against FastRNN and VPRNN. It's more like a simple ablation section to show both components in the model are functioning. However, this is far from showing the model is valid. It failed to compare to any modern orthogonal RNN architectures or modern gated RNNs. For example, the Penn Treebank perplexity results are all far from basic LSTM/GRU performance. Therefore, the experiments cannot justify the method. 

summary_of_the_review: The method proposed in the paper has limited novelty. The theory section in the paper has a meaningful contribution.
The experiments are not convincing. 
Overall, the paper is not appropriate to publish on ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes to approach and partially solve (under certain constraints) the exploding gradient problem of RNN. In practice, the authors combined two existing methods (residual connection + unitary RNNs) in a single novel model. Demonstrations are given for the exploding issue and for the motivation of parameter constraints. The new model, called SGORNN, is then tested on two toy tasks and on a largish one (language modelling). SGORNN outperform models that are based either on residual connection or unitary recurrent connections only. 
main_review: The idea of combining residual connection (scalar gates) and unitary RNN (orthogonal recurrent matrices)  is simple, well-documented, well-explained and well-demonstrated. In fact, this is one of these very nice simple ideas that can make the lives of many researchers way easier when playing with RNN (**only**). Indeed, the implementation of this solution (such as given in supplementary materials) is pretty straightforward. 

Nevertheless, I am concerned by the impact and scalability of this approach. The given experiments are two simple toy-tasks (one would have been sufficient to validate the model conceptually) and a medium one with language modelling with the Penn TreeBank dataset. Unfortunately, and appart from the first toy-task, the experiments fail, imho, at giving insights on how well this model will behave in practice.

The major improvement of this work comes from the possibility to avoid exploding gradients and marginally improve the generalisation capabilities of the model. Hence, we are expecting experiments that demonstrate these points in realistic conditions. Exploding gradients on the addition task is not the same gradient-explosion-level that one may observe with large scale datasets. How is SGORNN behaving with complex tasks that are known to cause gradient explosion, like speech processing or very-long sequence processing with large input vectors and large architectures ? How is SGORNN performing against existing RNN on well-known sequence modelling task ? Penn TreeBank with a perplexity of 130 is quite high. We do not require a SoTA results, this is out of the scope of such a paper, however, it remains that a new researcher can not have a clear idea of if this idea will work and scale well in his/her own complex scenario. In practice, most of the time, we are not just doing additions. An other point along these lines is that Vanilla RNN aren't the must-go solution right now. Practically speaking, RNN in general (LSTM, GRU others) are even being left in favour of self-attention. Hence, to have real impact on the community, this work must be extended to architectures that are widely used and still suffer from gradient explosion (GRU). 

**Strengths**
- Well written
- A nice, simple and effective (under certain conditions) idea.

**Weaknesses**
- Real impact on the community quite limited as only vanilla RNN are concerned by the study.
- Experiments fail at proving the points addressed by the proposed method (at scale and with realistic conditions).

**Minor remarks:**

Contrast for pictures are not well suited for black and white printers ! Impossible to distinguish the curves.
The conclusion isn't really a conclusion but more of discussion + conclusion. It would be better to split it in two different parts. 
 

summary_of_the_review: This paper presents a nice, simple and well-motivated idea. However, it fails at demonstrating how impactful it could be to the community with experiments that are not supporting fully the claims at scale or in realistic conditions.
review_title: null

Meta-review: This paper considers the exploding gradient problem in RNNs. The proposed network SGORNN can be seen as an extension to the FastRNN model by adding orthogonal weight matrices. 

I recommend rejection for this paper mainly for two reasons. 

First, as mentioned in the review of Reviewer 815o and Reviewer W7nS, adding orthogonal constraints into FastRNN should not be considered as a significant technical contribution. 

Second, more importantly, the experiments of the paper are not that convincing. All reviewers raise concerns about this issue. I also do not see the point of comparing the proposed model with a baseline LSTM model of much larger parameter size. I can’t think of a reason to do so. Also I think the small datasets will not give you a lot of meaningful insights in comparing the models – PTB for example, is a rather small dataset for language modeling and the results presented there are far from well. The numbers look really bad, reflecting the quality of how these experiments are done ( https://arxiv.org/pdf/1707.05589.pdf ).
Decision: Reject