SGORNN: Combining Scalar Gates and Orthogonal Constraints in Recurrent Networks
Authors: William Keith Taylor-Melanson
Martha Dais Ferreira
Stan Matwin
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Recurrent Neural Networks
Exploding Gradient Problem
Deep Learning Generalization
Orthogonal RNNs

Abstract: Recurrent Neural Network (RNN) models have been applied in different domains, producing high accuracies on time-dependent data. However, RNNs have long suffered from exploding gradients during training, mainly due to their recurrent process. In this context, we propose a variant of the scalar gated FastRNN architecture, called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). SGORNN utilizes orthogonal linear transformations at the recurrent step. In our experiments, SGORNN forms its recurrent weights through a strategy inspired by Volume Preserving RNNs (VPRNN), though our architecture allows the use of any orthogonal constraint mechanism. We present a simple constraint on the scalar gates of SGORNN, which is easily enforced at training time to provide a theoretical generalization ability for SGORNN similar to that of FastRNN. Our constraint is further motivated by success in experimental settings. Next, we provide bounds on the gradients of SGORNN, to show the impossibility of (exponentially) exploding gradients. Our experimental results on the addition problem confirm that our combination of orthogonal and scalar gated RNNs are able to outperform both predecessor models on long sequences using only a single RNN cell. We further evaluate SGORNN on the HAR-2 classification task, where it improves slightly upon the accuracy of both FastRNN and VPRNN using far fewer parameters than FastRNN. Finally, we evaluate SGORNN on the Penn Treebank word-level language modelling task, where it again outperforms its predecessor architectures. Overall, this architecture shows higher representation capacity than VPRNN, suffers from less overfitting than the other two models in our experiments, benefits from a decrease in parameter count, and alleviates exploding gradients when compared with FastRNN on the addition problem.

Under review as a conference paper at ICLR 2022
SGORNN: COMBINING SCALAR GATES AND OR- THOGONAL CONSTRAINTS IN RECURRENT NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Recurrent Neural Network (RNN) models have been applied different domains, producing high accuracies time-dependent data. However, RNNs have long suffered from exploding gradients during training, mainly due to their recurrent process. In this context, propose variant of the scalar gated FastRNN archi- tecture, called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). SGORNN utilizes orthogonal linear transformations at the recurrent step. In experiments, SGORNN forms its recurrent weights through a strategy inspired by Volume Preserving RNNs (VPRNN), though our architecture allows the use of any orthogomal constraint mechanism. We present simple constraint on the scalar gates of SGORNN, which is easily enforced at training time to provide a theoretical generalization ability for SGORNN similar to that of FastRNN. Next, we provide bounds on the gradients of SGORNN, to show the impossibility of (ex- ponentially) exploding gradients. Our experimental results on the addition prob- lem confirm that our combination of orthogonal and scalar gated RNNs are able to outperform both predecessor models on long sequences using only single RNN cell, and outperforms LSTM. We further evaluate SGORNN on the HAR-2 clas- sification task, where it improves upon the accuracy of FastRNN, VPRNN, and LSTM using far fewer parameters than FastRNN or LSTM. Finally, we evalu- ate SGORNN the Penn Treebank word-level language modelling task, where it again outperforms its predecessor architectures and shows comparable perfor- mance LSTM using far less parameters. Overall, this architecture shows higher representation capacity than VPRNN, suffers from less overfitting than the other two models in our experiments, benefits from a decrease in parameter count, and alleviates exploding gradients when compared with FastRNN on the addition problem.
INTRODUCTION
Recurrent Neural Networks (RNN) are models for sequence processing that have found applica- tions in many areas, such as statistical machine translation (Cho et al. 2014), credit score caleu- lation (Babaev et al. 2019). vehicle mobility prediction (Liu & Shoji 2019). and music gener- ation (Agarwala et al. 2017). Despite their wide applicability, RNNs have long suffered from exploding gradients, mainly due the repeated application of the network's hidden weights ma- trix during gradient computation (Hochreiter & Schmidhuber 1997). Several strategies have been proposed solve this problem, including models with orthogonal/umitary constraints on the hid- den weights TTaylo-Melansee et al.) [202 Casado & Martinez-Rubio, [219 Zhang et al.) 2018] Vorontsov et al. [017] Arjovsky al.] 2016) Jing et al.] 2016). Such orthogonal models usually have fewer parameters than standard RNN architectures, which reduces the representation capacity and leads to less accurate models. Specifically, Taylor- Melanson et al. (2021) noted that stack of two RNN cells is required to accurately model the addition problem for RNNs using the orthogonal VPRNN architecture when sequences become very long, and Rusch & Mishra (2020) noted that the EXPRNN architecture (Casado & Martinez-Rubic 2019) fails to converge on the addition prob- lem for very long sequences. To combat the problem of reduced representation capacity, Ding et al. 2019) proposed L.TTM-based architecture called the gated orthogonal recurrent unit (GORU)

---

Under review as a conference paper at ICLR 2022
which was experimented with using the EURNN orthogonal parametrization (Jing al. 2016 While the GORU architecture is a very useful and novel orthogonal RNN construction, it adds many parameters when compared to typical orthogonal RNNs, and thus may not be suited for embedded or mobile applications.
Recently, Kusupati et al. (2019) proposed the FastRNN architecture, which adds residual connec- tion to a standard recurrent network using a scalar gating mechanism. FastRNN outperforms several well-known unitary RNN models on many classification tasks, These scalar gated RNN models have a generalization error gap which scales with the product of the length of the sequences being processed and the non-resicee non-residual scalar gate, which is established through constraints introduced on the scalar gates of the network that are generally not enforced while training. Because of this, the assumptions required to prove the generalization of FastRNN cannot guaranteed to be satisfied in practice under the original formulation of the architecture.
Although the VPRNN architecture and many other RNNs alleviate exploding gradients by utiliz- ing orthogonal constraints, the problem of increasing representation capacity for orthogonal RNNs without greatly increasing parameter counts still remains. Moreover, even with the benefit from the weighted residual connection, FastRNN can still suffer from exploding gradients when applied to long sequences as shown in Section 3.T and Rusch & Mishra (2020) noted that FastRNN fails to converge when training long sequences for the RNN addition problem. In order to solve these problems, we propose the Scalar Gated Orthogonal Recurrent Neural Network (SGORNN), which is variant of FastRNN that uses orthogonal linear transformations at the recurrent step.
The constraints on the scalar gates of FastRNN required for generalization are dependent on bounds of the Frobenius norm of the hidden weights matrix, and of the sequence length of the data being processed. Both of these quantities may not be well-defined in practice, making it difficult to ensure generalization. In contrast with FastRNN constraints, we present constraint that relies only on the scalar gates of the SGORNN model, which is feasible to be employed in practice and results in generalization gap with similar growth that of FastRNN. Our theoretical results also confirm that SGORNN is immune to the (exponentially) exploding gradients so often seen when training RNN models.
In summary, SGORNN can ensure its theoretical generalization by applying the constraints we provide, which also alleviates exploding gradients. Our addition problem experiment shows that SGORNN converges for long sequences with a single RNN cell, outperforming both predecessor architectures despite having far fewer parameters than FastRNN. Using real-world data, SGORNN shows superior generalization to its predecessor models and comparable performance to LSTM models with far more parameters.
The remaining sections of this paper organized follows. In Section 2 describe the pro- posed SGORNN architecture, followed by the theoretical formulation on generalization and explod- ing gradients. Experiments are presented and discussed in Section In Section 4 present the conclusions obtained from the experiments and formulation, as well as proposed future work. Fi- nally, Appendix [ contains detailed information on the mathematical formulation used provide the generalization bound for SGORNN, as well as the alleviation of exploding gradients.
2 SGORNN
The proposed Scalar Gated Orthogonal Recurrent Neural Network (SGORNN) uses orthogonal lin- ear transformations at the recurrent step. Given an input sequence {xt)T C R of length T > 0, define the hidden states h, E R of a SGORNN at time t as:
h ad (Wx + d(O),,1 + +b) 3h,-1-
where θ holds the recurrent parameters of the architecture, Φ a mapping from recurrent parameters into the orthogonal matrices, W E kxxn and U <(+) E Ax are matrices, b E I&th is bias vector, and σ is a nonlinear activation function, such as rectified linear unit (ReLU). We use the notation U- ((+) for consistency with Kusupati al. 2019) during analysis. As with FastRNN, the parameters a, B E (0, are set by the sigmoid function. For simplicity, we assume hn is a vector of zeros, though, in theory, it can be made be a trainable value well.
2

---

Under review as a conference paper at ICLR 2022
For the sake of experimentation, we choose o(+) be defined as the hidden weights matrix of the VPRNN architecture (Tallo-Meelanson et al. 2021). a recently developed orthogonal RNN which is known outperform several other unitary/orhooonal RNNs classification tasks. It is worth mentioning that, although we use the VPRNN mapping in our experiments with SGORNN, an al- ternative orthogonal constraint such that of EXPRNN (Casado & Martinez-Rubio 2019) (with
á = exp) can also be applied to the proposed architecture. Following from Taylor-Melanson et al. 10211 the recurrent matrix is defined as:
(0)) = R Q1 RA-1 1
(2)
which is case of SGORNN with do = RT/2xk, such that R, performs pairwise 2D rotations
of the input vector using parameters from the jth column of o, and Q; is a fixed permutation
matrix. This formulation is inspired by feed-forward predecessor (MacDonald et al., (02T)) should be noted that, in the publicly available implementation of the VPNN layers from MacDonal
al. 2022) written in Keras (Chollet et al., 2015) that we require to implement SGORNN
modification to the permutation structure of U is present in the code This detail is ignored to
simplify notation, but is justified in the supplementary material for the sake of proving Corollary [
The constraint we propose for the scalar gates of SGORNN to ensure generalization takes the fol- lowing form:
B S 1 - 2a.
(3)
At first glance, this constraint may seem to limit the representation capacity of the model as FastRNN takes / a 2a by assumption during theoretical analysis. However, typically < 1, in which case 3 + 2xx R 3 + a, showing that approximation to the standard formulation of the scalar gates remains. This constraint is empirically justified in later sections.
Because our proposed constraints a and B are independent of all other parameters (or bounds), they are very simply enforced at training time by applying Equation In practice, the parameter / is clipped after applying the sigmoid function, so that it lies in the half-open interval (0, 1 2ax..
2.1 GENERALIZATION BOUND
Initially, state the constraint required for the generalization of FastRNN, to contrast with the constraints we require to guarantee the generalization of SGORNN and avoid exploding gradients. It is assumed for FastRNN that ||U|| Ru for some Ry 0, and the required constraint takes
the form:
1
a S min 4T|D U1l 1|' 4TRU ' |112
(4)
In the above equation, D is real number derived from the parameters and activation function of the model. From this, Kusupati al. (2019) established an O(TT) generalization bound for FastRNN which becomes independent of T since a = O(1/T). However, establishing this constraint in practice difficult for several reasons. First, the constraint is dependent on the sequence length T of the model, which may be ill-defined in real-world scenarios, and can lead to vanishing for very long sequences. Second, relies on restricting the function space to those RNNs with ||]]] I Rv, so that Ru is well-defined for use in Equation In addition, in many cases, choosing such an Ru before starting the training may decrease representation capacity drastically or, again, lead to vanishing a. Finally, the constraint becomes undefined for orthogonal matrices ( (|Ull| = 1). These issues motivate the use of our constraint from Equation 3 for SGORNN, which depends only on the of the scalar gates a and 8, and can be enforced training time through simple clipping operation without any prior assumptions about the sequence length or other parameters of the RNN. We define the class Fr of SGORNN networks with ReLU activations follows. Given bounds
'The implementation of SGORNN and experiment code is provided in supplementary ZIP archive See the VPNN Python implementation httph:tt/////ttt /tte::://iitii wtwpwhwww...//ttttteeeeeeo wttp://w///ww.... www.w//ayw..l....aa.a/
3

---

Under review as a conference paper at ICLR 2022
Rw, Rx >> 0, any fixed a, 3 E (0. 1) with 3 S 1- - 2a, mapping from parameters orthogonal matrices, and an input space X' of sequences (x] where |x l2 $ Rx, the SGORNN function class is defined
Fr - ((Tx hr-1)) + Nz (xr + ((0)hr -h | S Rw; θ dom(o))- (5)
It implicit in the definition of the SGORNN function class that bias vectors are omitted or absorbed into the weights matrix, a is the ReLU activation, and that the dimensions of hy-1,Xy: x,,, and W are appropriate. The Rademacher complexity this function class Fr leads to the following theorem, which describes the generalization ability of SGORNN models with ReLU activation functions and with a single output neuron using any 1-Lipschitz activation function. This provides the equivalent of the FastRNN generalization gap result for our new proposed architecture:
Theorem Let ht denote the final hidden state of SGORNN model from the function class Fr for predetermined bounds Rw, Rx. For any vector v E IR with || S Rv, and any 1-Lipschitz activation function y denote f E By o o Fr as a single-neuron output model such that f(X) o (v' hr). Then, denoting L
R + [0, Br] as any bounded 1-Lipschitz loss function for some BL > O, we have with probability at least 1 - å over the draw of an i.i.d. sample S = ((X(9, ) from any input-outpun distribution, the following generalization gap holds:
1 Σ f(x()) + 4C oT ((L().((X)) S
log 2/6
vm + +3BLY
(6)
where C = Rx Ro Rv represents the boundedness of the model parameters and input data, in the analysis of FastRNN (Kusupati al. 2019), and the expectation is taken over the input-oupuut distribution.
The presented theoretical result for SGORNN shows that the generalization of the model is inde- pendent of U and O, which were not used to compute the bounds on and 33 and do not appear in the definition of C. This is because the spectral norm for matrices is consistent with the Euclidean vector norm and ||U|| lg = 1 for SGORNN, Further, the constraints on the scalar gates from Equa- tion do not depend on the sequence length T of the model. A proof of Theorem provided in Appendix A.1
2.2 EXPLODING GRADIENTS
Next, we analyze the gradients aw As W and U are the only parameter matrices interacting with the loss through temporal layers, we study the maximum entry norm ||-Ilmax of the loss function with respect to these parameters to show that SGORNN does not suffer from exploding gradients through time when applying a weaker This analysis leads to our second theorem:
aL and 0L of a loss function L, which is assumed to be I-Lipschitz. 88
constraint and B than the one from Equation 3
Theorem Let E By o Fr be SGORNN model with ReLU activation functions (as in Theorem 1 with trivialization mapping satisfying
(e(@) 80.5
S M4 all indices i,j, and
max
with 3 < a. Then, the maximun parameter gradients of a Lipischittz loss L satisfy:
0L
<CMalaT)).
0e
äL
< Cal Vh Rw
(8)
OW
max
Where and Rww are defined as in Theorem
From such a theorem, we obtain the following corollary concerning our chosen implementation of SGORNN:
4

---

Under review as a conference paper at ICLR 2022
Corollary The VPRNN trivialization satisfies Theorem with Mo 4yhlogo(h), and so the
gradients of the parameters @ satisfy:
0L d
4ch
log, (h)2())))T)
(9)
max
The theoretical bounds on the gradients above follow from the gradient computations provided in FastRNN (Kusupati et al. 2019) combined with knowledge of the VPRNN architecture (Taylor-
Melanson et al. 2021). The proof of Theorem 2 provided in Appendix A.2 while the details of
Corollary are in Appendix A.33 A.3
should be noted that the above bounds differ from the findings of Arjovsky et al. (016)), who provided gradient bounds general unitary RNN models independent of T. This is because the gradient computation of FastRNN (and consequently SGORNN) is greatly influenced by the residual connection through and B. Regardless of this complication, have shown that SGORNN does not suffer from the exponentially exploding gradients often seen when training RNNs.
3 EXPERIMENTS
3.1 ADDITION PROBLEM
As proof of concept. the first experiment considered was the addition problem for RNNs, which was first proposed by Hochreiter & Schmidhuber (1997). This problem takes the form of a regression task, typically on the output vector hy of recurrent network. The input sequence x, 2D sequence x (u, ce) with ut being uniformly distributed in [o, 1] (i.i.d.) and c representing binary sequence. This binary sequence is mostly filled with 0s, having only two elements positions i and j set as 1, in which is chosen uniformly in [1, T/2] and j chosen uniformly in [T. Intuitively, the addition problem gives the sum of two random target positions in the sequence, that the RNN model must recognize and maintain (remember) over time.
The addition problem has been used as a benchmark for many RNN models which aim process long sequences, as the randomly generated data introduces long-term dependencies (Li et al. 2018
Taylor-Melanson et al. 202T Hochreiter & Schmidhuber 1997 AArovssy et al. 2016). A model applied on the addition problem is considered converged if it can consistently and considerably beat the baseline (MSE) error of approximately 0.167 on unseen data (Taylo- Melanson et al., 2021 Li et al. 2018), which is the error when the mean of the target (y 1) is always predicted. In experiments, all data is randomly generated as performed in the recent literature (Tylor-Mes Melanson al. 2021 Li al.) 2018). The regression target y over the sequence is the of the random numbers as indicated by the indices i, defined mathematically as:
v(X)
U+C: = di + Mj.
(10)
We compared the proposed SGORNN with VPRNN, FastRNN, and LSTM using the same model configuration and batch size for all architectures. In this experiment, all models had a hidden state dimension of 128, and the SGORNN and VPRNN used k 14 rotational sublayers, which is the maximum number allowed based on the constraint from VPRNN architecture TaylnonMelanson et al. 202T). Every model consisted of just one RNN layer with ReLU as the activation function. The constraint from Equation 3 was imposed on the SGORNN model so that the model satisfies Theo- rem]] [] The RMSprop optimizer (Tieleman Hinton 2012 was used, which the learning rate was decayed linearly to zero during training. Similarly to experiments conducted by Mhylor-Melansee (ct al. (2021), models were trained with a batch size of 64 for a maximum of 20, 000 training steps, and 10 batches of unseen data were used record validation MSE after every 100 training steps. We note that all models trained by Li et al. (2018) that converged on the addition problem did so within 20, 000 training steps, motivating
choice of cutoff for measuring convergence.
The goal of this experiment was to show which the models tested are able to converge on different
sequence lengths. For this reason, initial learning rates of each model were first set to 10-2, and were
5

---

Under review as a conference paper at ICLR 2022
decreased by a factor of 10 each time model could not converge. For example, if model could not converge with the initial learning rate of 10-² was retrained with an initial learning rate of 10-3 This was allowed to be done twice for each model, that all initial learning rates came from the set {10-2 10-3, 10-1). In this way, we performed a basic search for the maximum learning rate allow the convergence of the models. However, due issues such exploding gradients, some models could not converge for this task. In the case of partial convergence (e.g. VPRNN with T' 1000, see Figure 2 a learning rate decrease was performed ensure better convergence could not be obtained. For such a scenario, the result for the best learning rate was reported. Figures 1 2 and3 the validation error obtained for each model using the sequence lengths 500, 1000, and 5000 respectively.
T= 500
Validaation M55
Train Steps 100)
Figure 1: Addition problem results for the three models tested with 7 = 500.
T= 1000
Training Sleps 1000
Figure 2: Addition problem results for the three models tested with T 1000.
For the shortest sequence length, T = 500, all three models were able to converge using the largest initial learning rate of 10-2, though FasIRNN and SGORNN provided the most efficient conver- gence. In the case of sequences of length T 1000, VPRNN failed to converge with learning rates of 10-2 and 10-4, though it achieved partial convergence with a learning rate of 10-3, Also with T = 1000, SGORNN converged efficiently with the largest learning rate of 10- FastRNN suffered from exploding gradients and divergent loss using all learning rates, even with the small- est learning rate of 10-4 FastRNN also achieved partial convergence for T 1000 as shown in Figure but was unable to reach an efficient solution before exploding gradients occurred. Lastly, for the largest sequence length of T = 5000, we observed that FastRNN suffered from exploding gradients and consequently divergent MSE loss for all initial learning rates. VPRNN also failed to converge for T 5000 regardless the initial learning rate chosen, though it did not suffer from exploding gradients. In contrast this result, SGORNN still converged efficiently using the largest initial learning rate of 10-2 2 and did not suffer from exploding gradients. The LSTM model failed for all sequence lengths greater than T 500 regardless of the chosen learning rate.

---

Under review as a conference paper at ICLR 2022
COAR OORD
Validation
Treining Staee ( 200)
Figure 3: Addition problem results for the two previously convergent models, with T 5000.
Table I: Parameter counts and performance of the three RNN models tested on the addition problem. Partial convergence for VPRNN and FastRNN is indicated by parentheses.
Model Name
Parameter Count
Max Sequence Length
Exploding Gradients
VPRNN
1,409
1000 (partial)
Never
SGORNN
1,411
5,000
Never
FastRNN
16,899
1,000 (partial)
T T000
LSTM
67,201
500
Never
3.2 HAR-2 CLASSIFICATION
The HAR-2 classification task a binary classification task derived from a human movement data set collected using smartphones (Anguita et al. 2013). The data set consists of 9-dimensional sequences of length = 128, having 7, 352 training instances and 2, 947 testing instances. We used this dataset to train and evaluate SGORNN, FastRNN, VPRNN, and LSTM models, conducting performance comparison between them. All models were single-layer RNNs with 128 hidden units, except for the LSTM which was chosen to have 64 hidden units to roughly match the number of parameters of the FastRNN, similar to how hidden state sizes were chosen for a language modelling task in GORU (Jing et al.) 2019). The SGORNN and VPRNN models had A 14 rotational sublayers, which is the maximum allowed following from the constraint presented by (021). We used RMSprop optimizer (Tieleman & Hinton 2012) with an initial learning rate of 10- Models were trained for 300 epochs using a batch size of 100, and the learning rate was decreased by factor of 10 after 200 epochs. This strategy is similar
et al.)
the one employed for FastRNN (Kusupati et al. 2019) with the main difference being our lack of hyperparameter tuning. We trained each model 10 times using different random seeds The constraint from Equation was imposed on the SGORNN model to ensure good generalization based on Theorem [ This experiment shows that our proposed constraint from Equation can applied in practice using a real-world data set.
As shown Table SGORNN achieved the highest mean test accuracy among the three models trained. Although this accuracy is not drastically above that of FastRNN LSTM, across the 10 training runs, the standard deviation of the test accuracies measured was the lowest for SGORNN. This suggests that SGORNN had both greater performance and more stable training than the other models, while requiring far fewer parameters than FastRNN or LSTM.
We use the difference between the training and test performance (for both accuracy and loss) to estimate the degree of overfitting of the models trained on this task. Plots of these estimates of overfitting illustrated in Figure 4 Interestingly, VPRNN has the largest gap between training and testing accuracy, although this gap does not significantly differ from that of FastRNN or LSTM.
To simplify reproducibility, the seed for was made be the integer i (1,,...., 10).

---

Under review as a conference paper at ICLR 2022
Table Parameter counts and accuracy of the three RNN models tested the HAR-2 classification problem, including the standard deviation across 10 training runs.
Model Name
Parameter Count
Test Accuracy (# Standard Deviation)
VPRNN
2,305
92.29 + 0.96%
SGORNN
2,307
94.07 0.2%
FastRNN 17.795
93.77 + 0.49%
LSTM
19,009
93.45 + 1.22%
Further, the accuracy gap for both FastRNN, VPRNN, and LSTM seems to increase training goes while the gap for SGORNN seems to decrease and eventually converge to a stationary value between 3% and 4% (a). Even more interesting the degree of overfitting of FastRNN with respect to the entropy loss, where the decrease in learning rate after 200 epochs caused the gap to grow much faster. VPRNN, FastRNN, and LSTTM have an entropy gap growing with the number of epochs, while the SGORNN model loss gap remains nearly constant and has very small variance (Figure 4b). This behavior suggests that, as training goes on, the degree of overfitting of SGORNN decreases remains constant while increases for all other models.
Overlitiing on HAR-2 (Accuracy over 10 runs)
Overfitting HAR-2 [Entropy over 10 runs!
Accuracy)
Epochs
Epochs
(a) HAR-2 Accuracy
(b) HAR-2 Entropy
Figure 4: The observed estimate of overfitting (difference between train/test performance) for the three models trained on HAR-2. Solid lines indicate the 10-run mean, and the standard deviation over 10 runs is displayed using shaded regions.
3.3 PENN TREEBANK LANGUAGE MODELLING
The Penn Treebank (PTB) data set is annotated corpus for natural language processing tasks (Mar- cus et al. 1993). The word-level version of the data set consists of approximately 929, 000 training tokens, 74, 000 validation tokens, and 83, 000 testing tokens. The most common 10, 000 tokens used, with the rest of the tokens replaced by <unk> tokens. Newline characters are replaced with <eos> tokens. We use this data output of the RNN at time t is probability distribution over the token to be observed at time t + calculated using softmax activation function on a fully-connected output layer.
evaluate our models on language modelling task, where the
We take our sequence length and model architecture from FastRNN (Kusupati et al. 2019), process- ing sequences of length 300 and fixing the dimension of the trainable embeddings of each model to be the same as that of the hidden states of the RNN being trained. Our models were all single-cell RNNs with 256 hidden units. Because the classification layer of these language models accounts for such a large proportion of the overall parameters, the LSTM model was allowed to remain at 256 hidden units (unlike the previous HAR-2 2 task). The SGORNN and VPRNN models had A = 16 rotational sublayers. The constraint from Equation 3 was enforced SGORNN ensure gener- alization according to Theorem [ We used the RMSprop optimizer (Tieleman & Hinton 2012),
with the same number of epochs and learning rate decay scheme with the HAR-2 experiment
8

---

Under review as a conference paper at ICLR 2022
As attempt to prevent overfitting in all models, we applied dropout of 0.3 to the embeddings given as input to the RNN cells. The metric used to evaluate models is the test perplex- ity, which measures the level of uncertainty of the language models (lower is better). It is seen in Table 3 that SGORNN outperforms both of its predecessor models, and shows comparable perfor- mance to LSTM despite having only small fraction of the number of parameters within its RNN cell.
(Section 3.2)
Table 3: Parameter counts and test perplexity of the RNN models trained on the PTB data set for word-level language modelling. Parameter counts only include the size of the RNN cells.
Model Name
Parameter Count
Test Perplexity
VPRNN
67,840
257.88
SGORNN
67,842
132.26
FastRNN
131,330
146.75
LSTM
525,312
126.78
4 CONCLUSIONS
This paper has presented a new RNN architecture which combines orthogonal constraints with scalar gating mechanism. This new RNN, which we call SGORNN, was implemented for the pur- pose of experimentation using an orthogonal transformation scheme from recent literature called VPRNN. However, as only Corollary relies directly on the structure of the VPRNN mapping, our main theoretical results from Theorems I and 2 such as the EXPRNN mapping (Casado & Martinez-Rubio 2019).
can be applied with different mapping strategies,
Our theoretical analysis presented both bounds on the gradients of SGORNN and generalization gap bound for Lipschitz loss functions. The gradient bound for SGORNN shows that gradients do not explode exponentially through time, but instead are (()T))) in the worst case when S 1- This constraint to control gradients is weaker than the one required for generalization of our archi- tecture, in that enforcing the generalization constraint also enforces the gradient bound constraint. In theory, through further controlling bounds on a, we can fully control the upper gradient bound presented. The constraint on a (Equation 3 depends only on the other scalar gate B, rather than the (1//T) constraint required for generalization of FastRNN, which depends on bounds con- structed for other trainable parameters. Because of this dependence, our constraint for generalization was easily enforced through a clipping operation for the three tasks in our experiments.
In our experiments, we first trained models on the addition problem, where SGORNN was able to converge on sequences of length T = 5000 using less than 1, 500 parameters, while FastRNN suffered from exploding gradients and both VPRNN and LSTM failed to converge outright. This is the smallest number of parameters that was found in the literature for RNNs able process such long sequences on the addition problem. Further, it shows that SGORNN may require just one RNN tasks where other architectures with reduced number of parameters such as IndRNN (Li et al. 2018) and VPRNN (Taylor-Melanson et al.) 2021) have been noted to require two stacked cells. On the HAR-2 classification task, the SGORNN model had both the highest accuracy and suffered from the least amount of overfitting of all models considered. On the PTB word-level language modelling task, SGORNN outperformed both VPRNN and FastRNN in terms of test perplexity, and shows comparable performance to our LSTM. It is worth noting that our test perplexity for FastRNN is higher than that reported by Kusupati et al. (2019), which may be attributed to differences in hyperparameters. Based on the experiments, we conclude that the SGORNN architecture brings enforceable generalization and well-controlled gradients to the
cell to perform accurately
FastRNN architecture, and brings increased representation capacity to orthogonal RNN architectures such as VPRNN, reinforcing our thcoretical results.
Future work on SGORNN will include further investigation of generalization ability, well in- vestigation of alternative constraints on the scalar gates. Additional architectures such as SGORNN models with standard gating mechanisms may also be investigated. Finally, convergence analysis of SGORNN and the investigation of mappings to orthogonal matrices other than that of VPRNN will also be the topic of future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
N. Agarwala, Y. Inoue, and Axel Sly. Music composition using recurrent neural networks. 2017.
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and J Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. 01 2013.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks, 2016.
Dmitri Babaev, Maxim Savchenko, Alexander Tuzhilin, and Dmitrii Umerenkov. Et-rnn: Applying deep learning to credit loan applications. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2183-2190, 2019.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3NNov):4334433 482, 2002.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning the- ory. In Summer school machine learning, pp. 169-207. Springer, 2003.
Mario Lezcano Casado and David Martinez-Rubio. Cheap orthogonal constraints in neural net- works: A simple parametrization of the orthogonal and unitary group. CoRR, abs/1901.08428, 2019. URL URRphttp//aaxiis otg/ 1901 08428
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehree Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406. 2014. URL
1078
François Chollet et al. Keras.
2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott A. Skirlo, Max Tegmark, and Marin Sol- jacic. Tunable efficient unitary neural networks (EUNN) and their application to RNN. CoRR, abs/1612.05231, 2016. URL/htt::///rxix org/ abs/1612. 1112/12112.00111111
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated orthogonal recurrent units: On learning to forget. Neural Computation, 31(4): 765-783, 2019. doi: 10.1162/necoa.0111
Aditya Kusupati, Manish Singh, Kush Bhatia, Ashish Kumar, Prateek Jain, and Manik Varma. Fast- gmn: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network, 2019.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn): Building a longer and deeper rnn, 2018
Wei Liu and Yozo Shoji. Deepvm: Rnn-based vehicle mobility prediction to support intelligent vehicle applications. IEEE Transactions on Industrial Informatics, 16(6):399744066 2019.
Gordon MacDonald, Andrew Godbout, Bryn Gillcash, and Stephanie Cairns. Volume-preserving neural networks. In 202/ International Joint Conference on Neural Networks (IJCNN), pp. 1-9, 2021. doi: 10. 119111CNN52387.20002 21t.2.222..5112
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993. URL
htttpss//seeLanthooloy....
Mehryar Mohri and Andres Munoz Medina. Learning theory and algorithms for revenue optimiza- tion in second price auctions with reserve. In International Conference Machine Learning, pp. 262-270. PMLR, 2014.
T. Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies, 2020.
10

---

Under review as a conference paper at ICLR 2022
William Taylor-Melanson. Gordon MacDonald, and Andrew Godbout. Volume-preserving recurrent neural networks (vprnn). In 2021 International Joint Conference on Neural Networks (IJCN)), pp. 1-10, 2021. doi: 10. 10.1109/1CNN55238.22200 22219992222
T. Tieleman and G. Hinton. Lecture 6.5 6.5-RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies, 2017.
Jiong Zhang, Qi Lei. and Inderjit S. Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization, 2018.
A PROOFS OF STATED THEOREMS AND DETAILS OF COROLLARIES
In the following proofs, we follow notation similar to (Kusupati et al. 2019). This is different from that of (Taylor-Melansee
al. 2021) who denote the recurrent weights matrix by V rather than U. We choose to denote the empirical Rademacher complexity of a function class F by Rm(F), indicate that the average taken over random sample of size m. This differs slightly from both (Kusupati et al., 2019) and (Bousquet et al. (2003) who use n instead of m to denote the size of the random sample, since we use describe the input data dimension in Section 2
A.1 PROOF OF THEOREM 1
We first begin with a description of the one-sided Rademacher complexity bound framework used. For any probability distribution D over input-output pairs = (X, y) 2, any bound B, > 0, and any loss function L 2 ý [0, Br] taking in training instance and prediction V, define a class of functions over any class of predictors taking values y as:
Lo (e, (X,y) - f(y))(X)) : FFF
(11)
so that é / Lo then & 2 [0, Br.] is mapping from input-output pairs to loss values. The empirical Rademacher complexity of this class of functions is defined as in (Bousquet et al. 2003) (m))
on sample S = (=(1), (m)) drawn i.i.d. from 1) as:
Σ (ll()())
Ry, (L F) Eg
sup EyE m
(12)
The main result require prove the generalization of SGORNN is the one-sided Rademacher complexity bound from Bartlett & Mendelson (2002), which is also used by FastRNN (Kusupati al. 2019). It states that, for all ò 0, with probability least 1 over the draw of S, for any y L F and distribution D over Z:
1 m Σ Ee((20))+
log 2/8
BenDle
+ + 3B,
(13)
2m
Theorem D Let hy denote the final hidden state of a SGORNN model from the function class Fr for predetermined bounds Rww Rxx For any vector v E h with |[v S Ry, and any 1-Lipschitz activation function By denote f E By o FT as single-neuron oulput model such that f(X) ay(vV T hr). Then, denoting L R some Br, > O, we have with probability at least 1 å over the draw of an i.idd sample S (XX ()), ())) (t) from any input-output distribution, the following generalization gap holds:
[0, Br) as any bounded I-Lipschitz loss function for
+ 4C af
log2/6
((L().((X) < m
Σ
vm + +3BL
(14)
11

---

Under review as a conference paper at ICLR 2022
where C Ry Ryy represents the boundedness of the model parameters and input data, in the analysis of FastRNN (Kusupati et al. 2019), and the expectation is taken over the input-ouppus distribution.
It just remains to establish bound Rm (L F). As before, denote f(X( = o, hy the output of the model on given input sequence ( = (x1 ...., x where h denotes the final hidden state of the SGORNN the given input. Denote the class of SGORNN networks (outputting
hr) as Fr, so that F = dy o v Frr the overall class of hypothesis functions. Because both
and by are assumed to be 1-Lipschitz, we have by the Ledoux- Ledoux-Talagrand contraction (the needed version of which has proof in (Mohri & Medina 2014)) that:
Rm o F) S Rm (F) = Rm ovo Fr) S (
(15)
By definition of the empirical Rademacher complexity, we then have that:
Rm FI) = E,
1
Σ cV hT =
(16)
v.
Next, recall that ||v|l2 S Ry and |ilr S Rw. There is no explicit restriction placed on θ in the definition of Fr. Note that:
Rm o FT) E.
sup v,W,e m
E
(
(17)
m
S Eg
sup
||v122
(18)
v,W,e m
m
S RylE
sup w.e
E;
(19)
Now, follow similar strategy employed for FastRNN (usupati et al. 2019). Denoting Rm (Fr) as the above expected value, we have modified the notation of an inequality for Rm (Fr), provided by Kusupati et al. (2019), capture the supremum recurrent parameters O:
20
m
m
!! (20)
Rm S 3Rm (Jr-)) + m
Ex
sup w
Σ G
W W
+ E
sup w.e
Σ
c Uh
Ledoux-Talagrand Talagrand contraction to remove dependence on the activation function a. The first expected value in this equation bounded by Rw /' using the same argument as Kusupati et al. (2019). Because U is orthogonal, the second expected value in Equation 20 is:
Equation 20 relies on analog of the Ledoux-
E,
sup w.e
Σ
Uh (8)
E
sup
U
w.e
1=1
2=1
Ex
sup w.e
R (Fr-1)
So that, by substituting the bounds on the expected values, Equation 20 becomes:
Rm (F) $ (8 2a) (Fr-1) + R Rx
(21)
/m
12

---

Under review as a conference paper at ICLR 2022
Now, using the fact that B 1 - 2a, the coefficient of Rm (Fr-111) in the above recurrence is at
most 1. Assuming hg =0 0 for all i, we get that Rm (Fo) = and consequently:
Rm (Fr) S 2nT Rw Rxx
(22)
Vm
Setting C = Ry Rw Rx combining Equation 19 and Equation 22 gives bound on the Rademacher
complexity of L F of:
Rm (L o F) S 2C at mm'
(23)
and finally, the desired bound holding with probability at least 8 comes from combining Equa- tions and 23
(z(00) 4C aT Vm + 3B, V
log 2/0 2m
E Eg~Dle((s)
(24)
which is equivalent to the original statement of the theorem, (z)) is simply shorthand for Lyy, f(X)).
A.2 PROOF OF THEOREM 2
The proof relies the following formulas, taken from the formulation presented by Kusupati et al. (2019):
0L. D:
T
T-1
a
Σ
II
(aUTDA DK+1 + (T) (Vny L)h
(25)
0L
T
DD, D2
II (aU D + 31) (Vn,
(26)
OW
t=1
where Dj = diag(o' (Wx,, x + Uhg-1)) is the Jacobian matrix of the pointwise activation function σ. Note that for ReLU, D3 is a diagonal matrix with binary vector on the diagonal.
Theorem 2 Let E oy o Fr be SGORNN model with ReLU activation functions (as in
Theorem I with trivialization mapping satisfying
I for all indices i,j, and
80.
max
with 3 < a. Then, the maximum parameter gradients of a 1-Lipschitz loss L. satisfy:
aL
<CM.((TT)².
(27)
00
Iaa
0L
S CaT Vh. Rw
(28)
oW
Where and Ro are defined as in Theorem
We begin by establishing the first bound involving θ Note that by the chain rule:
aL
Σ aL OU
(29)
00, kj
aU d0,
T.
Note also that we simply denoted U = ((O), s0:
0T.
aL @
Σ
(30)
00,
aU 00,,
T.M
13

---

Under review as a conference paper at ICLR 2022
Thus, since
(r(@)r 00
I Ma:
01.
01
0L
01. 00 2
S
M
Σ
Ma
Mah
(31)
:0:j
0U TM
T,
1,1
where we use the fact that || $ V | and | S vh X|||
for any matrix
Next, use the subadditivity and submuliipicatiiityy of the spectral norm applied to Eq.
25
aL ØU
S a
Della
II (a U |TT|2 DD-+lll + + (111))))
[(Vn,L)hT
Because ReLU is I-Lipschitz, [Dall2 I for any k. Further, | = |T]| = 1. Thus, the product in the formula has an upper bound of (a++3)T-- < 1. also have that ([ 1.),,11 ll S || Llla 11122 Thus:
aL aU
Sa
Sal
112
2
It is then left to bound
VV 2 and ||b,- 1112. First, for the hidden state:
h h-1111
|00 (xx Uh;22 5h4-2
$ a [axl- + Uh,-2)ll + 8 |
a Wx. + Uh,
h
a Rw Rx + || + 2
aR Rx (a + 3) [he2
aR Rx |1h-2
Solving the above recurrence gives |l,,1112 S aTRwRx. Next for the gradient, assuming a 1- Lipschitz loss, we get that:
0L 0(hr)
0L 09 Oy a(rrj
0L Oi oy hT) 2 (v hT) a(hT);
h
< |/
(32)
So that L12 $ Ry. Plugging these back in gives:
0L /U
< Rx = C(aT)).
2
and consequently the gradient bound:
aL
I CMoh(aT)²
(33)
00,g
as desired (the statement regarding the max norm follows since we chose arbitrary i,j). Moving on to the bound on the gradient for W. get using the same strategy that:
14

---

Under review as a conference paper at ICLR 2022
al aw
ØL
aL
S vh
aw
OW
hhox hr 7/122
|xel2
Vh
S Ra CaTl
Rw
as desired.
A.3 DETAILS OF COROLLARY
I
Taylor-Melasson al. (021
showed that using the VPRNN trivialization, for any and AQ:
(Ø(() 6( + 1 S k ||
(34)
where k = 2 2 h1. Note that k S 4 loga h for h 2 2 which can be safely assumed. Now, consider a perturbation ÁO with zero entries everywhere but the i, jtth entry, so that:
O a0ij
(o(O), 6(0 + AO)
lim Λ0,
(35)
AG;j
The numerator has the upper bound:
())),, o0 40) s (6( - 6(0 A0)1 S Vāllo(()) - <( + 40)12
while the denominator has the lower bound (since A has only nonzero entry):
ا = ||00| > ||0||2-
Therefore, assuming the gradient of é is defined, and the limit exists, it must satisfy:
vh (lo(O) o(O A0)Ik2 S SkVT 4vh log, h.
sup Ae
(36)
|| |
by the derivations presented for VPRNN (Tylor-Melanse Melanson et al. 2021 This completes the deriva- tion of the Md bound for the VPRNN trivialization mapping.
A.4 JUSTIFICATION OF CODE IMPLEMENTATION
The public code implementation of the VPRNN layer we required for our experiments adds some additional permutations the matrix decomposition of U o(O). Regardless of how many adds, we can define the mapping ş of the implementation as Ø Q Q for permutation matrices
Q Q2, where & is a typical VPRNN mapping using suitable permutation matrices. This definition due to the fact that the product of two permutation matrices is also a permutation matrix. Using the spectral norm 11- get the following inequality since Q1, Q2 orthogonal:
|000+ AQ) - -500
Q A0) ²
< ( 000++80 (00+ - -0))
< 0000 + AQ) - (
k|140112-
15

---

Under review as a conference paper at ICLR 2022
So, the result required for Corollary holld for the implementation provided and the extra permuta- tions can be ignored.
16


Reviews:
[{"review_id": "zME7iL4EzsJ", "reviewer": "Reviewer_815o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and easy to follow. However, the proposed method seems to be a simple extension to the FastRNN model, and has limited novelty. The only addition to the FastRNN model is the use of orthogonal weight matrices, which is also a well-established line of research in the field. The experimental results could also be improved by adding more baseline methods, including RNNs with orthogonal weight matrices, as listed in the introduction section. ", "summary_of_the_paper": "In this paper, the authors propose an RNN architecture named SGORNN. The high-level idea is to add a residual connection between hidden states, and to parametrize the weight matrix by the product of a series of rotation and permutation matrices so that the weight matrix is orthogonal. The proposed model can be seen as an extension of the FastRNN model, the theoretical analyses of which also apply as a result. The proposed model is evaluated on a few tasks including the addition problem, HAR-2 classification, and PTB.", "summary_of_the_review": "The proposed method seems to be a simple extension of an existing method (FastRNN) and thus has limited novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "uPXZ7Epgg0H", "reviewer": "Reviewer_W7nS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed model has very limited novelty. It simply put the orthogonal constraint on a standard FastRNN. Combining orthogonal transition matrices with other mechanisms in RNNs has been studied in the literature (arXiv:1706.02761), but the authors failed to mention any of them.\n\n2. The orthogonal constraint method used in the paper is from VPRNN (Taylor-Melanson et al., 2021), which seems problematic. It is not proven to cover the entire orthogonal space and appears to have heavily redundant operations. Since the paper's main contribution is to combine orthogonal constraint with FastRNN (Kusupati et al., 2019), it is necessary to show that the combination works for ANY orthogonal parametrization methods (arXiv:1511.06464, arXiv:1612.00188, arXiv:1612.05231, arXiv:1901.08428).\n\n3. The theory part is very impressive. It starts with FastRNN's original theorems and derives a convincing conclusion based on the orthogonal condition. \n\n4. Experiments only compared the proposed model against FastRNN and VPRNN. It's more like a simple ablation section to show both components in the model are functioning. However, this is far from showing the model is valid. It failed to compare to any modern orthogonal RNN architectures or modern gated RNNs. For example, the Penn Treebank perplexity results are all far from basic LSTM/GRU performance. Therefore, the experiments cannot justify the method. \n", "summary_of_the_paper": "This paper proposed a novel recurrent network architecture called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). The model is a combination of (1) orthogonal transition matrix and (2) fixed residual connection as in FastRNN (Kusupati et al., 2019).\nThe authors theoretically studied the generation bound and gradient exploding condition of the model. \nLastly, experiment results show that the proposed model outperforms standard FastRNN and a version of vanilla orthogonal RNN on three tasks: (1) synthetic adding problem, (2) HAR-2 classification, (3) Penn Treebank word-level language modeling.  ", "summary_of_the_review": "The method proposed in the paper has limited novelty. The theory section in the paper has a meaningful contribution.\nThe experiments are not convincing. \nOverall, the paper is not appropriate to publish on ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "m8PVLyPwxE", "reviewer": "Reviewer_aPoP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of combining residual connection (scalar gates) and unitary RNN (orthogonal recurrent matrices)  is simple, well-documented, well-explained and well-demonstrated. In fact, this is one of these very nice simple ideas that can make the lives of many researchers way easier when playing with RNN (**only**). Indeed, the implementation of this solution (such as given in supplementary materials) is pretty straightforward. \n\nNevertheless, I am concerned by the impact and scalability of this approach. The given experiments are two simple toy-tasks (one would have been sufficient to validate the model conceptually) and a medium one with language modelling with the Penn TreeBank dataset. Unfortunately, and appart from the first toy-task, the experiments fail, imho, at giving insights on how well this model will behave in practice.\n\nThe major improvement of this work comes from the possibility to avoid exploding gradients and marginally improve the generalisation capabilities of the model. Hence, we are expecting experiments that demonstrate these points in realistic conditions. Exploding gradients on the addition task is not the same gradient-explosion-level that one may observe with large scale datasets. How is SGORNN behaving with complex tasks that are known to cause gradient explosion, like speech processing or very-long sequence processing with large input vectors and large architectures ? How is SGORNN performing against existing RNN on well-known sequence modelling task ? Penn TreeBank with a perplexity of 130 is quite high. We do not require a SoTA results, this is out of the scope of such a paper, however, it remains that a new researcher can not have a clear idea of if this idea will work and scale well in his/her own complex scenario. In practice, most of the time, we are not just doing additions. An other point along these lines is that Vanilla RNN aren't the must-go solution right now. Practically speaking, RNN in general (LSTM, GRU others) are even being left in favour of self-attention. Hence, to have real impact on the community, this work must be extended to architectures that are widely used and still suffer from gradient explosion (GRU). \n\n**Strengths**\n- Well written\n- A nice, simple and effective (under certain conditions) idea.\n\n**Weaknesses**\n- Real impact on the community quite limited as only vanilla RNN are concerned by the study.\n- Experiments fail at proving the points addressed by the proposed method (at scale and with realistic conditions).\n\n**Minor remarks:**\n\nContrast for pictures are not well suited for black and white printers ! Impossible to distinguish the curves.\nThe conclusion isn't really a conclusion but more of discussion + conclusion. It would be better to split it in two different parts. \n \n", "summary_of_the_paper": "This work proposes to approach and partially solve (under certain constraints) the exploding gradient problem of RNN. In practice, the authors combined two existing methods (residual connection + unitary RNNs) in a single novel model. Demonstrations are given for the exploding issue and for the motivation of parameter constraints. The new model, called SGORNN, is then tested on two toy tasks and on a largish one (language modelling). SGORNN outperform models that are based either on residual connection or unitary recurrent connections only. ", "summary_of_the_review": "This paper presents a nice, simple and well-motivated idea. However, it fails at demonstrating how impactful it could be to the community with experiments that are not supporting fully the claims at scale or in realistic conditions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FaoLsyDFeX1", "reviewer": "Reviewer_NMkH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thought this was a good submission. The model architecture is relatively novel yet simple to implement, the bounds are easier to satisfy compared to FastRNN or VPRNN, but most importantly the model's performance is far superior to the baselines. The experiments were well done in that a relatively large hyperparameter sweep was done in order to induce gradient explosions. \n\nThe one issue I have is that perhaps slightly less relevant, harder baselines should have been chosen so as to show how much work there is to do to get a SGORNN-like architecture to converge to performance in more typical models. \n\nAll in all, I think this paper provides an interesting theoretical contribution that the community would appreciate.", "summary_of_the_paper": "Authors demonstrate a new RNN-based architecture that avoids exploding gradients via careful selection of primitives and hyperparameters. The authors motivate their selections with gradient magnitude bounds and show proofs for them. Finally, the authors demonstrate their model's superior performance against several RNN-based baselines directly relevant to their architecture.", "summary_of_the_review": "An interesting model with nice properties that does relatively well empirically, but more baselines (eg, orthogonal RNNs) should be included before the paper is accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the exploding gradient problem in RNNs. The proposed network SGORNN can be seen as an extension to the FastRNN model by adding orthogonal weight matrices. 

I recommend rejection for this paper mainly for two reasons. 

First, as mentioned in the review of Reviewer 815o and Reviewer W7nS, adding orthogonal constraints into FastRNN should not be considered as a significant technical contribution. 

Second, more importantly, the experiments of the paper are not that convincing. All reviewers raise concerns about this issue. I also do not see the point of comparing the proposed model with a baseline LSTM model of much larger parameter size. I can’t think of a reason to do so. Also I think the small datasets will not give you a lot of meaningful insights in comparing the models – PTB for example, is a rather small dataset for language modeling and the results presented there are far from well. The numbers look really bad, reflecting the quality of how these experiments are done ( https://arxiv.org/pdf/1707.05589.pdf ).
Decision: Reject