Title: On orthogonality and learning recurrent networks with long term dependencies
Authors: Eugene Vorontsov
Chiheb Trabelsi
Samuel Kadoury
Chris Pal
Venue: ICLR 2017
Keywords: Deep learning

Abstract: It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and can therefore be a desirable property; however, we find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance. This paper explores the issues of optimization convergence, speed and gradient stability using a variety of different methods for encouraging or enforcing orthogonality. In particular we propose a weight matrix factorization and parameterization strategy through which we we can bound matrix norms and therein control the degree of expansivity induced during backpropagation.

TL;DR: While orthogonal matrices improve neural network stability during training, deviating from orthogonality may improve model convergence speed and performance.

Review discussion:
1- Have you tried any language modeling tasks? How about the adding problem?
2- How do you compare the running time of the algorithm to gradient descent, for example when the model has 1000 hidden units?
3- Have you compared your methods such as vanilla RNN or Identity RNN?
Yes. A vanilla RNN performs similarly to the factorized model with margin=none. We found that an IRNN performed very poorly on the copy task. We performed an additional set of experiments on the sequential MNIST tasks using the SVD factorized form of the transition matrix (margin=none) but initialized with identity (like the IRNN) -- this gave 40.0% accuracy on the ordered task and 53.5% accuracy on the permuted task, which is much worse than the other experimental configurations in Tables 1 and 2.
We measured the wall clock time per epoch for non-factorized RNNs with 128, 500, and 1000 hidden units. With regular gradient descent, the times are 28.0 ± 2.6, 73.1 ± 7.9, and 146.8 ± 0.3, respectively. With geodesic gradient descent, they are 42.4 ± 8.6, 100.7 ± 15.5, and 270.7 ± 18.6 respectively. These times include all updates to the network, not only the time to update the transition matrix. Given that there is a matrix inverse in the Cayley transform we were pleasantly surprised by these wall clock times.
We are working on some Penn TreeBank and adding problem experiments.
1 - Have you tried simply using a Cayley parametrization directly for U and V (i.e., represent only the positive entries of the necessary skew-symmetric matrices and construct an orthogonal matrix on the fly, backpropagating w.r.t. the parameters of the skew-symmetric matrices)? I've tried this in the past and it seems to work. Would be curious to know your thoughts on the tradeoffs vs. the geodesic gradient descent updates (especially around training speed).
2 - Have you experimented with/observed any benefits in convergence speed (presumably not in wall time) when applying the approach to other architectures, e.g., deep feedforward networks?
We have tried a Cayley parameterization on W, but for the copy task we examined here it did not work well, it behaved much worse compared to geodesic gradient descent which converged relatively quickly to a good solution.
To be specific, we ran experiments with the Cayley parameterization on W, testing on the copy task (length 100) without using a nonlinearity. This setup required a low learning learning rate for stable training; at a learning rate of 0.00001 (RMSprop), accuracy plateaued at ~25% after 5000 epochs. With geodesic gradient descent, the model converged (solving the task) after less than 80 epochs. We also tried Cayley parameterizations for U and V with an identity S matrix and this had similar convergence problems for the copy task.
Barring numerical issues in the theano update calculation, we considered that the worse performance of the direct Cayley parameterization may possibly be explained by regions of high curvature along the Stiefel manifold along which the step size could effectively become negligible. Geodesic gradient descent maintains step length, mapping the step along a geodesic.
In terms of computational cost, both are similar as for both there is a matrix inverse in the update step, an O(n**3) operation.
Not yet, but we are very eager to examine such architectures.
In fact there is some recent work on the arxiv that introduces “Stiefel layers” in feedforward networks (arxiv 1611.05927); they constrain the fully connected layers of a LeNet to have orthogonal weight matrices, for example. This appears to improve performance on the Cifar, STL, birds, aircrafts, and cars datasets for those authors.
Vanishing and exploding gradients makes the optimization of RNNs very challenging. The issue becomes worse on tasks with long term dependencies that requires longer RNNs. One of the suggested approaches to improve the optimization is to optimize in a way that the transfer matrix is almost orthogonal. This paper investigate the role of orthogonality on the optimization and learning which is very important. The writing is sound and clear and arguments are easy to follow. The suggested optimization method is very interesting. The main shortcoming of this paper is the experiments which I find very important and I hope authors can update the experiment section significantly. Below I mention some comments on the experiment section:
1- I think the experiments are not enough. At the very least, report the result on the adding problem and language modeling task on Penn Treebank.
2- I understand that the copying task becomes difficult with non-lineary.
However, removing non-linearity makes the optimization very different and therefore, it is very hard to conclude anything from the results on the copying task.
3- I was not able to find the number of hidden units used for RNNs in different tasks.
4- Please report the running time of your method in the paper for different numbers of hidden units, compare it with the SGD and mention the NN package you have used.
5- The results on Table 1 and Table 2 might also suggest that the orthogonality is not really helpful since even without a margin, the numbers are very close compare to the case when you find the optimal margin. Am I right?
6- What do we learn from Figure 2? It is left without any discussion.
Thank you for reviewing our work. I will address your comments in sequence:
We have added extensive experiments to the draft. Among these, we added the adding problem and Penn Treebank character prediction. These experiments are summarized in a general post above.
We find that the copying task becomes extremely difficult when using a tanh or ReLU activation function on the transition matrix, requiring hundreds of thousands of iterations and a soft orthogonality constraint on a short sequence. We have performed a number of experiments with nonlinearities on the copy task and reported the results and discussion in the appendix. We note that in prior work, the copy task was solved either without enforcing tanh or ReLU activation functions. Henaff et al. described a solution mechanism for an RNN without a transition nonlinearity and trained RNNs without a nonlinearity to solve the task. Arjovsky & Shah et al., as well as Wisdom et al., used a nonlinearity ("modReLU") that is initialized as an identity function and is effectively learned.
Thus, we experimented with a parameteric ReLU for which we chose different slope values or let the slope be learned. We found that the nonlinearity learns to nearly become an identity function and that forcing it to be less so makes the task unsolveable.
Finally, we tested a norm-preserving non-linearity (OPLU) which allows us to retrieve the same results as without a non-linearity, on the copy task. We also test this activation function on the MNIST tasks (p. 8).
Thank you for spotting that – we've added the numbers now. We used 128 hidden units in all of the originally published experiments. We used 512 hidden units in the PTB experiments, and 128 in the other newly added experiments.
We have added a section into the appendix that evaluates the running time of our method compared to regular SGD for three different model sizes. All times were averaged over 3 runs.
On MNIST the performance of the RNN without a margin was indeed very close to the top performance with a margin. Clearly, the results in Table 1 and Table 2 show that for this task it is detrimental to strongly limit deviation from orthogonality.
On the other hand, orthogonal initialization allows an RNN to perform on this task far better than Glorot normal initialization or even identity initialization (it is also orthogonal but not random and with no variance). This supports the notion put forth by Saxe that orthogonal initialization helps optimization by promoting signal flow in the beginning of training.
Furthermore, this supports our proposed intuition that restricting weights to be orthogonal could be detrimental for model performance. We have now added an extended discussion on this, considering both MNIST and PTB results, to section 3.1.2.
We describe the figure jointly with the results from tables 1 and 2. It shows that larger margins allow for faster convergence on the MNIST tasks and that RNNs with large margins perform almost identically to an RNN without a margin as long as the transition matrix is initialized as orthogonal. Because the first point has been demonstrated with the copy task experiments and the second point can be shown with the results in the tables, we moved the figure into the appendix, thus removing some redundancy.
The paper is well-motivated, and is part of a line of recent work investigating the use of orthogonal weight matrices within recurrent neural networks.
While using orthogonal weights addresses the issue of vanishing/exploding gradients, it is unclear whether anything is lost, either in representational power or in trainability, by enforcing orthogonality.
As such, an empirical investigation that examines how these properties are affected by deviation from orthogonality is a useful contribution.
The paper is clearly written, and the primary formulation for investigating soft orthogonality constraints (representing the weight matrices in their SVD factorized form, which gives explicit control over the singular values) is clean and natural, albeit not necessarily ideal from a practical computational standpoint (as it requires maintaining multiple orthogonal weight matrices each requiring an expensive update step). I am unaware of this approach being investigated previously.
The experimental side, however, is somewhat lacking. The paper evaluates two tasks: a copy task, using an RNN architecture without transition non-linearities, and sequential/permuted sequential MNIST. These are reasonable choices for an initial evaluation, but are both toy problems and don't shed much light on the practical aspects of the proposed approaches. An evaluation in a more realistic setting would be valuable (e.g., a language modeling task).
Furthermore, while investigating pure RNN's makes sense for evaluating effects of orthogonality, it feels somewhat academic: LSTMs also provide a mechanism to capture longer-term dependencies, and in the tasks where the proposed approach was compared directly to an LSTM, it was significantly outperformed. It would be very interesting to see the effects of the proposed soft orthogonality constraint in additional architectures (e.g., deep feed-forward architectures, or whether there's any benefit when embedded within an LSTM, although this seems doubtful).
Overall, the paper addresses a clear-cut question with a well-motivated approach, and has interesting findings on some toy datasets.
As such I think it could provide a valuable contribution.
However, the significance of the work is restricted by the limited experimental settings (both datasets and network architectures).
Thank you for your evaluation.
We have added additional experiments that will hopefully better clarify and motivate our discussion. We summarize these in a general post above. For the copy task, we investigated the use of nonlinearities, as discussed in the Appendix. We supplemented the synthetic copy task with the adding task. We performed additional experiments on the MNIST tasks that support the notion that the transition matrix learns to compensate for the contraction of the tanh nonlinearities.
Finally, we additional real world data in the form of PTB character prediction.
It would indeed be very interesting for us to investigate the effect of orthogonality constraints in LSTMs. We are looking into applying related concepts into future work on such models.
However, we focus this work on demonstrating potential costs to orthogonality constraints (reduced representational power, slower convergence) at a basic level. By first performing this analysis in basic RNNs, we avoid the complications of LSTMs that may make the results less interpretable and position our work as a followup to recent publications that promote orthogonality constraints in simple RNNs (Arjovsky & Shah 2015- arxiv 1511.06464 ; Wisdom 2016 - arxiv 1611.00035 ; Henaff 2016 - arxiv 1602.06662). Continued study with LSTMs will be a logical and interesting progression and will require extensive focus that would place it beyond the scope of this conference paper. We look forward to exploring that direction.
This paper investigates the impact of orthogonal weight matrices on learning dynamics in RNNs. The paper proposes a variety of interesting optimization formulations that enforce orthogonality in the recurrent weight matrix to varying degrees. The experimental results demonstrate several conclusions: enforcing exact orthogonality does not help learning, while enforcing soft orthogonality or initializing to orthogonal weights can substantially improve learning.
While some of the optimization methods proposed currently require matrix inversion and are therefore slow in wall clock time, orthogonal initialization and some of the soft orthogonality constraints are relatively inexpensive and may find their way into practical use.
The experiments are generally done to a high standard and yield a variety of useful insights, and the writing is clear.
The experimental results are based on using a fixed learning rate for the different regularization strengths. Learning speed might be highly dependent on this, and different strengths may admit different maximal stable learning rates. It would be instructive to optimize the learning rate for each margin separately (maybe on one of the shorter sequence lengths) to see how soft orthogonality impacts the stability of the learning process. Fig. 5, for instance, shows that a sigmoid improves stability—but perhaps slightly reducing the learning rate for the non-sigmoid Gaussian prior RNN would make the learning well-behaved again for weightings less than 1.
Fig. 4 shows singular values converging around 1.05 rather than 1. Does initializing to orthogonal matrices multiplied by 1.05 confer any noticeable advantage over standard orthogonal matrices? Especially on the T=10K copy task?
“Curiously, larger margins and even models without sigmoidal constraints on the spectrum (no margin) performed well as long as they were initialized to be orthogonal suggesting that evolution away from orthogonality is not a serious problem on this task. ” This is consistent with the analysis given in Saxe et al. 2013, where for deep linear nets, if a singular value is initialized to 1 but dies away during training, this is because it must be zero to implement the desired input-output map. More broadly, an open question has been whether orthogonality is useful as an initialization, as proposed by Saxe et al. , where its role is mainly as a preconditioner which makes optimization proceed quickly but doesn’t fundamentally change the optimization problem;
or whether it is useful as a regularizer, as proposed by Arjovsky et al. 2015 and Henaff et al. 2015, that is, as an additional constraint in the optimization problem (minimize loss subject to weights being orthogonal). These experiments seem to show that mere initialization to orthogonal weights is enough to reap an optimization speed advantage, and that too much regularization begins to hurt performance—i. e. , substantially changing the optimization problem is undesirable. This point is also apparent in Fig. 2: In terms of the training loss on MNIST (Fig. 2), no margin does almost indistinguishably from a margin of 1 or . 1.
However in terms of accuracy, a margin of .1 is best. This shows that large or nonexistent margins (i.e., orthogonal initializations) enable fast optimization of the training loss, but among models that attain similar training loss, the more nearly orthogonal weights perform better. This starts to separate out the optimization speed advantage conferred by orthogonality from the regularization advantage it confers. It may be useful to more explicitly discuss the initialization vs regularization dimension in the text.
Overall, this paper contributes a variety of techniques and intuitions which are likely to be useful in training RNNs.
Thank you for your review. We appreciate your observation that the initialization vs regularization/constraint trade-off could be highlighted more clearly. We address this on page 7 of the updated draft and in the conclusion. We also added new experiments (as summarized in the general post above), including some on new tasks, that should help clarify the analysis.
Following your suggestion, we experimented with a gain of 1.05 applied to pre-activations for a model that is constrained to be purely orthogonal. With this configuration, we could nearly match the best scores for both MNIST tasks, even though we were using a model with margin m=0. We did not, however, apply this gain on the copy task. Since we did not observe a consistent shift of the spectral distribution to a mean of 1.05 for that task, there was little motivation to do so.
Furthermore, the lack of a nonlinearity on that task meant that the notion that this gain would counterbalance the contraction of nonlinearities inapplicable. We did, however, further support this notion with additional experiments using a norm-preserving non-linearity (OPLU). We find that while other nonlinearities make the copy task extremely difficult to solve, the OPLU allows us to arrive at exactly the same results for all margins as we had when not using a nonlinearity.
Furthermore, the OPLU allows the RNN to achieve equally high performance for all margins on both MNIST tasks; training with no nonlinearity works but significantly worse.
We have updated the draft, adding new experiments, including some on two new tasks: the adding task and PTB character prediction. The PTB results allowed us to slightly expand our discussion on initialization, constraints, and on singular spectrum evolution (p.6-8). The experiments are briefly summarized in a standalone post.
As requested, we have performed a number of additional experiments and added them to the paper draft. The new experiments can be summarized as follows:
[Adding task]
We explored the use of different spectral margins on a synthetic adding task with a transition nonlinearity. A purely orthogonal model (margin=0) and one with no margin failed to converge beyond the baseline level on this task;
however, models with non-zero margins began to converge beyond the baseline.
[Penn Treebank]
We have explored the use of different spectral margins for character prediction on Penn Treebank using two setups. (1) A subset (about 23%) of the data containing sentences with up to 75 characters. (2) A subset (over 99%) of the data containing sentences with up to 300 characters.
[Parametric ReLU on the copy task]
We explored the use of a trainable non-linearity on the copy task and found that it learns to nearly become an identity function. We also experimented with manually setting a leaky ReLU. (This is in the Appendix)
[OPLU]
We have tested a norm-preserving activation function on the copy task and on the MNIST tasks. We found that it allows an RNN’s performance on the copy task to match that of an RNN without a nonlinearity. On MNIST, it allows all spectral margins to perform equally well. We discuss this further in the draft.
[MNIST with an orthogonal RNN + gain]
As suggested following the observation that the singular spectra tend toward distributions with mean 1.05 on the MNIST tasks, we trained an RNN with a transition matrix constrained to be purely orthogonal and applied a 1.05 gain on the hidden pre-activations. We found that this allowed purely orthogonal models to nearly match the top scores on these tasks instead of underperforming as they do without that gain.

Meta-review:
role: PC
data: comment: The work explores a very interesting way to deal with the problem of propagating information through RNNs. I think the approach looks promising but the reviewers point out that the experimental evaluation is a bit lacking. In particular, it focuses on simple problems and does not demonstrate a clear advantage over LSTMs. I would encourage the authors to continue pursuing this direction, but without a theoretical advance I believe that additional empirical evidence would still be needed here.

Decision: Reject