Lipschitz Recurrent Neural Networks
Authors: N. Benjamin Erichson
Omri Azencot
Alejandro Queiruga
Liam Hodgkinson
Michael W. Mahoney
Venue: ICLR 2021 Poster (2021)
Keywords: recurrent neural networks
dynamical systems
differential equations

Abstract: Viewing recurrent neural networks (RNNs) as continuous-time dynamical systems, we propose a recurrent unit that describes the hidden state's evolution with two parts: a well-understood linear component plus a Lipschitz nonlinearity. This particular functional form facilitates stability analysis of the long-term behavior of the recurrent unit using tools from nonlinear systems theory. In turn, this enables architectural design decisions before experimentation. Sufficient conditions for global stability of the recurrent unit are obtained, motivating a novel scheme for constructing hidden-to-hidden matrices. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks, including computer vision, language modeling and speech prediction tasks. Finally, through Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.

Published as a conference paper at ICLR 2021
LIPSCHITZ RECURRENT NEURAL NETWORKS
N. Benjamin Erichson
Omri Azencot
Alejandro Queiruga
ICSI and UC Berkeley er wrw.chson@berreeeee eddu edu
Ben-Gnrion University azencotêcs. cs bgu. ac.il
Google Research a afq@google, com
Liam Hodgkinson
Michael W. Mahoney
ICSI and UC Berkeley
ICSI and UC Berkeley
1 Liam. iiam.hookk www..o..i.nnnnbbeeeee..y.ooommmm
mmahoney@state htt.://stttteeeeeeeeeeeeee wer.eley. edu
ABSTRACT
Viewing recurrent neural networks (RNNs) as continuous- dynamical sys- tems, we propose a recurrent unit that describes the hidden state's evolution with parts: well-understood linear component plus Lipschitz nonlinearity. This particular functional form facilitates stability analysis the long-term behavior of the recurrent unit using tools from nonlinear systems theory. In turn, this en- ables architectural design decisions before experimentation. Sufficient conditions for global stability the recurrent unit are obtained, motivating novel scheme for constructing salden-to-hidder matrices. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on range of bench- mark tasks, including computer vision, language modeling and speech prediction tasks. Finally, through Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more robust with respect input and parameter perturbations compared to other continuous -teeuuuuu--meeeee RNNs.
INTRODUCTION
Many interesting problems exhibit temporal structures that can be modeled with recurrent neural networks (RNNs), including problems in robotics, system identification, natural language process- ing, and machine learning control. contrast to feed-forward neural networks. RNNs consist of one more recurrent units that designed to have dynamical (recurrent) properties, thereby enabling them to acquire some form of internal memory. This equips RNNs with the ability to discover and exploit spatiotemporal patterns, such symmetries and periodic structures (Hinton 1986). How- ever, RNNs are known to have stability issues and are notoriously difficult to train, most notably due the vanishing and exploding gradients problem (Bengio al. 1994 Pascanu al. 2013).
Several recurrent models deal with the vanishing and exploding gradients issue by restricting the tiddenn weight matrix be element of the orthogonal group (Arjovsky al. 2016 Wisdom et al.) 2016 Mhammedi et al.) [2177] Vorontsov et al.) 2017 Percano-Casadee & Martinez, Rubio 2019). While such an approach is advantageous in maintaining rongerange memory, limits the expressivity of the model. To address this issue, recent work suggested to construct hidden-to- hidden weights which have unit norm eigenvalues and can be nonnormal (Kerg et al., 2019). Another approach for resolving the exploding/vanishin ctt al. (2020), who formulate the recurrent units as a differential equation and update the hidden states based on the difference between predicted and previous states.
gradient problem has recently been proposed by KyKK
In this work, we address these challenges by viewing RNNs as dynamical systems whose temporal evolution is governed by an abstract system of differential equations with external input. The data are formulated in continuous time where the external input is defined by the function x = x(t) e R., and the target signal is defined y (() Rd. Based insights from dynamical systems theory, we propose a continuous-timg time Lipschitz recurrent neural network with the functional form
{ h Asa77 h tanh(W 8w
ow Bw.ww
h + Ux b)
(1a)
y Dh,
(1b)

---

Published as a conference paper at ICLR 2021
where the mudden-to-hidde matrices Ag.y E RNNN and Wa,y E RNN are of the form
{
Apana = (1 3A)(MA + MA) + BA(MA - MA) - YAll
(2a)
Wsw.ow = (1 BM + M + 9w (MW - MW) I
(2b)
where AA. Bw E [,,1], YA T > are tunable parameters and MA, Mw E TRNNN are trainable matrices. Here, h h(t) IR is function of time that represents an internal (hidden) state, and ak(l) is its time derivative. The hidden state represents the memory that the system has of its = 8t
past. The function in Eq. ① is parameterized by the hidden-to-hidden weight matrices A E N and W E.R &RNX the input-to-hidee to--ddde- encoder matrix E R-VX,, and an offset b. The function in Eq. (lb) is parameterized by the hidden-to-on to-ouuuut decoder matrix D E R Nonlinearity is introduced via the I-Lipschitz tanh activation function. While RNNs that are governed by differen- tial equations with an additive structure have been studied before (Zhang et al. 2014), the specific formulation that we propose in T and our theoretical analysis are distinct.
Treating RNNs as dynamical systems enables studying the long-term behavior of the hidden state with tools from stability analysis. From this point of view, an unstable unit presents an exploding gradient problem, while a stable unit has well-behaved gradients over time (Miller & Hardi, 2019). However, a stable recurrent unit can suffer from vanishing gradients, leading to catastrophic for- getting (Hochreiter & Schmidhuber 1997b). Thus, opt for stable model whose dynamics do not (or only slowly do) decay over time. Importantly, stability is also a statement about the robust- ness of neural units with respect to input perturbations, i.e., stable models are less sensitive small perturbations compared to unstable models. Recently, Chang et al.] (2019) explored the stability of linearized RNNs and provided local stability guarantee based on the Jacobian. In contrast, the particular structure of our unit (I) allows us to obtain guarantees of global exponential stability using control theoretical arguments. In turn, the sufficient conditions for global stability motivate a novel symmetric- skeww decomposition based scheme for constructing hidden-to-hidee hidden matrices. This scheme alleviates exploding and vanishing gradients, while remaining highly expressive.
In summary, the main contributions of this work are follows:
First, in Section 3 using control theoretical arguments in direct Lyapunov approach, we provide sufficient conditions for global exponential stability of the Lipschitz RNN unit (Theorem []. Global stability is advantageous over local stability results since it guarantees non-exploding gradients regardless the state. In the special case where A is symmetric, we find that these conditions agree with those in classical theoretical analyses (Lemma [.)
Next, in Section 4 drawing from our stability analysis, we propose a novel scheme based on the symmetric-skew decomposition for constructing hidden-to-hidden matrices. This scheme mitigates the vanishing and exploding gradients problem, while obtaining highly expressive coadeen-to-iiddee matrices.
In Section 6
we show that our Lipschitz RNN has the ability to oulperform state-of-the- art recurrent units on computer vision, language modeling and speech prediction tasks. Further, our results show that the higher-order explicit midpoint time integrator improves the predictive accuracy as compared to using the simpler one-step forward Fuler scheme.
Finally, in Section [].. we study our Lipschitz RNN via the lens of the Hessian and show that it is robust with respect to parameter perturbations; we also show that our model is more robust with respect to imput perturbations, compared to other continuous-time RNNs.
2 RELATED WORK
The problem of vanishing and exploding gradients (and stability) have a storied history in the study of RNNs. Below, summarize two particular approaches to the problem (constructing uni- tary/orthogonal RNNs and the dynamical systems viewpoint) that have gained significant attention.
Unitary and orthogonal RNNs. Unitary recurrent units have received attention recently, largely due to /Arjovsky et al. (2016) showing that unitary hidden-to sanden-to-hiddee to-hidden matrices alleviate the vanishing and exploding gradients problem. Several other unitary and orthogonal models have also been pro- posed (Wisdom et al. 2016 Mhammeei et al., 2017: Ding et al. [2017] Vorontsov et al. 2017) Jose et al. 2018). While these approaches stabilize the training process of RNNs considerably, they also
2

---

Published as a conference paper at ICLR 2021
limit their expressivity and their prediction accuracy. Further, unitary RNNs are expensive train, as they typically involve the computation of a matrix inverse at each step of training. Recent work by Cascano-Casado & Martinez-Rubio (2019) overcame some of these limitations. By leveraging concepts from Riemannian geometry and Lie group theory, their recurrent unit exhibits improved expressivity and predictive accuracy on range of benchmark tasks while also being efficient to train. Another competitive recurrent design was recently proposed by Kerg et al. (2019). Their ap- proach is based on the Schur decomposition, and it enables the construction of general nonnormal sidden-to-hidder matrices with unit-norm eigenvalues.
Dynamical systems inspired RNNs. The continuous time view of RNNs has a long history in the neurodynamics community as it provides higher flexibility and increased interpretability (Pineda)
1988 Pearlmutter 1995, ZZhng et al. 2014). In particular, RNNS that are governed by differen-
tial equations with an additive structure have been extensively studied from theoretical point of
view (Funahashi & Nakamura 1993 Kim et al., 1996 Chow & Li 2000; Hu Wang 2002] Li
et al. 2005: Trischler & D Eleuterio 2016 See Zhang et al. (0014) for comprehensive survey of continuous-tiee RNNs and their stability properties.
Recently, several works have adopted the dynamical systems perspective to alleviate the challenges of training RNNs which are related to the vamishing and exploding gradients problem. For non- sequential data, Ciccone al. (2018) proposed negative-definite parameterization for enforcing stability in the RNN during training. Chang et al. (2019) introduced an antisymmetric hidden-to- hidden weight matrix and provided guarantees for local stability. |Kag et al. (2020) have proposed differential equation based formulation for resolving the sxploding/vanishing gradients problem by updating the hidden states based on the difference between predicted and previous states. NNu al. 2019) employed numerical methods for differential equations to study the stability of RNNs.
Another line of recent work has focused on continuous- time models that deal with irregular sam- pled time-series, missing values and multidimensional time series. Rubanova et al. (2019) and De Brouwer et al.] (2019) formulated novel recurrent models based on the theory of differential equations and their discrete integration. [Lecnner & Hasani] (2020) extended these ordinary differ- ential equation (ODE) based models and addresses the issue vanishing and exploding gradients by designing an ODE-model that is based on the idea of long short-term memory (LSTM). This ODE-LSTM outperforms the continuous-time LSTM (Mei & Eisner 2017) as well as the GRU-D
model (Che al. 2018) that is based on a gated recurrent unit (GRU).
The link between dynamical systems and models for forecasting sequential data also provides the opportunity incorporate physical knowledge into the learning process which improves the gener- alization performance, robustness, and ability to learn with limited data (Chen et al. 2019).
STABILITY ANALYSIS OF LIPSCHITZ RECURRENT UNITS
One of the key contributions in this work is that we prove that model (1 is globally exponentially stable under some mild conditions on and W. Namely, for any initial hidden state we can guaran- tee that our Lipschitz unit converges to equilibrium if it exists, and therefore, gradients can never explode. We improve upon recent work on stability in recurrent models, which provide only a local analysis, see e.g., (Chang et al./ 2019). In fact, global exponential stability is among the strongest notions of stability in nonlinear systems theory. implying all other forms of Lyapunov stability about the equilibrium h* (Khalil) 2002, Definitions 4.4 and 4.5).
Definition 1. A point h² is an equilibrium point of h [(,, 1) if s(h", ,)) 0 for all t. Such point is globally exponentially stable if there exists some C > O and X > 0 such that for any choice of initial values h())) E R.N.
[|((t) h"ll S Ce Ce-M1)1(0)- h"ll. for any t 2 0.
(3)
The presence of a Lipschitz nonlinearity in ( plays an important role in our analysis. While we focus on tanh in our experiments, our proof is more general and applicable to models whose nonlinearity a()) is an M-LLipschitz function. Specifically, we consider the general model
h Ah + a ( Ux + +b,
(4)
for which we have the following stability result. In the following, we let miin and o Tmax denote the smallest and largest singular values of the hidden-to-hidden matrices, respectively.
3

---

Published as a conference paper at ICLR 2021
Theorem 1. Let h be equilibrium point of differential equation of the form
4 for some
I E R.. The point h' is globally exponentially stable if the eigenvalues of A =
2 (A + A
strictly negative, W is non-singular and either (a) min (min(A (A >> Max (W): (b) is
W + W A is positive definite. monotone non-decreasing, W + W is negative definite, and A A
The two cases show that global exponential stability is guaranteed if either (a) the matrix A has cigenvalues with real parts sufficiently negative to counteract expanding trajectories in the nonlin- earity: or (b) the nonlinearity is monotone, both A and W yield stable linear systems ù = Au, ò oe, and A, W have sufficiently similar eigenvectors. In practice, case (b) occasionally holds, but is challenging to ensure without assuming specific structure on A, W. Because such assumptions could limit the expressiveness of the model, the next section will develop tunable formulation for A and W with the capacity to ensure that case (a) holds.
In Appendix A.I vantage of this approach is that the driving input x is permitted to evolve in time arbitrarily in the analysis. The proof relies on the classical from control theory- - to our knowledge, these tools have not been applied in the modern RNN literature, and we hope our proof can illustrate their value to the community.
provide proof of Theorem using direct Lyapunov approach. One ad-
lemma and circle criterion
In the special case where A is symmetric and (t) constant, we show that we can also inherit criteria for both local and global stability from a class of well-studied Cohen-Grossie -oorfill models.
Lemma 1. Suppose that A is symmetric and W is nonsingular. There exists diagonal matrix D E NNN, and nonsingular matrices L,, V E N such that an equilibrium of 14 is (globally exponentially) stable if and only if there is a corresponding (globally exponentially) stable equilib- rium for the system
D2 + LV Ux Ux+
(5)
For thorough review of analyses of 5 see (Zhang al. 2014). In this special case, the criteria in Theorem I coincide with those obtained for the corresponding model ( However, in practice, we
will not choose A be symmetric.
SYMMETRIC-SKER HIDDEN-TO- MATRICES
In this section we propose novel scheme for constructing hidden-to-hidder matrices. Specifically, based on the successful application of skew-symmetric hidden-t-hhidee weights in several recent recurrent architectures, and our stability criteria Theorem T we propose an effective symmetric- skew decomposition for hidden matrices. Our decomposition allows for simple control of the matrix spectrum while retaining its wide expressive range, enabling us to satisfy the spectral con- straints derived in the previous section on both A and W. The proposed scheme also accounts for the issue of vanishing gradients by reducing the magnitude of large negative eigenvalues.
Recently, several methods used skew-symmetric matrices, Le., S 0 to parameterize the recurrent weights W E R N see e.g., (Wisdom et al. 2016 Chang et al. 2019). From a sta- bility analysis viewpoint, there two main advantages for using skew-symmetric weights: these matrices generate the orthogonal group whose elements are isometric maps and thus preserve norms [Lezcano-Casaal & PaatinezzRubio 2019); and the spectrum of skew-symmetric matrices is purely imaginary which simplifies stability analysis (Chang et al. 2019) The main shortcoming of this parametrization is its reduced expressivity, these matrices have fewer than half of the parameters of full matrix (Kerg et al. 2019). The latter limiting aspect can be explained from a dynamical sys- tems perspective: skew-symmet symmetric matrices can only describe oscillatory behavior, whereas matrix whose eigenvalues have nonzero real parts can also encode viable growth and decay information.
To address the expressivity issue, we aim for hidden matrices which on the one hand, allow to control the expansion and shrinkage of their associated trajectories, and on the other hand, will be sampled from a superset of the skew-symmetric matrices. Our analysis in Theorem [ guarantees that Lips- chitz recurrent units maintain non non-expanding trajectories under mild conditions on A and W. Un- fortunately, this proposition does not provide any information with respect to the shrinkage of paths. Here, opt for system whose expansion and shrinkage can be easily controlled. Formally, the latter requirement is equivalent to designing hidden weights S with small (A, i = 1, 2.......A N, where R2)) denotes the real part of A system of the form 4 whose matrices A and W exhibit
4

---

Published as a conference paper at ICLR 2021
(a) 0.5, (b) 3 0.75, 7- 0.01
(c) 3 0.85, 0.01
8 1.0,
Figure 1: Vector fields of hidden states that are governed by Eq. 1 trained for simple pendulum dynamics. (a), an unstable model is shown. In (b) and (c), it can be seen that we yield models that are asymptotically stable,i.e., all trajectories are attracted by equilibrium point. In contrast, (d), a skew-symmetric parameterization leads to a stable model without an attracting equilibrium.
small spectra and satisfy the conditions of Theorem will exhibit dynamics with moderate decay and growth behavior and alleviate the problem of exploding and vanishing gradients. To this end, we propose the following symmetric-skew decomposition for constructing hidden matrices:
SB,7
B) (M + M²)
(M (OM-M") -11
where M is weight matrix, and B E [0.5, 1), 7 > are tuning parameters. In the case (3,7) (1,0), we recover a skew-symmetrie matrix, i.e., S1.0 + ST.0 = 0. The construction S3,, 7 is useful as we can easily bound its spectrum via the parameters and 7, as we show in the next proposition. Proposition 1. Let Sp. satisfy (6). and let M } M M The real parts RA,(Sp,)) of the eigenvalues of S3,7, well the eigenvalues of Seyy SB.7 S3,50 lie in the interval
[11 - M)mmin (Maym)-
Ammax (amaxxMeyyu (Meym) 7
A proof is provided in Appendix A.2 We infer that 3 controls the width of the spectrum, while increasing 7 shifts the spectrum to the left along the real axis, thus enforcing eigenvalues with non- positive real parts. Choosing our hidden-to-hidden matrices to be ABaru and Waw.ow of the form (6) for different values of BA, 3w aA, 7w. we can ensure small spectra and satisfy the conditions of Theorem I as desired. Note, that different tuning parameters and affect the stability behavior of the Lipschitz recurrent unit. This is illustrated in Figure [] where different values for and y are used construct both Ag,y and Wa,y and applied to learning simple pendulum dynamics.
One cannot guarantee that model parameters will remain in the stability region during training. However,
show that when is taken be close one, the eigenvalues of A and W (which dictate the stability of the RNN) change slowly during training. Let F& denote the change in function F depending on the parameters of the RNN ( after one step of gradient descent with step size with respect to loss [(y). For matrix A, we let > (A) denote the k-th singular value of A. We have the following lemma.
Lemma 2. 3 1T, max; Ass (A max A (W = - B)2).
Therefore, provided both the initial and optimal parameters lie within the stability region, the model parameters will remain in the stability region for longer periods time with high probability as
1. Further empirical evidence of parameters often remaining in the stability region during training are provided alongside the proof of Lemma 2 in the Appendix (see Figure
5 TRAINING CONTINUOUS-TIMA LIPSCHITZ RECURRENT UNITS
ODEs such as Eq. (T] can be approximately solved by employing numerical integrators. In sci- entific computing, numerical integration is well studied field that provides well understood tech- niques VLeue 2007). Recent literature has also introduced new approaches which are designed with neural network frameworks in mind (Chen et al. 2018).
To learn the weights A, W, U and b, we discretize the continuous model using one step of numerical integrator between sequence entries. In what follows, subscript i. denotes discrete time indices,
5

---

Published as a conference paper at ICLR 2021
At represents the time difference between pair of consecutive data points. Letting f(h,t)
Ah tanh(Wh Ux((s) + b) that h(t)) f(h, t), the exact and approximate solutions for h4+1 given h2 are given by
het1 he ++21
f(h(s), = h4 /
Ah(s) tanh Un(()) + b) ds
(7)
~ he + At scheme [f, hrs At]
(8)
where scheme represents one step of a numerical integration scheme whose application yields an approximate solution for It Se p+ f(h(s), s)ds given he using or more evaluations of
We consider both the explicit (forward) Euler scheme,
h++1 + At Ahe + At tanh(2)),
as well the midpoint method which is two-stage explicit Runge-Kutta scheme (RK2),
h4-1 + At Ah At tanh(Wh Uz, +),
(10)
where h = h, + At/2 Ad, + At/2 tanh(=;) is an intermediate hidden state. The RK2 scheme can potentially improve the performance since the scheme more accurate, however, this scheme also requires twice as many function evaluations compared to the forward Euler scheme. Given a B and y that yields a globally exponentially stable continuous model, At can always chosen so that the model remains in the stability region of forward Euler and RK2 (Le (eeue 2007).
EMPIRICAL EVALUATION
In this section, evaluate the performance of the Lipschitz RNN and compare it to other state-of- the-art methods. The model is applied to ordered and permuted pixel-by-pixel MNIST classification, random well to audio data using the TIMIT dataset. We show the sensitivity with respect initialization in Appendix B Appendix B also contains additional results for: pixel-by-pixel CIFAR- 10 and noise-padded version CIFAR-I0; as well for character level and word level prediction using the Penn Tree Bank (PTB) dataset. All of these tasks require that the recurrent unit learns long-tern term dependencies: that is, the hidden-to-lle to-hidden matrices need to have sufficient memory to remember information from far in the past.
6.1 ORDERED AND PERMUTED PIXEL-BY-PING MNIST
The tixel-by-pixe MNIST task tests long range dependency by sequentially presenting 784 pixels to the recurrent unit, i.e., the RNN processes pixel at time (Le et al. 2015). At the end of the
Table 1: Evaluation accuracy on ordered and permuted pixel-by-pixel MNIST.
Name
ordered
permuted
N
params
LSTM baseline (Arjovsky al.
2016
97.3%
92.7%
128
R-6KK
MomentumLSTH (gguyen al. 2020
99.1%
94.7%
256
270K
Unitary RNN (Arjovsky al. 2016
95.1%
91.4%
512
299K
Full Capacity Unitary RNN Wisdom et al.
2016
96.9%
94.1%
512
2,20K
Soft orth. RNN (Vorontsov et al. 2017
94.1%
91.4%
128
188K
Kronecker RNN "Jose et al, 2018
96.4%
94.5%
512
R 1KK
Antisymmteric RNNN Cann el al.
2019
98.0%
95.8%
128
10K
Incremental RNN (Kag et al. 2020
98.1% 18.1%
95.6%
128
K/K//KK
Exponential RNN [Lezcano- asado & Martinez-Rubio.
2019
98.4%
96.2%
360
6<
Sequential NAIS-Nel (Ciccone al. 2018
94.3%
90.8%
128
18K
Lipschitz RNN using Euler (ours)
99.0%
94.2%
64
9
Lipschitz RNN using RK2 (ours)
99.1 1%
94.2%
64
9
I ipschitz RNN using Fuler (ours)
99.4 %
96.3%
128
<34K
I Lipschitz RNN using RK2 (ours)
99.3%
96.2%
128
34K

---

Published as a conference paper at ICLR 2021
Table Evaluation on TIMIT using layer models. The mean squared error (MSE) computes the distance between the predicted and actual mog-magnitudes of each predicted frame in the sequence.
Name
val. MSE
test MSE
# params
LSTM (Iellrrch et al.
2018
13.66
12.62
158
^2200K
LSTM (Nguyen al. 2020
9.33
9.37
158
=:200K
MomentumLSTM [guuyen et al.
2020
5.86
5.87
158
=200K
SRLSTM (Nguycn et al.
2020
5.81
5.83
158
~200K
Full-capacity Unitary RNN (Wisdom al. 2016
14.41
14.45
256
~200K
Cayley RNN (Helfrich
2018)
7.97
7.36
425
2200K
Exponential RNN (Lezcano- & Martinez- Rubio 2019
5.52
5.48
425
2220K
Lipschitz RNN using Fuler (ours)
2.95
2.82
256
1188K
Lipschitz RNN using RK2 (ours)
2.86
2.76
256
1988K
sequence, the learned hidden state is used to predict the class membership probability of the input image. This task requires that the RNN has sufficient long-term memory in order discriminate between different classes. A more challenging variation to this task is to operate on a fixed random permutation of the input sequence.
Table providess a summary of our results. The Lipschitz RNN, with hidden dimension of N = 128 and trained with the forward Euler and RK2 scheme, achieves 99.4% and 99.3% accuracy on the ordered pixel-by-pixel MNIST task. For the permuted task, the model trained with forward Euler achieves 96.3% accuracy, whereas the model trained with RK2 achieves 96.2% accuracy. Hence, our Lipschitz recurrent unit outperforms state-of-the-an RNNs on both tasks and is competitive even when a hidden dimension of N 64 used, however, can be seen that a larger unit with more capacity is advantageous for the permuted task. Our results show that we significantly outperform the Antisymmetric RNN (Chang et al. 2019) on the ordered tasks, while using fewer weights. That shows that the antisymmetric weight paramterization is limiting the expressivity of the recurrent unit. The exponential RNN is the next most competitive model, yet this model requires a larger hidden-to-hine to-hideen unit perform well on the two considered tasks.
6.2 TIMIT
Next, consider the TIMIT dataset (Garofolo) 1993) to study the capabilities of the Lipschitz RNN for speech prediction using audio data. For our experiments, we used the publicly available implementation of this task by LLzeano-Casade & Martinez-Rubio (209)). This implementation applies the preprocessing steps suggested by |Wisdom et al. (2016): (i) downsample each audio sequence to 8kHz; (ii) process the downsampled sequences with short-time Fourier transform using a Hann window of 256 samples and a window hop of 128 samples; and (iii) normalize the log- magnitude of the Fourier amplitudes. We obtain a of frames that each have 129 complex-valued Fourier amplitudes and the task is to predict the log magnitude of future frames. To compare our results with those of other models, we used the common train validation test split: 3690 utterances from 462 speakers for training, 192 utterances for validation, and 400 utterances for testing.
Table 2 lists the results for the Lipschitz recurrent unit as well as for several benchmark models. can be seen that the Lipschitz RNN outperforms other state-of-the-art models for fixed number of parameters (< 200K). In particular, LSTMs do not perform well on this task, however, the recently proposed momentum based LSTMs (Nguyen et al. 2020) have improvemed performance. Inter- estingly, the RK2 scheme leads to a better performance since this scheme provides more accurate approximations for the intermediate states.
7 ROBUSTNESS WITH RESPECT TO PERTURBATIONS
An important consideration beyond accuracy is robustness with respect to input and parameter perturbations. We consider Hessian- based analysis and noise-response analysis of different continuous-time time recurrent units and train the models on MNIST. Here, we reshape each MNIST thumbnail into sequences of length 98 that each input has dimension R& We consider this
7

---

Published as a conference paper at ICLR 2021
simpler problem that all models obtain roughly the same training loss. Here stochastic gradient decent (SGD) with momentum to train the models.
Eigenanalysis of the Hessian provides tool for studying various aspects of neural net- works (Hochreiter Schmidhuber 1997a Sagun et al., 2017 Ghorbani et al., 2019). Here, we study the Hessian H spectrum with respect the model parameters of the recurrent unit using Py- Hessian (Yao et al., 2019). The Hessian provides us with insights about the curvature of the loss function L. This is because the Hessian defined the derivatives of the gradients, and thus the Hessian eigenvalues describe the change in the gradient of L as we take an infinitesimal step into given direction. The eigenvectors span the (local) surface of the loss function at a given point, and the corresponding cigenvalue determines the curvature in the direction of the cigenvectors. This means that larger eigenvalues indicate a larger curvature, i.e., greater sensitivity, and the sign of the eigenvalues determines whether the curvature will be positive or negative.
To demonstrate the advantage of the additional linear term and our weight parameterization, we compare the Lipschitz RNN to two other continuous-time recurrent units. First, we consider simple neural ODE RNN (Rubanova et al. 2019) that takes the form
= tanh(Wh + Ux + b),
y= Dh,
(11)
where W is simple bidden-to-hiddee matrix. As a second model we consider the antisymmetric RNN (Chang et al. 2019). that takes the same form as (II). but uses skew-symmetric scheme to parameterize the hidden-to-hidden matrix as W = (M - MT) - II, where M is a trainable weight matrix and 7 is tunable parameter.
T the largest eigenvalue Amax H) and the trace of the Hessian tr(H). H.).The largest eigen- value being smaller indicates that our Lipschitz RNN found flatter minimum, as compared the simple neural ODE and Antisymmetric RNN. It is known that such flat minima can be perturbed without significantly changing the loss value (Hochreiter & Schmidhuber 1997a). Table 3 also
ports the condition number Ki x(H) = Xmne Am) (ff of the Hessian. The condition number x(H) provides
Amax(H)) (H)
a measure for the spread of the eigenvalues of the Hessian. is known that first-order methods can large (Bottou & Bousquet, 2008). The condition number and trace of our Lipshitz RNN being smaller also indicates improved robustness properties.
slow down in situations where
Next, study the sensitivity of the response yr time T in terms of the accuracy with spect to sequence of perturbed inputs { .rr} E R We consider three different pertur- bations. The results for the artificially constructed perturbations are presented in Table showing that the Lipschitz RNN is more resilient to adversarial perturbation. Here, we have considered projected gradient decent (PGD) (Goodfellow et al., 2014) method with L and the DeepFool
Table 3: Summary of Hessian based robustness metrics and resilience adversarial attacks.
Model
PGD
DF2
DF
Amax
tr(Hf)
k(I))
Neural ODE RNN
88.5%
69.6%
44.5%
0.30
4.7
37.6
Antisymmetric RNN
84.7%
83.4%
44.3%
0.24
4.8
35.5
Lipschitz RNN (ours)
93.0%
89.2%
54.1%
0.14
3.1
23.2
Neural ODE RNN
Antisymmetric
Test accuracy
Lipschitz RNN
Unitary RNN
0.6
0.6
0.4
0.2
0.0
0.8
of noise
amount of noise (b) Salt and pepper perturbations.
(a) White noise perturbations.
Figure 2: Sensitivity with respect different input perturbations.
8

---

Published as a conference paper at ICLR 2021
1.00
1.00
0.99
0.98
0.98
accuracy
0.97
0.97
0.96
0.95
0.95
0.94
0.94
0.93
0.93
0.92
permuted
0.91
permuted
0.90
0.0
0.2
0.4
0.6
0.8
1.0
12
14 1.6
1.8
2.0
0.50
0.55
0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1,00 ablation parameter, ,
ablation parameler, a
(a) Effect of the linear term.
(b) Effect of Eq. 6
Figure 3: The ablation study examines the effect of the linear term Ah (in (a)) and the importance of the Skew-Sy -iem-Symmetric Decomposition for constructing the hidden-to-hidden matrices (in (b)).
method (Moosavi-eezfooli et al. 2016) with le and Ioe norm ball perturbations. We construct the adversarial examples with full access the models, using 7 iterations. The step size for PGD is set to 0.01.
Further, Figure shows the results for white noise and salt and pepper noise. can be seen that the Lipschitz unit is less sensitive to input perturbations, as compared to the simple neural ODE RNN, and the antisymmetric RNN. addition, we also show the results for an unitary RNN here.
7.1 ABLATION STUDY
The performance of the Lipschitz recurrent unit is due to main innovations: (i) the additional linear term; and (ii) the scheme for constructing the hidden-to-hinn to-hidden matrices and W Eq. (6). Thus, investigate the effect of both innovations, while keeping all other conditions fixed. More concretely, we consider the following ablation recurrent unit
h++1 = h; + a Aht + E tanh with 2t = Wh; Ux; + b,
(12)
where a controls the effect of the linear hidden unit Both A and W depend on the parameters y-
Figure 3a studies the effect of the linear hidden unit, with 3 0.65 for the ordered task and B = 0.8 for the permuted task. In both cases we use y 0.001. It can be seen that the test accuracies of MNIST tasks clearly depend on the linear hidden unit. For a our models reduces to simple neural ODE recurrent units (Eq. (TT). The recurrent unit degenerates for > 1.6, since the external input is superimposed by the hidden state. Figure 3b studies the effect of hidden-to-hidden matrices with respect to 3. (0.65, 0.70} achieves peak performance for the ordered task, and 8 {0.8, 0.85) does for the permuted task. Note that / = 1.0 recovers an -emmetymet hidden hidden-to-hiddee to-hidden matrix.
both the ordered and permuted
can be seen that 3
CONCLUSION
Viewing RNNs as continuous-tiin dynamical systems with input, we have proposed a new Lipschitz recurrent unit that excels on range of benchmark tasks. The special structure of the recurrent unit allows us to obtain guarantees of global exponential stability using control theoretical arguments. In turn, the insights from this analysis motivated the symmetric-skew decomposition scheme for structing hidden-to-hidden matrices, which mitigates the vanishing and exploding gradients prob- lem. Due the nice stability properties of the Lipschitz recurrent unit, we also obtain model that is more robust with respect to input and parameter perturbations as compared to other continuous- time units. This behavior is also reflected by the Hessian analysis the model. We expect that the improved robustness will make Lipschitz RNNs more reliable for sensitive applications. The theo- retical results for our symmetric-skew decomposition of parameterizing hidden-to-hiddee matrices also directly extend to the convolutional setting. Future work will explore this extension and study the potential advantages these more parsimonious hidden-to-hidee matrices in combination with our parameterization in practice. Research code is shared via

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like thank Ed H. Chi for fruitful discussions about physics-informed machine learning and the Antisymmetric RNN. We are grateful to the generous support from Amazon AWS and Google Cloud. NBE and MWM would like to acknowledge IARPA (contract W91 19F20C00)))) NSF, ONR and CLTC for providing partial support of this work. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pp. 1120-11221 2016.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
Rajendra Bhatia. Matrix analysis, volume 169, Springer Science & Business Media, 2013.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, pp. 161-168, 2008.
Bo Chang, Minmin Chen, Eldad Haber, and Ed Chi. AntisymmeriicNN:: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, 2019.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag. and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Tian Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, pp. 6571-6583, 2018
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference on Learning Representations, 2019.
Tommy W. Chow and Xiao-Dong Li. Modeling of continuous time dynamical systems with input by recurrent neural networks. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 47(4):575-578 2000.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. Nais-net: Stable deep networks from non-autonomous differential equations. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31, pp. 3025-3035. Curran Asso- ciates, Inc., 2018. URL httpl//tttss/pprrcedinn.ggece neurips wtt.:c/pppeeeeee. ptcppppe//221//11ee 3
pdf
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-obsevve time series. In Advances in Neural Information Processing Systems, pp. 7379-7390 2019.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801-866, 1993.
John Garofolo. TIMIT acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. In International Conference on Machine Learning, pp. 2232- 2241, 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Wolfgang Hahn. Stability of motion, volume 138. Springer, 1967.
10

---

Published as a conference paper at ICLR 2021
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled Cayley transform. In International Conference on Machine Learning, pp. 1969-1978, 2018.
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent orthogonal networks and long-memory tasks. volume 48 of Proceedings of Machine Learning Research, pp. 2034-2042, New York, New York, USA, 20-22 Jun 2016. PMLR. URL httt:::pppoceeeiiin..ee www.......pp.....e..e..e.e
henaffi6 html
Geoffrey E. Hinton. Learning distributed representations of concepts. In Conference of the Cognitive Science Society, volume I, pp. 12, 1986.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1 1997a.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997b.
Roger A. Horn and Charles R. Johnson. Matrix analysis. Cambridge University Press, 2012.
Sanqing Hu and Jun Wang- Global stability of class of continuous-time recurrent neural net- works. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 49 (9):1334-1347, 2002.
Li Jing, Yichen Shen, Tena Dubcek, John Peurifoy, Scott Skirlo, Yann LeCun, Max Tegmark, and Marin Soljačić. Tunable efficient unitary neural networks (EUNN) and their application to RNNs. In International Conference on Machine Learning, pp. 1733-1741, 2017.
Cijo Jose, Moustapha Cisse, and Francois Flcuret. Kronecker recurrent units. In International
Conference Machine Learning, pp. 2380-2388, 2018.
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. RNNs incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learning Representations, 2020.
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoic. Non-normal recurrent neural network (nnRNN): Learn- ing long time dependencies while improving expressivity with transient dynamics. In Advances in Neural Information Processing Systems, pp. 13591-13601, 2019.
Hassan K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 2002.
Young H. Kim, Frank L. Lewis, and Chaouki T. Abdallah. Nonlinear observer design using dynamic recurrent neural networks. In Proceedings of 35th IEEE Conference Decision and Control, volume I, pp. 949-954. IEEE, 1996.
Aditya Kusupati, Manish Singh, Kush Bhatia, Ashish Kumar, Prateek Jain, and Manik Varma. Fast- gmn: A fast, accurate, stable and tiny kilobyte sized gated recurrent neural network. In Advances in Neural Information Processing Systems, pp. 9017-9028, 2018.
Quoe V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: (50.004450.00 2015.
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irregularly-sapple time series. arXiv preprint 20Ki:20006.44110 2020.
Randall J. Veque. Finite Difference Methods for Ordinary and Partial Differential Equations.
Society for Industrial and Applied Mathematics, 2007. doi: 10.1137/1.97888988777888
Mario Sezcano-Casado and David Murtinez-Rubio Cheap orthogonal constraints in neural net- works: A simple parametrization of the orthogonal and unitary group. In International Confer- ence on Machine Learning, pp. 7703, 2019.
Xiao-Dong Li, John K. L. Ho, and Tommy W. S. Chow. Approximation of dynamical variant systems by continuous- recurrent neural networks. IEEE Transactions on Circuits and Sys- tems II: Express Briefs, 52(10):656-661 2005.

---

Published as a conference paper at ICLR 2021
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: the Penn Treebank. 1993.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating mul-
tivariate point process. In Advances in Neural Information Processing Systems, pp. 6754-6764, 2017.
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogo- nal parametrisation of recurrent neural networks using Householder reflections. In International Conference Machine Learning, pp. 2401-2409, 2017.
John Miller and Moritz Hardt. Stable recurrent models. In International Conference on Learning Representations, 2019.
Seyed-Mohsen Moosavi-Dezfoolii Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method fool deep neural networks. In Conference on Computer Vision and Pattern Recognition, pp. 2574-2582, 2016.
Tan M. Nguyen, Richard G. Baraniuk, Andrea L.. Bertozzi, Stanley J. Osher, and Bao Wang. Momen- tumrnn: Integrating momentum into recurrent neural networks. arXiv preprint arXiv:2006.069/ 2020.
Murphy Yuezhen Niu, Lior Horesh, and Isaac Chuang. Recurrent neural networks in the eye of differential equations. arXiv preprint arXiv:1904 12933, 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318, 2013.
Barak A. Pearlmutter. Gradient calculations for dynamic recurrent neural networks: survey, IEEE Transactions Neural networks, 665):1212-1228, 1995.
Fernando Pineda. Dynamics and architecture for neural computation. Journal of Complexity, (3):216-245, 1988.
Yulia Rubanova, Tian Chen, and David Duvenaud. Latent ordinary differential equations for irregulatly-sampled time series. In Advances in Neural Information Processing Systems, pp- 5321-5331, 2019.
Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. arXiv preprint arXiv: 04454, 2017.
Shankar Sastry. Nonlinear systems: Analysis, stability, and control, volume 10. Springer Science, 2013.
Adam Trischler and Gabriele M. T. D' D'Elutterio. Synthesis of recurrent neural networks for dy- namical system simulation. Neural Networks, 80:67-78, 2016.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learn- ing recurrent networks with long term dependencies. International Conference on Machine Learning, pp. 3570-3578, 2017.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances in Neural Information Processing Systems, pp- 4880- 2016.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney, PyHessian: Neural networks through the lens of the Hessian. arXiv preprint arXiv:1912.07145. 2019.
Huaguang Zhang, Zhanshan Wang, and Derong Liu. A comprehensive review of stability analysis of continuous-time recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, 25(7): 1262, 2014.
12

---

Published as a conference paper at ICLR 2021
A PROOFS
A.1 PROOFS OF THEOREM
T AND LEMMA
1
There are numerous ways that one can analyze the global stability of (4) through the related model 5). many of which are discussed in Zhang et al. (2014). Instead, here we shall conduct a direct order to obtain cleaner conditions, and a more straightforward proof that readily applies in the time-inhomogeneowe setting.
approach and avoid appealing to diagonalization
Our method of choice relies on Lyapunov arguments summarized in the following theorem, which can be found as (Khalil 2002 Theorem 4.10). For more details on related Lyapunov theory, see also Hahn 1967 y: Sastry (2013).
Theorem 2. An equilibrium he for = f(t, h) is globally exponentially stable if there exists continuously differentiable function V (0, x R [0, 00) such that for all h RRN and 2 0,
OV
0V
1-11" & 1 S V(t,
h" II
and
Oh S
-kisl/ 1"11"
Ot
for some constants k1, 122, k3, 0r > O. and V(h) for h h²
To simplify matters, we shall choose a Lyapunov function V RN + [0, oo) that independent of time. The most common type of Lyapunov function satisfying the conditions of Theorem is of the form (hh) = (h ht P - h²), where P is a positive definite matrix. One need only show that (h) < (h )² Q(h (h- h²) for some other positive definite matrix Q to guarantee global exponential stability.
The construction of the Lyapunov function V that satisfies the conditions of Theorem 2 is accom- plished using the Kalman-Yakubowe Yakubovich- lemma, which is statement regarding strictly positive real transfer functions. We use the following definition, equivalent to other standard definitions by (Khalill 2002 Lemma 6.
Definition 2. A functio G C - NNN is strictly positive real if it satisfies the following:
(i) The poles of G(s) have negative real parts.
(ii) G(iw) G((-iw)" is positive definite for all R. where i V
(ii) Either G(oo) + G(oo)T is positive definite it is positive semidefinite and lim w² MT matrix M such that M (G(o) + G(oo)"
G(-iw)"") M f is positive definite for any N x (N q) full-rank where q rank([G(oo) + G(
The following is presented in Khalii 2002 Lemma 6.3).
Lemma 3 (Kalman- Sikkubovich-Popory Let A, W RN + RN be full-rank square matrices. There exists symmetric positive-definite matrix P and matrices L, U and a constant such that
= PA+ ATP = -T eP
-= WT
UTU=0 =0
if and only if the transfer function G(s)) W(s A) is strictly positive real. this case, we may take = 2/s, where p > 0 is chosen so that G - n) remains strictly positive real.
A shorter proof for case (a) is available to us through the (multivariable) circle criterion - the following theorem is corollary of (Khalil 2002 Theorem 7.1) suitable for our purposes.
Theorem (Circle Criterion). The system of differential equations
h Ah + (t,,, Wh)
is globally exponentially stable towards equilibrium at the origin if (|b(t, y)ll < M for some M. > and Z(s) = [I+ MG(s)][ - MG(+)]-1 is strictly positive real, where G(s) W(sT A)-1
13

---

Published as a conference paper at ICLR 2021
Both the Yakubovich- lemma and the circle criterion classical results in control theory, and are typically discussed in the setting of feedback systems (Khalil 2002 Chapter 6, 7)- Our presentation here is less general than the complete formulation, but makes clearer the connection to RNNs. With these tools, we state our proof of Theorem 1
To begin, we shall center the differential equation about the equilibrium. By assumption, there exists ht such that Ah (W + Ur(t) + b). Letting h = h - h², we find that
Proof of Theorem 1
= Ah + (Wh Uz(t) +)
Ah + Ah + (Wh + Wh + Ux(()++b)
= Ah + δ (Wh Wh + Ur(t) + - oW Ux(tt) b).
(13)
will suffice to show that 13 is globally exponentially stable at the origin.
Let us begin with (a). The proof follows arguments analogous to (Khalil 2002 Example 7.1).
Let G(s) = ((A s7) denote the transfer function for the system 13
Letting
(t, = a(x Wh + Ux(() + 6) oWWh + Un((t)++ +))
since a is M-Lipschitz, we know that (t, =>]| < M ||]||| for any E IRN. Therefore, let Z(s) II + MG(s)][[ - M MG((s]-
denote the transfer function in the circle criterion. Our objective to show that Z(s) is strictly positive real - by Theorem 3 this will guarantee the desired global exponential stability of 4 First, we need to show that the poles This only occur when G(s) itself has poles or / MG)(s) s) is singular. The former case occurs precisely where A - s/ singular, which occurs when is an eigenvalue of A. Since + A is assumed to be negative definite, A must have eigenvalues with negative real part by Lemma 4 and so the poles G(s) also have negative real parts. The latter case is more difficult to treat. First, since o max (AB) S maax (A)omax (B) and max (B+ (B-1)- min (B)
Z(s) have negative real parts.
T max (W)
O max (G(s)) S
(14)
m (A - =s])
Therefore, we observe that
amin -G((s))
2 Tmax (MG(s))
2 Ma (G(s))
Ma (W)
> Omin sT)
From the Fan-Hoffman inequality (Bhatia 2013 Proposition III.5.1), we have that
Tmin (A - xl) -0 (8/ - A) 2 Amin R(s)) A+ AT
A +
A"
= R(s) + Amin
2
2
and since + AT is negative definite, for any with R8(s) 2 0,
A A++A"
m (A - (8) 2 R(s) + amin
> T (A
(15)
2
Since miin miin (A Momax (W), it follows that mmin (I MG(s)) 0 whenever has non- negative real part, and so the poles of Z(s) must have negative real parts.
Next, we need to show that Z(iw) + ( (i is positive definite for all E R. Observe that Z(iw) + 7(- = [T MG BAL_--GG[AN MG [T ( + MG(- = - -GG
M
- MG
From Sylvester's law of inertia, we may infer that Z(iw) + ( ("-ww)) is positive definite if and only 7 + Y,, is positive definite, where Y,, of Y,, lie strictly within the unit circle, that is. Fuax (Yw) < 1 for all w E R, then I + Y., will necessarily be positive definite. From (14) and (15). we may verify that
G(iw) ) If we can show that the eigenvalues
T Taax (W)
Tmax (
sup 0 maax (G(w)) WER
< sup w WER J (A - iwl)
Tmin (Asym)
14

---

Published as a conference paper at ICLR 2021
Therefore,
Mo max (W) o min (Asym)
2
maxx (Yw) S M max (G(-tu)) J (G(x)))()))))))))))))
1,
by assumption. Finally, since 2(x)) + Z(xo)* = 2/ is positive definite, (()) is strictly positive real and Theorem applies.
Now, consider case (b). The proof proceeds in two steps. First, we verify that the transfer function G(s) ( sf) sf)-1 satisfies the conditions of the Galman-Yakuboovic-Poooe
lemma. Then, using the matrices P. L. U, and the constant inferred from the lemma, a Lyapunov function is constructed which satisfies the conditions of Theorem guaranteeing global exponential stability. Once again, condition (i) of Lemma 3 straightforward to verify: G(s) exhibits poles when is an eigenvalue of A, and so the poles of G(() also have negative real parts. Furthermore, condition (iii) is easily satisfied with M = I since G((x)) + G(oo)* 0. To show that condition (ii) holds, observe that for any w E R. letting A- (A-1)² for brevity,
G(iw) + G(-iw)" = W (A ) + (A (wl) "w"
iA iwl) (A iw1) [[A iw/) + w (A W*
Since the inner matrix factor is Hermitian, Sylvester's law of inertia implies that G(iw) + G(-iw)" is positive definite if and only if
B == (A + " + W (A (d))
is positive definite. Since B. is a Hermitian matrix, it has real eigenvalues, with minimal eigenvalue given by the infimum of the Rayleigh quotient:
Amin (w))
inf of B ||0|==1
inf v² o² W + WT A)U + iwr p ||0|==1
= innf oT W + WTAee
[|[|==
= = WA" W A).
By assumption, ATW + WTA has strictly positive eigenvalues, and hence B. and G(iw) + G(-iw)² are positive definite. Therefore, Lemma applies, and we obtain matrices P, L, U and constant e > with the corresponding properties.
Now may construct our Lyapunov function V. Let Wh and
(t) (<o(t) + Wh + Ux((t++ oWWh" + Uz(() + b),
so that à = Ad + u. Since a is monotone non-decreasing, axx) - a 2 0 for any x 2 y. This implies that for each 1, ...... N, 2; and u; have the same sign. In partieular, v u 0. Now, let V(h) = 1²2 Ph be our Lyapunov function, noting that is independent of t. Taking the derivative of the Lyapunov function over [33 and using the properties of P. U,U,
(h) T." Ph i'
WF A 4 2ht Pa
-W'(L"" 22 (L WT
2ht
= =-(LR) + (Lh)_Uu (Uu)² (Lh) - nI -TTTUU-
Un U00
= --CLhtUu) (Lh + Uu) eh Ph 2 u.
Since oF 2 and (Ih + (Th + Uu) 2 0. follows that (h) < -cAmin ( 1// and hence global exponential stability follows from Theorem and positive-defnittness of P. L
To finish off discussion regarding the results from Sec. 3, we provide a quick proof Lemma using simple diagonalization argument.
15

---

Published as a conference paper at ICLR 2021
Proof of Lemma Since A is symmetric and real-valued, by (Homn & Johnson, 2012 Theorem 4.1.5), there exists an orthogonal matrix P and a real diagonal matrix D such that = PDPT Letting ph" where h satisfies A since h P2,
that
1 = P PDP h p" ooWh Ux+
D= PT
@ (W P= + Ux b).
Therefore, satisfies (5) with L pr and V W.P both of which are nonsingular by orthogo- nality of P. By the same argument, for any equilibrium h², taking x- = p"hh',
D= + PT oW + Un + b) = P" (PDP"h* h²- + +oWWh + +Ux + b))
= -P' + +oWh + Un b)) = 0.
implying that 2" an equilibrium of (5) Furthermore, since
||2 - 21 (P* he'h-p"'h' p"h"""""'PP h =
(h h²) = h 1²1
from orthogonality of P. Because every form of Lyapunov stability. both local and global, including global exponential stability, depend only on the norm || 11 (Khalil 2002 Definitions 4.4 and 4.5), h" is stable under any of these forms if and only if 2 is also stable.
We remark that the proof of Lemma D can extend to matrices A which have real eigenvalues and are diagonalizable. These attributes are implied for symmetric matrices. However, they can be difficult to ensure in practice for nonsymmetric matrices without imposing difficult structural constraints.
A.2 PROOF OF PROPOSITION 1
The proof of Proposition D relies on the following lemma, which we also have made use of several times throughout this work.
Lemma 4. For any matrix A R.NNN, the real parts of the eigenvalues RX,(A) are contained in the interval [Amin ( (A9m),, Aunax (A+ym)), where A8ym = (A A²).
Proof. Recall by the min-max theorem, for (u, v) u"v, v. where u² is the conjugate transpose of u,
the upper and lower cigenvalues of A + AT satisfy
Amin (A + A² = inf
(u, (A + A²")o) = inf
Av) + (Au, v),
wECN ||||
VEEN ||
Amax (A+ A4+ vECN, |||==
sup
(u. (A + J²)))= sup vEEK
|= Av) + (Au, v).
Let {x(A) = + iw be an eigenvalue of with corresponding eigenvector e satisfying ||l||| 1. Since Au (u + iw)u,
(v, Av) (Av, Av) + (v, Ad) 22((,, Av) 2u/1o|12 2u.
Hence, Amin (A + A²)
I u S Amax (A A A²).
By construction, $ S3.y + 53,, = (1- M) - -y,, and so from Lemma 4 both the real parts (S,())))) of the eigenvalues of S3,7 as well as the eigenvalues of Saym lie the interval
Proof of Proposition
7
[Amin(S SSym)) 8,7 Amax (5
(SevmL)
Dmin ((1 B) M - -T), Amax ((1 - (3 Mamm - 77)]
3 < 1, for any eigenvalue À geym with corresponding eigenvector v, 3,y
+ B
(1 1- yv λe, and so My v
implying that A+y
1-3 is an eigenvalue of M and DAmiu (Meym), Amaase (Meym)) ]. In particular, we find that
therefore
in contained
8,7 [Amin (5 Amax (5 C [(1 3)mmin (M/y)) - 7, (1 - 3) Amax (M)
(16)
as required. Finally, 3 = 1, then [6 still holds, since both intervals collapse to the single point (-7).
16

---

Published as a conference paper at ICLR 2021
25
magniude
0.0
-25
08
0.7
0.5
0.7
0.9
tuning parameter, ,
tuning parameter, ,
(a) N=64
(b) N=128
Figure 4: Empirical evaluation of the theoretical bounds 16 The red lines track the largest real part and the blue lines track the smallest real part of the eigenvalues of the hidden-to-hidden matrix Ag. Each line corresponds to a different hidden-to-hidee matrix of dimension N = 64 in (a) and N 128 (b). The dashed black lines indicate the theoretical bound for each trial.
Figure illustrates the effect of 3 onto the eigenvalues of Ag,y with the largest and smallest real parts. It can be seen, both empirically and theoretically, that the real part of the eigenvalues con- verges towards zero as tends towards one, i.e., we yield skem-symmetric matrix with purely imaginary eigenvalues in the limit. Thus, for sufficiently large parameter 3 we yield system that approximately preserves an "energy" for limited time-horizon
(As(Aa)) 0, for 1,2,.... N.
(17)
A.3 PROOF OF LEMMA
First, it follows from Gronwall's 's inequality that the norm of the final hidden state [[4(T)]] is bounded uniformly in B. From Weyl's 's inequalities and the definition of AB,y-
max As (A s A (1 a
By the chain rule, for each element M of the matrix MA.
θL 0L 0y(T) öh(T) 0L (T(
0M Ty(T) Oh(T) 0M Ty(TT) 0M
Now, for any collection of parameters 0,,
d
Oh 00,
Oh
0A h + sech? (Wh + Ux+D b)
Öh
OW
Σ Σ
W Σ
Σ
=
dt
00:
00,
00:
00,
and from Gronwall's inequality.
ah(T)
AA
aWan
)
Σ
+
Σ
|
00;
00;
20;
T
Since As Ma
AL
+
AL 8MA
AMA
| M sym
< A
M
sym
A
aL
0L
s 8
OM
OM
< 8
aL oy
|a (0l D)T
dA + 0Ag,
0M
dM
Since
0 0(Mgh)
follows that
0M4
2M1
0Ag.
dAa
(((MA)) OMC
h(M
2(1
3)
+
0M4
aM
aM
17

---

Published as a conference paper at ICLR 2021
and so ||AsMA' O B)), and therefore max; (A = (18(1-1) Similarly, for the matrix Mw:
max
AGAR (W
$ (1 M
10
@Wa.
AWB
S 6(1 - B)
[D] |b||| A.l.ellllllllaaa...
aMW
aM
OL
@(Mwh)
((M
25(1 33)2
[D]] ||] 1+1W-,,11
By
amW
aMW
and hence max;
A (W
- -
In Figure 5 we plot the most positive real part the eigenvalues As.y and W3.7 during training for the ordered MNIST task. As increases, the eigenvalues change less during training, remaining the stability region provided by case (b) of Theorem for more the training time.
0.3
real part eigenvalue
maxRMAAs, ma
maxiMAA P
0.2
0.2
max3AA(W2.>
0.1
0.2
0.3
-0.2
-02
-0.3
-03
-04
-0.5
-05g
number of epoch
number of epoch
(a) 8 = 0.65
(b) 3 0.95
Figure 5: The red lines track the largest real part of the eigenvalues of the sadden-to-hidden matrix Ag,r and the blue lines track the largest real part of the eigenvalues of W3,y. We show results for two models trained the ordered MNIST task with varying 3.
B ADDITIONAL EXPERIMENTS
B.1 SENSITIVITY TO RANDOM INITIALIZATION FOR MNIST AND TIMIT
The hidden matrices are initialized by sampling weights from the normal distribution (O0 a), where is the variance, which can be treated as tuning parameter. In our experiments typically chose a small σ; see the Table for details. To show that the Lipschitz RNN is insensitive to random initialization, we have trained each model with 10 different seeds. Table shows the maximum, average and minimum values obtained for each task. Note that higher values indicate better perfor- mance the ordered and permuted MNIST tasks, while lower values indicate better performance on the TIMIT task.
B.2 ORDERED PIXEL-BY-PINEL AND PODSE- CIFAR-
The pixel-by-pixel CIFAR-10 benchmark problem that has recently been proposed by (Chang al., 2019). This task is similar to the pixel-by-pixel MNIST task, yet more challenging due to the increased sequence length and the more difficult classification problem. Similar to MNIST, we flatten the CIFAR-10 images to construct sequence of length 1024 in scanline order, where each element of the sequence consists of three pixels (one from each channel).
A variation of this problem is the noise poise-padded CIFAR-10 problem (Chang et al. 2019). where consider each row of an image as input at time step t. The rows from each channel are stacked so that obtain input of dimension R Then, after the 32 time step which process the 32
18

---

Published as a conference paper at ICLR 2021
Table 4: Sensitivity random initialization evaluated over 10 runs.
Solver
Task
Minimum
Average
Maximum
N
# params
Euler
ordered MNIST
98.9%
99.0%
99.0%
64
RK2
ordered MNIST
98.9%
99.0%
99.1%
64
R99
Euler
ordered MNIST
99.0%
99,2%
99,4%
128
<34K
RK2
ordered MNIST
98.9%
99.1%
99.3%
128
<34K
Fuler
permuted MNIST
93.5%
93.8%
94.2%
64
RK2
permuted MNIST
93.5%
93.9%
94.2%
64
Fuler
permuted MNIST
95.6%
95.9%
96.3%
128
3:44
RK2
permuted MNIST
95.4%
95.8%
96.2%
128
#34K
Euler
TIMIT (lest MSE)
2.82
2.98
3.10
256
198K
RK2
TIMIT (test MSE)
2.76
2.81
2.84
256
198K
Table 5: Evaluation accuracy on pixel-by-pixel CIFAR-10 10 and noise padded CIFAR-10.
Name
ordered
noise padded
params
1.STM haseline by (Chang et al.
2019
59.7%
11.6%
128
69K
Antisymmetric RNN (Chang et
2019
58.7%
48.3%
256
36K
Incremental RNN (Kag et al. 2020
54.5%
128
Lipschitz RNN using Fuler (ours)
60.5%
57.4%
128
34K/46K
Lipschitz RNN using RK2 (ours)
60.3%
57.3%
128
34K/46K
Lipschitz RNN using Euler (ours)
64.2%
59.0%
256
134K/158K
Lipschitz RNN using RK2 (ours)
64.2%
58.9%
256
134K/158K
row, start feed the recurrent unit with independent standard Gaussian noise for 968 time steps. At the final point in T = 1000, we use the learned hidden state for classification. This problem is challenging because only the first 32 time steps contain signals. Thus, the recurrent unit needs to recall information from the beginning of the process.
Table provides a summary of our results. Our Lipschitz recurrent unit outperforms both the in- cremental RNN (Kag et al., 2020) and the antisymmetric RNN (Chang et al. (019) by significant margin. This impressively demonstrates that the Lipschitz unit enables the stable propagation of signals over long time horizons.
B.3 PENN TREE BANK (PTB)
B.3.1 CHARACTER LEVEL PREDICTION
Next, we consider character level language modeling task using the Penn Treebank Corpus (PTB) [Marcus al. 1993). Specifically, this task studies how well model can predict the next character in a sequence of text. The dataset is composed of train / validation test set, where 5017K characters are used for training, 393K characters are used for validation and 442K characters are used for testing. For our experiments, we used the publicly available implementation of this task by Kerg et al. (2019), which computes the performance terms of mean bits per character (BPC).
Table 6 shows the results for back-propagation through time (BPTT) over 150 and 300 time steps, respectively. The Lipsehitz RNN performs slightly better then the exponential RNN and the non- normal RNN on this task. (Kerg al.) 2019) notes that orthogonal not particular well-suited for this task. Thus, not surprising that the Lipschitz unit has a small advantage here.
matrices
For comparison, we have also tested the Antisymmetric RNN (Chang et al. 2019) on this task. The performance of this unit is considerably weaker compared to our Lipschitz unit. This suggests that the Lipschitz RNN is more expressive and improves the propagation meaningful signals over longer time scales.
19

---

Published as a conference paper at ICLR 2021
Table 6: Evaluation accuracy on PTB for character-level prediction for different sequence lengths
T. The indicate results that were adopted from Kerg et al. 22099)
Name
TPTA 150
TPTr
300
# params
RNN baseline by (Arjovsky et al. 2016
2.89
2.90
= =1.22M
RNN-orth (Henaff ct al. 2016
1.62
1.66
1.32M
EURNN (ing et a.. 201
1.61
1.62
*1.22M
Exponental KNN (Leccano- Lezcano-Cusado & Martinez- 2019
1.49
1.52
*1.32M
Non-normal RNN (Kerg el al. 2019
1.47
1.49
1..22M
Antisymmteric RNN
1.60
1.64
1..2MM
Lipschitz RNN using Fuler (ours)
1.43
1.46
1.32M
B.3.2 WORD-LEVEL, PREDICTION
In addition character- level prediction, we also consider word-level prediction using the PTB cor- pus. For comparison with other state-of-the-aae units, we consider the setup by Kusupati et al. (2018), who use sequence length of 300. Table shows results for pocpppropagttion through time (BPTT) over 300 time steps. The Lipschitz. RNN performs slightly better than the other RNNs on this task and the baseline LSTM for the test perplexity metric reported by Kusupati et al./ (2018).
Table Evaluation accuracy on PTB for word-level prediction. The indicate results adopted from Kusupati et al. (018). Note that here the parameters for the hidden-to-hidder units are reported.
Name
validation perplexity
test perplexity
N
# params
LSTM (*)
117.41
210K
SpectraIRNN (*)
52.5K 130.20
24.8K
FastRNN (*)
127.76
FastGRNN-LSQ (*)
115.92
52.5K
FastGRNN (+)
116.11
52.5K
Incremental RNN Kag et al. 2020
115.71
29.5K
Lipschitz RNN using Hiler (ours)
124.55
115.36
160
50K
C TUNING PARAMETERS
For tuning we utilized a standard training procedure using a non-exhaustive random search within the following plausible ranges for the weight parameterization 3 = 0.65, 0.7, 0.75, 0.8, [0.001, 1.0]. For Adam we explored learning rates between 0.001 and 0.005, and for SGD we con- sidered 0.1. For the step size explored values in the range 0.001 1.0. We did not perform automated grid search and thus expect that the models can be further fine-tuned.
The tuning parameters for the different tasks that we have considered are summarized in Table g
For pixel-by-pixel MNIST and CIFAR- 10, we use Adam for minimizing the objective. We train all models for 100 epochs, with scheduled learning rate decays epochs {90). We do not gradient clipping during training. Figure shows the test accuracy curves for our Lipschitz RNN for the ordered and permuted MNIST classification tasks.
For TIMIT we use Adam with default parameters for minimizing the objective. We also tried Adam using betas (0.0, 0.9) well as RMSprop with a 0.9, however, Adam with default values worked best in our experiments. We train the model for 1200 epochs without learning-rate decay. Similar Kerg et al. (2019) we train our model with gradient clipping, however, we observed that the performance of our model is relatively insensitive to the clipping value.
For the character level prediction task, use Adam with default parameters for minimizing the objective, while we use RMSprop with a = 0.9 for the word level prediction task. We train the model for 200 epochs for the character- task, and for 500 epochs for the word-level task.
20

---

Published as a conference paper at ICLR 2021
Table 8: Tuning parameters used for our experimental results and the performance evaluated with 12 different seed values for the parameter initialization of the model.
Name
N
Ir
decay
/
Tw
a
Ordered MNIST
64
0.003
0.1
0.75
0.001
0.001
0.03
0.1/64
Ordered MNIST
128
0.003
0.1
0.75
0.001
0.001
0.03
0.1/128
Permuted MNIST
64
0.0035
0.1
0.75
0.001
0.001
0.03
0.1/128
Permuted MNIST
128
0.0035
0.1
0.75
0.001
0.001
0.03
0.1/128
Ordered CIFARIO
256
0.1
0.2
0.65
0.001
0.001
0.01
6/256
Noise-padded CIFARIO
256
0.1
0.2
0.75
0.001
0.001
0.01
6/256
TIMIT
256
0.001
0.8
0.8
0.001
0.9
12/256
PTB character-level 150
750
0.005
0.8
0.5
0.001
0.1
12/256
PTB character-level 300
750
0.005
0.8
0.5
0.001
0.1
12/256
PTB word-level
160
0.1
0.8
0.9
0.001
0.01
10/256
1.00
1.00
0.55
0.35
TCSE TOSEacoe accuracy
accuracy
0.90
0.30
0.85
0.85
0.00
0.80
0.7
Lischii RNN (Ne64)
0.75
Lipecinic RNN (N#664)
(N-128)
Upschttz RNN (N-128)
0.70
20
40
60
80
100
0.70
20
40
50
80
100
epochs
epachs
(a) Ordered bxxll-y-pixel MNIST
(b) Permuted pixel-by-pixel MNIST.
Figure 6: Test accuracy for the Lipschitz RNN for different classification tasks.
21


Reviews:
[{"review_id": "yip6SqFJes7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a new continuous-time RNNs which appears to be extremely similar to CT-RNN (Funahashi et al. 1993). They then constraint the network representation to account for learning long-term dependencies. \n\nPositive: The formulation of the constraint is very clear and sound. \n\nPositive: The analysis of the stability of the model and other properties is rigor enough and to the best of my knowledge sound and correct. \n\nNegative: From the experimental setting, reported values in tables, and code, it seems like the authors tuned the hyperparameters on the test set which I consider a bad practice that violates the code of conduct! I suspected this, and therefore, ran the code myself. With a few changes to the hyperparameters from the tuned one reported in the table the performance of the proposed model dropped significantly!\n\nI would suggest the authors to create a fair testing scheme for all baselines and report the experimental results more accurately.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VtRbRkYH7vF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have increased my score to reflect the revisions\n\n---------\n\nThis paper presents yet another architecture for fully connected RNNs or infinitely deep networks based on the integration of a continuous time dynamical system, where a projection of the weights is used to guarantee stability, hence a fixed point and a finite Lipschitz constant. \n\nPositives: The maths presented in the paper is correct and their results are nicer than the considered (not exhaustive) baselines.  \n\nNegatives:\n\n1) This idea has been widely explored and exploited by now. Adding a linear term and using a different solver is not enough in my opinion to make an innovative contribution. \n\nIf you wish to present this as an ablation study, then perhaps you need to benchmark against existing solutions.\n \nFor instance, in this (missing) reference a very similar network is presented and analysed. \n\n@incollection{NIPS2018_7566,\ntitle = {NAIS-Net: Stable Deep Networks from Non-Autonomous  Differential Equations},\nauthor = {Ciccone, Marco and Gallieri, Marco and Masci, Jonathan and Osendorfer, Christian and Gomez, Faustino},\nbooktitle = {Advances in Neural Information Processing Systems 31},\npages = {3025--3035},\nyear = {2018},\npublisher = {Curran Associates, Inc.},\nurl = {http://papers.nips.cc/paper/7566-nais-net-stable-deep-networks-from-non-autonomous-differential-equations.pdf}\n}\n\nI will refer to this as [1]. While this paper was about unrolling the stable RNN to generate a deep Lipschitz classifier, and was not used for sequence to sequence task, the architecture you propose and the claims are so much similar to [1] that this demands for a direct comparison. \n\nIn the above paper, stability projections are presented for an architecture that is essentially the same, minus the additional linear component here. Paper [1] should be included as a baseline. You have it already implemented for fully connected layers. \n\n2) It is not very clear how this additional linear component would help in practise, as your architecture is fundamentally discretised with Euler which results in yet another generalised res-net. It has been shown that ResNet works much better than their predecessor,  Highway networks, because of the direct skip connection and better gradient flow. \nWhile the missing reference [1] (NAIS-Net) preserves that connection, It feels like your linear term would get rid of the skip connection and prevent the technique from being used in very deep networks or very long sequences due to vanishing gradients. \n\n3) What is the motivation for using a continuos-time approach? Your results are invalidated by the forward Euler, unless a very small hyperparameter epsilon is introduced. Why not just compute the projection in discrete time as done in [1]. Your stability will not hold for sparse in time data-points, because the Euler step would become too big and this is effectively an RNN that cannot handle different sampling time while preserving stability. \n\n4) It seems strange to compare to NODEs as they are meant to be used for something else. In particular, NODEs are designed to work without inputs but just by estimating the initial condition for the ODE and then \"unrolling\". \nIf you add input signals to the NODE, which I guess is what you mean by \"NODE RNN\", how do you train it with the adjoint method? This should be made very clear in the paper. \n\n5) Your results are limited to a fully connected architecture, while [1] has shown a method to have a Lipschitz RNN for convolutional layers. Can you generalize to that as well?\n\nI don't feel the contribution here is relevant enough to be included in the conference. \n\nIf the above points are clarified in a convincing way and both the theoretical justification and the ablations performed with respect to [1], then I could consider improving my score. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BzTmHRdhJGK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Considering a continuous time RNN with Lipschitz-continuous nonlinearity, the authors formulate sufficient conditions on the parameter matrices for the network to be globally stable, in the sense of a globally attracting fixed point. They provide a specific parameterization for the hidden-to-hidden weight matrices to control global stability and error gradients, consisting of a weighted combination of a symmetric and a skew-symmetric matrix (and some diagonal offset). The authors discuss numerical integration by forward-Euler and RK2, and thoroughly benchmark their approach against a large set of other state-of-the-art RNNs on various tasks including versions of MNIST and TIMIT.  Finally, they highlight improved stability of their RNN against parameter and input perturbations.\n\nWhat I like about this paper is that it provides a solid theoretical basis and a principled, insightful parameterization that let’s one control separately the size of the real and the imaginary parts of the eigenvalues of A and W. The paper contains a number of interesting thoughts, and a really extensive comparison to other state-of-the-art models on several benchmarks.\n\nIn general, I feel however that this work is relatively close to the 2019 ICLR paper by Chang et al.; in several ways it feels like a more or less straightforward extension of this previous work.\n\nIt also remains a bit unclear to me how it’s ensured in practice that the matrices obey to the required conditions in the training process. Sect. 5 is not really about training, but just about numerically solving the ODE. From Appendix C it seems the scalar parameters $\\beta, \\gamma$ controlling the influence of the symmetric vs. skew-symmetric parts and the offset are not learned at all but just fixed after grid-search? The component matrices B, C, on the other hand it seems are not restricted at all but just initialized such that the theoretical conditions are likely, but not necessarily, met? This seems somewhat unsatisfying as there are in fact no guarantees that the global stability conditions will be met in practice, and tuning the model may require (potentially extensive) meta-parameter search?\n\nAnother drawback in my mind is that enforcing global fixed point dynamics is quite restrictive. For instance, this rules out cycles and many other interesting dynamics in the model’s intrinsic behavior. Apparently, from the authors’ empirical tests, this seems not to be required for solving this particular set of tasks. Which is somewhat puzzling to me as it appears this assumption should strongly curtail the model’s expressiveness.\n\nIn the tables and figures I missed statistics. No standard errors or confidence bands were provided, or how many runs were performed. If it’s all from a single run, can I be certain the numbers are not just lucky draws?\n\nNevertheless, given the overall convincing and extensive empirical results, I’m slightly leaning toward acceptance.\n\nMinor issues:\n- The authors sell the additional linear term in their RNN as a novelty, while in fact it’s rather standard in continuous RNN (older papers by Barak Pearlmutter, Song & Wang 2016, arxiv.org/abs/2006.02427, arxiv.org/abs/1910.03471)\n- RK2 is generally not sufficient for more involved (stiff) dynamical problems; so the reason it works well here may lie in the fact that the model’s intrinsic dynamic is indeed very simple\n- What is an unstable unit? I guess the authors mean that the RNN is not globally stable?\n- Sect. 2,  dynamical systems inspired RNN: It may be important to note that formulating a RNN as ODE does not solve the exploding/vanishing gradient or stability problem per se (nor is it immediately clear to me why it should actually make it easier).\n- Theorem 1: The $\\sigma$ refer to the matrix eigenvalues in this case?\n- What is a superset of skew-symmetric matrices?\n- Second-to-last pg. of Sect. 7 was unclear to me, i.e. what exactly was done and evaluated here, maybe cos I’m not familiar with some of the cited methods.", "extra_scores": {}, "extra_text": {}}, {"review_id": "15cjBR4BLHY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents some novel contributions regarding recurrent neural networks. \nBuilding on the work of Chang et al. (2019),  the authors provide a global convergence result for the hidden representation of a family of recurrent neural networks using standard techniques from the Lyapunov analysis of dynamical systems.\nThe requirements of the theorem are met  (within the limits of discretization) by their proposed algorithmic scheme. \nNumerical evaluation on a variety of benchmarks shows that the proposed algorithm yields systematic improvement over other RNN approaches. \nFor all of the above reasons, I recommend the acceptance of the paper. \n\nSome concerns to be addressed:\n- The connection between stability and trainability or refer to Chang et al. (2019) if their analysis applies here. \n- specify the functions \\sigma_min and \\sigma_max used in Theorem 1\n- specify the meaning of the one-arg function f(h^*) as opposed to the 2-arg f(h,t) appearing in Definition 1. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Solid work on extending AntisymmetricRNN and expanding its expressivity while controlling the global stability of the recurrent dynamics. It contributes to the growing interest in continuous-time RNN formulations that can deal with exploding gradient problem, and worthy of ICLR poster presentation. Three reviewers were positive and one was slightly negative. Authors added additional experiments and strengthened the manuscript significantly during the review process.
Decision: Accept (Poster)