Recursive Construction of Stable Assemblies of Recurrent Neural Networks
Authors: Leo Kozachkov
Michaela M Ennis
Jean-Jacques Slotine
Venue: ICLR 2022 Submitted (2022)
Keywords: stability
deep learning
RNN
combinations
modularity
sparsity
negative feedback
sequence learning
contraction analysis

Abstract: Advanced applications of modern machine learning will likely involve combinations of trained networks, as are already used in spectacular systems such as DeepMind's AlphaGo. Recursively building such combinations in an effective and stable fashion while also allowing for continual refinement of the individual networks - as nature does for biological networks - will require new analysis tools. This paper takes a step in this direction by establishing contraction properties of broad classes of nonlinear recurrent networks and neural ODEs, and showing how these quantified properties allow in turn to recursively construct stable networks of networks in a systematic fashion. The results can also be used to stably combine recurrent networks and physical systems with quantified contraction properties. Similarly, they may be applied to modular computational models of cognition. We perform experiments with these combined networks on benchmark sequential tasks (e.g permuted sequential MNIST) to demonstrate their capacity for processing information across a long timescale in a provably stable manner.

Under review as a conference paper at ICLR 2022
RECURSIVE CONSTRUCTION OF STABLE ASSEMBLIES OF RECURRENT NEURAL NETWORKS
An Anonymous authors Paper under double-blind review
ABSTRACT
Advanced applications of modern machine learning will likely involve combi- nations of trained networks, are already used in spectacular systems such Deep.indds AlphaGo. Recursively building such combinations in an effective and stable fashion while also allowing for continual refinement of the individual networks as nature does for biological networks will require new analysis tools. This paper takes a step in this direction by establishing contraction properties of broad classes of nonlinear recurrent networks and neural ODEs, and showing how these quantified properties allow in turn to recursively construct stable networks networks in systematic fashion. The results can also be used to stably combine recurrent networks and physical systems with quantified contraction properties. Similarly, they may be applied modular computational models of cognition. We perform experiments with these combined networks on benchmark sequential tasks (e.g- permuted sequential MNIST) demonstrate their capacity for process- ing information across long timescale in provably stable manner.
INTRODUCTION
Neuro-inspired machine learning has profoundly altered many fields such computer vision, nat- ural language processing, and computational neuroscience (Bengio et al., 2017; Hassabis et al., 2017). While models trained with e.g. deep learning are remarkably powerful, they are for the most part 'black boxes'. This opaqueness can be dangerous in cafety-critical applications, such as autonomous driving or human-centered robotics, and it limits conceptual progress. In the case of recurrent models, one difficulty is that providing a certificate of stability is currently impossible or computationally impractical. Given that stability a fundamental property of dynamical systems and is intimately linked to concepts of control, generalization, data-efficiency, and robustness being able to guarantee the stability of a recurrent model is an important step towards making sure deep models behave
expect them to.
In this spirit, there has been recent flux of work focusing on applications of contraction analy- sis (Lohmiller & Slotine, 1998) recurrent models. Loosely speaking, dynamical system is said to be contracting if any two of its trajectories converge to each other exponentially, regardless of initial conditions. primary advantage of contraction analysis is that it is directly applicable to non-autonomous systems, which the vast majority of recurrent models are, allowing in turn modular cnnrraction-preserving combination properties to be derived (Slotine & Lohmiller, 2001; Slotine, 2003). We include a mathematical primer on contraction analysis in Section Al.
Within this line of research, our paper has two main aims: 1) To provide simple contraction con- ditions for continuous-time recurrent neural networks and 2) To show how these continuous-time contraction conditions imply combination property. Using both aims, we proceed implement stable combination networks that exhibit state-of thee-----therrtttt (SOTA) performance on multiple sequential image classification tasks with a small number of trainable parameters.
1.1 PREVIOUS WORK ON RNN STABILITY
To briefly review the current literature application of contraction analysis recurrent models, first note that the 'Echo-State Condition' introduced in Jaeger (2001) is equivalent to discrete-tima time contraction in the identity metric. A later generalization of this condition included diagonal metric

---

Under review as a conference paper at ICLR 2022
(Buchner & Young, 2006). In the context of neuroscience, contraction analysis has been applied to analyzing the dynamics of winner- take-all networks (Rutishauser et al., 2011; 2015) as well as networks with synaptic plasticity (Kozachkov al., 2020). In the machine learning context, Miller and Hardt recently rederived an 'echo-state property" for discrete recurrent models, and went on to prove that these contracting recurrent models could be well-approximated by feedforward networks in certain cases (Miller & Hardt, 2018). More recently still, in series of papers Revay, Wang, and Manchester applied contraction analysis discrete-time recurrent networks (Revay & Manchester, 2020; Revay et al., 2021; 2020a), expanding the class of models considered in Miller & Hardt (2018).
In addition to contraction (which amounts to strong form of non aan-autooomous exponential stability) there has been a considerable amount of work attempting to enforce weaker forms of stability in RNNs, such as autonomous stability (Erichson al., 2021; ; Chang et al., 2019). Unfortunately. as we discuss in Section 2.3, autonomous stability does not in general imply non-autonomous stability, so it is not clear what stability properties these RNNs possess when driven with external input.
There is also line of work which uses orthogonal weight matrices to avoid the vanishing/exploding gradient problem during training. Orthogonality is typically ensured during training via param- eterization (Arjovsky exploiting the fact that the matrix exponential skew-symmetric matrix is orthogonal, as is done in Lezcano-Casade & Martinez-Rubi (2019). These works focus parameterizations of individ- ual RNNs which ensure stability is preserved during training. By contrast, our work focuses on 'combined' RNNs (i.e network of networks) and provides a parameterization the connections between stable subnetworks such that training preserves the overall network stability (Figure 1).
al., 2016; Lezcano-Casado & Martinez-Rubo, 2019) for example, by
Xb
Xa
Feedback
Feedback
Hierarchical
Hierarchical
Figure 1: Our stability certificate implies modularity principle. It may be used to recursively construct complicated 'networks of networks' while automatically maintaining stability.
1.2 COMBINATION NETWORKS
The combination and reuse of primitive "modules" has enabled a great deal of progress in computer science, and is also a key theme in biological evolution, particularly apparent in cortical structure of the human brain. In fact, it is thought that the majority of traits that have developed over the last 400+ million years are the result of evolutionary forces acting on regulatory elements that combine core components, rather than mutations to the core components themselves. This mechanism of action makes meaningful variation in population phenotypes much more feasible to achieve, and is appropriately titled "facilitated variation" (Gerhart & Kirschner, 2007). In addition to the biolog- ical evidence for facilitated variation, computational models have demonstrated that this approach produces populations that are better able to generalize to new environments (Parter al., 2008), an ability that will be critical to further develop in deep learning systems.
While the benefits of building modular systems are clear (Simon, 1962), in DeepMind's AlphaGo for example (Silver et al., 2016). there is no general guarantee that a combination of stable systems will itself be stable. Thus the tractability of these evolutionary processes hinges on some mechanism for ensuring stability of combinations. Because contraction analysis tools allow complicated con- tracting systems to be built up recursively from simpler elements, this form stability well suited for biological systems (Slotine & Lohmiller, 2001; Slotine & Liu, 2012). Note also that contracting combinations can be made between systems with very different dynamics, as long as those dynam- ics are contracting. Here, describe two common forms of contracting system combinations hierarchical and feedback - that automatically guarantee overall system stability (Figure 1).
Ultimately, cognitive models are moving increasingly toward study of multi-area dynamics, but many questions remain on how to best train and evaluate such networks (Yang et al., 2018; Yang &
2

---

Under review as a conference paper at ICLR 2022
Molano-Mazon 2021). Understanding how different brain regions interact harmoniously to produce a unified percept/action will require new ideas and analysis tools.
MATHEMATICAL RESULTS
2.1 CONTRACTION CONDITIONS FOR INDIVIDUAL, CONTINUOUS-TIAE TIME RNNS
We consider the following continuous-time RNN, evolving according to the following equations:
- Wo(x) + u(())
(I)
where x E IR", Ø is a static nonlinearity such that 0 S Ø' S u(t) is some input (potentially time- varying) and O. We do constrain the sign of Example nonlinearities d(x) that satisfy the constraints are tan)(ax). log(1 + e"), and max(0, x), with g a. 1, respectively. Note that this class of RNNs equivalent another commonly used class of RNNs where the terms Wx + appears inside d6.). See Section A2 or (Miller & Fumarola, 2012) for details. Our mathematical results therefore apply equally well to both types RNNs.
We seek stability certificate for this system in terms of the recurrent weight matrix W. We are specifically interested restricting W such that the RNN globally contracting. We do this with the goal of recursively combining these contracting RNNs with other contracting RNNs to make large, complicated, stable 'networks of networks'. Here we derive five restrictions on W which ensure contraction for the continuous-tin time RNN defined by equation 1. To the best of our knowledge these contraction conditions are novel. All proofs can be found the supplemental Section A4.
Theorem (Absolute Value Restricted Weights). Let W denote the matrix formed by taking the dlement-wise absolute value of W. If there exists positive, diagonal P such that:
W - I) + (g W 1) <0
then equation 1 contracting metric P. Moreover, if Wa S O, then [W]s: may be set zero to reduce conservatism.
Note that when = 1, Theorem can be checked by checking for linear stability of [W| -
Theorem (Symmetric Weights). J/W WT and g gW < I, then (1) is contracting.
Theorem (Product Diagonal and Orthogonal Weights), If there exists positive diagonal matri- ces P1 and P2, as well as Q Q" > 0 such that
W = PPQ
then (I) contructing in metric M (PIQP))
Theorem (Triangular Weights), If triangular and Hurwitz, then (1) is contracting in diagonal metric.
Theorem 5 (Singular Value Restricted Weights). If there exists positive diagonal matrix P such that:
2 PW-PW-P-
then (I) is contracting in metric P.
2.2
THE MODEL: NETWORK OF NETWORKS
The RNN equation a subnetwork of our model. Our goal is to combine these subnetworks into 'network of networks' in manner that preserves the stability of the underlying modules. In particular, we seek to parameterize and learn the inter-module connections. To do this, we prove and make extensive use of the following theorem:

---

Under review as a conference paper at ICLR 2022
Theorem 6 (Network of Networks). Consider collection of p subnetwork RNNs governed by equation 1. Assume that these RNNs each have hidden tillen-to-hiicee hidden weight matrices (W1..... Wpl and are independently contracting in metrics {M My). Define the block matrices W = BlockDiag( 1W....., p)) and M = BlockDiag(M, 1.. Mp), as well as the overall state vector x" (x] ..-2))
x²) ). Then the following "network of networks' is globally contracting in metric M:
- = - Wo(x) + u(t) L
L= B
Where B is an arbitrary square matrix. Note that we are agnostic to the particular contraction condition here the subnetwork RNNs can satisfy any of the theorems in the preceding section.
2.3 ON STABILITY THEOREMS FOR RNNS
Several recent papers in machine learning, e.g (Haber & Ruthotto, 2017; Chang et al., 2019), claim that a sufficient condition for stability of the nonlinear system:
x
f(x,
that the associated Jacobian matrix Jxx, t) negative, i.e:
has cigenvalues whose real parts are strictly 8x
8f
(x, max Re(Re (J (x, t)) Z -a
with o > 0. This claim is generally false. For counter-example, see Section 4.4.2 (Slotine & Li, 1991).
However, in the specific case of the RNN (I), it appears that the eigenvalues of the symmetric part of W do provide information in (Matsuoka, 1992) it was shown that if W, = (W + W has all its eigenvalues less than unity, and is constant, then (1) has a unique fixed point that is globally asymptotically stable. is easy to see that this condition also implies that the real parts of the eigenvalues of the Jacobian are uniformly negative. Moreover,
global stability in number of applications. For example,
(Chang et al., 2019) was shown that setting the symmetric part of W. . 1 (W- +T) almost equal to zero (yet slightly negative) led to rotational, yet stable dynamics in practice. This leads us to the following theorem, which shows that if the slopes of the activation functions change sufficiently slowly function of time, then the condition in (Matsuoka, 1992) in fact implies global contraction of (1).
Theorem 7. Let D be a positive, diagonal matrix with Dii positive diagonal matrix. If:
ddg and let P be an arbitrary, du."
(gW - 1)P P - 1) 1 cPP
and
Dg 3D
for B > 0, then (1) is contracting in metric with rate 3
has been conjectured that diagonal stability of gW I is a sufficient condition for global con- traction of the RNN 1 (Revay et al., 2020b), however this has been difficult to prove. To better characterize this conjecture, we present Theorem 8, which shows by way of counterexample that diagonal stability of g W does NOT imply global contraction in a constant metric for (1).
Theorem 8. Satisfaction of the condition gW eym - < is NOT sufficient to show global contraction of the general nonlinear RNN (I) in any constant metric. High levels of antisymmetry in W can make it impossible to find such a metric, which we demonstrate via a 2 x counterexample
0
-0
of the following form, with 2 W
4

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
is possible that imposing a strong stability constraint on the RNNs precludes them from perform- ing well on tasks that require information processing over long timescales. To investigate whether or not this occurs with our stable combination networks, we trained a variety of provably stable RNNS on benchmark sequential image classification tasks sequential MNIST, permuted sequential MNIST, and sequential CIFAR10 as these are frequently used to measure an RNN's s ability to use information from across a long sequence (Le al., 2015; Chang al., 2019). For all the tasks, images are presented pixel-by-pixee and the network makes a prediction at the end of the sequence. In the case of permuted seqMNIST, the pixels are presented fixed but random order.
Recall that our proposal is to constrain the stability of subnetworks in a way that makes training the combined network (using Theorem 6) stability The particular choice of subnetwork constraint influences the resulting training process and performance. We selected two stability con- straints from our results above to experiment with: Theorems and 5. the case of Theorem 1, we -o-hideen weight matrices, only training the con-
did not train the individual subnetworks' nections between subnetworks (Figure 2A). In the case of Theorem 5 we trained all the parameters of the model (Figure 2B).
We refer to networks with subnetworks constrained by Theorem as "Sparse Combo Nets' (be- cause, as we will detail below, sparse hidden-to-hidden weights more readily satisfy Theorem I). Similarly, we refer to networks with subnetworks constrained by Theorem 5 as 'SVD Combo Nets'. Throughout the experimental results refer combination networks of different sizes using the notation network'. Such a network consists of distinet subnetwork RNNs, with each such subnetwork RNN containing n units. For all subnetworks we use the ReLU activation function.
Sparse Combo Net
81
SVD Combo Net
(2x3 scheraatc
(2x3 schematic) tiknel
Usear Tesptleeesssoockk
Liber
Nogothi
m
Figure 2: Summary of network architectures. Nonlinear subnetworks are constrained to meet either Theorem via sparse initialization (A) Theorem via direct parameterization (B). Linear negative feedback connections are trained between the subnetworks according to 6. See Section AS for additional details on model definitions.
Our networks performed better than comparably to other stable RNNs across all tasks, especially when leveraging sparsity in the subnetworks. Our best test accuracies of 96.94% on permuted seqMNIST and 64.75% on seqCIFARIO 10 not only set but are also competitive with the overall SOTA while having many fewer parameters (Table I).
SOTA for stability- guaranteed RNNs,
3.1 NETWORK INITIALIZATION AND TRAINING DETAILS
For the Sparse Combo Net we were not able find a direct parameterization use during training, but it is straightforward to randomly generate matrices with particular likelihood of meeting the condition by selecting an appropriate density level and limit on entry magnitude. As every RNN meeting this contraction condition has well-defined stable LTTI system contracting in the same metric, it is not only easy to verify, but also easy to find a contraction metric to use in our training algorithm (Theorem 6). Solving for M in -I MA + A M will produce valid metric for any stable LTI system A (Slotine & Li, 1991). therefore randomly generate fixed subnetworks satisfying Theorem and train only the linear connections between them (Figure 3), well as the linear input and output layers.
5

---

Under review as a conference paper at ICLR 2022
A)
Becu dor Meyk
WiT hoge Mare
Figure 3: Example 3 x 16 Sparse Combo Net. The hidden-to-hie to-hidden nonlinear weights (W in Theorem 6) are initialized based on a set sparsity, and do not changed over training (A). The linear inter-subnetwork connections (L in Theorem 6) are constrained to be antisymmetric (with respect the overall network metric), and are updated in the training process (B).
For the SVD Combo Net, we ensured contraction by directly parameterizing each of the We (i 1,2... p) as:
W. - -,
(3)
where .; is diagonal and nonsingular, U; and V, are orthogonal, and Σ; is diagonal with Σ, E [0. (). We ensure orthogonality of U, and V, during training by exploiting the fact that the matrix exponential of skew-symmetric matrix is orthogonal, as was done in Lezzcan--Casaoo & Martinez- Rubio, 2019). The network constructed from these subnetworks using Theorem contracting in metric M B .)))
For both networks, we constrained the matrix B in Theorem 6 to reflect underlying modularity assumptions. particular, we only train the off-diagonal blocks B and mask the diagonal blocks. We do this to maintain the interpretation of L as the matrix containing the connection weights between different modules, as diagonal blocks would correspond to self-comnections. Furthermore, only train the nower-trianullat blocks of B while masking the others, to increase training speed.
Unless specified otherwise, all networks were trained for 150 epochs, using an Adam optimizer with initial learning rate le-3 and weight decay 1e-5. The learning rate was cut le-4 after 90 epochs and to le-5 after 140. After identifying the most promising settings, we ran repetitions trials on the best networks for 200 epochs with learning rate cuts after epochs 140 and 190. Detailed information on network initialization and hyperparameter tuning is provided in Section A3.
3.2 RESULTS
The Sparse Combo Net architecture achieved the highest overall performance on permuted seqM- NIST and SeICIFAR10 (96.94% and 64.75% best performance, respectively). On both these tasks, the Sparse Combo Net achieved SOTA for stable RNNs (Table 1). Furthermore, we were able to reproduce these scores several repetitions (Section 3.2.3). Along with repeatability of results, we also show that the contraction constraint on the connections between subnetworks (L in Theorem 6) is important for performance, particularly in the Sparse Combo Net.
Additionally, we profile how various architecture settings impact performance of our networks. We found that the sparsity of the hidden-to-hicee to-hidden weights in Sparse Combo Net had large impact the final network performance (Section 3.2. In both networks, we found that increasing the total number of neurons improved task performance, but with diminishing returns (Section 3.2.2).
3.2.1 EXPERIMENTS WITH SPARSITY
Because of the link between sparsity and stability well the biological relevance of sparsity, we explored in more detail how subnetwork sparsity affects the performance of Sparse Combo Net. We number of additional experiments the permuted SegMNIST task, varying sparsity level while holding network size and other hyperparameters constant. Here we use "n% sparsity level" to refer network with subnetworks that have just n% of their weights non-zero.
We observed a large (> 5 percentage point) performance boost when switching from a 26.5% spar- sity level 10% sparsity level in the 11 32 Sparse Combo Net (Figure 4), and subsequently

---

Under review as a conference paper at ICLR 2022
sMNIST Repeats
PSMNIST
SCIFARTO
Seq MNIST Best
PerSeq MNIST Best
Seg CIFAR Best
Stable RNN?
Repeats
Repeats
Name
Params
Mean (n)
Mean (n)
Mcan (n)
[Min]
[Min]
[Min]
LSTM (Chang al., 2019)
68K
97.3%
92.7%
59.7%
Transformer (Trinh al., 2018)
0.5M
98.9%
97.9%
62.2%
Antisymmetric (Chang et al., 2019)
36K
98%
95.8%
58.7%
96.85%
64.37%
Sparse Combo Net
130K
99.04%
96.94%
64.75%
(4)
(3)
[96.65%]
[64.04]
Lipschitz (Erichson et al., 2021)
99.2%
95.9%
134K
99.4%
96.3%
64.2%
(10)
(10)
[99.0%]
[95.6%]
Dense IndRNN
83K
99.48 %
97.2%
(Li et al.,
2020)
CKConv
(Romero et al.. 2021)
IM
99.32%
98.54%
63.74%
Trellis
(Bai al.,
8M 99.2%
98.13%
73.42%
2019)
Table I: Published benchmarks for sequential MNIST, permuted MNIST, and sequential CIFARIO best test accuracy. Networks grouped into three categories: basclines, best performing RNNs with claimed stability guarantee*, and networks achieving the current SOTA on each task. Within each grouping, networks are ordered by number of trainable parameters (for CIFAR10 if differed across tasks). Our network is highlighted. Where possible, we include information on repeatability. *For stability guarantees in machine learning, Section 2.3.
decided test significantly sparser subnetworks 16 32 Sparse Combo Net. We trained net- works with sparsity levels of 5%, 3.3%, and 1%, as well as 10% for baseline comparison (Figure S3A). 3.3% sparsity level produced the best results, leading to our SOTA performance on per- muted SeqMNIST. For component RNN size of just 32 units, this sparsity level is quite small, with only one or two directional connections per neuron on average (Figure S1).
Performance During Training, Permuted seqMNIST
Valiontion Accuracy
90
80
70
26.5% Sparsity
10.0% Sparsity
60
20
60
100
120
140
Epochs
Figure 4: Permuted SeqMNIST performance over the course of training for two 11 x 32 Sparse Combo Nets with different sparsity levels.
The 16 x 32 Sparse Combo Net with 3.3% density also performed SOTA for stable RNNs on se- quential CIFAR10. Not only the test accuracy the highest to date for provably stable RNN, was higher than the million parameter CKConv network, which holds the record for permuted SeqMNIST accuracy (Table 1). Our network has 130 thousand trainable parameters by comparison.

---

Under review as a conference paper at ICLR 2022
As sparsity had such a positive effect on task performance, did additional analyses to better derstand why. We found that decreasing the magnitude of non-zero elements while holding sparsity level constant decreased task performance (Figure S3B), suggesting that the effect is driven in part by the fact that sparsity enables higher magnitude non-zero elements while still maintaining stability.
It is also worth noting that upon investigation of the subnetwork weight matrices across these trials, the sparser networks had substantially lower maximum eigenvalue of W I, suggesting that stronger stability can actually correlate with improved performance on sequential tasks. This could be due to a mechanism such as that described in (Radhakrishnana et al., 2020).
3.2.2 EXPERIMENTS WITH NETWORK SIZE
Understanding the effect of size on network performance is important to practical application of these architectures. For both Sparse Combo Net and SVD) Combo Net, increasing the number of subnetworks while holding other settings constant (including fixing the size of each subnetwork at 32 units) was able to increase network test accuracy on permuted seqMNIST a point (Figure 5).
The greatest performance jump happened when increasing from one module (37.1% % Sparse Combo Net, 61.8% SVD Combo Net) two modules (89.1% 1% Sparse Combo Net, 92.9% SVD Combo Net). After that the performance increased steadily with number of modules until saturating at ~ 97% for Sparse Combo Net and 95% for SVD Combo Net. Because the internal subnetwork weights are not trained in Sparse Combo Net, it is unsuprising that its performance was substantially worse at the smallest sizes. However Sparse Combo Net surpasses SVD Combo Net by the x network size, which contains a modest 384 total units.
Rinal Validation Accuracy
Performance of Subnetworks, Permuted SeQMNIST
98
96.8
96.7
Sparse Combo Net
96.0
96
SVD Combo Net
94.4
93.6
90
25
Number Subnebworks
Figure 5: Permuted SeqMNIST performance plotted against the number of subnetworks. Each sub- network has 32 neurons. Results are shown for both Sparse Combo Net and SVD Combo Net.
For the Sparse Combo Net specifically we did additional experiments, both replicating the observed effect of network size using subnetworks with 16 units each (Figure S2A), and evaluating how task performance varies with modularity of a network fixed have 352 total units (Figure S2B). In the modularity experiment we observed an inverse U shape, with poor performance of a 1 x 352 net and an 88 net, and best performance from 44 8 net. Note that this experiment compared similar sparsity levels across the different subnetwork sizes. In practice we can achieve better performance with larger subnetworks by leveraging sparsity
way not possible smaller subnetworks.
The use of sparsity in subnetworks to improve performance suggests another interesting direction that could enable better scalability of total network size enforcing sparsity in the linear feedback weight matrix (L). We performed pilot testing of this idea in a 24 x 32 Sparse Combo Net, vary- ing the number of feedback connections that were fixed at while all other settings remained as they were in our prior experiments (Table S7). We obtained 65.14% test accuracy sequential CIFAR 10 using a network with only 50% the possible feedback connections non-zero, surpassing previous best and using smaller number of epochs to do so.
3.2.3 REPEATABILITY AND CONTROLS
Because the Sparse Combo Net does not have the connections within its subnetworks trained, net- work performance could be particularly susceptible to random initialization. Thus we ran repeatabil- ity studies on both permuted sequential MNIST and sequential CIFAR10 using our best network set- tings and an extened training period. Across four permuted seqMNIST trials with the same network
8

---

Under review as a conference paper at ICLR 2022
settings (16 x 32 with subnetwork sparsity level of 3.3%), best test accuracy always fell between 96.65% and 96.94%, a range much smaller than the differences seen with changing sparsity settings and network size. Three of the four trials showed best test accuracy 96.88%. despite some vari- ability in early training performance (Figure S4). Similarly, across three seqCIFAR 10 trials with the same network settings, best test accuracy always fell between 64.04% and 65% (Figure S5). We also ran 9 sequential CIFAR 10 trials using shorter training period to demonstrate reproducibility of the early training curve (Figure S6).
As control study, we tested how sensitive both Sparse Combo Net and SVD Combo Net were to the stabilization condition on the interconnection matrix (L in Theorem 6). To do this we constrained the hidden-to- to--idden weights of the networks to satisfy their respective contraction conditions. How- ever, instead of constraining L in a way that ensures contraction of the overall system, we set I. B and did not constrain B during training. In both cases we found that this hurt network performance on the permuted SeqMNIST task, demonstrating the utility of the contraction condition. For the 24 x 32 Sparse Combo Network we saw a decrease from 96.7% test accuracy to 47.0% test ac- curacy. For the 24 x 32 SVD Combo Network we saw a decrease from 94.9% to 94.56%. The disparity in performance decrease makes sense when considering that the hidden-to-hiee to-hidde weights of SVD Combo Network are trainable, while those of the Sparse Combo Network are not. Whatever instabilities are introduced by the lack of constraint the inter-subnetwork connections cannot be adequately compensated for in the Sparse Combo Network.
4 DISCUSSION
Most work on stability of task-trained RNNs has focused on single RNNs. Here we leverage tools from nonlinear control theory to derive novel single-RNN stability conditions which enable the re- cursive construction of stable assemblies of RNNS. In particular we show that certain stability condi- tions for individual RNNs allow for a simple parameterization of connections between these RNNs that automatically preserves stability during training. We then show that these modular 'network of networks' repeatedly perform better than existing stable RNNs on key sequence classification tasks such as permuted SeqMNIST and seqCIFAR10. 10. We also provide control studies that show the stabilizing parameterization of connections between subnetwork RNNs is important for high performance.
There are numerous future directions enabled by this work. For example, Theorem 7 suggests that a less restrictive contraction condition W in terms of the eigenvalues of the symmetric part is pos- sible and desirable. Furthermore, the beneficial impact of sparsity on training these stable models suggests potential avenue for additional experimental work in particular adding a regulariz- ing sparsity term during training. As 'network of network' approaches are becoming increasingly popular, our methodology is relevant variety of task types, including reinforcement learning applications. Moreover, because sparsity and modularity have important theoretical and empirical implications in neuroscience (Kozachkov et al., 2020; Slotine & Liu, 2012) we expect to be able to apply the present work to those topics as well.
Ultimately, our work represents a step forward in understanding the stability properties of recur- rent neural networks. Stability is fundamental property of dynamical systems, and is inextricably linked to concepts such as generalization, control, predictability, and robustness. Therefore, as systems trained with deep learning become more modular, complex, and integrated into our lives, understanding the conditions under which these systems are stable will become increasingly im- portant. Furthermore, it will be necessary to identify training techniques that can reliably generate provably stable RNNs with minimal performance loss. We provide a proof-of-concee of the use of combination networks to achieve this goal, and show that sparsity enables additional performance gains.
Acknowledgements This work benefited from stimulating discussions with Michael Happ and Quang-Cuong Pham.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference Machine Learning, pp. 1120-1128. PMLR, 2016.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. ICLR, 2019.
Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning, volume 1. MIT press Mas- sachusetts, USA:, 2017.
Michael Buehner and Peter Young. A tighter bound for the echo state property. IEEE Transactions on Neural Networks, 17(3):820-224, 2006.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricrnn: dynamical system view on recurrent neural networks. arXiv preprint arXiv:1902.09689, 2019.
Yimian Dai, Stefan Oehmcke, Fabian Gieseke, Yiquan Wu, and Kobus Bararrd Attention acti- vation. arXiv preprint arXiv:2007.0772292 2020.
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. ICLR, 2021.
John Gerhart and Mare Kirschner. The theory of facilitated variation. Proceedings of the National Academy of Sciences, 104(1):8582-855) 8589, 2007. doi: 1107://pas.00000351100
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspiree artificial intelligence. Neuron, 95(2):245-253 258, 2017.
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note. 2001.
Leo Kozachkov, Mikael Lundqvist, Jean-Jacques Slotine, and Earl K Miller. Achieving stable dy- namics in neural circuits. PLoS computational biology, 16(8)e1007659, 2020.
Dmitry Krotov and John Hopfield. Large associative memory problem in neurobiology and machine learning. arXiv preprint arXiv:2008.060 06996, 2020.
Quoc V Le, Navdeep Jaitly, and Geoffrey Hinton. A simple way initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 1504.00440000 504.00941, 2015.
1.XXvv1504.0000
Mario Sezcano-Casadee and David Martinez-Rubio. Cheap orthogonal constraints in neural net- works: A simple parametrization of the orthogonal and unitary group. In International Confer- ence on Machine Learning, pp. 3794-3803. PMLR, 2019.
Shuai Li, Chris Cook, and Yanbo Gao. Deep independently recurrent neural network (indran). arXiv preprint arXiv:1910.0625// 2020.
Winfried Lohmiller and Jean-Jacques E Slotine. On contraction analysis for non-linear systems. Automatica, 34(6):683 6966 1998.
Kiyotoshi Matsuoka. Stability conditions for nonlinear continuous neural networks with asymmetric connection weights. Neural networks, 5(3):495-500, 1992.
John Miller and Moritz Hardt. Stable recurrent models. arXiv preprint arXiv: 10369, 2018.
Kenneth D Miller and Francesco Fumarola. Mathematical equivalence of common forms of firing rate models of neural networks. Neural computation, 24(1):25-31,201 2012.
Kumpati S. Narendra and Robert Shorten. Hurwitz stability of metzler matrices. IEEE Transactions On Automatic Control, 55(6): 1484-1477, 2010.
10

---

Under review as a conference paper at ICLR 2022
Merav Parter, Nadav Kashtan, and Uri Alon. Facilitated variation: How evolution learns from past environments to generalize to new environments. PLOS Computational Biology, 4(11), 2008.
Adityanarayanan Radhakrishnana, Mikhail Belkin, and Caroline Uhler. Overparameterized neural networks implement associative memory. PNAS, 117:27162-271 2020.
Hubert Ramsauer, Bernhard Schäft, Johannes Lehner, Philipp SeidI, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint Ttl::0088.022177 2020.
Max Revay and Ian Manchester. Contracting implicit recurrent neural networks: Stable models with improved trainability. In Learning for Dynamics and Control, pp. 393-403. PMLR, 2020.
Max Revay, Ruigang Wang, and Ian R Manchester. A convex parameterization of robust recurrent neural networks. IEEE Control Systems Letters, 5(4):1363-1368 2020a.
Max Revay, Ruigang Wang, and Ian R Manchester. Lipschitz bounded equilibrium networks. arXiv e-prints, pp. arXiv-2010, 2020b.
Max Revay, Ruigang Wang. and Ian Manchester. Recurrent equilibrium networks: Unconstrained learning of stable and robust dynamical models. arXiv preprint 1040::2204 2021.
David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoom. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.026111v,, 2021.
Ueli Rutishauser, Rodney Douglas, and Jean-Jacques Slotine. Collective stability of networks of winner-take-all circuits. Neural computation, 23(3):735-773, 2011.
Ueli Rutishauser, Jean-Jacques Slotine, and Rodney Douglas. Computation in dynamically bounded asymmetric systems. PLoS Comput Biol, 11(I)=1100403,, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-49,, 2016.
Herbert A. Simon. The architecture of complexity. Proceedings of the American Philosophical Society, 106(6):467-482, 1962. ISSN 0003-049X.
Jean-Jacques Slotine. Modular stability tools for distributed computation and control. Int. Adap- rive Control and Signal Processing, 17(6), 2003.
Jean-Jacques Slotine and Yang-Yu Liu. The missing link. Nature Physics, 8(7):512-513, 2012.
Jean-Jacques Slotine and Winfried Lohmiller. Modularity, evolution, and the binding problem: view from stability theory. Neural Networks, 14(2):137-145, 2001.
Jean-Jacques E Slotine and Weiping Li. Applied Nonlinear Control. Prentice-Hall, 1991.
Nicolas Tabareau and Jean-Jacques Slotine. nlin/0601011, 2006.
Notes on contraction theory. arXiv preprint
Trieu H. Trinh, Andrew M. Dai, Minh-Thang Luong, and Quoc Le. Learning longer-term depen- dencies in mnns with auxiliary losses. arXiv preprint 1r3fv:1803.00144433..2 2018.
Eric W. Weisstein. Positive definite matrix. URL
wow.frl..wwwwwrrmmmmmmmommm wolfram. com/
Posi tiveee ioeffiniteeee initeMatriii html.
Guangyu R Yang and Manuel Molano-Mazon. Next-generation of recurrent neural network models for cognition, Apr 2021. URL assarxiv.com/ wwtpw//m//w......mm
Guangyu Robert Yang. Igor Ganichev, Xiao-Jing Wang, Jonathan Shlens, and David Sussillo. A dataset and architecture for visual reasoning with working memory. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 714-731, 2018.
11

---

Under review as a conference paper at ICLR 2022
APPENDIX
A1 CONTRACTION MATH
It can be shown that the non-autonomous system
x == ==f(x,t)
contracting if there exists metric M(x, t) = (x (xx > 0 such that uniformly
M+ MJ + J M 3 -M
where J of and 3 0. For details the main reference (Lohmiller & Slotine, 1998). Similarly, non-autonomous discrete-time system
X = f(x(,!)
contracting if
J J'M MM, 1 -8M,
A A1.1
FEEDBACK AND HIERARCHICAL COMBINATIONS
Consider two systems, independently contracting in constant metrics M, and M2, which are com- bined feedback:
x f(x, + By
(Feedback Combination)
y g(y, () Gx
If the following relationship between B, G, M1, and M2 is satisfied:
-
then the combined system is contracting as well. This may be seen as a special case of the feedback combination derived in (Tabareau Slotine, 2006). The situation is even simpler for hicrarchical combinations. Consider again two systems, independently contracting in some metrics, which are combined in hierarchy:
x f(x,t) t) ggg(y: f) + h(x
(Hierarchical Combination)
where h(x,t) is a function with bounded Jacobian. Then this combined system is contracting in diagonal metric, shown in (Lohmiller & Slotine, 1998). By recursion, this extends to hierarchies of arbitrary depth.
A2 TWO DIFFERENT RNNS
Note that in neuroscience, the variable in equation (1) is typically thought of vector of neural membrane potentials. It was shown in (Miller & Fumarola, 2012) that the RNN (1) is equivalent via an affine transformation another commonly used RNN model,
r + の Wy + b(t))
(4)
where the variable y is interpreted as a vector of firing rates, rather than membrane potentials. The two models are related by the transformation x Wy + b, which yields
w(-y O (Wy + b)) + -x Wo(x)
12

---

Under review as a conference paper at ICLR 2022
where v = b + rb. Thus b is low-pass filtered version of v (or conversely, may be viewed as first order prediction of b) and the contraction properties of the system are unaffected by the affine transformation. Note that the above equivalence holds even in the case where W is not invertible. In this case, the two models are proven to be equivalent, provided that b(0) and y()) satisfy certain conditions-which are always possible to satisfy (Miller & Fumarola, 2012). Therefore, any contraction condition derived for the z (or y) system automatically implies contraction of the other system. We exploit this freedom freely throughout the paper.
A3 SPARSE COMBO NET SUPPLEMENTARY MATERIALS
All networks described in the main text were trained using a single GPU on Google Colab. An exported Colab notebook with the code replicate all experiments is provided in the supplementary attachment.
To obtain the metric necessary for training the linear connections, scipy-integrate.quas was used with default settings to solve for M in the equation -I = MW + WW as described in the main text. This is done by integrating e 'Qe Wedl from 0 to oc. For efficiency reasons, and due to the guaranteed existence of a diagonal metric in the case of Theorem 1, integration was only performed to solve for the diagonal elements of M. Therefore check was added prior to training to confirm that the initialized network indeed satisfied Theorem 1 with metric M. However, it was never triggered by our initialization method.
Initial training hyperparameter tuning was done primarily with 10 x 16 combination networks on the permuted seqMNIST task, starting with settings based on existing literature on this task, and verifying promising settings using a 15 x 16 network. Initialization settings were held the same throughout, matching what was later done for the size comparison trials (described below). The results of all of the attempted trials are reported in Table $1. Once hyperparameters were decided upon, the trials reported on in the main text began.
To report on the number of trainable parameters, we used the following formula:
22-M 2
+i
+
Where n is the total number of units in the M x C' combination network, o is the total number of output nodes for the task, and i is the total number of input nodes for the task. Thus for the 16 x 32 networks highlighted here, we have 129034 trainable parameters for the MNIST tasks, and 130058 trainable parameters for sequential CIFAR1O.
Note that the naive estimate for the number of trainable parameters would be n² + corresponding to the number of weights in L, the number of weights in the feedforward linear input layer, the number of weights in the feedforward linear output layer, and the bias terms for the input and output layers, respectively. However, because of the combination property constraints on L, only the lower triangular portion of a block off off-diagonal matrix is actually trained, and L is then defined in terms of this matrix and the metric M. Thus we subtract M C2 to remove the block diagonal portions corresponding to nonlinear RNN components, and then divide by 2 to obtain only the lower half.
After training was completed, we inspected the state of all networks described in the main text, pulling both the nonlinear (W) and linear (L) weight matrices from both initialization time and the final model. For W. we confirmed it did not change over training, and inspected the max real part of the eigenvalues of [W] in accordance with Theorem 1. The densest tested matrices tended to have Amaz ([)) > 0.9, while the sparsest ones tended to have Amar ([W)) < 0.1. For L, we checked the maximum element and the maximum singular value before and after training. In general, both went up over the course of training. but by modest amount.
A3.1 NETWORK SIZE AND MODULARITY COMPARISON
For the size comparison trials (Figure 5A), the nonlinear RNN weights were set by drawing uni- formly from between -0.4 and 0.4 with 40% density using scipy.sparse. random, and then zeroing out the diagonal entries. These settings were chosen because they resulted in ~ 1% of 16 by 16
13

---

Under review as a conference paper at ICLR 2022
Sparse 32 Unit Component RNNs from Top Performing Network
Figure S1: Weight matrices for each of the 32 unit non- linear component RNNs that were used in the best perform- ing 16 x 32 network on per- muted sequential MNIST.
weight matrices meeting the Theorem condition. During initialization only the matrices meeting this condition were kept, finishing when the desired number of component RNNs had been producing a block diagonal W like pictured in Figure 3. This same initialization process was used throughout our experiments.
First, we held static the number of units and initialization settings for each component RNN, and tested the effect of changing the number of components in the combination network. 1, 5, 10, 15, 20, 22, 25, and 30 component RNNs were tested in this experiment (Table S2). Increasing the number of components initially lead to great improvements in test accuracy, but had diminishing returns test accuracy consistently hit ~ 93% with large enough number of modules, but neither loss nor accuracy showed meaningful improvement past the 22 16 network (Figure S2A). Interest- ingly, early training loss and accuracy became substantially worse once the number of components increased past a certain point, falling from 70% to 43% epoch test accuracy between the 22 x 16 and 30 x 16 networks.
To better understand how the modularity of the combination networks affects performance, the next experiment held the number of total units constant 352, selected due the prior success of the 22 x 16 network, and tested different allocations of these units amongst component RNNs. Thus x 352, 11 32, 44 8, and 88 networks were trained compare against the 22 16 (Table S3). Increasing the modularity improved performance to a point, with the 44 x 8 network resulting final test accuracy of 94.44%, while conversely the 11 x 32 resulted decreased test accuracy (Figure S2B). However, the 88 x 4 network was unable to learn, and a 352 x 1 network would theoretically just be scaled linear anti-symmetric network.
Because larger networks require different sparsity settings meet the Theorem 1 condition, these were not held constant between trials the modularity comparison experiment (Figure 5B), but rather selected in the same way between trials looking for settings that keep density and scalar balanced and result in 1% of the matrices meeting the condition. The scalar was applied after sampling non-zero entries from uniform distribution between -1 and 1. The resulting settings were 7.5% density and 0.077 scalar for 352 unit component RNN, 26.5% density and 0.27 scalar for 32 unit component RNN, 60% density and 0.7 scalar for unit component RNN, and 100% density and 1.0 scalar for 4 unit component RNN.
14

---

Under review as a conference paper at ICLR 2022
A)
Performance vs Network Size
100
Epoch 1
9501
92 90
98
Permuted segMNIST Test Accuracy
90
Epoch 150
B5 82
38
5
10
15
20
25
30
Number Component 16-unit RNNs
B)
Performance vs Network Modularity
100
93.01
94:44
Epoch
Permuled seqNNIST Test Accuracy
Epoch 150
40
30
20
0.09
10
20
40
60
80
Number Component RNNs for the 352 Total Units
Figure S2: Performance Sparse Combo Nets on the Permuted seqMNIST task by combination network size. We test the effects final and first epoch test accuracy of both total network size and network modularity. The former is assessed by varying the number of subnetworks while each subnetwork is fixed 16 units (A), and the latter by varying the distribution of units across different numbers of subnetworks with the total sum units in the network fixed at 352 (B). Note that these experiments were prior optimizing the sparsity initialization settings. Experiments on total network size were later repeated with the final sparsity settings (Figure 5). The results of both the size experiments are consistent.
A3.2 SPARSITY SETTINGS COMPARISON
Density and scalar settings for the component nonlinear RNNs were initially chosen for each net- work size using the percentage of random networks that met the Theorem I condition. For scalar a component network would have non-zero entries sampled uniformly between -s and
When we began experimenting with sparsity in the initialization, we split the previously described scalar setting into two different scalars one applied before a random matrix was checked against the Theorem condition, and one applied after a matrix was selected. Of course the latter must be < 1 to guarantee stability preserved. The scalar was separated out after we noticed that at 5% density, random 32 by 32 weight matrices met the condition roughly 1% of the time whether the scalar was or 100000 condition even if multiplied by factor of 10000. Therefore
85% of sampled matrices using scalar 10 would continue to meet
wanted mechanism that could bias selection towards matrices that are stable due to their sparsity and not due to magnitude constraints, while still keeping the elements reasonable size for training purposes.
15

---

Under review as a conference paper at ICLR 2022
Ultimately, both sparsity and magnitude had clear effect performance (Figure S3). Increases in both had positive correlation with accuracy and loss through most parameters tested, seemingly driven by the increase in non-zero element magnitude enabled by sparsity. Best test accuracy overall was 99%,,, which was obtained by both a 16 x 32 network with 5% density and entries between -5 and 5, and 16 32 network with 3.3% density and entries between -6 and 6. The latter also achieved the best epoch test accuracy observed of 86.7%%
A)
Best Network Sparsity Level Performance
Distribution Magnitude
Networks Training
Performance 3.3% Density 16x32 Networks Over Training
97
97
denaitty by scaler
3.0 scalar
Pormubed saqMMSS Test Accurary
06
scaarr 10% donotry. 1.0 scalor
06
95
Verm seatthis
93-
93
92
92-
91
0
20
40
60
80
1000
120
140
80
100
120
140
Epoch
Epoch
Figure S3: Permuted SeqMNIST performance by component RNN initialization settings. Test ac- curacy is plotted over the course of training for four 16 32 networks with different density levels and entry magnitudes (A), highlighting the role of sparsity in network performance. Test accuracy then plotted over the course of training for two 3.3% density 16 32 networks with different entry magnitudes (B), to demonstrate the role of the scalar. When the magnitude becomes too high however, performance is out of view of the current axis limits.
A3.3 REPEATABILITT
To further improve performance once network settings were explored on permuted seqMNIST, an extended training run was tested on the best performing option. Settings were kept the same as above using 3.3% density 16 x 32 network, except training just a single learning rate cut occurring after epoch 200 (exact number of epochs varied based on runtime limit). This experiment was repeated four times and resulted 96.99 96.94% best test accuracy (Figure S4).
for over 200 epochs, with
We performed a similar repetition task for the seqCIFARI0 task. To avoid hitting runtime limits, we performed three trials where each network was trained for 200 epochs with learning rate cuts at epochs 140 and 190. All networks exceeded 64% test accuracy, with the best performing network achieving 64.75% (Figure S5). We also trained a larger number of networks over smaller number of epochs (Figure S6), as well as explored additional hyperparameter tuning on this task (Table S6).
16

---

Under review as a conference paper at ICLR 2022
A)
Replicabllity Analysis Constant Settings Performance of 16x32 Networks with Extended Training
Permuted seqMIIST Test Accuracy
90
94
92
90
88
50
100
150
200
250
Epoch
B)
Replicability Analysis Constant Settings
Performance of 16x32 Networks with Extended Training
Figure S4: Permuted seqM- NIST performance on re- peated trials. Four differ- 16 x 32 networks with 3.3% density and entries be- tween and were trained. (A) depicts test accuracy for each the networks over the course of training. (B) depicts the training loss for the same networks.
Permueze ScoMNIST Training Loss
0.35
0.30
0.20
0.15
0.05
50
100
150
200
250
Epoch
A)
Replicability Analysis Constant Settings
Replicability Analysis Settings
Performance of Networks Extended Training
Performance Networks Training
70
60
1.8
50
R09CIAIR10 Training
3590113110 test
1.4
40
30
1.0
20
0.8
100
150
150
200
Figure S5: SeqCIFARIO performance repeated trials. Three different 16 x 32 networks with 3.3% density and entries between -6 and 6 were trained for 200 epochs, with learning rate divided by 10 after epochs 140 and 190. (A) depicts test accuracy for each of the networks the course of training. (B) depicts the training loss for the same networks.
A3.4 TABLES OF RESULTS BY TRIAL
Table SI shows all trials run on permuted sequential MNIST before beginning the more systematic experiments reported in the main text, presented in chronological order. Notably, networks did not require an extensive hyperparameter tuning process.
Tables S2 and S3 report additional details the size and modularity experiments (Figure S2).
Tables S4 and S5 report additional details on the sparsity experiments described in the main text (Figure 4), including results of all relevant trials, as some were left out of the main text for brevity.
Table S6 reports the results of all trials of different hyperparameters on the sequential CIFAR10 task, in chronological order.
17

---

Under review as a conference paper at ICLR 2022
Replicability Analysis Constant Settings
Performance Networks
Training
SEOCIFAR1O Test Accurary
55
50
80
100
120
140
Epoch
Figure S6: seqCIFARIO performance repeated trials with shorter training (done complete more trials). Nine different 16 x 32 networks with 3.3% density and entries between -6 and 6 were set up to train for 150 epochs, with learning rate divided by after epochs 90 and 140. Most of these networks hit runtime limit before completing, however they all got through at least 100 epochs and all had test accuracy exceed 61%. This figure depicts test accuracy for each of the networks over the course of training.
Finally, Table S7 reports the results of our pilot trial on introducing sparsity into the linear feedback connection matrix.
Final Test
Adam WD
Initial LR
Size
Epochs
LR Schedule
Acc.
10x 16
150
5c-5
5e-3
0.1 after 91
84%
10x 16
150
le-5
le-2
0.1 after 50,100
85%
15x 16
150
2e-4
5e-3
0.1 after 50,100
84%
10x 16
150
2e-4
le-2
0.5 every 10
81%
10x 16
200
2e-4
le-2
0.5 after 10 then every 30
81%
0.75 after 10,20,60, 100 then
10x 16
171
5e-5
1e-2
84%
every 15
15x
179
1e-5
1e-3
0.1 after 100,150
90%
Table S1: Training hyperparameter tuning trials, presented in chronological order. indicates that training was cut short by the 24 hour Colab runtime limit. LR Schedule describes the scalar the learning rate was multiplied by, and at what epochs. The best performing network is highlighted, and represents the training settings we used throughout most of the main text.
18

---

Under review as a conference paper at ICLR 2022
Final Test
Epoch
Final Train
Size
Test
Final
Epoch
Final
Acc.
Acc.
Loss
Size
Test
Test
Train
1 X 16
38.69%
24.61%
1.7005
Acc.
Acc.
Loss
3 x 16
70.56%
40.47%
0.9033
1x352
40.17%
26.97%
1.662
x 16
77.86%
47.99%
0.7104
11 x1x
89. 12%
61.29%
0.3781
10x 16
85.82%
61.38%
0.4736
22 16
93.01%
70.11%
0,2073
15x 16
90.28%
69.09%
0.3156
44 x 8
44%
25.78 75%
0.1500
20 16
92.26%
71.72%
0.2392
88 x 4
10.99%
10.99%
2E+35
22x 16
93.01% %
70.11%
0.2073
25 16
92.99%
61.81%
0.2017
Table S3: Results for different distributions of 352 total units across combination net- work. This number was chosen based prior 22 x 16 network performance. For each component RNN size tested, the same proce- dure was used to select appropriate density and scalar settings.
30x16
93.16%
43.21%
0.1991
Table S2: Results for combination networks containing different numbers of component 16-unit RNNs. Training hyperparameters and network initialization settings were kept the same across all trials.
Final Test
Epoch
Final Train
Size
Density
Scalar
1 Test
Acc.
Acc.
Loss
11 32
26.5%
0.27
89. 12%
61.29%
0.3781
11 x32
10%
1.0
94.86%
70.67%
0.1278
22 16
40%
0.4
93.01 %
70.11%
0.2073
22 16
20%
1.0
95.27%
76.58%
0.0924
22 16
10%
1.0
94.26%
71.53%
0.1425
44 x 8
60%
0.7
94.44%
25.78%
0.1500
44 x 50%
1.0
95.05%
30.52%
0.1180
Table S4: Results for different initialization settings varying sparsity and magnitude of the compo-
nent RNNs for different network sizes.
Pre- select Scalar
Post- select Scalar
Final Test
Epoch
Final Train Loss
Density
1 Test
Acc.
Acc.
10%
1.0
1.0
95.87%
73.67%
0.074
5%
10.0
0.1
95.11%
73.10%
0.1311
5%
10.0
0.2
96. 15%
82.50%
0.0051
5%
10.0
0.5
96.69%
75.76%
0.0001
5%
6.0
1.0
96.41%
21.55%
3.3E-5
5%
7.5
1.0
16.75%
11.39%
3068967
3.3%
30.0
0.1
96.24%
83.89%
0.0005
3.3%
30.0
0.2
96.54%
86.79%
4E-5
1%
10.0
1.0
96.04%
81.2%
0.0001
Table S5: Further optimizing the sparsity settings for high performance using a 16 x2 network. The final scalar the product of the pre-selection and post-selection scalars. Note that the 5% density and 7.5 scalar network was killed after 18 epochs due to exploding gradient.
19

---

Under review as a conference paper at ICLR 2022
Pre- select
Post- select Scalar
Best
Adam WD
Initial LR
Density
Epochs
LR Schedule
Test
Scalar
Acc.
3.3%
30
0.2
150
le-5
le-3
0.1 after 90,140
64.63%
3.3%
30
0.2
34
le-5
5e-3
after 90,140
35.42%
5%
6
1
150
le-5
le-3
0.1 after 90,140
60.9%
5%
10
0.5
150
le-5
le-4
0.1 after 90,140
54.86%
3.3%
30
0.2
150
le-5
5e-4
after 90,148
61.83%
3.3%
30
0.2
200
1c-6
2e-3
0.1 after 140,190
62.31%
3.3%
30
0.2
186*
lc-5
le-3
0.1 after 140,190
64.75%
3.3%
30
0.2
132
le-6
le-3
after 140, 140
62.31%
5%
10
0.5
195
le-5
1e-3
0.1 after 140, 490
64.68%
Table S6: Additional hyperparameter tuning for the CIFARIO task, presented in chronological order. indicates that training was cut short by the 24 hour Colab runtime limit, or in the case of high learning rate was killed intentionally due to exploding gradient. LR Schedule describes the scalar the learning rate was multiplied by, and at what epochs. The best performing network is highlighted.
Best
Test
Feedback Density
Best Overall
Size
Epochs
Acc. Through
Test Acc.
85 Epochs
x2
100%
86
52.7%
52.7%
22
75%
88
56.49%
56.48%
24
66.6%
89
58.844 38.84%
58.84%
22
50%
124
65.14%
58.01%
24
33.3%
129
61.86%
56.05%
22
25%
92
54.26%
50.54%
24
0%
130
39.8%
38.38%
16x
100%
150
64.63%
55.82%
Table S7: Results from pilot testing the sparsity of negative feedback connections in 24 x 32 Sparse Combo Net. Feedback Density refers to the percentage of possible subnetwork pairings that were trained in negative feedback, while the remaining inter-network connections were held at O. All networks were trained with the same 150 epoch training paradigm as mentioned in the main text, but were stopped after hitting 24 hour runtime limit. Decreasing Feedback Density is promising path towards further improving performance as the size of Sparse Combo Nets is scaled.
20

---

Under review as a conference paper at ICLR 2022
A4
PROOFS FOR MAIN RESULTS
A4.1 PROOF OF THEOREM
Our first theorem is motivated by the observation that if the y-system is to be interpreted as vector of firing rates, it must stay positive for all time. For a linear, time-invariant system with positive states, diagonal stability is equivalent to stability. Therefore a natural question is if diagonal stability of linearized y-system implies anything about stability of the nonlinear system. More formally, given an excitatory neural network (i.e Vij, Wi 2 0), if the linear system
x -x + g Wx
is stable, then there exists a positive diagonal matrix P such that:
P - 1) + (oW P)
The following theorem shows that the nonlinear system (1) is indeed contracting in metric P, and extends this result to more general W by considering only the magnitudes of the weights. Theorem 1. Let [W] denote the matrix formed by taking the element-wis absolute value of W. If there exists positive, diagonal P such that:
W| +) - PP <
then equation 1 is contracting metric P. Moreover, if Wa S then [W] may be set zero to reduce conservatism.
This condition particularly straightforward in the common special case where the network does not have any self weights, with the leak term driving stability. While it can be applied to a more general W, the condition will of course not be met if the network was relying on highly negative values on the diagonal of W for linear stability. As demonstrated by counterexample in the proof of Theorem I, it can impossible to use the same metric for the nonlinear RNN such cases.
Theorem I allows many weight matrices with low magnitudes or generally sparse structure to be verified as contracting in the nonlinear system equation I, by simply checking a linear stability condition (as linear stability is equivalent to diagonal stability for Metzler matrices too (Narendra & Shorten, 2010)).
Beyond verifying contraction, Theorem actually provides metric, with little need for additional computation. Not only is of inherent interest that the same metric can be shared across systems in this case, it is also of use in machine learning applications, where stability certificates are be- coming increasingly necessary. Critically, it feasible to enforce the condition during training via L2 regularization on W. More generally, there are variety of systems of interest that meet this condition but do not meet the well-known maximum singular value condition, including those with hierarchical structure.
Proof. Consider the differential, quadratic Lyapunov function:
= 6x P8x
where P > O is diagonal. The time derivative of V is:
À P8x 28x PJöx 28x" "Pix + 28x PWD1x
where is diagonal matrix such that D the right follows:
déh
2 0. We can upper bound the quadratic form on de
8x PWDdx
P.W P.W.,DDorrD Dōör,o S
Σ P.W.sa
Σ (
S PP [x]
If WW; S O, the term P.W,,, Sn,1² contributes non-positively to the overall sum, and can therefore be set to zero without disrupting the inequality. Now using the fact that P is positive and diagonal, and therefore dx P8x 18x T 5x|, we upper bound V as:
21

---

Under review as a conference paper at ICLR 2022
V $ |6x (-2P + P/W
+ W = 6x (P W - D) + ((W) - 1)))]
where | - 11 and W| - 0 if W. S and W|a = Wa if W > O. This completes
the proof. Note that W - I is Metzler, and therefore will be Hurwitz stable if and only if P exists (Narendra Shorten, 2010).
It is also worth noting that highly negative diagonal values in W will prevent the same metric P from being used for the nonlinear system. Therefore the method used in this proof cannot feasibly be adapted to further relax the treatment of the diagonal part of W. The intuitive reason behind this that in the symmetric part the Jacobian, SWD+Dw"P
P. the diagonal self weights will also be scaled down by small D, while the leak portion - P remains untouched by D. Now actually demonstrate a counterexample, presenting 2 x symmetric Metzler matrix W that is contracting in the identity in the linear system, but cannot be contracting in the identity in the nonlinear system equation 1:
9
2.5
W
2.5
0
To see that it is not possible for the more general nonlinear system with these weights be contract- ing in the identity, take D
0 0
Now
1
-1
1.25
(WD) syym
1.25
-1
which has a positive eigenvalue of 1
A4.2 PROOF OF THEOREM 2
While regularization may push networks towards satisfying Theorem strictly enforcing the dition during optimization is not straightforward. This motivates the rest of our theorems, which derive contraction results for specially structured weight matrices. Unlike Theorem these results have direct parameterizations which can easily be plugged into modern optimization libraries.
Theorem 2. W = WT and g W < I, then (I) is contracting.
When W is symmetric, (1) may be seen as continuous-tine -time Hopfield network. Continuous-time Hopfield networks with symmetric weights were recently shown to be closely related to Transformer architectures (Krotov & Hopfield, 2020; Ramsauer et al., 2020). Specifically, the dot-product atten- tion rule may be seen as discretization of the continuous-tin time Hopfield network with softmax activation function (Krotov & Hopfield, 2020). Our results here provide a simple sufficient (and above remark) condition for global exponential stability of given trajectory for the Hopfield network. In the case where the input into the network is constant, this trajectory is fixed point. Moreover, each trajectory associated with unique input is guaranteed to be unique. Finally, we note that our results are flexible with respect to activation functions so long as they sat- isfy the seste-rentriction condition. This flexibility may be useful when, for example, considering recent work showing that standard activation functions may be advantageously replaced by attention mechanisms (Dai et al., 2020).
nearly necessary,
Proof. We begin by writing W R P for some unknown R R and P = = 0. The approach of this proof is to show by construction that the condition g < implies the existence of an R and P such that the system is contracting in metric P. We consider the version of the RNN, which as discussed above is equivalent to the x version via an affine transformation.
Consider the contraction condition:
22 + MDW + w DM -3M
22

---

Under review as a conference paper at ICLR 2022
with 3 > 0. Substituting in the definitions of W and M, this condition becomes:
2P + PD(R P) + RR P)DP -BP
Simplifying the terms and collecting them all on one side, the above may be written as:
- 2)P + RDP + PDR 2PDP 3
via the Schur complement, the above term will be satisfied if:
(2 3)P RDP (2PDP) -PDR
- P2 RDR = - 3)
) 0
2
We continue by setting - RR with -² - a 272-35. so that above inequality satisfied. At this point, we have shown that if W can be written as:
W R - - RR
then (I) is contracting in metric M 7 RR. What remains to be shown is that if the condition:
9 gW - -
satisfied, then this implies the existence of an R. such that the above is true. To show that this indeed the case, assume that:
1
2
4-
Substituting in the definition of y, this is just the statement that:
2(2 8)
W 0 0
4g
Setting 3 2A >> 0, this yields:
(1- gW
Since W is orthogonal, we have the eigendecomposition:
1 V 4,2
I-W
I-W
- I-
4-2
where VTV = and a diagonal matrix containing the eigenvalues of W. Denote the symmetric rquare-root of this expression S:
= V
I I- AJV" =
472
Which implies that:
- W STS
422
We now define R in terms of S as follows:
R=
S +
7 27
Which means that:
1 I
1
1
I-W W (7R
I)(
I)
4-2
27
27
Expanding out the right side, we get:
1
W = R R
4-2
Subtracting + from both sides yields:
W = R - RRR
As desired.
23

---

Under review as a conference paper at ICLR 2022
A4.3 PROOF OF THEOREM 3
Theorem 3. If there exists positive diagonal matrices Pi and P2, as well as Q - Q² > 0 such
that
W = PPPPPP
then (I) contracting in metric M (PIQP:)-
Proof. Consider again a differential Lyapunov function:
V ox Mox
the time derivative is equal to:
V -V + 6x MWDdx
Substituting in the definitions of W and M, we get:
2V - 5x PT P2Dxx < -V
Therefore V converges exponentially to zero.
A4.4 PROOF THEOREM 4
Theorem 4. y - I is triangular and Hurwitz, then (1) is contracting in diagonal metric.
Note that in the case of a triangular weight matrix, the system (1) may be seen as a feedforward (i.e hierarchical) network. Therefore, this result follows from the combination properties of contracting systems, However, our proof provides a means of explicitly finding metric for this system.
Proof. Without loss generality, assume that W is lower triangular. This implies that Wij = i S Now consider the generalized Jacobian:
F= IWD+"
with I' diagonal and r, = el where to:
0. Because I is diagonal, the generalized Jacobian is equal
F= F=
Now note that:
(TWT )ag = e Wije Wije²
Where S j, we have Wi, = by assumption. Therefore, the only nonzero entries are where 2 j. This means that by making arbitrarily small, we can make rWr-1 approach a diagonal matrix with Wa along the diagonal. Therefore, if:
max gW 1 < 0
the nonlinear system is contracting. Since W is triangular, Wú are the eigenvalues of W, meaning that this condition equivalent g being Hurwitz.
A4.5 PROOF OF THEOREM 5
Theorem 5. If there exists positive diagonal matrix such that:
T PW 0
then (I) contracting in metric P.
24

---

Under review as a conference paper at ICLR 2022
Note that this is equivalent to the discrete-time diagonal stability condition developed in (Revay & Manchester, 2020), for a constant metric. Note also that when M = I, Theorem 5 is identical to checking the maximum singular value of W, previously established condition for stability of equation However a much larger set of weight matrices are found via the condition when M P instead.
Proof. Consider the generalized Jacobian:
F = P1/JJP 1/2 + 1t/ppp/////////
where D is diagonal matrix with D the generalized Jacobian we get:
Using the subadditivity of the matrix measure μ22 of dr."
de:
12 (F( S -1 n2 (2
Now using the fact that (2(() (-)) S || - ||2 we have:
Po FF 1 w////pp-///
S -1 -1+ l 1/2
Using the definition of the 2-norm, imposing the condition (2(F) < 0 may be written:
W'W - P
which completes the proof.
A4.6 PROOF THEOREM 6
Theorem 6 (Network of Networks). Consider collection of p subnetwork RNNs governed by equation 1. Assume that these RNNs each have and are independently contracting in metrics (M1,.... Mp}. Define the block matrices W = BlockDiag W....., W) and M = BlockDiag(Me I Mp). as well as the overall state vector XT (xf ...x.) x2) ). Then the following 'network of networks' is globally contracting in metric M:
weight matrices (W1...., p
- - W (x(x) + u(() L
(5)
L= B M
Where B is an arbitrary square matrix.
Proof. To see that this overall system is contracting in constant metric M, consider the correspond- ing differential Lyapunov equation (Lohmiller & Slotine, 1998):
Amaz (MM MJ +
J'
0
(6)
Amaz (2(3 M+ MWD).) + Aman - B"
M) 50
where J is the Jacobian equation 5 and D;: = ø' (xxx) is diagonal matrix containing the slopes of the activation functions. The last inequality follows from the original assumption that the individual subnetworks are contracting in metrics (M,...., M2 with rate B.
A4.7 PROOF OF THBOREM 7
Theorem 7. Let D be positive, diagonal matrix with Die positive diagonal matrix. If:
and let P be arbitrary, ddg "zy
(gW IJP P( - cPP
25

---

Under review as a conference paper at ICLR 2022
and
D D -3D
for B, > O, then (1) is contracting in metric with rate B.
Proof. Consider the differential, quadratic Lyapunov function:
V PDáx
where D > 0 is as defined above. The time derivative of V is:
V = PDDx + 8x (-2PD PDWD + PDWD
The second term on the right can be factored as:
5x" ' PW W'
8x D(D P PW D'D
8x DP' - 1 - 11PDDxx
-x PD2
where the last inequality was obtained by substituting in the first assumption above. Combining this with the expression for V. we have:
V S 6x PDox - cg ca-cg-o0x -8x PD
PD28x
Substituting in the second assumption, we have:
V. < 5x 5x PDD cg- "D= < -85x - PDox -BV
and thus V converges exponentially to with rate 3.
A4.8 PROOF OF THEOREM 8
Theorem 8. Satisfaction of the condition
g sym I
is NOT sufficinnt to show global contraction of the general nonlinear RNN (1) in any constant metric. High levels of antisymmetry W can make it impossible to find such a metric, which we demonstrate via 2 x 2 counterexample of the form
0
c
W
c
with 2
< 0 is equivalent to the condition for contraction of the system with linear activation in the identity metric.
W-WT Note that g sym g 2
The main intuition behind this counterexample is that high levels of antisymmetry can prevent constant metric from being found in the nonlinear system. This because D a diagonal matrix with values between O and so the primary functionality it can have in the symmetric part of the
26

---

Under review as a conference paper at ICLR 2022
Jacobian is downweight the outputs of certain neurons selectively. In the extreme case of all or values, we can think of this as selecting a subnetwork of the original network, and taking each of the remaining neurons be single unit systems receiving imput from the subnetwork. For given static configuration of D (think linear gains), this is hierarchical system that will be stable if the subnetwork is stable. But D evolve
time when nonlinearity is introduced, we would need to find a constant metric that can serve completely distinct hierarchical structures simultaneously- which is always possible.
Put in terms of matrix algebra, D can zero out columns of W, but not their corresponding rows. So for a given weight pair Wajs WHi, which has entry in W sgm = " if D; 0 and D, 1, the i.j entry in (WD)sgre will be guaranteed have lower magnitude if the signs of Way and wys are the same, but guaranteed to have higher magnitude if the signs are different. Thus if the linear system would be stable based on magnitudes alone D poses no real threat, but the linear system requires antisymmetry to be stable, D can make proving contraction quite complicated (if possible all).
2
Proof. The nonlinear system globally contracting in a constant metric if there exists symmetric, positive definite M such that the symmetric part of the Jacobian for the system, (MWD) sym M negative definite uniformly. Therefore (MWD) sym M must hold for all possible D if M is constant metric the system globally contracts in with any allowed activation function, as some combination of settings obtain particular can always be found.
Thus to prove the main claim, we present here a simple 2-neuron system that is contracting in the identity metric with linear activation function, but can be shown to have no M that simultaneously satisfies the (MWD) sym M < 0 condition for two different possible D matrices.
To begin, take
0 -2] 2 0
W
Note that any off-diagonal magnitude 2 2 would work, as this is the point which 1. of one of the weights (found in W Waym when the other is zeroed) will have magnitude too large for (WD) be stable.
Looking at the linear system, we can see is contracting the identity because
W syy
0 -1 0
Now consider (MWD) sym
M with D taking possible values of
o D2
0 1
D
and
0
We want to find some symmetric, positive definite M and (MWD2) M are both negative definite. m
a
m
such that (MWD)) 1)em)
M
b
Working out the matrix multiplication, we get
2m
b m
(MWD,) M
b
-
and
(MWD2) ssgm M
-a -(a + m))
-(a + m) -2m
27

---

Under review as a conference paper at ICLR 2022
We can check necessary conditions for negative definiteness these two matrices, as well as for positive definiteness on M. to try to find an M that will satisfy all these conditions simultane- ously. In this process will reach contradiction, showing that no such M can exist.
A necessary condition for positive definiteness in real, symmetric n x n matrix X is zµ > 0, and for negative definiteness Iş O. Another well known necessary condition for definiteness of a real symmetric matrix is [as + Ezsl > ( + Tzal = 2 2/z Vi + j. See (Weisstein) for more info on these conditions.
Thus we will require a and to be positive, and can identify the following conditions necessary for our 3 matrices to all meet the requisite definiteness conditions:
2m < a
(7)
-2m < b
(8)
12m - (a + b)l > 2|b ml
(9)
- 2- b) > 2/a + ml
(a
(10)
Note that the necessary condition for M be PD, a a+ > ml, not listed, as it is automatically satisfied if equation 7 and equation 8 are.
It is easy to see that if m 0, conditions equation 9 and equation 10 will result in the contradictory conditions a > and b > a respectively, so we will require a metric with off-diagonal elements. To make the absolute values easier to deal with, will check
0 and
O cases independently.
First we take m > 0. By condition equation 7 we must have a > 2m, so between that and knowing the signs of all unknowns
positive, can reduce many of the absolute values. Condition equation 9 becomes a + 2m > [2b 2m/] and condition equation 10 becomes a + b + 2m > must also have b m, condition equation further reduces to a + 6 2m > 2b 2m, which is equivalent to > b. Therefore we have again reached contradictory conditions.
2a 2m, which is equivalent to
i
A very similar approach can be applied when m < 0. Using condition equation 8 and the known signs we reduce condition equation 9 to 2 2/ml + a a++6 2b + 2/m,, i.. b. Meanwhile condition equation 10 works out to a + 6 m| > 2a - 2 ml i.c. > a.
Therefore it is impossible for a single constant M to accommodate both D; and D2, so that no constant metric can exist for W to be contracting when a nonlinearity introduced that can possibly have derivative reaching both of these configurations. One real world example of such a nonlinearity is ReLU. Given a sufficiently high negative input to one of the units and sufficiently high positive input to the other, D can reach one of these configurations. The targeted inputs could then flip at any time to reach the other configuration.
An additional condition we could impose on the activation function is to require to a strictly increasing function, that the activation function derivative never actually reach O. We will now show that a very similar counterexample applies in this case, by taking
and D2. = c 0
D =
Note here that the W used above produced a (WD)sym that just barely avoided being negative definite with the original D2 and Dy, will have increase the values the off-diagonals
28

---

Under review as a conference paper at ICLR 2022
bit for this next example. In fact anything with magnitude larger than 2 will have some 0 that will cause a constant metric to be impossible, but for simplicity we will now take
W. 0
4
0
Note that with W., even just halving one of the off-diagonals while keeping the other intact will produce (WD)sym that not negative definite. Anything less than halving however will keep the identity metric valid. Therefore, we expect that taking c in D1. and D2, to be in the range 0.5 x will also cause issues when trying to obtain constant metric.
We will now actually show via a similar proof to the above that M is impossible to find for W. when e v 0.5. This result is compelling because it not only shows that does not need to be particularly small value, but it also drives home the point about antisymmetry the larger in magnitude the antisymmetric weights are, the larger the where we will begin to encounter problems.
Working out the matrix multiplication again, now get
4m
2b m 2nc
(MW.Di+)e M
m 2
-mm
and
4me - a (22a + 2bc)
-(2a + m - -2be)
(MW, D2. M
-4m
Resulting in two new main necessary conditions:
44m a 4me/ > 2/22 m 201e
(11)
|4me a - b 4ml > 2/2a + 2be
(12)
As well as new conditions on the diagonal elements:
4m
(13)
4m - 6<0
(14)
We will now proceed with trying to find a, b, m that can simultaneously meet all conditions, setting = 0.5 for simplicity.
Looking m O, we can see again that M will require off-diagonal elements, as condition equa- tion 11 is now equivalent to the condition a + > |46 2a| and condition equation 12 is similarly now equivalent to a
|4a- 2b).
Evaluating these conditions in more detail, if we assume 46 > 2a and 4a > 26, we can remove the absolute value and the conditions work out to the contradicting 3a 36 and 36 3a respectively. As an aside, if e > 0.5, this would no longer be the case, whereas with < 0.5, the conditions would be pushed even further in opposite directions.
If we instead assume 2a > 4b, this means 4a > 2b, so the latter condition would still lead to b > a, contradicting the original assumption of 2a 4b. 2b 4a causes contradiction analogously. Trying 4b 2a will lead to the other condition becoming b >> 2u, once again contradiction. Thus a diagonal M is impossible
So now we again break down the conditions into m > 0 and m < cases, first looking at m > 0. Using condition equation 13 and knowing all unknowns have positive sign, condition equation 11
reduces to a + 2m > 44b - 2(a + m)) and condition equation 12 reduces to + + 2m > 4a - 2(b m.). This looks remarkably similar to the 0 case, except now condition equation 11 has - 2m added to both sides (inside the absolute value), and condition equation 12 has 2m added to both sides in the same manner. If 4b 2(a + m) the 22 term each side will simply cancel,
29

---

Under review as a conference paper at ICLR 2022
and similarly if 4a 2(b m) the +2m terms will cancel, leaving us with the same contradictory conditions as before.
Therefore we check 2(a + m) 46. This rearranges to 2a 2(2b m) > 2(b -m). so that from condition equation 12 we get b > a. Subbing condition equation 13 in to 2(a + m) > 4b gives 8b < 4a 4m < 5a i.e. b < aa, contradiction. The analogous issue arises if trying 2(b - m) > 4a. Trying 2(a + m) = 4b gives m = 25 a, which in condition equation 12 results
in 56 a 66a 6b), while in condition equation 13 leads to 5a 8b, so equation 12 can further
Ia and b < g is a contradiction. Thus there reduce to 5b a > 6a - 66 i.e. 11b > 7a. But b > TI
no way for > to work.
Finally, trying m < O, we now use condition equation 14 and the signs of the unknowns reduce condition equation 11 to a aa+bb+tt b+22mm] > 14b-((a- |m])] and condition equation 12 to a + b- 2/m] > |4a 2 + mml)]. These two conditions are clearly directly analogous to in the m > 0 case, where b now acts as a with condition equation 14 being b >> 4/m]. Therefore the proof is complete.
A5 CODE FOR MODELS USED IN EXPERIMENTS
destettione for sponce
modile (Liggtntoggssdiie):
folloning taufds/d (iphhla)) u(k]
showe
ity. klick diggonl, is "ountroe mekiitt fos sateds and is
def koik (aelf, input size, hieden, alzes, output alez, alsha, denalty, select mult, asleet mols):
iuper (ww.kmnknnnnlllll..c.. thet, Anitt
vell. aaae imprisiie
self hidden sioes hisles sizes
selt. hidden size- mutorp..ur(hhedee stzea3]
ae2f puspes size outpos adse
self.elahe + alpha
inteiative Giseer Saset and wape layors, to be trvincl), slony with brisess
self seightt_Sh mn.rorrester (torchhrrrrrronee (orch. (1.2// sgrt(self. hidder_size, (self, sideersise, self sieet_s
self welght to nr.Paraanter(torch nomal (h, the sarttseff hides stre), cuttuut site, self histen steel))
aelf bise sn mn.rrrmetarrtarr www.cc.norraaa/..... 2/tp.arrt(aeil hidden si,s), (t,af/ output adsell)
aelf aaffbiia.Jh mn.eerencorrttee (torchhorrmmmaae (0 a/tp./ hisdon at hidden adac]))
Ascl ragistrr (' Lmmds
specify
based the nasie inctiactaation scntioned
gererate_fnttten hutssli_unnsteertteteee teren.tewerae
desstty, pre_select_eitt, post)elect_eelt))
self argisiitto bontsstrrrrffftt Juffurthe
self sdlfregist buffertt
regfire buffer (" (_im)
comatee the conerrttee betueee cateyerver, rhce will teatowd.
Laae create aaek aftee_A sttsten iegiitro (ul (awak)
self Lrrain (see ieast toassttordduoorrddaoao chw.ooooooo harp:////nmpsrtttcp.eeeeeeeeeeee Widden/seessddddeeeeessssss (wp.sum/seer.....oocc kidten_stzes), rp..om(seeIf (ideen_s
der fors tepurg:
state turch.zeros( Chrprtscirtt..sppppeeee. shapeee).sseff set.te))seerrhhhoeeee hhldder seeee dexice = selfvdevice)
state hart-tytpee_eeeenee (ec(state)
Inpute anputtanbted())
Acutputz 1)
range (fo((a)))))
Turetaeetate cate.u.T topute[s] 8 (catt.wweght.....) aate.itee_))) pastes.. (celt.N (aaebed) tatr
state è (cetf.wetght_...y
eoutpute [hy]
return state
Figure S7: Pytorch Lightning code for Sparse Combo Net Cell
30

---

Under review as a conference paper at ICLR 2022
inefine networt
class mndssmtlly Se( [((Liighttnnghbbbe):
Pytorch nodule for training the following system:
taurdx/dt krphi(x) u(t)
6, phi sonlinearity, block diagonal, "contracting" conbination
def Init (self, Input_size, Nidiensizes, output_siie,ellhho.A)
super (nwssssembly__. init)___nit
self. Input_sizeee input_size
self sizeennnsiisss sizes hidden sizes
austop.sum(hiddensiiee
wwtptutttttsteeeee output_size
self.alphe alpha
#trput to hldden weights
self weight nm..araneter (otrch..ornaa(.... httpp///rttsssiite wwwdeheennaeeee inidden to output weights
(self (se2),
www.deennssaeee self-inpud salf.input_ssae)
self wwwght_thhhh in.Pananeeer (torch.normal no. (0/p//0,//___eeee (ttpppptttsssf hiddenn si2e), (self outputt_stz,, self heff.hiddennsiie (ize)))
self registe but.fereeffeussssii
("V_aask", create_nask_gieee Baaii_giee___rrrriile (ettrrchhheeeee((mm) me).:mmmm)))))).ooolllo.. b))
sporoncter Loatton of nidlen-to-hiiee networks
self.U in. in.aarnneter forchheye(ssi. www...idddeenn..eeeeeeieeeeeeeee [ize])
self.V in.aranstert cruttrrtorccheee (torch. eye waidenn -idddddd_isee))
self Signa mm.arrnnter (torch.ones one. hidden (12e,)))
self Phi m. Ph.PPameeerr (torch. normal (/mp, ssel( www.hhidddsiiiin (esdden_sieeeenooo (wii)))iiiitiiddmsseeee....
self seif.sing_ve__
wtwp_assss_se_eiiee_ bbool))
self regiiter buffer ("S_offset", (seliet",.ee.iiiiiee (self.sing_.e ppp_ "tarch.onessseef Srrrcchonee(eee.SSieee... (tt.s......e.ssse....e
#truinable Enter-subnetworb weights
saf.L_trrae ns. (self . waw.ttttrr....cllccccc. (t.1////a..//./oppppaee http://pttppp.eearrree. ()) (sp.sum(es), rp.sun(ns))))
mblases] self
Parameter(torchh normall 1/mpp www.sseetr.......so.. hidden hidden_stze),dee st,estttee.......e......_. (t.sef wwt.ttttstteeee (ize)))
self bh (araanterrtorch.orre (0,1//p.s00)))pp httpppssrttsseefef.....es//s seeel..iiieesseee (t, size))
forward (self, Input):
Bookk the Left and right orth "stem" nutrices to produce diagonat bbock structure U_naked selt www.hhhetteee. mask
V.naseed self www.//el......... Vaask
syme-symetrie
these nutrices:
dexponentiate shew symestyctiic
produce Left and right orthogonat nutrices
natrix exppee__sssseee U_Jasked.TT
torch.natris, paskke_ V_nasked.T)
dget diagonat of singutor votue notris, with etments between (9,1)
torch.exp(-see exp(f Signas5 -self -self.5_ S_offset)
#construct the weight notrix contracting netric determined
torch.diag wwwed(seef...leoeeee
torch.diag csbed())
Berchdiagg.mbes((/sett
dget nstric netric Inverse
torch.diag, (self (e))f)))
iiin diach.diag.oe mbbe selt se11e)) 22)
fget mosk for antaork-to-netaan coupling
Laakked self.l_treee wrw.ssttrassssf.._eee
#intcialize hldden
unbind(1)
(wwputtrtttt.tuupeee sellf self)..s...lleeee siidden_size),ddee self.device) state state. tateeyyeee_steeeeeeee _s(state)
spropogate Input through the dynetics and store outputs for 1 in (ene@(len())puts))
-state F.relu(state @ M.T inputs[A] @ (self.weight.th.)) self.bias.hh) state é (L.aasked.T (1 Leaasedd) M.in)) self aeph-al
by (salf.weight_ by,
Figure S8: Pytorch Lightning code for SVD Combo Net cell.
A5.1 NETWORK DIAGRAMS
31

---

Under review as a conference paper at ICLR 2022
A)
Sparse Combo
Net
Example)
Computer
Gerporee
Full Architacture
Weights Legend
Unear Feodiorward Input Layer
Hidden Combo Recurrent
Trained Incar
Nogatt
B)
SVD Combo Net
D412
(2x3 RNN Example)
UNB
Full Architecture
Weights Legend
Linear Feedlorward Input Layer
Traind norlincar
Layer
noem constrained)
Trained Uineer
Outppt Layer
Nogotiiee Scedbock
Figure S9: Detailed architecture diagrams for Sparse Combo Net (A) and SVD Combo Net (B).
32


Reviews:
[{"review_id": "bNvm2aiVBy", "reviewer": "Reviewer_hNbM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\nHow to assemble a network of RNNs is an interesting problem. The theorems on contraction properties are helpful to people thinking about provably stable RNNs.\n\n\nWeaknesses\n(1)\tSection 3.2.1 is pretty dry to read. Reporting results from many individual AxB networks seem unnecessarily.\n\n(2)\tThe authors showed performance comparison with other types of networks in Table 1. I think it would be quite informative to show performance of networks where everything is kept the same, except that the RNNs are no longer provably stable. \n\n(3)\tIt would also be good to know what happens if all connection weights are trained, not just the connections between modules. Does the performance actually decrease despite having more parameters?\n\n(4)\tThe provably stable part is kind of separated from the training modular network part. How closely are they related? Is having stable RNN modules particularly important for sparsely connected modular networks?\n\n", "summary_of_the_paper": "The authors studied contraction properties of continuous-time recurrent neural networks. They further showed that a network of provably stable RNNs (net of nets) can be trained to reach competitive performance on several benchmarks, including sequential CIFAR10, even when only connections between modules are trained.", "summary_of_the_review": "Overall, this is an interesting paper that takes a less common approach to RNNs: provable stability and net-of-nets. The results are at places more difficult to read, but overall it is clear.\n\nI want to add that I cannot evaluate whether the mathematical derivations are correct.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b6dKVELqYLo", "reviewer": "Reviewer_V8Ay", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I thought that the empirical results were rather convincing for what is primarily a theoretical contribution. The authors first thoroughly investigate various permutations of their modular sparse combination network framework (# RNNs vs size of each using absolute value weight constraints) and do another investigation of their alternative SVD weight constraint network (which doesn’t perform as well or train as quickly). Most importantly, they then show that they can best SOTA algorithms on some of the common (albeit easier) benchmarks in the field, even under (and perhaps because of) these constraints. \n- The theoretical contribution is quite powerful. There has been a lot of recent work in networks with many individual recurrent components, such as the aforementioned AlphaGo or the more general recurrent independent mechanisms (RIMs) framework, but for the most part, they rely on intuitive explanations and empirical results over theoretical guarantees. Clearly specialized RNN modules can be quite powerful, but RNNs are notoriously unstable and difficult to learn, and learning such models end-to-end is tricky. If we can apply these constraint conditions and still achieve good performance (which seems like it could be realistic, particularly in the absolute value constraint case), then we can develop sets of useful modules and mix-and-match to the task in question. This paper doesn’t answer all of the intermediate questions, but the stability analysis is a key step.\n- The proofs in the appendix are well-done and easy-to-follow, given a sufficient math background.\n\nWeaknesses:\n- This paper is very dense and difficult to follow. It took me a few reads to really understand the value of network stability and how it’s achieved in this case. The appendix is a mandatory read as are some of the references. None of the use cases are particularly intuitive. I think I would have liked to see a graphical representation of the sparse combo network (rather than the weight matrices in Figure 2), some pseudocode for the algorithms (tossed in the Appendix), and maybe an example case of an unstable network assembly diverging. I also feel like my familiarity with AlphaGo and other methods gave me more of an insight into how this would help in practice than the actual paper did.\n- As much as I liked the empirical results that were provided, they’re all of a kind: sequential image prediction. I would have liked to see at least one application in a different domain (NLP, RL, continuous control, etc).\n", "summary_of_the_paper": "This paper is primarily a theoretical contribution to the construction of assemblies of recurrent neural networks. We know that combinations of learned modular components can be powerful and far more tractable than learning bespoke models from scratch, particularly in applied domains (e.g. AlphaGo). Yet so far, we have no theoretical guarantees that these combinations will actually remain stable. This paper develops the theory behind provably-stable combinations of RNNs using weight constraints and feedback mechanisms. Then, using fixed RNNs generated according to these constraints (leaving the connections between them as antisymmetric learnable parameters), the authors show that their sparse combination network is able to achieve SOTA performance on sequential image classification benchmarks with far fewer learned parameters and the previous stability guarantee.", "summary_of_the_review": "Overall, I would accept this paper. Although it was difficult to follow and required a lot of consultation with the literature, I do ultimately think that this is a direction that DL algorithms are going in and that the theoretical and practical results from this work could be quite powerful. To make the paper better, I would like to see some results in a different domain and more effort towards improving the readability. Too often, valuable theoretical works go underutilized because they’re difficult to understand or don’t seem relevant to the empiricists and engineers who could build on them.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VbqbyCMrUnF", "reviewer": "Reviewer_q4Si", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The theoretical results seem interesting, although not very surprising. However, I think there is disconnection between the theoretical results and the proposed model: not all theorems are relevant to the proposed model.\n\nFor the proposed model (Section 3.1), I might have missed something, but I fail to fully understand the model; the presentation could be improved to help the readers. For example, the mentioning of \"subnetworks\" at the beginning of Section 3.1 is not defined/explained. It's unclear to me how the subnetworks are combined. I can only infer from the \"recursive construction\" in the title and Figure 2 that the resulting weight matrix is a block matrix. \n\nFurthermore, the idea of parametrizing orthogonal weight matrices by exponentiating skew-symmetric matrices is not novel and has been explored in expRNN [1].\n\nThe writing of the introduction section could also be improved. Instead of discussing AlphaGo and modules in evolution, the reader might benefit from a more thorough literature review of the RNN trainability and long-term dependence.\n\n[1] Lezcano-Casado, Mario, and David Martınez-Rubio. \"Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "In the paper, the authors study stable architectures for RNNs. On the theoretical side, the authors present a series of conditions such that a weight matrix of an RNN is contractive. On the modeling side, the authors propose RNN architectures that have contractive weight matrices. The proposed methods are evaluated on benchmark datasets including sequential MNIST, permuted MNIST, and sequential CIFAR-10.\n", "summary_of_the_review": "The theoretical results in the paper seem interesting. However, the presentation of the proposed model is not clear, and the model itself does not seem novel. Overall, I think the paper needs improvement to meet the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LQO7txHQ0E0", "reviewer": "Reviewer_RFwT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Theoretical results\n\nThe theorems 1-5 constitute an evolutionary step in the understanding the conditions of stability. The authors also show a counterexample for the common belief that linear contraction leads a sufficient condition for non-linear stability. These results are then (partly) used to construct provably stable RNN combinations.\n\n## Experimental claims\n\nThe experimental section, while interesting, seems to lack a main takeaway. Also some very dubious choice of reporting in the table.\n\n* The authors claim that for the result shown on the Tab. 1, they run the Perm-MNIST trial 4 times and the results fall between 96.65 and 96.94. I was frankly shocked to find that they choose to only report 96.94 on Tab.1! Running the same experiment multiple times and only reporting the best case scenario is not good practice and leads to misunderstanding at best. I would recommend that the authors report mean + variance. (If one were to only report the best case, one could get much better performance than is achievable on average by running the experiment many many times.)\n\n* Some claims are backed up by only a single data point. For example, the claim that increased modularity benefits performance to some point is only backed up by the fact that 44x8 performs better than 22x16 in Sec. 3.2.1. To draw a significant conclusion and demonstrate a trend, the authors can perhaps look at 50x7 and 39x9. \n\n* In general, more data points and error bars would help convince the reader that the conclusions are real and not flukes. \n\n* The results of 'performance vs network size' and 'performance vs modularity' for Sec. 3.1 and 3.2 are opposite each other. In 3.1 increased size makes the network better monotonically but in 3.2 it is inverted U shape. Similarly in 3.2 modularity makes the performance better monotonically, but it is inverted U in 3.1. What is the conclusion to be drawn here? \n\n* In general, the experiment section is a little hard to read and can use a summary of the main conclusions and clearly demarcated paragraphs and sections of the experiment that demonstrate each point.\n\n\n\n### Clarity and other minor points\n\n* The paper is at times very clear and at times very confusing. For example, the discussion of stability and contraction are clear but then in the paragraph at the top of page 3, the authors use the symbol g for two different thing in the same paragraph. \n* Theorem 7 should be slightly reworded so that it is clear that the first inequality is a condition and not a statement (this is rather obvious in hindsight but for a new reader it is very confusing).\n* For denoting multiplication, I would suggest using $\\times$ instead of x (e.g. 22x16 etc.)\n* Why is there a section 3.2.1 instead of just 3.2?\n* I would suggest an overall re-read of the paper to maximize readability.\n\n\n", "summary_of_the_paper": "The submission proposes new theorems showing the stability of a class of RNNs. Further by combining these RNNs into hierarchical and feedback superstructures, the submission achieve SOTA performance on a number of tasks.", "summary_of_the_review": "The paper constitutes an evolutionary step in understanding and designing stable RNNs. The theoretical results are novel and noteworthy. Unfortunately the experimental results lack a clear conclusion and at times do not follow best practices (i.e. reporting only the best run out of many).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the context of recurrent neural networks, the motivation of the paper is to explore the "space" between fully trained models and almost not trained models, e.g. echo state networks, using a formal approach. In fact, a modular approach has proven to be very successful in many practical applications, and in addition brain seems to adopt this strategy as well. The addressed theoretical issue is stability of the network (i.e., the network implements a contraction map.) Specifically, it is assumed that a network is composed of a set of subnetworks that meet by construction some stability condition, and the problem is to design a mixing weight matrix, interconnecting the latent spaces of the subnetworks, able to give stability guarantees during and after training. Some novel stability conditions are proposed as well as two different approaches to design a successful mixing weight matrix. The original submitted paper was not easy to read, and after revision major problems with presentation have been resolved, although the current version looks more like an ordered collection of results/statements than a smooth and integrated flow of discourse. The revision has also addressed some concerns by reviewers on the role of size and sparsity of the modules, as well as the sensitivity of the stabilization condition on the mixing weight matrix has been experimentally assessed, obtaining interesting results. Overall the paper reports interesting results, however the novelty of the contribution seems to be a bit weak, e.g. stability conditions on recurrent networks (although different from the reported ones) were already presented in literature. Also the idea of exploiting, in one of the proposed models,  the fact that the matrix exponential of a skew-symmetric matrix is orthogonal to maintain the convergence condition during training, is not novel. Moreover, the experimental assessment does not provide a direct comparison, under the same architectural/learning setting, of the novel stability results versus the ones already presented in literature. Empirical results are obtained on simple tasks (using datasets with sequences of identical length), and relatively small networks, which limits a bit the scope of the assessment, as well as it is not clear if the observed improvements (where obtained) are statistically significant (especially when compared with results obtained by networks with the same order of parameters.) The quality of the assessment would increase significantly by considering datasets with sequences of different lengths, and involving more challenging tasks that do require larger networks.
Decision: Reject