Recursive Construction of Stable Assemblies of Recurrent Neural Networks

1 INTRODUCTION . Neuro-inspired machine learning has profoundly altered many fields such as computer vision , natural language processing , and computational neuroscience ( Bengio et al. , 2017 ; Hassabis et al. , 2017 ) . While models trained with e.g . deep learning are remarkably powerful , they are for the most part ‘ black boxes ’ . This opaqueness can be dangerous in safety-critical applications , such as autonomous driving or human-centered robotics , and it limits conceptual progress . In the case of recurrent models , one difficulty is that providing a certificate of stability is currently impossible or computationally impractical . Given that stability is a fundamental property of dynamical systems – and is intimately linked to concepts of control , generalization , data-efficiency , and robustness – being able to guarantee the stability of a recurrent model is an important step towards making sure deep models behave as we expect them to . In this spirit , there has been a recent flux of work focusing on applications of contraction analysis ( Lohmiller & Slotine , 1998 ) to recurrent models . Loosely speaking , a dynamical system is said to be contracting if any two of its trajectories converge to each other exponentially , regardless of initial conditions . A primary advantage of contraction analysis is that it is directly applicable to non-autonomous systems , which the vast majority of recurrent models are , allowing in turn modular contraction-preserving combination properties to be derived ( Slotine & Lohmiller , 2001 ; Slotine , 2003 ) . We include a mathematical primer on contraction analysis in Section A1 . Within this line of research , our paper has two main aims : 1 ) To provide simple contraction conditions for continuous-time recurrent neural networks and 2 ) To show how these continuous-time contraction conditions imply a combination property . Using both aims , we proceed to implement stable combination networks that exhibit state-of-the-art ( SOTA ) performance on multiple sequential image classification tasks with a small number of trainable parameters . 1.1 PREVIOUS WORK ON RNN STABILITY . To briefly review the current literature on application of contraction analysis to recurrent models , we first note that the ‘ Echo-State Condition ’ introduced in Jaeger ( 2001 ) is equivalent to discrete-time contraction in the identity metric . A later generalization of this condition included a diagonal metric ( Buehner & Young , 2006 ) . In the context of neuroscience , contraction analysis has been applied to analyzing the dynamics of winner-take-all networks ( Rutishauser et al. , 2011 ; 2015 ) as well as networks with synaptic plasticity ( Kozachkov et al. , 2020 ) . In the machine learning context , Miller and Hardt recently rederived an ‘ echo-state property ’ for discrete recurrent models , and went on to prove that these contracting recurrent models could be well-approximated by feedforward networks in certain cases ( Miller & Hardt , 2018 ) . More recently still , in a series of papers Revay , Wang , and Manchester applied contraction analysis to discrete-time recurrent networks ( Revay & Manchester , 2020 ; Revay et al. , 2021 ; 2020a ) , expanding the class of models considered in Miller & Hardt ( 2018 ) . In addition to contraction ( which amounts to a strong form of non-autonomous exponential stability ) there has been a considerable amount of work attempting to enforce weaker forms of stability in RNNs , such as autonomous stability ( Erichson et al. , 2021 ; Chang et al. , 2019 ) . Unfortunately , as we discuss in Section 2.3 , autonomous stability does not in general imply non-autonomous stability , so it is not clear what stability properties these RNNs possess when driven with external input . There is also a line of work which uses orthogonal weight matrices to avoid the vanishing/exploding gradient problem during training . Orthogonality is typically ensured during training via a parameterization ( Arjovsky et al. , 2016 ; Lezcano-Casado & Martınez-Rubio , 2019 ) – for example , by exploiting the fact that the matrix exponential of a skew-symmetric matrix is orthogonal , as is done in Lezcano-Casado & Martınez-Rubio ( 2019 ) . These works focus on parameterizations of individual RNNs which ensure stability is preserved during training . By contrast , our work focuses on ‘ combined ’ RNNs ( i.e network of networks ) and provides a parameterization on the connections between stable subnetworks such that training preserves the overall network stability ( Figure 1 ) . 1.2 COMBINATION NETWORKS . The combination and reuse of primitive “ modules ” has enabled a great deal of progress in computer science , and is also a key theme in biological evolution , particularly apparent in cortical structure of the human brain . In fact , it is thought that the majority of traits that have developed over the last 400+ million years are the result of evolutionary forces acting on regulatory elements that combine core components , rather than mutations to the core components themselves . This mechanism of action makes meaningful variation in population phenotypes much more feasible to achieve , and is appropriately titled “ facilitated variation ” ( Gerhart & Kirschner , 2007 ) . In addition to the biological evidence for facilitated variation , computational models have demonstrated that this approach produces populations that are better able to generalize to new environments ( Parter et al. , 2008 ) , an ability that will be critical to further develop in deep learning systems . While the benefits of building modular systems are clear ( Simon , 1962 ) , as in DeepMind ’ s AlphaGo for example ( Silver et al. , 2016 ) , there is no general guarantee that a combination of stable systems will itself be stable . Thus the tractability of these evolutionary processes hinges on some mechanism for ensuring stability of combinations . Because contraction analysis tools allow complicated contracting systems to be built up recursively from simpler elements , this form of stability is well suited for biological systems ( Slotine & Lohmiller , 2001 ; Slotine & Liu , 2012 ) . Note also that contracting combinations can be made between systems with very different dynamics , as long as those dynamics are contracting . Here , we describe two common forms of contracting system combinations – hierarchical and feedback – that automatically guarantee overall system stability ( Figure 1 ) . Ultimately , cognitive models are moving increasingly toward study of multi-area dynamics , but many questions remain on how to best train and evaluate such networks ( Yang et al. , 2018 ; Yang & Molano-Mazon , 2021 ) . Understanding how different brain regions interact harmoniously to produce a unified percept/action will require new ideas and analysis tools . 2 MATHEMATICAL RESULTS . 2.1 CONTRACTION CONDITIONS FOR INDIVIDUAL , CONTINUOUS-TIME RNNS . We consider the following continuous-time RNN , evolving according to the following equations : τ ẋ = −x + Wφ ( x ) + u ( t ) ( 1 ) where x ∈ Rn , φ is a static nonlinearity such that 0 ≤ φ′ ≤ g , u ( t ) is some input ( potentially timevarying ) and τ > 0 . We do not constrain the sign of φ . Example nonlinearities φ ( x ) that satisfy the constraints are tanh ( ax ) , log ( 1 + ex ) , and max ( 0 , x ) , with g = a , 1 , 1 respectively . Note that this class of RNNs is equivalent to another commonly used class of RNNs where the terms Wx + u appears inside φ ( · ) . See Section A2 or ( Miller & Fumarola , 2012 ) for details . Our mathematical results therefore apply equally well to both types of RNNs . We seek a stability certificate for this system in terms of the recurrent weight matrix W. We are specifically interested in restricting W such that the RNN is globally contracting . We do this with the goal of recursively combining these contracting RNNs with other contracting RNNs to make large , complicated , stable ‘ networks of networks ’ . Here we derive five restrictions on W which ensure contraction for the continuous-time RNN defined by equation 1 . To the best of our knowledge these contraction conditions are novel . All proofs can be found in the supplemental Section A4 . Theorem 1 ( Absolute Value Restricted Weights ) . Let |W| denote the matrix formed by taking the element-wise absolute value of W. If there exists a positive , diagonal P such that : P ( g|W| − I ) + ( g|W| − I ) TP ≺ 0 then equation 1 is contracting in metric P. Moreover , if Wii ≤ 0 , then |W |ii may be set to zero to reduce conservatism . Note that when g = 1 , Theorem 1 can be checked by checking for linear stability of |W| − I. Theorem 2 ( Symmetric Weights ) . If W = WT and gW ≺ I , then ( 1 ) is contracting . Theorem 3 ( Product of Diagonal and Orthogonal Weights ) . If there exists positive diagonal matrices P1 and P2 , as well as Q = QT 0 such that W = −P1QP2 then ( 1 ) is contracting in metric M = ( P1QP1 ) −1 . Theorem 4 ( Triangular Weights ) . If gW− I is triangular and Hurwitz , then ( 1 ) is contracting in a diagonal metric . Theorem 5 ( Singular Value Restricted Weights ) . If there exists a positive diagonal matrix P such that : g2WTPW −P ≺ 0 then ( 1 ) is contracting in metric P . 2.2 THE MODEL : NETWORK OF NETWORKS . The RNN in equation 1 is a subnetwork of our model . Our goal is to combine these subnetworks into a ‘ network of networks ’ in a manner that preserves the stability of the underlying modules . In particular , we seek to parameterize and learn the inter-module connections . To do this , we prove and make extensive use of the following theorem : Theorem 6 ( Network of Networks ) . Consider a collection of p subnetwork RNNs governed by equation 1 . Assume that these RNNs each have hidden-to-hidden weight matrices { W1 , . . . , Wp } and are independently contracting in metrics { M1 , . . . , Mp } . Define the block matrices W̃ ≡ BlockDiag ( W1 , . . . , Wp ) and M̃ ≡ BlockDiag ( M1 , . . . , Mp ) , as well as the overall state vector x̃T ≡ ( xT1 · · ·xT2 ) . Then the following ‘ network of networks ’ is globally contracting in metric M̃ : τ ˙̃x = −x̃ + W̃φ ( x̃ ) + u ( t ) + Lx̃ L ≡ B− M̃−1BTM̃ ( 2 ) Where B is an arbitrary square matrix . Note that we are agnostic to the particular contraction condition here – the subnetwork RNNs can satisfy any of the theorems in the preceding section . 2.3 ON STABILITY THEOREMS FOR RNNS . Several recent papers in machine learning , e.g ( Haber & Ruthotto , 2017 ; Chang et al. , 2019 ) , claim that a sufficient condition for stability of the nonlinear system : ẋ = f ( x , t ) is that the associated Jacobian matrix J ( x , t ) = ∂f∂x has eigenvalues whose real parts are strictly negative , i.e : max i Re ( λi ( J ( x , t ) ) ≤ −α with α > 0 . This claim is generally false . For a counter-example , see Section 4.4.2 in ( Slotine & Li , 1991 ) . However , in the specific case of the RNN ( 1 ) , it appears that the eigenvalues of the symmetric part of W do provide information on global stability in a number of applications . For example , in ( Matsuoka , 1992 ) it was shown that if Ws = 12 ( W + W T ) has all its eigenvalues less than unity , and u is constant , then ( 1 ) has a unique fixed point that is globally asymptotically stable . It is easy to see that this condition also implies that the real parts of the eigenvalues of the Jacobian are uniformly negative . Moreover , in ( Chang et al. , 2019 ) it was shown that setting the symmetric part of Ws = 12 ( W + W T ) almost equal to zero ( yet slightly negative ) led to rotational , yet stable dynamics in practice . This leads us to the following theorem , which shows that if the slopes of the activation functions change sufficiently slowly as a function of time , then the condition in ( Matsuoka , 1992 ) in fact implies global contraction of ( 1 ) . Theorem 7 . Let D be a positive , diagonal matrix with Dii = dφidxi , and let P be an arbitrary , positive diagonal matrix . If : ( gW − I ) P + P ( gWT − I ) −cP and Ḋ− cg−1D −βD for c , β > 0 , then ( 1 ) is contracting in metric D with rate β . It has been conjectured that diagonal stability of gW − I is a sufficient condition for global contraction of the RNN 1 ( Revay et al. , 2020b ) , however this has been difficult to prove . To better characterize this conjecture , we present Theorem 8 , which shows by way of counterexample that diagonal stability of gW − I does NOT imply global contraction in a constant metric for ( 1 ) . Theorem 8 . Satisfaction of the condition gWsym − I ≺ 0 is NOT sufficient to show global contraction of the general nonlinear RNN ( 1 ) in any constant metric . High levels of antisymmetry in W can make it impossible to find such a metric , which we demonstrate via a 2× 2 counterexample of the following form , with c ≥ 2 : W = [ 0 −c c 0 ]

Summary: In the paper, the authors study stable architectures for RNNs. On the theoretical side, the authors present a series of conditions such that a weight matrix of an RNN is contractive. On the modeling side, the authors propose RNN architectures that have contractive weight matrices. The proposed methods are evaluated on benchmark datasets including sequential MNIST, permuted MNIST, and sequential CIFAR-10.