Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control
Authors: Neehal Tumma
Mathias Lechner
Noel Loo
Ramin Hasani
Daniela Rus
Venue: ICLR 2024 spotlight (2024)
Keywords: Low-rank
sparsity
closed-loop
recurrent neural networks
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Low-rank and sparse recurrent matrices of RNNs can help generalization to closed-loop settings and distribution-shifts

Abstract: Developing autonomous agents that can interact with changing environments is an open challenge in machine learning. Robustness is particularly important in these settings as agents are often fit offline on expert demonstrations but deployed online where they must generalize to the closed feedback loop within the environment. In this work, we explore the application of recurrent neural networks to tasks of this nature and understand how a parameterization of their recurrent connectivity influences robustness in closed-loop settings. Specifically, we represent the recurrent connectivity as a function of rank and sparsity and show both theoretically and empirically that modulating these two variables has desirable effects on network dynamics. The proposed low-rank, sparse connectivity induces an interpretable prior on the network that proves to be most amenable for a class of models known as closed-form continuous-time neural networks (CfCs). We find that CfCs with fewer parameters can outperform their full-rank, fully-connected counterparts in the online setting under distribution shift. This yields memory-efficient and robust agents while opening a new perspective on how we can modulate network dynamics through connectivity.

Published as a conference paper at ICLR 2024
LEVERAGING LOW-RANK AND SPARSE RECURRENT CONNECTIVITY FOR ROBUST CLOSED-LOOK CONTROL
Neehal Tumma? Mathias Lechner
Noel Loo" Ramin Hasani Daniela Rus"
Harvard University
MIT CSAIL
ABSTRACT
Developing autonomous agents that can interact with changing environments is an open challenge in machine learning. Robustness is particularly important in these settings as agents are often fit offline on expert demonstrations but deployed online where they must generalize to the closed feedback loop within the environment. In this work, we explore the application of recurrent neural networks to tasks this nature and understand how a parameterization of their recurrent connectivity influences robustness in closed-loop settings. Specifically, we represent the recur- rent connectivity as a function of rank and sparsity and show both theoretically and empirically that modulating these two variables has desirable effects on network dynamics. The proposed low-rank, sparse connectivity induces an interpretable prior on the network that proves to be most amenable for class of models known as closed-form continuous neural networks (CFCs). We find that CfCs with outperform their full-rank, fully-connected counterparts in
fewer parameters the online setting under distribution shift. This yields memory-efficiee and ro- bust agents while opening perspective how can modulate network dynamics through connectivity.
INTRODUCTION
Building models that are robust under natural distribution shift has long been goal in artificial intelligence (Taori et al., 2020). Existing techniques have sought to address this challenge through various approaches, including domain adaptation (Farahani et al., 2020), transfer learning (Zhuang et al., 2019) and data augmentation (Perez & Wang, 2017). However, these techniques come with drawbacks: domain adaptation and transfer learning can be computationally expensive, and data augmentation often suffers from robustness gains that are not uniform across corruption types (Ford et al., 2019). More generally, machine learning systems tend to perform poorly under distribution shift because approaches like these only serve to ameliorate a problem that is rooted in the model architecture itself
Natural learning systems serve to address this problem by interacting with their environment to understand the world. They make use of biologically-insprred frameworks that account for distri- bution shifts by modeling temporal structure (Hasani et al., 2020b). These models are particularly well-suited in paradigm where they are fit offline, passive datasets using imitation learning, but deployed in closed-lood loop, active testing settings (de Haan et al., 2019). This domain is commonly re- ferred to as the open-loop to closed-loop causality gap (Lechner al., 2022) which generalization requires the agent to learn coherent representation of its world (Lechner et al., 2020). One par- neural networks (CfCs) were
ticularly effective framework known as closed-form continuous-tin shown to outperform many state-of-the oh-tte-art recurrent models in closed-loop settings (Hasani et al., 2021). These models leverage a sparse wiring structure induced at initialization, yielding compact and interpretable networks (Chahine et al., 2023). However, the reason to why sparse connectivity useful in closed-loog systems is poorly understood. In this work, we will address this question by proposing and analyzing interpretable parameterization of connectivity in CfCs.
To do so, we turn to another class of models: low-rank recurrent neural networks (low-rank RNNs). A low-rank RNN is network whose recurrent connectivity matrix is low-rank. Deriving inspiration

---

Published as a conference paper at ICLR 2024
from neural activity in the brain, these networks have provably low-dimensional patterns of activity which makes them successful in many simple neuroscience- based tasks (Mastrogiuseppe & Ostojic, 2018). In this paper, leverage ideas from previous work CfCs and low-rank RNNs in order to devise a novel parameterization recurrent connectivity that improves robustness in closed-loop settings and offers interpretable measures for the network dynamies it incites. In doing so, we show the following:
Parameterizing recurrent connectivity as a function of rank and sparsity yields provable and interpretable network dynamics (Section 3.1)
Low-rank and sparse recurrent neural networks can outperform their full-rank, fully- connected counterparts under distribution shift in closed-loop environments (Section 4.1)
Pruning by inducing sparsity and pruning by enforcing low-rank structure are distinct in the types of network dynamics each induces (Section 4.3)
CfCs more amenable low-rank and sparse connectivity prior than other canonical recurrent architectures such as LSTMs (Section 4.4)
RELATED WORK
In this section, we describe previous works that are closely related to the core findings the paper.
Robustness in closed-loop settings. A class of continuous-time recurrent models known liquid neural networks (Hasani et al., 2020a) have been shown achieve sfate-of-the-aee performance un- der distribution shift by explicitly accounting for external interventions to environment conditions. Liquid neural networks are prime example natural learning systems that learn robust, and prov- ably causal, representations (Vorbach et al., 2021). CICs specific instance of liquid neural networks that can leverage sparse connectivity to learn robust representations in the closed-loop causality gap setting.
Outside modifying the model itself, other approaches to training robust models under an imitation learning frame- work include augmentation strategies, human interven- tions (Ross al., 2010), goal-conditioning (Codevilla et al, 2017), reward conditioning (Srivastava et al., 2019), task-embedding (James et al., 2018) and meta-learning (Finn et al., 2017). These advances fail to consider the structure of the underlying policy; neural networks improve the robustness of the decision- making process in autonomous agents, leading to better generalization under the same training distribution (Vor- bach al., 2021).
RNN in open-loop system
W
W
W
contrast, liquid
RNN in closed-loop system
Relating connectivity and network dynamics. A long- standing goal in computational neuroscience is to under- stand the relationship between structure and function in the brain (Sporns, 2013). We can ask the analogous ques- tion in the context of artificial neural networks: what role does model connectivity at initialization play in learned network dynamics? Amongst recurrent neural networks, echo state networks (Goodfellow et al., 2016) present one example of a model that induces static sparsity order to modulate the complexity of dynamics along the recur- rent dimension. However, unlike our case, echo state networks fix their recurrent connectivity during training. Other works have found that sparse networks can yield provably more robust models (Guo et al., 2018) that gen- eralize across many tasks (Chen et al., 2022) outside the recurrent setting. With respect modulating network rank, the advent of the low-rank RNN frame- work (Mastrogiuseppe & Ostojic, 2018) one the first attempts at explicitly modeling the low- dimensional dynamics of the brain. Some follow-ups on this work include examining how low-rank
XPa
X
P
W
W
Figure 1: Open-loop systems
ceive ground-truth observations æ;- Closed-loop systems receive observa- tions nı (a;-1) that are a function of the previous actions the agent takes. There is no external feedback to correct the agent in the closed-loop setting.
2

---

Published as a conference paper at ICLR 2024
connectivity arises in full-rank settings (Schuessler al., 2020) and analysis of how some tasks are more amenable to low-rank connectivity than others (Dubreuil, 2022). Our work borrows ap- proaches and intuition from many of these prior studies, but is distinct with respect to the proposed connectivity, analysis of network dynamics and task setting.
Pruning at initialization. Neural network pruning typically either removes connections in costly iterative training-retraining paradigm (Frankle & Carbin, 201 8) or modifies the objective function to promote sparsity (Goodfellow al., 2016) which can lead to difficulties in optimization. Recently, pruning at initialization has emerged as an approach to resolve these issues. Pruning at initialization attempts take a randomly initialized network and remove weights before training (Wang et al., 2021). Various approaches include using connectivity sensitivity (Lee et al., 2018), maintaining dynamical isometry (Lee et al., 2019) and training on supermasks (Ramanujan al., 2019). Gener- ally, these techniques outperform sparsity induced randomly at initialization. Another approach to pruning using rank decomposition. One example is low-rank matrix factorization, which is good at reducing size but at the cost of performance (Sainath et al., 2013). The most effective low-rank compression techniques usually leverage a low-rank approximation on the full-rank matrix either during or after training (Xu al., 2020). In contrast, in this work, propose parameterization of connectivity that randomly induces sparsity and leverages low-rank decomposition of the recurrent weights initialization- the most straightforward and least costly form of pruning.
Alternative RNN parameterizations. Existing works proposed other recurrent architectures such as Lipschitz RNNs (Erichson et al., 2021), antisymmetric RNNs (Chang et al., 2019) and Cay- ley RNNs (Helfrich et al., 2018) which, like us, propose alternative parameterizations of recurrent connectivity. However, unlike our study, these works introduce changes to the underlying network architecture, whereas our work focuses on the applications of low-rank and sparse recurrent connec- tivity in more canonical recurrent networks. This in line with previous works in the closed-loop causality gap setting (Chahine et al., 2023) which also restrict the scope of their models as such.
3 PARAMETERIZATION OF CONNECTIVITY
Consider standard RNN whose input is denoted by H E R" and hidden state is given by h, E R for time step hidden size and input size n. Then the functional form of the RNN is given by
h; = tanh( W - + Win Vn + b)
such that Wrea E Rhxh denotes the recurrent weights, W Winp E Raxh denotes the input weights and b denotes the bias. We now propose alternative parameterization of connectivity for the current weights. Consider singular value decomposition of Wree given by Wree = UEVT The canonical rank r approximation of Wree is U,E, V such that Σ, consists of the top-r singular values and U, and V consist of the corresponding singular vectors. Using this low-rank approxi- mation, we can construct the following parameterization of the recurrent weights for a given rank and sparsity level s:
Wree (r. 8) (2()) O M(s)
(2)
where Wi 122 W. = 1/2) and M F, is random binary mask such that each entry Mi, ~ 1 Bernoull((s). This low-rank and sparse parameterization of Wree is a generalization of the parameterization studied in Herbert & Ostojic (2022) to recurrent connectivity of arbitrary rank and trainable weights W,, W2 as opposed to restricting ourselves to the setting of rank-1 1 matrices and fixed weights in which dynamical analysis is more tractable, but the network is constrained in expressivity. Note that the random mask M is fixed throughout training.
3.1 NETWORK DYNAMICS AT INITIALIZATION
We propose this parameterization of recurrent connectivity as it allows us to modulate aspects of the network that instill inductive bias, which proves to be beneficial for performance in the closed- loop setting, particularly under distribution shift. We care about three measures of Wree(r, s) in particular: the spectral radius, the spectral norm and the rate of decay of the singular value spectrum.
3

---

Published as a conference paper at ICLR 2024
hxh
hxr
h xxh
h h
rx h
rank-r SVD
sparsify
W (r)
W rec
WG (r)
M(s)
W (r, s) rec
Figure 2: Parameterization of recurrent connectivity as a function of rank and sparsity.
The spectral radius Amaz of W (T, s) is widely accepted as a proxy for the rate at which the gradi- ent evolves backwards in time (Pascanu et al., 2012). Namely, if Xmaz < 1, an RNN has vanishing gradient (refer to A.5.1 for details). In many time series applications, having constant error flow desirable property, as arbitrary data sequences may have long-term relations. However, in the case of many closed-loop systems, learning long-term dependencies can be detrimental in the on- line setting due to the short-term causality inherent to the task (Lechner et al., 2020). To put this into context, in this work we consider agents learning to play games in various Arcade Learning Environments (ALEs). Agents that perform well in ALEs often use frame stacking (Horgan et al., 2018a) 8a) which enforces a strict short-horizer temporal prior by leveraging look-back window of only a few input frames. Hence, networks like LSTMs or GRUs may capture spurious long-term dependencies and thus learn inadequate models. In contrast, the vanishing of gradients that tends to be more pronounced in RNNs counterintuitively enhances the performance of the agent, as it places a short-horizon prior on the temporal attention span of the network.
We will motivate the importance of the singular values in the distribution shift setting. Consider an SVD on the recurrent weights given by (re s) = UVVT, Furthermore, consider perturbation sampled uniformly at random from the set of norm-1 vectors. If we apply this perturbation to the hidden state h2 then to measure robustness we want to quantify the deviation between W Wree (r, s) and Wree(r, (h, + e). This is given by Wrea s)e = UVVTe. e. Note that both U and VT are unitary matrices and thus do not affect the magnitude of e. This means that Σ captures the nature of the transformation on e. The two relevant aspects of the transformation are its magnitude and direction. The spectral norm, the maximum singular value, measures the former (i.e. smaller spectral norm implies better robustness). To quantify the latter, examine the rate which the singular values decay. This provides a proxy for the effective number of directions e is expanded in (i.e. faster decay implies better robustness). For details this argument, refer to A.5.2.
Now that we have motivated the spectral properties of W (rec(T,s) s), we next aim to understand how they change as function of rank and sparsity. We consider two random initialization schemes for W (ree(r, s): Glorot uniform spectral initialization (GU-spec) and orthogonal spectral initialization (ortho-spec). To thoroughly motivate the proposed parameterization, we provide theoretical proof of the relationships between rank/sparsity and spectral radius/spectral norm for both initialization schemes where possible. For the cases we do not prove, we provide an empirical analysis instead.
Theorem 1.
Given recurrent weights with parametrization shown in Equation (2) and initialization scheme specified in parentheses, we prove the the following:
The spectral radius of Wree(T, 8) decreases as a function of s (GU-spec)
The spectral radius of Wree(T, 8) increases as a function of r (ortho-spec)
The spectral norm of Wree(T, s) decreases as a function of (GU-spec)
The spectral norm of W Wree(r, 8) is constant as a function of (ortho-spec, GU-spec)
For proofs of the above properties and an empirical analysis for the unproven cases (which follow the same trends as the proven cases), refer to A.6. And since our theoretical arguments do not readily extend to the full singular value spectrum, sparsity for this as well (A.7) which shows that the rate of singular value spectrum decay increases as function of rank and decreases function of sparsity.
provide an empirical analysis a function of rank and
4

---

Published as a conference paper at ICLR 2024
3]
CTC
LSTM
RNN
GRU
CNN
0.0
0.5
0.8
0.0
sparsity
0.5
sparsity
0.5
sparsity
0.3
sparsity
0.5
sparsity
0.89
roward
0.82
0.2
0.2
0.2
0.76
0.0
0.0
0.:
0.0
0.0
0.69
16
16
27hul
51
CfC
LSTM
RNN
GRU
CNN
0.8
1.30
sparsity
0.5
Sparsity
sparsity
sparsity
sparsity
0.5
0.58
1:54
0.2
3.0
0.0
0.0
0.0
16 rank
27
1.5
10 rank
27 md
16 rank
27 ful
3.5
16
27
1.5
16
27hul
rank
rank
Figure 3: Online performance of recurrent networks under different ranks and sparsities in Seaquest environment. For offline performance, refer to A.9. a) In-distribution rewards in the online, closed- loop setting normalized by the rewards obtained by the expert in- in-distribution. b) Rewards averaged across 5 distribution shifts, normalized by the rewards obtained by the expert under distribution shift.
4 EXPERIMENTS
Experimental setup. We study the offline-online generalization gap of various recurrent architec- tures parameterized by low-rank, sparse connectivity under an imitation learning framework. In par- ticular, we measure the performance of our models in the Arcade Learning Environment (Bellemare et al., 2012) and MuJoCo (Todorov et al., 2012). For ALEs, we run experiments in the Seaquest and Alien environments and for MuJoCo ment, we train deep-Q network to generate expert trajectories that we then use to our recurrent networks in offline setting. We evaluate the recurrent networks online, closed-loop when de- ployed into the environment, both in-distribution and under distribution shift. For more details on the experimental framework, refer A.II.
explore the HalfCheetah environment. For each environ-
Models. We examine the following recurrent architectures: RNNs, LSTMs, GRUs, CNNs and CfCs. RNNs, LSTMs (Hochreiter & Schmidhuber, 1997) and GRUs (Cho al., 2014) refer to their canonical implementations. CNNs serve to address whether supervision along the recurrent dimension is even necessary. CfCs (Hasani al., 2021) are modified version of an RNN that admit the following functional form:
(1t) (F(hh-,,, @ G(h 0G) + (F H( Bi, 01)
Here, H and G are vanilla RNNs and F can be interpreted interpolates between the state-space trajectories of H and on per-element basis the hidden vector h(t) (refer to A.4 for details). Finally, recall in Section 3.1 that we proposed low-rank, sparse parameterization of connectivity (r, s) for the recurrent weights generalize this parameterization to the other recurrent architectures by simply parameterizing all recurrent weights in the network in the form Wree(r, s). For specifics, refer to A.11.
adaptive gating mechanism that
vanilla RNN. We
4.1 PERFORMANCE IN CLOSED-LOOP SETTINGS
We first explore the impact of our proposed connectivity parameterization for various ranks r and sparsities in the Seaquest environment in the online, in-distribution setting. We observe that the best performing models in-distribution tend to be low-sparsity, high-rank CíCs and LSTMs (Figure 3a). GRUs, on the other hand, tend to perform poorly for most (r,s) which aligns with previous work which also shows that GRUs are not particularly well-suited for closed-loop settings (Chahine et al., 2023). Due to the poor performance of GRUs relative to LSTMs, cussion and analysis to CfCs, LSTMs and RNNs. The most notable finding from the in-distribution results is observed in the high-sparsity models; in particular, find that LSSTMs tend to perform
will restrict most of our dis-
5

---

Published as a conference paper at ICLR 2024
b)
CTC
CC
CC 0.5)
CTC
2.0
15
(041)
0.5
0.0
rank
LSTM
LSTN 64 0.0)
LSTM (4 - 0.21
LSTM - 0.5)
LSTM 66 - 0.80
100 (15
100 NAN
100 (F
1.0
0.5
0.0
rank
Figure 4: a) Spectral radius of recurrent weights W (re s) in trained CIC and LSTM networks across ranks and sparsities (#1 SE). b) Frobenius norm of recurrent gradients G. as function of time (#1 SE). Note that the norms are plotted in log space and translated to decay from 0 to enable easier comparison. For details on the computation, refer to A.
much better than CfCs at high sparsities. Analogously, RNNs, which in general tend to perform worse than CfCs, show similarly worse performance at high sparsities relative to LSTMs. We offer intuition for these findings in Section 4.2 where we explore the relationship between sparsity and the recurrent memory horizon of the network.
Under distribution shift, find that the best performing models are low-rank, low-sparsity CfCs. More generally, across all recurrent architectures, we find that the low-rank models tend to perform on par with, and in many cases actually outperform, higher rank ones. This demonstrates that we can construct models with fewer parameters at initialization yet still learn more robust models. Interestingly, however, we find that the means through which we prune at initialization matters: in particular for high-sparsity models, we do not achieve the same robustness that we observe in low- rank ones. We will provide justification for the apparent disparity between pruning by increasing sparsity and pruning by decreasing rank in Section 4.3. For analogous results in the Alien and HalfCheetah environments, refer to A.9.
4.2 MODULATING RECURRENT MEMORY
Here, we aim to gain intuition for the results we observed in the online, in-distribution setting and in particular offer an explanation as to why the only effective form of pruning in-distribution was inducing sparsity in LSTMs. Recall in Section 3.1 we motivated the parameterization Wree(r, s) with respect to three measures, one of which was spectral radius, proxy for model's attention span across time. In particular, we showed at initialization that the spectral radius of Wrec(r, 8) decreases function of sparsity and increases after training (Figure 4a). In particular, in both CfCs and LSTMs, the low-rank and highly-sparse models tend to admit solutions with recurrent weights of low spectral radius.
function of rank. We first note that these trends persist
Comparing CfCs and LSTMs, we note that CfCs tend to admit solutions with significantly lower spectral radii. This is perhaps not surprising considering that LSTMs explicitly promote more consistent error flow over time (Jozefowicz et al., 2015) via their forget gate, and are thus more likely to attend to distant observations. So, assuming that the spectral radius is reasonable proxy for the recurrent memory-horizon horizon of model, it appears that LSTMs tend to have longer-term mem- ah.
= ||Jx|| || as this is the true measure of gradient propagation backwards time. Since || J&ll depends not only on the recurrent weights but also on the hidden state (for details, refer to A.4), it is possible that the disparate functional forms of CfCs and LSTMs mitigates the efficacy proxy for attention across time (for more details this intuition, refer A.5.1). Let us define G. = JJ- Jh-++1- Assuming that time represents the end of the time series, G. repre- sent the gradient backpropagated t steps in time, which when analyzed as function of t provides a direct measure of how much importance a model assigns to inputs as a function of the time that has passed since it last observed them. In particular, we compute log || |.el function of l and
ory than CfCs. However, note that the quantity we care about in practice is
Th,
spectral radius as a

---

Published as a conference paper at ICLR 2024
a)
rankr 1
16
64
5 0.0
5 0.2
5 0.5
$ 0.8
1.0
1.0
3.0
01 mar
Ore
Dra
0.0
0.0
0.0
20 sorted
sorted
sorted
sarted
b)
sparshy
---0.5
CTC
LSTM
RNN
0rax 2.1
Gregx 15
1.5
2.
Figure 5: a) Decay rate of singular value spectrum of CfC recurrent weights as a proxy for robust- ness. b) Spectral norm of the recurrent weights as a proxy for robustness (#1 SE).
find that gradients decay significantly faster across time in CfCs than LSTMs (Figure 4b). One, this demonstrates that spectral radius is effective measure of recurrent memory across architectures in this task setting. Two, we observe that in their full-rank, fully-connected forms, LSTMs have a long recurrent memory-horizon, while CfCs do not. In the context of closed-loop task, we know that is beneficial to limit the network's 's temporal attention span (Lechner et al., 2020). Since an LSTM's recurrent gradient in its baseline form does not decrease particularly fast across time, inducing spar- sity at initialization pushes the network into the vanishing gradient regime, reducing its attention across time. This result provides support as to why LSTMs are quite effective at navigating envi- ronments in-distribution in spite of high-sparsity in their recurrent weights (Figure 3a). In contrast, CICs in their baseline form already lie the vanishing gradient regime: intuitively, inducing spar- sity is less effective when the baseline network inherently possesses affinity to selectively attend to past observations. This aligns with an analogous trend observed in RNNs which are networks that, like CfCs, amenable to learning a vanishing gradient (Figure 19).
By analyzing the spectral radius of Wrec(r, s) as well as the recurrent gradients Gt, we have char- acterized sparsity in its ability to modulate the network's recurrent memory-horizon However, note that we also observe that low-rank LSTMs tend to admit solutions with lower spectral radius as well (Figure 4b). Yet, low-rank LSTTMs tend to perform worse in-distribution (Figure 3a) than their sparse counterparts. To understand why there is a disparity between reducing rank and increasing sparsity, we turn to an analysis of the singular values, which are best motivated the distribution shift setting.
4.3 ROBUSTNESS UNDER DISTRIBUTION SHIFT
In Section 4.1, we found that in the distribution shift setting, pruning by reducing rank improved performance (most notably in CICs) whereas pruning by increasing sparsity did not. This is in stark juxtaposition to the in-distribution trends in which inducing sparsity in LSTMs was the only form of pruning that did not worsen performance (Section 4.2). Here, we attempt to interpret these results by understanding both why CfCs appear to be the most robust model type and why pruning by reducing rank is distinct from pruning by increasing sparsity.
Recall in Section 3.1 we reduced the robustness of the hidden state under perturbation to two mea- sures: the spectral norm and the decay of the singular value spectrum of W Wree(r, 8) (for intuition, refer to A.5.2). Regarding the former, across models we observe that CfCs have lower spectral norms than both RNNs and LSTMs (Figure 5b, Figure 23). While this offers intuition as to the robustness standalone measure. This is because in practice, distribution shifts
of CfCs, it is not sufficient are applied to the input H,, which in turn corrupts the hidden state h4+1- Thus, while it remains important to analyze the spectral norm of the recurrent weights, it is also pertinent to analyze the spectral norm of the input weights. In doing so, we find that while there exist marginal differences across architectures, the input spectral norm does not vary the extent observe in the recurrent
7

---

Published as a conference paper at ICLR 2024
spectral norm (Figure 20b). This, along with the fact that CfCs learn significantly lower spectral norms in their recurrent weights, offers intuition as to the heightened robustness of CfCs under dis- tribution shift. In addition, the (albeit loose) relationship between weights with lower spectral norms corresponding to networks with lower Lipschitz. constants provides even further support as to why CfCs tend express more robust functions (elaborated upon in A.5.2).
Next, we address the observed disparity between sparse and low-rank recurrent connectivity. Across ranks and sparsities, the spectral norm decreases as function of increasing sparsity and decreas- ing rank (Figure 5b), aligning with the trends induced at initialization. In Section 4.2, we similarly showed that spectral radius decreases as function of increasing sparsity and decreasing rank in the trained networks. Thus. we cannot explain the apparent disparity between low-rank and sparse connectivity via these measures, and instead turn to the decay of the singular values. Again, aligning with the prior induced at initialization, we find that the decay of the singular values increases as function of increasing sparsity but decreases as function of decreasing rank (Figure Sa). And this is precisely where the two methods of pruning differ: since sparsity reduces the rate of spectral decay, the resulting transformation (induced by Wree) on a perturbed version of the hidden state vector expands in more directions (i.e. increasing the effect of the perturbation); lowering the rank does the opposite. We can make this notion more concrete by analyzing the dimensionality of the spte- trajectories of each model. Note that the trajectory induced by h, can be decomposed into two parts: the recurrently-drive portion Wt and the input-driven portion Wing Under this formu- lation, we can consider the full stace trajectory h2 to be driven by Wfuli = Wree Wimp].
To measure the complexity of these state-space trajectories. we ran PCA on them to measure their effective dimensionalities. We find that with decreasing rank, the dimensional- ity of the recurrently-driver trajec- tories decreases, whereas with in- creasing sparsity it increases (Fig- ure 6a, Figure 6b). Hence, our intuition is as follows: since the
spasly --- 012
CVC Wae
CFC Wre
EFC W/2
explaieed vat
0.8
explaines
0.5
05
0.7
0.0
rank
rank
rank
LSTM
LSTM W
W..
coolaredd var.
1.0
esplannee roed
activity along the recurrent axis is
constrained to lie in the subspace spanned by the vectors comprising (r, 8). the recurrent dynamics of low-rank recurrent networks are and hence sim- pler. We can imagine that in the in- distribution setting, it is not desir- able to arbitrarily constrain the net- work's capacity to learn recurrently. This provides justification as to why, in spite of the low-rank LSTMs also having a shorter recurrent memory-horizon, inducing sparsity into LSTMs does not worsen the in-distribution performance of the network like constraining rank does (Figure 3a). In contrast, recall that under distribution shift, we observed the opposite: inducing sparsity at initializa- tion was detrimental whereas constraining the network to be low-rank improved robustness. Con- straining the recurremtly-driven portion of the state-space is one means of reducing the network's s variability in the presence of input perturbations. Supervision along the recurrent axis is pivotal when the agent is faced with environmental occlusions, as it needs to lean on some notion of the past to make a decision in the present. By making the recurrent state more robust, we are better able to generalize under distribution shift.
0.0
15
0.0
2.5
27
1uB
25
27 NI
Figure Effective dimensionalities of the recurrent, input and full state-space trajectories collected during online test- ing, in-distribution, as measured by the explained variance of the top 5 principal components. a) CfC state-space dynamics 1+1 SE). b) LSTM state-space dynamics (#1 SE).
We make one final note across the model axis to further justify why CfCs appear to be inherently more robust than LSTMs and RNNs. In particular, in both the input-driven and full state-space trajectories (each of which are little affected by changes in the recurrent connectivity), we find that for all pairs (r, s). the dimensionality of LSTM trajectories is much higher than that of CfC trajectories, offering additional justification for the robustness observed in CICs. But what is perhaps more surprising is that RNNs, despite their simpler functional form, also posses higher-dimenssooall state-space dynamics than CfCs (Figure 18). Since RNNs and CfCs differ only with respect to the time constant network F, we explore this gating mechanism further in A.2.
8

---

Published as a conference paper at ICLR 2024
We
AW
CFC
LSTM
RNN
15
15
15
||W||=
|||||
[[W]]F
27 ful
1.0
27
rank
rank
rank
Figure 7: We individually consider the Frobenius norm of the weights at initialization Wo and the change in the weights after training AW. Note that the models shown here have no sparsity.
4.4 EXPLORING THE TASK DIMENSION GAP
We now have demonstrated two things: one is the efficacy of the proposed connectivity parame- terization in units of its interpretability connectivity has on each of the networks we examined: in particular, by disentangling the effects of rank and sparsity, we showed why LSTMs are more amenable sparse connectivity in-distribution whereas CfCs tend to show the most promise with low-rank connectivity under distribution shift. Here, we further our intuition on the performance of CfCs under distribution shift by understanding why they tend to be the most amenable architecture to low-rank recurrent connectivity.
modulator of network dynamics. Second is the impact
As we have noted, the efficacy of this parameterization rests upon the ability of the network to adhere to the prior throughout training. And recall that we found that the spectral norm of the recurrent weights CiCs remained much closer to in the full-rank, fully-connected network than in either LSTMs or RNNs (Figure 5). To better understand this, we borrow from analysis conducted by Schuessler et al. (2020) in which they decomposed the recurrent weights as follows: Wree = Wo + AW where Wo denotes the weights at initialization and AW denotes the change in the weights after training. In their setting, the purpose of the decomposition was to demonstrate that in set of simple tasks, the Frobenius norm of the weights at convergence [[Weell F is dominated by the norm of the weights at initialization [o F.. In spite of the recurrent connectivity they used being full-rank, they found that the changes in the weights learned during training were in fact low-rank as measured by ||AW]|.. In our analysis, we find the opposite: ||ollle certainly does dominate the norm of the final weights and is in fact lower than |WW]| (Figure 7). This reinforces the notion of task dimension put forth by Schuessler et al. (2020) which describes the rank of the saining-induced connectivity changes as function of the task the network trained on. In particular, in their work, they showed that the task dimension of the simple tasks they examined was low and hence a network with unconstrained, full-rank connectivity learned low-rank changes. In contrast, in our setting we consider significantly more complex task domain which incites higher- rank changes in the recurrent connectivity. This brings forth the notion of a task dimension gap between the offline, open-loop and online, closed-loop settings: namely, the networks examined are trained offline without being exposed to distribution shifts and hence learn higher-rank changes have shown, succeeding in the closed-loop setting under
in their connectivity. In contrast, distribution shift means learning lower-rank dynamics. Thus, networks that are able to abide to our low-rank prior and avoid learning high-rank changes in connectivity distribution shift. This is precisely where CfCs supersede LSTMs and to some extent RNNs as well. We find that despite each network starting the same Wolee, ||WW]] is lowest in CfCs.
better generalizing under
5 CONCLUSION
In this work, we investigated the use of a low-rank, sparse parameterization of recurrent connectiv- ity in various architectures as a means of improving model robustness in closed-loop environments. We showed that this type of connectivity was most amenable to CíCs and also showed promise in more canonical networks like LSTMs and RNNs. Furthermore, we demonstrated the interpretability of this prior by analyzing the network dynamics it induces a function of both rank and sparsity. Our results represent an application in pruning recurrent networks at initialization to improve per- formance under distribution shift.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Research was sponsored by the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agree- ment Number FA8750- 19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is autho- rized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we extensively detail the experimental setup in the ap- pendix as well as provide information regarding the analyses we conducted. The bulk of these details can be found appendix A.11 which describes how we constructed the dataset, how the models were trained (including details on hyperparameters), the evaluation metrics for the models and the intittonnimplemmnatioon regarding the analyses that we performed on the trained models. A key portion of our results is driven by an initialization scheme we proposed that deviates from default initializers given in existing open-source implementations. We thoroughly describe how and why our proposed initialization differs in appendix A.10, the reader leverage it to reproduce our results. Regarding our theoretical work, we provide proof's for the claims we made in the main portion of the paper which be found in appendix A.6. In that section, clearly delineate the cases in which we were unable to prove certain claims and had to resort to an empirical analysis instead.
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large- scale machine learning. /2th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 16), pp. 265-283, 2016. files/comm hwnfeerence/sddi1////
URL https:
org/ system/
bbadi.pdf.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning vironment: An evaluation platform for general agents. CoRR, abs/1207.4708, 2012. URL
http:///rxiie org/ abs abs/1207.... 4708.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. URL pattt:///axivy abs/1606. 01540.
Makram Chahine, Ramin Hasani, Patrick Kao, Aaron Ray, Ryan Shubert, Mathias Lechner, Alexan- der Amini, and Daniela Rus. Robust flight navigation out of distribution with liquid neural networks. Science Roboties, 8(77))eadc882,, 2023. doi: 10.1 206/sciroboties.adee660 https: http:///ww..ciiiee wttp:///t/////////tt/t//p//t/t/p 10. 1121/112/80 scii robot ics adc8892.
URL
Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi. Antisymmetriernn: A dynamical system view on recurrent neural networks, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), NeurlPS, pp. 6572-6583, 2018. URL htttp://bbpp: uni-trier: de/ db/ con //ni/ppsss ppsp/siiii ps2018 html #ChenRBD18.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang. Santosh Balachandra. Haoyu Ma. Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generalization from more efficient training. 2022.
10

---

Published as a conference paper at ICLR 2024
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406.1078, 2014. URL 1078.
Felipe Codevilla, Matthias Müller, Alexey Dosovitskiy, Antonio M. López, and Vladlen Koltun. End-to-end driving via conditional imitation learning. CoRR, ab/11700.0201, 2017. URLhttp: //arxiv..rrg www.vvorg/aaae abs/ / 1710. 02410.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. CoRR, abs/1905. 1979, 2019. URL http: //tpx/aavvvv////////a///a/aaaava www.riii.gg//.ooo..o.o... abs 905. 11 979..9
Valente A. Beiran M. et al. Dubreuil, A. The role of population structure in computations through neural dynamics, 2022.
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference Learning Repre- sentations, 2021. URL https::///// // http://ppppeeeeeeeee/eeeeeeeeeet .nw wtt.:viwwwwwee/tuttteee.
Eorum? id= -
Abolfazl Farahani, Sahar Voghoci, Khaled Rasheed, and Hamid Arabnia. A brief review of domain adaptation, 2020.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imi- tation learning via meta-learning. CoRR, hess1709.04905, abs/ 2.bs/1709. 04905.
Nic Ford, Justin Gilmer, Nicholas Carlini, and Ekin Dogus Cubuk. Adversarial examples are natural consequence of test error in noise. CoRR, abs/1901.10511. 2019. URL httt://///a
org/abs/ 1901. 10513.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net- taabs/180.. /1003.00... 03635.
works. CoRR, 000/1803.,3635, 2018. URL http:
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved ad- versarial robustness. CoRR, 0bss/8100096111 2018. URL httt:://axxiv wrw.iioorrr//..oooo..o. org/ abs/ 11100. 09619.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time- constant networks, 2020a.
Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. A natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4082-4093, PMLR, 13-18 2020b. URL https::/ppocceeesesssst
mlr www.s/pppsssseeeeeee wttppp/ssss//.aaaaarraaaaea
html.
Ramin M. Hasani, Mathias Lechner, Alexander Amini, Lucas Liebenwein, Max Tschaikowski, Ger- ald Teschl, and Daniela Rus. Closed-form continuous-depth models. CoRR, abs/2106.13898 2021. URL https: /awpivivvv.//////////aa/////aavv org/ abs abw.///.................mm.aem.b 12101.13888........
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform, 2018
Elizabeth Herbert and Srdjan Ostojic. The impact of sparsity in low-rank recurrent neural net- works. bioRxiv, 2022. doi: 10. 10.1101/2022.033144481111 URL https: //www biorxiv. org/ccontet/eeri seanteent/arrln httrrlly/2222222 01/0////2222 2222 03. 31........ 486515.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
11

---

Published as a conference paper at ICLR 2024
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, 0bbs/1803.00933, 2018a. URL http:://rxila org/ abs/ / 1803 00933.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, abs/1803.00993, 2018b. URL http:/ /ttt:://aaxiv.../ org/ abs / 1803. 00933.
Todd Huster, Cho- Yu Jason Chiang, and Ritu Chadha. Limitations of the lipschitz constant defense against adversarial examples. CoRR, abs/1807.09705. 2018. URL http://arxix
org/aba. 110771 09705.
Stephen James, Michael Bloesch, and Andrew J. Davison. Task-embedded control networks for few-shot imitation learning. CoRR, abs/1810.03237, 2018. URL httttp://axxi..ortttt abs/
1810.03231
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent net- work architectures. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 2342-2350, Lille, France, 07-09 Jul 2015. PMLR. URL https: httpt::ss/pprceeeingge...e mlr.. press/vas /v3// jozefowiczii...o html.
Mathias Lechner, Ramin Hasani, and Alexander Amini. Neural eircuit policies enabling auditable autonomy, 2020.
Mathias Lechner, Ramin Hasani, Alexander Amini, Tsun-Hsuan Wang, Thomas A. Henzinger, and Daniela Rus. Are all vision models created equal? study of the open-loop to closed-loop causality gap, 2022.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: single-shot network pruning based on connection sensitivity. CoRR, abs/1810.02340,, 2018. URL htttp://arxiv.oooo
abs/ 810.1... 02340.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propaga- tion perspective for pruning neural networks initialization. CoRR, Abs/1906.00330 2019. URL http:://arxt orooobrg//..... htth/bbs/1999 /9906. 06307.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Gold- berg, and lon Stoica Ray rllib: composable and scalable reinforcement learning library. CoRR, abs/1712.09381, 2017. htsp:://xxx/...oo
org/ abs 09381.
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, (293)6609-23.e29 2018. ISSN 0896-6273. doi: URL https: : //www/ sciencedirect. com/
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/1211.5063. 2012. URL http: /ttp:///axivvii/ii org/ abs/ 1211 11111....1 .5633.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification us- ing deep learning. CoRR, abs/1712.04621, 2017. URL 04621.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models, 2023.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? CoRR, abs/1911. 2019. URL http:// rxiv. org/ abs/ 911. 13299.
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1001.0686, 2010. URL http///xxxvvo//o/e abs/10 0686.
12

---

Published as a conference paper at ICLR 2024
Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high- nigh-dimensional output targets. In 2013 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 6655- 6659, 2013. doi: 2L.10090CASSPP2213..6636
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation, 2019.
Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in rnn. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 13352-13362. Curran Associates, Inc., wwwpinngn.nuuurraa neurips. cc/ httpp/peeeeeeeeeeeee/eeeeeeeeeee ppppp//ppe////////////p Bad75-Paper .
2020. URL
9ac1 steee/ccc1322ffFfc463333
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.063411 2017. URL 1707. 06347.
org/ abs/
Olaf Sporns. Structure and function of complex brain networks. Dialogues in Clinical Neuroscience, 15(3):247-26, 2013, doi: 3...1187/DDD
URL https: / http//doo.oo/// org/
B///B8//DCCS./ 2013. osporns. PMID: 24174898.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaskowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. CoRR, abs/1912.02877, 2019. URL
org/ abs/ 1912. 02877.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mcasuring robustness natural distribution shifts in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 18583-18599 neurips ec cccppapprriiiieeeeeeeeeeeeeeeeee files/paper/2020
Curran Associates, Inc.,
2020. URL https::/ File/d8330 /at.//88300055511117552222200000 7ele44ee77776d00000 (bwdd0--dddpppoee Pwper. pdf..
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 1.111///ROS.2212.36660000
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Yyарunov expo- nents for mns: Understanding information propagation using dynamical systems tools, 2020.
Charles J Vorbach, Ramin Hasani, Alexander Amini, Mathias Lechner, and Daniela Rus. Causal navigation by continuous-time neural networks. A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: http:///peervvieeee net / forum? i wdd.dcccss222....
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. CORR, abs/2103.0646/ 2021. URL hwt.tttt///// httpp:://axxi....tt// wrt. 22103.2//13 06460.
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi. Yiran Chen, Weiyao Lin, and Hongkai Xiong. TRP: trained rank pruming for efficient deep neural networks. CoRR, abs/2004. 14566, 2020. URL https: //arxiiivorrr http/bb//2000/ 14566.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu Hui Xiong, and Qing He. comprehensive survey on transfer learning. CoRR, abs/191 2019. URL
http:/ /ttp/iiivvv.////a//////////aaava wwwoorggbbbb.. 19111.111. 00111.26855. 625...
Karol Žyczkowski and H. J. Sommers. Truncations of random unitary matrices. Journal of Physics A, 33:2045-2057, 1999. URL https://api semanticschol. ar. Corg/CorrpssD:
27605911.
13

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
DIMENSIONALITY ANALYSIS UNDER DISTRIBUTION SHIFT
8)
spersity -
0.0
CIC Wrec
CTC Who
CFC Who
expleine VeL.
explaimed val.
expleinec vec
0.6
0
0.29
16
16
1.4
511
rank
rank
b)
Sparsity -a- --- 0.2 -.505 -
RNN Wee
RNN Way
RNN Wao
oxplained var.
explained S35
explained var.
8.40
0.0
9.35
0.4
0.50
16
1al
16
27
fa1
5
16
(
rank
rank
tank
sparsity 0.0 - 0.2 0.50.5 -0-08
LSTM Wess
LSTM Winp
LSTM Wis
3.0
oxplained var.
ropisined var.
explained varr
0.35
0.4
0.5
0.30
0.4
Figure 8: Effective dimensionalities of the recurrent, input and full state-space trajectories collected during online testing, under the noise distribution shift (for details, refer to A.I1), as measured by the explained variance of the top 5 principal components. a) CfC state-space dynamics (#11 SE). b) RNN state-space dynamics 1#1 SE). <) LSTM state-space dynamics (+11 SE).
Recall in Figure 6, we computed the effective dimensionalities of the recurrent, input and full state- space trajectories across models, ranks and sparsities collected during an online, closed-loop simu- lations of the agents in an in-distribution setting. In doing so, we found that the input and full state space dimensionalities were not particularly affected by changes in the recurrent rank sparsity. Here, we will show that the same does not hold under distribution shift and understand why this is a desirable property of the parameterization of recurrent connectivity.
In Figure 8, we find that in CICs, RNNs and LSTMs, the effective dimensionalities of the input and full state spaces increases with the rank of the recurrent weights. This is interesting as demonstrates the ability of the parameterization to modulate state-space dynamics outside of the drriiee subspace of activity. But it also begs the question as to why we do not observe this effect in the in-distribution setting.
The primary distinction is that under distribution shift, we can imagine that adding noise to the inputs causes the input state-space trajectory to evolve in random directions, raising its dimensionality. Having a robust recurrent state presumably allows for the filtration of some of this noise, which reduces the effect it has on future model inputs. Furthermore, since examine these trajectories in the context of a closed loop system, the input is itself a function of the previous hidden state, By making this function more robust via low-rank prior, this in turn reduces the dimensionality of the input state-space trajectory (and by proxy the full state-space trajectory as well).
One final note is that while we observe this trend as a function of rank, we do not observe it as a function of sparsity. In particular, modulations in the dimensionality of the recurrent state-space via changes in the sparsity of the recurrent weights do not appear impact the dimensionality of
14

---

Published as a conference paper at ICLR 2024
input full spate-spaces (Figure 8). While this certainly requires further exploration, possible hypothesis that this is caused by the counteracting effects of sparsity on robustness. Namely, recall that sparsity both reduces the spectral radius which shortens the temporal attention span of the network, making the model more robust across time. However, it also reduces the decay rate of the singular value spectrum, making it less robust at any given point in time. These effects potentially offset one another and prevent the sparsity in the recurrent weights from modulating the dimensionality of the input trajectory.
A.2 TIME CONSTANT ANALYSIS
sparsity
0.0
0.2
0.5
0.8
0.35
0.30
00.5
0.25
W
0.20
0.15
1
16
27
full
rank
Figure 9: Average absolute deviation from 0.5 of time constant vectors in trained CfC models.
In Section 4.3, we demonstrated that on average, CfCs both learn lower spectral norms in their re- current weights and express lower dimensional state-space trajectories than RNNs. Recall, the func- tional form of a CfC differs from an RNN via the time constant network (,, which learns vector of weights used to interpolate between G and H. Note that if F(h--1, 2z, [1, 1..... 1], the network reduces to that of an RNN by placing all its weight on a single trajectory. Because the dynamically adapt its inter-
time-constant learned by F is function of the input, the network polation weights under distribution shift by changing how much it weighs the trajectory induced by G versus the trajectory induced by H. While rationale beyond the effects of the time-constant mod- ule on the overall network dynamics warrants further exploration, the notion of gating mechanisms improving performance in neural networks is not new one; recent work done in convolutional language models shows similar findings (Poli et al., 2023).
In any case, better understand the time constant network, here we provide an analysis on what role modulating recurrent sparsity and rank has on the learned interpolation weights. In particular, let we denote the interpolation weights that are the output of F time We compute
w. 0.5| ΣT
Eh
(e 0.5
hT
which measures the average deviation of the weights from 0.5. We consider this metric means of understanding how much the interpolation between G and H tends to rely upon only one of the trajectories (as we observe in the of RNN) versus evenly combining them (as would be the case if F' learns to output [ ...... D. We find that as a function of increasing rank, the deviation of the time constant from 0.5 decreases (Figure 9). This means that low-rank CfCs tend to place
more of their weight on one trajectory than the other which demonstrates the ability of the time constant network align with the simpler recurrent dynamics promoted in the low-rank setting.
A.3 TASK DIMENSION CONTINUED
Here, we provide some additional commentary on our discussion of the task dimension gap in Sec- tion 4.4.
15

---

Published as a conference paper at ICLR 2024
At high level, the notion of task dimension gap which characterizes the disconnect between agents learning passively in the offline setting and learning actively in the online setting warrants further exploration and formalization. Furthermore, forth by Schuessler et al. (2020) itself is more aptly decomposed into recurrent task dimension and input task dimension. In particular, we have shown, changes in recurrent connectivity essarily synonymous with changes in input connectivity (Figure 6). A given task may necessitate learning connectivities of different ranks along the input axis and recurrent axis: disentangling the two is necessity in understanding the dynamics of recurrent neural networks. Another point to note that recurrent task dimension distinct from attention span along the recurrent axis, which can also be considered function of the task. We showed that while rank and sparsity both reduce tem- poral attention span, they have opposing effects on the dimensionality of recurrently-driven activity. Noting this distinction is pivotal in properly characterizing recurrent dynamics.
note that, the notion of task dimension put
not
A.4 DERIVING RECURRENT GRADIENTS
In this section, we provide details on the derivation for J, 8h,_ in RNNs, LSTMs and CfCs. ah Jr is the Jacobian of the hidden-state dynamics which captures information about the rate which the gradient is propagated across time. We motivate this further when we discuss the relationship between Jj and the memory horizon of recurrent network in Section A.5.
A.4.1 RNN
Recall that the functional form of an RNN is given by
h tanh Wh,h-1 W, +
where W, denotes the recurrent weights. W, denotes the input weights and denotes the bias. Then, if we let @ denote row-wise multiplication in the case of vector and matrix, then we have that
0h. 0h
= (1 (anh?(h,-1)) @ W
A.4.2 LSTM
The functional form of an LSTM is given by
i, W + Whth-1 + b:)
f. W, + -
= (W + Whoh, bo)
g: tanh W W..:: Whch bb)
f. @ C--1 i + 9:
h o, @ tanh(c;)
Following the notation from Vogt al. (2020), if we let
W. Whaht-1
where is determined by the gate and O denotes elementwise multiplication in the case of two vectors and row-wise multiplication in the case of vector and matrix, then we have that
16

---

Published as a conference paper at ICLR 2024
0f (0())) 3 (1 (yr))l Wis
0h-1
Oi.
0h (3))) 0 o Wh
do. (3)) @ (1 - ) 0 Who
0h
0g: [11 tanh (y.))l (ynn)
1 Whe
dh.-1
dc; Ofc c 0h
di.
o tanh(y)) + o tanh (1-aanh² o Whe +
dh 0h--1
Ōh: do:
dc:
2 tanh(c)) + 0, O tanh (c))) 0 Oh,-
Oh.-1
0h.1
A.4.3 CFC
The functional form of CfC is given by
h(t) = 2,, 0F)) @ G(h 2,, 0G) + [F-h 0,) Hh4-1, 01))
where 8r, G, Oz refer to the parameters in each module as given by the following equations:
F(h--1, 0x) = Way., + Whyhh-1
G(h4-1,, Dt, 0g) tanh (i + Whght-1 + by) H(h;-1, 21, 0H) = tanh ( + Winhe-1 + bb)
We note that in the work that originally proposed the CfC architecture (Hasani et al., 2021), F is referred to as a liquid time constant network due to its motivation as time constant in dynamical system. In general, time constants are more thoroughly motivated in the continuous- in which a neural network is used to model the derivative of the hidden state as opposed to the hidden state itself (Chen et al., 2018). In that setting, time constant represents a parameter that characterizes the speed and coupling sensitivity of an ODE that models a system. However, our work exists the discrete-time setting in which we can no longer interpret the time constant as parameter of continuous-ting time system. Instead, interpret F as adaptive gating mechanism, as discussed in Section 4.
setting
If we let
W.. Wh. + b. y- where is determined by the gate, then it follows that
Th. ays) @ (1 tanh2 (y,)) Why tanh(y,) (o(ys) (1- W
Oh.11
[1 +[--o(yr) @ tanh2 (yh) Whh. + tanh(yy) @ [1 (a(y)) o oly)) Wxy)l
A.5 MOTIVATING SPECTRAL ANALYSES RECURRENT MODELS
In this section, we motivate the analyses employed in this paper in order to analyze the dynamics of the various recurrent networks we examined. In particular, we will motivate the low-rank, sparse parameterization of W from the perspective of modulating the spectral radius and spectral norm (and more generally the eigenspectrum and singular value spectrum) of the recurrent weights at initialization and then extend this line of reasoning the analyses performed the trained models. For specifics on the implementation details and computation performed for these analyses, refer to A.15.5.
17

---

Published as a conference paper at ICLR 2024
A.5.1 RECURRENT MEMORY HORIZON
We leverage our parameterization of W Wree (r, s) as function of rank and sparsity in order mod- ulate the spectral radius, spectral norm and singular value spectrum of the recurrent weights.
Here, we argue that the spectral radius of (ree s) is a pertinent measure for the memory horizon of the network across time. We will demonstrate why in the context of an RNN as the computations are most tractable under this functional form. partieular, in an RNN, the recurrent gradient in 8h, Jr ah, reflects how much the network's hidden state is updated based on information from the past. A higher recurrent gradient suggests the network is paying more attention to distant inputs during training, while a lower recurrent gradient implies less reliance such distant information.
The relevant quantity in backpropagation through time in an RNN is
Th. Oh,
Oh,
II 1212k
II (1 tanh (h:-1)) 3 Wh. 12
0h
(refer to A.4 for details on the derivation of RNN recurrent gradient). We can re-express the element- wise product in the expression for ah. as the product of two matrices follows:
Th.-1
diag((1 (anh"(h.-1))
where the (1) operator constructs a diagonal matrix where the elements of are placed along the diagonal. Then, we have that
Oh:
1 diag tanh (h:-))) [Wall
Th,-1
by the sub- sub-multiplicativity of a matrix norm. As a function of rank and sparsity, if assume that [diag[(1 tanh? (hx-1))] is reasonably unaffected by the changes in Wree(r, 8) at initialization, then variation in the bound arises only from [|Wr
Drawing from the analysis presented in Pascanu (2012), if we assume that the relevant variation in the bound as function of rank and sparsity comes only from ||Wil and that Wn, PDP-1 is diagonalizable, then we approximately have that
0h; 0h
<
diag((1 - tanh tanh? (hah))) WR R (W,, | 1212
II
PD
where P denotes a matrix of eigenvectors and D denotes a diagonal matrix with the eigenvalues on the diagonal. It follows that for sufficiently large l the eigenvalue of leading magnitude. One can argue this more formally using the power iteration method, details of which can be found in Pascanu al. (2012). We further note that this notion of modifying the spectral radius in order to modulate attention across the time in recurrent networks not a new one. One prominent example can be found in echo state networks which like our parameterization induces sparsity in the recurrent weights in order to control how much attention the model pays to distant inputs.
k. PD is dominated by
Extending this mathematical argument to LSTMs and CfCs is less straightforward given the more intricate gating mechanisms present the functional forms each model. While reason- able to hypothesize that decreasing the spectral radius in these architectures will result in faster decay of gradients across time, it is unclear how fast/slow this decay is relative to the analysis pre- sented above for RNN. Furthermore, the assumption made in the argument above for gradient decay in RNNs rested upon ignoring the portion of the gradient influenced by the hidden state: |diag[[1 [(1 - tanh? could potentially learn hidden-state vectors of small magnitude, causing the gradients to decay even faster. And this assumption is even less reasonable in LSTMs and CfCs after training, again due to the nuanced gating present in each architecture.
(hanh((h;1))) (h:-1))) ))]l| This assumption becomes less reasonable after training as the network
Thus, while we still examine the spectral radii each architecture, we also conduct a more nuanced recurrent memory analysis on all the architectures by computing the norm of the recurrent gradients
18

---

Published as a conference paper at ICLR 2024
backpropagated through time. In particular, if we let Jr an, denote the recurrent Jacobian, then we can take the cumulative product of the Jacobians as follows:
[Jz,
JJJ., J2,
J2J1]
Taking the norm of each of the matrices in the list above gives us a concrete measure of the extent to which a given model attends to its past observations as a function of time. This enables us to evaluate the effect of the recurrent weight's spectral radius on the
of given architecttre which allows us to make comparisons not only within a given architecture across ranks and sparsities, but also across the different recurrent architectures we analyze (i.e. RNN vs LSTM vs CfC).
A.5.2 ROBUSTNESS UNDER DISTRIBUTION SHIFT
In the last section, we motivated the parameterization of (r, from the perspective of the spectral radius and the implications it has on the attention profile of the network across time. In this section, we will motivate the parameterization instead from the perspective of the singular value spectrum of Wrec (r, s) and understand the implications it has on the robustness of the network under distribution shift.
Before motivating the analysis of the singular value spectrum, we first clarify the nature of the distribution shifts in the context of this work. Here, distribution shifts are applied to the input image, which is then fed through a set of convolutional layers before entering the recurrent portion of the network (A.11). The perturbed input denoted by xf then corrupts the hidden state hy via robustness the update rule for the hidden state specified by the recurrent model. To simplify analysis, we do not consider the convolutional layers and instead restrict the scope of our analysis to the input weights and recurrent weights of the recurrent network. Since both œ; and h, are affected by distribution shift, in practice we care about the robustness induced by both Winp and Wree. However, recall that our parameterization only constructs W Wreer,, s) function of rank and sparsity, while maintaining the structure of W imp as full-rank and fully-connected. We note that we did try extending the parameterization as a function of rank and sparsity to inpp. but found that doing so was quite detrimental to performance (results not shown). Thus, at initialization, we only modulate the robustness across the recurrent axis via the recurrent weights. Nonetheless, we still examine the spectral properties of the input weights after training to understand whether modulating the rank and sparsity of the recurrent weights implicitly affects the input weights during learning.
Now, let more formally understand why modulating the spectral properties of the weights has an impact on robustness. Consider a perturbation e applied to the hidden state h; such that |/e]| = 1 (in practice recall that the perturbation is actually applied to I; which later corrupts he, but apply the perturbation directly to h2 to simplify our argument). Under distribution shift, we care about robustness against all possible perturbations since we can imagine e being sampled from some arbitrary distribution over unit vectors. In particular, we want to understand how Wrech, differs from W Weee(h, (h, e). To motivate the importance of the singular value spectrum, consider SVD on Wree as follows:
Wrea (h, + +e) = UEV +e)= +UEV
Note that the quantity we care about for measuring robustness is UΣVT e. Since VT is unitary matrix, V = e has the same magnitude as e. Similarly, U is also a unitary matrix, so the only transformation that affects the magnitude of e the scaling performed by the singular value matrix Σ. Now. we can reduce the robustness of Wree to two things: the magnitude of the expansion induced by Σ and the effective number of directions in which is expanded.
First, we will discuss our approach to measuring the magnitude of the expansion induced by the recurrent weights. A canonical measure of the expansion induced by a matrix given by the spectral norm which is equivalent to the leading singular value. In this case, the spectral norm of W tells us that (he + e) deviates most from We if is the norm-1 vector parallel to the singular vector corresponding to the largest singular value of Wrech,, We can interpret this as a worst-case (i.e. adversarial) analysis of robustness. The spectral norm of Wree is further tied to robustness via its relationship to the Lipschitz constant of the full network the function learned by the network. In particular, a trivial upper bound for the global Lipschitz constant of neural network is computed by multiplying the spectral norms of all the weights in
measure of the smoothness of
19

---

Published as a conference paper at ICLR 2024
the network (Szegedy et al., 2014). However, this has been shown in many cases to be poor proxy for network robustness due to the looseness of the bound (Huster et al., 2018). Unfortumately, since computing tight bounds for network-wich Lipschitz constants is NP-hard (Scaman & Virmaux, 2019), we maintain that it is reasonable to assume that a lower spectral norm in the recurrent weights results in lower global Lipschitz constant at initialization. Of course, it is possible that during training, the spectral norm of other weights in the network increase and potentially counteract a decrease in the spectral norm of the recurrent weights induced at initialization. In practice, since the distribution shift is applied to the network input, it is also pertinent to analyze the input weights W Wimp in the recurrent module. One thing to note is the limitation of this analysis when making comparisons across architectures. In particular, recall that we have made the assumption that within given architecture, across ranks and sparsities, models that have input and recurrent weights with lower spectral norms tend to express functions with lower I Lipschitz constants (in which case perturbation applicd to the input would have less of an effect on the output). However, this assumption becomes less reasonable across architectures given the fact that the functional form varies significantly across RNNs, LSTMs and CfCs. Optimally, we would actually compute the Lipschitz constants in order to make such comparisons more viable, however we are unable to do this given the NP-hardness: of the problem. Hence, while we still aim to make comparisons across architectures via an analysis of the spectral norm of the weights, is important acknowledge the potential limitation in this approach.
Next, we will discuss our approach to quantifying the effective number of directions which is expanded. Note that in order to remain robust against the many potential directions the perturbation vector e can lie in, it is desirable for the singular values of rec to decay rapidly, as this implies that only few singular values (i.e. only a few directions corresponding to the top singular vectors) contribute significantly to the transformation. Hence, we analyze the decay of the sorted spectrum of singular values normalized by the leading singular value proxy for the effective numbers of dimensions that contribute to the transformation of e by Wren. Again, it is possible that during training. the singular value spectrum of the input weights Winp counteracts the prior induced on the recurrent singular value spectrum at initialization and thus also must be examined. As with the spectral norm analysis discussed above, we also acknowledge the potential limitation in making comparisons of spectral decay across different recurrent architectures.
While the decay of the singular value spectrum provides good proxy for the directionality com- ponent of robustness, it does so only for a single point in time. In actuality, we want to understand the directions the hidden state evolves in across the entire trajectory of our model in order to have robustness measurement that takes into account all points in time. To do so, we perform a dimen- sionality analysis on the hidden state-space trajectories collected over the course of a simulation in the closed-loop environment. In particular, we are interested in three state-spaces; the recurrently- driven state-space, input-driven sate-space and full state-space. A canonical state-space trajectory analysis collects the h, over time (i.e. the full state-space) and performs PCA in order to measure the effective dimensionality of the trajectory. We extend this analysis by decomposing the full state- space into the portion driven by the recurrent weights, Wrech, and the portion driven by the input weights W W. This allows us to disentangle the effects of the proposed parameterization of the recurrent weights into its individual effects on the recurrent and input state-spaces. Furthermore, note that this dimensionality analysis enables us to make viable comparisons across recurrent ar- chitectures whereas this is potential limitation of analyzing only the decay of the singular value spectra in the recurrent and input weights.
So, we have now motivated the spectral norm and decay of the singular value spectrum of both the recurrent and input weights from the perspective of constructing a model that is robust to distribution shift. This was done under the framework of assuming a perturbation applied to æ; which also results in perturbation applied to h; which affects the output decision of the model at time step t. However, we can also ask how does this perturbation affect the model into the future: namely, how does the perturbation applied to h, affect h4- for t* > t. We can answer this question by understanding how information is propagated through the network across time. But note that this is precisely the intention of analyzing the time-horizon of the recurrent memory discussed in A.5.1. So, not only does modulating the recurrent memory of the model serve the purpose of enforcing a short-horizon temporal prior necessary to model the short-term causality inherent to closed-loop environment, it also makes the network more robust across the time dimension. Thus, we have two measures of robustness: one at the current point in time and another for all time points into the future.
20

---

Published as a conference paper at ICLR 2024
A.6 THEORETICAL ANALYSIS OF SPECTRAL RADIUS AND SPECTRAL NORM INITIALIZATION
In this section we provide proofs for the spectral radius and spectral norm, p PWW), ||]] respec- tively, for connectivity matrices W at initialization. For clarity, we iterate that p p(W) max; [a,]. i.e. the largest norm of eigenvalues of W, and [W] = max; F; the largest singular value of W. We consider sparse networks with Glorot uniform initialization and orthogonal initialization in ap- pendices A.6.2 and A.6.4, respectively, and orthogonal low-rank matrices in appendix A.6.3. Note that in our experiments, we only consider networks with orthogonally initialized recurrent weights which we motivate further in A.10. Also, note that in the cases where we are unable to provide proof, we still perform an empirical analysis.
A.6.1 GLOROT UNIFORM AND SPARSE
Glorot Uniform Sparse
Spectral radius W. P(W)
Spectral norm of W. [|W]
10
2.0
Theoretical value
1.5
0.4
0.5
0.2
0.0
0.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.3
1.0
Sparsity, 5
Sparsity, s
Figure 10: Spectral radius and spectral norm of uniform-sparse matrices as a function of sparsity, 8, for matrices initialized with Glorot uniform initialization, with n 512. We additionally plot the theoretical predicted value for large n.
We consider weight matrices W with dimension n n and consider the limit as n + co. Recall that Glorot uniform initialization scheme initializes weights with entries iid Wij ~ resulting in an entry-wise variance of L. We generate sparse matrix by Vn
Unif v3
element-wise multiplying matrix, T sampled from the Glorot uniform initialization by spar- sity map M, where My Bernoulli(p
s), with being the sparse factor. Our final weight matrix is given by W W @ M. For sparsity s, the resulting variance of entries given by I=s, As the entries of W are all iid, we can apply the Kürko-Ginibe circular law for large n, which states that the eigenvalues of W converge to a uniform disk in the complex plane with radius VI VI-S. s, so we expect p(s) = VI-s. s. This analysis follows closely with that of Herbert & Ostojic (2022).
For [W], we apply the Marchenko-Pastie law, which states that the distribution of eigenvalues of AAA values converges to (AAT(A) = 2 (A+ (JA-A](A-2-) 1xe(A .21]. with A o² (1 1)2, A has entries iid from distribution with zero mean and variance o For our setting we let A VnW, so o² 1 We see that the maximal eigenvalue of AAT is thus X = 4(1 s), and the upper bound for the largest singular value of W is || 2/ - -s). We observe good empirical agreement with these values in Figure 10.
A
A.6.2 ORTHOGONAL AND SPARSE
In the orthogonal-sparse case, we cannot apply the same arguments in appendix A.6.4, as the entries of an orthogonal matrix are no longer iid. However, we note that the entries of orthogonal matrices approximately Gaussian when considered individually (Zyczkowski & Sommers, 1999). Thus, we expect with high sparsity, the correlations between entries break down and the entries of the matrix behave iid. Thus, expect for large values of s, p(W) and ||] to have the same behavior
21

---

Published as a conference paper at ICLR 2024
Orthogonal + Sparse
Spectral radius of W. p(W)
Spectral norm of W. [[W]
o(W)
[[
1.0
Thaoretical
0.0
0.2
0.6
0.8
0.0
0.6
Sparsity,
Sparsity,
Figure 11: Spectral radius and spectral norm of orthogonal- matrices function of sparsity, 8, with n = 512. We additionally plot the theoretical predicted value for large n for the uniform- sparse initialization. We see that orthogonal-sparse matrices behave like uniform-sparse ones at high sparsities.
in appendix A.6.4. We verify this empirically in Figure 11, where see that for large values of s, P(W) ~ vT- and ||W]| = 2v1-a s, as with appendix A.6.4. Interestingly, || || is non- monotonic in s. Calculating the exact forms of p(W) and W|| is interesting direction for future work.
A.6.3 ORTHIOGONAL AND LOW RANK
Orthogonal Low Rank
Spectral radius of W, p(W)
Spectral norm of W. [|W]|
Theoretical
1.04
0.8
1.02
p(W)
1.00
0.4
0.98
0.96
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Rank ratio,
Rank ratio,
Figure 12:
Spectral radius and spectral norm of orthogonal-low-ranl matrices as a function of rank ratio, r, with n = 512. We additionally plot the theoretical predicted value for large n.
Recall that generate low rank connectivity matrix W, we first generate orthogonal matrix W then perform an SVD of it W = USV" = Σn Σ u,"," and then truncate it at rank to get W = > u. with & S n. For ||[] note that because all the singular values are 1, truncating the SVD does not change the maximum singular value so ||] = 1 for all rank ratios
r
and V I. It is sufficient to consider this particular SVD, other SVDs correspond to arbitrary rotations of U and V. which will not affect the final eigenvalue distribution in expectation. For this particular SVD, W [ W 2 ...., W .0.....00 0). i.e. take the first & columns of W and replace the remaining entries with Due to the large zero block this matrix, W has the same non-zero eigenvectors and eigenvalues the k k principal submatrix of W W 1:k.1:kk Zyczkowski &
For p(W), ), we note that one valid SVD of W is W W with U = 5 S
Sommers (1999) studies the eigenvalues of the principal submatrix of random orthogonal matrices
for eigenvectors need concatenate the remaining zeros have the correct dimension
22

---

Published as a conference paper at ICLR 2024
and shows that the distribution of these eigenvalues has mean E[A] V for rank ratio r Furthermore, Var(X]
fixed r (Zyezkowski & Sommers, 1999), have p(W)= = /r for large n. Empirical verification of this is provided in Figure 12 for = 512.
A.6.4 GLOROT UNIFORM AND Low RANK
Glorot Uniform Low Rank
Spectral radius of W, p(W)
Spectral norm of W. [[W]]
1.0
2.0
1.5
P(W)
0.6
1.0
0.4
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Rank ratio,
Rank ratio,
Figure 13:
Spectral radius and spectral norm of uniform-low-rank matrices function of rank ratio, r, with n = 512.
As with the orthogonal-low-rand case, low-rank approximation does not affect the spectral norm of W when using Glorot Uniform initialization scheme. Proving results for the spectral radius of low-rank uniform initialization more difficult due to the more complex eigenstructure of uniformly initialized matrices compared to orthogonal ones. Thus, we leave it to future work to formally prove the relationship between rank ratio and spectral radius. We observe empirically Figure 13 that spectral radius is increasing with rank ratio, like with the orthogonal-low cahhogonll-low-radd -ank case.
A.7 SINGULAR VALUE SPECTRUM AND EIGENSPECTRUM AT INITIALIZATION
Recall in A.6, we provided theoretical results for the spectral norm and spectral radius of randomly initialized matrices as function of rank and sparsity in order to provide theoretical guarantees regarding the proposed parameterization of the recurrent weights Wree(T, s) at initialization. As explained in A.5.2, also motivated the parameterization with respect to the entire the spectrum of singular values due to its connection with robustness. Our theoretical results regarding the spectral norm a function of rank and sparsity do not extend the full spectrum of singular values. So, here we provide empirical evidence regarding the decay of the singular value spectrum of the recurrent weights function of rank and sparsity (we also provide plots of the decay of the eigenspectrum of the recurrent weights to demonstrate the similarity between the two related, yet distinct spectra).
A.7.1 ORTHOGONAL INITIALIZATION
Here, we examine the singular value spectrum for the orthogonally initialized recurrent weights across various ranks and sparsities. We note that as sparsity increases, the rate at which the sin- gular value spectrum decays decrease (Figure 14). As discussed in A.5.2, this raises the effective dimensionality of the transformation induced by the recurrent weights on the hidden-state vector. Similarly, as rank increases, the rate at which the singular value spectrum decays also decreases (Figure 14). This is interesting increasing sparsity removes parameters from the model, whereas increases rank adds parameters to the model. Hence, we observe that pruning by increasing sparsity and pruning by decreasing rank distinct with respect to the decay each induces in the singular value spectrum of the recurrent weights. Furthermore, note that we observe the exact same patterns with respect to the recurrent eigenspectra well (Figure 15).
23

---

Published as a conference paper at ICLR 2024
16
full
W (5 0.0)
Wer (s 0.2)
Wsec 0.5)
W (5 0.8)
2.0
2.0
2.0
2.0
1.0
1.0
1.0
1.0
0.0
0.0
0.0
20
40
20
sorted
sorted
sorted
Figure 14: Singular value spectra of orthogonally initialized recurrent weights across various ranks and sparsities sorted decreasing order.
rank
27 full
W/e: (5 0.0)
Wwx 0.2)
Wie (s 0.5)
Wie: (5 0.8)
3.5
[A.]
[A/] 1.0
[A]]
[A]
0.5
0.5
0.0
0.0
20
40
Figure 15:
Eigenspectra of orthogonally initialized recurrent weights across various ranks and spar- sities sorted in decreasing order.
A.7.2 GLOROT UNIFORM INITIALIZATION
Here, we examine the singular value spectrum for the Glorot uniform initialized recurrent weights across various ranks and sparsities. The decay patterns observed with respect to the singular value and eigenvalue spectra the orthogonally initialized weights hold here as well (Figures 16, 17).
rank
5
16 27 fill
W (5 = 0.0)
Wre: (5 = 0.2)
Wree (5 0.5)
Wiee (S = 0.8)
1.5
1.5
1.5
3.5
0)
0
0.5
0.5
0.5
0.5
0.0
0.0
2.0
0.0
sarted
sorted
sorted
sorted 1
Figure 16: Singular value spectra of Glorot uniform initialized recurrent weights across various ranks and sparsities sorted in decreasing order.
rank 1
16 27 full
Wrec (s 0.0)
Wrec (s = 0.2)
Wree (s 0.5)
Wre: (s 0.8)
AM11
AA/]
A1
[
0.0
0.0
0
40
60
20
60
40
20
60
sorted
Sorted
)
sorted
sorted i
Figure 17: Eigenspectra of Glorot uniform initialized recurrent weights across various ranks and sparsities sorted in decreasing order.
24

---

Published as a conference paper at ICLR 2024
A.8 TRAINED NETWORK DYNAMICS RESULTS
In this section, we provide plots for various analyses we conducted on the trained networks that were not included the main portion of the paper.
sparsity
RNN Wiee
RNN Weep
RNN Went
expleined ver.
0.55
0.75
0.50
0.6
0.4
0.45
0.40
0.0
16 full 0.60
Figure 18: Effective dimensionality of RNN state-space trajectories across various ranks and spar- sities. Note that the dynamics of RNNs are higher-dimensional than the dynamics we observe in CfCs. This, alongside the disparity in spectral norm, offers intuition as to why we find that CiCs tend to outperform RNNs under distribution shift.
a)
b)
GRU
18
f6
50
10g 1C.1
-25
-15
-20.
2.5
54
75
160
25
56
75
10.
25
50
75
18.0
25
5.0
75
220
Figure 19: a) Spectral radius recurrent weights W Wreerr 8) trained RNN networks across ranks and sparsities. b) Frobenius norm of recurrent gradients G, as function of time. Note that the norms are plotted in log space and translated to decay from 0 to enable easier comparison. For details the computation, refer to A.1I.
sparsity - 0.0 0.2 - -
CFC
LSTM
RNN
3.0
Omax 2.5
2.5
Tyax 2.5
rank
rank
rank
Figure 20: Spectral norm of the input weights across models, ranks and sparsities.
25

---

Published as a conference paper at ICLR 2024
rank
5 16
27
64
CFC (S 0.0)
CTC (s = 0.2)
CfC (s = 0.5)
CfC (s
0.8)
0:
01 Oma
01 Omeo
0 mma
20
40
60
20
40
60
20
40
60
20
40
60
sorted
sorted
sorted
sorted
0.0)
0.2)
0.5)
0.8)
1.0
0 mmax 0.5
Unoo 0.5
Omax 0.5
0.0
0.0
80
20
sorted
sorted
softed
sorted
RNN (s 0.0)
RNN (s 0.2)
RNN (s 0.5)
RNN 0.8)
1.0
1.0
1.0
1.0
0.5
Omar 0.5
Dmjo 0.5
Gmax 0.5
0.0
0.0
0.0
0.0
sorted
Sorted
sorted
sorted
Figure 21: Decay of the recurrent singular value spectrum across models, ranks and sparsities as measured by normalizing the sorted spectrum by the spectral norm. We find that the prior induced at initialization holds at convergence: namely, as sparsity increases the rate of decay of the spectrum decreases and as rank decreases the rate of decay of the spectrum increases.
rank /
5 16
27
64
CfC (s = 0.0)
CfC (s = 0.2)
CfC (s = 0.5)
CfC (s = 0.8)
0
0 mmar
0) Daar
0 Fmax
60
60
20
40
20
80
00
20
40
60
sorted
sorted /
sorted
sorted
0.0)
0.2)
0.5)
= 0.8)
1.0
1.0
0
Omar
Daaa 0.5
Smax 0.5
0.0
0.0
0.0
20
sorted
Sorted
sorted
Sorted
RNN (s 0.0)
RNN (s 0.2)
RNN (s 0.5)
RNN (s 0.8)
1.0
1.0
1.0
1.0
Gwaw 0.5
Orar 0.5
Oreo 0.5
Tmax
0.5
0.0
0.0
0.0
0.0
sorted
Sorted
sorted
sorted
Figure 22: Decay of the input singular value spectrum across models, ranks and sparsities as mea- sured by normalizing the sorted spectrum by the spectral norm. We find that the decay is quite similar across models, sparsities and ranks, which aligns with many of our findings suggesting that the input weights are little affected by changes in the rank and sparsity the recurrent weights.
26

---

Published as a conference paper at ICLR 2024
rank.
CfC 0.0)
CfC (5
CfC 0.5)
CfC (5
2-
sarted
sorted
sorted
sarted
LSTM 0.0)
LSTM (5 0.2)
LSTM 0.5)
LSTM (5 0.8)
07
00
20
50
40
20
sorted
Sorted
RNN (s 0.0)
RNN (5 0.2)
RNN (s 0.5)
RNN (5 0.8)
0
0
0,
0
20
40
50
20
40
60
20
40
60
20
40
sarted
sorted
sorted
sorted
Figure 23: Singular value spectrum of the recurrent weights of the trained models across various ranks and sparsities sorted in decreasing order.
rank.
CfC 0.0)
CfC (s
CfC 0.5)
CFC (5 0.8)
2-
2
Sorted
sorted
sorted
Sorted
LSTM 0.0)
LSTM (s 0.2)
LSTM 0.5)
LSTM (s 0.8)
07
0,
20
50
20
60
40
RNN [s 0.0)
RNN (S 0.2)
RNN (s 0.5)
RNN (5 0.8)
07
07
0:
0
20
40
50
20
40
60
20
40
60
20 Sorted
40
sorted
Sorted
sorted
Figure 24: Singular value spectrum of the input weights of the trained models across various ranks and sparsities sorted in decreasing order.
27

---

Published as a conference paper at ICLR 2024
rank
27
64
CfC (5 0.0)
CfC (s 0.2)
CfC (5 = 0.5)
CfC (5 0.8)
1.5
1.5
1.5
1.5
M 1.0
1/.00
1.//1.0
[A,
0.5
0.5
0.5
0.5
0.0
20
40
60
20
40
20
40
60
20
40
sorted
sorted
sorted
sorted
LSTM 0.0)
LSTM (s = 0.2)
LSTM (5 0.5)
LSTM (5 0.8)
1.5
1.5
1.5
1.11.0
1/.00
1.
M.
0.0
0.0
20
40
60
20
40
20
60
40
sorted /
sarted
sorted
sorted
RNN
RNN 0.2)
RNN 0.8)
1.0
W/
1.0
|
0.5
0.5
0.0
0.0
0.0
0.0
sorted
sorted
sorted
sorted
Figure 25: Eigenspectrum of the recurrent weights of the trained models across various ranks and sparsities sorted in decreasing order of magnitude.
model
cfc
1dbm
Sppeained vBc.
explaimed var.
expleined ver.
10 S.
espleiees
orplained
48
20
20
22
40
20
43
60
20
40
00
PC
PC
PC
1,15 0.2
F= 5, 0.2
F= 16, S 0.2
r= 27, 0.2
[= full,, 5 0.2
explaned ver.
empllai necl war
explained
12 1
explained
PC
PC
PC
PC
FC
1, 0.5
5, 0.5
16, 0.5
27.
0.5
full, 0.5
erplained var.
exploimed Velt 12 1
expleined wea..
3.0
Erploined
2
1: R.
02
stolained var.
explaiined vaar
explained
explained
emplained
66-
&
02
60
35
10
20
43 60
20
PC
PC
PC
PC
Figure 26: Dimensionality analysis of full state-space trajectories collected from the online, closed- loop in-distribution simulation analyzed in Figure 6. Here, plot the explained variance curve over all principal components to provide more complete view of the effective dimensionality of the trajectories, opposed only the top PCs.
28

---

Published as a conference paper at ICLR 2024
A.9 FULL RESULTS
A.9.1 ONLINE AND OFFLINE PERFORMANCE
Here, we provide offline validation/test loss, online in-distribution rewards and online rewards under distribution shift for each Gym environment (Brockman et al., 2016) that we ran experiments on: Seaquest, Alien and HalfCheetah. For each environment, we considered 5 types of models: CICs, RNNs, LSTMs, GRUs and CNNs. For each the recurrent architectures we considered models of various different ranks and 4 different sparsities. Full details on the experimental setup given in A.
CfC
LSTM
RNN
GRU
CNN
1.02
sparsity
sparsity
sporsity
sparsity
sparsity
0.5
rewarD
02
16 rank
27
27
16
27 1.11
1.5
16
27
1.5
27hul
rank
rank
rank
rank
CrC
LSTM
RNN
GRU
CNN
12.8
0.8
sparsity
0.5
Sparsity
0.5
Sparsity
0.5
sparsity
0.5
sparsity
reward
0.82
0.2
0.2
02
0.66
0.0
0.30
LSTM
RNN
GRU
CNN
D0
0.0
D.
0.
0.020
Sparsity
0.5
sparsity
0.5
Sparsity
0.5
Sparsity
05
sparsity
0.588
0.644
0372
0.2
0.
0.556
0.0
0.0
0.0
0.0
155
27
fuil
1.5
27
27hul
rank
rank
rank
Figure 27: Online and offline performance of recurrent networks under different ranks and sparsities in the Seaquest environment. Note that online performance of Seaquest was presented in the main portion of the paper, but here include it alongside the offline losses evaluated validation/test set. a) In-distribution rewards the online, closed-loop setting normalized by the rewards obtained by the expert in-distribution. b) Rewards averaged across 5 distribution shifts, normalized by the wards obtained by the expert under distribution shift. c) Loss of model evaluated on a validation/test
29

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0.2
0.5
0.8
CIC
1
0.86 (0.03)
0.93 (0.04)
0.88 (0.03)
0.8 (0.05)
5
0.81 (0.06)
0.86 (0.05)
0.79 (0.06)
0.71 (0.06)
16
1.01 (0.06)
0.92 (0.06)
0.79 (0.05)
0.9 (0.05)
27
0.91 (0.06)
0.97 (0.04)
0.88 (0.06)
0.83 (0.06)
full
0.97 (0.04)
0.76 (0.08)
0.7 (0.05)
0.76 (0.06)
LSTM
0.99 (0.04)
0.83 (0.04)
0.9 (0.06)
0.92 (0.05)
5
0.94 (0.03)
0.94 (0.05)
0.88 (0.06)
0.88 (0.05)
16
0.94 (0.04)
0.94 (0.07)
0.9 (0.06)
0.9 (0.07)
27
1.0 (0.03)
0.99
0.9 (0.06)
0.88 (0.06)
full
0.9 (0.05)
0.98
0.9 (0.06)
0.9 (0.06)
RNN
0.94 (0.06)
0.92 (0.06)
0.89 (0.05)
0.9 (0.05)
5
0.96 (0.05)
(0.05)
0.94 (0.06)
0.92 (0.06)
16
0.94 (0.06)
(0.
0.88 (0.06)
1.07 (0.02)
27
0.88 (0.05)
0.83 (0.05)
1.01 (0.06)
0.91 (0.03)
full
0.92 (0.04)
0.92 (0.04)
0.92 (0.05)
0.86 (0.06)
GRU
0.81 (0.06)
1.04 (0.04)
1.0 (0.05)
(0.04)
5
0.85 (0.04)
0.84 (0.06)
0.85 (0.06)
0.91 (0.04)
16
0.92 (0.06)
0.92
0.78 (0.05)
0.94 (0.05)
27
0.79 (0.05)
0.95 (0.06)
0.79 (0.06)
0.84 (0.05)
full
0.82 (0.04)
0.95 (0.06)
0.93 (0.06)
0.94 (0.05)
CNN
1
5
16
27
full
0.77 (0.05)
Table 1: Mean episodic in-distribution rewards in Seaquest environment normalized by rewards obtained by expert policy. Normalized rewards are given +1 SE which is shown in parentheses.
30

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0
0.2
0.5
0.8
CfC
1.12 (0.05)
1.03 (0.03)
0.95 (0.02)
0.93 (0.03)
1.12 (0.03)
1.26 (0.03)
0.83 (0.02)
0.76 (0.03)
16
0.76 (0.03)
0.85 (0.03)
0.73 (0.03)
0.8 (0.03)
27
0.95 (0.02)
0.91 (0.03)
0.74 (0.02)
0.77 (0.03)
full
1.28 (0.03)
0.95 (0.03)
0.59 (0.04)
0.98 (0.03)
LSTM
0.77 (0.03)
0.68 (0.04)
0.76 (0.03)
0.91 (0.02)
0.67 (0.04)
0.71 (0.04)
0.82 (0.03)
0.89 (0.02)
16
0.66 (0.05)
(0.04)
0.62 (0.05)
0.62 (0.05)
0.7 (0.04)
0.63 (0.05)
0.74 (0.04)
0.66 (0.05)
full
0.58 (0.05)
0.67 (0.04)
0.69 (0.04)
0.78 (0.05)
RNN
0.78 (0.02)
(0.02)
0.7 (0.04)
0.87 (0.02)
0.78 (0.02)
(0.02)
0.7 (0.03)
0.83 (0.03)
16
0.96 (0.02)
0.76
0.73 (0.03)
0.73 (0.03)
0.71 (0.04)
0.68 (0.04)
0.62 (0.04)
0.71 (0.04)
full
0.51 (0.03)
0.61 (0.04)
0.61 (0.04)
0.78 (0.04)
GRU
0.8 (0.05)
0.67 (0.05)
0.61 (0.04)
0.67 (0.04)
0..400.05)
(0.05)
0.76 (0.04)
0.66 (0.04)
16
0.76 (0.04)
0.66 (0.04)
0.62 (0.04)
0.69 (0.04)
27
0.73 (0.05)
0.69 (0.04)
0.67 (0.04)
0.73 (0.04)
full
0.78 (0.04)
0.76 (0.04)
0.7 (0.04)
0.6 (0.04)
CNN
16
27
full
0.58 (0.04)
Table 2: Mean episodic rewards under distribution shift in Seaquest environment normalized by performance of the expert policy under distribution shift. Rewards are given #1 SE which is shown in parentheses.
31

---

Published as a conference paper at ICLR 2024
CFC
LSTM
RNN
GRU
CNN
0.
sparsity
sparsity
0.
sparsity
sparsity
0.5
0.5
0.67
0.2
02
0.2
0.2
0.2
0.52
0.0
0.37
rank
rank
rank
rank
rank
b
CfC
LSTM
RNN
GRU
CNN
sparsin
0.5
0.5
sparsity
sparsity
0.5
0.5
0.68
0.2
0.51
0.40
20
15
rank
rank
rank
rank
rank
LSTM
GRU
CNN
0.8
0.8
0.8
0.8
0.8
0.640
sparsity
0.5
sparsity
0.5
sparsity
0.5
sparsity
0.5
sparsity
0.5
0.626
0.12
0,598
0.0
0.0
0.0
0.0
16
2M
16 rank
27
16
27
1.5
16
27
15
16
27h
rank
rank
Figure 28: Online and offline performance of recurrent networks under different ranks and sparsities the Alien environment. a) In-distribution rewards the online, closed-loop setting normalized by rewards obtained by the expert in-distribution. a) In-distribution rewards in the online, closed-loop setting normalized by the rewards obtained by the expert in in-distribution. b) Rewards averaged across 5 distribution shifts, normalized by the rewards obtained by the expert under distribution shift. c) Loss of model evaluated on a validation/test set.
32

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0
0.2
0.5
0.8
CIC
0.59 (0.07)
0.81 (0.14)
0.95 (0.11)
0.83 (0.19)
0.52 (0.1)
0.84 (0.14)
0.66 (0.13)
0.98 (0.14)
16
0.94 (0.14)
0.45 (0.05)
0.42 (0.05)
0.47 (0.04)
27
1.1 (0.18)
0.94 (0.19)
1.01 (0.17)
0.58 (0.08)
full
1.2 (0.21)
(0142)
0.47 (0.07)
0.7 (0.11)
LSTM
(0.12)
(0.1))
0.92 (0.12)
0.81 (0.12)
0.9 (0.12)
(0.15)
0.8 (0.15)
1.1 (0.18)
16
0.87 (0.13)
0.9 (0.12)
1.09 (0.13)
0.84 (0.16)
27
1.11 (0.14)
(0.15
(0.14)
0.92 (0.12)
full
(0.14)
0.95 (0.14)
L.15 (0.12)
0.88 (0.17)
RNN
0.77 (0.12)
0.73 (0.11)
0.8 (0.1)
0.8 (0.11)
(0.11)
(0.1I)
0.82(0.1)
0.78 (0.14)
16
0.54 (0.1)
(0.11)
0.97 (0.06)
0.76 (0.11)
27
0.92 (0.11)
(0.12)
0.75
0.71 (0.11)
full
0.97 (0.09)
(0.13)
0.97 (0.11)
0.57 (0.1)
GRU
0.81 (0.11)
(0.11)
0.71 (0.11)
0.52 (0.11)
(0.09)
0.87 (0.12)
0.83 (0.12)
0.44 (0.06)
16
0.8
(0.1))
(..1)
0.64 (0.12)
0.54 (0.1)
0.67 (0.11)
0.55 (0.1)
0.79 (0.11)
full
(..1)
(0.12)
0.77 (0.14)
0.89 (0.11)
CNN
16
27 full
0.64 (0.09)
Table 3: Mean episodic in-distribution rewards in Alien environment normalized by rewards ob- tained by expert policy. Normalized rewards are given +1 SE which is shown in parentheses.
33

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0.2
0.5
0.8
CfC
0.81 (0.04)
0.52 (0.02)
0.62 (0.03)
0.77 (0.05)
1.1 (0.02)
0.78 (0.02)
0.67 (0.02)
0.51 (0.03)
16
0.8 (0.02)
1.05 (0.05)
0.3 (0.02)
0.55 (0.03)
27
0.48 (0.02)
(0.02)
0.66 (0.02)
0.73 (0.01)
full
0.44 (0.01)
0.41 (0.02)
0.6 (0.03)
0.6 (0.02)
LSTM
0.63 (0.03)
0.58 (0.05)
0.88 (0.05)
0.66 (0.06)
0.87 (0.06)
0.08 (0.0)
0.81 (0.06)
0.69 (0.05)
16
0.81 (0.06)
0.08
0.56 (0.03)
0.8 (0.04)
0.72 (0.06)
0..
0.64 (0.05)
0.63 (0.05)
full
0.65 (0.06)
0.3 (0.01)
0.76 (0.06)
0.66 (0.04)
RNN
0.68 (0.03)
(..45)
0.65 (0.03)
0.62 (0.03)
0.69 (0.06)
(0.04)
0.82 (0.05)
0.62 (0.04)
16
0.65 (0.03)
0.68
(0.05)
0.6 (0.04)
0.8 (0.04)
(0.05)
0.74 (0.06)
0.72 (0.05)
full
0.58 (0.04)
0.77 (0.04)
(0.05)
0.49 (0.02)
GRU
0.1 (0.0)
0.94 (0.05)
0.82 (0.03)
0.71 (0.04)
0.95 (0.05)
00.08))))
0.76 (0.05)
0.77 (0.05)
16
1.04 (0.03)
0.95 (0.05)
0.08 (0.0)
0.69 (0.03)
27
0.73 (0.04)
0.87 (0.05)
0.67 (0.05)
0.82 (0.07)
full
0.69 (0.05)
0.08 (0.0)
0.78 (0.03)
0.86 (0.05)
CNN
16
27
full
0.67 (0.04)
Table Mean episodic rewards under distribution shift in Alien environment normalized by per- formance of the expert policy under distribution shift. Rewards are given +1 SE which is shown in parentheses.
34

---

Published as a conference paper at ICLR 2024
CfC
RNN
GRU
CNN
0.
0.8
0.8
sparail
0.5
Sparsity
0.5
0.5
sparsity
0.5
Sparsity
0.06
reward
0.90
0.2
0.2
0.99
0.0
0.0
0.0
0.0
0.0
27h
rank
CTC
LSTM
RNN
GRU
CNN
0.1
08
Sparsio
0.5
Sparsity
0.5
0.5
sparsity
0.5
sparsity
0.5
0382
02
02
02
0.
0.036
0.0
0.010
rank
CfC
LSTM
RNN
GRU
CNN
0.8
0.8
0.8
0.0
0.34250
0.14220
Sparsity
0.5
sparsity
0.5
0.3
2.5
sparsity
0.34202
0.14778
0.2
022
0.2
0.2-0
0.14154
0.0
0.0-9
0.0
0.14330
27
16
27 hill
Online and offline performance of recurrent networks under different ranks and sparsi- ties in the HalfCheetah environment. a) In-distribution rewards in the online, closed-loop setting normalized by the rewards obtained the expert in-distribution. b) Rewards averaged across distribution shifts, normalized by the rewards obtained by the expert under distribution shift. c) Loss of model evaluated validation/test set.
Figure 29:
35

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0.2
0.5
0.8
CIC
1
0.945 (0.005)
0.929 (0.003)
0.924 (0.004)
0.92 (0.003)
5
0.941 (0.003)
0.926 (0.003)
0.91 (0.002)
0.936 (0.002)
16
0.932 (0.002)
0.941 (0.003)
0.919 (0.002)
0.963 (0.005)
27
0.937 (0.004)
0.874 (0.004)
0.914 (0.003)
0.883 (0.004)
full
0.934 (0.003)
0.816 (0.006)
0.877 (0.002)
0.847 (0.003)
LSTM
1
0.933 (0.003)
0.94 (0.003)
0.927 (0.003)
0.936 (0.003)
5
0.95 (0.003)
0.925 (0.004)
0.937 (0.002)
0.962 (0.003)
16
0.937 (0.003)
0.991 (0.003)
0.968 (0.005)
0.905 (0.003)
27
0.931 (0.003)
0.965 (0.004)
0.945 (0.003)
0.872 (0.003)
full
0.951 (0.003)
0.875 (0.003)
0.889 (0.008)
0.842 (0.009)
RNN
0.943 (0.004)
0.917 (0.007)
0.93 (0.005)
0.903 (0.003)
5
0.941 (0.003)
0.946 (0.003)
0.885 (0.003)
0.921 (0.003)
16
0,941 (0.002)
0.942 (0.003)
0.954 (0.003)
0.922 (0.004)
27
0.944 (0.003)
0.917 (0.003)
0.9 (0.029)
0.861 (0.006)
full
0.938 (0.003)
0.92 (0.002)
0.971 (0.003)
0.924 (0.021)
GRU
1
0.937 (0.002)
0.938 (0.004)
0,932 (0.003)
0.937 (0.002)
5
0.944 (0.002)
0.935 (0.003)
0.938 (0.003)
0.927 (0.003)
16
0.93 (0.003)
0.934 (0.003)
0.927 (0.003)
0.943 (0.003)
27
0.94 (0.002)
0.943 (0.003)
0.926 (0.011)
0.934 (0.001)
full
0.937 (0.004)
0.936 (0.003)
0.949 (0.003)
0.935 (0.002)
CNN
1
5
16
27
full
0.90 (0.003)
Table Mean episodic in-distribution rewards in HalfCheetah environment normalized by rewards obtained by expert policy. Normalized rewards are given +1 SE which is shown in parentheses.
36

---

Published as a conference paper at ICLR 2024
Sparsity
Model
Rank
0.2
0.5
0.8
CIC
0.842 (0.009)
0.919 (0.001)
0.903 (0.003)
0.852 (0.004)
5
0.915 (0.002)
0.869 (0.007)
0.848 (0.004)
0.82 (0.005)
16
0.86 (0.007)
0.838 (0.008)
0.871 (0.003)
0.908 (0.0)
27
0.822 (0.008)
0.858 (0.001)
0.813 (0.006)
0.898 (0.003)
full
0.9 (0.001)
0.891 (0.005)
0.812 (0.006)
0.795 (0.003)
LSTN
1
0.871 (0.001)
0.894 (0.0)
0.86 (0.003)
0.872 (0.004)
5
0.849 (0.006)
0.885 (0.001)
0.88 (0.0)
0.909 (0.003)
16
0.875 (0.002)
0.847 (0.008)
0.83 (0.006)
0.918 (0.001)
27
0.848 (0.006)
0.893 (0.002)
0.882 (0.0)
0.903 (0.003)
full
0.888 (0.001)
0.848 (0.004)
0.905 (0.001)
0.86 (0.003)
RNN
1
0.861 (0.003)
0,854 (0.002)
0.806 (0.009)
0.846 (0.003)
5
0.87 (0.003)
0.841 (0.006)
0.875 (0.001)
0.824 (0.008)
16
0.843 (0.006)
0.841 (0.007)
0.819 (0.009)
(0.005)
27
0,852 (0.004)
0.903 (0.004)
0.844 (0.006)
0.891 (0.0)
full
0.82 (0.009)
0.869 (0.002)
0.884 (0.0)
0.819 (0.009)
LSTM
1
0.866 (0.001)
0.843 (0.003)
0.85 (0.003)
0.872 (0.001)
5
0.872 (0.003)
0.849 (0.006)
0.882 (0.0)
0.849 (0.004)
16
0.878 (0.002)
0.858 (0.006)
0.887 (0.001)
0.888 (0.001)
27
0.846 (0.004)
0.831 (0.007)
0.859 (0.004)
0.862 (0.004)
full
0.891 (0.001)
0.864 (0.0)
0.872 (0.003)
0.877 (0.001)
CNN
1
5
16
27
full
0.836 (0.003)
Table Mean episodic rewards under distribution shift in HalfCheetah environment normalized by performance of the expert policy under distribution shift. Rewards are given +1 SE which is shown in parentheses.
37

---

Published as a conference paper at ICLR 2024
A.10 INITIALIZATION OF RECURRENT WEIGHTS
We leverage an initialization scheme consistent across each of the recurrent networks in order to control for potential confounding effects that could arise due to differences present in the default initializers for the recurrent weights. We will first discuss what the default initialization scheme looks like for the recurrent networks we considered and then motivate our proposed method of initialization.
Default initialization schemes. We define the default initialization schemes of recurrent weights in RNNs, LSTMs and GRUs as those implemented by TensorFlow (Abadi et al., 2016) and the default initialization of recurrent weights in a CFC as given by the open-source implementation presented in Hasani et al. (2021). The default initializers for these models differ in couple key areas. For one, RNNs, LSTMs and GRUs are orthogonally initialized whereas CiCs are initialized from a Glorot uniform distribution.
2.00
1.75
spectral norm
1.50
1.25
1.00
0.75
0.50
Glorot uniform
0.25
orthogonal
0.00
20
40
60
80
100
120
matrix size
Figure 30: Spectral norm of random square matrices drawn from orthogonal and Glorot uniform distributions for various sized matrices.
In Figure 30, we investigate the spectral norm of random matrices drawn from orthogonal and Glorot uniform distributions. Due to the significant disparity between the spectral norms inherent to the distributions, we opt to standardize the initialization scheme across all recurrent models by drawing the weights from an orthogonal distribution. We motivate this further in the description of our initializer.
Another concern is that RNNs, LSTMs and GRUs are distinct with respect to how they are orthogo- nally initialized: in particular, consider [Wo]. W,, WT, W2, Wa). [Wo W,, W2] which represent the concatenated recurrent weights RNNs, LSTMs and GRUs respectively. Rather than initialize each W. to be orthogonal, TensorFlow draws the concatenated set of matrices from an orthogonal distribution. The following result demonstrates why this is problematic in the context of our work.
[A B) denote the concatenation of the two matrices. Then, |||| |C|| and B < ||]|| where || denotes the spectral norm.
Lemma 1. Consider matrices A E Raxy and E Raxg Let C
| If we constrain to be of unit norm, then || = su Al/l Let denote the unit norm vector that maximizes ||al/ Then, we can construct another vector the concatenation of z which contains entries and zero-vector which contains q entries. Note that by construction m is also unit norm. It follows that Az < ||Cm|] which means that the maximum attainable value of ||Cx||| is greater than or equal to the maximum attainable value of | over all unit vectors æc Thus, ||||| S ||C||| An analogous argument can be made to show that B < ||C]]
Proof. We will first consider the case of matrix A. By definition, ||||| = sup
[z 0 0-... 0] which represents
By Lemma 1, it follows that the default initialization in LSTMs and GRUs produces submatrices W; with spectral norm less than the spectral norm of the default initialized recurrent weights in RNNs. We will address this discrepancy in our proposed initialization scheme.
38

---

Published as a conference paper at ICLR 2024
Orthogonal opectral-initializttio scheme. Orthogonal ppectral-nnitializatio refers to the initial- ization scheme we utilized for the models all the results we presented. First and foremost, this standardizes the distribution of the recurrent weights to be orthogonal. Note that also formalize an analogous initializer which we call Glorot uniform spectral-initializatiie which is detailed in the next subsection.
In practice, we use the orthogonal distribution as it leans on the intuition of our connectivity prior in which show empirically that networks initialized at lower spectral norms tends converge to solutions with lower spectral norms as well. Because we care about the robustness of the network as measured by spectral norm convergence and found that orthogonal weights have lower spectral norm than Glorot uniform ones (Figure 30), we opt to draw from the orthogonal distribution.
With respect to the concatenation issue discussed above, we address this by instead drawing each recurrett weight matrix from an orthogonal distribution as opposed to drawing the full, concatenated set of matrices from one. This ensures equivalent spectral norms across all recurrent weights and models.
Finally, the ppectal-iniitializaioo refers specifically to the rank-r SVD performed on the full-rank weights Wree to generate W1. W2 (Section 3.1) for training. We opted to perform spectral de- composition as opposed to drawing W1 and W₂ from orthogonal distributions individually because Wi W2 is no longer orthogonally distributed. In addition, note that multiplying the two matrices does not preserve the spectral norm of Wree. In contrast, by the Eckhart- Young-Minsky theorem, a rank-r SVD is the most efficient approximation of Wrec under the spectral norm.
Note that the drawback of this initialization scheme is that we are unable to prove all the results posed in Theorem due to the dependence between entries in an orthogonal matrix (appendix A.6.2). In contrast, we are able to prove Theorem in the case of uniformly distributed matrices, which we motivate next.
Glorot uniform spectral-initiaaizztion scheme. The Glorot uniform ppectral-initializee refers to the same construction the orthogonal spectral-initializer. except recurrent weights are drawn from Glorot uniform distributions instead. Note that this distribution is valuable if one wants theoretical guarantees for all the properties given in Theorem 1. However, this comes the cost of empirically less robust models as evidenced by the learned weights possessing higher spectral norm relative to their orthogonally initialized counterparts (results not shown).
A.11 DETAILS OF EXPERIMENTAL SETUP
In this section, we extensively detail each the experiments and analyses that were presented the main portion of the paper.
A.11.1 GENERATING EXPERT TRAJECTORIES
Arcade learning environments. Within the set of ALEs, we considered the Seaquest and Alien environments. These environments in particular were chosen learning agents that achieved performance competitive to state-of-the-an using out-of-the-boo mod- els provided by RLlib (Liang al., 2017). For the ALEs, we train an Ape-X DQN (Horgan al., 201 1018b) model using the Atari pre-processing framework detailed in Horgan et al. (2018b). We em- ploy the default implementation of the Ape-X DQN model given by RLlib which is tuned version of the model presented in Horgan et al. (2018b). After the model is trained to perform sufficiently well in the environment, we use it to generate expert trajectories. Specifically, we generate 100 roll- outs of the trained model's 's observations and actions taken in the environment which were later used to fit the recurrent models we examine in an imitation learning framework.
able train reinforcement
MuJoCo. Within the set of MuloCo environments, we considered the HalfCheetah environment. Again, this environment was chosen based on its amenability to models provided by RLlib. For the MuJoCo environment, we employ an implementation of proximal policy optimization (PPO) given by RLlib (Schulman et al., 2017). We also leverage the default set of hyperparameters given by RLlib to train the model. As we did with the ALEs, generate 100 rollouts of the trained model's observations and actions taken in the environment which were later used to fit the recurrent models using imitation learning.
39

---

Published as a conference paper at ICLR 2024
A.11.2 IMITATION LEARNING FRAMEWORK
For each recurrent architecture, we construct a model to be offline on the generated expert tra- jectories. We have layers in the network that act to preprocess the observations before entering the recurrent portion of the network as well as output layers from the recurrent portion in accordance to the action space specified by the environments, Since the observation and action spaces in ALEs and MuluCo environments are distinct, we construct different network architectures for each.
First, consider the network architectures used for ALEs. In their form, observations given by a 210 160 x dimensional image, but they are preprocessed using the Atari preprocessing specified by Horgan al. (2018b). This yields new observations of dimension 84 84 x 3 which are used as inputs to the model.
layer type
activation relu relu
Conv(64, k=5, s=2) Conv(128, k=5, 2=2) Conv(128 k-5, 2=2) relu
AveragePooling TimeDistributed recurrentNet FC(numActions)
none
none
none
softmax
Table 7: ALE network architecture
In the table above, denotes the kernel size and s denotes the stride of the convolution. recur- rentNet refers to the recurrent network module that varied and is discussed in further detail below. numActions refers to the number of actions the agent can take in the environment. Since the action space is a discrete categorical variable that take numActions different values, the network is trained using categorical cross-entropy loss. We detail the model hyperparameters and corresponding grid search in A.11.3.
Next, we will consider the network architecture employed for MuJoCo networks.
layer type FC(256) FC(256)
activation relu
relu
TimeDistributed
none
recurrentNet
none
FC(nmAcction))
tanh
Table 8: MuJoCo network architecture
Since the action space for MuJuCo environments in continuous, the output is continuous variable that can take numActions different values and the network is trained using MSE loss. Further- more, since the action space bounded within the interval [1,, a tanh activation function is applied to the output layer.
A.11.3 MODEL HYPERPARAMETER
Below is a table with the model hyperparameters that were used in both the ALE networks and MuloCo networks. Parameters in square brackets represent grid search over which the best per- forming model was chosen. Note that only offline performance the validation/test set was used to determine the best performing model the grid search. Parameters in curly braces are dimensions across which the network individually evaluated for comparison.
The architecture the convolutional head in the ALE networks was used on the basis of work presented in Lechner et al. (2022) which conducting extensive grid searching across convolutional architectures over many ALEs. Analogously, the dense head used in the MuloCo architecture was drawn from Hasani et al. (2021).
40

---

Published as a conference paper at ICLR 2024
hyperparameter optimizer hidden size
value
Adam (31 0.9, $2 0.999)
64
learning rate epochs rank sparsity
[5 x 10 10-5, 1 10 10-4,11-0000 5 10 *]
150
(1, 5, 16, 27, full)
{0, 0.5, 0.8}
Table 9: Model hyperparameters
In all the environments, we considered models of rank 1,5, 16, 27, full and sparsities of 0, 0.2, 0.5, 0.8. Note that the recurrent weights in the full rank networks are not rank decomposed, they for the low-rank networks. These ranks and sparsities were chosen such that for each rank we examined, we also looked at a sparsity level for which the number of recurrent parameters in each parameterization is roughly equivalent. For example, since the networks we constructed had hidden dimension of size 64, a recurrent matrix with a sparsity level of 0.5 has the same number of parameters as one with rank of 16. We did not present results of networks with a very high recurrent sparsity like 0.95 (which would roughly be the sparse analog to rank-1 networks) due to optimization difficulties encountered, particularly LSTMs and GRUs. In order to ensure that the performance of given network wasn't t due to favorable random sparse mask, we trained each model 3 times from different random seeds and averaged over the model performance.
Finally, also employed an initialization scheme for the recurrent weights distinct from the default initialization schemes given in open-source implementations. For details on this, refer to A.
A.11.4 EVALUATION METRICS
Trained models were evaluated offiin validation/test set with respect to their cross-entropy loss in the case of ALE networks and mean-squared error in the case of MuJoCo networks. However, in practice, we care about how the agent performs when deployed in the environment closed-loop. The online rewards were computed by performing 10 simulations of the agent in the environments and averaging over the rewards from each episode. The in-distribution setting refers the environments in which the observations are not modified aside from preprocessing performed by RLlib.
In the distribution shift setting, for the ALE observations we perform 5 types of perturbations the agent's 's observations: cutout, brighten, darken, noise, and blur. In cutout, we remove some pixels from the center of the input image. In brighten, in the positive direction and do the same for darken but shift the pixels in the negative direction. To noise the image, we draw random noise from a uniform distribution and add it to the input image. For distribution shifts in the MuJoCo environments, we perform 3 types of perturbations: noise, dropout and offset. To noise the observations, we draw random noise from normal distribution and add it to the observation vector. To blur the observations, we perform Gaussian blurring on the image. To perform dropout, we mask a subset of the dimensions in the observation vector. To perform offset, we, with equal probability, shift the observation vector in the positive or negative direction by an amount constant across all dimensions. For each of the inputs during closed-loop navigation, apply the distribution shift with probability p 0.1.
shift all the pixel values by constant amount
To evaluate the models, both in-distribution and under distribution shift, we normalize the rewards by the performance of the expert policy in order to convert the rewards into units that allow to benchmark the performance of the imitation learning agents while also making comparisons across recurrent models and connectivities.
A.11.5 DETAILS ON NETWORK ANALYSES
In this section, we further elaborate on the chosen modes analysis in the trained recurrent net- works and in where the metric computation is potentially ambiguous elaborate the methodology used to compute it. All the plots shown in the paper regarding these analyses were performed models trained in the Seaquest environment.
41

---

Published as a conference paper at ICLR 2024
Spectral analyses. For recurrent networks that have more than set of recurrent weights (CfCs, LSTMs, GRUs), any of the spectral statistics that were computed (i.e. eigenspectrum, recurrent weights singular value spectrum, input weights singular value spectrum, etc.) we compute and plot a single statistic that represents the average over the statistics of each of the individual recurrent weight matrices.
On a separate note, with respect to measuring the decay of the singular value spectrum, note that we normalized the sorted set of singular values by the spectral norm. This was done in order to explain away the magnitude of the transformation, since we separately motivate and analyze the spectral norm. Doing so also ensures each of the decay curves start which enables comparison across models, ranks and sparsities.
Recurrent gradient analysis. In Section 4.2, we briefly discussed the analysis performed analyze the evolution of the recurrent gradients across time. Here, we provide some additional details that were not addressed in the main portion of the paper.
Firstly, we computed the norm of the gradients in log space due to the exponential decay associated with the gradients over time (which happens practice because gradient propagation multiplica- tive). Secondly, note that each of the recurrent gradients start evolving from 0. We enforced the decay to start from 0 since we do not care about the starting value of the gradients but rather how this gradient evolves in units relative to the start value (i.e. the rate of decay matters, but not the value it started decaying at). So, we translated the curves to begin at O, irrespective of their starting value.
end of time,
Effective dimensionality of trajectories. To understand the complexity of the dynamics across recurrent models, connectivity ranks and connectivity sparsities, we collected the trajectories of the model during the simulation of the agent in the online, closed-loop setting and fit a PCA on the entire set of trajectories. We then computed the explained variance of the top 5 principal components (PCs) as a proxy for the effective dimensionality of the trajectories. This is canonical approach to analyzing state-space trajectories in recurrent networks (Lechner al., 2020).
However, we extended the canonical analysis beyond just the full state-space trajectories by de- composing the trajectories into recurrently-driver activity and input-driven activity. In particular, we considered Wrecht-1 as the recurrent activity and W WmppZ as the input activity. We analo- gously aggregated these trajectories individually and computed the explained variance of the top PCs. This gives separate measurements for what the dimensionality of activity looks like in the recurrent subspace versus the input subspace and allows us to explicitly disentangle these two axes.
For recurrent networks that have more than one set of recurrent (and input) weights (CfCs, LSTMs, GRUs), we computed the explained variance of the top 5 PCs individually for each set of weights and then averaged across them to produce a single number summary for the complexity of network dynamics.
A. 12
MODEL PARAMETER COUNTS
model
parameter count
CIC
61632
LSTM
81726
RNN
20544
GRU
61632
Table 10: Parameter counts of the fully-connected, full-rank versions of each recurrent model. Input size 256, hidden state size 64.
42


Reviews:
[{"review_id": "YnKEHSBPh4", "reviewer": "Reviewer_p6DM", "summary": "Paper analyzes how the sparsity and rank of recurrent connectivities effects the robustness of using these models for closed-loop control.", "questions": "See weaknesses section", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Originality\n- The setting that this paper analyzes appears to be novel\n\nQuality\n- Paper presents a variety of in-depth analysis experiments, which seek to understand the effect of rank and sparsity on different aspects of the model", "weaknesses": "- In its current form, I found it a bit challenging to parse the main contributions of the paper. I think the reason might be that Section 3 (Parameterization of Connectivity) only details the form of the proposed connectivity and the theoretical ramifications on spectral radius and norm, without any mention of the main empirical findings in the paper. It is not until section 4 (Experiments), where the paper mentions the specific findings within each subsection. Even there, I found it difficult to get concrete takeaways from the experiments, because the results are usually describe in great detail without a high-level point. For example, in Sec 4.1, it seems like low-sparsity, high-rank CfCs and LSTMs are good in online settings, and LSTMs tend to be better than CfCs and RNNs at high sparsities, and low-rank, sparsity CfCs tend to be good under distribution shift. From reading this section, it was not clear to me which configuration of sparsity, rank, and architecture was most effective? In general, I think it would be helpful to distill the main takeaways from the experiments and incorporate them into Section 3, before explaining the details of the experimental setup and results.\n- Right now, all of the neural network models used in the experiments use some kind of temporal connection. However, I believe this is not the standard architecture used when solving the tasks used in the experiments (Seaquest, halfcheetah, etc.). It would be helpful to to include an additional baseline using standard architectures (conv net, fully connect, etc) to get a sense for the return that a \"vanilla\" approach can get, and to better appreciate the significance of the robustness gains made by the additional sparsity and low rank formulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zli5u4SPDB", "reviewer": "Reviewer_59d7", "summary": "This paper show how the spectral radius and norm depend on sparsity and the rank of the recurrent weights, using these as a proxy to robustness and to introduce an inductive bias towards better performance for networks trained in a causality gap setting.\nBetter performance under distribution shifts is demonstrated for low-rank and sparse recurrent neural networks for networks trained on various environments under an imitation learning framework.\nFinally, it is shown that closed-form continuous-time neural networks are more amenable to a low-rank and sparse connectivity prior than the canonical recurrent architectures (vanilla RNNs and LSTMs).", "questions": "About the eigenspectrum analysis.\nWhy is it the case that the closer the distribution of eigenvalues is to uniform, the more balanced the attention profile of the recurrent weights is across dimensions of the eigen-transformed?\nAnd why does this have implications for the dimensionality of recurrent state-space dynamics?\n\n\nIn Figure 5, 6, 7 etc, what does the colored region mean? Is it the variance? Is it a confidence interval? If there are only three networks per parameter setting shouldn't they all be shown?\n\nIs it really counterintuitive that with increasing sparsity, the dimensionality of the recurrent trajectories increases? Isn't sparsity just functioning as effectively increasing the rank?\n\nIt is not clear why higher dimensional dynamics would lead to more robust networks, as claimed in Figure 16. This seems to be in disagreement with the claims about constraining the network to be low-rank improved robustness in Section 4.3.\n\nHow is $\\Delta W$ calculated in Figure 7? If it is just the change of the parameter across training, the caption should mention that instead of saying that the change in weights \\emph{during} training is shown.\n\nWhat number of parameters do the different networks have? In particular, how many more parameters do Cfcs have as a result of having two vanilla RNNs in them and another gating mechanism? Be more clear about how $F$ is parameterized. How could the increase in the number of parameters explain the increase in performance?\n\n\nShouldn't a higher spectral norm contribute to higher dimensional dynamics? For a low spectral norm the network would quickly collapse onto a low-dimensional manifold. How do you explain then the RNNs have higher spectral norm and lower dimensionality in their dynamics?\n\nThe last claim of Theorem 1 is only proved for orthogonal initialization?\n\nWhat are the parameters used for the Adam optimizer? The default ones? Mention.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Application of existing analyses to understand robustness and generalization performance to a new connectivity framework is novel. Furthermore, the analysis of  decomposing the activity into recurrent and input driven components seems novel.  The proof of the theorems are sound. All methods are well described in a way that allows for reproducible, especially if code will be made available. The main aim, to better understand why why sparse connectivity is useful in closed-loop systems is a very important problem to tackle.\nThe theoretical contributions are significant to an even broader setting than is considered in the paper.", "weaknesses": "Already existing analysis and tasks.\nA more comprehensive comparison to other networks trained on the tasks is missing.\n\nSome existing analyses to measure robustness, for example as the loss in performance as a function of noise level is missing.\nOverall, it is not entirely clear why some of the proxies used in the paper are sufficient to be used to asses robustness.\nThe proxies are only partially justified to be good measures of robustness. Further analyses of the dynamics, in greater detail than dimensionality, would be important to have certainty about the robustness properties of the trained networks.\nThere are many other measures for assessing the effect of perturbations on dynamics, for example Lyapunov exponents and these should be calculated for a more complete analysis.\n\nBecause there is no assessment of the learning dynamics itself or the process of learning itself the vanishing gradient analysis is a misnomer. I understand the main point of these analyses to be some memory component or transient (convergent/divergent) dynamics itself instead of it being related to the question whether the network architecture can support vanishing gradients. That said, also for vanishing gradients Lyapunov exponents are a good method for analysis.\n\nIt is unclear why a particular number of samples were considered to be sufficient. For example, why are 3 randoms sufficient to rule out that a favorable random sparse mask is not influencing the results? \nFurthermore, for such high spaces for the task, averaging over 5 perturbations seems insufficient.\n\nTheorem 1 lacks the case where the recurrent matrix does not have full connectivity to begin with and the influence of the sparsity parameter $s$ on the spectrum in that case.\n\n\\paragraph{Reward}\nIt is difficult to asses to what (good) performance the reward range in Figure 3 corresponds to. Would it be possible to show what kind of actions the highest and lowest performing networks correspond to at least? The performance of the agents is a very relative concept for which a number is not sufficient to understand what the networks actually are doing when they perform the task (supposedly well). What is the maximum number of rewards?\n\n\\paragraph{Reward ranking}\nIt seems that the reward ranking is a difficult to asses proxy for performance on the perturbed environments. For a full picture it would be better to show what amount of reward decrease the different shift cause. How do the distribution shift change the reward range for example? If all networks perform badly, but the Cfc slightly better, can we still claim that they are robust?  \nThe best performing networks on the online performance measure seem not to be performing particularly well on distribution shifted versions of the task. How does the ranking look like if the performance of the in-distribution reward is also taken into account?\nFinally, to see what kind of perturbation is most damaging for the different networks would be very insightful. Performance per perturbation type could also show what kind of robustness the different networks display.\n\nFor showing these ranking, it would be perhaps better to show the actual (average) rank as a number instead of a color coded version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzuPh9Apma", "reviewer": "Reviewer_XFJy", "summary": "The paper investigates the robustness of recurrent-based agents on non-stationary environments. In particular, the paper focuses on how sparsity and rank of different recurrent neural networks (RNNs) impact network dynamics. The paper provide theoretical insight on the property of the weight matrices in RNNs with respect to the rank and sparsity, which is used to support their empirical findings. Specifically, by considering the spectral radius and the spectral norm of the weight matrices, the paper suggests that smaller spectral radius and norm can yield better generalization under distribution shift under their experiments.", "questions": "See comments above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper does a thorough investigation on how the sparsity and rank of the weight matrices impact the generalization of an agent under distribution shift.\n\t- The paper focuses on many axes in the experimentation.", "weaknesses": "**Comments**\nAddressing the following comments will increase my score:\n- The paper is very difficult to read---while the paper provides some high-level intuition for particular quantities (e.g. low sparsity corresponding to the rate of vanishing gradient), I feel there is a big gap between every subsection. In other words, I often have trouble making the connections between the results. Furthermore, even with the large text of paragraphs, often the most important information is ambiguous, and I list them in the following:\n\t- I don't think I completely understand this experiment. What is $W_{full}$? I expected the full state-space trajectories mean running PCA on the states gathered over time, and the two former corresponding to $W_{rec}h_t, W_{inp}x_t$.\n\t- Theorem 4.1 seems to relate the weight matrices based on particular initialization schemes. I understand that in figure 4 the paper appears to agree with theorem 4.1, but I fail to understand why this theorem needs to be in the main paper as it does not explain any insight on the generalization/robustness directly.\n\t- The scale of the subplots should be consistent for easier comparison (e.g. figures 4, 5, and 6.)\n- The proof of theorem 4.1 is unclear to me. While I see that the paper uses random matrix theory to obtain result on the spectral radius and spectral norm based on the sparsity, I don't completely understand how we obtain the statements for rank $r$ for both initializations.\n- Regarding the reward ranking metric: Why not normalized return? Ranking is not too meaningful regarding how generalized the agent is---it is only relative performance. An alternative is to provide the expert performance as a baseline.\n- On figure 5, the paper only indicates the variance of the data captured by the top 5 principal components (PCs), we lose the information on the remaining variance captured by the proceeding PCs. As a result, using only the top 5 PCs to claim higher/lower effective dimensionality seems incorrect. For example, comparing two methods $M_1$ and $M_2$, if we assume that $M_1$'s top 5 PCs capture 0.7 explained variance while $M_2$'s capture 0.5, we cannot guarantee that $M_1$ is always capturing more variance as we increase the number of PCs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5YmuNuS51r", "reviewer": "Reviewer_R5dM", "summary": "Low-rank and sparse initial recurrent connectivity have been found to be two powerful priors when training RNNs to imitate an expert and later deploy them (in control tasks). This work aims to better understand the reasons behind this behavior. From a theoretical point of view, it links sparsity levels and rank at initialization with properties of the recurrent connectivity matrix. Empirically, it studies in depth how those two properties affect the dynamics of the different networks and analyzes how they affect generalization to distribution shifts.", "questions": "c.f. Weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written. Theoretical results are simple and insightful for the result of the result. The empirical analysis is thoroughly done. It nicely combines analysis tools developed in dynamical systems, computational neuroscience, and deep learning to better understand the observed behavior.", "weaknesses": "The analysis focuses on the spectral properties of the connectivity matrix. While I can understand why this is a reasonable choice for understanding the impact on low-rank and sparsity for a single architecture, it seems limited when it comes to comparing different architectures. Ideally, one would need to study the spectral properties of the recurrent Jacobian. The recurrent Jacobian would heavily depend on the recurrence connectivity matrix for all architectures, but important differences might still remain. The paper is currently missing a discussion of this point.\n\nMinor: \n\n- It would be great to know which kind of distribution shift you are using in your experiments. I could not find this information.\n- the work of Herbert Jaeger or Wolfgang Maass is probably a better reference for echo state networks than the Deep Learning Book of Goodfellow et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)