Operative dimensions in unconstrained connectivity of recurrent neural networks
Authors: Renate Barbara Krause
Matthew Cook
Sepp Kollmorgen
Valerio Mante
Giacomo Indiveri
Venue: NeurIPS 2022 Accept (2022)
Keywords: recurrent neural networks
computation through dynamics
dimensionality

Abstract: Recurrent Neural Networks (RNN) are commonly used models to study neural computation. However, a comprehensive understanding of how dynamics in RNN emerge from the underlying connectivity is largely lacking. Previous work derived such an understanding for RNN fulfilling very specific constraints on their connectivity, but it is unclear whether the resulting insights apply more generally. Here we study how network dynamics are related to network connectivity in RNN trained without any specific constraints on several tasks previously employed in neuroscience. Despite the apparent high-dimensional connectivity of these RNN, we show that a low-dimensional, functionally relevant subspace of the weight matrix can be found through the identification of \textit{operative} dimensions, which we define as components of the connectivity whose removal has a large influence on local RNN dynamics. We find that a weight matrix built from only a few operative dimensions is sufficient for the RNN to operate with the original performance, implying that much of the high-dimensional structure of the trained connectivity is functionally irrelevant. The existence of a low-dimensional, operative subspace in the weight matrix simplifies the challenge of linking connectivity to network dynamics and suggests that independent network functions may be placed in specific, separate subspaces of the weight matrix to avoid catastrophic forgetting in continual learning.

Operative dimensions in unconstrained connectivity of recurrent neural networks
Renate Krause, Matthew Cook, Sepp Kollmorgen, Valerio Mante" and Giacomo Indiveri'
Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland Corresponding authors: rekrau @ ini ch. valerio ietii.inii.illll.. "Equal contribution
Abstract
Recurrent Neural Networks (RNNs) are commonly used models to study neural computation. However, comprehensive understanding of how dynamics in RNNs emerge from the underlying connectivity is largely lacking. Previous work de- rived such understanding for RNNs fulfilling very specific constraints their connectivity, but it is unclear whether the resulting insights apply more generally. Here we study how network dynamics are related to network connectivity in RNNs trained without any specific constraints on several tasks previously employed in neuroscience, Despite the apparent high-dimensional connectivity of these RNNs, we show that low-dimensional, functionally relevant subspace of the weight matrix can be found through the identification of operative dimensions, which we define as components of the connectivity whose removal has a large influence on local RNN dynamics. We find that weight matrix built from only a few operative dimensions is sufficient for the RNNs to operate with the original performance, implying that much of the high- high-dimensional structure of the trained connectivity functionally irrelevant. The existence of low-dimensional, operative subspace in the weight matrix simplifies the challenge of linking connectivity to network dynamics and suggests that independent network functions may be placed in spe- cific, separate subspaces of the weight matrix to avoid catastrophic forgetting continual learning.
Introduction
central goal in neuroscience is to understand how groups of tightly interconnected neurons generate the complex network dynamics that underlies behavior. To this end, ever larger experimental datasets on neural anatomy, neural activity and the corresponding behavior are collected and analyzed and new experimental tools to extend the amount and quality of such data are continuously developed. However, in most settings it remains an open question how the underlying neural connectivity is able to generate the observed neural dynamics. Progress in how inspect and interpret these complex datasets, and in particular on the relation between neural structure and function, may come in particular from new theoretical frameworks 2
Artificial Recurrent Neural Networks (RNNs) are a promising tool to develop such theoretical frame- works in well-controlled and flexible setting 4 5). Previous work RNNs explained how a specifically designed connectivity can give rise to desired network dynamics [6] 7 g] Further
theoretical work RNNs with random, recurrent weights provided detailed insights into the proper- ties of neural dynamics emerging from largely unstructured connectivity (9] 10
I1 12 [3] More
recent work related structure function in feedforward networks 14
15
RNNs with specific
connectivity constraints. For example, network motifs in threshold-iee linear networks are used to predict
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the existence of fixed points of the dynamics [6]. Similarly, a principled understanding of dynamics and the role of different cell classes in computations can be achieved for RNNs with oow-dimeniional weight matrices (low-rank RNN 17 [8] ) At present, it remains unclear if and how the resulting findings can be generalized to RNNS which are not subject to such constraints.
In this work, we study how the network dynamics are related to the network connectivity in vanilla RNNs that are trained using gradient-based approach without imposing any specific constraints on the network weight matrix. We find that the weight matrix of the trained RNN is consistently high- dimensional, even when trained on tasks resulting in dynamics that are wwwdiimnsiooal.. Notably, Now-dimensional subspace within the high-dimensional weight
we are nonetheless able to identify matrix that sufficient perform the trained task. We identify this functionally relevant subspace of the connectivity through the definition of set of operative dimensions, which we define as components of the network connectivity that have large impact on computationally relevant local dynamics produced by the network.
This ability to identify functionally relevant subspaces in weight matrices improves our understanding of how the network connectivity generates the observed network dynamics and thereby makes RNNs into a more interpretable model for neuroscience and machine learning applications.
2 Results
We perform our analyses on vanilla RNNs trained without regularization terms, using the standard RNN equation:
-x + Wr, + Bu, + o
(I)
where x, E RNN are the linear activities of the N hidden units over time with r, tanh(x;). W E R NxN is the recurrent weight matrix of the hidden units and T E R is the time constant (T 10ms, dt- 1ms). We consider RNNs of N 100 noisy units, where each element of o drawn from a Gaussian distribution N( = 0, a = 3. 1623 Rs 0.1). The network output is defined as:
7, Yr,
with output readout weights Y E Z The task-dependent, time-varying inputs u, E R are projected onto the hidden units with input weights B E N Note that only the inputs vary across different conditions within task. For any given condition, the cost is defined as:
1 2マ 2/
cost
(2) 2x))
ZT
where (1) is the desired output. All network weights (B, W. Y) are randomly initialized, and networks are trained to minimize the summed costs across all conditions.
The RNNs are trained separately on two previously proposed tasks: context-dependent integration
19 and sine wave generation (20 (see appendix section A.3.8 for additional results on sequential MNIST). The dynamics of RNNs trained on these tasks is well understood, and was shown to be largely independent of the type of employed RNN (21) context-dependent integration, the RNN receives two noisy, sensory inputs (between -1 and 1) and two static, context inputs (0 or 1). The network is trained select one of the two sensory inputs (depending on the currently active context input; i.c. select input sensory; in context,) and integrate it over time (Fig. [a b). The network should reach choice, or choice; the average of the contextually relevant sensory input positive or negative, respectively. In the sine wave generation task, the RNN receives one static input and is trained to output a sine wave whose target frequency given by the level of this static input (Fig. [ gi for details on task structures see section I.I.I. For both tasks we trained 20 RNNs (different random initial connectivity) with gradient- optimization to minimize the cost (Eq. 3 for details see section A.1.2)
2.1 High-variance dimensions
To characterize the relation between network connectivity and dynamics, we first consider the functional relevance of high-variance dimensions (as in analyses of low-rank RNN [7]) defined as the left singular vectors of the RNN weight matrix W:
W = EN w.s.
(4)

---

b
ConTExtE dngendentinteennter
choices
variance cipla ned (S)
chakcs
aorsory
cholcey
& g
context,
contesty
choices
Chokeey ContONT
cholcn
rCcurrent
welghts
g
vorance hed (9)
h
verance explined Pit A O m 5
irpuuss
curput
con
& &
0.3
Emme
weights
recurment
10
25 50 PCIW)
T0o
25
50 rank
100
PCXI
Figure 1: High-variance dimensions of the connectiviry. (a) Task schematic of the context-dependent integrator. (b) Low-dimensional projection of condition average trajectories for example context- dependent integrator. (c) Variance explained (in activity space) by individual PCs of the network activity X over all input conditions, shown at different stages of training. (d) Variance explained (in weight space) by individual PCs of the weight matrix W at different stages of training. (e) Network output cost (Eq. 3) of networks with reduced-rand weight matrices W.C. for k 1 : N (Eq. 5). averaged over input conditions (shaded area: median absolute deviation (mad) over trials). (f-j) Analogous to (a-e), but for sine wave generator. (b-e) and (g-j) representative network per task.
where w; and the left and right singular vectors respectively, and 8; the associated singular values. In general, we refer to the i-th left singular vectors of a matrix M E NxC (C E R) as the principal components of the matrix, PC(M);
We first ask whether the weight matrices of the trained networks are low or high-dimensional. by assessing the amount of variance in the connectivity (sî) explained by principal components PC(W); In the trained networks, variance explained falls off slowly over PC(W);, implying that the weight matrices W are digesiimenii The variance explained by individual PCs changes little during training (Fig. [Tt, shown for column dimensions; colors: training iteration) but is strongly influenced by the rank of the untrained network (Fig. S6). Despite having -igh-dimensional connectivity, the network activities (X [x1,...xy, Xy] and R = rr.....r :r..:.ry:: both further concatenated over all input conditions) are consistently now-dimensional. Activity is low-dimensional even before training, as W initialized with a spectral radius of (Fig. DT. h; colors: training iteration). Throughout training, more than 95% of the variance in the network activities X can be described by less than dimensions. Thus, even though the underlying weight matrix (being high-dimensional) could support activity in any region of state space, the network activity generally only occupies low-dimensional subspace. This finding raises the question of whether all dimensions in W are truly necessary to perform the task.
To assess the functional importance of single dimensions of W, we construct aaduced-ranke approxi- mations of W from only subset of all PC(W); and then assess the performance of the resulting RNN (as 22 ). Each low-rank approximation is constructed by including only the first PC(W);
W PC = D W.s.v.
(5)
RNN performance based on WPC evaluated as above to obtain the network output cost (Eq. 3
We find that a large number of PC(W); are consistently required to reach a similar performance as the original network. For the two example networks, 88 PCs are required to achieve original performance for context-dependent integration, and 86 PCs for sine wave generation (Fig. I we define original performance as 4 times the cost of the corresponding full-rank RNN, Fig. S7 shows more example networks). Furthermore, performance does not improve monotonically with increasing rank, but rather displays sudden jumps at specific ranks (Fig. [Te. j). These jumps suggest that some high- variance dimensions are more relevant than others in driving the network output. These putative differences in functional relevance are not obviously mirrored the amount of variance in W explained by each high-variance dimension (Fig. [H. i). In other words, the amount of variance in connectivity space explained by given dimension W does not appear to directly correspond to its functional relevance.
3

---

choice
choice,
local dynumicss frand 60)
cholce,
choicey
choloh
0.5
time
time
Lond
X alove points thal
ronk(W)
Pank(W)
cank ranki
choices
choico
choices
choices
Figure 2: Local and global dynamics. (a) Network output cost for reduced-rank weight matrices
W in an example network for context-dependent integration. Analogous to Fig. l.. but shown separately for choice; and choice? trials in context, (b) Average trajectories for the conditions in (a) in the full-rank and reduced-rank (A: = 60) network (grayish and reddish curves, see legend). Blue lines show recurrent dynamics in the reduced-rank network initialized at locations corresponding to slow points of the dynamics the full-rank network (blue crosses). (c) Same as (b) but for reduced-rank network with & = 61.
Overall, it appears that the inferred high- nigh-variance dimensions may be ill-suited to shed light on the relation between RNN connectivity and dynamics. These findings point two possible scenarios: the functionally relevant subspace of the weight matrix in these unconstrained RNNs is genuinely high- dimensional, precluding simpler descriptions of the most relevant components of the connectivity; or high-variance dimensions PC(W); may in general not be well-aligned with functionally relevant dimensions of the connectivity.
2.2
Operative dimensions
2.2.1
Definition of operative dimensions
To distinguish between these two scenarios, we devised an approach to directly identify functionally relevant dimensions of the weight matrix. We refer to this new type of dimensions as the operative dimensions of the connectivity.
The definition of operative dimensions based on the key insight that sudden jumps in performance (cost, Eq. 3) can be caused by small changes in the local dynamics. Here we illustrate this for one example network trained on context-dependent integration (Fig. [2.. We focus on 2 out of input conditions, for which the cost shows prominent jumps at specific ranks k, specifically when
transitioning from 60 to k = 61 (Fig. 2 red circles).
To understand this sharp increase, we examine the activity trajectories produced by the original full-rank and the reduced-rank RNN for the two considered input conditions (k 60 in Fig. 2 k = 61 in Fig. (2.. The well-perfor,ing, full-rank RNN produces trajectories that start in the center and move out endpoints on the right (choice,; solid black) or left (choicez; dashed gray). The RNN with & = 60 closely matches these dynamics (Fig. [b; red curves, solid and dashed). On the other hand, the RNN with ào erroneously produces trajectories that move the left endpoint for both conditions (Fig. 2c; overlapping red curves). This discrepancy between choice, trajectories in the original full-rank and 61 RNN underlies the large jump in cost (Fig. 2h).
Critically, we found that the large difference in cost between the two reduced-rank networks is due to only small change in the underlying local, recurrent dynamics. To visualize the local dynamics in the two reduced-rank RNNs, we generate brief activity trajectories (Fig. bb.c; blue curves) initialized the location of identified slow-points of the dynamics in the original full-rank RNN (blue crosses; identified as in [19). Here, we isolate the recurrent contribution to the dynamics by running the RNN without any external inputs (Eq. T with u, that the local recurrent dynamics is substantially different across networks only in the region of state space that activity travels through at the onset of the trial (Fig. [2..c dashed square). This small local difference is sufficient to redirect the choice; trajectories for & = 61 towards the wrong endpoint. This mistake cannot be corrected by recurrent dynamics at other locations state space, resulting in the large cost. In summary, the functional relevance of individual dimensions of W is hard to assess based on changes the cost, but rather may be better judged based on its effect on local network dynamics.
o, 0). The resulting trajectories suggest

---

Based on this insight, we define operative dimensions as dimensions in W that have large impact on the local dynamics if removed from W (Fig. B.). Given an arbitrary unit vector a E we define W as the matrix of rank N - obtained by removing the dimension a from the column space of W (orthogonal projection):
W W aaa W)
In the absence of noise, the dynamics of the resulting reduced-rank RNN are then given by TK: = -X; Wr; + Bu: At location x, the activity evolves to Xe+1 Nr S over one time-step. Likewise, the state the full-rank network evolves to Xt+1 = X, + X,dt, where we have set o₁ = in Eq. T We quantify the change in local dynamics brought about by removing as:
Af X 8
as illustrated graphically in Fig. Ba. We then define the operative dimensions of W based on Af in two steps. In the first step, we infer a set of local operative dimensions at each of P sampling locations y, E N in state space. The sampling locations are chosen as an evenly distributed subset of the states X2 explored by the condition average trajectories of the full-rank network (Fig. [p. g; only a subset of P >> 100 locations shown). This choice of sampling locations ensures that we consider only local dynamics that is likely to contribute to solving the task at hand. The first local operative dimension d at location y, is defined as the dimension a that maximizes Af:
d argmax (rngmax(Af)(a.... (x=x)))
(8)
a
Up to N - 1 further local operative dimensions d.j at the same location y, are defined in the same way, but under the additional constraint that they need to be orthogonal all previously identified local operative dimensions at that location:
dijj argmax ()) constrained by: d. 0, Vi <i
In the second step, we define the global operative dimensions by combining the local operative dimensions dij from all sampling locations operative dimensions are scaled by their local & and concatenated into one matrix L.
J. 1 P). Specifically, the local
I. = [1,
dx.
(10)
where Afi, A A/x=,,,add.)) The i-th global operative dimensions q: is then defined as the i-th left singular vector of L:
(II)
where Si are the singular values, and P; the right singular vectors of L. The subspace spanned by the global operative dimensions q, consists of all left singular vectors with 9: > O. Note that these steps result operative dimensions the column space W. which are referred to as column dimensions in the figures. We employ analogous approach to define operative dimensions of the row space of W which yields global operative row dimensions q (row dimensions in figures; see section A.2.2).
2.2.2 Operative dimensions identify a low-d functional subspace of the connectivity
To quantify the functional relevance of the global operative dimensions, proceed as for the high- variance dimensions above (Eq. 55) by constructing reduced-rac -rank approximations of W from only a subset of the operative dimensions. For the column dimensions, the reduced-rank. approximations wor given by:
WOP 10/11.((4 W)
(12)
Network activity OP based on WOP is then computed as above (Eq.
Unlike the varianar dimensions, the global operative dimensions identify a low-dimensional subspace that is sufficient for the RNN to achieve the original performance in both tasks (Fig. 3p-c e-f; solid lines, qualitatively similar results on sequential MNIST are shown in section A.3.8 We find that 15 column and 27 row dimensions are sufficient to achieve original performance for context- dependent integration; and 29 column and 41 row dimensions for sine wave generation. Thus, the RNNs are functionally low-rank even though the underlying weight matrix is high-dimensional.
5

---

d
recuraant flors- fieid
ronko 1
valluree expaateed RR
60
conten dependen strrccer
redluced ronkc 1
cofume dimensions
g
zom pirahiu loctioos
Rine
aerspling
alana Enaindory
100
ingettooy
evarage
varlance ald
dictarce (eat)
samping locations
goss
Opetalive dimenvenn
dimmsionn
20
maximbo A
10
20
10
30
75 rank/W 1
10c
PCLL)
mink// 0P
Figure 3: Operative dimensions. (a) Definition of operative dimensions based on local recurrent dynamics along condition average trajectory. Arrows show the recurrent contribution to the dynamics for the full-rank and several reduced-rank networks (colors, see legend). Local operative dimensions maximize Af. Here inputs u, and noise o, set to zero. (b) Rank of global operative column dimensions, estimated with PC analysis on concatenated local operative dimensions (Eq, TO and 11 (c) Network output cost of networks with reduced-rant weight matrix W for h = 1 N (Eq. 12p. (d) State distance between trajectories in the full-rank network and in networks with reduced-rank weight matrix w (b-d) Based on global operative column dimensions and averaged over 20 networks per task; shaded area: mad. Network output cost obtained with internal and input noise, state distance without any noise. (e-g) Same as (b-d) for global operative row dimensions.
This clear difference between the global operative dimensions and the high-variance dimensions is also reflected in their weak alignment to each other (Fig. $8). This finding implies that much of the network connectivity these trained networks not required to perform the task.
Notably, the identified functionally relevant subspace of the connectivity is sufficient to generate the original network trajectory, not just the network activities along the output direction. The average distance between the network trajectories of the reduced-rank and the full-rank networks decreases rapidly as the global operative dimensions are sequentially added to the weight matrix (Fig. 3d, g: stote distance = Ei= ||x; x. This observation follows from the definition of operative dimensions, which focuses not on changes network output, but rather on the recurrent dynamics at sampling locations lying along the entirety of the condition average trajectories. We obtained similar results for networks at different stages training (Fig. $(4) and with different types of connectivity and architecture (Fig. $15). Similarly, even though operative dimensions defined based how much they alter the local dynamics when removed from W, the first few operative dimensions are sufficient to preserve the dominant local, linear dynamics (Fig. $17
Unlike for high-variance dimensions (Fig. D j), RNN performance and state distance change smoothly with increasing rank of the global operative dimensions (Fig. Bd, f-g). However, adding the first few dimensions the weight matrix often hurts performance. This effect can happen because the global operative dimensions are not sorted based on when they are required during individual trials, but rather by their impact on local network dynamics. Indeed, the first few operative dimensions are mostly required at state-space locations explored late in the trial, but are insufficient to sustain the dynamics required early in the trial (Fig. $2I,, i.
Identifying the operative dimensions critically relies on the correct choice of sampling locations. First, we illustrate the importance of the choice of sampling locations by defining global operative dimensions based on random sampling locations in state space (Fig. [3.. f; dotted lines). For the same choice of rank, the resulting dimensions yield much poorer performance compared the operative dimensions defined as above, emphasizing the importance of capturing the local network dynamics within the functionally relevant part of state space. Second, even when sampling locations are defined along the condition average trajectories, they need to be sampled at high enough density. When too few sampling locations are chosen along the trajectories, the identified global operative dimensions are less effective at reproducing network output and trajectories (Fig. S18).

---

high dimensiinal generator
explsined
Matte ditemme Gasil
outpue
80
Ben.
output 2
cafume
Bo.
outpur
bimé
vallnoo cpllare (1)
50
numberr Inputs Aitslpuut
varfannce explained 691
800
State siraace (nau)
50
20
70w diminsiane
80.3
20
PCXX)
PCIL)
mank
Figure 4: Operative dimensions and network complexity. (a) Task schematic for high-dimensional sine wave generator networks trained to output 1-5 sine waves simultaneously. (b) Variance explained (in activity space) by individual PCs of the network activity X over all input conditions. (c) Rank of global operative column dimensions, estimated with PC analysis concatenated local operative dimensions L (Eq. and and ITI (d) Network output cost of networks with reduced-rank weight matrix
WOP for k = 1 N (Eq. 12 (e) State distance between trajectories in the full-rank network and in networks with reduced-rank weight matrix W (c-e) Based on global operative column dimensions; averaged over networks per number of inputs/outputs; shaded area: mad. Network output cost obtained with internal and input noise, state distance without any noise. (f-h) Same as (c-e) for global operative row dimensions.
One consequence of the tight link between local network dynamics and operative dimensions is that the number of operative dimensions required to approximate particular network may increase with the complexity of the network activity, and of its inputs and outputs [23]. To illustrate this relation, we systematically varied the complexity RNN computations by training sine wave generators with varying numbers of input and output signals (between - 5 inputs U and outputs Z; Fig. 4. As expected, the network activity x; becomes increasingly high-dimensional for increasing values of U and Z (Fig.Eb) and correspondingly the number of global operative dimensions required to achieve the original performance increases as well (Fig. 4p-e and f-h). obtained similar effect when increasing the dimensionality of the inputs into the RNNs, while keeping the dimensionality of the output fixed. (Fig. T3). Analytical considerations also imply that, vanilla RNNs described by Eq. [ the number of global operative dimensions is tightly linked to the dimensionality of the network activity R (section A.2.4 and A.2.5).
2.2.3 Operative dimensions relate functional modules to weight subspaces
Past studies have shown that RNNs can implement complex, context-dependent computations by "tiling" activity state space into separate functional modules [21 19 24). Dynamics within individual modules is often approximately linear, but different approximately linear dynamics, corresponding to different input-output relations, are implemented across modules (20).
Our definition of operative dimensions is well-suited to ask whether the existence of functional modules activity space has a correspondence at level of the structure the connectivity. To obtain a more fine-grained mapping from function to structure, the global operative dimensions can be generated based on different subsets of local operative dimensions. Specifically, sampling locations can be grouped based on their functional meaning. i.c. based which condition average they belong to. The resulting sets of function-specini global operative dimensions can then link different network functions particular subspaces in the weight matrix.
We demonstrate this approach for the context-dependent integrator network. We inferred global operative dimensions from local operative dimensions that were collected either separately by context, but pooled over choice, or separately by choice, but pooled over context (details section A.2.6).
We refer to the resulting context-dependent global operative column dimensions q, (contex),, and the choice-dependent dimensions as q, (choice,) (i N, = 2; row dimensions: q. We compared these dimensions directly by measuring their pairwise alignment (subspace angle,
7

---

a
texty
didce
oor bext2
choloes
20
column
36
dimensoon
0.5
0.5
20
%
50 renkiw
(contexti)
(choice))
rankit (11))
contesti)
choice) choice2
18
20
dimressoos
10
055
%
25
50
75
100
25
5g
75
100
9. (ranteed))
Fank/W
ronktW ronki
4.2
422"
Figure 5: Function-speciff global operative dimensions. (a) Subspace angle between global operative column dimensions defined separately for context, and contextz. (b) Subspace angle between global operative column dimensions 9; defined separately for choice; and choices. (c-d) Same as (a-b) for global operative now dimensions 9. (e) Network output cost averaged separately over trials of context, or contexty for networks with reduced-rank weight matrix consisting of the first & function- specific global operative column dimensions from context, k.j. context j 1). (f) Network output cost averaged separately over trials of choice; choice? for networks with reduced-rank weight matrix consisting of the first & function-specifie global operative column dimensions from choice, (WOPone, choice j = 1). (g) Same as (e), but based on global operative column dimensions k.j from context 2. (h) Same as (f), but based on global operative column dimensions from choice 2.
Fig-5a-m -1dd. We find that the function-specific global operative dimensions are only weakly aligned across the two contexts, but more strongly aligned across choices, suggesting that the RNN uses different weight subspaces to implement the sensory integration in each context, but reuses weight structures to implement the sensory integration across choices.
To further support this conclusion, we checked if the global operative dimensions inferred from one functional module can support accurate computations another module. We first constructed module specific, reduced-rank approximations of W as above (Eq. 12 which we refer as W (context-
k.j
specific) and Wk. WOPcn. sehoice-specifiee We then measured network performance across conditions when using these different reduced-rane -ank approximations (Fig. (p- In agreement with the subspace- angles above, we find that networks based on operative dimensions defined in context, perform poorly in context,, and vice versa (Fig. 5e. g) whereas operative dimensions from a given choice yield comparatively good performance also in the other choice (Fig. 5}. h). These observations are in line with previous findings [19] that the context-dependent integrator implements approximate line attractor in each context, with mostly preserved dynamics along given line attractor (i.e. across choices), but rather different dynamics between attractors (i.e. across context). Interestingly, the identified structure in the underlying connectivity, while clearly revealed by the operative dimensions (Fig. (5), does not become apparent when studying the weight matrix contributions of individual network units directly (Fig. $19
3 Discussion
In this work, we present an approach for inferring the operative dimensions of an arbitrary weight matrix in an RNN. The operative dimensions of the connectivity are defined based on their impact on the computationally relevant, local recurrent dynamics of the network. We find that for the examined tasks, the operative dimensions span a oow-dimensional subspace of the connectivity space that is sufficient to produce accurate outputs reduced-rank network.
Relation to state of the art. Our framework of operative dimensions extends recent work based on RNNs that by construction are explicitly low-rank, which already showed that many tasks of relevance to neuroscience can be solved with RNNs that rely entirely low-rank connectivity [[7].
8

---

Unlike in this previous work, the networks we analyzed have a connectivity that is not constrained- the dimensionality of the underlying weight matrix high at initialization, and remains high throughout training (Fig. [d. i). This high dimensionality persists even though the weight updates brought about by training are low-rank [22]. However, our analyses showed that only a low-dimensional subspace of the effectively nigh-dimensional connectivity is used to solve the task, whereby the remainder of the structure in the weight matrix plays essentially no role in solving the task at hand. This finding opens the possibility to apply the conceptual framework of low-rank networks [17], which has provided valuable and direct insights into the relation of connectivity and dynamics, also to other types of RNNs.
Operative dimensions amount to a form of model reduction 25 26 that emphasizes the preservation of attractor dynamics [27 28 for the special case of a model parameterized as neural network. Critically, our approach to relating the model structure to its function relies on characterizing the dynamics locally, along explored activity trajectories, similarly to methods to quantify non-linear
dynamics based on Lyapunov or bred vectors [29
More generally, our work makes a contribution towards increasing the interpretability of artificial neural networks. Past work had shown that unconstrained, trained RNNs need not be considered as a "black box", but rather that the computations implemented by many RNNs can be understood at the level of population dynamics. through the interaction of inputs and dynamical primitives like attractors and saddle points [20] 33 [24]. Recent work related the implemented dynamical primitives to the underlying connectivity [77 in constrained RNNs. We have shown that establishing
such relations is in principle also possible through the definition of condition-dependen operative dimensions (Fig-E) 5 without any specific constraints on the connectivity. Current opinions assume that such increased understanding and interpretability of artificial networks is desirable both to increase acceptance of the resulting machine learning applications throughout society, but also potentially to design better artificial system that overcome biases and limitations resulting from current approaches (see 3 35 for reviews). Nonetheless, potentially negative societal impacts of increased interpretability cannot be ruled out. For instance, if used on RNNs trained on personal data, operative dimensions may potentially facilitate the extraction of private information that was otherwise hidden in high-dimensiona weight matrices.
Limitations. One limitation of our approach to identifying operative dimensions is that it relies on studying local recurrent dynamics based on a somewhat arbitrary definition of sampling locations in activity state space. By design, the inferred operative dimensions will be sufficient to reproduce the full-rank dynamics only at these specific locations in state space. Note that there is no explicit requirement for the operative dimensions to reproduce the desired network output. We picked sampling locations along the condition average trajectories, based on the assumption that these average trajectories provide a good coverage of all the relevant local dynamics. In practice, this assumption may not hold in all RNNs. For one, average trajectories may travel through state-space regions that are not visited by any single trials, leading one to under-sample functionally relevant regions. For another, single-trial and average trajectories may also reflect dynamics that is not functionally relevant, and thus lead to over-sampling of regions that are not directly involved in generating the output. Further work may address alternative approaches for choosing sampling locations to optimize their functional relevance. To some extent, the adequacy of sampling locations can be tested empirically by systematically varying the number and location of sampling locations to optimize the performance of the inferred reduced-rank networks (Fig. $18).
Another potential limitation of approach is that it may not be equally effective in identifying functional subspace of the connectivity across all types of RNNs and in more complex tasks. While here we have focused on vanilla RNNs, we found that a oow-dimensional functional subspace can be identified in such RNNs for a variety of network architectures, including with non-overlapping populations of excitatory and inhibitory neurons, or the use of different non-linear activation functions (Fig. ŞT5). The properties of learned dynamics in the kind of tasks we employed is largely conserved across different types of networks (GRU, LSTM; [1] 24). ), in particular the role of "tiling" activity space into distinct computational modules. This observation implies that our approach to relating connectivity and function at the level of local dynamics is at least meaningfully applicable even in these different kinds of networks. While we find that operative dimensions can retrieve a functionally relevant subspace also for richer task like sequential MNIST (Fig. ST6).. it remains an open question whether these approaches will extend to harder Al problems.
9

---

Conclusion Operative dimensions can infer a functionally relevant subspace within high- dimensional RNN connectivity that is sufficient perform the task at hand. On one hand, this observation might benefit practical applications of RNNs as computational tool, e.g. for continual learning in RNNs, by specifically protecting functionally relevant subspaces of the weight matrix to avoid catastrophic forgetting [36), or for weight compression, by storing the weight matrix as linear combination of the global operative dimensions [37 [2]. On the other hand, the ability to identify functionally specific subspaces in the network connectivity may improve the applicability of RNNs as model in neuroscience, as simplifies the critical challenge of linking the properties of the connectivity to the network dynamics and may thus provide guidance on how to relate structure to function in complex biological datasets.
Code Availability
We used Matlab to perform the data analysis. All the code and models required to reproduce the main analyses supporting our conclusions are available in matlab and python at: lnttps///gitlab. com/ neuroinf mrtrronnf//perratiieemmeennionoo
Author Contributions
R.K. and V.M. conceived the idea for the study, R.K. performed the analyses. M.C. and G.I. provided feedback throughout project. R.K. and V.M. wrote the manuscript. R.K., V.M., M.C. and G.I. reviewed the manuscript.
Acknowledgements
We thank all group members (Neural Computation and Cognition (V.M),, Cortical Computation (M.C.) and Neuromorphic Cognitive Systems (G.I.) for their valuable feedback and discussions.
Funding
This work was supported by the EU-H2020 FET project CResPACE (Grant No. 732170, G.I.), grants from the Simons Foundation (SCGB 328189, V.M.; SCGB 543013, V.M.) and the Swiss National Science Foundation (SNSFPPOOP3 V.M.).
References
[1] Peiran Gao and Surya Ganguli. On simplicity and complexity in the brave new world of large-scale neuroscience. Current opinion in neurobiology, 32:148-155, 2015.
[2] Danielle S Bassett and Olaf Sporns. Network neuroscience. Nature neuroscience, 20(3): 353-364, 2017.
[3] Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiolggy, 46:1-6, 2017.
[4] David Sussillo. Neural circuits computational dynamical systems. Current Opinion in Neuro- biology. 25:156-163, 2014. ISSN 0959-4388, doi: http:///bb.orr//1..0166.....obb URL htttp::///eett sci htt.e//ccceeeeteoc/c///a////aeea http:///caaacoeerrcceco////////a Theoretical and computational neuroscience.
1.000.
[5] L.F. Abbott. Theoretical neuroscience rising. Neuron, 60()):899-455, 2008. ISSN 0896-6273, doi:
2008. 10..0........0..........0....... URL https://www.es. wwtp://wcienncdeiee.om
[6] John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-255 1982.
[7] Rani Yishai, R Lev Bar-Or, and Haim Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, 92(9):3844-388) 1995.
10

---

[8] Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A: Mathematical and general, 21(1):257, 1988.
[9] Carl Van Vreeswijk and Haim Sompolinsky. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, 27445293):1244-1725 1996.
[10] Francesca Mastrogiuseppe and Srdjan Ostojic. Intrinsically-generaae fluctuating activity in excitatory-inhibiorry networks. PLoS computational bioloy, 13(4): 1005498, 2017.
[11] Johnatan Aljadeff, Merav Stern, and Tatyana Sharpee. Transition to chaos in random networks with cell-tyee-specifcc connectivity. Physical review letters, 1 114(8):08800,, 2015.
[12] Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, 61(3):259, 1988.
[13] Alexander Rivkind and Omri Barak. Local dynamics in trained recurrent neural networks. Physical review letters, 118025) 2017.
[14] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
[15] Naday Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. arXiv preprint arXiv:2201. 12760, 2022.
[16] Carina Curto, Jesse Geneson, and Katherine Morrison. Fixed points of competitive threshold- linear networks. Neural computation, 31(1):94-155, 2019.
[17] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computa- tions in low-rank recurrent neural networks. Neuron, 99(3):609-633 623, 2018.
[18] Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. Complementary roles of dimensionality and population structure in neural computations. biorxiv, 2020.
[19] Valerio Mante, David Sussillo, Krishna Shenoy, and William Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7474):78 84, 2013.
[20] David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-649, 2013.
[21] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems, 32, 2019.
[22] Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in mns. Advances in neural information processing systems, 15:1332--13622 2020.
[23] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, page 214262, 2017.
[24] Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with jacobian switching linear dynamical systems. Advances in Neural Information Processing Systems, 34:16700-16713, 2021.
[25] Bart Besselink, Umut Tabak, Agnieszka Lutowska, Nathan Van de Wouw, H Nijmeijer, Daniel J Rixen, ME Hochstenbach, and WHA Schilders. A comparison of model reduction techniques from structural dynamics, numerical mathematics and systems and control. Journal of Sound and Vibration, 332(11):4403- 4422, 2013.
[26] Peter Benner, Serkan Gugercin, and Karen Willcox. A survey of projection- based model reduction methods for parametric dynamical systems. SIAM review, 57(4) 531, 2015.
[27] Ivan Ivanov, Ranadip Pal, and Edward R Dougherty. Dynamics preserving size reduction mappings for probabilistic boolean networks. IEEE Transactions on Signal Processing, 55(5): 2310-2322, 2007.
[28] Aurélien Naldi, Elisabeth Remy, Denis Thieffry, and Claudine Chaouiya. Dynamically consis- tent reduction of logical regulatory graphs. Theoretical Computer Science, 412(21).220--2211. 2011.
11

---

[29] Francesco Ginelli, Pietro Poggi, Alessio Turchi, Hugues Chaté, Roberto Livi, and Antonio Politi. Characterizing dynamics with covariant Iyapunov vectors. Physical review letters, 99 1130:130601, 2007.
[30] Christopher L. Wolfe and Roger M Samelson. An efficient method for recovering Yyaрunov vectors from singular vectors. Tellus A: Dynamic Meteorology and Oceanography, 59(3): 355-366, 2007.
[31] Zoltan Toth and Eugenia Kalnay, Ensemble forecasting at nme: The generation perturbations. Bulletin of the american meteorological society, 74(12)2317-2330) 1993.
[32] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systems, 32, 2019.
[33] Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang, On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiation and Plasma Medical Sciences, 5(6): 741-760, 2021.
[34] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
[35] Quan-shi Zhang and Song-Chup Zhu. Visual interpretability for deep learning: survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27-39, 2018.
[36] Lea Duncker, Laura Driscoll, Krishna V Shenoy, Maneesh Sahani. and David Sussillo. Organiz- ing recurrent network dynamics by task-computttion to enable continual learning. Advances in neural information processing systems, 33:14387-14397, 2020.
[37] Genta Indra Winata, Andrea Madotto, Jamin Shin, Elham J Barezi, and Pascale Fung. On the effectiveness of low-rank matrix factorization for Istm model compression. arXiv preprint vrXiv:1908.09900 2019.
[38] James Martens and Ilya Sutskever. Learning recurrent neural networks with hessian-free optimization. In JCML, 2011.
[39] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):
1735-1780, 1997.
[40] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint 1:X9.:1409.1255 2014.
[41] Herbert Jaeger. Adaptive nonlinear system identification with echo state networks. Advances in neural information processing systems, 15, 2002.
[42] Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural compu- lation, (4(11):2531-2566 2002.
[43] H Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Training excitatoryinnhibioryy recurrent neural networks for cognitive tasks: simple and flexible framework. PLoS computational biology,
2016.
[44] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144 2012.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Main claims are what is shown in the main results
figure (Fig. 3p.
(b) Did you describe the limitations of your work? [Yes] See section 2.2.2 and limitation section in discussion (section[3) 3).
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read and adhere the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See equations in main and detailed descriptions in appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] See various sections in appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes] Code. models and example set of operative dimensions are provide.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]] Exact code to reproduce results provided, including all parameters. Additionally, details are explained in respective sections of text and appendix.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] Mean + median absolute deviation are reported where possible and details described in the corresponding figure caption.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section A.2.7
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Yes, we cited the corresponding papers.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes] Yes, we provide the code to reproduce our results.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A] We are not using any human-derived data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We are not using any human-derived data.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ydd29rWKkgF", "reviewer": "Reviewer_QY7N", "summary": "The authors study how unconstrained training of recurrent neural networks results in structured synaptic connectivity. Specifically, they define a novel decomposition of the synaptic matrix that reveals this structure. In contrast to a naïve decomposition of the matrix, this novel form is relatively low-dimensional. \nThe new suggestion is to define operative dimensions as those directions in weight space that lead to the maximal disturbance to neural dynamics. These are defined locally, around points of interest, and then integrated into a global measure. \n", "questions": "1.\tBesides reference 20, there are also relevant works in the feedforward domain. A couple of examples: \na.\tSaxe, Andrew M., James L. McClelland, and Surya Ganguli. \"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks”\nb.\tTimor, Vardi, Shamir, “Implicit Regularization Towards Rank Minimization in ReLU Networks”\n2.\tL38-39: “… it remains unclear … findings can be generalized to RNN … not subject to such constraints”. – This sentence highlights the need to better relate to existing literature.\n3.\tEquation 1 and Line 57. If I understand correctly, the random noise is drawn independently for every dt. Stochastic differential equations require scaling the noise with sqrt(dt). If dt is not changed, this just amounts to an “incorrect” scaling of the noise. But, in general, it is not an exact implementation of equation 1, and could lead to problems when checking sensitivity for dt for instance.\n4.\tIt is not entirely clear to me why the operative dimensions reveal cases similar to figure 2. If a change in W causes a trajectory to cross a separatrix, the local changes might be small, but the resulting global (t -> infty) changes could be large. The discussion in section 2.2.1 explains why a small change in local dynamics can lead to dramatic performance changes. But operative dimensions search for large local changes.\n5.\tL228: Another relevant reference here is Smith, Linderman and Sussillo “Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems”\n6.\tIs there any relation between these results and the eigenvalue spectrum of W, or of its linearized versoins? If I understand correctly, the authors show that singular value decomposition is uncorrelated to these dimensions. On the other hand (following the jacobian switching linear systems reference above, and the results in A.2.3), is there a relation to the eigenvectors of the linearized dynamics around the points used?\n", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis is an important problem, and there are few studies tackling it. The suggested approach is, to the best of my knowledge, novel.\nWeaknesses:\nA main weakness is novelty with respect to reference 20 (Schuessler et al, NeurIPS 2020). That paper tackled the same problem, and showed both numerically and analytically that the changes to connectivity (delta W) are low rank. There are certainly differences between the papers, but given the similarity, this is hardly discussed.\nAnother relevant reference (that is not cited) is Smith, Linderman and Sussillo “Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems”, NeurIPS 2021. In that paper, roughly speaking, trained networks are approximated by several local dynamical systems. It would seem that the analysis of A.4.3 of Vanilla might be extended to general networks by using the Jacobian at each point in phase space.\nThe paper also has very little analytical results – specifically, the question of why this low dimensionality develops is not addressed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m_CfEF-4Xp6", "reviewer": "Reviewer_emwn", "summary": "This paper works towards an understanding of how weight structure influences the dynamics of recurrent neural networks (RNNs). It is shown that the principal components (PCs) of the weight matrix do not provide a good compression that preserves dynamics, instead a more dynamically-informed low-rank compression of the weights into \"operative dimensions\" can do so. These are defined (locally, along the attractor, similar to Lyapunov exponents) as the direction which maximizes norm of the divergence of trajectories. Compression of the W matrix using either PCs or operative dimensions is compared with a few RNN tasks used in computational neuroscience.", "questions": " If the RNN dynamics are linearized, I believe that there is a direct connection to the singular vectors of the dynamical map. Maybe PC directions, in this special case, correspond to operative dimensions? This seems like an obvious connection to explore. Also, spending a bit more time in the text talking about the methods you use to optimize Eqn 8 and 9 would be helpful.\n\nDo you see alignment of the operative dimension components with the B matrix components? The alignment (or lack thereof) to the W matrix components is explored extensively; this seems like an obvious one to also explore.\n\nIn the discussion, it would be worth discussing connections to model reduction methods, eg for PDEs and high-dim ODEs.\n\nPreservation of attractor basin boundaries seems like something these operative dynamics clearly address, but I don't think this was discussed.", "limitations": "I found the authors did a good job discussing limitations and potential ethical issues.\nHowever, I do think they could mention that their results are limited to studying very simple tasks, much simpler than the AI type tasks where RNNs are employed in practice.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this paper to be very well-written and interesting. I didn't read the supplement due to lack of time. The way of defining operative dimensions is elegant and seems to be somewhat numerically efficient (it is tested for weight matrices with 1000s of dimensions). I liked how Fig 2 shows how these dimensions affect the attractor structure of the RNN, and I enjoyed the discussion.\n\nThe main weaknesses I found were in the tasks that were chosen. These tasks are somewhat artificial although I like their grounding in neuroscience-related tasks. For the NeurIPS audience, it would be interesting to see some whether operative dimensions offer insight into a more \"AI\" RNN task. Perhaps this is beyond the scope of the current paper.\n\nI think the paper would be strengthened by exploring the connection between your ideas and more classical dynamics ideas such as Lyapunov vectors or bred vectors. You are essentially using the same idea, calculating directions of maximal growth in trajectories, except you are doing it in the weight space rather than the state space. This is interesting.\n\nSomething that could describe the low-dimensionality that is observed here is the dimensionality of the control signal u/matrix B. It could be that this low-dimensional drive forces the RNN onto an attractor of similar dimension. That could explain the patterns you observe in Fig 4. I imagine there is extensive literature that could explain this effect. \n\nSome detailed comments:\nThroughout the text you use \"RNN\" to refer to \"recurrent neural networks\" in the plural. I think you should change this to \"RNNs\" for the plural, which is more common.\n\n148: \"dimension of the connectivity a\" suggest changing to unambiguous language \"unit vector a\"\n\n150, eqn 6: maybe worth mentioning this is an orthogonal projection\n\n283: \"This past work had however not\" awkward language. Also, I believe the low-rank RNN work from Mastrogiuseppe and Ostojic did relate to attractor structure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LqIy6HjE81V", "reviewer": "Reviewer_Q6im", "summary": "This paper presents a framework and method to analyze the connectivity of unconstrained RNN weights. They find that while recurrent weights are high-dimensional, the network dynamics that actually solve simple neuroscience tasks (i.e. ones that map stimulus inputs to one of two choices) are low-dimensional. These low-dimensional subspaces are estimated by removing different dimensions and examining the effect on activity state space. These are referred to as operative dimensions, which contain functional modules for the task. The operative dimensions can be related back to the recurrent connectivity matrix W by the construction of a reduced-rank connectivity W^{OP}. Finally, they illustrate how this framework allows one to relate different functional modules to subspaces in the weight matrix, which in essence relates function to structure.", "questions": "* Fig. 3C shows that network performance gets worse with the first few operative dimensions before it gets better. The authors hypothesize in lines 201-206 that the first few operative dimensions may be required at state space locations explored late in the trial, rather than at the beginning of the trial. Is there any evidence to support this?\n* It is interesting that the activity of the RNN is low-dimensional even before training. Why is this?\n* Any thoughts from the authors on how this might scale to stacked RNN architectures?\n\nSuggestions\n* The citations in the text are numbered, but the reference list is not. Please fix.", "limitations": "The authors thoroughly discuss the limitations to their approach, notably their choice to sample the average trajectory in activity state space. They also provide supplementary analyses to address some of the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, with sufficient detail and supporting material in the appendix. The math is straightforward and clear. The methods are certainly based on some prior work, but the general framework and findings are novel and interesting. The authors are also very thorough with additional analyses and explore nearly all the related questions that arose as I read the paper.\n\nTheir findings are significant in that they apply to unconstrained RNNs, which brings the literature closer to real-world uses of neural networks. They also could be applied to other areas in machine learning (continual learning or weight compression). And, as the authors note, the findings could also make it easier for neuroscientists to use RNNs as models of biological datasets. This paper could clearly have impact on several sub-areas.\n\nOne of the most interesting findings, in my opinion, is that the number of operative dimensions increases with with task/network complexity. It will be interesting to test this point further by independently varying stimulus complexity and output complexity. For example, how many more dimensions are required as one increases the number of nonlinearities required to produce some output? To be clear I am not suggesting that the authors conduct another analysis now -- there is already more than enough in the current paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8y1lY04VfPI", "reviewer": "Reviewer_49zS", "summary": "The paper presents the operative dimensions -- a novel, more effective tool (e.g. compared to PCA of W, X or R) to characterize the functionally relevant subspaces of RNN weights. The tool is well designed (Sec 2.2.1), easy to compute (appendix), and able to identify previously unknown properties of RNNs, e.g. the existence of low-dimensional functional subspaces (Sec 2.2.2) and the weight-space functional modules (Sec 2.2.3).", "questions": "- The column and row dimensions’ results are always both presented in the main text. Their difference is however not clearly explained. E.g., what do they capture differently? Are they complementary to each other, e.g. stronger when combined? Why should we use both? Etc.\n- choice_1 and choice_2 don’t seem to be explicitly defined in the paper. Do they mean when the output is positive or negative? If so, do sensory_1’s mean and sensory_2’s mean always have opposite signs? Or, do they mean when sensory_1 or sensory_2 is picked by the RNN? If so, how is the detection done? And shouldn’t context_1 always lead to choice_1 (and context_2 to choice_2) when the RNN is fully trained?", "limitations": "The authors have reasonably addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ (Originality) The proposed tool (operative dimensions) is novel to the best of my knowledge, as well as the two main findings regarding previously unknown properties of RNNs (as summarized above).\n+ (Quality) The proposed tool is well designed (as dimensions with highest impact on local dynamics when removed), easy to compute (closed-form for column dimensions), and carefully validated with lots of experiments and analyses in the main text and appendix.\n+ (Clarity) The writing is overall clear and easy to follow. The figures are well-made and very helpful.\n\nWeaknesses\n- (Significance) Although the two tasks (context-dependent integration and sine wave generation) are well studied and commonly used in computational and theoretical neuroscience papers, it’s hard to predict if the proposed tool (validated on them) and results (derived from them) really generalize to harder, more realistic tasks. Additional experiment(s) using e.g. sequential MNIST, TIMIT, or any of the possible applications described in Sec 3, could greatly enhance the significance of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept