Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics
Authors: William Qian
Jacob A Zavatone-Veth
Benjamin Samuel Ruben
Cengiz Pehlevan
Venue: NeurIPS 2024 poster (2024)
Keywords: neuroscience
recurrent neural networks
dynamical systems
Primary area: neuroscience_and_cognitive_science

Abstract: One of the central goals of neuroscience is to gain a mechanistic understanding of how the dynamics of neural circuits give rise to their observed function. A popular approach towards this end is to train recurrent neural networks (RNNs) to reproduce experimental recordings of neural activity. These trained RNNs are then treated as surrogate models of biological neural circuits, whose properties can be dissected via dynamical systems analysis. How reliable are the mechanistic insights derived from this procedure? While recent advances in population-level recording technologies have allowed simultaneous recording of up to tens of thousands of neurons, this represents only a tiny fraction of most cortical circuits. Here we show that observing only a subset of neurons in a circuit can create mechanistic mismatches between a simulated teacher network and a data-constrained student, even when the two networks have matching single-unit dynamics. In particular, we show that partial observation of models of low-dimensional cortical dynamics based on functionally feedforward or low-rank connectivity can lead to surrogate models with spurious attractor structure. In total, our results illustrate the challenges inherent in accurately uncovering neural mechanisms from single-trial data, and suggest the need for new methods of validating data-constrained models for neural dynamics.

Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics
William Qian Jacob Davatone-Veth3... Benjamin S. Ruben', Cengiz Pehlevan 2,3,4 'Biophysies Graduate Program, Kempner Institute for the Study of Natural and Artificial Intelligence, John A. Paulson School of Engineering and Applied Sciences,
*Center for Brain Science, Department of Physics, Harvard University
Cambridge, MA 02138
jzavatonevethāfas haree harvard. cpehlevandseas harvard. edu
Abstract
One of the central goals of neuroscience is to gain a mechanistic understanding of how the dynamics of neural circuits give rise to their observed function. popular approach towards this end is to train recurrent neural networks (RNNs) to reproduce experimental recordings of neural activity. These trained RNNs are then treated as surrogate models of biological neural circuits, whose properties can be dissected via dynamical systems analysis. How reliable are the mechanistic insights derived from this procedure? While recent advances in population-lewe levell recording technologies have allowed simultancous recording of up to tens thou- sands of neurons, this represents only tiny fraction of most cortical circuits. Here we show that observing only subset of neurons in a circuit can create mechanistic mismatches between simulated teacher network and data-constrained student, even when the two networks have matching single-unit dynamics. In particular, partial observation of models of dimensiooi cortical dynamics based functionally feedforward or low-rank connectivity can lead to surrogate models with spurious attractor structure. Our results illustrate the challenges inherent in accurately uncovering neural mechanisms from single-trial data, and suggest the need for new methods of validating data-constrained models for neural dynamics.
Introduction
recent years, advances in recording techniques have brought forth a deluge of neural data. Simul- tancous measurements of the activity of hundreds thousands of neurons now be obtained at high spatiotemporal resolution [1-3]. These methods are increasingly deployed to perform longitudi- nal recordings in animals executing quasi-naturalistic behaviors complex tasks [2-7], meaning that one may not have recourse to repeatable trial structure when analyzing these data [8]. critical question for contemporary systems neuroscience then arises: How mechanistic insights about the neural dynamics underlying animal behavior be extracted from large-scale recordings [3, 5, 7, 9, 10]?
Given access only single, non-repeatable measurement of neural activity, a natural question is whether one could construct a reliable vílico surrogate model the dynamics of the measured neural circuit. As this surrogate model would not be subject to measurement limitations, it could be used to generate hypotheses about the corresponding biological neural populations, and to simulate how such populations might behave under various external inputs perturbations. natural approach to constructing a surrogate model is to optimize a recurrent neural network (RNN) to mimic the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

recorded neural activity. In recent years, this approach has gained broad popularity, and has been applied to data from many species and recording modalities [9, 11-20].
Data-driven models of neural dynamics are constructed under a number of less-than-ine teaaeeaa--deal conditions. First, in single-trial settings, one has access to measurements of dynamics only in restricted condition or set of conditions, so it may be impossible to observe how changes in input or in internal state affect the circuit of interest [8]. Second, the system of interest is only partially observed: one can only usually record from a subset of the neurons in a given circuit, and certainly cannot simultancously record all of their inputs [7]. Third, measurements of neural activity are noisy, and may be biased to capture only certain components of neural activity. For example, intrinsic indicator dynamics mean that calcium imaging may effectively low-pass filter neural activity, even if one is only interested in firing rates and not precise spike times [21, 22]. Fourth, it can be challenging to account for the presence and structure of intrinsic neuronal noise. Finally, there will always be a significant mismatch between the single-unit dynamics of a model and biology, as models abstract away or ignore biophysical details to enable efficient optimization and simulation [17, 18].
Even in the unrealistic scenario where the activity of every relevant neuron is recorded, exactly inferring synaptic weights from dynamical measurements alone extremely challenging [23]. A more modest hope is that data-constrained models should be able to capture the mechanistic dynamical properties of ground-trum -trut circuits at a qualitative level- that is, to recapitulate slow time scales, unstable directions, oscillatory dynamics, and attractors [9. 12-14, 17, 19, 24, 25]. Given a data- constrained model, one can identify attractor properties using dynamical systems analysis [13, 26, 27]. These macroscopic dynamical properties are of substantial neuroscientific interest, as now-dimensional attractors are believed to underlie observed neural activity across variety of neural circuits and tasks [24]. In particular, line attractors sets of stable fixed points organized along lines in neural activity space-have been proposed to underlie cognitive functions requiring short-term or working memory, including sensory integration, decision making, and even aggressive behavior [12, 13, 24, 26, 28, 29]. Importantly, low-dimensional structures in data should be relatively robustly detectable even under partial observation [7]. so there is reason to be optimistic that data-constrained models could correctly recover line attractor dynamics. Indeed, several recent papers have used contsroontr models with -wwdimmensioall latent RNN dynamics to propose that line attractors underlie the accumulation of internal drives and of external reward [12, 13, 30, 31].
However, despite some positive examples [14, 191, previous works have not mapped out how partial observation affects whether data-driven modeling can accurately recover oow-dimensional attractor structure. To address this question, in this paper, we consider teacher-student setup in which activity from one RNN is imitated by another, and show that partial observation can induce mechanistic mismatches even under relatively ideal conditions where the input to a circuit is either perfectly known or white noise, and where the single-unit dynamies of the student match the teacher. Our primary contributions are as follows:
In $2. we begin with motivating example: we show that data-constren tinguish between two mechanisticalll-dsstnne models in a stimulus intuurratnnnntnnnnn task. Both a line attractor network [28] and functionally feedforward chain [32] are identified as line attractors. We then turn to the analytically-traatian show that when the teacher is an approximate line attractor, the student will recover this struc- ture. In contrast, when the teacher connectivity is non-normal, the student may learn spurious approximate attractor structure. We illustrate this with two biologically-motvve motivated examples: functionally-feedforvee
modeling fails to dis-
setting of noise-driven linear RNNs ($3). In $3.1, we
integrators ($3.2), and networks with low-rank connectivity ($3.3).
Then, in $4, we explore how these insights generalize beyond the linear setting. Focusing on the example of nonlinear low-rank networks, we show that partial observation once again can induce overestimation of cigenvalue magnitude. Here. though. this can result in spurious attractor structure including additional stable fixed points and limit cycles.
Our results begin to illuminate the inductive biases of fata-constrained RNNs trained under partial observation towards particular mechanisms of generating long timescales. They suggest that caution warranted in inferring mechanism from data-constrained models, and underscore the primacy of direct activity perturbations for validating putative attractor dynamics [25].
2

---

a
b
1)
ii)
MaL00E sundea
0.5
Ipp
MooMME (AA)) Stmmas Gali
0.5-
0.83-
-0.3
0.2-
ii))
Ground
iv)
Line altracctrr score: 5.67
Ground truth
iV)
Line attractor score: 6.13
Naure Sapponn (A.2
time constant T.
400
300
Presporse IA
time consznnt
.300
280
200
223
Leamned
200
Learned
100
100
25
Dimension
Dimension
Learned
vi)
Ground Truth
Learned
vi)
Truth
19
PC2 (A.U.)
PC2 (AUJ)
PC2 (A.
PC2 (A.U.)
20-
100
00.
10
05
100
PC1 (A)
-100
PC1 (A.U.)
1:0
0.0
PC1
(A.U.)
20
PC1 (A.U.)
50
102
Figure 1: Data-constr models fail to distinguish between mechanistically different sensory integration circuits. a. Recovery of a line attractor through data-constrained modeling. i). Schematic of integrator network, showing the subsampled neurons (blue). and its interpretation as set of independent self-excitatory modes. ii). Input signal (lop) and its integral (bottom) as estimated by the network (green) and computed exactly (black). iii). Example activity traces from the true network (top) and an LDS fit to observations of 5% of its neurons (bottom). The agreement is excellent. iv). Spectrum of time constants for the data-constrained LDS model (main figure) and for the top five time constants of the true circuit (inser). Both show a single large time constant, indicating approximate line attractor dynamics, though the cata-conntrained model underestimates that of the true network. v-vi). Flow field in the space of the top two principal components of activity for the LDS model (v) and line attractor network (vi). Shading indicates the magnitude of the flow, while arrows indicate its direction. Observed activity is shown by dots colored by their time. The learned flow field shows good qualitative agreement with the ground truth; both networks have a slow line along which the observed activity is driven. b. As in but for functionally-feecffoovaa integrator circuit. As diagrammed in (i), this network can be thought of as a set of non-self-excitinn modes which are connected in feedforward chain. Though this network solves the integration task (ii) and the LDS fit is good (iii), the LDS identifies a single long time constant that is not present in the true dynamics (iv). The learned (v) and ground-truth (vi) flow fields correspondingly do not match, with the activity lying off the slow line of the true dynamics. See Appendix G for detailed experimental methods.
2 A motivating example: data-constrained modeling of integrator circuits
The circuit basis for temporal integration of scalar sensory inputs is longstanding question in systems neuroscience [12, 24, 28, 29, 32-41]. Though many models for integrator circuits have been proposed [28, 32, 37, 40, 42], two linear RNN models are perhaps the most prominent: the line attractor [28, 40J, and the feedforward chain [32, 37J. Both of these models have extremely simple dynamics
Tă = -2 + Jz + bu.
for state z E RR.,, recurrent weights E RDD,, and input u(t) E IR encoded through b E RDD However, they posit structurally distinct mechanisms for how memories can be maintained beyond the single-unit time constant T. In classic line attractor networks, the recurrent weights are chosen to be symmetric, and one eigenvalue of J is tuned to be precisely equal to one, with the rest being less than one. Then, by choosing the input weights b to align with the corresponding cigenvector, one obtains perfect integrator of the signal u(t) [28] (App. A). However, this model suffers from a substantial fine-tuning problem: slight mis-tuning of the weights causes exponentially large deviations from perfect integration [28, 32, 37]. In contrast, functionally feedforward chain maintains a memory by iteratively passing signals from one mode of activity to the next (Fig. 1; App. A) [32, 37J. Most
3

---

simply, a literal feedforward chain has connectivity Ji = 6.41.5. However, one can encode modes in distributed patterns of neural activity rather than single neurons, so that this structure not obviously apparent in recordings (see App. A for details). Such networks are more robust to mistuning of synaptic strengths than line attractor networks, but they can only sustain a memory over (TTD) time. Importantly, the dynamics of such a network are highly non-normal; the recurrent connectivity matrix has all eigenvalues equal to zero. Here, inspired by [36], we add skip connections from each mode to the last mode in the chain (Fig. 1; App. A). This guarantees that, like the line attractor network, the activity produced by the functionally feedforward network is approximately low-dimensional.
Given the simplicity and ubiquity of these models, we first asked whether data constrannd modeling could robustly distinguish between them. We constructed a model sensory integration task, which networks of both architectures could effectively solve (Fig. 1). Using standard variational inference methods [13, 18], we fit recordings of 5% of the neurons from each network with a latent linear dynamical system (LDS), which models the neural activity as a linear projection of a now-dimensiona latent RNN [18] (App. G). These models explicitly encode the prior belief that population activity low-dimensional. In this case, we used 5 latent dimensions.
Though the data-constrained models do an excellent job of capturing the activity recorded from both the line attractor and the feedforward chain, analyzing the latent dynamics matrices reveals that both networks are interpreted as approximate line attractors (Fig. 1). In particular, the spectrum of eigenvalues Å; of each LDS dynamics matrix induces a spectrum of decay time constants 7; = 7/
RR. (in continuous time; see App. A and B.3) [13, 27]. Previous works have identified networks with large gaps between the top two timescales as approximate line attractors [13, 27]. As simple metric, Nair et al. [13] defined a "line attractor score" loga (fi /f2), and interpreted scores greater than indicative of approximate line attractors. The LDS models fitted these mechanistically different integrator circuits each have a single slow direction, with a line attractor score in excess of 6 (Fig. 1). However, visualising the flow fields of the ground truth and data-constrained models shows that the dynamics of the line attractor are qualitatively recovered well, while the model fit to recordings of the feedforward chain shows strong mismatch as it discovers a spurious line attractor (Fig. 1). Therefore, luta-driven modeling fails to distinguish circuit hypotheses for this simple task.
An intuitive explanation for why the data-driven model fails mechanistically reproduce the functionalll-feedforvval chain is immediate: if the single-unit time constants are fixed and the data- constrained model has fewer neurons, it cannot realize a feedforward chain with sufficiently long memory [32, 37]. The only way to manufacture long memory timescales with a small number of latent neurons is through large eigenvalues. This is potentially fundamental obstacle the ability of latent space models to recover neural mechanisms; we will return to this point in the Discussion.
3 A tractable model setting: noise-driven linear networks
Motivated by the observations of the previous section, we now seek a setting in which we can analytically study the structure of the student RNN's weight matrix. Whereas in $2 we assumed the teacher networks were driven by a known cow-dimesiional signal, here we consider the case in which the teacher and student are driven by isotropic Gaussian noise. This is optimistic assumption, it means that the teacher network will explore all directions in its phase space evenly over the course of single long trial [37].
Coneretely, we consider a teacher-student setup in which both networks are rate-based linear RNNs driven by isotropic Gaussian noise, or, cquivalently, their activity evolves according to multivariate -ubleinnUhleeneek (OU) processes [43]. The teacher has D neurons and a recurrent weight matrix B, such that the dynamics of its firing rate vector ztt) E R.. is
z + Bz + {t)
where E(t) is white Gaussian noise. The student's 's dynamics are identical, except that it has d neurons, recurrent weights A, and driving noise (4). such that its rate x(1) E TRd evolves as
= - + Ax (l)..
Then, the task is to estimate the student's dynamics matrix A given access only to partial observations of the teacher's activity For simplicity, we assume that we observe the first neurons of the teacher
AAll code is available at https:///itthii httpp:://gihuu....////aaa//////a htwp///ggga////ttttoooaaaaaaaaaa

---

a
b
1
ii)
1
Learned
ii)
Ground Truth
10
Ground Truth Learned
Ground Truth Leamed
PC2 (A
PC2 (AUU
16-
25
-1
-1
-20
1
O
1
-1
1
(A.U))
R(A)
R(A)
iii)
iv)
ii)
Learned
iv)
Ground Truth
1
10
Ground Truth Learned
Ground Truth Leamedd
20-
PC2 (A.U))
10
PC2
-1
1
-1
O R(A)
1
-1
1
(A.U.
R(A)
Figure 2: Partial observation of symmetric teacher networks does not lead to spurious attractor dynamics in a data-constrained student network. Ground truth teacher (red) and learned student (blue) dynamics matrix eigenvalues. (i). symmetric teacher without attractor structure. (ii)..iv): symmetric teacher that is approximate line attractor. (ii); for infinite observation time. (ii). for a finite observation time window. b. Flow fields of learned (student) and ground truth (teacher) networks for a finite observation window. (i): symmetric teacher without attractor structure. (ivi). symmetric teacher that is an approximate line attractor. All plots correspond to 5% partial observation. See Appendix G for detailed experimental methods.
network for time T, i.e. we observe
x x (t) Pz(t) for t [0, and P= (la. Odx(D-d))-
Assuming an isotropic Gaussian prior A:j miid. No0, 1/(T)) scaled such that the long-time limit well-defined, we show in Appendix B that the maximum a posteriori (MAP) estimate of A can be computed explicitly in terms of empirical covariances of x (t) [44-77]. To make the problem analytically tractable, we focus on the limit T co, where these covariances can be computed using classical results stationary states of OU processes [43, 48]. We assume that the eigenvalues of the teacher's weight matrix B have real part strictly less than one, such that it admits a stable stationary state with covariance S = lim4-+00 E[z(()z(()]] Then, in the limit of long observation window T oc, the MAP estimate the student's 's dynamics matrix can be written in terms of the stationary covariance S (App. B)
Ax = PL=PBSPTTPP (PSPT + pla)-1
This result is stated continuous time; we also give the corresponding result for discretized dynamics in Appendix B. In the fully-observed case, the zero-ridge limit of the MAP recovers the teacher dynamics matrix, i.e., limpyo Abola=D B. The stability condition means that can consider best approximate line attractors with arbitrarily large but not infinite time constants, but this is not a substantial limitation [28, 32,37J. 37].
To determine when this data-driven modeling approach recovers the mechanistic structure of the teacher, our task then to analyze the spectrum of Aso for various choices of B, as for linear networks the eigenspectrum fully determines the (approximate) attractor structure [28]. Coneretely, letting X; and X be the eigenvalues of the teacher and student dynamics matrices, respectively, we want to compare the resulting spectra of timescales 7, = T 7/|l RAI and = 7/ 1 RÅ.l. We are primarily interested in whether the existence or non-existence of slow timescales can be accurately recovered.
3.1 Normal dynamics
We begin by considering teacher networks with normal connectivity matrices (BB = BT B). This includes attractor networks like the idealized line attractor, which have symmetric connectivity (B B T). and when driven by noise have equilibrium stationary state [43, 48). As such matrices
5

---

b
Line attractor score
0 104
10
0- 202
Time constant i
08
402
0.-
0- 902
0.
0- 1033
972
02-
0.0
10th
2014
10
102
10
201
Subsampling fraction
(Teacher size)
(Teacher size)
Figure 3: Heavily subsampling a feedforward chain leads to line-attractor-lik student dynamics. Line attractor score as function of subsampling fraction d/D D for teacher networks of varying sizes D. b. Real parts of the top two eigenvalues of a = 25 student's 's dynamics matrix for varying teacher network size D. As in b., but showing the time constants corresponding the top two cigenvalues. Beyond a threshold value of D, the separation increases rapidly. Thus, the student shows two mechanistic mismatches: First, it learns dynamics matrix with non-vanishing eigenvalues. Second, at sufficiently low subsampling fraction the top two eigenvalues are separated by substantial gap, yielding anneaatractor-liie dynamics. See Appendix G for detailed experimental methods.
have orthogonal cigenspaces, the dynamics of normal teacher network can be viewed as a set of non- interacting modes with decay timescales determined by the real parts of the eigenvalues (App. A).
For such teachers, we show in Appendix C that partial observation does not lead to overestimation of timescales under MAP inference. Ordering the eigenvalues of B descending order of their real parts as 1 > ()(1)) 2 R(2)) 2 ... 2 (. the eigenvalues Å of the student's dynamics matrix Aso satisfy R(AA) > R(A:) R(AD) for all 1 < i S d. However, this positive recovery result does not exclude the possibility that the spectrum of the student's dynamics matrix will have qualitatively distinct gap structure, which would lead to incorrect inference of approximate attractor mechanisms.
In the special case of an ideal line attractor, this does not happen: if the teacher is a symmetric approximate line attractor, then the student will be as well. Concretely, suppose that B is symmetrie, with cigenvalues satisfying À
E, < 1, and X; < for > 2, and that the eigenvector in corresponding to the leading eigenvalue (the direction of the approximate line attractor) is randomly oriented or delocalized. Then, the eigenvalues of the student dynamics matrix satisfy Å 2 A1 - D/dd) and d S X2 (App. C.3). This implies that approximate line attractors can be recovered even under heavy partial observation long as the deviation of the teacher dynamics from a perfect line attractor is small. In Figure 2, we illustrate this successful recovery, and show that it is not qualitatively affected even if the observation time is finite. This successful recovery is consistent with what we found in the driven setting in Figure 1.
3.2 Non-normal dynamics: Feedforward amplification
Our results for normal teacher dynamics in $3.1 show that the student can correctly recover line attractor dynamics, matching our motivating observation in Figure 1. However, we recall that we found that non-normal network performing integration through feedforward amplification was incorrectly recognized as also being a line attractor. While it is challenging to analyze general non- normal teacher matrices in the noise-driven setting [43, 48]. we can show that this mismatch again emerges for feedforward chains. In particular, we show in Appendix D that the dynamics of a student of fixed size approach that of line attractor as teacher size increases. Assume that the teacher is perfect feedforward chain with connectivity B,y = 81+,,-- Then, as D - oo for fixed d, the student dynamics matrix A in the limit of long observation time and vanishing regularization approaches 8.+1, + 6,g6.j, hence its leading eigenvalue approaches 1, while the others tend to zero (App. D).
As illustrated in Figure 2, the qualitative conclusion that partial observation leads to timescale overestimation does not change even when the observation time is finite. We remark that the fact that the student becomes closer and closer to a line attractor as D increases is consistent with the intuitive argument given at the end of Section 2: if the number of observed neurons is fixed and small, the only way for the student network to capture the long integratiion window of the feedforward chain is through tuning its eigenvalues create long timescales. In Figures and G.3, substantiate this intuition by showing how the estimated timescales depend on the size of the teacher network relative to the student.

---

a
ii)
Ground Truh
Opun
N
Ground Trth
Neura Activity IAU.
M B
MN
M
MN
iv)
5%
(R(A)
50%
GT
1.0 iv)
5%
SR(A)
50%
GT
1.0
PC3 (A
PC3 CAUL)
0.5
0.5 $
PC2 (AU))
PC2 (AA)
PCE (AU.
PC2 (uu)
PC (AU)
eC2 (AU.
PCI (AU.))
PC1 (A.U.)
PCI (AU.
MCI (A UJJ
PCI J
PCI (AU)
Figure 4: Spurious slow directions in data constraind student models for low-rank teacher dynamics. Learning from rank-2 teacher. i). Schematic of teacher weights. ii). Ground truth teacher (red) and learned student (blue) dynamics matrix eigenvalues at 5% subsampling. Note the presence of single learned outlier eigenvalue with real part near 1. iii). Activity traces for the teacher (red) and student (blue) networks. iv). Example student network dynamics for 5% and 50% subsampling compared the ground truth (GT)). Here, points along the trajectory student dynamics rapidly converge to a line and then decay slowly towards the origin, consistent with the outlier eigenvalue observed in (ii). b. As in a, but for a rank-3 teacher network. Correspondingly, the student learns two outlier eigenvalues, and two slow directions. See Appendix G for detailed experimental methods.
colored by their time. The
3.3 Low-rank non-normal dynamics
As second neuroscience-insse inspired example of non-normal teacher dynamics, we consider low-rank connectivity. In recent years, low-rank RNNs have emerged as popular models for cortical dynamies [15, 16, 25, 49, 50]. Importantly for our purposes, they yield low-dimensional population activity, and hence are again relatively ideal scenario for data-constrained modeling under partial observation 50J. However, we find that connectivity that is both non-normal and low-rank can also give rise to severe timescale overestimation in the student network.
As a particularly simple example of low-rank teacher dynamics, we consider the case in which B = MNN is rank r < D, with M, N E Dx having null overlap M 'N 0rxr and orthogonal columns M M = N - Then, B is a non-normal matrix with all-zero eigenvalues. When y >> 1, the stationary covariance S of the teacher network's activity will have precisely r large eigenvalues of order 7², separated from a bulk of eigenvalues that are of (1) with respect to y (App. E). In this large-7 regime where the teacher's activity is approximately low-dimensional, the student's learned dynamics matrix has eigenvalues approaching 1, with the rest approaching zero (App. E). Therefore, the student learns an r-dimensional hyperplane attractor. Importantly, this can occur when y² is chosen such that B has order-1 elements. We show how this effect depends subsampling fraction G.4.
In simulations, we observe a finite observation time effect whereby only 1 of the learned eigenvalues are near when process noise is small (Fig. G.1). Consequently, fitting a student network to non-normal teacher with null overlap connectivity of rank as described above can result in the spurious discovery approximate (r 11) dimeesiiol hyperplane attractors. We illustrate this explicitly for the cases network leads to the spurious discovery of approximate line attractor and plane attractor dynamies, respectively, despite nearly perfectly recapitulating the observed activity (Fig. 4). Consistent with these observations, we show that latent LDS models fit to the same teacher activity via more sophisticated variational inference methods also learn a few vastly enlarged timescales (Fig. G.5).
and = 3, where observing only 5% of the neurons in the teacher
4 Mismatched attractor structure in data-constrained nonlinear networks
Though the linear networks studied $3 are analytically tractable, they are of course inherently limited in the types of attractor dynamics they can display. We therefore asked which qualitative insights from the linear setting carry over to nonlinear networks where one allows for nonlinear
7

---

firing rate transfer function 6. That is, we again consider a student-teacher setup, but now the teacher and student dynamics are 72 -z + (o(z) (t) and Fx = -x+ -x+ AΦ(x) + (tt), respectively (App. B). Our focus is again on low-rank networks, both for their usage as models for cortical processing and for the fact that their approximately low-dimensional activity makes them a natural candidate for data-driven modeling under partial observation [15, 16, 49].
While it is less straightforward to relate attractor dynamics to the eigenspeetrum of B in the nonlinear setting, one can still use spectral information to gain insight into the dynamics near the trivial fixed point at the origin. Specializing to ((2) = tanh(z), the Jacobian of the teacher dynamics at the origin is simply - ID + B, and that of the student is analogously -Ia + A. Therefore, connectivity matrix cigenvalues of real part greater than 1 would imply that the fixed point at the origin is unstable, and thus the network must support other dynamical behavior (e.g, other stable fixed points, limit cycles, and/or chaos). In the linear case, eigenvalues of real part greater than 1 were never spuriously discovered, as that would yield exponentially divergent activity. However, with saturating nonlinearity, this extreme eigenvalue overestimation is no longer pathological. Indeed, when we infer the weights of student network using MAP estimation, we find that at small subsampling fractions the student can learn eigenvalues of real part greater than one from a teacher with no such eigenvalues if the teacher connectivity is non-normal. Strikingly, this can lead to the discovery of spurious limit cycles (Fig. 5a) and fixed points (Fig. G.6).
At this point, one might ask whether the eigenvalue overestimation phenomenon we have observed is an artefact of the estimation methods (thus far, MAP and LDS variational inference) on which we have focused. The conceptual argument given at the end of $2 suggests that this should hold more broadly for low-dimensional student networks learning from partial observations of high-dimensional teachers, but this is a heuristic argument, not a rigorous test. We therefore applied several other commonly-used inference methods [14, 51, 52] to fit student dynamics in the low-rank nonlinear teacher setting, All of the methods produced mismatched attractor structure, with many showing
propensity to overestimate eigenvalues, yielding spurious limit cycles (Fig. 5). We show that these effects persist for more general teacher weight matrices and for student networks with additional hidden units to account for unobserved neurons in Fig. G.2.
5
Discussion
In this paper, we have shown partial observation can lead data-constrained models to incorrectly identify the mechanistic basis for slow recorded neural dynamics. We found that, while attractor- like networks can be faithfully recovered even when only small fraction of neurons are recorded, data-constrained models can learn spurious attractor structure from non-normal transient dynamics.
As noted in $2, an intuitive explanation of our results for linear networks is that low-dimensional dynamical systems are limited in the longest timescales they could generate through functionally feedforward integration, and thus are inherently biased towards line- attractor-like mechanisms when fit to observations of slow dynamics. Though our focus has been on partial observation as a driver for this dimensional restriction, most approaches to data-constrained modeling with latent dynamics explicitly bias model selection towards smaller latent spaces. In particular, it is standard to select the smallest latent space dimension that captures more than certain threshold fraction of the variance in the data [13, 18]. This will necessarily favor amproximaate-ttrracton like solutions. Indeed, if one applied such a model selection procedure to the integrator models studied in Figure one would select at most a two-dimensional latent space (see Figure G.8), and thus fall victim to the failure mode noted there. This bias in model selection procedures illustrates wider issue: benchmarking and model selection based on explained variance for restricted set of measured dynamics alone are not necessarily sufficient to diagnose mechanistic mismatches [23, 53]. It highlights a tension between the desire to recapitulate mechanism and our intuitive conception of low dimensionality as a signature of model parsimony.
Here, we have focused on the setting in which one measures dynamics over a single trial for inputs that are either fully known white noise. Previous works have shown how failure account for unobserved inputs can lead to incorrect eigenvalue estimation [19]. Moreover, most previously- proposed methods to disentangle input-driven versus autonomous dynamics require repeatable trial structure [10, 19, 54]. Our work focuses only on the effects of unobserved neurons, and does not attempt to address this additional source of mechanistic mismatch. Observation of network under
8

---

b
a
Gouudd Leaned
Leamee
Actalyy
MAP
CORNN S
-1
10%
RRAA)
1.0
5%
1.0
(4.1)
PC3 1A.1
555
0.5
PCZ
PCZ (A.))
PC (a.u.
(A-M
PCZ (AUL)
PC2 (AUU
PCI (A.U.)
PC2 (AU.)
PC) (A.U.)
PCI (A.U))
PC2 (AU.)
PC2 (A.U.)
1
d
ii)
BPTT
FORCE
Ground
Learned
Learred
-1
0
10%
JUA)
50%
1.0
10%
3U(A)
50%
GT
1.0
(A.U))
PC3 (A.U.)
0.5
0.5
(C2.
PC2 (AU.)
PC2 (A.A)
PC2 (AU)
PC2 (AU))
PC2 (&U)
PCI (A.U))
PC1 (AVJ)
PC1 (AU.)
ACI (AU.)
Toe T
(AU) PC1
Figure 5: Eigenvalue overestimation leads to spurious limit cycle discovery across diverse inference methods. a. Learning from a noise-driven teacher with low-rank non- non-normal connectivity using MAP estimation. i). Ground truth teacher (red) and learned student (blue) dynamics matrix eigenvalues at 10% subsampling. Note the conjugate pair of learned eigenvalues with real part greater than 1. ii). Activity traces for the teacher (red) and student (blue) networks at 10% subsampling. iii). Example student network dynamics for 10% and 50% subsampling compared to the ground truth (GT). b. As in a. but for the "Convex Optimization of Recurrent Neural Networks (CORNN)" algorithm proposed by [14]. Since CORNN was proposed for leaky-rate (as opposed leaky-current) dynamics, modify the student and teacher dynamics accordingly. A spurious limit cycle is fit at 5% subsampling. As in a, but using backpropagation through time (BPTT). d. As in but for the recursive-least- squares based FORCE algorithm [51]. See Appendix G for detailed numerical methods.
restricted set of input conditions poses particularly striking challenge if it has a spectrum of heterogeneous integration timescales. inputs drive activity along only a subset dimensions, easy to imagine how heterogeneous spectrum of time constants could be reductively interpreted as low-dimensional attractor structure [38, 39, 55]. When studying circuits whose upstream inputs are not well-understood dynamically, inferences about circuit-intrinsic attractor structure become even more tenuous. As our work shows that data-constran conttranntta models can fail to correctly distinguish between mechanisti- cally different hypotheses for the circuit basis of slow dynamics, an important question how one should validate putative attractor structure. Though we do not address this issue in the present work, obvious candidate for conclusive validation of attractor dynamics of course direct experimental perturbation of neural activity. Daie, Svoboda, and colleagues have identified feedforward amplifi- cation (in lieu of fine-attractor-lik dynamies) in anterior lateral motor cortex using the correlation structure of responses to optogenetic perturbations [36]. O' Shea, Duncker, and colleagues have used targeted optogenetic perturbations to interrogate putative low-dimensional dynamics in primate mo- tor cortex [25]. There, they show that data-constrained models with unconstrained weight matrices do not readily predict perturbation responses, while those with weight matrices constrained to be low- rank capture the fast recovery of the dynamics to dimeeneiioonll subspace. After the completion of our work, Vinograd et al. [56] have begun to interrogate putative line attractor dynamics in hy- pothalamus using similar perturbations. An important question for future work will be determine how specifically targeted a patterned optogenetic perturbation must be in order to distinguish between the line attractor and functionally weweenalllleeeeffwaarrra. integrator networks studied here. Another important

---

question is how to relate patterned stimuli in silico and in vivo, particularly for latent variable models where possibly ill-posed inversion of the latent state to observed neuron mapping would be required.
Finally, we remark that our work relates to a broader question of inference for partially-observed dynamical systems. There is substantial literature on performance guarantees for parameter estimation for linear systems, which largely focuses on predictive accuracy rather than qualitative features [44-47, 57J. There are also host of methods which leverage delay embeddings; in addition to predictive accuracy, these methods prioritize accurately inferring the dimensionality and topological structure of an underlying system [20, 58-60]. However, such methods are "eqquation free", and thus are not ideal for identifying how dynamical variables are coupled [61]. Consequently, such methods might not be suitable for distinguishing between mechanistically distinct circuit hypotheses that generate similarly low dimensional neural activity. Indeed, delay embeddings do not appear to distinguish between the two integrator circuits studied here (Fig. G.7). We leave a more detailed investigation of delay embedding-basse based approaches for future work.
Acknowledgments and Disclosure of Funding
We thank Farhad Pashakhanloo and Mitchell Ostrow for helpful comments on previous version of our manuscript. JAZV and CP were supported by NSF Award DMS-2134157 and NSF CAREER Award IIS-2239780. JAZV was further supported by a Junior Fellowship from the Harvard Society of Fellows. CP was further supported by Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] Shreya Saxena and John P Cunningham. Towards the neural population doc- trine. Current Opinion in Neurobiology, 55:103-111, 2019. ISSN 0959-4388. doc.ittss//ooooog///000066...o.p
URL https: //wp::///wwww//// /www. sciencedirectt. com/ science http://rrriiaae/////544111119100 Machine Learning, Big Data, and Neuro- science.
[2] Paul Masset, Shanshan Qin, and Jacob A Zavatone- Veth. Drifting neuronal representations: Bug or feature? Biological Cybernetics, pages 1-14, 2022. doi:doi.org/ 1007//004222--01 00916-3.
[3] Anne E. Urai, Brent Doiron, Andrew M. Leifer, and Anne K. Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature Neuroscience, 25(1):11-19 01 2022. ISSN 1546-1728 doi: 1..300/ 11t////11199211111 -21-009000- URL https://do..org:: 4038/s41595 021 -20-8000---0000--9
[4] Ashesh K Dhawale, Rajesh Poddar, Steffen BE Wolff, Valentin A Normand, Evi Kopelowitz, and Bence Ölveezky. Automated long-term recording and analysis of neural activity in behaving animals. eLife, 6:e27702, 2017.
[5] Saurabh Vyas, Matthew D. Golub, David Sussillo, and Krishna V. Shenoy. Computation through neural population dynamics. Annual Review of Neuroscience, 43(1):249-275, 2020. doi: 10.1 www.aannree--uoooo92666-----
[6] Markus Meister. Learning, fast and slow. Current Opinion in Neurobiology, 75:102555, 2022. ISSN 0959-4388. http:///s/..bbooo.....0o011o.oo. sciencedirect wtt/ wttp:///cciicccaac.ee.////////aa
URL https:://///// //www.
[7] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. bioRxiv, 2017. 101:10.11011214266 URL https:://ww:t biorxiv. org/ content/eearyy/201/111 12/214262.
[8] Alex H. Williams and Scott W. Linderman. Statistical neuroscience in the single trial limit. Current Opinion in Neurobiology. 70:193-205, 2021. ISSN 0959-4388. /tthttpp//dbb.org//00000......oo
URL https: //www. scwencedi com/
ocinnee/rrriceeppp/////sss222222 Computational Neuroscience.
10

---

[9] Lea Duncker and Maneesh Sahani. Dynamics on the manifold:
Identifying computational recoriiis Current Opinion in Neurobiology, 70:163-
dynamical activity from neural population recordings.
170, 2021. ISSN 0959-4388 wwtpttpps//ooooo///////////////p www. www.scieneedieeee http://ww.aaac.aaaa.aaaaaa//aaaa Computational Neuroscience.
URI. https: //
[10] Aniruddh R. Galgali, Maneesh Sahani, and Valerio Mante. Residual dynamics resolves re- current contributions to neural computation. Nature Neuroscience, 2662):326-338, Feb 2023. ISSN 1546- 1726. 141598-022-011300-1
(0338//415990022-01200000 URL atttps://dol www.o..g//...
[11] Chethan Pandarinath, Daniel J. O'Saa, Jasmine Collins, Rafal Jozefowicz, Sergey D. Stavisky, Jonathan C. Kao, Eric M. Trautmann, Matthew T. Kaufman, Stephen Ryu, Leigh R. Hochberg, Jaimie M. Henderson, Krishna V. Shenoy, L. F. Abbott, and David Sussillo. Inferring single- trial neural population dynamics using sequential auto-encoders. Nature Methods, 15(10): 805-815, October 2018. ISSN 154877105. 10::10.1030//1111111111 10c10...00//41159-00---0001 //doi.org////// 1038/ 841192-118-00009---5
URL https:
[12] Emily I.. Sylwestrak, YoungJu Jo, Sam Vesuna, Xiao Wang. Blake Holcomb, Rehecca H. Tien, Doo Kyung Kim, Lief Fenno, Charu Ramakrishnan, William E. Allen, Ritchie Chen, Krishna V. Shenoy, David Sussillo, and Karl Deisseroth. Cell-tye--peeiffc population dy- namics of diverse reward computations. Cell, (511)))566883877..8 2022. ISSN 0092- 8674. dot.https://oii.rgg/// 101p000000000......2222222..11/0 019, URL https: http::////w..ciieeeiieeettt sciencedirec.... com/science/
[13] Aditya Nair, Tomomi Karigo, Bin Yang, Surya Ganguli, Mark J. Schnitzer, Scott W. Lin- derman, David J. Anderson, and Ann Kennedy. An approximate line attractor in the hy- pothalamus encodes an aggressive state. Cell, 186(1)1788-133e15, 2023, ISSN 0092-8674. dat.ttpss//oo..oo///00000....... scieecee/atticee//////0022222222
URL https: http:://wwwwwww... www.:scceeneedrecc.o com/
[14] Fatih Dine, Adam Shai, Mark Schnitzer, and Hidenori Tanaka. CORNN: convex opti- mization of recurrent neural networks for rapid inference of neural dynamics. In A. Oh, Neumann, A. Globerson, Saenko, M. Hardt, and Levine, editors, Advances Neu- ral Information Processing Systems, volume 36, pages 51273-51301. Curran Associates, Inc., 2023. URL
nwwriss.eenpppppee. cc/ cwtpcper_pppppeeeeeeeeeeeee wtlllffflllllleeeeeee ptteee//paper/22///ieee pdf.
a1
[15] Adrian Valente, Jonathan W Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank RNNs. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 24072-24086. Curran Associates, Inc., 2022. URL https: httr:::://prooeedngg..... neurips cctpccccppa.eeee..cc hec/pape__seeee /a2e///022////lle 90776@llaattt00eeeecccccaeee---. Paper- Confeere-oneeeceeee pdf..
[16] Adrian Valente, Srdjan Ostojic, and Jonathan W. Pillow. Probing the Relationship Between Latent Linear Dynamical Systems and Low-Rank Recurrent Neural Network Models. Neural Computatioon, 34(9):1871-1822 08 2022. ISSN 0899-7667. 166://cc.///cco.... a 01522. URL https://doi..rggee http:/.o.112//ecooorr.o a_01222
[17] Matthew G. Perich, Charlotte Arlt, Sofia Soares, Megan E. Young, Clayton Mosher, Juri Minxha, Eugene Carter, Ueli Rutishauser, Peter H. Rudebeck, Christopher D. Harvey, and Kanaka Rajan. Inferring brain-wide interactions using data-constrained recurrent neural network models. bioRxiv, 2021. 10.110000 12. 12............ 8..42338. URL https: www. biorxiv org/ 12. 423348.
cotteatt/arrry/00///////////
[18) Scott Linderman, Annika Nichols, David Blei, Manuel Zimmer, and Liam Paninski. Hierarchical recurrent state space models reveal discrete and continuous dynamics of neural activity in elegans. bioRxiv, 2019. 10.:100111 URL https:://///. :/ltts::/oi ///ww. biorxiv. org/ cogg/otn/t/////// eatly/20199/0//2//221550

---

[19] Parsa Vahidi, Omid G. Sani, and Maryam M. Shanechi. Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior. Proceedings of the National Academy of Sciences, 121(7))a221888712) 2024. doi:10. (00.1077/paa222228887111 URL https: / //www pnas orggdoi/absse http://abs/11..0000..00 1073/ 107.//aas.pp......a pnas. 2212887121.
[20] Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational system dynamics from neural data with recurrent neural networks. Nature Reviews Neuroscience, 24 (11):693-710, Nov 2023. ISSN 1471-0048, doi:10. 1008//41583-002-00774444 URL https: / //doi.org//i 1038/ 4 00---88--000007--00
[21] Yan Zhang, Márton Rózsa, Yajie Liang, Daniel Bushey, Ziqiang Wei, Jihong Zheng, Daniel Reep. Gerard Joey Broussard, Arthur Tsang. Getahun Tsegaye, Sujatha Narayan, Christopher J. Obara, Jing-Xuan Xuan Lim, Ronak Patel, Rongwei Zhang, Misha B. Ahrens, Glenn C. Turner, Samuel S.-H. Wang, Wyatt L.. Korff, Eric R. Schreiter, Karel Svoboda, Jeremy P. Hasseman, Ilya Kolb, and Loren I.. Looger. Fast and sensitive GCaMP calcium indicators for imaging neural populations. Nature, 61 5(77755)) March 2023. ISSN 1476-4687. doi: 10.8//4//4156 023-05828-9. URL https:://di..oor///
1038/ s41586 000-58666--------0 028-05828-9.
[22] Carsen Stringer and Marius Pachitariu. Computational processing of neural recordings from calcium imaging data. Current Opinion in Neurobiology, 55:22-31, 2019. ISSN 0959-4388.
1.005. URL https: //www//ww/////////////// www.:ssiieeddreccc.. com/
cciacc//ariicee//////99988888819 Machine Learning, Big Data, and Neuro- science.
[23] Abhranil Das and Ila R. Fiete. Systematic errors in connectivity inferred from activity in strongly recurrent networks. Nature Neuroscience, 23(10)) October 2020. ISSN 1546-1726. 100:10.113800 100:1000003344445522200000066660
URL https: http://oooooooo////////.e/ org/ooog//// /030/ 00t0://4415500000000----
[24] Mikail Khona and Ila R. Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(124:74,, December 2022. ISSN 1471-0048, 10.1000//41588000222 00642-0. URL hattps://don http:///oo.or///.//.// 10300.038800 1038/
[25] Daniel J. O'Shea, Lea Duncker, Werapong Goo, Xulu Sun, Saurabh Vyas, Eric M. Trautmann, Ilka Diester, Charu Ramakrishnan, Karl Deisseroth, Maneesh Sahani, and Krishna V. Shenoy. Direct neural perturbations reveal dynamical mechanism for robust computation. bioRxiv, 2022. doi: 10. 1000/2022.12.1.5207766 URL https: //wttps:://ww.... biorxiv. wwwrrxvv..oggceooo org/ htt///oonet/earrly/// 12.16...... 12.166.207688.
[26] David Sussillo and Omri Barak. Opening the Black Box: Low-Dimensional Dynamics in Aigh-Diimesiooal Recurrent Neural Networks. Neural Computation, 25(3):626 649, 03 2013. ISSN 0899-76677. doi: 10. 1we1/1111/COo.. a a_00499. URL https://doi. org/10. 11.10.....1....... a_ a_00409.
[27] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dy- namies. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, wtwppiinw............ppar.rrprre wwcpppppe_filleeeec hit.less/pperr221///iieee. pdf.
Inc., 2019. URL
(28) H. S. Seung. How the brain keeps the eyes still. Proceedings of the National Academy of 1996. doi: 10. (0073/paas.33.23.3332... Sciences, 93(23):13339-11) 20007//ppa///// 93. 23. 13339. pnas. wwtes/org//oo/aaie wttp///i//ab//oooeo/eeeee/aaaaet
URL https: //www.
[29] Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome. Context-dependent computation by recurrent dynamics prefrontal cortex, Nature, 503(7474):78 84, Novem- ber 2013. ISSN 1476-4687. 10. 10.1100//auur112777 12742. URL https://doi. http:///doooog/1..10111 org/10.108// nature12742.
[30] Mengyu Liu, Aditya Nair, Nestor Coria, Scott W. Linderman, and David Anderson. En- coding of female mating dynamics by hypothalamic line attractor. Nature, Aug 2024. ISSN 1476-4687. doi:
URL. https://doi. htt////ii.org/...0003..3
12

---

[31] George Mountoufaris, Aditya Nair, Bin Yang, Dong Wook Kim, Amit Vinograd, Samuel Kim, Scott W. Linderman, and David Anderson. A line attractor encoding a persistent internal state requires neuropeptide signaling. Cell, 2024/09/24 XXXX. ISSN 0092-8674. 10 101.1.0001/.cccl2222..000 0850.. URL. https:///oi...... org 1016/j cel.. c 2024 08. 015.
[32] Mark S. Goldman. Memory without feedback a neural network. Neuron, 61(4):221-634, 2009. ISSN 0896-6273. doi:htips://doi.cm
URL https:
scweececineei com/
[33] Timothy Doyeon Kim, Thomas Zhihao Luo, Tankut Can, Kamesh Krishnamurthy, Jonathan W. Pillow, and Carlos D. Brody. Flow-field inference from neural data using deep recurrent networks. bioRxiv, 2023. 100100000// 11.14.567136 URL http:: /top::://www... www.xii.... 11. 567136.
org/content/earl] wtteeeerryyyrre/eeeeee /tt////22111////1/
[34] Thomas Zhihao Luo, Timothy Doyeon Kim, Diksha Gupta, Adrian G. Bondy, Charles D. Kopec, Verity A. Elliot, Brian DePasquale, and Carlos D. Brody. Transitions in dy- namical regime and neural mode underlie perceptual decision-making bioRxiv. 2023. URI. https://www https://www.biorxiv
101.001110022221...5....000 2023/11/00/2023 15..1.....2.7
biorxiv org/content/earlly
[35] Kayvon Daie, Karel Svoboda, and Shaul Druckmann. Targeted photostimulation uncovers circuit motifs supporting short-term memory. Nature Neuroscience, 24(2):259-25) 265, Feb 2021. ISSN 1726. doi:10..100 11111100000/411300000007777 $41593-020 020-0-00----0---333
www.oi.orgg//... URL https:///ol /doi. org wtg///..10. 10//110.188 1038/
[36] Kayvon Daie, Lorenzo Fontolan, Shaul Druckmann, and Karel Svoboda. Feedfor- ward amplification in recurrent networks underlies paradoxical neural coding. bioRxiv, 2023, 10i: 00t.:...100222000001 08.55........ URL.htpss//wwwx biorxiv wwwg wwwtoot//eattttloeoo 2023 0023/08/07/2023 08. 542026.
[37] Surya Ganguli, Dongsung Huh, and Haim Sompolinsky. Memory traces in dynamical systems. Proceedings of the National Academy of Sciences, 189749):1870-11875, 2008. doi: 10i:10.100 (0..1073/pas.0004455 105. URL httppa///wwer pnas. wttp////aas///...././//o 1073/0.......mm 0804451105.
[38] Emre Aksay, Itsaso Olasagasti, Brett D. Mensh, Robert Baker, Mark S. Goldman, and David W. Tank. Functional dissection of circuitry in neural integrator. Nature Neuroscience, 10(4):494- 504, 04 2007. ISSN 1546- 1726. 1et1100.18//11877 1877. URL https: org/10.10000 nn1877.
[39] Andrew Miri, Kayvon Daie, Aristides B. Arrenberg. Herwig Baier, Emre Aksay, and David W. Tank. Spatial gradients and multidimensional dynamics in a neural integrator circuit. Nature Neuroscience, 14(9):155--115) 09 2011. ISSN 1546-1726. doi: 10. 10..0.0///....... 28888. URL https://doi. 1ttp://///oooooo//...///// 1038/nn nn. 2888.
[40] Stephen C. Cannon, David A. Robinson, and Shihab Shamma. A proposed neural network for the integrator of the oculomotor system. Biological Cybernetics, 49(2):127-133 Dec 1983. ISSN 1432-0770. d. 1000/000000///00000333 URL https: org/10. 1007 /BF0032399.
[41] Natalie A Steinemann, Gabriel M Stine, Eric M Trautmann, Ariel Zylberberg, Daniel M Wolpert, and Michael N Shadlen. Direct observation of the neural computations underlying a single decision. bioRxiv, 2024. doi: 101110002....002249000002 httpa////aarrly///////////////// 2202..00220022 02, 490321.
URL htttps://wwat www.bii.oov........o......i..b.o org/
[42] Alexei A. Koulakov, Sridhar Raghavachari, Adam Kepecs, and John E. Lisman. Model for a robust neural integrator. Nature Neuroscience, 588)7775-782, Aug 2002. ISSN 1546-1726. doi: 10. 1038/nn893. URL httts::///to..ooe http:/ooo..../g.// 1038 1038//0883..../.3...13
[43] Crispin W Gardiner. Handbook of stochastic methods, volume 3. Springer Berlin, 1985.
[44] Max Simchowitz, Horia Mania, Stephen Tu, Michael I. Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning
13

---

Theory, volume 75 of Proceedings of Machine Learning Research, pages 4794473, PMLR, 06- 09 Jul 2018. URL httpp::/pprcceeeeeeeee
mlr wptss//7///imccco/////p
html.
[45] Tuhin Sarkar, Alexander Rakhlin, and Munther A. Dahleh. Finite time Iti system identification. Journal of Machine Learning Research, 22((6): 2021. URL httpttp:/// htbp:://mmrrr htll//mloooooo arg/papers/ v22/19- 725. html.
[46] Anastasios Tsiamis and George J. Pappas. Linear systems can be hard to learn. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 2903-2910, 2021.
2687778778..
[47] Tuhin Sarkar and Alexander Rakhlin. Near optimal finite time identification of arbitrary linear dynamical systems. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5610-5618. PMLR, 09-15 Jun 2019. URL otthttts: www.httsss/prroceeiiiiee p/ppceeedcc....... 19a. html.
mlr.
[48] Claude Godrèche and Jean-Mare Luck. Characterising the nonequilibrium stationary states of Omstein-Uhllnbeck processes. Journal of Physics A: Mathematical and Theoretical, 52(3):
035002, 2018. 100110000///111111222210 1aaaf990
[49] Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojie. The role of population structure in computations through neural dynamics. Nature Neuroscience, 25(6):783-794, June 2022. ISSN 1546-1726. doi: 100.000038//4153-0222000000 / //doi.org/10 1038/ 014155330222 022-01088-4.
URL https:
[50] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computa- tions in low-rank recurrent neural networks. Neuron, 99(3):609-623.3 e23.e29, 2018. ISSN 0896- 6273, /ow.hhtps:://oi..rrg//// 100. 1016/j www.:...eeooo.1..... 0.0.0003. URL https: http: /eattpp::/ww..... http://www..ciieeeirrrr..
http/sciennc//rrticllc http://cacrar//////////6///00310
[51] David Sussillo and L.F. Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544-557, 2009. ISSN 0896-6273.
URLhttps:// hattps::/www.sciieeee www.sccenceiirect.oo com/
[52] Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82-89, 2019.
ISSN 0959-4388. URL hhttps://wwe sciencedirect. com/ Machine Learning, Big Data, and Neuro-
http:t/ss//bb.iao////....11oo.oo wcweecce/atiiioccooocccocooiiaee ccccace//rrrccee//////9999999000 science.
[53] Poornima Ramesh, Basile Confavreux, Pedro J. Gonçalves, Tim P. Vogels, and Jakob H. Macke. Indistinguishable network dynamics can emerge from unalike plasticity rules. bioRxiv, 2023, 100.1001110////11.11111... 2023 2023/11/04/2023// 566168........
URLhttps://www biorxiv. org/ content/earrly
[54] Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and lla Fiete. Beyond geometry: Comparing the temporal structure of computation in neural circuits with dynamical similarity analysis. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 33824-33837. Curran Associates, wwwpeeeddddiinnnnnueeeeeeeeeeeee nwurips. cc/ ccppper pappr_rpprrrrrr files/ wtilse/ppper/002//////// /aper//023/fille .dd.
Inc., 2023. URL httpsss//prrooeeeiinn
[55] Mati Joshua, Javier F. Medina, and Stephen G. Lisberger. Diversity neural responses in the brainstem during smooth pursuit eye movements constrains the circuit mechanisms of neural integration. Journal of Neuroscience, 33(15))663336647, 2013. ISSN 0270-6474.
URL https://www. jnw.u//ooo/......cc....... org/content/
33315/6633.
[56] Amit Vinograd, Aditya Nair, Joseph H. Kim, Scott W. Linderman, and David J. Ander- son. Causal evidence of line attractor encoding affective state. Nature, Aug 2024. ISSN 1476-4687. 341586-024 024-07915-x.
10355..001111111111555555555 1038:44556--0444099154
URL https://doi. ott./10.110000
14

---

[57] Yuyang Zhang, Shahriar Talebi, and Na Li. Learning www-dimessionl latent dynamics from nigh-dimensional observations: asympootssss and lower bounds. arXiv, 2024.
[58] William Gilpin. Deep reconstruction of strange attractors from time series. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 204-216. Curran Associates, Inc.. 2020. URL httptttt//ooocceddiiiiii neurips. cc/pappr_ftee Satpprrfflee/pppee///00/ http:/ppppppppp/eeeeee Paper.pda.
021bc//e2200111183330000666
[59] William Gilpin. Chaos as an interpretable benchmark for forecasting and data-driven modelling. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2021. URL https: /ttp://////eeeeeeee/eeeeee
jtbjYJrr.............
[60] William Gilpin. Generative learning for nonlinear dynamics. Nature Reviews Physics, 6(3): 194-206, March 2024. ISSN 2522-5820. doi:
URL https:
/ //doi. www.oo.gg.......o/o..o/ 1038 (020///2254---2222 -205544--2----0 006-80088-2.
[61] George Stepaniants, Alasdair D. Hastewell, Dominic J. Skinner, Jan F. Totz, and Jörn Dunkel. Discovering dynamics and parameters of nonlinear oscillatory and chaotic systems from partial observations. arXiv, 2023.
[62] Rajendra Bhatia. Perturbation bounds for matrix eigenvalues. SIAM, 2007.
[63] Suk-Geun Hwang. Cauchy's ' interlace theorem for eigenvalues of hermitian matrices. The American mathematical monthly, 111(2)117--15,, 2004.
[64] DLMF. NIST Digital Library of Mathematical Functions. http://dlmff.iss http://dlmf.niss.oovv... gist.go//..ttttnn..o Release 1 1.1.1 of 2021-03-15 2021. URL http:/ mnsst. mist.gov/ /. F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.
[65] Murray Dow. Explicit inverses of Toeplitz and associated matrices. ANZIAM J., 41()) E215, January 2003. URL http:////aaa: austms. org. au/V44/E019
[66]
Lloyd N Trefethen. Pseudospectra of matrices. Numerical Analysis, 91:234 266, 1991.
[67]
Alexander Atanasov, Jacob Zavatone-Vett and Cengiz. Pehlevan. Risk and cross validation in ridge regression with correlated samples. arXiv preprint arXiv:2408.04607 2024.
[68] Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Coumapeaa, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Bretr, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Mand Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585:357-366 362, 2020. doi: 100.1008/44118600200-0000
[69] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero. Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. 10138/44 592-019-0686-2.
[70] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32, Curran Associates, Inc., 2019. URL http:::ttp////tttt /ttp:///ssstttttttsssss//oooooo http://eegg///rr neurips. cc/paper_ files/
Paper pdf.
15

---

A Introduction to integrator models
In this appendix, we provide a brief, pedagogical introduction the integrator models used as motivating examples in $2. We recall from the main text that both models have dynamics
- Jz + bu
for state z TR,, recurrent weights E DxD, and input u(t) E R encoded through vector b E DR They differ only in the choice of weight matrix J. These linear dynamics are of course exactly solvable, yielding
(tt) - elU-IDN//z(() f ds (U-IDilt-a///b((
5
A.1 Line attractor
The construction of the classic line attractor network as popularized by Seung [28] starts by assuming that J is symmetric, such that admits an orthogonal eigendecomposition with real eigenvalues
OAO
for 00 000 To and A diag A1 ....... AD) for X3 E R. We assume that the eigenvalues
ordered as \ 2 A 2 ... 2 AD. For the system to be stable, we must of course have X; S 1 for all Then, letting
z(t) = 0 ztt)
and
b = O
be the projections the state and encoding vector into the eigenvector basis, we have
E₁(1) = (())=--(),, + 8, f f ds -t- (t>))/
where we have introduced the timescales
Then, it is easy to see that if for some j we have Xn = 1, the corresponding timescale Tj will be infinite and the activity ž (t) along that dimension will perfectly integrate u(t). If integrating u(t) in a way that is stable to perturbations of the network is our only goal, then activity along other dimensions should decay in time, meaning that we should have all other eigenvalues be strictly less ÅD. Moreover, should have b, 0 for all & > 1, i... the
than one, i.c., 1 = Å1 X2 input should be aligned to the top eigenvector of For the decay to be fast, we want the gap between Y and X2 be large. The classic line attractor network achieves this very simply, choosing
i=
Jij 11/(D 1)
such that it has eigenvalue with multiplicity corresponding to an eigenvector proportional to 1p. and eigenvalue -D - 1) with multiplicity
[28, 40].
However, in realistic setting, it will not be possible to fine-tune the top eigenvalue exactly to 1, and there will be some decay along the integration dimension. Therefore, one must consider approximate line attractor dynamics, for which A 1 for some eгrоr & > O, while the other eigenvalues are far smaller, i.e., X >> A2 2 2 Ap [13, 27, 28, 40]. This network is exceptionally sensitive to the error E, with \ = - E one has = T/c, and the error between the true integral of uut) and the readout from the approximate attractor network is exponentially large in time. Yet, so long as A > X2, perturbations along the approximate integration dimension will still decay exponentially more slowly than those along other dimensions.
16

---

In Figure/ 1, we generated connectivity J such that the largest eigenvalue is close to 1, and all other eigenvalues are Specifically, we used J QAQ
for
1 10-3
i=j=1
Ay
0.2
i=j i
0
and Q matrix generated with entries Qiy (O, V Note that for realism, we have relaxed the symmetry constraint, and instead use connectivity that can be related a corresponding symmetric approximate line attractor via a similarity transform. We use D = 500 as the size of the network.
A.2 Fuunttooalll-ffeddforarrdd integrator
The exquisite sensitivity of the line attractor network to small perturbations of the synaptic weights has motivated theoretical investigation of panoply of alternative integrator circuits. Restricting our attention simple linear networks, the most prominent proposal is approximate integration through functionally-fcedforaar non-normal integration [32, 37]. This model starts with the following algebr- observation: if J is non-normal (i.e., JJ T # HJJ)), J). though one loses orthogonal diagonalizability, one can still consider the Schur decomposition
J OTO
where O is orthogonal and T is upper triangular. As proposed by Goldman [32], the Schur decom- position is more conceptually useful tool for interpreting non-normal dynamics than the eigende- composition, preserves the orthogonality of the modes. In partieular, while if the dynamics normal 7 is diagonal and each mode only excites itself, if J is non-normal a given mode may interact 'later' modes hidden feedforward structure, revealing a circuit basis for non non-normal amplification.
As the simplest example of this structure, Goldman [32] considered a hidden chain structure
Tij 6z+1,5-
As T is strictly upper triangular, all eigenvalues of / vanish. Considering the mode decomposition
(tt) xxt)
and
b O
as we did in the symmetric case, we have the mode-wide dynamics
r1)+ 1 + 2,( + b)+1m(t).
This gives sequential low-pass filtering of the input, which allows approximate maintenance of memory over (T) time [32, 37]. Importantly, this mechanism is inherently far less sensitive to small variations in the weights than the line attractor.
For the functionally feedforward network in Figure 1, we use
Tij 1 f6,,1 - -1.).
Here, 3 controls the strength of skip connections that further amplify the output mode of activity. We select = 0.5 that, like the line attractor network, the activity produced by the functionally feedforward network is approximately low- www-dimensional. We generate O as an orthonormal matrix uniformly random with respect to the Haar measure, and use D 500 the size of the network. For input weights, we use the sum of the Schur modes b = ED O..., where O.,, denotes the Schur mode. Then, any readout proportional to the mean Schur mode will then solve the integration task up to a constant rescaling. To achieve the correct readout scale for 3 = 0.5, we used readout weights 0.7 O, where = DP O,,; denotes the mean Schur mode.
B MAP inference of connectivity in noise-driven RNNs
In this Appendix, we lay out the procedure sketched in $3 for maximum a posteriori (MAP) inference of connectivity in noise-driven RNNs that underlies our analytical results. We first consider the continuous time setting directly, and then the discretized case.
17

---

B.1
Continuous time
We first consider the continuous- time setting. We recall from the main text that we consider teacher- student setup, where the teacher has D neurons and a recurrent weight matrix B, such that the
dynamics of firing rate vector zt)) R is
7ă = - + BØ(z) + E(t)
where {()) is uncorrelated Gaussian noise with Elg(()] 0 and (IE())())))] 20 SDID, and e is a possibly nonlinear transfer function, which we take to act elementwise. Again, we assume d-dimensional student with recurrent weights A, such that its rate x(t) E R2 evolves
7X - - -Ao(x) + (tt),
where n1()) is d-dimensional white noise with [[n(()] 0 and E[n(()((s)]] 2075(1 s)la.
Assuming d < D, we observe the first neurons of the teacher:
(t) Pz(t) for [0. and P (Ia. Oax(D-a)).
Our goal is to infer the student's weight matrix A given these observations.
To do so,
MAP inference. Our starting point is the likelihood of observing trajectory {x (t) :t E [o, TT} given a particular weight matrix A, which using the path integral representation of Itō process can be written non-rigorously
(- dt 1| (t) + x AØ(x (0))
p((x'he obs (t) : e [0, A) & exp
20
Here, have used that
6(x*0)) ((Pz) = Pı((z)
simplify the notation. To make the problem analytically tractable, we choose isotropic Gaussian prior over the elements of A:
of
Ag Lidd N
Tp
where p > O. We have chosen this parameterization of the prior variance because it makes the log- posterior density particularly simple:
o7 log p( A
ohs E [0, T)})
T
dt
T |
ob (t) Ao(x (t)) 2
p||AI|
We remark that we have proceeded rather cavalierly our treatment of functional density, but this procedure can equally well be viewed as ridge-regularized least-squares estimation. We will also arrive at the same characterization of the log- tog-posterior density as the continuous-tine time limit of the discrete setting in the subsequent subsection.
As the oog-posterior density is quadratic, is easy to read off that the MAP estimate of A is
L'' dl T
obs (t) obs (1))))))) (0) [[ di Tx
Ag
T (1)66
+ pTa
where we add a subscript T to emphasize the observation window. Using the dynamics of x obs (1) Pz((), re-write this in terms of the teacher's dynamics
A = P BCr + f-' dt T (()))((()))
PT [PCrP] + +lld
(B.1)
where
Cr= f
the empirical covariance of the teacher network activity.
18

---

So far, we have let ø be general. However, we now specialize to the linear setting Ø(2) = in which the student and the teacher are Crmstein-Uhlenben (OU) processes. Then, we have the formal solution
z(1) = = S/ ds ds
and, at least in the long-time limit, we can leverage the classical theory of such processes [43, 48]. Provided that all eigenvalues of the dynamics matrix - + B have negative real part, this process will converge to Gaussian stationary state with equal- covariance
E, (z(z(t)z(t)
]1=
which solves the Lyapunov equation
(LD- -D-B)S B)S S(ID B3) 200
or equivalently given by the matrix integral
S 20 dt -0- (ID (lb-B)t e -ID-B)
In the stationary state, the time-lagged correlation given by
C(r) = E,(z(t)z(t + 1
((r) - (LD-
Moreover, one adds infinitesimal linear perturbations to the dynamics as a(t) (~In + B)z 17(t) + h(t).
one has that the linear response to perturbations of the system in the stationary state is given by RTy(T)= oR, (2: + +)] (UD-BB)
(t)
so that
C(r) R(T)S.
Thus, we will have
lim Or = S T+000
and we claim that The vanishing of this term follows from the observation that
/' dt
lim Jn 7 =())((() = T-200
J'" dt {(t))(t)
(00Z0) =0
E
T
while by the Itô isometry
(1. dt T ((t)a(())
( dt T E(t)a(t)
= 1 fooe ( dt " Blec,((0)
Thus, from (B.1), conclude that the MAP estimated student dynamics matrix in the long time limit takes the form
Å = PBSP (PSPT +laa
(B.2)
As an aside, if B is symmetric matrix, the process will be reversible, and the stationary state an equilbrium. In this case, setting of for brevity, the stationary covariance takes the relatively simple form
S s= 1 dt e (Ip (Ip-Bi B) B)
(B.3)
In this case, we can gain some intution for the effect of partial observation directly from considering the stationary covariance. Consider a generic symmetric weight matrix, partitioned according to the
19

---

observed and non-observed neurons:
Bo00
Bon
B
Bon
Bun
We can then write the marginal covariance matrix the observed neurons as
S0
PT Boo (LD Bun) B Bonl-
We can then interpret
Boo + Bon(ID-d - Ban) BB
on
sort of effective weight matrix that accounts for the effect of feedback through the unobserved neurons on the stationary state.
B.2 Discrete time
We now consider the discrete-time setting, in which the teacher and student are both AR(1) processes. Our goal here to show that taking the continuum limit of the resulting estimate of the dynamics matrix recovers the result obtained directly in continuous time. Letting
be the discretization scale, the teacher's dynamics
(1 a) aBó(z V2c..
(B.4)
where &, ~ (O, ID is isotropic Gaussian noise, while those of the student are
X = -1- + aAp(xx-1) + Van.
where ni I is isotropic Gaussian noise. The likelihood of some observed data {xe He is then given by
T
P((x hEM))))
II exp ==1
(1(1 o)); + a xx 113
20
which precisely the time-sliced analogue the functional density considered above. Again assuming an isotropic Gaussian prior the entries of A, we obtain the corresponding loss function
T
L -- T I1(1
-=
Σ
111--axx + aAp(x(-1) - xil 2 + p||A||
t=1
where P corresponds to the strength of the prior/regularization. We then can arrive at the MAP estimate of the dynamics matrix
(pT p a² D
Ar= (1-)-1))) -1-
Again assuming that the observed data {xe HEElZ are produced via partial observations of the teacher activity
X
P P (Iaxd Oax(D-a;),
we can then describe the learned dynamics matrix solely in terms of properties of the teacher RNN;
BCT + Ar = =
( PT (pla a² PCpPT)-
T
where
1 Cr
Σ
It is now easy to see that the continuum limit of this discrete-time estimate converges in distribution the continuous- time result.
20

---

In discrete time. it is easy to see that
T
E
T
Σ
-
0
1=1
and
E
Σ S (((z_1)
1
E
Sco(Ze1)
T
:l (2
2 1 d
Blo( 1,
T2
B 10
Elo(=
T2
t=1 s<t
o T
Bl 11
Sa (CT))
T
as Z independent E. Then, so long as CT remains bounded, this correlator tends probability to zero as T 0o.
We thus arrive at the MAP estimate of the student dynamics matrix in the long time limit:
Å- = a2 (pla + a² PT) the discrete time analog of (B.2). If we specialize to the linear case, letting
J = (1 ID + aB
such that
2, = Jzz-1 + V2
have the formal solution
2: zo 22
JL- E
B.3 A note on time constants
We note an equivalence between the time constants
RA.
used in this work and the discrete time analog used in previous work [13, 30J.
1
|X
where X are the eigenvalues of the discrete-time dynamics matrix (1 aD + aB, which in terms of the eigenvalues A; of B has eigenvalues X 1 ad; Thus, [X]
v(I (1-a+ aRA (a3A,)² Taylor-expanding the logarithm yields
T (At) (1 (1-RA,
RA, ((a)
21

---

or, in the true continuous timeenuuuu----eeeeeeeee limit,
lim ( (1) A 1 R1,
which matches the continuous-time time constants. For a < 1 - RA, we therefore may use the continuous-time result with negligible error.
C Normal teacher
In this Appendix, we derive the two results on normal teachers claimed in $3.1 of the main text: that the student eigenvalues are contained within the support the teacher spectrum, and that an approximate line attractor is recovered by MAP inference even under severe partial observation.
Assuming that BB B B, for large T, we can simplify the teacher covariance follows:
1
CT
11-12-11
T
2005
T t-1
J Zuzz (JT)-1
(JT)
T ME J (JT
as only the diagonal noise terms should contribute. Further, the first sum above should also tend to 0 assuming that only has eigenvalues with modulus < Note that if small, this condition is equivalent to the real parts of eigenvalues of B being < 1. Simplifying the remaining term, we have
200
T t-1
2002
T 1-1
J (JT)-1--
(JJ (JJTY-
T
T
1=
2=
200
(JJT)
where the first equality above follows from normality of B. Summing this Neumann series, this simplifies to
2005
T
t=2
20o
[
((
ID) (oll))CJTTTT Sustyer (ID
- -40- 2)
The only term that remains in the limit T - 00 is
2002
=o -
(ID
T
yielding
Co = JJ
Expanding.
JJ (1 - T + (1-a)a B + B +
= In + 2a<B, - Ip) +o)+ (BB - -2B)),
where we have defined B, B+B T the symmetric part of B. This yields the simplification
2
1 - BBT
C.. of -n- B, + (B,
2
For a & 1, we recover the continuous time result for symmetric stated in (B.3).
Thus, we arrive at the expression
Å PB - P(Ip -B,)-
(pla
where the regularization has been re-scaled appropriately absorb constants.
22

---

Since B is normal, it can be diagonalized over C as B = UAU Observe that B+B URRAA where R denotes the real part. We then have that
B+
dm = PUA(ID R(AA)
UU*P ( - -AA-RAAA-UUPPP -UPT))
D X (uia())
(pta Σ 1- 1
Σ
1- R(A)))
R(a,) (i
(C.1)
where Uid represents the truneated/projected ith eigenvector of For brevity, define K (pla + T- "i
We first analyze the symmetric case B BT, where the argument is slightly simpler. The more general normal case is addressed in C.I. In this case, we have:
D
Axo = Σ
A Xa
(ind(uiaa)) K-
(C.2)
Using the fact that for general matrices P and Q. PQ and QP have the same cigenvalues, we instead analyze the spectrum of
D
A K-1/2
x X
(idd(
K-1/2
Using that A 2 X for all we have
\ Σ Σ1- (ia(uidd) K-1/2
A 3 1-12
1/2
where PIC Q denotes that P Q is negative semidefinite.
we further suppose that مA1 2 O, we have the relation
D Σ i=1
1-222
A u (u(
K K-1/2 3 1/2
pdila+
X
u (uf K-1/2
1 A;
K (X 1/2
Aa
Thus, if مYd 2 0, all eigenvalues of A satisfy Ò; < Å₁. Similarly, if ΩÂp S 0, all eigenvalues of Åx satisfy \; 2 \n. Both upper and lower bounds are necessarily satisfied simultaneously in the ridgeless limit -
C.1 More general normal matrix case
Suppose B has y pairs of complex eigenvalues Any Koy with corresponding eigenvectors u u as well as D - 2p real eigenvalues År, with corresponding eigenvectors u"s We can then rewrite (C.1) as
R((,,))
p
3(ee)) 1 - R()e,s
D-2p
dry
A_ =
2
Fe +2 R((),))
Ge +
Σ #=1
(uf - dr,
K-1
1=1
1=1
where
Fs, = (R s Ru + (1 ((u
and
Go (
3( Mu
-Similarly, we can also express K in terms of real components:
D-2p
K= p + 2
R(
F
+
Ar²
Lnd
(u
1=1
23

---

We then study the spectrum of
R(Ag;) R()ey)
3(e;;) R(Acg)
D-2p
Ary Arg
A'o K-1/22
2
Pos + 2 Σ
Gej
(u ( K K-1/2
Ao no longer symmetric because of the skew-symmetric components Gey. However, we can still analyze the symmetric component
R(A)))
D-2p
= K-1/2
Pez +
Σ
(u
T
K K-1/2
R((2e)
Ars
j=1
As before, under the condition (oR(A)) 2 O. we have the ordering
p
R(A1)
D-2p
9((1))
(Alco) KK (AR(AI)da +
Σ
Pe +
u
(u
K K-1/2
R(Argg)
Xr,
j=1
j=1
= -K-
Ra(A))aa
We can then observe that v T TA'' - Ra) I 0 for arbitrary v E R since the symmetric component of A'e La is NSD. This implies that R(A)) < R(A1) for all j.
Similarly, under the condition PyD S 0, R(²,) 2 (AD)) for all j. Both upper and lower bounds again hold sinultaneously for - 0, regardless of the teacher spectra.
C.2 Stronger result for symmetric teachers
In the symmetric case B B² with p O, can also show stronger result that 7Ú Ag for all j E {1,...d) d):
Observe that for any j > 1, that
D
A X
T K-1/2
A/c K-1/2
Σ di di K-1/2+ + K-1/2
Σ
u
u
Let x-) denote the jth largest eigenvalue of Applying Weyl's inequality, we have
X
K-1/2
K K-1/2
A1
K-1/2
X
u
ind
T
K K-1/2
A
ド=j
D
A
A1
K/1/2
K
Σ
ui.d
ui:d
K-
(C.3)
A
which follows since the first term the RHS of rank < j 1. We also have that
D
D Σ
u uiaauie :) x-1/2
1/1 1/2
Σ
A
u
uidd T K K-1/2 1/2
A X
X,
=
Thus, we can bound (C.3) above by A;. yielding the result
Å A
(C.4)

---

C.3 Line attractor recovery
Suppose the teacher near perfect symmetric line attractor. In particular, let B B have eigenvalues Å1 = 1 - E, & 1, and X < for i 2 2. For simplicity, assume - 0. In this case, we can express (C.2)
Ans - (uia)) T
X u (iaauuia)
(
1 Σ
Σ
T
Uisd (uizd)
(C.5)
Denote Pi = Ei= x-x, u (ui and =
Em r-x, u (uf From Weyl's perturbation
bounds on symmetric matrices [62], we can bound the eigenvalues of the "numerator" as follows:
1
(T + Ps) - ("- u (la(
A1
u (
||Pallop
A2
(C.6)
X2
where (C.6) follows from the Cauchy interlacing theorem [63]. This yields the bound on the top eigenvalue of the numerator,
A - (u + Pi
1-
A2
u
1
A2
We can obtain a similar bound on the largest eigenvalue of the "denominator":
X1 (u + P3 S |
(
1
A2
the case where X; 2 0 (e.g., no timescale faster than the intrinsic timescale a single neuron), use bounds on the eigenvalues of products of PSD matrices to obtain the following:
d = (Aoo) 2 Xn (Num) Ja
X (Num)
(A1 (Den)) -1
|
1- Ao )
Aa
1
8
A
1 + Aa
(1
2
1 au 1
- A2
E(1 + A2)
A1
| [1 X2)
where we have used 'Num' and 'Den' as shorthand for the factors in (C.5), Assuming cigendirections are randomly oriented, ||u )
From result (C.4), we have an upper bound on the second largest eigenvalue
d A2
Thus, under the stated assumptions, we can conclude Å 2 X1 O and Åo S A2.
D Feedforward chain
In this Appendix, we derive the approximation for the learned dynamics matrix resulting from partial observations of feedforward chain that state in $3.2.
25

---

Suppose the teacher matrix has structure B QMQ' for My 8,41, and = For convenience, we focus on the continuous- time limit. In this limit, the stationary covariance
1 /
E
lim T-00 T a(t)z(t)T dt
satisfies the relation
/
e- (lo-"")))) 20 /
Ex -20
c. nh dt
By the nilpotency B, we have that
D-1
(Bt)a
D Σ
(Ml)
e
B1
Q
QT
n!
n!
==
n=
Σ (M
i isi
[(-)]]
ne!
0
i>j
ij
We then have that
(2k-i-s
Me
Σ
ij
(k t)((k - 5}
k=max((s))
Defining M
Mo lis = 10 -2t
McM 1] dt
Σ
(2k 1l--i- -i-1
-- )
22k- ------ j+1
k=max((-)
we can express the stationary covariance as
I 20²
The learned dynamics matrix is then given by
À = PQME PQQSMQTP
P Ia 20
For simplicity, we consider the Q = ID case, with p + 0. A will satisfy:
Å (PEMp)) PMM PT
(D.I)
Observe that [PE ME for S i,j S d, and that
is D - i= D
[MM 1ij
And thus, for D, (PM [E that A. = 5,41, + ōzgā for some constants âj. This yields the following:
leti for 1 S i.j & d. We can then make the ansatz
i d 1
( (PSM )
àx M
d
The first d rows of D.I are equal under this ansatz. The elements gg are then chosen such that the t row of D.1 matches, yielding that they must satisfy the following linear relation:
26

---

Σ àg Σ "11ks lks 1IE isjsd
k=
à i[S"I1.d,1d lidd1:dd (E*))4114)) J441,1dd
Also note that Å has the form companion matrix, and thus has eigenvalues given by the roots of the polynomial S()) = Aa Ei= 2 Since à o O, we can say that Å will have nonzero eigenvalues.
D.1 Structure of the subsampled stationary covariance
When d < D and very large, [PE _ with constant differences between diagonals. Specifically, claim that for 1 S i.j < D,
is well approximated as having a Toeplitz structure
i - jl
2
VD
To show this, we must obtain asymptotics for
2k 12--i-1 1)
Σ
i2t-
k-i
k=max((.)
when <új S as D + co for fixed d. is easy to confirm that this sum is symmetric in i and j. as
2k - (k ( + (k
k-i (
2k - - )
k: k-
k k-j
Consider the lower triangular elements, letting j i - q for q {0, 1, ..... - 1}. After shifting 4- k i, we have
D-i bii-q
2k + q
M
22k+
k=
then easy see that the diagonal elements (q are weighted sums of central binomial coefficients:
1
2k
M 11,8
2
2
k
Then, using the bounds [64]
4
2k
1²
<
V
k
v n
we have that
1
D-i
1
D D-i
<
M
<
4VT
Vk
2
2/TT
V
k=
Using asymptotics for generalized harmonic numbers [64], we have
D
Σ
2VD
=
V
VD
1=1
For any fixed i, this immediately yields
MM
+
O
VD
27

---

Now, consider the off- off-diagonal elements, for q E f1,2..... , 1}. We remind ourselves that the sum of interest
2
2k + ạ
Σ h=
22k+
k
Using the recurrence
2k + 9)
2k + q
2h + g
k + q
k
we have
2k +
2k + q - 1)
k
k
so
D-i
2k 9
D D-i
2k + q --11
22k+g+1
2 (g-1))+1
k
k=
k=0
which shows that the matrix elements are non-increasing as one moves away from the diagonal:
M - (13-1-1))
Moreover, we have from the same recurrence the weak lower bound
2k + 1)
2)
2k
1)
k
whence
W EMKiing 2
M 1 11-1-1-1)--
These bounds show that all elements of the truncated covariance matrix must be of the same order. To show that the subleading term is of the desired form, consider the difference between successive diagonals, which using the above identities may be expressed as
D-
Σ M 11--(0-1)
|DM 1i.i
Σ
2k +
2 g+
2k q
1:
h=0
Using the abovementioned bounds on central binomial coefficients, we have the bound
q
2k + 9)
2k +
< 22k+ 2k + 9 k + 4/2)
22k+ 2k +
k
9
2m (2k + +/2
which shows that the series is convergent - with an OLIvDD) remainder. In particular, letting n D i + 1, as this bound monotone decreasing in k, we have
1
2k +
4
2 g++
2k + a
k
2x
(2k + 3/22
km
==
1
[ dk
q 2 (2k: + 3/22
v 2m (2n 3/22
1
/2T (2n 3/
/ (2n + 2/22
28

---

What remains is to compute the infinite sum, which evaluates to
q
2k
9
2 2k +
for q 2 1. Therefore, we have
1
[ (9- P.
vD)
hence in combination with previous result for the diagonal terms conclude that
D +0
2
VD
or, restoring the indices, we obtain the claimed result that
D i-
M J0
This shows that the subsampled stationary covariance matrix is approximately Toeplitz.
D.2 Structure of the student dynamics matrix under heavy subsampling
Now, we consider the structure of the student's dynamics matrix the < regime. The inverse of the form of Toeplitz matrix by which the stationary covariance is approximated is known to take the form [65]:
-1
Olc++Oaa
O(e)-O(a)
2 -1
0
-1
-1
0
1TM1pTl- R
-1
0
-1
O(c)+O(d))
O(c)+0(d)
where DDT.. n.. We also have that PME PT PPMPPT + 1 where =
1
j). Thus,
-= PME TT (PSM 1T ~ la RIPS pTJ-
Taking the large approximation, we find that the learned student dynamics approaches the form
As 81+1,5 5.adij-
In other words, approaches a feedforward chain of size except with the activity of the start of the chain never decaying. The largest learned eigenvalue in this limit is 1, while the others vanish identically.
We note that in practice, the sensitivity of the eigenvalues of feedforward chain connectivity matrices small perturbations would cause multiple the learned eigenvalues to be significantly larger than 0. In particular, the e-pseudospectrum of feedforward chain of length d has radius the order el/d [66].
E Low rank
this Appendix, we derive the results on MAP inference low-rank null teachers stated in $3.3.
Consider a low-rank teacher of the form B = M M e Pxr, N E Dxr, If N A = 0rxr and T N M T M I2,,, then B has all eigenvalues, but nonnormal. Here 22 scale parameter, which in some sense controls the degree of non-normality (scales the norm of the
29

---

commutator [B, ] ] BB" B
B).
B). We compute the stationary covariance of the teacher process,
suppressing factors of of by setting og 1:
0 = 2
e e B) -Un-- B
de
/
e
dt
0
j0
2
jo
e 21 exp
mitiy
exp
momi dl
Observe that m commutes with mjnz due to the N M constraint, Thus, can write
=x=2
=2
2t
exp (m;n
II exp (mam 1) dt
k=1
0
2
21
(ID + mmm () II (ID nam t)
t=1
k=1
x
2
[ID + (L) + t)dt
e "ID + 2B,1 + BB (d
where B, B-B" Performing this integral yields the solution
y = ID + Bg + BBT.
2
E.1 Spectrum of the stationary covariance
Our first goal is to determine the eigenvalues and eigenvectors of E To do so, suppose that R a unit-norm eigenvector of Σ with eigenvalue A. Then, it must satisfy
Nou u 1 MN u+ NM 1 2 M Au.
As M and N span orthogonal r-dimensional subspaces of R one possibility is that u lies in the (D 212-22r)dinnn diimensional complement of those subspaces, in which case it must have eigenvalue Thus, D has eigenvalue with multiplicity D - 2r. Now consider the case in which u lies in the union of the subspaces spanned by M and N. Make decomposition
Ma Nb,
where a, b E R", The unit-norm condition is
1 | 2 = (|a||2 + ||b|13),
while the eigenvector condition becomes
1
n u Ma + Nb u + M b + マ N + 5 Ma
2
A[Ma+ + Nb+
Acting with T have
1 a +
? bb + -7 a = Aa
2
2
while acting with T we have
b+ - 2 = Ab.
2
30

---

Together these conditions imply that b = ta, which gives a coupled set of equations for and A:
1
+ 2 +
+ 1 At. 2
This linear system has solutions
\ = 77 - 7² 12V44 44+ 7
t -2 + 71
which each must correspond orthogonal r-dimensional eigenspaces. Therefore, we at last conclude that the eigenvalues of X are I with multiplicity D 2r and ) each with multiplicity r. When y > this gives an r-dimensional 'signal' eigenspace with eigenvalue
+ + + 7
7 + + +0
x
(D -ir-dimmnsional 'null' eigenspace with eigenvalue 1, and an r-dimensional 'suppressed" eigenspace with eigenvalue
A_ + - 22
1
+
As result, increasing 7 will push the effective dimensionality of activity in the stationary state closer r.
E.2
Spectrum of the learned dynamics matrix for large y
We now turn to our main goal, which is to approximately determine the eigenvalues of the learned dynamics matrix after subsampling. Using our result for the stationary covariance, find that the learned dynamics matrix in the infinite time limit is given by
= PB(LD B. BB++ BB.++BBB)P (P(ID B. -
1,+,BB )
(M + 22 MMT) lat
MNT + NNT
MMM
2
2
where M = PM denotes M truncated to the first d rows. Since M is of rank r, will have at non-zero eigenvalues.
The relevant regime is when of the normalization condition N N M M = 2 in any fixed dimension must have Ne. Og7). My = O(y). We can then consider making & parametrically large, which case we have
> 1. such that the activity is approximately dow-dimensional. Because
À 11 MArT + O
where HT is the orthogonal projector onto the r-dimensional span of M MM Here, have used the fact that " ~ O(r*) and MT O(?2), so the former terms will dominate at large y.. Therefore, it follows that as becomes large the non-zero eigenvalues of tend to one. This argument relies on fixing all dimensions.
case of interest is when 22 ~ O(D/VF) for D > r; with this scaling, the elements of are O(1) with respect to D and r.
31

---

E.3 Low rank teachers with nontrivial eigenspectra
While the null overlap case is analytically tractable, we would like to also consider non-normal low rank teachers with nontrivial eigenspectra. To do this, we can specify an overlap matrix Q with the desired spectra, such that ' M Q. For a fixed M (e.g. selected with random entries). we can satisfy the desired overlap via selecting N as N
QMT where MT M+ + V( MM+) generalized pseudoinverse. Here, V E T is arbitrary, and M+ the Moore-Penrose pseudoinverse. Observe that in the case where M has orthogonal columns and V = 0,xD, M M+ .M yielding that B MN is normal so long as Q is normal.
Thus, there are then two ways to make B = MNT = MQMI non-normal; one is to choose choose V O,xD and make Q non-normal. Another way is select nonzero V, e.g. with entries drawn from (00 ).) Then, one can construct highly non- non-normal B even with normal overlap matrix. We use the latter method to generate non-normal low rank teacher connectivity that supports oscillatory dynamics. Specifically, for Fig. 5, we construct teacher connectivity by generating M
/0.5
-0.5
with random entries ~ NN 1). Q
and σv = VD
12 for D = 500.
0.5
0.5
F Regularization
In the analysis thus far, we have focused on learning the dynamics matrix in the ridgeless limit. However, using a high enough regularization may help prevent the discovery of spurious slow modes of dynamics from non-normal transients. Indeed, this necessarily true by observing that in the opposing limit f oc, the learned dynamics matrix collapses to the zeros matrix. One can attempt to estimate an optimal regularization parameter that yields predictive fit via cross-validation, but is unclear whether this would necessarily resolve the eigenvalue overstimation issues described here. Obtaining an accurate estimate of the optimal ridge from cross-validation on windows correlated timeseries data is statistically challenging [67]. and ill-defined if the timeseries is non-stationary. Furthermore, such problems arising from an inherent expressivity issue of the student network- such cannot
when fitting -iw-demensional LDS model to a functionally feedforward simply be resolved through regularization. We leave a more detailed investigation of regularization- based strategies for ablating spuriously discovered attractor-like dynamics to future work.
G Numerical methods and supplemental figures
All of our numerical simulations are implemented Python 3.9. 18 using NumPy 1.26.2 [68], SciPy [69], and PyTorch [70]. They were not computationalll--ntenn 12 hours in total to run on a consumer Dell XPS laptop equipped with an Intel CMre 17-13700H processor. Code to reproduce all experiments is available at https:/ //gtth////tt/t///// cow com/wqiann) DataConstrainedRNNe.
and required less than
For simplicity, we use = 1 in all numerical simulations. Unless stated otherwise, we use teacher network size of 500 in all numerical experiments.
We integrate the student and teacher RNN dynamics via Euler integration with a timestep Al = 0.01. Under the discretization scheme of B.4, in all experiments, select the noise parameters of the 0.02
student and teacher dynamics as 0, = og /2
In the examples of Fig. 1, we generate ground truth network activity by iterating the dynamics for a duration T 5000 x A..
In the purely noise-driven experiments with finite observation time windows, we fit student networks to ground truth teacher activity generated over duration T = 30000 x At.
For MAP inference, we use a regularization parameter p = 0.001 in all experiments. In experiments involving the long time limit T + co, we use SciPy [69]'s built-in Lyapunov solver to compute the stationary covariance of the teacher activity.
For all LDS models, we run the fitting procedure for 200 iterations using the implementation provided by the authors of [18] under MIT License GitHub.? For the experiments in Fig. 1, the input
Shttps://gitae htt.ccubb..oo//iieeeemaaaaaa///a
32

---

signal was explicitly passed to the fitting procedure. For the purely noise-driven teacher setting of Fig. G.5, no imput signal was provided.
Fig. 5: We note that CORNN is applicable only for fitting RNNs that obey slightly different dynamics (leaky-rate instead of leaky-current), of the form
(Ax) + (()
We thus modify student and teacher accordingly in experiments involving CORNN [14]. We run the CORNN fitting procedure for 1000 iterations using the implementation provided by the authors as free software on GitHub.
For backpropagation cockpropagatiio-ttroug----tt through-time, we train student network dynamics matrices observed teacher activity via truncated BPTT, splitting the observed 30000 timepoints into segments of 500 contiguous timepoints. Student networks were then trained for 100 epochs learning rate 0.01, batch size of 16, and a teacher forcing ratio of 0.5.
For experiments with FORCE, we train student networks for 1000 iterations with parameters == 1.5, Po = 0.01. We adapt the implementation provided by the authors [17] under GNU GPL license Github. To increase agreement of the fitted activity, we restricted the observation window of the fitted network to the first 5000 timepoints; fitting to the full observation window yielded fits that failed to capture prominent initial transients of the observed activity.
For experiments involving low-rank teacher networks with null overlap, we use 72 0.2 D for the VF linear networks (Fig. 4) and 7² 0.4 D. for the nonlinear networks (Fig. G.6). This scaling ensures
Vr
elements of B are order-1 (see E.2). For low-rank teacher networks with nontrivial eigenspectra (Fig. 5), see E.3.
hatepe///gthhhb.co////cctttttttd htttp:///gthhb..cca///aaaaaaaaa
33

---

Rank 2
Rank
1.0
0.5
0.0
(R(A)
Rank 4
Rank
Finite time
1.0
8-88x
T + 8
0.5
ERRRRR
0.0
5
10
5
10
Dimension
Finite time effects for learning from low-rank teachers with null overlap connectivity. The top ten largest learned eigenvalues are shown for students learning from teachers of rank 2 S S and null overlap connectivity (all 0 eigenspectrum). All plots correspond connectivity parameter y = D/v /r with teacher size D = 500 and 5% partial observation. Each point indicates an average over randomly selected teacher networks. Error bars are standard error of the mean, but are mostly too small to see.
Figure G.I:
MAP
BPTT
BPTT + hidden
1
Ground Truth
Ground Truth Learned
Ground Truth Learned
Learned
3(A)
301
0
1
11
-1
-1
0
1
-1
0
1
R(A)
R(A)
R(A)
GT
1.0
Neural Activity U..
Neural Activity (A.U.)
PC3 (A.U.)
Neural Activity
0.5
PC2 (A.U.)
160
160
160
PC1 (A.U.)
10%
10%
10%
PC3 (A.U.)
PC3 (A.U.)
PC3 (A.U.)
PC2 (A.U.)
PC2 (A.U.)
PC2 (A.U.)
PC1 (A.U.)
PC1 (A.U.)
PC1 (A.U))
Figure G.2: Eigenvalue estimation under partial observation extends more generally to unstructured non-normal teacher connectivity, and occurs even when hidden neurons are added during training. Each column corresponds to an inference method (MAP, BPTT, BPTT with 200 additional hidden neurons) at 10% partial observability. Top row: ground truth teacher (red) and learned student (blue) dynamics matrix eigenvalues. Middle row: Activity traces for the teacher (red) and student (blue) networks. Bottom row: Example student network dynamics, as shown by randomly sampled initial conditions. Right: Ground truth dynamics.
34

---

Line attractor scor
Overiap
Singular
3-
qurous
13-4
16
10-2
13-4
10
20-4
Subsampling
Subsampling fraction
Subsampling fraction
Line attractor score
spuniss
10-4
10
10tl
30-1
109
10-2
16-3
100
Subsampling fraction
Subsampling fraction
Subsampiing fraction
Line attractor score
10-
spurios
12th
10
10 20t1
20"
10th
10ah
10"
Subsampling fraction
Subsampling fraction
Subsampling fraction
Figure G.3: Properties of learned student dynamics matrices for functionally feedforward teachers in the long time limit T + co. Each row corresponds to functional skip connection strength {0.25, 0.5, 13. Left: Line attractor score versus subsampling fraction (d/D). The green line indicates line attractor score of 1. Middle: Top two singular values of the learned (student) and true (teacher) dynamics matrices as function of subsampling fraction. Right: Normalized overlap (absolute cosine similarity) of the learned left and right singular vectors corresponding to the largest learned singular value (ū,, 0, respectively) with the truncated top left and right singular vectors of the true network ([u]] [ui]t.a. respectively). The red curve shows how the expected overlap would approximately scale for arguments with randomly selected entries. All plots show averages over 20 randomly selected teacher networks. The shaded regions indicate +1 standard error of the mean, and in some cases too small to see.
35

---

Line attractor score
10-
60
0.8
20-
00w dd
04
sponiss
10-1
10-1
10-1
Subsampling fraction
Subsampling fraction
Subsampling fraction
Line attractor score
100
8
1.-
0.0
000 di
0.6
Odkile 90
04-
spuriss
20
02-
0.0
Subsampling fraction
Subsampling fraction
Subsampling fraction
Line aftractor score
200
1.0
oleehu dut
Singular
0.6
0.2
0.0
Subsampling
Subsampling
Subsampling
Figure G.4: Properties of learned student dynamics matrices for rank-1 I teachers with null overlap connectivity in the long time limit T > cc. Each row corresponds
different value of y (72 10.1, 0.2, D). Left: Line attractor score versus subsampling fraction (d/ D). The green line indicates line attractor score of Middle: Top two singular values of the learned (student) and true (teacher) dynamics matrices as function of subsampling fraction. Right: Normalized overlap (absolute cosine similarity) of the learned left and right singular vectors corresponding to the largest learned singular value (ū), 01, respectively) with the truncated top left and right singular vectors of the true network ([u1] [ei]1.d, respectively). The red curve shows how the expected overlap would approximately scale for arguments with randomly selected entries. All plots show averages over 20 randomly selected teacher networks. The shaded regions indicate +1 standard error the mean, and in some cases too small to
a
o
30
Une attractar scone 8.00
scors:
O.se
Ground Tuth
Learned
dine condaar
20
20
20
25
15
M
M.
10
10
M
B
MN
Dimersion
Dimension
Dimension
Figure G.5: LDS fits the low-rank linear teacher networks considered Fig. (a. rank and b. rank 3, respectively). Left: A schematic of teacher connectivity. Middle: The top five time constants of the ground truth teacher connectivity, Right: The time constants learned by an LDS model fit to the teacher activity. All fits were performed at 5% partial observability.
36

---

SOUUII Scttwry (A 1A.0
i)
Leamed
344
Leaned
Neural Acttvit
0 R(AA)
mi)
5%
50%
1.0
5%
50%
1.0
PC3 (A.1)
PCI (A.))
0.5
0.5
PC2 (AU))
PC2 (AU)
PC (AU)
PC2 (AU)
PC2 (C2
PC2 (AU)
PCI (AUU
PC1 (AU.)
PC1 (A.U))
PCI (AU
RC1 (AUU
PC1 (A.U)
Figure G.6: Spurious or missed stable fixed points learned from rank-3 teachers with null overlap connectivity in the nonlinear setting. i). Ground truth teacher (red) and learned student (blue) dynamics matrix eigenvalues at 5% subsampling. ii). Activity traces for the teacher (red) and student (blue) networks 5% subsampling. iii). Example student network dynamics for 5% and 50% subsampling compared to the ground truth (GT).b. As a, but for another example teacher network with rank 3 null overlap connectivity.
a
0.s
PC2 (A..))
05
PC2
50
0.0
150
0.9
PC1 (A.U.)
PC1 (A.U.)
Figure G.7: Delay embeddings for integrator circuits performing the integration task of Fig. 1. a. Top two PCs of delay-embeddee observed activity from the line attractor performing the integration task. Each row corresponds to different number of delays (5, 10, 20), and each column corresponds to delay interval in (5, 10, 20) Av. b. As in a, but for the functionally feedforward chain.
b
Line attractor score
4-
4
Line attractor
3-
3
2
10
10
Subsampling fraction
Latent dim.
Figure G.8: Sweep of LDS models fit to a functionally feedforward chain performing the integration task of Fig. 1. Line attractor score versus subsampling fraction for LDS models fit with maximal latent dimension d 1. b. Line attractor score versus LDS latent dimension at full observability (d D). In both plots, each point represents average over ten fits.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims make in our paper, and its limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We emphasize the limitations of the settings of our experiments and theoretical results throughout, well as the fact that many of our theoretical results are asymptotic approximate. We test finite-size deviations from those approximations in our experiments.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
38

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide complete derivations of our results in Appendices B, C, D, and E. We note that many of our analytical results are approximate, as we emphasize throughout.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe numerical results in detail in Appendix G. Moreover, include code to reproduce all results with the initial submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility.
39

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For initial submission, we include code to reproduce all experiments as a ZIP file. For publication, we will make code available on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
cesiionoolcci for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp..ccc ppppulic/guuddeiee
/CodeSubmieeeeeee C simmmissii ierionPolll onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details in Appendix and in the code included with our submis- sion.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For plots showing average metrics over multiple instantiations, we include either shaded regions or error bars denoting one standard error the mean. In some cases, the error bars significance, so we do not include emro bars. Our empirical results are proofs of concept rather than claims regarding frequency.
small see. For other results, we do not make claims of statistical
Guidelines:
40

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //terr//neuussseee http:///rrrr..c.pppplie/ee/////e CaabbiicEEthiie csCuidelineen ines?
Answer: [Yes]
Justification: Yes, affirm that our research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not anticipate our work will have societal impact.
41

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (c.g., dep.oymmnt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper involves neither data nor models that have risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide citations and license information for all code used in Appendix G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
42

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
43

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "klS1OX8tCp", "reviewer": "Reviewer_njnf", "summary": "Deriving a mechanistic understanding of neural circuits from observations (neural recordings) is a fundamentally ill-posed problem. This paper explores and exposes this issue in controlled theoretical settings. Specifically, the authors focus on two aspects: the intrinsic biases of data-constrained surrogate models and the partial observability of the data. They demonstrate these pitfalls in both linear and non-linear systems.", "questions": "Please refer to the weaknesses section above.", "limitations": "Please refer to the weaknesses section above.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a beautifully written paper, clear and concise. The examples are well motivated. The theoretical and experimental analyses are mathematically tight. Moreover, the issue of fitting models to partially-observed data and deriving mechanistic insights from such surrogate models is a pertinent one for neuroscience today, given the developments in recording techniques and data-driven modeling.", "weaknesses": "While the problem formulation is intuitive and well-motivated, the settings considered here are exceptionally constrained. This paper would greatly benefit from and be practically more useful if the authors provided scenarios where observed data has more than one plausible mechanistic explanation and how one would go about falsifying each hypothesis. The authors allude to perturbation analyses as a solution, but once again, the interpretations are unclear when we lack a \"ground truth.\" \n\nAre there existing biological neural datasets that the authors can use to expose possible misinterpretations of circuit mechanisms?\n\nThis question of system identification has been of long-standing interest in neuroscience. Coupled with surrogate modeling, there is a big literature on simulation-based inference and adjacent topics that would be worth adding as discussion in this current manuscript. Some representative references:\n\nFlexible and efficient simulation-based inference for models of decision-making. Boelts et al. (2022)\n\nFast Inference of Spinal Neuromodulation for Motor Control using Amortized Neural Networks. Govindarajan et al. (2022)\n\nThe frontier of simulation-based inference. Cranmer et al. (2020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of our manuscript, and are grateful for their positive assessment. Below we respond to their comments on the Weakenesses of our work: \n\n* We are glad the reviewer found our formulation of the problem intuitive, and agree that the settings we consider are constrained. However, we would argue that they are not exceptionally so. In particular, we emphasize that the integrator networks in Figure 1 do in fact have multiple plausible mechanistic explanations; by examination of the activity and network outputs alone we could not discern that the example in 2a is a line attractor, while that in 2b is a feedforward chain. Given the mismatch in the fitted and ground-truth flow fields, one could experimentally distinguish these hypothesis based on perturbing the activity of the integrator neurons in this circuit. This could be accomplished, for instance, through optogenetic stimulation (see e.g. O'Shea, Duncker, et al. 2022, or very recent work by Vinograd et al. 2024 posted after the NeurIPS submission deadline). Carefully investigating when perturbations can sensitively distinguish between mechanistic hypotheses is an important topic, but lies beyond the scope of the present work. \n\n* We agree with the reviewer that it would be interesting to expose possible misinterpretations of circuit mechanisms in existing biological datasets. However, for the purpose of the present paper we prefer to maintain our focus on synthetic data with known mechanisms, so that we can map out when mismatch in model mechanism arises. This is also motivated by the fact that conclusively demonstrating misinterpretation of mechanism in biology would require experimental manipulations; as we are not in a place to carry out such experiments, for the moment we prefer to take a more conservative approach rather than suggesting particular targets for future study. We will a discussion on this broader point, to our work. \n    \n* We thank the reviewer for drawing our attention to the simulation-based inference literature, and will expand the discussion section to reflect its connections with our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6aE8za3fS", "reviewer": "Reviewer_VFVu", "summary": "The authors address provide a cautionary tale for how data-constrained RNN models can mis-identify the dynamical structure underlying neural population computations when only a subset of the neural population is observed (\"partial observation\").  Empirical case studies are provided, whereby data-constrained student networks are trained from partial observation of synthetic teacher networks. The analyzed dynamics of the trained student networks differed from those of the teacher network in several teacher network setups. When teacher networks performed integration of a stimulus input via a feedforward chain dynamic, student networks were found to be biased to recover a line attractor dynamic under certain conditions. Mismatched student-teacher dynamics were also shown when teachers were linear noise-driven networks with non-normal dynamics or low-rank connectivity. The final case study provided details similar propensities for mismatched student-teacher dynamics when all networks are non-linear.", "questions": "- If I understood correctly, the student models applied throughout were designed to provide a one-to-one mapping from hidden units in the student RNN to observed teacher unit activity. Other data-constrained architectures do not rely on such design constraints--in particular, LFADS [11] and Neural ODEs [33] attempt to reproduce single-trial neural recordings (or teacher activity) via a readout from the RNN hidden activity space, thus allowing the RNN expressivity to be decoupled from the size of the observed neural population.  To what extent might some of the described mismatches be artifacts of the limited expressivity of RNN models whose individual units are constrained to directly match observed teacher unit activity? To be concrete, in the integration example, too few units in the student network was explained as a cause for the student network mis-identifying a line attractor motif when the teacher actually employed a feedforward chain (lines 127-132; and Figure 1b). Would a mismatch still appear by when fitting partial observations using a less-constrained architecture like LFADS?\n- Minor: The intro paragraph beginning on line 34 lists a number of challenges for data-constrained models. Why is it important for us to consider all of these limitations when only the partial observability is addressed in this work? There may be an opportunity to clarify the exposition here.\n- Minor: Line 66: \"even under relatively ideal conditions where the input to a circuit is either perfectly known or white noise\". In the white noise case, were known inputs not provided to the student networks? This sentence could benefit from clarification.", "limitations": "The authors did not include an explicit \"Limitations\" section, and beyond a brief sentence in the author checklist, limitations of the techniques are not discussed explicitly in the manuscript. I would encourage the authors to consider stating some of the limitations of their studies and the particular model class chosen for the student RNNs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Originality: The overall message and cautionary tale is indeed original (although see related work in Weaknesses).\n\n- Quality: The case study examples are well chosen and illustrative of the potential for mismatched student-teacher dynamics. The experiments appear well controlled and visualizations are intuitive and elegant.\n\n- Clarity: Overall, the writing is clear and appropriately didactic. \n\n- Significance: Recent years have seen a proliferation of data-constrained model development and interest in interpreting the corresponding trained models. Thus, this cautionary tale message is quite significant and timely. While perhaps beyond the scope of this work, solutions to the problems exposed, even if only hypothesized, could improve the significance of this work.", "weaknesses": "- Minor: There is a line of related work from Chethan Pandarinath's group on interpretable data-constrained models that should be referenced if not explicitly discussed. For example: Sedler, Andrew R., Christopher Versteeg, and Chethan Pandarinath. 2023. “Expressive Architectures Enhance Interpretability of Dynamics-Based Neural Population Models.” Neurons, Behavior, Data Analysis, and Theory, March, 1–22. https://doi.org/10.51628/001c.73987.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our work. Below we address some of the concerns and questions that they raise:\n#### Weaknesses:\n * We thank the reviewer for pointing out the work of Pandarinath's group, which we unfortunately missed citing in our initial manuscript. We will add the suggested references. \n#### Questions:\n* While the analytically tractable setup we describe in Section 3 is indeed constrained to a one-to-one mapping from units in the student and units in the teacher, we would like to emphasize that the motivating experiment described in Section 2 was actually performed with a latent variable model (LDS) that does not have such a constraint (See lines 109-113). Further, for our results on low-rank non-normal teacher dynamics (Section 3.1), we showed experiments in a supplementary figure that found that the spurious slow directions we derived analytically for the one-to-one mapping setup still occurred in LDS models (Supplementary F.5). We did not experiment with LFADS because it is a spiking model, whereas we focus on rate-based models throughout. Moreover, to the best of our knowledge, there's not a straightforward way of extracting spectral information from LFADS weights to make claims about attractor dynamics, whereas this can and has been done for LDS. \n* We describe the limitations of data-constrained models in detail here to help frame why the insights that are derived from such models are often rather course-grained (attractor-like properties rather than individual synaptic weights). Further, it motivates our analytical setup in Section 2, which includes two of the four stated limitations (partial observation and neuronal noise), and justifies why we focus on recovery of spectral properties over other possible criteria (e.g., weight recovery). Lastly, the discussion of these limitations supports the overall message that even in a best-case scenario where the other two limitations are ignored, data-constrained RNNs are inductively biased towards fitting (possibly spurious) attractor-like solutions.\n* In the white noise case, the students were only driven by noise; there are no known additional inputs. We will reword that sentence to make this distinction clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hpv1uqM7Zf", "reviewer": "Reviewer_Tqmw", "summary": "The authors investigate whether dynamical systems models that are statistically fit to neural data recapitulate the same dynamical mechanisms of the circuit. Commonly, neuroscientists only measure activity from a small fraction of neurons within a circuit. Under this constraint, the authors show examples where the model learns fundamentally different dynamical mechanisms. If true, this would be of great significance to the computational neuroscience community and an important cautionary tale.", "questions": "See \"Weaknesses\".", "limitations": "Limitations are adequately discussed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are a lot of good things to say about this paper. The motivating question is very interesting and important. The mathematical framing and analysis is elegant. The introduction and discussion sections are skillfully written to target the computational neuroscience community, and I think the paper has the potential to be very impactful within this community after some revisions.", "weaknesses": "I very much liked the big picture idea of this paper. However, in various parts of the paper, the writing is unclear and left me confused on details. The interpretation of certain findings should be clarified. I am optimistic about being to raise my score during the discussion period if these weaknesses can be addressed.\n\n* In section 2, the authors fit a linear dynamical system model with 5 latent dimensions (line 113) to 5% of neurons sampled from a feedforward chain circuit. The model fails to recapitulate the feedforward chain and instead learns a line attractor. Critically, there are two distinct but non-mutually exclusive explanations for this result: (a) 5 latent dimensions is not enough to capture the line attractor, or (b) the partial observation of 5% of the circuit is insufficient. The title and abstract of the paper exclusively favor explanation (b). However, on lines 127-132 the authors do not mention (b) and instead put forth (a), \"[if] the data constrained model has fewer neurons, it cannot realize a feedforward chain with sufficiently long memory\". Here, the recurrent dynamics in the LDS happen within a 5-dimensional latent space, so I assume that the authors mean \"fewer neurons\" to mean the relatively low dimensionality of the latent space.\n\n* I would like to see the result in section 2 more systematically investigated as follows. Let `N` denote the number of neurons in the teacher circuit (feedforward chain). Let `d` denote the latent dimensionality of the LDS model, and let `n` denote the number of neurons that are measured / observed for fitting the LDS model. I am interested to see whether the LDS model still fails to learn a feedforward chain when `N` is large (e.g. maybe 1000) and `n = d` is big but still a small fraction of the overall system (e.g. `n = d = 100`). Additionally, it would be elucidating to see an example where `N = n` (i.e. fully observed circuit), but a low-dimensional dynamical model is assumed `d < n`. The purpose of these additional experiments is to clarify whether the problem is that `n` $\\ll$ `N` (as suggested by the title / abstract), or whether the real problem is that `d` $\\ll$ `N`, or whether both are a problem.\n\n* Figure 3 seems to get at some of these concerns, but I think the paper will be much clearer if the above comments are addressed in Figure 1 as well.\n\n* In Figure 3, it is still a little unclear to me whether subsampling, per se, is the problem. If I understand the construction of the teacher network properly, as `D` increases, the timescale of the feedforward chain also increases. It is harder for the student network to match this timescale with a chain, leading it to instead learn a line attractor. I would like to see an experiment where the timescale of the feedforward chain is kept constant while increasing `D` (e.g. by inserting multiple copies of each neuron in the feedforward chain). In this case, when you randomly sample `d` neurons for the student network, do the learned dynamics still fail to capture a feedforward chain for large `D`? An affirmative answer would seem to be more in agreement with the main claims of the title and abstract. A negative answer would still be interesting, but would change the interpretation of the paper to be more in line with the idea that a mismatch in dimensionality is the core of the problem instead of sub-sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful critiques of our work, which have helped us further strengthen our paper. Below we respond to their comments on Weaknesses of our work:\n* The reviewer brings up an excellent point---that both partial observation and the selection of a latent dimension much smaller than the number of observed neurons could contribute to why the feedforward chain in the experiment of Section 2 is misidentified as a line attractor. We set up the experiment to parallel how data-constrained models are often fit in practice, with both partial observation and an explicit bias towards smaller latent dimensions, but appreciate the concern that this leaves room for multiple explanations as to why the mismatch we found occurs. However, since partial observation even in the absence of an explicit bias towards small latent dimensions already bounds model dimension to be smaller than that of the teacher network, the quoted explanation we provide was intended to implicitly also support explanation (b). Nonetheless, we will provide further clarification on this point and our interpretation of the findings here.\n* We agree that the experiment you propose would be fruitful in disentangling explanations (a) and (b), and as such have performed a sweep of LDS models of various latent dimensions and observability conditions, fit to the same functionally feedforward chain as in Fig. 1b. \n    \n    In particular, using the suggested notation, we fit models with a latent dimension $d = n-1$ (the $-1$ accounts for the fact that the LDS fitting procedure we use requires latent dimension to be strictly less than the observation dimension) to a functionally feedforward teacher circuit performing the same integration task (as in Fig. 1b). We performed fits at various values of $n$, up to the suggested fraction of $n = 0.1N$. In addition, as suggested, we fit LDS models of small latent dimensions $d \\in \\{2, 3, \\dots, 6\\}$ at full observability.  \n\n    The results demonstrate that the mechanistic mismatch we observe persists even if partial observability is the only driver of restricted model dimension. Further, the second experiment demonstrates that  a model of small latent dimension will still learn a line attractor-like mechanism even at full observability. This finding is consistent with the quoted explanation of lines 127-132. \n\n* We agree that an experiment with a fixed length feedforward chain while increasing $D$ could further clarify how the results of Fig. 3 should be interpreted. We interpret the suggested connectivity for such an experiment proposed by the reviewer as resembling synfire chain connectivity. Specifically, we consider teacher networks with connectivity $B = \\frac{1}{k}\\delta_{i+1,j} \\otimes \\mathbf{1}_{k\\times k}$, where $k$ is the number of copies of each neuron in the chain. Here, the $\\frac{1}{k}$ factor is included to ensure that all the nonzero singular values of $B$ remain equal to $1$ as the number of copies $k$ is scaled up. We compute the learned student matrix for a random selection of $d=50$ observed neurons in the limit of long observation time $T$, for fixed chain length $D/k = 50$ and values of $k \\in \\{2^0, 2^1, \\dots 2^6\\}$. For each value of $k$, we report the top two eigenvalues, time constants, and singular values of the learned student networks, averaged over $10$ random observed neuron selections, sampled without replacement. \n\n    The results are consistent with the interpretation that neither a feedforward chain, nor a line attractor (or any persistent timescale mechanism) are learned when $D$ is increased in this manner. Specifically, as $k$ is increased, the learned timescales approach the intrinsic neuronal timescale $\\tau$, ruling out persistent timescale mechanisms, and the learned singular values fall well below $1$, ruling out a feedforward chain-like structure. Thus, as in Fig. 3, the student still does not learn a feedforward chain, but fails to do so in a different manner.    \n\nPlots for the latest experiments are included in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the referees for their careful reading our our manuscript, and are gratified by their positive appraisal of our work. 

Here, we would like to address two common points of concern. 

## Title and Framing

First, we would like to clarify why we frame the title and abstract to focus on partial observation rather than  dimensionality mismatch in general: 

Partial observation is a near-universal property of data-constrained models in neuroscience, whereas restricted latent dimensionalities only apply to a subset of such models (many methods exist for fitting one-to-one mappings between student units and observed teacher units, such as FORCE, BPTT, and the MAP inference procedure we describe). Our latest experiments, the quoted reasoning of lines 127-132, and the second paragraph of the Discussion all suggest that dimensionality mismatches arising from restricted latent dimensions also contribute to the observed mechanistic mismatches. However, our analytical results only apply to the setting without the additional latent dimensionality restriction. Extending our theoretical analysis to latent variable models is an important objective of our future work. Consequently, to reflect our contributions accurately, we focus on partial observation in the title and abstract, using wording that does not rule out other possible sources of mismatch. We reiterate that we do discuss restricted latent dimensions as an additional driver of bias towards attractor-like solutions in the Discussion (paragraph 2). We nonetheless agree that this point should be expanded on, and will discuss it further in the context of our latest experiments. 

## Solutions

Second, we agree with the reviewers that it is important to provide an avenue to resolve the issues we highlight in our work. As we mentioned in our manuscript and discuss in our reply to Reviewer njnf, one approach to validating putative attractor structure is to examine responses to perturbations of neural activity, for instance using optogenetics. However, we believe that it is important to focus the present manuscript on documenting the space of failure modes before we can formulate a useful proposal as to how they can be addressed. In our revised manuscript, we will expand our discussion of possible approaches using targeted optogenetic perturbations, and caveats thereof.

Meta-review: This paper theoretically and numerically studies the implications of only partial observations for inferring dynamical systems models from neuroscience data. Theoretical analysis with linear models reveals that with non-normal dynamics certain mechanistic scenarios cannot be discerned under partial observations, or are wrongly identified. Numerical analysis with nonlinear models reveals further issues with spurious attractor structure. All referees felt this is an important contribution in the field of neuroscience, where dynamical systems methods are widely used yet practitioners are often not (sufficiently) aware of such issues, and technically the paper appears to be solid. 

Personally, I admittedly had more reservations with this work, but will go with the referees in my vote. Perhaps in their revision the authors may want to consider the following points:

1) Partial observations are a standard problem in any empirical setting that has been widely discussed in the nonlinear dynamics and physics literature (simple google search). Different solutions for it were advanced, from simple preprocessing by optimal delay embedding (which often already does a fair job) to reconstructing unobserved variables in the model’s latent space (a problem of course if one starts with too few latent states to begin with, as apparently much neuroscience work does). Personally, I found it a bit surprising that hardly any of this literature was discussed. Also, whether simple delay embedding could already alleviate the problem in this neuroscience setting would have been an interesting piece of information for practitioners (and simple to do).

2) Another thing the authors may want to check is whether optimal regularization parameters were chosen for the reconstruction methods. This was somewhat unclear, also why different settings were chosen for different methods, and whether other adjustments of the regularization may substantially improve the recovery of the observed dynamics.
Decision: Accept (poster)