Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks
Authors: Anushri Arora
Jonathan W. Pillow
Venue: NeurIPS 2025 poster ()
Keywords: low-rank rnns
dynamical systems
computational neuroscience
gaussian processes
Primary area: neuroscience_and_cognitive_science

Abstract: Low-rank recurrent neural networks (RNNs) provide a powerful framework for characterizing how neural systems solve complex cognitive tasks. However, fitting and interpreting these networks remains an important open problem. In this paper, we develop new methods for efficiently fitting low-rank RNNs in ''teacher-training'' settings.  In particular, we build upon the neural engineering framework (NEF), in which RNNs are viewed as approximating an ordinary differential equation (ODE) of interest using a set of random nonlinear basis functions. This view provides geometric insight into how the choice of neural nonlinearity (e.g. tanh, ReLU) and the distribution of model parameters affects an RNN's representational capacity. We show that this perspective leads to an online training method that achieves higher accuracy with smaller networks than previous methods such as FORCE, and outperform backprop-trained networks of similar size while requiring substantially less training time. We then consider the problem of finding minimal and maximal low-RNNs for approximating a target dynamical system.  We show that a variant of orthogonal matching pursuit (OMP) can be used to find the smallest RNN for a dynamical system of interest. At the other extreme, a dual space formulation allows for efficient fitting of infinite low-rank RNNs, which provide a Gaussian Process (GP) prior over dynamical systems.  We use the resulting GP marginal likelihood to optimize the hyperparameters governing neural activation functions, which leads to improved training performance even for finite RNNs. Finally, we describe active learning methods for low-rank RNNs, which  speed up training through the selection of maximally informative activity patterns.

Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks
Anushri Arora Department of Computer Science
Jonathan W. Pillow Princeton Neuroscience Institute Princeton University pillowsprinceton. edu
Princeton University ha1698@princetoo edu
Abstract
Low-rank recurrent neural networks (RNNs) provide powerful framework for characterizing how neural systems solve complex cognitive tasks. However, fitting and interpreting these networks remains important open problem. In this paper, we develop new methods for efficiently fitting low-rank RNNs in "leacher-training" settings. In particular, we build upon the neural engineering framework (NEF), which RNNs are viewed as approximating an ordinary differential equation (ODE) of interest using a set of random nonlinear basis functions. This view provides geometric insight into how the choice of neural nonlinearity (e.g. tanh, ReLU) and the distribution of model parameters affects an RNN's representational capacity. We show that this perspective leads to online training method that achieves higher accuracy with smaller networks than previous methods such as FORCE, and outperform backprop-trained networks of similar size while requiring substantially less training time. We then consider the problem of finding minimal and maximal low-RNNs for approximating a target dynamical system. We show that variant orthogonal matching pursuit (OMP) can used to find the smallest RNN for dynamical system of interest. At the other extreme, dual space formulation allows for efficient fitting of infinite low-rank RNNs, which provide a Gaussian Process (GP) prior over dynamical systems. We the resulting GP marginal likelihood to optimize the byperparameters governing neural activation functions, which leads improved training performance even for finite RNNs. Finally, we describe active learning methods for low-rank RNNs, which speed up training through the selection of maximally informative activity patterns.
Introduction
Recurrent neural networks (RNNs) are a popular tool for characterizing the computational properties of neural populations and the dynamics underlying complex cognitive tasks [166.. Previous work has proposed a variety of methods for training RNNs to implement a dynamical system or generate target signal of interest, including reservoir computing 81. FORCE (9 10, and back-propagation [HH However, trained RNNs remain difficult to interpret. Common approaches tend to use search methods to identify fixed points or "slow points", and then use dimensionalliy-reduie redducttonn methods to visualize projected flow fields around these points [12) [4]. However, fixed-poin finding algorithms are difficult to apply to nigh-dimensional systems, and often unclear how accurately low-D projections reflect a network's 's true dynamics 2 15
To overcome these difficulties, recent work has focused "low-rank RNNs", in which the recurrent weight matrix is constrained to have low rank [3] [6, [7]. This literature has shown that wide variety of tasks can be implemented in low-rank RNNs, which exhibit dynamies whose dimension limited by the rank of the recurrent weight matrix and number of input dimensions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A parallel arm of research has focused on methods for embedding now-dimessional quantities into high- sigh-dimessional network activity [8 Of particular relevance our work is the Neural Engi- neering Framework (NEF) which provides an analytical method for determining connection weights between neurons that implement a desired function or dynamical system. This achieved via distributed representations, where each neuron characterizes the target function through a random nonlinear projection, forming diverse tuning curves across the population. Network connection weights are then obtained by finding the optimal linear combination of these nonlinear projections via least-squares regression, enabling complex function approximation.
This approach is closely related to sparse linear approximation methods that seek to use overcom- plete library of basis functions to fit the ODEs [24 25 For instance, SINDY proposes using sparse regression
pre-specified polynomial basis to reconstruct system's dynamies [26]. Moving- window approaches [27] employ a sliding-horizon scheme: solving a constrained optimization at each time window, and using statistical tests to prune irrelevant basis functions, thereby maintaining parsimony over time. More recently, (28) considered linear splines as basis functions, preserving network expressivity while yielding analytically tractable update rules that improve interpretability. However, these methods rely on explicit parametric representations of the system dynamics. In contrast, neural-network Gaussian processes (NNGPs) offer a nonparametric alternative, encoding priors over function classes defined by infinite-width neural networks 21 While this has tradi- tionally been limited feedforward networks, recent work has extended it to recurrent settings via untied-weight constructions [32 34].
Here present unified framework for designing and training low-rank RNNs that implement target dynamical system of interest. We begin with an offline method based on NEF, where each neuron acts nonlinear basis function and least-squares regression embeds known ODE into the network. This yields a geometric view of universal approximation, illustrating how nonlinearity choices (tanh vs ReLU) shape representational capacity. We then extend this an online setting using a recursive least-squares update, and show that our method outperforms FORCE and BP in network accuracy and convergence speed. Next, we address the critical question of basis selection: using a variant of orthogonal matching pursuit (OMP), we identify the minimal set of basis functions needed implement a given dynamics. We then consider infinite low-rank RNNs, which converge to a Gaussian process (GP), enabling principled initialization by maximizing the marginal likeli- hood over hyperparameters governing neural activation functions. Finally, we introduce an active learning strategy to efficiently select informative datapoints, further reducing training time and data requirements
2 Background: low-rank recurrent neural networks
Consider a population of d rate-based neurons with membrane potentials x = [. Ia] and firing rates o(x) = (6(x1)..... Φ(xa)] where o6-) is a scalar nonlinearity mapping the membrane potential to firing rate (e.g., sigmoid, tanh, ReLU). The dynamics of generic RNN are given by vector ordinary differential equation and a linear output:
x= - Jo(x) + Bu, z <=W(xx)
(1)
where J e Rd is the recurrent weight matrix, B e paxdi the input matrix, u is a -imeensosio input signal, and W E Rdoue xd a readout matrix. This network becomes a low-rank RNN if the recurrent weight matrix J has reduced rank r < d, thus factorizing as:
J= M
m.n where {M,N E Rdxr
(2)
Here m,, are columns of M, respectively. In this case, the network state ætt) evolves within a subspace of at most + drn dimensions [3] 6), with activity in the remaining dimensions decaying due to the term (-x). Thus, the network state vector can be expressed as:
x(t) = Mx(t)) + Bv(t),
where xc(t) represents latent recurrent activity and vtt) denotes low-pass filtered inputs 6 17
Finally, R = F(K, u) represents the differential equation that governs the low-dimensional recurrent dynamics, where F is nonlinear function of the latent state k and input
!Code: https://gituub..om/mmusss - Training- of- Minimal-am -adi--nddd--iii Maximal- Lowall---oW-o-o Row--ank-aa- Lank-RNNs.git

---

3 An alternate view of low-rank RNNs
Standard approaches to training low-rank RNNs involves optimizing the parameters {N, M, B, W] via nack-propagation [77] Here we consider an alternative approach, which amounts to solving least squares regression problem with set of random nonlinear basis functions.
We begin by considering the problem of embedding an arbitrary low-dimensional dynamical system into a low-rank RNN. Specifically, we wish set the model parameters so that z obeys the dynamics of an particular "target" ODE:
= gz)
for some function g- We will then identify this output with the latent vector defining the network's activity in the recurrent subspace: ztt) - kt)). This implies that the dimensionality of the output is equal to the rank of the network, r = dout, and constrains the output weights to be the projection operator onto the column space of M, that is, W = (M M)- (If the target output is lower dimensional than the rank of the network r, we can truncate z to take only its first dous elements).
We are then left with the problem of setting the network weights M, N, and input weights B so that the latent vector z(t) evolves according to (eq. 4 For simplicity, consider the rank-1 case where z is scalar. This corresponds to an RNN weight matrix J = mn Assume that the input is also scalar, and that the input vector b E Rd is orthogonal to m (although we relax this constraint in SI C The network state can then be decomposed as a time-varying linear combination of m and b [T6] 17 (eq. 3
x(1) = mz(() + bv((),
(5)
where (t) represents the low-pass filtered input, resulting from the linear dynamical system V -V + u(). The fact that m and b are orthogonal means that we can write the dynamics that govern the latent variable explicitly as:
= + n 6(mz + bv)
a result shown previously in 17. and which is schematized in Fig. D Our goal of embedding an arbitrary ODE (z)) into the network can be now viewed as setting the model parameters so that
g(z) + n (ommm+++ bb)
To achieve this, note that the right-hand-side can be viewed linear combination of terms 6(m,z + b,v) with weights n, for E (1,...,d d}. Each of these terms can be viewed as a nonlinear basis function in z. If Φ is the hyperbolic tangent function, each such term is a shifted, scaled tanh function in z, where m, is the slope and b,u is the offset. This means that we can view the problem of embedding g(z) into low-rank RNN as the problem of setting and b to build an appropriate set of basis functions, and setting n so that the linear combination of basis functions approximates g(z) + This approach formalizes the connection between low-rank RNNs and the NEF [18 21). and shows that a low-rank RNN corresponds to a neural ODE with a single hidden layer 37).
Already, this perspective makes an important limitation clear: if the inputs v(t) are zero, the basis functions are all odd-symmetric (that is, g gm,,z) = g(- (m,2) for all z), crossing the origin only at zero. (see Fig. [C). Because -z is also odd-symmetric, and the linear combination of odd-symmetric functions is -ymmmtrcetiic, this means that in the absence of inputs, the network can only capture the odd-symmetric component of g(z). A low-rank RNN is therefore not a universal approximator unless it has inputs, or equivalently, different biases or offsets to each neuron (similar to general RNNs). If the di is instead taken to be ReL.U, the problem is even function with non-zero slope on either z > 0 or z < O. Thus the network can only approximate g(z) that are piecewise linear functions broken at the origin. (SI Fig. 7
severe: each basis function is linear
If we set the filtered input to be the constant v = 1, we see that the problem of embedding an arbitrary ODE in a low-rank RNN amounts to fitting the ODE in a basis of shifted and scaled basis functions in z. To achieve this, we propose to sample the scales (elements of m) and offsets (elements of b) to obtain a random basis, and then fit by least- regression, namely:
(olzgriam T + + (Zarramm T + b bT b (Zyriamm T + +b T (g(Zgria) + Zyria) (8)
where Agrid denotes a grid of points at which we wish to fit gz)) Note that we could use weighted least squares if we care more about accurately approximating certain regions of g(z), or add a small
3

---

A
low-rank RNN: standard Input
lntent space view
C
basie functions (no input)
besis functions (wi inpudt
integrated Input
+ b))
m
311
6/mm
401
D
target ODE
Iasont
latintt
2 darivativv
basis
doldi
adiivity
actidtyy
basis functions
+ nm ((x) + bu 2= d mz bv)
Two equivalent views of low-rank RNNs. (A) Standard view of rank-1 RNN with 8 neurons x and I latent dimension z. (B) Alternate view of the same network, now framed in terms of the dynamics of latent z. This shows that a low-rank RNN equivalent to a neural ODE with a single hidden layer [35]. (C) Basis functions obtained by sampling slope parameters m, ~ (N(,, 1), but without input (v, 0). (D) Attempting to fit an example ODE using this basis recovers only the odd-symmetric component, since all basis functions are odd symmetric. (E) Adding inputs allows basis functions have random horizontal offsets. Here we sampled the input weights b, NN0 1) and set input v, 1. (Note that this could also be obtained by using per-neuron "biases"), (F) Least squares fitting of n using the basis from (E) provides good fit to the target ODE.
Figure 1:
ridge penalty if the design matrix (whose columns are given by the basis functions evaluated at Zigrid) ill-conditioned.
Fig. D shows an illustration of this approach for an example ODE, here chosen to be a cubic polynomial with two stable fixed points and one unstable fixed point. Note that the network cannot approximate g(z) when the inputs are set to zero (Fig. -CD)), but can do so with near-perfect accuracy when both
the m vector and the (constant) inputs are drawn from Gaussian distribution (Fig.) B-F).
3.1 Multi-dimensional dynamical systems
We can apply this same regression- based approach to higher-dimensional nonlinear dynamical systems, where rank r = (z) > 1. In two dimensions, the basis functions are given by 6(1,,)) + 22 + b:), which are scaled, shifted tanh functions with random orientation (Fig. ( Approximating a 2D dynamical system with rank-2 RNN can then be written as the problem of fitting two different nonlinear functions gi (z) and (z) using two different linear combinations of the same 2D basis functions:
9 (z) [2)
21
I (M +b] (M( bb]
g(2)
+
n2 [
(9)
22
where M = [m m2] is a d x 2 matrix whose columns define the slope and orientation of each basis function, b is once again a column vector of offsets, and we have assumed constant input (v = 1). Note once again that if we do not include inputs, the basis functions are all radially odd-symmetric around the origin. Thus, once again, the RNN will only be able to capture radially odd-symmetric g(z), and is not universal approximator unless we include nonzero offsets bv * O.
To embed a given multi-dimensional ODE g( (z) into low-rank RNN, we once again generate a random basis by sampling the elements of M E Igxx2 and b E Rd from Gaussian distribution. The problem factorizes into learning each column vector n, for each dimension of the g. we have:
n; (o(2 (ø(ZgriaM' + + of M(ZrriaMT + - 16( + +b (g(Zgria) + Zgria), (10) for 1,2. This differs from the ID case above only that Zgrid is now a r-column matrix of grid points, where each row contains the coordinates of single point in z. Note that these grid points need not be uniformly sampled: we could sample them from an arbitrary distribution, or use a collection of points from simulating the ODE from variety of starting points (SI E.2).
Fig. 2 shows an application to an example 2-dimensional nonlinear ODE, in this case containing a stable limit cycle. Note that this 2D system is highly nonlinear and not radially odd-symmetric, so once again, embedding the system in low-rank RNN fails if we do not include inputs (or per-neuron
biases, SI A.2)

---

A
example 2D basis functions
target flow field
target des/dt dt
target day/ dd
22
fitted flow field
fitted dzy/dt
fitted day/d
72
simulated activity of 10 example RNN units
E
target and simulated latents
activiyy (
RNN 21
time
16
lime (6)
10
Figure Embedding 2-dimensional nonlinear ODE into rank-2 RNN. (A) Example basis functions obtained by sampling M and b coefficients from a zero-mean Gaussian, producing randomly oriented, scaled, and shifted hyperbolic tangent functions. (B) target two-dimensional nonlinear dynamical system, containing a stable limit cycle on a circle of radius one, represented as flow field (left), dzj or by its component functions (z) at and g2 (z) dz2 (right). (C) Least squares fitting of weight vectors n1 and n2 produces near perfect match to the target flow field, and functions 91 and 92. (D) Output firing rates o(x;) for 10 example units (i.e i E {1..... 10}) during the red example trajectory shown panel C. (E) Simulated trajectories from the true ODE (blue trace in panel B) and latent variable of the fitted RNN (red trace from panel C), plotted as a function of time, showing good agreement between the target ODE and the RNN output. Note that fitting was closed-form formm and did not require backprop-through-ting
di
Comparison with backpropagation (BP) and FORCE
Recursive least squares for online fitting: The previous sections present an offline learning approach for via basis design matrix composed of all points the grid (z) which the target ODE (g(z)) evaluated. Here, we adapt this framework to online learning format used by most state-of- the-art RNN training algorithms. In online learning, each timestep / 21,2.... generates an output i; which should match the target trajectory state z, Given M target trajectories: 2),iE [1, -00.....] M). the objective minimize prediction error between network outputs and target states.
-T,, ..),,, the network
We note, our NEF approach can be modified for such setting via online recursive updates to the weight vector n. This translates to recursive least-squares (RLS) optimization scheme (SI- Algorithm D summarizes our implementation).
To evaluate the performance of our method in online learning settings, we compare against widely-used training algorithms for RNNs: FORCE learning 10] and BP. Fach baseline a enjjectory-trcckinn framework, where the goal is to learn vector field dynamics from simulated low-dimensional trajectories.
Comparison:
We first evaluate on the classical sine wave generation task, originally used to benchmark FORCE [9. Our rank-2 networks, with targets comprising both the sine signal and its cumulative integral, consistently outperform FORCE-trained full-rank networks (weights were drawn from (0, 2)) across all tested network sizes (Fig. A). Notably, approach achieves lower MSE while requiring fewer neurons than FORCE, which needs larger networks to suppress chaotic activity to accurately reproduce oscillatory dynamics (for additional comparisons with FORCE see SI
E.3
next evaluate on a binary decision-making task modeled by bistable attractor ODE, Both our method and networks trained with BP, learn from teacher trajectories that start random initial

---

A
Comparison against FORCE eine tack
Loss
True vs Target Reconstruction (N+62)
10
TDREF
Target
Figure 3: (A) Training error as function of network size for RNNs trained via FORCE and method the sine- wave task. Right panel shows qualitative reconstruction for a network with 32 neurons. (B) Train and test error for RNNs trained to perform a bi- nary decision-making task us- ing bi-stable attractor dynam-
MSE
10
10
16
10
Number of nourons
Time
Comparisan against baciprop on decision making task: bi stable antractor
Train
Test Loss
10 POu Dlan
Our method
PAEK method
(tui rank)
MSE
10
frann
ics 38 39]
80
Epochs
conditions and converge to one of two fixed points (training details: SI E As shown in Figure 3 B. our approach achieves lower test error across comparable network sizes than both full-rank and low-rank BP-trained networks. Furthermore, our RLS-based approach converges with substantially reduced training time. These results highlight the efficiency our framework for capturing structured low-dimensional dynamics.
5 Finding the smallest RNN for given dynamical system
Thus far, we've introduced both offline and online methods for fitting low-rank RNNs to target dynamics, assuming access to predefined set of nonlinear basis functions. This naturally leads to a fundamental question: how should this basis set be chosen? Equivalently, what the minimal network size required to accurately approximate given dynamical system? To address this, return to the offline setting, where the full target dynamics (z)) are known advance and is scalar. Our goal is to identify the smallest low-rank RNN-i.e. -ie. the minimal number of neurons d' < d, that can accurately implement the function (z). Formally, we pose this as a sparse approximation problem: find minimal subset of basis functions & m. b) and corresponding weights such that the network can faithfully reproduce the dynamics. Mathematically, this implies selecting the best d' entries from & (m, b), to create basis:
@(m;, z + b;,)
D (m, b)
where (ii,..., in S (1,,,...,,4
Φmmig biy
Then, linear weighting (no n') is learned using least squares regression, where:
g(z) ~ -y * n' Pe (m, where {n'
d' vector)
(11)
To achieve the desired optimization of approximating (Z). we begin with a large enough ((m, b) obtained by sampling from a uniform grid of values for (m, b). We then follow an iterative approach, wherein at each iteration I, we greedily pick basis function i; with the highest alignment to the current residual estimate of g(z). This is done via an adaptation the well-established orthogonal matching pursuit (OMP) framework. A more detailed description of this process is provided in SI- Algorithm
is worth noting, changing the original basis set (mm, b) to Tp (m, b), could result in the algorithm converging different minima (global minima in &' d'mm, b) could be different from global minima in & (m, b)). However, the basis sets are equivalent, we observe similar performance across simulations (SI- Fig 13
We also introduce a continuous extension of this framework 40
Unlike standard OMP, which selects from discrete set of basis functions, continuous-OM refines each selected basis function by

---

A
(scaled) functions selected during OMP
approsimatin enror
Inpar decdy
10
Continuous OMP
B
tanget
2
FINN
-6
-6
Figure 4: Finding the smallest RNN for particular nonlinear dynamical system using orthogonal matching pursuit (OMP). (A) Scaled basis functions selected after 3,3,53 5, and 10 iterations of OMP, along with the linear decay term for example ODE (shown below). (B) Target ODE (black) and RNN fit after each step of OMP. (C) Mean squared error (MSE) between target ODE and RNN approximation function of the number of RNN neurons added by OMP, and our continuous OMP (COMP) method.
optimizing its parameters. Specifically, after each greedy selection, we optimize the corresponding (m, b) values by performing gradient descent the ordinary least-squares objective (Eqn [1]. Thus we search continuous space of basis functions rather than being limited to a predefined discrete set, offering greater flexibility in finding the optimal representation of g(z).
In Fig 4 we apply this method a simulated ID ODE, with two stable fixed points and one unstable fixed point. The first row shows the greedily-added basis functions, multiplied by their corresponding learned linear weightings. The bottom row shows their linear combination against the true underlying ODE. Through this iterative process, we observe with just 5 neurons our network almost perfectly reconstructs the target ODE. Finally, we note our COMP method of optimization allows for fine- tuning of the basis functions' parameters, therefore leading to more accurate approximations with fewer neurons (Fig. 4C shows sharper drops in MSE for fewer neurons). We also apply this method to higher dimensional dynamics (SI Dj.
Critically, previous work (c.g., 33 17) have similar dynamics which are learned using BP with much larger networks (typically 512 neurons). Our method instead provides an empirical framework find the minimum number of neurons needed to fit dynamics within estimated margins of error.
6 Infinite low-rank RNNs
So far we have taken a "primal space" view of fitting low-rank RNNs, in which we optimize the linear weights N over a fixed nonlinear basis order fit a dynamical system interest. In this section we will instead rely on dual space view in order to optimize the distribution of nonlinear basis functions employed in this representation. This view will turn allow us to consider low-rank RNNs where the number of units goes to infinity.
For simplicity, we focus on a rank-1 network with constant filtered input (v) given by:
0(=)
n. p(m;2 + bav)
(12)
Rather than fixing and optimizing the parameters (n, m, b) directly as we've 'e done previously, in this section we instead take bayesian perspective by placing prior distributions over the network weights. This in turn also makes (2)) a random variable, through which we can then characterize the distribution over functions expressible by the network. More precisely, instead of solving the regression problem in the primal form (Eqn B). the dual space view considers the similarity between inputs and z' in feature space denoted by the covariance kernel depicted in Eqn 13
K(2, x²) (m [m(m T z b bT (mm iz'' + +b +bTv)
(13)

---

Furthermore, the Central Limit Theorem ensures that in the limit of infinite basis functions (or neurons) d + oc, the network output converges to a Gaussian Process: g(2) S GP K), thus revealing that an infinitely wide low-rank RNN is exacily equivalent Additionally, the specific form of the induced kernel parameterized by the choice of nonlinearity and the distribution over (m, f). Specifically, when á erf(.) and the weights (m, b) (O, E). we recover the well known aresin analytic kernel [30
GP over state dynamics.
Kx(2,2) = sin -
E=' (1+22 Ex) (1+22 E+!)
(14)
where the kernel hyperparameters D fo of mmb), characterize the distribution of this basis.
Thus, given noisy observations of the form y; - g(z)) + e, with (O, oz), we perform GP regression and maximize the log marginal likelihood:
logp(y z, 2 Jy (Ks + + log Ky o log 2n.
Optimizing this objective E selects basis distribution that is statistically aligned with the data. Figure compares GP regression fits under two such choices. Panel A shows the result using standard normal prior over (m, b), is common in the literature 3 T6 6 17] the fit is poor and uncertainty remains high, especially away from the training points. Panel B shows the result after optimizing Σ: the posterior mean better matches the target function, and uncertainty is markedly reduced. This improvement stems from a data-adaptive kernel matrix that more faithfully captures the structure of the underlying dynamics. Finally, we train finite rank-1 networks using backprop- through-time (BPTT) and our online method (RLS) on trajectories generated from the ODE depicted Panels A and We demonstrate that in both cases, Gooopiime bases yield lower training MSE than saaddrd-norral -normal bases (Panel C). These findings represent averages over seeds of initialization; for additional training details see SI
E.2
Together, these results underscore the sensitivity of low-rank models to their initialization and show how principled, inference-based approach can mitigate this sensitivity without relying on extensive trial and error.
A 0.15
GP coveriance (standard)
larget ODE postanir predictive
Train Emrr:: Optimized basis Standard Normal
dz ati
BPTT standard basis standard basls
BPTT opt basis
10
opt basis
obsenvations
MSE (log scale)
B 0.15
GP covoriance loptibeec)
posterior predictive (optimized hyperparams)
16
dz
dt
100
Number of neurons
0.15
Figure Low-rank RNN as a Gaussian process. (A) Targed ODE and posterior predictive distribution for an infinite RNN given set of observations (red small). Here the GP covariance function (right) assumes a standard normal prior over basis parameters m and b. (B) Infinite RNN predictive distribution with optimized GP covariance hyperparameters (values obtained: of 0.6, og = 4.5, amb = 0.01), and resulting GP covariance (right). (C) Comparison of networks trained by initialization from optimal basis distribution vs the standard normal distribution. Rank 1 networks trained with BPTT & our online method (RLS) achieve lower train MSE. Standard basis implies network parameters were initialized via standard normal distributions. Opt basis corresponds to intializing network weights with the GP found hyper-paree parameters.

---

7 Active learning for low-rank RNNs
Another consequence of the dual space view is that it naturally suggests active learning strategy for sample-efficien fitting of low-rank RNNs. That is, if we can measure the ODE at a limited number of locations in latent space and input space where should take those measurements? Here we build on previous work on adaptive experimental design that proposed selecting inputs that maximize information gain about the parameters of interest [42)45). If we assume independent Gaussian noise in our measurements of /l, then the maximally informative location (z, v) is the one that maximizes our uncertainty about the weights n, which in turn corresponds to the maximal eigenvector of the posterior covariance.
Given the design matrix & ó(mz bv) and target vector y g(z) z, the posterior covariance over n under a Gaussian prior is proportional to:
Σ = + T + + -
(15)
where À is the ratio of measurement noise variance prior variance. For candidate data-point (2 the predictive distribution is Gaussian with variance:
02(2, = -(m= + bo) 6o(mm + be)
(16)
The most informative point for the trial is the maximizer of the predictive variance:
(=" = arg max 6(m= + bu) T + A7) - ( Φ(m= + be), (a,0)
(17)
which corresponds to the point in (2, space with maximal projection onto the top eigenvector of Σ. Intuitively, (17) targets regions where current uncertainty is maximal, therefore yielding the highest expected information gain per data point. Each selected point is added the dataset, and the linear weight n, is updated using the posterior mean given the data collected so far.
We illustrate this process in Fig. for system with non-normal dynamies. The heat map represents (2, v). while red arrows depict the estimated vector field after each active data acquisition. Initially, uncertainty is highest in unexplored regions (yellow); as more samples are acquired, the learned dynamics converge rapidly to the true ODE. Furthermore, it is worth noting that within this setting can also compute the minimum number of samples required bound depends on the dimensionality of the kernel function over the support of our system (i.e., the eigenspectrum of & T over grid of z, U E [-1,+1]), and the level of observational noise (46)
obtain a desired MSE; this
In conclusion, by exploiting the closed-form posterior the dual (kernel) space, we obtaín computationallyy efficient and statistically grounded acquisition rule that significantly reduces the number samples required to fit the system dynamics.
true ODE
B error in RNN fit
C
after measurement after 3 measurements
point
selected
high
MSE
Information
gain
low
2
measuremants
Figure 6: Active learning for low-rank RNNs. (A) Target ODE flow field over latent and input reflecting non-normal dynamics. (B) Mean squared error of rank-1 1 RNN approximation function the number active learning measurements. (C) Fitted ODE (red arrows) after (left) and 3 (right) measurements selected using active learning. Heatmap shows the predictive variance (heatmap) used to compute the most informative point for the next trial (yellow dot).
8 Discussion
In this paper we have introduced several novel methods for low-rank RNNs, including: (1) an online learning rule for training RNNs that substantially outperforms FORCE and backprop; (2)

---

methods for finding minimal RNNs (i.e., RNNs with the fewest units) using an extension of OMP; (3) infinite low-rank RNNs using the equivalent Gaussian Process; and (4) active learning methods for identifying the most informative points in state space to quickly learn approximation to ODE of interest. Note that the third contribution provides a novel solution to the problem of optimizing the nonlinear basis functions, that is, optimizing the distribution over M and B parameters so that the model can accurately implement an ODE of interest. We propose maximum marginal likelihood of the equivalent GP kernel hyperparameters can thus be used to obtain improved initializations for low-rank RNNs, even finite models that are ultimately trained with gradient based methods.
These results also open up several promising directions for future work. We have focused on the problem of embedding an ODE of interest; an important related problem is infer low-rank networks underlying high-dimensional neural observations, problem recently considered in [37]. Second, our active learning method assumed the ability to sample arbitrary locations in latent space, which unrealistic in practice. A promising future direction is to develop control theoretic methods to find maximally informative inputs the system 44 Finally, we examined GPs equivalent to infinite RNNs for specific choices of nonlinearity (namely erf, which we explored here, or ReLU 447), but designing kernels that best approximate the data and then extending these kernel functions to closed-form analytic solutions remains open problem 488)
Altogether, our contributions establish a theoretical and practical foundation for designing inter- pretable and sample-efficient low-rank RNNs, with broad applicability in both machine learning and neuroscience.
10

---

References
Paul Miller, Carlos D Brody, Ranulfo Romo, and Xiao-Jing Wang. recurrent network model of somatosensory parametric working memory the prefrontal cortex. Cerebral Cortex, (3(11):1208-1) 2003.
[2] Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology, 46:1 6, 2017.
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3):609-623) 2018.
[4] R. Schaeffer, M. Khona, L. Meshulam, International Brain Laboratory, and Ila R. Fiete. Reverse-engineering recurrent neural network solutions to hicrarchical inference task for mice. bioRxiv, pages 2020-06, 2020.
[5] Lea Duncker and Maneesh Sahani. Dynamics the manifold: Identifying computational dynamical activity from neural population recordings. Current opinion in neurobiolggy, 70:163-170, 2021.
[6] Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe. and Srdjan Ostojic. The role of population structure computations through neural dynamics. Nature neuroscience, 25(6))883-74, 2022.
[7] Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148:34, 2001.
[8] Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: new framework for neural computation based on perturbations. Neural Computation, 14:2531-2560 2002.
[9] David Sussillo and L. F. Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544455) 557, Aug 2009. doi: n0.10166/.euuron220099 2009.07.09..... URL http://t:/ doi. orgiorr/10 1016/j www.rn..ono..oo 20..000.000000 07.018
[10] Brian DePasquale, Christopher J. Cueva, Kanaka Rajan, G. Sean Escola, and L. F. Abbott. full-force: A target-based method for training recurrent networks. PLOS ONE, 13(2):1-8,, 02 2018. doi: 10.1371/ journal.pone.011 0191527. URL /tt.pttt::///o.. org/100 1171/ p191.019152
0191527
[11] Barak A Pearlmutter. Dynamic recurrent neural networks. 1990.
[12] David Sussillo and Omri Barak. Opening black box: oow-dimensional dynamics in high- dimenniionnn recurrent neural networks. Neural computation, 25(3):626 -49, 2013.
[13] Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-89, 2019.
[14] Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computa- tion by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78 84, 2013.
[15] Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems, 34:4738-4750, 2021.
[16] Manuel Beiran, Alexis Dubreuil, Adrian Valente. Francesca Mastrogiuseppe, and Srdjan Ostojic. Shaping dynamics with multiple populations in low-rank recurrent networks. Neural Computation, 33(6):1572- 1615, 2021.
[17] Adrian Valente, Jonathan Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank rnns. Advances in Neural Information Processing Systems, 35:24072-22086, 2022.
[18] Chris Eliasmith and Charles H Anderson. Neural engineering: Computation. representation. and dynamics in neurobiological systems. MIT press, 2003.
[19] Terrence C Stewart. A technical overview of the neural engineering framework. University of Waterloo. 110, 2012.
[20] Martin Bocrlin, Christian K. Machens, and Sophie Denève. Predictive coding of dynamical variables in balanced spiking networks. PLoS Comput Biol, 9(1): 11 2013, doi: 10. 1 1003258. URL http:// //xx doi. org/10...10. 13711 pcbi 1003258

---

[21] Omri Barak and Sandro Romani, Mapping low-dimensional dynamics to high-dimensional neural activity: A derivation of the ring model from the neural engineering framework. Neural Computation, 33(3):
827-852, 2021.
[22] Larry F Abbott, Brian DePasquale, and Maoul-Martin Memmesheimer. Building functional networks of spiking model neurons. Nature neuroscience, 19(3):350-33) 2016.
[23] Alireza Alemi, Christian Machens, Sophie Deneve, and Jean-Jacques Slotine. Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[24] Markus Heinonen, Cagatay Yildiz, Hearik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ode models with gaussian processes. In International conference on machine learning, pages 1959-1968. PMLR. 2018.
[25] Harish S Bhat, Majerle Reeves, and Ramin Raziperchikolaci. Estimating vector fields from noisy time series. In 2020 54th Asilomar Conference on Signals, Systems, and Computers, pages 399-606. IEEE, 2020.
[26] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Sparse identification of nonlinear dynamics with control (sindyc). PTACC-PapeSSOLLii 49(18)7100-71 2016.
[27] Fernando Lejarza and Michael Baldea. Data-driven discovery of the governing equations of dynamical systems via moving horizon optimization. Scientific reports, 12(1):11836, 2022.
[28] Manuel Brenner, Florian Hess, Jonas M Mikhaeil, Leonard F Bereska, Zahra Monfared, Po-Chen Kuo, and Daniel Durstewitz. Tractable dendritic rans for reconstructing nonlinear dynamical systems. In International conference on machine learning, pages 23200. Pmlr, 2022.
[29] Christopher Williams. Computing with infinite networks. Advances in neural information processing systems, 9, 1996.
[30] Radford M Neal and Radford M Neal. Priors for infinite networks. Bayesian learning for neural networks. pages 33, 1996.
[31] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv: 2017.
[32] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
[33] Greg Yang. Sealing limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arxiv:/902.0445 2019.
[34] Xiang Sun, Scongyoon Kim, and Jung-Il Choi. Recurrent neural network-indueed gaussian process. Neurocomputing 509:75-84, 2022.
[35] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[36] Arthur Pellegrino. N Alex Cayco Gajic, and Angus Chadwick. Low tensor rank learning of neural dynamics. Advances in Neural Information Processing Systems, 36: 1674-11702, 2023.
[37] Matthijs Pals, A Erdem Sagtekin, Felix Pei, Manuel Gloeckler, and Jakob H Macke. Inferring stochastic low-rank recurrent neural networks from neural data. arxiv [cs. 1g), 2024.
[38] Kong-Fat Wong and Xiao-Jing Wang. A recurrent network mechanism of time integration in perceptual decisions. Journal of Neuroscience, 26(4):1314 13241138,8 2006.
[39] Thomas Zhihao Luo, Timothy Doycon Kim, Diksha Gupta, Adrian Bondy, Charles D Kopee, Verity Elliot, Brian DePasquale, and Carlos D Brody. Transitions in dynamical regime and neural mode underlic perceptual decision making. bioRxiv, pages 2023 10, 2023.
[40] Chaitanya Ekanadham, Daniel Tranchina, and Eero P Simoncelli. Recovery of sparse translationiinvaiiant signals with continuous basis pursuit. IEEE transactions on signal processing, 59(10)4735-44) 4744, 2011.
[41] Karin C Knudson, Jacob Yates, Alexander Huk, and Jonathan W Pillow. Inferring sparse representations of continuous signals with continuous orthogonal matching pursuit. Advances in neural information processing systems, 27, 2014.
12

---

[42] David J.C. MacKay. Information-basse objective functions for active data selection. Neurul Computation. 4(4):590-604, 1992.
[43] Hendrik Kuck, Nando de Freitas, and Arnaud Doucet. Sme samplers for bayesian optimal nonlinear design. In 2006 IEEE Nonlinear Statistical Signal Processing Workshop, pages 99 102. IEEE, 2006.
[44] Horia Mania, Michael I. Jordan, and Benjamin Recht. Active learning for nonlinear system identification with guarantees. Journal of Machine Learning Rexearch, 23(32):1-30, 2022. URL http:tt::// org/
papers/ Sopers/v23/20- 1003/20-801 807 .htll
[45] Aditi Jha, Zoe Ashwood, and Jonathan W Pillow. Active learning for discrete latent variable models. Neural computation, 36(3):33-477, 2024.
[46] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: review. Statistical science, pages 273-304, 1995.
[47] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
[48] James Benjamin Simon, Sajant Anand, and Mike Deweese. Reverse engineering the neural tangent kemel. In International Conference on Machine Learning, pages 20215-20231. PMLR, 2022.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have elaborated on all claims made in the abstract and introduction in the main paper, and they reflect the paper's contributions accurately
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss limitations in the discussion/concllsioo section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
AnA]
Justification:
We do not have any theorems or lemmas, as this is not a theory paper
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided details all parameters and modeling details in the main paper.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release code with the final version of the manuscript. However, we provide sufficient details to replicate our framework.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee////eeeuubiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper focuses on closed form solution via regression and are repro- duce using the classical ridge regression solution, thereby not requiring any specific error bars/solvers. We provide details on initial conditions which allow reproducing our results. Additionally, for networks trained via BP provide necessary details the task and data used, with additional training details (epochs trained etc) in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our paper focuses on closed form solution via regression and are repro- duce using the classical ridge regression solution, thereby not requiring any specific error bars/solvers. We provide details initial conditions which allow reproducing results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [Yes]
Justification:
We provide details in the main/supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper adheres to the NeurIPS code of conduct.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not envision any societal impact of this work
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The answer NA means that the paper does not existing assets
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The answer NA means that the paper does new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
19

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method did not involve LLMS for any important, original non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httpp:///eerrrppccccc//ooteerese for what should or should not be described.
20

---

A Choice of nonlinearity
If we consider a network with a rectified-linear instead of a tanh nonlinearity, the restrictions on the network's representational capacity in the absence of inputs are even more severe (Fig. 7 In this case, the basis functions are all scaled and axis- axis-flipped relu functions that intersect the x axis at 0. Thus they can only represent piecewise linear functions composed of two pieces with a knot at zero. Adding inputs (or per-neuron biases) allows the network to have universal approximation capabilities.
A basis functions (no input)
C
basis functions (w/ input)
2
0
-2
0
2
-2
0
2
Z
Z
B
D
target ODE basis
5
5
dz/dt
dz/dt
0
-5
-2
0
2
-2
0
2
Z
Z
Figure 7: Representational capacity of 1D low-rank RNN with rectified- (relu) nonlinearity.
(A) Set of basis functions obtained by taking random coeffients m, ~ ~N(0, 1) but without input (v, 0). (B) Attempting to an example ODE using this basis recovers only piecewise linear fit with kink at zero. (C) By adding inputs, basis functions have random offset well as slope. Here we set v, and sampled the input vector coefficients b, ~ (O, 1). (D) Least squares fitting of in the random basis from (C) provides a high-accuracy approximation to the target ODE.
A.1 Comparison of activation functions in estimating ODEs
In this section, we explore the low-rank RNN's ability to approximate different types of dynamics (i.e function classes), with different activation functions (i.e basis functions). Our discussion above highlights how relu units can approximate functions through piecewise linear components. Non-zero inputs create basis functions which can be used to compose ODEs with "knots" the shifted offsets. Alternatively, through our discussion in Section 3] we note tanh units provide smooth non-linear basis functions. The non-zero inputs create shifted basis functions, which perform a similar role, with smooth compositions. Following this intuition, if an ODE consists of smooth non-linear components can be hypothesized that tanh units would have higher performance. Whereas, the ODE consists of piecewise linear dynamics, relu units would prove to be more optimal. To validate this, we simulate two such ODEs Fig. [ Trivially, in the case of large enough number of basis functions, networks comprising of relu or tanh units can approximate any function (i.e they behave as universal approximators). However, to assess performance, we estimate the smallest networks in both cases that can fit the ODE within pre-defined margin of error. As expected, the ODE with smoother non-linearities can be fit with smaller tanh networks than relu networks (the opposite true for piecewise linear ODEs).
21

---

A
ODE 1
Comparison of activation functions
tanh neurons
relu neurons
relu neurons
2
MSE
MSE tanh MSE relu
densyy
target ODE fit in basis
dzigt
10"
MSE..05
5
12
0 2
6
-6
0
2
B
ODE 2
Comparison of activation functions
tanh neurons
tanh
relu neurons
25 neurons
10"
Byner decoo
MSE tanh MSE relu
=
MSE
target ODE basis
dzlet
MSE,
10
25
Number of neurons
Figure 8: Performance comparison of tanh v/s relu in approximating different ODEs. (A) Depicts an ODE with two stable fixed points and one unstable fixed points (better fit with tanh non-linearity). (B) Depicts an ODE with a shifted knot and two linear components (better fit with RELU non- linearity). First column represents MSE (for network with tanh and relu activations) as a function of the number of neurons in network. Neurons are added using OMP. The top row of the right column shows scaled basis functions selected via OMP. The bottom row shows fits (orange) of the target ODE (black) at the marked iterations of OMP.
A.2 Absence of inputs for limit cycle
Section 3.J depicts a limit cycle embedded into the low-rank RNN using our framework. The specific ODE of non-linear and non-symmetric system is give
dx
(=f + 2f)
20 21 0.35
dl
/ +2 +
dy
1 += V (
+ 0.5
dt
where is small constant added for numerical stability. The constant values in each dimension make the underlying ODE non odd-symmetric.
In this section we show the inability of an RNN without inputs to appropriately approximate this function. In Fig. 9 the first column represents contour plots of the target ODE for each dimension. The overlayed vertical and horizomtal dashed red lines depict X =
0, Y 22 0 respectively. Note, there is a slight (left and upwards) shift in the contour plots, indicating the non-radial symmetry. This is introduced by adding a constant negative decay in 21 and positive correction in 22. The second and third columns represents the fitted ODEs for an RNN with and without inputs respectively. It can be observed the RNN without inputs is unable to create offsets in any dimension, thus failing at recovering the underlying ODE. To further highlight this we simulate a sample trajectory from the polar coordinates of limit cycle (detailed in Section 3.T) in the last row of Fig. | As expected, the low-rank RNN with inputs almost perfectly overlaps the trajectory, unlike the low-rank RNN without inputs.
22

---

Target de
Fitted 1dd with
Fitted 1/dh without inputs
Target dz2/dh
Fitee cr2/dt with inputs
Fitted dz2 et without inputs
Sample Trajectory
RNN with inputs
RNNN without inputs
True True RNN 21 RNN 22
Time
Figure 9: Influence of inputs in capturing non-symmetrical limit cycle
B Online recursive least square (RLS) algorithm
Here we provide additional details recursive least-square algorithm. Specifically, for an observed target trajectory {2.)T=1 with the initial state Zo:
Algorithm Online RLS for Low-Rank RNNs
1: Inputs: target trajectory (2,)?= step size dt.
2: Initialize: Basis parameters: m, bias b, weights n +- 0, Precision P P --1
for 1 to 7 do
4: Basis vector Oz + o mz + b)
5: Target derivative V/ 2, Zz
(finite difference)
dl
6: Prediction 01 +- n 01 2t
7: Error e, 3/1 D1
8: Gain
I+ p
9: Weight update <- k4 Ct
10: Covariance update +- ki o,
11:
end for
C General formulation & application to binary decision making task
We apply our framework to specific group of binary decision making tasks commonly observed in systems neuroscience. In this task, a rat accumulates evidence of auditory pulses over time from clicks on its left and right side. At the end of the stimulus period, the rat must turn to the side which produced more clicks, and is rewarded for inferring this correctly. has been shown that multiple underlying dynamical portraits could represent this behavior [39] We thus show applicability of our method by using recover the autonomous and input driven dynamics on four separate synthetically generated dynamic portraits linked this task [39]. Here, intuitively, the input dynamics encode
23

---

for the accumulation of evidence based on the clicks, and a final decision to turn is made once the accumulation value reaches a specific attractor in the network. For instance, if the instrinsic dynamies encode a bi-stable attractor, each of the end points represent specific decision, and the inputs move the dynamies along a line between them [38 Additionally, consistent with previous studies, we model our simulations to provide equal weights to left and right clicks but with opposite magnitudes.
We model four flow fields representing intrinsic dynamics, namely a bi-stable attractor, line attractor, a non-canonical line attractor and the flow field inferred from 39 More formally, they are given as follows
Bistable attractors:
dæj = 102, 21) =1)dt + cudt
d
1022dt
Classic DDM line attractor:
21 E (-0.7,0.7) dz1 1021 (0.7 21)(0.7 21)
cudt
2 $ (-0.7,0.7)
dæ2 3022
(18)
Non-canonical line attractor:
dæ 5
d = - 522dt + cudt
Unsupervised model:
dz1 521 (0.85 + (0.85 - 21)dt + cudt
dæ2 (0055 + 0.1)(21 1.222)
Here, 21, 22, represent the two latent dimensions, u represents the magnitude of the input clicks, and represents its positive or negative.
Critically, we observe the input dynamics lie in dimension parallel the recurrent activity. Or alternatively, drive the system in the dimensionality spanned by the recurrent activity. We thus present general formulation of our equations model these input dynamics. Following Eqn for scalar 2, we now not only observe orthogonal (b = bperp) neuron specific inputs, but additional input dynamics that influence the recurrent activity (Dpor. spans the same direction as m), thus updating Equ :s:
x(t) mz(() + bpar Vpar(t) + DperpVperp(e (t)r
(19)
where (t), perp (t) represent the low-pass filtered inputs which drive activity along and perpendicular to the recurrent dimensions respectively.
Our goal of embedding the ODE g(z) into the network can now be viewed as setting the model parameters so that
g ( + n (m byar Vpar(1) + DperyUperplee
(20)
This allows us to follow a similar setup to our discussions in Sec. 3, with the exception that auditory inputs are applied along bpar or b Dperp- or both.
As shown in Fig 10, each row represents one of the above dynamical regimes. The first column represents the dynamics along or 22, and the RNN fitted version. Next, we model two right (or positive) clicks at 0.5 and = 1 second and single left (negative) click t 2.5 second. The second column represents the ODE when we start from (z = 0), pushed by these input dynamics, for our fitted RNN dynamics (Eqn 6 against the true ODE (computed using Euler method). Lastly, we also recover the underlying flow fields, as indicated by the last column. In Fig 11 we embed a non-canonical lite-attractor in which input axis is perpendicular the line attractor and non-normal dynamics give rise to movement along the line attractor. We successfully embedded all three of these systems with rank RNNs. Lastly, we also embed a system with rotational dynamics between fixed points with integration along the diagonal between them. This is done through rank 2 RNN with inputs along each of the directions spanned by bpar (Fig. BJ. This proves the flexibility of our framework in embedding dynamics associated with neuroscience tasks.
24

---

Bi-Stable Attractor
1D Flow Field
Auditory inputs driving trajectory.
2D Flow Field learnt by 1D RNN
Arbitrayy Dimension 1.82
0.8
1////
Z
-1.0
1.5
0.0
0.5
1.0
1.5
2.0 2.5 3.0
3.5
4.0
-1.0
1.0
Time in seconds
RNN Learnt Dimension
Targes CDE
RNA
piP
Unstable FP
B
Line Attractor
Flow Field
Audilory inpuls driving trajectory
2D Flow Field leemnt by 1D ANN
10.8
Right Click value
0.84
1.0
-0.9
value
1.0
Abbrraay Dimonsion
-1.0
2
1.0 0.0 0.5 1.0 1.5
Time in seconds
2.0 255 3.0
3.5
4.0
-1.0
RNN Learnt Dimension
1.0
Figure 10: Two different dynamical portraits for binary decision making task: (A) bi-stable attractor ODE and (B) line attractor ODE. First column represents the true underlying ODE and the RNN estimate learned using least squares. Second column depicts a sample trajectory driven by momentary input clicks. A right click creates drift towards the positive stable fixed point where left click, towards the negative stable fixed point for A. For B, accumulation along the line takes place with no diffusion. Third column represents the fow-field estimated by the RNN.
A Non-canonical RNN line attractor flow field
B
Unsupervised-modd model flow field
trajectory
Point Attraclor
Input Dimension
Line Attractor
Z1
Z1
Input pulse train
Input pulse train
0.5
inp dim
0.8
z1dim
22 dim
21 dim
Rank 1 RNN
Rank 2 RNN
-0.5
-0.5
time (s)
time (s)
Figure 11: Two additional dynamical portraits for binary decision making task. Top: flow-field for each ODE, with input driven trajectory highlighted in blue. Bottom: true and fitted trajectories over time for each dimension.
25

---

D OMP algorithm and it's application to a limit cycle ODE
The detailed algorithm for OMP is given here:
Algorithm 2 OMP for finding smallest RNN
1: Select grid of values
2: Create global basis set @ using uniformly sampled m, values.
3: Initialize weights using linear decay term only; no (2 ( 1)= g(z).
4: Initialize residual: ro = g(z) nig * 100=(2 (-z)
5: Initialize solution basis set, du 0.
6: At each iteration t:
1. Find basis vector with highest correlation with residual:
arg max || r-llllll
(21)
2. If entry is not in the solution basis:
Add new entry to the solution basis, D +- D;
Solve to find new linear weights [no, n']l using Eqn 8
Solve find new linear weights n2 using Eqn g
Compute the updated residual:
noo (-2) n +;
g(2)
(22)
3. Check for termination based on predefined sparsity threshold
d' len(
(23)
D.1 Smallest RNN for 2D Limit Cycle
Following our discussion on the multi-dimensional case and the smallest RNN (Sec. 3.1 5 apply our framework to learn the smallest number of neurons needed fit an RNN for the limit cycle flow-field (equations provided in SI. A.2p. Specifically, we apply both our OMP and COMP method to greedily add neurons that best approximate the target ODE. Fig 12 highlights the advantage of our COMP and OMP methods in designing small RNNs. Additionally we note, for our COMP method, with just 20 neurons the ODE fits are qualitatively similar the true target, with much lower MSE values compared to OMP. This showcases the advantage of COMP as the basis parameter space increase.
we
D.2 Parameter distribution of random basis for OMP
In this section delve into the role of the distribution from which the random basis is sampled. As shown in Section 5 each basis function approximates the ODE (g(z)) over some finite domain (z). Thus, first, is critical the basis functions span the domain of the function being approximated. Second, these functions need not be odd symmetric and hence basis functions need to also be shifted capture these movements. As shown Fig 3 as long as these properties are met (i.e both the uniform grid and standard normal generate basis functions in the same domain, with the same offset ranges), the exact underlying distribution from which the basis functions are drawn does not play critical role when finding the smallest low-rank RNN. This can be seen as the MSE values follow similar trends with greedy addition of basis functions (panel D). Qualitatively, this can also be observed via similar reconstruction of the ODE across iterations of OMP (panel A,B). Note however, by changing the distribution from which drawn the exact basis picked are different, as the global optimal basis are no longer the same (panel C).
26

---

MSE vs Number of Neurons
1etromac10
nourons-
OMP COMP
frted dt
dilld
10 Neurons
15
M mag m Iine MiN
Tupedary
Figure 12: COMP and OMP for 2d Limit Cycle. The first row depicts the estimated flow-field for dimension one. The second row depicts the estimated flow-field for dimension two. Last row shows true vs the RNN reconstructed trajectory for both OMP and COMP. We show fits and reconstructions for 5, 10, 20 neurons that added via OMP and COMP respectively. Right most panel depicts the mean squared error (on log-scale) as a function of the number of neurons. Note COMP shows much steeper drops in MSE, compared OMP.
Neuron target ODE ii bussia
ODE fits with OMP
5 Neurons
C
Scaled basis functions with
D
approximation error
dzeet
Basis: Normal Distribution
Besis: Uniform Distribution
MSE
largel ODE!
6
neurons
Figure 13: Influence of distribution of basis functions. (A,B) RNN estimated fit of Bi-stable attractor
ODE the true underlying ODE over 1,3,5 iterations of OMP. the top row basis functions (both m; and b,) are generated over a uniform grid spanning +1 to -1. Alternatively the bottom row consists of basis functions generated from standard normal (i.e m, ~ NN 1), and b, ~ (0, 1)). (C) Scaled basis functions selected after 5 OMP iterations, with the top row drawn from the dniform distribution and the bottom row from the normal distribution. (D) Mean squared error (MSE) between target ODE and RNN approximation as function of the number of RNN neurons added by OMP (blue: basis functions drawn from uniform grid, green: basis functions drawn from standard normal).
E Additional training details
E.1
Backprop comparison for binary decision making ODE
In Section both our framework and the networks trained with Backprop are trained and tested from of teacher trajectories, which are simulated from the underlying ODE (eg., in the binary decision-making task, trajectories originate from random initial conditions and evolve toward one of two fixed points). To generate each trajectory, we used Euler integration with a time step of dt 0.01 over a duration of 4 seconds, yielding 400 time steps per trajectory. A total of 160 unique initial conditions were uniformly sampled along the z-axis. Of these, 150 were used for training and the remaining 10 for testing. Each of the networks trained via BP in Fig were trained for 15 epochs, with batch size of 10 per epoch. Thus, total of 150 gradient steps were performed (performance plateaued at 100 gradient steps). Additionally, each of the networks were trained over three random seeds of initializations, where parameters were initialized from standard normal distributions. The values reported in Fig 3 correspond to the best performing seed. Lastly, to compare performance we trained our networks via our online RLSS implementation. Once training was complete (using the 150 training trajectories), we used the final weight to test performance.
27

---

We report training times in Table !
Note, our framework provides significantly faster training.
Table I:
Training Time For Binary-Decision Making Task
Model Type
Size
Time(s)
Low Rank (r = 1)
5
62,419
Low Rank (r = 1)
10
62,468
Full Rank
2
29.161
Full Rank
3
29.209
Full Rank
5
29.025
Full Rank
29.392
Full Rank
50
28.998
Our Model
5
0.069
E.2 Target tracking task: Compare networks with optimal bases vs standard normal bases
To compare performance for networks trained with optimized basis against those trained with the stan- dard normal basis, we derive target matching problem from the ODE presented in Figure (Panels and B). Specifically, we used a set of four starting locations in z given by: [7.92, 4.72, 2.3, -7.36]. We then rolled out the dynamics (as per the ODE) using an Euler integration for total of 1000 time-steps, with dt 0.1. Each of these trajectories (originating from each of the start locations) converged to specific stable fixed point. We then compared rank 1 RNNs trained via Online Method (RLS) and those trained with BPTT. Our RLS method achieves near perfect performance after single epoch, while BPTT trained networks trained for epochs. MSE results presented are averaged over 5 random seeds of initialization.
E.3 FORCE Comparisons
E.3.1
Lorenz Attractor Trajectory
Following our discussion in Sec. 4 present additional evaluation using more challenging target trajectory: the Lorenz attractor, a well-known chaotic system defined by the set of coupled nonlinear differential equations
dx
dy
dz
= oly - x),
= x1p 2) y,
zy B2,
(24)
dt
dt
dt
10, p == 28, and B -8/3. We simulated the trajectory using Fuler integration with a time step of dt = 0.01 for a total of T = 3000 steps (30 seconds of data), and used the scaled (t) signal as the target output (as in (9) While our method was trained on (and able to reconstruct) all three variables Z = [z(t), yt). and 2(t)]. only reconstructions of z(t) are shown for comparison, as this is the only signal used in the FORCE training objective 9 Specifically in our case, this can be formalized
using standard parameters
rank-3 RNN, written the problem of fitting three different
nonlinear functions de dy and functions:
da using three different linear combinations of the same 3D basis
di de
de
da dt dy
x
n (MZ + b)
(MZ + b)
(25)
I
(MM + bb]
where M [m; mymg]] is d x matrix, is once again column vector of offsets, and we have assumed a constant filtered input, v =
We compare networks trained using our online learning framework against those trained with the FORCE algorithm. For FORCE, we used a recurrent gain of = 1.5 and trained each network across 10 random seeds; method was similarly evaluated with the same number of seeds. Fig 14Irppors mean squared error (MSE) averaged across seeds, along with representative trajectory reconstructions.
As previous experiments, our method consistently achieves lower MSE with fewer neurons. Qualitatively, this is also evident in the trajectory reconstructions: networks produce more
28

---

accurate fits at smaller sizes (N = 16, 64) than FORCE, which fails to reliably capture the signal until larger networks (for instance showed with N 1024). Additionally, this further highlights that our networks converge more rapidly than those trained with FORCE.
Comparison against FORCE on Lorenz Attractor
MSE on trajectory
10a
10a²
MSE
FORCE Our Online Method
10
10a
10'
10'
Number of neurons
Target
FORCE
True vs Target Reconstruction (N=16)
1.00
Our Online Method
0.50
x(t)
0.90
-0.75
10
20
True vs Target Reconstruction (N=64)
0.
0.50
0.25
0.75
1.00
6
10
True vs Target Reconstruction (N=1024)
1.00
mas may
wwy
10
Time (s)
Figure 14: Top Row: Mean squared error (MSE) in log scale across networks of varying size (left). Bottom Rows: True vs reconstructed x(t) trajectory from the Lorenz system, (green our method, blue FORCE) for networks of size 16, 64, 1024.
29

---

E.3.2 Application to noisy data
We further validate our framework by discussing it's application to noisy data. Specifically, we simulated a target signal for total of = 1000 seconds. This signal was sampled with discrete time bin of di 0.1, thus 10,000 time steps of the target wave were obtained. We then added independent gaussian noise: N ~ (0, 0.05), at each time step the trajectory to obtain the noisy target (also used in FORCE 19). While the target output is 1-d signal, we used the trajectory and it's two cumulative sums to fit our networks (thus needing rank 3 network). We trained RNNs using RLS method, FORCE and BPTT (rank 3, 4). To ensure consistent comparison, in Fig. [3 report the training MSE (averaged across 5 seeds) after one epoch: a single pass over the 10, 000-step sequence, across all methods. We see highest performance with RLS method. Additionally, both RLS and FORCE achieve near-perfect performance in a single pass, while BPTT typically requires 5-8 epochs to reach comparable errors.
A
Target Signal: Noisy Sine Wave
B
FORCE vs BPTT vs Our Online Method (RLS)
BPTT (rank BPTT (rank FORCE
RLS (rank 3)
MSE (log scale)
10
z(0) o
-1
200
600
800
1000
10
Time (seconds)
8
16
64
128
256
512
1024
Number of neurons
Figure 15: Networks fit on a noisy sum of four sinusoids target signal. A: Target trajectory sampled for T 1000 seconds. B: Training Mean Squared Error (in log scale) of networks trained with FORCE, BPTT and our Online Method (RISS)
E.3.3
Additional discussion online (RLS) method and the FORCE Framework
Below we discuss some ways in which our methodology differs from the FORCE/FULL-FORCE training schemes.
9 10
1. Initialization with Full-Rank weight matrix: FORCE/FULL-FORCE methods require full rank-initializan iaakiiitializations. and learn low-rank updates on this initilalization over time. This comes at the cost of interpretibility as the dynamics of such networks need to be analysed post training through methods such as PCA. On the other hand, our offline and online framework directly models the latent low-dimensional dynamic and doesn't ever need full-rank initializations.
2. Senstive to initialization and requires multiple epochs: One of the motivations of FORCE is that it introduces the benefits of training networks that exhibit chaotic activity prior to training. While powerful, it is observed this results in these networks needing multiple epochs/iterations. Additionally, these networks exhibit stochastic results based on initializa- tion (e.g based on the parameter g). In contrast, methods provide deterministic closed form updates.
3. Doesn't directly embed an ODE, but produces a set of target trajectories: A stark difference between offline framework is unlike other training methodologies similar to FORCE and FULL-FORCE that can only be trained against target trajectories, we can also directly model the underlying ODE. Thus, in cases where such a hypothesized ODE exists, we can represent the entire space the low-dimensional dynamic.
F Embedding higher dimensional ODEs
To highlight the ability of our framework to embed higher dimensional ODEs, we train a network to perform the bibit flip flop task [2]. Specifically, we consider the case where = 10. In this case,
30

---

the network receives 10 binary inputs at random times, and produces 10 binary outputs. The network expected maintain it's output the inputs received are the same, or alternatively switch it's output if the opposite input pulse is presented. We successfully embed such task using rank 10 network with 10 neurons. We design our networks such that each neuron is required to keep track of single input pulse, thus requiring 10 neurons. Note, this could be arbitrarily scaled up to any number of imput pulses, by simply scaling up the number of neurons in the network. In Fig. [16 each neuron is presented with random positive or negative input pulses in blue, to which it must respond appropriately. Our method generates the required output as seen via the activity (x (x)) of 4 example neurons from this network. The network simulated for T 50 seconds, sampled dt 0.01. Concretely, this was achieved by rolling out the equivalent high-dimensional network dynamics given by:
x x= + MN d(x) + Bu
(26)
In our case, M, N, B are randomly drawn from standard normal distributions. represents the input pulse train seen blue, and the vector of neural activity x(t) represents the desired output.
10 bit flip flop task
Neuron
Neuron 2
1
Input
Neuron
Activity
Neuron 3
Neuron 4
0
10
20
30
40
50
10
20
30
40
50
Time (seconds)
Time (seconds)
Figure 16: RNN generating appropriate response for the 10 bit flip-flop task. Neuron Activity, 2:() of example neurons is shown.
31


Reviews:
[{"review_id": "0f2qJlte3N", "reviewer": "Reviewer_AJnD", "summary": "The authors propose a training algorithm, inspired on both low-rank RNNs and the Neural Engineering Framework (NEF), that allows several advances:\n- it is significantly faster than backpropagation, both in theory and in practice,\n- it allows to find the smallest RNNs (the ones with the least amount of neurons) able to do a given task, as supported by a sound theory,\n- it can be applied in certain cases to infinitely large networks, building a link to the neural tangent kernel regime,\n- it can be extended into an active learning algorithm for data-efficient learning from dynamical trajectories.\n\nThe authors also bring some theoretical insights by bridging the frameworks of low-rank RNNs and NEF into a unified framework.", "questions": "1) (important) Could the authors clarify exactly how their algorithm differs from NEF technically, if there are differences. I think this would help readers fit the given algorithm within their view of RNN learning algorithms, and clarify those papers contributions.\n2) (minor) lines 99-100, I do not understand if the authors meant to truncate the rank instead.\n3) lines128-129: I do not understand the claim hat the network can only approximate PWL functions, since PWL functions are themselves universal approximators.\n4) (minor suggestion) equations (8) and (10) could be simpler to read if a simbols were assigned to the design matrices in each, but that is a matter of personal taste.\n5) (important) how is the variance of m and I chosen initially and how does it affect learnability? This seems to be overlooked in the text.\n6) (important) the tasks demonstrated here are always a little too simple for this venue. The absence of a truly multi-step, or higher-dimensional, or contextual tasks could raise doubts about the applicability of the method in more realistic settings. Please consider this comment in relationship with question 5) as it seems warranted that any basis will fare well in the simple tasks demonstrated, but things might not be so simple in more complex settings.\n7) Could the authors clarify exactly how optimisation of the covariance in section 6 relates to the absence of details on the choice of m and I in the intial setting? In particular, it seems to me that reference [3] does insist on non-trivial and non-standard normal choices for the basis vectors, so the claim as such might be misleading, although the method proposed does very convincingly find good parameters. In particular, this relates to the limitations of a given chosen basis: can a basis made of independent and standard normal coefficients in m and I be sufficient for all tasks? How about a basis of correlated coefficients? These questions are not sufficiently addressed.", "limitations": "Yes, the authors cover limitations well.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Quality: the work is of good quality, with extensive theoretical developments that are constantly backed by sound experiments, and a sufficiently exhaustive literature review. I think most claims are supported although a few points could require more evidence (see questions).\n\nClarity: the paper is very clear and well-organised and can be read in a linear fashion without necessarily referring to the cited papers. There seems to be enough information to reproduce the results independently.\n\nSignificance: by bridging a number of theoretical frameworks about vanilla RNN training, this paper brings new theoretical insights to a very active area at the intersection of computational neuroscience and ML. The contributions are numerous on both the theoretical and practical sides, and most importantly they start building a bridge between the micro and statistical scales of neural networks learning theories (ie. the very few neurons-scale, and the NTK/infinite width-scale) which is one of the major gaps in NN learning theories. On the other hand, the learning algorithm itself is very similar in its implementation to the NEF algorithm, and the novel contributions on that specific aspect could be clarified. Moreover, the authors insist on their algorithms being able to fit a dynamical system, but it is not completely clear whether they think it can be used to learn a task directly, which is something the previous algorithms including FORCE, NEF, low-rank RNNs, aimed to do.\n\nOriginality: by bridging many different aspects of NN learning theory and demonstrating their links in practice, the authors demonstrate a very deep understanding of the field and a vision to unify some of its different frameworks into a sounder theoretical foundation. I don't think I have seen previous work bridging all these different frameworks.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and enthusiastic assessment on the clarity, originality, and significance of our work. We’re pleased that they found our paper clear and well-organized and appreciated both our new theoretical and practical insights. We’re especially encouraged by their recognition of our effort to bridge multiple frameworks in RNN learning theory and to provide new insights supported by sound experiments!\n\nBelow we address specific questions raised:\n1. **Difference from NEF:** We apologize for the lack of clarity on this front. Our presentation is exactly that of NEF in the case of the entire ODE being known. We would like to highlight, although the NEF presentation is indeed low-rank, we haven’t seen any explicit literature highlighting this (to the best of our knowledge). Additionally, we believe our online RLS approach is the first formulation to adapt NEF for the target matching case. While this appears straightforward, we believe that we’re the first to present this. Similarly, our other novel contributions such as finding the smallest RNN (OMP/COMP), linking the framework to the GP formulation and active learning are all novel connections to the NEF methodology. We will include a detailed discussion on this in our final version of the paper.\n2. **lines 99-100:** We meant to highlight a case where a task/target may need more latent dimensions than the output signal. For instance, a 1-d periodic signal would need at least 2-d latents to represent oscillations. We apologize for the confusion, and will simplify our text to more clearly indicate this.\n3. **lines 128-129:** This refers to our discussion on networks with the relu non-linearity. Here, we highlight, similar to tanh networks in the absence of per neuron inputs (or biases), the basis functions are piecewise functions about origin only, and can thus only universally express functions that are not offsetted from the origin (Fig 7 in the appendix). \n4. **Equations 8, 10:** We thank the reviewer for bringing this to our attention, we’ll clean up our notation in the final version of the paper!\n5. **Variance of m,I:** We deeply apologize for the lack of details, which we will include in the final version of the paper. For the initial comparisons in Fig 3 (for our method and BPTT), we drew $m, I$ from $\\mathcal{N}(0, 1)$ and for FORCE we used $\\mathcal{N}(0, \\frac{1}{d})$, where $d=$ number of neurons.\n6. **Tasks presented are too simple:** We agree with the reviewer that the tasks presented in the paper are fairly simple. Our goal was to present simpler toy tasks in the main paper to highlight our theoretical advancements and to develop intuition for our presented results. Below we discuss various additional/more complex tasks that we have included.\n    - **3-d example, contextual tasks:** The reviewer is absolutely correct in highlighting this, as indeed we haven’t considered high-dimensional dynamical systems. However, the primary motivation for our work is to extend the theoretical foundations of the low-rank RNN literature (e.g., Mastrogiuseppe & Ostojic, 2018; Beiran et al., 2021; Dubreuil et al., 2022; Valente et al., 2022) & the “task training” RNN literature (e.g., Mante & Sussillo 2013; Sussillo & Barak 2013; Luo, Kim, et al., bioRxiv 2023), which has focused on training RNNs to perform **decision-making tasks (e.g., appendix Section C of our paper).** Thus, while these tasks aren’t high-dimensional (i.e typically rank 1 or 2) , they're challenging for animals to perfect & perform. Additionally, while the ODEs considered in the main paper are either 1 or 2 dimensional, in **Section G.2 of the appendix, we do show application to a 3-dimensional system (the Lorenz attractor), along with a comparison to FORCE.** \nWe would like to add, moreover, that there is **no theoretical impediment to applying our framework to higher dimensional systems**; the regression-based approach we have described is fully general, and thus equally applicable to systems of higher dimensionality. To illustrate this point, we will apply our method to 10, 20, and 40-dimensional systems in the revised manuscript. In fact, our marginal-likelihood based approach to fitting infinite RNNs is especially advantageous in higher dimensions, where selecting an optimal basis for a particular nonlinear function is even more critical. We feel this will substantially strengthen the paper, illustrating application to dynamical systems much higher-dimensional than those commonly considered in most neuroscience tasks.\n    - **Noisy/Complex Target:** During this rebuttal period we have also conducted **new analyses on a more complex: noisy sum of four sine-waves target signal**. Specifically: we simulated a target signal (T=1000s, dt =0.1s, thus 10,000 samples) of a sum of four sinusoids. We then added independent gaussian noise (N(0,0.05)) at each time step of the trajectory, thus obtaining a noisy target as seen in FORCE. While the output is a 1-d signal, we used the trajectory and two cumulative sums as our target (thus requiring a rank 3 network), for our RLS method. We compared our method to low-rank RNNs trained with BPTT (where all weights were updated), as well as FORCE networks (note: all weights were sampled from $N(0,\\frac{1}{\\sqrt d})$ as referenced by FORCE). To ensure a consistent comparison, we report the training MSE (averaged across 5 seeds) after one epoch (a single pass over the 10,000-step sequence). **Both RLS and FORCE achieve near-perfect performance (RLS has best performance) in a single pass, while BPTT typically requires 5–8 epochs to reach comparable errors.** Here, we report BPTT results after one epoch to highlight the difference in sample efficiency (as we can’t include images of the complete training loss curve).\n\n| Number of Neurons| Method | Train Error (MSE)  | Std Error |\n|----------|----------|----------|----------|\n| 8 | RLS (r=3)  | 0.012 |0.006 | \n| 8 | FORCE  | 0.525 | 0.064 |\n| 8 | BPTT (r=3) | 0.107 | 0.106 |\n| 8 | BPTT (r=4) | 0.087 | 0.0919 |\n| 16 | RLS (r=3)  | 0.012 | 0.006 |\n| 16 | FORCE  | 0.530 | 0.029 |\n| 16 | BPTT (r=3) | 0.047 | 0.048 |\n| 16 | BPTT (r=4) | 0.070 | 0.0768 |\n| 64 | RLS (r=3)  | 0.011 | 0.006 |\n| 64 | FORCE  | 0.275 | 0.056 |\n| 64 | BPTT (r=3) | 0.016 | 0.003 |\n| 64 | BPTT (r=4) | 0.017 | 0.006 |\n| 128 | RLS (r=3)  | 0.011 | 0.006 |\n| 128 | FORCE  | 0.127 | 0.058 |\n| 128 | BPTT (r=3) | 0.016 | 0.003 |\n| 128 | BPTT (r=4) | 0.016 | 0.003 |\n| 256 | RLS (r=3)  | 0.011 | 0.006 |\n| 256 | FORCE  | 0.024 | 0.006 |\n| 256 | BPTT (r=3) | 0.023 | 0.011 |\n| 256 | BPTT (r=4) | 0.019 | 0.009 |\n| 512 | RLS (r=3)  | 0.011 | 0.006 |\n| 512 | FORCE  | 0.014 | 0.002 |\n| 512 | BPTT (r=3) | 0.022 | 0.013 |\n| 512 | BPTT (r=4) | 0.020 | 0.009 |\n| 1024 | RLS (r=3)  | 0.011 | 0.006 |\n| 1024 | FORCE  | 0.012 | 0.001 |\n| 1024 | BPTT (r=3) | 0.019 | 0.004 |\n| 1024 | BPTT (r=4) | 0.017 | 0.003 |\n- **Comparison with optimized v un-optimized basis:** Finally, we have also included **new analysis** on RNNs trained with standard normal vs optimized basis. We reframe this problem as a target matching problem. Specifically, we used a set of four starting locations given by $7.92, 4.72, 2.3, -7.36$. We then rolled out the dynamics (as per the ODE in Fig 5) using an Euler integration for a total of 1000 time-steps (with dt=0.1). Each trajectory converged to a specific stable fixed point. We then compared low-rank RNNs (rank=1) trained via our RLS method and those trained with BPTT (for 5 epochs) on standard normal initialized basis and the optimal basis distributions found via our GP framework. Below we summarise the Train MSE. **Note, in both finite size RNNs trained with our RLS method and those trained with BPTT, the optimized basis results in lower train errors.**\n\n| Number of Neurons| Method | Train MSE |\n|----------|----------|----------|\n| 2 | RLS  |  0.847|\n| 2 | RLS (opt)  | 0.125 |\n| 2 | BPTT  | 6.519 |\n| 2 | BPTT  (opt)  | 0.211|\n| 5 | RLS  | 0.021 |\n| 5 | RLS (opt)  | 0.001 |\n| 5 | BPTT  | 0.508 |\n| 5 | BPTT  (opt)  | 0.199|\n| 10 | RLS  | 0.041 |\n| 10 | RLS (opt)  |0.001 |\n| 10 | BPTT  | 0.250 |\n| 10 | BPTT  (opt)  |0.187 |\n| 50 | RLS  | 1e-3|\n| 50 | RLS (opt)  | 1e-3 |\n| 50 | BPTT  | 0.239 |\n| 50 | BPTT  (opt)  | 0.1907|\n| 100 | RLS  |1e-4 |\n| 100 | RLS (opt)  | 1e-5 |\n| 100 | BPTT  | 0.200 |\n| 100 | BPTT  (opt)  | 0.188|\n\n7. **Optimization of cov, ref [3], limitations of a chosen basis:** We appreciate the reviewer for giving us a chance to address this. The choice of $m,I$ matches the distribution which we originally used in our comparison. As per our understanding, ref [3] and follow up work (Dubreuil, et al. 2022; Valente, A. et al. 2022 ) suggests a joint normal distribution with $m,I$ (not necessarily with $0$ mean as we have used in our implementations and theoretical results). \n    - **Mean $0$ distributions:** While for this version we have worked with $0$ mean distributions (as this cleanly relates to the arcsin kernel), including the mean as a kernel hyper-parameter has been left to future work. We will clarify our claim in the paper, thanks for pointing this out!\n    - **Limitations of a choses basis:** The reviewer brings up a very interesting point on *when* correlations with $m, I$ are needed. While we agree this is an intriguing question (one which we do not answer in this work), we provide an empirical framework to optimize such a basis, to be as effective as possible in approximating an ODE. We do however agree with the reviewer and will include an explicit discussion on this in the final version of our paper! \n\nLastly, we believe that our framework offers the **same capabilities** (with smaller, faster trained and more interpretable networks) as that of FORCE/NEF or BP (in the case of target matching tasks), as demonstrated in our results in Fig 2, appendix G.2 and our new analysis on a noisy target trajectory."}, {"review_id": "Rq3as0n1yj", "reviewer": "Reviewer_YAqz", "summary": "This paper aims to design low-rank recurrent neural networks to approximate specified low-dimensional nonlinear dynamical systems, drawing on ideas from the neural engineering framework. Its main results are: 1. to show that this procedure outperforms other strategies for online training of a low-rank RNN, 2. to use an OMP-based method to find the smallest number of neurons required to approximate a given system, 3. to consider the limit of an infinitely-large network, in which their method becomes kernel regression with an NNGP-type kernel, and 4. to show that the kernel regression formulation allows for active learning.", "questions": "1. The formulation of Section 3 amounts to approximating the flow of interest using random feature ridge regression, and I think the paper could be enhanced by explicitly leveraging what is known about regression for random features. For instance, there is a long line of work on how many random features are required to achieve optimal error rates with respect to the number of observations, notably the work of [Rudi and Rosasco (2017)](https://proceedings.neurips.cc/paper_files/paper/2017/hash/61b1fb3f59e28c67f3925f3c79be81a1-Abstract.html). This literature has seen noticeable developments in recent years with the development of sharp asymptotic descriptions for the error assuming random sampling, see e.g. [Defilippis et al. (2024)](https://proceedings.neurips.cc/paper_files/paper/2024/hash/bd18189308a4c45c7d71ca83acf3deaa-Abstract-Conference.html) or [Atanasov et al. (2024)](https://arxiv.org/abs/2405.00592). Thus, reifying this link could shed new light on the issue of how many neurons are required to approximate a given system. \n\n2. In Section 6, it might be interesting to comment on the fact that one can engineer kernels by clever choice of nonlinearity $\\phi(\\cdot)$. See [Simon et al. (2022)](https://proceedings.mlr.press/v162/simon22a.html). However, this would likely run into issues of biological plausibility. \n\n3. The discussion of active learning is very interesting, but I wish the authors said more about the effect of sampling in $(z,v)$ in the *offline* setting. Under what assumptions on the dynamics function $g(z)$ can you say something concrete about the number of samples required? In the context of question (1), this should be possible if one fixes a probability distribution over $z$. \n\n4. In the comparison with backpropagation through time, can you provide some details on what parameter initialization was used? I am curious if you can see a minimal-rank bias of GD from small initialization that recovers the minimum rank required to solve the task, as has been observed in various recent works from Omri Barak's group and in linear networks by [Bordelon et al. (2025)](https://arxiv.org/abs/2503.18754). \n\n5. I wish the authors devoted more prose to the various reasons why one would want to embed a known ODE into a recurrent neural network, and what new insights this particular method could enable. For instance, do you think one could use this framework to find novel low-rank RNN implementations of commonly-studied computations? Elaborating on these broader motivations might help build a more cohesive narrative for the paper's results.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "On the whole, I think this paper collects a set of interesting, albeit not entirely surprising, results. My main concern is that the authors do not make full use of their mapping of low-rank RNN training to a feedforward network approximation problem, as I detail below and under **Questions**. With the mapping in hand, I don't find some of their claims to be particularly novel given what is known in the feedforward network literature. For instance, the limitations on expressivity without inputs reflect the known limitations on the expressivity of feedforward networks without bias terms (in the ReLU case this is something of a folklore theorem, see e.g. [this blog post](https://james-simon.github.io/blog/the-expressivity-of-shallow-relu-nets/) or [this recent TMLR paper](https://openreview.net/forum?id=Ucpfdn66k2)). More positively, I think the authors could make use of the mapping to obtain stronger results, e.g. they could use known results about random feature models to give better estimates of how many neurons are required. With all of this in mind, I am in favor of accepting this paper, as I think the not-quite-cohesive whole into which it combines mostly existing ideas is nonetheless sufficiently interesting.", "was_revised": true, "final_justification": "I had no major concerns with the initially submitted manuscript, and the authors have adequately replied to my comments and those of the other referees. I maintain my positive assessment.", "rebuttal": "We thank the reviewer for their time and positive assessment of our work. We especially appreciate their constructive suggestions on improving our work. \n\nBelow we address specific questions raised by the reviewer:\n1. **Optimal error rates and number of neurons:** We’re grateful for being pointed to this literature, it certainly relates to our empirical results on OMP/COMP, and active learning. We agree with the reviewer that connecting our results with the theoretical literature on optimal error rates will certainly strengthen the paper, and we will add a discussion (and citations) to this in our final version!\n2. **Clever kernel choices:** Yes certainly! Thanks for pointing this out, we will include a reference to this.\n3. **Effect of sampling (z,I):** Our active learning framework builds on previous results in bayesian optimal/adaptive experimental design. Specifically, we cast the problem in a linear-gaussian setting and use the Bayes A-optimal criteria (i.e maximizer of predictive posterior variance) to choose the next point. Within this setting, we can compute the *minimum number of samples* required to obtain a desired MSE; this bound depends on the dimensionality of the kernel function over the support of our system (i.e., the eigenspectrum of $\\Phi(x)^T \\Phi(x)$ for a grid of ‘x’ values over this support) and the level of (Gaussian) observation noise in each sample (see Chaloner & Verdinelli 1995). We will include an explicit discussion on this in our final version! \n\n    To further validate our empirical results we also performed **additional analysis comparing our active learning method to random sampling** through for the ODE presented in Fig 6. Note, **our method achieves significantly lower error rates much quicker**. Below we summarize our results (the first data point is selected randomly for our method as well for fair comparisons), MSE is computed on average over $5$ random seeds:\n| Number of data-points | Sampling Method | MSE (True ODE vs Fitted) | \n|----------|----------|----------|\n| 1 | Random | 0.814 |\n| 1 | Ours | 0.814|\n| 2 | Random | 0.820 |\n| 2 | Ours | 0.806 |\n| 3 | Random | 0.478 |\n| 3 | Ours | 0.010 |\n| 4 | Random | 0.012 |\n| 4 | Ours | 0.006 |\n| 5 | Random | 0.011 |\n| 5 | Ours | 0.006 |\n| 6 | Random | 0.008 |\n| 6 | Ours | 0.005 |\n| 7 | Random | 0.007 |\n| 7 | Ours | 0.003 |\n| 8 | Random | 0.005 |\n| 8 | Ours | 0.002 |\n| 9 | Random | 0.002 |\n| 9 | Ours | 0.0009 |\n| 10 | Random | 0.002 |\n| 10 | Ours | 0.0008 |\n\n4. **Initial parameter initializations and minimal rank bias:** We deeply apologize for the lack of details, which we will include in the final version of the paper. For the initial comparisons in Fig 3 (for our method and BPTT), we drew $m, I$ from $\\mathcal N(0,1)$. FORCE needed initializations $\\mathcal N(0,\\frac{1}{d})$, where $d =$ number of neurons. For the original set of experiments, the results presented averaged over $3$ seeds of initialization.\n    - **Minimal Rank Bias:** We thank the reviewer for bringing this to our attention. We’re currently working on training full-rank networks to reproduce a noisy target ((which requires a minimum of rank 3 per our framework). We will check how different initializations correlate with the effective rank of the recurrent matrix (similar to Bordelon et al. (2025)). We'll be sure to include this comparison in the final version of the paper! \n\n5. **Discussion on embedding known ODEs:** We apologize for our lack of discussion on this topic and will include a section in our camera-ready version. Here we provide a brief explanation on this. We believe our work provides an alternative way to accelerate neuroscientific research. Specifically, typically such analyses of task dynamics is done in a data-driven manner. That is, task-trained RNNs interpret governing dynamics of specific tasks linked to brain computation (e.g., Kanitscheider & Fiete 2017; Turner, E, et al. 2021; Luo, Kim, et al. 2023). While this approach has numerous advantages, our lack of understanding of how such networks embed solutions leads to degeneracies in the network/task solution space therefore often leading to incorrect/evolving conclusions. It is for this reason, we propose our work as an alternative — “hypothesis-driven” analysis. To elaborate, we propose designing ODEs to solve tasks (or have access to trajectories from such ODEs), which our method then embeds into the low-rank RNN. As highlighted in our work, this reverse first process allows us to make interesting and concrete explanations regarding the network connectivity, the minimal number of neurons, ideal network initializations and a first step to closed loop experiments via active learning."}, {"review_id": "nNmfdScn1K", "reviewer": "Reviewer_XXEJ", "summary": "This paper introduces an alternative perspective on low-rank RNNs. Considering the dynamics in the latent space, you can view low-rank RNNs as approximating an ODE with weighted combinations of $N$ basis functions. By randomly selecting and fixing these basis functions, you can reduce the problem of fitting a low-rank RNN to a regression problem. Online fitting basis weights with RLS converges more quickly and achieves higher performance with fewer neurons than FORCE or BPTT. This perspective also allows for finding the smallest possible RNN given a basis that approximates a given ODE with low error. The paper further shows how an infinitely-wide low-rank RNN is equivalent to a GP and optimizing the GP kernel can help find data-adaptive basis distributions. Finally, the paper show how to select data points for active learning with minimal samples.", "questions": "1. I assume it is still worse, but can you compare the RLS training speed to BPTT on low-rank networks while updating only $\\textbf{n}$ and leaving $\\textbf{m}$ and $\\textbf{I}$ fixed?\n2. How does the RLS-based method fare compared to FORCE and BPTT when observed trajectories have some noise?\n3. Does the RNN that is fit to PCA from the bistable attractor actually qualitatively recover those dynamics?\n4. Can you get similar errors with similar network sizes to OMP/COMP with just normal offline backprop (with learning the bases)? Or backprop on a larger low-rank RNN while encouraging sparsity in the rows of $\\textbf{n}$ (without learning the bases)?\n5. Do optimized basis distributions lead to faster training times with FORCE/BPTT/RLS for finite-size low-rank RNNs?\n\nThese last questions are probably unreasonable to address in the short rebuttal period, but I am simply sharing them anyway out of curiosity:\n1. How does minimal number of neurons scale with latent dimensionality, both for good performance on RLS-based online learning and for OMP/COMP method? (Admittedly I am not sure what kind of systems you should use for these. However, I think William Gilpins's `dysts` package includes some chaotic attractors with as many as 6 dimensions.)\n2. Given a low-rank RNN fit to some neural data (e.g., from Pals et al. 2024 or Valente et al. 2022), does this basis function perspective offer any kind of new insight or interpretability? Is there, for example, interesting structure in the selected basis functions?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\nI found this view on low-rank RNNs to be quite interesting. It leads to several clever methods and ideas that the paper demonstrates are pretty effective, at least in simpler cases. The paper itself is also quite well-written and easy to understand. In all, I think there are a lot of exciting directions that the ideas presented here can be further developed and I am excited to see what comes of them.\n\n**Weaknesses:**\nFrom a neuroscience perspective, I feel that practical applications of the results shown here are somewhat limited. \n* All examples in the paper are fairly idealized and very low-dimensional (mostly 1 or 2D). I imagine the number of bases required grows exponentially with latent dimensionality and that the RLS-based method might struggle when the fixed random bases are insufficient to actually approximate the system. The main examples also do not have any kind of noise from what I can tell. I would think the RLS-based method still works well, but there is no evidence of that.\n* Fitting an RNN to a known latent space ODE feels to me to be a rare use case in computational neuroscience. As the authors note, RNNs are more commonly used to either fit high-dimensional neural observations or perform some (simplification of) a neuroscientific/cognitive task, and in neither case do we know what the latent dynamics should be. The example in the appendix fitting the low-rank RNN to PCA trajectories from noisy neural observations is not very compelling to me. The development of latent dynamics models was motivated in part by the noisiness of single-trial PCA, and I don't see how fitting the RNN really offers anything that the PCA did not already provide.\n* Similarly, finding the minimal-size RNN that implements an ODE is also not a problem I have seen before or see much benefit in solving, and it requires full knowledge of the latent dynamics. I am happy to be convinced otherwise, though.\n* Infinite low-rank RNNs, while a nice theoretical/conceptual framework, are not really practical. Though the example with optimizing the covariance is cool, I would like to see whether the benefits of this initialization transfer to fitting actual finite low-rank RNNs.\n* Though the paper mentions interpretability a few times, I don't think it made a particularly strong case that it offers new methods/insights for neuroscientific interpretability. I think the perspective of approximating latent dynamics with per-neuron basis functions is nice, but I would like to see an example where, for instance, a low-rank RNN fit to neuroscientific data can be analyzed and interpreted in this fashion. \n\nIn all, I think this work is interesting and exciting, but the results and examples are quite idealized and simplified. Of course, these limitations are clearly acknowledged and extending these ideas beyond idealized settings to actual practical application is (rightfully) left to future work, but I would like to see at least some preliminary results pushing in that direction to have more confidence that these ideas are applicable to challenges neuroscientists face with noisy data and complex higher-dimensional dynamics.", "was_revised": true, "final_justification": "The work is interesting, thorough, and well-presented. The additional text and experiments strengthen the results and should make the contributions of the work clearer to readers. However, though the authors reasonably argue for the potential value of their approach, the general applicability of this work to neuroscience remains to be seen. For this reason, I recommend a score of 5.", "rebuttal": "We thank the reviewer for their time and positive assessment on the originality and clarity of our work. We’re extremely grateful that they recognize the value of our contributions and share our excitement about the future directions this line of research can take! \nBelow we address the specific weaknesses and questions raised by the reviewer:\n \n**Limited applicability to neuroscience**: We appreciate the reviewer for giving us a chance to address this, and apologize for our lack of discussion along these lines in the paper. We believe our work provides an alternative way to accelerate neuroscientific research. Specifically, typically such analyses of task dynamics is done in a data-driven manner. That is, task-trained RNNs interpret governing dynamics of specific tasks linked to brain computation (e.g., Kanitscheider & Fiete 2017; Turner, E, et al. 2021; Luo, Kim, et al. 2023). While this approach has numerous advantages, our lack of understanding of how such networks embed solutions leads to degeneracies in the network/task solution space therefore often leading to incorrect/evolving conclusions. It is for this reason, we propose our work as an alternative — “hypothesis-driven” analysis. To elaborate, we propose designing ODEs to solve tasks (or have access to trajectories from such ODEs), which our method then embeds into the low-rank RNN. As highlighted in our work, this reverse first process allows us to make interesting and concrete explanations regarding the network connectivity, the minimal number of neurons, ideal network initializations and a first step to closed loop experiments via active learning. We do however agree with the reviewer, that this is limited to tasks for which experimentalists/computational collaborators can effectively hypothesize such underlying low-dimensional dynamics. We will add a discussion on this in the final version of the paper.\n\nBelow we address specific weaknesses highlighted by the reviewer:\n1. **Fairly idealized, low-dimensional and noise-free examples**: The reviewer is correct in their intuition, as the latent dynamic dimension increases the required basis dimensionality also increases. While the examples in the paper are mostly 1 or 2D; we have included the **3D Lorenz attractor example and comparison of our RLS method with FORCE in the appendix (Section G.2.1). Note, in this case 16 neurons (for our RLS method) suffices for almost perfect reconstruction**. \nTo the reviewers **second point on noise-free examples, we have included new analyses on a noisy target** from a sum of four sinusoids (as done in the original FORCE paper). As can be seen from our analysis, the reviewer’s intuition is correct, i.e our RLS method converges faster and with far fewer neurons ( details under response to Q2 below). \n2. **Fitting low-rank RNNs to known ODEs/ PCA on him dim noisy data**: We thank the reviewer for highlighting this. On fitting low-rank RNNs to known ODEs – we point the reviewer to our response to the main weakness highlighted by them (Limited applicability to neuroscience). Once again, while we agree designing such an ODE for a potential task might not always be possible, we believe that in the cases that it is (such as typical 2AFC decision making/ cognitive tasks studied in this literature), our framework provides a rigorous and insightful solution.\n    - Point on PCA: we agree with the reviewer, our goal was simply to illustrate that our framework can also fit such trajectories if desired. However, we believe embedding the latent dynamics into a trained low-rank RNN offers capabilities beyond what PCA (since it’s a linear dimensionality reduction) alone provides such as understanding the influence of perturbations etc.\n3. **Minimal size RNN:** The reviewer is correct in their assessment, the main advantage of OMP/COMP is from a computational perspective of fitting small networks. \n4. **Infinite low-rank RNNs:** We appreciate the reviewer for highlighting this! To address this comment we’ve conducted **new analysis**, please see our result details under Q5 below. \n5. **Fit to real data:** The reviewer’s assessment is correct. Our goal was to present interpretability from a computational perspective. I.e our method provides a geometric/algebraic description on how each neuron (in the artificial network) is a non-linear basis function of the ODE of interest. We agree there *might be* intriguing neuroscientific interpretations of this framework, however exploring these directions is left for future work.\n\n**Questions:**\n1. **Training time RLS, BPTT (only n):** We simulated a target signal (T=1000s, dt =0.1s, thus 10,000 samples) of a sum of four sinusoids. We then added independent gaussian noise $\\mathcal N(0,0.05)$ at each time step of the trajectory, thus obtaining a noisy target as seen in FORCE. While the output is a 1-d signal, we used the trajectory and two cumulative sums as our target (thus a rank 3 network). We trained low-rank networks with BPTT only allowing ‘n’s weights to be updated. Networks were trained between 5-8 epochs (i.e to achieve similar error as our method) and average running time is reported below across 5 seeds. **Note, our RLS method converges faster.**  \n| Number of Neurons| Method | Time (s) |\n|----------|----------|----------|\n| 8 | RLS (rank 3)  | 0.07 |\n| 8 | BPTT  ( rank 3)  | 105.28 |\n| 8 | BPTT  ( rank 4)  | 83.29 |\n| 16 | RLS (rank 3)  | 0.12 |\n| 16 | BPTT  ( rank 3)  | 70.01 |\n| 16 | BPTT  ( rank 4)  | 59.35 |\n| 64 | RLS (rank 3)  | 0.28 |\n| 64 | BPTT  ( rank 3)  | 138.82 |\n| 64 | BPTT  ( rank 4)  | 128.69 |\n| 128 | RLS (rank 3)  | 0.64 |\n| 128 | BPTT  ( rank 3)  | 153.50 |\n| 128 | BPTT  ( rank 4)  | 28.92 |\n| 256 | RLS (rank 3)  | 1.41 |\n| 256 | BPTT  ( rank 3)  | 131.4 |\n| 256 | BPTT  ( rank 4)  | 140.03 |\n| 512 | RLS (rank 3)  | 8.34 |\n| 512 | BPTT  ( rank 3)  | 130.0 |\n| 512 | BPTT  ( rank 4)  | 130.93 |\n| 1024 | RLS (rank 3)  | 41.21 |\n| 1024 | BPTT  ( rank 3)  | 129.89 |\n| 1024 | BPTT  ( rank 4)  | 128.98 |\n2. **RLS v FORCE v BPTT:** We use the same noisy sum of four sine-waves target discussed above. RNNs are trained with BPTT and all weights are updated. To ensure a consistent comparison, we report the training MSE (averaged across 5 seeds) after **one epoch (a single pass over the 10,000-step sequence)**. Both RLS and FORCE achieve near-perfect performance in a single pass **(note our method has highest performance)**, while BPTT typically requires 5–8 epochs to reach comparable errors. Here, we report BPTT results after one epoch to highlight the difference in time efficiency (as we can’t include images of training loss curves).\n| Number of Neurons| Method | Train Error (MSE)  | Std Error |\n|----------|----------|----------|----------|\n| 8 | RLS (rank 3)  | 0.012 |0.006 | \n| 8 | FORCE  | 0.525 | 0.064 |\n| 8 | BPTT (rank 3) | 0.107 | 0.106 |\n| 8 | BPTT (rank 4) | 0.087 | 0.0919 |\n| 16 | RLS (rank 3)  | 0.012 | 0.006 |\n| 16 | FORCE  | 0.530 | 0.029 |\n| 16 | BPTT (rank 3) | 0.047 | 0.048 |\n| 16 | BPTT (rank 4) | 0.070 | 0.0768 |\n| 64 | RLS (rank 3)  | 0.011 | 0.006 |\n| 64 | FORCE  | 0.275 | 0.056 |\n| 64 | BPTT (rank 3) | 0.016 | 0.003 |\n| 64 | BPTT (rank 4) | 0.017 | 0.006 |\n| 128 | RLS (rank 3)  | 0.011 | 0.006 |\n| 128 | FORCE  | 0.127 | 0.058 |\n| 128 | BPTT (rank 3) | 0.016 | 0.003 |\n| 128 | BPTT (rank 4) | 0.016 | 0.003 |\n| 256 | RLS (rank 3)  | 0.011 | 0.006 |\n| 256 | FORCE  | 0.024 | 0.006 |\n| 256 | BPTT (rank 3) | 0.023 | 0.011 |\n| 256 | BPTT (rank 4) | 0.019 | 0.009 |\n| 512 | RLS (rank 3)  | 0.011 | 0.006 |\n| 512 | FORCE  | 0.014 | 0.002 |\n| 512 | BPTT (rank 3) | 0.022 | 0.013 |\n| 512 | BPTT (rank 4) | 0.020 | 0.009 |\n| 1024 | RLS (rank 3)  | 0.011 | 0.006 |\n| 1024 | FORCE  | 0.012 | 0.001 |\n| 1024 | BPTT (rank 3) | 0.019 | 0.004 |\n| 1024 | BPTT (rank 4) | 0.017 | 0.003 |\n3. **Qualitative recovery:** Yes, we achieve MSE within 10^-3, and the flow-fields when reproduced do also qualitatively match.\n4. **Sparsity of n:** This is an interesting performance comparison which we will work on including in our final version of the paper! Thank you for bringing this to our attention.\n5. **Optimized basis:** As per the reviewer’s suggestion we have also included **new analysis on finite sized low-rank RNNs trained with standard normal vs optimized basis** for the ODE depicted in Fig 5 of our paper. We re-frame this problem as a target matching problem. Specifically, we used a set of four starting locations given by $[7.92, 4.72, 2.3, -7.36]$. We then rolled out the dynamics (as per the ODE in Fig 5) using an Euler integration for a total of 1000 time-steps (with dt=0.1). Each of these trajectories converged to a specific stable fixed point. We then compared low-rank RNNs (rank=1) trained via our RLS method (trained for 1 epoch) and those trained with BPTT (trained for 5 epochs) on standard normal initialized basis and the optimal basis distributions found via our GP framework. Below we summarise the Train MSE. **Note, in both finite size RNNs trained with our RLS and those trained with BPTT, the optimized basis results in lower train errors.**\n| Number of Neurons| Method | Train MSE |\n|----------|----------|----------|\n| 2 | RLS  |  0.847|\n| 2 | RLS (opt)  | 0.125 |\n| 2 | BPTT  | 6.519 |\n| 2 | BPTT  (opt)  | 0.211|\n| 5 | RLS  | 0.021 |\n| 5 | RLS (opt)  | 0.001 |\n| 5 | BPTT  | 0.508 |\n| 5 | BPTT  (opt)  | 0.199|\n| 10 | RLS  | 0.041 |\n| 10 | RLS (opt)  |0.001 |\n| 10 | BPTT  | 0.250 |\n| 10 | BPTT  (opt)  |0.187 |\n| 50 | RLS  | 1e-3|\n| 50 | RLS (opt)  | 1e-3 |\n| 50 | BPTT  | 0.239 |\n| 50 | BPTT  (opt)  | 0.1907|\n| 100 | RLS  |1e-4 |\n| 100 | RLS (opt)  | 1e-5 |\n| 100 | BPTT  | 0.200 |\n| 100 | BPTT  (opt)  | 0.188|\n\n**Additional Questions:**\nWe thank the reviewer for bringing these to our attention. These suggestions seem very interesting! While not in the scope of this rebuttal, we will include these analysis in future work!"}, {"review_id": "hZNFl6D7s0", "reviewer": "Reviewer_bQjY", "summary": "This paper presents a unified framework for efficiently training low-rank recurrent neural networks (RNNs) to approximate target dynamical systems. The development of these interpretable, sample-efficient low-rank RNNs has important implications for machine learning and neuroscience applications.", "questions": "1. The author presents the case of v=1 in Equation (8), but the approximation of g(z) appears to perform poorly. For Iv obtained from a Gaussian distribution, how should \\hat{n} be computed?\n\n2. How to effectively approximate g(z) when the dimensionality of the target dynamics is high? In other words, is this low-rank RNN only applicable to certain low-dimensional dynamical systems?\n\n3. This paper embeds arbitrary ODEs into an RNN network through Equation (7). What specific tasks can this RNN network be used for? If the goal is merely to analyze time-series data, why not directly apply the approximation from Equation (7) to the raw data instead of going through the RNN encoding process?\n\n4. Other questions are detailed in the Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths：**\n\n1. Methods are well-explained with simple examples.\n\n2. Shows advantages over standard methods (FORCE/BP) in speed and network size.\n\n\n**Weaknesses:**\n\n1. Doesn't discuss how methods perform on high-dimensional systems.\n\n2. Only compares to older methods (FORCE/BP), not other newer low-rank RNN techniques.", "was_revised": true, "final_justification": "After reviewing the author's rebuttal, most of my concerns have been addressed. I am pleased to raise my score to 4.", "rebuttal": "We thank the reviewer for their time & valuable feedback on our paper. We especially appreciate their positive remarks on the advantages of our method compared to FORCE/BP and the clarity of our methods. \nBelow, we address specific weaknesses and questions raised by the reviewer, and humbly request the reviewer to consider raising their score if they are satisfied with our responses.\nWeaknesses: \n1. **Performance on high dimensional systems:** The reviewer is absolutely correct in highlighting this, as indeed we have not considered high-dimensional dynamical systems. However, the primary motivation for our work is to extend the theoretical foundations of the low-rank RNN literature (e.g., Mastrogiuseppe & Ostojic, 2018; Beiran et al., 2021; Dubreuil et al., 2022; Valente et al., 2022) & the “task training” RNN literature (e.g., Mante & Sussillo 2013; Sussillo & Barak 2013; Luo, Kim, et al., bioRxiv 2023), which has focused on training RNNs to perform decision-making tasks (e.g., appendix Section C of our paper). Thus, while these tasks aren’t high-dimensional (i.e typically rank 1 or 2) , they're challenging for animals to perfect & perform. Additionally, while the ODEs considered in the main paper are either 1 or 2 dimensional, in Section G.2 of the appendix, we do show application to a 3-dimensional system (the Lorenz attractor), along with a comparison to FORCE. \n\nWe would like to add, moreover, that there is no theoretical impediment to applying our framework to higher dimensional systems; the regression-based approach we have described is fully general, and thus equally applicable to systems of higher dimensionality. To illustrate this point, we will apply our method to 10, 20, and 40-dimensional systems in the revised manuscript. In fact, our marginal-likelihood based approach to fitting infinite RNNs is especially advantageous in higher dimensions, where selecting an optimal basis for a particular nonlinear function is even more critical. We feel this will substantially strengthen the paper, illustrating application to dynamical systems much higher-dimensional than those commonly considered in most neuroscience tasks.\n\n2. **Comparison to older methods (FORCE/BP):** We thank the reviewer for highlighting this. However, to the best of our knowledge, the majority of the literature we build on (referenced in point above) uses these frameworks for training low-rank RNNs, which is why we focus our comparison on them.\n\nQuestions:\n1. **v=1, poor approximation and \\hat{n} computation:** We apologize for the confusion. For v=1, our approximation is nearly perfect (please see Fig 1 panel E,F). The approximation is poor in the case of no inputs (v=0), as highlighted in Fig 1 Panel C,D. Lastly, if Iv is obtained from a Gaussian distribution, the computation remains linear in ‘n’, thus maintaining the same Least Squares solution presented in equation 8.\n2. **Approximation of g(z) in high-dimensional settings:** Thank you for pointing this out. The approximation is achieved in the same way as the examples presented in the paper. We briefly discuss how this approximation is achieved in the following two cases:\n\n    **2.1.  If the complete ODE is known:** In the case of a high-dimensional dynamic, if the complete ODE is known, g(z)’s approximation would simply be the evaluation of the different dimensions of ‘g’. Thus each component of g(z) would be stacked to obtain the complete output. A complete description of this can be seen through our exposition of the 2-d limit cycle in Sec 3.1 (additionally Sec G.2 of the appendix extends this to the 3-d case as well). \n\n      **2.2.  Only trajectories are available (target matching scenario):** Alternatively, in the case of target matching, where trajectories from the ODE (g(z)) are available (eg see Sec 4 of the paper); we can approximate g(z) via finite differencing (Euler method). Thus $g(z) \\approx \\frac{z_t - z_{t-1}}{dt}$, where $dt$ represents the Euler bin.\n\nThus the **framework presented is applicable to any dimensional ODE**, however the corresponding size of the target and design matrices would increase.\n\n3. **Application of method to tasks and broader applicability:** Thank you for this question. The goal of our framework isn’t restricted to analyzing time-series data; rather it is designed to train a low-rank RNN to replicate a target dynamical system. Keeping this view in mind, our framework goes beyond simply approximating g(z), but rather provides an interpretable and efficient way to train low-rank RNNs. \n\n      **3.1.** Specifically as highlighted in the literature (Elia,et al.,2021; Barak 2017; Vyas, Saurabh, et al. 2020), this trained model becomes a proxy for the real system. This allows us to do experiments such as identifying minimal neurons needed, neuron ablations, perturbation analysis etc which might not be possible to carry out in the true biological or physical system (or might be costly/time intensive). Thus, such a proxy would accelerate scientific understanding of such systems. However, such models have typically been seen as black boxes, and we believe our work provides stronger theoretical and practical insights into the internal representations (and learning dynamics) of these networks.\n\n      **3.2.**  Second, we also present an active learning strategy which provides a principled way to identify informative data points, thereby reducing experimental costs."}]

Author rebuttal:

Meta-review: All reviewers agree with an accept recommendation for this paper. The AC does not find any reason to overturn the consensus in the discussion, and thus recommends accept.
Decision: Accept (poster)