Title: Efficient Training of Minimal and Maximal Low-Rank Recurrent Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: low-rank rnns

dynamical systems

computational neuroscience

gaussian processes
Abstract: Low-rank recurrent neural networks (RNNs) provide a powerful framework for characterizing how neural systems solve complex cognitive tasks. However, fitting and interpreting these networks remains an important open problem. In this paper, we develop new methods for efficiently fitting low-rank RNNs in ''teacher-training'' settings.  In particular, we build upon the neural engineering framework (NEF), in which RNNs are viewed as approximating an ordinary differential equation (ODE) of interest using a set of random nonlinear basis functions. This view provides geometric insight into how the choice of neural nonlinearity (e.g. tanh, ReLU) and the distribution of model parameters affects an RNN's representational capacity. We show that this perspective leads to an online training method that achieves higher accuracy with smaller networks than previous methods such as FORCE, and outperform backprop-trained networks of similar size while requiring substantially less training time. We then consider the problem of finding minimal and maximal low-RNNs for approximating a target dynamical system.  We show that a variant of orthogonal matching pursuit (OMP) can be used to find the smallest RNN for a dynamical system of interest. At the other extreme, a dual space formulation allows for efficient fitting of infinite low-rank RNNs, which provide a Gaussian Process (GP) prior over dynamical systems.  We use the resulting GP marginal likelihood to optimize the hyperparameters governing neural activation functions, which leads to improved training performance even for finite RNNs. Finally, we describe active learning methods for low-rank RNNs, which  speed up training through the selection of maximally informative activity patterns.


Peer reviews:
comments: SUMMARY:
This paper presents a unified framework for efficiently training low-rank recurrent neural networks (RNNs) to approximate target dynamical systems. The development of these interpretable, sample-efficient low-rank RNNs has important implications for machine learning and neuroscience applications.

STRENGTHS AND WEAKNESSES:
**Strengths：**

1. Methods are well-explained with simple examples.

2. Shows advantages over standard methods (FORCE/BP) in speed and network size.


**Weaknesses:**

1. Doesn't discuss how methods perform on high-dimensional systems.

2. Only compares to older methods (FORCE/BP), not other newer low-rank RNN techniques.

QUESTIONS:
1. The author presents the case of v=1 in Equation (8), but the approximation of g(z) appears to perform poorly. For Iv obtained from a Gaussian distribution, how should \hat{n} be computed?

2. How to effectively approximate g(z) when the dimensionality of the target dynamics is high? In other words, is this low-rank RNN only applicable to certain low-dimensional dynamical systems?

3. This paper embeds arbitrary ODEs into an RNN network through Equation (7). What specific tasks can this RNN network be used for? If the goal is merely to analyze time-series data, why not directly apply the approximation from Equation (7) to the raw data instead of going through the RNN encoding process?

4. Other questions are detailed in the Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an alternative perspective on low-rank RNNs. Considering the dynamics in the latent space, you can view low-rank RNNs as approximating an ODE with weighted combinations of $N$ basis functions. By randomly selecting and fixing these basis functions, you can reduce the problem of fitting a low-rank RNN to a regression problem. Online fitting basis weights with RLS converges more quickly and achieves higher performance with fewer neurons than FORCE or BPTT. This perspective also allows for finding the smallest possible RNN given a basis that approximates a given ODE with low error. The paper further shows how an infinitely-wide low-rank RNN is equivalent to a GP and optimizing the GP kernel can help find data-adaptive basis distributions. Finally, the paper show how to select data points for active learning with minimal samples.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nI found this view on low-rank RNNs to be quite interesting. It leads to several clever methods and ideas that the paper demonstrates are pretty effective, at least in simpler cases. The paper itself is also quite well-written and easy to understand. In all, I think there are a lot of exciting directions that the ideas presented here can be further developed and I am excited to see what comes of them.\n\n**Weaknesses:**\nFrom a neuroscience perspective, I feel that practical applications of the results shown here are somewhat limited. \n* All examples in the paper are fairly idealized and very low-dimensional (mostly 1 or 2D). I imagine the number of bases required grows exponentially with latent dimensionality and that the RLS-based method might struggle when the fixed random bases are insufficient to actually approximate the system. The main examples also do not have any kind of noise from what I can tell. I would think the RLS-based method still works well, but there is no evidence of that.\n* Fitting an RNN to a known latent space ODE feels to me to be a rare use case in computational neuroscience. As the authors note, RNNs are more commonly used to either fit high-dimensional neural observations or perform some (simplification of) a neuroscientific/cognitive task, and in neither case do we know what the latent dynamics should be. The example in the appendix fitting the low-rank RNN to PCA trajectories from noisy neural observations is not very compelling to me. The development of latent dynamics models was motivated in part by the noisiness of single-trial PCA, and I don't see how fitting the RNN really offers anything that the PCA did not already provide.\n* Similarly, finding the minimal-size RNN that implements an ODE is also not a problem I have seen before or see much benefit in solving, and it requires full knowledge of the latent dynamics. I am happy to be convinced otherwise, though.\n* Infinite low-rank RNNs, while a nice theoretical/conceptual framework, are not really practical. Though the example with optimizing the covariance is cool, I would like to see whether the benefits of this initialization transfer to fitting actual finite low-rank RNNs.\n* Though the paper mentions interpretability a few times, I don't think it made a particularly strong case that it offers new methods/insights for neuroscientific interpretability. I think the perspective of approximating latent dynamics with per-neuron basis functions is nice, but I would like to see an example where, for instance, a low-rank RNN fit to neuroscientific data can be analyzed and interpreted in this fashion. \n\nIn all, I think this work is interesting and exciting, but the results and examples are quite idealized and simplified. Of course, these limitations are clearly acknowledged and extending these ideas beyond idealized settings to actual practical application is (rightfully) left to future work, but I would like to see at least some preliminary results pushing in that direction to have more confidence that these ideas are applicable to challenges neuroscientists face with noisy data and complex higher-dimensional dynamics.\n\nQUESTIONS:\n1. I assume it is still worse, but can you compare the RLS training speed to BPTT on low-rank networks while updating only $\\textbf{n}$ and leaving $\\textbf{m}$ and $\\textbf{I}$ fixed?\n2. How does the RLS-based method fare compared to FORCE and BPTT when observed trajectories have some noise?\n3. Does the RNN that is fit to PCA from the bistable attractor actually qualitatively recover those dynamics?\n4. Can you get similar errors with similar network sizes to OMP/COMP with just normal offline backprop (with learning the bases)? Or backprop on a larger low-rank RNN while encouraging sparsity in the rows of $\\textbf{n}$ (without learning the bases)?\n5. Do optimized basis distributions lead to faster training times with FORCE/BPTT/RLS for finite-size low-rank RNNs?\n\nThese last questions are probably unreasonable to address in the short rebuttal period, but I am simply sharing them anyway out of curiosity:\n1. How does minimal number of neurons scale with latent dimensionality, both for good performance on RLS-based online learning and for OMP/COMP method? (Admittedly I am not sure what kind of systems you should use for these. However, I think William Gilpins's `dysts` package includes some chaotic attractors with as many as 6 dimensions.)\n2. Given a low-rank RNN fit to some neural data (e.g., from Pals et al. 2024 or Valente et al. 2022), does this basis function perspective offer any kind of new insight or interpretability? Is there, for example, interesting structure in the selected basis functions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to design low-rank recurrent neural networks to approximate specified low-dimensional nonlinear dynamical systems, drawing on ideas from the neural engineering framework. Its main results are: 1. to show that this procedure outperforms other strategies for online training of a low-rank RNN, 2. to use an OMP-based method to find the smallest number of neurons required to approximate a given system, 3. to consider the limit of an infinitely-large network, in which their method becomes kernel regression with an NNGP-type kernel, and 4. to show that the kernel regression formulation allows for active learning.\n\nSTRENGTHS AND WEAKNESSES:\nOn the whole, I think this paper collects a set of interesting, albeit not entirely surprising, results. My main concern is that the authors do not make full use of their mapping of low-rank RNN training to a feedforward network approximation problem, as I detail below and under **Questions**. With the mapping in hand, I don't find some of their claims to be particularly novel given what is known in the feedforward network literature. For instance, the limitations on expressivity without inputs reflect the known limitations on the expressivity of feedforward networks without bias terms (in the ReLU case this is something of a folklore theorem, see e.g. [this blog post](https://james-simon.github.io/blog/the-expressivity-of-shallow-relu-nets/) or [this recent TMLR paper](https://openreview.net/forum?id=Ucpfdn66k2)). More positively, I think the authors could make use of the mapping to obtain stronger results, e.g. they could use known results about random feature models to give better estimates of how many neurons are required. With all of this in mind, I am in favor of accepting this paper, as I think the not-quite-cohesive whole into which it combines mostly existing ideas is nonetheless sufficiently interesting.\n\nQUESTIONS:\n1. The formulation of Section 3 amounts to approximating the flow of interest using random feature ridge regression, and I think the paper could be enhanced by explicitly leveraging what is known about regression for random features. For instance, there is a long line of work on how many random features are required to achieve optimal error rates with respect to the number of observations, notably the work of [Rudi and Rosasco (2017)](https://proceedings.neurips.cc/paper_files/paper/2017/hash/61b1fb3f59e28c67f3925f3c79be81a1-Abstract.html). This literature has seen noticeable developments in recent years with the development of sharp asymptotic descriptions for the error assuming random sampling, see e.g. [Defilippis et al. (2024)](https://proceedings.neurips.cc/paper_files/paper/2024/hash/bd18189308a4c45c7d71ca83acf3deaa-Abstract-Conference.html) or [Atanasov et al. (2024)](https://arxiv.org/abs/2405.00592). Thus, reifying this link could shed new light on the issue of how many neurons are required to approximate a given system. \n\n2. In Section 6, it might be interesting to comment on the fact that one can engineer kernels by clever choice of nonlinearity $\\phi(\\cdot)$. See [Simon et al. (2022)](https://proceedings.mlr.press/v162/simon22a.html). However, this would likely run into issues of biological plausibility. \n\n3. The discussion of active learning is very interesting, but I wish the authors said more about the effect of sampling in $(z,v)$ in the *offline* setting. Under what assumptions on the dynamics function $g(z)$ can you say something concrete about the number of samples required? In the context of question (1), this should be possible if one fixes a probability distribution over $z$. \n\n4. In the comparison with backpropagation through time, can you provide some details on what parameter initialization was used? I am curious if you can see a minimal-rank bias of GD from small initialization that recovers the minimum rank required to solve the task, as has been observed in various recent works from Omri Barak's group and in linear networks by [Bordelon et al. (2025)](https://arxiv.org/abs/2503.18754). \n\n5. I wish the authors devoted more prose to the various reasons why one would want to embed a known ODE into a recurrent neural network, and what new insights this particular method could enable. For instance, do you think one could use this framework to find novel low-rank RNN implementations of commonly-studied computations? Elaborating on these broader motivations might help build a more cohesive narrative for the paper's results.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a training algorithm, inspired on both low-rank RNNs and the Neural Engineering Framework (NEF), that allows several advances:\n- it is significantly faster than backpropagation, both in theory and in practice,\n- it allows to find the smallest RNNs (the ones with the least amount of neurons) able to do a given task, as supported by a sound theory,\n- it can be applied in certain cases to infinitely large networks, building a link to the neural tangent kernel regime,\n- it can be extended into an active learning algorithm for data-efficient learning from dynamical trajectories.\n\nThe authors also bring some theoretical insights by bridging the frameworks of low-rank RNNs and NEF into a unified framework.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: the work is of good quality, with extensive theoretical developments that are constantly backed by sound experiments, and a sufficiently exhaustive literature review. I think most claims are supported although a few points could require more evidence (see questions).\n\nClarity: the paper is very clear and well-organised and can be read in a linear fashion without necessarily referring to the cited papers. There seems to be enough information to reproduce the results independently.\n\nSignificance: by bridging a number of theoretical frameworks about vanilla RNN training, this paper brings new theoretical insights to a very active area at the intersection of computational neuroscience and ML. The contributions are numerous on both the theoretical and practical sides, and most importantly they start building a bridge between the micro and statistical scales of neural networks learning theories (ie. the very few neurons-scale, and the NTK/infinite width-scale) which is one of the major gaps in NN learning theories. On the other hand, the learning algorithm itself is very similar in its implementation to the NEF algorithm, and the novel contributions on that specific aspect could be clarified. Moreover, the authors insist on their algorithms being able to fit a dynamical system, but it is not completely clear whether they think it can be used to learn a task directly, which is something the previous algorithms including FORCE, NEF, low-rank RNNs, aimed to do.\n\nOriginality: by bridging many different aspects of NN learning theory and demonstrating their links in practice, the authors demonstrate a very deep understanding of the field and a vision to unify some of its different frameworks into a sounder theoretical foundation. I don't think I have seen previous work bridging all these different frameworks.\n\nQUESTIONS:\n1) (important) Could the authors clarify exactly how their algorithm differs from NEF technically, if there are differences. I think this would help readers fit the given algorithm within their view of RNN learning algorithms, and clarify those papers contributions.\n2) (minor) lines 99-100, I do not understand if the authors meant to truncate the rank instead.\n3) lines128-129: I do not understand the claim hat the network can only approximate PWL functions, since PWL functions are themselves universal approximators.\n4) (minor suggestion) equations (8) and (10) could be simpler to read if a simbols were assigned to the design matrices in each, but that is a matter of personal taste.\n5) (important) how is the variance of m and I chosen initially and how does it affect learnability? This seems to be overlooked in the text.\n6) (important) the tasks demonstrated here are always a little too simple for this venue. The absence of a truly multi-step, or higher-dimensional, or contextual tasks could raise doubts about the applicability of the method in more realistic settings. Please consider this comment in relationship with question 5) as it seems warranted that any basis will fare well in the simple tasks demonstrated, but things might not be so simple in more complex settings.\n7) Could the authors clarify exactly how optimisation of the covariance in section 6 relates to the absence of details on the choice of m and I in the intial setting? In particular, it seems to me that reference [3] does insist on non-trivial and non-standard normal choices for the basis vectors, so the claim as such might be misleading, although the method proposed does very convincingly find good parameters. In particular, this relates to the limitations of a given chosen basis: can a basis made of independent and standard normal coefficients in m and I be sufficient for all tasks? How about a basis of correlated coefficients? These questions are not sufficiently addressed.\n\nLIMITATIONS:\nYes, the authors cover limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}