Title: Learning Dynamics of RNNs in Closed-Loop Environments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Recurrent Neural Networks

Closed-Loop

Learning Theory

Dynamical Systems

Neuroscience
Abstract: Recurrent neural networks (RNNs) trained on neuroscience-inspired tasks offer powerful models of brain computation. However, typical training paradigms rely on open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop a mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- or open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast to open-loop ones, are governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the agent-environment interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the learning dynamics of recurrent neural networks in closed-loop scenarios and compares it to the corresponding ones in open-loop scenarios. It finds that they significantly differ, with open-loop dynamics failing to properly solve the closed-loop problem mid-training. The paper uses a simplified model and task to thoroughly investigate these dynamics theoretically, as well as provides empirical evidence that the findings holds on a (slightly) more complicated task.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper is **insightful** and **well-written**. It is **well connected** to existing work and nicely motivates why the closed-loop setting is worthwhile studying. The theoretical analysis of the learning dynamics, while specific to a particular task and simple network, is well executed and **provides non-trivial insights on interesting learning dynamics**.\n\nDespite the limited scope of the analysis (which is needed to make progress and thus totally fine for me), I find the main weakness of the paper to be that it is **overly technical**. The big picture and the technical details are well done, but an in-between level which summarizesinterprets/justifies their results in a more intuitive way.\n\nQUESTIONS:\nFrom my understanding, the authors use a network trained on the task as the teacher for the open-loop setting. Why not training it on the **optimal control policy** (which should be tractable). Given that trained networks do not reach 0 loss, their predictions might contain some strange artifacts that affect learning dynamics. Can the authors comment on that and perform, if possible, a quick experiment to verify that?\n\nCan the authors provide intuition on l. 250 and on why low frequencies are learned faster?\n\nCould the analysis provided by the authors be extended to **higher-dimensional systems**? Additionally, would there be some **properties** about the task at hand under which the authors expect their **results to significantly change**?\n\nThe huge generalization gap of open-loop dynamics that the authors report could motivate studies in other domains. For example, LLMs are often trained through imitation learning (ie open loop), but are used in a closed loop scenario. If similar findings hold in this scenario, this would imply that a medium amount of imitation learning can actually be harmful for final performance. Overall, I feel that there are **connections to other areas of ML research**, and discussing some of them could make the paper interesting to a larger audience.\n\nFinally, here are a few recent theory papers that the authors may find interesting and relevant to their paper. [1](https://proceedings.neurips.cc/paper_files/paper/2024/hash/fbb07254ef01868967dc891ea3fa6c13-Abstract-Conference.html) [2](https://arxiv.org/abs/2407.07279) [3](https://openreview.net/forum?id=KGOcrIWYnx) Some recent papers on learning dynamics of linear RNNs, from a more deep learning perspective than the literature the authors cite. [Here](https://journals.aps.org/prx/abstract/10.1103/PhysRevX.15.021051) is another theory paper on learning dynamics in a (simplified) RL scenario.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes recurrent neural networks (RNNs) trained to control a dynamical system in a closed-loop environment where the RNN dynamics influence the external state dynamics. The authors show that the learning trajectories of RNNs trained with gradient descent differ substantially between open-loop and closed loop environments, with open loop training leading to large overfitting peaks. The authors provide simulations in nonlinear RNNs and a three-stage analysis of the learning dynamics for linear RNNs trained in closed-loop on a double integrator task $\\ddot x(t) = u(t)$. In the first stage, the RNN learns a \"negative position\" controller where $u(t) \\approx - x(t)$. In the second stage, the model begins to infer velocity after a plateau in the performance. In the final stage, the model refines the scale of the readout weights, leading to convergence. The authors also describe multi-frequency tasks in the closed loop setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper studies an important problem (how neural networks learn to control an agent that interacts with its environment), which has until now evaded theoretical study. \n2. The paper has an in depth analysis of the learning dynamics in three distinct stages and supports their results with simulations. \n3. The authors also extend the analysis to multi-frequency tasks.  \n4. There are many other interesting results in their Appendix which provides comparisons between SGD and Adam, comparisons to optimal control, etc. \n\nWeaknesses:\n\n1. To make theoretical analysis tractable, the authors focus their theoretical efforts on a rather double integrator system. \n2. A few of the mathematical / modeling assumptions seem slightly dubious (see questions), but I am happy to be convinced if I am wrong or misunderstanding. \n\nOverall, this is a novel paper studying an important problem in the theory of RNNs and I advocate acceptance.\n\nQUESTIONS:\n1. Throughout the paper, the authors often try to compare gradient descent dynamics on the parameters, possibly implicitly assuming that they should coincide to gradient descent with respect to the order parameters (such as $\\sigma$ or the eigenvalues of $P$, etc). I am skeptical of this claim. For example, a gradient with respect to weights $W$ will not coincide with a gradient with respect to an eigenvalue of $P$. Could the authors provide some clarification on this point?\n2. I did not understand the motivation to study $L = \\alpha L(t) + (1-\\alpha) L(0)$. What value of $\\alpha$ should correspond to the actual experiments? \n3. In the multi-frequency task, what sets the timescale of learning each distinct frequency? In general, if the model requires generating $m$ distinct frequencies, do the dynamics close in a $m$ dimensional subspace?\n\nLIMITATIONS:\nThe authors do a good job of discussing the limitations of their work in the discussion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze the learning dynamics of a closed-loop RNN performing a double integrator control task. They (mostly) work in the setting of a linear RNN, and perform their analysis based on the spectrum of a simplified version of the resulting coupled linear system. They identify unique dynamics emanating from the coupling with the environment, and distinguish these dynamics from an open-loop model that is trained in a student-teacher setup.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe authors study an important problem, and provide good intuition for the different training stages. I didn't find any major flaws in their analysis, and I am convinced this project has great potential.\n\n## Major Weakness\n1. Studying the setup for a linear RNN, especially in the continuous-time description, seems to provide the opportunity for very explicit analyses, for example, based on the explicit loss functions and gradients, generalizations to a whole class of control problems and the effects of regularization. While the authors provide valuable insights into the specific setups they chose, it seems that their analysis is unfinished.\n\n## Minor Weaknesses\n2. Fig. 2 (b) and (c) are hard to read and understand, because of the contrast, the size and the unexplained arrows.\n3. In line 145 on page 5, the statement about generality seems hard to justify.\n\nQUESTIONS:\n1. In equation (1), the authors chose a very specific problem. How much of their analytic analysis can be extended to a setup with somewhat more general matrices $\\mathbf{A}$, $\\mathbf{B}$, $\\mathbf{C}$ ?\n2. On line 70, pg. 2, the authors assume that $N$ is large. Where is that assumption used?\n3. In fig. 1 (c), why is the initial train loss lower than the initial test loss?\n4. In line 98, when the authors state that they use SGD for training, do they mean BPTT?\n5. How is the gradient clipping taken into account for the analytical analysis?\n6. In fig. 2 (a), can the authors characterize the resonance in the open loop training, and understand the transition to stability?\n7. In line 165 on page 2 on page 5, can the authors support the statement that $u_t \\propto -\\text{pos.}$? Fig. 3 (b) by itself doesn't seem sufficient to support this.\n8. On line 168, how do the authors support the claim that the velocity is not represented in the RNN?\n9. In equation (14), the surrogate loss seems ad hoc. If the authors claim that the gradient induced by this loss is correlated to the actual gradient, can they support this?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}