Learning Dynamics of RNNs in Closed-Loop Environments
Authors: Yoav Ger
Omri Barak
Venue: NeurIPS 2025 poster ()
Keywords: Recurrent Neural Networks
Closed-Loop
Learning Theory
Dynamical Systems
Neuroscience
Primary area: neuroscience_and_cognitive_science
TL;DR: We develop a mathematical theory of closed-loop learning in RNNs, showing how it fundamentally differs from conventional open-loop supervise training.

Abstract: Recurrent neural networks (RNNs) trained on neuroscience-inspired tasks offer powerful models of brain computation. However, typical training paradigms rely on open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop a mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- or open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast to open-loop ones, are governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the agent-environment interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.

Learning Dynamics of RNNs in Closed-Loop Environments
Yoav Ger
Omri Barak omri. barakgbrak@gmill.... com
yoav. Snav.ger@campus. gerecammus.tecchiion ac
Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion, Israel Institute of Technology
Abstract
Recurrent neural networks (RNNs) trained on neuroscience-insprree tasks offer powerful models of brain computation. However, typical training paradigms rely open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast open-loop ones, governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the eevernenviro interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.
Introduction
Recurrent neural networks (RNNs) are widely used to study dynamic processes in both neuroscience and machine learning 222 3 5 Their recurrent architecture, reminiscent of brain circuits, enables them to form internal representations such as memories of past inputs [6] 7.. When coupled with an environment, these memories can support the formation of internal world models that help agents achieve their goals. This capability is critical in real-world settings, which are often ambiguous and only partially observable. However, the conditions under which these models emerge during training remain poorly understood, particularly in closed-loop -eviroomnvonttntttt interactions, where outputs influence subsequent inputs.
Despite the ubiquity of closed-loop interaction in biological learning, most RNN training to date has relied on supervised learning in an open-loop setting, with theoretical work primarily focused on characterizing the properties of the resulting solutions [8] 9 [O 01 13 15). Motivated by recent analytical understanding of feedforward networks (16). several studies have begun to
investigate the learning dynamics themselves, though they remain confined to simplified open-loop
regime [17 [8 20. 21 (22. These approaches typically assume i..d. inputs that are independent of the network's 's outputs, thereby omitting the feedback- environments in which biological agents Iearn. As a result, the learning dynamics that arise in closed-loog settings remain largely unexplored. In particular, it is unclear whether such conditions induce distinct learning trajectories or solution classes compared to conventional open-loop training.
Training agents in environments is typically approached through reinforcement learning (RL), en- compassing methods such as policy gradients and actor-critic algorithms [23 24 25). While RL has
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

achieved remarkable success in complex tasks, the complexity of modern architectures and training pipelines often obscures theoretical understanding of the learning dynamics [26] 27 28 Here, we
focus on a simple, tractable setting where a single recurrent network is trained with policy gradients on tasks that preserve key features of agent-environment learning. These include feedback that induces correlated inputs and partial observability, requiring internal representations.
Using this setting, we develop a mathematical framework for learning in closed-loop environments. We begin by showing that closed-loop and open-loop training produce fundamentally different learning dynamics, even when using identical architectures and converging the same final solution. To investigate this divergence, we focus on the largely understudied dynamics of closed-looy RNNs. Specifically, we show that tracking the eigenvalues of the coupled agent-environne environment system (rather than the RNN alone) is both necessary and sufficient to uncover the structure of the learning process, which unfolds in distinct stages reflected in the spectrum and training loss. Notably, closed-loop learning gives rise to natural tra between two competing objectives: myopic, short-sighted policy improvement and long-term system-level stability. Finally, we demonstrate that similar learning dynamics arise in more complex motor control task, with RNNs progressing through stages similar to those observed in human experiments.
2 Preliminaries
We begin by introducing our framework, comprising the environment (a double integrator task) and the agent (an RNN), which is trained under either closed-loop or open-loop conditions. Throughout, use bold lowercase letters for vectors (c.g.. z), bold uppercase letters for matrices (e.g.. W), and non-bold letters for scalars. The identity matrix is denoted by We define the scalar overlap between two vectors p and q as 0 dpq p q. The spectral radius of matrix is denoted by p(:), and we write (A) for the characteristic polynomial of a matrix with eigenvalues A.
Double integrator task Our task environment is the classic discrete-time double integrator control problem 29 30), described by the following linear state-space model:
B 0 [
Et+1 = Ax; + B Bug where
(I)
Here, 2, = (z It (2)
(1)
E 222 represents the position and velocity of unit mass, while u, E RR is
the control signal applied at each time step to steer the mass toward the target state x= = (0, 0) T.
Note that acceleration is precisely equal to the control input (i.e., ø = uy). Since we are interested in partial observability, our observation matrix satisfies rank CC) < (An. Specifically, only the position is measured,
y/1
C Et,
C =
0]
(2)
so that yr 2((' The goal to generate sequence of control signals that minimizes the average squared Euclidean distance to the target state without expending excessive force,
T
T
min 01r
Σ T
x=
+
3
1=1
where 3 balances the cost of state error against control effort.
RNN model We model our RNN-based agent as large network of N recurrently connected neurons that interact via continuous firing rates. We adopt a discrete-time RNN to align with the discrete nature of the control task. The network state dynamics and output are given by:
h = o0 W he m m)-1,) u++1 = 2 h
where h E RR- is the hidden state of the RNN, db-) denotes the activation function, W RNNNN the recurrent weight matrix, and m, E Nxxi are the input and output weight vectors, respectively. Here, 1++1 is the mass position and #441 the control signal. We also validate our findings continuous-timg time RNN model (Appendix A.I.
2

---

(a) Llosed-Loop
(b)
Closed-Loop Loss Landscape
(c)
Closed-Loop
Bay
Stage 1
Stage 2
Stage 3
10
Train
101
Test.
10
m
Start
Test Loss
10
10
Epoch
10"
10"
Open Loop
Open Loop
Train
End
Toacher
10
nd y
W
Clu. u)
0
10"
Test (closed-lopp)
Train
10"
10
10
10
10
Epoch
Figure I: (a) Schematic of learning setups: In the closed-loop setting (top), the RNN output is fed into the environment, which evolves according to the system dynamics and produces the next imput to the network. In contrast, the open-loop setting (bottom) lacks feedback: student RNN is trained via supervised learning to imitate a pre-trained teacher mapping i.i.d. inputs y to target outputs u. (b) Conceptual illustration of our hypothesis: optimization trajectories for closed-loop (blue) and open-loop (red) RNNs explore different regions of the closed-loop loss landscape (darker colors indicate higher loss). (c) Summary of results: the closed-loop RNN (top) progresses through three distinct learning stages, while the open-loop RNN (bottom) shows a sharp test loss peak. Note that the open-loop train loss (dashed black) drops by roughly one order of magnitude while the closed-loop test loss (red solid) drops by four, highlighting its limited sensitivity to the environment. Both panels use log scale; dashed and solid lines show train and test loss on right and left axes, respectively.
Closed-loop learning setup In this setup, the RNN agent learns through repeated interaction with the environment. At each time step, the environment's 's state æ evolves according to Eq. [] and projection of this state, y, is fed as input to the RNN. From the agent's 's internal state, h. an output is produced, which serves as control signal for the environment, thus closing the feedback loop (Fig. [a top). The initial mass state is sampled uniformly: 2g (1[[-2111))
2 2).. The RNN trained via policy gradients [31 32 to minimize the objective in Eq. [ This process can be interpreted as form of adaptive control 333, where policy learning and system identification occur simultaneously (see Appendix B). While classical control often includes a control penalty, we adopt an unregularized version (34] by setting = O. This simplification does not affect our main findings, though it may influence the final solution the RNN settles on (see Stage and Appendix A.2)
Open-loop learning setup Following the standard teacher-student framework [35] 36). we formu- late open-loop supervised setup in which a student RNN does not interact with the environment but is instead trained to imitate an onput-output mapping defined by a teacher network pre-trained on the task (Fig. Th; bottom). To ensure i.i.d. training data, we inject white noise inputs into the teacher and use its outputs as targets. Formally, the training objective is given by min + Σi= (ū. uz)², where i, and u, denote the student's 's prediction and the teacher's 's target, respectively. Note that the double-integrator is special control task that admits an analytically optimal solution (see LQR in Appendix B). Accordingly, we also supplemented our results using the corresponding LQR controller as teacher (Appendix A.5
Training details and RNN initialization Both the closed-loop and open-loop RNNs share the same architecture and are initialized identically. Each network consists of N 100 neurons, an episode length of T 50, and the hyperbolic tangent activation function, Φ(-) = tanh. The input and output weight vectors were initialized independently from (0. 1/N), ) and the recurrent weight matrix W was initialized from (00 9² /N). where g controls the initial recurrent strength. Training was performed using stochastic gradient descent on W. m, and z, with learning rate of 1 = 10-3 batch size of 100, and gradient clipping (2- (2-norm capped at 1) avoid exploding gradients.
3

---

(a)
Test Loss
(b)
Closed-loop Loss & Stability
(c)
Zoom-
0,04
Peak
0.5
10"
Stage 1
Unstable (asc.)
Stage
60
Toss
Stage
0.0
Start
0.00
100
Stage sige-sugl
Lo00
Stage 3
20
Closed-Loop
0.9
Uustable (real)
0.04
10
Open-Toop
Stable
Eind
Starp 3
10
10
10
10
-0.9
0.0
0.5
-0.04
0.00
Epuch
Figure 2: (a) Closed-loop test loss (reproduced from Fig. [ Despite identical architectures, the two RNNs exhibit distinct learning dynamics under closed-loop and open-loop training. The closed-loop RNN progresses through three distinct learning stages, whereas the open-loop RNN displays a sharp loss peak. Gray and yellow markers indicate the end of Stages 1 and 2, respectively, for the closed-loop RNN. (b) Numerical estimation of the effective RNN gain Ker during training. projected onto the (k1. k2) plane (closed-loop shown in blue; open-loop in red). Although both RNNs begin and end in similar regions, they follow markedly different trajectories across the loss landscape. The background color represents the logarithm of the loss; darker regions correspond higher loss. The solid black curve separates three distinct stability regimes. (c) Zoomed-in view of panel (b), highlighting the divergence point between the trajectories (gray marker). Gray and yellow markers match those in panel (a). Notably, the end of Stage 2 for the closed-loop RNN (yellow marker) coincides with stability phase transition in the (k1, K2) plane.
3 Closed-loop and open-loop RNNs exhibit different learning dynamics
We begin with a simple question: all else is equal, will two identical RNNs-one trained in closed-loop setup and the other in open-loop setup-exhibit the same learning trajectories? We hypothesize they will not, as the open-loop agent lacks feedback from the environment and is therefore blind to the consequences of its actions. As result, producing actions that closely match those of the teacher (and thus reduce loss in the open-loop sense) may be detrimental once the environment introduced (b)..Tb). To test this, train identical nonlinear RNNs (initialized with g = 0.1) under the two paradigms and find that their learning dynamics diverge substantially (Fig. [+). While both RNNs show similar initial improvement, the closed-loop RNN quickly enters plateau phase. In contrast, its open-loop counterpart exhibits sharp peak in closed-loop test loss, indicating breakdown in performance. Although the loss eventually recovers, it is accompanied by only a minor improvement in open-loop training loss, underscoring the setup's insensitivity to control consequences (Fig- Te: bottom). We further verified that this divergence between elosed- and open-loop learning is not specific to the double- intelee- task, but persists across broader class of control problems that generalize (Appendix (A.)).
To formalize this divergence concretely, we build on concepts from control theory. Specifically, we consider the closed-loo) stability of the double integrator system under linear state feedback (37). defined by:
1 0
1
Ma = A BK =
k
(5)
k1
K2
Sweeping over a range of feedhack gains (k1, k2). we simulate system trajectories and compute the total squared distance from the target state. Additionally, by analyzing the eigenvalues of Ma we identify regions in parameter space corresponding to stable and unstable dynamics (Fig. P2p and Appendix B.4). Next, embed the nigh-dimensiooal nonlinear RNN policy into this interpretable 2D space, we sample trajectories from both RNNs during training, recording the full system state ær and the corresponding control output u, (noting that the RNN observes only the position of the mass). We then fit a linear model of the form:
(6) yielding an empirical estimate, which we denote the effective feedback gain, K [k] k2). Note that this is only an approximation. An exact analytical map to (ki, k2) is possible only when the RNN is linear and the dynamics lie in effecttvely wooddimensional subspace (Appendix C.9
~ k2z (2)

---

As shown in Fig. 2p, and consistent with our hypothesis, the two RNNs follow distinct trajectories within this oowcimmnsiooal subspace defined by the closed-loop system. Closer inspection of early training (Fig. 2 reveals three notable observations. First, the divergence between open- and closed- loop learning emerges early, at the end of Stage (Fig. 2 gray marker). Second, the progression from Stage I to Stage 2 of the closed-loop RNN unfolds in Zig-Zag trajectory, reflecting non-monotonic updates in the effective feedback gain. Third, the transition from Stage 2 to Stage 3 in the closed-loop RNN (when the plateau ends) coincides with shift from unstable to stable dynamics in the (k1, k2) plane (Fig. 2 yellow marker).
Before shifting our focus to the closed-loop RNN, we note that the dynamics of learning in the open-loop setup not trivial itself. the network were linear, the open-loop loss would be equivalent to learning a linear filter via gradient descent. This setting was recently analyzed analytically [18], and we reproduce similar results in Appendix A.4 for completeness. Next, to better understand the three observations outlined above, we focus on the less explored case of closed-loop learning.
4 Analytic results of closed-loop learning dynamics
Having shown that closed-loop RNNs exhibit distinct learning dynamics from their open-loop counterparts, now turn our analytical treatment, beginning with three simplifying assumptions. First, we replace the nonlinear activation with a linear one, x6-) = id. Second, consistent with prior work [17 (8), observe that training induces low-rank connectivity, often rank-1 (Appendix (A.3). and thus we set W u with u, v E _Nx1 Third, the input weights m are randomly set and fixed during training. These assumptions simplify analysis without sacrificing generality.
4.1 Coupled closed-loop system derivation
To analyze the interaction of the closed-loop RNN, we adopt unified framework, standard in control theory, that combines the dynamics of both the RNN and the environment. Since both are linear, stack the environment state æ and the RNN hidden state h, into a joint state vector s (zz, h₂) yielding the coupled linear system:
env. dynamics
control ouipul
B
81+1 Paa St: where P
m MCA
W
Linput feedhack
dynamics
The matrix P E (22++))((++N) defines the full closed-loop dynamies, with stability governed by its eigenvalues. For general W, these are found by solving self-consistent characteristic equation (Appendix C.2). In the specific case of rank-1 connectivity (i.c., W = wv )) the hidden state he
remains confined to the subspace spanned by m and and can be expressed as h, = (m) m u. This allows us to formulate a reduced four-dimensional system:
[ I
0
0
(eff)
E R
(eff) 8141
Pes s (eff)
where
Pet
1
0zm
Tzu
(8)
[m)
1
0
0
Tum
Tou.
This effective system is governed by just four scalar order gerameters-overlage denoted by a-with eigenvalues satisfying:
xp(A) 3 (- (-Ovu 2) X2 + (20 J2m 1) (avm 7zm -ara azu Fem)
While P (with W Tu and Per produce identical authin-episode trajectories due to spectral equivalence, their learning dynamics are not similar: gradients applied to the order parameters in Par do not directly correspond to weight updates the full RNN. Nonetheless, we numerically find that learning in the reduced system captures key features of high-dimensional RNN learning, as we show next, stage by stage. Appendix C we provide full derivation of both systems and demonstrate their spectral identity (within-episode).
5

---

(a)
(b)
(c)
(d)
(c)
Stage
Trajectory
cig(P)
Theoretical
Empirical
0.2
16
50-
75
Stmmiation Thecy
Proshtin
Loss
16"
Imag.
0.0
50
0.0
16"
25
-0.2
-50
-0.2
10
10
50
200
0.9
1.0
1.1
-0.3
0.0
0.3
0
Epoch
Time-Step
Roal
am
Epoch
Figure 3: Stage 1 Negative position policy. (a) Training loss rapidly decreases during Stage (b) Example trajectory of mass position under RNN control at the end of Stage 1, exhibiting oscillatory divergence; note, the red dashed line indicates the episode length used for training. (c) This oscillatory instability is explained by inspecting the eigenspectrum of P. which reveals the emergence of a dominant camplex-conjusee
pair during training with p(P) > 1. Arrow indicates trajectory; darker marker denotes final position. (d) The effective loss (dashed), predicted by the order parameter am alone, shows strong agreement with simulation (solid). (e) As predicted by the asymptotic loss landscape in (d), despite initializing many RNNs with different initial overlaps, all networks converge to small and negative Jem by the end of Stage 1.
4.2 Three stages of learning; control, representation, and refinement
Stage 1 Negative position policy In the first stage, the loss rapidly decreases (Fig. 3a) as the RNN adopts a "negative-position" " policy, Ur x -position (Fig. Bp). This behavior resembles a proportional (P) controller (37), where the control signal is directly tied to the instantaneous position error. While this provides immediate corrective feedback, lacks any internal representation of the mass's velocity (i.e., no derivative integral components in PID controllers). Consequently, this policy induces instability in the combined system P, as evidenced by an eigenvalue with magnitude greater than one (Fig. 3 which dominates the loss during this stage.
More specifically, since gradient descent induces only minimal change to W uu during Stage 1, we set it to zero, which simplifies Eq. to to take a quadratic form:
(A() X2 2X + (1 02m
(10)
with cigenvalues given by = VTz, where Jm quantifies the alignment between the input and output vectors. This corresponds two types of instability, as illustrated in Fig. [p: real dominant eigenvalue when 0zm >> 0, and a complex-conjugat enables closed-form approximation of the loss in Stage as a function of 0zm alone, which shows excellent agreement with the empirical loss (Fig. 3t).
pair when 2zm < 0. This analysis
In addition, the analysis reveals an asymmetric loss landscape around omm 0, with a sharper rise for positive overlaps compared to negative ones. This asymmetry drives learning toward small negative values of Jzm during this stage (Fig. (e. Taking the limit T' + Do gives Jmm - For finite T, however, the asymmetric shape of the loss pushes Jzm to small negative values. This can also be understood as the need construct early-episode control at the expense of long- long-term behavior, as we expand next. A full derivation of Stage appears in Appendix C.5
Stage 2 Building world model In Stage the loss enters plateau phase (Fig. 4a), which ends when the RNN learns to steer the mass to the target via a highly underdamped response with sustained oscillations (Fig. 44). This phase concludes when the closed-loop dynamics stabilize, as the dominant eigenvalues of P enter the unit disk (Fig. 4b). Reaching this stable regime requires the RNN infer velocity, hidden variable not directly available from the input. To understand how stability is achieved, we examine how gradient descent updates alter the spectral structure of P.
Unlike Stage 1, the recurrent weights, W uv² T_ now evolve and must be included in the spectral analysis. From Stage 1, we know that the characteristic polynomial admits a complex-conuugat pair of roots, implying that the third root is real. Applying Vieta's formula to xP()) yields:
A112 (20 Jam + + (-oou- 2) As X
(11)
where \ and A denote the complex and real roots of Eq. 9 respectively. To approximate λs. we apply first-order perturbation theory around the non-zero eigenvalue Tvu of the rank-one connectivity

---

(a)
(b)
(c)
(d)
Stage 2
Trajectory
eig(P)
Theoretical
Empirical
0.20
0.25
0.25
T-0
10"
Loss
103
Position
Amm.
Imog.
Emeg
Imng-
105
WWW
1o.13
0.13
T
0.13
10
10
10
10
50
200
0.98
1.00
0.975
000
0.975
Epoch
Time-Seer
Real
Real
Real
Figure 4: Stage 2- Building a world model. (a) Training loss enters a plateau phase. (b) Example trajectory of mass position under RNN control at the end of Stage 2, showing an underdamped response with sustained oscillations that eventually converge to the target (note the time-step on the x-axis). (c) Empirical evolution of one of the dominant complex-conjugate eigenvalues during Stage exhibiting vig-7ag trajectory inward toward the unit disk (dashed). Arrow indicates trajectory; darker marker denotes final position. (d) Theoretical trajectories of the dominant eigenvalue optimizing the surrogate loss across different values: optimizing only short-term loss (a = 0) drives the eigenvalue upward along the imaginary axis; optimizing only long-term loss (a = 1) causes direct descent; intermediate values (0 < 1) yield trajectories consistent with empirical behavior; red marker indicates the initial condition. (e) Empirical validation: initializing the RNN from the same starting point (red dot) and varying episode length T produces trajectories that qualitatively match the theoretical surrogate predictions.
matrix, yielding:
A ~ Teu + (ovw)2 20 Jzm
Ozu vvm
+ 0
(12)
Substituting Eq. 12 into Eq. II yields a closed-form approximation for the dominant eigenvalue norm of P. denoted Loc. Taking powers of this expression serves as a surrogate loss that captures the system's asymptotic behavior as - co. To test whether this quantity alone governs learning, we applied gradient descent to Loe (with respect to the order parameters). We found, however, that the resulting dynamics were inconsistent with simulation: the dominant eigenvalue descends vertically along the imaginary axis (Fig. 41: a 1). deviating from the empirical inward trajectory (Fig. 4p)
This discrepancy be attributed to the fact that although the empirical simulation ultimately minimizes foo, the RNN must also establish short-term control. To capture this transient behavior, introduce second term in the surrogate loss. A natural choice is the loss an early timestep; for t = 2, the second term can be written compactly as (see Appendix C.6 for other values):
L2 = 20 .m + 20 +
(13)
We now detine the full surrogate loss that interpolates between short- and long-horizon terms:
Lsurrogate =a- Loo + (1-a) L2, a [
(14)
This blended objective better reflects the dual goals of the second stage: stabilizing the long-term dynamies and achieving short-term immediate control. As expected, setting a = O. optimizing purely for short-term control, drives the dominant eigenvalue upward along the imaginary axis, in direct opposition to the desired asymptotic behavior (Fig. fit = 0). In contrast, interpolating with 0 < < produces trajectories consistent with empirical observations, where the eigenvalue moves inward toward the unit disk. To further validate this prediction, we initialized multiple RNNs from the same initial condition while varying episode length T: shorter episodes pushed the dominant cigenvalue upward, while longer episodes caused to descend, as predicted by theory (Fig. 4.)
Our theory also explains the two observations described above. First, the zig-zag motion in the empirical trajectory (Fig 4 can be attributed to the competing influences of the two loss components, whose gradient directions in the complex plane are nearly opposed. While such zig-zag patterns are well documented in ill- illconditioned optimization (e.g., due to large Hessian condition numbers [39 40)). here they result from a structured conflict between short- and long-ten objectives, hallmark of closed-loop learning. Notably, this conflict is mitigated by adaptive optimizers such as Adam [41] (see Appendix C.7p. Second, the divergence between open- and closed-loop test loss may arise because the open-loop RNN, at least initially, does not optimize for the long-term loss component, allowing output weights to grow unboundedly. A full derivation of Stage 2 appears in Appendix C.6
7

---

(a)
Stage 3
(b) 3
Trajectory
(c)
eig(P)
(d)
Theoretical & Empirical
10
10"
Position
Loes
10
Imag
Loss
B
10
10a
Simmiation Tussyy
10
10
16
10
10"
Epoch
Time-Step
Real
Epach
Figure 5: Stage 3 Policy refinement. (a) Final drop in training loss marks the onset of Stage 3. (b) Example trajectory of mass position under RNN control the end of Stage 3, exhibits fast and non- oscillatory dynamics. (c) Spectrum of the closed-loop matrix P: the dominant complex-conjugee pair contracts further inward, while A grows, indicating the emergence of a second slow mode. Arrow indicates trajectory; darker marker denotes final position. (d) Training loss during Stage 3 for full nigh-dimensional model (solid) and the dimeninnsional effective model (dashed black), across different regularization strengths / (colored), initialized from identical Stage 2 endpoints. The effective model shows excellent agreement with the RNN learning dynamics throughout Stage 3.
Stage 3 Policy refinement Once the closed-loop matrix P stabilizes, the RNN enters final phase of refinement. Training loss drops again (Fig. 5)), and mass trajectories become fast and non-oscillatory toward the target (Fig-(5)) Notably, the third real eigenvalue A increases and begins to influence behavior (Fig. (8). This marks shift from Stage 2 where |aal < [A1], indicating the emergence of second slow mode alongside the dominant complex pair. Learning dynamics in this stage are reproduced by directly optimizing the order parameters of the effective model. This is illustrated by varying the control regularization (see Appendix AA.2) and comparing the results to the full model. Additional analysis confirms that, despite operating in stable regime of P. Stage 3 learning remains shaped by the same competing objectives as Stage (Appendix C,8).
5 Generalization to multi-frequency tracking tasks
(a)
Tracking Task
Test Loss
(c) Trajectory
(4)
eig(W) (e)
Zoom-In Platears
W
MMM
500
1000
1100
Stare
NO 6MMNN
Tim
Froq, Dependent Learning
Cleed-Toop
Ope-TTop
Nom
3305
10
10
10"
10
10
Epoch
Time- Step
Re
Frequency
Figure 6: (a) Closed-loop RNN trajectory (dashed blue) tracking 2D multi-frequence target (solid
black). (b) Test loss across epochs for closed-loop (blue) and open-loop (red) RNNs. Closed-loop RNN exhibits stepwise drops aligned with acquisition of higher-frequency components (wi Wa
Wy < W4); open-loop shows a loss peak followed by convergence. (c) RNN (solid) and reference (dashed) trajectories for individual frequencies, epochs marked in (b). (d) Eigenvalue spectra of W at the same epochs, revealing frequency-specin with loss decomposed into W/ and w2 (right and left axes), showing a trade-off between controlling learned and acquiring new frequencies. (f) Frequency-depender learning: both RNN (blue) and humans (green) acquire higher frequencies more slowly. RNN acquisition times were computed from the markers in (b), normalized by total training time; human data from 42 (see Appendix D)
adaptations. (e) Zoom in on the first plateau in (b),
To test the generality of our findings beyond the simple double integrator task, we trained RNNs on a two-dimensional tracking task inspired by human motor control studies [42 43 In this task, the RNN must learn to control cursor to follow reference trajectory whose motion is defined by sum of sinusoids with different temporal frequencies (Fig. 64; see Appendix D for full details).
8

---

Consistent with the simpler task, closed-loop RNNs exhibited stage-like learning dynamics; training progressed through discrete plateaus, with each drop the loss corresponding to the acquisition of a component (Fig. [p; colored markers). These transitions were validated by testing the RNN on individual frequency components: Fig. 6 shows that each marked epoch aligns with accurate tracking of newly acquired frequency. while Fig. (6 reveals corresponding adaptations in the spectrum of W (see also Figs. [8 [9 in Appendix D). In contrast, open-loop RNNs displayed a test loss peak before convergence, consistent with the double integrator.
Furthermore, the loss plateaus in closed-loop RNNs reflect a trade-off between maintaining control over previously acquired frequencies and learning new ones, which require changes in W. This is evi- dent in crossover, where performance on wi temporarily declines as w2 is acquired (Fig. (ob). Finally, the frequency-dependée
acquisition times of closed-loop RNNs closely tracked trends observed in human participants performing the same task under mirror-reversal perturbation ( (2 Fig. (
suggesting a shared inductive bias favoring the early acquisition of low-frequency components.
Discussion
Learning in biological agents inherently occurs through closed-loop interactions, where future sensory inputs depend on past outputs (i.e., actions). However, most studies train networks, including RNNs, in open-loop supervised settings that omit this feedback structure. To address this gap, we studied the learning dynamics of RNNs under closed-loop conditions in simplified setting.
Our key finding that closed-loop learning is fundamentally distinct from its open-loop counterpart. Using our analytical framework, we show that closed-loop RNNs must balance competing short- and long-term objectives. This trade-off temporarily stalls progress, producing a training loss plateau that resolves only when changes in W support an internal representation of hidden variables, such as velocity or frequency. Once formed, these representations allow the output weights z to grow, enabling further improvement. In contrast, open-loop RNNs permit unchecked growth of z early in training. yielding initial gains but ultimately leading to a loss peak when evaluated under closed-loop, nutoof-distribution conditions.
In line with prior work on simplicity bias in neural networks, learning often unfolds by first acquiring simple structures before more complex ones 12 16, 45] We observed a similar pattern both in policy (e.g., proportional before PID) and in internal representations (e.g., low before high frequencies). Our closed-loop framework offers novel insight: stage-like learning arises from the need balance short-term policy gains with longer-term demands requiring internal models of the environment. This trade-ofi off is reflected in loss plateaus, which mark distinct stages of learning. Finally, consistent with recent findings open-loop RNNs (i.e., uncorrelated inputs) [18] learning proceeds by first aligning weights to task-relevant directions, followed by scaling. Interestingly, in closed-loop RNNs, this scaling is delayed and emerges only after sufficient internal representation is formed, as indicated by changes in W.
Our work complements recent theoretical advancements aimed at broadening learning theory be- yond traditional supervised paradigms [34] 46 (47 (88 and architectures beyond feedforward net- works 14
50 51 52 Closely related studies have investigated learning dynamics under RL
rules but typically considered weak agent-ent enverommnio coupling or were restricted to feedforward architectures 46 47 Other studies examined implicit biases in networks trained on finear-quadrane regulator (LQR) tasks but largely relied on full-state feedback and used feedforward architectures (34). Recent empirical research on RNNs trained with closed-loop feedback highlights their potential to replicate aspects of biological motor learning, yet lacks an analytical characterization of the underlying learning dynamics [53]. Our approach addresses these limitations by examining recurrent networks trained on closed-loop tasks with only partial-state feedback, thereby requiring the RNN to develop internal representations of hidden state variables to achieve task goals.
We acknowledge several limitations. Our analysis employed simplified architectures, linear assump- tions, and direct gradient computations to enable tractable mathematical derivations. Extending our results to scenarios involving sparse rewards or approximate gradients, typical of broader reinforce- ment learning settings, remains an important future direction. Furthermore, the reduced system was accurate only within an episode, and future work could derive effective learning dynamics for these order parameters. Finally, although our findings qualitatively extend to nonlinear RNNs, applying this framework to fully nonlinear agents and environments warrants further investigation.

---

To conclude, our study highlights the importance of incorporating closed-loop dynamics into models of biological learning and demonstrates the usefulness of low-dimensional effective models in understanding RNNs' learning dynamics. We anticipate that this analytical perspective can inform future studies exploring more complex agent-environment interactions.
Acknowledgments
This work was partially supported by the Israel Science Foundation (1442/21) and the Human Frontier Science Program (RGP0017/2021), both awarded OB
Code Availability
All code was implemented in Python using PyTorch [54] and is available on GitHub: https
References
[1] Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology, 46:1-6, 2017.
[2] Saurabh Vyas, Matthew Golub, David Sussillo, and Krishna V Shenoy. Computation through neural population dynamics. Annual review of neuroscience, 43(1):249-275, 2020.
[3] Alex Graves, Abdel-rahman Mohamed. and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645-6649, leee, 2013.
[4] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference machine learning, pages 1310-1318. Pmlr, 2013.
[5] Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670-26698, PMLR, 2023.
[6] John Hopficld. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy sciences, 29(5):2554-25 2558, 1982.
[7] Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744-766, 2022.
[8] David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-649) 2013.
[9] Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome, Context-dependent computation by recurrent dynamics in prefirnnaa cortex. nature, 503(7474):77- 2013.
[10] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances neural information processing systems, 32, 2019.
[11] Adrian Valente, Jonathan W Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank mns. Advances Neural Information Processing Systems, 35:24072-2444 24086, 2022.
[12] Elia Turner and Omri Barak. The simplicity bias in multi-task mns: shared attractors, reuse of dynamics, and geometric representation. Advances in Neural Information Processing Systems, 36:25495-25507, 2023.
[13] Laura N Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 2797)1349-1333, 2024.
10

---

[14] Friedrich Schuessler, Francesca Mastrogiuseppe, Srdjan Ostojic, and Omri Barak. Aligned and oblique dynamics in recurrent neural networks. eLife, 13:RP93060, 2024.
[15] Marino Pagan, Vincent D Tang, Mikio C Aoi, Jonathan W Pillow, Valerio Mante, David Sussillo, and Carlos D Brody. Individual variability of neural computations underlying flexible decisions. Nature, pages 1-3, 2024.
[16] Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 2013.
[17] Friedrich Schuessler, Francesca Mastrogiuseppe. Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in mns. Advances in neural information processing systems, 33:13352-13362. 2020.
[18] Blake Bordelon, Jordan Cotler, Cengiz Pehlevan, and Jacob A Zavatone-Veth Dynamically learning to integrate in recurrent neural networks. arXiv preprint arXiv:2503.18754 2025.
[19] Lukas Fisenmann, Zahra Monfared, Niclas Göring, and Daniel Durstewitz. Bifurcations and loss jumps in rnn training. Advances in Newral Information Processing Systems, 36:70511-70547, 2023.
[20] Owen Marschall and Cristina Savin. Probing learning through the lens of changes in circuit dynamics. bioRxiv, pages 2023-09, 2023.
[21] Alexandra Maria Proca, Clémentine Carla Juliette Dominé, Murray Shanahan, and Pedro AM Mediano. Learning dynamics in linear recurrent neural networks. In Forty-second International Conference on Machine Learning, 2024.
[22] Fatih Dinc, Ege Cirakman, Yiqi Jiang, Mert Yuksekgonul, Mark Schnitzer, and Hide- nori Tanaka. A ghost mechanism: An analytical model of abrupt learning. arXiv preprint arXiv:2501.02:: 2025.
[23] Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint arXiv:1707.08
[25] Jane X Wang. Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex meta-reinforcement learning system. Nature neuroscience, 21(6):860-863 2018.
[26] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-55 533, 2015.
[27] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 162(6419):1140-1144 2018
[28] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, 2025.
[29] Benjamin Recht. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2(1):253-279 2019.
[30] Johannes Friedrich, Siavash Golkar, Shiva Farashahi, Alexander Genkin, Anirvan Sengupta, and Dmitri Chklovskii. Neural optimal feedback control with local learning rules. Advances in Neural Information Processing Systems, 34:16358-16377 2021.
[31] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning, Machine learning, 8:229-256, 1992.

---

[32] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. International conference on machine learning, pages 1467-1476. PMIR, 2018.
[33] Anuradha M Annaswamy. Adaptive control and intersections with reinforcement learning. Anmual Review of Control, Robotics, and Autonomous Systems,
2023.
[34] Noam Razin, Yotam Alexander, Edo Cohen-Karlik, Raja Giryes, Amir Globerson, and Nadav Cohen. Implicit bias of policy gradient in linear quadratic control: Extrapolation to unseen initial states, arXiv preprint arXiv:2402.07873 2024.
[35] Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):0056, 1992.
[36] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual review of condensed matter physics, 11():501-552, 2020.
[37] Eduardo D) Sontag. Mathematical control theory: deterministic finite dimensional systems, volume 6. Springer Science & Business Media, 2013.
[38] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computa- tions in low-rank recurrent neural networks. Neuron, 99(3):609-623, 2018.
[39] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
[40] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1421v91412.6980, 2014.
[42] Chen Avraham and Firas Mawase. Feedback-feedforvve dynamics shape de novo motor learning. bioRxiv, pages 2025-04, 04, 2025.
[43] Christopher S Yang, Noah J Cowan, and Adrian M Haith. De novo learning versus adaptation of continuous control in a manual tracking task. elife, 10:e62578, 2021.
[44] Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. In International Conference on Machine Learning, pages 28843-28863. PMLR, 2023.
[45] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International conference on machine learning, pages 5301-5310. PMLR, 2019.
[46] Nishil Patel, Sebastian Lee, Stefano Sarao Mannelli, Sebastian Goldt, and Andrew Saxe. RI perceptron: Generalization dynamics of policy learning in high dimensions. Physical Review X, 15(2)022151,, 2025.
[47] Blake Bordelon, Paul Masset, Henry Kuo, and Cengiz Pehlevan. Loss dynamics of temporal difference reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[48] Ann Huang, Satpreet Harcharan Singh, and Kanaka Rajan. Learning dynamics and the geometry of neural dynamics in recurrent neural controllers. In Workshop on Interpretable Policies in Reinforcement Learning @ RLC-2024, 2024.
[49] Edo Karlk,Karlik, Itamar Menuhin-Gruman, Raja Giryes, Nadav Cohen, and Amir Globerson. Learnigg low dimensional state spaces with overparameterized recurrent neural nets, arXiv preprint 2rXiv:2210.14000 2022.
[50] Yuhan Helena Liu, Aristide Baratin, Jonathan Cornford, Stefan Mihalas, Eric Shea-Brown and Guillaume Lajoie. How connectivity structure shapes rich and lazy learning neural circuits. ArXiv, pages arXiv-2310, 2024.
12

---

[51] Udith Haputhanthri, Liam Storan, Yiqi Jiang. Tarun Raheja, Adam Shai, Orhun Akengin, Nina Miolane, Mark Schnitzer, Fatih Dine, and Hidenori Tanaka. Understanding and controlling the geometry of memory organization in rans. arXiv preprint arXiv:2502.07256, 2025.
[52] Jakub Smékal, Jimmy TH Smith, Michael Kleinman, Dan Biderman, and Scott W Linder- man. Towards a theory of learning dynamics in deep state space models. arXiv preprint arXiv:240..07279, 2024.
[53] Barbara Feulner, Matthew G Perich, Lee E Miller, Claudia Clopath, and Juan A Gallego. A neural implementation model of feedback-basee motor learning. Nature Communications, 16(1):1805, 2025.
[54] Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 19Xiv:19/..01703, 2019.
[55] Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods. Courier Corporation, 2007.
[56] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. ASME, 1960.
13

---

Appendix overview
The appendix is organized follows:
Section A Supplementary experiments: reproducing our main findings using continuous- time RNN; assessing the impact of regularization (non-zero 9); demonstrating the sponta- neous emergence of low-rank structure in unconstrained RNNs; replicating the open-loop setup results from [88; using an alternative LQR teacher; extension to a more general family of control problems (k-integrator).
Section B
Background on key concepts from control theory.
Section Full derivation of the closed-loop matrix and Pen, including showing their
equivalent characteristic polynomials. Effective losses for Stages and 2, along with
supplemental analyses.
Section D Additional implementation and training details for the tracking task, along with supplementary plots.
14

---

A Supplementary experiments
A.1
Continuous-time RNN
To align with canonical models in neuroscience, we replicate our main findings using a continuous- time RNN (CTRNN), governed by:
T h h + o (Wh + my),
u u= T
The double integrator task likewise formulated in continuous time and discretized using Euler integration:
A/
0
A
B
A
We use fixed step size Al 0.1 and time constant matrix takes the form:
1.
The resulting closed-loop system
A At +
Bz
p=
As shown below, the continuous-time RNN exhibits the same wase learning dynamics (Fig. 7 feedback gain trajectories (Fig. /bc). and spectral transitions (Fig. 7d-ff as in the discrete-time case. Stage transitions are marked on the test loss (a) and aligned with the corresponding marker in effective gain (b-c). Note that in iatinuous-time systems, stability requires all eigenvalues to have negative real parts.
(a)
(b) Clased-loop Loss & Stability (c)
Test Loss
Zoom-In In
0.5
Closed-Locg Loop
Hastable (cer.)
0.08
10
Ope--Topp
Stage
Loss
Stage
0.0
Star:
Stage
60
Loss (logg)
20
Ked
0.00
40
Stage
Unetable [eal))
20
10
-0.9
Stable
-0.08
10"
10
10"
10
-0.9
0.0
0.5
-0.05
0.00
Epoch
kg
(d)
(e)
eig(P)
eig(P)
eig(P)
0.02
Stage
0.024
Stage
0.15
Stage 3
Imag
Imag.
Imag-
0.00
0.020
0.00
-0.02
0.016
-0.15
-0.004
0,000
0.004
0.000
0.002
-0.2
0.0
0.1
Real
Real
Real
Continuous-time RNNs show qualitatively similar dynamics the discrete-time case. (a) Test loss reveals three distinct learning stages in the closed-loop RNN; the open-loop RNN shows sharp loss peak. (b-c) Effective feedback gains trace distinct trajectories for each RNN in the (k1, k2) plane. Markers denote stage transitions and align with panel (a). (d-f) Spectral evolution of the closed-loop matrix follows the same progression: emergence of complex eigenvalues, zig-zag motion toward stability, and eventual stability. Note that in continuous time, stability requires all eigenvalues have strictly negative real parts.
Figure 7:
15

---

A.2 Regularization
In classical control. penalizing control effort helps balance precision and energy use. Following [4] we set 3 in the main text to isolate learning dynamics without regularization. To assess the impact of regularization, we trained networks with increasing control penalties (3 0.1, 1), keeping all other parameters the same.
As shown helow, higher / increases final train loss (Fig. (a) and produces ries (Fig. [p), reflecting reduced control magnitude. Nonetheless, key features of the learning process, including plateaus (Fig. [8)), convergence to stable feedback gains (Fig. [8)), and the emergence of low-rank structure in the recurrent weights (Fig. [i-f), remain intact. Thus, we conclude that regularization shifts the final solution but does not disrupt the dynamics of closed-loop learning.
underdamped trajecto-
A.3 Low-rank structure
To characterize the structure of the trained recurrent weights W (initialized with = 0.1), we plot their eigenspectra at convergence (Fig. Bi-f). Across all 8, the spectra dominated by single outlier with a compressed bulk near zero, consistent with an emergent low-rank structure 17 38 This observation motivates the low rank assumptions used in our analytic derivations.
(a)
(b)
(c)
Test Loss
Trajectory
Closed-loop Loss & Stability
10
1.0
10'
Position
Unelabble (coe.)
Loss
10
3 3=0 0
0.5
3 = 0.1
Unstabie (roal)
10
3 1
-0.7
Stable -0.7
10
10
10
10"
20
40
Epoch
Stee-Step
(d)
(e)
cig(W)
cig( W)
cig( W)
0.4
B= 0
0.4
0=1.1 0.1
0.4
B= 1
Imag.
Imag.
Imag.
0.0
-0.4
-0.4
-0.4
-0.7
-0.3 Real
0.0
-0.7
-0.3 Real
0.0
-0.7
-0.3
0.0
Real
Figure 8: (a) Higher 3 increases final test loss while preserving the three-stage learning dynamics. (b) Trajectories become more underdamped with increasing B. (c) Effective feedback gains (k1, kz) converge to the stable region. (d-f) Eigenspectra of W show consistent low rank structure across B.
16

---

A.4 Open-loop dynamics
In the special case where the RNN has linear activation and is trained in an open- setup, the learning task reduces to imitating linear temporal filter defined by teacher network. As recently analyzed by [ this setup admits a detailed analytical treatment. In particular, when initialized with small weights, gradient descent drives the emergence of pair of outlier eigenvalues in the recurrent matrix W that escape the spectral bulk and converge toward complex-conjugal pair
A = C. iods
where and are determined by the teacher RNN. Although our main text focuses on the less understood closed-loop learning regime, we replicate here the core findings [18] for completeness (Fig. ( Following [18], we use continuous-time RNN (see A.I). In this case, the filter induced by the teacher can be written explicitly as
(t) = Te m,
where W, m, and z denote the teacher's 's recurrent matrix, input vector, and output vector, respectively, We then train student RNNs with increasing initial connectivity strength g to imitate this response, using Gaussian white noise as input, exactly matching the open-loop setup described in the main text.
(n)
Train Loss
(b)
eig W)
(c)
eig( W)
(d)
eig W)
10
0.1
Imng.
Inngg.
Trrag..
10
0.00
10
0.1
0.2
10
10
10"
Real
Real
Real
Epoch
Figure 9: (a) RNN student training loss for initial connectivity strengths g 0.0, 0.1, 0.2. Higher leads to faster convergence. (b-d) Training induces a pair of complex-conjugate outlier eigenvalues in the student RNN's recurrent matrix W, which escape the bulk and converge to the target X, (red dots) specified by the teacher, consistent with [88] Gradient coloring reflects training epoch, with darker colors indicating later times. Side and bottom panels show the real and imaginary projections of the leading eigenvalues over training epochs. Note that the majority of the bulk are effectively unchanged.
17

---

A.5 LQR teacher
The cuuule-integrator task analyzed in the main text admits an analytically optimal Linear Quadratic Regulator (LQR) policy (see Appendix B). To test whether the open-loop learning dynamics depend on the teacher, we replaced the RNN teacher used in the main text with this optimal LQR controller and trained the student RNN while the LQR governed the environment. As shown in Fig. [0 the resulting loss trajectory closely matches that obtained with the RNN teacher, with the open-loop dynamics still showing sharp loss peak midway through training.
Test Loss
10
Loss
10
Closed-Loop
10 -1
Open-Loop (LQR Teacher)
10
10
10
10
Epoch
Figure 10: Open-loop learning with an LQR teacher reproduces the transient loss peak.
18

---

A.6
K-integrator task
To test the generality of our findings, we extended the inteee-nteegrttor setup by introducing the discrete-time k-integrator family, a cascade of integrators in which A, B, C' take more general forms:
E++1 Ag + But,
y+ =
Ca2
Here Ag feck is upper-triangular with ones on both the main diagonal and the first superdiagonal:
(Ax)is = { i j=jor
otherwise,
the input matrix B R assigns each of the last min(k; b) state components a distinct control input:
Bij { i i= k - 1+,
otherwise.
and the observation matrix C' E IR selects the first state components:
{ Ci to otherwise.
i= ie {I, {l.....c).
This formulation generalizes the double-integrator task analyzed in the main text 2, b= 1) to arbitrary order and dimensionality, allowing systematic variation of controllability (b) and observability (c). shown Fig. [T] the qualitative gap between closed- and open-loop learning persists across this broader class of systems. In partially observable settings (c < k), open-loop RNNs exhibit transient loss peaks and sometimes fail converge, whereas closed-loop RNNs show plateau-like trajectory before reaching convergence. Furthermore, under full-state feedback (c k), both training modes behave similarly.
k 2
b
|
k 2
b b=2
2 FD
PO
Closed-loop
Loss
Loss
10"
10
Open-looy loop
10
10
10
10
10
10
10
10"
Epoch (x5)
Epoch
(x5)
18
k
3
b
10
3
b
c=2
10'
k=
3
b=
PO
PO
FO
Loss
Losg
Loss
10
10
10"
10
10
10
10
10
10
10"
10"
10"
10
10
10
Epoch
(x5)
Epoch
(x5)
Epoch (x5)
10
b
10"
4
c
PO
FD
Loss
Loss
Loss
11
10"
10"
10
10
10
10
10"
10
10"
10
10"
10 Epoch (x5)
10
10"
Epoch (x5)
Epoch (x5)
Figure 11: Open- closed-loop learning across the k-integrator family. Training loss for closed- (blue) and open-loop (red) RNNs across varying system order (k), number of actuators (b), and observed variables (c). Partial observability (PO, orange) yields sharp loss peaks and occasional non-convergence in open-loop RNNs, while closed-loop RNNs exhibit plateau dynamics before converging. Under full observability (FO, green), both regimes show nearly identical dynamics.
19

---

B Control theory
In classical control theory, it standard treat control, estimation, feedback, and system identifi- cation as separate and sequential design problems. In contrast, our case, the closed-loop RNN must implicitly and simultaneously solve all of these problems during learning. For completeness, we provide the reader with a brief overview of each component.
B.1 Linear-Quadratie Regulator (LQR)
A fundamental instance of optimal control is the Linear-Quadratee Regulator (LQR) 55 where the system dynamics are linear and the cost function is quadratic. The problem formulated as (for finite horizon)
min MOOT
Q u, Ru))
2++1 A + B u, ut
where H; RR is the state, ur E Ih is the control input, and w, is process noise. The matrices A and B describe the system dynamics, while Q and R are positive semi-definite cost matrices. is well known that A, B, Q. and R are known in advance, then the optimal control law takes the form of linear state feedback:
U --KK It.
where the gain matrix K given by
K B MB) MA and M is given by analytically solving the discrete algebraic Riccati equation,
M Q A A AB B BBB B MA
B.2 Kalman estimation
common extension of LQR, particularly relevant biological learning, arises when the state æ, is not directly observable. Instead, we obtain noisy measurements,
C 2; + V²
where e; is measurement noise. By the separation principle, estimation and control can be designed independently, first estimating æ; and then applying the same state-feedback law described above.A standard choice for estimation is the Kalman filter (56), which updates the state estimate 2; as
c1+1 = A B u + L, (y:- - C.t,)
The Kalman gain Lg chosen to minimize estimation error, with the gain and covariance update equations:
L. A- (CE,CT + E:11 - ( L,)), E, W
Here, E, quantifies uncertainty in the state estimate, V is the measurement noise covariance, and W the process noise covariance. The estimated &, replaces æ, in the LQR control law, known LQG (Linear- Quadratic-Gaustte control, ensuring optimality under partial observability.
B.3 System identification
The results above assume that the system matrices are known in advance, enabling optimal estimation and control. However, in many practical settings-including our case- thes- matrices must be learned from observed data. A common approach is to treat system identification and control design as separate steps. First, the system identification problem is solved to estimate a nominal model, specifically the matrices A and B, from data:
min A.B
|0+11
Ar,
Bu,
Once the nominal model obtained, it can be used to compute the optimal controller K using the LQR formulation.
20

---

B.4 Closed-loop stability of the double integrator
fundamental requirement in feedback control is ensuring closed-loop stability, as instability leads to unbounded state growth. The system is stable if the eigenvalues of the closed-loop matrix.
Ma A BK
le within the unit disk (for discrete time systems), ensuring that 2; asymptotically converges to zero in the absence of process noise. In our task environment, the system dynamics are given by the discrete-time double integrator:
1
0 1
A
B
0 1
For feedback policy of the form Uy 4 where K R1x2 the closed-loop system matrix is given by:
1
Ma A BK
[&]
k1
K2
To determine stability, we compute the eigenvalues of Ma by solving its characteristic polynomial:
XMa (A) = det(AA - Ma) x² - (2 k2) + (1 A2 k1)
which are given by the quadratic formula:
( - k2) +
(22 - 2) 4(1 K2 + kn
Stability of the system requires that both eigenvalues lie strictly within the unit circle, i.e., [x This condition partitions the (k1, k2) parameter space into three distinct regimes (see Fig, 12a). The blue region corresponds to stable dynamics, where both eigenvalues are inside the unit circle. The orange region denotes oscillatory instability, in which the eigenvalues are complex conjugates with magnitude exceeding one. The green region corresponds to real instability, where one real eigenvalue satisfies | 2 1.
1.
(a)
(b)
Stability Regions
Trajectory
100
Slable
50
Position
0
Unstable (osc.)
0
-50
-1
Unstable (real)
-100
-1
1
0
10
20
Time Steps
Figure 12: (a) Stability regions in the (ki, ki)) space. The green region represents real unstable modes, the orange region corresponds to unstable oscillatory dynamics, and the blue region indicates stable dynamics. The red dots mark specific parameter choices for which trajectories are simulated. (b) Example trajectories of the first component of the state (position) corresponding to the red dots in (a), illustrating how different policies impact the closed-loop system behavior.
21

---

Coupled system P
C.1 P matrix derivation
The environment dynamics are given by the linear space equations in Eq. and Eq. 2 and the RNN dynamics follow the update rule Eq. f Assuming the RNN takes linear activation function. we derive below the combined update equations that define the full closed-loop dynamics matrix P. as introduced in the main text (Eq. 7
21+1 Ax; + B Ut = Ax, + (z he)
-Ax + B = h2s
h441 Wh; + m 1/+++
Wh: + m (Cæ++1)
Wh, + C(Ax; + B ha)
=mCAx,
m CB T + W
m A2 Wh,
Or in matrix form:
env. dynamics
control outpat
B
8141 St.
where
CCA
W
Linput feedback
RNN dynamics
C.2 Characteristic polynomial of P for general W
To analyze the stability the closed-loop system, we examine the eigenvalues of P, which satisfy
det (P A12+N)
Because P has distinct block structure, apply Schur's determinant identity:
A
B
det
det(A) -
C
D
Applying this to P - AlN+2 gives:
A
Al2
B
AIN ) det ((=
1 1- 1=) x
1
det(P - 11+NN) = det
m C'A
W
m1
m1
m2
m2
1-> d (1""
1 1-
det
W
A
AIN
mn
mN
The determinant of the x 2 block
det (15 1 1 (1-
With further simplification, this factorization leads to (1-A) 2)² det ( (1- x)²
M= AIN)
Thus, the cigenvalues of P are given by the solutions to
- 2 = or det W (I- mz -INN 0,
yielding self-consistent eigenvalue equation in X, which defines the spectrum of P.
22

---

C.3 Rank-one connectivity
Next, we consider the case where W is low-rank, specifically rank-one, and can be expressed as an outer product:
W = u
where u, v E RN11 this case, the second term of the characteristic equation reduces to:
det u v + XI= m X
Noting that the matrix u
[1-2]² m z
is at most rank-2, we introduce the rank-2 matrices:
U = u m (N x 2),
V
(2 N)
(1 ))2 T
so that:
X
UV uv
(1 X)2
Applying the determinant lemma for rank-2 matrices, we have:
dct(-AIx + +VV = (-1)a det I2 x Vv
1
where:
- Tvu
1 x vvm
L
-I-X- Ozu
XI- azm
Taking the determinant and setting it to zero yields the characteristic polynomial:
(A = x + (-ovu 2) + v2 Jm + 1)A + (Tou Jzm Tvu 2zu Tem)
a cubic equation as defined in the main text Eq. 9 The resulting cubic equation is fully governed by four scalar quantities, pairwise inner products between RNN vectors, and can be viewed as low-dimensional representation of the full P system.
23

---

C.4 Low-dimensional effective system
Under the simplifying assumption that the recurrent weights are rank-1, the hidden state h2 evolves within the subspace spanned by m and ut and can be written as
Kig (mm) m + Kig
h
(m) and Ki
(a) define effective coordinates for the hidden state, which collect into reduced
Here, Kig
four-dimensional system state:
(1)
Zt (2)
(eff)
81
Rig (m)
IR
Kt (u)
The environment evolves according
F++1
(1)
xi
(2)
2+1
(2)
= 24
(2)
Jzm k2)
ozu Ri (u)
and the RNN hidden state according to
h++1 mC Ax; + uo
with CA = [1 1], yielding
(m) Mt-1
(1) z4 +
22)
(u) M++1
(m)
(u)
Ju Rit + ou Kig
Together, these updates define reduced effective system
I
(eff)
Poff 85
(eff)
Pott
0
dzm
zuu
++1
0
0
dpm 0mu
The characteristic polynomial of Pett is
+ X..., (A) 3 + (-ovu - 2) + (2 - Ozm + 1)A +
(ovuTzm Tvu - OzuTum):
which exactly matches the characteristic polynomial of the full matrix P (Eq. This confirms spectral equivalence, in which Par preserves the eigenvalues and thus captures the essential dynamics of the full system.
24

---

C.5
Stage
In the absence of initial connectivity, we have W uv T (equivalent setting g 0), the characteristic equation Eq. reduces to:
Xp = X² 2A + (1 - 22m)
With eigenvalues given by
== 1 1 Vmm
Note that the square root is real if Jzm exhibits complex-conjugate eigenvalues.
2 0, and imaginary if Jzm < O, in which case the system
System stability We found that the system has two non-trivial eigenvalues, given by = 1+ / am
/axm-
Since [1 + za 2 I, the spectral radius of p(P) and therefore the system is unstable. If < 0, the eigenvalues are complex with magnitude exceeding I, resulting in unbounded oscillations. If omm O. the eigenvalues are precisely 1, implying marginal instability, If om > a single eigenvalue is real and greater than 1, leading to exponential divergence. Overall, the system's behavior either grows exponentially or oscillates with increasing amplitude. depending on the sign of Jzm-
Loss approximation
The loss used train the RNN agent given by
T 2r x: 101 T Σ (1t 2 (x,
We seek a closed-form expression for L in terms of 2mm As T grows large, any mode with an eigenvalue A < decays to zero, and in the limit T - co, the influence of the initial state becomes negligible. We consider two cases:
1. Jmm The system has one dominant eigenvalue A with the state evolving as 01 V1 where qı is the corresponding eigenvector and an is the projection coefficient of the initial state. The loss simplifies to
T
1q1
1=
Recognizing this as a geometric series with ratio Af and ignoring the initial state contribution, we obtain
2(++1)
L
X
2. 0 7zm. < 0: The system has complex-conjugate eigenvalues with
|11/ = 1 + [m
Decomposing the coefficient on into real and imaginary parts, æi + ib, the state evolves
as
H; a crosst//ozmmn + bbsintty (axm)))
which expands to
2 a² cos 2 (tv 2mm)) I) b² sin' (t (zm)) + 2ab cos(t (a/m)) sin(t ((zmm)))))))))
Using cos 0 + sin? 1, averaging out oscillatory terms over time, and ignoring initial state contributions, the loss simplifies
|> (((++)
L ~
1 1- 121
25

---

C.6 Stage
In this stage, the recurrent weights W uv evolve, and we must consider the full characteristic cubic equation shown in the main text:
(A = x + (-Ovu - 2) + (20va - Jmm + 1)A + (Tou Jzm - Ovu azu dom
Lo approximation The characteristic equation above takes the general cubic form:
a1 + 6Λ -
where the coefficients a, b, c, d are defined above. Denoting the roots as X1 X2 = X1 (complex conjugates), and À R, this structure allows to establish direct relationship between |1 and A using Vieta's formulas, which relate the polynomial' 's roots to its coefficients. From Vieta's ' first formula, the of the roots is:
+ A a
Since A1 + X2 = 2e(()1), follows that:
A
Re(A1) = 2
Vieta's second formula gives the sum of the products of root pairs:
A1 + dιds + d2d3
Using d1 = 12r and d + A2 Re(1)), we have:
|a/2 + 21g Re(A1)
Substituting the expression for Re(A1), we obtain:
A
|^ 2 2
2
Simplifying gives:
|1
As
Substituting the coefficients in terms of the order parameters reproduces Eq. 11 of the main text. This yields direct relationship between the magnitude of the complex eigenvalues and the real root Ag. expressed entirely through the four scalar overlap parameters.
Perturbation approximation of Ag We now derive closed-form approximation for the real eigenvalue Ag- Since we take W = uv we know that the unperturbed matrix has a single non-zero eigenvalue dou- To quantify how the feedback term modifies this eigenvalue, apply first-order perturbation theory, treating uv as the unperturbed matrix and (I-Ap mz as perturbation. Expanding the characteristic equation XA()) 0 around Twus and using the first-order correction formula:
A ~ Ao Xp(2o) Xx)) with Ao Tuu
We obtain:
A R5 Geu
Ozu Tum
+ 0(22)
(Avw) 2 20
Jam
This expression captures the leading-order shift in the eigenvalue due the low-rank input-output term, with many terms in the numerator and denominator canceling upon substitution.
26

---

Le approximation To approximate the early-time contribution to the loss, we define second objective based on powers of the system matrix P. Specifically, we extract the upper-left x 2 block of P (i.e., A block) and define:
2 L. = Σ (P.)) f,=1
This quantity approximates the expected loss at single early timestep under random initial conditions. To show this, consider the evolution a; p' with target state x 0, the instantaneous loss quadratic in æ;. Since Zo is random and consists of independent, zero-mean components with identical variance, the expected early-time loss simplifies as cross-terms cancel out in expectation. Specifically, after applying Pt, the first component of the state satisfies:
Pfi T + P12 Tu
and the expected squared norm is:
E (x) {2) 2P [(( + (P12 (E( (2))
where we used B[z0
( = by independence at The same argument applies to the second
coordinate, yielding:
x (Pi)) + (P (P (P)))
where the proportionality constant depends on the variance the initial state. For example, for = 1,
P P12,1:2 = ( 1) = == 12+ 0- + 12 =3
fort 2 (Eq. 13
p2
(
2 Jem + 1)
+1)
L2 = 20 sm 20 mm + 6
1:2.1:2
amm
and for t= 33
P3 1:2,1:2
Tem +
o 2am +
Tem Cem:2u su + 20
Om su + 30 +
C3 (aem 1)2 + (a + 3)² + (zu + 20 + (oomozu + 30 + 1)²
We therefore used = 2 in the main text (Eq. 13 it is the lowest order that already includes the order parameters in its expansion. Note that / can also reproduce the results of Fig. 411 but requires different tuning of the a parameters due the distinct scaling of the loss.
27

---

C.7 SGD vs. Adam
The choice of optimizer influences learning dynamics in Stage With SGD, the training loss shows oscillations due to shifts in the output weights between more and less negative values. This reflects trade-off between early- and late-episode control. As illustrated on the right (Fig. [3). the differing curvature along the output weight direction and the recurrent weight direction W causes updates to alternate between prioritizing short-term performance (green region, lower right) and long- term stability (red region, upper left), producing characteristic zig-zag trajectory. This trade-off is also evident empirically: more negative output weights improve short-term performance but cause divergence later in the episode (green in Fig. 14.),
W
Lo > Loc
Z
Lo < Le
whereas less negative weights reduce late-episode instability
at the expense of higher early error (red Fig. 14a). The
resulting zig-zag pattern in the loss (Fig- 14
b) driven by
Figure 13: Illustration of zig-zag learning dynamics.
sharp sign changes the projected gradient V 2 m (Fig. 141).
Furthermore, discussed in the main text, alternating changes
in the overlap m (Fig. (4c) contribute to zig-zag motion of the dominant eigenvalues of P across
the complex plane (Fig. 140).
(b)
(e)
(d)
(e)
Trajectory
Train Loss
Overlap
Gradicnts 2
cig(P)
400
MM
-0.0175
0.5
Position*
200
M
4.15
E
-0.0200
0.0
Imag-
0.15
44.10
0.0225
4.05
-0.5
0.12
50
10 Epoch
10
10
0.98
1,00
Time-Step
Epoch
Epoch
Real
Figure 14: (a) Squared position (2) over time, showing alternating trajectories from epochs with more (red) vs. less (green) negative output weights. (b) Zoom-in to the training loss exhibits oscillations during Stage 2. (c) Overlap m, alternating between and less negative values. (d) Projected gradient V m, showing sign flips across epochs. (e) Dominant eigenvalues of P, shifting across the complex plane in response.
In contrast, Adam uses running averages gradients to stabilize updates, eliminating oscillations and producing smooth convergence during Stage (Fig. 155). Note, however, that while Adam mitigates the loss plateaus and zig-zag patterns observed with SGD, does not eliminate them entirely. Plateaus may still emerge, typically shorter requiring mild parameter tuning. In more complex tasks, find that such plateaus arise more readily, even when using Adam (Appendix D
(s)
(b)
(c)
(d)
(e)
Trajertory
Train Loss
Overlap
Gradients T
0.5
M
0.15
Position?
-0.033
Imag-
3.6
0g
-0.44g
-0.5
0.12
10
0.98
1.00
Time-Step
Epoch
Epach
Epoch
Rtal
Figure 15: Adam smooths updates and eliminates oscillations, yielding stable convergence during Stage 2.
28

---

C.8 Stage
Loss dynamics remain shaped by competing forces. Although Stage 3 operates in stable regime, the final drop in training loss reflects ongoing tradeoffs between early- and late-episode performance, as in Stage 2. This is supported in two ways. First, RNNs initialized within this regime and trained using the open-loop setup exhibit divergent (kı, 12) trajectories compared to the closed-loop path (Fig. 16p). Second, decomposition of the total loss into early and late episode segments reveals that improvements in one often coincide with degradation in the other (Fig- T6p). This confirms that closed-loop learning remains uniquely structured, even after reaching the stable regime.
(a)
(b)
Closed-loop Loss & Stability
Alternating loss shifts (start vs. end)
0.4
0.02
0.0
Unstable (osc.)
Start
and
0.00
-0.02
-0.8
End
Unstable (real)
Stable
-0.8
0.0
0.4
-1.0
-0.5
0.0 AL start
0.5
k1
Figure 16: (a) Numerical estimation of the effective RNN gain K The blue curve shows the closed- loop evolution during Stage 3. Colored curves depict open-loop training from different initialization points sampled along this trajectory. Despite being initialized the same stable region, the open-loop paths diverge from the closed-loop trajectory, highlighting their distinct learning dynamics. (b) Decomposition of the loss into early and late episode segments. Each point compares the change in early loss s and late loss Aedd between consecutive epochs. The negative correlation indicates an alternating pattern: improvements one segment typically coincide with degradation in the other.
29

---

C.9 Eigenvector-based recovery of (k1,
Under the rank-1 assumption, the full closed-loop matrix P E R(2-N) reduces exactly to the 4 x 4 effective matrix Per (see C.4). During Stages 1-2, the spectrum of Per is dominated by Complex-conjugee pair λ1, A1. We denote the corresponding cigenvector by qı E C and separate it into real and imaginary parts:
qR
Re(g)),
qi = Im
These vectors linearly independent and span the slow invariant subspace. Each eigenvector be decomposed into an environment projection and a hidden projection by.
[ (1)
(2)
(m)
KR [
q -- xx (m))) Ki (
QR
FR
where the first two entries represent the environment projection (position and velocity), and the last two correspond to the projection of the RNN hidden state onto m and u. From the second row (i.e.,
velocity update) of the eigenvalue equation Perq. *1q.. we obtain:
(A1 1) R k k (1)
+ k2ak (2)
(A1 1)
k (1)
(2)
This defines linear system for the feedback gains denoted Kexact Let
(1)
(A1 1)
R
so that
Q K Exxaet = u
Kexaci Q
Comparison between the analytic Kexact recovery and the numerical estimates of Ketf (main text Eq. O shows excellent agreement during Stage 2, with deviations emerging Stage (see Fig. [7]. Note that the analytic recovery relies on the eigenvectors of Pen, which do admit a closed-form expression, though is lengthy and not written in compact form.
When eigenvector-base recovery exact? The map- ping P - (k1, is exact when the closed-loop dynam- ics are effectively two-dimensional. This condition is met throughout Stages 1-2, and most clearly during Stage 2, where the dynamics enter an oscillatory phase dominated by conjugate pair. We further know that in this phase, the third eigenvalue can be approximated by Ag R Tuu and remains small with aeal < |A1l.
Stage-2
Stage-3
0.0
-0.2
-0.4
kg
Stage 3: correlated third mode As training progresses into Stage third eigenvalue A moves toward the unit circle and is no longer negligible, breaking the earlier ap- proximation. However, empirically, we found that this mode remains strongly correlated with the dominant con- jugate pair, and the three directions continue to span nearly rank-2 subspace. In this setting, projecting onto (k1, ke) is longer exact but remains a good approxima- tion due to persistent alignment among the leading modes, Therefore, the effective gains Kar can still be estimated numerically (see main text).
Auplytic
-0.6
200
400
Epoch
Figure 17: Numerical estimates of the effective feedhack gains Ket during train- ing (solid lines) compared with eigenvector- based recovery K (dashed black). Agree- ment is exact during Stage 2. and begins to diverge in Stage 3 with the emergence of third slow mode. The red vertical dashed line denotes the transition from Stage 2 to
Higher-order regimes If three more slow modes emerge without strong correlation-e.g.- genuine rank-3 structure-no fixed pair (ki, A2) can capture the closed- loop behavior. The controller then implements a higher-order policy, and projecting onto un -ki=(") kz,,??) discards essential structure.
To conclude, exact recovery holds when the dominant conjugate pair is isolated by a spectral gap (i.e., |Aa < Axl). In Stage 3, approximate recovery remains valid due to residual alignment. Beyond this regime, capturing the full closed-loop dynamics requires additional controller states or alternative representations.
30

---

D Tracking task
We train RNNS track two-dimensional target trajectories composed of the of two sinusoids along each axis. Below, we describe the dynamics of the target position (reference signal r), cursor position 2, and RNN controller h. Following 422 43], the reference signals were generated as:
(t) = di cos( + +1) + ag cos(wat + da). Ty (t) = ai cos(w2t + d2) + a2 cos(w4t t + (4)
where amplitudes a = d2 = 2.31, the phases o were sampled uniformly from [-x, v at each episode, and the angular frequencies
W1 2T 0.10, W2
0.20, W3 2x x 0.30, W1 2 0.40
corresponding to increasing frequencies along each axis. The trajectories had a total duration of 30 seconds, sampled at 10 Hz (nsamples 300), and linear ramp was applied to the amplitude over the first second.
Each trajectory obeyed linear state-space model
0
0
0
0
0
0
w
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
i(1) Rr((), R
0
0
0
0
1
0
0
0
W2
0
000
0
0
0
0
with output
1
0
1
0
00 0
(r(()) Ctr(t), Cn
0
0
0
0
discretized with time step At 0.1 seconds R. exp RAt).
A. with Agent and environment The agent was modeled by a continuous-time RNN (see
N 100 hidden units and linear activation, discretized using Euler's method with step size At 0.1. The RNN had input weight matrix M E x recurrent weight matrix W E R N X and output two-dimensional plant. Each axis (æ, 2) and (y. 3) followed independent continuous- dynamics and was simulated using Euler's method:
weight matrix Z RN Nx2, The environment
At 1
0 A
Lt+1 Axx Bug,
=
=
B
where x, = (z, i. y) is the full plant state and u, IR2 are the accelerations commanded by the cursor, while à and ý represent
agent. Here, x and y denote the horizontal and vertical positions its velocities and the agent is tasked with matching the cursor to the time-varying target (rz (t), (t)). The observation matrix E p2xd projects the full plant state onto observed positions:
C
0 0
At each time step, the RNN received a 4-dimensional input vector consisting of the observed cursor position and the current target position (a=(t), v(t), (z),,,, and output 2-dimensional control vector u, = (uz, uyy) corresponding to the commanded accelerations along each axis.
Closed-loop dynamies The full closed- loop system, combining the reference generator, the plant, and the RNN controller, was captured by the block transition matrix:
Ra
0
P=
A A
(B + B)Z
A AT
A
A)) AIW W
where denotes the block-diagonal sum over the two axes and C maps plant states observed positions.
31

---

Training We trained the RNN controller under two regimes: closed-loop and open-loop. In both cases, training used the Adam optimizer [4] (learning rate 10 batch size 100), minimizing time-averaged loss over T = 30 seconds.
Closed-loop In this regime, the RNN interacted with the environment and was trained minimize the squared position tracking error:
T Σ t=1
Lelosod
T |x(() - rt) ll((t) (t))1
Target trajectories each episode. At each timestep, the RNN received the observed cursor position and the current target (2r, y/s Ta (1),, ry (4)). produced control vector, and influenced the system. The hidden state was reset between episodes. Plant states were clamped to [-10, 10) to prevent divergence. Fixing the input weights M did not affect convergence in this setup.
a) generated as described above, with random phases Φ; ~ (-(,-, resampled
Open-loop Here, a randomly initialized student RNN was trained to match the outputs of a pre-trained teacher (from the final closed-loop epoch). Both networks received the same input (z((), v(t), Ta (()), where (ar(),, v()) evolved under teacher control. The student was optimized minimize the squared difference in outputs:
Lopen T Σ Ustuudem (t) Ueacher (t)
This setup only converged when the teacher drove the plant. Using Gaussian white noise in place of state inputs led to unstable or ineffective student policies. In contrast to the closed-loop case, learning the input weights M was essential for the student match the teacher.
Loss crossover To highlight the trade-off between maintaining control over a previously learned frequency and acquiring a new one, which requires changes in W, tested the RNN from checkpoint taken during the first loss plateau (marked by (e) in Fig. 6b) on individual frequencies. As shown in Fig. Ge. this reveals crossover where performance on w1 temporarily declines as W2 is acquired.
Human data To compute the green Data curve in Fig. 6 we extracted summary data from 42 which reported distinct participants' learning rates across frequencies (lower frequencies acquired more rapidly than higher ones). For the first four frequencies (wi -w:), we identified the first time block at which the mean orthogonal gain (averaged across participants) exceeded 0.3, and normalized this value by the total duration (20 blocks). Orthogonal gain quantifies the component of motor output aligned with the target direction after mirror reversal.
32

---

(a)
Closed-Loop
(b)
Open-Loop
10
W1
W3
W2
W4
Loss
10
10
10
10
10
10"
10
10
10
10
10"
10
Epoch
Epoch
Figure 18: Frequency-speciffc loss decomposition during training the multi- fullli-freuuecyy tracking task. (a) Closed-loop training exhibits stage-like convergence: loss decreases sequentially for each frequency component (4i--4), reflecting their progressive acquísition. (b) In contrast, open-loop training, loss initially rises, then decreases across all frequencies more uniformly, reflecting less structured acquisition.
Train Loss
eig W))
W2
2.5
25
2.5
2.5
Loos
10
0.0
10
2.5
-2.5
0.0
25
-2.5
20
25
20
2.5
2.5
20
2.5
W
25
2.5
Loss
0.0
8
0.0
0.0
2
0.0
10
-2.5
2.55
-55
10
-2.5
0.0
2.5
20
20
2.5
2.5
M
25
2.5
Loos
10
0.0
0.0
0.0
10"
10
-2.6
-2.
25
2255
20
-25
-2.5
-2.5
2.5
2.5
25
2.5
2.5
Loos
10"
Iung
0.0
0.0
10"
2.5
-2.5
+2.5
-2.5
0.0
2.5
20
20
20
20
Epoch
Real
Time
Time
Time
Time
Figure 19: Supplement Fig. illustrating frequency-specifie RNN learning the tracking task. Each row shows (left to right): closed-loop RNN training loss, eigenvalue spectrum of the recurrent weights W. and RNN outputs for isolated frequency components wi evaluated at key epochs during closed-loop training (indicated by vertical colored lines the first column). Colored vertical lines match the markers in Fig. 6b.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that closed-loop learning differs substantially from open- loop learning. This is supported first empirically and further by our theoretical framework. The distinction also generalizes more human-inspired task.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are explicitly addressed in the Discussion section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: We highlight key theoretical assumptions in the main text and provide com- plete and thorough mathematical derivations of our closed-loop learning framework in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Most implementation details, including task specifications, RNN parameters, and hyperparameters, are provided in the main text. The appendix contains all remaining information.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
35

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: The main contribution is theoretical, with full mathematical derivations pro- vided. We provide a detailed description the simulations that enable reproduction; the code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hniips.cc/public//ree
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details either provided in the main text the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This a theory-focused paper; all simulations are included illustrate and validate precise mathematical predictions derived from our framework. We do not run benchmarks or make claims about statistically significant differences between methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All simulations are lightweight and can run on an aff-the-shelt laptop. No specialized hardware extensive computational resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////..aaee..cccc///////aaa
Answer: [Yes]
Justification: All authors have reviewed the NeurlPS Code of Ethics and confirm that the work complies with its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This is a theoretical study that does not involve the development deployment new technology. such, we do not anticipate direct societal impacts, either positive or negative.
37

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is a theoretical study and does not involve the release of data or models with potential for misuse. Therefore, safeguards are not applicable this context.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: Prior work is appropriately cited where relevant.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This is theory-focused paper. No new assets introduced. Code for the illustrative simulations will be released upon acceptance, but it is not the primary contribution.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This a theory-focused paper. No human subjects were involved, and no data were collected.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This a theory-focused paper. No human subjects were involved, and no data were collected.
Guidelines:
39

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing and editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
40


Reviews:
[{"review_id": "GNqQ10WTDd", "reviewer": "Reviewer_4A6e", "summary": "In this paper, the authors analyze the learning dynamics of a closed-loop RNN performing a double integrator control task. They (mostly) work in the setting of a linear RNN, and perform their analysis based on the spectrum of a simplified version of the resulting coupled linear system. They identify unique dynamics emanating from the coupling with the environment, and distinguish these dynamics from an open-loop model that is trained in a student-teacher setup.", "questions": "1. In equation (1), the authors chose a very specific problem. How much of their analytic analysis can be extended to a setup with somewhat more general matrices $\\mathbf{A}$, $\\mathbf{B}$, $\\mathbf{C}$ ?\n2. On line 70, pg. 2, the authors assume that $N$ is large. Where is that assumption used?\n3. In fig. 1 (c), why is the initial train loss lower than the initial test loss?\n4. In line 98, when the authors state that they use SGD for training, do they mean BPTT?\n5. How is the gradient clipping taken into account for the analytical analysis?\n6. In fig. 2 (a), can the authors characterize the resonance in the open loop training, and understand the transition to stability?\n7. In line 165 on page 2 on page 5, can the authors support the statement that $u_t \\propto -\\text{pos.}$? Fig. 3 (b) by itself doesn't seem sufficient to support this.\n8. On line 168, how do the authors support the claim that the velocity is not represented in the RNN?\n9. In equation (14), the surrogate loss seems ad hoc. If the authors claim that the gradient induced by this loss is correlated to the actual gradient, can they support this?", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe authors study an important problem, and provide good intuition for the different training stages. I didn't find any major flaws in their analysis, and I am convinced this project has great potential.\n\n## Major Weakness\n1. Studying the setup for a linear RNN, especially in the continuous-time description, seems to provide the opportunity for very explicit analyses, for example, based on the explicit loss functions and gradients, generalizations to a whole class of control problems and the effects of regularization. While the authors provide valuable insights into the specific setups they chose, it seems that their analysis is unfinished.\n\n## Minor Weaknesses\n2. Fig. 2 (b) and (c) are hard to read and understand, because of the contrast, the size and the unexplained arrows.\n3. In line 145 on page 5, the statement about generality seems hard to justify.", "was_revised": true, "final_justification": "The authors study a problem of considerable relevance to the field of learning dynamics. While their analysis relies on surrogate losses and approximating gradient descent dynamics through order parameters, their results are intuitive and interesting. I (narrowly) recommend acceptance, given the more general analysis the authors have performed in the meantime.", "rebuttal": "We thank the reviewer for appreciating the importance of our work. \n\n**R: Studying the setup for a linear RNN, especially in the continuous-time description, seems to provide the opportunity for very explicit analyses, for example, based on the explicit loss functions and gradients, generalizations to a whole class of control problems and the effects of regularization. While the authors provide valuable insights into the specific setups they chose, it seems that their analysis is unfinished.**  \n\n**A**: We appreciate the reviewer’s acknowledgment of the insights we provide, but we believe some aspects of the critique may stem from misunderstandings, which we are happy to clarify. \n\n1. **\"continuous-time description\"** - Neither our main text nor theoretical analysis involves a continuous-time model. Both the RNN and environment are defined in **discrete time** (lines 71–72: _“We adopt a discrete‑time RNN to align with the discrete nature of the control task.”_). The Appendix (A.1) contains continuous time results as a robustness check and is not central to our analysis. \n\n2.  **\"based on the explicit loss functions and gradients\"** – The loss function is not given explicitly, as it depends on solving the eigenvalues and eigenvectors of a 4D matrix (or equivalently, unfolding the RNN and environment dynamics over $T$ steps). This is precisely why we used a surrogate loss (see also our response to **Reviewer iiBW**).\n\n3.  **\"generalization to broader control problems\"** - See our response to your **Q1** below (k-integrator) and to **Reviewer xBz7** (MiniGrid). \n\n4. **\"effects of regularization\"** - We have a dedicated Appendix (section A.2) on this topic in the original submission. \n\n**R: Fig. 2 (b) and (c) are hard to read and understand, because of the contrast, the size and the unexplained arrows.**  \n\n**A**: We will improve the contrast and size in the final version. As for the arrows, we are unsure which unexplained arrows the reviewer is referring to. **Fig 2b** includes arrows linking \"Start\" to \"End\" to indicate the direction of training over epochs. **Fig. 2c** shows the same arrows as in Fig. 2a. The caption also explains the relationship between the gray and yellow markers across panels. If the reviewer has more specific concerns about figure clarity, we would be happy to address them. \n\n**R: In line 145 on page 5, the statement about generality seems hard to justify.**  \n\n**A**: In line 145, we write: _“these assumptions simplify analysis without sacrificing generality”_, referring to our use of a rank-1 linear RNN with fixed input weights to enable analytical treatment. Crucially, this simplified model still exhibits the same 3-stage learning dynamics as the full-rank nonlinear RNN with all weights trainable. Thus, the core phenomena are preserved, and the claim of generality is supported by our results. We will clarify this more explicitly in the revision. \n\n**R: 1. In equation (1), the authors chose a very specific problem. How much of their analytic analysis can be extended to a setup with somewhat more general matrices ${\\mathbf{A}, \\mathbf{B}, \\mathbf{C}}$?**\n\n **A**: We agree that our starting point is a specific problem. We chose the double-integrator because it is a well-established benchmark in control theory (e.g., Recht, 2019; Friedrich et al., 2021), which allowed us to make concrete progress on the underexplored question of how RNNs learn in closed-loop settings. \n\nTo our knowledge, one relevant prior work is Razin et al., 2024 (cited on line 279), which considers arbitrary $\\mathbf{A}$ and $\\mathbf{B}$ but assumes **full-state feedback**. While other work may exist, this study motivated our focus on the **partially observable** regime ($\\operatorname{rank}(C) < \\operatorname{rank}(A)$; Eq.2, line 65), which is also more aligned with biologically inspired settings. Notably, even in the minimal double-integrator task, this setup reveals a clear divergence between closed- and open-loop learning - the central phenomenon our paper aims to explain. \n\nAs the reviewer requested, we ran **additional extensive experiments** to test the generality of our findings by introducing the _k_-integrator family - a cascade of integrators where $\\mathbf{A}, \\mathbf{B}, \\mathbf{C}$ take more general forms, spanning both partially observable and full-state feedback cases. In total, we evaluated **36 additional configurations** beyond the one in the main text, varying the state dimension ($k \\in {2, 3, 4, 5}$), number of actuators ($b$), and number of observed variables ($c$). Key results are summarized below:\n\n- Partial observability ($c < k$): we recover both key phenomena reported in the original submission- a loss plateau in closed-loop RNNs and a sharp loss peak in open-loop RNNs.  \n\n- Full-state feedback ($c = k$): As expected, the loss curves of closed- and open-loop RNNs are quite similar, with much less pronounced plateaus or sharp peaks.\n\n**R: 2. On line 70, pg. 2, the authors assume that $N$ is large. Where is that assumption used?** \n\n**A**: This is the common scenario in theoretical RNN/neuroscience studies, and hence we used it. The assumption that $N$ is large is not critical to our analysis. We used $N = 100$ (line 94), but also tested $N = 50$, $200$, and $500$, and found the learning dynamics to be invariant to $N$. \n\n**R: 3. In fig. 1c, why is the initial train loss lower than the initial test?**  \n\n**A**: The discrepancy arises because we mistakenly recorded the **test loss at epoch 0** _before_ any updates, while the **train loss at epoch 0** was evaluated _after_ the first update. Thanks for pointing this out, and we will fix this in the final version. \n\n**R: 4. In line 98, when the authors state that they use SGD for training, do they mean BPTT?** \n\n**A:** Backpropagation through time is a method to compute gradients. Stochastic gradient descent is a method to choose samples on which the gradient is computed. These refer to different parts of the training process. We use BPTT as a way to compute gradients for SGD. If the reviewer had a different concern in mind, we’d be happy to clarify further. \n\n**R: 5. How is the gradient clipping taken into account for the analytical analysis?**  \n\n**A**: Gradient clipping is used in practice to prevent numerical overflow from unstable closed-loop policies - a simulation artifact, not a meaningful physical effect. We can also avoid clipping by clamping the environment (i.e., restricting position or velocity from becoming too large). In the multi-frequency task (Section 5), where we test the generality of our findings, no gradient clipping was applied - yet both key phenomena (loss plateau in closed-loop, sharp loss peak in open-loop) still appear clearly (Fig. 6b). In Figs. 4 and 5, we applied gradient clipping to the order parameters for the same reason.\n\n**R: 6. In fig. 2 (a), can the authors characterize the resonance in the open loop training, and understand the transition to stability?**  \n\n**A**: The term **resonance** is not used anywhere in the paper, nor is it conceptually relevant to our analysis, so it's unclear how to respond to this specific point. **Fig. 2a** shows a transient peak in error over time, which may bear some visual resemblance to resonance curves. The transition to stability, however, is thoroughly described as Stage 2 of learning (Section 4.2), with dedicated illustrations in Figs. 2b and 4.\n\n**R: 7. In line 165 on page 2 on page 5, can the authors support the statement that $u(t) \\propto -\\text{pos}$.? Fig. 3 (b) by itself doesn't seem sufficient to support this.**  \n\n**A**: Yes, the statement is supported both numerically and analytically in the paper: \n\n1.  **Numerically**: In Figs. 2 b,c, during Stage 1, the closed-loop RNN converges to effective linear gains with $k^{\\text{eff}}_1 < 0$ and $k^{\\text{eff}}_2 \\approx 0$, which corresponds to a policy of the form $u_t \\propto -\\text{pos}$. \n\n2. **Analytically**: In Section 4.2 (Stage 1), we assume small initial recurrent weights ($W \\approx 0$), so the output is dominated by the input–output overlap $\\sigma_{zm}$. With $\\sigma_{zm} < 0$, the RNN implements $u_t \\propto -\\text{pos}$. \n\n3.  **Validation**: Our analysis shows that in Stage 1, the loss depends only on $\\sigma_{zm}$, enabling an exact loss derivation (Fig. 3d). This is later confirmed in simulations: RNNs with different initial $\\sigma_{zm}$ values all converge to $\\sigma_{zm} < 0$ by the end of Stage 1 (Fig. 3e). \n\n**R: 8. On line 168, how do the authors support the claim that the velocity is not represented in the RNN?** \n\n**A**: This is addressed in Section 4.2 (Stage 2). Briefly, the RNN receives **only position** as input - **velocity is never observed** (lines 64–66). Early in training, with negligible recurrent weights ($W \\approx 0$), the effective gain on velocity is near zero ($k_2^{\\text{eff}} \\approx 0$), and the policy reduces to $u_t \\propto -\\text{pos}$ (see response to Q7 above). Since the RNN lacks both velocity input and internal memory early on, it cannot represent velocity. Only later, as $W$ increases, does the network begin to infer velocity by integrating across time (e.g., via $x_t^{(1)} - x_{t-1}^{(1)}$). \n\n**R: 9. In equation (14), the surrogate loss seems ad hoc. If the authors claim that the gradient induced by this loss is correlated to the actual gradient, can they support this?** \n\n**A**: We agree that the surrogate loss in Eq. (14) is ad hoc. Its value lies in its tractability and ability to provide intuition: it depends only on order parameters (e.g., the $\\sigma$’s) and captures the key tradeoff in closed-loop learning - short-term control ($\\mathcal{L_0}$) vs. long-term stability ($\\mathcal{L_\\infty}$) - modulated by a single tuning parameter $\\alpha$. We also refer the reviewer to our response on the surrogate loss provided to **Reviewer iiBW**."}, {"review_id": "UJZ5y5qWCR", "reviewer": "Reviewer_iiBW", "summary": "This paper analyzes recurrent neural networks (RNNs) trained to control a dynamical system in a closed-loop environment where the RNN dynamics influence the external state dynamics. The authors show that the learning trajectories of RNNs trained with gradient descent differ substantially between open-loop and closed loop environments, with open loop training leading to large overfitting peaks. The authors provide simulations in nonlinear RNNs and a three-stage analysis of the learning dynamics for linear RNNs trained in closed-loop on a double integrator task $\\ddot x(t) = u(t)$. In the first stage, the RNN learns a \"negative position\" controller where $u(t) \\approx - x(t)$. In the second stage, the model begins to infer velocity after a plateau in the performance. In the final stage, the model refines the scale of the readout weights, leading to convergence. The authors also describe multi-frequency tasks in the closed loop setting.", "questions": "1. Throughout the paper, the authors often try to compare gradient descent dynamics on the parameters, possibly implicitly assuming that they should coincide to gradient descent with respect to the order parameters (such as $\\sigma$ or the eigenvalues of $P$, etc). I am skeptical of this claim. For example, a gradient with respect to weights $W$ will not coincide with a gradient with respect to an eigenvalue of $P$. Could the authors provide some clarification on this point?\n2. I did not understand the motivation to study $L = \\alpha L(t) + (1-\\alpha) L(0)$. What value of $\\alpha$ should correspond to the actual experiments? \n3. In the multi-frequency task, what sets the timescale of learning each distinct frequency? In general, if the model requires generating $m$ distinct frequencies, do the dynamics close in a $m$ dimensional subspace?", "limitations": "The authors do a good job of discussing the limitations of their work in the discussion.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. This paper studies an important problem (how neural networks learn to control an agent that interacts with its environment), which has until now evaded theoretical study. \n2. The paper has an in depth analysis of the learning dynamics in three distinct stages and supports their results with simulations. \n3. The authors also extend the analysis to multi-frequency tasks.  \n4. There are many other interesting results in their Appendix which provides comparisons between SGD and Adam, comparisons to optimal control, etc. \n\nWeaknesses:\n\n1. To make theoretical analysis tractable, the authors focus their theoretical efforts on a rather double integrator system. \n2. A few of the mathematical / modeling assumptions seem slightly dubious (see questions), but I am happy to be convinced if I am wrong or misunderstanding. \n\nOverall, this is a novel paper studying an important problem in the theory of RNNs and I advocate acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and for recognizing both the novelty and importance of the problem addressed. We also appreciate your support for the acceptance of our work. Please find below our point-by-point responses to your questions.\n\n**R: Throughout the paper, the authors often try to compare gradient descent dynamics on the parameters, possibly implicitly assuming that they should coincide to gradient descent with respect to the order parameters (such as $\\sigma$ or the eigenvalues of $P$, etc). I am skeptical of this claim. For example, a gradient with respect to weights $W$ will not coincide with a gradient with respect to an eigenvalue of $P$. Could the authors provide some clarification on this point?**\n\n**A**: We fully agree with the reviewer's keen observation that gradients w.r.t. the full RNN weights ($\\boldsymbol{W}$) do not, in general, coincide with gradients taken w.r.t. the low-dimensional order parameters such as $\\sigma$ or the eigenvalues of $P$. This point was noted in the original submission (lines 159–160):  \n\n*\"Gradients applied to the order parameters in $P_{\\text{eff}}$ do not directly correspond to weight updates in the full RNN\".*\n\nOur goal, rather, was to show that the order parameters serve as a compact and interpretable proxy for understanding learning dynamics, as they clearly capture the three learning phases observed in the full model. It turns out that despite this mismatch, the loss dynamics in Stage 3 are almost identical when using the order parameters, and in Stage 2, they differ only in the timing of the plateau.\n\nFurthermore, since the original submission, we have begun to develop exact closed-form expressions for the weight updates in terms of the order parameters alone. Preliminary results indicate that this allows an exact match to the high-D dynamics and has implications beyond the closed-loop setting. We believe that adding these results would make an already large appendix even larger and would neither allow for a full discussion of these implications, nor for a proper review. We are, however, open to different views on this matter.\n\n**R: I did not understand the motivation to study $L = \\alpha L(t) + (1-\\alpha) L(0)$. What value of $\\alpha$ matches the actual experiments?**   \n\n**A**: We introduced the blended loss as a mathematically tractable surrogate that approximates the full loss and provides intuition about the competing objectives the RNN faces in Stage 2. The full loss averages over all timesteps from $0$ to $T$, but it is analytically unwieldy and not easily readable. Our surrogate replaces it with two interpretable components, expressed in terms of the order parameters:\n\n1.  Short-term control - minimizing initial position error, captured by $\\mathcal{L}_0$, which has a simple closed-form (Eq. 13).\n\n2.  Long-term stability - pushing the dominant eigenvalues of $P$ inside the unit disk, captured by $\\mathcal{L}_\\infty$ (Eq. 11), based on the norm of the leading eigenvalue. Although this can be computed exactly at the onset of Stage 2, it later becomes the root of a cubic polynomial and offers limited additional insight.\n\nOptimising either term in isolation does not reproduce the empirical eigenvalue trajectory (Figs. 4 c-e):  \n1. $\\alpha = 0$ (short‑term only) - drives the dominant eigenvalue upward along the imaginary axis.  \n2. $\\alpha = 1$ (long‑term only) - yields a direct vertical descent. Note that gradient descent on the order parameters causes only the imaginary part to decrease, and not towards the origin.\n\nBlending the two $0 < \\alpha < 1$ produces inward trajectory (Fig. 4d), validating $\\mathcal{L}_{\\text{eff}}$ as a faithful low‑dimensional surrogate.\n\n**A:** As for your question regarding the value of $\\alpha$ - In the experiments, the RNN is trained on finite episode lengths $T$, and $\\alpha$ can be thought of as a knob that replicates the effect of training with various $T$. In Fig. 4e, we validate this theoretical prediction by training identical initial‑condition RNNs with various episode lengths $T$ and showing that they qualitatively produce the trajectories predicted by our surrogate loss using $\\alpha$ (Fig. 4d). \n\nWe do not claim to derive an exact mapping between $\\alpha$ and $T$ - the dominant eigenvalue trajectory could depend on other factors as well. We merely claim that the chosen $\\alpha$ in our surrogate loss ($\\mathcal{L}_{\\text{eff}}$) provides a good qualitative fit to the phenomenon observed during Stage 2 in the high‑D model.\n\nFurthermore, motivated by your question, we **ran new numerical experiments** sweeping over a finer grid of surrogate parameters $\\alpha$ and episode lengths $T$. For each, we computed the phase of the dominant complex eigenvalue. The resulting scatter plot traces a smooth curve, revealing a clear $\\alpha - T$ correlation that provides additional validation for our surrogate-loss intuition.\n\n**R: In the multi-frequency task, what sets the timescale of learning each distinct frequency?\nIn general, if the model requires generating $m$ distinct frequencies, do the dynamics close in a $m$  dimensional subspace?**\n\n**A**:  Regarding your first question about the timescale of learning each frequency, we find this phenomenon particularly intriguing. Learning appears to unfold in discrete, well-delineated stages, marked by sharp drops in training loss. Empirically, the end of each plateau aligns precisely with the appearance of a new complex-conjugate eigenvalue pair in the recurrent matrix $\\boldsymbol{W}$ (Fig. 6d, Fig. 17), whose imaginary part matches the newly acquired frequency. In the multi-frequency task, we did not attempt to derive exact analytical timescales, as this section aimed primarily to test the generality of the double-integrator findings, using fewer modeling constraints (e.g., Adam optimizer, full-rank weights).\n\nFor the double-integrator task, we attempted a more analytical approach. Our analysis suggests that the single plateau ends in a bifurcation that stabilizes the coupled system, but solving for the precise timescales requires handling the eigenstructure of a cubic polynomial—analytically cumbersome even with symbolic math. We believe this is an important and promising direction for future work.\n\n**A**: Regarding your second question, whether dynamics close in an $m$ dimensional subspace. We ran a **new analysis** to test this. Specifically, we pooled hidden states from 100 evaluation rollouts and performed PCA at five checkpoints: epoch 0 and the four loss drops shown in Fig. 6b. The results revealed a clear pattern: each loss drop introduces a new principal component with substantial variance. At epoch 0, the first PC alone explains ~80% of the variance, while by the final loss drop, the top $m$ PCs are needed to explain >90% of the variance."}, {"review_id": "8q6ruB2jQl", "reviewer": "Reviewer_xBz7", "summary": "The paper studies the learning dynamics of recurrent neural networks in closed-loop scenarios and compares it to the corresponding ones in open-loop scenarios. It finds that they significantly differ, with open-loop dynamics failing to properly solve the closed-loop problem mid-training. The paper uses a simplified model and task to thoroughly investigate these dynamics theoretically, as well as provides empirical evidence that the findings holds on a (slightly) more complicated task.", "questions": "From my understanding, the authors use a network trained on the task as the teacher for the open-loop setting. Why not training it on the **optimal control policy** (which should be tractable). Given that trained networks do not reach 0 loss, their predictions might contain some strange artifacts that affect learning dynamics. Can the authors comment on that and perform, if possible, a quick experiment to verify that?\n\nCan the authors provide intuition on l. 250 and on why low frequencies are learned faster?\n\nCould the analysis provided by the authors be extended to **higher-dimensional systems**? Additionally, would there be some **properties** about the task at hand under which the authors expect their **results to significantly change**?\n\nThe huge generalization gap of open-loop dynamics that the authors report could motivate studies in other domains. For example, LLMs are often trained through imitation learning (ie open loop), but are used in a closed loop scenario. If similar findings hold in this scenario, this would imply that a medium amount of imitation learning can actually be harmful for final performance. Overall, I feel that there are **connections to other areas of ML research**, and discussing some of them could make the paper interesting to a larger audience.\n\nFinally, here are a few recent theory papers that the authors may find interesting and relevant to their paper. [1](https://proceedings.neurips.cc/paper_files/paper/2024/hash/fbb07254ef01868967dc891ea3fa6c13-Abstract-Conference.html) [2](https://arxiv.org/abs/2407.07279) [3](https://openreview.net/forum?id=KGOcrIWYnx) Some recent papers on learning dynamics of linear RNNs, from a more deep learning perspective than the literature the authors cite. [Here](https://journals.aps.org/prx/abstract/10.1103/PhysRevX.15.021051) is another theory paper on learning dynamics in a (simplified) RL scenario.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, the paper is **insightful** and **well-written**. It is **well connected** to existing work and nicely motivates why the closed-loop setting is worthwhile studying. The theoretical analysis of the learning dynamics, while specific to a particular task and simple network, is well executed and **provides non-trivial insights on interesting learning dynamics**.\n\nDespite the limited scope of the analysis (which is needed to make progress and thus totally fine for me), I find the main weakness of the paper to be that it is **overly technical**. The big picture and the technical details are well done, but an in-between level which summarizesinterprets/justifies their results in a more intuitive way.", "was_revised": true, "final_justification": "Solid paper. The authors addressed my (minor) concerns. I keep my score as it is (5).", "rebuttal": "Thank you very much for taking the time to carefully review our work. We appreciate your positive assessment. Your comment regarding the non-trivial insights into learning dynamics precisely captures the message we aimed to convey. Please find below our point-by-point responses to your comments and questions. \n\n**R: From my understanding, the authors use a network trained on the task as the teacher for the open-loop setting. Why not training it on the optimal control policy (which should be tractable). Given that trained networks do not reach 0 loss, their predictions might contain some strange artifacts that affect learning dynamics. Can the authors comment on that and perform, if possible, a quick experiment to verify that?** \n\n**A**: Yes, that’s correct—the teacher in our analysis is an optimized RNN. However, our results also hold for optimal control teachers. We were initially motivated by the fact that these control problems admit an analytically optimal teacher via the Riccati equation (i.e., LQR; Appendix B.1). We used an RNN teacher to ensure a fairer comparison between open- and closed-loop RNNs. Following the reviewer’s suggestion, we **performed additional experiments** using an LQR teacher and found that the sharp loss peak in the open-loop setting still persists, confirming that our results extend to this setting as well. \n\n**R: Can the authors provide intuition on l. 250 and on why low frequencies are learned faster?** \n\n**A**: The phenomenon reflects a well-documented inductive bias of neural networks to learn simpler structures in the training data before acquiring more complex components. This has been shown in various settings, including networks learning low-order moment statistics [Refinetti et al., 2023], dominant singular modes [Saxe et al., 2013], and lower frequencies before higher [Rahaman et al., 2019] (all cited in the original submission, line 265). \n\nIn our case, we show that the RNN acquires a new frequency precisely when a complex conjugate outlier emerges in the spectrum of the recurrent weights (i.e., $W$), whose imaginary part matches the learned frequency (Fig. 6d, Fig. 17; original submission). Under rich learning (i.e., small initialization), outlier eigenvalues first reach lower frequencies (smaller imaginary parts) before progressing to higher ones. \n\n**R: Could the analysis provided by the authors be extended to higher-dimensional systems? Additionally, would there be some properties about the task at hand under which the authors expect their results to significantly change?** \n\n**A**: Yes, our analysis can be extended to more complex settings. Below we describe **two new experiments** that show our results hold in more complex scenarios: \n\n1. We conducted preliminary experiments with GRU-based RNNs in a customized, higher-dimensional MiniGrid environment featuring _partial observability_. The agent starts in an ambiguous state and must localize itself to reach the goal via the shortest path. Closed-loop RNN (trained via online policy gradient) produced near-optimal behavior, whereas open-loop imitation RNN diverged and remained suboptimal, failing to develop a rich internal state. While preliminary, these results align with our theoretical findings and suggest broader applicability to nonlinear, high-dimensional tasks.\n\n2. In response to **Reviewer 4A6e**, we systematically generalized our setup by introducing the _k_-integrator family - a cascade of integrators where the matrices $\\mathbf{A}, \\mathbf{B}, \\mathbf{C}$ take more general forms, and the state, actuator, and observation dimensions scale to higher values (for details, see Q1 response to Reviewer 4A6e). Briefly, across **36 additional configurations**, we found that under _partial observability_, the main results of the paper hold: closed-loop RNNs exhibit a loss plateau, and open-loop RNNs show a sharp loss peak. Under _full-state feedback_, these effects largely disappear.\n\nIn summary, we believe that _partial observability_ is a critical property underlying the divergence between closed- and open-loop RNNs. \n\n**R: The huge generalization gap of open-loop dynamics that the authors report could motivate studies in other domains. For example, LLMs are often trained through imitation learning (ie open loop), but are used in a closed loop scenario. If similar findings hold in this scenario, this would imply that a medium amount of imitation learning can actually be harmful for final performance. Overall, I feel that there are connections to other areas of ML research, and discussing some of them could make the paper interesting to a larger audience.**  \n\n**A**: We find this direction very intriguing and are excited by the possibility that our findings could have broader implications, including for domains like LLM training. Our original motivation was to gain analytical traction on the underexplored problem of closed-loop learning in RNNs - potentially relevant to both learning theory and theoretical neuroscience. More broadly, the open- vs. closed-loop distinction is indeed relevant in other domains. For example, in the robotics and RL literature, works on imitation learning and behavioral cloning have acknowledged the pitfalls of ignoring feedback during training (Haan et al., 2019). While LLMs are beyond the scope of this work and our expertise, we appreciate the insightful suggestion and agree that exploring such connections could be a valuable direction for future research. \n\n**R: Finally, here are a few recent theory papers that the authors may find interesting and relevant to their paper. [1] , [2] , [3] Some recent papers on learning dynamics of linear RNNs, from a more deep learning perspective than the literature the authors cite. [Here] is another theory paper on learning dynamics in a (simplified) RL scenario.**  \n\n**A**: We thank the reviewer for pointing out these interesting and relevant works. Some are already cited in the original submission - for example, [3] and the RL perceptron paper [Here], which was one of the early motivations for this project. We will carefully review the remaining suggestions and make sure to integrate them into the revised manuscript."}]

Author rebuttal:

Meta-review: The paper studies training RNNs in open-loop (supervised) or closed-loop (influencing it's own next input) setting, performing a double integrator control task. First, the paper studies that the same RNN, under these two settings, undergoes drastically different learning stages. 

Strengths: The paper is clear and well-written and provides non-trivial insights into these understudied learning dynamics. 
Weaknesses: Some reviewers would have welcomed more empirical investigations outside the mathematical tractable setting this works studies. I find this not necessary. 

The paper is clearly well-executed and, after the rebuttal, no (major) weaknesses remain.
Decision: Accept (poster)