Dynamical phases of short-term memory mechanisms in RNNs
Authors: Bariscan Kurtkaya
Fatih Dinc
Mert Yuksekgonul
Marta Blanco-Pozo
Ege Cirakman
Mark Schnitzer
Yucel Yemez
Hidenori Tanaka
Peng Yuan
Nina Miolane
Venue: ICML 2025 poster (2025)
Keywords: Dynamical system theory
computational neuroscience
working memory
Primary area: applications->neuroscience_cognitive_science

Abstract: Short-term memory is essential for cognitive processing, yet our understanding of its neural mechanisms remains unclear. Neuroscience has long focused on how sequential activity patterns, where neurons fire one after another within large networks, can explain how information is maintained. While recurrent connections were shown to drive sequential dynamics, a mechanistic understanding of this process still remains unknown. In this work, we introduce two unique mechanisms that can support this form of short-term memory: slow-point manifolds generating direct sequences or limit cycles providing temporally localized approximations. Using analytical models, we identify fundamental properties that govern the selection of each mechanism. Precisely, on short-term memory tasks (delayed cue-discrimination tasks), we derive theoretical scaling laws for critical learning rates as a function of the delay period length, beyond which no learning is possible. We empirically verify these results by training and evaluating approximately 80,000 recurrent neural networks (RNNs), which are publicly available for further analysis. Overall, our work provides new insights into short-term memory mechanisms and proposes experimentally testable predictions for systems neuroscience.

Dynamical phases of short-term memory mechanisms in RNNs
Bariscan Kurtkaya" Fatih Dine 234 Mert Yuksekgonul
Marta Blanco-Pozo 26 Ege Cirakman"
Mark Schnitzer 267 Yucel Yemez'
Hidenori Tanaka" 89 89 Peng Yuan" 10 Nina Miolane
Abstract
1. Introduction
Short-term memory is essential for cognitive pro- cessing, yet our understanding of neural mecha- nisms remains unclear. Neuroscience has long fo- cused on how sequential activity patterns, where neurons fire one after another within large net- works, can explain how information is maintained. While recurrent connections were shown to drive sequential dynamics, a mechanistic understand- ing of this process still remains unknown, In this work, we introduce two unique mechanisms that can support this form of short-term memory: slow- point manifolds generating direct sequences or limit cycles providing temporally localized ap- proximations. Using analytical models, we iden- tify fundamental properties that govern the se- lection of each mechanism. Precisely, on short- term memory tasks (delayed cue-discrimination tasks), we derive theoretical scaling laws for criti- cal learning rates function of the delay period length, beyond which no learning is possible. We empirically verify these results by training and evaluating approximately 80,000 recurrent neural networks (RNNs), which are publicly available for further a Overall, our work provides new insights into short-term memory mechanisms and proposes experimentally testable predictions for systems neuroscience.
Learning to respond based on delayed information a fun- damental challenge in both neuroscience and machine learn- ing (Constantinidis & Klingberg. 2016; Bengio et al., 1994). In machine learning, early studies in the 1990s identified this difficulty as the problem of learning long-term depen- dencies, particularly in recurrent neural networks (RNNs) trained on tasks requiring information retention across time (Bengio et al., 1994). Despite significant progress in neural network architectures, this issue remains central design- ing systems capable of effective temporal reasoning.
To address long-term dependencies, machine learning re- searchers developed specialized architectures such as Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and, more recently, Transformers (Hochre- iter, 1997; Chung et al., 2014; Vaswani, 2017). These mod- els introduced mechanisms that enhance memory retention, e.g., by learning neuronal timescales. In contrast, biological neurons communicate through action potentials and are gov- erned by biophysical parameters, such as membrane time constants, that determine the decay of activity over time (Jonas et al., 1993; McCormick et al., 1985; Miles, 1990; Isokawa, 1997: Geiger al., 1997). This fundamental dif- ference raises a key question: how do biologically inspired neural systems learn to associate events separated in time?
This question has regularly been studied under central topic in neuroscience: short-term memory, fundamental cognitive function that enables organisms to temporarily store and manipulate information. This process essential for adaptive behavior and is disrupted in several neurologi- cal and psychiatric conditions such Alzheimer's disease (Bondi et al., 2017), schizophrenia (Lee & Park, 2005) and post-traumatic stress disorder (Scott et al., 2015) among others. Yet, while short-term memory is a central cogni- tive function, the mechanisms that support it remain poorly understood in systems neuroscience (Serences, 2016).
'qqual contribution t Supervision. 'Koc University. Turkey *CNC Program, Stanford University, Stanford, USA GGoomtrric Intelligence Lab, UC Santa Barbara, Santa Barbara, USA 'Kavli Institute for Thcoretical Physies, UC Santa Barbara, Santa Bar- bara, USA CComputer Science, Stanford University, Stanford, USA "James H. Clark Center for Biomedical Engineering & Sciences, Stanford University. Stanford, USA 'Howard Hughes Medical Institute, Stanford University. Stanford, USA 'Phi Lab, NTT Re- search, Sunnyvale, USA "Center for Brain Science, Harvard Uni- versity, Cambridge, USA "Institute for Translational Brain Re- search, Fudan University, Shanghai, China. Correspondence to: Fatih Dinc <fdinc @ ucsb.edu>.
In the mammalian brain, short-term memory supported by a dynamic network spanning cortical and subcortical regions such as the prefrontal, parietal, and sensory cor- tices, and the hippocampus (Todd & Marois, 2004; Curtis & D' D'Esposito, 2003; Harrison & Tong, 2009; Gnadt & Ander- sen, 1988; Baeg et al., 2003; Cavanagh et al., 2018; Meyers
Proceedings of the 4gnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Dynamical phases of short-term memory mechanisms in RNNS
al., 2008). Theories explaining this core process can be broadly grouped into three levels. At the synaptic trans- mission level, memory may be stored in "activity-silen" states, maintained by transient synaptic changes induced by plasticity (Mongillo et al., 2008; Stokes, 2015; Kozachkov et al., 2022; Masse et al., 2019; Brennan & Prockt, 2023). At the neuronal activity level, persistent spiking has been linked to memory maintenance during delay periods (Gnadt & Andersen, 1988; Barash et al., 1991; Curtis & D Esposito, 2003; Baddeley & Hitch, 1974) (Fig. IA). Finally, at the population level, short-term memory has been associated with sequential patterns of neural activation (Baeg et al., 2003; Harvey et al., 2012; Rajan et al., 2016; Wang, 2021) (Fig. IB).
ent memory tasks or training conditions? Q3: And how do these learned mechanisms vary with the delay duration, Le.. the time during which the information must be held in memory?
We set out to answer these questions through a computa- tional theory perspective, generating predictions for in-vivo experiments that we then test in-silico. To achieve this, we trained approximately 80,000 recurrent neural networks (RNNs) perform classical system neuroscience experi- ments designed to assess short-term memory capabilities, while also studying simple toy models and low-rank RNN models as they performed a simpler short-term memory task (Fig. ID-E). Our contributions can be summarized as follows: (i) Using interpretable dynamical system mod- els stripped down to their most essential components for solving delayed activation task (Fig. ID-E), we illus- trate how RNNs can develop distinct strategies during learn- ing, depending on the task and optimization parameters (Figs. 2, 3, and S1), (ii) We show how changing one of the simplest tasks in a trivial manner, e.g., by adding a post-reaction period, can qualitatively change the learned short-term memory mechanisms. This finding is partieularly relevant cautionary tale for the systems neuroscience community modeling animal behaviors by drawing compar- isons to RNNs (Mante et al., 2013) (Fig. 3). (iii) Utilizing
In this work, we study mechanisms that can support mem- ory maintenance at the final population level (Fig. 1B-C). Increasing evidence suggests that memory stability and ro- bustness can emerge from dynamic, distributed neural population activity and is governed by underlying attractor states (Cavanagh et al., 2018; Spaak et al., 2017; Meyers et al., 2008; Stroud et al., 2024; Brennan & Proekt, 2023). Yet several theoretical, and empirically relevant, questions remain unresolved: Q1: What mechanisms support mem- ory maintenance through sequential neural activity? Q2: What determines which mechanism is favored under differ-
A Persistent activity model
Sequential activation
Limit cycles
D
Delayed cue discrimination task
Memory
Memory
Cue (i)) Delay (ii) Reaction (iv) Post-
Reaction
Simulation with RNNs
Memory encoding window
Maintenance window
Input and output values
Cue signal
Target response
Response window
=
Cue
Outpul
Distinct groups of neurons are persistently active during maintenance of particular short-term memory
Memory maintenance happens via the same group sequentially active neurons, readout neurons subserve distinct memory-guided responses
Sequences can become periodic ascillaioons the post-trial period when limit cycles are used for storing shori-term memory
deloy
1050
p0st
dalay
1000
post
Figure 1. Studying short-term memory mechanisms with biologically motivated neural network models. A-C A visualization of short-term memory maintenance in neural activations. A Earlier models predict that fixed and persistent activities of a subset of neurons during the delay period could encode the content corresponding to distinct memories (red vs blue populations) (Fuster & Alexander, 1971; Atkinson & Shiffrin, 1968). B Memory contents can also be stored within a sequentially firing neuron population (Rajan et al., 2016). where distinct readout neurons decode distinct memories. C In this work, we show that limit cycles, i.e., periodic orbits, can locally approximate the memory mechanism in panel (B). However, if left undisturbed, limit cycles may result in spurious periodic responses in the post-reaction window (magenta). D-E An illustration of the delayed cue-discrimination task and its simulation with RNNs. D The task includes two correct cue-reaction matches (blue-left, red-right) and consists of four periods: (i) Cue period, where one of the two cues is presented; (ii) Delay period, with no expected reaction; (iii) Reaction period, where the cue class needs to be outputted; and (iv) an optional post-reaction period at the end of the trial. E We designed an equivalent task for RNNs, in which two input channels are mapped to the output channels and for a given trial only a single input is provided in the form of a rectangular pulse of duration Tin. RNNs can be trained to follow the stages (i-iv) similar to the animals in their target responses. When the cue period is omitted, we refer to this task delayed activation task.
2

---

Dynamical phases of short-term memory mechanisms in RNNS
analytical models, we demonstrate theoretical scaling laws and synthesize a phase diagram of mechanisms for solv- ing the task, which is function of the task properties and optimization parameters (Figs. S2 and 4G). (iv) Through a large-scale study of RNNs we extract the same scaling laws in RNNs trained to perform delayed cue-discrimination tasks, which are quantitatively consistent with our theoret- ical explorations within error bars. We show that RNNs have little to no trouble learning modified task with delayed output, providing further evidence that training bio- logically inspired neural networks to delay their outputs is fundamentally difficult (Figs. 4, S3, S4, S5 and S6).
and Wout E Rg-NoutXNvw and dout E RR-ouu for an Nout- dimensional output. For notational simplicity, we assume
0 until our empirical discussions in Section 3.4.
RNN models of sequential Recent work has iden- tified the generation of neural sequences as a potential key component of short-term memory (Rajan et al., 2016). This distributed sequence can be used to store the short-term memory content and explicitly keep track of time, which contrasts with earlier theories of short-term memory main- tenance relying on the persistent activities of some neurons (Fuster & Alexander, 1971; Atkinson & Shiffrin, 1968) (Fig. IB). Previous work has provided insights into how these sequences are generated by examining neuronal-level mechanisms (Rajan et al., 2016; Sommer & Wennekers, 2005; Laje & Buonomano, 2013), often focusing on the learned connectivity structures (Rajan al., 2016; Laje & Buonomano, 2013). However, the population- causal mechanisms underlying neural computation, which could be investigated through latent dynamics, remain largely un- explored.
2. Bacckrouud
Systems neuroscience often utilizes animal models, e.g., mice, rats, birds, and flies, to study cognitive functions including short-term memory (Serences, 2016). Alterna- tively, it is often possible to design artificial models such as RNNs, which can be trained on observed neural activi- ties or tasks that are traditionally performed by animals in laboratories (Mante et al., 2013; Sussillo & Barak, 2013; Yang et al., 2019; Masse et al., 2019; Dubreuil et al., 2022). Then, once RNNs are trained, their computation can be reverse-engineered to investigate the nature of the neural computation performed by biological networks, generat- ing hypotheses that can later be tested with experimental procedures (Walker et al., 2019; Finkelstein et al., 2021). Consequently, RNNs have recently become a key compo- nent of computational neuroscience studies (Dubreuil al., 2022; Valente et al., 2022; Finkelstein et al., 2021; Sussillo & Barak, 2013).
To study the latent mechanisms of sequential neural activ- ities, focus on class of low-rank RNNs (i.e., when W W=Ex- (z)) are constrained to be low-rank). In this class, the computation performed by RNNs can be stud- ied with the latent dynamical system (Dubreuil et al., 2022; Valente et al., 2022; Dine et al., 2025a; 2023):
(t) = - (t)
+
Σ
n
(k)
x
F=1
(2)
K
X tanh
Σ
m
1 (2((t))
+ wim + +
Here, we consider a family of RNNs that are biologically relevant and whose parameters can be interpreted as func- tional connections between neurons (Perich & Rajan, 2020; Perich et al., 2021). Specifically, we study RNNs defined by the following equation:
where the latent variables are defined as (t)(t) = n (k)TT((t) r(t) (Dinc et al., 2025a). Intuitively, by constraining the rank of the weight matrix, W, one can project the computation performed by N neurons down to a K K-dimensional latent dynamical system. In this work, we use this paradigm (i.e., training low-rank RNNs) to illustrate the distinct mecha- nisms that support sequence generation, they emerge dur- ing training. Then, our large-scale experiments generalize our findings to full-rank RNNs with no explicit restrictions.
ri(t)= = -rt) + tanh(Wr(() + u() +b e), (1)
where r(t) E R-V refers to the firing rates of N neu- rons, u(t) E RN pre-defined Nin -in-dimensional inputs, W E RNNN Win E N* weight parameters with b E R. corresponding bias terms, and e some noise that is, in practice, sampled i.i.d from a Gaussian distribu- tion. The output is taken as a projection of the neural ac- tivities as ō(t)= f r(t) bout), where f() is ei- ther identity or sigmoid function in this work (see below)
3. Results
3.1. Latent mechanisms underlying neural sequences
As first step, we set out to answer our first question (i.e., Q1: What mechanisms support memory maintenance through sequential neural activity?) using a simple short- term memory task, stripped of any extrancous components, that can be solved using neural sequences (Fig. 1D-E). Specifically, we focused on one of the simplest tasks called a delayed activation task (Dinc et al., 2025b). In this task,
'The dataset is publicly available through (00.5228//eenoo.1555522 15529757. To obtain this dataset, we used several computers with NVIDA RTX 3090 GPUs or equivalent, which roughly amounts to 230 kg CO2 emission, 930 km driven by an ICE car, 115 kg coal burned, 4 tree seedlings sequesting carbon for 10 year as computed via wit.hloooogtuuubbiiipeeee
3

---

Dynamical phases of short-term memory mechanisms in RNNS
X Trial start
Response start
Trial end
Absolute firing rates
A
T = 75 ms, T = 10 ms,
Latent state (during trial)
Latent state trial)
Decision boundary
Normalized values
10
Activity of the second latent variable
10
Neuron
104
Loss function
104
Gradient norm
10
10a
100
Number of training epochs
Activity of the first latent variable (a.u.)
Time (ms)
dalay
ms, reap
10 ms,
10
Activity of the second latent variable(a.u)
Normalized values
10
Newron ID
10a
Loss function
10
Gradient norm
100
10P
10'
Number training epochs
Activity of the first latent variable (a.u.)
Time (ms)
75 ms, T resp
10 ms, a 10
dalay
Aclivity of the second latent variable(a.u)
Normalized values
Neuron ID
Loss function
10a
Gradient norm
100
10'
10'
10
100
Number of training epochs
Activity of first latent variable (a.u.)
Time (ms)
Figure 2. Changing optimization parameters may result in RNNs learning distinct attractor mechanisms. We trained a rank-2 RNN perform the delayed activation task, in which RNNs initialized a particular (Methods) should delay their output hy Taelay for Trep time interval. The task does not constrain the network activity or the output outside of these time windows. A Left. The normalized gradient norm and the loss function values. Middle. Flow maps the learned latent dynamical systems demonstrating attractor mechanism. The shaded areas around the decision boundary correspond to network outputs within 0.75 (outer green boundary) 0.25 (outer red boundary). Right. Absolute firing rates of the network, whose latent dynamics were illustrated in middle column. Dotted line corresponds to the start of the response period. Parameters: N 100 neurons, 7 10ms, At 5ms, Tdelay 75ms, Tresp = 10ms, a for stochastic gradient descent using otherwise the default parameters Pytorch (Paszke al., 2017). B-C Same in panel (A), but trained with learning rates of 10- and 10-4, respectively.
(learned)
the network is initialized particular state, which ran- domly selected in the state space (Methods), and has withhold its response (ó(t) 0) for Taclay time win- dow. Afterwards, the network is forced to output response (ô(t) = 1) for Tresp time window.
points. This transition is often structured, iee, the same latent trajectory would be followed the system passes through these regions (Sussillo & Barak, 2013), and there- fore can lead to stable and sequential activations of neurons (Fig. IB). The second mechanism relies on the generation of limit cycles, which are defined as closed trajectories that represent sustained, periodic oscillations. In this mecha- nism, when the oscillation periods are much larger than the trial windows, neural sequences can emerge during the task execution without repetition (Fig. 1C). But, once the task concludes, the limit cycles would continue to oscillate and regenerate the sequential activities, whereas the slow- slow-points are transitionary and would not repeat (Fig. IB-C).
Two distinct mechanisms can subserve neural sequences- Broadly, we set out to study two latent mechanisms that may subserve the observed neural sequences. The first mecha- nism relies the generation of set of slow-points, neural states in which the neural dynamics vary slowly (F(t) z 0). In this mechanism, neural sequences emerge during the slow transition of the latent dynamical system through these slow-

---

Dynamical phases of short-term memory mechanisms in RNNS
Since a minimum of two dimensions is required generate oscillatory behavior in dynamical systems (Strogatz, 2018), we start by studying the learned latent representations in an RNN with rank K = 2 (corresponding to two-dimensional latent dynamics). The output, 6(t), of the RNN is defined as a nonlinear projection of the latent variables using a sigmoid function s()) For the largest learning rate (Fig. 2A), this rank-2 RNN was incapable of solving the task despite the nearly-zero gradient'. As the learning rates decreased, it was capable of solving the task with limit cycles with moderate learning rate (Fig. 2B), but utilized slow-points (also referred to as "SP manifolds" for generality) for the lowest learning rate (Fig. 2C). Notably, both mechanisms produced neural activity sequences during the task window (Fig, 2, right panels).
behavior within the task window, but can show distinct prop- erties outside; and the mechanisms we identified belong to this class. For the rest of this work, we will formalize this connection between the learned latent mechanisms and the task and/or optimization parameters. In doing so, we will generalize these findings beyond anecdotal observations via theoretical investigations and large-scale experiments.
3.2. Influence of task design on the learned mechanism
Behavioral experiments in systems neuroscience often con- tain arbitrary design choices that are deeply embedded in the experimental paradigm, yet their implications are rarely scrutinized. Consider the delayed cue discrimination task, a task commonly studied in neuroscience experiments with mice (Ebrahimi et al., 2022). In the machine learning con- text, it consists of four distinct periods (Fig. ID-E): cue, delay, reaction, and an optional post-reaction period (i-iv). Upon observation of a cue, the networks have to wait for brief period and output a response in the corresponding output channel. Typically, there are two possible cues and hence two output channels. If there is post-reaction period, then the networks should learn to output zeros after the re- sponse window concludes. In experimental context, while animals typically indicate their response by licking either a right or left spout, the reward or punishment is often di- rectly provided at the conclusion of the behavior without the post-reaction window. The distinction, as we show below, matters for the final learned solution.
Despite leading equivalent neural activities within the trial window, the two mechanisms had qualitatively different behaviors after the trial concluded (Fig. 2). Limit cycles had led to recurrent generations of the same sequences, whereas the SP manifolds converged to a persistent activity style representation after the trial concludes. We will make this observation rigorous in our large-scale studies below. For now, we provide insights into another relevant question: Do RNNs commit to learning one mechanism from the start, or are they capable of changing their inner mechanisms during the training?
Learned latent mechanisms can evolve during learning- We next studied the latent dynamical systems of a rank- 2 RNN as it was training to solve the delayed activation task (Fig. S1). Interestingly, the loss function showed characteristic periods of learning, corresponding to changes in mechanisms (Fig, S S1A). Specifically, after the first jump in the loss function, a SP manifold was formed in the latent dynamical system (Fig. SIB; left). Here, a line of slow- points allowed the network output to stall until Tamlay. Then, the SP manifold had acquired a curved structure (Fig. SIB: middle), reminiscing an onset to a rotating solution. Finally, after the oscillations in the loss function values ceased, limit cycle had emerged in the latent dynamical system (Fig. SIB; right). Hence, we confirmed that RNNs could change their inner mechanisms during learning, i.e., what started as a SP manifold solution later evolved into a limit cycle.
Minor modifications to the task requirements can change the learned mechanisms-We now set out to answer our second question (i.e., Q2: What determines which mechanism is favored under different task or training conditions?). To do so, we perform a minor change to the delayed activation task by requiring a short post-reaction waiting period and illustrate how minor changes in experimental design can fundamentally alter the mechanisms networks tend to learn during training. In the modified delayed activation task, (rank-2) RNNs are required to switch their responses at the end of the response period and output zero for a brief duration before the trial concludes (Fig. ID-E). In networks trained without this modification (Tpstt = Oms), the latent dynamical systems can learn both slow-point manifolds (for shorter delays) and limit-cycles (for longer delays), similar to Fig. 2. However, when we introduce even a brief post-reaction period (Tpast = 30ms), the networks predominantly learn limit cycle solutions, despite achieving similar task performance (Fig. 3A-B). We quantified and confirmed these findings with a broader empirical study with 1600 RNNs (Fig. 3C; via Fisher's exact test with Bonferroni corrections). Thus, in rank-2 RNNs solving the delayed activation task, increasing the delay period promoted the emergence of limit cycle solutions, and this tendency was dramatically amplified by the addition of a rest-reaction
In reality, since functional connections in the brain can rapidly reorganize within the start and end of trial (Ebrahimi et al., 2022), the two mechanisms can become practically inseparable in biological neural networks. In other words, there is an equivalence class of mechanisms that can generate neural sequences that have the same local
'This phenomenon has been studied in details by previous work (Dine et al., 2025b), in which the leamnngg dynamics can get stuck in no-learning zones with near-zero gradients when very large learning rates are used.
5

---

Dynamical phases of short-term memory mechanisms in RNNS
A
Training a rank-2 RNN on
B
Fully trained network for T
Fully trained network
C
An empirical study with
two versions of the DA task
0 ms
for T pos:
30 ms
X Trial start
100 networks per condition
post
Normalized loss
function values
10
Activity of the second latent variable(a.uu)
Response start
Fraction of networks with full rotations
10
Trial end
T ms
Latent state (during trial)
10
30 ms
Latent state (after trial)
10
Decision boundary
100
40
60
8D
100
Number of training epochs
Activity of the first latent variable (a.u.)
Delay period (T dway)
Figure 3. Changing trivial task parameters may result in RNNs favoring limit cycle solutions. A We performed experiments similar those in Fig. 2, but now with an added post-reaction period poos)) B Learned latent dynamics for the fully trained networks in panel (A). Parameters for A-B: As in Fig. 2A, with Tactay = 30ms, Tresp 10ms, Tpost 0 or 30ms, and a = 10 C Fraction of networks developing rotating solutions (i.e., limit eycles) when solving the two versions of the task as a function of delay Taclay- Parameters: = 0.005, varying levels training epochs corresponding to Taelay, otherwise the same as in panels A-B. Comparisons: Fisher's exact test with Bonferroni corrections. *** ""p 0.001, and ""p < 0.01.
window. This demonstrated that seemingly minor changes in the task design can favor learning distinct short-term memory maintenance mechanisms.
Earlier work has considered a oie-dimensional toy system, the normal form of a saddle-node bifurcation, = x² + + (Strogatz, 2018), to study the learning dynamics of slow- point formation (Dinc et al., 2025b). This system approx- imates the local behavior of one-dimensional flow maps G(v, u) near their minima, where slow-points (also ob- served in Fig. 2) arise. To perform a similar analysis with the oscillatory dynamics observed in Figs. 2 and 3, we will consider a second toy model in this work: a ttwo-diennioonal system exhibiting a limit cycle, one of the simplest settings in which periodic orbits, and hence oscillations, can emerge (Strogatz, 2018). Both toy models are meant qualita- tively mirror the behavior of the respective latent flow maps G(v, u) that may be learned by an RNN (Fig. 2). While these toy systems are not exhaustive, they capture key fea- tures of the latent mechanisms that RNNs may converge to, and allow us to dissect their learning dynamics in a controlled and analytical setting, which we discuss next.
3.3. A theoretical scaling analysis of mechanistic phases
So far, we used simple examples to demonstrate how RNNs can use different latent mechanisms to generate neural se- quences, that RNNs can change their inner mechanisms dur- ing the course of their training, that the choice between these mechanisms is not random but depends on specifics of the experimental design. Now, to provide an answer to the third question (i.e., Q3: How do the learned mechanisms vary with delay duration?), ?), we turn to a set of -iw-dimeniional dynamical models and study their learning dynamics.
Why study low-dimensional toy models?-Recen work has shown that training (full-rank) RNNs often converges to effectively low-dimensional solutions (Valente et al., 2022), a property that appears to generalize across many real-world networks (Thibeault et al., 2024). As discussed in Eq. (2), these low-dimensional computations can often be inter- preted as latent dynamical systems. In this view, training the RNN parameters (W, W b) amounts to shaping a latent flow map of the form:
Extracting scaling laws from toy models-To start with, in Appendix 12. we reproduce the findings of the earlier work on slow- manifolds for a more general version of the de- layed activation task with Tadlay * Tresp. This analysis re- veals a maximum learning rate, also reported in (Dine et al., 2025b), beyond which the learning dynamics destabilize and even an approximation of the desirable solution cannot be reached. Our analysis reveals that this maximum learn- ing rate follows power law such that asp ~ O (app delay ), where 3sr E [4.5]. The exact value Asr achieves within this interval depends whether Taelay and Tresp are compara- ble, with larger delays leading to more substantial decreases (approaching 3sp - 5 in the limit Taelay >> Tresp). Such a steep decrease with the desirable learning rate can explain our anecdotal observations in Figs. 2 and 3C that slow- points become rarer for large Taelay high learning rates, which will be substantiated in the next section.
+*(t)= G(x(t), u(t); W, wii,,)) 66)
where G governs the evolution of low-dimensional latent variables x(t) in response to input u(t). Hence, studying how toy models, which do not necessarily share the same architectural assumptions as RNNs but still aim to learn similar local dynamics, may provide intuition for how these flow maps may be learned. Thus, we set out to analyze simple low-dimensional dynamical systems that capture the two mechanisms floo-pooinss and limit cycles) we have observed in trained RNNs.
6

---

Dynamical phases of short-term memory mechanisms in RNNS
Now, we focus on the limit cycle model. For simplicity, we fix the radius to an attractive fixed-point, and allow periodic oscillation in the frequency such that the equations of the motion for the toy model become (See Appendix S2.2 for derivations) s=(t) = sin(2 where r E R is a learnable parameter. The network output is defined as 6(x) = (2: < 0), where (-)) is defined as the Heaviside function. By inspection, the rotations have a half-period of Thaif 2r' and the model outputs zero for the first Thair while outputting one for the second half. An example flow map is shown in Fig. S2A; changing the parameter, r, modulates the frequency/period of the cycle.
(Figs. I, 2, and S1), and therefore can offer further insights into learning dynamics in more complicated architectures. Below, we provide one example case, and show empirically that similar phase spaces emerge in full-rank RNNs trained on delayed cue discrimination tasks.
3.4. Empirical scaling analyses of latent mechanisms
As a final step, we tested the theoretical predictions of the toy models, including the proposed scaling laws and the phase space of latent mechanisms, by training full-rank RNNs on delayed cue-discrimination tasks (Figs, 4, S3, S4, and S5). Similar to low-rank RNNs, full-rank RNNs also learned to solve the task using two distinct mechanisms (Fig. 4A; Fig. S3A,B). Solutions involving sustained oscillations exhibited minimal power at low frequencies, as the dynam- ics were dominated by high- figh-fryy components (Fig, S3C). Based on this observation, we developed a mecha- nism discrimination index that automatically classified the learned RNNs into three groups: (i) networks relying on flow-point manifolds, (ii) networks employing limit cycles, and (iii) networks that failed to perform the task (see Fig. S4 for representative examples).
In Appendix S2.2, we compute an analytical loss function for the limit cycle model in the limits Tresp = Taelay and Teep < Tdelay. We illustrate the latter case in Fig. S2B, note the exact correspondence between the analytical curve and the numerical simulations. For both cases, the loss function has a global optimum at 2Tacta However, it has kink around the global minimum and quickly flattens to constant value for r M[Taaay + -.mp Hence, if the learning rate is larger than some critical value, the oscilla- tions around the glohal minimum can throw the parameter to value r
"(Tasty terminating the training process due to the zero gradient despite the fact that the model cannot solve the task. Our analysis reveals that this PLe maximum learning rate scales following af delay where $LC E [2. 3). Notably, given that $LC S $sp, the limit cycle solution has a better scaling than the slow-point one, providing a theoretical explanation for our observation in Fig. 2. Specifically, since the RNN in Fig. 2 was trained using a learning rate right at the transition (compare to Fig. S1), it is not surprising that slow point mechanism even- tually evolved into a limit cycle. Moreover, this theoretical prediction is also consistent with the experiments in Fig. 3C, where increasing the delay period for a fixed learning rate resulted in more RNNs finding limit cycle based solutions. Hence, both learning rate and the duration of the delay can influence the learned latent mechanism during training.
Large-scale RNN experiments reproduced the toy model predictions then applied this classification to examine whether different mechanisms consistently emerged across varying delay durations. In line with theoretical predictions, increasing the length of the delay period preferentially led to the emergence of limit cycles (Fig. 4B). These oscillatory so- lutions disappeared beyond critical learning rate for given delay, with the transition scaling as 8r.c = 2.72 + 0.07 (Fig. 4C), in close agreement with predictions (see Fig. S3D- E for the procedure used to compute critical delays). For shorter delays, RNNs predominantly developed slow-point manifolds, with the transition scaling as 3gp = 4.05 + 0.10. also matching the theoretical range (Fig. S3). However, when we trained RNNs on modified task variant that in- cluded a post-reaction window, slow-point solutions were no longer observed, and only oscillatory dynamics emerged (Fig. 4D-F). To summarize these transitions, we plotted phase diagrams (Fig. 4G-I) derived from both the theoret- ical model and the full-rank RNN experiments, capturing the conditions under which each mechanism emerged and their associated scaling laws.
Further insights from toy models- The distinct scaling of ÔLC and 8sp also highlights the existence of an interest- ing phenomenon: phase space of mechanisms that can be learned by RNNs. This phase space (which will be discussed in Fig. 4G) depends on the delay time (a task parameter) and the learning rate (an optimization parameter). On the other hand, it is important to not over-interpret the toy model results. Specifically, increasing the dimensionality of the latent dynamical system can allow more efficient solutions to the task, and it may be possible for RNNs to utilize other (diverse) mechanisms for storing memories (e.g., those us- ing inherent transient dynamics in high-dimensional neural networks (Jaeger, 2002; Liu et al., 2025; Maass et al., 2002: Ichikawa& & Kaneko, 2021)). Fortunately, the toy models we discussed utilize mechanisms that are observed in practice
Control auperiments-Fnalllyy we performed two additional control experiments. In Fig. S5, we repeated the analysis using RNNs with slower intrinsic dynamics (i.e., larger T), and in Fig. S6, we tested version of the task that no longer required memory maintenance. In the latter case, the dependence on task duration nearly vanished, and the scaling observed in Fig. 4 was substantially weakened (3 0.38 + 0.02). These findings reinforce the idea that the computational mechanisms adopted by RNNs are not fixed
7

---

Dynamical phases of short-term memory mechanisms in RNNS
Scaling analysis in delayed cue discrimination tasks with no post-reaction period (T
post 0ms)
Loss function values V
Memory
B
C
mechanism
Scaling of the
Fraction RNNs
0.1
Not learned
Learning rates
104
no-solution boundary
Limit cycle
SP manifold
10
B 2.72 0.07.
R 0.98,
p=100
10-
1000
100
200
300
400
30
100
300
Number of epochs
Delay (ms)
Delay (ms)
Scaling analysis in delayed discrimination tasks with post-reaction period (T post
50ms)
E
Scaling of the no-solution boundary
Loss function values
Fraction of RNNs
Leaming rates
0.1
10*
10
B = 2.65 0.23,
R2 0.93, p<10=
109
0
1000
0
100
200
300
400
30
100
300
Number of epochs
Delay (ms)
Delay (ms)
Phases of learned short-term memory mechanisms
G
Toy models
H
Learning rate (logscale)
T post
0 ms
Learning rate (logscale)
T
post
50 ms
Scaling curves
10
10-
Learning rate
10
Phase no learning
(phase boundaries)
Slow point
10
Limit cycle
1
10a
10?
Phase
Phase Only limit cycles
10
10
10
0
200
400
200
400
0
200
400
Delay (ms)
Figure Emergent phases attractor mechanisms RNNs performing delayed cue-discrimination. A Loss curves corresponding different memory mechanisms observed during training. Distributions of learning rates and resulting memory mechanisms function of the delay period. Each line represents distinct learning rate; each run is initialized with a different random seed. C Power-law fits shown log-log scale, where the y-axis indicates the maximum learnable delay and the z-axis the corresponding critical learning rate. Each dot denotes the cutoff delay value beyond which learning fails (see Methods for how these values are estimated). D-F Same as in panels A-C, but for tasks with post-reaction windows. G The phase diagram of mechanisms predicted the toy model. H-I Phase diagrams corresponding to panels B-E, where color intensity reflects the proportion of trials converging to cach of the three dynamical phases. As predicted by the thcoretical analysis in the original task (H) reveals three distinct regimes, whereas the inclusion of a post-reaction window (I) biases the model toward limit cycle dynamics. To generate these phase diagrams, we trained over 65,000 RNNs across varying learning rates, delay intervals (Taslyy, random seeds (see Table S1). For full parameter details, refer to Section S4.
but shaped by the task's demands. In particular, memory duration, learning rate, and structural constraints (refer to Q3 above) jointly determine the preferred mechanism.
core components of the solutions. These sequences arose through two distinct mechanisms: slow-point manifolds and limit cycles. While the use of slow-points to approximate sequences is well aligned with prior intuitions (Khona Fiete, 2022), the emergence of limit cycles in this context represents novel prediction of our framework. Although limit cycles have previously been observed in memory tasks (Pals et al., 2024), they typically appear when periodicity is embedded in the task design. In contrast, the delayed activa- tion and delayed cue-diserimination tasks used here had no explicit periodic components. Thus, the appearance limit cycles in these tasks suggests an alternative, previously un-
4. Discussion
In this work, set out to answer three fundamental ques- tions about the population-leva mechanisms that can sup- port short-term memory in neural networks.
What mechanisms support memory maintenance through sequential neural activity? In the memory tasks studied here, neural activation sequences consistently emerged
8

---

Dynamical phases of short-term memory mechanisms in RNNS
derappreciated, mechanism for generating sustained neural sequences in the absence of external rhythmic input or oscillatory output.
date longer memory spans can complicate optimization and destabilize learning (Dinc al., 2025b). Unlike biological systems, which operate under biophysical constraints such as membrane time constants and action potentials, artifi- cial networks must learn timescales through training, which is not always possible even if the underlying network is theoretically capable of solving a particular task.
What determines which mechanism is favored under dif- ferent memory tasks or training conditions? Our analyti- cal results revealed that the selection between slow-point manifolds and limit cycles depended on both the structure of the task and the learning rate used in the optimization. We later confirmed these predictions in large-scale simu- lations using full-rank RNNs (Table S1). This finding is particularly striking in light of recent interest in dynamical systems approaches in neuroscience, where considerable effort has gone into identifying and manipulating attractors and manifolds that underlie neural activity (Liu et al., 2024; Vinograd et al., 2024). The sharp divergence in dynam- ical solutions (slow-points versus limit cycles) emerging from minor changes in task design underscores a critical point: the structure of the task can strongly shape the neural dynamics that emerge.
Overall, by framing the challenge through dynamical sys- tems theory, our work highlights core trade-off between memory duration, learning speed, and network stability. Un- derstanding this tradc-off offers a principled explanation for the persistent difficulty of learning long-term dependencies and opens the door to new strategies that balance biological plausibility, computational efficiency, and learning perfor- mance.
Concluding remarks- Our findings carry important impli- cations for experimental neuroscience. We have shown that both slow-points and limit cycles can serve as effec- tive mechanisms for generating sequential neural activity. However, in the context of a dynamic brain, where synap- tic plasticity continually reshapes functional connectivity (Ebrahimi al., 2022), it remains unclear how to determine whether observed sequences are supported by sow-points or limit cycles. More broadly, how can we distinguish be- tween dynamical solutions that produce equivalent activity patterns during a trial, but diverge in their behavior once the trial ends? To our knowledge, this question (concerning equivalence of distinct latent mechanisms subserving task execution, but later diverging after the task conclusion) has not been explicitly addressed in experimental designs within systems neuroscience. Tackling it would require a shift in how experiments are conceptualized and conducted. As a starting point, our work illustrates how (even minor) ma- nipulations of task structure can lead to distinct underlying mechanisms.
This raises important questions about how we interpret dy- namical structures observed in neural recordings. Can we conclude that one type of solution is more fundamental than another simply because it appears in specific task variant? Our findings highlight the need for caution. They suggest that conclusions about the nature of neural computations must be grounded not only in observed dynamics but also in a careful analysis of how task parameters may constrain or bias the underlying solutions. What may appear to be a core computational motif could, in fact, reflect a contingent outcome of the task design itself.
How do the learned mechanisms vary with the delay dura- tion? A key insight into this question comes from exam- ining how critical learning rates (beyond which learning ceases to be effective) scale with delay length. In simple delayed-responss tasks, we found that the critical learning rate allowing successful training decreases as a power-law function of the delay. As the delay increases, smaller and smaller updates are required, ultimately making training impractically slow for long durations. This constraint may underlie the historical difficulty of capturing long-term de- pendencies in RNNs, as highlighted in early work (Bengio et al., 1994).
Beyond neuroscience, the phenomenon of staged learning dynamics and algorithmic phase transitions has been widely documented in artificial neural networks, including Trans- formers trained on modular arithmetic (Liu et al., 2022; Nanda et al., 2023), language modeling (Wei et al., 2022; Lubana et al., 2024), and in-context learning (Raventós et al., 2024). Within this growing literature, our study of RNNs provides a complementary perspective, grounded in dynam- ical systems theory, that connects computational behavior to interpretable latent mechanisms. Recent efforts aim to formalize learning progress in feedforward networks using quantities analogous to order parameters in physics (Park et al., 2024). Our results go a step further by identifying literal phase transitions between qualitatively distinet com- putational strategies in dynamical systems. Future work may explore how these mechanisms can be implemented or approximated by feedforward architectures.
The introduction of gated architectures such as LSTMs and GRUs partially addressed this issue by enabling networks to learn adaptive timescales (Hochreiter, 1997). These ar- chitectures could perhaps bypass the power-law constraint by dynamically modulating their internal time-seales, and thereby extend their memory retention periods. However, even if successful, this flexibility would lead to trade-offs. Extending timescales indefinitely slows down network re- sponses, limiting their utility for rapid decision-making. In addition, expanding the parameter space to accommo-
9

---

Dynamical phases of short-term memory mechanisms in RNNS
Acknowledgements
intraparietal area. temporal properties: comparison with area 7a. Journal of neurophysiology, 66(3):1095-1108, 1991.
We would like to thank members of the Geometric Intelli- gence Lab for their helpful feedback on an earlier version of the project and Dr. Boris Shraiman for fruitful discus- sions the final manuscript. EC and BK's internships were supported in part by a grant from the Feldman-McClelland Open-a-Door fund of the Pittsburgh Foundation. BK thanks the Impact Scholarship and Research Scholarship for Crit- ical Thinkers programs from the Bridge to Turkiye Funds for supporting his visit to Stanford University. FD expresses gratitude for the valuable mentorship he received PHI Lab during his internship at NTT Research and acknowledges funding from Stanford University's ' Mind, Brain, Compu- tation and Technology program, which is supported by the Stanford Wu Tsai Neuroscience Institute.. MJS gratefully acknowledges funding from the Simons Collaboration on the Global Brain and the Vannevar Bush Faculty Fellow- ship Program of the U.S. Department of Defense. NM ac- knowledges funding from the National Science Foundation, award 2313 150. PY was additionally supported by the Foun- dation of the Shanghai Municipal Education Commission (No. 24RGZNA01). This research was supported in part by grant NSF PHY-2309135 and the Gordon and Betty Moore Foundation Grant No. 2919.02 to the Kavli Institute for Theoretical Physics (KITP). Some of the computing for this project was performed the Sherlock cluster. We would like to thank Stanford University and the Stanford Research Computing Center for providing computational resources and support that contributed to these research results.
Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Bondi, M. W., Edmonds, E. C., and Salmon, D. P.
Alzheimer's disease: past, present, and future. Journal of the International Neuropsychological Sociery, 23(9-10); 818-8 801-831,11 2017.
Brennan, C. and Proekt, A. Attractor dynamics with activity- dependent plasticity capture human working memory across time scales. Communications psychology, 1(1):28, 2023.
Cavanagh, S. E., Towers, J. Wallis, J. D., Hunt, L. T., and Kennerley, S. W. Reconciling persistent and dynamic hy- potheses of working memory coding in prefrontal cortex. Nature communications, 9(1): 2018.
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: wexXiv::412.35555 1412.3555 2014.
Constantinidis, C. and Klingberg, T. The neuroscience of
working memory capacity and training. Nature Reviews Neuroscience, 17(7):438-449, 2016.
Curtis, C. E. and D'Esposito, M. Persistent activity in the prefrontal cortex during working memory. Trends in cognitive sciences, 7(9):415-423, 2003.
Impact Statement
This paper contributes to machine learning and computa- tional neuroscience by making unprecedented number of trained RNNs publicly avilabblemodee that required several weeks of compute time to generate.
Dine, F., Shai, A., Schnitzer, M., and Tanaka, H. CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL https://///// htt.ttpps::ppppeeerrtiiiiee /pwp///peeeeeeeeeeee/ee/ee/oo.ep net/ forum? i id=GGIA1p9EDT.
References
Atkinson, R. C. and Shiffrin, R. M. Human memory: A proposed system and its control processes. Psychology of Learning and Motivation, 1968.
Dine, F., Blanco-Pozo, M., Klindt, D., Acosta, F,, Jiang, Y., Ebrahimi, S., Shai, A., Tanaka, H., Yuan, P., Schnitzer, M. et al. Latent computing by biological neural net- works: A dynamical systems framework. arXiv preprint arXiv:2502. /4337, 2025a.
Baddeley, A. D. and Hitch, G. Working memory. vol- ume 8 of Psychology of Learning and Motivation, pp. 47-89. Academic Press, 1974. doi: https://doi.org/10. 1016/S0079- (016//00799-7221088)644444 (08)660555...
Dine, F., Cirakman, E., Jiang, Y., Yuksekgonul, M., Schnitzer, M. J., and Tanaka, H. A ghost mechanism: An analytical model of abrupt learning. arXiv preprint arXiv:2501.022. 2508077.7. 2025b.
Baeg, E., Kim, Y., Huh, K., Mook-Jung, I., Kim, H., and Jung, M. Dynamics of population code for working mem- ory the prefrontal cortex. Neuron, 40(1):177-188, 2003.
Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F., and Ostojic, S. The role of population structure in com- putations through neural dynamics. Nature Neuroscience, pp. 1-12, 2022.
Barash, S., Bracewell, R. M., Fogassi, L., Gnadt, J. W., and Andersen, A. Saccade-related activity in the lateral
10

---

Dynamical phases of short-term memory mechanisms in RNNS
Ebrahimi, S., Lecoq, J., Rumyantsev, O., Tasci, T., Zhang, Y., Irimia, C., Li, J., Ganguli, S., and Schnitzer, M. J. Emergent reliability in sensory cortical coding and inter- area communication. Nature, (05(7911):713-1))
Laje, R. and Buonomano, D. V. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nature neuroscience, 16(7):925-933, 2013.
2022. Finkelstein, A., Fontolan, L., Economo, M. N., Li, N., Ro- mani, S., and Svoboda, K. Attractor dynamics gate cor- tical information flow during decision-making. Nature Neuroscience, 24(6):843 850, 2021.
Lee, J. and Park, S. Working memory impairments in schizophrenia: a meta-analysis. Journal of abnormal psychology. 114(4):599, 2005.
Liu, C., Jia, S., Liu, H., Zhao, X., C. Xu, B., and Zhang, Recurrent neural networks with transient tra- jectory explain working memory encoding mechanisms. Communications Biology, 8(1):137, 2025.
Fuster, J. M. and Alexander, G. E. Neuron activity related to short-term memory. Science, 173(3997):652-6553 1971.
Geiger, R., Lübkee J., Roth, A., Frotscher, M., and Jonas, P. Submilisecood ampa seceptor-mediater signaling at principal neuron-interneuron synapse. Neuron, 18(6): 1029-1023, 1997.
Liu, M., Nair, A., Coria, N., Linderman, S. W., and An- derson, D. J. Encoding of female mating dynamics by hypothalamic line attractor. Nature, pp. 1-3, 2024.
Liu, Z., Michaud, E. J., and Tegmark, M. Omnigrok: Grokking beyond algorithmic data. The Eleventh Inter- national Conference on Learning Representations, 2022.
Gnadt, J. W. and Andersen, R. A. Memory related motor planning activity in posterior parictal cortex of macaque. Experimental brain research, 70:216-220, 1988.
Lubana, E. S., Kawaguchi, K., Dick, R. P., and Tanaka, H. A percolation model of emergence: Analyzing trans- formers trained on a formal language. arXiv preprint arXiv:2408.1 12578, 2024.
Harrison, S. A. and Tong, F. Decoding reveals the contents of visual working memory in early visual areas. Nature, 458(7238):632-635 2009.
Harvey, C. D., Coen, P.,, and Tank, D. W. Choice-specific sequences in parietal cortex during virtual-navigation decision task. Nature, 484(7392):62-688, 2012.
Maass, W., Natschläger, T., and Markram, H. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural com- putation, 14(1 1):2511-2560, 2002.
Hochreiter, S. Long short-term memory. Neural Computa- tion MrTs-ress, 1997.
Mante, V., Sussillo, D., Shenoy, K. V., and Newsome, W. T. Context-dependen computation by recurrent dynamics in prefrontal cortex. nature, 503(77444:78-844 2013.
Ichikawa, K. and Kaneko, K. Short-term memory by tran- sient oscillatory dynamics in recurrent neural networks. Physical Review Research, 3(3):033193, 2021.
Masse, N. Y., Yang. G. R., Song, H. F,, Wang, X.JJ, and Freedman, D. J. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nature neuroscience, 22(7): 1197))1199-167,, 2019.
Isokawa, M. Membrane time constant as a tool to assess cell degeneration. Brain Research Protocols, 1(2):114-116 1997.
Jaeger, H. Adaptive nonlinear system identificatinn with echo state networks. Advances in neural information processing systems, 15, 2002.
McCormick, D. A., Connors, B. W., Lighthall, J. W., and Prince, D. A. Comparative electrophysiology of pyrami- dal and sparsely spiny stellate neurons of the neocortex. Journal of neurophysiology 54(4):782-806, 1985.
Jonas, P., Major, G., and Sakmann, B. Quantal components of unitary epscs the mossy fibre synapse ca3 pyrami- dal cells of rat hippocampus. The Journal of physiology, 472(1):615-66)) 1993.
Meyers, E. M., Freedman, D. J., Kreiman, G., Miller, E. K., and Poggio, T. Dynamic population coding of category information in inferior temporal and prefrontal cortex. Journal of neurophysiology. 10763);44771441,, 2008
Khona, M. and Fiete, R. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744- 766, 2022.
Miles, R. Synaptic excitation of inhibitory cells by single ca3 hippocampal pyramidal cells of the guinea-pig in vitro. The Journal of physiology, 428(1):61-77, 1990.
Kozachkov. L., Tauber, J., Lundqvist, M., Brincat, S. L., Slotine, J.-J., and Miller, E. K. Robust and brain-like working memory through short-term synaptic plasticity. PLoS computational biology, 18(1))e1010776, 2022.
Mongillo, G., Barak, O., and Tsodyks, M. Synaptic theory working memory. Science, 11945869):1543-1555 2008.

---

Dynamical phases of short-term memory mechanisms in RNNS
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretabilliyy. arXiv preprint arXiv:2301.05221 2023.
Spaak, E., Watanabe, K., Funahashi, S., and Stokes, M. G. Stable and dynamic coding for working memory in pri- mate prefrontal cortex. Journal of neuroscience, 6503-6516, 16, 2017.
Pals, M., Macke, J. H., and Barak, O. Trained recurrent neural networks develop phase-locked limit cycles in working memory task. PLOS Computationall Biology, 20 (2)xe101 1852, 2024.
Stokes, M. G. 'activity-silent working memory in prefrontal cortex: dynamic coding framework. Trends in cognitive sciences, 19(7):394-400 2015.
Park, C. F., Lubana, E. S., Pres, I., and Tanaka, H. Compe- tition dynamics shape algorithmic phases of in-context learning. arXiv preprint arXiv:2412.01003 2024.
Strogatz, S. H. Nonlinear dynamics and chaos: with appli- cations to physics, biology, chemistry, and engineering. CRC press, 2018.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z.,, Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. In 3/st Confer- ence on Neural Information Processing Systems, 2017.
Stroud, P., Dunean, J., and Lengyel, M. The computa- tional foundations of dynamic coding in working memory. Trends in Cognitive Sciences, 2024.
Sussillo, D. and Barak, O. Opening the black box: low- dimensional dynamies in high-dimensional recurrent neu- ral networks. Neural computation, 25(3):626 (4666664,9-9 2013.
Perich, M. G. and Rajan, K. Rethinking brain-wide interac- tions through multi-region 'network of networks' models. Current opinion in neurobiology, 15:146-151, 2020.
Thibeault, V., Allard, A., and Desrosiers, The low-rank hypothesis of complex systems. Nature Physics, pp. 1-9, 2024.
Perich, M. G., Arlt, C., Soares, S., Young, M. E., Mosher, C. P., Minxha, J., Carter, E., Rutishauser, U., Rudebeck, P. H., Harvey, C. D., et al. Inferring brain-wide inter- actions using data-constrained recurrent neural network models. bioRxiv, pp. 2020-12, 2021.
Todd, J. J. and Marois, R. Capacity limit of visual short- term memory in human posterior parietal cortex. Nature, 428(6984):751-754 2004.
Rajan, K., Harvey, C. D., and Tank, D. W. Recurrent net- work models of sequence generation and memory. Neu- ron, (0,))00())188 142, 2016.
Valente, A., Pillow, J. W., and Ostojic, S. Extracting com- putational mechanisms from neural data using low-rank mns. Advances in Neural Information Processing Sys- lems, 35:24072-24086, 2022.
Raventós, A., Paul, M., Chen, F., and Ganguli, S. Pretrain- ing task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Schäfer, A. M. and Zimmermann, H. G. Recurrent neural networks universal approximators. In Artificial Neural NetWorks-ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part 16. pp. 632-640. Springer, 2006.
Vinograd, A., Nair, A., Kim, J., Linderman, S. W., and An- derson, J. Causal evidence of a line attractor encoding an affective state, Nature, pp. 1-3, 2024.
Walker, E. Y., Sinz, F. H., Cobos, E., Muhammad, T. Froudarakis, E., Fahey, P. G., Ecker, A. S., Reimer, J., Pitkow, X., and Tolias, A. S. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12):2060 2065, 2019.
Scott, J. C., Matt, G. E., Wrocklage. K. M., Crnich, C., Jor- dan, J., Southwick, S. M., Krystal, J. H., and Schweins- burg, B. C. quantitative meta-analysis of neurocogni- tive functioning in posttraumatic stress disorder. Psycho- logical bulletin, 141(1):105, 2015.
Wang, X.-J. 50 years of mnemonic persistent activity: quo vadis? Trends in Neurosciences, (4(11))888-90) 9028888890000000 2021.
Serences, J. T. Neural mechanisms of information storage in visual short-tern term memory. Vision research, 128:53-61 67, 2016.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D.,. et al. Emergent abilitics of large language models. arXiv preprint arXiv: 07682, 2022.
Sommer, F. T. and Wennekers, T. Synfire chains with conductance-based neurons: internal timing and coordina- tion with timed input. Neurocomputingg 65-66:449-454 2005. ISSN 0925 2312. doi: https///oi.org/10 1016/ j j.neucom. 20..2.0.401..11 Computational Neuroscience: Trends in Research 2005.
Yang, G. R., Joglckar, M. R., Song, H. F., Newsome, W. T., and Wang, X.-J Task representations in neural networks trained to perform many cognitive tasks. Nature neuro- science, 22(2):297-306, 2019.
12

---

Dynamical phases of short-term memory mechanisms in RNNS
Methods
S1. Theoretical details on attractor mechanisms subserving short-term memory
S1.1. A short background on dynamical system theory
A dynamical system describes how the state of a system evolves over time. Formally, it can be represented by a differential equation of the form:
a(t) f(x(t), s<to) zo,
where zo denotes the initial state, and è' represents the time derivative of the state variable x E R
In neuroscience, researchers often examine neural firing rates IR" through the lens of dynamical systems. This approach helps uncover how temporal dynamics unfold within dimee-dimensional neural state spaces, as characterized by dynamical systems theory.
Since firing rates are themselves components of the system's state, in this study we simulate such systems using leaky firing-rate biologialy interpretable neural networks. One might argue that RNNs may not be expressive enough to learn such complex dynamics and that more sophisticated architectures like Long Short-Term Memory networks (LSTMs) or Transformers might perform better. However, this concern is addressed by the universal approximation property of RNNs, which states that, given sufficient number of neurons, even vanilla RNNs can approximate any dynamical system (Schäfer & Zimmermann, 2006). Additionally, compared to more complex models, RNNs offer greater interpretability due to their simpler and shallower architecture.
As previously mentioned, neuroscientists often study neural activity through the lens of attractor dynamics. In this framework, researchers analyze how systems evolve over time by characterizing their underlying attractor structures. As we focus on several commonly studied attractors from dynamical systems theory (Strogatz, 2018), we describe the fixed-point attractors, slow-point manifolds, and limit cycles below.
Fixed-Point Attractors: In a fixed-point attractor, the system evolves toward a specific state where it remains indefinitely. Once the state is reached, the time derivative becomes zero, i.e., i = 0.
Slow-point manifolds are particularly relevant to memory-related mechanisms in neuroscience. These manifolds describe regions of the state space where the system changes very slowly, satisfying å z O. Unlike fixed points, which trap the system, slow points allow it to persist along a trajectory over time. This property enables the system to maintain information temporarily, as illustrated in Fig. S4D.
Slow-Point Manifolds:
Limit Cycle Attractors: Limit cycle attractors describe systems that exhibit sustained oscillatory behavior. These trajectories form closed loops in state space and are often associated with rhythmic or repeated tasks. In such cases, à / O, as shown in Fig. S4C. However, as demonstrated in our experiments, limit cycles can closely approximate slow-point manifolds during the trial period.
Importantly, single dynamical system may contain multiple attractor types, and the specific trajectory follows can depend on the initial condition zo. Nonetheless, identifying and understanding these fundamental attractor structures is essential for uncovering the mechanisms underlying neural population dynamics.
S2. Theoretical studies of scaling relationships via toy models
In this section, we provide the additional details and calculations regarding the toy models we introduced in the main text.
S2.1. The ghost model
The first model we considered is the ghost model, which effectively forms slow point that can be used to store the duration of the delay before the network takes an action. This model was introduced in a prior work (Dinc et al., 2025b), whose results we briefly summarize below. Then, we generalize the methodology introduced in the earlier work, which we will eventually use to study the additional toy models we introduced this work.
13

---

Dynamical phases of short-term memory mechanisms in RNNS
12.1.1. SUMMARY OF THE PREVIOUS WORK
The toy model starts with one-dimensional dynamical system:
Tā(l) = x (1) +r,
(S1)
where x E IR is the state variable and r E IR is a learnable parameter. Here, T is a fixed time scale, which we effectively set to one such that time is computed in dimensionless units (in units T). This system is very well known in the traditional dynamical system theory literature the standard form for the saddle-node bifurcation (Strogatz, 2018). Specifically, for < the dynamical system has two fixed points x² --- the on the left being attractive and the other repulsive. However, for r 0, the system has fixed-points. Hence, "saddle-node bifurcation" is said to take place as is varied between positive and negative values (Strogatz, 2018). Since local minima of most functions can be approximated by quadratic term, this is considered the standard form for the saddle-node bifurcation and can be used to approximate the emergence/collusion of two fixed-points.
Though traditional work often focuses on changes as r varied by hand, (Dinc et al., 2025b) has taken an alternative approach and considered learning this parameter through task. Specifically, the output of the network is defined as ö(z: (z =*), where (-.) is the Heaviside function and x² is some fixed-parameter (which is later taken analytical calculations). The model is initialized zt 0) = 0. Then, the goal of the model is output 0 until some time T and then for another T times. In mathematical terms, o(t) = O - 7). Unlike the prior work (Dine et al., 2025b), since our goal is to understand the scaling of the effective learning rates with respect to the delay period lengths, we define a normalized loss function as:
in
1 T ((r) T 1. (ō(x(t)) o(t)) dt.
(S2)
It is possible to compute this loss function exactly in the limit x" oo, which is follows (Dine al., 2025b):
VT for 4
[(r)= =
(S3)
otherwise.
This loss function achieves the global minimum at r* as:
such that L(r*) = 0. One can compute the analytical gradient
47
m 377
for
167
<
17T,
x²
VL(r)=
for
x²
(S4)
477322
477
for < 167
Here, the zero gradient for r < also observed in other toy models of our interest, is referred to as a "no learning zone." Once model is caught in this region, there is no returning back.
167 is practically undesirable, since the loss function is still quite high. This flat region,
2²
Moreover, in this toy model, an interesting phenomenon emerges around the global minimum. Specifically, the derivative
4 becomes:
2T
from the left,
At = 47 VC(r)
(S5)
from the right.
These expressions differ from (Dine et al., 2025b) only in the sense that the normalization of the loss function introduces a (T)-1 coefficient to the gradient computation. Notably, this loss function has a kink at its global optimum. Thanks to this kink, (Dinc et al., 2025b) has defined maximum (effective) learning rate a* for any a >> a", the model would be thrown from the global minimum to the zero gradient regime, after which the learning would come to an halt despite non-desirable loss value of 1. This learning rate can be computed as:
3x
3rd
ag hoss V((r)
a aghast
T
(S6)
167
32
Thus, for a normalized loss function, the effective learning rate for the ghost point toy model scales following (T)))). Below, we extend this toy model with asymmetric response (Trosp) and delay (Taelay) times.
14

---

Dynamical phases of short-term memory mechanisms in RNNS
12.1.2. GHOST MODEL WITH ASYMMETRIC RESPONSE TIME
For the problem of interest, we simply replace the loss function with:
1
Jo Fdnnlyy
Tanlay Tresp
f(r)= = Tacluy
(ó(x(1)) - 0)2 dt +
(ō(x((t) - 1)2 de,
Tresp Thstay
(S7)
Fanlay
Tantay +Trepp
o(z(1)) dt + Tresp
[1 - (()
Tdelay
Thatay
As shown by previous work (Dinc et al., 2025b) (by integrating the dynamical system equations), in the limit x² + oc, the network output can be written as ō(t) = ( t"), where ta 2/F' This creates three distinct regimes of analytical calculation: i) t* I Teelay, ii) Taelay t Tdelay Tresp: and iii) Tdelay Tresp t²
Let us start with the first case, t" & Taelay (i.e., Z 4T
dalay
/ d1
Tactay
Tactay + Trmp
[(r) Teelay
o(x(1))
T
T resp
Taclay
=
=0
(S8)
Tdelay
t
π
dt
Tdelay c
Tdelay
2T v
Next, we consider the second case, Taolay S 1" < Taclay + Tresp (i.e., 4(Taotay [Trup)
7²
S S
4T ):
delay
Tdalny
1
(" [1 ((x(t)))
Tasta +
[(r)
o(x(t))
[1 - (-(xxtt))
Tdelay
Tresp Jaalay
T Tresp dc-
=0
(S9)
Taelay
Taolay
di
Tresp Taclay
T Tresp
T Tresp
resppp /r
Trep
Finally, we consider the third case, Taelay + Tresp S (i.e., r S
A(Tacloy Trrep)
Tacley o(x(t)) dt +
1 Tresp Taclar
Taolay + Tweap
C(r)
[1 1 ((z(()))
Taolay
(S10)
1
Taulasy TTeup
dt = 1.
Tresp Taclur
Bringing all these together, the loss function becomes:
for
r S T(T TTTaalo +T
C(r)
T/
π
Tanlay
Trap
for
4[Tachy + Trapp) 2 r
<
47
dolay
7
(S11)
2Taclay for r 47.
Firstly, we note that this loss function is continuous and achieves its global minimum We again compute the derivative near the global minimum:
x²
such that L(r*) 0.
47 Jeies
T²
delay Towp 27 q
273
from the left,
At
VL(r)
(S12)
47 delay
from the right.
15

---

Dynamical phases of short-term memory mechanisms in RNNS
Similar before, one can compute the critical learning rate due to the kink the global optimum. First, noting 7b 4(Taclay + Teapp?2 we arrive at:
π
x² d2Tolay + T T reesp
Testay >> T Tramp T Tresp
(S13)
4T
4(Tallyy
T rresp) )2
4T delay (Tdelay + T Tresp) )
27 delay
delay
Then, the critical learning rate can be found as:
π T resp
x=2 πt TTTrp
ghust--as
VC((I)]
2T
delay
2T
delay
4
rrsp T
delay'
(S14)
In words, if the delay period is significantly longer than the response period, the critical learning rate scales O delay) )
S2.2. Limit cycle model
Similar to a ghost model, another potential strategy for delayed response involves creating limit cycle attractors. Though there multiple dynamical system forms can use to model these attractors, we focus on a simple toy model as follows:
Tp(t) = (1 -1 (())((t)
r②(l)= = 2nr,
(S15)
where and f are polar coordinates and r is the trainable parameter as before. Here, T is fixed time scale, which we effectively set to one such that time is computed in dimensionless units (in units of 7). We assume that the system is initialized at p = I and 0- /2, and primarily focus on zt!) = p(t) cos()(t)). Below, we discuss how we train this model to solve the delayed response task.
S2.2.1. TOY MODEL SETUP
Since م 1 is a fixed point of the limit cycle system with p(t)|p=1 0 and ((t) 2xrt 2.2 the time evolution of the x-coordinate follows:
xxf) = sin (2TTt),
(S16)
where is learnable parameter. Next, define the output of this system as o(x(()) (=((() 0). For the delayed response task, we make slight modification and assume that the delay is longer than the reaction time such that o(t) O(T - - and the total task time is Trotal = Tdelay + Tresp. Then, the loss function becomes:
1
L (0(x(1) <0) 0)? di + Tresp
Tantay
Tanlay Tresp
[(r)= = Taclay
(0(x(1) <0) - 1)² dr,
Tontay
(S17)
1
Tattay
1
Tarkay + Trepy
Jo
1(sin(2xrl) < 0) dl +
(sin(22Tl) > O)
Taclay
Tresp Taolay
When Tresp S Tdeley. simple inspection reveals that the global optimum of this problem is achieved when Taeley corresponds to the half the period, i.e., r 1
2T At this value, the network outputs 0 until Taelay, and for another Taelay 2 Tresp. leading to [(r*) = O. In Fig. 3B, we show an example plot for the loss function. Notably, locally, this plot shows a kink at the global minimum and a no-learning zone for r < Tb, similar to the ghost model above. However, in reality, the global loss landscape is far more complicated and no global no-learning zone exists.
As we discuss below, it is also possible to show that local no-learning zone exists at < rb 2(Tacia as for any smaller r satisfying r > -rb, the system will only output 0 throughout the whole trial, hence a flat loss region exists with the value of for all E [-rb. Moreover, using geometrical reasoning, we can analytically compute the loss function 1 for + < r S TTme under the condition that Taelay > Tresp and for r 4T and < when Taelay T Tresp T. First, we start with the limit that Taelay = Tresp = T, and then later consider the long delay limit.
S2.2.2. ANALYTICAL CALCULATIONS WITH EQUAL DELAY AND RESPONSE TIMES
Noting that Thait 1 is the half period of the oscillations, we start by considering the regime- - - 1. in which य Thair 2 2T. In this case, x(t) = 1 or 0 for the full 2T duration, depending on whether r is positive or negative. In both cases, the loss function is L(- पт S r S + 1, Le.. flat.
16

---

Dynamical phases of short-term memory mechanisms in RNNS
Next, we consider the interval for which T Thall 2T, Le., क7 Srs 전 Then, for the times T 1 S Thals the model outputs zero, whereas it should have outputted one. For this interval, the loss function then becomes:
Thalf T
1
1 2T
[(r)
1 for 4T
Srs
(S18)
2Tr
37. In this case, the network outputs zero for halr times (making an error in T Thatt out of these), and one for another Thall, only to come back to zero for another 27 half times. Thus, in total, the network makes mistakes in 3(7 Thait) out of the 27 total times. This leads the loss function:
Finally, we consider the interval 2T S hall S T, which corresponds to 27 S r S 3
3
37halt
3
[(r)
for
S
r -
(S19)
2Tr
2T
4T
Bringing all these together, we arrive at the loss function for
47
1
r
47
3
as:
for
1
S 37
37
[(r)
27T for
47 S r 27
(S20)
for
S
2
Here, it is clear that for r* = 27. the global minimum is achieved such that L(r*) 0. We can also compute the gradient at this point as:
- -22
27 from the left,
Atr
VC(r)
(S21)
2T
6T from the right.
Next, given that rb 47 compute effective learning using the kink at the global optimum
1" rb 1
ai.
24 T-2
(S22)
[[([T]] 4T 6T
Interestingly, creating a limit cycle to solve the delayed response task leads to a better scaling with the trial time. Notably, a ghost Eq. (S6) has a much higher pre-factor than ai in Eq, (S22). This means that for small T, may be favorable to learn ghost points, whereas for large limit cycles can preferable.
S2.2.3. ANALYTICAL CALCULATIONS WITH LONG DELAYS
Now, consider the limit Tdelay > Tresp- and compute the analytical loss function for dnlay +Tresp S r Taolay TTres
Firstly, similar to before, the loss function is flat for the region 2[Taeeyy Frery S r S 2(Tasay TTapp since Thalf 2 Taelay + T Tesp. In other words, the model outputs either zeros (for 0) ones (for < 0) throughout the full window, leading to [(r)= 1.
Next, we consider the case where Taolay S Thalf S Taelay + Tresp- i.e., 2(Taslay TTepp) network outputs zero for Thalt. which is longer than Taelay and thus leading to an error contribution for times Thail Teelay Since Thale < Taelay + T resp and 2T 2 2T 2 Taelay + Tresp. the network correctly outputs one for the rest of the trial. Hence, the loss function becomes:
SrS
Tiaaay In this case, the
Thaie Taelay Tresp
Taelay
C(r)
for 2(T + Tresp)
Sr
(S23)
Tresp
2Tresp"
Treep
2T
Thirdly, we consider the case
< Thall I Taelays i.e. Tiasla Sr Tantay Tresp Since Tdelay 2 Thatr the model starts outputting ones even before Tdelay is reached. Hence, there is a contribution to the loss function for the time duration Taelay Thatf. However, since 2Thalf 2 Taclay + Tresps the model correctly outputs the ones during the response window. Hence, the loss function becomes:
Testay TTepp
2
Thalf
1 Sr 5
1
L(r)= = 1
=1
for 2Tdelay
(S24)
Taelay
2T
Taelay Tresp
17

---

Dynamical phases of short-term memory mechanisms in RNNS
Then, since hall 2 T delay the network will incorrectly output one for the duration Taeley - Thltf Moreover, since 2Thalt S Taelay + Tesp and by assumption T Teep S Ldalax (and thereby Tresp S halt)) the network will output zero incorrectly for the time interval Taclay + T resp halt.. Then, the loss function here becomes:
Finally, consider the case Tielay < halt V Himer i.e., Taelay + Teepp
Taelay
Thaif Tdelay
Taolay + Tresp half
Taolay
1
((r) =
2
for
Teesp
Tresp
Tr
reresp
Tadlay
Tresp
delay (S25)
Bringing all these together, we arrive at the (local values of the) loss function as:
1
for for for for Tactay +Tecop
t r 2(Taclay +Toop)
Tacioy T Tieop
S r. S 2Taclay
[(r)
2r
2(Taclyy +Trop)
(S26)
1 2r Taclly
2Taciay I r S Taolay + +Tccpp
2 +
Taalay
S r I Ticlay
Tresp
2rTade
rTecop
Here, it is clear that for r" gradient at this point as:
1 2Taelay"
the global minimum is achieved such that [(r*) = 0. We also compute the
1 2Tdaluy
- Tasaar
from the left,
Atr
VL(r)
Troop
(S27)
2Taelay
from the right.
Next, given that Tb- T[Taly T
compute the distance between the minimum and the no-learning zone as:
T Tresp
Trcm & Taclay
T Tresp
Tb 2Talla
2(Teclay + Tresp)
2Tally (Tdelay + Tresp)
2T 2T delay
(S28)
Then, compute the critical learning as
Tb VC(r))]
T Tresp
1
T
Tresp T
aicaas
2T 2Tdelay
delay
4
delay
(S29)
In words, if the delay period is significantly longer than the response period, the critical learning rate scales as ~ O is worth noting that another candidate for the critical learning rate can be computed from the kink r' which provides the same scaling.
delay )
Treep + Taclay
S3. Experimental details on empirical evaluations
$3.1. Simulating RNN models
In this section, describe how implemented the update rule for the leaky firing-rate RNN defined in Equation (1). Since the experiments are conducted in discrete time, we use the forward Fuler method to update the firing rates:
r(t + 1)= af(((t), u(t)
(S30)
Here, denotes the firing rate, f(-) corresponds to the update function defined in Equation (I), and a is the step size, computed as a = At/r. The update function can be written more explicitly as:
r(t + 1) (1 r() + tanh(Wr(t) W u(t) + +b+
(S31)
With this update rule, we can compute the firing rate at the next time step. The only remaining component is computing the gradients from the predicted output ((t) to the target output o* (t), as described in Sections $3.2 and $3.3.
18

---

Dynamical phases of short-term memory mechanisms in RNNS
S3.2. Task details
Short-term memory is fundamental cognitive process essential for mammalian survival. Several well-established studies have proposed theories on memory maintenance that have deepened our understanding of short-term memory, illustrated in Fig. 1. However, while these theories focus solely on memory maintenance and response processes, they do not fully explain the underlying mechanisms. This knowledge gap raises a critical question: Which mechanisms govern short-term memory in the brain? To address this question, we investigate short-term memory through two key properties: memory maintenance and information processing. Accordingly, we conduct experiments on two well-ostablished neuroscience tasks: the delayed activation and delayed cue-diserimination tasks. We determined these tasks because the first task exclusively examines memory maintenance, while the second incorporates both information processing and the retention of input stimuli generate the relevant response. Furthermore, to enhance transparency and ensure the reproducibility of experiments, we provide detailed outline of all task properties in this section.
S3.2.1. 1. DELAYED ACTIVATION TASK
The delayed activation task allows us to isolate the memory component by assuming that cue processing has already occurred.
In this setup, the task consists of three distinct periods: the delay period (Tdelay), ), the reaction period (Tresp), and the optional post-reaction period (T postt) Since the cue period omitted, the input remains zero throughout the trial, making the task entirely dependent on the intrinsic dynamics of the recurrent model rather than external stimuli.
The expected ground truth responses, denoted as O* for oz 03. ..., of are defined over the total duration T Tdelay + resp + T post as follows:
1, if E Tresp
of =
(S32)
0, otherwise
This task is inherently simpler than the delayed que-diserimination task, as it isolates the challenge of maintaining information in memory without requiring cue processing. By setting the input internal state dynamics correctly generate the expected response during the reaction period. The optional post-reaction period allows for additional investigation into how the network dynamics stabilize after completing the required response.
at all times, the model must rely solely its
S3.2.2. DELAYED CUE- DISCRIMINATION TASK
In contrast to the delayed activation task, the delayed cue-discrimination task involves two input channels and two output channels. It consists of four distinct periods: the cue period (Tin); delay period (Taelay), reaction period (Tres)): and an optional post-reaction period (Tpost) This task allows us to investigate both stimulus processing and memory components simultaneously.
The input cue for the delayed cue-discrimination task, denoted as
U = ((1, vi), (uh, wz), (ul, vz), .... (uz, )
defined over the total duration T= Tin + Taelay + Tresp + T poot as follows:
or or ift Tin
U1 (0.0),
(S33)
otherwise
The expected ground truth outputs, denoted as are defined as:
O = o.)), oz (,)(())))) (o51 oz (o77 of
19

---

Dynamical phases of short-term memory mechanisms in RNNS
o = Woon (0,0), otherwise
if E Tees
(S34)
This task requires the model to process the cue presented during the cue period, maintain it through the delay period, and then generate the appropriate response during the reaction period. The optional post-reaction period allows for additional investigation into how the network dynamics stabilize after completing the required response.
S3.3. Training details
To ensure the reproducibility of our experiments, we provide a comprehensive description of the training configurations used in this study. All recurrent neural network (RNN) architectures were implemented using the PyTorch framework. Unless stated otherwise, models were trained using the stochastic gradient descent (SGI) with the momentum value of 0.9 and weight decay value of 10-7, Most of the experiments were conducted on computing systems equipped with an Intel 19-10X CPU and Apple M2 CPU. To further enhance transparency and support future research, we will publicly release complete training pipeline GitHub.
Training with long delay intervals, i.e., Tdelay > Tresp presents challenges when compared to shorter delay times due to the model's tendency to learn to always output 0. This issue arises from an inherent imbalance in the loss function: since the model is expected to output 0 for the majority of the trial and only 1 during the response period, a naive mean-squared error (MSE) loss would favor minimizing errors in the dominant interval, leading to trivial solutions where the model outputs at all times. This imbalance aggravated when the total trial duration Tia Taelay Tpost is significantly greater than Tresp.
To mitigate this issue, we trained all RNNs using weighted MSE loss, where different time intervals were assigned corresponding weights. The weight function was defined as follows:
10
if E Tresp
WMSE
10
Tima"
otherwise
(S35)
Tams
This weighting scheme ensures that the loss function does not disproportionately favor the dominant 0 responses and appropriately emphasizes the importance of correct responses during the reaction period.
S3.4. Details on large-scale experiments
In this study, we conducted several short-term memory experiments investigate underlying memory mechanisms. Specifically, we examined rank-2 RNNs trained the delayed activation task, both with and without post-reaction period (Fig. 2, 3, and S1). To extend our observations to full-rank RNNs, increased task complexity by training models on the delayed cue-discrimination task, again with and without a post-reaction period (Fig. 4, S3, and S4). Additionally, to explore the role of the time constant r and memory- reeattdd components, we evaluated full-rank RNNs under various configurations (Fig. S5 and S6). Further implementation and training details are provided in Sections S4 and S3.3. As a summary, the configurations and number of total runs for each experiment are presented below:
Table S1. Lcrge-scale experiment contigurations and the number of total runs.
RNN
TASK
LEARNING RATE
Taelay
# SEEDS
EXPERIMENTS
RANK-2
DELAYED ACT. w/wO Tpost
5 10-3
30 ms
100
1600
FULl
CUE-DISC. wo/ Taelay
log(10 3,1] (20)
Tresp
lin [20, 400) ms (20)
20
8000
FULL
CUE-DISC. w/ LONGER T
log[10- (10)
lin 800) ms (21)
20
4200
FULL
CUE-DISC, WOl Tpost
log[10
10 (15)
10
lin[0, 400) ms (21)
100
31500
FULL
CUE-DISC, WOLT T Trost
log [0 1.6 10 )
(15)
[in[0, 400) ms (21)
100
31500
FULL
CUE-DISC. W/ Tpost
log
10 10
10- (15)
(15)
[in 400) ms (21)
20
6300
20

---

Dynamical phases of short-term memory mechanisms in RNNS
S3.4.1. DETECTING LEARNED MECHANISMS SUBSERVING SHORT-TERM MEMORY IN RNNS
We designed Cetrics-Reaction Accuracy (RA) and Reaction Reliability (RR)-to evaluate the performance of our recurrent neural network (RNN) models. Reaction Accuracy quantifies the model's ability to discriminate the input cue by identifying the class corresponding to the maximum output channel at each time step, while Reaction Reliability assesses the degree of signal distinction between the classes.
More formally, let O E IR x2 represent the model's output vector over the reaction interval, and G E xx denote the ground truth labels over the same interval, where is the total number of time steps.
Reaction Accuracy (RA): This metric assesses how accurately the model produces the correct response during the response period (Tres)). : computed as follows:
N arg max(O;) - G.
RA
(S36)
Here, arg (Of returns the index of the output channel with the highest activation at time step t, and Ge denotes the ground truth class label that time.
Reaction Reliability (RR): This metric complements Reaction Accuracy by not only evaluating whether the model selects the correct response during the response period T resp: but also by measuring how confidently does so. is computed as follows:
1
N
OLG E Of
RR
(S37)
Here, Or,G, denotes the model's output corresponding the ground truth class G, at time step t, and the denominator normalizes this value by the total output across both output channels at that time step.
Mechanism Discrimination Index (MDI): The two metrics, RA and RR, allow us determine whether the model has successfully learned the task. If model fails to learn, we categorize its behavior as either "Unstable Training' or 'Final Model Failure', as illustrated in Fig. S3A-B and Fig. S4A. However, these metrics alone do not clearly distinguish between limit cycles and slow-point manifolds (see Fig. S3A-B and Fig. S4C-D). Therefore, to reliably differentiate these two mechanisms, we further analyze the frequency distributions of correlation coefficient matrices computed from the first two principal components the models' firing rates, as shown in Fig. S3C.
Since limit cycles closely approximate slow-point manifolds during the trial period, it is challenging distinguish them based solely on firing rates observed within that interval. To overcome this limitation, we extend the analysis window by concatenating the post-reaction period during inference, thereby revealing the underlying dynamical structure of the firing rates, as illustrated in Fig. S4C-D. Next, we apply principal component analysis (PCA) to the firing rates and extract their first two principal components (Fig. S4). We then compute the Pearson product- momunt correlation coefficient matrix from these principal components.
Due to the repetitive structure of limit cycles, their correlation coefficient matrices exhibit predominantly high-frequency components. In contrast, slow-point manifolds produce correlation matrices characterized by consistently high correlations, corresponding predominantly to low-frequency components. After calculating the frequency components, we define hyperparameter to establish a threshold for discrimination between low-frequency and high-frequency, and compute the Mechanism Discrimination Index (MDI) as follows:
MDI PSD
(S38)
PSDrotal
where PSDif is the power spectral density of low-frequency components, and PSDrotal is the total power spectral density. Using this approach, we reliably discriminate between limit cycles and slow-point manifolds, as demonstrated Fig. S3C.
21

---

Dynamical phases of short-term memory mechanisms in RNNS
As summary, defined four categories classify the outcomes of RNN training:
Final Model Failure: An RNN is considered to have failed to converge if its mean accuracy and reliability have never previously reached exceeded 0.8 (Fig. S4A).
Unstable Training: Training is classified as unstable if, after initially achieving accuracy and reliability values above 0.8, the performance subsequently falls below 0.6 for more than 10% the epochs following the initial success (Fig. $4B).
Limit Cycle: An RNN is classified exhibiting limit cycle if its mean accuracy and reliability during the final 10% of epochs are 2 0.8, and the Mechanism Discrimination Index (MDI) is less than 0.5 (Fig. S4C).
Ghost Point: An RNN is classified as exhibiting ghost point if its mean accuracy and reliability during the final 10% of epochs are 2 0.8, and the Mechanism Discrimination Index (MDI) is greater than or equal to 0.5 (Fig. S4D).
The first two categories collectively constitute the "Not Learned" classification.
S3.4.2. PLOTTING THE SCALING LAWS
To investigate the scaling relationship between the learnable delay interval and the learning rate, we performed the following analysis.
First, for each learning rate, we extracted the model's performance across different delay intervals from the large-scale experiments (Fig. 4B, E). We then fit saturating eunetion-either an exponential saturation or sigmoid-to the performance curves over delay (Fig. S3D, E). This yielded a cutoff delay value a", representing the point at which performance saturates or exhibits sharp transition. Specifically, the exponential saturation fit was defined as:
so [1 11-xxp(-a(z expp-a((x zo))
if < mm
f 2o.
(S39)
otherwise
The fits were applied to performance metrics (e.g., reaction accuracy reliability) over delay intervals using scipy.optimize. curve fit.
Next, we analyzed how the cutoff delay values a" scale with learning rate a. We performed a log-log linear regression of the form:
log(a) = 5 log(a") log(A)
(S40)
The critical exponent / was estimated using scipy. stats. linregress and validated by an additional log-log via curve fft. To estimate confidence intervals, we performed Monte Carlo sampling using the parameter covariance matrix. The resulting fit and its 95% confidence interval were visualized log-log axes, shown in Fig. 4C, F and Fig. S3F.
This analysis enabled us to extract empirical scaling laws for slow-point manifolds and limit-cycles, which were then compared to theoretical predictions.
S3.4.3. PLOTTING FIRING RATES
To visualize firing rates, we collect the hidden states of the leaky firing-rate RNN during inference. To reveal the underlying dynamical structure, we extend the trial by appending post-reaction interval (Pport) that is nine times longer than the original duration, defined as T Tin + Taelay + Tresp-
After collecting the full firing rate trajectory, we sort neurons based on the timing of the peak activation in their absolute firing rates. Specifically, each neuron's 's activity is scanned to identify the time step at which its firing rate reaches its maximum value. Neurons that do not exceed a predefined activation threshold are pushed to the end of the ordering. This procedure ensures that neurons are visualized the order in which they become active, highlighting sequential structure in the network's 's dynamics.
22

---

Dynamical phases of short-term memory mechanisms in RNNS
S4. Reproduction details
To investigate the emergent cecchanisms--secificclll slow-point manifolds and limit cycles-we conducted experiments using low-rank recurrent neural networks (RNNs) (Background). Given that minimum of two dimensions required for the formation of limit cycles, we set the low-rank dimensionality to K 2. The delayed activation task was chosen the training paradigm. The learning rate was initialized at a = 10-2. Additionally, we set the time decay constant to = 10 ms, the time step to Al = 5 ms, the delay interval to Taclay 150 ms, and the response interval to T Tresp = 50 ms. The architecture was trained for 150, 000 epochs with a total of N = 100 neurons. All weights and biases, as described in Eq. (2), were optimized throughout training. For this experiment, noise was set to = 0, and firing rates were initialized from a multivariate Gaussian distribution with mean μ4, = and standard deviation o, = 0.1. Training was performed using stochastic gradient descent (SGD) with PyTorch's s default hyperparameters.
Figure 2:
Figure 3: This figure highlights the impact of even slight changes in task configuration on network dynamics. To demonstrate this, we trained identical RNNs with N 100 neurons and rank of K 2. by increasing the post-reaction period interval from T post Taelay 30 ms, Teep 10 ms, and For the experiments in Fig. SIC, we set 5 x 10-3 and trained the networks for 500, 000 epochs while varying the delay interval as Tdelay E {30, 40, 50, 60, 70} ms. Moreover, SGD is used with the default hyperparameters of PyTorch.
ms to T post 30 ms. We applied similar configuration to Fig. 2A, except for setting
10-3 for the experiments in Fig. S-A-B, training the networks for 10 epochs.
In large-scale RNN experiments, we developed primary training pipeline to systematically investigate the effects of learning rate, delay interval length, and post-reaction interval. Apart from these variations, all other training configurations in Fig. 4 were set as follows: number of neurons N = 100, noise initialization drawn from a Gaussian distribution with mean fre = O and standard deviation d, = 10-3, and firing rate initialization drawn from a Gaussian distribution with mean J4, = 0 and standard deviation a, = 10-1, Additionally, task-related parameters were fixed, with the cue interval set to Tin 30 ms, the reaction interval to Treep = 50 ms. the time decay constant to 10 ms, and the time step to At = 5 ms. Since lower learning rates require longer training durations, we implemented adaptive schedule for the number of training epochs based on the learning rate. The number of epochs was determined using the following equation:
Figure 4:
num_epochs = max 103, 30 a
(S41)
The delay interval was gradually increased in increments of 20 ms, covering the range Tdelay E [0,420] ms. In our initial experiments, learning rates were sampled from a logarithmic scale with equal spacing in the range a E [10-3, 10-1). To further investigate the effects of intermediate learning rates, we additionally sampled more values logarithmically within the range [10-1.6, 10 10-0.5] (see Table S1 for details). Besides, Fig. 4C,F presents the estimated cutoff delay values, which are described in detail in Section 2.5. Moreover, in Fig. H, I, we constructed phase diagram of emergent mechanistic phases derived from the results shown in Fig. 4C-D.
Figure S2: We analyze the emergence of the limit cycle model in the delayed activation task through analytical methods (Supplementary 1.2). Figure S2A illustrates the vector field of the dynamical system, which is derived from Eq. (S15) with r = 0.1. Additionally, Figure 4G presents two loss curves: the analytical curve is obtained from Eq. (S17), while the numerical curve is computed using Eq. (S20), with parameters set to Taelay = 60 ms, Tresp = 10 ms, and di = 10-2 Finally, Figure S2B is constructed by plotting analytically derived scaling laws, with the response interval set to Tresp = 10 ms....
Figure S3: This figure builds on the results presented in Fig. 4 to illustrate the RA and MDI metrics, Additionally, highlights the scaling laws associated with slow-point manifolds observed in the large-scale RNN experiments.
Figure S4: We randomly sampled the example classes of dynamical systems from the large-scale RNN experiments.
Figure 55: In this experiment, we use the same training configuration as in Fig. 4, except for the delay intervals and the value of r. We set 5 = 20 ms and sampled Taelay from the range [0, 840] ms in increments of 40 ms. (See Table S1 for more details.)
23

---

Dynamical phases of short-term memory mechanisms in RNNS
Figure S6: In this figure, the only differences from the large-scale experiments that the number of training epochs is set to 3000, Tadlay = 0 ms, and the response interval resp is varied instead. This setup allows to examine the importance of the memory component. (See Table SI for more details.)
The code to reproduce these figures is publicly available at http://gtthbbbbommmaaiiiiccccaaa pta.sissssl.sssssss..ssss the data jointly with the code at https://doi.org/10.553.. 1/22//////ooo111510
24

---

Dynamical phases of short-term memory mechanisms in RNNs
Figures and captions
25

---

Dynamical phases of short-term memory mechanisms in RNNS
A
Training a rank-2 RNN
B
Epoch 40,000 (SP manifold)
Epoch 67,000 (Transition)
Epoch 130,000 (Limit cycle)
on a delayed-activation task
latent variable(a.uu u.).) Outputs
T Bokay
X Trial start
Target
Network
Normalized loss function values
Response start
Activity of the second
Trial end
10"1
Latent state (during trial)
10
Latent state (after trial)
10a
Decision boundary
Number of training epochs
Activity of the first latent variable (a.u.)
Figure SI. RNNs can switch between distinct short-term memory mechanisms during training. A To illustrate that different mechanisms can employed by dynamical systems delay their responses, we trained rank-2 RNN to perform the delayed activation task. In this task, networks initialized at given state (Methods) should delay their output by Taelay- Parameters: N 100 neurons, T = 10ms, Åt 5ms, Taslay = 150ms, Tresp = 50ms, a = 10 -2 for stochastic gradient deseent using otherwise the default parameters in PyTorch (Paszke et al., 2017). The dotted lines correspond to epochs, for which the latent dynamical systems illustrated in Panel (B). During the training, two distinct mechanisms emerge in the RNN, which we probe by plotting the two-dimensional latent dynamical systems. Left. Early in the training. the network lowers the loss values by designing line of slow-points (referred SP manifold). towards which other stats are attracted. The latent states after the trial concludes end up in final attractive fixed-point. Middle. As training progresses, the SP manifold slowly dissolves into circular pattern, which deem settles into the final solution. Right. After further training, a limit cycle emerges in the latent dynamical system. In this structure, even after the trial concludes, latent states continue evolve periodic manner. The shaded areas around decision boundary correspond to network outputs within 0. 75 (outer green boundary) and 0.25 (outer red boundary).
transient mechanism before the network
26

---

Dynamical phases of short-term memory mechanisms in RNNS
Second model variable (y)
output = 1
output 0
T = 60 and T = 10
delay
reep
Analytical curve Numerical data
Normalized loss function values
0.01 Parameter values (r)
First model variable (x)
Figure S2. A theoretical scaling analysis of short-term memory mechanisms via toy models. The illustration of the flow map corresponding to the toy limit cycle model for = 0.1. The red circle denotes the attractive limit cycle, the purple line corresponds to the decision boundary of the output, and the green circle denotes the starting point of the state. B An example loss curve for the limit cycle toy model. with analytical curve and empirical data points. Note the kink at the global minimum. C The different dependencies of the effective learning for types of mechanisms create phase space function of delay and learning rate values. In general, there are three main phases of short-term memory mechanisms for large Taelay >> Tresp. For this figure, we picked Tresp 10ms.
27

---

Dynamical phases of short-term memory mechanisms in RNNS
A
B
C
Memory
mechanism
Frequency (normalizzed,
40
Loss function values
Final model fails
Reaction accuracy
Unstable learning
Limit cycle
Limit cycle
SP manifold
SP manifold
0.5
1000
1000
0
Number of epochs
Mechanism discrimination index
E
F
failing to solve the task
Learning rates
developing SP manifolds
Learning rates
0.316
0.316
Scaling of SP manifolds
Fraction of RNNs
0.019
Fraction of RNNs
0.019
Learning rates
10
0.001
0.001
10a
B= 4.05 0.10,
R 0.99, p<10
10
200
400
200
400
30
100
300
Delay (ms)
Figure Details of large-scale RNN experiment. Different memory mechanisms exhibit distinct loss curves during training. While limit cycles and slow-point manifolds successfully converge to local minima. the "Final Model Failure" and "Unstable Learning" classes do not achieve stable convergence due to high learning rates. As further illustrated, only the limit cycle and slow-point manifold mechanisms achieve consistently high reaction accuracies. C The Mechanism Discrimination Index (MDI) (see Methods) allows reliable differentiation between limit cycles and slow-point manifolds, even though their dynamics appear similar during trial period. D Lowering the learning rate enhances the model's ability to learn tasks with longer delay intervals. The solid curve represents the corresponding exponential saturation (sec Methods), Additionally, decreasing the learning rate increases the likelihood of models adopting slonttoint manifolds. As predicted analytical models, results from the large-scale RNN experiments exhibit similar power-law relationship (Trelay 2 Taslav 2 Thllay for learning slow slow-point manifolds for large learning rates. However, for larger delays (Taelay 100ms), these solutions preferentially suppressed, presumably in favor of the limit cycle based solutions.
28

---

Dynamical phases of short-term memory mechanisms in RNNS
X Trial start
A
Latent state in trial
Reaction metrics
Network outputs
Response start
Absolute firing rates
Latent state after trial
Trial
Accuracy Reliability
100
Class: Final
model faliure
Output
Neuron ID
PC (a.u.)
Output 2
Target
0
1000
2
Training epochs
Time (s)
Time (s)
1 (a.u.)
B
100
Class: Unstable learning
Output 1
Output
Neuron ID
PC (a.u.)
Target
Accuracy
Reliability
1500
Training epochs
Time (s)
Time (s)
PC 1 (a.u.)
C
100
Class: Limit cycle
(a.u)
Yeuron ID
Accuracy
Reliability
1000
Training epochs
Time (s)
Time (s)
PC 1 (a.u.)
100
Class: Slow-point manifold
Output
Output 2
Neuron ID
PC 2 teau)
Target
Accuracy Reliability
0
3000
0
2
Training epochs
Time (s)
Time (s)
PC 1 (a.u.)
Figure $4. The four main classes identified in the large-scale RNN experiment. This figure illustrates representative examples from each four primary classes: A Final model failure, Unstable learning. C Limit cycle, and D Slow-point manifold. focused on trained models from experiments with delay interval of 180 ms, which specifically selected because it exhibits all four classes, as illustrated Fig. 4B. The first column presents the reaction metrics. Lee. Reaction Accuracy and Reaction Reliability (see Methods), The sccond column shows the network's outputs alongside the corresponding ground truth signals. The third column displays the firing rates individual neurons. The final column illustrates the first two principal components derived from these firing rates. Training parameters used for each representative example a 0.01, seed = 85. Additional parameters shared across all models are Tin = 30 ms, Tastay = 180 ms, Tresp 50 ms. Al 5 ms. 10 ms, Cupise 0.001, a. 0.1, and hatch size per channel 64 (see Methods).
follows: A 0.1, seed 93; B z 0.01931, seed 41; Ca O.I. seed 82: and
29

---

Dynamical phases of short-term memory mechanisms in RNNS
A
B
C Limit cycle
No learning
SP manifold
Memory
Loss function values
0.1
mechanism
Learning rate (logscale)
10
Final model fails
Fraction of RNNs
Unstable learning
Limit cycle
SP manifold
10-a
10
Number of epochs 1000
0
200
400
600
800
0
400 Delay (ms)
800
Delay (ms)
Figure RNN experiments with increased т. this figure, we examine effect of increasing the time constant to 20 ms while keeping all other hyperparameters identical to those used in the large-scale delayed que-discrimination task experiments (Fig. 4). These experiments reveal that increasing does not alter observed scaling laws. As Fig. S3, we observe similar loss curve patterns across all classes under the larger setting. B Consistent with previous results, the distribution of dynamical classes across different random seeds remains largely unchanged with the increased 7. C The phase diagram for 20 ms shows scaling laws consistent with those observed in the smaller regime.
30

---

Dynamical phases of short-term memory mechanisms in RNNS
A
B
Loss function values
Network
Leamingg rate (logscale)
10
performance
Learning rates
10-1
Not learned
10
Scaling of the no-solution boundary
Correct
= 0.38 0.02, R =
10-1
p=10
1
10
1000
2000
3000
200
400
40
100
400
Number of epochs
Response window (ms)
Figure S6. Scaling laws disappear in fixed-response task experiments with RNNs. In this experiment, we remove the delay period from the delayed cue-discrimination task isolate the effect of memory and assess its role in scaling laws. As shown in the figure, the removal of the delay component eliminates the previously observed scaling hehavior, highlighting memory as critical factor underlying these laws. A As in previous tasks, observe qualitatively similar loss curves between learned and correct networks. B Phase diagram showing the classification of not learned and correct networks in the fixed-responss task. The x-axis represents the length of the response interval, during which the model required generate consistent output over time. Notably, removing the delay period (and thus the memory requirement) leads to substantial loss of scaling structure. C We compute the scaling law using the same methodology Fig. 4C,F and Fig. S3F.
31


Reviews:
[{"review_id": "TERd4LvgDl", "reviewer": "Reviewer_pHmu", "summary": "This paper studies computational RNN models of a classic neuroscience working memory task–the delayed response task–along with two, very simplified and tractable, dynamical system models capable of learning the task through adaptation of a scalar parameter. The paper studies the role that changes in the delay time, the response time, and an optional post-response period (where no output from the given model is expected) time play in how the models studied learn to solve the task. The main results of the paper is that RNNs tend to learn two different solutions to the task, one making use of a slow point the other a limit cycle, and that the solution learned will depend on the learning rate, the length of the delay and response periods, and the presence of a post-response period. It is observed that for longer delays and higher learning rates a limit cycle tends to be learned instead of a slow point solution. Interestingly, the loss function of a toy dynamical system model (the normal form of the saddle-node bifurcation) of learned slow-point dynamics requires a smaller learning rate or delay period to be learned, while a toy dynamical system for the limit cycle dynamics (sine function) is capable of learning with faster learning and longer delay periods. This relationship of better scaling w.r.t. learning rate and delay period for limit cycle solutions is proposed to underlie the phenomena observed in RNNS. Interestingly, the observed scaling in trained RNNs on the task without the post-response period is roughly what was predicted by the toy model. Lastly, for the numerical experiments, a very large sample of RNNs was trained and the authors plan to make these publicly available to facilitate future work.\n\nThe authors argue that their analysis of the learning of delay periods sheds light on the difficulty of learning long time dependencies in machine learning, and that it will help inform future neuroscience research by demonstrating how task parameters can have a significant effect on learning.\n\n## Update After Rebuttal\nI am satisfied by most of the authors clarifications and believe the paper is relevant to neuroscience and RNN-related learning and have thus updated my score to a 4. The remaining issue for me is that I still believe the authors could have done a better job connecting the toy models and the low-rank RNNs mathematically; this is why I have not rated the paper a strong accepted. Lastly, please disregard my mistaken replacement of $\\dot{x}$ with $x$ in one of the comments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and highlighting our approach's strengths—especially our effort to simplify a complex problem into an interpretable framework. Their suggestions have greatly shaped our revisions. Below, we address all specific concerns and weaknesses.\n\n**Q1** That is an excellent point. As the reviewer notes, models based on dynamic synaptic variables like STSP represent a fundamentally distinct mechanism from the activity-based attractor models we focus on. In an earlier draft, we mentioned both classes—such as Mongillo et al. (2008)—and then narrowed the focus to activity-based models (Fig. 1). Based on the reviewer’s feedback, we restored and expanded the neuroscience context. In addition to intro, discussion mow highlights that STSP mechanisms may allow networks to bypass the learning constraints we identify for fixed-weight attractor models (e.g., Masse et al., 2019). The Conclusion links to recent AI models like Mamba, which integrate memory via local mechanisms that may functionally resemble STSP.\n\n**Q2** This point is addressed in our reply to Reviewer CYoj.\n\n**Q3** A key motivation for releasing all trained models is to enable reuse without retraining. We now clarify that the dataset required ~1 month of GPU time. Due to space constraints, we briefly outline three research directions this dataset enables (with more to be added in the Outlook):\n\n- Curriculum learning: Training pre-trained models for longer delays with reduced learning rates may enable faster convergence and offer insight into curriculum/transfer learning. The toy models could support new curriculum strategies with theoretical guarantees—relevant for experimentalists (including co-authors) who find it difficult to train mice on seconds-long delay STM tasks.\n\n- Model initialization: Exploring how weight initialization affects convergence to slow-point vs. limit-cycle strategies. For example, initializing near an existing solution (e.g., SP) may bias learning toward that strategy and potentially shift phase boundaries (Fig. 5).\n\n- Robustness: Comparing robustness by perturbing neurons (e.g., mimicking optogenetics) may reveal key differences (oscillations vs decay) between slow-point (SP) and limit cycle (LC) strategies.\n\nWe also refer the reviewer to our response to Reviewer 59aj’s second question for more on the dataset’s utility.\n\n**Q4** Testable predictions. Our work predicts that both SP and LC dynamics can generate sequential neural activity, but differ in structure: SP dynamics converge to a slow-point attractor by trial end, whereas LC dynamics continue producing trial-type-specific sequences beyond reward. This distinction is testable by analyzing neural activity after task completion. While Rajan et al. (2016) explored sequences in RNNs, we link them to distinct attractor types. For instance, Fig. 4 predicts that extending the post-response period biases dynamics toward LC (oscillatory) or SP (ramping) solutions. Although this period can’t be fully removed in experiments, it can be disrupted—e.g., via bulk optogenetic inhibition of task-relevant regions, providing a plausible way to test this prediction.\n\nRelation to recent studies: We will cite the suggested works and place greater emphasis on neuroscience connections (previously omitted due to space). The reviewer’s interest—as a neuroscientist—strongly encouraged this expansion.-\n\n**Q5**  We thank the reviewer for raising this thought-provoking possibility of a third dynamic regime—one in which a network operates in a limit-cycle (LC) mode during the task but undergoes a modulatory shift in excitability toward the end of the trial that disrupts the remaining cycle, effectively mimicking a slow-point (SP) solution in the end. This is indeed how we would expect LC solutions to be utilized in practice. Distinguishing such a modulated LC from a true SP is an exciting experimental challenge and could reveal important aspects of synaptic learning rules and neuromodulatory control. We outline three levels of experimental inquiry to test this hypothesis in vivo (briefly):\n- Exp 1: The assumption of functional reorganization can be tested by combining constant-power bulk optogenetic activation with - calcium imaging at different trial stages.\n- Exp 2: Genetically encoded fluorescent sensors (e.g., dLight, GRAB-ACh) can be used to track neuromodulatory signals during task performance. A signal peaking in the post-response window would support the proposed mechanism.\n- Exp 3: To establish causal roles, one could block or genetically delete the relevant receptors and observe changes in post-trial neural dynamics or task performance.\n\n**Final remarks** Thank you for the style suggestions and pointing out the typos. We incorporated all suggested edits and reviewed the manuscript for clarity, consistency, and polish. Please let us know if any points need clarification. We sincerely appreciate the reviewer’s enthusiasm and attention to detail.", "claims_and_evidence": "In the reviewer’s view the claims seem well supported.", "methods_and_evaluation_criteria": "Yes, they seem appropriate.", "theoretical_claims": "I checked the proofs in section S1.1 and they appear sound.", "experimental_designs_or_analyses": "I did not check any of the code. The experimental designs seem appropriate for the questions that are being asked.", "relation_to_broader_scientific_literature": "There are several recent papers studying the “curse of memory” (difficulty learning long-timescales) in RNNs, the effect that long-memory tasks have on learning, and how to ameliorate it, that the reviewer believes could be relevant to cite: \n- Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks – Li et al. (2022) JMLR\n- Recurrent neural networks: vanishing and exploding gradients are not the end of the story – Zucchet & Orvieto (2024) NeurIPS\n- Generalized teacher forcing for learning chaotic dynamics – Hess et al. (2023) PMLR\n\nThe author also believes it could be useful to discuss models of working memory that rely on dynamics in variables other than firing rate, see for example the reference in Q1 of “Questions for Authors” section.", "essential_references_not_discussed": "The reviewer is reasonably well versed with current neuroscience models of working memory (Working models of working memory – Barak & Tsodyks (2014) Current op. In Neurobiology). \n\nThe researcher is currently engaged in research on learning in the neuroscience-related firing-rate RNNs studied in the paper, and is therefore quite familiar with relevant literature, including:\n- Flexible multitask computation in recurrent networks utilizes shared dynamical motifs – Driscoll et al. (2024) Nature Neuroscience\n- Generating Coherent Patterns of Activity from Chaotic Neural Networks – Sussillo & Abbott (2009) Neuron\n- A neuronal least-action principle for real-time learning in cortical circuits – Senn et al. (2024) eLife\n- Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics – Qian (2024) biorXiv,\n\nalong with the papers mentioned above", "other_strengths_and_weaknesses": "## Strengths\nThe paper does a fantastic job distilling a difficult problem down to a model that is simple enough for mathematical analysis but still seems to capture certain key aspects of the problem. The insights into two different strategies an RNN can use to solve a delay task, and how task structure and learning rate can influence these, are interesting and, in the reviewer’s view, worthy contributions.\n## Weaknesses\nThe two main weaknesses, in the reviewer’s view, are: (1) that the simple dynamical systems studied are a substantial departure from an RNN and little justification is provided for the choice of simplified systems (Question 2, below); (2) how machine learning connections are primarily discussed in the discussion when the paper seems more relevant to computational neuroscience (Question 4, below); (3) the lack of discussion of certain alternative models of working memory (Question 1, below).\n\nNote: the reason for the score is because the reviewer would like to see their comments and questions addressed before being able to recommend it for acceptance. If these are addressed satisfactorily the reviewer will certainly increase their score.", "other_comments_or_suggestions": "- In the abstract: the reviewer finds “limit cycles providing temporally localized approximations” a little confusing, and wonders if clarity could be increased for this sentence.\n- In the abstract, the authors mention: “we derive theoretical scaling laws for critical learning rates as a function of the delay period length, beyond which no learning is possible.” It could be worth specifying that this derivation is in simplified dynamical system models rather than directly from RNNs\n- Line 38, RHS: “studied” => studying\n- Line 45, RHS: “periodicity” => periodic\n- Line 73: LHS: I suggest: “Using interpretable dynamical system models stripped down to their most essential components for solving a delayed activation task” => “Using low-rank RNNs”. More concise and avoids confusion with analytical dynamical system models also studied in the paper\n- Line 91, RHS: I suggest changing $W/W^{in}$ to $\\{W, W^{in}\\}$ to avoid notational confusion with division\n- Equation 3: “$x$” => $\\dot{x}$\n- Line 359 RHS: exponents 2 and 3 should be -2 and -3.\n- Line 663: “an halt” => a halt\n- Eq S13: $T_{delay} >> T_{delay}$ => $T_{delay} >> T_{resp}$", "questions_for_authors": "1. To the reviewer’s understanding there is an alternative hypothesis for encoding working memory: that of short term plasticity (e.g. Working models of working memory – Barak & Tsodyks, 2014; see section on “Short term synaptic plasticity). Could the authors add mention of this hypothesis to the paper, or provide a compelling argument as to why it is irrelevant?\n2. While compelling empirical evidence is provided for the relevance of the toy dynamical system models studied to learning in RNNs, the toy models are not rigorously connected to RNNs–at least beyond the sentence “It is worth noting that increasing the dimensionality of the dynamical system can allow more efficient solutions to the system, but the toy models we discuss can be thought as approximate bounds on what can be achieved.” Could the authors provide some deeper insight into this limitation, and the differences they expect between the behaviour of the toy models and of RNNs? This could be a useful limitation to include in the discussion.\n3. The authors suggest that a key impact of their paper is the number of models the paper makes available. The reviewer wonders at the utility of providing 35,000 models all trained on a simple delay task. Could the authors propose different examples of ways this dataset could be used? The reviewer would also be curious about the environmental impacts of training this many models.\n4. From the reviewer’s perspective this paper is primarily focused on learned computational mechanisms for solving a continuous-time delayed response task and therefore more relevant for neuroscience than machine learning. However, the authors spend the vast majority of the discussion talking about machine learning. The reviewer believes the paper would be more valuable if it spent more time discussing neuroscience applications, relevance to experimentalists, and potential hypotheses-to-test that the paper might generate; in particular, ways of distinguishing slow-point vs. limit cycle mechanisms experimentally. Would the authors be able to provide more discussion time on these neuroscience implications?\n5. In biological networks one typically has modulatory signals that can increase or reduce network excitability. Do the authors have ideas for if and how a network with a limit cycle and excitability modulation (that can shut off the limit cycle after the response period) might be distinguished from the slow point mechanism, and how such a modulated limit cycle might compare with the two studied mechanisms?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxeCTSc3vv", "reviewer": "Reviewer_CYoj", "summary": "This paper analyzes the emergent mechanisms of short-term memory maintenance in task-optimized recurrent neural networks. The paper presents an analysis of a toy model and performs large-scale experiments to show that similar features emerge in actual task-optimized networks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and for recognizing that both the theoretical and empirical components are well-executed. We understand the main concerns to be (1) the clarity of the connection between theory and experiments, and (2) the perceived novelty of our contributions. We have revised the manuscript accordingly and address both below.\n\n**(1)** Due to space constraints, we had to condense this connection in the original submission. Here, we clarify the rationale and why toy models are *directly* relevant for the RNNs.\n\n**In short**, consider the models in Eq. (S1) and (S15) of the main text. These approximate the latent trajectory of a rank-one and rank-two RNN around a local minima, respectively. The scaling laws, derived from these models, accurately predict the transition boundaries observed in large-scale RNN training (new result, $\\beta = 4.05 \\pm 0.1$, $R^2 = 0.99$), when RNNs are learning those particular solutions in their latent subspaces.\n\nHere, it is worth noting that we empirically tested the scaling laws in the (realistic) full-rank RNNs, whereas the toy model theoretically models the latent trajectories in the low-rank RNN. The theoretical connection is an open question in this field and is a current hot topic (Valente et al., 2022). However, Valente et al. 2022 has empirically shown that trajectories of full-rank RNNs trained on behavioral tasks can often be reproduced by their low-rank counterparts that are sufficient to solve the task. In our case, rank-one RNN is sufficient to solve the task with a slow-point manifold, so it is not surprising that the toy model predictions explain (qualitatively) the existence of and (quantitatively) the boundary scalings of the phase diagram.\n\n*Longer rationale justifying, e.g., why the slow point model is a realistic approximation:*\n- RNNs are universal approximators of dynamical systems, and the rank of their recurrent weight matrix determines the dimensionality of the latent dynamics they can implement (Beiran et al., 2020).\n- Recent work has shown that even complex tasks can be solved with low-rank RNNs (Dubreuil et al., 2022), and that full-rank RNNs are often effectively low-rank in practice (Valente et al., 2022). In our empirical study, we similarly observe that both low- and full-rank RNNs converge to low-dimensional solutions, which take the form of either slow-point manifolds or limit cycles.\n- The slow-point toy model is the **normal form** (for reference, see Nonlinear Dynamics and Chaos by Strogatz) for a saddle-node bifurcation, which approximates one-dimensional dynamical systems around their local minimum (slow-point formation), including those implemented by the rank-one RNNs in Eq. (2).\n- Finally, learning the weights in rank-one RNNs implementing a slow-point corresponds to varying the parameter of this normal form, which gives rise to the phase diagrams derived in Fig 4C (and observed in full-rank RNNs in Fig 6).\n\nWe will briefly incorporate this explanation into the revised manuscript, with the rigorous mathematical details in the supplementary.\n\n**(2)** We agree with the reviewer that *studying attractors as short-term memory mechanisms in RNNs is one of the oldest problems in computational neuroscience*. However, our contribution is not the problem itself, but our novelty lies in findings and mechanisms that were previously not known in this long-standing field. To be specific, here are a few of the novel findings in our work:\n- To our knowledge, this is the first study to systematically analyze how latent mechanisms emerge during learning (e.g., such a description does not exist for the STSP based models, also cited by the reviewer).\n- We identify a sharp transition between slow-point and limit-cycle solutions depending on task delay and learning rate. To our knowledge, we are the first to show that there is a phase diagram of strategies (w.r.t., optimization and task parameters).\n- Unlike prior work, which often focuses on complicated working memory tasks (e.g., saccades in the references cited by the referee), our custom task design focuses on short-term memory and balances simplicity and expressiveness, allowing us to analyze dynamical structures at scale.\n- We show that small changes (e.g, a post-response) can qualitatively shift the memory strategy learned. \n- Our large-scale dataset (35,000+ trained RNNs, first dataset of its size reported and to be made public) will be released to support further studies on training dynamics, robustness, and learning strategies.\n\n**Misc** We thank the reviewer for pointing out the omission of synaptic models of working memory, which we now explicitly cite. Please also see our response to Reviewer pHmu. We also see that you asked for an ethics review. How could we address your concerns?\n\n**Final remarks** We hope these clarifications address the reviewer’s concerns and are grateful for the feedback, which significantly improved the paper’s presentation and scope.", "claims_and_evidence": "The theory and numerical experiments are each separately well-executed and interesting, but their connection is difficult to pin down. Typically, theoretical analysis of RNNs is designed to be directly comparable with the outcomes of numerical experiments. However, in this work, the theory seems to target a different model entirely. While the experiments exhibit some qualitative similarities to the theoretical toy model, they feel somewhat disjointed, making it challenging to establish a clear link between theory and practice.", "methods_and_evaluation_criteria": "Yes, somewhat (see above).", "theoretical_claims": "Yes, all of them.", "experimental_designs_or_analyses": "The RNN experiments make sense, although I did not look at the code in detail.", "relation_to_broader_scientific_literature": "This paper would be of interest to computational neuroscientists, as well as deep learning researchers working on interpretability.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The work is not particularly novel, as studying attractors as short-term memory mechanisms in RNNs is one of the oldest problems in computational neuroscience. I appreciate the attempt at theory, although as I explain above the connection between the numerical experiments and the theory developed is unclear to me.", "other_comments_or_suggestions": "For completeness, the introduction should acknowledge that many neuroscientists propose a synaptic basis for working memory, which relies on mechanisms distinct from those presented in the authors' work. For example:\n\n- https://www.cell.com/trends/trends/cognitive-sciences/fulltext/S1364-6613(15)00102-3\n- https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010776\n- https://www.nature.com/articles/s44271-023-00027-8?fromPaywallRec=false", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CM277I2K07", "reviewer": "Reviewer_59aj", "summary": "The paper investigates the strategies that recurrent neural networks (RNNs) use to maintain short-term memories via sequential firing. The authors trained low-rank and full-rank RNNs on delay-response tasks and identified two distinct mechanisms: slow-point (SP) manifolds and limit cycles. They found that introducing a post-response period significantly biases the strategy toward limit cycles. Additionally, they derive a scaling law relating the critical learning rate to the delay period length.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, as well as their recognition of our methodological rigor and insightful contributions to understanding short-term memory mechanisms in artificial networks. Please find our responses below:\n\n**Q1** While learning rate is an abstract optimization hyperparameter, prior literature indicates that it can be influenced by both external and internal biological factors. See below for a brief discussion that we will add:\n\n> Interestingly, our findings that longer delays require smaller learning rates parallels observations in neuroscience, where tasks with longer temporal gaps between cues and outcomes often pose greater credit assignment challenges. During cue-reward delays, dopamine activity has been suggested as a mechanism for solving credit assignment through eligibility traces. For instance, synaptic-level eligibility traces have been evidenced by dopamine's role in potentiating synaptic changes within precise temporal windows (Shindou et al., 2019; Yagishita et al., 2014). Additionally, dopamine may facilitate credit assignment through activity ramps that increase as subjects approach rewards (Fiorillo et al., 2003, Mikkael et al., 2022, Krausz et al., 2023). But more explicitly, a recent study (Coddington et al., 2023) showed that dopamine can directly modulate learning rates to support effective decision-making. Similarly, our results align with studies showing that the duration of the inter-trial interval—analogous to our post-reaction time—can also affect learning rates. Specifically, another neuromodulator, serotonin, has been found to modulate learning rates following long inter-trial intervals (Iigaya et al., 2018).\n\nThank you for this exciting connection that significantly improves the impact of our work for the neuroscience community.\n\n**Q2** We appreciate this insightful question. Our task was intentionally designed to isolate memory maintenance mechanisms, independent of representational content, to expose the underlying structure of the solution space (e.g., phase transitions). In tasks like delayed cue discrimination (Fig. 5), inputs determine the initial condition of the latent state, which then evolves along either a slow manifold or a limit cycle. In both scenarios, the identity of the cue is encoded in the trajectory, e.g., distinct slow-point manifolds.\n\nHow feature representations are integrated into these mechanisms is a compelling direction for future work. For instance, one could introduce a contextual variable that modulates the delay period for the same stimulus. This would allow investigation into whether the RNN reuses its existing slow point or limit cycle structures, or develops new attractors for each context. We view questions like these as natural next steps—and our publicly available dataset of trained RNNs is well-suited for such follow-up studies. We will add a paragraph to the outlook about this interesting future direction.\n\n**Weakness: hyperparameter sensitivity and generalizability** In the revised draft, we include a new experiment in which we double the neuronal time constant $\\tau$ (one of the hyperparameters in the RNN architecture). The results remain virtually unchanged, with the same phase diagram emerging. We will report these findings in a supplementary figure. We also note that a recent work (Park et al. 2025; ICLR 2025) has empirically found a similar phase diagram of algorithmic strategies, though in a completely different context.\n\nWhile testing all hyperparameters is computationally infeasible given the scale of our experiments, we now include the following paragraph in the Discussion to explicitly state this limitation and outline future directions:\n\n> In this work, primarily constrained by the immense compute required (about a month of a standard GPU time), we fixed architectural parameters and activation functions, which is a limitation of this work. Future work could explore the effects of different activation functions, network size (although recent results suggest that larger networks may reduce efficient learning rates; see Dinc et al., 2025), self attention-like mechanisms, and short-term synaptic plasticity on the emergence of different memory strategies. We conjecture that, since the toy models highlight the fundamental constraints associated with the latent dynamical systems learned within these architectures, phase diagrams we observe in this work may present universally across models (see also Park et al., 2025).\n\n**Generalizability and relevance of the toy models** We agree that this is central to the broader impact of our work and should have been stated more clearly. Please also see our response to Reviewer CYoj.\n\n**Final remarks** Please let us know if any further clarifications are needed. We sincerely appreciate the reviewer’s feedback and support. We agree that stronger alignment with neuroscience is important and have made necessary revisions.", "claims_and_evidence": "Their major claim was supported mostly by their results on artificial RNNs, but the biological relevance of these mechanisms remains unclear, especially the usage of limit cycles.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "I did not verify every detail of their mathematical arguments, I reviewed the toy model derivations, and they appear to be correct.", "experimental_designs_or_analyses": "Yes. The experiments include both low-rank and full-rank RNNs, as well as large-scale training with over 35,000 networks, ensuring robust findings within artificial RNNs.", "relation_to_broader_scientific_literature": "The paper is well-situated within the computational neuroscience and machine learning literature, particularly in the study of RNN dynamics. The discussion of memory maintenance mechanisms in artificial networks is insightful. However, the connection to empirical neuroscience is underdeveloped.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "- Strengths:\n      - Rigorous mathematical derivation.\n      - Large-scale empirical validation with 35,000+ trained RNNs.\n      - Acute to realize that traditional RNNs can be different from brain doing task by post-action period.\n- Weaknesses:\n      - Weak connection to biological plausibility and experimental neuroscience.\n      - Limited discussion of hyperparameter sensitivity and generalizability.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. What do you think would be the biological correspondence of the learning rate in your model?\n2. In Figure 5, the network needs to maintain feature-specific information rather than a featureless memory. How do SP manifolds or limit cycles encode and transmit feature representations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the strategies being used by RNNs to maintain short-term memories and identified two mechanisms of slow-point manifolds and limit cycles. The paper has done thorough analysis of the trained network and how it is dependent on training parameters. Reviewers appreciate the distillation of a complex problem down to a simple model that is plausible for math analysis. Although there are concerns from reviewers (missing citations and discussions, weak connection to experimental neuroscience, etc), overall, I think contribution of the study outweighs the weakness and decide to accept this paper. Please revise the manuscript based on reviewers’ suggestions.
Decision: Accept (poster)