Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
Authors: William Qian
Cengiz Pehlevan
Venue: ICLR 2026 Poster (2026)
Keywords: recurrent neural networks
computational neuroscience
dynamical systems
Primary area: applications to neuroscience & cognitive science

Abstract: Training recurrent neural networks (RNNs) to perform neuroscience-style tasks has become a popular way to generate hypotheses for how neural circuits in the brain might perform computations. Recent work has demonstrated that task-trained RNNs possess a strong simplicity bias. In particular, this inductive bias often causes RNNs trained on the same task to collapse on effectively the same solution, typically comprised of fixed-point attractors or other low-dimensional dynamical motifs. While such solutions are readily interpretable, this collapse proves counterproductive for the sake of generating a set of genuinely unique hypotheses for how neural computations might be performed. Here we propose Iterative Neural Similarity Deflation (INSD), a simple method to break this inductive bias. By penalizing linear predictivity of neural activity produced by standard task-trained RNNs, we find an alternative class of solutions to classic neuroscience-style RNN tasks. These solutions appear distinct across a battery of analysis techniques, including representational similarity metrics, dynamical systems analysis, and the linear decodability of task-relevant variables. Moreover, these alternative solutions can sometimes achieve superior performance in difficult or out-of-distribution task regimes. Our findings underscore the importance of moving beyond the simplicity bias to uncover richer and more varied models of neural computation.

Published as a conference paper at ICLR 2026
DISCOVERING ALTERNATIVE SOLUTIONS BEYOND THE SIMPLICITY BIAS IN RECURRENT NEURAL NETWORKS
William Q Cengiz Pehlevan
BBophhysiss Graduate Program
"Kempner Institute for the Study of Natural and Artificial Intelligence
'John A. Paulson School of Engineering and Applied Sciences *Center for Brain Science
Harvard University, Cambridge MA 02138
wi 11 ilia amq anêg. harvara. edu, c hawaaassaaaaarr.o
ABSTRACT
Training recurrent neural networks (RNNs) to perform neuroscience-style tasks has become a popular way to generate hypotheses for how neural circuits in the brain might perform computations. Recent work has demonstrated that task- trained RNNs possess strong simplicity bias. In particular, this inductive bias often causes RNNs trained on the same task to collapse on effectively the same solution, typically comprised of fixed-point attractors or other low-dimensional dynamical motifs. While such solutions are readily interpretable, this collapse proves counterproductive for the sake of generating a set of genuinely unique hypotheses for how neural computations might be performed. Here we propose Iterative Neural Similarity Deflation (INSD), a simple method to break this in- ductive bias. By penalizing linear predictivity of neural activity produced by standard task-trained RNNs, we find an alternative class of solutions to classic neuroscience-sty! RNN tasks. These solutions appear distinct across battery of analysis techniques, including representational similarity metrics, dynamical sys- tems analysis, and the linear decodability of task-relevant variables. Moreover, these alternative solutions can sometimes achieve superior performance in diffi- cult -itoo--distribution task regimes. Our findings underscore the importance moving beyond the simplicity bias to uncover richer and more varied models of neural computation.
INTRODUCTION
Developing recurrent models of neural computations has become an increasingly popular approach to generate hypotheses for neuroscience (Mante et al., 2013; Rajan et al., 2016; Maheswaranathan et al., 2019; Yang et al., 2019; Sylwestrak et al., 2022; Daic et al., 2023; Beiran et al., 2023; Nair al., 2023; Driscoll et al., 2024; Javadzadeh al., 2024; Genkin al., 2025). In particular, current neural networks (RNNs) trained on neuroscience-style tasks offer insight into possible so- lutions that may be implemented at an approximate level by biological neural circuits. Such RNNs are typically trained via backpropagation through time (Werbos, 1990) or FORCE (Sussillo & Ab- bott, 2009), methods that seem bear little resemblance the way learning proceeds in biological circuits (Crick, 1989; Lillicrap et al., 2020). Nonetheless, resemblances between solutions found by artificial and biological networks have the potential shed light on shared principles of neural computation that emerge despite these differences (Mante et al., 2013; Yamins et al., 2014; Sussillo al., 2015; Kell et al., 2018; Banino et al., 2018; Schrimpf al., 2020; Feather et al., 2023; Jensen et al., 2024; Pagan et al., 2025).
Central to this research program is the ability to produce multiple competing hypotheses that can then be evaluated on equal footing via comparisons against experimental data (Barak et al., 2013; Sussillo et al., 2015; Soldado-Mo Magranee et al., 2024; Pagan et al., 2025; Huang et al., 2025). Ideally, particular task would be sufficient to yield diverse range of solutions
training multiple RNNs for this purpose. Yet, this strategy faces major obstacles scenarios where training procedures overwhelmingly bias RNNs towards particular kinds of solutions.

---

Published as a conference paper at ICLR 2026
Recent work has shown that task-trained RNNs exhibit bias towards simple volutions-solutionns that use a minimal arrangement of cow-dimensional dynamical structures such as fixed point attrac- tors and limit cycles, and reuse dynamical motifs where possible (Turner & Barak, 2023; Driscoll et al., 2024; Hazelden et al., 2025). These types of solutions have desirable properties including parsimony and flexibility, and often lend themselves relatively straightforward interpretation via analysis techniques such as targeted dimensionality reduction, low-rank approximation, and dynam- ical systems analysis (Sussillo & Barak, 2013; Mante al., 2013; Dubreuil et al., 2022; Valente et al., 2022; Driscoll et al., 2024). However, for many neuroscience can be strong enough to cause different networks trained on the same task to collapse to effectively the same, minimal solution, a phenomenon referred to dynamic collapse (Hazelden et al., 2025). Despite the desirable properties of such solutions, it remains far from clear that this bias towards simplicity is always aligned with the inductive biases of biological circuits. For example, RNNs trained on simple memory tasks ubiquitously find solutions using persistent activity held in stable attractor states
tasks, this simplicity bias
et al., 2019; Turner & Barak, 2023; Driscoll et al., 2024; Hazelden et al., 2025), yet population- recordings have shown that the neural representations underlying memory functions be highly dynamic (Spaak al., 2017; Lundqvist et al., 2018; Daie et al., 2023; Ritter & Chadwick, 2025). These observations raise an important question: how can RNNs be trained to generate unique hypotheses for recurrent computations that go beyond the simplicity bias?
The most natural toolkit for generating different task solutions includes varying hyperparameters such as the initialization scale, training seed, and model architecture. The initialization scale in particular has been shown affect lazy versus rich learning in RNNs (Schuessler et al., 2020; Liu et al., 2023; Bordelon et al., 2025), well as the emergence of "aligned" or "oblique" solutions (Schuessler al., 2024). However, dynamic collapse can still be observed even when RNNs are initialized in the highly chaotic regime (Hazelden et al., 2025). While varying these basic knobs is sometimes sufficient to generate a multitude of qualitatively distinet solutions, (Turner et al., 2021: Huang al., 2025; Murray, 2025; Kurtkaya et al., 2025), many classes of realistic solutions likely still inaccessible through these means. For instance, Pagan et al. (2025) found that a large population of RNNs trained on the same context-dependent decision making task populated only one corner of the solution space compatible with neural data. Moreover, solutions obtained by varying architectural details can appear representationally distinct, but often implement the same underlying dynamical solution, as revealed by fixed-point topology (Maheswaranathan et al., 2019).
In this paper, we propose simple method for generating unique solutions to RNN tasks, extending beyond solutions discoverable by standard means. This method, which we call Iterative Neural Similarity Deflation (INSD), loosely analogous to the Gram-Schmidt procedure but in the space of RNN solutions. By iteratively penalizing the linear predictivity of neural activity produced by previously trained RNNs in an online fashion, we find solutions that diverge from the prototypical solutions to classic neuroscience-style tasks. We show that the alternative solutions generated in this manner not only use distinct representational geometry as expected, but also use different dynamical motifs and encode task variables more nonlinearly. Across all tasks, these solutions forgo the usage of fixed point attractors and slow manifolds for keeping track of task-relevant information, and instead tend to maintain task-relevant information in dynamically evolving subspaces of activity. Surprisingly, we find that these alternative solutions can sometimes achieve superior performance when tested in difficult -iitt--iistiibuttinn task conditions.
2
METHODS
2.1 SETUP AND TRAINING PROCEDURES
We consider rate-based RNNs obeying the dynamics
dx dt
x + Wr + (()
(1)
where x E N represent neural activations over N units. W E NXN is the recurrent weight matrix, Jin E N and u(t) E N..... are the input weights and inputs, respectively, r ((x))
2

---

Published as a conference paper at ICLR 2026
the "firing rates", and ó is an elementwise nonlinearity which we take to be tanh. The output is given by yt) = J r"t)) (t), for readout weights J E R Mouu X.N.
For each task, we first train a reference RNN to minimize the mean squared error
1 L 7 J./ (v(() (1)
(2)
averaged over different input conditions utt), via batch gradient descent over the parameters {W, Jo We initialize the recurrent weights as W5 NN g2 /N), where 9 is a gain parameter. The input and output weights are both initialized with entries drawn from (0,11/N)
We then apply neural activity similarity penalty to subsequent RNNs trained on the same task. In particular, for each batch of input conditions, firing rates R1 E (BL))) and R2 E (B are collected from the reference RNN and the second RNN, respectively, where the batch (B) and discrete timestep (I,) dimensions have been flattened. These firing rates are then projected into their respective readout null spaces, yielding R2 and R The second RNN is then trained with the loss
L' = AS AS(R), ),
(3)
where S is some neural similarity measure, and À hyperparameter representing the strength of the penalty. We project firing rates to readout nullspaces prior applying the similarity penalty because allowing it to operate on the output potent component of activity would be counterproductive to solving the task. In particular, if the reference RNN achieves near perfect outputs y(t) y*)), then to achieve similar task performance, the second RNN's activity must necessarily be able to linearly predict the output potent component of the reference RNN's activity. This procedure can be continued iteratively, with third RNN penalized with respect to both previous RNNs via a loss
(4) and so on. We refer to this procedure as Iterative Neural Similarity Deflation (INSD), and label RNNs trained in this manner alt- 2lt-2, etc. This approach for explicitly encouraging different task solutions somewhat resembles the Barlow Twins method (Zbontar et al., 2021) in computer vision and the method of linear adversarial concept erasure (Ravfogel et al., 2022) algorithmic fairness.
C" = C ( R+) + S(R, RR)]
For comparison, we also train a population of "standard" RNNs on each task. For simplicity, we use the same architecture for all RNNs, training ten RNNs with different seeds for each initialization scale g E [0.01, 0.5, 1.0, 1.5]. A more detailed sweep including architecture, hyperparameters, and nonlinearities can be found in Maheswaranathan et al. (2019). Training details are specified in A.
2.2 NEURAL SIMILARITY MEASURES
There exists large variety of neural similarity measures that could be used for the similarity penalty, each with their own advantages and drawbacks (Raghu et al., 2017; Kornblith et al., 2019; Williams al., 2021; Harvey et al., 2024a; Williams, 2024; Cloos et al., 2024; Harvey et al., 2024b). For our purposes, we seek metric which is invariant to relabeling or rotation of neural axes, and for which forwards and backwards passes can be efficiently computed online.
For many neural similarity measures, solving a task while maintaining low neural similarity with respect to a reference network admits a trivial yet undesirable solution: subspace of activity im- plements version of the reference solution, while the remaining degrees of freedom simply inflate the dimensionality of the neural activity with task- iireeeateev dynamics. In particular, centered kernel alignment, representational similarity analysis (RSA), and linear predictivity scores in the direction of [reference RNN > penalized RÑN] can all be driven arbitrarily close to in this manner (see A.2). To avoid this solution, reference RNN] as the similarity penalty. This asymmetry between predictivity and predictability in their sensitivity irrelevant dynamics has also been noted in recent work latent variable mod- els of neural activity (Versteeg et al., 2024; Dabholkar & Barak, 2025). We remark that canonical correlation analysis (Hotelling, 1936; Raghu al., 2017) can also avoid this undesirable solution, although the extra whitening step incurs slight additional computational cost.
linear predictivity in the opposite direction [penalized RNN -
We define linear predictivity as (X, Y) = 1 -ii [XM-Y
where Ux
Y 2
X(X x)+ T E (BL)) projects the column space of X. As the input matrices
3

---

Published as a conference paper at ICLR 2026
a
b
ref. RNN
alt-1 RNN
VT
Output
Output
0
0.5
context que
ref.
all-1
PC2
PC2
2.5
PC1
PC1
fixed point
Ineg
d
2.5
-2.5 0.0 2.5
linear predictivity
1.0
9
init. seed avg
0.8
ambadding
Model
0.6
Model
1.0
0.5
0.5
g= 1.5
NBS
1.0
all RNN
-0.25
g= 1.5
RMN
alx2 RNN
0
0.5
1.0.
L.S
al11 RMN
alt2 RMN
0.5
3h.2 RNN
MDS
Model
Model
RNNs yield distinct solutions to context-dependent integration. a. Task schematic: two noisy stimuli are passed as input. In each trial, only the input stream selected by the context cue needs to be integrated, while the other ignored. b. Example trajectories shown along the first two PCs and the output axis for the reference RNN (left) and alt-1 RNN (right), respectively. Trajectories are colored by time and relevant context (Ctx A: viridis, Ctx magma) during the corresponding trial. Fixed points (green x x's) and unstable oscillatory leading eigenmodes (red bars) are shown. c. Representative examples of cigenvalue spectrums for Jacobians computed at fixed points found for the reference RNN (left) and alt-1 RNN (right). d. Left: Linear predictivity matrix across RNNs at different initialization scales and seeds trained on the task, along with the alt- 1 and 2 RNNs. Right: same, but with scores for the standard RNNs averaged seed. e. MDS embedding of the DSA dissimilarity matrix computed across the same RNNs as in d.
Figure 1: Similarity-penaliii
initialization
often rank-deficient in our usage, for numerical stability, we also add small ridge regularizer when computing the similarity penalty: S(X, Y)
U Y
where Ux,p = X(X X -
2.3 DYNAMICAL SYSTEMS ANALYSIS
We probe the dynamical properties of task solutions via numerically solving for fixed points, in (Sussillo & Barak, 2013). In line with previous studies (Sussillo & Barak, 2013; Maheswaranathan al., 2019; Driscoll et al., 2024; Kurtkaya et al., 2025), we include approximate fixed points, also referred to as slow points. Where relevant, we also report the stability. eigenvalue spectrum and leading eigenmode(s) that govern the linearized dynamies in the vicinity of each fixed point.
3 RESULTS
We analyze and compare similarity-penalizee solutions across three neuroscience-styke tasks that have been well studied in the literature (Barak al., 2013; Mante al., 2013; Maheswaranathan et al., 2019; Schuessler et al., 2020; Smith et al., 2021; Krause et al., 2022; Valente et al., 2022; Costacurta al., 2024; Driscoll al., 2024; Huang et al., 2025; Pagan et al., 2025). These tasks span context-dependent processing, discrete and analog memory, and delayed output production. Each of these tasks is associated with prototypical solution that has been reported across multiple studies, which briefly describe for each task. Task parameters are specified in A.3.
Ceneext-de integration. We begin by studying RNNs trained on context- deeeedt-de inte- gration (Fig. la). For this task, the network receives two streams of noisy input stimuli and fixed context cue. For a short duration Tpe. only the one-hot encoded context cue is shown. Thereafter, the context from A VH, o² /dt) (following the convention in (Mante al., 2013; Schuessler et al., 2024)). For each trial, the stimuli coherences μ4 are sampled from e(-Pmaas [mna]]- At each timestep, the
remains on, while the noisy input stimuli are sampled independently at each timestep
4

---

Published as a conference paper at ICLR 2026
a
b
ref. RNN
alt-1 RNN
subspace alignment
regression subspace
Cox
Cbx A
Ctx
ref RNN
Sim Stm.Acm Adh
5tmm. A
cos
all-2 RNN
Choice
Choice
Choice
Choice
0
Sim coh.
5tm
Stim
8lim
Ok
Choice
Chokc
Choiks
Choioe
weight subspace
Cbx A
Cix B
Clx
Cix
Ctx A/B decoding
A
1.0
test acouracy
Jul
0.5
linear RBF
56m B coh.
0.0
standard alt-1
all-2
Figure 2: Linear encoding of task-relevant information degraded in similarity -penalized RNNs. Task: context-dependent integration. a. Averaged trajectories plotted on different sets of axes, colored by the coherences of the input stimuli. Top row: axes directions estimated via pre- dicting current target output (choice), stimulus A coherence, and stimulus B coherence via linear regression over neural activity aggregated over trials and time points. Bottom row: same averaged trajectories, but plotted on axes of the input and output weights. each quadrant, left and right plots correspond to context A and B trials, respectively. Colorbars are normalized so that 1 corresponds to the minimun/maximmm coherence value. b. Alignment hetween the regression and weight sub- spaces, as measured by the cosine of the principal angles. Grey dots represent alignments computed for the population of standard RNNs. c. Decodability of the relevant context from neural activity under linear RBF kernel regression, report the standard error of the mean. The grey dotted line represents the baseline accuracy.
quantified by test accuracy a heldout set. Error bars
network must output the cumulative sum (scaled by di) of all inputs received far in the stimu- lus channel selected by the context cue. RNNs trained on this task and its binary decision making variant have consistently been found learn two lines of fixed points (line attractors), one for inte- grating the relevant stimulus in each context (Mante et al. (2013); Maheswaranathan et al. (2019); Smith et (2021); Krause al. (2022); Pagan et al. (2025)).
To assess the properties of solutions, as in (Maheswaranathan et al., 2019), we first probe all trained networks using task trials of varying stimuli coherences, turning off stimuli noise for visual clarity. observed that all standard RNNs found the aforementioned pro-
In line with previous findings, totypical solution, regardless of initialization scale and training seed. We illustrate this solution for reference RNN in Fig. lb (left), showing activity trajectories plotted on the axes of the first two principal components and the readout. During the context-only period, trajectories quickly segregate into separate regions of state space. Then, in each context, activity is driven along line of approx- imate fixed points that densely tile the span of trajectories observed in that context. In contrast, similarity penalized RNNs yielded solutions characterized by oscillatory dynamics (Fig. 1, right). Activity in each context was readily distinguishable by the shape of trajectories, rather than the por- tion of state space they occupy. Moreover, activity was no longer driven along slow/fixed points. Instead, unstable fixed points with oscillatory eigenmodes were found, but were not used (at least directly) for remembering the cumulative input in either context. Comparing the eigenspectrums of the Jacobians at representative fixed points for both networks confirmed that marginally stable linearized dynamics were only present for fixed points of the reference RNN (Fig. Ic). For brevity, defer the trajectory and eigenspectrum plots for the alt-2 RNN to the Appendix (Fig. A.1).
We compute linear predictivity scores both directions between all pairs of models, including the population of standard RNNs and models produced by two iterations of INSD. We find that the representations used by standard RNNs are all highly linearly predictive of each other, with only slight deviations from perfect predictivity observed when predicting models of high initialization scale from models of lower initialization scale (Fig. ld). Further, similarity penalized RNNs were markedly less predictive and less predictable with respect to standard solutions. To quantify rela-
5

---

Published as a conference paper at ICLR 2026
ref. RNN
alt-1 RNN
a RNN
X fixed point
ii
ii
PC3
unstable (complex)
unstable (real)
stable (real)
PC1
PC2
.012 Die
Out 3
lv
Out
Out 2
Figure 3: Sustaining discrete memory states without fixed-point attractors. Task: 3-bit flipflop. Example trajectories plotted the principal component (i, ii, iii) and output (iv, v, vi) axes, shown for the reference (i, iv), alt-l (ii, v), and alt-2 (iii, vi) RNNs Fixed points (green x's) and their leading eigenmodes (colored bars) are shown. A larger marker size is used for stable fixed points. b. Example timeseries of network output for all three networks.
tionships between the solutions beyond geometrical similarity, we also compute their Dynamical Similarity Analysis (DSA, Ostrow et al. (2023)) dissimilarity matrix, visualizing the scores via a multi-dimeesional scaling embedding (Fig. le). This embedding reveals a degree of clustering by initialization scale. However, similarity-penalized solutions achieve a dynamical dissimilarity with respect to the standard population that far exceeds the scale of variability observed across clusters.
Next, we analyzed population responses via projecting activity trajectories onto task-relevant sub- spaces. For the reference and alt-1 RNNs, we first construct a regression- -based subspace comprising of the "stimulus A", "stimulus B", and "choice" axes. These directions were estimated via linearly regressing the coherences of stimulus A, stimulus B, and the task target, respectively, from neural activity aggregated across timesteps and 5000 trials. Consistent with prior studies (Mante et al., 2013; Smith et al., 2021; Pagan et al., 2025), projecting the averaged trajectories of standard RNNs onto this set of axes revealed temporally stable and consistent encoding of the coherences of both input stimuli, regardless of the selected context (Fig. 2a, top left). In contrast, for the alt-1 RNN, the coherence of the relevant stimulus in each trial could still be linearly decoded somewhat con- sistently, but estimates of the irrelevant stimulus were often inconsistent with actual trial conditions subspace, projecting aver-
(Fig. 2a, top right). We repeated these analyses, but for a weight- aged trajectories onto the axes [J Jig, Joujj defined by the input and output weights of each RNN We again find that, for the standard solution, stimuli coherences for both relevant and irrelevant stimuli can be stably distinguished under these axes (Fig. 2a, bottom left). However, for the alt-1 RNN, the directions encoded by the input weights poorly captured the coherences of both stimuli, regardless of context (Fig. 2a, bottom right). To assess the relationship between the weight and re- gression subspaces, we quantified their alignment via computing the principal angles between them (Fig. 2b). Across all models, the leading overlap was near unity, likely due to the high alignment be- tween the regression "choice" axis and Jour weight axis. Although the standard RNNs demonstrated varying degrees of moderate alignment between the remaining axes, these angles were near orthog- onal for both the alt-1 and alt-2 RNNs. Finally, we assessed the extent to which task the most basic task variable-can be accurately decoded from activity. Consistent with the geometric picture of Fig. 1,, we find that context is linearly decodable at high accuracy for standard RNNs, whereas the alt-1 RNN (and to lesser extent, alt-2 RNN) requires additional nonlinear featurization of representations for context to be decodable at similarly high accuracy (Fig. 2c).
3-bit flipflop. We next seek alternative solutions on 3-bit flipflop, simple discrete memory task. For this task, three input channels are given. At each timestep, each channel independently has probability p having an upward or downward spike of magnitude 1/dt, with both directions having equal probability. The target output for the network begins at 0 for all channels, and thereafter tracks the sign of the last spike in each channel. Trained RNNs consistently learn the most minimal and sensible solution: fixed point attractors arranged in a cube associated with each of the 8 main output states (aside from the starting outputs at 0), as well as saddle points whose unstable directions are aligned with edges of the cube to facilitate state transitions (Barak al., 2013; Maheswaranathan

---

Published as a conference paper at ICLR 2026
a
farget
memory
ii
response
1.0
0.5
Output
Output
ref.
0.0
incer predictiity
Loavtty
PC1
PC1
PC2
PC2
point
Iii
DSA enbedding
alt-1
Output
Output
NO93
-0.25
2-15
2-10
PC1
PC1
1st.
PC2
PC2
0.5
0.5
10
HDB
Figure 4: Similarity-penalized RNNs find dynamic, rather than persistent, encoding of analog memories. a. Task: MemoryPro. an angle encoded as a 2D vector is passed as input during a stimulus phase, followed by a memory phase where the angle input is absent. Only once the fixation cue (grey) is removed must the network output the angle that was observed. All inputs are noisy. b. Example trajectories divided by the memory (i, iii) and response (ii, iv) phases, shown for the reference RNN (i, ii) and alt-1 RNN (iii, iv). All plots are along the first two memory phase PCs and the cos output axis, with trajectories colored by time. The start and end of every trajectory is colored by the target output angle. Fixed points (green x's) and unstable oscillatory eigenmodes (red bars) are shown. The right corner of each subplot shows the direction encoding the target angle over time, estimated by linear regression. c. Linear predictivity matrices comparing standard RNNs at different initialization scales to similarity penalized RNNs (alt-1, alt-2). Scores involving standard RNNs averaged with respect to the initialization seed. Left: base linear predictivity scores. Right: linear predictivity scores when activity is first projected to the readout nullspace. d. MDS embedding of the DSA similarity matrix computed across the same RNNs as in c.
et al., 2019; Ostrow et al., 2023). We plot trajectories of solutions as well as fixed points for a reference, alt-1 and alt-2 RNN trained on this task. We confirm that the reference RNN indeed learns the standard solution involving the cube stable fixed points, and saddle points that transition between them (Fig. 3a.i). Moreover, the geometrical structure of activity in PCA space is minimal in the sense that it mirrors the cube-like geometry of the task output. For the similarity penalized RNNs, however, observed trajectories longer show this geometry in PCA space, and instead follow oscillations generated by unstable fixed points with complex leading eigenmodes (Fig. ia.ii,iii).
Despite these apparent differences in representational geometry, all three networks must ultimately produce cube-like geometry when trajectories are projected onto the output subspace; this is de- manded by the structure of the target output of the task. Thus, compare the solutions found more aptly, we also plot trajectories and fixed points on the output axes of cach RNN. This reveals that, even in the output subspace, similarity-penaan properties of fixed points. In this example, the alt-1 RNN lacks fixed points that stabilize any of the output states, instead showing two groups of unstable fixed points with oscillatory eigenmodes, and saddle points that appear to transition between them (Fig. 3a,v). The alt-2 RNN recovers the presence of fixed points at each corner, but they are no longer stable/attractive (Fig. 3,).). Moreover, the directions of saddle points that line the edges of the cube are often misaligned. These differ- ences in dynamical motifs manifest as slight but noticeable imperfections in the output produced by the similarity-penalized RNNs (Fig. 3b). We also assess the similarity of representations across all RNNs using linear predictivity and DSA (Fig. A.2). Similar to the findings for context-dependent integration, all standard solutions found to be perfectly linear predictive of each other, whereas the similarity-penalized RNNs occupy disparate areas of the DSA MDS embedding.
RNNs exhibit distinct arrangements and stability
Lastly, turn our attention the MemoryPro task (Fig. 4a). The RNN receives three piecewise constant inputs: a fixation cue and 2 stimuli channels encoding an angle. For each trial, the angle 0 is sampled from u[- Following Driscoll al. (2024), at train time, stimuli and response onsets and offsets variable. Specifically, after a delay of length Tad U[TT Tatll
MemoryPro.
7

---

Published as a conference paper at ICLR 2026
context-dependent
flipflop
MemoryPro
-1.
Test
1.2
1.4
10"
Tra-- 2.0
4.0
Trass maa--10m 10.0
1.0
0- 0.2
o- 0.3
o- 0.4
10'
spcuracy
MSE
10"
MSE
10
0.5-
10"
10"
0.0
1
0.0
0.5
0.2
1.0
noise scale
noise
frac. memory load
ref. RNN 0 alt-1 RNN
alt-2 RNN
trained values
Figure 5: Similarity penalized models can outperform standard models in difficult task regimes. a, b. Mean squared error on the context- context-dependent integration (a) and flipflop (b) tasks, respectively, across different noise scales and trial length scaling Tauit. The grey dotted lines in- dicate the noise scale used during training. c. Accuracy on the MemoryPro task versus fractional memory load, at different noise scales We assess accuracy using the same criteria as in (Driscoll et al., 2024). Small blue dots represent the scores achieved by the population of standard RNNs.
the angle stimuli (sin 0 cos 0) are shown for a duration Tsum ~ [ following which the fixation cue removed and the response period begins. For a duration Tresp ~ [[[Tnsp: Fespl. the network must output the angle seen during the stimuli phase, also as a 2D) vector. The network must also produce an output that tracks the fixation cue. All three inputs are also subjected to independent noise at each timestep. drawn from (0, o²). Previous studies consistently report the following prototypical solution: during the memory phase, angles are encoded along ring manifold of persistent states in the output nullspace, stabilized by ring attractor. During the response phase, this ring of fixed points quickly rotates to become output potent (Driscoll et al., 2024; Costacurta et al., 2024; Hazelden et al., 2025).
stim: Tstinl Then, the stimuli are
turned off for a duration T meem mTmm - Tnem].
To probe the properties of solutions, we plot trajectories collected over trials with various target angles and in the absence of input noise (Fig. 4b). As in Driscoll et al. (2024), we use the axes of the first two memory phase PCs and the cos 9 output channel, separating trajectories by the memory and response phases. Our results confirm that standard RNNs ubiquitously find the prototypical solution involving a ring attractor that rotates outwards, shown for the reference network (Fig. 4b,i.ii). We also plot the direction in activity space that best predicts the target angle via linear regression at each timestep, confirming that memorized angles are statically encoded (Fig. 4b,i,ii. bottom right). In contrast, the alt-1 RNN exhibits rotational dynamics during the memory phase that nonetheless maintains the relative ordering of trajectories by their corresponding target output (Fig. 4b, iii). The ring of fixed points is no longer present, and is instead replaced by a line of unstable fixed points with oscillatory leading eigenmodes. Linear decoding analysis reveals that the direction encoding the target angle is indeed rotating with the activity (Fig. 4b, iii, bottom right). Moreover, this direction even acquires output potency at times, despite the fact that the output potent component is, by task necessity, a low-variance fraction of the activity during the memory phase. During the response phase, these trajectories continue to oscillate, but rotate to become output potent (Fig. 4b, iv). We defer the corresponding plots for the alt-2 RNN to the Appendix (Fig. A.3).
As done for previous tasks, we compute linear predictivity and DSA dissimilarity scores between all pairs of models across the standard and similarity- penalized RNNs. While the linear predictivity of standard solutions from similarity-penalizee similarity-penalzzed solutions is degraded, we find that it is still significantly above zero (Fig. 4c, left). However, this partial predictivity is ablated once activities are projected into their respective readout nullspaces. This indicates that the only component of activity that the similarity penalized models can predict from standard solutions is that which is necessary to solve the task, namely, the output potent component. An MDS embedding of the DSA dissimilarity matrix confirms that the mniilarity-penalizee RNNs achieve dynamically dissimilar solutions (Fig. 4d).
Assessing solutions by their performance under atypical task conditions. Across all three tasks, we found solutions that appear distinct from standard solutions by variety of measures. However, are these solutions actually functionally distinct, or are they merely approximating the standard solution in ways that are difficult to discern? To answer this question, we tested all models under task conditions seldom or never seen during training. For the context- context-dependent integration task, we measured task performance across different noise scales of the input stimuli. We also introduce and sweep over the parameter Tmuhs a factor that uniformly scales the duration of trials. We conduct similar performance sweep for the flipflop task. For the MemoryPro task, again sweep the
8

---

Published as a conference paper at ICLR 2026
imput noise scale o, but also sweep the fractional memory load, which we define Tain Tuen tune this parameter, fix the duration of the pre-stimulus and response phases, as well as the total duration of the stimulus and memory phases combined. We then adjust the timing of the transition from the stimulus to the memory phase to produce test trials of varying fractional memory loads.
T To
We report model performance across these sweeps in Fig. 5, as well as the corresponding effective dimensionality of activity as measured by the participation ratio in Fig. A.4. Across all tasks, we find that standard RNNs typically outperform seen during training (Fig. 5). However, we also observe many cases where similarity-penallzee RNNs outperform standard models. For instance, for the context-dependent integration task, the alt-2 RNN moderately outperforms the population of standard RNNs in highly noisy conditions, all the while remaining robust to lengthened trial durations. For the flipflop task, although we observe near-identical performance across most models, the alt-2 RNN achieves moderate but significant gain in relative performance when noise is high. Lastly, for MemoryPro, we observe that the alt- RNN significantly outperforms standard RNNs on the most difficult trials, where both noise and memory load are high, but significantly underperforms the population under low memory loads. The alt-2 RNN only matched or underperformed the population under all conditions, suggesting that it simply failed to learn the task as well. Altogether, these performance deviations confirm that moillarty-penallin models indeed produce solutions that are functionally distinct.
aomilarity-pnnalizn
RNNs when tested under conditions
4 DISCUSSION
Generating rich set of diverse hypotheses that can be tested against experimental data is founda- tional for progressing our understanding of the brain. Motivated by recent observations of dynamic collapse in task-trained RNNs (Maheswaranathan et al., 2019; Driscoll et al., 2024; Hazelden al., 2025), we propose a method called Iterative Neural Similarity Deflation (INSD) for expanding the space of accessible solutions. Across three neuroscience-style tasks, extensively study and com- pare the solutions generated by iteratively penalizing the linear predictivity of past solutions. These analyses revealed alternative solutions that did not directly use simple dynamical motifs such as fixed point attractors or continuous slow manifolds to store information. Instead, similarity-penalize RNNs tended to produce activity characterized by quasi-periodic oscillatory modes. Further analy- sis revealed that these oscillations were not simply nuisance dynamics that emerged as a peculiarity of the similarity penalty, but rather actively supported the dynamic encoding of task-relevant infor- mation. These solutions are reminiscent of a theory proposed by Park et al. (2023) on how memories can be stably maintained in the phase difference between two oscillations, rather than through per- sistent attractor states. In the same vein, recent work by Ritter & Chadwick (2025) argues that optimally efficient and noise-robust working memory requires high-dimensional rotational dynam- ics, and further finds signatures of such dynamics in monkey prefrontal cortex. These observations are consistent with our finding of improved robustness for some penaaiize solutions.
For context-dependen integration, unlike similarity-penalized RNNs, standard RNNs produced so- lutions where task-relevan information was stably represented in linear subspaces, consistent with neural data recorded during analogous tasks (Mante et al., 2013; Pagan al., 2025). Thus, a natural concern is that mnnilarity-penaliie RNNs may produce solutions whose population coding proper- ties are not realistic. However, we argue that being able also find unrealistic solutions crucial for probing when and why simple solutions align with biology. Moreover, in principle, one could con- struct networks that interpolate between standard and similarity-penalizee solutions. Most simply, this could be achieved by an RNN with two populations of neurons, one dedicated to implementing each solution. Much as how ensembling is used in machine learning to reduce variance and improve generalization, such mixed models may possibly enjoy greater robustness, all the while maintaining more realistic linear encoding properties at the population level. We leave a more detailed investi- gation of this idea to future work.
Finally, we acknowledge that linear predictivity is an imperfect measure of both dynamical similarity and functional equivalence (Ostrow et al., 2023; Qian et al., 2024; Braun et al., 2025). The recently proposed Dynamical Similarity Analysis (DSA, Ostrow et al. (2023)) has been shown to effectively identify RNN solutions whose dynamical properties are only superficially distinct, while other met- rics often fall short. However, computing this metric as a similarity penalty in an online fashion would be prohibitively computationally expensive. Despite the limitations of linear predictivity, we

---

Published as a conference paper at ICLR 2026
found that penalizing the predictivity of representations used by standard RNNs sufficient to generate solutions with distinct dynamical features and unique task performance profiles. While we did not perform an extensive sweep over the strength of the predictivity penalty here, solutions that differ from standard solutions in more fine- fine-grained ways may potentially also be discoverable if the penalty is set to be small. For instance, for context-dependent decision making, more granular form of solution degeneracy can be characterized in terms of the relative arrangements of input vectors and eigenvectors of the linearized dynamics across contexts (Pagan al., 2025). Since the penalty we use relates to neural activity, our approach likely cannot fully capture degeneracies in the relative arrangements between left eigenvectors and input vectors, but may be able to explore degeneracies in the relative arrangements of the right eigenvectors (integration directions) across contexts, which do directly manifest in activity.
A limitation of our study is that we focus on simple single-task settings where standard solutions invoke attractor dynamics. Future work should investigate tasks that require transient dynamics, such as timing tasks, where standard RNN solutions are already somewhat varied (Turner et al., 2021; Beiran et al., 2023; Huang et al., 2025). Experiments in multitask settings would also be insightful for understanding whether greater task demands make it more difficult find solutions that are not linearly predictive of reference solutions (Cao & Yamins, 2024; Huang et al., 2025). More generally, this "capacity" for distinct solutions is expected to depend on task difficulty and network size, and could be systematically explored using INSD. Such experiments could serve as a way empirically assess ideas such as the Contravariance Principle and the Platonic Representation Hypothesis (Cao & Yamins, 2024; Huh et al., 2024), both of which assert that sufficiently difficult or constraining task demands necessitate representational convergence across systems.
ACKNOWLEDGMENTS
We thank Jacob A. Zavatone- Veth and David G. Clark for insightful discussions and comments on a previous version of this manuscript. W.Q. is supported by a Kempner Graduate Fellowship. C.P. is supported by an NSF CAREER Award (IIS-2239780), DARPA grants DIAL-FP-038 and AIQ- HR001 12520041, the Simons Collaboration on the Physics of Learning and Neural Computation, and the William F. Milton Fund from Harvard University. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
REFERENCES
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap. Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like repre- sentations in artificial agents. Nature, 557(77055:4299443, May 2018 ISSN 1476-4687. doi: 10. 10.1038/s4158-00 018-0102-66
Omri Barak, David Sussillo, Ranulfo Romo, Misha Tsodyks, and L. Abbott. From fixed points to chaos: Three models of delayed discrimination. Progress in Neurobiology, 103:214-222, April 2013. ISSN 0301-0082. doi: 10. pneurobio.20133.0000
Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mehrdad Jazayeri, and Srdjan Ostojic. Para- metric control of flexible timing through low-dimensional neural manifolds. Neuron, 111(5): 739-753.e8, March 2023. ISSN 0896-6273, doi: 10. 10.1016/..eurron.0222..
Blake Bordelon, Jordan Cotler, Cengiz Pehlevan, and Jacob A. Zavatone- Veth. Dynamically Learn- ing to Integrate in Recurrent Neural Networks, March 2025.
Lukas Braun, Erin Grant, and Andrew M. Saxe. Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks. In Forty- Second International Conference on Machine Learning, June 2025.
Rosa Cao and Daniel Yamins. Explanatory models in neuroscience, Part 2: Functional intelligibilityy and the contravariance principle. Cognitive Systems Research, 85:101200, June 2024. ISSN 1389-0417. doi: cogg6//.ogsss.2223..0
10

---

Published as a conference paper at ICLR 2026
Nathan Cloos, Moufan Li, Markus Siegel, Scott L. Brincat, Earl K. Miller, Guangyu Robert Yang, and Christopher J. Cueva. Differentiable Optimization of Similarity Scores Between Models and Brains. In The Thirteenth International Conference Learning Representations, October 2024.
Julia C. Costacurta, Shaunak Bhandarkar, David M. Zoltowski, and Scott Linderman. Structured flexibility in recurrent neural networks via neuromodulation. In The Thirty-eighth Annual Con- ference Neural Information Processing Systems, November 2024.
Francis Crick. The recent excitement about neural networks. Nature, 337/6203) January 1989. ISSN 1476-4687. doi: 10.1038/3371290
Kabir Dabholkar and Omri Barak. When predict can also explain: Few-shot prediction to select better neural latents, August 2025.
Kayvon Daie, Lorenzo Fontolan, Shaul Druckmann, and Karel Svoboda. Feedforward amplifica- tion in recurrent networks underlies paradoxical neural coding. bioRxiv, pp. 2023.08.04 0...0.8...2266. August 2023. doi: 10. 10.1101/2023.00 00.42......0..5222..
Laura N. Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):1349-136, July 2024.
ISSN 1546- 1726. doi: 10.10038/41593-004-016666
Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamics. Nature Neuroscience, 25(6):783-794, June 2022. ISSN 1546-1726. doi: 10. 10..0///41133-022-000000
Jenelle Feather, Guillaume Leclerc, Aleksander Madry, and Josh H. McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Na- ture Neuroscience, 26(11):0017-2044, November 2023. ISSN 1546-1726. doi: 10.1038/
1-15933-23-00444-00
Mikhail Genkin, Krishna V. Shenoy, Chandramouli Chandrasekaran, and Tatiana A. Engel. The dynamics and geometry of choice in the premotor cortex. Nature, 645(8079): 168-176, September 2025. ISSN 1476-4687. doi: 10.1008/s41586-11 (0088//4158-0005-99999.99
Matthew D. Golub and David Sussillo. FixedPoinFindder Tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of Open Source Software, 3(31): 1003, November 2018 ISSN 2475-9066.
doi: 10.21105/jos.000000.
Sarah E. Harvey, Brett W. Larsen, and Alex H. Williams. Duality of Bures and Shape Distances with Implications for Comparing Neural Representations. In Proceedings of UniReps: The First Workshop on Unifying Representations in Neural Models, pp. 11-26, May 2024a.
Sarah E. Harvey, David Lipshutz, and Alex H. Williams. What Representational Similarity Measures Imply about Decodable Information. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, October 2024b.
James Hazelden, Laura Driscoll, Eli Shlizerman, and Bric Shea-Brown. KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks, July 2025.
Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-37, December 1936. ISSN 0006-3444. doi: 1i..099/biomee22.34..333..
Ann Huang, Satpreet Singh, Flavio Martinelli, and Kanaka Rajan. Measuring and Controlling Solution Degeneracy across TrakeTr Recurrent Neural Networks, May 2025.
Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. Position: The Platonic Repre- sentation Hypothesis. In Proceedings of the 41st International Conference on Machine Learning, pp. 200270011-222 July 2024.
Mitra Javadzadeh, Marine Schimel. Sonja B. Hofer, Yashar Ahmadian, and Guillaume Hennequin. Dynamic consensus- consensus-building 2024.
building between neocortical areas via long-range connections, December
11

---

Published as a conference paper at ICLR 2026
Kristopher T. Jensen, Guillaume Hennequin, and Marcelo G. Mattar. A recurrent network model of planning explains hippocampal replay and human behavior. Nature Neuroscience, 27(7):1340- 1348, July 2024. ISSN 1726. doi: 10. 01/88//41593002--0110
Alexander J. E. Kell, Daniel L. K. Yamins, Erica N. Shook, Sam V. Norman- and Josh H. McDermott. A Task-Optimiie Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals Cortical Processing Hierarchy. Neuron, 98())x630-644.e16 May 2018. ISSN 0896-6273. doi: 10. 1010116 /001//..neuron... 2018 8.03.044.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neu- ral Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pp. 3519-3529, May 2019.
Renate Krause, Matthew Cook, Sepp Kollmorgen, Valerio Mante, and Giacomo Indiveri. Opera- tive dimensions in unconstrained connectivity of recurrent neural networks. Advances in Neural Information Processing Systems, 35:1 15777777-77085, December 2022.
Bariscan Kurtkaya, Fatih Dinc, Mert Yuksekgonul, Marta Blanco-Pozo, Ege Cirakman, Mark Schnitzer, Yucel Yemez, Hidenori Tanaka, Peng Yuan, and Nina Miolane. Dynamical phases of short-term memory mechanisms in RNNs. In Forty-Second International Conference on Machine Learning, June 2025.
Timothy Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, June 2020. ISSN 1471-0048, doi: 10.10038441533-00--077--33
Yuhan Helena Liu, Aristide Baratin, Jonathan Corfford, Stefan Mihalas, Eric Todd SheaBrown, and Guillaume Lajoie. How connectivity structure shapes rich and lazy learning in neural circuits. In The Twelfth International Conference on Learning Representations, October 2023.
Mikael Lundqvist, Pawel Herman, and Earl K. Miller. Working Memory: Delay Activity, Yes! Persistent Activity? Maybe Not. Journal of Neuroscience, 38(32):7013-700 August 2018. ISSN 0270-6474, 52011400. doi: 10. N0.553YNEURRSCC2448.1
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and individuality in neural dynamics across large populations of recurrent networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78 -84, November 2013. ISSN 1476-4687, doi: 10. 10.8/////t 12742.
Keith T. Murray. Phase codes emerge in recurrent neural networks optimized for modular arithmetic, July 2025.
Aditya Nair, Tomomi Karigo, Bin Yang. Surya Ganguli, Mark J. Schnitzer, Scott W. Linderman, David J. Anderson, and Ann Kennedy, An approximate line attractor in the hypothalamus encodes aggressive state. Cell, 186(1):178-1934 193.e15, January 2023. ISSN 0092-8674, 1097-4172. doi:
20.10166..el1..222.111.
Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and lla Fiete. Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis. Ad- vances Neural Information Processing Systems, 36:33824-3333 December 2023.
Marino Pagan, Vincent D. Tang, Mikio C. Aoi, Jonathan W. Pillow, Valerio Mante, David Sussillo, and Carlos D. Brody. Individual variability of neural computations underlying flexible decisions. Nature, 639(8054):421 March 2025. ISSN 1476-4687. doi: (0.1038/s41.88 586-024-08433--
Memming Park, Ábel Ságodi, and Piotr Aleksander Sokół. Persistent learning signals and working memory without continuous attractors, August 2023.
William Qian, Jacob A. Zavatone- Veth, Benjamin S. Ruben, and Cengiz Pehlevan. Partial obser- vation can induce mechanistic mismatches in data-constrained models neural dynamics. Ad- vances in Neural Information Processing Systems, 37:67467-67510 December 2024.
12

---

Published as a conference paper at ICLR 2026
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In Advances in Neural Information Processing Systems, volume 30, 2017.
Kanaka Rajan, Christopher Harvey, and David W. Tank. Recurrent Network Models of Sequence Generation and Memory. Neuron, 90(1):128-14) 142, April 2016. ISSN 0896-6273, doi: 10.1016/j. neuron.2016.02.009.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D. Cotterell. Linear Adversarial Con- cept Erasure. Pruceedings of the 39th International Conference on Machine Learning, pp- 18400-18444 18421, June 2022.
Laura Ritter and Angus Chadwick. Efficient Working Memory Maintenance via High-Dimensional Rotational Dynamics. bioRxiv, pp. 0.55099088.648336 September 2025. ISSN 2692-8205. doi:
10.1101/2025.09. 000.00225..9.08.778883333
Martin Schrimpf. Jonas Kubilius, Ha Hong. Najib J. Majaj. Rishi Rajalingham, Elias B. Issa, Kohi- tij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L.. K. Yamins, and James J. DiCarlo. Brain-Score: Which Artificial Neural Network for Object Recog- nition is most Brain-Like? bioRxiv, pp. 407007, January 2020. doi: 10.1101/47707.
Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in RNNs. In Advances Neural Information Processing Systems, volume 33, pp. 13352-13362, 2020.
Friedrich Schuessler, Francesca Mastrogiuseppe. Srdjan Ostojic. and Omri Barak. Aligned and oblique dynamics in recurrent neural networks. elife, 13, October 2024. doi: 10.7554/eLife. 93060.2.
Jimmy Smith, Scott Linderman, and David Sussillo. Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems. Advances Neural Information Processing Systems, volume 34, pp. 16700 11700-16711 2021.
Joana Soldado-Magraner Valerio Mante, and Maneesh Sahani. Inferring context-dependent compu- tations through linear approximations of prefrontal cortex dynamics. Science Advances, 10(51); ead14743. December 2024. doi: 10. 066/sciadv.adl4744.
Eelke Spaak, Kei Watanabe, Shintaro Funahashi, and Mark G. Stokes. Stable and Dynamic Coding for Working Memory in Primate Prefrontal Cortex. Journal of Neuroscience, 37027):6503-6516 July 2017. ISSN 0270-6474, 1529-2401.
doi: 10.553/NNEURSSCC3366666....
David Sussillo and L. F. Abbott. Generating Coherent Patterns of Activity from Chaotic Neural Networks. Neuron, 63(4):544-557, August 2009. ISSN 0896-6273. doi: 10. 00.10166j.eeuro.22009 07.018.
David Sussillo and Omri Barak. Opening the Black Box: Low-Dimensional Dynamics High- Dimensional Recurrent Neural Networks. Neural Computation, 25(3):626-666 649, March 2013.
ISSN 0899-7667. doi: 10. 10.1162/NECO. 4.0009.
David Sussillo, Mark M. Churchland, Matthew T. Kaufman, and Krishna V. Shenoy. A neural net- work that finds a naturalistic solution for the production muscle activity. Nature Neuroscience, 185)):025-1033, July 2015. ISSN 1546-1726. doi: 10.1038/nn.4044
Emily L. Sylwestrak, YoungJu Jo, Sam Vesuna, Xiao Wang, Blake Holcomb, Rebecca H. Tien, Doo Kyung Kim, Lief Fenno, Charu Ramakrishnan, William E. Allen, Ritchie Chen, Krishna V. Shenoy, David Sussillo, and Karl Deisseroth. Cell-type-speciff population dynamics of diverse reward computations. Cell, 185(19):5688-388 -58...22-7777 September 2022. ISSN 1097-4172. doi:
10. 10.1016/j.cell 10.10166f.ee11.222208..ooo
Elia Turner and Omri Barak. The Simplicity Bias in Multi-Tash RNNs: Shared Attractors. Reuse of Dynamics, and Geometric Representation. Advances in Neural Information Processing Systems, 36:25495-25507, December 2023.
13

---

Published as a conference paper at ICLR 2026
Elia Turnrr, Kabir V Dabholkar, and Omri Barak. Charting and Navigating the Space of Solu- tions for Recurrent Neural Networks. In Advances in Neural Information Processing Systems, volume 34, pp.
2021.
Adrian Valente, Jonathan W. Pillow, and Srdjan Ostojic. Extracting computational mechanisms from neural data using low-rank RNNs. Advances in Neural Information Processing Systems, 35: 24072-24086, December 2022.
Christopher Versteeg, Andrew R. Sedler, Jonathan D. McCart, and Chethan Pandarinath. Expressive dynamics models with nonlinear injective readouts enable reliable recovery of latent features from neural activity. In Proceedings of the 2nd NeurlPS Workshop on Symmetry and Geometry in Neural Representations, pp. 255-278, August 2024.
P.J. Werbos. Backpropagatton through time: What it does and how do it. Proceedings of the IEEE, 1550051 October 1990. ISSN 1558-2256. doi: 10.1109/5.8837.
Alex H. Williams. Equivalence between representational similarity analysis, centered kernel align- ment, and canonical correlations analysis. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, October 2024.
Alex H. Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized Shape Metrics on Neural Representations. In Advances in Neural Information Processing Systems, November 2021.
Daniel I.. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimize hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111228)8619-862, June 2014. doi: 10.1073/pnas. 1 403112111.
Guangyu Robert Yang. Madhura R. Joglekar, H. Francis Song, William T. Newsome, and Xiao-Jing Wang. Task representations neural networks trained to perform many cogni- tive tasks. Nature Neuroscience, 22(2):297-306, February 2019, ISSN 1546-1726. doi:
10.10038/41593-018--1111
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, pp. 12310-12320, July 2021.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 TRAINING AND OTHER MISCELLANEOUS DETAILS
For all experiments, we use RNNs with 128 neurons. All RNNS are trained in PyTorch. We use the Adam optimizer with a learning rate of 10-3 3 a weight decay of 10-6, and a batch size of 32. For the strength of the similarity penalty, we use À = 0.05 throughout. When computing linear predictivity, use p 10 as the ridge regularizer. RNNs trained as part of the INSD procedure are initialized at the scale g = 1. All networks are trained for a minimum of 10 iterations, with training terminating when the loss stops improving. Training runs were primarily done using 4th Generation Intel Xeon CPUs; GPU acceleration was not necessary.
For computing DSA dissimilarity matrices, 2023). Across all tasks, we used a rank of 100, 8 delays, and a delay interval of 10 timesteps. The delay parameters
the open source package from (Ostrow et al.,
selected to be compatible with trials of duration 100 timesteps, used for context-dependent integration and 3-bit flipflop.
For finding fixed points, use the open source package FixedPointFindr (Golub & Sussillo, 2018). We report approximate fixed points with velocities spanning = 5 x 10-1 to q = 10-9, and subsample redundant fixed points by adjusting the uniqueness tolerance parameter. As in (Driscoll et al., 2024), we report fixed points over a wide range of velocity tolerances to best account for variations in relevant timescales across the different tasks.
Training and analysis code is publicly available at
A.2 A BRIEF NOTE ON NEURAL SIMILARITY PENALTY LOOPHOLEES
We model the scenario described in the main text as follows: representations X, Y RRPx, Suppose that the representations in X contained in low dimensional subspace of dimension k < N, P. We represent this by factorizing X = LW, where L E Px are the latent representations and W E R* Suppose further that Y is com- posed of identical latents, along with some irrelevant noise in other dimensions. We write this as Y [LQ Z], where Q 1* is an orthogonal matrix, Z R Pxd represents the irrelevant noise, and d = N - k. For simplicity, we model the entries of Z as drawn i.i.d from ((0, 1). Below, we compute and describe the behavior of various similarity metrics on these inputs at large N, P, and Ø.
we are given two sets of neural
1. CENTERED KERNEL ALIGNMENT (CKA)
We focus on linear CKA:
|X 1 Y
CKA(X, Y)
(5)
X X
Y
We expand the numerator as I X
LQ
02| X Z 2
We also expand |Y L
L
2 L 2 +
Z 212.
At large N. we can approximate Zz /d +
Ip. This allows the simplification
=
Tr XX Z)= Tr(ZZ" XX d) X 1 and 1 Z d.P. At large o, drop subleading terms in a, giving
2dd X22
0O (VE
k
CKA(X,Y) 2dd P
(P)
(6)
X
X
where the final inequality follows from the bound
X 12 < V X||
Thus, CKA between otherwise identical representations can be suppressed through irrelevant noise.
A.2.2 REPRESENTATIONAL SIMILARITY ANALYSIS (RSA)
We take RSA to refer to the cosine similarity between the squared Euclidean distance representa- tional dissimilarity matrices (RDMs), as in Williams (2024).
15

---

kd k P Published as a conference paper at ICLR 2026
Let D -||z; 2yl? and D 13 yyll² represent the P x P RDMs. We have:
(D* (D)
RSA(X, x)= | D
(7)
We can write Di, Dij + D where S = =QQ Dropping terms subleading in o, have
(D*,D")
RSA(X, Y) | D2
(8)
Note that D 2d for * At large N (and therefore large d), expect concentration, yielding D /d + 2(J I), where J is x P matrix of ones. Thus, we have
(D PPP = -
Eij D
RSA(X, Y)
(9)
Thus, RSA is also suppressed by irrelevant noise.
A.2.3 LINEAR PREDICTIVITY [REF. + PENALIZED]
As in the main text, define the projection operator Ux X(x x)+x²
xT. We have that
2 (X,Y) = |
(10)
We can write | ||S|| + a² Ux where we have used that Ux S, by construction, S = LQ is contained the column space of = LW. Similarly, we have [xx 2 large Finally, large N, 12
|| || a² 2 2 yielding (2(X, Y)
have that
Ux 2
x 2 [2]
X
(11)
E
2|
Pd P'
demonstrating that linear predictivity in this direction is also suppressed by irrelevant noise.
A.2.4 LINEAR PREDICTIVITY [PENALIZED REF.
Consider the opposite direction:
JU 1
2 (YYX)
(12)
|
Since the column space of Y contains that of X, we have X |X| yielding ?²(, = 1. Thus, perfect linear predictivity is maintained.
A.3 TASK PARAMETERS
Context-dependent integration: We use timestep of dt = 0.1, a context-only duration Tpee = 2.5 (25 timesteps), and total trial duration T' 10 (100 timesteps). We set the noise scale to VO.1.
3-bit flipflop: We use a timestep of di = 0.2, and a total trial duration of T = 20 (100 timesteps). We set p 0.1 as the spike probability per timestep.
MemoryPro: We use a timestep of dt = 0.2. Mirroring timing parameters selected in (Driscoll al., 2024), we set T Trip 3/ dt, T Teip 7/dt, Trim T'em 2/dt, and T T = 16/dt. We use a noise scale of a = 0.1. As in (Costacurta et al., 2024), we scale down the output channel corresponding the fixation target by factor of 0.8.
16

---

Published as a conference paper at ICLR 2026
A.4
ADDITIONAL FIGURES
a
C
Ctx
Ctx B
Stin,
Output
Choice
Choice
cot.
Stm.
PC2
PC1
Charce
Choice
d
Ctx
Ctx B
10
Ctx
Ctx B
Sim
Imag
J
J
-10
-10
0
Sim.
Real
Figure A.1: Properties of the alt-2 RNN for the coneext-dependent integration task. a,b. Anal- ogous to Figs. lb.c. As for the alt-1 RNN, we observe oscillatory dynamics, well as fixed points with unstable oscillatory modes. However, these oscillatory modes are of much higher frequency. c,d. Analogous Fig. 2a. Unlike the alt-1 RNN, average trajectories plotted in the regression sub- space to some extent maintain the relative ordering of the coherences of both stimuli. This is likely explained by the alt-2 RNN still retaining degree of linear predictivity of standard RNN represen- tations, something that was entirely absent for the alt-1 RNN (Fig. Id). However, representations in the weight subspace reveal consistent representation of stimuli coherences.
linear predictivily
DSA embedding
activity
readout nullspace
g 0.01
p.-.S.
g
04
MDS2
1.5
28:1 Rea
-0.2
nef. RNN
p 9.1
g
ga.33
0.00
0.22
0.50
3rd RNN
MDS
Modar
2
Model
Figure A.2: Similarity measures across standard and
models trained on the 3-bit flipflop task. Figures are analogous to those in Fig. 4c,d. cemilarity-penalizze RNNs retain some degree of linear predictitty of standard RNNs, but that effect ablated once representations projected to readout nullspaces. As for other tasks, also observe DSA embdding that significantly separates the solutions monilarity-penaliiee RNNs from those found by standard RNNs.
17

---

Published as a conference paper at ICLR 2026
b
memory
response
alt-2
Output
PC1
PC1
PC2
PC2
Figure A.3: Properties of the alt-2 RNN for the MemoryPro task. Figures are analogous those in Fig. 4b. We again observe oscillatory dynamics supported by center of unstable fixed points. This RNN does poorly on the task relative to the RNNs shown Fig. 4b, as indicated by the activity itself prematurely acquiring significant output potence during the memory phase.
b
context-dependent int.
flipflop
Participation Ratio
Taoe
Taue
Truo
Participation Ratio
Tiuut 1.0
Taue 4.0
Tiaa 10.0
7.5
5.0
2.5
noise scale
noise scale
6
MemoryPro
Participation Ratio
a= 0.1
o= 0.2
o= 0.3
==
0.4
ref. RNN
alt-1 RNN
all-2 RNN
11119900
trained values
0.2
frac. memory load
Figure A.4: Effective dimensionality over different task conditions. Plots are analogous to those
in Fig. 5, but instead report the participation ratio, computed over neural activity during the task.
18

---

Published as a conference paper at ICLR 2026
A.5 LLM USAGE STATEMENT
Large language models were used sparingly for the sole purpose of trimming and polishing text. All technical contributions, experiments, analyses, and figures are the authors' own.
19


Reviews:
[{"review_id": "8gCOee6PSU", "reviewer": "Reviewer_ReZK", "summary": "The manuscript presents a new approach to train RNNs for the purpose of generating new hypotheses about neural mechanisms in neuroscience. Current approaches for training RNNs on neuroscience tasks tend to result in similar types of solutions, often relying on low-dimensional arrangements of stable fixed points and relatively simple dynamics around them. The authors’ use an iterative training procedure to find a family of RNNs that all solve a given task, but do so by implementing different dynamics and solutions. Specifically, each subsequent RNN in the family is trained with a loss that combines task accuracy with a penalty for solutions that are similar to those implemented by the previously trained networks. The authors apply this procedure to three well known and often used tasks of relevance to neuroscience and show that the resulting RNNs implement solutions that differ from those obtained with standard training approaches.", "questions": "Can the authors show in more detail how the alternative RNN solve their tasks? In particular, can they explain why the additional dynamics that distinguishes them from the standard networks is required to solve the task? \n\nStandard RNNs can often be understood with fixed points and dynamics around them, and the authors argue that the same descriptions to not apply to the alternative RNNs. But then how can these be understood and used to generate testable hypotheses?\n\nCan the authors’ approach be used to identify the various solutions to context dependent integrations found by Pagan et al?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, clear, and technically strong.\n\nGiven the increasing use of RNNs as models of neural computations, the paper addresses an important question, and develops a creative approach that may increase the number of hypotheses that can be generated with RNNs.\n\nThe paper puts forward a new metric to compare the dynamics of different RNNs that overcomes some of the limitations of previously proposed metrics.", "weaknesses": "The dissimilarity measure employed by the authors to compare RNN dynamics overcomes some of the limitations of previously proposed measures, in particular by preventing RNNs to simply add irrelevant dynamics in new subspaces to otherwise unchanged dynamics analogous to that in the standard RNN solution. However, it is not clear to me whether their dissimilarity can prevent more subtle ways to superficially modify the dynamics without changing its essence.\n\nFor example, one could imagine a scenario where an alternative solution to the context-dependent integrator appears much more high-dimensional and “curved” than in the standard solution, and yet locally the nature of the interaction between the inputs and slow dynamics are preserved. If such a solution was possible, a comparison of the trajectories alone would not adequately capture the fact that locally the solutions are quite similar in essence. \n\nMore generally, whereas the standard solutions are well understood at a variety of levels of description (e.g. the local relative arrangement of inputs, left and right eigenvectors in the context dependent integrator), the authors do not reach the same level of understanding for the alternative solutions. Without such a full understanding, it is challenging to decide how “different” the various solutions really are. \nIt does not help that the alternative RNN produce dynamics that seems substantially more complex than that in standard RNNs, and so understanding the underlying mechanisms (which ultimately is required to generate new hypotheses about brain function) may be more challenging. \n\nThe authors’ approach for generating alternative solutions is quite heavy handed, in that it forces solutions with very different trajectories, whereas it is known that the different mechanisms can produce very similar trajectories (recent Pagan et al work). Presumably, these different solutions could not be found with the approach presented in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RP5UZboSsD", "reviewer": "Reviewer_SAv4", "summary": "The authors propose an algorithm for training a population of RNNs on the same task, in a manner that encourages diverse solutions. This is done by adding a cost term that ensures activity in the space orthogonal to the output is orthogonal to previous networks. They show this generates solutions that differ in both geometry and topology.", "questions": "Line 148: this is somewhat similar to the argument of bijectivity in latent variable inference (Versteeg et al 2023, Dabholkar et al 2025).\n\nWhat is the capacity to produce different solutions? How many can there be, and what does this depend on? For instance, Training a 1-bit flip flop can be easily done with a 2-unit GRU. Can the method work already there (or maybe in 3D), thereby allowing phase space analysis?\n\nHow robust are these to noise in the actual dynamics, or in connectivity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "When RNNs are used as hypothesis generators, it is crucial to explore the space of hypotheses. This is usually done in a somewhat heuristic manner, and a more systematic approach is very important.\n\nThe idea is novel, original, elegant, and seems to work.", "weaknesses": "The other part of hypothesis generation is understanding the mechanism behind each RNN. This part was lacking. How exactly do the alternative RNNs solve the different tasks? In Turner 2021, the mechanism is explained. While this does not always have to be possible, it would be interesting to see a case where it is.\n\nAre the alternatives more or less similar to experimental data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VfAkD1DJSH", "reviewer": "Reviewer_657y", "summary": "This paper introduces Iterative Neural Similarity Deflation (INSD), a method for overcoming the simplicity bias that often causes recurrent neural networks trained on neuroscience-style tasks to converge to the same fixed-point or low-dimensional dynamical solutions. The authors propose penalizing the linear predictivity between neural activities of new and previously trained networks, effectively forcing networks to explore alternative representational geometries and dynamics. They test INSD on three canonical tasks—context-dependent integration, 3-bit flip-flop, and MemoryPro—and show that it yields qualitatively different solutions characterized by oscillatory or high-dimensional dynamics rather than static attractors. These alternative networks occasionally perform better under challenging, out-of-distribution conditions, suggesting that the INSD procedure can reveal functionally distinct computational strategies.", "questions": "1. The paper focuses primarily on relatively simple, neuroscience-style tasks. Is there a specific reason the authors did not extend their analysis to larger or more complex settings, such as multi-task RNNs or higher-dimensional architectures? It would be valuable to understand whether the proposed approach scales effectively beyond toy examples.\n\n2. The authors might consider situating their work in relation to recent theoretical ideas such as the [Contravariance Principle](https://arxiv.org/pdf/2104.01489) and the [Platonic Representation Hypothesis](https://arxiv.org/abs/2405.07987), both of which posit that sufficiently difficult tasks drive convergent representational structures across systems. Could the proposed INSD framework be used to empirically probe these claims—for instance, by testing whether multiple, distinct dynamical solutions can coexist for more complex tasks where such representational convergence is expected?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I really liked this paper. There is a notion in computational neuroscience that certain tasks somehow \"induce\", out of necessity, a particular type of solution in a network. This paper shows that there are many different potential solutions to a particular task.", "weaknesses": "The examples are all toy. I would be extremely interested to see how this technique performs in larger-scale settings, such as when training SSMs and/or Transformers on Long Range Arena (or a similarly difficult benchmark).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)