Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs
Authors: Manuel Brenner
Georgia Koppe
Venue: Accepted by TMLR ()

Abstract: Sequence modeling tasks across domains such as natural language processing, time-series forecasting, speech recognition, and control require learning complex mappings from input to output sequences. In recurrent networks, nonlinear recurrence is theoretically required to universally approximate such sequence-to-sequence functions; yet in practice, linear recurrent models have often proven surprisingly effective. This raises the question of when nonlinearity is truly required. In this study, we present a framework to systematically dissect the functional role of nonlinearity in recurrent networks -- allowing to identify both when it is computationally necessary, and what mechanisms it enables. We address the question using  Almost Linear Recurrent Neural Networks (AL-RNNs), which allow the recurrence nonlinearity to be gradually attenuated and decompose network dynamics into analyzable linear regimes, making the underlying computational mechanisms explicit. We illustrate the framework across a diverse set of synthetic and real-world tasks, including classic sequence modeling benchmarks, an empirical neuroscientific stimulus-selection task, and a multi-task suite. We demonstrate how the AL-RNN's piecewise linear structure enables direct identification of computational primitives such as gating, rule-based integration, and memory-dependent transients, revealing that these operations emerge within predominantly linear dynamical backbones. Across tasks, sparse nonlinearity plays several functional roles: it improves interpretability by reducing and localizing nonlinear computations, promotes shared (rather than highly distributed) representations in multi-task settings, and reduces computational cost by limiting nonlinear operations. Moreover, sparse nonlinearity acts as a useful inductive bias: in low-data regimes, or when tasks require discrete switching between linear regimes, sparsely nonlinear models often match or exceed the performance of fully nonlinear architectures. Our findings provide a principled approach for identifying where nonlinearity is functionally necessary in sequence models, guiding the design of recurrent architectures that balance performance, efficiency, and mechanistic interpretability.

Published in Transactions on Machine Learning Research (01/2026)
Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs
Manuel Brenner1.2.
Georgia Koppe 1,46
Interdisciphnaayy Center for Scientific Computing, Heidelbery University Frrsst Stringmann Institute of the Max Planck Society, Frankfurt 2 Dept. of Psychiatrr, Goethe University Frankfurt 4 Hector Institute for Al in Psychiatry and Dept. of Psychiatry and Psychotherapy, Central Institate for Mental Health Mannheim 9 Herrte Institute for Al in Brain Health, Tübingen
Reviewed on OpenReview: // openrevie.. net/ forum? id= qI2Vt9P9r1
Abstract
Sequence modeling tasks across domains such as natural language processing, time-series forecasting, speech recognition, and control require learning complex mappings from input to output sequences. In recurrent networks, nonlinear recurrence is theoretically required to universally approximate such sequence-to-ssequene recurrent models have often proven surprisingly effective. This raises the question of when nonlinearity is truly required. In this study, we present a framework to systematically dissect the functional role of nonlinearity recurrent networks- allowing to identify both when it computationally necessary, and what mechanisms enables. We address the question using Almost Lincar Recurrent Neural Networks (AL-RNNSs), which allow the recurrence nonlinearity to be gradually allenuated and decompose network dynamics into analyzable linear regimes, making the underlying computational mechanisms explicit. We illustrate the framework across diverse set of synthetic and real-world tasks, including classic sequence modeling benchmarks, an empirical neuroscientific stimulus-selectoon task, and a multi-task suite. We demonstrate how the AL-RNN's piecewise linear structure enables direct identification of computational primitives such as gating, rule-based integration, and memory-depended transients, revealing that these operations emerge within prodominantly linear dynamical backbones. Across tasks, sparse nonlinearity plays several functional roles: it improves interpretability by reducing and localizing nonlinear computations, promotes shared (rather than highly distributed) representations in multi-task settings, and reduces computational cost by limiting nonlinear operations. Moreover, sparse nonlinearity acts as useful inductive bias: low-data regimes, or when tasks require discrete switching between linear regimes, sparsely nonlinear models often match or exceed the performance of fully nonlinear architectures. Our findings provide a principled approach for identifying where nonlinearity is functionally necessary in sequence models, guiding the design of recurrent architcctures that balance performance, efficiency, and mechanistic interpretability.
functions; yet in practice, linear
1 Introduction
Modeling sequences with temporal dependencies remains a core challenge in machine learning, with applications spanning natural language processing, speech recognition, vidco analysis, control, and forecasting (Hochreiter and Schmidhuber, 1997; Vaswani al., 2017; Lillicrap al., 2019; Lim et al., 2021). Sequence modeling tasks often depend long-range memory (Bengio et al., 1994), requiring models to store, transform, and retrieve

---

Published in Transactions on Machine Learning Research (01/2026)
information across time. Classical results have long pointed to the necessity of nonlinear recurrence for such operations, showing that recurrent neural networks (RNNs) with nonlinear dynamics are both universal approximators and Turing complete (Siegelmann and Sontag, 1995).
Yet, recent work has revived interest in linear recurrent models, particularly due to their computational advantages (Orvieto et al., 2023). Linear systems support parallel computation and scale efficiently (Martin and Cundy, 2018), and when equipped with expressive input and output mappings perform surprisingly well on long- horizon tasks (Gu et al., 2022; Orvicto et al., 2023). These findings revive deeper question: How can we systematically assess to what extent nonlinearity is necessary for temporal computation, and what role it plays?
This question is especially relevant when considering interpretabilily. Linear systems, despite their dynamical limitations (Strogatz, 2018), are analytically tractable and structurally transparent, making them appealing in settings where understanding the internal dynamics of a model as important as its raw performance (as, for instance, in neuroscience; Sussillo and Barak (2013); Durstewitz el al. (2023)). Nonlinear systems, by contrast, afford richer dynamics, but typically introduce a higher degree of complexity. Reflecting this trade-off, the landscape of sequence models has grown into a diverse collection of architectures combining linearity and nonlinearity in different ways (Patro and Agneeswaran, 2024), for instance through mechanisms such as input-dependent parameter nonlinearities (Gu and Dao, 2023) or gating mechanisms (Mehta et al., 2023). These innovations, while empirically successful, often blur the functional roles of linear and nonlinear components, obscuring their individual contributions to sequence modeling.
Motivated by this, we aim to provide framework that disentangles how, when, and if nonlinearity is functionally necessary, and to demonstrate its utility for uncovering the computational mechanisms underlying sequence modeling. We tackle this question using the Almost-Linear RNN (AL-RNN), a model initially proposed for dynamical systems reconstruction (Brenner et al., 2024a). Here, we repurpose the AL-RNN as a natural tool to investigate a wide class of sequence modeling tasks. It is uniquely suited for this purpose because it makes noulinearity both controllable and observable. Built from both linear and piecewise-lineat units, the AT-RNN allows to lune the strength of nonlinear recurrence by varying the number of nonlinear units. At the same time, its piecewise-linear structure partitions the hidden state space into distinct linear subregions. Tracking which subregion is active over time yiclds symbolic bitcode trace of the computation, which we use to analyse the switching structure that implements temporal processing. Finally, the linear dynamics within each subregion enable complementary dynamical analyses (e.g., fixed points and stability) (Eisenmann et al., 2024), connecting the bitcode-level view to interpretable dynamical mechanisms.
We apply the AL-RNN across a diverse set of classic memory benchmarks, a multi-task setting (Driscoll et al., 2024), as well as an empirical dataset of neural recordings from a rodent performing a stimulus selection task, using each setting to demonstrate how nonlinear mechanisms can be systematically identified, while exploring scenarios where linearity suffices, minimal nonlinearity is beneficial, nonlincarity is required. In doing so, we reveal task-specific nonlinear mechanisms such as gating, rule-based integration, and memory-dependent transients, often embedded within a mostly linear dynamical backbone.
strong
In most single task settings, reducing nonlinearity matches or even exceeds the performance of fully nonlinear models while retaining interpretability and making computational motifs and mechanisms directly traceable. In the multi-task sctting, forces reuse of small set of nonlinear computational motifs, promoting shared representations across tasks, improving sample efficiency and exposing a mechanistic map of cross-lask structure. Collectively, our results suggest that sparse nonlinearity provides a useful inductive bias that enables the emergenec of nonlincar motifs where needed, while preserving prodominantly lincar dynam- ics. Overall, our study sheds light on the minimum necessary conditions for memory in AL-RNNs and related architectures, offering a principled framework guiding the design of sequence models that are both computationally efficient and analytically tractable.
2 Related Work
The ability to represent, retain, and manipulate memory over time has been a central challenge in sequence modeling. Early RNNs, such as those proposed byman (1990) and Jordan (1997), introduced the paradigm
2

---

Published in Transactions on Machine Learning Research (01/2026)
of learning temporal dependencies through hidden state recurrence. However, these simple RNNs struggled to capture long-range dependencies due to vanishing and exploding gradients (Bengio et al., 1994). This limitation was addressed by architectures like Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (Chung et al., 2014). Such gated RNNs, and more recent extensions such as Long Expressive Memory (Rusch et al., 2022), established gated recurrence as cornerstone of sequence modeling, supporting stable memory retention across extended time spans. These models successfully mitigated gradient issues, but the introduction of nonlinearity through gating also increased the complexity of their internal state dynamics. To address this, an alternative line of work has focused on combining the expressivity of nonlinear dynamics with the analytical tractability of linear models. This gave rise to piecewise linear and switching models such as switching linear dynamical systems (Fox et al., 2008; Linderman et al., 2016; 2017) and piecewise linear recurrent neural networks (PLRNNs) (Durstewilz, 2017; Brenner et al., 2022; 2024a). These architectures segment the state space into locally linear regions, enabling rich dynamical behaviors through structured transitions between subspaces while preserving mathematical tractability.
While the computational complexity of RNNs scales with O(T), T being the sequence length, the inherently sequential nature of RNNs makes their training inefficient to parallelize on modern GPU hardware. The Transformer architecture (Vaswani et al., 2017) has therefore largely replaced recurrence with global self- attention across many sequence-modelligg tasks. Self-attention enables parallel sequence processing without relying on a persistent hidden state. This parallelism, however, comes at the cost of computational efficiency, scaling with O(T²). Further, it models memory only within a finite context window. As a result, Transformers lack causal, evolving memory, making them harder to interpret and less inherently efficient. for very long-range dependencies (Patro and Agnceswaran, 2024).
This quadratic scaling has motivated the exploration of more efficient alternatives for long-sequence modeling. Recent interest has focused on structured linear state-space models (SSMs). These models implement linear recurrence, which can be unrolled using spectral methods or convolutions. Advanced parallel scanning techniques, such as Blelloch's scan (Blelloch, 1990), enable linear RNNSs to achieve highly efficient paral- lelization, reducing complexity to Oloog T') (Martin and Cundy, 2018). Theoretical advancements have even demonstrated that deep linear RNNs with nonlincar mixing layers are universal approximators of regular sequence maps, broadening their applicability to complex temporal tasks (Orvieto et al., 2023; 2024). Modern SSMs like $4 (Gu et al., 2022), grounded in HiPPO theory (Gu et al., 2020), leverage these principles to match, or even surpass, Transformers on long-range sequence benchmarks (Tay et al., 2021). The Legendre Memory Unit (LMU) (Voelker et al., 2019) similarly enforees a fixed lincar memory subspace derived from continuous- dynamics, combined with a nonlinear update. Recent extensions have roduced more efficient initialization schemes and streamlined designs (Gupta et al., 2022; Smith et al., 2023; Hasani et al., 2023), enhancing both stability and performance. Some models, like Mamba (Gu and Dao, 2023) and Gated SSMs (Mehta et al., 2023), reintroduce nonlinearity through structured gating mechanisms while preserving the parallelizable structure of linear SSMs. Sequence models also increasingly incorporate hybrid architectures that blend recurrent, convolutional, and attention mechanisms (Peng, 2023; Qin et al., 2023; Poli et al., 2023; Lieber et al., 2024; Sieber et al., 2024). However, this proliferation of architectures, driven primarily by performance and efficiency, makes mechanistic interpretation increasingly challenging, as the structural assumptions cach model encodes often remain implicit.
3 Method
Almost-Linear Recurrent Neural Networks To dissect the roles of linear and nonlinear recurrence, we adopt an SSM-inspired architecture: the recently proposed AL-RNN (Brenner et al., 2024a). In this model, latent dynamics evolve under a combination of linear and PWT. transition functions, modulated by external inputs. Nonlinearities are applied only to a subset P & M of the hidden state, making AL-RNNs a simplified variant of piecewise lincar RNNs (Durstewitz, 2017; Brenner et al., 2022). An AL-RNN is defined by:
2 = A + W (x-1= + hh
(1)
3

---

Published in Transactions on Machine Learning Research (01/2026)
where 21 E RM is the system's latent state. The function (=(-) applies nonlinearity selectively to only the last. P units:
@ (z,) = [2,,, P.t: ((2M-((1))))))))) (....M..).......(.....). ((2M,)))
(2)
where o(-) a scalar nonlinearity. The original AL.RNN formulation (Brenner et al., 2024a) used the ReLU nonlinearity 6(2) max(0,2) 2). While we also tested other nonlinearities including GeLU, tanh, and hardtanh, only PWL functions like ReLU preserve the property that the system operates define A E RMMx a diagonal matrix where entries corresponding to the linear (first M P) units are set to zero, A = diag(O,.... ...... 0, aM -M--P1,..... RMM) so that linear self-connections are only assigned to the nonlinear units. The matrix W E RMxM captures interactions based on the partially nonlinear transformed state (z), E R weighs the K-dimensional external inputs 8t; and h E TRM bias term.
composition of linear
dynamical systems. To avoid redundant paramelerization,
M,t
linear units (L=4)
M=6
piecewise linear units (P=2)
01
11
Z M
00
10
# (((r)
[Ra,e)n Far P20 max (0, FAA -MM--41,)) max((, =M.,))
11
2e+1 1z
Ze+11 w
24+1 W 2
Figure 1: Ilustrationn of the AL-RNN and bitcode assignment. The example displays a 6-dimensional AL-RNN with four linear and two PWL units (left). The PWL units partition the state space into four lincar subregions (right). Bitcodes encode positive (1) and negative (0) activation values for these units. Each subregion corresponds to a unique bitcode (00, 01, 10, 11) and governed by a distinct linear dynamical system (DS) with its own recurrence matrix Was An example trajectory (bottom) traverses sequence of these subregions time, experiencing discrete switches in dynamics when crossing subregion boundarics (marked by dashed vertical lines). Within each subregion, the dynamics remain linear.
Computational Mechanisms in RNNs Sequential tasks require recurrent networks to store information over time and to manipulate it in ways that depend on both past states and incoming inpuls. These functions can be understood through the dynamics that govern hidden states and the computations they enable, implementing objects such as slow modes, attractors, and transient trajectories. From computational perspective, these structures implement operations such as accumulation, recall, gating, and context-dependent routing.
Linear RNNS exhibit a restricted but analytically tractable set of dynamics, governed by the spectral properties of the recurrence matrix (Strogatz, 2018). Autonomous dynamics are limited to fixed points and modal decompositions: slow modes (eigenvalues near the unit circle), which support information storage through leaky integration graded retention, complex eigenpairs, which generate oscillatory modes for rhythmic phase-sensitive coding, and unstable modes, which can produce exponential growth decay depending gain and initialization (Seung, 1996; Ratcliff and McKoon, 2008; Gold and Shadlen, 2007; Henaff et al., 2016). These autonomous motifs can support stable (or deliberately unstable) memory traces but, because the update is globally linear, do not. by themselves implement state-dependent reconfiguration. Input- driven dynamics in linear RNNs therefore reduce to uniform accumulation or filtering, lacking mechanisms for gating, selective recall, or context-dependent switching. A key strength lies in their analytic transparency and stability when dominated by slow modes, but their computational repertoire fundamentally limited compared to nonlinear models.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Nonlinear RNNs differ both in terms of the autonomous dynamics they can generate, and how they can integrate external inputs. Autonomous nonlinear recurrence enables multistability, where multiple attractors allow for distinct, stable internal states that support e.g. associative memory (Hopfield, 1982; Amari, 1977). More complex motifs such as k-cycles, chaotic attractors, itinerancy, and metastability generate structured trajectories that do not converge to a single fixed point, supporting sequential processing and structured recall (Rabinovich et al., 2008; Tsuda, 2015). In terms of input integration, thresholded activations can act as switches, enabling state-dependent transitions and context-dependent routing (Balaguer-Ballee Ballester et al., 2011; Wang, 2008). This allows identical imputs to produce different outcomes depending on the internal state. Nonlinear gating, in turn, selectively integrates or suppresses information (Ackley et al., 1985). Nonlinear RNNs therefore excel when tasks require flexible memory or complex temporal structure, but their richer dynamics also make them harder to infer, analyze, and track compared to linear models.
Piecewise Linear RNNs aim to reconcile these trade-offs by combining the analytical tractability of linear dynamics with the computational expressivity of nonlinear modcls. From a dynamical systems perspective, piecewise linear systems offer principled and general framework: any sufficiently smooth nonlinear DS can be approximated arbitrarily well by partitioning the state space into linear subregions (Storace and De Feo, 2004). The AL-RNN, which we employ here, implements the simplest version of this idea that can be Icarned automatically from data in an end-to-end fashion. It makes no assumptions about the switching mechanism itself, and requires no prior specification of how many subregions are needed or where transitions should occur. The AL-RNN partitions the state space into 2" linear subregions separated by switching boundaries, within which the dynamics are linear and analytically tractable (sce Fig. 1). Each linear subregion of the AL-RNN can be uniquely identified by a binary bitcode of length P, corresponding to the on/off (positive/negatiive) activation state of the P ReLU units (Fig. 1,rihh)). Since the ReLU nonlincarity partitions cach dimension at zero, this yields 2f possible configurations of active units. By tracking which bitcodes occur during inference, we can quantify and visualize how many of these subregions are actually used by the network, offering a compact representation of its functional complexity. This PWL structure thus inherits the advantages from both paradiguns: the analytic transparency of linear systems within each subregion, combined with the representational flexibility of nonlinear models. Further, transitions between subregions are explicitly detectable, and within each regime, the system's 's dynamies can be analytically characterized in terms of fixed points and stability properties (Durstewitz, 2017; Eisenmann et al., 2024) (sce Appx. A.2).
negative)
To promote the reconstruction of both fast and slow time scales and associated memory in the latent space, we incorporate a manifold regularization term based on Schmidt et al. (2021) (see Appx. A.1.2 for details and performance comparisons). The AL-RNN processes latent inputs {st)t=177 derived from raw inputs {# (for details, see Appx. A.1.3), and task outputs are decoded from the latent states {ath=r the final state ør via linear readout layer. This consistent use of linear decoding is central to our interpretability framework. By constraining the decoder to be linear, we ensure that the recurrent dynamics themselves must carry the full computational burden of the task solution. Since the AL-RNN implements a first-order Markov process, this task solution naturally takes the form of a DS, mirroring the classical computational neuroscience perspective on how stateful recurrence encodes computation through evolving trajectorics (Durstewitz et al., 2023).
Training Details
4 Experimental Results
Task Selection and Experimental Structure To probe how recurrent nonlinearity shapes memory, we selected set of tasks spanning those that can be solved by linear dynamics and those that require nonlinear computation, allowing us to highlight distinct linear and nonlinear memory mechanisms (Sect. 3) engaged by different computational demands. In addition, to assess the robustness of the framework under more challenging conditions, we apply it to multi-task setting and to an empirical neuroscientific datasct. Our primary goal is interpretability rather than performance optinization, making direct architectural comparisons less straightforward. However, to contextualize AL-RNN performance, we included standard gated RNN basclines (LSTM, GRU) on the performanco-oriented benchmarks (sMNIST, Speech Commands, Copy, Addition), and investigated the influence of sparse nonlinearity within S4 and a Linear Transformer on the Copy and Addition task (see Fig. 27 and Table 2). We also compared AL-RNNs with different activation
5

---

Published in Transactions on Machine Learning Research (01/2026)
functions (ReLU, GeLU, tanh, hardtanh), finding qualitatively similar results (see Table 3), though some systematic differences emerge, discussed further in Sect. 4.3.
4.1 Modility-Specific Encoding Enables Simple Linear Integration
We begin with a class of tasks that require gradual accumulation of information over time to support a final decision. At each time step the AL-RNN's latent state 2; integrates incoming representations over time, and the class label 9 is predicted from the final state 2r. We test this sctup across images, audio, and text, employing apmaan-approoprate encoders to map raw imputs to latent representations: pretrained GloVe embeddings (Pennington et al., 2014) for text, 1D convolutional layers for sequential pixels, and spectral features for audio (see Appx. A.1.3). While these encoders necessarily differ in their treatment of modality-specifn structure, they serve the common purpose of transforming heterogeneous inputs into shared representational space amenable to temporal integration. For instance, the GloVe embeddings provide scmantically meaningful representations where valenced words (c.g., 'good" vs. "bad") are already separable, putting focus on the temporal integration of information rather than learning semantic structure from scratch. By constraining the recurrent dynamics to operate on these normalized representations and requiring classification through a lincar readout, we enforce that the main computational mechanism, the temporal integration itself, is expressed explicitly in the AL-RNN's dynamics, revealing that despite the diversity of input modalities, the underlying task solution converges to the same dynamical principle across all three domains.
b
C
Regative Prefiiu
periible
pffrrt
Froquetio mrqquencc
P=10
$3.3
-20
$3.0
0.0
125
Munbbo of Fendinece anRoS 1r1
PC1
PwO, test eccuracy=87
P=50, lest aocuracym .5%
Pe 1o00 test Bccuracy= 0%
00.4
034410 150
97.6
07.2
10
20
10100
K2
Figure 2: Top row: Sentiment classification on IMDb reviews, a: Test accuracy (y-axis) as a function of nonlincar units, P. b: Trajectory through latent space showing sentiment-relevant keywords guiding classification. c: The dominant first PC clearly separates positive and negative reviews. d: Bitcode frequencies are highly concentrated on a small subset of linear subregions. Bottom row: Digit classification on sequential MNIST, e: Test accuracy (y-axis) a function of nonlinear units (x-axis; mean + std over 10 seeds). f: Final latent state projection onto the first 3 PCs show that nonlinearity partitions latent space by class.
For binary sentiment classification using the IMDb dataset (Maas et al., 2011), the integration mechanism is remarkably simple, and increasing nonlinearity has minimal effect on performance (Fig. 2a; Welch's f-test. comparing
vs P 100: t 1.34, p p= 0.20). During a negative review (Fig. 2b), large updates to the latent state occur almost exclusively when highly valenced keywords such as worst or terrible are encountered, suggesting that memory integration is dominated by sparse lexical cues already emphasized by the input embedding. Accordingly, the dynamics are dominated by a single slow mode: approximate line attractor with one eigenvalue near and all others substantially smaller The eigenvector associated with the slow mode is almost perfectly aligned (cosine similarity > 0.999), which alone accounts for 98%
We the "line attractor' functional sense: the dynamics exhibit very slow integration along dominant mode (eigenvalue 1) that preserves information over resk-relevant timescales. Strictly speaking, true line attractors require

---

Published in Transactions on Machine Learning Research (01/2026)
of the total variance, indicating that the system's activity is effectively constrained to a one-dimensional (semtiment-relatel) integration axis (Fig. 2c), mirroring prior observations in the literature (Maheswaranathan et al., 2019). The distribution of nonlinear bitcodes is correspondingly degenerate (Fig. 2d); a handful of codes account for almost all examples.
We observed similar slow-mode integration mechanisms in image classification (Sequential MNIST= sMNIST) and audio classification (Speech Commands), detailed in Appx. A.4.2. While the overall dynamical mechanism is very similar, in both cases, the data manifold is higher-dimenstonall and requires more complex latent representations. Here, sparse nonlinearity provides modest but significant improvement (Fig. 2e for sMNIST, Fig. 13 for Speech Commands).
For sMNIST, bitcode analysis reveals a clear mechanism by which noulinearily alfects performance: each of the 10 digit classes aligns with its own set of closely neighboring bitcodes, enabling class-specific linear integration dynamics (Fig. 10). While all classes are processod through slow modes with cigenvalues near 1 (Fig. 9), small differences in these eigenvalues, arising from distinct linear subregions, cause trajectories to diverge into more well-separated clusters (Fig. 2f). Performance peaks at intermediate nonlinearity and declines for fully nonlinear models, which overdisperse the data manifold (Fig. 12). The fact that this same slow-mode integration structure emerges across text, images, and audio demonstrates a universal computational motif for evidence accumulation tasks, consistent. with similar mechanisms observed in biological systems (Seung, 1996; Ratcliff and McKoon, 2008; Maheswaranathan et al., 2019).
4.2 Minimal Nonlinearity Stabilizes Transients For Structured Memory Recall
Next, to evaluate the AL-RNN's ability to sustain stable internal representations over time, we tested it on the classic copy task (Fig. 3a). During the encoding phase, the network receives a random sequence of Naym distinct symbols, presented one per time step. This is followed by a delay of length D time steps with no input, and then a "cue" signal indicates that the network should reproduce the original sequence over the next Neeq time steps (the "recall phase"). Performance is measured as the number of sequences recalled correctly. This task is particularly interesting from a neuroscience perspective, as it parallels experimental paradigms used to study working memory in prefrontal corlex, where stable task-relevant information must. be maintained over delay intervals despite complex and heterogeneous underlying dynamics (Murray et al., 2017; Rajan et al., 2016).
In this configuration (Neym = 4, Neeg = 8 and D = 200), we first. find that purely linear models perform above chance, suggesting they can partially solve the task. This aligns with theory: linear systems can support marginally stable cycles when eigenvalues lie on the unit circle, enabling long recurrence times that could, in principle, encode sequence and timing. However, such solutions are highly fragile to noise and perturbations (Strogatz, 2018), confirmed by the observation that despite extensive hyperparameter tuning, linear models fail to solve this task with high accuracy (Fig. 3c). Minimal nonlinearity therefore greatly improves performance and robustness. A single nonlinear unit (P = 1) often yields perfect recall, with performance declining for increased nonlinear units and being worst for fully nonlinear models (Fig. 3c).
To understand the underlying computational mechanism underlying successful solutions, we analyzed a representative AL-RNN with P = 1 that perfectly solves the task, i.e. decodes all encoded symbols with 100% accuracy on the test set. We find that the PWL unit leverages two distinct linear subregions to separate mooding//ecoddig vs. storage dynamics. During the encoding and decoding phases, the network remains confined to the primary linear subregion optimized for integrating and recalling symbols (Fig. 3b, top). In contrast, during the delay phase, the net transitions to the second linear subregion, where it implements a complex transient cycle that stores the encoded sequence. Because the encoded sequence uniquely determines the entry point into the cycle's transient, the network preserves this phase information throughout the delay, allowing the sequence to be decoded from the corresponding exit point at recall.
Crucially, we find that sparsely nonlinear models often only use small subset of available linear subregions (Fig. 3d, Fig. 16 for a full statistical analysis). Fig. 17a illustrates why fully nonlinear models fail to converge
continuous manifold of neutrally stable fixed points, while our linear models instead have single fixed point with one very slow decay mode.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Input
Delay (200 time steps)
Recall
Input
Delay
Recall
Syme P
Input
Target
Prediction
Steps
Symbo
PWL Unit
-10
Time
200
400
600
800
1000
1200
1400
Time Stops
100
d
0.20
accuracy
Frequency
0.4
P=10
00115
60
9.10
40
0.2
0.05
13
34
15
Bitcode
Component
Figure 3: a: Structure of the copy lask and two example trajectories for P= b: Top: Activity of the PWL unit (P=1) for the input sequences in The latent activity follows complex limit cycle primarily located in one linear subregion (PWL unit negative), which switches to the second subregion during decoding (PWL unit positive). Bottom: Autonomous activity of the AL-RNN in the absence of inputs encodes a 100-cycle, with its transient located only in one subregion (PWL. unit negative). c: Symbol-wise recall accuracy (mean std over 10 seeds) vs. number of PWL units P. d: Histogram of binary "bitcodes" during recall for P = 10, ratcentrated on small subset out of 21 1024 possible bitcodes. e: Explained variance ratio of PCs of latent network activity (for P 1) indicates relatively nigh-dimensional, complex dynamics.
to good solutions: their loss curves are highly irregular and feature frequent bifurcations, directly linked to high gradient norms throughout training. For trained models, solutions are spread out across hundreds of subregions (bitcodes), failing to learn simple task solutions (Fig, 18a). For results on the variable delay task, where the recall cue occurs after a random interval, see Sect. A.4.3.
4.3
Nonlinearity Enables Gating in the Addition Task
10-1
b
0.6
Input
Heme IeputDulppe
C
1.0
10-2
80
0.5
Prediciad Dulput
Trrgee Mosited Input
PWL Unit
20-3
10"4
Prodicted
10-5
Targer
Number nonlinear units
0.0
PWL Unit
20
80
100
Stops
Figure 4: a: Performance function of units P (mean std over 10 seeds). b: Latent trajectories in PWL space with P 2 noulinear units. The second linear subregion is selectively activated only at the two masked time points ('input"), indicated by sharp transitions across quadrant boundaries. Outside these events, the network remains in a single linear regime, leading to smooth integration of the cumulative sum. c: Time course of masked inputs (red), network outputs (grey/black), and target values (blue) for two example trials, where the input occurs either early (light grey) or late (black) in the sequence. In both cases, the trajectory initially follows linear drift prior input, then transitions sharply to an elevated integration path with the offset. same slope Alfsee to reach the larget after 100 steps.
Next, we consider the addition problem (Hochreiter and Schmidhuber, 1997), an established test of selective memory. The input consists of continuous stream of random numbers, along with sparse binary mask
8

---

Published in Transactions on Machine Learning Research (01/2026)
indicating two marked time points. The goal is to output the sum of the two marked times, ignoring all other inpuis. Crucially, the mask is not directly fed into a separate gating mechanism, but the AT-RNN must internally learn to accumulate inputs only at the relevant times. This requires the network to integrate incoming values differently depending on the input. This task is impossible to solve by a linear AL-RNN (sec Appx. A.4.1 for a formal proof), as accurately identified by our pipeline (Fig. 4a). In contrast, with just a single nonlinear unit (P 1) the AL-RNN achieves magnitud lower error.
Fig. 4b illustrates that nonlinearity enables the emergence of an internal gating mechanism, where the AL-RNN learns to selectively 'open' and 'close' an internal integration pathway conditional on the input mask. To sum the inputs, the network leverages a slow linear drift that accumulates over the full sequence length (Fig. 4c), while nonlinear units rapidly reposition the state onto the appropriate integration trajectory when gated imputs occur. This mechanism cleanly separates the functional contributions of linear and nonlinear units and aligns with theoretically derived optimal PWL solutions for this task previously described in the literature (Monfared and Durstewitz, 2020; Schmidt et al., 2021). Interestingly, as before, oven with increasod nonlinearity, the network preserves this strategy (Fig. 14a), distributing linear integration and gating across two distinct small sets of subregions. While fully nonlinear models can outperform sparsely nonlinear models (Fig. 18b), some fully nonlinear models fail to capture this simplicity of the task mechanism effectively, fragmenting the linear accumulation mechanism (Fig. 18b, middle vs. right), an effect that is conserved across training and test set (Fig. 14c). In line with this, we find that fully nonlinear models frequently "linearize" units by shifting their mean activity far from zero, effectively enforcing linear dynamics but at the cost of more difficult optimization (Fig. 14b).
Interestingly, all tested nonlinearities showed similar performance pattern: marked improvement at P 1, modest gains with sparse nonlinearity, stable plateaus at intermediate P, and worse performance at full P (Table 3). What varied was how cleanly interpretable mechanisms emerged. The addition task requires combining two linear processes: slow drift accumulating time, and selective integration of marked inputs. PWL activations like ReLU naturally separate these into distinct lincar subregions, whereas tanh must approximate the separation by shifting activity between near-linear and saturated regimes. This softer partition is harder to learn and yields roughly an order of magnitude higher MSE, highlighting the inductive bias of PWL units for switching-linear tasks.
4.4 Nonlinearity Facilitates Routing in Contextual Integration
Beyond simple integration, we examined context-dependent tasks where the model must respond differently to identical stimuli based on preceding rule cues. We lested this in two seltings: a simple benchmark requiring conditional integration of a stimulus sequence, and neuroscientific task involving joint modeling of both behavioral decisions and neural population activity from rodent auditory cortex (PFC-1 dataset, Rodgers and Weese (2014)). For the latter, we jointly trained the AL-RNN to both solve the task and reconstruct neural population activity from the same latent states.
In both settings, linear models fail completely, plateauing at chance performance, while introducing even a single nonlinear unit enables near-perfect accuracy (Appx. A.4.5& A.4.4). In both cases, the central mechanism was straightforward to interpret: nonlinearity routes different task rules into distinct linear subregions, implementing context-dependent gating that lincar dynamics fundamentally cannot achieve. While the overall task exhibits nonlinear dependencies, requiring opposite responses to identical inputs depending on context, the piecewise structure allows each subregion to implement its own linear solution to its respective subtask. Building on this insight that nonlincarity enables computational routing, we next explore how sparse noulinearity structures computational reuse across multiple related tasks.
4.5 Shared Nonlinear Structure in a Multi- Task Paradigm
A nat ural extension of these results is to examine multi-task learning, where networks must balance lask-spe- cific specialization with cross-tast task generalization. This setting is particularly relevant from a neuroscientific perspective, as biological neural circuits routinely solve multiple related tasks with shared neural resources. We trained AL-RNNs on suite of 11 cognitive benchmark tasks adapted from Driscoll et al. (2024), pre- sented in randomly interleaved trials (see Appx. A.3 for details). The suite includes pro- and anti-response
9

---

Published in Transactions on Machine Learning Research (01/2026)
20 samples per task
samples task
500 samples per
Dokuffo
Dakyyled
Readhh
5A
Cathe
36
ChaisModd
52
A3
Coarmedd?
b
PINL Un
PVAL Units
Reacthro
100k
Catha
ChemrMozz
2 5 9011
1.3. 4.5 5.6 7.6.7. 91011
2 4
323.5 5.6 7 91011
3 7 91001
3 5.6 910E1
2.3.4 4 5 7 91001
185t
185t
Thsk
Thakk
Thrk
TAsk
TAsk
0.
02
0.4
96
0.0 100
Samileity
Figure 5: AL-RNN training multi-task paradigm. (a) Test accuracy (mean across 5 models per setting) across 11 cognitive tasks as function of P (PWL units), for different training set sizes. For limited data (20 samples per lask), sparse nonlinearity (P (P=1 to P = 8) outperforms both linear and fully nonlinear models by reusing structures tasks. For medium data (50 samples per task), more nonlinearity provides additional benefit, while sparse nonlincarity (P to P 16) performs best. With abundant data (500 samples per task, right), this advantage diminishes as fully nonlinear models can afford task-speeific solutions, though sparsely nonlinear architectures remain competitive. between bitcode distributions (mean across 5 independent training runs). Sparse nonlinearity (P= 2, reveals interpretable structure where related tasks share subregions- for instance, pro variants cluster together while anti variants form a separate group. As nonlinearity increases, this structure vanishes as the model has sufficient capacity to learn essentially independent representations for cach task, reducing overlap and eliminating the interpretable similarity structure visible lower P.
(b) Task similarity via. Jensen-Shannon divergence
variants of delayed response, reaction lime, and calegory decision lasks (requiring context-dependent mir- ror-image mappings), match and non-match variants of delayed match-to-sample, two context-integration tasks that demand selective attention to one of two sensory modalitics, and a go/nogo task probing simple threshold detection. These lasks share commnn temporal structure and input format, but require distinct computations, making them a natural testbed for understanding how nonlinearity supports flexible reuse of computations across tasks.
Fig. 5a shows test accuracy across the 11 tasks as function of the number of PWL units P and mumber of task samples used for training. While significant portion of the multi-task suite requires nonlincarity, with linear models (P=00 0) solving only two of eleven tasks, single PWL unit (P = 1) achieves signifi- cant performance gains across nearly all tasks simultancously. The pro/ant /nti task pairs make this particularly transparent. Fach variant individually solvable by a linear system (e.g., mapping stimulus to the same vs. opposite response), but single lincar network cannot implement both mappings once, because would need to 'flip' the input-output relationship based on context. Introducing just one nonlinear unit provides this context-dependent switch: it allows the network to route pro and anti computations into different linear regions and thus solve both tasks simultancously. Crucially, the same single PWL boundary is leveraged to achieve this separation for multiple different tasks simultaneously (see Fig. 24 for bitcodes and Fig. 25 for example latent. trajectories).
The advantage of sparse nonlincarity depends strongly on the training data. With very limited training data (20 samples per task), sparse nonlinearity (P to 8) substantially outperforms both linear and fully nonlinear architectures across most tasks. At intermediate data levels (50 samples per task), sparse models
10

---

Published in Transactions on Machine Learning Research (01/2026)
maintain strong performance while fully nonlinear models begin to catch up. With abundant data (500 sam- ples per task), fully nonlinear models achieve the best performance as they can fully converge to task-specific solutions without the constraint of shared subregions. Even in this high-data regime, however, sparse models (P=2 to 8) remain competitive.
This pattern aligns with the hypothesis that sparse nonlinearity is particularly advantageous in the low-data regime by forcing the network to reuse a small set of nonlinear computations across tasks. To test this "reuse' hypothesis, examined the network's distribution of bitcodes on the test set, and quantified similarity via Jensen-Shannon divergence (Fig. 5b, see Fig. 24 for examples of bitcodes). The resulting similarity matrices reveal how tasks share nonlincar resources: at P= 1, the structure is relatively stark with binary either-or distributions, while P = 4 and P = feature more nuanced relationships, where pro variants cluster logether, anti variants form a separate group, and complex tasks like match-to-samin to-sample develop distinct signatures. This shared structure helps explain the sample efficiency advantage by enforcing cross-task generalization when data is scarce. At full nonlinearity (P 64), this structure disappears a ncar-diagonal matrix as tasks are spread across independent subregions without recurring shared patterns.
4.6 Nonlinear Decoding Enables Compositional Generalization
Finally, we investigate the SCAN benchmark (Lake and Baroni, 2018), which requires to systematically recombine learned action primitives according to syntactic structures, tapping into executive functions such as memory manipulation and control. In this setting, a sequence of linguistic commands (e.g., run opposite. left after walk right) must be mapped to a corresponding sequence of low-level actions (c.g., turn right, walk, turn left, turn left, rwn; Fig. 8). The SCAN grammar is composed of conjunctions ( (and) and temporal dependencies (after), requiring the model to either execute commands sequentially or invert their order during docoding (Lake and Baroni, 2018).
a
100
Avg. Action Sequence Length
40
128
Length
Sequar Accuracy
30
PIDecoderi
70
16
60
20
30
16
32
64
128
20
0000
0001
0010
0011
0100
0101
0110
0111
1070
1001
1010
1011
1101
1110
1111
(Eenoccr)
Bitcode
0.3
Relative Frequency
1.00
Word
Frequency
0.75
after
0.2
and
0.50
0.25
0.0
ento
anto
ime
sree
0.00
0080
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1110
1111
Bitcode
Ditcode
Figure 6: a: Sequence-level accuracy (in %) number of nonlinear units P in encoder (x-axis) and decoder (y-axis). b: Bitcode distribution of initial states of the decoder indicate dominant subspace for which the first two units are zero. c: Average action sequence length associated with initial states lying in different subregions. d: Relative frequency of commands containing and and after within subregions shows clear syntactical separation.
We adopt a modular encoder-decoder architecture (illustrated in Fig. 8), assigning two separate AL-RNNs to encode the commands and decode the resulting latent state into action sequences, separately varying the number of nonlinear units P in the encoder and decoder AL-RNNs (out of total of 128 units each). Purely linear models remain stuck around 20 30% sequence accuracy (Fig. 6a). Sparse nonlinearity (P=1,44
11

---

Published in Transactions on Machine Learning Research (01/2026)
while improving performance remains insufficient to solve the task reliably. Performance peaks when both encoder and decoder include moderate nonlinearity (P = 32), with near-perfect accuracy (> 99.9%), and then slightly deteriorates for fully nonlinear models (P- = 128, ~ 96%). In addition, we observe a clear asymmetry: a linear encoder can suffice, achieving up to 99% accuracy when paired with a sufficienll nonlinear decoder (P 64), while a linear decoder remains below 80% regardless of the encoder.
To better understand this behavior, we analyzed the latent structure produced by the encoder. Evven when the encoder is purely linear, the final encoded states map onto a large number of distinct subregions in the decoder's PWL space: thus, while the encoder itself remains linear, end-to-end training organizes the latent reprosentations such that the decoder's nonlincarity can be exploited effectively.
Second, the combination of a fully nonlinear encoder (P = 128) and a moderately nonlinear decoder (P = 4) can also achieve > 99% test accuracy while enabling a clear syntactic interpretation of the latent units. We found that this behavior emerges from the interaction between the expressive encoder, which maps syntactic constructs onto the PWL latent space of the decoder, effectively leveraging its subregions for decoding (Fig. 6). Specifically, a compact subspace of four subregions, with the first two PWL units inactive, captures most initial states for shorter sequences (Fig. 6b). In contrast, more complex and lengthier command sequences are consistently mapped to a separate latent subspace, where nonlinear dynamics become more pronounced (Fig. 6c). Within this second subspace, we observe a clear syntactic segregation, with commands containing and and after being mapped to mutually exclusive linear subregions (Fig. 6d). Additionally, we find that during decoding, transitions between composite commands in an action sequence are synchronized with sign flips in the PWL units (Fig. 26). Collectively, these findings suggest that while symbollc encoding in SCAN can remain largely linear, compositional generalization depends the decoder's PWL structure tiling latent space into modular, syntax-aware subregion. This decomposition reveals a pathway toward interpreting how abstract syntactic operations are organized in more complex sequence tasks.
Table 1: Recurrent motifs identified by AL-RNNs across tasks, linking dynamics to computation.
Task
Motif(s)
Computation
IMDb
Slow mode
Scutiment integration
sMNIST, SpeechCmd
Clase-dependent slow modes
Evidence accumulation & pattern-spocifi routing
Addition
Slow mode + gating
Selective integration of marked inputs
Copy (fixed delay)
Limit cycle + subregion transition Limit cycle + subregion transitions
Stable storage & timed recall
Copy (variable delay)
Flexible storage & sequence decoding
depeeeetde integration
Rule-specific slow modes
Bule-based integration
CRCNS-PFC-1 (rodent data)
Rule-specific slow modes
Rule-based routing
SCAN
Syntax-depnndent subregion partitioning
Syntax- based modular decoding
Multi-task training
Task-specific subregion allocation
Compasitional reuse nonlinear motifs
5 Discussion
This work asks whether-ane how sys-esystmatiiallyy attemuating nonlincarity can make the computational mechanisms underlying sequence learning tasks explicit and tractable. Instead of treating nonlinearity a binary design choice, we study how progressively tuning its strength reveals which aspects of temporal computation genuinely require nonlinear recurrence, and which can be supported by predominantly linear dynamics.
We pursuc this question using the AL-RNN, whose piccewise-linear structure allows nonlincarity to be preciscly controlled and directly observed. Varying the number of nonlinear units enables a step-wise interpolation between linear and nonlinear recurrence, while the induced partitioning of state space enables bitcode-basse based analyses and analytical characterization of the underlying dynamics. This combination allows us to directly identify mechanisms such as gating, switching, and regime changes through which nonlinear recurrence contributes to sequence compulation.
Across tasks, we observe a consistent organizational principle, summarized in Table I: memory itsolf is often implemented via a slow linear mode, while computational operations are layered on top through sparse nonlinear mechanisms, often realized as discrete switches/ /gates. The PWL structure of the AL-RNN makes
12

---

Published in Transactions on Machine Learning Research (01/2026)
these mechanisms transparent, directly linking switches in model dynamics to computation. Our multi-task experiments further reveal that sparsely nonlinear models enable sample-efficien transfer learning, with computational reuse across tasks directly linked to shared linear subregions. This suggests that multi-task settings provide a promising avenue for understanding how neural systems discover and share computational primitives across related problems.
Beyond reducing interpretability, we find that for some tasks, fully nonlinear models indeed perform worse than mix of linear and nonlinear units, which could link to less stable optimization. We interpret this useful inductive bias of sparsely nonlinear models, resting on converging observations: low-P models tend to yield compact, interpretable solutions, whereas fully nonlinear models fragment computation across many subregions; fully nonlinear models often end up "linearizing" units during training, which is harder to optimize than starting with linear units directly; and these effects are robust across activation functions. While tailored regularization or initialization could in principle mitigate these gaps, our results provide mechanistic insight into why sparse nonlincarity can perform better "out of the box", and suggest that the sparse introduction of nonlinearity in hhewiis linear backbones could be leveraged as design principle for more sophisticated sequence modeling architectures.
A natural concern is that our findings may be overly specific to the AL-RNN and its PWL activations. We use the AT-RNN in a deliberately reductionist. way- not to argue that PWT nonlinearities are the "right" way to solve these tasks, but to break the model down into the simplest components needed for understanding. The PWL formulation makes it easy to see when and how the network's dynamics change, which in turn makes the underlying computations more transparent. Our goal is therefore not necessarily invariance at the level of low-level mechanisms, since different nonlinearities can realize the same computation in different. ways, but invariance at the level of functional roles. From this perspective, the patterns observe largely linear memory dynamics interrupted by occasional nonlinear events that gate, switch, or change regimes are not specific to the AL-RNN, but reflect common strategies that recur across tasks and activation functions.
More broadly, this structured separation of linear and nonlinear dynamics provides a useful lens for analyzing latent. neural dynamics where simílar mechanisms underlie storage, gating, and context-dependent computation (Durstewitz et al., 2023; Wang, 2008; Mante et al., 2013; Cho et al., 2014). Our results demonstrate how structured models can expose such mechanisms cven in noisy, real-world settings, and suggest that modular, sparsely nonlinear latent representations reminiscent of cortical organization ion (Fusi et al., 2016; Goyal el al., 2021)- offer a promising pathway toward mechanistic interpretability in both artificial and biological systems. More broadly, the structured separation of linear and nonlinear dynamics provides a useful lens for analyzing latent neural dynamics where similar mechanisms underlie storage, gating, and context-dependent computation (Durstewitz et al., 2023; Wang, 2008; Mante et al., 2013; Cho et al., 2014). Our results demonstrate how structured models can expose such mechanisms even in noisy, real-world settings, and suggest that modular, sparsely nonlinear latent representations reminiscent of cortical organization (Fusi et al., 2016; Goyal et al., 2021) offer promising pathway toward mechanistic interpretability in both artificial and biological systems, suggesting potential applications to understanding how language models perform complex reasoning.
Limitations and Future Directions Our study covers a structured but limited set of sequence tasks. While the identified dynamics capture important computational strategies, many real-world tasks, particularly in language, may blend these mechanisms more intricately. While we conducted preliminary analyses of sparse nonlincarity in more complex architectures, including $4 (Gu et al., 2022) and linear Transformers (Katharopoulos et al., 2020), understanding the role of nonlinearity in these models presents distinct chal- lenges, since here, linear recurrence and linear attention mechanisms are integral to optimization and cannot be straightforwardly replaced with AL-RNN-style sparse PWL reccurence. Moreover, when nonlinearity is introduced through layer-wise transformations rather than within the recurrence itsclf, the connection to interpretable dynamical objects (fixed points, eigenspectra, flow fields) becomes less clear. Although we observed benefits of strategically placed sparse nonlinearity even in these architectures for the tasks tested (Fig. 27), a comprchensive mechanistic understanding of these effects requires future work. Extending our interpretability framework to modern architectures represents a promising path toward bridging mechanistic understanding with state-of-the-art sequence models.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Acknowledgements
This rescarch funded by the German Rescarch Foundation (DFG) within the Collaborative Research Centre 265 subprojects A06 and B08, the Federal Ministry of Research, Technology and Space (BMFTR), the Baden-Wirttembergg Ministry of Science as part of the Excellence Strategy of the German Federal and State Govermments, and the DYNAMIC Centre, funded by the LOEWE program of the Hessian Ministry of Science and Arts (Grant Number: LOEWE
Reproducibility
Code for experiments and datasets publicly available under http:://ttaaa.occaaaaaaabeeeeeee Nonlinear i
References
TensorFlow Datasets, a collection of ready-to-use datasets. https://www.eenoorr/t htw.://..too.o....ooorodaaaasss.
David H. Ackley Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltzmann machines.
Cognitive Science, 91):1177-66, January 1985. ISSN 0364-0213. doi: 10.1016 (001100/500011011335)80011222220
URL https:tt////....... /wwp//wwwww// wttp::ww......eeeeeee..ee.eeeete sciencedirect. http///ccccaaaettc////////////20
Shun-ichi Amari. Dynamics of pattern formation in lateral-inhibiition type neural fields. Biological Cybernetics, 27(2):77 87, June 1977. ISSN 1432-0770. doi: 10.1007/BF00337220 URL https://dee 3rg/10.1007/ BF00337259.
Emili Balaguer- Baaguuer-,,l Christopher C. Lapish, Jeremy K. Seamans, and Daniel Durstewitz. Attracting Dynamics Frontal Cortex Ensembles during Memory-Guided Decision-Making. PLOS Computational Biology, ((5)=e100205) May 2011. ISSN 1553-7358. doi: /0.137//jourual.pche //journals. plos. org/ wwtp/o/gg///oo/o.ooooooooooooooo waloscompioo//attcceeeaaaa 1 pcbi 1002057. Publisher: Public Library of Science.
URL https:
Y. Bengio, Simard, and P. Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE transactions on neural networks, 5(2):157-166, 1994. ISSN 1045-9227. doi:
Guy E Blelloch. Prefix sums and their applications. In Sythesis of parallel algorithms, pages Kaufmann Publishers Inc., 1990.
60. Morgan
Manuel Brenner, Florian Hess, Jonas M. Mikhaeil, Leonard F. Bereska, Zahra Monfared, Po-Chen Kuo, and Daniel Durstewitz. Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems. Tn Proceedings of the 39th International Conference on Machine Learning, pages 2292 2320. PMLR, June mlr.
2022. URL https://proceedinee htw.cceddingg......rrres/eeeeeee httpss//v6/bbreeee2222 html. ISSN: 2640-3498.
Manuel Brenner, Christoph Jürgen Hemmer, Zahra Monfared, and Daniel Durstewitz. Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction. Aduances in Neural Infor- mation Processing Systems, 37:36829 36868, December 2024a. URL httpp:///pptt https://proceeddngaa neurips. cc/
paper http://ppp00000//..../.../000..0 oat.://eeeee..treee00oooeeeeaaee Abstract-Confereeeeec html.
Manuel Brenner, Florian Hess, Georgia Koppe, and Daniel Durstewitz. Integrating Multimodal Data for Joint Gencrative Modeling of Complex Dynamics. In Proceedings of the 41st International Conference on Muchine Learning, pages 4482 4516. PMLR, July 2024b. URL https:://rroeeeee //tpp:s:///occeiing..... mlr hatr.prres/v235. brenner24a. html. ISSN: 2640-3198.
Manuel Brenner, Elias Weber, Georgia Koppe, and Daniel Durstewitz. Learning Interpretable Hicrarchical Dynamical Systems Models from Time Series Data. In International Conference on Learning Representations (ICLR), 2025. URL https: https:///pperrevee http:///pppaeeeeeeeeeeeeerrreeaa net//forummi.--yy220aaaa..
14

---

Published in Transactions on Machine Learning Research (01/2026)
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder- for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734. Association for Computational Linguistics, October 2014. doi: 10.3115/11/D14-1177.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, December 2014. URL http://arxiv. http:/iww..rg/b//////a1// 3555. arXiv:1412.3555 [cs].
S. Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4): 357 366, August 1980. ISSN 0096-3518, doi: ieee. feee..rg/doo. wttg/document/116332
URL
Laura N. Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation recurrent nelworks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):13449 13613399---31,,,--11 July 2024. ISSN 1546-1726. doi:
10.0088/4411938-04--0666688 URL https://www http:://wwwwwwwwuueeeeeeeee www.rnatuee.... com/ /ttp//artiiieeeeeeeeeeeeeeeeeeee http:///rrriceeee//---2222222220
Daniel Durstewitz. A state space approach for piecewise-linear recurrent neural networks for identifying computational dynamics from neural measurements. PLOS Compulational Biology, (3(6):e100554)
June 2017. ISSN 1553-7358, doi: 10..8T1/journa.ppci..00005
URL https: /ttps://journals plos. org/ 1371/journal wwwpwoowww.a...a....ooooo.oreeee pttpp................ 1005542.
ploscompbiol/atticcll http//arrcccccctiocecee a
Publisher: Public Library of Science.
Daniel Durstewitz, Georgia Koppe, and Max Ingo Thurm. Reconstructing computational system dynamics from neural data with recurrent neural networks. Nature Reviews Newroscience, 24((1):693-71) November 2023. ISSN 1471-0048. doi: 10.1038/s41583 100-1-53-00000000--111 URL https://www. https:://wwwnaturee nature. com/articles// 100-88-023-007-00
Publisher: Nature Publishing Group.
Lukas
Eisenmann, Zahra Monfared, Niclas Göring, and Daniel Durstewitz. and loss jumps in RNN training. Advances in Neural Information Processing Sys-
Bifurcations
hwwpooceednngg.neurrp.. wwttrris.ccppappppe ftllss/paper/222///11 tems, 36, 2024. URL wabstracct-Coffeenncc Conference html.
d13300227990011177700222228 wat.:ec1t855-AAtraaat.t.
Jeffrey Elman. Finding Structure Time. Cognitive Science, 14(2):179 211, 1990. ISSN 1551-
6709. doi: 10.1207 111007//1551167090g1112222 URL https://onliieelbbrarr.. wiley com/doi/aat abt/aabb........................ $15516709cco1402_11
Emily Fox, Erik Sudderth, Michael Jordan, and Alan Willsky. Nonparametric Bayesian Learning of Switching Linear Dynamical Systems. In Advances in Neural Information Processing Systems, vol- ume 21. Curran Associates, Inc., 2008. URL https:://// //pp://sppppp//p/e//e nips cc/paper filess/paper/2008/naae /haoo//
9504411222644335a78l86666666. wttpt//tbbbtattt....... html.
Stefano Fusi, Earl K Miller, and Mattia Rigotti. Why neurons mix: high dimensionality for higher cognition.
Current Opinion in Neurobiology, 37:66-74, April 2016. ISSN 0959-4388 doi: 10.1016/j.comb.0011 201..016.0000000000.
http://wwwwsccicceeeeeee URL https:
com
Joshua I. Gold and Michael N. Shadlen. The neural basis of decision making. Annual Review of Neuroscience, 30:535 574, 2007. ISSN 0147-006X. doi: /tt///amrrev..u...2555 29.051605.113033.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural Production Systems. In Advances in Neural Informa- tion Processing Systems, volume 34, pages 25673 25687. Curran Associates, Inc., 2021. URL https: Abstttaatt-bbttttt.........aaa.e html.
http://www...aaa//////aaa/aaaaaa //proceedings iwwcchnngn.nuuueee
Albert Gu and Tri Dao. Mamba: Lincar-Time Sequence Modeling with Selective State Spaces, December 2023. URL http:://axtt /tt////www...rr/a/////11.......0 arXiv:2312.007722 (cs).
15

---

Published in Transactions on Machine Learning Research (01/2026)
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent Memory with Optimal Polynomial Projections. In Advances in Neural Information Processing Systems, volume 33, pages 1474 1487. Curran Associates, Inc., 2020. URL https://proceeeee Abstract. html.
neurips. http:/ssscpppppe////////aeeeee hassh/
Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference Learning Represenlations (ICLR), 2022. URL htwph:tt//////to /ttpps://openrevie..
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal State Spaces are as Effective as Structured State Spaces. Advances in Newral Information Processing Systems, 35:22982
22994, December 2022. URL https://proceednngs http://proooonng....irroo.p neurips......... www..ps.cc.pppprrrr..ro.. /ttppp/rttleeepppe///2///////// 01500fffffebbtloroo444444t-----0 ottp://cceeetttttaattttttoto-tae
html.
Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid Structural State-Space Models. In International Conference on Learning Representations (ICLR), 2023. URL httpss//openreviewt wwt.teeoooo.ddde//te........
Mikael Henaff, Arthur Szlam, and Yann LeCun. Recurrent Orthogonal Networks and Long-Memort Tasks. In Proceedings of The 33rd International Conference Machine Learning, pages
PMLR, June
2016. URL https: httppp://pprcceeee http///pooeeeeeeeeeee//// mlr. www....peee//....... press/v48/henaffe html. ISSN: 1938-7228.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Nearal Computation, 98):17355 November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.0
URL
neco 1997.9977.. 9.8 1735.
J J Hopfield. Neural networks and physical systems with emergent collective compulational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(4))2554-2558, April 1982. ISSN 0027-8424. URL https: //wp://wwwww//// ncbi nlm nih. gwtt./ooomm///mm/mm http///m/aatticcee////333333
Michael I. Jordan. Serial order: A parallel distributed processing approach. Neural-network models of Amsterdam, Netherlands, 1997. ISBN
cognition: Biobehavioral foundations. North-Holland/Elle. 978-0-444--1991-4 doi: (0.1006/50166-4115()))))10111
Pages: 495.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: fast. autoregressive transformers with lincar attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML 200 pages 5156 5165. JMLR.org, July 2020.
Yoon Kim. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirioal Methods in Natural Language Processing (EMNLP), pages 1746-1751. Association for Computational Linguistics, October 2014. doi: 10.3115/41/D14-118
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.680 6980 [es},
January 2017. URL http://arxiv org/ abs/aaa////////////////////aaa/a 6980. arXiv: 1412.6980.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeee recurrent networks, June 2018. URL 1rXiv:1711.00350 [cs).
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. pradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278 2324, November 1998. ISSN 1558-2256. doi: 10.1109/5.726791. URL https://ieeexploxxs ieee. org/ /ocummmt//267991.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A Hybrid Transformer-Mambee Language Model, July 2024. URL http://arrtt http://tttwwtpraaa///a//aaaoaaas 2403. 19887. arXiv:2403. 19887 (cs).
16

---

Published in Transactions on Machine Learning Research (01/2026)
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019. URL http://arxiv .og.org//aao.. ha//ab//1550 /109// 100p0/9999.9.1. arXiv:1509.0297 [cs].
Bryan Lim, Sercan Ö. Ank, Nicolas Loeff, and Tomas Pfister. Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(:)11748 1764, October 2021. ISSN 0169-2070. doi: http://.jfoorcast222222... atticcle//11//11199000000000
URL https:
wttp://teee.c.cccccicec....o com/science/
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian Learning and Inference in Recurrent Switching Lincar Dynamical Systems. In Proceedings of the 20th International Conference on Arlificial Intelligence and Stalistics, pages 914 922. PMLR, April 2017. URL https://prroceeding mlr, www.ppressevve.e /ww//1///eea////aaa html. ISSN: 2640-3498.
Scott W. Linderman, Andrew C. Miller, Ryan P. Adams, David M. Blei, Liam Paninski, and Matthew J. Johnson, Recurrent switching linear dynamical systems, October 2016. URL http:://araii wwg.i.iowi/oao..ooooooooooooooor org. [stat).
08466. arXiv:1610.0m
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Araual Meeting of the Association for Computational Lánguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL attp:://wws acwweb. org/
Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, and David Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systems, 11-15696-15708 December 2019. ISSN 1049-5258. URL
attps://wwl ncbi nlm. mlm.nih. wtt.g/////aticle/////eee/eeee//e
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://github.
con/pytorchhvisiion, 2016.
Valcrio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome. Context-dependent computation by recurrent dynamics prefrontal cortex. Nature, 503(4474):78 84, November 2013. ISSN 1476-4687.
doi: 10.1038/ m222//222re12 URL attps://wwt nature. com/ wwttiiceeseeaauree12222222 Publisher: Nature Publishing Group.
Eric Martin and Chris Cundy. Parallelizing Linear Recurrent Neural Nets Over Sequence Length. In International Conference on Learning Representations (ICLR), February 2018. URL https://openrevien
Harsh Mehta, Ankil Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long Range Language Modeling via Gated State Spaces. In International Conference on Learning Representations (ICLR), 2023. URL met//forum/id=5MMXYYYCC https://openreviee http::///ppeeeeeeeeeeeetoouaaaaa
Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with RNNs. Advances in Neural Information Processing Systems, 35:11297- 11312, December 2022. URL https://proceeddngs wtwpooeeddingsnnerippps meeriip.cc/pappr files//ppprr/222//ess 498551881100bbaaa111889444444449 hat.aeaaaaaaaee00eeecccccaaaeeae Conferenceeceeeeeeeeee html.
Zahra Monfared and Daniel Durstewitz. Transformation of ReLU-based recurrent neural networks from discrete-time to continuous-time. In Proceedings of the 37th International Conference on Machine Learning, pages 6999 7009. PMLR, November 2020. URL https://prooeeddngss mlr wtt.sssllr.r.....ee..eee html. ISSN: 2640-3498.
John D. Murray, Alberto Bernacchia, Nicholas A. Roy, Christos Constantinidis, Ranulfo Romo, and Xiao-Jing Wang. Stable population coding for working memory coexists with heterogencous neural dynamics prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America, 114(2): 394 399, January 2017. ISSN 1091-6490. doi: 10.1073/pnaa pttpp/a......991111111 1619449114.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Antonio Orvicto, Samuel L. Smith, Albert Gu, Amushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting Recurrent Neural Networks for Long Sequences. In Proceedings of the 40th International Conference on Machine Learning, pages 26670 26698. PMLR, July 2023. URL https: //proceedings...
www.eeddiinnmmm........m.mm.mm.e www.m.r.p.... htt..ppees//v22/rrreetttt. html. ISSN: 2640-3498.
Antonio Orvieto, Soham De, Caglar Gulcehre, Razvan Pascanu, and Samuel L. Smith. Universality of Linear Recurrences Followed by Non-linear Projections: F'nite- Width Guarantees and Benefits of Complex Eigenvalues, June 2024. URL http:://arae httpppp//awxwv..o//aaa/////s//// wtg/ab///307...o// 11888.
Badri Narayana Patro and Vijay Srinivas Agneeswaran.
Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges, April 2024.
URL http://arxiv, wwt.aawwwwwww.a/////rr/rr.aaaart
wtt./oog/////o/ooeooooe /20//////...../////////////aa/ 16112. arXiv:2404.66112 [cs].
Bo et al. Peng. RWKV: Reinventing RNNSs for the Transformer Era. In Findings of the Association for Computational Lúnguistics: EMNLP 2023, pages 14048-14077 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10. 118555/v112000 fiw.iiiiin..n mundinng-eemnl.996
Jef[rey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Repre- sentation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532 1543, Association for Computational Linguistics, October 2014. doi: 10.3115/r1/D14-111
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Re. Hyena Hierarchy: Towards Larger Convolutional Language Models. In Proceedings of the 40th International Conference on Machine Learning, pages 18043-28077 PMLR, July 2023. URL https::t///ppp htttps:/prroceediggs mlr pttess/v222/ol:/// html. TSSN: 2640-3498.
Zhen Qin, Songlin Yang. and Yiran Zhong. Hicrarchically Gated Recurrent Neural Net- work for Sequence Modeling. Advances in Neural Information Processing Systems, 36:33202 33221, December 2023. URL https://proceeddnng httpneceeennnnnnnnneeeeeeeeeeeee neurips. wewrips.cc/paper.rco fiapssspppeeee/2233/3 6efess4457eccc99ttteeddeeeeeeeee
.hml.
Misha Rabinovich, Ramon Huerta, and Gilles Laurent. Transient dynamics for neural processing. Science, 351(5885):48 50, 2008.
Kanaka. Rajan, Christopher D. Harvey and David W. Tank. Recurrent Network Models Sequence Generation and Memory. Newron, 90(1):128 142, April 2016. ISSN 0896-6273. doi: http://.neurooo221.222..o. URL http://wwww.cnccceereettttt
Roger Ratcliff and Gail McKoon. The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks. Neural computation, 20(4):873 -922, April 2008. ISSN 0899-7667. doi: 101.////cc.200..11.
URL https: : //www//wwwwww/// ncbi nlm. mih, http:///cc/aacccee///////.//////
Chris C. Rodgers and Michael R. DeWeese. Neural correlates of task switching in prefrontal cortex and primary auditory cortex in novel stimulus selection task for rodents. Neuron, (2()):1157-1170, June 2014.
ISSN 1097-4199. doi:
T Konstantin Rusch, Siddhartha Mishra, N Benjamin Erichson, and Michael W Mahoney. Long expressive memory for sequence modeling. In International Conference on Learning Representations, 2022.
Dominík Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durstewitz. Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies. Tn International Confer- ence Learning Representations (ICLR), 2021. URL btttps://pperreeeee net/forum?id= -otorummidd__XX/xxIIOOt XYzxxPIQu6s
H Sebastian Seung. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences, (23):13339-1333) 1996.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Jerome Sieber, Carmen A. Alonso, Alexandre Didier, Melanie N. Zeilinger, and Antonio Orvieto. Un- derstanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neu- ral Networks. Advances in Neural Information Processing Systems, 37:134534 134566, December 2024. doi: 10. 000552202/079017--2220 URL https::/procceddhiieee mewrrinnsgnn.nnnn........... cc/paper saaeir_feee__pppe fillss/paperr2224/aaas f27 h0tp://5500000/.0000aaaaaa000a00
html.
H. T. Siegelmann and E. D. Sontag. On the Computational Power of Neural Nets. Journal of Computer and System Sciences, 50(1):132-15) 150, February 1995. ISSN 0022-0000. doi: 10.1006 10006/jcss.1995.0111. 1et.s9995.... 1013. URL
https:/ betps://www. www.sciencedrrecttttt.mmmmm../.. http:///......cac.c.c/aaaaaa///t
Jimmy T. H. Smith, Andrew Warrington, and Scott Linderman. Simplified State Space Layers for Sequence
Modeling. In International Conference Learning Representations (ICLR), 2023. URL https://
openreview fent/forrmiddd
M. Storace and O. De Feo. Piecewise-linear approximation of nonlinear dynamical systems. IEEE Transactions Circuits and Systems I: Regular Papers, 51(4):830-84) 842, April 2004. ISSN 1558-0806. doi: 10. 10.1109/TCSL. 2004.823664.
Steven H. Strogatz. Nonlinear Dynamáes and Choos. CRC Press, May 2018. ISBN 978-0-429-96111-- doi:
10.1201/9750429492553, URL https:// //www www.oooraaacis.ccmmoooooooo www.book/bbboooecoooooooooooo..a htt.a//c..ooooooo/////0/00001111 9780029991121...
David Sussillo and Omri Barak. Opening the Black Box: Low-Dimensional Dynamics in High- Dingh-Dimesiooall Recurrent Neural Networks. Neural Computation, 25(3):626-64)) March 2013. ISSN 0899-7667. doi:
10.1162/NRCO a 00409. URL https://doi. httlddii.ogg//////// 10. www111622NCCC0. _000000___0009....___
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena : A Benchmark for Efficient Transformers. In International Conference on Learning Representations (ICLR), 2021. URL http:p::///prreieew...r net// forum? Drrum?id=VVeeW-gC222
Ichiro Tsuda. Chaotic itinerancy and its roles in cognitive neurodynamics. Current Opinion in Newrobiology, 31:67-71, April 2015. ISSN 1873-6882. doi: 10. 10.1016/j.comb. htt.://.oomb.21444. 2014.08.01..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss:/paapps //pppers nips ct/ccc/pppaae...ccceececcccc/e files/ patps///pppe//1/////// haaa://ff02222//761110000000a0a0 htt.raaab.......aaaaaaaaaetet... html.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre Memory Units: Continuous- Repre- sentation in Recurrent Neural Networks. In Advances in Neurol Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https: / //ppp//ppppppaeee www..nippcc........ cw/pppprppppppeeeeeeeeeeeee..eee wwtpppccepppeeeeeeeeeeeeeeeeeeee www.fflllppaaeeeee tatles/paper22111 2219//hahhh///////// Abstract html.
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaune Lajoie. On æyapunov exponents for RNNs: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathemalics and Slatislics, 8, 2022. ISSN 2297-4687. URL http:////wwwoooieeriiiiia org/articlss. 30.3389/FFB. 20..2002.2222 818799.
Xiao-Jing Wang. Decision making in recurrent neuronal circuits. Newron, 60(2):215 234, October 2008. ISSN
1097-4199. doi: 10. n0.1016//..cuuro.2008...
Pete Warden. Speech Commands: A Dataset for Limited-Vocabulrr Vocabulary Speech Recognition, April 2018. URL https://arxiva wwtpivv.org/bbb////ee0/ 03200v1..
Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. A C-LSTM Neural Network for Text Classification, November 2015. URL httt:://axxv..../// wttpi/.orr//aa//11111. 08630. XXiv:1511.00630 [es].
19

---

Published in Transactions on Machine Learning Research (01/2026)
A Appendix
This appendix provides further methodological details and extended results. Sect. A.1 describes training procedures, regularization, encoders, and spike data modeling. Sect. A.2 covers analytical methods for bitcode analysis, stability, Lyapunov exponents, and flow fields. Sect. A.3 details all datasets. Sect. A.4 presents extended results across all tasks, including analysis of a stimulus selection task on rodent spike data (A.4.5).
A.1 Methodological Details
A.1.1 Training Details
Optimization and Hyperparameters The primary locus of this study was investigate how nonlinearity affects performance. To ensure that our observations were not confounded by latent dimensionality, we used relatively high-dimensional AL-RNN models, scanning over several values of the latent size M in the range of 10 to 100 (up to 128 for SCAN given the higher complexity of the (ask). This allowed us to operate in a regime where performance was not limited by the latent state capacity, and where range of values for P could be tested. We also conducted targeted sweeps over the caaniflld-attaccte regularization strength r E {0, 0.01, 0.1, 0.5, 1.0, 5.10} (see Sect. A.1.2). In practice, т = 0.1 consistently yielded good results, and we therefore fixed this value across tasks to reflect that our results are not dependent on overly specific hyperparameter tuning. Importantly, the qualitative performance patterns across different levels of nonlinearity were robust to variations in both M and т. The main hyperparameter of interest remained the number of nonlinear units P. which was systematically varied across experiments to assess its impact task performance and dynamical properties. Hyperparameter scans for P are reported directly in the results figures of the main text.
For optimization, we employed standard settings consistent with the original AL-RNN implementation (Brenner et al., 2024a), including initialization strategies from the official repository. We initialized the self-connection weights in A with small random values to prevent premature convergence to unstable dynamics, while the weights W, the bias term h, the input matrix C and readout matrix D (mapping to logit scores for classification) were sampled from Gaussian distribution with zero mean and standard deviation of 0.01. For optimization, we employed Adam (Kingma and Ba, 2017) with learning rate of 0.001, which was further reduced during training by a cosine annealing learning rale scheduler. We used these settings for all datasets and did not perform fine-grained tuning.
During training, the model parameters were trained using standard backpropagation through time (BPTT). An independent validation set, comprising 10% of the training data, was used for model selection. For datasets with pre-existing train/test splits (e.g., sMNIST, Speech Commands, IMDb, SCAN), this was taken from the training set. For datasets without predefined splits, a validation set was artificially created. Other hyperparameters, such as batch size and learning rate scheduler, were set to conventional values that ensured stable convergence across all tasks.
Hardware Usage All models presented in this study are relatively lightweight, using single-layer AL-RNNs with a maximum 128 units. Due to this efficiency, all experiments could be conducted on a single CPU, with total training times for each model not exceeding 12 hours. Hence, the memory footprint remained well within the capacity of standard computing setups (less than 8GB RAM). The modest computational requirements ensure that approach is easily replicable on conventional hardware.
A.1.2 Manifold Attractor Regularization
To promote the reconstruction of both fast and slow time scales and associated memory - in the latent space, we regularize a subset Mreg & M of the latent states, as suggested in Schmidt et al. (2021), according to:
Mi,y
Meog M
Miog
Creg =T
Σ i=1
(Ax 11)
W )
++
Σ
h
(3)
1=1
1=1
20

---

Published in Transactions on Machine Learning Research (01/2026)
sMNIST
Speech Commands
100
98
Test eccuracy (%%)
96
94
92
85-
80
0.1 Regularization strongth
0.5
1.0
5.0
10.0
0.01
0.05 0.1
0.3
3.0
5.0
Regularization strangth
Figure 7: Manifold Attractor Regularization (MAR, Schmidt et al. (2021)) leads to a strong performance boost on sMNIST and Speech Commands Task, while not being overly sensitive the exact value of the regluarization strength T.
Here, À represents the matrix with the effective diagonal term for each unit: for linear units, the diagonal elements are derived directly from the W matrix, as A is zero in this case. For nonlinear units, the diagonal contribution is computed as the sum of their respective entries in A and W. This ensures that the regularization correctly constrains self-connections to be near 1, irrespective of the unit's lype. The regularization loss added to the total loss with a scaling factor T for each setting.
By constraining the self-connections of certain units to be near and suppressing their cross-conncctions, these sub-units function near-perfect integrators, capable of encoding arbitrarily slow time scales. Dynamically, this regularization encourages the formation of continuous manifold of marginally stable fixed points, supporting long-term integration without requiring architectural modifications such as gating, It enables efficient memory retention even in simple RNN architectures, and has been shown to outperform LSTMs and other RNN architectures on long-range tasks (Schmidt et al., 2021). As recommended by Schmidt et al. (2021), we applied the regularization to half of the latent units M, hence primarily targeting the linear subspace.
A.1.3 Encoders for Text, Audio, and Images
To incorporate domain-specific inductive biases, we used custom encoders for each modality, optimized for capturing the relevant temporal and structural patterns inherent in the data.
IMDB For the IMDB sentiment classification task, used pre-trained 100D GloVe embeddings (Pennington et al., 2014) to initialize word vectors, followed by multi-layer perceptron (MLP) nonlinear encoder. This additional nonlinear mapping allows the model to capture high-level semantic abstractions from raw word embeddings that aid with sentimen iment classification (Kim, 2014; 1; Zhou el al., 2015) and enable the AL-RNN to focus on evidence integration, leading to a dynamically simple linear process.
For the sMNIST task, we applicd a scries of stacked local 1D convolutional layers to the pixel sequences. These 1D convolutions respect the sequential nature the input while extracting local stroke-based features that are beneficial for digit recognition.
sMNIST
Speech Commands For the Speech Commands dataset, we transformed the raw audio signals into Mel Frequency Cepstral Coefficients (MFCCs), widely used feature representation for speech processing (Davis and Mermelstein, 1980). MFCCs provide compact and noise-robust representation, emphasizing the perceptually relevant aspects of speech. This transformation also downsamples the sequence length based on the MPCC hyperparameters. In our setup, we select parameters that preserve moderate sequence lengths comparable to sMNIST (784 time steps). The transformed MFCC features are then passed through a series of stacked local 1D convolutional layers to extract temporal patterns. To ensure consistency across samples, the MFCC features are normalized using global mean and standard deviation, computed only over the training data.
21

---

Published in Transactions on Machine Learning Research (01/2026)
A.1.4 Training on Spike Data
To train on the neural spike data in Sect. A.4.5 and Appx. A.3, we embedded the AL-RNN within the Multimodal Teacher Forcing (MTF) framework (Brenner et al., 2024b), which enables training directly on non- Gan-Gausiaan observations through likelihood-bed based optimization. Specifically, we employed 1D temporal convolutional encoder to map observed spike trains to the latent space, followed by Poisson decoder defined
log p(c, z,) = -- (Cts log Aui - Aas- - log((ct) with X = exp + b),
E(ers
where 2, is the latent state the AL-RNN, dec the predicted spike rates, and c₁ the observed counts. The MTF framework incorporates a latent consistency loss that penalizes the distance between inferred and prodicted latent trajectories by
Leansistency =
Σ (enc)
[|
2,
(gee)) ||?
where zt (cne) is the latent trajectory inferred from the encoder and z (gen) the trajectory generated by the AL-RNN. Given the short duration of the spike train trials (30-40 time steps) and their high noise levels, we found that training without teacher forcing but relying solely on the consistency loss yielded the best results, likely because teacher forcing caused overfitting to individual spike events.
To account for drift in trial-specific spike statistics while preserving shared task representation, we hierarchically parameterized the decoder following the approach in (Brenner et al., 2025), but applied this structure exclusively to the decoder parameters. Specifically, each trial was associated with a five-dimensional feature vector 160) (with its dimension determined by PCA, Fig. 23) that generated trial-specific decoder weights and biases via learned linear projections. The resulting hierarchical Poisson decoder was defined as
logr(c; 10, = E6 ces log A
A -log(en!)) with X (i) softplus(B!) + b()),
where B6) and 660 are decoder parameters generated as linear projections from ((),
This design choice ensures that the AL-RNN learns a consistent dynamic structure across trials. Hierarchizing the RNN itself could allow to absorb task variation into the dynamics. By restricting flexibility to the output mapping, the AL-RNN remains responsible for solving the core disambiguation problem imposed by the context-dependdntt lask, while the decoder accounts for slow drift or baseline shifts firing statistics.
Across all trials, we observe total of 783,840 time points. For the results presented, we use an AL-RNN with M 10 latent dimensions, yiclding 220 trainable parameters for the recurrent model. The hicrarchical decoder includes Nicat 5 learnable features per trial, which, combined with the shared linear projections, results in total of 5,375 trainable parameters. Given the large number of observed data points relative to the model's capacity, this setup remains well within a regime where overfitting is unlikely.
A.2 Analysis of Trained Models
Bitcode Analysis To quantify the distribution of latent states across the lincar subregions, we extract a subset of the PWL units from the full latent state vector. Let ž E R. denote the last P compouents of the full latent state at time step I, corresponding to the PWL units. We represent their activation patterns as bitcodes, where each component is assigned a binary value based on its sign (I if positive, 0 otherwise). Formally, the bitcode for &N is defined as:
P 12 > 0] P
b.
(4)
where I[|] is the indicator function. The empirical distribution of these bitcodes across a set of latent states is then given by:
(b
p(b) = SveRn(()
(5)
where n(b) is the count of bitcode b and B is the set of all observed bitcodes.
22

---

Published in Transactions on Machine Learning Research (01/2026)
Stability Analysis In each linear subregion of the AL-RNN, its fixed points and their stability can be analytically computed. For a given bitcode, we first compute the masked recurrent weight matrix:
W masked
W @ (bitcoode),
(6)
where M (bitcode) applies masking to the connectivity matrix W, effectively zeroing out the outgoing connections of inactive nonlinear units.
Fixed points are then computed by solving the linear system:
(A W masked 1)= --
using a standard linear solver (implemented with numpy. www.nllggg................l.... ..ove... If the matrix singular, no fixed point is considered valid. To assess the stability of each fixed point, we compute the eigenvalues of the Jacobian matrix:
J = A+ W masked
(8)
The fixed point is classified as stable all eigenvalues satisfy [Ae] < 1, which were computed via the eigenvalue decomposition using numpy. linalg. eigvals.
Maximum Lyapunov Exponent The maximum Lyapunov exponent, Amax, characterizes the average rate of divergence of infinitesimally close trajectories in DS. It is formally defined as:
T-2
Amax
lim T-x00
1 T
log
II J r=
where Jr-r represents the Jacobian of the system at time step T - |II denotes the spectral norm, and the product of Jacobians accumulates the local expansion or contraction at each step. For the AL.-NN, these Jacobians are analytically tractable (Eq. 8) within each subregion. To approximate this exponent numerically, we evolved the trained model forward for 5, 000 time steps from a randomly sampled initial condition, discarding initial transients of 500 time steps. Since the product of Jacobians grows exponentially in chaotic systems (Mikhaeil et al., 2022), we applied the algorithm outlined in (Vogt et al., 2022), which maintains nunerical stability by re-orthogonalizing the Jacobian products at regular intervals via QR decomposition.
Variance Analysis To quantify the spatial distribution of class manifolds in the AL-RNN latent space for the sMNIST lask, we computed four metrics based on the final latent states of each digit class (Fig. 12). The final latent states were first normalized dimension- wise to account for scaling differences and then projected onto their PCs. The number of components was dynamically selected to capture least 80% of the total variance. We first evaluated the Coefficient of Variation (CV), which is defined as the ratio of the standard deviation to the mean of the class-specific variances w: CV = a(w) dispersion, indicating how spread out the class-specific variance is in relation to its average. We further computed the Gini coefficient, which captures inequality in the distribution of variance across class manifolds,
=(t)- The CV provides a measure of relative
given by G N. EN 1n:-1: The Gini coefficient specifically quantifies how unevenly the variance is distributed among the different classes. Third, we calculated the Max-Min Ratio, defined as the ratio of the max(v)
-N-
maximum to the minimum class variance, defined Rmax / min(x) This ratio highlights the degree of disparity between the most and least represented class manifolds, providing direct measure of distribution extremes. Finally, mcasured the Shannon Entropy of the variance distribution: H(v) Σ.", P: log where Ps represents the normalized variance for each class. Lower entropy values imply that variance is concentrated within limited number of classes, while higher entropy suggests a more even spread across all class manifolds.
Flow Field Approximation To approximate the continuous latent dynamics of the AL-RNN, we computed the flow field over a two-dimensional grid of points in the latent space. At cach grid point, we applied the AL-RNN's step (Eq. 1) to estimate the local velocity vectors. This provides an approximation of the underlying continuous flow. For Fig. 20, used grid density of 30 x 30 points.
23

---

Published in Transactions on Machine Learning Research (01/2026)
A.3 Datasets
IMDb The IMDb dataset (Maas et al., 2011) consists of 50,000 movie reviews, evenly split between training and test sets, with sentiment labels (positive or negative) for binary classification. It is available on Kaggle at httpss//wwwr kaggle. cow com/ddaasets////// aakkshi25mppthhi -athhh//ddddddattt--ooo mere-500-mviii-eiieee review was tokenized and converted to word vectors using pre-trained GloVe embeddings (Pennington et al., 2014). The sequences are truncated or padded to a fixed length of 128 tokens during training and testing- We did not apply any further text preprocessing beyond tokenization, as the GloVe representations handle most standard normalization. Training and evaluation follow the standard 25,000/25,, 2,,000005,,00 train/Les split..
Each
The sequential MNIST (sMNIST) dataset transforms the standard MNIST (Lecun et al., 1998) digit images into sequences by flattening each 28 x 28 image into a 1D sequence of length h 784. We used the dataset as provided in torchvisiion datasets (maintainers and contributors, 2016). The pixel intensity values are normalized to the range [0, 1] before being passed to the AL-RNN. The dataset is split into 60, 000 training sequences and 10, 000 lest. sequences, following the standard MNIST partitioning.
SMNIST
Speech Commands The Speech Commands dataset (Warden, 2018) consists of 1-second audio recordings with a sampling rate of 16kHz of spoken words from a fixed vocabulary of commands: ["down", "go", "left", "no", "off", "on", "right", "stop", "up", "yes"]. We used the version of the dataset pro- vidod by TensorFlow Datasets (TFD), which includes standardized preprocessing, splitting, and metadata.
Copy Task The Copy Task is a synthetic memory benchmark where the model is required to store and reproduce a random sequence of discrete symbols after a delay period. Each input sequence consists of a random sequence of symbols represented in one-hot encoding, followed by blank segment of the same length, and finally a cue signal indicating when to begin recall. The target output is an exact reproduction of the original symbol sequence during the recall phase. Sequence lengths are fixed during training. The training set comprised 1000 sequences, while the test set contained 200 sequences. Given that the task is defined over 4 symbols and 8 time steps, the total number of unique sequences is 4$ = 65, 536. This represents a substantial combinatorial space, ensuring that the relatively small training set could not fully cover the distribution of possible sequences, thereby preventing the model from simply memorizing the training examples. Reported test accuracies reflect the percentage of correctly predicted symbols in the test sequences after the delay period.
Addition Problem In the Addition Problem, popularized in (Hochreiter and Schmidhuber, 1997), the goal is to sum the values at the marked positions and output the result at the end of the sequence. This task tests the model's ability to integrate sparse, context-dependent information across long temporal spans. The two randomly chosen indices indicating the values to be summed are presented in the first half of the sequence. The AL-RNN is trained to produce the sum of the two digits at the final time step. The training set comprised 2000 examples, while the test set contained 200 examples. Reported test errors quantify the mean-squared error between the predicted and the correct sum.
Contextual Multistability The Contextual Multistability task a custom-designed integration task inspired by neuroscience paradigms of context-dependent decision-making (Mante et al., 2013). Each trial begins with one-hot context cue, indicating which integration policy the model should adopt. A sequence of noisy evidence is presented across several time steps, and the model is required to integrate this evidence according to the initial context. At the final time step, recall cue triggers the model to produce a decision based on the integrated evidence. If the initial context is inverted, the decision boundary is reversed. The AL-RNNN is trained to predict the correct label based on its final latent state. We trained on 1000 example sequences, and tested on 200 sequences.
Prefrontal Cortex Task Recordings The CRCNS PFC-1 dataset (Rodgers and DeWeese, 2014), includes singlo-unit recordings from medial prefrontal cortex and auditory cortex of rats performing a context-dependent stimulus selection task. We focused on the longest available session (Day 4) from the first rat (CR12B), comprising approximately 900 trials. For this rat, only auditory cortex (AC) neurons were recorded. Fach
24

---

Published in Transactions on Machine Learning Research (01/2026)
trial involves auditory stimulus comprising pitch component (high low warble, presented bilaterally) and spatial cue (broadband noise from left or right). The correct response- "go" or deeeeds on the current task rule: either localization (respond based on side) or pitch discrimination (respond based on pitch). The task is structured in repeating blocks. The task was structured into repeating blocks of four trial types that defined the lask stage.
1. Block 1: spatial rule with spatial-only stimuli (left vs. right broadband noise),
Block spatial rule with compound stimuli (both spatial and pitch cues)
3. Block 3: pitch rule with pitch-only stimuli (low vs. high warble, bilaterally presented)
Block pitch rule with compound stimuli
We aligned each trial from one second before stimulus onset to the decision point (center-port withdrawal for go trials). No-go trials lacked an explicit decision timestamp, so we truncated them to match the duration distribution of go trials, avoiding length-based confounds between trial types. Spike times were binned at 50ms resolution, resulting in spike count vectors for cach neuron over the trial window.
External inputs were encoded one-hot vectors: two dimensions for the spatial cue (left (eft/right) /rihhtt two for pitch (low/high), and four for the current block (task stage). To mimic the contextual cue structure used in memory tasks, task stage information was only provided during the first 100 ms of the trial, reflecting the fact that rats had learned the current rule through block structure and early trial cues. Only trials with correct. responses were retained to ensure that lask-trained and spike responses were consistent with each other.
SCAN Task The SCAN dataset (Take and Baroni, 2018) is a synthetic benchmark designed to evaluate systematic compositional generalization
eqquence-to-scquene learning. In SCAN, natural language commands are translated into sequences of low-level actions in a navigation environment. For example, the command "jump twice" is mapped to the action sequence "JUMP JUMP", while more complex command like 'walk around right" is interpreted as 'RTURN WALK RTURN WALK RTURN WALK RTURN WALK'. The mapping is deterministic and unambiguous, with each command generated by a phrase-structure grammar that defines valid language-action mappings. The commands are constructed from a small set of primitives:
Primitive Actions: walk, run, jump, look.
Directional Modifiers: left, right, opposite.
Frequency Modifiers: twice, thrice.
Compositional Operators: and, after.
We specifically use the Simple Split of the SCAN datasct, where 80% of all command-action pairs are randomly assigned to the training set, while the remaining 20% are reserved for evaluation. The primary challenge is to generalize compositional patterns from seen instructions to novel ones. For instance, the model learns 'walk' and 'jump', along with the modifier "twice", it should generalize to "jump twice" even if that specific combination was not observed during training. We follow the canonical implementation of SCAN for preprocessing and evaluation, where commands tokenized into one-hot vectors. During training, input sequences encoded with the encoder AL-RNN and predicted using the decoder AL-RNN, which runs freely from the initial state provided by the encoder AL-RNN. Reported test accuracies quantify the amount of correctly predicted test sequences.
Multi-Task Training We jointly trained AL-RNNs on 11 cognitive benchmark tasks adapted from Driscoll et al. (2024) presented in randomly interleaved trials. The task suite includes pro- and anti-respouse variants of delayed response (remember a stimulus direction across a delay, then respond the same or opposite direction), reaction time (immediately respond in the same or opposite direction as presented stimulus), and category decision (classify stimulus angle above below threshold, then respond with the
25

---

Published in Transactions on Machine Learning Research (01/2026)
Input tokens Xt run opposite left after walk right
Output tokens Vt
turn right, walk, turn Left, turn left, run
Linear token embedding
1 Predicted Yt DZt
St =xt
token logits
Encoder AL-RNN
Decoder AL-RNN
dec
Feenc (zt-i.s))
Zt dec = Fodec (z(-1)
enc dec
ZT
Zo
Figure 8: Outline of the the encoder-decoder architecture used for solving the SCAN task.
corresponding or reversed category mapping). We also included match and non-match versions of delayed match-to-sample, where the network must compare stimuli separated by delays and respond based on whether they match within 45 tolerance. Two context integration tasks require selective attention to one of two simultaneously presented sensory modalities while integrating noisy evidence across sequential presentations. Finally, a go/nogo task tests simple threshold detection, requiring response when stimulus amplitude exceeds 0.5 and fixation maintenance otherwise. Together, these tasks probe working memory, response inhibition, selective attention, and context-dependent computation while sharing common temporal structure and input encodings that encourage discovery reusable computational strategies.
A.4 Further Results
Table Comparison of different models and activations across tasks.
Model Activation
Addition MSE
Copy Acc.
sMNIST Acc.
SpeechCmd Acc.
AL-RNN (GELU)
2+1-10 10-4
96 3%
98.2 0.2%
99.2 0.2%
AL-RNN (ReLU)
3 1.5 10-4
92 3%
98.2 + 0.2%
99.1 + 0.2%
AL-RNN (tanh)
3
90 3%
98.0 + 0.2%
99.0 + 0.2%
AL-RNN (hardtanh)
4 5-
88 + 6%
97.9 = 0.3%
99.0 + 0.1%
LSTM
8.8 + 10-10-
54 -4=10 12%
98.5 + 0.2%
99.1 + 0.1%
GRU
6.3 + 10-4
69 + 7%
98.0 + 0.2%
99.0 + 0.1%
Table 3: Performance across tasks for different activation types and sparsity levels.
Model
Copy Acc.
Addition MSE
Contextual Integration Acc.
Linear (P 0/50)
58 + 3%
0.15 + 0.01
~ 50%
Sparse (GeLU, 33//50)
+ 3%
2+1 10-1
95%
Sparse (ReLU, 33/50)
92 + 3%
3 10
95%
Sparse (lanh, P = 3/50)
90 3%
913.10 10-4
95%
Sparse (hardtanh, p =3/50)
88 + 3%
415.00 10
~ 95%
Full (GeLU, P -50/50)
38 + 6%
2
2 95%
Full (RcLU, 50/50)
39 + 5%
10-4
95%
Full (lanh, p 50/50)
58 + 6%
3.6 10-3
95%
Full (hardtanh, P= 50/50)
46 + 4%
6 5 10
~ 95%
A.4.1 Impossibility of Solving the Addition Problem with Linear RNNs
Proposition 1. An AL-RNN without ReL.U nonlinear ily cannot solve the addition problem.
26

---

Published in Transactions on Machine Learning Research (01/2026)
Proof. Consider linear RNN where d(2) 2, yielding the dynamics
2 = A + Cs + h,
(10)
where A= A W and 5t = (51,1, 82,1) T. The output at time T is given by
yT B2T B
AT 20 +
Σ AT (Cs: +11
(11)
This can be rewritten as linear functional of all inputs:
T
yT (1 + + +
(12)
where a BAT 3, = BAT and y includes contributions from zo and h. Crucially, the coeffi- cients and 8r are fixed functions of time, independent of the input data.
For the addition problem, the target output must satisfy yr = 81,4 + 81,t2, where 4 and le are the (random) time points at which 92,4, = $2,2 1. This requires the effective coefficient of 81, in the output to
a off -(1 if s2, =1
[1
=1 -10 if 82,1 =
(13)
However, in the linear system, 0₂ is fixed and cannot depend on 82,-- Since t and t2 are randomly positioned across trials, no fixed sequence {ar]T= can satisfy at = 04, 1 and a = 0 for t & {ti,t2) for all trials simultaneously. Therefore, a linear PLRNN cannot implement the conditional integration required by the addition problem, it lacks the ability to gate the contribution of $1, based on the value of $2
A.4.2 Classification Tasks
sMNIST In the sMNIST task, 28 x 28 digit images are flattened into sequential time series of length 784, with one pixel input per time step. Classification is based solely on the final latent state. Bitcode analysis provides a clear mechanism by which noulinearity increases performance: each class aligns with its own set of closely neighboring bitcodes. For instance, for P = 50, the average variation in the bitcode within each class was just 0.06 0.02 (see Fig. 10, 0.5 indicates chance level), meaning that on average 48 out of 50 PWL units had the same sign at the final state. Between classes, PWL units different bitcodes. We observed that switching between these codes during sequential processing is aligned with the onset of visually informative stroke patterns (see Fig. 11 for example illustrations). This allows the AL-RNN to implement class-specific linear integration dynamics: while all digit classes are processed through slow modes with maximum eigenvalues close to 1 (Fig. 9c), subtle class-dependent differences in these cigenvalues, arising from the distinct linear subregions occupied by each class, implement differentiated integration rates. Since the input projection itself is linear (via the C matrix), is the autonomous dynamics within each subregion that reshape the latent manifold: local flows with slightly divergent eigenvalues drive trajectories for different. digit classes apart in latent space, resulting in well-separated clusters that facilitate discrimination without requiring the input integration mechanism itself to be nonlinear (Figs. 2f & 12).
27

---

Published in Transactions on Machine Learning Research (01/2026)
P-5
12
P=1
08
b
Exlkved Welonee
0.20
P-50
P-100
1.2
0.15
0.10
Principal Componment
LA0T
Figure 9: a: Mean latent trajectories of the first PC for each digit class the sMNIST task. The trajectories reveal a slow accumulation of evidence along the primary axis, similar to sentiment classification (Fig. 2c). Interestingly, distinct branching patterns visible: the digit '6' diverges first, reflecting its characteristic structure in the initial image segments. Digits '8' and '9" share overlapping paths until around the midway point, aligning with their visual similarity in the upper halves of the images. b: Explained variance ratio of the first 15 PCs indicates a more complex structure compared to the 2D sentiment attractor in Fig. 2. (c) Maximum eigenvalue analysis across digit classes for models with varying nonlinearity. Linear and minimally nonlinear models (P =P=11 1,5) maintain maximum eigenvalues consistently near 1.0 across all digits, indicating that uniform slow mode shared across all classes. Models with moderate nonlinearity (P= preserve eigenvalues close unity but show class-speeifie variations, demonst raatnnnstratingggg that nonlinearity enables subtle, digit-specific modulation of the slow manifold while maintaining the overall slow integration character. Fully nonlinear models (P = 100) exhibit systematically larger cigenvalues across all classes, providing a mechanistic explanation for the overdispersion and reduced performance observed at high P.
SMNIST
Speech Commands
6bom
yes up stop right
Speech Command
0.8
Actieton poloot
Commerds:
0.8
0
right:
on
0
stop
go
0.0
0.1
0.2
0.3
0.4
PWLL Indixx
Avg. Deviation
Pinal State RaLLi unit indix
Activationn wan probubiity
Speeech Commend
D0 left:
0.8
St00
Cammandi
1ight
0.6
Mumbers
0n
00f
no
right
left
stop
down
0.0
0.2
0.3
0.4
0.5
0.0
0.0
0.1
0.2
0.3
0.5
Average
PWL Indexx
Average
unit index
Figure 10: Analysis of class-specific bitcode activation for the sMNIST task (left) and Speech Commands (right). Left subpanel: Average bitcode activation probability per class for the final states žσ for two representative models of = 50 (top) and 100 (bottom). High and low probabilities indicate that bitcode activations are consistent within class. Right subpanel: Average within-class deviation from these predominant activation patterns (rows in the left figure). Notably, comparing P = 50 (bottom) to the fully nonlinear model P 100 (top), the within-class variability of bitcode activations is reduced. This reduction in variation implies more stable and distinct bitcode representations, enhancing the robustness and reliability of the model's classification outcomes.
28

---

Published in Transactions on Machine Learning Research (01/2026)
Digit 0
Digit 1
Digit 2
Digit 3
Digit 4
2
Digit
Digit 6
Digit
Digit 8
Digit
7
Figure 11: Analysis bitcode activations for the sMNIST task. Fach panel shows the average Hamming distance from the reference state (0, 0, 0) for model with p = bitcode activations, aligned across time for all 10 digits. The PWL units switch primarily during visually informative stroke segments, effectively gating nonlinear processing. Digits with sharp vertical onsets (e.g., "1", "4" and "6") display early bitcode activations, whereas digits with round strokes (e.g., 90", "8") activate later. Digits with similar overall shapes (e.g., "8" vs. "9") diverge in their bitcode representation toward the end of the sequence. These qualitative differences illustrate how the different linear subregions the AL-RNN support the dynamic discrimination of the different digits.
a
P-0
N-50
1-0
Denoty
P-50
1-0
P-100
Contfsest of virddon
Gesi Confiiciett
10"2
P-100
Fe200
P+50
2.35
Via MWN Rutie
E1tlapy
Figure 12: Analysis of class manifolds for the sMNIST task. (a) Kernel Density Estimates of key metries across nonlinear configurations (P 0, 50, 100). The fully linear model (P = 0) exhibits tightly packed, uniform class variance (low Gini coefficients and Max-Min ratios). Moderate nonlinearity (P = 50) achieves optimal balance with CV s 1, producing spatially distinct manifolds. The fully nonlinear model (P 100) shows high Gini cocfficients and wide Max-Min ratios, indicating uneven variance distribution where some manifolds are over- compresede while others are spread out. (b) Final latent states projected onto the first three PCs for frequently confused digits 3 (orange) and 5 (green). At P = 50, manifolds are spatially distinct and well-separated. At 100, they become elongated and entangled, with digit particularly ceer-compressed, directly correlating with observed misclassification patterns.
29

---

Published in Transactions on Machine Learning Research (01/2026)
b
P-O, test accuracy-96.88.
P-10, test accuracy-P8.6X
P-50, test accurecy-99.58
P-100, test accuracy-f1.9K
P0
7550
off
1825
570
500p
563
Up
yes
1901
Figure 13: Same figure as Fig. but for the Speech Commands task. a: Accuracy plotted against number of nonlinear units. Increasing nonlinearity leads to moderate increases, while performance deteriorates for fully nonlinear models. b: Final latent states projected onto the first 3 PCs highlight how nonlinearity partitions the latent space according to class labels.
Speech Commands To verify that the trends observed in sMNIST generalize across modalities, we evaluated the AL-RNN on the Speech Commands dataset (Warden, 2018), a ten-class audio classification task using spoken commands sampled at 16 kHz. We used the same evaluation setup as for SMNIST, with classification based on the final latent state after processing the input sequence. As in the visual domain, we observed that increasing nonlinearity improved classification accuracy up to an intermediate number of nonlinear units, after which performance declined for fully nonlinear models (Fig. 13a). Latent states from models with moderate nonlincarity again exhibited clear class-specific clustering in PC space (Fig. 13b), and bitcode alignment within classes remained high (Fig. 10) closely mirroring the gating behavior observed in sMNIST (Sect. A.4.2). These results confirm that the latent integration and partitioning mechanisms learned by the AL-RNN generalize robustly across visual and auditory modalities.
A.4.3 Additional Results Addition and Copy Task
Autonomous Dynamics in the Copy Task To understand how the flow field implements these com- putations, we further examined the autonomous dynamics of the system. These establish a global k-cycle with a procise period of 100 time steps (exactly half the delay phase duration), as confirmed by a neutral maximum Lyapunov exponent (Amar R 0, Fig. 3b, botoom, see Appx. A.2 for methodological details), for which trajectories neither diverge exponentially nor converge to fixed point. While the initial transient dynamics are constrained entirely to the lower subregion, the fully developed k-cyele spans both subregions, oscillating between them in a structured manner. This cycle is stabilized through the interplay of two different dominant eigenvalues of the local Jacobians: one with contracting virtual fixed point (Amax & 0.992) and the other around a diverging fixed point (Amar & 1.005) which drives the initial transient dynamics. Intuitively, once the trajectory has settled onto the orbit that alternates between those two subregions (as visible in Fig. 3b, bottom, where the PWL unit repeatedly crosses the switching boundary), the net effect. over full cycle is neutral expansion Xmaz 0), which stabilizes the limit cycle.
Copy Task With Variable Delay In the main text, analyzed the copy task under fixed delays between encoding and recall. We additionally implemented variable delay version of the task, in which the cue to initiate recall occurred after randomly chosen interval. In this setting, solutions discovered under fixed delays did not transfer successfully, but required different strategy for solving the task. However, consistent with the fixed delay task, we find that successful AL-RNNs still maintain latent activity within a single lincar subregion throughout the delay period, indicating that storage continues to rely on stable linear dynamics. However, the decoding mechanism is substantially more complex: during the recall phase, trajectories now transition across multiple linear subregions, rather than remaining confined one. Analysis of the resulting bitcodes shows that symbol identity is reflected in the sequence of subregion switches, with decoding accuracy from the bitcode alone reaching around 60% (compared to a 25% chance level) (Fig. 15b). These results indicate that variable timing shifts the computational burden from storage to decoding, Whereas fixed-delay tasks can be solved by maintaining activity in a single regime and reactivating stored content at a fixed time, the varlable-delay setting requires a more flexible, state-dependent decoding process.
30

---

Published in Transactions on Machine Learning Research (01/2026)
25000
10000
20000
P=1
8000
P=3
P=50
Count
15000
Count
6000
Count
10a
10000
4000
5000
2000
10a
0
101
100
000
110
111
001
010
104
20000
P=5
P=10
15000
Count
Count
10000
5000
102
00000
01,00
00100
10011
(9011
P0010
60110
6011
01200
01011
0,00
0.50
0.75
1.00
= mask, only masked
b
C
0.995
10-2
Latent Activation 236
-2
lest Loss
-3-
10-3
-4
-6-
10-4
20
40 60
100
10-3
10-2
Time Steps
Training Loss
Figure 14: a: Distribution bitcode subregions in the addition task for different levels of nonlinearity (P) for the best-performing models. A small subset of dominant subregions (blue) encode linear integration dynamics, while sparse gating implemented by a few low-froquency subregions (red). The relative ratio of subregion activity for masked time points shows clear separation between integration and gating functions, with region implementing both. While more nonlinear settings (P 10, 50) distribute integration dynamics across multiple neighboring regions, this could introduce instability compared to the compact representations with sparse nonlinearily, polentially explaining performance decreases in fully nonlinear models. Note that for 10&:00 the y-axis scaled logarithnically. b: Networks often "lincarize" certain units, pushing individual activations far from the piecewise- linear boundary such that only & small subset effectively switches. This inductive bias simplifies the optimization landscape by operating in the linear regime where possible, which may explain why fully nonlinear models less robust. Correlation between training and test losses across runs (r = 0.995). Each point represents trained model; the dashed line indicates y Near-perfect correlation with slightly higher test. losses rules out classical overfitting.
31

---

Published in Transactions on Machine Learning Research (01/2026)
a
b
100
Unit 1
Unit 2
0.8
3
Mean Activation
Unit 4
PWL Unit
0.6
Unit 5
Symbol
Unit 7
60
Unit 8
0.2
Unit 9
50
Unit 10
10
20
30
40 50
0.0
-3
Steps relative to cue onset
Figure 15: Results for the variable delay copy task. a: Symbol recall accuracy as function of the number of PWL units Performance rises steeply with sparse nonlinearity and peaks at intermediate values of P. Error bars denote standard deviation across seeds. b: Mean activation of individual PWL units aligned to cue onset (dashed red line). During the delay period (steps < 0), activity remains confined to a largely linear regime. After cue onset, recall requires structured transitions across multiple subregions, reflected in distinct. activation patterns across units and time steps.
104
1.0
0.8
Cumulative Mass
Coefficient
Linear Regions
P=1
10a
P=3
Possible Regions Fffective Regions
0.4
10a
P=5
P=10
0.2
101
0.4
P-25
P=50
0.0
10a
10
20
30
40
50
0
10
20
30
40
50
100
101 Bitcode Rank (sorted by probability)
10a
P
Figure 16: Distribution of linear subregions during decoding stage for the copy task. (a) The Gini coefficient of bitcode distributions as function of P, reflecting the imbalance in subregion usage. As P increases, the Gini coefficient rapidly rises, indicating that only limited subset of regions is dominantly occupied. (b) The effective number of linear regions occupied by the AL-RNN, plotted alongside the theoretical maximum (dashed gray line) computed as min(2", total samples). Despite the potential to explore exponentially many regions, the network only populates small fraction. (c) Cumulative mass plot for the bitcode distributions sorted by probability, visualized across different values of P.
32

---

Published in Transactions on Machine Learning Research (01/2026)
a
Copy
20
324
80
Task
Poll
415
P-25
-8.
1-90
0.
25.0
1000
2005
3000
40000
1.0
Epech
10
2>
0.75
-10
0.50
Addition Task
0.00
2:394
0.181
10-2
PASS
-8.5
100
2385
0.164
0050
Eoech
Spachh
+1.00
0.00L
0.51
1.0
10
Figure 17: Training dynamics and performance of AL-RNNs copy task and addition task. (a) Loss (left) and gradient norm (right) trajectories for models with varying numbers of P PWL units out of 50 (see y-axis labels). Copy task (top): Linear models (P 0) show smooth training but fail to reach low loss values. With sparse nonlincarity (P 1,5), some models achiove good performance with relatively smooth training, though bifurcations begin to appear. Fully nonlinear models (P = -0)) exhibit highly irregular training with frequent bifurcations and consistently elevated gradient norms. Addition task (bottom): Loss drops faster for models with more nonlinearity, likely since higher model capacity means solutions discovered more rapidly, followed by fine-tuning. Models with minimal nonlinearity (P =P= 1) often fail to find solutions, possibly because fewer nonlinear units reduce the probability of stumbling upon the correct mechanism. Intermediate configurations (P- = 10, 25) still show training instabilities where losses suddenly jump up. Fully nonlinear models converge fastest but maintain higher gradient norms throughout. (b) Test accuracy for purely lincar nelworks across latent dimensions M and regularization strengths т. Linear networks completely fail on the addition task, performing at chance level across all settings. On the copy task, accuracy ranges from 40 70% with apparent random fluctuation across hyperparameters, never approaching optimal performance. This demonstrates that the results from the main text overly sensitive to hyperparameter search.
33

---

Published in Transactions on Machine Learning Research (01/2026)
a
Max (best model)
Best madel P+3, 100% test accuracy
Best modcl P=50, 48% test accuracy
Best eccuracy 196)
100
0.
Copy Task
80
26033
0.031
10
25
50
8.0
100
Number of nonlinear units (P)
Subregtan
b
Min (best model)
Best model P-50
Worst model P-50
10
Addition Task
Test MSE
10-3.
40
30
10-5
5
10
25
50
Bicodds during Gacing
Bitcodes
Number of nonlinear units (P)
Max (best model)
MAD
128
100
128
1.00
Xwy
Seip Mice Accunery MAD
SCAN Task
Sequencs Accuracy
Accurscy
90
(Decoder)
70
P (Dacoder
32
10
26
60
18
80
70
60
8
16 (Encoder)
32
64
128
(Ersoddrr)
70
Train Accuracy
30
90
100
Figure 18: Performance and bitcode distributions on copy laski addition lask, and SCAN task. (a) Copy task performance across nonlincarity levels P. Best model accuracy (left) peaks at intermediate P P= 3, reaching 100% accuracy. Subregion usage for the best. P = 3 model (middle, 100% accuracy) concentrates in small number of regions, while the best P = 50 model (right, 48% accuracy) fragments activity across hundreds of subregions. (b) Addition task test MSF. The best individual model fully nonlinear (P= 50, left), while median performance was slightly worse with full nonlinearity. Comparison of the the best 50 model (middle) vs. worst performing models (right) shows that. the well-performing model concentrates gating in few subregions with strongly skewed distributions, while the worst model spreads activity across many subregions. (c) SCAN task accuracy. Best models (left) achieve near-perfect performance across all P values, including P = 128 (fully nonlinear). However, mean absolute deviation (MAD, middle) increases substantially for fully nonlinear models, indicating higher training variability. Train-test accuracy correlation (right) shows this variance is not due to overfitting but reflects training sensitivity. Fully nonlinear models can still find optimal solutions but do so less reliably.
34

---

Published in Transactions on Machine Learning Research (01/2026)
4.4.4
Nonlinearity Enables Internal Task Switching Through Context- Dettedeeennnt Routing
Decision
a
Cue 1
5/
s;
Class 1,
Σ Sj <
Class 2
-0.84
0.92
1.03
-0.82
1,06 0.92 0.87
Cue 2
s;
Class
Σ St <
Class 1
b
C
Uneer Unit
Linozr Lhit
Context Class
100
15
Context 1- Class 2
Accuracy (N)
Context Class
PNU Unit
10
Context 2 Class 2
-10
Nonlineartty
Time Steps
Time
gelu
hardtanh relu
PWL Unit
PVA Umit
80
10
tanh
Numbee Units (P)
10
-10
-5
Linear Unit
0
5
10
50
Steps
100
25
Time Steps
100
Context
Context 2
Figure 19: a: Task schematic: Depending on the initial context cue (blue or red), the network must classify the sign of the integrated input sequence with a reversed decision rule. b: Test accuracy as function of nonlinear units, P. c: Final latent states for both contexts with a linear decision boundary (dotted line, red and grey regions indicate classes). Right: Latent trajectories over time (black for Class 1, red for Class 2) cumulatively integrate evidence. The initial context determines the activation of the PWT, unit, selecting the lincar subspace: the PWL unit is positive in Context 1 and mostly negative in Context 2, inverting the temporal integration process to enable linear separability.
We assess the network's ability to switch between tasks using contezt-dependent integration task (Fig. 19a), design inspired by behavioral paradigms in neuroscience for studying flexible decision-making (Mante et al., 2013). Each trial begins with a one-hot context cue, followed by a sequence of scalar inputs (T'ee 100). This task requires the network to switch between two internal decision policies based on a contextual cue. Purely linear AT-RNNs cannot. solve this task, and plateau at 50% accuracy (Fig. 19b), independent of how many linear units are provided. In contrast, the task is almost perfectly solvable with just one linear and one PWL unit, achieving up to 96% accuracy.
We illustrate the central mechanism with one successfully trained model with one linear and one PWL unit. Here, the PWL unit primarily operates in two distinct subregions, based on the initial cue (Fig. 19c). In Context 1, the nonlinear unit essentially mimics linear integration, residing within a single subregion for the entire sequence, while the linear unit accumulates the input symmetrically, In Context 2, the nonlinear unit transitions to a different subregion, effectively inverting the sign of evidence accumulation within the lincar unit, achicving the required task switching, leading to lincar separability of the final states (Fig. 19c, right). This mechanism exemplifies context-dependent routing: the nonlinear unit acts as switch, directing identical inputs through distinct internal linear dynamics depending the initial context. See Fig. 20 for flow field and example trajectories.
A.4.5 Nonlinearity Enables Task Switching in a Stimulus Selection Task in Joint Task-Neural Modeling
We next evaluated a structurally similar setting, this time training the AL-RNN not only on the task but also to jointly reconstruct neural population activity from the CRCNS PFC-1 dataset (Rodgers and Weese, 2014). This dataset records single-unit activity in rodent auditory cortex during flexible stimulus-selection task, in which animals alternate between two rules: choosing the correct response based either the spatial origin or the pitch of the sound (Fig. 21a). The two stimulus dimensions are presented simultancously but independently, with either a low or high warble (pitch cue) played on both sides, and pure tone (side cue) played either on the left or right. Depending on the current rule, the animal must attend to either the pitch (low = "go", high- 'no-go") or the side (left 'go', right = "no-go") of the auditory input. This sctup creates contextual ambiguity: for example, an identical stimulus combination of a high-pitched warble and a tone the left requires different responses ("go" under the spatial rule, "no-go" under the pitch rule). Hence
35

---

Published in Transactions on Machine Learning Research (01/2026)
Cum. Sum of Evidence
5
Wwm W
15
Context 1, Class
Context Class
Context
Context 2, Class
Es, 0 Class
0
10
Context 2, Class
Unstable Fixed Point
PWL Unit
Decision Boundary
-5-
5
-10-
Context
$.500 <.0 Class
0
20
40
60
80
100
-5
Time Steps
-10
-5
5
10
Linear Unit
Figure 20: Flow field and latent trajectories in the linear and nonlinear subspaces for both contexts (Context 1: blue, Context 2: red) in the contertual indegration lask. The left panel depicts the cumulative sum of evidence for two example sequences, with context inverting the decision rule. The right panel shows the vector field, where thick lines represent paths taken when no input is provided except the initial context. cue, gradually dristing lowards the decision boundary in the respective context. Thinner lines illustrate the two example trajectories under continuous input (corresponding to the cumulative sum of evidence shown in the left panel). In these cases, evidence integration drives the trajectories along context-specific pathways-positive accumulation in Context 1 and inverted accumulation in Context 2. An unstable fixed point (green dot) near the origin demarcates the boundary between the two decision regions.
the task studies linear and nonlinear neural computations in context-dependent behavior. Preprocessing steps and selection criteria are detailed in the Appx. A.3. To ensure a consistent task representation, we restricted training and evaluation to the 723 trials in which the animal made the correct choice.
Task-training the AL-RNN We first trained an AL-RNN to solve the task based purcly on structured temporal input, independent of neural data. Each trial mimicked one of the 723 empirically observed trials, beginning with a brief context cue indicating the active rule (spatial or pitch), presented for 100 ms (two time steps), followed by a ono-sccond delay. Then, an auditory stimulus was presented (one-hot encoded for pitch and side), and the model was required to produce binary decision (go/no-go), which was read out linearly from the latent. state at the empirically observed decision time St,te (see Appendix A.3). Despite the task's contextual nature, stimuli based on their proceding rule cuc, and lincar 3D models often solved the task perfectly. Systematic evaluation confirmed that increasing capacity beyond this dimension does not yield substantial gains in task accuracy alone (Fig. 21c, left), and that errors were largely restricted to contextually ambignous stimuli (e.g., Right Low under the spatial rule; Fig. 21c, right).
find that a 2D model is already often sufficient to linearly separate identical
Task Information Improves Spike Reconstruction We next used the same AL-RNN to jointly model both the animal's behavior and the neural spike trains. That is, the network received identical inputs as in the task-only training context cue, delay, and stimulus and was required not only to produce the correct. docision at the appropriate time, but also to reproduce the observed spiking activity across neurons from the same latent states of the AL-RNN (Eq. 1). To achieve this, we trained the AL-RNN within the Multimodal Teacher Forcing (MTF) framework (Brenner et al., 2024b), which allows direct training non- Gaussian spike data via a Poisson decoder. The decoder is hicrarchically conditioned on a 5-dimensional, trial-specific feature vector (Brenner et al., 2025), enabling it to capture slow drift in spike statistics while preserving a shared task representation in the AL-RNN (see Appx. A.1.4 for training details and Fig. 23 for an analysis of the learned features).
We found that including task information markedly improved spike reconstruction (Fig. 21d), suggesting a strong alignment between latent decision dynamics and observed neural variability. Quantitatively, spike rates
36

---

Published in Transactions on Machine Learning Research (01/2020)
a
Spatial Rule
Pitch Rule Low Go
b
---
Spatiel Rure, No Go
Spatial Rure, Go
Pitch Rulc. No Go Pitch Rule,
Stimulus Noisa Burst (LehtRight)
Distractor Notsa Burst (Leht-ighht
Don't Go
MMW
WWW
Stimulus
WW
Go
WW
Warble [HighLoow
(Hig
Don't
1.00
-3
Uneer Unit
0.95
Blokk
0.90
1.0
Spatial Rule
0.85
0.5
Left
0
0.0
Rigie
Le Le-il RAoa
R--
0.8
3.2
2.0
Inpure
Time [6]
0.975
0.000
Muntring talx
0.20
Mear NEE
0.045
Spaae Cemierio
0.360
Pitch Rule
0.225
0.40
0.358
High
0.000
0.20
2.356
0.05
10
Figure 21: a: Schematic of the contextual stimulus selection task used in the PFC-1 dataset (Rodgers and DeWeese, 2014). In each trial, animals responded to auditory stimuli based either the spatial origin (left/right) or the pitch (high/low), depending on the current task rule. b: Latent trajectorios from a task-trained 2D model for two identical stimuli under different rules, illustrating context-dependent separation at decision time by the subregion boundary (black horizontal line). c: Task performance across models of increasing nonlinearity (P) and different latent dimensions (M) (left) and misclassification counts by stimulus and block (right). d: Mean firing rate MSE with and without inputs. c: Joint training performance when modeling both task and spike data: lest accuracy (left), mean spike rate MSE (middle left), and coefficient of variation (CV) MSE (middle right) and mean spike correlation (right) function of the number of nonlincar units. f: Mcan activation over time of a representative nonlincar unit that differentiates rules and modulates stimulus-driven activity conditionally.
and coefficients of variation were well captured (see Fig. 22a). To evaluate the model's temporal reconstruction, we correlated recorded and generated spike trains, yielding a mean correlation of r = 0.36 0.002. This approached the correlation between separate Poisson samples drawn from the same latent trajectory (r 0.371 + 0.057), which serves approximate upper bound given the high intrinsic variability of sparse cortical spiking.
Nonlinearity Supports Integration In Joint Modeling In the joint setting, nonlinearity began to play more significant role. Increasing the number of nonlinear units improves both task accuracy and spike reconstruction (r < -0.37, = 0.005, low firing rate MSE/higher task accuracy are better, Fig. 21e). In contrast, linear models plateau around 87% accuracy and exhibit an almost identical error pattern: they consistently respond "go" all trials with left or low cues, regardless of the active rule, thus failing resolve the contextual ambiguity (see Fig. 21c, right). In many such nonlinear models, we observed the emergence of single units that effectively gated the task rule, becoming active in rule context and inactive in the other, with stimulus integration occurring only within the active subregion (Fig. 21f). Quantitatively, we examined the linear subregion active at the time the input is first presented (after one second) and compared which subregions were active in blocks 2 and 4 (the two conditions where both spatial and pitch cues are present. but different rules are active; see Appx. A.3). In successful models (> 99% accuracy), trials from these blocks were routed into almost entirely distinct subregions (on average, 97% of trials fell into block-specific lincar subregions), while in unsuccessful models this separation was much less pronounced (only 32%). Task accuracy was strongly correlated with this separation score (ratio of trials falling into non-overlapping subregions for different rules, 0.66). This shows a clear link between successful models leveraging nonlinearity to assign different task rules to distinct linear regions, allowing them to react differently to identical stimuli depending on context. This mechanism is illustrated for a successful task-trained 2D model in Fig. 21b.
37

---

Published in Transactions on Machine Learning Research (01/2026)
b
a
Ground Truth
30
30
1.0
Predicted Mean (Hz)
Neuron
20
20
10
0
10
AL-RNN with Task Inputs
30
Neuron
20
0
0
10
20
30
10
True Mean (Hz)
0.00
0.25
0.50
0.75 Time (s)
1.00
1.25
1.50
C
5
0.994
Spatial Rule, No Go
Predicted CV
4
1.5
Spatial Rule, Go
Pitch Rule, Go
1.0
Pitch Rule, Go
3
PC
0.5
2
1
-0.5
1
2
3
4
5
-2
-1
0 PC
2
True CV
Figure 22: Predicted vs. true mean firing rate (top) and coefficient of variation (CV; bottom) across all neurons for example model trained with task inputs (M 10, P = 3). The model accurately captures both first- and second-order spike statistics. b Example raster plot comparing true and reconstructed spike trains for single trial for the same model (M = 10, P = 3). Example PCA-projected latent trajectories for the four ambiguous input conditions from Fig. 21, showing clear separation based on context.
a
b
Go
No-Go
10
r= 0.865
Varfance
Predicted per
1.5
PC3
0.0
0.0
Principal Component
Tue Mean Trial (Hz)
-1.5
PC2
0.0
PC1
Figure 23: a: Explained variance the five PCs of the learned 5D trial-specific feature vectors, used to generate trial-spocific decoder parameters. b: Predicted vs. true trial-averaged firing rates for all neurons. c: PCA projection of the feature vectors into the first three PCs, colored by Go (blue) and No-Go (red) labels. Trial-level task information partially encoded in the decoder feature space; a linear classifier trained the first three components achieves 86% accuracy.
38

---

Published in Transactions on Machine Learning Research (01/2026)
Bitcode similarity matrix
Task-specific bitcode distribution
B-1
Delayho
Gaff
b
ManntSandee
1.0
P=1
Delonge
0.8
P=2
Delighe
0.6
buath
Correlation
Cale
P-8
P=16
Gelteg
0.2
32
oer
P=64
0.0
fractine
P=1
P=2
P=4
P=8
P=16
P=32
P=64
Figure 24: Bitcodo usage from AL-RNN training on a multi-task paradigm. a: Left: Bitcode similarity matrices computed from Jensen-Shannon divergence between lask-specific subregion distributions, as in Fig. Right: Task-specific bitcode usage distributions corresponding to each P level. For p= 1, only possible bitcodes exist (representing the two sides of single PWL boundary), and tasks cleanly partition between them, e.g., DelayPro and Anta requirements (see also Fig. 25 for example latent trajectories). At. P= 5, with 2 = 32 possible bitcodes, distributions broaden but remain concentrated on small subsets, with some tasks showing distinct peaks (e.g., Match2Sample, CtxIntMod1/2) /2) while others overlap. At P = 64, with 204 possible bitcodes, distributions become extremely sparse and dilfuse across the lop 200 most-used regions, and tasks show almost no shared structure. b: Consistency of learned representations across runs with 20 samples per task via correlation of task similarity matrices (from a) across 5 independent training runs. Sparse nonlinearity (P 4 16) produces highly robust subregion assignments. Representations also remain similar across adjacent sparsity levels (e.g., P = 4 to P = 8 shows correlations ~ 0.5 0.7), indicating that in the low data limit, the shared nonlinear structure is robustly captured.
opposite subregions, consistent with their opposite but individually linear task
39

---

Published in Transactions on Machine Learning Research (01/2026)
Delay Task (Anti) Opposite Orientation
Delay Task (Pro)
10
-5
-10
N'
-15
-20
-5
-25
Time Steps
Time Steps
Contextual Integration Modality 1
Contextual Integration Modality
10
5
-5
-10
-15
-20
10
20
30
40
50
60
70
10
20
30
40
50
60
70
Time Steps
Time Steps
Figure 25: Latent trajectories of an AL-RNN with P 1 trained on the multi-task suite for the pro and anti versions of the delay task, and the two modalities in the contextual integration task. The network leverages the single PWL boundary (black horizontal line) to separate the task mechanism for the two different. task versions.
40

---

Published in Transactions on Machine Learning Research (01/2026)
.walk around left after look around thrice"
PWL Unit
PWL Unit
1.0
Piaign 0)
Turn Left
0.5
Walk
PWL Uni
Distance from Switch
Distance from Switch
Unit 3
Plsign it =0
PWL Unit PWL
PWL Unit
0.0
10
Action Steps
Distance from Switch
Distance from Switch
Figure 26: Latent Dynamics of PWL Units during composite actions in the SCAN task. a: Visualization of an example SCAN command walk oround right thrice after jump around left thrice. Top: Executed actions over time, illustrating the switching behavior between distinct subtasks (look, turn left, walk). Middle: Activation of PWL Unit 1 throughout the sequence, highlighting characteristic sign flip from nogative positive precisely before the transition point which triggers the switch to the next subtask (shaded background region). Bottom: Bitcode representation for the four PWL units, where black denotes negative activation and white represents positive activation. Notably, PWL Unit 1 consistently flips to positive at the switch, marking the boundary between subtasks. b: Statistical analysis of PWL unit activations around switch points between composite subtasks, averaged across all SCAN sequences. The x-axis represents time steps relative to the switch point (t 0). Bars indicate the average probability positive activation for each PWL unit (mean + sem). The dashed red line is the global bascline activation for each unit. All PWL units showcase characteristic shifts in their sign around switch points: Unit initiates switches with brief peak to positive activity, Unit 2 transitions from predominantly negative to predominantly positive activation (and vice versa. for Unit 4), and PWL Unit 3 synchronizes its oscillations tightly with subtask boundaries.
41

---

Published in Transactions on Machine Learning Research (01/2026)
Addition Task
Copy Task
a
Standard Uner Turatomar
Standerd Linear Tansforer
Trans- former
Sparse Norlhoar
Sperie roe Hont
0.1
2p-1
10
10-3
3.5
05
10-1
10
0.4
Zanced Lawer Tardbaral
Standrt
Liew
20
4C
03
302
40
100
23
100
b
10F
My Uiner
Fully Unear
3.0
S4
10.
10-1
10
10
Standdrr 54
82
Soor card
300
kpach
1000h
Czec
Soech
Figure 27: Comparison of three architectural variants for both Linear Transformer (Katharopoulos et al., 2020) and S4 (Gu al., 2022) models: Fully Linear models contain no nonlinear activations anywhere in the archi- tecture, Standard models include nonlincarity in every layer (RcLU in feedforward networks for Transformers, GELU in every S4D block for S4); and Sparse Nonlinear models only use nonlinearity at the final layer after all attention or state-space blocks, keeping intermediate computations linear. Note that the optimization is not directly comparable to the training of the AL-RNN. While we implemented causal versions of these architectures to maintain meaningful task structure, even with causal masking, these models fundamentally dilfer from finst-order Markovian RNNs, with their multi-layer structure provides a view the sequence that enables linear operations to solve tasks like addition (which our proof shows is impossible for purely linear Markovian modcls). (a) Lincar Transformer performance. On the addition task (left), standard Transformers with distributed nonlinearity converge slower and converge at higher error than both fully linear and sparse nonlinear variants, which perform comparably. Test performance closely mirrors training, with standard Transformers reach ~00 MSE while others achieving near 10-4, On the copy task (right), sparse nonlinear models substantially outperform both alternatives, reaching near perfect accuracy (>95%) on both train and test sets. Standard Transformers show intermediate performance, while fully linear models converge much slower and exhibit high variance. (b) S4 model performance. For the addition task (left), all variants converge to similar final performance (~10-3 MSE) with, with the nonlinear variants performing slightly better. On the copy task (right), all three variants achieve near- wear-perfect performance, though sparse nonlinear models converge slightly faster than the alternatives. Overall, the architecturo-dependor benefits of sparse nonlincarity observed in RNNs partially transfer to iirention-basee models but are less pronounced in state-space models for the two considered tasks.
42


Reviews:
[{"review_id": "sO0UTDx8xN", "reviewer": "Reviewer_Htgm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses the contributions of nonlinearity to computation.\nAL-RNNs are trained on different tasks and their dynamics and computational strategy is analyzed and in particular linear and nonlinear networks are compared.\n \n\nThe paper investigates many different tasks, some of which are quite complicated. \nThe analysis of the trained networks is fairly extensive.\n\n\nOn the other hand, the claims in the paper are about RNNs in general, even though only one specific architecture was tested with specific hyperparameters.", "claims_and_evidence_yn": "No", "claims_explanation": "1. \"Nonlinearity Reshapes Slow Modes in Temporal Integration\"\n\n\n\"Similar slow-mode or line attractor based integration mechanisms\", while the dynamics in these networks looks more like an approximate line attractor (i.e. a slow (invariant) manifold), see also \nSágodi, Á., Martín-Sánchez, G., Sokol, P. and Park, M., 2024. Back to the continuous attractor. Advances in Neural Information Processing Systems, 37, pp.66856-66906.\n\nWhat is the contribution from the nonlinear part to slow dynamics? Slow invariants can develop in nonlinear networks (Ságodi, 2024). \n\nFinally, it is unclear how tokenization influences the learned computation. \n\n\"Bitcode analysis provides a clear mechanism by which nonlinearity increases performance: each class aligns with its own set of closely neighboring bitcodes\". It is unclear how the first statement follows from the second.\n\n\"These results support the view that nonlinearity in this setting acts not as a core computational mechanism, but as a tool for improving expressivity\"\nWhat are the set of core computational mechanisms? How do you measure expressivity?\n\n\n\n2. \"Minimal Nonlinearity Stabilizes Transients For Structured Memory Recall\"\n\"This cycle is stabilized through the interplay of two distinct spectral properties: one with a contracting virtual fixed point ($\\lambda_{\\text{max}}\\approx$ 0.992) and the other around a diverging fixed point ($\\lambda_{\\text{max}}\\approx$ 1.005) which drives the initial transient dynamics.\"\nIs this the eigenspectrum of the cycle? Or of two different fixed points?\nBut how does that then \"stabilize the cycle\"?\nAnd where is this information about the spectrum coming from, i.e., how was the Lyapunov spectrum calculated?\nFinally, it is not readable from Fig. 4b that the Lyapunov exponent is approximately 0. \n\nIt remains unclear how the networks trained on the Copy Task With Variable Delay solve the task. \n\"with decoding accuracy from the bitcode alone reaching around 60\\%\" seems to indicate that the subregions are not the correct level at which decoding takes place.\n\n3. Nonlinearity Enables Gating in the Addition Task\nIt is clear that nonlinear networks can implement gating mechanisms more easily.\nHowever, it is possible to let purely linear model implement one, especially if the task length is nonvariable. \n\n\n4. \"Nonlinearity Enables Task Switching in a Stimulus Selection Task in Joint Task–Neural Modeling\"\nTo show task switching in trained models and the contribution from nonlinearity to this, it would be better to use a more simple task. \nThis task, involving real-world data, although very interesting and important to show the usefulness of AL-RNN, is too complex to clearly test the effects of the nonlinearity in the trained networks.\nFor example, this task shows that sparsity is a bad inductive bias, see Fig. 6e (against the claims of the authors). \n\n\n5. \"Nonlinear Decoding Enables Compositional Generalization\nThis is again an interesting setting to test.\"\n\n6. \"This may partially explain the efficiency of linear SSMs on long-range tasks\".\nIt is surprising that this is the authors takeaway from the paper, while they showed that purely linear networks were insufficient on some of the tasks under consideration. \n\n\n7. \"highlights a useful inductive bias of sparsely nonlinear model\"\nThere is some evidence coming from the networks trained on the different tasks that combining linear units with nonlinear ones is beneficial. \nHowever, whether it is a certain size of (non)linear is optimal or whether it is sparsity is unclear.\n\n8. \nOverall, it is unclear whether the contributions are specific to AL-RNN or whether they are something general.\n\n\nFor example: \n\"The PWL structure of the AL-RNN makes these mechanisms transparent,\ndirectly linking switches in model dynamics to computation.\" \nI am not sure where it is show that there is a direct link.\n\n\n\n\n\nClaims about used methods:\nIt is claimed that AL-RNN allows fine-grained control.\nFine-grained control of the nonlinearity would suggest that along a certain parametrization of the nonlinearity you slightly vary it and describe how the representations change (possibly smoothly). \n\n\nClaims about/with proposed metrics:\n1. It is unclear what the bitcode is showing about the dynamics and about the contribution of the nonlinearity (beyond the subregion type analysis which is specific to PWLs, but  it cannot be applied to the other nonlinearities)\n\n\nClaims that need ablation experiments: \n1. \"Purely linear AL-RNNs cannot solve this task\" and \"Our pipeline accurately identifies that, without nonlinearity, the network fails on this task\". This is only established for the current hyperparameter setting.\nThe following hyperparameters should be explored to make these statements stronger.:\ni. Total hidden size. \nii. tau\niii. Mreg\niv. Task related: length, variable timing etc. \n\n2. \"tanh cannot enforce a clean partition\" this is probably also total size dependent.\n\n\n3. Contributions to slow modes from nonlinearity.\nThe paper lacks investigation of slow modes in the nonlinear part of the network. PWL networks can also create line attractors, see for example\nSágodi, Á., Martín-Sánchez, G., Sokol, P. and Park, M., 2024. Back to the continuous attractor. Advances in Neural Information Processing Systems, 37, pp.66856-66906.\n\n\n\n\n\nQuestions: \n1. Task implementation: \na. Integration relies on reliable end of task. This leads to the summation being decomposed into an immediate and temporally unrolled component.\n\nii. \n\n2. \"such solutions are highly fragile to noise and perturbations\"\nBut the networks are trained without noise, so why is robustness important?\n\"Minimal nonlinearity therefore greatly improves performance\nand robustness.\" There are no experiments to test robustness properties of either the linear or nonlinear networks.\n\n3. \"These results indicate that variable timing shifts the computational burden from storage to decoding.\" The computational burden of what? Storage is still relevant in this task, right?\n\n4. Why were different Mregs chosen for the different tasks?\n\n\nSmall claims:\n1. \"At $P=50$, class manifolds are spatially distinct and well-separated\"\nNeeds: density or alpha or some quantification of average overlap rather than a single example with a visualization based evaluation with sub-optimal visualization.\n2. \"we analyzed a representative AL-RNN with P = 1 that perfectly solves the task\". In terms of MSE? Cross-entropy?\n\n\n\n\n### Clarity\n\n1. Figure 3:  \"first 3 PCs\" but the axes are missing so the projection looks 2D.\n1. The description of the bitcode is confusing: \"To quantify the distribution of latent states across the linear subregions, we extract the last $P$ components of each latent state trajectory\"\nseems different from Eq. 3.", "audience_yn": "Yes", "audience_explanation": "How tasks are solved and how choices to the network architecture (e.g. nonlinearity) are a very important topics in successfully training RNNs and making them interpretable through post-training analysis.", "requested_changes": "## Main requested changes\n\n1. Other architectures should be tested to support the general claims or the claims should be about the AL-RNN architecture specifically.\n\n2. Ablation experiments should be performed (see above).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "phZTqkC5m5", "reviewer": "Reviewer_8uNP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the functional contribution of nonlinearity in modern sequence models, analyzing how nonlinear components influence representation power, stability, and expressivity. The authors decompose sequence architectures into their linear and nonlinear operators and systematically evaluate their roles in capturing temporal dependencies and hierarchical features. Through empirical and theoretical analysis, the paper provides deeper insight into when and why nonlinearities are essential and explores whether linear substitutes can approximate certain nonlinear effects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "For the most part, yes, for the claims made about AL-RNNs on the selected tasks, but not yet fully convincing as a general prescription for sequence modeling. The submission presents clear, targeted evidence that sparse nonlinearity in AL-RNNs yields interpretable, robust, and often optimal behavior on the studied benchmarks, but several gaps limit the broad applicability of these conclusions.", "audience_yn": "Yes", "audience_explanation": "Yes, the analysis of nonlinearity in modern sequence models, which examines how nonlinear components influence representation power, stability, and expressivity, motivates future sequence modeling, especially recurrent networks.", "requested_changes": "1) Run a small set of experiments that replicate the “sparse nonlinearity” vs “full nonlinearity” comparison on at least one structured SSM (e.g., S4/Mamba) and one Transformer-style model or token-mixer. Report whether the same inductive bias holds and discuss required modifications.\n\n2) For cases where fully nonlinear models underperform, include training/validation curves, gradient norms, and any bifurcation/instability indicators. Try and report results for simple mitigations (e.g., progressive increase of nonlinearity, targeted initialization, or regularization) to clarify whether failures are representational or optimization-related.\n\n3) Add experiments on a larger, practically relevant dataset (e.g., long-range language/audio/time-series) to probe whether the sparse-nonlinearity benefit scales to more realistic inputs and longer contexts. \n\n4) Quantify bitcode/subregion usage and its variability across seeds and model sizes (plots or heatmaps). Add targeted ablations suggested by results (e.g., encoder vs decoder nonlinearity for SCAN, isolated removal of nonlinear units in critical layers) and visualize representative latent trajectories to support mechanistic claims.", "broader_impact_concerns": "See Above", "additional_comments": "**Weakness:**\n\n1) While conceptually strong, the scope of datasets used for evaluation seems somewhat narrow.\n\n2) Some theoretical sections are mathematically dense and may benefit from more intuitive explanations or diagrams.\n\n3) The connection between the empirical findings and practical architecture design recommendations could be made more explicit.\n\n**Questions:**\n\n1) Have the authors explored whether nonlinearity can be adaptively modulated during training — for example, through learnable activation strengths — to balance efficiency and expressivity dynamically?\n\n2) Could similar insights be extended to continuous-time or operator-based models (e.g., Neural Operators), where nonlinearity may play a different role in dynamics propagation?\n\n3) How do nonlinearities affect stability and gradient flow in long sequence modeling, particularly in architectures without explicit recurrence?\n\n4) Do the findings suggest an optimal trade-off between linear and nonlinear components that can guide model architecture design in future sequence models?\n\n5) Can the decomposition approach proposed here be used to identify redundant nonlinear layers or to design interpretable linear surrogates for efficiency-critical applications?\n\n6) Failure modes of full nonlinearity and training remedies: The author shows that fully nonlinear models often perform worse and sometimes “linearize” during training. Can you (a) characterize when and why this occurs (e.g., optimization landscape, bifurcations), and (b) test whether targeted initialization, regularization, or curriculum (e.g., start linear then gradually increase P) mitigates those failures?\n\n7) The central claim is that sparse nonlinearity is a useful inductive bias. Do these findings transfer to structured linear SSMs (S4, Mamba) or Transformer-like architectures with nonlinear mixing?", "extra_scores": {}, "extra_text": {}}, {"review_id": "qeJa4iVdAX", "reviewer": "Reviewer_V8Av", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** This paper aims to understand the importance of non-linearity in recurrent models solving integration and memory-based tasks. The authors consider a recently proposed architecture, AL-RNN, to study this. AL-RNN allows controlling the number of non-linear units vs. linear units through the hyperparameter $P$, where $P=0$ indicates fully linear and $P=N$ for an RNN with $N$ units indicates fully non-linear. Furthermore, the AL-RNN formulation allows one to interpret the learned dynamics or solutions through bitcode analysis, i.e., looking at frequencies of activations over timesteps in specific piece-wise regions of the AL-RNN state space. The authors find in most tasks that at least a minimal amount of non-linearity improves performance, including some tasks where non-linearity is necessary in order to solve the task. They also show that fully non-linear models do not perform as well as those with less non-linearity, due to optimisation difficulties. Overall, the paper offers an experimental account on where and how much non-linearity is useful in integration and memory-related sequence tasks.\n\n**Strengths:**\n* The authors have carried out experiments on a diverse set of memory-related tasks, analysing in each the effect of non-linearity on task performance.\n* Using an AL-RNN allows the authors to finely control the amount of non-linearity, enabling direct study of the main question.\n* The AL-RNN formulation also allows in \"symbolic\" interpretability of the activations of the RNNs through bitcode analysis (i.e., 0 if the ReLU is inactive and 1 if active), allowing the authors to track the evolution of network activity over neighbouring regions/bitcodes to interpret the computations being performed.\n* In general, the paper and experiments are presented well.\n\n**Weaknesses/Suggestions/Questions:**\n* The non-linearity here is restricted to ReLU or its variants due to the AL-RNN formulation. There is not much in terms of studying the roles of other non-linearities and gates. Could the authors comment on this? We do see in the addition task that you have gating enabled by non-linearity, however there could conceivably be tasks that LSTMs with forget gates or GRUs perform better than ReLU RNNs, e.g., some word- or character-level language modelling tasks with long-range dependencies. Could the authors comment on this?\n* Is there an explanation for why, in the simpler fixed-delay copy task, any non-linearity more than $P=1$ consistently degrades performance?\n* It isn't completely clear to me how to reconcile the worse performance of fully non-linear models in this paper on some tasks (except IMDB, see Fig. 2b) with the general observation in deep learning that non-linear transformers generally outperform linear models such as structured SSMs (ref. Sec. 5 on linear models' efficacy in long-range tasks). It is conceivable that there might be optimisation issues which could be alleviated by regularisation or sparsity constraints (Khona et al., 2023). Furthermore, it could just be a consequence of the tasks considered here (variants of integration and working memory recall) being much simpler than the computations in real-world tasks, causing more expressive models to overfit or over-linearise non-linear units without appropriate regularisation.\n* Continuing from the previous points, while several tasks considered, each network is trained only on a single task or at best a multi-context task with a single underlying computational motif – and the tasks considered are variants of integration tasks or memory recall tasks, while networks in practice must implement more than just these computational motifs. Several previous works have studied recurrent neural networks trained to perform multiple cognitive tasks concurrently, where multiple dynamical motifs such as line attractors, ring attractors, fixed points, etc. may be implicated. Perhaps training individual models on multiple cognitive tasks (e.g., Yang et al., 2019; Khona et al., 2023; Driscoll et al., 2024) involving several computational motifs might further indicate the importance of non-linearity. It might also be interesting to see just how much non-linearity would be required in such a context, and how it relates to the amount of shared structure across these tasks (perhaps, e.g., more shared structure -> reuse of dynamics across contexts -> fewer partitions of state space required -> less non-linearity required?). The impact of regularisation and sparsity constraints is something to be accounted for as well.\n* [Minor] The intro/related works could potentially also include references to hybrid attention + recurrence models, interleaving attention layers with linear SSMs while still achieving strong performance, e.g. Lieber et al., 2024. This relates to several points in the paper where a combination of both linear and non-linear computations results in best performance.\n* [Minor] The authors have considered only AL-RNNs, and this is a limitation they have acknowledged, it might be interesting to extend these results to variants of linear attention (see Katharopoulos et al. 2020; could use $\\phi(Q) = [elu(Q_i)\\text{ if }i < n - p\\text{ else }Q_i]_{i=0}^{n-1}$ and similarly for $K$).\n\n**References:**\n1. Khona, Mikail, et al. \"Winning the lottery with neural connectivity constraints: Faster learning across cognitive tasks with spatially constrained sparse rnns.\" Neural Computation 35.11 (2023): 1850-1869.\n2. Yang, Guangyu Robert, et al. \"Task representations in neural networks trained to perform many cognitive tasks.\" Nature neuroscience 22.2 (2019): 297-306.\n3. Driscoll, Laura N., Krishna Shenoy, and David Sussillo. \"Flexible multitask computation in recurrent networks utilizes shared dynamical motifs.\" Nature Neuroscience 27.7 (2024): 1349-1363.\n4. Lieber, Opher, et al. \"Jamba: A hybrid transformer-mamba language model.\" arXiv preprint arXiv:2403.19887 (2024).\n5. Katharopoulos, Angelos, et al. \"Transformers are rnns: Fast autoregressive transformers with linear attention.\" International conference on machine learning. PMLR, 2020.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I did not identify any major flaws in this paper, the experiments seemed mostly clear to me. The authors come up with 4 different contexts in which they study the central question of what non-linearity enables in sequences modelling and I believe the experiments and analyses provide evidence of having answered that question. A non-binding suggestion I have is to include a widely used set of multiple cognitive tasks in order to further study how much non-linearity matters in multi-task settings and also in tasks beyond integration. Otherwise, I just have questions and clarifications I believe the authors can address.", "audience_yn": "Yes", "audience_explanation": "The submission is relevant to both sequence modelling in general and specifically to recurrent models used in computational neuroscience as well. In particular, the submission touches upon the following axes from TMLR's CfP:\n* Experimental and/or theoretical studies yielding new insight into the design and behaviour of learning in intelligent systems – this is an experimental study on how non-linearity relates to task efficacy in sequence modelling/memory-related tasks.\n* New approaches for analysis, visualization, and understanding of artificial or biological learning systems – the submission applies an existing model, AL-RNN, in a new context to shed light on when/how much non-linearity is useful in memory tasks.", "requested_changes": "See the questions section of my initial review. In summary:\n* Clarify the lack of analysis on other non-ReLU-like non-linearities.\n* Clearer explanation for why increasing non-linearity worsens performance on the fixed-delay copy task.\n* Comment on regularisation and sparsity in alleviating issues with fully non-linear models.\n* Experiments on other cognitive tasks using different dynamical motifs, and multi-task RNNs trained on several cognitive tasks at once rather than separately.\n* [Minor] Reference to/discussion on hybrid attention + SSM models such as Jamba and others.\n* [Minor] Potentially experimenting with variants of linear attention to broaden the scope/impact and move beyond just recurrent models.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)