Emergence of Spatial Representation in an Actor-Critic Agent with Hippocampus-Inspired Sequence Generator
Authors: Xiao-Xiong Lin
Yuk-Hoi Yiu
Christian Leibold
Venue: ICLR 2026 Poster (2026)
Keywords: neuroscience
deep reinforcement learning
place cells
navigation
Primary area: applications to neuroscience & cognitive science
TL;DR: A minimal, sparsely driven sequence generator in actor-critic agent not only supports successful navigation but also gives rise to hippocampus-like spatial representations.

Abstract: Sequential firing of hippocampal place cells is often attributed to sequential sensory drive along a trajectory, and has also been attributed to planning and other cognitive functions. Here, we propose a mechanistic and parsimonious interpretation to complement these ideas: hippocampal sequences arise from intrinsic recurrent circuitry that propagates transient input over long horizons, acting as a temporal memory buffer that is especially useful when reliable sensory evidence is sparse.
We implement this idea with a minimal sequence generator inspired by neurobiology and pair it with an actor–critic learner for egocentric visual navigation. Our agent reliably solves a continuous maze without explicit geometric cues, with performance depending on the length of the recurrent sequence. Crucially, the model outperforms LSTM cores under sparse input conditions (16 channels, $\sim2.5\%$ activity), but not under dense input, revealing a strong interaction between representational sparsity and memory architecture.
Through learning, units develop localized place fields, distance-dependent spatial kernels, and task-dependent remapping, while inputs to the sequence generator orthogonalize and spatial information increases across layers. These phenomena align with neurobiological data and are causal to performance. Together, our results show that sparse input synergizes with sequence-generating dynamics, providing both a mechanistic account of place cell sequences in the mammalian hippocampus and a simple inductive bias for reinforcement learning based on sparse egocentric inputs in navigation tasks.

Published as a conference paper at ICLR 2026
EMERGENCE OF SPATIAL REPRESENTATION IN AN ACTOR-CRITIC AGENT WITH HIPPOCAMPUS-INSPPIRED SEQUENCE GENERATOR
Xiao-Xiong Lin"
Yuk-Hoi Yiu
Christian Leibold
Faculty of Biology & Bernstein Center Freiburg BrainLinks- BiiinLiink-BBainTooon Tools Center University of Freiburg, Freiburg, Germany
ABSTRACT
Sequential firing of hippocampal place cells is often attributed to sequential sensory drive along a trajectory, and has also been attributed to planning and other cognitive functions. Here, we propose mechanistic and parsimonious interpretation to com- plement these ideas: hippocampal sequences arise from intrinsic recurrent circuitry that propagates transient input over long horizons, acting as temporal memory buffer that is especially useful when reliable sensory evidence is sparse. We imple- ment this idea with minimal sequence generator inspired by neurobiology and pair it with an actor-critic learner for egocentric visual navigation. Our agent reli- ably solves a continuous maze without explicit geometric cues, with performance depending on the length of the recurrent sequence. Crucially, the model outper- forms LSTTM cores under sparse input conditions (16 channels, 2.5% activity), but not under dense input, revealing strong interaction between representational sparsity and memory architecture. Through learning, units develop localized place fields, distance-dependent spatial kernels, and task-dependent remapping. while inputs to the sequence generator orthogonalize and spatial information increases across layers. These phenomena align with neurobiological data and are causal to performance. Together, our results show that sparse input synergizes with seguence-generatnne dynamics, providing both mechanistic account of place cell sequences in the mammalian hippocampus and a simple inductive bias for reinforcement learning based on sparse egocentric inputs navigation tasks. Code: https: http://ggtttt.tt/// com wwwp///ixxiooggi/////t... hippos lam
INTRODUCTION
Hippocampal place cells track the animal's location during navigation (O'Keefe Dostrovsky
1971 1) and they fire in sequence reflecting the behavioral order of the place fields (Foster& & Wilson
2007). Spatial locations are thereby thought to serve as anchors for episodic memories (Aronowitz
Nadel 2023). view reinforced by observations of "look-ahead" sequence replay linked to trajectory planning (Foster& & Wilson| 2007 Kay et al.] 2020). On the other hand, hippocampal neurons are also found to fire successive moments not explainable by location alone (Eichenbaum 2014).
suggesting the place cell sequences could reflect timing rather than spatial input.
The intertwined spatial and temporal representations in hippocampus has been touched upon in many recent computational models. Successor representations interpret place cells as predictive states (Stachenfeld al.) 2017 Mattar & Daw 2018); reservoir models emphasize pre-existing
dynamics in shaping place cell sequences (Leibold 2020); probabilistic approaches model place cells
as latent states inferred from successive inputs (Raju et al. 2024]; and self-supervised methods refine spatial tuning by exploiting temporal smoothness of the trajectory (Wang et al., (224)). While these approaches reproduce place-like activity and even sequential patterns, they rarely address explicitly where hippocampal sequences originate. Many bottom-up studies instead emphasize that structured representations emerge from circuit dynamics and physiological constraints, suggesting mechanistic
"Correspondence 1 xxxggggmaal com

---

Published as a conference paper at ICLR 2026
A
B
D
* reward E
fixed module
trained module
Varn Theta
Vis. Encoder
DG
CA3
Decoder
FC 16
128
MaxPool
Z-score BNC
ReLU
ResBlocks2
Block c-
Thresholdi
FC 128
CA
Block 2-22222520
Block 3-32 cb
Landmark
tuning
Broad place tunting
ReLU
Value
2C666 ReLU
Action(5)
Theta Cycle
Place
Direct input
Field
Influence vin recurrence
96x7243
Figure 1: Model summary. A: Illustration of theta sequences observed in rodent hippocampus, In each theta cycle, R 3 neurons are activated and the activation propagates over L = 4 theta cycles in sequence
R 1 6 neurons. B: The theta sequences are thought to be driven by sequential inputs despite the recurrent connections in hippocampus. C: The recurrent connections could support generating long horizon sequential activity without sequential external inputs. D: A virtual environment (19 x 19 tiles) was constructed using DeepMind Lab with walls randomly placed on 15 % of the tiles. Wall layouts are kept fixed across episodes, with an invisible reward near the bottom right. E: The agent receives a first person perspective visual input that is processed via an encoder (shallow ResNet with 3 convolutional blocks; matching the SOTA in DeepMind Lab environment (Espeholt et al. 2018), pretrained and fixed in experiments). These outputs linearly mapped to F = 16 features (FC: fully connected layer), and then sparsified using batch normalization and high thresholding (T 2.43), such that the percentage of activation (~ 2.5%) matches the sparse activity of DG granular cells that project to CA3. CA3 is modeled via sequences of neuron activations, each sequence evoked by a distinct DG input feature. The activity of all CA3 neurons are then flattened and linearly mapped to the decoder multilayer perceptron. CA3 is hard coded to isolate the effect of long range integration. The DG and decoder modules are trained.
models are needed to link anatomy, dynamics, and representational patterns (Lin et al. 2023 Buzsáki
& Tingley 2018 Schaefler et al. 2022).
We propose a parsimonious account directly addressing this question: hippocampal sequences arise from intrinsic recurrent circuitry in CA3 that can propagate activity over long timescales in the absence of input (Fig. [ACC). The CA3 sequence generator receives sparse inputs from dentate gyrus (DG) yield localized spatial codes in CA3 that support navigation (Leibold 2020 2022). The sparse-input regime is not incidental but central: reflects the ecological reality that navigation often guided by only few reliable landmarks amid abundant sensory noise; the biology of DG granule cells, which fire at extremely low rates; and the computational advantages of high-capacity, low-interference codes that promote compositionality and generalization.
This mechanism mirrors recent key ideas from machine learning. State-space models and structured linear RNNs preserve long-range information by expanding inputs into -igheniineni temporal feature space before compressing them via shallow nonlinear readouts (Fu et al. 2022 Gu et al. 2020, 202T). Our model resonates with this principle: DG sparsification provides a low-activity code that is sustained and expanded by intrinsic recurrence, offering rich set of features for downstream policy learning.
To test this hypothesis, we implement an agent with a DG-like sparsification module, a recurrent sequence generator (CA3 proxy), and an actor-critie learner for egocentric navigation. We show that sequence generation and sparse input synergize, outperforming LSTMs of comparable size in the sparse-input regime, while LSTMs remain competitive under dense input. Moreover, place fields, DG orthogonalization, and task-dependent remapping emerge naturally during training.
These results suggest that a sequence-basee reservoir, inspired by CA3, is well suited for constructing spatial representations from sparse wwwbbadwidth inputs. The synergy between sparse coding and intrinsic sequence dynamics thus offers both mechanistic explanation for hippocampal sequences and simple inductive bias for reinforcement learning in navigation tasks.
2

---

Published as a conference paper at ICLR 2026
2 METHODS
2.1 VIRTUAL ENVIRONMENT
We consider an agent navigating continuous environment with sparse obstacles and uniform visual textures. The environment was simulated using DeepMind Lab (Beattie et al.) 2016) (Figure IP). The wall layouts were kept fixed for repeated episodes unless otherwise stated. The environment was designed such that spatial relations between locations cannot he trivially inferred from the similarity of their corresponding visual features and that it allows multiple possible trajectories from a given location to the target.
In each episode, the agent was initially placed at a random location at least units from the goal. An episode is finished when the agent reaches the goal or maximum of 7200 frames (with action repeat=8 corresponding to a maximum of 900 action steps, i.c. time steps in the model). For the map displayed in Fig. D the reward placed near the bottom right corner and then moved to the lower left corner to test generalization.
2.2 VISUAL PROCESSING
The visual encoder is intended to mimic visual cortical preprocessing, extracting general-pun purpose visual features for the hippocampus. We pretrained a ResNet encoder (He al.) 2015) combination with an LSTM core as in Espeholt et al. (2018) and kept it fixed when training our hippocampus model. We also tested a variant using the layer 2 output from ResNet pretrained on Imagenet. The model produces similar performance (Fig. A2
2.3 HIPPOCAMPUS MODEL
Dentate gyrus sparsification module. A main cortical input the hippocampus is routed through the dentate gyrus (DG) (Amaral et al. 2007). DG activity is characteristically sparse, with only about 2-5% of granule cells active in given environment (Henze al., 2000 Leutgeb & Leutgeb] 2007). We model the DG as linear projection of visual features followed by batch normalization (running estimates mean and variance, retention rate 0.95 per minibatch) and high activation threshold to match the sparsity of activity (~ 2.5%).
CA3 as a sequence-generating shift register. We model CA3 as a linear RNN shift register that propagates inputs as theta sequences (Fig. D and E, CA3) and, for this paper, we keep it fixed to isolate the effects of intrinsic sequence generation from effects potentially induced by recurrent plasticity. The DG provides input u, R. over F features. Each feature is assigned dedicated prewired sequence length f, so the CA3 state is X, E Ft Motivated by hippocampal theta sequences (Dragoi & Buzsáki 2006 Foster & Wilson 2007 Leibold 2020), we parameterize the total number of sequence units as & L-L+ (R 1), where L is the number of theta cycles and R the number of active units per cycle.
Single-feature dynamics (F 1), Let R denote the CA3 state for a single feature and Wr the corresponding DG input. Sequence propagation
Ø141
Sz. + J the
(1)
where S E ExE is the shift operator and J E I fx1 injects the input into the first R slots:
0
1
0
0
S
J
0.0 O.O. 0 e
(2)
0
Ext
Thus a transient input u, creates activity in the first R positions which is then shifted one step per timestep along the length-( register.
3

---

Published as a conference paper at ICLR 2026
A
1-64,
B
new Rew.
C
Sparse Input
D
Dense Input
R-8
new mup
Stccess Rate
R-8
block path
Success Rate
CA33
RandRNN
SSM LegS
LSTM
R=1
1.0
1.0
1.0
0.5
0.5
Saale Len (frames)
0.5
Stable Len (frames)
0.0
0.0
5000
5000
5000
5000
0
Sucvess
0.5
Sixcess
0.5
1.0
0.0
0.0
0.0
0.0
3 1e8 Environment Frames
0
1e8
1e8
Environment Frames
Environment Frames
Environment Frames
Figure 2: Training performance A Agents with different sequence length L and repeat R. Line and shaded area are mean and s.e.m. across random seeds. Success Rate: fraction of episodes where the goal is reached. Len: number of frames to reach the goal. Stable success: the rate of having 100 consecutive successful episodes. Metrics were Gaussian smoothed, a = 6 x 10 frames. The best performing agent with L = 64 and R = 8 across all seeds was tested for its transfer learning. New Rew.: new reward location at the lower left corner. new map: a randomly generated map with the same statistics. block path: new walls are added block paths to the reward, while the previous blocking walls were removed. C and Performance of agents with different recurrent modules and inputs. C: Sparse input. D: Dense input where natch-normalization and high thresholding was removed. CA3: our CA3 model with L = 64 and R = 8. RandRNN: randomly initialized fixed RNN of the same state size. SSM LegS: fixed SSM HiPPO-LegS from Gu al. (2020) with the same state size. LSTM: trainable LSTM with matching number of parameters.
Multiple features (F > 1). Each feature dimension evolves independently under the same dynam- ics. Stacking all F sequences with Kronecker expansion gives the blokksttruttured update
A = IF @ S, B Ir ? J,
where F is an identity matrix that has the size of the number of DG features. and are the same as the single feature case (For extended explanation, see Appendix B.2). and the full CA3 dynamics
X AX, + B un
with fixed recurrent matrix A and input matrix B.
2.4 ACTOR CRITIC NETWORK
The activity of hippocampal units is fed into a decoder module with two linear/fully connected (FC) layers with ReLU activation. Actions and value are computed the linear readout from the decoder module. We trained the agent with a standard advantage actor actor-critic objective ppolicy-gradiens + value-baseling + entropy regularization) as implemented in Sample Factory (Petrenko al. 2020 Espeholt et al. 2018).
3 RESULTS
3.1 BEHAVIORAL PERFORMANCE
Training the naive agent 64, R 8) on one fixed reward location the maze exhibits robust performance measures after about 350 million frames (Fig. 2A)). Thus our model effectively maintains information about sparse inputs in its trajectories of the RNN-like dynamics across time like in reservoir (Jaeger & Haas, 2004). This view is further supported by our simulations with reduced sequence length showing inferior performance (Fig. A)). In the most extreme case (L = 1, R = 1) without sequences, essentially a pure feedforward architecture corresponding to brain with DG output bypassing CA3, the agent did not achieve robust behavior. While agents with reduced sequence lengths (L 1, 16, R 8) still express reasonable success rates and trajectory lengths, their behavior is considerably more unstable as evidenced by the low fraction consecutive successes (Fig. Z The agent's performance is stable within wide range of R and becomes more sensitive
4

---

Published as a conference paper at ICLR 2026
DG-CA3 Agent
0.00
0,02
0.04
LSTM
Frequency
Epoch 0
Epech I
Epoch 2
Epoch
Epoch
Epoch 5
Epoch 6
new Rew.
Agent
20
20
Figure 3: Evolution of occupancy over the course of learning. Color represents normalized measure of the amount of time points the agent spent at a location. Mean head directions at a location were visualized in arrows. From epoch 5 onwards, the agent has preference to reach the bottom right corner (independent of the random starting location) and proceed to the goal (red star) from there. The last panel shows the behavior of fully trained dense LSTM agent.
Table I: Performance comparison across architectures. Training steps indicate the number of environment frames (in millions) required to reach 80% success. "X" indicates that the threshold was not reached in all random seeds within 350M frames.
Input
CA3
Random RNN
HiPPO-LegS
HiPPO-LegT
HiPPO-LagT
LSTM
Sparse (Steps)
173.6=77.6
X
X
X
X
X
Sparse (Succ.)
0.16:10.0
0.51
0.52+0.11
0.57
0.65+10.10
0.56+0.06
Dense (Steps)
X
x 0.65+
133.9::27..6
Dense (Succ.)
0.71++0.07
0.78++0.15
0.64+0.21
0.38+00.02
0.93
Transfer learning for new reward location and blocked paths, however, requires only about 50 million frames, and transfer learning to map requires about 150 million frames. These indicate that the agent was able to form a generalizing representation of the map and task (Fig. 2
R when running speed is slower (Fig. A1 and Tab. A3
The agent equipped with a sequence-generating module (CA3; eq 4 learns faster under sparse DG input than agent in which CA3 is replaced by LSTM core, resembling the SOTA architecture of Hessel et al. (2019) on the DMLab-30 benchmark (Beattie et al., 2016), from which our environment adapted. Our CA3 module also outperforms state-space model HiPPO-LegS (Gu al. 2020) and, as an additional control, randomly initialized RNNs, indicating that its theta-sequence dynamics provide a distinct advantage (Fig. (C).
Crucially, this advantage is confined to the sparse-input regime. Under dense input, LSTMs perform better-consiie consistent with prior reports (Hessel et al. 2019) and our CA3 module performs worse than HiPPO-LegS and random RNNs (Fig. 2D). We also tested other variants from Gu et al. (2020) and they performed similar to HiPPO-LegS (Tab. [ Fig. A4). These results highlight a specitic synergy between sparse representations and intrinsic sequence generator.
3.2 OCCUPANCY
We divided the training into 6 epochs and evaluated the behavior and learned representations at these checkpoints. The agent develops stable trajectory after 4 to 5 epochs. Between Epoch 4 and Epoch 5, the agent learned to get around the obstacle in the upper part of the right edge and started to spend more time at the lower right corner before reaching the reward site (Fig 3 The agent appears to develop a strategy of visiting locations with salient input/landmarks and converging from different starting locations to the habituated paths, even after switching to new reward location. This is consistent with the typical strategies employed by animals in familiar enviroments (Gibson et al.) 2013). In comparison, the LSTM agent under dense sensory input has fewer converging trajectories before reaching the reward (Fig. 3 and Fig. ATT] likely due the goal information being more readily available in the visual input, thereby implementing strategy more related to visual search.
3.3 PLACE FIELD ANALYSIS
In order to understand how the agent executes a successful strategy and whether the representations in the hippocampus-insppees model would also align with the known physiology of place cells,
5

---

Published as a conference paper at ICLR 2026
A
B
DG Input
CA3
Decoder layer
LSTM units
no.
no.
no.
no.
SI-0,03
S1-0.1
SI-0.09
SI-0.07
S1-0.04
Feature
SI-0.10
S1-0.62
SI=0.50
SI-0.38
SI-0.36
Feature
SI-0.03
$1=0.1
S1-0.10
SI-0.07
S1=0.0s
Feature 8
SI-0.11
S1-0.57
SI-0.31
S1-0.15
SI-0.12
Feature 12
Figure 4: A Spatial tuning of DG and CA3 units from epoch 6. Pixel coordinates correspond to environment (black crosses correspond to walls). Each row shows the CA3 units ordered by their positions in the activity sequence. We selected 4 out of the 16 feature sequences for visualization. Spatial Information (SI): bits per time step. B Spatial tuning of randomly selected Decoder layer 1 units from epoch 6. C Spatial tuning of example LSTM output units in the dense LSTM agent. Colormaps are min-max scaled.
Epoch 0
2
5
6
new
1.0
Correlation
DG unit
0.5
0.0
DG unit
Place field Entropy
7.5
bit/stop
50
50
CA3 unit no. in sequence
Figure 5: A Correlation (color code) of the 16 DG activity maps evoked by the visual features. Throughout training, DG input to CA3 became increasingly orthogonal. The spread of CA3 place field measured by entropy. The units are ordered by their positions in a sequence. Each sequence is indicated by a unique color across epochs. C Same as but for the spatial information of CA3 units.

---

Published as a conference paper at ICLR 2026
further computed activity maps of the units for the different layers of the model network. For the input units (DG), we (by construction) observe sparse activation that is distributed across few locations in the map that is further refined in the subsequent linear readout in decoder layer 1 (Fig. ABB). Hidden units in the LSTT agent show non-localized tuning, distinct from typical place cells (Fig. 4C) More interestingly, learning the weights from the visual encoder DG leads orthogonalization, such that DG population activity over time develops unique representation of individual locations (Fig. (AA).
Units farther away from DG input in CA3 sequences show broader spatial tuning (Fig. AA; quantified in Fig. 5B) matching previous reports (Jung & McNaughton 1993] Parra- Barrr--Brrrroooo et al. 2021). The place fields of DG and CA3 units gradually stabilize through learning and exhibit remapping after adapting to a new reward location, measured by shifts of the place field center of mass (Fig-AAD)) In contrast, the input to LSTM agents did not show orthogonalization of the learned input projection (Fig.AA A14).
3.4 SPATIAL INFORMATION ANALYSIS
Qualitative inspection of the spatial tunings in Fig. iis also quantitatively supported by the distribu- tions of spatial information (SI) of the neural activity maps: CA3 units activated 16 steps after the input have larger SI (Fig. (C). This corroborates our training results that agents with sequence length L 16 are able to acquire the task reasonably well (Fig. 2 Units activated later in sequence exhibit smaller spatial information (Fig. 5C).
Tracking the distribution of SI rates across training epochs, we observe a growing long tail indicative for the development of a neural space representation throughout all layers (Fig. A88). Furthermore, ST also increases with increasing hierarchical level of the network layer until the first layer of the decoder network.
To test whether SI causal to behavior, we selectively permute the output weights of Decoder layer I units. When the permutation was done the 32 units with the lowest SI, the performance was not impaired with respect to success rate and trajectory length (100% and 1065 frames). When the permutation was done on the 32 units with the highest SL. the success rate dropped by 4.9% and the average trajectory length increased from 1065 frame to 2794 frames.
3.5 POPULATION-LEVEL REPRESENTATIONS
We next examined population-level representations. We employ a novel method inspired by popu- lation vector (PV) correlation and representational (dis)similarity analysis (Kornblith et al.) 2019 Kriegeskorte & Weil 202T). It measures the mean population vector correlation a pair of location bins grouped by the spatial displacement Az, Ay. This measure can be interpreted as a spatial kernel learned by the network (Fig. 6). An unbiased representation of spatial geometry would show a kernel function that is isotropic, and smoothly and monotonically depends the distance between locations bins. Conversely, the representation could be restricted to specific location bin pairs, displacement along specific orientation, specific spatial frequency, simple visual resemblance. Kernels from all layers (DG, CA3, and Decoder layers) exhibit some dependence on distance throughout learning. However, CA3 kernels are smoother than the DG kernels. Both DG and CA3 show lower correlation values compared to the decoder layer. Notably, the Decoder layer I develops the most pronounced spatial tuning (Fig. AA), consistent with the single-unit selectivity observed in Fig. A8 After the reward location was changed ("new""), kernels became less sharply defined, indicating that the rep- resentations are disrupted although behavior is adapted. Most layers in the agent with LSTM core and dense input did not show spatial kernels with strong displacement dependency. Only the LSTM output units showed gradually refined spatial kernel during learning but is strongly non-isotropic (Fig. B Fig. A15). The dependence of representational similarity on distance and orientation is quantified in Fig. 6C
Over training, stable place code gradually emerges and remaps after learning the new reward location, indicated by the correlation of population vectors across training epochs (Fig. A9). Interestingly, the similarity between fully trained networks (epoch 6) and the novel reward condition ("new") is higher than between the naive (epoch 0) and trained (epoch 6) networks, suggesting that the agent acquires generalizable knowledge about the arena's 's spatial layout.
7

---

Published as a conference paper at ICLR 2026
A
B
DG-CA3 Agent
LSTM Agent
CA3 Final
CA3 new Rew.
Early
Mid
Final
new Rew.
Final
LSTM Final
1.0
Input
Decoder 1 CA3 DG
0.8
PV correlation
LSTM Decoder Output
0.6
0.4
17-
0.2
0
-17
0.0
177
17
1020 Orientation Distance
Ax
Figure 6: Kernel representation of distance. A Mean Pearson correlation (color code) of pairs of population activity vectors (PV) as function of spatial displacement. Rows correspond to different layers (brain regions) the network, columns to learning stages (epochs). Same as A, but in dense LSTM agents. C PV correlation as function of distance (left) and orientation (right).
DISCUSSION
In this study, presented minimal model of the hippocampus that enables navigation in vision- based virtual environment and reproduces key phenomena of spatial representation observed in the mammalian hippocampus.
The cornerstone of our model - the sequential connectivity in CA3 biologically plausible and inspired by established findings theta sequence firing (Dragoi & Buzsáki 2006 Foster & Wilson 2007; Yiu et al., 2022). By constraining CA3 recurrent connectivity as a dynamical reservoir and limiting training input and output weights via reinforcement learning, we clearly isolate the effect of this theta sequence generator.
Agents equipped with the CA3 sequence module achieved superior navigation from sparse input (analogous to DG activity) compared to LSTMs of similar complexity or SSM-based cores. Crucially, this advantage was regime-dependent: under sparse input, CA3 dynamics and DG sparsification synergized to support robust navigation, whereas under dense input, conventional recurrent cores such as LSTMs performed better. Performance also degraded substantially when CA3 sequence length was ablated or shortened (Fig. 2). highlighting the functional role of theta sequence generator.
Analysis of neural activities revealed clear parallels with experimental observations of place cell properties, including robust place field formation (Fig. 4. population-level encoding of spatial distance (.. progressive orthogonalization of DG outputs (Fig. 55)), and dynamic remapping triggered by changes in reward locations (Fig. A9 A10 consistent with experimental findings (Leutgeb & Leutgeb 2007
Fenton 2024).
Interpretation. The CA3 module expands sparse DG codes into a temporally smoothed canonical basis set, providing long-horizon history without the indiscriminate feature mixing of fully connected RNNs. This is especially beneficial under sparse input, where immediate sensory signals are limited and long-range context is critical for policy learning. The repetition parameter R functions built-in prior of the temporal smoothness of latent states, thereby smoothing the CA3 spatial tuning via the agent's 's movement, at the cost of imprecise timing of input memory.
At first sight, sparsifying input to a recurrent core seems purely Eatrimentall-eaviine the agent "blind" most of the time. Yet also filters out noisy, non-informative cues, making each supra-threshold input more reliably tuned to restricted regions of space. Through sequence propagation and policy learning, spatially smooth representations can then be stabilized. This mechanism is consistent with habitual trajectories: neuronal activity late in a sequence only remains informative if the policy converges onto consistent paths.
8

---

Published as a conference paper at ICLR 2026
Under dense input, LSTMs perform better, likely because the abundant weak signals can be accumu- lated to inform space, reducing the need for long-horizon buffering of strong signals. These intuitions supported by comparative experiment where different levels of Gaussian noise were added to the pixel input to suppress weak signals. The DG+CA3 agent suffers less from increasing noise level than the LSTM agent (Fig. A5 Tab. A4).
Overall, these results suggest that different recurrent architectures are suited to different sensory regimes: sparse coding naturally complements sequential expansion, whereas dense input favors mixing-oriented recurrent dynamics. The agents' behaviors also diverged, resembling memory-driven navigation in our model versus more sensory-drie -rivven "visual search" navigation with LSTMs.
Biological and computational relevance. We proposed parsimonious account of hippocampal theta sequences: they can intrinsically maintained without requiring external input. While actual theta sequences in biological systems likely arise from multiple factors and vary across contexts (Chance 2012: Schlesiger et al. 2015 Yiu & Leibold 2023 Ahhadi et al. 2025). our abstraction nonetheless reproduced hallmark hippocampal phenomena, including robust place fields (Fig. progressive DG orthogonalization (Fig. 5A), distance-dependdnt population kernels (Fig. [ remapping after goal changes (Fig. A9 and Fig. ATO), and, by construction, theta sequences (Foster & Wilson 2007). These representational effects were linked to performance and align with evidence on sparse DG activity and CA3 sequences (Leutgeb & Leutgeb 2007 Fenton 2024).
Importantly, the visual encoder can be viewed as capturing the visually driven components of entorhinal cortex (EC) input to DG and CA3: the dorsal "where" and ventral "what" streams that project to medial and lateral EC (Wang et al. (2011). The depth sensor signal fed to Decoder can be seen as temporoamonic pathway from EC to CAL. To reflect the multi-moda nature of EC, additional modalities could be incorporated as input features.
Beyond explaining brain phenomena, our brain-inspired minimal model also proved useful as a module in competitive deep RL agents. The CA3 shift register can be viewed as sparsely active reservoir that generates linite-length temporal bases, contrasting with the rotational modes of Legendre SSMs and the fading modes of Laguerre SSMs. Its structure resonates with recent shift-diagonal architectures (Fu et al., 2022). but tuned to sparse sensory regimes and navigation tasks. Thus, the model not only explains how theta sequences and place cells form but also provides normative account of the computational effectiveness this mechanism for navigation.
The comparison between hippocampus-inspee demonstrate the distinction of learned strategies and representations, even though the performance indicators are comparable (Figs. A15A14A1 A1ZA13
model and LSTM agent with dense input
Biological Predictions. Our model predicts that larger environments or sparser inputs require longer sequences for successful navigation, consistent with developmental adaptability of sequence length (Wikenheiser & Redish 2015 Farooq & Dragoi 2019). More broadly, it suggests that hippocampal spatial representations could rely largely on intrinsic eequennce-geerratin
circuitry, with experience primarily shaping feedforward and readout connections. It offers an explanation for how place cells can persist despite lesions of entorhinal cortex (Hales et al.] 2014, Schlesiger et al., 2015). The parsimonious nature of our model also provides unified mechanism how navigation upon sequences, no matter how they arise and how they differ across species. This is particularly important in species without prominent theta oscillations, e.g. bats show hippocampal sequences locked to wingbeats (Forli et al. 2025).
be built
Future Directions. Several extensions follow naturally. On the biological side, incorporating local plasticity rules into the DG-CA3 pathway or CA3 readouts would align the model more closely with known mechanisms while preserving the benefits of prewired dynamics. Interactions with path integration in medial EC could further clarify their complementary roles in spatial cognition. Moreover, theta sequences have also been observed other brain areas, raising the possibility of hierarchical coordination across regions.
These ideas resonate with developments in machine learning: structured dynamics akin to SSMs and linear attention increasingly used in modern sequence models, including LLMs (Fu et al. 2022 Katharopoulos et al. 2020). suggesting that hippocampal-like motifs may illuminate principles of efficient long-range computation. The simplicity of model also makes it interpretable: sequentially

---

Published as a conference paper at ICLR 2026
connected CA3 units can be seen structurally representing trajectories, which could in turn motivate algorithms that minimize internally measured trajectory length without relying on external reward.
On the ML side, combining sparse sequential reservoirs with learned SSMs may yield hybrid architectures capable of adapting flexibly across imput regimes. More broadly, the bottom-up ap- proach-focusing focusing eircuit motifs rather than top-down expert informed objectives- path toward scaling to larger networks and more complex tasks.
RL for neuroscience. Our framework showcases how modern RL can serve testbed for computational neuroscience. By embedding a biologically inspired hippocampal circuit within an end- to-end reinforcementlleaminng agent, the model enables to study brain computation in the context of the full perception-action loop, rather than in isolated modules for sensory processing, cognition, or motor control. Our approach thereby relates to ongoing discussions machine learning regarding the "Bitter Lesson", which cautions against relying on handcrafted intermediate representations in favor of scalable end-to-end optimization. Importantly, our structural priors (sparseness, sequences) kernels, and remapping
do not prescribe the representational solution; place fields, distance- emerge through end-to-end learning. In this sense, physiological constraints act as an inductive bias that narrows the hypothesis space without sacrificing scalability, providing a middle ground between unconstrained black-box box models and heavily engineered representational schemes. This perspective imposing minimal structural constraints, such as sparse connectivity, rather than making apriori assumptions - is particularly relevant for association cortices such as hippocampus and prefrontal cortex, where emerging representational patterns shaped jointly by sensory inputs, movement statistics, and behavioral goals (Lin et al. 2023).
An important conceptual point concerns how our representations relate existing theoretical frame- works. Our CA3 representations relate to both successon--eprrssetatiin (SR) and reward/value- frameworks, but with important distinctions. As in SRs, CA3 activity policy dependent and shows a consistent temporal ordering: because the CA3 module propagates an intrinsic sequence, each unit predicts its downstream neighbors. This produces SR-like anticipatory structure, yet it arises solely from the fixed pessioooy-iispiree architecture CA3 does not learn a predictive map and receives no TD updates discounted occupancy signal.
At the same time, unlike many previous RL studies of hippocampus (Kumar et al. 2022 Leibold 2020), that takes allocentric input and analyze how policy shapes and utilizes them, our contribution complementary: we show how localized, Gaussian-like place tuning can emerge from egocentric observations when combined with sparse DG input and intrinsic CA3 sequences. Our work also distinguishes itself from many other work that takes egocentric input but where the spatial similarity structure is already reflected in the input (e.g. small room where each wall has a different color; Vijayabaskaran & Cheng (2022); Raju al. (2024)). Our environment is large maze, with uniform looking obstacles scattered in the arena, accounting for a more realistic discrepancy between visual representation and spatial representation.
The behavioral and representational divergence between the CA3-agent and the LSTM-agent illus- trates a promising direction for future comparative work. Differences in qualitative strategies- not only performance-car reveal how architectural constraints shape navigation, representational geom- etry, and generalization, providing insights that are difficult to obtain from performance benchmarks alone.
Conclusion A minimal, sparsely driven sequence generator in actor-critic agent not only supports successful navigation but also gives rise to hippocampus-lit spatial representations.
5 ACKNOWLEDGEMENTS
This work was funded by the German federal ministry for research, technology and space travel (BMFTR) under grant number 01GQ2510 and the Bainnlinks-Bratie Center.
REFERENCES
Siavash Ahmadi, Takuya Sasaki, Marta Sabariego, Christian Leibold, Stefan Leutgeb, and Jill K Leutgeb. Distinct roles of dentate gyrus and medial entorhinal cortex inputs for phase precession
10

---

Published as a conference paper at ICLR 2026
and temporal correlations in the hippocampal CA3 area. Nat. Commun., 16(1):13, January 2025.
David G Amaral, Helen E Scharfman, and Pierre Lavenex. The dentate gyrus: fundamental neu- roanatomical organization (dentate gyrus for dummies). In The Dentate Gyrus: Comprehensive Guide to Structure, Function, and Clinical Implications, Progress in brain research, pp. 3-790. Elsevier, 2007.
Sara Aronowitz and Lynn Nadel. Space, and not Time, Provides the Basic Structure of Memory. Oxford University Press, 2023.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefraneq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen, Deepmind lab, 2016. URI.httpa //tp:v//axvvvvv/ http://wwww.r........ wwtp/rag/a/bb/ee/
03801
György Buzsáki and David Tingley. Space and time: the hippocampus as a sequence generator. Trends in cognilive sciences, 22(10))853-86, 2018.
Frances S Chance. Hippocampal phase precession from dual input components. J. Neurosci. 32(47):
16693-703a, November 2012.
George Dragoi and György Buzsáki. Temporal encoding of place sequences by hippocampal cell assemblies. Neuron, 50(1):145-157) April 2006.
Howard Eichenbaum. Time cells in the hippocampus:
dimension for mapping memories. Nature Reviews Neuroscience, 15(11) 15(11):732-744 2014.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International conference on machine learning, pp. 1407-1416. PMLR, 2018
U Farooq and G Dragoi. Emergence of preconfigured and plastic time-compeessed sequences in early postnatal development. Science, 363(6423): (1866443):18--177 173, January 2019.
André A. Fenton. Remapping revisited: how the hippocampus represents different spaces, Nature
Reviews Neuroscience, 25(6):428-445 2024. ISSN 1471-0048 doi: 10.
URL ltttt:::///itt http://wwnatuteeeeeeoee http::/ww.a..acc../rrr//aa////// 024-8-77----0------------------ Publisher: Na- ture Publishing Group.
Angelo Forli, Wudi Fan, Kevin K Qi, and Michael M Yartsev. Replay and representation dynamics in the hippocampus of freely flying bats. Nature, 645(8082):974-998,, 2025.
David Foster and Matthew A Wilson. Hippocampal theta sequences. Hippocampus, 17(11): 1093-1099) 2007.
Daniel Y Fu, Tri Dao, Khaled K Saab, Ammin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
Brett Gibson, William N Butler, and Jeffery S Taube. The head-direction signal is critical for navigation requiring a cognitive map but not for learning a spatial habit. Current Biology, 23(16): 1536-1540, 2013.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances neural information processing systems, 33: 1474-1487) 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2/11..o 0a../..00396 2021.
Jena B Hales, Magdalene I Schlesiger, Jill K Leutgeb, Larry R Squire, Stefan Leutgeb, and Robert E Clark. Medial entorhinal cortex lesions only partially disrupt hippocampal place cells and hippocampus--ependeent place memory. Cell Rep., 933,8993, November 2014.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image 03385
recognition, 2015. URL htttps://axiiv.. http://///.aa...o//a////////////
A Henze, N Urban, and G Barrionuevo. The multifarious hippocampal mossy fiber pathway: a review. Neuroscience, 98(3):407-422) 2000
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Herbert Jaeger and Harald Haas. Harnessing nonlinecarity: predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78-8)) April 2004.
M W Jung and B L McNaughton. Spatial selectivity of unit activity in the hippocampal granular layer. Hippocampus, 3(2):16--122 April 1993.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnss: Fast autoregressive transformers with linear attention. In International conference machine learning, pp. 5156-5165, PMLR, 2020.
Kenneth Kay, Jason Chung, Mariclena Sosa, Jonathan S Schor, Mattias Karlsson, Margaret Larkin, Daniel F Liu, and Loren M Frank. Constant sub-second cycling between representations of possible futures the hippocampus. Cell, 180(3):552-565 567.e25, February 2020.
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519-3529.
PMLR, 2019. ISBN 2640-3498.
Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Na- ture Reviews Neuroscience, 22(1 1):70--718 11 2021. ISSN 1471-0048. doi: 10.1038/ 341583--22100500--3 number: 11 publisher: Nature Publishing Group.
Benjamin Kuipers. The spatial semantic hierarchy. Artificial intelligence, 119(1-2):191-333 2000.
M Ganesh Kumar, Cheston Tan, Camilo Libedinsky, Shih-Cheng Yen, and Andrew Y Y Tan. A nonlinear hidden layer enables actor actor-critic agents to learn multiple paired association navigation. Cerebral Cortex, 32(18))3917-3336 01 2022. ISSN 1047-3211. doi: 10.1093/cercoo//haa455..
URL /attps://doi..ot http::///oo....oooo00.000//////r
Christian Leibold. A model for navigation in unknown environments based on a reservoir of
hippocampal sequences. Neural Networks, 124:328-344 2020. ISSN 08936080. doi: 10.1016/ j.neumet.2220001.014, URL
eeevieeerrreeeeeee..........e... com retrieve/iii/
(S083368802030016)
Christian Leibold. Neural kernels for recursive support vector regression as model for episodic memory. Biological Cybernetics, 116(3):7773386 2022.
Stefan Leutgeb and Jill K. Leutgeb. Pattern separation, pattern completion, and new neuronal codes within continuous CA3 map. Learning Memory, 14(11):745-757, 2007. ISSN 1072-0502, 1549-5485, doi: 103/0/7 URL
cshlp. hotg///on////////// 14/11/745 Company: Cold Spring Harbor Laboratory Press Distributor: Cold Spring Harbor
Laboratory Press Institution: Cold Spring Harbor Laboratory Press Label: Cold Spring Harbor
Laboratory Press Publisher: Cold Spring Harbor Lab.
Xiao-Xiong Lin, Andreas Nieder, and Simon Jacob. The neuronal implementation of repre- sentational geometry in primate prefrontal cortex. Science Advances, 955))caad88685, 2023.
doi: 10.1116/sciad URL htthhttps:/ www.
Sciadv. ad5
Marcelo G Mattar and Nathaniel D Daw. Prioritized memory access explains planning and hippocam- pal replay. Nat. Neurosci., 21 1): 16))))60999000 1607-1617, November 2018.
J O' O'Keffe and J Dostrovsky. The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Res., 34(1):171-175, November 1971.
12

---

Published as a conference paper at ICLR 2026
Eloy Parra-Barrero, Kamran Diba, and Sen Cheng. Neuronal sequences during theta rely behavior- dependent spatial maps. Elife, 10296, 2021.
Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learn- ing. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7652- 7662. PMLR, 2020. URL html
mlr http://s.........../////////tt.e
Rajkumar Vasudeva Raju, J. Swaroop Guntupalli, Guangyao Zhou, Carter Wendelken, Miguel Eazzro-Grreille and Dileep George. Space is a latent sequence: A theory the hippocampus. doi: 10. 126/sciadv.adm84700 URL
Science Advances, :e(3)))adm84400 2024. ISSN 2375-2545 2548.
www.newww.........o..mm..o...eew org doi/ 11.0..1.100 / sci adm8
Rylan Schaeffer, Mikail Khona, and lla Fiete. No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit. Advances in neural information processing systems, 35: 35:16052-16067. 2022.
Magdalene I Schlesiger, Christopher C Cannova, Brittney L Boublil, Jena B Hales, Emily A Mankin, Mark Brandon, Jill K Leutgeb, Christian Leibold, and Stefan Leutgeb. The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity. Nat. Neurosci,, 18 (8):1123-1133) August 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
William Skaggs, Bruce McNaughton, and Katalin Gothard. An information-theorriic ap- proach deciphering the hippocampal code. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufman 1992. URL https:://procceddnnggg ba hic/baasbbec33aaaocttbSss333aae3
httpippp.cpppppppeeeeee filess/pape:/1992/
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel Gershman. The hippocampus as predictive map. Nature neuroscience, 20(1 1):1643-1653, 2017.
Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
Sandhiya Vijayabaskaran and Sen Cheng. Navigation task and action space drive the emergence of egocentric and allocentrie spatial representations. PLoS Comput. Biol., 18110),, 2022. doi: N0133//JOURNAL.PCB...0000 URL 1010320
Quanxin Wang, Enquan Gao, and Andreas Burkhalter. Gateways of ventral and dorsal streams in mouse visual cortex. Journal of Neuroscience, 31(5):1905-1918 2011.
Zhaoze Wang. Ronald W. Di Tullio, Spencer Rooke, and Vijay Balasubramaniaa. Time makes space: Emergence of place fields in networks encoding temporally continuous sen- sory experiences. Advances in Neural Information Processing Systems, 37:77836-37864, httpp//ppprr__ieees/pp/////////p 2024. URL hatp://aaeeserr//caaasaaaaaaa111 Cew.rractt-ooffeennceee html
Andrew M Wikenheiser and A David Redish. Hippocampal theta sequences reflect current goals.
Nature neuroscience, 18(2):289-294, 2015.
Yuk-Hoi Yiu and Christian Leibold. A theory of hippocampal theta correlations accounting for extrinsic and intrinsic sequences. Elife, 12(RP86837), October 2023.
Huk-Hoi Yiu, Jill K Leutgeb, and Christian Leibold. Directional tuning of phase precession properties in the hippocampus. J. Neurosci., 42(11):2282-2299 March 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A LIMITATIONS
Physiological Limitations Our model has fixed sequential connectivity in CA3, ignoring dynamic processes such as synaptic turnover, plasticity, and sequence length adaptations observed biologically. This simplification may limit the biological realism and generalizability of the model to adaptive neural processes. Additionally, the absence of direct entorhinal cortex inputs means have not modeled potential interactions between hippocampal sequences and detailed sensory-motor driven activity.
Conceptual Assumptions We assume sequential activity in CA3 arises exclusively from con- nectivity rather than sequential inputs, potentially oversimplifying hippocampal dynamics. This assumption excludes possible interactions between external sequential signals and intrinsic CA3 network dynamics, which could affect real-world predictive accuracy.
Reinforcement Learning Methodological Limitations Our gradient-basee based reinforcement learning approach lacks clear biological analogs, organisms likely utilize local Hebbian synaptic updates rather than global gradient propagation. Furthermore, the biological interpretation of the multilayer perceptron decoder remains unclear, posing challenges direct biological interpretations of our results. Additionally, our use asynchronous, batch-based training may not accurately reflect the temporal continuity real-time single-organis learning dynamics seen biologically. thus potentially limiting ecological validity.
Technical Limitations Our results depend on the environment specifications and hyperparameter settings. We chose reasonably difficult environment setting where the difference between architec- tures can highlighted. does not imply the effect can generalize to all environments. We tested the agent with L=64 and R=8 on 5 other randomly generated maps, it could converge at similar speed with the map shown other results (Fig. A3). We also tested different frame-skip number, larger frame-skip leads faster learning, and the effect of sequence length performance weaker (Fig.AA6) This is probably due to the fact that our environment has sparse obstacles and that larger frame-skip leads effectively longer sequence and memory. Most results in our report 4 policies for population based training and environments per worker. These parameters also change the dynamics learning. having more policies made agents with L=64 learn slower and L=8 learn slower. Further systematic investigation of these factors is required to establish broader applicability.
Taken together, these limitations highlight important areas for further experimental and theoretical investigation.
B IMPLEMENTATION DETAILS
B. ENVIRONMENT
Description
Look (dx)
Pitch (dy)
Strafe
Forward
Fire
Jump
Crouch
Forward
0
0
Strafe Left
0
-1
0
0
Strafe Right
0
0
Look Left + Forward
-20
0
0
0
Look Right + Forward
20
0
Table Al: Reduced action set used in DeepMind Lab experiments. Each action is defined over the 7-dimensional control space (look, pitch, strafe, forward, fire, jump, crouch).
Navigation task does not require complex actions, thus we reduced the action space used IM- PALA (Espeholt et al. 2018), to facilitate training speed (Table
14

---

Published as a conference paper at ICLR 2026
Since mainly interested in modeling the way-finding aspect of navigation (Tomman 1948 Kuipers] 2000), the MLP also receives average depth the pixels, nown-sampled to 10 horizontal pixels to aid motor control (going around walls and avoiding collisions). This procedure is supposed to mimic the sensory motor collision avoidance habits that do not depend on hippocampus. This depth information is directly routed to the decoder layer, the representations in the recurrent and its input are not directly influenced.
B.2 MULTI-FEATURE DYNAMICS
As more direct illustration for the recurrent dynamics of the sequence network X = X,+ us we, here, provide the matrices A and be for the multi-feature case:
S
0
J
0 S
0
0
j
0
0
A =
B
(5)
S
FEx FE
Fix
with
T 0
07
0.O..... 0 E
(6)
0
R times
times
ext
Note that the rows in A corresponding to the first time step of each feature only contain Os, i.c., sequences can only started from DG input and not from the dynamics itself.
B.3 ARCHITECTURE
Table A2: Network architecture of the CA3 agent.
Component
Details
Encoder (ResNet)
3 conv blocks with residual layers, 16-32 channels, max pooling Linear (3456 + 256), ReLU
MLP head
DG projection
Linear (2566 - 16), BatchNorm (momentum 0.05, affine), ReLU, intercept 2.43
CA3 (recurrent core)
Fixed shift-register reservoir, size (6x()+---))1111
Decoder
MLP: Linear (1136 + 10 + 128). ReLU, Linear (128 - 128), ReLU
Critic
Linear (128 + 1)
Actor
Linear (128 - 5)
CA3 agent has learnable parameters in DG projection and from CA3 to Decoder, the CA3 output in the full version has 1136 188=149504 parameters. LSTM has learnable parameters where m is hidden size and n is input and output size. Solving this gives hidden size of roughly 137, which we used in the implementation. SSM agents have matching state size with CA3 agent. They also have block-diagonal recurrent weights, where the weights in each block are obtained through the original implementation by Gu al. (2020) with sooo-rdder-oldd discretizations.
(m² + mn)
B.4 ACTOR CRITIC
We optimize actor-critic objective with advantage estimates in Espeholt et al. (0018 consisting of clipped policy loss, value regression loss, and an entropy bonus:
[(O, σ) EE, (miin(r(()).))) clip(r.(0), 1- c, + +)) +c, (R((R.VV((s))) E, E.HoTT sx))].
where re = (a; st) me (a, st).
Here RG and the advantage estimates A, are computed using V-trace returns (Espeholt et al. 201 8),
which correct for off-policy updates arising in the asynchronous actor-learner setup of Sample Factory (Petrenko et al. 2020). This formulation known Asynchronous PPO (APPO)-come) combines PPO's clipped surrogate objective (Schulman et al., 2017) with IMPALA's V-trace corrections, enabling scalable training with many actors while maintaining stable policy updates.
15

---

Published as a conference paper at ICLR 2026
B.5 TRAINING CONFIGURATION
We trained agents using sample -eltf---arryyyy Petrenko et al. 2020) with the following setup.
Environment. frameskip repeating action for frames until getting the next observation. Each run used 32 workers x envs/worker, with decorrelation up to 120s.
Algorithm. APPO EEseeolt et al. 2018). 7 = 0.99, rollout length 64, recurrence 64, batch size 2048, 2 batches/epoch. Optimizer learning rate x 10
Architecture. Visual encoder: pretrained ResNet on DMLab and the second layer of ResNet pretrained ImageNet DG: batchnorm ReLU, intercept 2.43, with 16 features. Decoder: 2 MLP layers of size 128.
Population Based Training. Enabled PBT with policies, replacement gaps 0.05 (relative) and 0.2 (absolute), mutation start after 10M steps, period every 2M steps. Policy lag tolerance set to 35.
Logging and checkpoints.
Training for 108k seconds, milestones every 5400s.
Miscellaneous. Seeds: [1111,2222,3333,44444
Device: CPU. Affinity pinning disabled. Inputs not normalized. Other parameters were default in sample-factory for DeepMind Lab.
B.6 SKAGGS' SPATIAL INFORMATION MEASURE
We quantify how much information unit's activity conveys about the agent's location in our discrete- time simulation by a "bits per step" version of Skaggs' spatial information (Skaggs et al. 1992) Let the environment be divided into spatial bins, and define:
Pi fraction of timesteps (steps) spent in bin (occupancy probability),
mean activity rate in bin i, measured in activity per step,
A = En X; overall activity rate (activity per step).
First, the information conveyed per timestep is
Ai
step
Σ Pr
A loga
[bits/step].
This measures the average reduction in positional uncertainty (in bits) each simulation step provides. Interpretation:
Uniform firing (A; À for all yields zero information (Useep Lspike 0). Elevated A; in particular bins gives positive contributions proportional p; A; (for bits/step) or P. (A,A) (for bits/spike).
Bins visited rarely (small p;) contribute less, guarding against over-weighing seldom-visited locations.
In our simulations. A; and À are estimated from spatially binned sum of activity divided by the number of steps in each bin. Bins with X; 0 omitted (treating 0- log 0).
16

---

Published as a conference paper at ICLR 2026
C
SUPPLEMENTARY TABLE
Table A3: The effect of parameter R at different running speed (frameskip). Performance measured the success rate at 150 million training frames. This summary of Fig. AT
Frameskip
R=1
R=4
R-8
R=12
R=16
R=32
8
0.866=0.006 0.788 0.785=101133
0.857=0.006
0.886=#0.089
0.880=0.036 0.901=0.114
0.806= 0.758=0.226
4
0.848
0.874=0.106
0.902=0.116
Table A4: The performance at different noise level. Performance is measured as the success rate at 100 million training frames. This is summary of Fig. A5
Model Noise
10
40
DG+CA3
0.693 + 0.137
0,492 + 0.167
0.539 + 0.108
0.685 + 0,186
0.477
0.072 0.373 0.008
Dense LSTM
0.638 0.063
0.707 + 0.178
0.495
0.010
0.449 + 0.064
D SUPPLEMENTARY FIGURES
frameskip-8
frameskip=4
Len (frames) Success Rate
Len (frames) Success Rate
0.75
R-1
0.75
0.50
R 4
0.50
0.25
R=8
0.25
R-12
5000
R=16
5000
R=32
1.0
1.0
Stable
Success
Stable Success
0.5
0.5
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Environment Frames
1c8
Environment Frames
1c8
Figure A1: Effect of parameter with different running speed, Training performance of CA3
agent with L=64 different R. Random seeds: [1111, 2222, 3333, 4444, 5555].
17

---

Published as a conference paper at ICLR 2026
1.0
Len (frames) Success
0.5
5000
Stable Success
0.5
0.0
Environment Frames
le8
Figure A2: Robustness of visual encoder. Training performance of CA3 agent with the ResNet layer output as visual encoder. Random seeds: [1111, 2222, 3333, 4444, 5555].
Len (frames) Success Rate
1.0
map1
0.5
map2
map3
map4
5000
map5
1.0
Stable Success
0.5
0.0
Environment Frames
1e8
Figure A3: Training performance on 5 other randomly generated maps. Lines with the same color are results from random seeds [0, 42]
18

---

Published as a conference paper at ICLR 2026
A
Sparse input
B
Dense input
Success Rate
1.0
1.0
legt
0.5
0.5
legs
lagt
0.0
0.0
Len (frames)
5000
5000
0
1.0
1.0
Success
Stable
0.5
0.5
0.0
0.0
Environment Frames
1e8
Environment Frames
1e8
Figure A4: Training performance agents with fixed SSM cores described by Gu et al. (020)). legt: Legendre bases with fixed memory horizon. legs: Legendre bases with infinite memory from beginning of an episode. lagt: Laguerre bases, i.e. memory with decay. All three SSMs were implemented with the same state size 64+8-1=71 states. The SSMs were fixed, with three additional learnable parameter controlling the input, recurrent scales and timestep length for discretizations, common practice described by Gu et al. (00T).
CA3 module, each input feature is expanded into
DG+CA3
Dense LSTM
noise level
Success Rate
1.0
Success Rate
1.0
0
0.5
10
0.5
Len (frames)
Len (frames)
5000
5000
1.0
1.0
Stable Success
Stable Success
0.5
0.5
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
2.0
Environment Frames
1e8
Environment Frames
1e8
Figure A5: Training performance of agents under different noise levels. Independent Gaussian noise of different sigmas was added the input image pixels (intensity 0-255). Random seeds: [1111,
2222, 3333, 4444, 5555].
19

---

Published as a conference paper at ICLR 2026
35 66 65 44 10
57
78
114
48
60
87
54
74 102 53 119]
Figure A6: Rate maps from decoder layer in the last training epoch. Showing the random 16 units of the 50 units with the best SI. Unit id displayed on top.
[119 34 57 104 36 114 102 84 78 69 123 49 97
122
Figure A7: Rate maps from decoder layer 2 in the last training epoch. Showing the random 16 units out of the 50 units with the best SI. Unit id displayed on top.
DG
Bew
0
Deender CA3
5
Training Epocc
0
4
3
Decoder 2
Spetial Information (bits per stop)
Figure A8: Distribution of SI rates for different layers (brain regions rows) over the progress of learning (colors) and change of reward site (purple). Y axes: unit fraction density.
20

---

Published as a conference paper at ICLR 2026
DG
CA3
Decoder
Decoder
1.0
tnining spoch
(PV corthx
0.5
0.0
training epoch
Figure A9: PV correlations across training. Although performance saturates by epochs 5-6, represen- tational changes continue. The last epoch shows the effect of learning new goal.
DG
CA3
CoM shift (bins)
CoM shift (bins)
2
0->1
4.55
5
6->new
0-21
4-75
5->6
6->new
Epoch transition
Epoch transition
Figure A10: Convergence and Remapping. The center of mass (CoM) shift of place fields of individual units from epoch to epoch reveals higher stability in DG and remapping for new goal.
2000
2000
2000
2000
2000
position
1000
1000
1000
2000
1000
2000
1000
2000
1000
2000
1000
2000
position
Figure A11: Occupancy map of LSTM agent's trajectories across training.
21

---

Published as a conference paper at ICLR 2026
Figure A12: Place fields from the LSTM core (randomly selected units). Each panel shows the normalized spatial activity of one unit.
Epoch
Epoch
Epoch 2
Epoch 3
Epoch
1.0
Correlation
0.5
0.0
Figure A13: Population vector correlation across epochs. Color indicates Pearson correlation: rows show epochs 0-4.
22

---

Published as a conference paper at ICLR 2026
Feature
Feature
Feature
Feature
Feature
Feature
Feature
Feature
Feature
Feature 9
Feature 10
Feature 11
Feature
Feature 13
Feature 14
Feature 15
Figure A14: Learned dense input features in LSTM agent. Each tile shows the spatial receptive field of one feature.
Epoch 0
1
2
3
4
1.0
Input
0.8
Pearson correlation PVs
LSTM Output
0.6
Decoder2 Decoderl
0.4
17
0.2
A
0
-17
0.0
-17
0
17
AX
Figure A15: Displacement-depeeddent Pearson correlation kernel of population vectors. Axes show
spatial shifts (Ax. Ay); values are averaged correlations.
23

---

Published as a conference paper at ICLR 2026
0.5
Sto00m Raz
0.6
& nan pulcio 4
&mmm polcio
& Auangiiiio
0.03
0.55
0.40
0.75
10
1.50
1.35
0.0
1.0
Traentey Legth (trrn))
0000
Treganery Laghttar
6002
4000
2003
0.50
175
0.3-
0.2
0.4
0.1
0.0
Figure A16: Training performance with different environment action repeats and number of policies population based training. Left: environment frame skip action repeat Right: environment frame skip / action repeat = 4. The frame skip controls the fine-grad Ane-graininess of actions, which is not required for the current navigation task. On the other hand, larger frame skip effectively make sequences propagate over longer traversals.
24


Reviews:
[{"review_id": "IdRPCMLdLC", "reviewer": "Reviewer_vHh4", "summary": "This paper proposes a neuroscience-inspired architecture that combines sparse input - mimiking dentate gyrus (DG) - with a recurrent CA3 sequence generator to model spatial navigation. The authors demonstrate that under sparse input conditions, the CA3 architecture — implemented as a fixed shift-register that propagates theta sequences — outperforms LSTM cores of comparable size on an egocentric visual navigation task in a maze environment. The model develops place cell-like spatial representations and generalizes well to new reward locations, obstacles, and maps after training. Critically, the performance advantage disappears under dense input conditions, suggesting a specific synergy between sparse coding and sequence dynamics that may account for the emergence of hippocampal theta sequences.", "questions": "I believe the CA3 architecture (central to the paper) explanation of Section 2.3 is not entirely clear and I would have preferred an expansion in the appendix, expecially of the \"Multiple Features\" case. In my opinion, the meaning of \"Each feature is assigned a dedicated prewired sequence of length l\" (line 134) requires clarification, as does the definition of $I_F$, $S$, and $J$ in the multi-feature scenario (line 157).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Neuroscience-AI integration:** The work effectively bridges computational neuroscience, AI, and reinforcement learning to account for the neuroscience-related problem of DG and CA3 synergy, theta sequences, and spatial representations emergence.\n\n**Task validity:** The navigation task appropriately mirrors real-world neuroscience experimental paradigms, with the agent navigating using vision to locate rewards.\n\n**Architectural interpretability:** The relatively simple architecture facilitates interpretation of individual components and minimizes confounding variables, allowing clear attribution of the effects observed.\n\n**Generalization capacity:** The trained model demonstrates robust transfer learning capabilities when adapted to new reward locations, novel obstacle configurations, and new maps (Figure 2).\n\n**Regime-dependent performance:** The demonstrated input-regime specificity provides evidence for the functional role of hippocampal sequences: CA3 outperforms LSTMs and alternative architectures under sparse input (~2.5% activity), while underperforming under dense input conditions. This specificity offers mechanistic insight into why theta sequences may have emerged under sparse input conditions.", "weaknesses": "**Incomplete parametric analysis:** The relationship between input density and CA3 performance (also w.r.t. LSTM or alternatives) requires systematic investigation. The paper tests R=1 only with L=1, leaving unexplored critical control conditions such as L=64 with R=4 (more sparsity) or L=64 with R=16 (less sparsity). A thorough parametric sweep varying R while holding L fixed would strengthen conclusions about the sparse-input CA3-DG synergy.\n\n**Missing entorhinal cortex input:** The model considers only DG input to CA3, omitting entorhinal cortex projections entirely. The absence of discussion regarding this major hippocampal input limits biological completeness.\n\n**Insufficient quantitative support for spatial tuning claims:** The claim of line 261 \"Units positioned later in CA3 sequences (e.g. columns no.48 and no. 64 in Fig. 4) show broader spatial tuning as reported experimentally\" lacks clear visual support in Figure 4 and requires quantitative validation.\n\n**Unclear novelty and utility of spatial kernel measure:** Section 3.5's spatial kernel analysis does not clarify whether this measure is newly introduced or standard in spatial representation studies. The section primarily describes how the measure behaves across architectural modules and training epochs without drawing clear conclusions from these observations.\n\n**(minor) Figure readability:** Font sizes in figures, particularly Figure 2, are too small for comfortable reading while going through the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rRupxXFAnP", "reviewer": "Reviewer_2sC9", "summary": "The authors introduce a biologically inspired model of the hippocampus focused on sequential connectivity from dentate gyrus (DG) to CA3, implemented within a reinforcement learning (RL) agent for visual navigation tasks. The model is designed with theta-sequence firing, while using sparse DG inputs. Navigation experiment demonstrates the model reproduces place field formation, remapping with reward changes, and orthogonalization of trajectories. The CA3 sequence generator helps the agent navigate more efficiently under sparse input conditions compared to standard RNN models.", "questions": "1. Is it not expected that a CA3 module with high-thresholded, 2.5% sparsity will outperform other RNNs with sparse input (as in Fig. 2C)? Previous work (Kumar et al. 2022 Cerebral Cortex) has shown improved learning with higher thresholding but only to a point; how does your contribution differ?\n\n2. Similarly, is it not trivial that different trajectories post-training will result in different CA3 sequences?\n\n3. Could a chaotic attractor or standard reservoir network (e.g. Kumar et al. 2022 Cerebral Cortex) achieve similar benefits as your sequence generator?\n\n6. To what extent do your representations align with the successor representation framework (Stachenfeld et al. 2017 Nature Neuro.), or are they more directly tied to reward/value learning signals (as in Kumar et al. 2025 ICML)?\n\nReferences:\n- Kumar et al. 2025 (https://icml.cc/virtual/2025/poster/46112)\n- Ito et al. 2015 (https://www.nature.com/articles/nature14396)\n- Pfeiffer & Foster 2013 (https://www.nature.com/articles/nature12112)\n- Gardner et al. 2018 (https://royalsocietypublishing.org/doi/full/10.1098/rspb.2018.1645)\n- Kumar et al. 2022 (https://doi.org/10.1093/cercor/bhab456)", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The architecture is constrained to reflect biological hippocampal anatomy, specifically the DG-to-CA3 connectivity and sparse input regimes.\n- The model captures some experimental observations including place field formation, population-level spatial encoding, and remapping to new rewards.\n- The baseline comparative analysis across different recurrent architectures suggests the functional significance of the sequence-generating module.", "weaknesses": "- Inconsistent performance: The CA3-based agent does not outperform LSTM or random RNNs in the dense input regime, and in some benchmarks, random RNNs do better than the CA3 module. I wonder about the generality and robustness of the sequence generation module. Is sequence generation necessarily a built in feature of the hippocampus or is it a consequence of something else? The current results do not support sequence generation in the hippocampus for faster learning.\n\n- Ambiguous role of RL signals: The connection between reinforcement learning algorithms (e.g., A2C/TD error) and the emergence of hippocampal representations is not clearly explored. How does learning a policy or value function specifically influence place fields emergence and remapping (Kumar et al. 2025 ICML)? Could we get the same phenomena if we do behavior cloning or other supervised learning regimes? I do not see the point of using RL in this study other than to model navigation behavior. \n\n- Insufficient clarity on planning: The authors do not analyze how the CA3 sequence generator supports planning, forward sweeps (Ito et al. 2015 Nature; Pfeiffer & Foster 2013 Nature), or predictive representations (Gardner et al. 2018). E.g. how do reward or environmental changes translate to different planning behaviors. Is your definition of a plan (rapid adaptation) different from policy learning (gradual learning)? \n\n- Comparative analysis is limited: There is limited discussion on the contribution of sequence generation versus input thresholding in the observed performance with sparse inputs. Why does the proposed model only perform better with input thresholding and worse with dense inputs? More rigorous ablation studies might be relevant to disentangle these factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wm4TUZcKGH", "reviewer": "Reviewer_EoBt", "summary": "This submission studies the computational role of hippocampal sequences by constructing a neural network model with recurrence and examining how it performs on spatial navigation tasks. The authors find that sequence length is short, the model does not learn the task well (and struggles to generalize). Increasing the sequence length improves performance. The authors' model outperformed several other ML models when the inputs were sparse, but not dense. Investigation of the model revealed that place cell like responses emerged, as did the decorrelation of inputs.", "questions": "1. How exactly is the CA3 recurrent model being implemented and used by the rest of the model?\n\n2. How do results showing a lack of theta oscillations in bats and birds line-up with the results the authors have shown?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Understanding hippocampal replay is an important open question and the authors tackle it with an interesting computational approach. \n\n2. The results demonstrating the interplay between sequence length and sparsity are interesting and will be of broad interest to computational neuroscientists. \n\n3. The authors have a detailed discussion section where they emphasize the potential impact their results could have on improving existing ML models. This will help broaden the impact of the work and help it fit more broadly in ICLR. \n\n4. The experimental design, with obstacles and repeated visual inputs, was nicely done and makes the work more aligned with actual ethologically relevant environments. \n\n5. The SI ablation study, where the weights were shuffled as opposed to just removed, was nice and convincingly shows spatial information is used.", "weaknesses": "1. The biggest weakness in my mind is that the explanation regarding the CA3 shift register was not so clear. In particular, I did not really understand: a) what it meant that each feature had a ``dedicated prewired sequence''; b) how the sequence was input to the decoder layers; c) what was learnable in the architecture. Later, in the Discussion, the authors mention that the sequence was a ``linear reservoir'', which I think clarifies some of these questions. But providing more details on this, in the section on the model, would be helpful. Perhaps including a figure demonstrating example sequences and a schematic highlighting how the sequence is passed to the next layer? There is also a minor typo in Eq. 2, where the number of 0s in J should be $L - R$, not $L - 1$. \n\n2. The authors mention in the abstract, introduction, and discussion section the remapping between different environments. I never saw that results mentioned in the results (the figure being alluded to only in the discussion). If that's an important result, it should be highlighted earlier in the results section. \n\n3. The writing in general felt terse. Not much detail was provided and many new paragraphs were used. This led to jumping between ideas/results and taking up space which then limited how much could be discussed. This was especially apparent in the Introduction, which did not feel very coherent or informative. \n\n4. Bats and some birds do not exhibit theta oscillations. Despite this, these animals are extraordinary spatial navigators. I don't think this diminishes the authors' work, but I do think it is: a) worth commenting on; b) worth discussing how and why a lack of theta can be reconciled with their model/results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)