A Model of Place Field Reorganization During Reward Maximization
Authors: M Ganesh Kumar
Blake Bordelon
Jacob A Zavatone-Veth
Cengiz Pehlevan
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement learning
Temporal Difference error
Hippocampus
Place cells
navigation
Primary area: applications->neuroscience_cognitive_science
TL;DR: Optimizing basis function parameters using the Temporal Difference error recapitulates several neural phenomena, and improves the speed and flexibility of policy learning

Abstract: When rodents learn to navigate in a novel environment, a high density of place fields emerges at reward locations, fields elongate against the trajectory, and individual fields change spatial selectivity while demonstrating stable behavior. Why place fields demonstrate these characteristic phenomena during learning remains elusive. We develop a normative framework using a reward maximization objective, whereby the temporal difference (TD) error drives place field reorganization to improve policy learning. Place fields are modeled using Gaussian radial basis functions to represent states in an environment, and directly synapse to an actor-critic for policy learning. Each field's amplitude, center, and width, as well as downstream weights, are updated online at each time step to maximize rewards. We demonstrate that this framework unifies three disparate phenomena observed in navigation experiments. Furthermore, we show that these place field phenomena improve policy convergence when learning to navigate to a single target and relearning multiple new targets. To conclude, we develop a simple normative model that recapitulates several aspects of hippocampal place field learning dynamics and unifies mechanisms to offer testable predictions for future experiments.

A Model of Place Field Reorganization During Reward Maximization
M Ganesh Kumar" 123
Blake Bordelon 123
Jacob A. Zavatone-Veth 24 Cengiz Pehlevan 123
Abstract
spatial receptive field such that the population activity can describe an animal's allocentric position within the environ- ment (Moser et al., 2015). Ablation studies demonstrate that the hippocampal representation is useful for learning to navigate to new targets (Morris et al., 1982; Packard & McGaugh, 1996; Steele & Morris, 1999). Importantl, each field's is spatial selectivity evolves with experience in new environment before stabilizing in the later stages of learning (Frank et al., 2004). Specifically, high density of place fields emerge at reward locations (Gauthier & Tank, 2018; Lee et al., 2020; Sosa et al., 2023), place fields elon- gate hackward against the trajectory (Mehta et al., 1997; Priestley et al., 2022), and individual field's spatial selec- tivity continues to change or "drift" even when animals demonstrate stable behavior (Geva et al., 2023; Krishnan & Sheffield, 2023; Kentros et al., 2004; Mankin et al., 2012; Ziv et al., 2013). Although disparate mechanisms have been proposed to model these phenomena, framework that can unify these phenomena and clarify their computational role remains elusive. Here, we propose a single normative model for spatial representation learning, based on the hip- pocampal CAI given its role in representing salient spatial information (Dong et al., 2021; Dupret al., 2010). Our key contributions are:
When rodents Iearn to navigate in novel envi- ronment, high density of place fields emerges at reward locations, fields elongate against the trajectory, and individual fields change spatial selectivity while demonstrating stable behavior. Why place fields demonstrate these characteristic phenomena during learning remains elusive. We develop a normative framework using reward maximization objective, whereby the temporal difference (TD) eтror drives place field reorgani- zation to improve policy learning. Place fields are modeled using Gaussian radial basis func- tions represent states in an environment, and directly synapse an actor-critic for policy learn- ing. Each field's amplitude, center, and width, well as downstream weights, are updated online at each time step to maximize rewards. We demon- strate that this framework unifies three disparate phenomena observed in navigation experiments. Furthermore, we show that these place field phe- nomena improve policy convergence when learn- ing navigate to single target and relearning multiple new targets. To conclude, we develop a simple normative model that recapitulates several aspects of hippocampal place field learning dy- namics and unifies mechanisms to offer testable predictions for future experiments.
We develop a two-layered reinforcement learning model to study spatial representation learning by place fields 1A). The first layer is a population of Gaussian radial basis functions that transform continuous spatial informa- tion into a relevant representational substrate or "state"", which feed into an actor-critic network in the second layer that uses these representations to learn actions that max- imize cumulative discounted reward. Besides the actor and critic weights, each place field's amplitude, center and width is optimized by the Temporal Difference error.
1. Introduction
A place field is canonically described as a localized region in an environment where a hippocampal neuron's 's firing rate maximal and robust across trials (O' Keefe & Dostrovsky, 1971; Oeeeee, 1978). Classically, each neuron has a unique
Our model recapitulates three experimentally-obsere neural phenomena during task learning: (1) the emergence of high place field density at rewards, (2) elongation of fields against the trajectory, and (3) drifting fields that do not affect task performance.
'The John A. Paulson School of Engincering and Applicd Sci- ences, Harvard University 'Center for Brain Science, Harvard Uni- versity 'The Kempner Institute for the Study of Natural and Artifi- cial Intelligence, Harvard University 'Society of Fellows, Harvard University. Correspondence to: M Ganesh Kumar <mganeshku- maa@scas. scas add>..
We analyze the factors that influence these representa- tional changes: a low number of fields drives greater spatial representation learning, the mean population fir- ing rate reflects the value of that location, and increasing
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Model of Place Field Reorganization During Reward Maximization
noise magnitude during field parameter updates causes monotonic decrease in population vector correlation but non-monotonic change in behavior.
Fields learn to encode future occupancy. As animals traverse 1D track, most CA1 fields increase in size and their center of mass shift backwards against the trajectory of motion (Mehta et al., 1997; Frank et al., 2004; Priest- ley et al., 2022). A proposal for this behavior is that fields initially coding for location 2; are learning to also encode the previous location 11 hence predictively coding for location occupancy p(xx+112z+) (Mehta et al., 2000; Stachen- feld et al., 2017). While algorithms such as the successor representation (Dayan, 1993) learn to predict the transition structure (Gershman, 2018), the representation is dependent on predefined navigation policy. Hence, a complete norma- tive argument- inguuent-including policy learning- for why fields exhibit this behavior is still lacking.
We demonstrate that optimizing place field widths and amplitudes enhances reward maximization and policy con- vergence. However, field parameter optimization alone is insufficient for learning to navigate to new targets. Intro- ducing noisy field parameter updates improves new target learning, suggesting functional role for noise.
2. Related Works
Anatomically constrained architecture for navigation. Learning to navigate involves the hippocampus encoding spatial information and its strong glutamatergic projections to the striatum (Lisman & Grace, 2005; Floresco et al., 2001). The ventral and dorsal regions of the striatum are associated with value estimation and stimulus-response as- sociations, functioning similarly to critic and actor, respectively (Niv, 2009; Joel et al., 2002; Houk et al., 1994). Additionally, dopamine neurons in the Ventral Tegmental Area influence plasticity in the striatal synapses (Reynolds et al., 2001; Russo & Nestler, 2013). This anatomical insight has led to the design of a biologically plausible navigation model, where place fields connect directly to an actor-critic framework, and synapses are modulated by the TD error (Arleo& & Gerstner, 2000; Foster et al., 2000; Frémaux et al., 2013; Brown & Sharp, 1995; Kumar et al., 2022). Recent evidence shows direct dopaminergic projections to the hip- pocampus to modulate place cell activity, strengthening the case for navigation models with adaptive place fields (Palacios- Filardo & Mellor, 2019; Krishnan et al., 2022: Kempadoo et al., 2016; Sayegh et al., 2024). How upstream information from the entorhinal cortex influences place field representations for policy learning needs clarity (Fiete et al., 2008; Bush et al., 2015). As new experiments challenge the canonical definition that a place cell only has one place field (Eliav et al., 2021), we study spatial representational learning using Gaussian place fields, instead of place cells.
Fields drift during stable behavior. After animals reach certain performance criterion in navigating to reward location, the spatial selectivity of individual place fields changes across days, even though animals exhibit stable behavior (Kentros et al., 2004; Mankin et al., 2012; Ziv et al., 2013; Geva et al., 2023; de Snoo et al., 2023). A proposal is that these fields continue to drift within a degenerate solution space while the overall representational manifold or the chosen performance metric remains stable (Qin et al., 2023; Pashakhanloo & Koulakov, 2023; Masset et al., 2022; Kappel et al., 2015; Rokni et al., 2007). Another proposal is that compensatory synaptic plasticity adjusts the readout to maintain stable decoding over time (Rule et al., 2020; Rule & O' L'Larry 2022). However, a model that demonstrates stable navigation learning behavior with drifting fields is absent, and the functional role of drift remains unclear.
3. Task and Model setup
Most navigational experiments involve an animal moving from a start location to target location to receive a reward, either in a one-dimensional (ID) track or a two-dimensional (2D) arena. Similarly, our agents receive their true position at every time step (t) described by the variable (scalar xa in ID, vector æ; in 2D), and have to learn a policy (r) that specifies the actions to take (ge) to move from a start location (e.g. Istart = -0.75, Fig. 1A green dash) to a target with reward values following a Gaussian distribution (x, = 0.5, a, = 0.05, Fig. 1A red area). The agent outputs one-hot vector 9r (left or right in ID and left, right, up or down in 2D). which causes its motion to be discrete, similar to trajectory in a grid world. To model smooth trajectories in continuous space as an animal's behavior (Foster et al., 2000; Frémaux et al., 2013; Kumar et al., 2022; 2024), we use a low-pass filter to smooth g, using a constant
Field density increases near reward locations. Density traditionally refers to the number of field centers of mass in a location. However, we also consider changes in the mean population firing rate, which includes variations in each field's width and amplitude. As animals learn to navigate towards a reward, a high density of place fields emerge at reward locations (Gauthier & Tank, 2018; Lee et al., 2020: Sosa et al., 2023). Reward location based reorganization was observed in hippocampal CA1 and not in CA3 (Dupret et al., 2010). Interestingly, recent study showed that place fields initially coding the reward location shifted backwards against the trajectory causing a decrease in reward coding fields, suggesting of a representation predictively coding for reward (Yaghoubi et al., 2024).
aenv = 0.2 after scaling for maximum displacement using
mmar 0.1:
I++1 = It + Dt 0t+1 = (1 - Qene De + OennVmmart (1)
2

---

A Model of Place Field Reorganization During Reward Maximization
To track an agent's reward maximization performance dur- ing navigational learning compute the cumulative dis- counted reward (G -i L r=+1++*) for the entire trajectory for each trial using y = 0.9 as the discount factor, which is similar to tracking the cumulative reward. The agent will continue to receive rewards by staying at the tar- get. Hence, the trial is terminated when either the maximum trial time reached Tmoz or when the total reward achieved Ei Pt reaches& threshold Rmaz-
probability distribution P as in Foster et al. (2000), making this policy stochastic. wy and W were initialized by sam- pling from a normal distribution NO 10- with other initializations (uniform or all zeros) reproducing similar results.
3.3. Reward Maximization Learning Objective
The objective of our agent is to learn policy π that maxi- mize the expected cumulative discounted reward J [G = E, E2, r're] over all trajectories т. To achieve this goal, our agent uses the standard actor-critic algorithm using the temporal difference residual (refer to App. A):
3.1. Place fields as spatial features
The agent represents space through N place fields, which have spatial selectivity modeled as simple Gaussian bumps: (2)
(#+)) = af exp(-I|<; - Au >
(5) which reduces variance and speeds up policy convergence (Sutton & Barto, 2018; Dayan & Abbott, 2005; Wang et al., 2018; Schulman al., 2017: Mnih et al., 2016). The TD residual is also biologically relevant, the responses of midbrain dopamine neurons resemble TD reward prediction error (Schultz et al., 1997; Starkweather & Uchida, 2021; Gershman & Uchida, 2019; Amo et al., 2022: Montague et al., 1996). The actor learns a reward maximizing policy by ascending the gradient of the policy log likelihood, mod- ulated by the TD residual. To accurately estimate the value function and critique policy learning using the TD error, the critic minimizes the squared TD error C E, [.)
JTD E, [ra + yo(2)+1) v
where a, x and set the amplitude, center, and width re- spectively. Two types of place field distributions were ini- tialized to tile the environment: (1) Homogeneous popula- tion with constant values for amplitudes ag = 0.5, widths 0; = 0.1, and centers uniformly tiling the environment A = -1,,.., 1] (Foster et al., 2000; Frémaux et al., 2013; Kumar et al., 2022; 2024). (2) Heterogeneous population with amplitudes, widths and centers drawn from uniform random distributions between [0, 1], [10 5,055,,0 0.11. [-1, 1] re- spectively. These ranges are consistent with experimental data where place fields were 20 cm to 50 cm wide in small to medium environments (Lee et al., 2020; Frank et al., 2004; Mchta et al., 1997; Sosa et al., 2023). 2D place fields have scalar amplitudes, two dimensional vectors for center, and square covariance matrices for the width (Menache al., 2005). Refer to App. A for further details.
As our agent uses a single population of place fields, these fields must learn spatial features that enhance both policy and value learning. The field parameters 0 = {a, X, o) and the policy weights W*, w" are updated by gradient ascent using joint objective modified from Wang et al. (2018):
3.2. Policy learning using an Actor-Critic
Vo,w n0 JTD =E | (Vo,w log ( (z2)
To model an animal's trial-and-error based learning behav- ior, adopt the reinforcement learning framework, specif- ically the actor-critic (Arleo & Gerstner, 2000; Brown & Sharp, 1995; Foster et al., 2000; Frémaux et al., 2013; Ku- mar et al., 2022; 2024). The critic linearly weights place field activity using a vector w to estimate the value of the current location:
+ (V . de],
(6)
with V Ww parameter gradients online, and provide the explicit update equations for each parameter in App. A. The learning rates for the actor-critic and place field parameters can be the same (Fig. S13). For theoretical analysis, we assume a separation of timescales between learning the actor-critic weights and updating place field parameters (App. B).
and Vw-L = O. We estimate all
x(x)==;; - N w (aa))
(3)
The value of a location corresponds to the expected cumu- lative discounted reward for that location. The actor has M units, each specifying movement direction. In the ID and 2D environments, M = 2 and M 4 respectively to code for opposing directions in each dimension e.g. left versus right and up versus down. Each actor unit aj linearly weights the place field activity such that the matrix W computes the preference for moving in the j-th direction: (x)) = E. (1:)) with the probability of taking an action computed using a softmax. A one-hot vector 9j is sampled from the action
4. Results
4.1. High field density emerges at the reward location
We first examine the neural phenomenon where a high field density emerges at the reward location over the course of learning. Field density is defined by the distribution of place field center of mass (COM) (Lee et al., 2020), which we estimate using Gaussian kernel smoothing.
P3 = Σxd exp(ax)
exp(a,)
(4)
Figure 1B shows how our agent's 's track occupancy (p(x)),
3

---

A Model of Place Field Reorganization During Reward Maximization
A
B
AA: T=50
AA: T=1200
AA: T=12000
M, da, Ao
8
[Ml
Actor
Critic
P
PR
V
D1500
0.538
T- 100, G.R.
T- 10000, G.R. 202.-227-
50000, G.R. -24
41, Aa, Ao
Goal Rep. (<00.2)
25
20-
D
15
0.4
50000
d(x)) -dx)
E
F
0.2
0.005
0.006
sim
Theory
Rasx R, 0.05
Rawx 5,R 0.05
0.000
Sin
0.004
0.0
Raw 9. 0.05
Raw 5, Raze 0.025
-0.05
Theory
0.002
Rasv R. 0.1
0.000
10a
10
10a
10a
10
10a
10c
N
T
T
Figure 1. Fields shift towards and amplify locations with high value. (A) The task is to navigate from the start (green dash) to the target (red area) to receive rewards. The agent has N place fields (blue) which synapse to an actor (red) and critic (green). The TD error ó modulates all parameter updates. (B) Example change in field centers for an agent on ID track when only optimizing field centers (AA). Initially (T' = 50), the agent spends high proportion of time (pRar (z), black) at the start location while place fields are uniformly distributed. After some learning trials (T = 1200), the agent still spends high proportion time the start high field density (d(z)) and mean firing rate (zz)) emerges at the reward location. As learning proceeds, the agent spends higher amount of time at the reward location but now field density and mean firing rate starts to increase at the start location (T = 12000). (Right) A high field density and mean firing rate emerge at the reward and start locations for 50 agents that were randomly initialized with heterogencous populations and when all parameters (X, a, a, w", W are updated using the same learning rate (1) = 0.0005). (C) Example change in field centers for an agent in 2D arena with start (green), target (red), and obstacle (gray). In the early learning phase, field centers (black dots) near the target shift closer, causing a high number of field centers (goal representation) to aggregate at the reward (10 agents with random seeds, right). In the later learning phase, field centers align along the trajectory. The start, reward locations and radius for goal representation (G.R.) are marked by green, red and blue circles. (D) When initialized with a heterogeneous field population, the enhancement of average field density (d(x)) at the reward location x, compared to non-reward location x' decreases as the number of fields increases. This density increases when the reward magnitude (Rmax) ) increases, and reward location's 's size (Raize) decreases. (E) Example field dynamics when an agent (N = 512) navigates a track. Fields initialized before (A, = 0.5, blue) and after (A, 0.6, orange) the target move forward and backward respectively, increasing the density near the target. (F) Fields closest to the reward (A; = 0.5: blue) show rapid amplification compared to other fields (A+ = -0.75, 0.0 green and purple). The first order perturbative prediction (theory) provides a good approximation. Shaded area and error bars are 95% CI over 50 seeds.
field density (d(x)), mean firing rate (f())), and individual field's spatial selectivity (o(z)) change when learning to navigate in a 1D track from the start Isttrt = -0.75 to the target at I,- = 0.5, when only optimizing place field centers (AA). In the early stages of learning (T = 50 to T = 1200), the agent with a homogeneous place field population spends a higher proportion of time at the start location with only sporadic exploration towards the reward. Despite this be- havior, high field density and mean firing rate emerges at the target after a few trials. Individual fields at the reward lo- cation shift closer to the target, as seen in Sosa et al. (2023), in contrast to fields at non-rewarded locations. As learning progresses and the agent spends a higher proportion of time at the reward (T 12000), field density and mean firing
rate at the start location also begins to rise, replicating the two-peaked field distribution in Gauthier & Tank (2018), with some fields shifting backwards from the reward loca- tion towards the start as in Yaghoubi et al. (2024). Figure IB (right) also shows that a high field density at the reward location followed by the start location robustly emerges in agents initialized with a heterogeneous place field popu- lation, and when all the field parameters (A, a, a)) as well as the actor (W" and critic (w") parameters are optimized using the same learning rate (17 = 0.0005).
Similar field dynamics are observed in 2D arena with an obstacle where agents have to navigate to a target from a starting location (Fig. 1C). When optimizing all the place field parameters in homogeneous population, a high field
4

---

A Model of Place Field Reorganization During Reward Maximization
density rapidly emerges at the reward location to increase goal representation (number of COM within 0.25 unit radius from target center) as seen in (Dupret al., 2010), followed by gradual reorganization of field density backward along the agent's trajectory.
the reward location to be amplified first, followed by the start location (Fig. 1F, Fig. S1). Therefore, these approximations predict fields shifting to the start and reward location with field amplification at locations of high value.
4.2. Fields clongate against the movement direction
Increasing the number of fields in a heterogeneous place field population reduced the average density and mean firing rate (Fig. ID, Fig. SI) that emerges near the reward location. This is because the number of fields increase, the agent goes into weak feature learning regime (Fig. S4) in which feature learning does not contribute to additional ad advantage. Conversely, the density and mean firing rate are proportional to the reward magnitude, and inversely proportional to the reward location width as a narrower target might require higher discriminability for the agent to maximize rewards.
We now turn to the next phenomenon where place field sizes increase and their centers shift backward against the move- ment direction as animals learn to navigate. This behavior suggests predictive coding for future occupancy, which can be learned through Hebbian association of fields (Mehta et al., 2000), or through the successor representation (SR) algorithm, which minimizes state prediction error for each place field to learn the transition probabilities (Stachenfeld et al., 2017). Here, we show that our proposed Reward Maximizing (RM) agent also recapitulates field elongation in a 1D track and 2D arena around an obstacle.
To understand why place fields exhibit these dynamics, we perform perturbative approximation to the place field pa- rameter changes under TD learning updates (Menache et al., 2005; Bordelon et al., 2024). In this approximation, we assume that the change to the field parameters is small, con- trolled by the number of fields, and by the large separation between learning rates. Focusing on the place field centers, we derive in App. B the approximation where 77λ 0.0001 is the learning rate for the field centers and 0.01 is the learning rate for the critic weights such that 17 << ry:
For comparison, we developed two agents: A) an SR agent that learns the transition probabilities in parallel to policy learning (Fig. 2A). The SR agent has a similar architecture to our (RM) agent (Fig. IA), with two key differences: 1) It has one set of place fields with fixed parameters, and only the synapses from these place fields to the actor-critic are optimized for policy learning. 2) There is a separate set of N successor place fields (1r) that receive input from the fixed place fields via synapses U which are optimized using the SR algorithm (App. C). We compare the learned succes- sor place fields to the learned place fields in our RM model, referring to them henceforth as place fields. B) Metric Rep- resentation (MR) agent (Fig. 2B) that estimates its current coordinates in an environment (21). This metric representa- tion enables navigation to recalled targets by vector subtrac- tion (Foster et al., 2000; Kumar et al., 2024). The coordinate readout weights and place field parameters are updated by gradient descent minimize the path antegration-derived TD error L R ] (#++1 - (2+ + a)) while only the actor and critic readout weights are updated to learn a policy, This objective allows place fields Φ(x:) to reorganize even in the absence of rewards. See App. D for the derivation.
(t) - (0) ~ 22 (3++
X- x-) + P. -. 2 (t). x
(7)
of
of
Under this approximation, each field's center shifts propor- tionally to the squared magnitude of the critic weights (w.), implying that fields at locations with a high value will shift at a faster rate compared to locations with a low value. In addition to the value of location, the agent's 's start location (modeled as a Gaussian with mean fgg = -0.75 and spread σ2) and the mean field center location X over time under the policy influence each field's displacement. As the reward location is visited frequently, we expect Ā zo 0.5. As the term within the square bracket changes sign depending on the field location, only the fields near the reward location will shift towards the reward, while the rest of the fields will move towards the start location. Due to these influences, the field density at the reward location will increase first followed by gradual increase in start location (Fig. IB.E). Additional approximations are needed to model the agent's trajectory and improve the simulation-theory fit for place field centers (See App. B).
All three agents (SR, MR, RM) recapitulate the phenomena seen in (Mehta et al., 1997): on average, place fields increase in size over learning (Fig. 2C), and the center of mass (COM) shifts backwards from their initialized positions (Fig. 2D). However, the place field dynamics evolve differently. All agents initially spend a high proportion of time at the start location and gradually learn a policy to spend a higher proportion of time the reward (Fig. 2E, Fig. S5E).
A similar perturbative analysis for amplitude yields: (t) (0) = 2 no
The SR, by design, tracks the transition probabilities of the agent's policy. Consequently, the SR population mean firing rate (x) closely aligns with the agent's probability of being in location PSR, showing a high positive correlation (Fig. 2F, blue). Since the MR representation is modulated
(8) when Ta < 1 and where Ta = 0.0001 is the learning rate for the a parameters. Thus place fields will be amplified at a rate similar to learning the value function, causing fields at
1 (t),
5

---

A Model of Place Field Reorganization During Reward Maximization
T=1000
T=3000
T=10000
T=50000
Askit
Fosiion
Foudd
MMMAAMM
Pee
1.0
Norm sizz
1.0
Norm A COM
0.5
0.5
SR
0.0
MR
I
0.0
102
10
10
10'
10
10
fan Page faM
far fam fse fune
fat
102
103 T
104
10a
10a
10"
T
Figure 2. Different objectives cause fields to elongate against the trajectory, but with different dynamics. (A) Successor Representa- tion agent architecture with successor fields (t)) a separate set of features that learned in parallel to navigational learning. (B) Metric Representation agent architecture where the path-integration derived temporal difference error drives place field reorganization, while policy learning occurs parallel. (C-D) The Reward Maximization (RM), Successor Representation (SR) and Metric Representation (MR) algorithms cause (C) field sizes to increase and (D) center of mass to shift backwards against the trajectory in a 1D track. Field changes were normalized separately to be between to visualization. (E) agents initially spend a high proportion time at the start location, and later learn to dwell at the target (black). Individual SR fields and mean firing rate (red) closely track the proportion of time the agent spends a location (top). MR fields reorganize only at the start location (middle). Conversely, individual RM fields and mean firing rate show inverse relationship against the proportion of time the RM agent spends at location in the early learning phase, start align in later phases (hottom). (F) SR agents show a consistently high, positive correlation (blue) between mean firing rate and proportion of time spent in location. MR agents' show non- monotonic increase correlation (green). Conversely, the RM agents' mean firing rate and time spent at a location become anti-come mnti-correlated before becoming positively correlated (orange). (G) The SR and RM mean firing rates (blue) become anti-correlated before becoming positively correlated at the later learning phase, while the SR and MR fields align momentarily before de-correlating (orange), and the RM and MR fields become anti-correlated (green). (H) Example change field selectivity by SR (top), MR (middle), and RM (bottom) agents in 2D arena with an obstacle. The RM agent's field elongation more pronounced than the SR and MR agents. Summary statistics in Fig. S6. Shaded area is 95% CI over 10 seeds.
by the agent's displacement d2, fields reorganize more at the start location since displacement is nonzero, causing higher mean firing rate. Conversely, displacement becomes zero at the reward location as the agent comes to a stop to maximize rewards, causing low field reorganization at the reward. Hence, MR fields fex (I) become positively correlated with par at the start location, but do not fully align with the agent's time spent the reward location (Fig. 2F, green). Conversely, during early learning, the RM agent exhibits a high population mean firing rate JØRM (x) at the reward location, which contrasts sharply with the proportion of time spent at that location, leading to a highly negative correlation between føns (z) and PRAS (Fig. 2F, orange). Interestingly, in the later phase of learning, f (I) and PRM become positively correlated.
(Fig. 2G, blue). Conversely, the mean firing rate correlation decreases monotonically towards zero for the MR and RM agents (Fig. 2G, green), while the correlation between SR and MR increases due to the alignment at the start location in the early learning phase before becoming uncorrelated in the later learning phase. A similar change in correlation is observed when comparing the individual field selectivity, and the spatial representation similarity matrix (Fig. S5FGG), Hence, while all three algorithms demonstrate qualitatively similar neural phenomenon, the dynamics of learning these representations are different, with SR and RM agents even- tually learning similar spatial representations.
In a 2D arena with an obstacle, the three agents show field elongation against the movement direction (Fig. 2H) while also accounting for the blockage of path by the obstacle. The RM agent shows significantly larger elongation of fields to span the entire corridor while the elongation of fields by SR is subtle and field elongation by MR is more pronounced
The mean firing rates learned by the SR and RM agents be- come negatively correlated during the early learning phase but become positively correlated at the later learning phase
6

---

A Model of Place Field Reorganization During Reward Maximization
only at the start location. See Fig. S6 for summary statistics.
4.3. Stable navigation behavior with drifting fields
A
T=25000
T= 75000
T= 125000
T= 150000
T= 195000
Fledd serted by Pele zotton by
The third phenomena that the model captures has been de- scribed as representational drift, where the agent demon- strates stable behavior but the spatial selectivity of individual place fields change over time (Fig. 3A. Fig. S8G), as seen in Ziv et al. (2013). Although our agent uses stochastic policy, both the navigation behavior (Fig. 3E, blue) and the population vector (PV) correlation (Fig. 3C, blue) are extremely stable.
T- 195300 7 225000
Similanity Matrie
To drive larger variability in the representation, we intro- duced Gaussian noise (çx) of varying magnitudes during place field parameter updates at every time step (0++1 0, + Et, see App. E). Increasing the noise magnitude led to a faster decrease in PV correlation but also disrupted agents' policy convergence for magnitudes greater than 10-3 (Fig. 3E purple, Fig. S7). Hence, we consider the noise magni- tudes between 10-4 and 10-3 to be relevant. As the noise magnitude increases, agent's 's reward maximization behavior remains stable (Fig. 3E) while the PV correlation decreases rapidly (Fig. 3C). This demonstrates that agents can opti- mize their policies to maintain stable behavior even though individual spatial selectivity is changing. Interestingly, the spatial representation similarity matrix remains more stable than PV correlation (Fig. 3B), even at higher noise mag- nitudes (Fig. 3D), and when the agents are not explicitly optimizing for representational similarity (Qin et al., 2023).
1.0
1.0
Prate - 0.0
0. 0.0001
2 0.5
3 0.5
0.00025
Prase 0.0005
B1 0,001
0.0
2.5x10
10
2x10
2.5x10
10a
2x10
E
Onois 0.0
noise 0.0001
naisc 0.001
50
50
10a
2x10s
105
2x103
10
2x10
F
0.2
R-0.541
Re- 256,
R-O, 188.
Varth)
px 10
0.2
p p=100
p 10
0.1
0.1
0.0
Unlike noisy field parameter updates, adding noise to the actor and critic synapses (w0+1 w + & and W = WT + + caused the reward maximization behavior, repre- sentation similarity and PV correlation to change at similar rates (Fig. S7), which is not as consistent with experiments (See Fig. S9 for comparisons to data).
Mean Amplitude
Figure 3. Stable representation similarity and anchor fields fa- cilitate consistent behavior. (A) Injecting Gaussian noise with magnitude dnoise 0.0001 into field parameters causes indi- vidual field's spatial selectivity to change across trials while (B) the representation similarity matrix (dot product of population activity) remains stable. (C) Injecting higher noise magnitudes (Tnoise 0.0001, 0.001) leads to a faster decrease in popu- lation vector correlation (Rpv) across trials while (D) the sim- ilarity matrix correlation (RRs) decreases at slower rate. (E) Agents' reward maximization performance (G) remains fairly sta- ble even when the noise magnitude increases. However, beyond Anoise = 0.001, performance hecomes highly unstable. Black dash indicates the trial at which PV and similarity matrix correla- tion was measured from. (F) Normalized variance in field parame- ters (0 (a, A, a)) between trials 25,000 to 200,000 quantifies the change in individual place fields spatial selectivity. With no noise (blue) or a larger noise magnitude (Tvosse = 0.001), fields with larger amplitude undergo greater change in its parameters. When Gnoise = 0.0001, we see the opposite trend, where fields with larger amplitude more stable than fields with smaller amplitude, suggesting fields with larger amplitudes acting as an- chored representations in an environment. Refer to Fig. S8 for other anoise values. Shaded area is 95% CI over 10 seeds.
We quantified this drifting behavior at the level of in- dividual neurons by summing the nonmalized (between [0, I)) variance in each field's parameters (E Var(0) = Var(ā) + Vār(a) + Vār(o)) across learning trials, and comparing this against the mean amplitudes for each field. When no Gaussian noise is added (Fig. 3F, blue), fields with higher mean amplitude showed higher variance in its parameters, which is expected since fields with higher amplitude are more likely to be involved in policy learn- ing. Conversely, with a small Gaussian noise, we see the opposite trend where fields with smaller mean amplitude showed a higher variance in parameters while fields with a higher mean amplitude were more stable (Fig. 3F, orange). At smaller noise magnitudes, there is strong positive corre- lation between higher amplitude fields and the magnitude of actor and critic readout weights (Fig, S8). This suggests that high-amplitude fields are more involved in policy learning and thus more stable, whereas less important fields can alter
7

---

A Model of Place Field Reorganization During Reward Maximization
their spatial selectivity to maintain stable behavior (Fig. 3E, orange), consistent with Qin et al. (2023). Increasing the noise magnitude beyond this magnitude causes place fields with large magnitudes and more involved in policy learning to drift more (Fig. 3P, purple), causing unstable behavior (Fig. 3E, purple).
vergence even the number of fields increased (Fig. 4B), Optimizing place field widths (α) contributed to the greatest improvement in maximum reward and largest decrease in the number of trials needed for policy convergence (Fig, 4A-B). Optimizing place field amplitudes (a) contributed to the next most significant improvement (Fig. 4A-B). In- terestingly, place field center (A) optimization did not con- tribute to a significant improvement in performance, and in fact caused a decrease in reward maximization perfor- mance and speed of policy convergence when optimized together with the amplitude parameter. Hence, optimizing field widths followed by amplitudes and lastly centers signif- icantly improved agent's 's reward maximization performance and increased the speed of policy convergence.
4.4. Field reorganization improves policy convergence
A
B
Fixed
40
40000
Acc
20
20000
AA, da
AA bi
AJ,
20000
10
10a
10
Optimizing field parameters using the auxiliary metric rep- resentation (MR) objective, inspired by Fang & Stachenfeld (2023), marginally improved policy learning (Fig. S15), However, as the number of place fields increase (Fig. 4B), the computational advantage afforded by place field op- timization extinguishes. Nevertheless, optimizing all the parameters small number of fields, e.g. 8, leads to a similar rate of policy convergence than with larger number of randomly initialized fields e.g. 128, which hints that representation flexibility could allow efficient learning in systems with few neurons.
C
x, 1,=00 0.5
X, - 0.0
x: - 0.75
x- 0.25
X; D.5
Guee
Gone 0.0001
Codaa 0. 0.00025
0. 0.0005
0.001
0-
3x10*
10a
1.2x110P 2x10*
2.3020
Figure 4. Field reorganization and noisy updates improve tar- get learning, (A) Optimizing all three field parameters, amplitude, width and center of randomly distributed fields allowed agents (N = 16, a = 0.1) to attain the highest cumulative discounted reward (G), while fields with fixed field paramcters attained the lowest. (B) Optimizing place field widths (o)), followed by field am- plitudes (a) and lastly field centers (A) caused the biggest decrease the number of trials needed for policy convergence (Ta245. at- tain running average of G = 45 over 300 trials). As the number of fields increased, the number of trials needed for policy conver- gence decreased and the computational advantage afforded by field optimization extinguished. (C) Agents need to navigate to a target that changed after 50,000 trials 2, {0.5, 0.0, 0.75, -0.25, 0.5). Without noisy field parameter updates, agents (N = 128, a = 0.1) struggled to learn new targets (blue, aaoise 0.0). Field updates with different noise magnitudes influenced the policy convergence speed and maximum cumulative reward for subsequent targets, with noise = 0.0005 (red) demonstrating the highest improve- ment. Shaded area is 95% CI over 50 seeds.
We now turn to the influence of noisy fields when learn- ing to navigate to new targets, inspired by Dohare et al. (2024). Agents now have to navigate from the same start location to a target that repeatedly changes location. Al- though all agents learned to navigate to the first (x, 0.5) and the second (x, 0.0) targets equally well, agents with- out noisy field updates struggled to learn the next three targets, and achieved a lower average cumulative reward (Figure 4C, blue). Increasing the noise magnitude led to a monotonic improvement in new target learning. Some place fields coding for the initial reward location shifted to code for the new reward location (Fig. S3), replicating the behavior of place cells that selectively code for rewards (Gauthier & Tank, 2018; Sosa et al., 2023). This behav- ior was suppressed in agents without noisy field parameter updates. However, noise magnitudes beyond a threshold (anoise = 0.001) caused average cumulative reward to de- crease. These results suggests that there is functional role for noise, especially for new target learning. We see a simi- lar improvement in reward maximization performance with noisy field updates in 2D arena with an obstacle when we either change the target or the obstacle location (Fig. S12).
As the reward-maximizie reward-maximizing model recapitulates experimentaaly--oserreed changes in place fields, it is natural to ask what computational advantage representa- tion learning might offer during reinforcement learning. To probe the contributions of each field parameter to policy learning, we perform ablation experiments. These ablations are particularly important due to the parameter degeneracies in the model: one can trade off the place field amplitudes and the critic and actor weights.
5. Discussion
We first considered the task of navigating to single fixed target. Agents with fixed place fields attained the lowest nav- igational performance with cumulative reward G platcauing at G = =3 (Fig. 4A), and showed the slowest policy con-
We present a two-layer navigation model which uses tun- able place fields feature inputs to actor and a critic for policy learning. The noisy parameters of the place fields
8

---

A Model of Place Field Reorganization During Reward Maximization
and the policy and value function learn to maximize rewards using the temporal difference (TD) error. Our simple rein- forcement learning model reproduces three experimentally- observed neural phenomena: (I) the emergence of high place field density at rewards, (2) enlargement of fields against the trajectory, and (3) drifting fields without influenc- ing task performance. We analyzed the model to understand how the TD error, number of place fields and noise mag- nitudes influenced place field representations. Lastly, we demonstrate that learning place field representations with noisy field parameters improves the rate of policy conver- gence when learning single and multiple targets.
drift rates needs to be explored. The current model provides starting point for this investigation.
The proposed model is not without limitations. First, we modeled single peaked place fields instead of the complex representations resulting from single "place" cells, which can be multi-field and multi-scale. Nevertheless, the pro- posed online reinforcement learning framework is general enough to accommodate other models of place cell descrip- tion (Mainali et al., 2024; Sorscher et al., 2023) e.g. Fig. S14, and can be extended to study representation learning in other brain regions e.g. medial entorhinal (Boccara et al., 2019; Wen et al., 2024) or posterior parietal (Suhaimi et al.,
Our goal was not to replicate every known mechanistic detail about place fields, but come up with a minimal biolog- ically grounded model that captures as many phenomena as possible. This required deliberate decisions omit certain granularities. While this necessitates some disconnect with neural mechanisms, it allows parsimony (Van Vreeswijk & Sompolinsky, 1996; Burak & Fiete, 2009; Hopfield, 1982) and interpretabiliity (Bordelon et al., 2024) to make experi- mentally testable predictions for neural systems (Montague et al., 1996; Schultz et al., 1997; Kumar et al., 2024). For instance, our model gives an alternative normative account for field elongation against the trajectory, which can be con- trasted with the successor representation algorithm (Raju et al., 2024; Kumar et al., 2024). As field dynamics are different in these two models, they could be distinguished by experiments that track fields over the full course of learn- ing (Fig. 2C-E, Fig. S6). Furthermore, place field width and amplitude optimization increases maximum cumulative reward and accelerates policy convergence (Fig. 4A-B).
2022) cortex.
Next, place field parameters are optimized by backpropa- gating the temporal difference error through the actor and critic components (Fig. S15). Since the motivation was to develop a normative model whose objective was to maxi- mize rewards, this was a reasonable starting point. However, this model must be extended using biologically-plane plauusbbe learning rules (Miconi, 2017; Murray, 2019; Lillicrap et al., 2016; Nøkland, 2016; Overwiening et al., 2025) before it can in any way be considered mechanistic (Lee et al., 2024; Starkweather & Uchida, 2021; Krishnan et al., 2022; Kem- padoo et al., 2016; Edelmann & Lessmann, 2018).
Although we explored a simple non-rewardddeeeenden ob- jective to drive place field reorganization, extending the model to other auxiliary objectives (Low et al., 2018; Scha- effer et al., 2022) to understand their influence in represen- tation learning for policy learning is the next step. While our computational experiments successfully demonstrated the model's effectiveness in reproducing three disparate phe- nomena, further work should test its robustness across other reinforcement learning algorithms e.g. policy gradient (Ku- mar & Pehlevan, 2024) and network architectures (Team et al., 2021; Kumar et al., 2025). Additionally, we need to explore how place field reorganization scales in larger, more complex environments (Hill et al., 2020; Lin et al., 2023; Nieh et al., 2021; Kumar et al., 2024) beyond the few environments we considered.
Most models that characterized representational drift were not studied under the context of navigational policy learn- ing (Masset et al., 2022; Pashakhanloo & Koulakov, 2023; Ratzon et al., 2024). We showed that increasing the noise magnitudes caused differrnnt drift regimes (Fig. 3F: Fig. S9D), and at very high noise levels navigation behavior started to collapse (Fig. 3C, Fig. S7). Importantly, we showed that fields in the noisy regime allowed agents to consistently learn new targets in both ID (Fig. 4C) and 2D (Fig. SI2A-B) environments, without getting stuck in local minima. The biological origins of adding noise to place field parameters can be attributed to noisy synaptic plastic- ity mechanisms (Mongillo et al., 2017: Kappel et al., 2015; Attardo al., 2015). Other mechanisms such as unstable dynamics in downstream networks (Sorscher et al., 2023) and modulatory mechanisms such dopamine fluctuations (Krishnan & Sheffield, 2023) could adaptively control drift rates. A difficult experiment that could directly verify our model to induce or constrain place field drift rates in an- imals and determine how this perturbation influences new target learning. How fluctuations in dopamine, stochastic actions and stochastic firing rates within place fields drive
Code Availability
The code for our agents and to reproduce all figures in this paper is available at: https://githhbb com/ Pehlevan-Group www.eeaaannrooopppaeeiiiieeeeeee httpuppppppaeeeeddeeeeeeeeeeeeee reo_geiiddrrrr_ereeeeeeeee
Author Contributions
MGK and CP conceptualized and designed the study. BB performed the theoretical analysis. MGK performed the simulation experiments and wrote the original draft. BB, JZV and CP revised the manuscript.
9

---

A Model of Place Field Reorganization During Reward Maximization
Acknowledgments
Burak, Y. and Fiete, I. R. Accurate path integration in
continuous attractor network models of grid cells. PLoS computational biology, 5(2):e 2009.
We would like to thank Albert Lee, Lucas Janson, Peter Dayan, Farhad Pashakhanloo, Shahriar Talebi, Paul Mas- set, well as the members of the Pchlevan, Ba, Janson and Murthy labs for useful insights. We also appreci- ate the discussions during the Analytical Connectionism Summer School 2024 and COSYNE 2025, This research was supported in part by grants NSF PHY-1748958 and PHY-2309135 to the Kavli Institute for Theoretical Physics (KITP). MGK and CP are supported by NSF Award DMS- 2134157. BB is supported by a Google PhD Fellowship. JAZV was supported by Junior Fellowship from the Har- vard Society of Fellows. CP is further supported by NSF CAREER Award IIS- 2239780, DARPA grant DIAL-FP-038, a Sloan Research Fellowship, and The William F. Milton Fund from Harvard University. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
Bush, D., Barry, C., Manson, D., and Burgess, N. Using grid cells for navigation. Neuron, 87(3):507-520, 2015.
Dayan, P. Improving generalization for temporal difference learning: The successor representation. Neural computa- tion, 5(4):613-224 1993.
Dayan, P. and Abbott, L. F. Theoretical neuroscience: com- putational and mathematical modeling of neural systems. MIT press, 2005.
de Snoo, M. L.,, Miller, A. M., Ramsaran, A. I., Josselyn, S. A., and Frankland, P. W. Exercise accelerates place cell representational drift. Current Biology, 33(3):R96-R97, 2023.
Dohare, S., Hermandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632(8026):768-774 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Neuroscience and/or Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dong, C., Madar, A. D., and Sheffield, M. E. Distinct place cell dynamics in cal and ca3 encode experience in new environments. Nature communications, 12(1):2977, 2021.
References
Dupret, D., O'neill, J., Pleydell-Bouverie B., and Csicsvari, J. The reorganization and reactivation of hippocampal maps predict spatial memory performance. Nature neuro- science, (38))995-1002, 2010.
Amo, R., Matias, S., Yamanaka, A.. Tanaka, K. F., Uchida, N., and Ochida-Uchida, M. A gradual temporal shift of dopamine responses mirrors the progression of tempo- ral difference еπrог in machine learning. Nature neuro- science, 25(8): 2022.
Edelmann, E. and Lessmann, V. Dopaminergic innervation and modulation of hippocampal networks. Cell and tissue research, 373:71 188:711-7760 11-277, 2018.
Arleo, A. and Gerstner, W. Spatial cognition and neuro- mimetic navigation: a model of hippocampal place cell activity. Biological cybernetics, 83(3):287-299 2000.
Eliav, T., Maimon, S. R., Aljadeff, J., Tsodyks, M., Ginosar, G., Las, L., and Ulanovsky, N. Multiscale representation of very large environments in the hippocampus of flying bats. Science, 3 372(6545))aabg4020, 2021.
Attardo, A., Fitzgerald, J. E., and Schnitzer, M. J. Imperma- nence of dendritic spines in live adult cal hippocampus. Nature, 523(7562):592-550) 2015.
Boccara, C. N., Nardin, M., Stella, F., O'Neill, J., and Csicsvari, J. The entorhinal cognitive map is attracted to 2019.
Fang, C. and Stachenfeld, K. L. Predictive auxiliary objec- tives in deep rl mimic learning in the brain. arXiv preprint 1rXii:2310.06088 2023.
goals. Science, 363(6434)
Bordelon, B., Masset, P., Kuo, H., and Pehlevan, C. Loss dynamics of temporal difference reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Fiete, I. R., Burak, Y., and Brookings, T. What grid cells convey about rat location. Journal of Neuroscience, 28 (27):6858 1116658-6871.. 2008
Brown, M. A. and Sharp, P. E. Simulation of spatial learning in the morris water maze by a neural network model of the hippocampal formation and nucleus accumbens. Hippocampus, 171: 1995.
Floresco, S. B., Todd, C. L., and Grace, A. A. Glutamatergic afferents from the hippocampus to the nucleus accumbens regulate activity of ventral tegmental area dopamine neu- rons. Journal of Neuroscience, 21(13):49155 4922, 2001.
10

---

A Model of Place Field Reorganization During Reward Maximization
Foster, D. J., Morris, R. G., and Dayan, P. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1):1-1, 2000.
Kappel, D., Habenschuss, S., Legenstein, R., and Maass, W. Network plasticity bayesian inference. PLoS computa- tional biology, I1(11):e1004485, 2015.
Frank, L. M., Stanley, G. B., and Brown, E. N. Hippocampal plasticity across multiple days of exposure novel envi- ronments. Journal of Neuroxcience, 24(35):7611-7689 2004.
Kempadoo, K. A., Mosharov, E. V., Choi, S. J., Sulzer, D., and Kandel, E. R. Dopamine release from the lo- cus coeruleus to the dorsal hippocampus promotes spa- tial learning and memory. Proceedings of the National Academy of Sciences, (3351))14435-144885 2016.
Frémaux, N., Sprekeler, H., and Gerstner, W. Reinforcement learning using a continuous time actor-critic framework with spiking neurons. PLoS computational biology, 9(); e1003024, 2013.
Kentros, G., Agmihotri, N. T., Streater, S., Hawkins, R. D., and Kandel, E. R. Increased attention spatial context increases both place field stability and spatial memory. Neuron, (2(2)2283-299) 2004.
Gardner, M. P., Schoenbaum, G., and Gershman, S. J. Re- thinking dopamine as generalized prediction error. Pro- ceedings of the Royal Society B, 285(1891)220181655, 2018.
Krishnan, S. and Sheffield, M. Reward expectation re- duces representational drift in the hippocampus. bioRxiv, 2023.
Gauthier, J. I.. and Tank, D. W. A dedicated population for reward coding in the hippocampus. Neuron, 99(1): 179-193, 2018.
Krishnan, S., Heer, C., Cherian, C., and Sheffield, M. Reward expectation extinction restructures and degrades cal spatial maps through loss of dopaminergic reward proximity signal. Nature communications, 13(1):66633 2022.
Gershman, S. J. The successor representation: its compu- tational logic and neural substrates. Journal of Neuro- science, 38(33):7193-7200 2018.
Kumar, M. G. and Pehlevan, C. Place fields organize along goal trajectory with reinforcement learning. Cognitive Computational Neuroscience, 2024.
Gershman, S. J. and Uchida, N. Believing in dopamine. Nature Reviews Neuroscience, 20( (0(11):703-714 2019.
Geva, N., Deitch, D., Rubin, A., and Ziv, Y. Time and experi- ence differentially affect distinct aspects of hippocampal representational drift. Neuron, 11 111(15):337--336 2023.
Kumar, M. G., Tan, C., Libedinskk, C., Yen, S.-C., and Tan, A. Y. nonlinear hidden layer enables actor-critic agents learn multiple paired association navigation. Cerebral Cortex, 32(18):300 (118):3917-3993 2022.
Gonzalez, W. G., Zhang, H., Harutyunyan, A., and Lois, C. Persistence of neuronal representations through time and damage in the hippocampus. Science, 365(6455): 821-825, 2019.
Kumar, M. G., Tan, C., Libedinsky, C., Yen, S.-C., and Tan, A. Y.-Y. One-shot learning of paired association navigation with biologically plausible schemas, 2024. URL www.thtpss:/eett pttttpss//axxi.orrr/ /abs/ / 2106. 03580.
Hill, F., Tieleman, O., Von Glehn, T., Wong, N., Merzic, H., and Clark, S. Grounded language learning fast and slow. arXiv preprint MXXi:20099007199 2020.
Kumar, M. G., Manoogian, A., Qian, B., Pehlevan, C., and Rhoads, S. A. Neurocomputational underpinnings of suboptimal beliefs recurrent neural network-based agents. bioRriv, pp. 2025 03, 2025.
Hopfield, Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 7955)22554- 1982.
Lee, J. S., Briguglio, J. J., Cohen, J. D., Romani, S., and Lee, A. K. The statistical structure of the hippocampal code for space as a function of time, context, and value. Cell, 183(3):620-66) 635, 2020.
Houk, J. C., Adams, J. L., and Barto, A. G. A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement. In Models of In- formation Processing in the Basal Ganglia. The MIT Press, 11 1994. ISBN 9780262275774. doi: 10.7551/ mitpress/4788.003.. 000.0000000000000.. URL https:/ /ww..ooorrr. 10. 7551 httpl/mmmpptsssss/e 4708. 003.0020,
Lee, R. S., Sagiv, Y., Engelhard, B., Witten, I. B., and Daw, N. D. feature-specific prediction error model explains dopaminergic heterogeneity. Nature neuroscience, 27(8): 1574-158,, 2024.
Lillicrap, T. P., Cownden, D., Tweed, D. B., and Akerman, J. Random synaptic feedback weights support error backpropagation for deep learning. Nature communica- tions, 7(1):13276, 2016.
Joel, D., Niv, Y., and Ruppin, E. Actor-crite models of the basal ganglia: New anatomical and computational perspectives. Neural networks, 15(4-6):535-555 2002.

---

A Model of Place Field Reorganization During Reward Maximization
Lin, Z., Azaman, H., Kumar, M. G., and Tan, C. Com- positional learning of visually-grounded concepts using reinforcement, arXiv preprint arXiv:2309.04500 2023.
Montague, P. R., Dayan, P., and Sejnowski, T. J. A frame- work for mesencephalic dopamine systems based pre- dictive hebbian learning. Journal of neuroscience, 16(5): 1936-1947, 1996.
Lisman, J. E. and Grace, A. The hippocampal-ve loop: controlling the entry of information into long-term mem- ory. Neuron, 46(5)-703-713, 2005.
Morris, R. G., Garrud, P., Rawlins, J. a., and O'Keefe, J. Place navigation impaired in rats with hippocampal le- sions. Nature, 297(5868):681
1982.
Low, R. J., Lewallen, S., Aronov, D., Nevers, R., and Tank, D. W. Probing variability in a cognitive map using manifold inference from neural dynamics. BioRxiv, pp- 418939, 2018.
Moser, M.-B., Rowland, D. C., and Moser, E. I. Place cells, grid cells, and memory. Cold Spring Harbor perspectives in biology, 7(2):a021808, 2015.
Mainali, N., da Silveira, R. A., and Burak, Y. Universal statisties of hippocampal place fields across species and dimensionalities. bioRxiv, pp. 2024-06, 2024.
Murray, J. M. Local online learning in recurrent networks with random feedback. Elife, 8:e43299, 2019.
Nich, E. H., Schottdorf, M., Freeman, N. W., Low, R. J., Lewallen, S., Koay, S. A., Pinto, L.,, Gauthier, J. L., Brody, C. D., and Tank, D. W. Geometry of abstract learned knowledge in the hippocampus. Nature, 595(7865):80- 84, 2021.
Mankin, E. A., Sparks, F. T., Slayyeh, B., Sutherland, R. J., Leutgeb, S., and Leutgeb, J. K. Neuronal code for ex- tended time in the hippocampus. Proceedings of the National Academy of Sciences, 109(47):19462-1946) 2012.
Niv, Y. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139-154 2009.
Masset, P., Qin, S., and Zavatone-Vett J. A. Drifting neu- ronal representations: Bug or feature? Biological cyber- netics, 1 116(3):253-26) 2022.
Nøkland, A. Direct feedback alignment provides learning
in deep neural networks. Advances in neural information processing systems, 29, 2016.
Mehta, M. R., Barnes, C. A., and McNaughton, B. L. apperience-dependenee asymmetric expansion of hip- pocampal place fields. Proceedings of the National Academy of Sciences, 94(16)881188821, 1997.
O' Keefe, J. The hippocampus as a cognitive map, 1978.
O'Keefe, J. and Dostrovsky, The hippocampus as a spatial map: preliminary evidence from unit activity in the freely- moving rat. Brain research, 1971.
Mehta, M. R., Quirk, M. C., and Wilson, M. A. Experience- dependent asymmetric shape of hippocampal receptive fields. Neuron, 25 (573)70777715, 2000.
Overwiening, J., Kumar, M. G., and Sompolinsky, H. Tedfa- 8: Temporal integration in deep spiking networks trained with feedback alignment improves policy learning. Cog- nitive Computational Neuroscience, 2025.
Menache, I, Mannor, S.. and Shimkin, N. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research, 134(1):215-235 2005.
Miconi, Biologically plausible learning in recurrent neu- ral networks reproduces neural dynamics observed during cognitive tasks. Elife, 6:c20899, 2017.
Packard, M. G. and McGaugh, J. L. Inactivation of hip- pocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neuro- biology of learning and memory, 65(1):65-72, 1996.
Mnih, V., Badia, P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 of Proceedings of Machine Learning Re- search, pp. 1928-1932 New York, New York, USA, 20- 22 Jun 2016. PMLR. URL httpp:::............tt...t...h : http///oeeedn// ngs. mlr. mlr.press// 4 8 mmihal6 html.
Palacios-Filardo, and Mellor, R. Neuromodulation of hippocampal long-term synaptie plasticity. Current opinion in neurobiology, 54:37 2019.
Pashakhanloo, F. and Koulakov, A. Stochastic gradient
descent-induced drift of representation in a two-layer neural network. In International Conference on Machine Learning, pp. 27401-27419 PMLR, 2023.
Mongillo, G., Rumpel, S., and Loewenstein, Y. Intrinsic volatility of synaptic connections a challenge to the synaptic trace theory of memory. Current opinion in neurobiology, 46:7-13, 2017.
Priestley, J. B., Bowler, J. C., Rolotti, S. V., Fusi, S., and Losonczy, A. Signatures of rapid plasticity in hippocam- pal cal representations during novel experiences. Neuron, 1 110(12):1978-199 2022.
12

---

A Model of Place Fieid Reorganization During Reward Maximization
Qin, S., Farashahi, S., Lipshutz, D., Sengupta, A. M., Chklovskii, D. B., and Pehlevan, C. Coordinated drift of receptive fields in hebbian/anti-hebbian network mod- els during noisy representation learning. Nature Neuro- science, 26(2):339-349 2023.
Schultz, W., Dayan, P., and Montague, R. A neural substrate of prediction and reward. Science, 275(5306):
1593-1599, 1997.
Sorscher, B., Mel, G. C., Ocko, S. A., Giocomo, L. M., and Ganguli, S. A unified theory for the computational and mechanistic origins of grid cells. Neuron, 111(1): 121 121-137, 2023.
Raju, R. V., Guntupalli, J. S., Zhou, G., Wendelken, C.,, Lázaro-Gredilla, M., and George, D. Space is latent se- quence: theory of the hippocampus. Science Advances, 10(31)eadm8470, 2024.
Sosa, M., Plitt, M. H., and Giocomo, M. Hippocampal sequences span experience relative rewards. bioRxiv, 2023.
Ratzon, A., Derdikman, D., and Barak, O. Representa- tional drift as a result of implicit regularization. Elife, RP90069, 2024.
Stachenfeld, K. L., Botvinick, M. M., and Gershman, J. The hippocampus as predictive map. Nature neuro- science, 20(1 1):1644-1653 2017.
Reynolds, J. N., Hyland, B. I., and Wickens, J. R. A cellu- lar mechanism of reward-related learning. Nature, 413 (6757), 2001.
Starkweather, C. K. and Uchida, N. Dopamine signals as temporal difference errors: recent advances. Current Opinion in Neurobiology, 67:95-105, 2021.
Rokni, U., Richardson, G., Bizzi, E., and Seung, S. Motor learning with unstable neural representations. Neu- ron, 54(():653-66) 2007.
Steele, R. and Morris, R. delendde impairment of seichingg-to-llce task with chronic and intrahippocam- pal infusion of the nmda- anda-antagonist d-ap5. Hippocampus, 9(2): 18-136, 1999.
Rule, M. E. and O'Leary, T. Self-healing codes: How stable neural populations can track continually reconfig- uring neural representations. Proceedings of the National Academy of Sciences, 119(7)=2106692119, 2022.
Stöckl, C., Yang, Y., and Maass, W. Local prediction- learning in high-dimensional spaces enables neural net- works plan. Nature Communications, 15(1):2344, 2024.
Rule, M. E., Loback, A. R., Raman, D. V., Driscoll, L. N., Harvey, C. D., and O' O'Leary, T. Stable task information from an unstable neural population. elife, be51122, 2020.
Suhaimi. A., Lim, A. W. Chia, X. W., Li, C., and Makino, H. Representation learning in the artificial and biologi- cal neural networks underlying sensorimotor integration. Science Advances, 8(22)xeab00984, 2022.
Russo, S. J. and Nestler, E. J. The brain reward circuitry in mood disorders. Nature reviews neuroscience, 14(9): 609-625, 2013.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Bradford Book, 2018
Sayegh, F. J., Mouledous, L., Macri, C., Pi Macedo, J., Le- jards, C., Rampon, C., Verret, L., and Dahan, L. Ventral tegmental area dopamine projections to the hippocampus trigger long-term potentiation and contextual learning. Nature Communications, 15(1):4100 2024.
Sutton, R. S., McAllester, D., Singh, S., and Mansour,
Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Schaeffer, R., Khona, M., and Fiete, I. No free lunch from deep learning neuroscience: A case study through mod- els of the entorhinal hhpppiiaaa-hpppocmppllllal circuit. Advances in neural information processing systems, 35:16052-16067, 2022.
Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J.. Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., et al. Open-ended learning leads to gener- ally capable agents. arXiv preprint arXiv:2107.12808. 2021.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. figh-ddmmeniionl continuous control using generalized advantage estimation. arXiv preprint 106.025066 2015.
Tolman, E. C. Cognitive maps in rats and men. Psychologi- cal review, 55(4):189, 1948.
Tse, D., Langston, R. F., Kakeyama, M., Bethus, I., Spooner, P. A., Wood, R., Witter, M. P., and Morris, R. G. Schemas and memory consolidation. Science, 316(5821): 76-82, 2007.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint
13

---

A Model of Place Fieid Reorganization During Reward Maximization
Van Vreeswijk, C. and Sompolinsky, H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, 274(5293) 1996.
Wang, J. X., Kurth-Nelson, Z.,, Kumaran, D., Tirumala, D., Soyer, H., Leibo, Z., Hassabis, D., and Botvinick, M. Prefrontal cortex meta-reinforcement learning system. Nature neuroscience, 21(6))860-868, 2018.
Wen, H., Sorscher, B., Aery Jones, E. A., Ganguli, S., and Giocomo, L. M. One-shot entorhinal maps enable flexible navigation in novel environments. Nature, pp. 1-8, 2024.
Yaghoubi, M., Nieto-Posadas, A., Mosser, C.-A., Gisiger, T., Wilson, É., Williams, S., and Brandon, M. Predictive coding of reward in the hippocampus. bioRxiv, pp. 2024- 09, 2024.
Ziv, Y., Burns, L. D., Cocker, E. D., Hamel, E. O., Ghosh, K. K., Kitch, L. J., Gamal, A. E., and Schnitzer, M. Long-term dynamics of cal hippocampal place codes. Nature neuroscience, 16(3):264-200) 266, 2013.
14

---

A Model of Place Fieid Reorganization During Reward Maximization
A. Details of the Place field-based navigation model
The code for initializing and training the model in ID and 2D environments, along with the code for analyzing neural phenomena and generating all figures, will be available on GitHub upon acceptance.
A.1. Place fields in 1D and 2D environments
The agent contains N place fields. In a ID track, each place field is described as
(x(x)) = a exp It x 12
(9)
2
with a, and a describing the amplitude, center and width, adapted from Foster et (2000); Kumar al. (2022: 2024). Most of the simulations were initialized with amplitudes a; 0.5 and widths o; 0.1, with centers uniformly tiling the environment À {-1,..., 1.. Nevertheless, similar representations emerge for amplitudes drawn from a uniform distribution between [0, 1] and widths uniformly drawn between [0.01, 0.25). This parameter initialization was used for ablation studies in Fig. 4. In 2D arena, each place field is described as
1
(x11) a exp
2, x.) 7
(10)
where E, is a 2x2 covariance matrix, adapted from Menache al. (2005). The off- aff-diagonals were initialized as zeros and diagonals initialized match the variance in the 1D place field description, i.e. En 0.12 to ensure field widths consistent in ID and 2D.
A.2. Reward Maximization Objective (Policy Gradient)
The objective of the model is learn a policy a parametrized by W and spatial features & parameterized by 0 that maximizes the expected cumulative discounted rewards over any trajectory {zo, 90, To, TI.T, TT, 2T+1) in finite-horizon setting, modeling the trial structure in neuroscience experiments
JC= = EETToo
Σ r'r = F. [G(T))
(11)
where y is the discount factor, Yt is the reward at time step after choosing an action 9: at state ny, and the time horizon finite with trials ending after a maximum 100 steps the ID track and 300 steps in the 2D arena.
To maximize the cumulative reward objective, we perform gradient ascent on the policy and place field parameters, &new fold + 17 VeJG
W new = W + V w.
(12)
where t7 and 17 are learning rates for f and T respectively. The gradients are derived using the log-derivative trick, Vo.w RR, (G(T)
(13)
= Vo,w
(14)
p( Wg"(()),W")GG
(15)
E, Va,w- logp(7 W (WT)G(T)],
(16)
where the trajectory 7 describes the state to state transitions. We expand the above using the Markov assumption that the transition to future states depend only on the present state and not on the states preceding it,
p(T W p(zo) II
1 W*)
(17)
==
logp(r|0, WT) log p(rop(())))))) +
(logp(ze+110)) + log T (g.22,; WT))
(18)
T
Va,w- log (((0, WT)
Σ
Vo.w log ( Ix WT).
(19)
1=
Since the gradients are not dependent on the state transitions, the last line excludes them. Substituting Eq. 19 into Eq. 16
15

---

A Model of Place Field Reorganization During Reward Maximization
yields
Vg.w- = EE,
Ve.w log 0 W*) G.
(20)
===
which completes the full derivation of the policy gradient theorem (Sutton et al., 1999; Sutton & Barto, 2018). The policy gradient objective was used by Kumar & Pehlevan (2024) to optimize the policy and place field parameters. However, this learning signal requires an explicit reward and policy gradient methods are slow to converge as they suffer from high variance due to:
Monte Carlo sampling: Agents need to sample an entire episode to estimate the expected return [-[G, I' M + r + ...] before updating the policy. This can introduce significant variance because the estimate based single path through the stochastic environment, which may not be representative of the expected value over many episodes.
No Baseline: The basic policy gradient algorithm computes the gradient solely based on the return G' from each trajectory. By introducing baseline (either constant b or dynamically evolving b, e.g. value function uz), which estimates the expected return from a given state, the variance of the gradient estimate can be reduced, because now the policy learns which action is better than the previous (concept of using an Advantage A, instead of rewards).
Value based methods (Sutton & Barto (2018), Chapter 3.5) were introduced address some of these issues. For instance, instead of sampling returns Gr, value functions Vú learn estimate the expected returns each time step t:
1/2 E[
(21)
which can reduce the variance during credit assignment. The combination of policy gradient with value-based methods lead us to the Actor-Critic algorithm.
A.3. Alternative reward maximization objective (Temporal Difference)
The optimal value function V2 reflects the true expected cumulative discounted rewards, hence the policy gradient objective can be rewritten
J = [G(T]] E
(22)
1=0
rt + Σ
(23)
1=1
J E Ir + + <Ve+1l
(24)
which yields the following self-consistency equation
Tt + 7V+1 - V/2 = 0,
(25)
argued by Sutton & Barto (2018); Frémaux et al. (2013).
Alternatives to policy gradient algorithms propose subtracting a baseline which can be fixed constant or a dynamically changing variable b,. Since we have the value function Vú, we can modify the objective to be
=A=E Vi E [A]]
IE Σ rt + vV+1 V/2
(26)
which gives us the Advantage function (Mnih et al., 2016; Schulman et al., 2015). This reduces the variance as the policy has to learn select actions that gives an advantage over the current value function. We get a learning objective function that is analogue maximizing the expected cumulative discounted returns while subtracting a baseline Eq. 11.
Vo.w = Σ V 1og 1x,; 0, WT). A
(27)
However, we have assumed that given the optimal value function V2 to critique the actor if it is doing better or worse than before. Instead, we can learn to estimate the value function is using critic by minimizing the Temporal Difference error
= Ti + YU++1 Vt.
(28)
16

---

A Model of Place Fieid Reorganization During Reward Maximization
The critic can learn approximate the true value function by minimizing the mean squared error between the true value function V2 and the predicted U/ the temporal difference error 6t
T
5 = E
(V(x+) - 0( w
(29)
T
(re + +VV((++))) - u( w²))
E
(30)
2
Since do not have the optimal value function Vr, we can approximate it by bootstrapping the estimated value function U and ensuring that we do not take gradients with respect to the time shifted value estimate v(z(+1) using stop gradient (sg):
T
LT
=
E
(Te + 70(0041 ri
v( 0, w w)
(31)
2
0
E
82
(0, w")
(32)
2
r=0
We minimize the temporal difference error using gradient descent for the critic estimate the value function
OLTD
08
V LDD
V0, w"),
(33)
86
dv
E
5i (-1) V V0 w²)
(34)
E
V g- v( w²) -
(35)
Notice the additional negative sign that pops out when you take the derivative of only with respect to et
06 a(r, + 770+1 vz)
(36)
dv
Se,
since Tr and 1 are treated as constants, we do not take their derivatives. Since we do not have the optimal value function V3 but a biased estimate Vt, we can use the temporal difference error as our reward maximization objective
JTD =E
Σ re + 7
=
Σ 82
(37)
As the value estimation becomes closer to the optimal value 0, + V,, this objective becomes similar to the advantage objective JTD JAA. Note that we are not directly maximizing the TD error during policy learning. Rather, we want to optimize the policy æ and place field parameters 0 by gradient ascent, using the biased estimate of the advantage function
vo JTD =
Vo,w" log (99 0, W*) 6g.
(38)
Li
An alternative interpretation is that during policy learning, the agent learns a policy to maximize the difference between the actual reward and the estimated value
A.4. Combined reward maximization objective for place field parameters
In our model (Fig. IA), actor 4 and critic w" weights are optimized separately, while the place field parameters @ overlap. The actor uses gradient ascent for Eq. 27, and the critic employs gradient descent for Eq. 35. Since we have a single population place fields, we optimize these parameters to support both objectives. Thus, we derive a combined objective function to update W w", and in a single gradient pass
17

---

A Model of Place Field Reorganization During Reward Maximization
w = V VWa qun JTTD
VW= ,00, 9 LD
(39)
E
W wn log (1 W" .0)
E
V W w 00 w", .06,
(40)
E
W" 000, + Vw= w oe( , W wr log (9) W"
00,,
(41)
E
V V loooloogooo (g W", .0) + V (z wo, 0)) 6
(42)
0 and LTD 0 since the respective objectives are not parameterized by w" and W respectively, This means that W is tuned to maximize JTD, w² is tuned minimize LTD and is tuned to balance both the objectives.
where V. JTT
Since most optimizers e.g. in Tensorflow, PyTorch perform gradient descent, not ascent, we can minimize the negative policy gradient Eq. 27, which is equivalent to the negative log likelihood
V W _0 oh
V Vwe w JTD + V w TD
(43)
T
W w log W" 0) 8z
+
V w (0 .))
(44)
E
V 0 - log (9:l:e W=, 0) 6r
+ E
Vw .m ww (x(; w", 00,
(45)
Vw any.0 o G
Vw _m . C.
(46)
which is the same update rule used in Wang et al. (2018); Mnih al. (2016) to train the actor and critic separately while the feature parameters are trained jointly.
It is also possible to initialize separate populations of place fields, each for the actor and critic. Alternatively, we only optimize place field parameters using the actor's objective while the critic uses the spatial features to learn the value function. The converse also possible where the place field parameters and critic weights are optimized to minimize the TD error while the actor learns a policy without optimizing the spatial representations, as we did in the perturbative approximation (App. B). From numerical experiments, optimizing place field parameters using both the actor and critic objectives allowed the agent to achieve the fastest policy convergence and highest cumulative reward performance (Fig. S15).
A.5. Online update place field and actor-critic parameters
Now, we derive an online implementation of Eq. which is the same as Eq. 42, so that all parameters are updated at every time step. Extending from Foster al. (2000); Kumar et al. (2022), the actor and critic weights are updated according the gradients
Aw Aw" (t + 1) = n(;)
(47)
where Øt.j St- P and 0.01. The gradient updates for place field parameters follow
AO 1) nod; (w) + W (t) g.) V2)(2;;0).
(48)
where we use a significantly smaller learning rate 1jo = 0.0001 so that the spatial representation evolves in a stable manner.
18

---

A Model of Place Field Reorganization During Reward Maximization
Specifically, each field parameter is updated according
T ó, (w, (t) + (e(()
(t.))
(49)
Aas, 7aa bp d(()))
2
(50)
ai
Jt A of
bp Adi 71x o, (x,))
(51)
where 622 is the TD error gradient that has been backpropagated through the actor and critic weights. Using just the w (t) 3 weights alone to backpropagate the TD emror influences the representation learned by the place field population and ultimately the navigation performance (Fig. S15).
There are two ways to optimize the place field width parameter. The first and straightforward method is to update the width parameter according to
(xt x.33 T
A To -5 0z (x4)
(52)
where k in ID place field. In 2D place field with h 2, while keeping the off-diagonals to zeros as in Menache et al. (2005). However, fields will only elongate along each axis. Instead, in our simulations, we optimized the off-diagonals using the same gradient flow equations. However, we needed to include additional constraints so that each place field's covariance matrix remains I) symmetric, 2) bounded, and 3)positive semi-deete deni-definite to perform matrix inversion. Specifically, the covariance matrix was bounded between [10-5, -5 0.5] to prevent exploding widths and gradients.
update the diagonal elements in the 2D matrix
19

---

A Model of Place Fieid Reorganization During Reward Maximization
B. Derivation for perturbative expansion
The dynamics of place field parameters are nonlinear and difficult to characterize analytically. To gain some analytical tractability, we impose a strong separation timescales between policy learning updates and place field parameter updates. To do so, set the learning rates for the actor-critic 7) to be much larger than the learning rates for the place field parameters :a: MA, fo < 17- In simulations,
77 0.01 and 7o 0.0001.
The critic estimates the value as
N
v(x)) =
Σ wroi (æt, 0.),
(53)
where 0; (arg; Ar, o.) are neuron specific parameters (amplitude, mean, and bandwidth respectively). We write w" as for clarity. To start with let's just consider
(i 0.) = at exp (-207(1
(s. - -2)=-
(54)
We consider a TD based update, which in the gradient flow (infinitesimal learning rate) limit can be approximated as
dt w(t) = (w) ww(()),
(55)
((()) = w;(() Vo. 0.)6,,
(56)
di
The key assumption make is that the dimensionless ratio of learning rates, e is perturbatively small
710 1,
(57)
7
where n7e the learning rate for the place field parameters 0: and n is the learning rate for the actor-critic. The matrix (t) E() (t) where Σ (ab( (a)e)) and Σ+ (=)r))))))))))11) depends on the equal time and time-step shifted correlations of features. The vector w M EEE where wn (x(x) R(x). We investigate simple perturbation series.
wtt) = wo(() + (t) + 2 w2 +
(t) = (t()) + c (t) + 22 +
(58)
and examine the dynamics up to first order in We will show that this recovers many qualitative features of the observed representational updates.
The leading zeroth order dynamics are
0o(t) 0 0, d (t) Mo (wy - wn)()),
(59)
dt where Mo = Σ(0) - (0) is the initial feature covariance under the initial policy.
dt
B.1. Place Field Amplitude
We start by asserting a separation of timescales between training readout weights and feature parameters during a simple TD learning setup
dt (t)() = Σ Mis(wy - ww),
(60)
dt (t) = ailt)
2
Σ 3
Mis(wy - wy),
(61)
The zero-th order solution to Eq. 55
Awo (t) = wv wt)(t) exp (- Mt) wy
(62)
wo(() = [I - -xpp [-Mt)]wy.
(63)
which can be substituted in to get the first order correction the dynamics for 0
20

---

A Model of Place Field Reorganization During Reward Maximization
an (1) = 2000" ④ [I exp (-Mi)) @ M exp (-Mt) wy
(64)
dt
Under the condition that oq - and M - M we can work out an exact expression terms of the eigendecomposition M = >x u
(n = >- (wv uue wv) (ux o oue) (1 (1-c-hat)
d + de (1 e (Aotteltt)
x
(65)
kt
approximate this at late times
lim (1) R 2wv ④ wy 60000
(66)
As + oo we can approximate this as lo (t)) RM 2(wv)² This indicates that neurons which are heavily involved in the reproduction of the value function are upweighted in their amplitude.
B.2. Field Center
Based on the place field center update equation and rewriting the terms as above,
(1)
Lt A wc))(()
Σ (n) - wwy-
(67)
dt
of
We need compute an average over spatial positions. We approximate the space position early in training as a Gaussian with mean 8n and variance oz
(Iz - Ar) 0²
(x(0))0(())
~ Mis A M.j. σ²
(68)
where Maj -3: 3/2 + ) (A+ + Xj) + 2/2 f1z) is the mean value of x obtained by the above Gaussian integral under
the approximation that p(x) Nizz O2) Approximating Xr the mean position of the tuning curves À obtain the following prediction
( (A(0) + 1
A(t) X(0) V
2(0)
O exp (-Mc))
(69)
Following the solution Eq. 63, we can approximate this at late times as
lim ((t) - x(0) ~ ew o 10
(
02
2
1 03
(
(200) X) _2
Pz
x
owv
(70)
Hence, in addition to the value of a location, three additional factors influence each field's ' displacement.
x(t) (0) ~ 7
2
(x o7 + Pv x,
W (t), 17 <
(71)
77
o2
02
where Ā the agent's 's expected location sampled from its policy, Rs = 7.775 is the starting location and Cr is the estimated spread of the trajectory. This analysis suggests that fields will be influenced by both the start location and the location where the agent spends higher proportion of time at. In later learning phases, this will be the reward location À 0.5. Consequently, only the fields near the reward location will shift towards the reward, while the rest of the fields will move towards the start location. We illustrate this perturbative approximation at carly and late times of training in Figure 5. The theory is quite accurate early in training, but fails at sufficiently long training time.
21

---

A Model of Place Field Reorganization During Reward Maximization
A
T- 100
B
T= 10000
0.0005
0.05
A(0)
0.0000
0.00
Sim
-0.0005
Theory
-0.05
Theory
-1.0
0.0
0.5
-1.0
-0.5
0.0
0.5
1.0
Figure 5. Difference in early versus late time perturbative approximation. Blue scatter points shows the magnitude and direction of change (N 256) field center position compared the position at which the fields were initialized (A, (0))). (A) early time, the perturbative expansion is good fit to the field center displacement, and captures the shift in fields towards the reward location 2, = 0.5 (red) (B) As learning proceeds, the approximation begins to break down for fields further from the reward location. Free parameters were fit with À = 0.535 and d= 0.45.
C. Successor Representation agent
The generalized temporal difference error is given by
8 (م())) (xx+1) 05 (xx).
(72)
with Mj representing the predicted successor representation and ((x)) representing the initialized place field representation that is not optimized.
(2.) Σ 1 (x))),
(73)
The successor representation is computed using a summation of the place fields with a learned matrix U that is positively rectified. The rectification is necessary have a non-negative representation.
AU 6x(x)) -
(74)
The matrix U initialized as an identity matrix and is updated using a two-factor rule using the TD error as in Gardner et al. (2018).
D. Metric Representation learning objective
The hippocampus is known to learn and represent spatial maps even in the absence of rewards, enabling rapid navigation to new locations when required (Tolman, 1948; Steele & Morris, 1999; Tse et al., 2007). This requires reorganization of place fields non-rewarded conditions, which has been proposed as a mechanism for learning a predictive map that estimates future spatial occupancy (Mehta al., 1997; Stachenfeld al., 2017). To describe this non-reward-baase reorganization, the successor representation algorithm (Dayan, 1993) has been used. More recently, an auxiliary predictive objective has been proposed (Fang & Stachenfeld, 2023).
Here, we present simple predictive objective for place field reorganization that is independent of rewards. We introduce a previously described objective called the Metric Representation (MR), which learns a low-dimensional representation of an environment using place field activity and a biologically plausible learning rule that is modulated by a path integration- derived Temporal Difference error. This representation allows an agent to predict its current coordinates 2(x;) and perform vector subtraction to rapidly navigate recalled goals (Foster al., 2000; Kumar al., 2024). However, representation learning not studied using this objective. Recently, similar objective learning rules, although as a high-dimensiona. representation (Stöckl et al., 2024).
proposed to learn spatial map using local
The dimensionality of the coordinate prediction 2(xx) is equal to the dimensionality of the environment, calculated through a linear summation of place field activity:
2j (xx) Σ (T)) Z
(75)
When the agent accurately predicts coordinates in the environment, the following path integration-derived self-consistency
22

---

A Model of Place Field Reorganization During Reward Maximization
equation holds:
2y(0++1) = 2j (2+) + az(())) ( - 2,(1)) - az (z( =0,
(76)
(77)
where (z(2,) is the true displacement of the agent in the environment. However, if the prediction is inaccurate, Eq. 77 can be reformulated into temporal difference error for each dimension j of the environment described by Foster et al. (2000); Kumar et al. (2024):
X 27(1++1) 2y(x:) - az()), X
(78)
This one step prediction error (x+) can be expressed as a loss function, similar to Fang & Stachenfeld (2023) without the temporal discounting factor:
C
Eigna Σ
E
Σ 2 ((0441:WMng 2(n; W -xx;))
(79)
which can be minimized by gradient descent by optimizing both the coordinate readout weights (RMR) and place field parameters (0 E {a, X. +))
V WERRO o = E
X V+((;;0))
(80)
1=0
The gradient updates were implemented in an online manner:
Wn + 1) = nx:
(81)
A + 170 W (xR(()xxVoo(la))
(82)
We can analyze how the different temporal difference residues (both the canonical reward -eerddde and newly proposed metric representation-basse influence place field reorganization and agent policy learning performance by propagating the residues through combination of actor, critic, and metric representation weights:
hp de (3 (t) + 8 W (t) Tt) - BMn W (t)
(83)
A + 1) = 1 Vo 8)
(84)
This can be done by setting the weighting of each component Se, Sr, BMR E {0, 1). Refer to Fig. S15 for policy convergence performance in both the ID and 2D environments when using different combinations to learn place field representations.
E. Details for noisy field updates
To induce drift, independently introduced noise field amplitudes, centers and width, well the synapses the actor and critic (0 E {a, a, w", WT)).
0c+1 = 02 Ec.
(85)
where the noise term & are independent Gaussian noises with zero mean and magnitude noise 10-1), We performed a noise sweep to determine how increasing the noise magnitude affected the agent's reward maximization behavior, population vector correlation and representation similarity. Refer to Fig. S7.
23

---

A Model of Place Field Reorganization During Reward Maximization
A
N= 16
B
N 258
T=1000
Fxed: T=5000
T-50000
T=50000
Bodd)
Booto
ZevoRDEICON
400 Poodo
600 Fwilt
400 Bwil
400
T-1000
4a: T-50CO
T-500C0
7-1000
T-5000
T-50000
XAW COW)
4601 Poeti
24
421 Pruhi
6001 Red
600
T=-000
4c: T=5000
T=500C0
11000
T=5000
T-0000
Xee IRODICON
*/C. Popln
M41 Pruho
Ad Rwib)
AC. T-1000
Ad. T-5000
4a, 6e: T-50300
No: T=1000
Ac. Ac: T=5000
No: T=50000
Budel
450 Budei
Pawnr
800 Dwe
600
800
AA: T=1000
AA: 5020
AA: T=500C0
M: 11000
T-5000
COR
6: T-50000
400 Poodo
600
600
AI An: T-1000
EAteROE COMG
A.. Ac: T-5000
ZAVISIDIION
AA, Aa: T-50000
4A, Ae: T-1000
Rudio
43. Aa: T=5000
4A. Ar: T-50000
800
AA Ac: 1000
24. Ao: 1m5000
AA. De: 1=50000
4A, Ae: T=1000
EMADRORICOMO
A1, Ac: T=0000
2A, Ac: T-50000
auc. Fudia
40c
K4
4A, S0, Ao: T-1000
MA La, dc: T-5000
MA. &0, Ao T-59000
M. Ac. Ar T=1000
M. No, Ar T=5000
W. Ao. frr 5-0000
CONO
400 podod
800 Par
800 Dwrlt
Z
Figure ST. Influence of place field parameter optimization for single seed. Example change in individual field's spatial selectivity (0(z), colored), mean firing activity at a location (2, (x(x)) field density which is the number of Center of Mass (COM) in a location location
after smoothing with a Gaussian kemel density estimate (gKDE) (ECOM), blue) and, the frequency of being (PRa)())) when optimizing different combinations of field parameters (a, X, a)) during reward maximization (RM). The location in which the highest value for mean firing activity, field density and frequency is attained indicated by a red, blue and black vertical dash line respectively. Optimizing (A) small number (N 16) and (B) large number of place fields yields similar high mean firing rate the reward location followed by the start location. However, the field density evolves diffcrently when the low field regime, (A) a high density emerges the reward location in the early stages of learning, but shifts to the start location at later stages of lcarning. This effect was observed a recent experiment where place fields which initially encoded the reward location, gradually shifted backward towards the corresponding start location. This shift led to decrease place fields specifically coding for the reward, suggesting that the hippocampal representation reorganizes to predictively code for the reward (Yaghoubi et al., 2024). Whether experiments demonstrate such misalignment between place field density and mean firing rate needs to be analyzed. Based on the ablation studies (Fig. 4A,B), mean firing rate will be a stronger indicator of learning performance than field density. (B) In the high field regime, a high field density the reward location remains stable throughout learning. Note that COM changes only when the place field centers are optimized (AX). Distribution is shown for single seed run for a homogencous place field population that has been initialized by with equal spacing between field centers (A E [-1, 1)), equal amplitude (a 0.5) and width (a 0.01). Refer Fig. S2 for general place field reorganization over different seeds.
24

---

A Model of Place Field Reorganization During Reward Maximization
A
Fields with constant space, width, amplitude
N = 4
=
=
0.6
0.3
0.4
0.8
0.20
0.15
0.6
Q.8
0.8
0.6
(KDE(COM)
0.4
0.4
0.10
0.05
0.4
0.2
0.05
0.2
0.0
0.00
0.508
0.538
0.48
-0.77
0.486
0.716
0.508
0.198
64
N
256
0.08
0.06
0.6
0.03
gKDE(COM
0.5
0.02
0.4
0.4
0.02
0.4
0.4
0.3
B
Heterogeneous place field population
N = 4
=
= 16
N 32
0.4
0.8
1.5
1.00
0.20
0.100
(KDE(COM)
Ed00
0.75
0.15
0.6
0.075
0.50
0.050
0.5
0.1
0.25
0.025
0.2
0.0
0.00
0.000
0.316
0.542
0.484
0.55
0.542
0.536
0.54
64
N
256
0.08
0.03
0.6
0.04
0.5
0.5
gEDE(COM)
0.02
0.04
0.02
0.4
0.4
0.02
0.02
0.3
0.01
0.3
0.3
0.2
Figure S2. Average change in field density and mean firing rate for different number of place fields. Vertical blue and red dash lines indicate the location with the highest density and mean firing rate, with the legend indicating the location (æ)) (A) Homogencous place field distribution amplitude (ax 0.5) and equal width (a = 0.01). (B) All place field parameters center (A), amplitude (ax), and width (ar) were initialized by sampling from uniform distribution between [-1, 1]. [0, 1]. [10 10-5, 0.1] respectively model heterogeneous place field population. Learning rates for the place field parameters and actor-critic were = 0.0001 and n = 0.01 respectively. Shaded area is 95% CI over 50 diff lifferent seeds.
initialized with ficld parameters similar to Fig, S1, equal spacing between field centers (A [-1, 1)), equal
25

---

A Model of Place Field Reorganization During Reward Maximization
A
T= 0
T= 50000
T= 100000
1.0
1.0
1.0
Velocity Vix)
dbx
0.5
0.0
0.5
0.0
0.5
0.0
0.0
0.1
0.0
0.1
0.0
0.1
1
11
1
B
Fields before learning
Fields before shift
Fields after shift
Tuning curves $00
1.0
0.5
1.0
-0.5
1.0
Location (x)
Location
Location (x)
C
1.0
After shift (x 0.2)
0.5
0.0
-0.5
-1.0
-1
0
1
Before shift (x, 0.75)
Figure A small proportion reward-encoding place fields shift the new reward location. Agents with 256 place fields and Gaussian noise injected to field parameters (Anoise 0.0001) were trained to navigate to reward location al 0.75 for 50,000 trials, thereafter reward location was shifted 2,, 0.2 for the next 50,000 trials. (A) Place field density at start of learning uniformly distributed (left) and increased near the first reward location at the end of the first 50,000 trials (center). After the shift in reward location, a high density of fields emerged at new reward location (right). The black line shows learned policy, where velocity of 0. (-)..) indicates moving right (left). Agents learn to navigate the reward location, both before and after the shift. (B) Example distribution of individual place fields before learning (left). before the shift (center) and after the shift (right). All place field parameters X, a, and
initialized by sampling from uniform distribution between [-1, 1), [0, 1]. [10 0.1]] respectively model heterogeneous place field population. Notice the backward shift of some place fields that were at the initial reward location to the new reward location. (C) About 2.6% of the place fields coding for the initial reward # 0.75 (green dots) shifted the new reward location at 2, = - 0.2 (about 19 of the 734 green dots are within the blue circle). Other place fields at x, = -0.2 increased their firing rate encode the new reward location. We large number of fields shifting backward, though not entirely to the new reward location. Shaded area shows 95% CI for 10 seeds of agents with 256 place fields each. Black and green dots show total 2560 place fields for all 10 agents.
26

---

A Model of Place Field Reorganization During Reward Maximization
A
Fields with constant space, width, amplitude
B
Heterogeneous
[|0rmal 8o1l/1|Bo11
[|0 Boll/11Bl11
place field population
100
10-1
10-1
10a
102
101
102
N
N
C
Rese 0.05, Fir 2V
0.3
Riaee Ruw 6.
0.4
Rear 5, Rose 0.05, Fit::30
Rinee S, Ryee 0 0.05, Fit: 1N
Reoss 9, Ruse 0.05, Fit: 2.0
(Ze())) Σd()))/N
Ruv 0.05, 3.
0.3
d(x)) d(x)
Ruos 0.025, -05
0.2
Riar 5, Raze 0.025.Fit:: 0000>>
0.2
Rear 5, Ruse 0.1.Fit: 0.1.Fi::4.8N 4.8N-14
Rosee 5, Ryee 0.1, Fit: 3.3
0.1
0.1
0.0
-0.1
0.0
10a
102
101
102
N
N
Figure 54. Weak feature learning with large number of place fields. Critic w and actor W weights were initialized by sampling from random normal distribution. (0, 10 3). despite the number of place fields N, similar Foster et al. (2000); Kumar et al. (2022); Frémaux et al. (2013). (A) Homogeneous place field population: Place field parameters were initialized with equal spacing between field
centers (À E [-1, 1-,11] 1]), equal amplitude (a 0.5) and equal width (a 0.01). (B) Heterogeneous place field population: All place field parameters center (A), amplitude (a), and width (ar) were initialized by sampling from uniform distribution between [-1, 1). [0, 1), [10-5, 0.11 respectively. (A-B) The sum of the 1.2 and final value decreases the number of fields available increases. Hence, as the number of fields increases. the change in each place field's parameter becomes smaller. This suggests a weak feature learning regime with large (C) Similar to Fig. ID. Density at the reward location d(z..) compared to non-reward location d(x') decreases with higher number of fields. (D) The mean firing rate at the reward location Σ ((x)) compared to non-reward location Σ ((x') decreases with a higher number of fields. (C-D) Density and mean firing rate at the reward location are proportional to the reward magnitude ( (Rmor), and inversely proportional to the size of the reward location (Rasse). Error bars show 95% CI over 50 different seeds.
for each place field's center X, amplitude and width between its initialized
27

---

A Model of Place Field Reorganization During Reward Maximization
A
Ro:T+g
5
Actor
Actor
CrRic
Postion
PL
Ph
6:17-10
R.:T- 10020
X
6633
2-000
SA
Fixed
-6492
(T-
(=00000
(T 10000)
(T0000
C
100
AM
0:55
03
3.66
1.15
3.02
4AMIT 1000)
6 (M 3000)
#M 10000)
( 50000)
0.03
5.2
6.75
0.50
0.000
8.000
-102
-0032
-2.44
17- 1000)
("
" 10000)
6 "T
010
-1.95
-003
0.075
177
323
107
337
107
10'
107
107
10
0.050
0.1
0.1
0.025
0.000
0.00
0.0
F
G
1.0
1.0
1.0
Pox Paye
Prur
0.8
feefie fou
0.5
Peu Pire
0.6
Figure S5. SR and MR agent architecture, and representation dynamics. (A) Successor Representation (SR) agent architecture to learn navigational policy and the SR place fields. Only the synapses from the initialized place field (ofized) the actor (red) and critic (green), and the synapses (U)) to the SR fields (17) were plastic. Refer to App. C for implementation details. (B) Left: Metric Representation (MR) agent architecture learns predict the agent's coordinates summation of place field activity, and its dimension is the same as the displacement in the environment. The agent lcarns to predict its coordinates by minimizing path integration derived temporal difference eтror Xt- The gradient updates performed on both the coordinate readout weights W and place field parameters X, ar. The agent learns to navigate to the reward location only by updating the actor critic weights, without influencing place field parameters Refer to App. for details. Hence, place fields in the MR agent will reorganize even in the absence of rewards. Right: Change in MR agent's coordinate estimation in 1D track across trials (T = 0, 10, 50000). Coordinate estimation was close zero during W WR initialization. After trial, the agent starts to show monotonic increase in coordinate estimation the agent moves from =1
environment. The coordinate readout is linear
1. By 50,000 trials, the agent's coordinate estimation becomes stable. (C) Average change for 16 and 64 place fields' size (firing rate greater than in the track) (top row) and center of mass (bottom row) when SR, RM and MR agents navigate in ID track with the absolute change reflected in the axis. Shaded area shows 95% CI over 5 seeds for agents with 16 and 64 place fields. (D) Spatial representation similarity matrix for SR (top row) and RM (middle row) and MR (bottom row) agents in ID track is visualized by taking the dot product of the place field activity each location. (E) Difference in correlation between the proportion of time spent in location between SR, RM, MR agents. (F) The correlation hetween the individual field firing learned by SR, RM, and MR agents rapidly diverge but remain positively correlated. (G) The correlation between the spatial representation similarity matrices (purple) leamnd by SR, RM and MR rapidly diverge in the early learning phase but stabilize and remain positively correlated in later phases.
28

---

A Model of Place Field Reorganization During Reward Maximization
A
(x)
B
(x
T=1000
T=1000
T=5000
T=5000
80300
19
T-10000
T=10000
073.00
T=50000
5000000
C
(x)
D
T=1000
0.3
T=5000
77:03
10
10'
18
101
10'
10"
T=10000
3391100
77.038
000
-0.03
-0.0075
T=49999
107
107
107
10
16
127
307
101
107
107
10'
77:0335
P.:T=
P 100
2500
P::T= 50000
0.02
0.0
0.01
0.0
-0.1
-0.2
-0.01
Py:T 9
Py:7= 100
T 2500
Py:T= 50000
0.02
0.2
0.0
-1
1
1
1
0
Figure S6. Field clongation in 2D arena. (A-C) 2D Place field distortion dynamics by SR (A), RM (B), and MR (C) agents as learning proceeds. Numbers in yellow the obstacle indicates (Field [D)-(Maximum firing rate). (D) Average change for 256 place fields' size (top row) and center of mass () (bottom row) when SR, RM and MR agents navigate in 2D arena with the absolute change reflected in the y axis. Area determined by computing the firing that was greater than 10 3 the arena. The 2D arena was divided into three sub-arcas to track COM movement !) away from the reward location, 2) the corridor from right to left, and 3) towards the start location. All three agents showed an increase field area and backward COM shift towards the start location. Shaded area shows 95% CI over 3 seeds. (E) Change in coordinate readout weights in 2D environment. Each plot indicates the synaptic weights W MR from place fields the (top row) and y dimensions explores the environment, the weights converge to reflect spatial map where the coordinate estimation for the X and Y axes increase monotonically when the agent moves left to right and bottom to top respectively, similar Foster et al. (2000); Kumar et al. (2024) which used a similar inth-integration TD error but with cligibility traces instead.
jt environment respectively. Weights were randomly initialized trial 0. the agent
29

---

A Model of Place Field Reorganization During Reward Maximization
A, 0
10
Stanserdd Devatoon
10a
Stanstrrd Devintoon
Stansard Devatton
Standard Devlatton
1.9
20-3
20-1
-0-4
10"
1012
10
10
10
10
10
10
10
10
10
10
10'7
10
10
10
10"
132
10
10"
10
10"
Cheene
A, a
0. a
A. 0. a
W,
Standard Deviston
Stadard Devietton
30
Standard Deviatton
30
Stadard Deviatton
20-2
101>
10-3
10
10
10
10
10"
10
10"2
Oueen
Wy
Wy
A, d, We. Wy
Strddrd Devatton
10a
Standrd Devirton
Stadord Deviatio
30
Rev
20-1
20-3
9
Ray/Ras
10-2
3072
10-2
Mapse
Cooes
Figure S7. Noise amplitude monotonically influences population noise with increasing magnitude [5x10-7, - 10* either in field parameters (a, A, σ) or Actor-Critic (W,, w.) influences the variance in Population Vector Correlation (Rev, blue). Spatial Representation Similarity which the product of field activity (RRs, orange) and cumulative discounted reward (G, green). Low variance of Rev and RRS indicates high correlation as learning progresses. Low variance in indicates stable performance. When increases before decreasing as noise amplitude increases, agent's navigation performance collapsed and the agent achieves 0 reward with low variance. high ratio of variance in population vector correlation and reward maximization behavior (Rrv/G; red) indicates that there is an optimal noise amplitude which causes high variance in population vector correlation (low PV correlation) while demonstrating stable performance. A similar analysis can be performed using representational similarity (Rev Ras, purple) to determine the optimal noise amplitude for high variance in population vector correlation but stable representation similarity seen in Qin al. (2023). Note that agents only optimizing for navigation behavior instead of representation similarity.
correlation and agent performance. Adding Gaussian
30

---

A Model of Place Field Reorganization During Reward Maximization
A
Onoise 0.0
0.0001
0.00025
Onoise 0.0005
Oraise 0.001
50
50
50
25
0
0
10
2x10
10
2x10
10a
2x10
10
2x10
10
2x10
B
0.2
Re0.54).
p=100
0.2
pi
Re-o o 286,
200
Rm 0 096.
0.3
Re0.156,
0.3
Varl0)
0.2
O
0.2
0.1
0.2
0.1
8
0.1
0.1
0.1
000
0.0
2000
10
0.0
1.5
1.0
1.5
2.5
5.0
7.5
0
50
75
Mean Amplitude
10000
Mean w? +Yw
6000
4000
0=10
p-130'
10000
p-030'9
p-10
p-0.012
4000
2000
2000
5000
10000
8
88
5000
0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0
2
0.0
2.5
5.0
7.5
0
25
50
75
Mean Amplitude
D
E
F
1.0
1.0
Active Fraction
1.0
Rev
0.5
Re5
0.5
0.5
0.0
0.0
0.0
2.5x104
10
2x105
2.5x10
10a
2x10
25000
100000
200000
T
T
G
2
1.0
% 1
0.5
-1
0.0
0
100000
200000
0
100000
200000
0
100000
200000
T
T
T
Figure S8. Influence of noisy fields on agent performance and field representation. (A) Reward maximization performance variability increases when noise magnitude increases. (B) With noise injection, variance in parameter update is initially positively correlated with field amplitude (bluc). When small amount of noise is added, fields with a larger mean amplitude show smaller variance in change in parameter while fields with smaller amplitude show higher variance. Conversely, when the magnitude of noise further increased (purple), fields with higher amplitude show higher variance in its parameters. (C) The correlation between mean amplitude and the magnitude the readout weights (sum over actions for squared actor weights and squared critic weights) is high and positively correlated when the noise magnitude is low. This correlation decreases and becomes weakly positive when Cnoisa 0.001. This supports the claim that in the low noise regime, fields with a high amplitude are more involved in policy learning and hence drift less or are more stable to maintain performance integrity. (D) Population vector correlation decreases at faster rate than the similarity matrix when noise magnitude increases. (E) Representation similarity correlation decreases as the noise magnitude increases, but at a slower rate than PV correlation. (F) Proportion of fields that are active (average fraction of fields with firing rate less than 0.05, 0.1,0.25) continues increase (G) Introducing Gaussian noise with zero mean and variance (0, 0.00025) to place field parameters caused each place field's center, firing rale and width fluctuate trials progressed. See App. E for details. This causes each ficld's spatial selectivity to change over time. Specifically, cach field's centroid (A) shifted from its initialized location, firing rates fluctuated (a²) causing fields to gain or lose selectivity, and most fields increased in size (a²) while some did not. The first two were observed by Qin et al. (2023) who analyzed Gonzalez et al. (2019). Each color corresponds to the dynamics of a specific field, with example fields shown.
with higher noise magnitude.
during updates 0++1
31

---

A Model of Place Field Reorganization During Reward Maximization
A
T= 25000
T= 75000
== 125000
T= 150000
T 195000
Fields sorted by
T=25000
Fields sorted by
T 125000
Fields sorted by
T= 195000
-1.0
1.0
-1.0
0.0
1.0
-1.0
1.0
-1.0
0.0
1.0
-1.0
1.0
X
X
B
Onoise
0.0
Onoise
0.0001
Onoise 0.00025
noose 0.0005
Onoise 0.001
0.75
0.75
Fraction
0.50
0.50
0.4
0.2
0.25
0.25
0.2
0.05
0.00
0.00
0.0
0.0
0.00
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
Centroid shift: A(AT) A(T) -AT 25000)
100
-T 500
AT 1000
2500
10-00
AT 50000
-7 100000
C
T= 0
T= 25000
T= 100000
T= 195000
Figure $9. Noisy place field parameter update replicates drift dynamics in neural data. (A) Place field centroids becomes distinctively different across trials, after stable navigation performance was attained at trial 25,000, similar to Ziy et al. (2013); de Snoo (2023). Each place field's centroid position was sorted according is added to place field parameters (a, X, o). place field optimization alone does not cause centroids to shift. Instead, adding small Gaussian noise (Trose E (0.0001, 0.00025, 0.0005)) replicates the gradual shift centroid position across trials (25,100 to 125,000) as seen Qin al. (2023); Ziv et al. (2013); Geva et al. (2023). When the noise magnitude is high e.g. Tnoise 2 0.001, centroids shift rapidly to a new location, similar to the random shuffie or null hypothesis seen Ziv et al. (2013); Qin et al. (2023): Geva al. (2023). (A-B) 10 agents initialized with different seeds have 640 fields in total. (C) Example
25,000. 125,000 and 195,000. (B)
no Gaussian noise
Analysis done for 64 place fields aggregated graph topology for one agent with N 64 place fields with Gaussian noise doose = 0.00025 added to field parameters. Each node indicates place field's centroid position across learning, and the edge is weighted by the normalized (between 0 1) cosine distance between each node that is less than 0.55. Red, green, blue, orange. black nodes indicate centroids initialized at the reward, start, end of track the reward, end of track the locations and the middle the track respectively. learning progressed, the cosine distance hetween each centraid changed and the ensemble representation rotated. Nevertheless, fields encoding the reward, start, and track were fairly stably as seen Gonzalez (2019), the greater separation of clusters support the phenomenon where a high density fields emerge at the reward and start locations.
32

---

A Model of Place Field Reorganization During Reward Maximization
A
N=4,=001 01
R=
N-32.-03
V- 64,0-00
42-
20
73-
Fand
02:
30-
10000
20004
10000
SCE30
10009
10830
2009
1050
14000
18900
10000
3009
43042
13500
30065
29000
10000
40030
30060
B
H=6.0-0:05
N d-
N 16.0
N- 22.0000000
K- 64.
43-
79
2000
20003
42081
3006
20400 19000
1.0.0
am0.05
f=0522.m00005
#=1206.om0p
R00d
AA.
A/. Ao. An
2004 1080
50003
10820
2000
14500
20000
X639
4201
2006
2000
C
Maximum cumulativv discounted
Maximum Reward
for pollcy convergarce
Policy Convergence
C.00
20000
C.06
10000
Redd 08
dar
dhh der dh de
0.0
Rhod
68
C.00
Figure S10. Influence field width and number fields on agent performance. (A) Fields initialized with 0.1 and (B) a = 0.05. Policy learning slower when initialized with smaller field width. (C) Influence of field parameter optimization maximum cumulative reward (left) and trial at which agent achieves cumulative discounted reward of 45 and above for the previous 300 trials (right). Correlation plot shows the p-value for pairwise t-test performed to determine the influence of fields parameters on learning performance.
the average
33

---

A Model of Place Field Reorganization During Reward Maximization
= == 0.025
N 8, o= 0.05
N 8, 0= 0.1
50
50
50
0-
0-
== 16,0 0.025
N 16.16..00 0.05
N 1= 0=0.1
0-
N 32, 0.025
N= 32, 0.05
N= 32. 0.1
50
50
50
N 64, a= 0.025
64,
0.1
N 128, a 0.025
N = 128, a 0.05
128, a- 0.1
50
50
50
0
N 256, 0.025
N = 256, 0.05
N 256, 0.1
50
50
50
50000 100000 150000
200000 250000
50000 100000 150000 200000 250000
50000 100000 150000
200000 250000
Figure SI1. Influence of noise target learning performance in 1D track. Increasing the number of place fields (N) and field widths (a) led to general increase in new target learning performance. When no noise was injected to field parameters (anoise 0.0, targets and to be stuck in local minima. Instead, noise magnitude of Jnaise 0.0005 allowed agents to maximize rewards throughout the 250,000 trials. Increasing the noise magnitude beyond this (Tnoise 0.001) negatively affected the agent's target learning performance, especially when the number of fields were low.
blue), most agents struggled learn navigate
34

---

A Model of Place Field Reorganization During Reward Maximization
A
[0.75, 0.75]
01.75, 0.75]
[0.75, 0.75]
[-0.55, 0.75]
40
20
0
25000
50000
75000
100000
125000
150000
175000 200000
T
B
Xabs 0-2 -1
Xobs
[-022002, -0.5,
Onoise 0.0
40
Onosse 0.0001
Onoise 0.00025
20
Cnoise 0.0005
Onoise 0.001
0
0
20000
40000
60000
80000
100000
T
Figure S12. Influence of noise on lcarning performance in 2D arena with an obstacle. (A) Agents started at the same location Pstart 0.75) and had navigate target that changed to a new location every 50,000 trials following the sequence (x, E [00.75, -0.75), (-0.75, 0.75), (0.75, 0.75), (-0.75, -.. Increasing the noise magnitude improved new target lcarning performance. Agents learned navigate target v,- (0.75 0.0) from
location Istart = (-0.75, 0.0) with an obstacle 0.5) for the first 50,000 trials. After which, the location of obstacle was shifted up to (Xmin 0.2, Zmaz 0.2, Mmii 0055, Ymuz 1.0) while start target location was the same. Agents with noise magnitude Onoise 0.00025 showed the highest average reward maximization performance followed by Onoise 0.0005. high noise magnitude (anaise 0.001) disrupted learning performance while agents without noisy field updates (Onoise 0.0) did not learn to navigate around the new obstacle. Note that field amplitudes and widths were clipped to be between 10 1 2] and [10 10055,00 0.5] respectively to ensure the Σ covariance matrix in 2D place fields remained valid for matrix inversion. Performance averaged
with coordinates (Imii -0.2, Imax 0.2, Mmin -1.0,
agents initialized with different number of 2D place fields (N (64, 144, 256, 576)) with the diagonals of the field width initialized with 0.01 and constant amplitude a = 1.0, over 30 different seeds. Shaded area is 95% CI.
35

---

A Model of Place Field Reorganization During Reward Maximization
A
N 128
N 256
N 512
0.6
GIDEICOM)
@EDE(COM)
0.6
@KDE(COM)
0.4
0.4
0.2
0.2
0.0
0.0
B
(x)
T=1000
000 1-
00 262
T=5000
0D 262
3333
T=50000 T=10000
56 TAT
00 P-268
3333
1TA 95
66 E-T
00 9-268
BE
Figure $13. Using the same learning rates the place field parameters and actor-critic recovers the same phenomena of high field density emerging at reward location followed by the start location, and field elongation against the agent's trajectory. (A) Each place field's amplitude, center width were sampled from a uniform distribution [0, 1]. [-1, model heterogeneous place field distribution. After learning, high density (number) of fields emerged the start (green dash) and reward (red area) location, similar to Fig. IB (right) and Fig. S2B. This phenomenon is consistent across different numbers place fields. Shaded area is 95% CI over 50 different seeds. (C) In 2D arena with obstacles, place fields elongate from the reward location (red circle) back to start location (green circle), while narrowing along the corridor with an obstacle (gray), similar to Fig. 2F. Learning rates for the actor, critic and place field parameters y ne 0.0005.
1, 1]. [10 3 .00.1] respectively
36

---

A Model of Place Field Reorganization During Reward Maximization
A
T=0
T= 10000
0.5
0.0
-1
0
1
-1.0
-0.5
0.0
0.5
1.0
X
B
N = 16
=
64
1.0
1.0
0.1
gKDE(COM)
0.2
M
0.5
0.5
0.0
0.0
0.0
0.546
0.586
0.502
0.52
Figure S14. Center-surround place fields reproduces the emergence of a high density of fields at the reward location. (A) Example of 16 center-surround fields uniformly distributed before (left) and after learning for 10,000 trials (right), with the learning rates for the center-surround place field parameters and policy network heing the same (n reward location while others elongated from the reward location back the location similar to Fig. 2C (bottom row). (B) A high field density (gKDE(COM)) and mean firing rate C ((z)) emerged at the reward location for N 16 (left) and N 64 (right) when using center-surrount fields. However, do not high density emerging the start location robustly. Further analysis needed verify if the representations learned by Gaussian basis functions and center senter-surround fields (difference of Gaussians) are similar, and if not seeds.
= 0.001). Place fields near the reward shifted to the
why. Shaded area 95%
37

---

A Model of Place Field Reorganization During Reward Maximization
A
N 4
N 8
N 16
N 32
Roed
Reed
50
Raed
50
Rxed
Actor
CHICC
40
Actir
CHIS
40
Actor
CTRC
40
Actor
CHtic
Hetric
RM
Metric
RM
30
RM
Hetric
RM
Hetric
20
RMM
20
HH
RM++M
RM
10
10
0
10
10f
10
10
102
10
103
20f
T
T
T
N 64
N = 128
N = 256
N - 512
50
50
50
40
40
Reed
Reed
Reed
Reed
Actor
10 30
Actor
30
Oftic
Cntic
Crttic
20
20
Metric
10
10
10
10
10
10
10
10
10a
10
B
N=
N 8
N 12
100
Reed
Raed
Reed
60
Actor
Cricc
75
Actor
CIDC
Actor
Citic
Metri
Netric
Metric
6
40
AC
30
50
MB
20
25
25
0-
0
10a
10f
10a
10
T
N 16
N= 20
N = 24
100
100
75
75
80
50
Actor
50
25
25
20
0-
10
10f
103
10
10a 10f
T
Figure S15. Difference in policy convergence when backpropagating temporal difference error to optimize place field parameters. We evaluated the speed of policy learning when optimizing (A) heterogencously distributed place field population in the ID track and (B) homogeneously distributed place field population in the 2D arena using: (1) fixed place field parameters (blue). (2) backpropagating the TD error 6, through the actor weights WT De (B_ 1, B. = 0, 8MAR R O, orange), (3) backpropagating the TD error through the critic weights w" (3, 0, 3. 1, 3,5 O, green). (4) backpropagating the path integration derived TD error X through the metric representation weights WMR (3,- = 0, 3,, = 0, Byr n = 1, red) while learning the value function and policy by optimizing only the readout critic and actor weights, (5) backpropagating the TD error through both the actor and critic weights, otherwise called the Reward Maximization agent (3g- = 1, Bg = 1, BMR = O, purple), (6) backpropagating the TD error through both the actor and critic weights and the path integration based TD error through the metric representation weights, (3x = 1, = 1, BMR = 1, brown). The combined RM+MR objective used for place field parameter optimization achieved the fastest policy learning. similar to Stachenfeld et al. (2017) when the number of fields was low (N = (4, 8, 16, 32) in ID and N = (4, 8) in 2D). With more fields, the reward maximization agent (RM,, purple) was almost as effective as the combined objective (RM + R, brown). Optimizing place field parameters using only the actor weights led to the slowest policy convergence (orange), nevertheless faster than using fixed place fields. The same learning rates were used when the number of fields were increased. Hence, tuning learning rates should improve the stability of policy learning, especially in the 2D arena for the agent with the combined RM+MR objective. Shaded area indicates 95% CI over 50 random seeds.
38


Reviews:
[{"review_id": "ZKopsVvt5h", "reviewer": "Reviewer_Qrmv", "summary": "This paper proposes a model that is inspired by place fields in the hippocampus and how it could be used to develop representations that can be used for reinforcement learning. The authors argued that their model aligns with phenomena observed in neuroscience experiments, specifically high density of activity around reward locations, elongation of representations from paths taken by the agents as well as stable policy learning despite representational drift. The authors evaluated their hypothesis on simple 1D linear track and 2D environments, which are commonly used in computational neuroscience studies. Experiments on different targets as different tasks were also used to study the effects of place field updates due to these changes. Analyses were done to determine where learning the parameters of the place field were better or worse than keeping them fixed. Overall, the paper contributed a model that shows potential on how place field like representations can be using for reinforcement learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We hope our response clarifies most of the concerns. \n \n>The writing of the paper is very dense\n\nWe will reduce the density to keep the description (e.g. parameter choices in methods, description of path integration based TD error in section 4.2, and figure captions) to a minimum and shift supplementary figures to the main paper. Additionally, we will get another page for the final draft. \n \n> figures are pixelated\n\nWe will use vector-based graphics e.g. .eps instead of .png for the revision! \n\n> is it surprising that the resulting representations are transformed Gaussians \n\nYes, we agree that the paper is more to advance the field of computational neuroscience and have made this edit “This paper presents work whose goal is to advance the field of Computational Neuroscience.” Transformed Gaussians are expected, but it is surprising that the TD objective is sufficient to capture the 3 different phenomena, that have been described by 3 disparate mechanisms i.e. (1) high field density is needed at salient stimulus (e.g. reward location) to maximize fisher information (Ganguli & Simoncelli 2014), (2) predictive coding of future location shows fields elongating and shifting backwards while enveloping obstacles (Stachenfeld et al. 2017) and (3) optimizing noisy parameters shows stable population code while individual place fields drift (Qin et al. 2023). Hence, it is not at all expected that the Gaussian basis function transformations seen in our model recapitulates all 3 phenomena including that shown by the SR algorithm.\n \n >phenomena…using pixel or other high-dimensional observations\n\n Yes, this is a good question. Fang & Stachenfeld 2024 showed that optimizing deep representations using the metric representation-like objective recapitulates the backward shift of features. Nevertheless, whether representations in deep networks follow the dynamics observed in the current model is part of our future work. With respect to the current paper, we believe that this is a different question from the original goal, which was to analytically study how representation learning evolves in predefined place cell descriptions. \n \n> environments are also very simple \n\nWe used simple tasks to match the environmental setup used in experiments so as to study and predict place field representation dynamics. We explored an elevated level of task complexity by changing the target or obstacle location to serve as predictions of place field representations to test in experiments. Nevertheless, it could be possible to have different place field behavior in different or more complex environments. As a follow up, we are working on: multiple rewards (Lee et al. 2020 Cell), uncertainty in reward distribution (Tessarau et al. 2024 bioRxiv) and, sequential two alternative forced choice (Yaghoubi et al. 2024 bioRxiv). We appreciate other complexities to explore based on the reviewer’s recommendation.\n\n \n> 1.Why SR fields decrease at the later stage? \n\n Fig. 2C shows how the individual place fields change when optimizing the parameters using the SR algorithm. In the early phases of learning, place fields at the start location (purple) increase in size since the agent spends a higher amount of time at the start doing random walks (black). As the agent learns to spend a higher amount of time at the reward location, the fields at the start location decreases in size while those at the reward location increases. However, the rate of decrease in SR field sizes at the start location (purple) is more significant than the rate of increase in field sizes at the reward (green), resulting in a slight mean decrease in field size in Fig. 2A. \n \n> 2.Why is the representation similarity matrix visualization different from the others? \n\nThis was an outlier example when the similarity matrix at T=75000 was slightly different from the other time points. But the population code becomes similar again as optimization continues. Fig. 3D shows the similarity matrix autocorrelation for the plots in Fig. 3B with $\\sigma_{noise}=0.0001$ remaining largely stable (orange) with small deviations. Subtle differences in representational similarity matrices have also been observed in experimental data (Fig. 5F, Qin et al. 2023). \n  \n> 3.Elongation of fields by SR being subtle \n\nIncreasing the SR discount factor (Eq. 71) to 0.95, 0.99, 0.999, 0.9999 led to a faster increase in successor field magnitudes rather than the field width. Yet, the increase in SR field width was still significantly smaller than the field elongation observed when optimizing using the TD error. The SR fields ($\\psi$) in our models are anchored to fixed place fields ($\\phi$ Eq. 72) and the SR fields learn to represent the transition probability between each of these fixed place fields. Increasing the distance between the fixed place fields ($\\phi$) or reducing their width could show a faster increase in the SR width compared to the increase in SR magnitudes.", "claims_and_evidence": "Yes, there were supported by the experiments and analyses described in the paper.", "methods_and_evaluation_criteria": "The proposed methods and evaluation makes sense to evaluate the hypotheses proposed by the authors.", "theoretical_claims": "While there aren't any mathematical proofs included in the main paper, I have looked through the math equations in the main paper. To the best of my knowledge, they seemed correct.", "experimental_designs_or_analyses": "Yes, I checked the experimental designs with are focused on simple 2D mazes. The inputs to the agents are place fields which are pre-determined using Gaussian distributions. Despite having multiple targets as different tasks, the structure of the environment remains fixed.", "relation_to_broader_scientific_literature": "There is a huge body of work look at place cells-like representations for reinforcement learning, particularly in the computational neuroscience field. This paper aligned well with many of such studies where the basis features are deemed to be gaussian-like, hence are pre-defined to be Gaussian distributions. Many of the related studies as the authors have included as their references also mainly study the efficiency and efficacy using simple navigation tasks.", "essential_references_not_discussed": "I believe that the authors have cited relevant references to the best of my knowledge.", "other_strengths_and_weaknesses": "# Strength\n1. The literature review is thorough and extensive. This helps the reader to understand well about the work done by the authors relate to broader field relating to representations inspired of place cells and reinforcement learning. \n2. Lots of analysis and ablation studies performed to convince the readers that the model proposed fulfilled the three criteria, which I will quote myself earlier: A) high density of activity around reward locations, B) elongation of representations from paths taken by the agents as well as C) stable policy learning despite representational drift. \n3. The math equations were presently clearly in both the main paper and the supplementary section. Details about the baseline models such as Successor Representation and Metric Representation were also provided. \n4. The captions of the figures provide clear descriptions of what the plots are showing. \n\n# Weakness\n1. The writing of the paper is very dense. There is a lot of information packed in the main paper, with constant and important references made to the supplementary section. It is clear that the authors have done a lot of work and are trying to pack the main paper as much as they can but this makes it harder for the reader to follow along. \n2. Many of the figures are pixelated if you zoom in or use a monitor to read the paper. I highly recommend using vector based graphics for your plots and figures for better visualisations and readability. \n3. Only simple environment with pre-defined features were considered. There is no evidence that this model can handle complex tasks and environments, making the proposed model somewhat toy-ish.", "other_comments_or_suggestions": "I feel that this paper is more suitable to be deemed as computation neuroscience contribution than wha the authors claim to be in their impact statement: \"Advance the field of Machine Learning.\" The reasons being that inputs are simple, pre-determined Gaussian features. Therefore, is it surprising that the resulting representations are transformed Gaussians as seen in Figure 2F?  \n\nSecondly, It is unclear if the phenomena observed could also be realised using pixel or other high-dimensional observations. \n\nThirdly, the environments are also very simple, ranging for 1D linear track to 2D maze. Would the results also hold or perhaps a different set of conclusion can potentially occur when the environment changes as the target changes?", "questions_for_authors": "I would be happy to hear from the authors regarding the points I made in the Weakness and the comments sections. Depending on how the discussions and rebuttal proceeds with other reviewers and myself, I would be open to increasing my score.  Here are some further questions: \n\n1. Figure 2A. Why does the SR field size decreases at the later stage? \n\n2. Figure 3B, second column where T = 75000. Why is the representation similarity matrix visualisation vastly different from the others in B? \n\n3. Line 318 in the left column. There is a statement on elongation of fields by SR being subtle. Why is this the case? Is this due to the discount factor? Would a different or higher discount factor in Eq. 71 leads to stronger elongation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAzs8eZxD4", "reviewer": "Reviewer_iexX", "summary": "In this work, the Authors develop a reinforcement-learning model of the place field organization. They consider three effects that have been observed regarding the place fields in biology (a higher density at reward locations, an elongation backwards along the trajectory, and a drift observed at the times of stable behavior). The authors show that their model reproduces these three effects.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We hope our response clarifies most of the concerns.\n\n>place fields in biology are organized similarly\n\nWe wanted to ask if a single, simple normative model can recapitulate several learning-induced changes in place field representations. We agree that there could be other mechanisms that elicit the same phenomena, which we briefly explored. Still, we think the proposed model is the most parsimonious in explaining the learning dynamics observed in place fields when animals are learning to navigate. \n\nWe do explore how other objectives also elicit similar representations in the subsequent section: i.e. path integration TD error (MR - metric representation) and state prediction (SR - successor representation) objectives. While the representations learned by these 3 different objectives could be similar (Fig. 2A,B), the dynamics of how place field representations change is different (Fig 2C,D,E), making this a prediction of our proposed model. \n\n>generality of the model \n\nWe would like to clarify that our single proposed model (noisy Gaussian basis function parameters modified by the TD error - Noise+TD) replicates all 3 phenomena without ad hoc modifications. Perhaps this was unclear, which we clarify below. \n\n>first model is formulated\n\nWhile our partial model (TD) was explained first (Fig. 1), our full proposed model (Noise+TD) recapitulates high field density at targets (Fig. 3B, Fig. S3A,B). \n\n>successor representation agents and metric representation agents are introduced \n\nWe clarify that the successor representation (SR) and metric representation (MR) agents were introduced as alternative comparisons to our proposed agent (Noise+TD) where field parameters were optimized only by the TD error, not by the SR and MR objectives. Both partial (Fig. 2) and full Noise+TD (Fig. 3B) agents show field elongation against the trajectory when all the place field parameters are optimized.\n\n>noise is introduced \n\nThe stochasticity in the partial TD model was insufficient for representational drift (Fig. S7). Hence, our full model Noise+TD includes noisy parameter updates. To reiterate, adding noise to place field parameters did not detract the model from demonstrating (1) high field density at the reward location (Fig, 3B and Fig. S3), and (2) field elongation (Fig. 3B). \n\nHence, our full proposed model (Noise+TD) replicates all 3 neural phenomena, without needing ad-hoc modifications. We hope this clarification addresses the reviewer’s main concern about the model’s generality, and we will clarify this in the manuscript.\n\n>consideration of alternative plausible models\n\nThe SR and MR objectives are alternative plausible models to show that different objectives can recapitulate the field elongation while the representation dynamics are different (Fig. 2E), making this a prediction of our model.\n\n>design choices (successor representations, noise) are not evaluated\n\nWe would like to clarify that we did evaluate various design choices. SR does not influence policy learning, as the policy influences SR fields instead. Hence, the influence of SR fields on policy learning was not evaluated. Instead, we explored the influence of MR objective in policy learning. Fig. S15 shows that optimizing place field parameters using MR improved policy convergence (red) compared to using fixed place fields (blue). However, the rate of improvement was not as significant compared to optimizing place fields using the TD objective (purple). Using MR as an auxiliary objective (brown) to reward maximization (TD error) showed a slightly faster policy convergence, consistent with Fang & Stachenfeld (2024). We described this result in lines 372-375 right column.\n\nWe refer the reviewer to Fig. 4C, Fig. S11 and Fig. S12 which show the functional role of noise in policy learning when the target shifts or obstacle location changes. Specifically, when the target consistently shifts to a new location, partial TD agents without noise (blue) fail to learn the new target locations, suggesting they are trapped in a local minima. Instead, Noise+TD agents can continually learn the newly shifted targets. Hence, noisy place field representations increase the agent’s flexibility to learn new targets. \n\nTo conclude, our full proposed  Noise+TD model replicates all 3 neural phenomena and demonstrates faster policy convergence when the task structure changes. \n\n >better informed by the literature\n\nWe refer the reviewer to the specific parts of the manuscript (Sec. 2 & 5) where we believe a discussion of the relevant literature has been included. It would be helpful if the reviewer could share other literature we missed to add as further discussion.\n\n>mapping of the actor-critic\n\nLines 68 to 84 left column\n\n>hippocampus and entorhinal cortex\n\nHippocampus: Lines 80 to 83 left column\nEntorhinal cortex: lines 84-87 right column, 393-394 right column\n\n>representation drift\n\nLines 66-82 right column, 434-403 left column", "claims_and_evidence": "While the model reproduces the three effects observed in the place cells in biology (and that is the claim put forward in the paper), there is no reason to believe that the opposite is true, i.e., that the place fields in biology are organized similarly to the proposed model. While the overall direction of the paper is interesting and, with the proposed set of tools, it could be possible to test the argument of biological relevance, this has not been done in this paper, sadly limiting its impact. I elaborate in the sections below.", "methods_and_evaluation_criteria": "While it is generally a good idea to consider biological observations and see whether a model reproduces them, several considerations should be typically put in place including the generality of the model (i.e. the same model should reproduce all the observations) and the consideration of alternative plausible models (i.e. the models that look equally plausible based on prior research but do not reproduce the observed phenomena). What I found problematic with the methods in this work is that the model is modified ad hoc: for the three phenomena observed in the place fields, there are three corresponding modifications of the model: the first model is formulated to reproduce the higher density of the place fields at the reward locations; then successor representation agents and metric representation agents are introduced to reproduce the elongation of the place fields backwards along the trajectory, and, finally, noise is introduces to reproduce the representation drift. This strategy raises the question of the proposed model's generality. Then, while ablation experiments are provided (which is a good thing), they only consider the properties of the place fields (i.e. the centers, widths, and amplitudes) while the aforementioned design choices (successor representations, noise) are not evaluated. Thus, while the model indeed reproduces the biological observations, there’s no evidence that biological place fields form in accordance with this model.", "theoretical_claims": "While the theoretical claims seem correct (as in: the model reproduces the said effects) and are confirmed by the simulations, this does not address the model’s applicability issue as raised above.", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria section above.", "relation_to_broader_scientific_literature": "While a lot of relevant literature is cited, the model design choices here, as well as the comparison with baseline models, could be better informed by the literature. Specifically, there’s substantial literature on the neuroanatomy of the reward circuit including the mapping of the actor-critic model that could be used / discussed. Additionally, there’s substantial literature on the place fields, including in works considering the hippocampus and the entorhinal cortex. These works could be discussed here and compared as to whether they reproduce the same three effects. Finally, there’s vast literature on the representation drift; whole conferences are held on that topic. This literature could also be considered and discussed here. Overall, considering and discussing this literature in the follow-up work on this project may make it a much stronger and more well-founded contribution than the current version.", "essential_references_not_discussed": "See Relation to Broader Scientific Literature section above.", "other_strengths_and_weaknesses": "The text is written clearly; the ideas and the scope of work are well-articulated, making it easy to read and review.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A\n\n_____________________________\nPost-rebuttal update: while I don't believe that my concerns have been addressed, they are mostly related to the significance of the results, which is subjective. Upon that consideration, I raise my score by one step.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1PMq068qaE", "reviewer": "Reviewer_iZMo", "summary": "This paper develops a reinforcement learning (RL) model to explain how hippocampal place fields reorganize during reward-based navigation. In the model, Gaussian radial basis functions (place fields) receive continuous spatial inputs, and feed into an actor-critic framework that learns to navigate in 1D and 2D environments. At each time step, the temporal difference (TD) error modulates both the actor-critic synapses and the place field parameters (amplitude, center, and width). Through online updates, the model captures three key experimental observations about place fields: (1) increased density at the reward location, (2) backward elongation against the movement direction, and (3) continuous drift of individual place fields even when behavior stabilizes. The authors show that place field reorganization under TD error significantly improves policy convergence by providing more discriminative spatial representations. Perturbative analysis clarifies why fields near high-value locations experience stronger shifts. Additionally, introducing noise to place field parameter updates leads to representational drift without disrupting navigational performance, and in fact aids adaptation to new reward targets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We hope our response clarifies most of the concerns.\n\n>proposed model will highly rely on reward. \n\nWhile the current model proposes a reward dependent objective, we have also proposed a non-reward-dependent objective (Metric Representation) which recapitulates place field elongation against the trajectory, but not a high density at the target (Fig. 2C). Optimizing place fields either using the MR objective as a standalone or as an auxiliary improves the policy convergence rate (Fig. S15, red and brown) compared to fixed representations (blue), consistent with Fang & Stachenfeld 2024.  \n\n>Tasks can be too simple. \n\nWe kindly refer the reviewer to our response to Reviewer 1 (Qrmv) that “the environments are also very simple”.\n\n>biologically implausible.\n\nThe question we were pursuing was whether there is a single, simple normative model that can recapitulate several learning-induced changes in place field representations. To do this, we felt some level of biological unrealism was permissible. Nevertheless, the reviewer is correct. We raised this biological implausibility as a limitation and have discussed avenues to make it plausible (e.g. random feedback with local learning rules). We are currently working on a biologically plausible representation learning model and preliminary results are promising. \n\n>1.remapping when reward has been changed?\n\nYes, when the target changes, some place cells that were coding for the initial reward location shift to the new reward location (Fig. S3), replicating the remapping phenomenon (Gautheir et al. 2018). Additionally, place fields that were not initially coding for reward but in the vicinity of the new reward location were recruited to code for the new target. Hence, the model does support partial remapping when the target changes. However, we do not see (1) the same proportion of reward place cells coding for the new target, and (2) reward coding place fields shift gradually to the new target location, whereas we could expect place fields to rapidly shift or jump to the new target location (Krishnan et al. 2022). The timescales for reward based remapping needs additional experimental data for verification.\n \n> 2.comparison between your RM model and other RL models? \n\nWe will add comparison to other RL models to the discussion namely in 3 aspects:\n\nReward maximization (RM) algorithms (TD vs Q): Our RM model maximizes rewards by optimizing the place field representations for policy and value estimation. Other RL (e.g. Q-learning or SARSA) algorithms seek to maximize cumulative discounted rewards. Due to the reward dependency in these RL objectives, we expect other reward-maximizing RL algorithms to learn similar representations to our actor-critic based model. \n\nArchitecture (GBF vs MLP): Deep RL models that use MLPs for representation learning have also shown high density at reward location and backward shift of features (Fang & Stachenfeld 2024 ICLR). Adding Gaussian noise to deep network parameters also elicit a form of representational drift (Aitken et al. 2022). Although we constrained place field tuning curves to Gaussian distributions to analytically study representation learning, we believe insights from these analyses can be translatable to deep RL models. \n\nLearning objectives (TD vs MR vs SR): The SR model learns transition probabilities based on the agent’s policy. Meaning, when the policy changes, the transition probabilities change, and the SR fields will subsequently change. Hence, SR fields do not influence policy learning, making this model inadequate to study representation learning for policy learning. If the agent has a reward maximizing policy, then the SR model recapitulates high field density at the reward location and field elongation against the trajectory (Fig. 2). Hence, the SR model seems to require 2 disparate components, making the SR model less parsimonious. Furthermore, the SR model is not a good candidate to study representational drift as SR fields have to be anchored to fixed representations (Eq. 72). Meaning, we should not expect SR field centroids to drift to a new location as observed in neural data (Fig. 3D Ziv et al. 2013; Fig. 5H Qin et al. 2023). Conversely, the MR agent is a reward independent objective (Foster et al. 2000) and learns to self-localize by path integration. Given its non-dependency on rewards, place fields do not reorganize to show a high field density at targets (Fig. 2C), though they cause fields to increase in size and shift backwards against the trajectory (Fig. 2A,B). We did not study representational drift using the MR objective in this paper, though this could be a future direction. \n\nWe hope these discussions would suffice on why the proposed model (Noisy GBF optimized by TD error) is a suitable candidate while being parsimonious and anatomically grounded to the biological neural circuits. We kindly request the reviewer to reconsider the score.", "claims_and_evidence": "The authors' major claim is their model can replicate three different phinomina observed in Place cells, which has been demonstrated by their simulation.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "I specifically looked at the perturbative expansions in the paper’s Appendix and found no apparent incorrectness.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "The authors situate their model in the intersection of hippocampal research and RL, demonstrating a reward-based alternative to SR or purely mechanistic place field accounts, while also addressing representational drift seen in modern neural data.", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "- Strength\n      - Unified three phenomina: clustering, elongating and drifting in one model.\n      - Provided a computational goal (gaining reward) for the hippocampus to form space representation.\n- Weakness\n      - The learning of the proposed model will highly rely on reward, which is not true for humans and animals. \n      - Tasks can be too simple.\n      - Directly applying RL algorithm on place cell via backpropogation can be biologically unplausible.", "other_comments_or_suggestions": "no", "questions_for_authors": "1. Does your model support any type of remapping when reward has been changed?\n2. Could you elaborate more on the comparison between your RM model and other RL models? Especially the reason why this specific algoritm will be able to explain all three phinomina while the others can not?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a reward maximization framework using temporal difference error to drive place field reorganization to improve policy learning. The model could reproduce three phenomena observed in place cells, which is a strength of the paper from a computational neuroscience perspective. Most reviewers think the paper has sufficient literature review and has done sufficient analysis and ablation studies. Although one reviewer was concerned about the experimental verification of the proposed framework, he eventually recognized the merit of the proposed testable predictions after rebuttal (I highly recommend reading the author-reviewers’ conversations). Combined, I recommend accepting this paper.
Decision: Accept (poster)