A Model of Place Field Reorganization During Reward Maximization

Authors: M Ganesh Kumar, Blake Bordelon, Jacob A Zavatone-Veth, Cengiz Pehlevan
Primary area: applications to neuroscience & cognitive science
Keywords: Place field, Reinforcement Learning, Successor representation, Representation drift

Abstract:
When rodents learn to navigate in a novel environment, a high density of place fields emerge at reward locations, fields elongate against the trajectory, and individual fields change spatial selectivity while demonstrating stable behavior. Why place fields demonstrate these characteristic phenomena during learning remains elusive. We develop a normative framework using a reward maximization objective, whereby the temporal difference (TD) error drives place field reorganization to improve policy learning. Place fields are modelled using Gaussian radial basis functions to represent states in an environment, and directly synapse to an actor-critic for policy learning. Each field's amplitude, center and width, as well as downstream weights, are updated online at each time step to maximize cumulative reward. We demonstrate that this framework unifies the three disparate phenomena observed in navigation experiments. Furthermore, we show that these place field phenomena improves policy convergence when learning to navigate to a single target and relearning multiple new targets. To conclude, we develop a normative model that recapitulates several aspects of hippocampal place field learning dynamics and unifies mechanisms to offer testable predictions for future experiments.

Decision: reject
Reviewer scores: 5, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730566009273

thread_type:
official_review

<thread_object>
creator: reviewer_lqgu

<content>
summary:
This paper presents a computational model for hippocampal place field reorganization during reward-based navigation learning. The model integrates reinforcement learning via actor-critic with place fields represented through Gaussian radial basis functions. The learning system is capable to create high-density place fields near reward locations, field elongation along the trajectory, and drift in place fields. The TD signal is used to tune field properties such as amplitude, centre, and width. As in similar other studies, the use of noise helps learning and generalization.

strengths:
- The paper is clearly written and well structured
- The paper makes an effort to provide a unified theory for multiple place field phenomena
- The integration of the actor-critic RL system with pace fields is biologically relevant and interesting
- The model is compared against the well known successor representation model
- The link between a normative approach via reward maximisation and the emergence of place field representation could be valuable to interpret biological dynamics. 
- The use of noise to modulate the learning flexibility is interesting and potentially insightful for neuroscience
- The results are provided with 10 seeds runs and confidence intervals

weaknesses:
- While the contributions are clear, it is less clear exactly in which why this model advances previous models and what we now understand better. I am also generally struggling with words such as 'recapitulate' that don't seem to mean much to me; do the authors means to say that this models reproduces for the first time the biological neural dynamics observed in the hippocampus? If so, it would be better to be more explicit. 

- The authors mentioned very briefly about the use of backpropagation and the future requirement for a biologically plausible rule. I feel that this part is not sufficiently expanded. Can the author speculate whether the model would work at all with alternatives, e.g. STDP-like rules? And if so, how would that affect the model and the ability to reproduce biological neural dynamics? 

- How foes noise in the model relate to biological systems? While there are a few relevant papers that are cited in that respect, it would useful to have a motivation that is more grounded in biological dynamics. E.g. it is known that neuromodulatory activity could be regulating exploratory or exploitative behavior, possibly affecting neural dynamics with different level of noise or signal to noise ratio. 

- It's not clear to me how the model would work without rewards, as this is the case in biology where place cells and place fields develop also in the absence of reward. It would good to have at least some indications on how the model could be adapted to work without an explicit reward. 

- Maybe I missed it, but the supplementary information says "Refer to the Github code repository for
implementation details." but I didn't find the link to it.

questions:
The list of weaknesses contains questions. The authors are encouraged to consider them and possibly update the paper to account for this points and provide responses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658668417

thread_type:
official_review

<thread_object>
creator: reviewer_j18n

<content>
summary:
This paper develops a reinforcement learning model that uses tunable place fields to simulate learning dynamics observed in rodent navigation. The model represents place fields as Gaussian radial basis functions connected to an actor-critic RL framework, updating field parameters to maximize accumulated rewards based on temporal difference error. The model replicates key experimental findings: increased place field density near rewards, field expansion along navigational trajectories, and field drift that doesn’t disrupt performance. The model suggests a potential framework for studying place fields learning dynamics, offering predictions for future hippocampal experiments.

strengths:
Explaining the emergence of three observations about place field reorganization in a normative approach. 

Comparing with competitive model, such as successor representation.

Ablation study to understand contribution of different place field parameters to policy learning.

Well written, well organized.

weaknesses:
The learning rate for place field parameters are 100 times slower than the actor and critic weight parameters. This separation of learning process by implementing different learning rates is not biologically grounded. Are there any potential biological mechanisms that might support different timescales of plasticity. 

The pace fields has been found to have more complex shapes, rather than simple gaussian shape. For example, the center-surround structure in the place field (which could be modeled as difference of gaussians) was shown to have important functional role (Sorscher et al 2022 Neuron). It is not clear how using more complex place field shapes might affect the model's results or conclusions. 

The number of place fields appears to be a crucial hyperparameter, as the model’s resemblance to experimental data is observed only within a specific range of values for this parameter. The authors examined the effect of the number of place fields on model performance in Supplemental Figures 6 and 7. However, it is important to explore methods to enhance the model’s robustness to variations in this parameter, which is relevant to the model's potential scalability to larger environmental spaces.

Randomness in field parameters, rather than in actor/critic weights, appears essential for the RNN to replicate field drifting and perform tasks with changing targets. This distinction highlights two types of noise in the RNN, each with distinct functional roles; however, a discussion on the possible biological origins of this noise and its functional distinctions is lacking. In the current work, field drifting is modeled by manually injecting noise into field parameters. Could the ring attractor mechanism identified in Sorscher et al. (2022, Neuron) provide a more mechanistic interpretation of field drifting?

questions:
The paper takes a normal approach, using gaussian radial basis function to model place fields. There are multiple works that modeling place field using RNN without specific place field parameter assumptions, but taking them as emergent features. Could the place field reorganization be explained in such approach? 

Could the authors consider incorporating center-surround structures or other non-Gaussian shapes and discuss how this might impact their findings?

The learning rate for place field parameters and the actor/critic weight parameters possess distinct time scales.  Could the authors explore how varying the learning rate ratio affects the model's behavior and ability to replicate experimental findings. Could the model still work after removing this assumption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731173651583

thread_type:
official_review

<thread_object>
creator: reviewer_ifil

<content>
summary:
This paper models the reorganisation of place fields on linear track by backpropagating the reward-based TD error gradient to the parameters of Gaussian place field basis features. The methods are sound and paper is presented well, however I have reservations about the conclusions drawn, particularly the links to changes in field density, neural drift and the reliance on a reward signal for the learning - which are at odds with certain aspects of the experimental hippocampal literature which the model is trying to explain

strengths:
The paper provides and executes a novel idea with the intention of explaining certain aspects of the hippocampal literature. The observed backwards skewing of place fields, opposite to the direction of motion, provides an alternative account of place field skews (Mehta et al 2000), which so far has mainly been explained by predictive mechanisms (e.g. Mehta et al 2000) like the successor representation (Stachenfeld et al 2017).

weaknesses:
The model claims to find increased place field density at reward locations - however if the definition of field density from the experimental literature was used, the authors would find their the model actually yields the opposite results to what is observed in the experimental literature. To be specific, Gauthier and Tank 2018 find increased density of field centre of masses - i.e. fields are more numerous at goal locations, however what the authors show is mean activity of place firing rate is greater at at the goal location, which is caused by a sparse number of place fields at the goal location with incredibly high firing rate (in order to create a ramping value toward the goal location). In fact, the majority of the place field centre of masses are shown to expand and skew backwards down the track, opposite to the direction of motion (as reported in the next result) thus if the authors used the density of field centre of masses - as the experimental literature does - they would actually find an increased field density in the locations on the track preceding the goal location, and a decreased field density at the goal.

The authors also claim the model captures observed neural drift of hippocampal place cells - this is quite a stretch. Place cells have been shown to be stable for very long periods of time - up to 153 days (Thompson & Best 1990). Indeed, reward expectation has actually been shown to decrease neural drift, opposite to the claims of the authors here (Krishnan & Sheffield 2023). Essentially what the the authors show is drift in the population vector over time - which is observed in the hippocampus - however the authors' drift is because place fields actually drift around the track, which is not observed in real life - place fields are generally stable. In fact, it is likely hippocampal neurogenesis, which is heavily triggered by exercise, is responsible for driving some features of the observed representational drift (Snoo et al 2023). 

Finally the comparison to successor representation (SR) is apt, however a favourable quality of the SR model is it is not dependent on an external reward signal to trigger these changes, and thus can explain phenomena such as latent learning when no reward is present. In fact the model presented here implies experimentalists would observe huge differences in hippocampus based on whether or not reward is present, when if fact periods of exploration in the absence of reward only improves future reward based learning (Tolman 1948)

questions:
Based on the above, I would recommend the authors:
1. Use the same measures on their data that the experimental results they are attempting to explain use
2. Consider dropping the neural drift part of the results, as the underlaying mechanism in the model is at odds with many aspects of the hippocampal literature
3. Consider focussing on a set of experimental results better suited to the model e.g. the Dupret et al 2010 cheeseboard task
</content>
</thread_object>

</thread>
<thread>
creation date:
1731193109644

thread_type:
official_review

<thread_object>
creator: reviewer_zibp

<content>
summary:
This paper uses a representation learning perspective to understand how the reorganization of place fields under experience and reinforcement supports RL. The authors aim to recapitulate three experimentally observed phenomena: the timeline of dense place fields emerging at rewarded locations, lengthening of fields against the direction of motion, and stable performance amidst place field drift. They also include a number of additional experiments and analyses characterizing features of the model.

strengths:
- Overall it's a nice approach to relating hippocampal activity to downstream RL computations. Also seems like a very plausible constraint on neural activity. The model is minimally complex while still allowing insight on representation learning
- Model is capturing some phenomena in the data very nicely.
- In particular, the model is capturing the robustness against drift and modeling potential normative benefits of drift is a nice contribution -- this has been hypothesized in a number of places (eg Driscoll, Duncker et al 2022) and is well-demonstrated in the deep learning context (as the paper cites), but I don't know of any computational work demonstrating it in the context of a hippocampal navigation model.

weaknesses:
Weaknesses

- unclear comparisons / mismatches with data.

First, the density of the place fields that is reported typically measures the number of fields with centers near the goal, not the total summed firing at a location. The prediction of increased place field density at the reward location should therefore be a larger number of place fields with centers near or before the goal. The plots in Fig 1 seem to actually show the opposite (with the possible exception of T=1000) -- place fields *spread out* near the goal location but get much larger. 

Also, Gauthier & Tank report that these cells are reward specialized, and therefore remap with reward. I believe that would not be a prediction of this model -- enabling this would require adding reward / reward-predictive landmarks as as an observation dimension and observe this.

Authors note that there is a rapid increase in the density of place fields near the goal location which quickly changes, with fields spreading out a lot and getting larger amplitude. I don't believe the latter observations are consistent with the data.

It appears that the number of place fields associated with non-rewarding, but frequently visited regions might go down. I believe this is not consistent with data, where home locations (and even salient landmarks) have many place fields. 

- robustness

Does the same thing happen each time? no error bars in a lot of the plots (particularly regarding density). 

- confusing benchmarks

Only one benchmark -- SR fields. This was described as predicting future location occupancy p(x_{t+1} | x_t), but I believe the SR is in fact encoding a discounted sum of future occupancy \sum_{t=0}^\infty \gamma^t p(x_{t+1}|x_t) (aka, the expected number of visits to future states). 

Another relevant benchmark is also Fang et al 2024 which also do representation learning in a deep RL framework, and show clustering at reward and skewing (but do not look at drift / skew against the direction of motion). 

- needs tidying in some places

eperience
hippocamppal
weighs -> weights
delta defined indirectly in expectation
"The agent’s transition in the environment is smooth as we use a low-pass filter using a constant" didn't understand this.

- argument about number of fields seems speculative. Authors argue it could be because it's in weak feature learning regime, but don't see any evidence of that (and an efficient coding hypothesis seems more likely).

- T's not matched up for Fig 2C?

Comments. 

- Consider adding dropout rather than / in addition to parameter drift, as i think that will lead to more distributed representations and more actual place field clustering near goal (and also consistent with neural activity, see low, lewallen et al 2018).

questions:
- What is going on with the giant place field in Fig 2C?
 
> "To determine an agent’s reward maximization performance during navigational learning we track the true cumulative discounted reward"

why not just reward rate?

- Place fields form in the absence of any reward, and many characteristics of place fields with respect to reward appear to also occur for salient landmarks. Time spent in a region of space and sensory complexity also alters place fields in the absence of reward. What happens to place fields in this model absent reward?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732586896962

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revisions

comment:
We would like to thank all the reviewers and area chairs for taking the time and effort in reviewing our manuscript and providing insightful feedback. We have clarified each reviewer's comments and have revised the manuscript with new experiments, figures, analyses and discussions. Below, we give an overview of the common and specific questions by each reviewer. If you have any additional comments or questions, please let us know so that we can address them before the deadline for revisions this Wednesday. We hope the modifications will be sufficient to reconsider the score. 
Kindly, Submission 10547

### Common concerns:
- Use the definition of density in the literature (Reviewers Zibp, ifiL): 
-- Modified Figs 1B, 1C, 1D, Sup. Fig. 1
-- New Sup. Fig. 2
-- Modified text 187-205 and Figure 1 captions

- Modeling drift dynamics observed in the literature (Reviewers Zibp, ifiL): 
-- New Sup. Fig. 8G

- Biological origins of noise (Reviewers J18N, LQgu): 
-- Added discussion in lines 510-513

- Feature learning regime (Reviewers Zibp, J18N): 
-- New Sup. Fig, 4

- Consideration of non-reward based objective for place field reorganization (Reviewers Zibp, ifiL, LQgu):  
-- Added discussion in lines 529-531

### Specific questions:
- Whether place fields remap when reward location changes (Reviewer Zibp): 
-- New Sup. Fig, 3

- Clarifying agent transition and field dynamcs (Reviewer Zibp):
-- Clarified lines 111-120
-- New Sup. Fig. 5E

- Whether using the same learning rate will change phenomena (Reviewer J18N): 
--New Sup. Fig. 12

- Other models of place fields e.g. center-surround (Reviewer J18N): 
-- New Sup. Fig. 13

- Biological plausibility of model (Reviewer LQgu): 
-- New Sup. Fig. 14.

- Code for agents (Reviewer LQgu): 
-- We have included a minimal working code as a .zip folder to train agents in the 1D track and 2D arena with obstacle.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733238412703

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Analyses

comment:
We sincerely appreciate all the reviewers' thorough evaluation and insightful feedback.

In response to Reviewer ifiL's recommendations, we have conducted additional analyses to further validate our model's ability to replicate the drift dynamics observed in neural experiments. The results of this analysis are presented in Supplementary Figure 9 based on the findings from Ziv et al. (2013, Nature Neuroscience), de Snoo et al. (2023, Current Biology), Qin et al. (2023, Nature Neuroscience), and Gonzalez et al. (2019, Science).

We trust that this additional analyses addresses the reviewer's concerns and respectfully ask for reconsideration of the score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a reinforcement learning model investigating how place fields, represented as Gaussian radial basis functions, reorganize through TD errors during navigation. The model aims to reproduce three key hippocampal phenomena: 1) increased place field density near rewards, 2) field elongation opposite to movement direction, and 3) stable place-field drift. Following author revisions, the model successfully demonstrated a higher number of place fields emerging at both reward and home locations, backed by theoretical derivations showing how fields dynamically balance between these locations. The authors also provided extensive robustness analyses across multiple experimental conditions and parameter settings.

However, several concerns led to the paper's rejection despite the authors' thorough revisions. While the authors addressed issues regarding place-field density measurements and home location representation, key limitations remained: the model's strict reliance on reward-based learning (unlike biological systems where place fields form without rewards), incomplete validation of drift dynamics against experimental data, and discrepancies in how reward-specialized cells remap compared to empirical observations. The reviewers noted that while both the successor representation (SR) and reward maximization (RM) models made similar predictions about field distribution and dynamics, additional data analysis would be needed to determine which better matches experimental observations. They recommended strengthening empirical validation and exploring non-reward learning mechanisms before resubmission.