A generative model of the hippocampal formation trained with theta driven local learning rules
Authors: Tom George
Kim Stachenfeld
Caswell Barry
Claudia Clopath
Tomoki Fukai
Venue: NeurIPS 2023 poster (2023)
Keywords: hippocampus
path integration
local learning
generative models
oscillations
inference
Helmholtz machine
wake-sleep
TL;DR: A simple and biologically plausible generative model of the hippocampal formation accounts for a variety of cognitive functions

Abstract: Advances in generative models have recently revolutionised machine learning. Meanwhile, in neuroscience, generative models have long been thought fundamental to animal intelligence. Understanding the biological mechanisms that support these processes promises to shed light on the relationship between biological and artificial intelligence. In animals, the hippocampal formation is thought to learn and use a generative model to support its role in spatial and non-spatial memory. Here we introduce a biologically plausible model of the hippocampal formation tantamount to a Helmholtz machine that we apply to a temporal stream of inputs. A novel component of our model is that fast theta-band oscillations (5-10 Hz) gate the direction of information flow throughout the network, training it akin to a high-frequency wake-sleep algorithm. Our model accurately infers the latent state of high-dimensional sensory environments and generates realistic sensory predictions. Furthermore, it can learn to path integrate by developing a ring attractor connectivity structure matching previous theoretical proposals and flexibly transfer this structure between environments. Whereas many models trade-off biological plausibility with generality, our model captures a variety of hippocampal cognitive functions under one biologically plausible local learning rule.

A generative model of the hippocampal formation trained with theta driven local learning rules
Tom M George'
Caswell Barry²
Kimberly Stachenfeld Claudia Clopath
Tomoki Fukai
Sainsbury Wellcome Centre, UCL, UK
Dept. of Cell and Developmental Biology, UCL,, UK *Columbia University, New York, NY 6Oknaawa Institute of Science and Technology, Japan
&Google DeepMind, London, UK
Bioengineering Dept., Imperial College, UK
tom. gom.george... 200uc1. ac uk
Abstract
Advances in generative models have recently revolutionised machine learning. Meanwhile, in neuroscience, generative models have long been thought fundamen- tal to animal intelligence. Understanding the biological mechanisms that support these processes promises shed light on the relationship between biological and artificial intelligence. In animals, the hippocampal formation is thought to learn and use generative model to support its role spatial and non-spatial memory. Here we introduce a biologically plausible model of the hippocampal formation tantamount to Helmholtz machine that we apply to a temporal stream inputs. A novel component of our model is that fast theta-band oscillations (5-10 Hz) gate the direction information flow throughout the network, training it akin to a high-frequency wake-sleep algorithm. Our model accurately infers the latent state high-dimensional sensory environments and generates realistic sensory predic- tions. Furthermore, it can learn to path integrate by developing ring attractor connectivity structure matching previous theoretical proposals and flexibly transfer this structure between environments. Whereas many models trade-off biological plausibility with generality, our model captures variety of hippocampal cognitive functions under biologically plausible local learning rule.
Introduction
Generative models seek create new data samples which are similar those from the training set. To so they must learn the probability distribution of the training data, comprising a rich, generalisable and accurate model of the world. Many of the recent advances in AI have involved types of generative models: VAFS [ GANs (2) diffusion models 3 and autoregressive models [ have seeded improvements in Al capabilities ranging from data compression
[5] image generation k and natural language 7 In neuroscience, the animal brain has long been known to exploit generative models The ability to generate representative sensory data samples can be used directly, for example during offline planning or memory recall. It can also be used indirectly to aid training of inference networks with the goal of processing rich, noisy and high dimensional streams of incoming sensory stimuli, as discussed in the predictive coding literature [TO]. In a sentence: "What cannot create [generate]. do understand [inference]" (R. Feynman).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

sloop
sloop
a Network
b
C Dynamics across time
Wass
(())
sleep
sleep
$O00
PA
PA
gA
gA
g
wake
5 Hz
Waa
POO00OO
W ply
PB
PA
g
((t)
Sensory stimul Wis
PB
PB
8B
8B
ztt)
Z 0000
time
Enwronment
gated
fixed
plastic
AWp. OK
PA
AWeys
BAys X
BA/B
not gated
A biologically plausible generative model is trained with theta frequency wake-sleep cycles and a local learning rule. Network schematic: high-D stimuli from an underlying environmental latent, 2, arrive at the basal dendrites of the sensory layer, p. and map to the hidden layer, g (this is the inference model, weights in green). Simultaneously, top-down predictions from the hidden layer g arrive at the apical dendrites of p (this is the generative model, weights blue). Neurons in layers and have three compartments. A fast oscillation, 0(t), gates which dendritic compartment basal (pp. 98) or apical (pA. 9A) - drives the soma. local learning rule adjusts input weights to minimise the prediction error between dendritic compartments and the soma. e This equates to rapidly switching "wake" and "sleep" cycles which train the generative and inference models. Panel displays just two updates per theta-cycle, in reality there are many (ôt << Tg).
Figure 1:
The hippocampal-cntoriee system (aka. hippocampal formation) - a brain structure implicated in spatial [TI and non-spatial [12] memory provides pertinent example. Its primary role seems to inference 33: : mapping sensory inputs into a robust and decodable representation of state (grid cells [14, place cells [1] etc. (15)). A generative model is thought to have dual role in learning: supporting offline tasks such as route planning [16] and memory consolidation [17], and online during behaviour with path integration [8]. Path integration enables the hippocampal network to maintain an up-to-date and accurate estimate of its position in the absence of reliable sensory data by integrating self-motion cues. A recent flurry of computational 19
20 21 and theoretical 22) 21
work has highlighted the importance of path integration as a key objective explaining hippocampal function and representations.
Existing computational generative models the hippocampal formation 2 24 account for many of its cognitive functions and internal representations but require non-trivial learning rules and message passing protocols which don't 't connect with known aspects of biology. Computational models of
path integration 25 26
27 have mostly focussed continuous attractor networks which, although
experimentally supported 28 alone lack the complexity or expressivity required of fully general model of the hippocampal memory system.
The primary contribution of this paper is to introduce a biologically plausible model of sequence learning in the hippocampus which unifies its capacities generative model of sensory stimuli and path integration under one schema. To do this we propose modeling the hippocampal formation as Helmholtz machine [ which learns predict sensory stimuli given the current hidden state and action (e.g. velocity). We propose a deep connection between the hippocampal theta oscillation B0 and the unsupervised wake-sleep algorithm Gi for training Helmholtz machines. Though this class of generative models isn't widely used, and lacks the scalability of the lastest transformer baasdd sequence learners, it excels in this context since is has many natural points of contact with biology (both in terms architecture and neural dynamics) yet still maintains the expressiveness afforded to models of the brain by deep neural networks.
In this paper we:
introduce new model of the hippocampal formation which learns the latent structure of an incoming stream of sensory stimuli analogous Helmholtz machine.
describe biologically plausible learning regime: -heetaossilllations gate information flow through multi-compartmental neurons which rapidly switches the system between "wake" and "sleep" phases. All plasticity is local.

---

train our model on stimuli from a biologically relevant spatial exploration task and show it learns to path integrate by developing a ring attractor connectivity structure (comparable to theoretical predictions and empirical results in deep recurrent neural networks trained with gradient descent). Learning generalises: when the agent moves to a new environment, path integration capabilities recover without needing to relearn the path integration weights.
Our model of the hippocampal formation simultaneously (i) accounts for its role as generative model of sensory stimuli, (ii) can learn to path integrate and (iii) can transfer structural knowledge between environments. The model, though here applied to the hippocampus, can be viewed as step towards a general solution for how biological neural networks many brain regions (for example visual cortex [TO] can learn generative models of the world
1.1 Related work
A recent generative model of the hippocampus, the the hippocampal formation be thought of as hierarchical network performing latent state inference. Medial entorhinal cortex (MEC) sits atop the hierarchy and learns an abstract representation of space which is mapped to the hippocampus (HPC) where it is bound onto incoming sensory stimuli. Once trained the system can act in a generative fashion by updating the hidden representation with idiothetic action signals and then predicting the upcoming sensory experience. The drawback of this model, and others which share similar philosophical approach B2 24
Machine 23 proposed that
is that it requires training via backpropagation through time (or equivalent end-to-end optimisation schemes, as in 24 without clear biological correlates, Related hierarchical network architectures have also been studied in the context of reinforcement learning
+ and hippocampal associative memory R4
Historically, hippocampal models of path integration have focused on continuous attractor networks (CANs) 25 26 271 21] in entorhinal cortex. A bump of activity representing location is pushed around the CAN by speed and/or head-direction selective inputs, thus integrating self-motion. CANs have received substantial experimental support (28) but few studies adequately account for how this structure is learned by the brain in the first place. One exception exists outside the hippocampal literature: Vafidis et al. (35] built model of path integration in the fly head-direction system which uses local learning rules. Here we go further by embedding our path integrator inside a hierarchical generative model. Doing so additionally relaxes the assumption (made by Vafidis et al. 35 and others 36)) that sensory inputs into the path integrator are predefined and fixed. Instead, by allowing all incoming and outgoing synapses to be learned from random initalisations, we achieve a more generalisable model capable of transferring structure between environments (see section 3.3)
Hippocampal theta oscillations have been linked to predictive sequence learning before
37
38
39
40 where research has focused on the compressive effects of theta sequences and how these interplay with short timescale synaptic plasticity. Instead of compression, here we hypothesize the role of theta is to control the direction information flows through the hierarchical network.
Finally, recent theoretical work by Bredenberg et al. AII derived, starting from principles of Bayesian variational inference, a biologically plausible learning algorithm for approximate Bayesian inference of hierarchical network model built from multi-compariin culti-comparttentaal neurons and trained with local learning rules using wake-sleep cycles. Here we build a similar network to theirs (i) extending it to a spatial exploration task and mapping the hidden layers onto those in the hippocampal formation, (ii) simplifying the learning rules and relaxing a discrete-time assumption - instead, opting for a temporally continuous formulation more applicable to biological tasks such as navigation and (iii) adapting the hidden layer to allow idiothetic action signals to guide updates (aka. path integration). Their work provides a theoretical foundation for our own, helping explaining why learning converges on accurate generative models.
2 A biologically plausible generative model trained with rapidly switching wake-sleep cycles and local learning rules
In sections 2 and t we give concise, intuitive descriptions of the model and experiments; expanded details can be found in the supplementary material.
'Code at httpt////tt.t.oooooooo/////////a

---

2.1
Basic model summary
We consider learning in an environment defined by latent state, =t)), which updates according to stochastic dynamics initially unknown to the network,
dz fz()).
(I)
These dynamics depends on the task; first we consider =t) to be set of mutually independent random variables and later we consider the more realistic task of an agent moving on a 1D track.
The network recieves sensory input which is function of the latent state into a sensory layer, p(), and communicates this to a hidden layer (aka "internal state"), g()). The network contains both an inference (aka. recognition) model which infers the hidden state from the sensory input (green arrows, Fig. TT) and a generative model which updates the hidden state with recurrent synapses and maps this back to the sensory layer (blue arrows). As we will soon identify these processes with Basal and Apical dendritic compartments of pyramidal neurons we label activations sampled from the inference model with the subscript B and those from the generative model with the subscript In summary
( + 8t) = p(=(()
Inference model
(t + 5t) = ggn (Wen P(t))
gA + ōt) = Aga ( gn B(t))
Generative model.
(4 ō1) pna (Wp, p. 8())
Wan WA on are matrices of randomly initialised and plastic synaptic weights. P maps the environmental latent into a vector of neural inputs. a's denote activation functions applied to the dendritic pre-activations either the identity (a(x) X) or rectified tanh functions (a(x) = max(0, (a)). A small amount of noise is added the dendritic activations to simulate realistic biological learning.
We believe that the widely adopted convention of modelling neurons as single-compartment per- ceptrons is limiting. By considering, in minimal extension, the distributed dendritic structure of real neurons we can tap into significant potential for explaining hippocampal learning. Theoretical
44 4 and experimental [46 47 48 research into credit assignment in biological neurons has identified different roles for basal and apical dendrites; basal dendrites are thought to receive bottom-up drive from sensory inputs whereas apical dendrites receive top-down drive from higher layers in the sensory hierarchy [9] Following this line of research - and matching an equivalent theoretical model of latent state inference described by [41 we identify the inference process with synaptic inputs into a basal dendritic compartment pyramidal neurons and the generative process with synaptic inputs into apical dendritic compartment. In summary, each p and g neuron in our model has three compartments: a somatic compartment, a basal dendritic compartment and an apical dendritic compartment (Fig. [D). Only the somatic activation is used for communication between layers (right hand side of Eqns. 2 and 33) while dendritic compartment activations are variables affecting internal neuronal dynamics and learning described below (Eqns. and
42
4
2.2 Theta oscillations gate the direction of information flow through the network
The dynamies of the somatic activations ptt) and g()) are as follows: the voltage in each soma is either equal to the voltage in the basal compartment the voltage in the apical compartment depending on the phase of an underlying theta oscillation. This achieved by a simple theta- theta-gating mechanism (Fig. [D)::
p(t) = ((t) (l-
g(t) (t)) (f) + (1- (tt)
(4)
where 0(() is a 5 Hz global theta oscillation variable defined by the square wave function:
0(1) {b.
1, t/t mod 1 < 0.5
(
if t/T mod 1 > 0.5
"These labellings conveniently match the notion that inferences are made from layers Below in the sensory hierarchy (bottom-up) whereas generative predictions arrive from Above (top-down).

---

for T' 1 and fo = 5 Hz, matching the hippocampal theta frequency (5-10 Hz) [50] According to this model theta-band oscillations the hippocampal local field potential gate which dendritic compartment drives the soma. Experimental 47 51 52 support for this assumption.
and modelling work [53] gives provisional
These local theta-dynamies have global consequences: the early phase (0(t) = 1) of each theta cycle can be thought of as "wake" phase where information flows upwards through the network from the environment to the hidden layer, sampling the inference model. The latter phase (0(t) = 0) of each theta cycle a "sleep" phase where information flows down from the hidden layer to the sensory units, sampling the generative model. These dynamics are displayed in Fig.
2.3 Hebbian-style learning rules train synapses to minimise local prediction errors
In contrast to comparable models which are optimised end-to-end using backpropagation through time our model learns synaptic weights according to a local plasticity rule which is a simplified variant of a rule proposed by Urbanczik and Senn 46
Incoming synaptie projections are continually adjusted in order to minimize the discrepancy between the somatic activation and the dendritic activation. The full learning rules are described in the supplement but simplified versions are given here:
dw x (g(t) (E()))p(1T
dl
dwpa dt
x (p(t) (A()))B(1T
dwas
(g(t) (t)))
(6)
dt
Notably this learning rule is equivalent for all plastic synapses in the model: p to g, g p and the recurrent g to g synapses (see Fig. [T). If a local prediction error is detected, for example the somatic activation larger than the dendritic activation, then the synaptic strength of inputs into that dendritic compartment which are positive/negative are strennthed/weakkendd to reduce the error. This model can equivalently be viewed type of Hebbian learning weight change is proportional to the correlation of pre- and post-synaptic activity (the first term)- regularised (by the second term) to prevent unbounded growth.
During the wake phase the weights of the generative model (Wp, and w Wg,) are trained and plasticity on the inference weights (Wgg) falls to zero. This occurs naturally because p P so there will be no basal prediction errors to correct. During sleep the reverse occurs; the weights of the inference model are trained and plasticity on the generative model falls to zero. Experimentally, apical activity is known to guide plasticity at basal synapses in CAI 4б). This alternating, coordinated regime of sampling and learning saample-ifeerence-train---- -errarinegeeerivee, then is a hallmark of the wake-sleep algorithm. It fundamentally differs from the forward and backward sweeps of backpropagation since neurons remain provisionally active at all times so the process of learning minimally perturbs perception. Also, whereas backpropagation sends error signals down through the network to train synaptic weights, here only predictions are sent between layers and error signals are calculated locally at each dendrite.
As discussed in section Bredenberg al. 41 mathematically derive learning rules similar to these starting from a loss function closely related to the evidence lower bound (ELBO). As such our identification of early- and late-theta phases "wake" and "sleep" cycles can be considered precise: from a Bayesian perspective our hippocampal model is minimising a modified ELBO loss (see supplement) thus learns find approximately optimal inference and generative models accounting from the temporally varying stimulus stream it is presented.
2.4 Velocity inputs into the hidden layer
For path integration, the hidden state needs access to an idiothetic (internally generated) velocity signal. To satisfy this we endow the hidden layer, with conjunctive velocity inputs, henceforth "conjunctive cells", as shown in Fig. B for leftward motion and geg for rightward motion. Each conjunctive cell receives input from the
Conjunetive cells are organised into two groups: gr, is responsible
hidden units and either the leftward (wz, max(0. (0, -i)) or rightward (UR = max (0, à)) component of the velocity. For the results shown this connectivity is one-to-one (Worlis [Wa.alis di but
5

---

a
(t)
Time, t min
Inference model
Generative model Tested during sleep
100005
Tested during
Prediction error
Stimulus / Hz
AMMA
Auto-correlation
10-1
(PA
pa)
(A
gB.
Time, min
10.1.5
PB
PA
Lag /
4
Training / min 30
network
sleep mode
Learning in environment of temporally varying latents. In this artifical task the latent space comprises of N₂ = 5 independent random variables with an autocorrelation decay timescale of s. Prediction errors (difference between apical and basal activations) in sensory and hidden layers reduce over training time, с Tested in wake mode (0 = 1) after training, the ground truth stimulus matches apical prediction for all stimulus dimensions (one shown) implying the network is efficiently "autoencoding"" the sensory inputs into and back out of the compressed hidden layer. Tested in sleep mode (# O, no environmental inputs), generated data from the hidden units, 9- have autocorrelation curve which matches that of the true latents implying statistically accurate generative model has been learned. More extensive samples from this model, before and after training, can be found in Fig. ST
Figure 2:
random connectivity works too, see supplement. Finally, conjunctive cells send return connections back the apical dendritic compartment of the hidden units via randomly initialised plastic synaptic weight matrix. This inputs are what drive the hidden units to path integrate.
This model takes inspiration from so-called conjunctive grid cells [54] found in the medial entorhinal cortex (MEC). These cells, though to be an integral component of the mammilian path integration system (2)). jointly tuned head direction and location much like the conjunctive cells in model. An important and novel aspect of our model is that synaptic weights between or into the hidden units are learned. This deviates from other models for example that by Burak and Ficte 27 (where all connectivity is predefined and fixed) or Vafidis et al. (35] and Widloski and Fiete 33 (where sensory inputs to the hidden units pre-defined and fixed). This is not only more realistic but affords the model flexibility to translate path integration abilities between environments without having to relearn them, form of transfer learning which we demonstrate in section 33
3 Results
3.1 Validation on an artifical latent learning task
We begin by testing the basic model (i.e. without conjunctive inputs, Fig. [T] on an artificial task. N₂ = latents, 2t),, are independently sampled from smooth, random process with an autocorrelation timescale 1 second (Fig. [4). The sensory layer, Np = 50, then receives a high-dimensional random linear mixture of the latents into the basal compartments:
(t) Az(t).
where E 50 and [A]i ~ NO A The hidden layer, g(t), is matched in size to the latent process, N₂ = N₂ 5, and all dendritic activation functions are linear. We train the model for 30 minutes of simulated time and track prediction errors, the difference between the basal and apical activations in the sensory and hidden layers, which reliably decreased throughout training (Fig. PD). We then perform two tests designed confirm whether the model has learnt accurate inference and generative models.

---

a
VH
Wix from
Wox from gvL
Wea from Evp
Evi
O000
EvR
MEC
K
CITY andiee
g BOOOO
Ween
SA
MEC
EA
POOOO
HPC
SR
00
Postion (uy]
a
Prediction error
(PA
P8
0.1
(A ge
Position on track m
40
Error / cm (Errrr) * SEM
sensory lesion
Training / min
30
10 Time /
30
The hippocampal model learns path integrate on ID track using ring attractor. Position selective (place cell) inputs drive basal dendrites of the sensory layer p (HPC). b Hidden units (MEC) are connected two sets of "conjunctive cells" which each connect back the hidden neurons (g) and either the leftward (for gu,) or rightward (for go,,) velocity of the agent allowing velocity information to enter the network. Synaptic strengths of the return connections from the conjunctive cells to the MEC hidden units, as well as those for the MEC recurrent connectivity (collective denoted Wg.). are randomly initialised and plastic. с After training, reordering the hidden units by the position of peak activity reveals ring attractor in the synaptic weight matrices. Centre-surrount recurrent connectivity stabilises an activity bump which is then "pushed" around the attractor manifold by asymmetric connections from the conjunctive cells, integrating velocity, Bands zero weights show MEC neurons which have become perpetually inactive (aka "died"). The bottom panel displays the matrix row-averages, utilizing the circular symmetry of the environment to align rows before averaging. Learning plateaus after 15 mins of simulated time. e Path integration ability is demonstrated in a lesion study: after 10 seconds in the normal oscillatory mode the network placed into sleep mode (aka generative mode), lesioning the resition-dependen sensory inputs. Despite this HPC continues to accurately encode position, evidence that the MEC ring attractor is path integrating the velocity inputs and sending predictions back to HPC. Lower panel shows the accumulated decoding error well the SEM over 50 trials.
Figure 3:
of
First, we set the dynamics of the model "wake" mode (9 1) and measure the basal and apical activations of one of the sensory neurons for 60 seconds. Close correspondence (Fig. 2 confirms that the network accurately "autoencodes" the high- high-dimensional sensory inputs through the compressed hidden layer. Since all activation functions are linear this implies that w "g and Wp, are pseudoinverses. Next, we place the network in "sleep" mode (0 = 0) and allow the generative model to run freely. The autocorrelation of the generated hidden states (g 0 0), displayed fully in the supplement) match that of the true environmental latents (z(t)), Fig. 1 implying the generative model has statisties closely matching those of the true underlying generative process.
3.2 Learnable path integration with hidden ring attractor
Next we turn our attention to the hippocampal formation's role in spatial navigation, and our central result. The environment consists of an agent randomly moving around 1 1D circular track (motion and cell data is generated using the RatInABox package (55)). The basal compartment of each HPC neuron is spatially tuned to single different Gaussian input however non-Gaussian
7

---

randomly spatially tuned inputs work as well (see supplement Fig. S2p)
(x(t) xi)
(t)a = exp
(8)
20
ar()) is the position of the agent and (2i) 1ير are the centres of the Gaussian inputs (a = 6 cm), intended to simulate hippocampal place fields, evenly spaced at 1 cm intervals along the track. MEC (i.e. the hidden layer, g(t)) is matched in size N, = N, 100 with rectified tanh activation functions on both dendritic compartments (o (z) = ga (z) = max(0, tanh(x))) and HPC (the sensory layer p(t)) is linear po (x) 2). Two populations of conjunctive cells (Fig. Ba & b) feed into the apical compartments of the MEC recurrent units. Random initialisation of w means that MEC neurons start off with random non-Gaussian spatial tunings. W9a and Wpa also randomly initialised.
The network is trained for 30 minutes with learning plateauing after 15 (Fig. Bit. A lesion study, designed to test path integration, is then performed as follows: First, the network is run for 10 seconds normally (i.e. with theta-oscillating periods of wake and sleep). Since the simulated HPC neurons receive place-tuned inputs uniformly ordered along the track (i.e. Iy > J > i) an activity heatmap of HPC reveals a bump of activity accurately tracking agent's 's position (Fig. D left). The network is then placed into a sleep phase (0- 0) for 20 seconds. This amounts to full sensory lesion since top-down MEC inputs, not bottom-up place-tuned sensory inputs, drive HPC. Despite the full sensory lesion, hippocampal activity remains approximately unperturbed and the activity bump continues to accurately track position, slowly accumulating errors (Fig. 3 right). Since our HPC layer has no recurrent connectivity it cannot support this post-lesion activity on its own. Instead feed-forward drive from an MEC ring attractor, which we turn our attention to now, is responsible for maintaining the HPC code.
To find the ring attractor we must first reorder the MEC cells. We do this according to the position of the peak of their receptive fields (defined in the supplement). After reordering, the recurrent connectivity matrix can be seen to have acquired a centre-surround connectivity profile. Nearby MEC cells were, average, strongly and positively recurrently connected to one another. Those far apart weakly inhibit one another (Fig. B left; band of strong positive weights along diagonal flanked by weak negative weights). This profile matches that of quasi-continuous ring attractor: local excitatory and long-range inhibitory connections stabilise a bump of activity on the attractor manifold in the absence of sensory input [56].. Weights from the conjunctive cells acquired asymmetric connectivity (Fig. Be. middle& & right) skewed towards the velocity direction for which they are selective. These asymmetric connections enable conjunctive cells to "push" the activity bump around the manifold, integrating velocity (see supplement for a visualisation of the MEC bump attractor). Theoretical work on ring attractors has demonstrated that for accurate path integration the asymmetric weights must be proportional to the derivative of the symmetric weights [56]. approximately observed here. A noteworthy observation is that some MEC neurons become perpetually inactive; this is a consequence of the fact that both top-down and bottom-up synapses into the hidden layer are plastic and can fall to zero (Fig. 3 bands of sero-weights) satisfying a trivial 9A = 9B = 0 solution for minimising the prediction eтror. Despite this, not all MEC neurons die and the surviving subset are sufficient for path integration. In supplementary section 5.4.2 we discuss additional results showing when the network learns robust path integrate under a variety of plasticity, initialisation and noise manipulations.
Crucially, what sets this model apart from others 19 using a conventional path-integration objective and backpropagation. Instead, it has been demon- strated how path integration can naturally arise in biologically constrained network subject to a much simpler (yet more broadly applicable) local objective, in cases where idiothetic velocity signals are available to the hidden layers.
20 21 2 is that the network is not optimized
3.3 Remapping: transfer of structural knowledge between environments
Finally, we demonstrate how our trained network can transfer structural knowledge which here means the ring attractor and thereby path integration between environments. We start by training the network as in section 3.2 the only diffence is that for simplicity we choose to fix Wan 8, giving rise to MEC representations which, like HPC, are unimodal (this constraint can be relaxed and, in the more general case, MEC units typically have multiple receptive fields, Fig (4ft, reminiscent of grid cells). We then simulate hippocampal "remapping" event by shuffling the sensory inputs to the HPC layer (Fig. 4 & b, top panel) and retraining the network for further 30 minutes but this time
8

---

Spatial receptive fields...
Remapping experiment
b
before remap
just after remap
after retrain
Bvu
Ev
100
(x
retrain Wen & Wps
HPC
POOO
remap Wps
O0
100
chance decoder
(Error) cmm
20
gi (x)
MEC
10
0
1
before
immediately
after retrain
0
shufile
after shuffle
Position, x m
Figure 4: Remapping and transfer structural knowledge between environments. a After training (as in Fig. 2 place cell inputs are shuffled to simulate a "remapping" event observed when an agent moves to new environment. The agent then retrains for an additional 30 minutes: during this period internal MEC weights, and weights from the conjuctive cells to MEC are held fixed while MEC +> HPC weights remain plastic. Recptive fields of the HPC and MEC neuronal populations at different stages in the experiment: Initially after remapping HPC and MEC inputs randomised. MEC relearns rate maps as they were before remapping but with a constant phase shift. Note: neurons ordered by the position of their peak activity the track before remapping and this ordering is maintained subsequent panels. e The error (# SEM over 50 trials) after second of path integration shown at different stages of the experiment. Although path integration is initially disrupted after remapping it recovers despite no relearning of the MEC synapses where the ring attractor is stored.
holding weights in the hidden layer. W. Only the HPC + MEC synapses (w ga & wwp. Waa remain plastic during retraining. Biologically this may accounted for by the observation that cortical plasticity is substantially slower than hippocampal plasticity [57]
During biological remapping events place cells remap independently whereas grid cells remap en masse with entire modules shifting by the same constant phase [58]. This observation is reproduced in our model: after retraining MEC units regroup with receptive fields as they were before remapping but with a constant phase shift along the track. This re-emergence of structure occurs because the ring attractor seeds bump of activity on the attractor manifold (during the "sleep" phases of retraining) onto which the shuffled HPC inputs then bind. Since nothing constrains where on the circularly symmetric attractor manifold this regrouping can initiate, only relative correlations, modulo phase shift, are preserved.
second after sensory lesion is tested just before remapping, just after remapping and after retraining (Fig. F+). After the remapping path integration abilities temporarily disappear because the MEC ring attractor is still tuned the old and invalid HPC receptive fields. After relearning and despite no adjustments to the MEC weights, w Wax where the ring attractor is stored - path integration recovers almost the level before remapping. This differs substantially from other which don't 't consider plasticity on the ring attractor inputs. In these models, adaptation new environment necessarily requires complete relearning of the ring attractor. Instead our model exploits the basic fact that movement (path integration) in one environment fundamentally the same as in another, one must simply learn new mapping to/from the ring attractor, "translating" it to fit the new sensory stimuli.
Decoding
local models of path integration learning 35 Bol
4 Discussion
We propose that the hippocampal formation resembles Helmholtz machine, simultaneously learning an inference and generative model of sensory stimuli. Like previous models 2 medial entorhinal

---

cortex (MEC) sits hierarchically above the hippocampus (HPC) to which it sends generative pre- dictions. Our model differs in the learning rules and neural dynamics: local prediction enrors are minimised between distinct dendritic compartments receiving bottom-up and top-down signals. Theta oscillations regulate internal neural dynamies, switching the network between wake and sleep phases. In a navigation task our MEC model forms a ring attractor capable of path integration. Despite simple learning rules and dynamics our model retains key cognitive capabilities of the hippocampal formation including the ability to transfer knowledge across different sensory environments.
Local learning rules are commonly recognised as essential in biologically plausible learning algo- rithms 43 However, the importance of learning scheduling how neural systems coordinate or multiplex distinct phases of forward and backward information flow often overlooked[59). Neural oscillations such as theta, hypothesized to temporally coordinate communication between neuronal populations 660], likely play an underexplored role this regard (neural "bursting" has also been pointed out as potential solution to multiplexing [6]). One advantage of the wake-sleep algorithm, which this study suggests neural oscillations can support, compared to forward and backward sweeps is that, during convergence, the two phases become highly similar, allowing learning to proceed without affecting perception.
While our discussion has primarily focused on theta oscillations as a mechanism for learning, they have also been proposed as mechanism for short-range future prediction via so-called "mind- travel" [2). During the latter phase of each theta cycle (i.e. the sleep phase) gain amplified velocity signals might rapidly drive the MEC activity bump along the manifold allowing the agent assess nearby upcoming locations. This complimentary proposition could neatly integrate into the framework proposed here and emphasizes the need for further investigation into the multifaceted functions of neural rhythms within the hippocampal/entorhinaa system.
Beyond theta oscillations, both faster gamma cycles [33 and the slower physiological states of sleep and wake [4]] have been associated with learning. Based on our model we suggest a tentative hy- pothesis that theta oscillations may be favored due to optimality criterion; whilst faster oscillations could be mechanism to prevent extreme drift during sleep that might disrupt learning their frequency might by upper bounded biophysically by the neural time constants associated with the biophysical processes supporting dendritic gating the soma. These ideas, their relevance to other brain regions involved in generative learning. 2D spatial dynamics, and offline memory consolidaaion/replayy remain exciting questions for future theoretical and experimental investigation.
10

---

References
[1] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2022.
[2] Ian J. Goodfellow, Jean Abadet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
[3] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages -265,, Lille. France, 07-09 Jul 2015. PMLR. URL |tt/ptpps//prroeeddngg mlr ppwss//3/ssoois di www.eiiiiss.n....mmmmmmmm htmm
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, (ed///066.376,, 2017.
URI. htt.ttttpp//riiiii http:///axvvv.or/eeee/ abt////s////// 03762
[5] Tom M George and Pietro Lio. Unsupervised machine learning for data encoding applied
to ovarian cancer transcriptomes. November 2019. doi: 10.1101/855593. URL https
/ //doi.org/10 1101/855593
[6] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot rext-to-image generation. CoRR, abs/2102.109922 2021.
URL CtL.https:// /arxiv.vvvv. wwtps///..rgaa////...../ 12092
[7] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
[8] Karl Friston. The free-energy principle: unified brain theory? Nature Reviews Neuroscience, 11(2):227-13, January 2010. doi: 10. 10.1038/nm2787. URL https://doi. 0ttp::////o1.000111000000000 10. 10388/
nrn2787
[9] Samuel J. Gershman. The generative adversarial brain. Frontiers in Artificial Intelligenee, September 2019. doi: 0..3389/ira.22119000113 URL https::/doi..rr http://6o...rr//1..3333/////
2019.00018
[10] Rajesh N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January 1999. doi: 1038/44/5 URL http:ttt///oo..or//////// 1038/4580)
[11] John OKеefe, Place units in the hippocampus of the freely moving rat. Experimental Neurology, 51(1):78-109, January 1976. doi: 10. 10.00166011444886666600555 https:s/:///doi org/
10. (7)1)/0001444486666))))))))
[12] Larry R. Squire. "memory and the hippocampus: A synthesis from findings with rats, monkeys, and humans": Correction. Psychological Review, (113)::82--822 July 1992. doi: 10.1037/ http:tttt///.....ot org 1037/000333333 295x. 99.99.9.888
0033-295x.99. 99.3x.999.588.
[13] Honi Sanders, Matthew A Wilson, and Samuel J Gershman. Hippocampal remapping as hidden state inference. elei 9, June 2020. doi: 10.7554/elife.51 140. URL https://tt:/
7554/elife. 51140
[14] Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I. Moser. Mi- crostructure of a spatial map in the entorhinal cortex. Nature, 436(7052):0018866, June 2005, doi: 10. 10.1038//aturc03721. URL https://///oi.. 1ttp:/0000003//mauur033335
[15] Edvard Moser, May-Britt Moser, and Bruce L McNaughton. Spatial representation in the hippocampal formation: history. Nature Neuroscience, 20(11)):4488 1464, November 2017.
doi: 10.1038/nn.4653, URL httl::tts://ee 1ttp://////i...011111101000.1 www8/nn...m...mm 4653
[16] Hugo J. Spiers and Eleanor A. Maguire. Thoughts, behaviour, and brain dynamics during navi- gation in the real world. Neurolmage. 31(4):1826-
July 2006. doi: 10. 1016/ 10. neuroimage. 2006.01.033 URL https://::/// org 10. 1016/001ojjjj www.:mmuuuraaeem.m htat::///ooo...r///000000oe.aaeo 2006. 01. .037
[17] Margaret F Carr, Shantanu P Jadhav, and Loren M Frank. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nature Neuroscience, 14(2): 147-153, January 1.11. doi: 10. 10.1038/nn.273. URL //doi org/ 10g/10.110 1038/nn. 2732

---

[18] B. L. McNaughton, C. A. Barnes, J. L. Gerrard, K. Gothard, M. W. Jung, J. J. Knierim, H. Kudrimoti, Y. Qin. W. Skaggs, M. Suster, and K. L. Weaver. Deciphering the hippocampal polyglot: the hippocampus as path integration system. Journal of Experimental Biology, 199 (1):173-185, January Januaryy 1996. doi: 10. 10.1242/jeb.199) 199.1.173. httpp::///:///...rrr//
1242/
199. 1.173
[19]
Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks perform spatial localization, 2018.
[20]
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney. Helen King, Koray Kavukeuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector- navigation using grid-like representations in artificial agents. Nature, 557(7705):429 433, May 2018. doi: 018-0088--100 0102-6
10.1038//41586-01 018-860-18------66
httpst ///o/ org/ 10. 1038/=4115000
[21] Ben Sorscher, Gabriel C. Mel, Samuel A. Ocko, Lisa M. Giocomo, and Surya Ganguli. unified theory for the computational and mechanistic origins of grid cells. Neuron, 111(1): 121-137.e13, 2023. ISSN 0896-6273. doi: hups://doi. wrw. www.0/011/.....n..ooo www.urronnn22......oo https://www sci aeciieneedieecc cwtee/tttcc.mmeeeeeeeeeeeeeeeeee wwt.c/o...ccccccceeecccccc wwwccinneaaatiiiaaii http//rtiiccceeeee /ttp:////////0000000000
URL
[22] William Dorrell, Peter E. Latham, Timothy E. J. Behrens, and James C. R. Whittington. Actionable neural representations: Grid cells from minimal constraints, 2023.
[23] James C.R. Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy E.J. Behrens. The tolman-cichenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation. Cell, 18495)1249- 1263.e23, November 2020. doi: 10. wetp://fccll.222..110 1016/j.cel1 2020. 10.024
URL https:///oi.
[24] Dileep George, Rajeev V. Rikhye, Nishad Gothoskar, J. Swaroop Guntupalli, Antoine Dedieu, and Miguel Lázaro-Gredilla. Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nature Communications, 12(1), April 2021. doi: URL https:///oi org/ 10. 1038/ $414674444777 020-14677-22--255---3
10. 00.1038/444467-02-2555555
[25] W. E. Skaggs, J. J. Knierim, H. S. Kudrimoti, and B. L. McNaughton. A model of the neural basis of the rat's sense of direction. Advances in neural information processing systems, 173-180, 1995. ISSN 1049-5258.
[26] Alexei Samsonovich and Bruce L. McNaughton. Path integration and cognitive mapping in continuous attractor neural network model. Journal of Neuroscience, 17(15):590--5920, 1997. ISSN 0270-6474.
doi: 10. (00122/NEEURRSSC.TT5555500000000
URL fatts://:://///
jneurosci org/ content/1//155555 5900
(27] Yoram Burak and lla R. Fiete. Accurate path integration in continuous attractor network models of grid cells. PLoS Computational Biology, 502)2e100029, February 2009. doi: 10. 1371/
journal.pcbi. 1000291. URL httpp:///ooi. 1ttp:/10..71///rrraaa pcbi. pcb1.10002911
[28) Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. arXiv, 2021. doi:
10. 4485550/arriv221122009778
[29] Peter Dayan, Geoffrey E. Hinton, Radford M. Neal, and Richard S. Zemel. The helmholtz machine. Neural Computation, 7(5):89 904, September 1995. doi: 10. 00.116///cco.1995...555
https//s:://oo org/10 1wl.0..1112cnccac 1995. 7 5.889
[30] Gyorgy Buzsaki. Theta oscillations in the hippocampus. Neuron, 33(3):325-340, January 2002. doi: 10. 10010066/089666225500000666888
URL
00586-x
[31] Geoffrey E. Hinton, Peter Dayan, Brendan Frey, and Radford M. Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 10.1 1211/sscenccc.7761883
May 1995. doi: https: org/ 1 / 7761831
[32] Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zambaldi, Dharshan Kumaran, Demis Hassabis, Caswell Barry, and Charles Blundell. A model of egocentric to allocentric under- standing in mammalian brains. November 2020. doi: 00.1101/2020.11.1123
URL
https: :stps://doi. org/10. 1101/20200001 11 378111
12

---

[33] Dongqi Han, Kenji Doya, and Jun Tani. Self-organization of action hierarchy and composition- ality by reinforcement learning with recurrent neural networks. Neural Networks, 129:149 199.149--22 September 2020. doi: 10. 10116/..eemee2222000... neunet. 2020. .0.0000600022
URI. CRR..tttss:: https:///dooo /rgg/...... 1016/]
[34] Sugandha Sharma, Sarthak Chandra, and Ila Fiete. Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19658-19682. PMLR, 17-23 Jul 2022. URL https
//proceedings.n mlr.
[35] Pantelis Vafidis, David Owald, Tiziano D' Albis, and Richard Kempter. Learning accurate path integration in ring attractor models of the head direction system. elife, 11 11:e69841, jun 2022. ISSN 2050-084X. doi: 10.7554/eL.ffe.69%5 eLife. 69841
URL. https://doi. org /0. 7554/
[36] John Widloski and lla R. Fiete. A Model of Grid Cell Development through Spatial Exploration and Spike Time-Dependen Plasticity. Neuron, (112):4814955 2014. ISSN 0896-6273. doi:
10.1016/j.neeroo.2200 20w.uuoon..01....0.1
[37] William E. Skaggs, Bruce L. McNaughton, Matthew A. Wilson, and Carol A. Barnes. Theta phase precession in hippocampal neuronal populations and the compression of temporal se- quences. Hippocampus, 6(2):149 172, 1996. doi: 10. 10.10000(scci))0098888888 16699996010001199996666449999999 aid- hod-hipo6>3.0.co22-4 2<149: hi14::aaddhippo63... co
URL https://doi https://doi.orr/100 10020..00000000 (str)/ssiii)) ottp://..00///111111111-11111110 1063
[38] M.R. Mehta, M.C. Quirk, and M.A. Wilson. Experience-Dependen Asymmetric Shape of Hippocampal Receptive Fields. Neuron, 25:707-715, 2000.
[39] Tom M George, William de Cothi, Kimberly L Stachenfeld, and Caswell Barry. Rapid learning of predictive maps with STDP and theta phase precession. eLife, 12, March 2023. doi: 11154//life.806666
httpss/////ai http://ooo.orggg//// wtt.a/....r......./////e/11..... 7554/ 80663
[40] Tom M George. Theta sequences as eligibility traces: A biological solution to credit assignment.
In International Conference Learning Representations 2023 (TinyPapers track), 2023. doi: http://d...orr/111455555//1.x.. vd16AYbem37
https://doi.owr 200.04440000000..233..... 224.
URL httpp:://ppeneeveer..eet//////
[41] Colin Bredenberg, Eero P Simoncelli, Benjamin S H Lyo, and Cristina Savin. Impression learning: Online representation learning with synaptic plasticity. page 13, 2021.
[42] Konrad P. Kording and Peter Konig. Supervised and unsupervised learning with two sites of synaptic integration. Journal of Computational Neuroscience, 11(3):20-215,, 2001. doi:
10. 10.1023/a: 1013776171177 URL http::ttts:::/o.orrtttt /0t
[43] Robert Urbanczik and Walter Senn. Learning by the Dendritic Prediction of Somatic Spiking, Neuron, 81(3):521-528 2014. ISSN 08966273. doi: 10. 10..101/...euurn2201......
URL
elsevier..... http://ee.oo.oererrrrreee//////a
[44] Joao Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcireuits approximate the backpropagation algorithm. In Advances in Neural Information
Processing Systems, pages 8721-8732, 2018.
[45] Blake Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current Opinion in Neurobiology. 54:28-36 36. February 2019, doi: 11110.66..cobb2211888..0000 URL http::///oo.oo.... http:s///oo..orgg000.0000/////./ conb. 2018 08.00...8 003.
[46] Katie C Bittner, Christine Grienberger, Sachin P Vaidya, Aaron D Milstein, John Macklin, Junghyup Suh, Susumu Tonegawa, and Jeffrey Magee. Conjunctive input processing drives feature selectivity in hippocampal CAI neurons. Nature Neuroscience, 18(8):1133-1142 July
2015. doi: 10.1038/nn.406.. URL https: /ttp::/:///ooor/1....o 1038/nn 4062
[47] Jurij Brankack, Mark Stewart, and Steven Fox. Current source density analysis of hippocampal theta rhythm: associated sustained potentials and candidate synaptic generators. Brain Research, 61 610 July 1993. doi: 10. (0000000000000099999))0000000
URL.
https: /stps://doi..o 1ttp://0011111000----8888800
90043-m
13

---

[48] Kenji Mizuseki, Anton Sirota, Eva Pastalkova, and György Buzsáki. Theta oscillations provide temporal windows for local circuit computation the wewpehiinnl-iipooaaaapaa loop. Neuron. 64(2):267-280. October 2009, doi: 10.1016/j ne0.1//..euronn220000...... URL. org
10.1016/J neuron 2009 08.0088.000 037
[49]
Matthew Larkum. Are dendrites conceptually useful? 2022. doi: 10.101 10.1016/j.neuree www.eoocccencce.222......
Neuroscience, 489:4-14, May
URL
neuroscience 2022. 01.03...00 .08
[50] David Foster and Matthew A. Wilson. Hippocampal theta sequences. Hippocampus, 17 (11):1033-199,, 2007. doi: 10. 10.1002/hipo.2344. URL
20645
[51] Christian Holscher, Roger Anwyl, and Michael Rowan. Stimulation on the positive phase of hippocampal theta rhythm induces long-term potentiation that can be depotentiated by stimulation on the negative phase in area cal. The Journal of Neuroscience, 17(16))6470-6477 August 1997. doi: 10.1523/jneuoocci. 1 17-16-06470.11 1997. URL https: hrg. 1523/ jneurosci. 16- 06470. 1997
[52] Yoko Yamaguchi, Yoshito Aota, Bruce L. McNaughton, and Peter Lipa. Bimodality of theta phase precession in hippocampal place cells in freely rumning rats. Journal of Neurophysiology, 87(6):2629 2642. June 2002. doi: 20022/....022...2222222 1152/ jn. 2002. 87. 2629
URL httLpttt/////...rr/////
[53] Michael E. Hasselmo, Clara Bodelón, and Bradley P. Wyble. A proposed function for hippocam- pal theta rhythm: Separate phases of encoding and retrieval enhance reversal of prior learning. Neural Computation, 14(4):793-817 April 2002. doi: 10208666779777001111
URL
https://doi. org/ 10. 10w111122/8800 01162/089976602317338888
[54] Francesca Sargolini, Marianne Fyhn, Torkel Hafting, Bruce L. McNaughton, Menno P. Witter, May-Britt Moser, and Edvard I. Moser. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science, 312(5774):558-76) May 2006. doi: 10.1126/scienc. //oo:
1125572. URL
www11112///eneeeee 1125572
[55] Tom M George, William de Cothi, Claudia Clopath, Kimberly Stachenfeld, and Caswell Barry. RatlnABox: A toolkit for modelling locomotion and neuronal activity in continuous environments. aug 2022. doi: 10. 10.1101/2022.08 00000/2022.0811055335544.. 10.50341.. URL https://doi. org/10
1101X2F2022.08 10. 503541
[56] K Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. The Journal of Neuroscience, (2-))221222216 March 1996. doi: 10. 10.1523/jneuoosei. 16-06-02112. 1996. URL htttps:///di....... org. 1e231/522nnue/ 16-06- 02000002112..0 1996
[57] Ceren Ergorul and Howard Eichenbaum. Essential role of the hippocampal formation in rapid learning of higher-order sequential associations. The Journa of Neuroscience, 26(15):411 4117, April 2006. doi: 0152///neurscii00411-02222222220
URL https://d/// or0.10...011.0.............
j nneurosci 0440004416600000 2006
[58] Marianne Fyhn, Torkel Hafting, Alessandro Treves, May-Britt Moser, and Edvard I. Moser. Hippocampal remapping and grid realignment entorhinal cortex. Nature, February 2007. doi: 10. 00.10/80aatur005601.
http::tttt:////t org 1038 nature05601
[59] Jordan Guerguiev, Timothy P Lillierap. and Blake A Richards. Towards deep learning with segregated dendrites. eLife, 6, December 2017. doi: 10.7554/elife.22901.
URL https
//doi.org/11 75w011005555411.....01 22901
[60] Pascal Fries. Rhythms for cognition: Communication through coherence. Neuron, 88(1):
220 200, October 2015. doi: 10. wew.66..eerron.215.00.000 URL http:/ttts///oorrgg/ee.
1016/j nwwron. 2015. .9.009... 034
[61] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning hierarchical circuits. Nature Neuroscience, 24111000-1111 May 2021. doi: 10.1039/:415933-02-005555 http:://doi..rr1100 100.00011101///11---1000 00857
URL
[62] Honi Sanders, César Rennó-Costa, Marco Idiart, and John Lisman. Grid cells and place cells:
An integrated view of their navigational and memory function. Trends in Neurosciences, 38
14

---

(12):763-775, December 2015. doi: 10.10166jttns.20151100000 URL https:///o..or.:::.. 1016/j.tine tinss. 20ins.2015..... .0.0..0.044
[63] Kwan Tung Li, Junhao Liang, and Changsong Zhou. Gamma oscillations facilitate effective learning in excitatory-inhibitory balanced neural circuits. Neural Plasticity, 2021:1-18, January
2021. doi: 10. (0.1155/0011/6668811
URL URt.httss:/// htttps://doi.orgg.
1155/2021/66632
[64] William E. Skaggs and Bruce L. McNaughton. Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science, 27122 March 1996. doi: 10.1 1 27..555...e......5.........
URL httpps://oii.rrg// wttp:/........1/.iii.ii...
271. 5257. 1870
15


Reviews:
[{"review_id": "9YkyxrX0hr", "reviewer": "Reviewer_PiE7", "summary": "The Hippocampus is postulated as a generative model to learn latent state representations and generate sensory predictions to solve spatial and nonspatial tasks. Theta-band oscillations are used to gate information flow into the generative model to modulate learning. A ring attractor develops within the generative model for path integration and flexibly transfer structures between environments. These computations are performed using biologically plausible learning rule.", "questions": "-\tWould the plasticity rule still work if it was purely Hebbian plasticity i.e. $dW/dt = p(t)*g(t)$ without the additional normalizing term?\n\n-\tThe authors contrasted the wake-sleep algorithm to backpropagation. Could the authors compare the intuition of the wake-sleep algorithm to contrastive Hebbian learning? This might be a way to describe contrastive Hebbian learning using a biologically plausible alternative.\n", "limitations": "The authors have sufficiently addressed their future directions. Model limitations and conditions at which the model breaks could have been further explored.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tScheduling neural dynamics with theta oscillator: Model neurons are usually described as input summation followed by a nonlinear transformation. Here, the authors expanded the computational dynamics of a neuron by coupling the membrane potential with the theta oscillator (Eq. 4). This affords fine-grained control or better scheduling of the neurons and plasticity using a central gating mechanism.  \n\n-\tBiologically plausible learning rule: The model seems to be an extension of the MESH architecture which uses the Pseduoinverse learning rule. Here, the network is using Hebbian like learning algorithms to learn a ring attractor for path integration and can dynamically relearn a new environment even with some of the weights are fixed, adding to the list of biologically plausible models.  \n", "weaknesses": "-\tNot a completely novel architecture: Although the multi-compartment neuron architecture controlled by the theta oscillator is novel, past works have described similar neural architecture (Sharma et al., 2022; Han et al., 2020) of having two hierarchical layers recurrently connected. I am curious to know how these neural architectures correspond to the author’s proposed architecture and if there can be some overlap in learning performance?\n\n-\tInsufficient analysis: It is not clear what computations the generative model performs using the ring attractor. The inference model seems to resemble place like receptive fields in Fig. 4b but so does the generative model? I anticipated grid like fields instead but this is not the case? It might also be interesting to show the synaptic weights ($Wgb$, $WpA$) evolve through learning and offer some insights to its computation, similar to Fig. 3c.\n\n-\tInclusion of network prediction prior to learning: Could the authors include the y axis information in Fig. 2b to show how much of the prediction error has been reduced through learning, and the activity of the inference (Fig. 2c) and generative (Fig. 2d) model prior to training to determine the difficulty of the task and the effectiveness of learning? \n\n-\tPath integration capability: Fig 3e demonstrates that the mean decoding error increases monotonically beyond 20 cm by 5 seconds. Can this high error still be claimed to be accurate path integration, where prior to the lesion, error was almost 0? What else could be integrated into the model such that the error does not increase as fast as in Fig. 3e?\n\n-\tApplication to a 2D navigation task: Authors demonstrated the model’s application to a 1D navigation task (left and right) which is rather limiting. Learning to path integrate in a 2D environment will be much more convincing. Furthermore, if the authors could demonstrate the agent ability to perform vector-based navigation by estimating its location using path integration and recalling a goal location from a memory system (Kumar et al., 2023; Foster et al., 2000), this will be a complete contribution to the research in biologically plausible spatial navigation using the entorhinal-hippocampal model. \n\n-\tInsufficient ablation studies: Ablation studies of the theta oscillator, multi-compartment neurons and individual layer plasticity will give us a better understanding of the roles played by each of these components for learning a ring attractor for path integration and remapping to a new environment. \n\nReferences:\n\nHan, D., Doya, K., & Tani, J. (2020). Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks. Neural Networks, 129, 149–162. https://doi.org/10.1016/j.neunet.2020.06.002 \n\nSharma, S., Chandra, S., & Fiete, I. R. (2022). Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold. ICML. http://arxiv.org/abs/2202.00159 \n\nKumar, M. G., Tan, C., Libedinsky, C., Yen, S., & Tan, A. Y.-Y. (2023). One-shot learning of paired associations by a reservoir computing model with Hebbian plasticity. ArXiv. http://arxiv.org/abs/2106.03580\n\nFoster, D. J., Morris, R. G., & Dayan, P. (2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1), 1–16. https://doi.org/10.1002/(SICI)1098-1063(2000)10:1<1::AID-HIPO1>3.0.CO;2-1 \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this detailed review which has led to a number of changes to the paper. We have addressed your comments and questions below but please respond with any additional questions which we’re happy to answer.  Apologies if our answers seem at all \"curt\", we are heavily constrained by the 6000 character limit but would like to cover all your points.\n\n> Not a novel architecture....Sharma et al., Han et al.,... \n\nWe weren't familiar with the work of Sharma et al. but agree their architecture is similar, including its relationship to HPC-MEC. Two differences stand out: firstly we focus on generative modelling of the temporally varying environmental stimuli but they focus on content addressable memory of static stimuli. Additionally we use multicompartmental neurons which \"multiplex\" top-down and bottom-up signals and a biologically plausible learning rule. It is exciting to see these architectures applied to hierarchical RL as in Han et al. Comparing learning performance between these models would be quite non-trivial as all three networks perform substantially different tasks (generative modelling, memory and RL). We'll add these references. \n\nWhilst there are numerous other examples of two layers recurrent architectures in ML we'd debate whether this is a \"weakness\". The combination of its features and results make our model novel even if the architecture alone is inspired by previous works. This allows us to transfer insight and intuition from these works directly our the neural system of interest, without reinventing the wheel. It is _because_ our model is related to existing generative models which allows for a deeper interpretation of hippocampal function. See general response section __Contributions__ for more on this point.\n\n> I anticipated grid like fields...\n\nLet us clarify: The MEC fields in fig 4b are a special case where we fixed HPC->MEC weights to the identity, hence why we see place-like fields in MEC (see line 520). In general our MEC representations are multimodal and grid-like (Fig S2a). Actually, they may be even more grid-like in larger environments when the attractor manifold would repeat itself due to the circular symmetry. We'll add a sentence to make this distinction clearer. Should the paper be accepted we'll be happy to add the temporal evolution of the weights as a figure to the supplement. \n\n> Inclusion of network prediction prior to learning...\n\nThank you; we will make your proposed changes to the figures. Currently the y-axis in Fig 2b is log-base-10 meaning the $g$ prediction error decreases by 10x and $p$ by slightly less. \n\n> Path integration capability...\n\nThis rate of error accumulation is only slightly worse that comparable models e.g. Sorcher et al. 2023, which reaches $\\sim$10 cm error after 200 cm distance-travelled (=> 4 seconds at our agent's speed). Apples-to-apples comparisons are hard: we focus on small networks of $\\sim$100 neurons and local learning rules which will always be outperformed by large networks (~4000 neurons) trained using backpropagation through time.\n\nSteps to improve our path integrator (which we'll mention in the paper) include\n\n* Gate some sensory data into soma during path integration, mirroring nature's lack of full sensory lesions (e.g. even in the absence of visual cues, here-unmodeled olfactory or tactile cues would persist).\n* Increase the number of neurons.   \n\n> Application to a 2D navigation task...\n\nSee general response section __2D environment__ for discussion and preliminary results.\n\n> ...vector-based navigation... \n\nWe agree vector navigation merits further study; being tied to grid cells and continuous attractors, akin to those within our network. Regrettably, time constraints prevented including these extensions in our manuscript as we focused on novel results instead of replicating existing ones.\n\n> Insufficient ablation studies...\n\nSome ablation studies have already been performed to stress test the model\n* Lesioning of sensory inputs (Fig 3). \n* Relaxing the constraint that HPC receive unimodal spatial inputs (Fig. S2b) \n* Relaxing the identity constraint on weights from MEC to the conjunctive units (Fig. S2c) \n\nIn the attached pdf you’ll find results for three new ablation studies, summarised in the general response section __Additional simulations__. These were performed in response to your review. Thank you for this valuable suggestion which has led to a geuinely interesting set of new results. We will include these (and any other ablation studies you feel important) in the supplement.\n\n> Would the plasticity rule work if it was purely Hebbian...\n\nNo. The second term can be viewed as a normaliser or equally as the weight-dependent term in the “target” equation $U-V(w)$ (see Urbanczik, Neuron, 2014) pulling the dendritic voltage towards the somatic voltage until they are equal. Without it there is nothing to stop the weights from continuing to change/grow once they reach their target value of $U$ and blowing up. We’ll include a sentence to illuminate this intuition in the paper. \n\n> ...compare...to contrastive Hebbian learning?\n\nWe aren't experts on CHL but were interested to read your comment and agree there are a number of intriguing similarities including alternative target-driven and non-target driven phases and two-term Hebbian-like learning rules. Without a more detailed mathematical analysis, which is arguably out of scope for the current paper, it is hard to say anything more strongly than this. Instead, may we suggest that we can flag these similarities in the paper for others to take note of and look forward to investigating this in future work as we likewise suspect there may be a deeper connection between the two. \n\nWe hope our responses have addressed your inquiries and that the changes have enhanced the manuscript. If so, we'd be grateful if you'd consider revising your score to reflect this. If there are any specific actions we could take to further improve it, please let us know.  "}, {"review_id": "tv8Jc1QRs8", "reviewer": "Reviewer_UjUg", "summary": "This work presents a neat model that incorporates aspects of hippocampal function under one umbrella: \n* first, the input from the environment (z) goes into the sensory layer (p) and activates the internal state (g) in a certain way, the model captures this an \"inference\" or \"wake\" stage of the training\n* next, a set of recurrent connections in the \"internal state layer\" g simulates the prediction-making, guessing the next internal state, simulating the predictive mechanism in navigation (or potentially other cognitive functions)\n* finally, a generative pathway goes the opposite way from g down to p and attempts to generate what the sensory input should look like\n\nTraining of the inference and the generative parts of the model is alternative with a 5 Hz square wave function, which reflects existing observations of the role of this oscillation in hippocampus.\n\nThe contribution of the paper is just to propose a model that describes the dynamics of simple dynamical processes, with potential extension to path integration.\n\n------------- Update after the rebuttal period -------------\n\nI would like to thank the authors for a very detailed rebuttal and being engaged in the discussion on both the technical and the ~philosophical levels.\n\nMy main criticism was based on the idea that just training a model that captures something is not enough, because there is an infinite number of models & architectures that will be able to do that. However, after the rebuttal period and skimming the paper again I came to think that the model presented in this paper achieves more that just capturing the dynamics by \"any means necessary\", but actually does so under heavy restrictions, and, while there are still multiple such models possible, the fact that the model still works under said restrictions makes this result interesting.\n\nI am raising my score to \"7: Accept\" as this work, in my estimation achieves exactly the required level of \"high impact on at least one sub-area\" (computational modelling of hippocampal formation) and has \"good-to-excellent evaluation, resources, reproducibility\".", "questions": "Fig 1: What would be the appropriate intuition for the g layer? If z is \"input\" and p is \"sensory\", then should we think of g as an \"internal state\"?\n\n95: Are there studies that support the idea of the theta rhythm acting as the direction switch? Are there recordings that somehow show the change in the direction of the flow of information?\n\n147: Although in the Introduction (line 102) you mention that there is not discrete time assumption and the whole thing works continuously, I still can't help by wonder how many weight updates \"fit\" into one 5 Hz cycle for \"wake\" and \"sleep\" steps? Is this number driven by the computational power of the simulation hardware, cramming as many updates as possible, or there is a biologically-informed constrain that says something like \"within one theta cycle synaptic connection can be updated only X times\"? How does your implementation of the model handle this? Note: while the formulation is continuous, the act of updating is still a discrete step that takes place at some point, and this is what my question is about, basically how many times within one cycle does the line of the code that does w = w + update is being executed?\n\n156: Is it correct to think of g->g recurrent connection as a step in the internal representation space where the brain tries to predict what the next (t+1) state will look like? And this internal representation is then used as a starting point for the generative part of the model to correctly generate the state of the environment z_t+1?\n\n161: What is the \"ground truth\" for training w_p_A and w_g_A in the wake cycle? How do we know what what we should train toward do? I understand that the training is local and works to minimize the differences between dendritic (apical for the wake cycle, right?) and somatic activation, but how do we know that the dendritic activations represent the \"true\" ones and that bring the the system closer to them will result in successful learning?\n\n161: Same question rephrased: what is considered to be correct output of the inference part of the model?\n\n164: What is the ground truth of the generative training step? Let's say I run 1 pass from g -> p -> z, where does the learning signal come from? Do we compare the generated z_t+1 with the actual one to know whether the generative step has generated the correct stuff? What is considered the \"correct\" output of the generative part of the model and what this output is compared with in order to be able to say \"yep, it has generated correctly\"?\n\n209: When you say that only the \"wake\" part of the model is activated, how do you conclude that is \"correctly autoencodes\"? I guess this question is still stemming from the ones above, but I fail to see what is considered the ground truth? Does it go all the way to z, or it is sufficient at this point to train w_g_B (as per Fig 1) to produce such a g_t, that after g->g step that will create g_t+1 this g_t+1 will correctly be reconstructed via w_p_A back into original p?\n\n209: Another question here is when you \"autoencode\" do you autoencode p_t into p_t or do you \"autoencode over time\" so that p_t autoencodes into p_t+1? If not the latter, then I am confused when does the g->g step happen and what's the intuitive meaning of that step.\n\n(Sorry for the avalanche of questions, I imagine these will be hard to parse, I tried to formulate them the best I could :) )\n\n233: If the hippocampal \"internal\" part of the model is trained to autoencode g->p, then why would we expect sensory lesion to affect that in any way? You say it is remarkable that is keep generating, but isn't it exactly the behavior we would expect? I am trying to understand why the fact that g->p keep working is \"remarkable\" when we turn the z off. Since z is not part of that machinery, isn't it trivial that g->p will keep doing it's thing? Please let me know what I am missing and if the observation is not trivial as I postulate, then would would be the trivial behavior that we would expect but that is remarkably not occurring?\n\n280: It would be interesting to confirm that indeed MEC has learned some basic property of path integrating by running an ablation experiment, where MEC is randomly initialized and for 30 minutes on the MEC <-> HPC is allowed to train. As with echo networks it might the case that learning MEC <-> HPC is always sufficient even is MEC internal state is random. In the ablation experiment you can check whether using pre-trained MEC is any different from using randomly initialized MEC (given the same approximate shape of the value distribution), would it result in longer re-training times, worse performance, etc or not.\n\n301: Would anything change (in terms of the attractor, performance, etc) in instead of 5-10 Hz oscillations between wake and sleep you would use 1 Hz? 20 Hz? Is the 5 Hz crucial to the success of the model, and if yes - what would \"break\" is another rhythm would be implemented?\n\n* What was the main reason for choosing very simple experimental environment for this work? A 2D navigation experiment would help show so much more and build empirical connection to place/grid cells.", "limitations": "There is not discussion of the limitations in the paper, and while I personally do not consider it a must-have, in this case it would be useful to help understand how far-reaching the authors deem their clams and results and which aspects of it rely on strong assumptions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This is a well-written paper, it has a good flow and is quite understandable. The proposed model combines various ideas about brain function in an elegant way and, while remaining simple, does manage to make those ideas to work in unison. The relevant work section gives great context for the work. The experimental work is well explained and documented.", "weaknesses": "As someone who is not coming from the attractor perspective I fail to understand the significance of an attractor appearing. In my mind (please help me understand why this is not the case) the appearance on an attractor is exactly the goal and purpose of training. Basically, the way I see it, if we have a certain dynamical process, and we have successfully trained a neural network to capture this dynamics (the model can predict s(t+1) from s(t)) then saying \"it formed an attractor\" is equivalent to saying \"the learning has converged\". But isn't that precisely the purpose of training the model? The mere fact of it converging (and forming an attractor that captures the training data trajectory on a manifold) is, of course, a good thing -- the learning was successful, but is not in any way unexpected or remarkable. After all this is precisely what we wanted to happen -- we make an effort to build a model and the learning process that captures the data, and if done correctly, that's exactly what it is going to do.\n\nIn this work the fact that the generative part of the model had formed an attractor (aka \"can generate the dynamical process correctly\") is presented as a significant outcome. But from the machine learning perspective this is a trivial result, that IS what happens when you train something successfully and is not sufficient to imply a special biological significance of the model.\n\nPut it this way: if I would make this model to have not 1 sensory and 1 hidden layer, but, let's say 3 sensory and 4 hidden layers -- it would, of course, also form some sort of an attractor (with a more complex underlying manifold and shape due to the fact that we have more layers), but that would not tell me anything about this model's biological plausibility.\n\nI guess what I would like to discuss with the authors and other reviewers is whether creating A machine learning model has scientific value, or (as I posit) we need to make a step further and offer a model that would satisfy more than just capturing the dynamics, but also make predictions or coincide with biological restrictions that we did not explicitly encode, etc. Then -- yes -- we could say that not only this model captures the dynamics, but it also the only variant (or at least belongs to a small family of variants) that also do X, Y and Z the way the brain does it.\n\nIn the olden days of modeling proposing a set of differential equations that capture the dynamics was impressive because it was not a given than such a set of equations can ever be found. The outcome of such scientific endeavor could fail if the scientist was not able to describe the dynamics. So when a model was found nonetheless it made the contribution significant. But can we really apply the same criteria to ML-driven modeling? Because in the case of ML it is almost a given, that a \"set of equations\" (now represented by an artificial neural network) will be found to satisfy the data. Which tells us nothing more that \"this data can be described\" (which is rather noncathartical) and the is likely to be an infinite amount of such models that would do so (even the same architecture, trained again from a different initialization starting point would likely produce a \"new\" model).\n\nI am looking forward to discussing this with other reviewers and the authors, to help me understand (1) am I wrong about the (in)significance of an attractor appearing and (2) whether it is too much and prohibitive for science to ask more from a model in order to count it as a significant contribution to the field than just it being able to capture the data, especially when this data is simulated?\n\nFor the authors, one way to make this critique constructive and actionable I guess would be to say more in the paper about why this particular model (it's architecture and other characteristic) is especially suited to described hippocampal dynamics and why all the same conditions would not be satisfied by any number of similar models.\n\nOther points:\n\n* It would be great to see some ablation experiments that would help understand that the observed experimental results are unique to the proposed model and would not emerge if some of the critical components of the model are turned off (and help assess their criticality).\n\n* I also ask this as a question below, but I wonder why 2D environments were not explored experimentally? It would add so much to the work, make it relevant to more people and help build connections to existing work on hippocampal function, in particular place/grid cells.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this incredibly thorough review. Your insightful comments have led to meaningful improvements. We respond point-by-point below but these first paragraphs are reserved to further your philosophical discussion about the goals of computational modeling. \n\n### General response about the merits of computational modeling.\n\nIn ML, models are “trained” to minimize an objective function almost always with some type of gradient descent. In this case, users “put in” to their objective functions exactly what they want, and often manage, to get out. As you said, in the olden days this was merit-worthy but less so today with the powerful algorithms/architectures available. Even language comprehension is arguably a solved problem. \n\nOn the other hand, in computational neuroscience people often describe “learning” which, at a coarse level, involves more simple learning rules reminiscent of those ubiquitously observed across the brain. End-to-end optimization of objectives with backpropagation is considered implausible. \n\nWe are strongly in the \"learning\" regime. Obviously with the right objective function and optimizer we could have achieved the same results. A network trained this way would perform better but it would be sufficiently trivial that, without further justification, we wouldn’t have submitted it to NeurIPS. In our case we view the attractor and other results of learning as _emergent_ properties. As we’ll elaborate in the point-by-point it was not the only possible solution and, although as scientists we were guided by intuition to suspect this solution, it was in no way “built in” to the optimisation procedure.\n\nOf course an objective function (sort of) exists – in our case we liken the system to a Helmholtz machine optimizing an ELBO-style loss – but that does not make the solution trivial since the learning rules, architecture and dynamics with which we (approximately) optimize this are heavily constrained by biology: No back prop, no long memory traces, no non-local information. The primary value of our paper is therefore demonstrating that the learning rules and architecture of HPC can be seen through the right mathematical lens as roughly the same as a well studied machine learning system (we don't think anyones done this before). To put it another way, we took a step towards discovering the objective function of HPC and (to us at least) this finding is profound; speaking to deeper organizational principles at play within the brain. \n\nAll said, we believe we are in firm agreement with you: the value is absolutely not in a model converging but in what this teach us about the brain. We believe the goal of modeling should be to (i) unify previously disparate concepts and (ii) make testable predictions. On both accounts we believe we are achieving this goal. Thank you again for opening the discussion on this important topic. If you agree we’d be happy to carry on the discussion below and if it’s made you reconsider the contribution of our paper we’d be grateful if you might revise your rating to reflect this. \n\nWe'll now respond point-by-point but the character limit means we won't fit all your questions in this box. We hope it is ok if we finish responding to your questions in a comment which we'll post after the rebuttal period closes. \n\n### Point by point response (part 1/4)  \n\n> ...I fail to understand the significance of an attractor appearing....an attractor is exactly the goal and purpose of training.... \n\nPerhaps it would help to begin by clearly stating a question which, prior to our work, we do not believe there would have been consensus within the field about the answer to: _“Can unsupervised local learning in a system with one hidden layer receiving unstructured spatial inputs learn transferable path integration or is back propagation through time and/or deeper non-linear architectures a requirement?”_ \n\nState-of-the-art before us was that relevant HPC modelling almost always used non-local learning rules and powerful optimization algos which find path integration / attractors as, in your words, this was “precisely the purpose of training” (e.g. Sorscher et al. 2023 and Banino et al. 2018). It's unclear if this was because authors weren’t interested in biological plausibility or if its because it was _necessary_ for successful training.\n\nTo reiterate our earlier point we don’t just put in what we get out. We dont just do literal gradient descent on a hand-crafted objective function. The appeal of the attractor is exactly that it emerged from a system so heavily constrained by the realities of biology, lending weight to the suggestion that our model may contain some truths about the brain. \n\nAn observation from Fig 3c may help: notice a subset of neurons have converged on a trivial but perfectly valid solution of zero weights. They “died”. Their existence neatly demonstrates that a functioning attractor manifold over all/most hidden units is not _the_ solution but rather _a_ solution. We are the first to demonstrate this solution can emerge under the constraints used. Also, (see points further down) the system doesn't predict $s(t+1)$ from $s(t)$ but $s(t)$ from $s(t)$ making the future-predictive nature of the attractor more remarkable.\n\nThis is why we respectfully disagree with the statement that  \n\n> saying \"it formed an attractor\" is equivalent to saying \"the learning has converged\". \n\nSure, learning converged. But nothing a priori guaranteed an attractor would be the fixed point.  In the remaining space we will add sentences to clarify our interpretation of this result and also reword instances where we have lent too far toward the interpretation that convergence alone is the key take home result, which it is not. \n"}, {"review_id": "Lql0SU72pY", "reviewer": "Reviewer_iw5X", "summary": "In this work, the authors give a continuous version of the 'impression learning' [1] and use the theta oscillation to modulate wake-sleep phase.\n\n[1] Bredenberg, Colin, et al. \"Impression learning: Online representation learning with synaptic plasticity.\" Advances in Neural Information Processing Systems 34 (2021): 11717-11729.", "questions": "The latent variable $z$ is not used in the current model, so just disregard it. Eq.(1) can be ignored. If you want to use $z$, there must be a decoder from the neural activity $r$ to $z$.", "limitations": "None", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "The authors links the theta oscillations to $\\theta_t$ in 'impression learning', which gives a neural implementation of the wake-sleep algorithm in Helmholtz machine.", "weaknesses": "From a computational perspective, the contribution lacks novelty.\n\nIn Seciton 5.2, the authors clearify their relationship to 'impression learning'. They list four differences:\n\n1. continuous version. \n\nThe discrete version is equivalent as long as the time steps are sufficiently small. And the authors did not provide the mathematical form of the Evidence Lower Bound (ELBO) for the continuous-time version.\n\n2. $\\theta_t$ can be ignored.\n3. disregard $1-k_t$.\n\nThese are only technical improvments.\n\n4. Link the theta oscillation (5Hz) to $\\theta_t$\n\nThis is a special case of  'impression learning'.  And the authors did not provide an explanation of how it is implemented from a computational standpoint. They merely established the connection without detailing the computational aspects.\n\nOverall, compared to 'impression learning,' I do not believe this work has enough novelty from a computational perspective. Although it provides a biological explanation of 'impression learning,' I do not think this paper is suitable for the current venue. Perhaps journals like Nature Neuroscience/Communications, or eLife would be more appropriate.\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you this review. We’d like to clarify our contributions in case they were misinterpreted. As the manuscript states; “The primary contribution of this paper is to introduce a biologically plausible model of sequence learning in the hippocampus which unifies its capacities as a generative model of sensory stimuli and path integration under one schema”. The intended goal was to shed light on brain function using insights from the ML and not – at least not primarily – to introduce an entirely new model of representation learning (although we make some minor contributions in this regard). \n\nA criticism leveled within this review is that our model doesn't have \"enough novelty from a computational perspective\". We’d argue this critique slightly misses the point and is oversimplified. Nobody has proposed to model hippocampus as a hierarchical Helmholtz machine nor implemented such a model with learning rules, dynamics and architecture so closely matched to biology, nor shown such a model works. This is a novel computational model.\n\nPerhaps the question comes down to scope rather than significance. We are grateful for your comment recognising this work as suitable for high ranking “journals like Nature Neuroscience/Comms”, however, we also believe that NeurIPS is _the_ preeminent venue for research at the interface of neuroscience and ML and that biologically plausible implementations of key ML algorithms lie well within its scope. A myriad of NeurIPS papers support this claim:\n\n* Pogodin et al., NeurIPS 2023, Towards Biologically Plausible Convolutional Networks\n* Greedy et al., NeurIPS 2022, Single-phase deep learning in cortico-cortical networks\n* Liu et al., NeurIPS 2022, Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators\n\n...being the most recent. See also:\n\n* Koren et al., 2022, Pemberton et al., 2022, Lipshutz et al., 2020, Bahroun et al., 2021, Clopath et al. 2007, Hasselmo et al., 1994, \n\n... all published in NeurIPS. \n\nNeurIPS mission statement is to “bring together researchers in machine learning, neuroscience, statistics, …” with listed relevant topics including “Neuroscience and cognitive science (e.g., neural coding)”. Given the comments and ratings from the other three reviewers we are confident that this sentiment is still held within the community today. \n\nIn light of this discussion and the additional comments/changes made below we’d be grateful if you’d consider revising your rating and will be happy to attend to changes you consider necessary. Thank you again for the review.\n\n### Point by point response \n\n> The discrete [and] continuous versions are equivalent...\n\nYou're correct, shifting to continuous equations was not fundamental however it allows us to directly compare to real world temporal phenomena e.g. we can say how long in seconds, rather than how many \"steps\", the path integrator is accurate for. \n\n> authors did not provide the mathematical form of ELBO in continuos-time \n \nAlthough we agree this is a great suggestion it would be less trivial than it may seem. Unlike, say, Bredenberg et al., we did not derive our learning rules starting from a loss function hence, given the modifications we made, its not guaranteed one would even exist. We'llcarry on giving the discrete formula in the supplement. \n\n> ...are only technical improvements.\n\nThese technical improvements are not, themselves, the core contributions of the paper though they may still be considered minor relaxations of past results. \n\nThese four point you raised were merely the changes we made to the impression learning architecture for experiment 1. Of course in later figures/experiments we made additional changes which are novel too. These include:\n\n* The addition of self-motion inputs, which is crucial for path integration, to the latent hidden layer. \n* A significant increase in the number of neurons per layer.\n* Training the network on stimuli deriving from a complex and biologically relevant spatial exploration task.\n* Continuous synaptic noise (see methods)\n* ...\n\n> no explanation of how it is implemented from a computational standpoint\n\nWe are confident all equations (including learning rates, initialisations, time constants and noise additions) have been detailed in the paper and supplement however if you inform us which equations are missing we’ll be happy to add them. Additionally, section “5.2 Relationship to online Bayesian Inference” connects to our model to impression learning.\n \n> ...I do not believe this work has enough novelty... \n\nTo reiterate we consider the core novel contributions of our paper to be\n\n* Creating an interpretable neural model for hippocampal function inspired by previous literature on Helmholtz machines. \n* Successful training of this model in biological regimes to demonstrate it replicates numerous aspect of hippocampal function. \n\n> ...this paper is more suitable for...journals like Nature Neuroscience/Communications...\n\nThank you, and we take this statement as implying the content of our work is good but that you feel the topic is out of scope for NeurIPS, in which case we respectfully disagree. Please see our above arguments. \n\n> The latent variable z is not used in the current model, so just disregard it...\n\nWe had difficulty understanding this comment - if you could clarify we would be more than happy to address this. z is an important variable as it represents the underlying environmental latent so it cannot be discarded. \n\nThank you for this response. You’re right that not _all_ components of this model’s architecture are entirely novel. This need not mean it lacks computational novelty outright, nor was creating a new ML model ever the intended goal. Even for readers with no interest in biology, bioplausible implementations of ML algorithms are within scope as these lead the way toward novel implementations that better our understanding of how artificial intelligence relates to its biological counterparts.\n"}, {"review_id": "tnB9DRCsxr", "reviewer": "Reviewer_Lhyx", "summary": "The paper is an application of the learning scheme derived in Bredenberg et al. 2021 to representation learning and path integration in the MEC and HPC. By adapting the aforementioned scheme to continuous time, and by relating the proposed oscillatory 'gating' signal to theta oscillations that have been observed to gate dynamics in HPC and MEC, the authors propose to model representation learning in these two brain areas as a form of rapidly alternating 'wake-sleep' learning where brief periods of generative prediction dynamics alternate with more standard position-driven dynamics and deviations between path-integrated predictions and sensory stimulus-driven inputs are used to drive learning at both apical and basal synapses. Subsequently they demonstrate that their MEC model is able to capture two interesting features, learned exclusively through local synaptic plasticity rules:\n\n1. The MEC is able to learn a ring attractor capable of path integration based only on motion cues that is able to accurately predict subsequent dynamics.\n2. This attractor can easily generalize to a new environment without modifications to its recurrent synaptic weights.", "questions": "Could you elaborate on simplifying assumptions made about hippocampal circuitry? It seems to me that the hippocampus plays no role in inference here (it just relays sensory information as an identity function without any plasticity and plays no role in path integration), whereas in your model the 'unlearned' sensory layer that prevents representational collapse could have been an earlier stage of sensory processing like the thalamus, allowing basal synapses in the hippocampus to learn as has been observed in experimental data (e.g. BTSP).\n\nWhat happens in this model during 'sleep'? I.e. if there are no vestibular cues provided to this network, does it still sample along its path integrator in a way that preserves spatial information? How does this relate to existing experimental information on sleep 'rehearsal' in the hippocampus?\n\nWhat happens if more noise is introduced into dynamics of the network during both inference and learning? In particular, I'm concerned about the absence of an explicit gate on plasticity. While it is conceptually an improvement to remove gating of plasticity tied to the theta oscillation, in the model of Bredenberg et. al 2021, the update is not strictly zero for inference parameters during the inference phase--it is only zero in mean (the same is true for the generative phase). As noise grows in the neural output, it is possible for increased variability in these updates, which could longitudinally compound errors or cause some form of representational drift. Have you tested for this possibility? If not, it may be important to note that the gate is only unnecessary in near-deterministic noise regimes.", "limitations": "There are no obvious negative societal impacts of the authors' work, and beyond the questions I've raised above, the authors do an adequate job of addressing the limitations of their work and explicitly relating their contribution to previous studies.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "While previous models have successfully captured the features I previously mentioned through various optimization schemes, they have not shown how such learning could occur through local, biophysically-motivated synaptic plasticity which can be cleanly related to any representation learning objective function (in this case, the authors relate their learning to a variant of the ELBO objective). This is the principle success of the paper, and in my mind could prove to be a very important model for learning orchestrated between the MEC and HPC.\n\nThe authors also relax several constraints on the learning derived from Bredenberg et al. 2021:\n1. They generalize learning dynamics to continuous time.\n2. They include self-motion information in the 'generative' part of learning dynamics, which allows for path integration.\n3. They relax constraints on gating of plasticity that are tied to theta cycles, simplifying learning and requiring fewer assumed variables that are locally available to synapses.\n\nFurthermore, the paper is clearly written and logically presented.", "weaknesses": "To me, the principal weaknesses of the paper come from additional analyses that the authors could have done, but did not do. I will list several below:\n\n1. The authors do not test their learning algorithm in more complex environments. Simple two-dimensional environments such as boxes or circles could be used to test whether or not the learning algorithm is sufficient to develop grid-like cells in MEC. More complex image-based environments could be used to test the scalability of the proposed learning algorithm, and could potentially test whether or not the proposed learning algorithm is also sufficient to develop place cell responses (rather than having them hand-provided for the system). Given that algorithms similar to Wake-Sleep do not scale as well to high-dimensional problems as backpropagation and that the brain is presumably able to learn place cell information indirectly from multimodal sensory input (vision, audio, vestibular inputs, etc.), this is a valid concern.\n\n2. The authors do not discuss testable predictions for their learning algorithm--I can identify several, which I would like to see discussed in more detail. In particular, though the authors suggest that after learning has converged the 'generative' and 'inference' phases of learning 'become indistinguishable,' this is both untested and unlikely to be true for nondeterministic systems and agents operating in partially observable environments with stochastic transitions. In particular, conditioned on the system being in the generative phase, stochasticity in the neural network and uncertainty in the environment should cause the precision of the path integration estimate to decrease throughout time (as the system advances deeper into the theta phase). In fact, the authors' results (Fig. 3e) appear to show exactly this phenomenon. In particular, this is closely related to results from Ujfalussy & Orbán,  Elife, 2022, which use similar phenomena to support the notion that some form of sampling tied to theta phase oscillations is occurring in the hippocampus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, we are glad that you find our contribution “very important” and “clearly written”. We respond to your key points below and including an additional simulation to answer one of your questions. Please kindly inform us if there's anything else required to boost your score.\n\n> ...The authors do not test their learning algorithm in more complex environments. e.g. two-dimensional...[or]... image-based environments.\n\nPlease see general response section __2D environments__ for discussion and preliminary results.\n\nRegarding image-based environments we share your suspicions that wake-sleep with no add-ons may not scale well. We made sure not to overstate our claims in the paper. In fact, the reason we chose to model hippocampus, not earlier sensory cortices, is precisely so we can ignore the complexities of these regions and focus on the core computations relating to structure learning in HPC. This is a similar approach to Sorcher et al. (2023), and Whittington et al. (2020). We acknowledge our proposal is only a partial solution and won't account for all learning within the brain.\n\n> The authors don't discuss testable predictions...In particular...that after learning has converged the 'generative' and 'inference' phases of learning 'become indistinguishable,' [which is] unlikely to be true for nondeterministic systems. \n\nThanks you for suggesting this. We agree and will make this point clearer in the manuscript by removing the word 'indistinguishable'. Of course environmental or neural stochasticity will always cause the generative predictions to “drift” and prevent total convergence. An additional point we’ll add is that we speculate the presence of this drift within each sleep-phase could explain why fast neural rhythms (5 Hz) are used, as they prevent drift accumulating and damaging perception.\n\nRegarding other testable predictions from our model, we list a few here:  \n\n* Decoupling HPC from sensory inputs shouldn't immediately damage path integration abilities since this comes from MEC.\n* Decoupling HPC from MEC should damage path integration but not inference. \n* The effect of inhibiting layers in the hippocampal formation optogenetically should depend strongly on which phase of theta we inhibit at. \n* Synaptic learning rules in hippocampal and entorhinal populations may be fundamentally similar.\n\n>Could you elaborate on simplifying assumptions made about hippocampal circuitry? It seems that the hippocampus plays no role in inference...\n\nThat's mostly correct although, as we demonstrate in Fig S2, the mapping from sensory to hippocampus doesn't have to be the identity . Also, hippocampus relays information _to_ MEC but also recieves information _from_ MEC and these synapses are all fully plastic. As you correctly identified HPC is not involved in the path integration mechanics which is precisely why the system can \"transfer\" this ability between environments once hippocampus has learnt to translate the new sensory inputs into the old MEC code. So its role is not trivial. \n \nYou're also correct that fixed sensory inputs prevent representational collapse in HPC but they wouldn't prevent collapse in MEC. Since MEC didn't collapse this suggests we could extend the model deeper to include a thalamus which drives HPC via learnable synapses. This is almost definitely closer to the what happens in the brain and would be an interesting future direction. For now we avoided this complexity to focus on our specific goals of answering questions about path integration and structure transfer. \n\n> What happens in this model during 'sleep' if there are no vestibular cues...\n\nIn our model, if there were no vestibular (aka velocity) inputs this would be equivalent to path integrating a zero velocity so the bump attractor would stay still. Synaptic noise could, in theory, cause this bump to randomly drift and this drift would preserve spatial information, i.e. it would not jump randomly from place to place. In practice, however, the quasi-continuous nature of our attractor makes each location a very slight local minimum, stabilising the bump against drift. This \"stickiness\" during path integration was also observed by Vafidis et al. (2021, eLife) and may be a problem for agents attempting to perform path integration at very low velocities.\n\nIn terms of how this relates to hippocampal \"rehearsal\", relevant hypotheses include:\n\n* Large quanities of noise overcome the quasi-continuous stickiness and drive drift during sleep/replay in the absence of vestibular inputs. \n* Top-down processes drive \"artificial\" vestibular inputs which push the bump around the manifold. This hypothesis was put forward, but not tested, in an opinion piece by Sanders et al. (2015, Trends in Neuro). \n* Feedback inhibition \"destabilises\" the bump causing it to move of its own accord (Chu et al. NeurIPS 2022) \n\nOur model is well place to answer these questions but, with the time constraints of this rebuttal, we look forward to tackling it in future work.\n\n> What happens if more noise is introduced into dynamics of the network during both inference and learning? In particular, I'm concerned about the absence of an explicit gate on plasticity...\n\nThis is an astute observation. One of the core philosophies we take is to avoid hand-crafting learning rules to include terms which may be non-trivial for biology to implement. Explicitly gating plasticity at sub-theta timescales is one such example. Instead we answer empirically, via simulation, whether this term in the theoretical learning rule strictly necessary? \n\nWe find that plasticity gating is _not_ necessary. To test this, and in response to your review, we ran the same path integration experiment but with __10 x more noise__. As shown in the attached pdf this results in noisier but still accurate path integration. Only once we increase noise to 100x does learning fails. Thank you for encouraging us to check this, we'll add this result into the supplement. "}]

Author rebuttal:
We thank all reviewers for their detailed and thoughtful comments and are glad they found it to be a "well written" and "logically presented" paper about a model which "could prove to be very important". We respond to each review individually but, for the benefit of all, here summarize three major aspects of our responses including additional experiments we have performed. 

### Contributions

To reiterate our contribution: this paper substantially builds on an existing literature modelling the hippocampal formation's role in navigation (including path integration) and transferable structure learning. We take the novel approach of formulating it as a Helmholtz machine constructed from hierarchical layers of multi-compartmental neurons and show it can be "trained" with simple, local and biologically plausible Hebbian learning rules which can be approximately derived starting from an ELBO objective function. 

Additionally, we show how the wake-sleep algorithm can be implemented by somatic gating between basal and apical inputs controlled by the hippocampal theta-oscillation. All in all our paper links deeply theoretical ideas about generative models (e.g. wake-sleep) to puzzling biological concepts (e.g. neural oscillations) in a manner we hope can seed fruitful discussion and progress.
 
We want to make clear that our intention to model HPC as a Helmholtz machine (as opposed to dreaming up or deriving an entirely new architecture) was a very concious choice. By not reinventing the wheel we can -- and hope the community will -- transfer insights from historical and recent ML research into generative models onto our understanding of the hippocampal formation. In doing so we avoid having to explain away lots of complexities with biologically implausible techniques such as back propagation as was done by others before us. 

Our contribution should be of interest to both neuroscientists and ML'ers, for whom bioplausible implementations of key algorithms shed light on how the brain, and perhaps future AIs, can implement ML models in more data- and energy-efficient ways. 

### Additional simulations

Reviewers suggested that additional ablation studies and experiments would strength our results. Here we summarise 3 additional experiments displayed in the attached pdf: 

* __Removing plasticity from recurrent MEC synapses__. (Fig R1a)  The result is that learning fails and path integration does not emerge indicating recurrent plasticity is crucial and that interlayer plasticity alone cannot bypass the need for a tuned attractor manifold. As suggested by reviewers PiE7 and UjUg.
* __Removing plasticity from HPC $\leftrightarrow$ MEC synapses__ (Fig R1b) results in HPC never achieving internal consistency between basal and apical inputs. Despite this the model can still path integrate. The interpretation here is that HPC simply relays sensory information into MEC but cannot then “translate” MEC predictions back to the sensory code. This would be a problem for a real behaving agent who could not then easily combine sensory or internal predictions and would need two decoders to predict position (one for when the hippocampus is in wake mode, and another for sleep). As suggested by reviewers PiE7 and UjUg.
* __Increasing the synaptic noise 10x__ (Fig R1c & d) confirms that learning path integration is robust up to substantial amounts of noise (Fig R1c & d). As suggested by reviewer Lhyx. 

We thank the reviewers for these suggestions leading to interesting new results which we will include in the manuscript. 

### 2D environments 

Three reviewers mentioned that our results would be more convincing in 2D. To test this __we performed some preliminary experiments in 2D__, with the results shown in the attached pdf. Now the agent moves in a 2D environment and has four sets of conjunctive cells in charge of controlling north, south, east and westward motion. Just like in 1D, after learning we observe the local "centre-surround" nature of the learned recurrent MEC synaptic connections (a hallmark feature of bump attractors, Fig R2b left), and also that the synapses from the four sets of conjunctive cells are each skewed in four opposite directions (a hallmark feature of path integration, Fig R2b right). Fig R2c shows path integration where every 1 second sensory input is provided to recalibrate the 2D bump attractor. In between these instances the activity bump remains stable (albeit only briefly).  This result indicates that path integration abilities are beginning to emerge in the 2D model. Note that, due to computational and time constraints, this model has not been tuned nor trained for the required amount of time and therefore it is likely we are not seeing optimal performance of the bump attractor.

To be clear, we do not intend these results as a comprehensive exploration of 2D and recognise they leave many questions unanswered. Unfortunately 2D takes substantially longer to simulate and tune due to the increased complexities of motion modelling and the much larger number of cells required. For these reasons, properly exploring 2D remains out of scope and we choose to focus only on 1D as the simplest setting where all core features of path integration can be tested comprehensively. We include them here as they give us some confidence that this approach would scale to 2 - and potentially higher - dimensions.


Meta-review: This paper presents an abstract model of hippocampal learning based on a sequential version of the Helmholtz machine. The authors show that model accurately infers the latent state of high-dimensional sensory environments and generates realistic predictions. They also show that it can be used for navigation and transfer between environments (using relatively toy environments).

Overall, the reviewers had concerns related to the novelty of the paper, the clarity of what it contributes beyond previous work, and the simplicity of the environments used. After a fairly extensive set of rebuttal discussions, 3/4 reviewers felt that the paper met the bar for acceptance. As such, and accept decision was reached.
Decision: Accept (poster)