Title: A generative model of the hippocampal formation trained with theta driven local learning rules
Venue: NeurIPS 2023
Accepted: yes
Keywords: hippocampus

path integration

local learning

generative models

oscillations

inference

Helmholtz machine

wake-sleep
Abstract: Advances in generative models have recently revolutionised machine learning. Meanwhile, in neuroscience, generative models have long been thought fundamental to animal intelligence. Understanding the biological mechanisms that support these processes promises to shed light on the relationship between biological and artificial intelligence. In animals, the hippocampal formation is thought to learn and use a generative model to support its role in spatial and non-spatial memory. Here we introduce a biologically plausible model of the hippocampal formation tantamount to a Helmholtz machine that we apply to a temporal stream of inputs. A novel component of our model is that fast theta-band oscillations (5-10 Hz) gate the direction of information flow throughout the network, training it akin to a high-frequency wake-sleep algorithm. Our model accurately infers the latent state of high-dimensional sensory environments and generates realistic sensory predictions. Furthermore, it can learn to path integrate by developing a ring attractor connectivity structure matching previous theoretical proposals and flexibly transfer this structure between environments. Whereas many models trade-off biological plausibility with generality, our model captures a variety of hippocampal cognitive functions under one biologically plausible local learning rule.

A generative model of the hippocampal formation trained with theta driven local learning rules

Abstract

Advances in generative models have recently revolutionised machine learning. Meanwhile, in neuroscience, generative models have long been thought fundamental to animal intelligence. Understanding the biological mechanisms that support these processes promises to shed light on the relationship between biological and artificial intelligence. In animals, the hippocampal formation is thought to learn and use a generative model to support its role in spatial and non-spatial memory. Here we introduce a biologically plausible model of the hippocampal formation tantamount to a Helmholtz machine that we apply to a temporal stream of inputs. A novel component of our model is that fast theta-band oscillations (5-10 Hz) gate the direction of information flow throughout the network, training it akin to a high-frequency wake-sleep algorithm. Our model accurately infers the latent state of high-dimensional sensory environments and generates realistic sensory predictions. Furthermore, it can learn to path integrate by developing a ring attractor connectivity structure matching previous theoretical proposals and flexibly transfer this structure between environments. Whereas many models trade-off biological plausibility with generality, our model captures a variety of hippocampal cognitive functions under one biologically plausible local learning rule.

1 Introduction

Generative models seek to create new data samples which are similar to those from the training set. To do so they must learn the probability distribution of the training data, comprising a rich, generalisable and accurate model of the world. Many of the recent advances in AI have involved types of generative models: VAEs [1], GANs [2], diffusion models [3] and autoregressive models [4] have seeded improvements in AI capabilities ranging from data compression [5] to image generation [6] and natural language [7]. In neuroscience, the animal brain has long been known to exploit generative models [8, 9]. The ability to generate representative sensory data samples can be used directly, for example during offline planning or memory recall. It can also be used indirectly to aid training of inference networks with the goal of processing rich, noisy and high dimensional streams of incoming sensory stimuli, as discussed in the predictive coding literature [10]. In a sentence: 'What I cannot create [generate], I do not understand [inference]' (R. Feynman).

<latexi sh

1_b

64="N

d

HK

S8JuO

C

j

k2TWXw

>A

B

c

VDL

gM

F

3U

f

Z

I

mr

y

n

0o

z

Q

PG

+

v

E

/R

7

p

5

9

q

Y

<latexi sh

1_b

64="w5q/

YWF

n

oyzP

CX

M3

v

I

>A

B8

c

VDLSg

U

2

+

R

Z

p

r

0

E

k

m

J

O

Q

K

9

7

u

f

N

G

T

H

j

d

<latexi sh

1_b

64="

wH

CVDKE8

3

zL

G

W

P

/uA

>

B

X

c

SgM

F

U

f

Zd

k

I

Omj

n

2

y

YN

J

oR

q

+

T

7

0v

5

r

p

9

Q

<latexi sh

1_b

64="O2

K95y wqSp

cLj

BC

WP

G

>A

+X

VD

gM

F

3

r

U

R

Z

T

u

H

I

Jk

N

0

8

m

f

v

E

z

7

d

n

o

Y

Q

/

<latexi sh

1_b

64="3CIQO9

JF

koB

dPT

uj

Vzqyf0

>A

+X

c

DLSgM

r

U

5

R

Z

W

7

w

2

mn

p

N

H

K

v

E

G

Y

/

8

<latexi sh

1_b

64="

yYT3dDc+LvJKk

7Q

2GXZ

o0

>A

B

V

S

N

F

rPU

I

gq

u

M

f

On

m

R

H

C

5

E

j

w

9

8

p

z

W

/

<latexi sh

1_b

64="AoL+

K/

wFyYv

V

d

rW

8

>

B

X

c

D

SgM

3

PU

5CR

Z

pOm

T

u

7

k0

z

I

n

j

f

J

GN

H

9

Q

2

E

q

<latexi sh

1_b

64="

Tj3Y

O/

WRnJ

F0

M

>A

B8

c

VD

U

f

Zdu

k

wI

mr

E

Ny

G

o

H

Qz

259

X

7

K

CP

p

+

<latexi sh

1_b

64="VpFy

Uc0

CO

R

YDWw+E27o

5k

>A

H

9S8N

L3

X

Z

Q

JKI

m

r

qj

u

g

/

B

dz

f

P

n

G

T

M

v

<latexi sh

1_b

64="mUSf/

A53Hqd7zK

gYo

OC

Z

>

B8

c

VDL

N

FJ

u

v

Qkp

G

2

T

9

M

E

R

+

0

W

P

j

r

w

y

X

n

I

<latexi sh

1_b

64="+G8

H5d27O

BF

Y

M3

Jf

D

Z

>A

c

V

LS

N

U

u

ovgQkp

RT

K9

zE

/

q

0

W

P

j

r

m

w

y

X

n

I

C

<latexi sh

1_b

64="+G8

H5d27O

BF

Y

M3

Jf

D

Z

>A

c

V

LS

N

U

u

ovgQkp

RT

K9

zE

/

q

0

W

P

j

r

m

w

y

X

n

I

C

<latexi sh

1_b

64="O2

K95y wqSp

cLj

BC

WP

G

>A

+X

VD

gM

F

3

r

U

R

Z

T

u

H

I

Jk

N

0

8

m

f

v

E

z

7

d

n

o

Y

Q

/

<latexi sh

1_b

64="

yYT3dDc+LvJKk

7Q

2GXZ

o0

>A

B

V

S

N

F

rPU

I

gq

u

M

f

On

m

R

H

C

5

E

j

w

9

8

p

z

W

/

<latexi sh

1_b

64="mUSf/

A53Hqd7zK

gYo

OC

Z

>

B8

c

VDL

N

FJ

u

v

Qkp

G

2

T

9

M

E

R

+

0

W

P

j

r

w

y

X

n

I

<latexi sh

1_b

64="+G8

H5d27O

BF

Y

M3

Jf

D

Z

>A

c

V

LS

N

U

u

ovgQkp

RT

K9

zE

/

q

0

W

P

j

r

m

w

y

X

n

I

C

<latexi sh

1_b

64="mUSf/

A53Hqd7zK

gYo

OC

Z

>

B8

c

VDL

N

FJ

u

v

Qkp

G

2

T

9

M

E

R

+

0

W

P

j

r

w

y

X

n

I

<latexi sh

1_b

64="r/dKX

o

WH

pGZ0JSC

Uj

>A

B83

c

VDL

gM

F

f

u

k

wI

Om

Q

7

Y

y

+

2

9

5

E

N

q

TP

<latexi sh

1_b

64="JE30

Ndc

RLBqP

T

7j zY

o

>A

CGH

VD

MwF

KX

k

2

I

5r

O

y

f

S

Q

/

W

v

n

9

p

m

8

+

Z

<latexi sh

1_b

64="

5LO0+qBGDT/Ky

R

7A

dwI

>

83

c

V

S

N

FJ

U

f

Z

ugk

Ep2r

X

m

M

Cj

z

n

o

v

H

W

9

P

Q

Y

<latexi sh

1_b

64="TX

kCJ+UZP

Dv

8pH

3/

w

>A

B

c

V

LS

N

F

f

dug

I

E

2r

7o

m

0

M

O

G

j

R

n

KW

9

5

Qy

Y

<latexi sh

1_b

64="TX

kCJ+UZP

Dv

8pH

3/

w

>A

B

c

V

LS

N

F

f

dug

I

E

2r

7o

m

0

M

O

G

j

R

n

KW

9

5

Qy

Y

<latexi sh

1_b

64="AoL+

K/

wFyYv

V

d

rW

8

>

B

X

c

D

SgM

3

PU

5CR

Z

pOm

T

u

7

k0

z

I

n

j

f

J

GN

H

9

Q

2

E

q

<latexi sh

1_b

64="3CIQO9

JF

koB

dPT

uj

Vzqyf0

>A

+X

c

DLSgM

r

U

5

R

Z

W

7

w

2

mn

p

N

H

K

v

E

G

Y

/

8

<latexi sh

1_b

64="

5LO0+qBGDT/Ky

R

7A

dwI

>

83

c

V

S

N

FJ

U

f

Z

ugk

Ep2r

X

m

M

Cj

z

n

o

v

H

W

9

P

Q

Y

<latexi sh

1_b

64="TX

kCJ+UZP

Dv

8pH

3/

w

>A

B

c

V

LS

N

F

f

dug

I

E

2r

7o

m

0

M

O

G

j

R

n

KW

9

5

Qy

Y

<latexi sh

1_b

64="

5LO0+qBGDT/Ky

R

7A

dwI

>

83

c

V

S

N

FJ

U

f

Z

ugk

Ep2r

X

m

M

Cj

z

n

o

v

H

W

9

P

Q

Y

<latexi sh

1_b

64="

VAWU

c

+

9XNHBG

2QMFIE

>

DLSg

3

rP

k

w

zZ

p0

j

d

J

yRTK

C/

fu

v

qY

O

m

7

o

8

n

5

<latexi sh

1_b

64="O9JD

kWm+IZ

2fuQ

dAT

/

VjRo

>

B

H

c

LSgM

F

3

rP

Y

q

0N

y

E

K

n

G

w

8

z

X

p

v

5

C

U

7

<latexi sh

1_b

64="AjUVywHkE+0

S

KRo dp

/98

>

B

c

DL

gM

F

3

2vq

I

QN

m

uC

5

G

P

Z

J

f

n

z

r

W

Y

T

X

7

O

a  Network

p

g

z

c  Dynamics across time

time

wake

wake

sleep

sleep

θ

(t)

5 Hz

g

(

t

)

p

(

t

)

z

(

t

)

= fixed

= plastic

∆

w

pA

∝

p

-

p

A

wake

sleep

b

w

pA

= gated

w

Environment

= not gated

p

=

∆

w

gA

/

B

∝

g

-

g

A

/

B

wake

sleep

g

=

w

pA

w

gA

w

gA

w

p

A

p

B

p

A

p

A

p

B

p

B

g

A

g

B

g

B

g

A

g

B

g

A

Sensory stimuli

Hidden

w

pB

w

pB

The hippocampal-entorhinal system (aka. hippocampal formation) - a brain structure implicated in spatial [11] and non-spatial [12] memory - provides a pertinent example. Its primary role seems to be inference [13]: mapping sensory inputs into a robust and decodable representation of state (grid cells [14], place cells [11] etc. [15]). A generative model is thought to have a dual role in learning: supporting offline tasks such as route planning [16] and memory consolidation [17], and online during behaviour with path integration [18]. Path integration enables the hippocampal network to maintain an up-to-date and accurate estimate of its position in the absence of reliable sensory data by integrating self-motion cues. A recent flurry of computational [19, 20, 21] and theoretical [22, 21] work has highlighted the importance of path integration as a key objective explaining hippocampal function and representations.

Existing computational generative models of the hippocampal formation [23, 24] account for many of its cognitive functions and internal representations but require non-trivial learning rules and message passing protocols which don't connect with known aspects of biology. Computational models of path integration [25, 26, 27] have mostly focussed on continuous attractor networks which, although experimentally supported [28], alone lack the complexity or expressivity required of a fully general model of the hippocampal memory system.

The primary contribution of this paper is to introduce a biologically plausible model of sequence learning in the hippocampus which unifies its capacities as a generative model of sensory stimuli and path integration under one schema. To do this we propose modeling the hippocampal formation as a Helmholtz machine [29] which learns to predict sensory stimuli given the current hidden state and action (e.g. velocity). We propose a deep connection between the hippocampal theta oscillation [30] and the unsupervised wake-sleep algorithm [31] for training Helmholtz machines. Though this class of generative models isn't widely used, and lacks the scalability of the lastest transformer-based sequence learners, it excels in this context since is has many natural points of contact with biology (both in terms of architecture and neural dynamics) yet still maintains the expressiveness afforded to models of the brain by deep neural networks.

In this paper we:

introduce a new model of the hippocampal formation which learns the latent structure of an incoming stream of sensory stimuli analogous to a Helmholtz machine.

describe a biologically plausible learning regime: Theta-oscillations gate information flow through multi-compartmental neurons which rapidly switches the system between 'wake' and 'sleep' phases. All plasticity is local.

train our model on stimuli from a biologically relevant spatial exploration task and show it learns to path integrate by developing a ring attractor connectivity structure (comparable to theoretical predictions and empirical results in deep recurrent neural networks trained with gradient descent). Learning generalises: when the agent moves to a new environment, path integration capabilities recover without needing to relearn the path integration weights.

Our model of the hippocampal formation simultaneously (i) accounts for its role as a generative model of sensory stimuli, (ii) can learn to path integrate and (iii) can transfer structural knowledge between environments. The model, though here applied to the hippocampus, can be viewed as a step towards a general solution for how biological neural networks in many brain regions (for example visual cortex [10]) can learn generative models of the world. 1

1.1 Related work

A recent generative model of the hippocampus, the Tolman-Eichenbaum Machine [23], proposed that the hippocampal formation be thought of as a hierarchical network performing latent state inference. Medial entorhinal cortex (MEC) sits atop the hierarchy and learns an abstract representation of space which is mapped to the hippocampus (HPC) where it is bound onto incoming sensory stimuli. Once trained the system can act in a generative fashion by updating the hidden representation with idiothetic action signals and then predicting the upcoming sensory experience. The drawback of this model, and others which share a similar philosophical approach [32, 24], is that it requires training via backpropagation through time (or equivalent end-to-end optimisation schemes, as in [24]) without clear biological correlates. Related hierarchical network architectures have also been studied in the context of reinforcement learning [33] and hippocampal associative memory [34].

Historically, hippocampal models of path integration have focused on continuous attractor networks (CANs) [25, 26, 27, 21] in entorhinal cortex. A bump of activity representing location is pushed around the CAN by speed and/or head-direction selective inputs, thus integrating self-motion. CANs have received substantial experimental support [28] but few studies adequately account for how this structure is learned by the brain in the first place. One exception exists outside the hippocampal literature: Vafidis et al. [35] built a model of path integration in the fly head-direction system which uses local learning rules. Here we go further by embedding our path integrator inside a hierarchical generative model. Doing so additionally relaxes the assumption (made by Vafidis et al. [35] and others [36]) that sensory inputs into the path integrator are predefined and fixed. Instead, by allowing all incoming and outgoing synapses to be learned from random initalisations, we achieve a more generalisable model capable of transferring structure between environments (see section 3.3).

Hippocampal theta oscillations have been linked to predictive sequence learning before [37, 38, 39, 40] where research has focused on the compressive effects of theta sequences and how these interplay with short timescale synaptic plasticity. Instead of compression, here we hypothesize the role of theta is to control the direction information flows through the hierarchical network.

Finally, a recent theoretical work by Bredenberg et al. [41] derived, starting from principles of Bayesian variational inference, a biologically plausible learning algorithm for approximate Bayesian inference of a hierarchical network model built from multi-compartmental neurons and trained with local learning rules using wake-sleep cycles. Here we build a similar network to theirs (i) extending it to a spatial exploration task and mapping the hidden layers onto those in the hippocampal formation, (ii) simplifying the learning rules and relaxing a discrete-time assumption - instead, opting for a temporally continuous formulation more applicable to biological tasks such as navigation and (iii) adapting the hidden layer to allow idiothetic action signals to guide updates (aka. path integration). Their work provides a theoretical foundation for our own, helping to explaining why learning converges on accurate generative models.

2 A biologically plausible generative model trained with rapidly switching wake-sleep cycles and local learning rules

In sections 2 and 3 we give concise, intuitive descriptions of the model and experiments; expanded details can be found in the supplementary material.

2.1 Basic model summary

We consider learning in an environment defined by a latent state, z ( t ) , which updates according to stochastic dynamics initially unknown to the network,

These dynamics depends on the task; first we consider z ( t ) to be a set of mutually independent random variables and later we consider the more realistic task of an agent moving on a 1D track.

The network recieves sensory input which is a function of the latent state into a sensory layer, p ( t ) , and communicates this to a hidden layer (aka 'internal state'), g ( t ) . The network contains both an inference (aka. recognition ) model which infers the hidden state from the sensory input (green arrows, Fig. 1a) and a generative model which updates the hidden state with recurrent synapses and maps this back to the sensory layer (blue arrows). As we will soon identify these processes with Basal and Apical dendritic compartments of pyramidal neurons we label activations sampled from the inference model with the subscript B and those from the generative model with the subscript A . 2 In summary

w g B , w p A , w g B are matrices of randomly initialised and plastic synaptic weights. ¯ p maps the environmental latent into a vector of neural inputs. σ 's denote activation functions applied to the dendritic pre-activations - either the identity ( σ ( x ) = x ) or rectified tanh functions ( σ ( x ) = max(0 , tanh ( x )) ). A small amount of noise is added to the dendritic activations to simulate realistic biological learning.

We believe that the widely adopted convention of modelling neurons as single-compartment perceptrons is limiting. By considering, in a minimal extension, the distributed dendritic structure of real neurons we can tap into significant potential for explaining hippocampal learning. Theoretical [42, 43, 44, 45] and experimental [46, 47, 48] research into credit assignment in biological neurons has identified different roles for basal and apical dendrites: basal dendrites are thought to receive bottom-up drive from sensory inputs whereas apical dendrites receive top-down drive from higher layers in the sensory hierarchy [49]. Following this line of research - and matching an equivalent theoretical model of latent state inference described by [41] - we identify the inference process with synaptic inputs into a basal dendritic compartment of pyramidal neurons and the generative process with synaptic inputs into an apical dendritic compartment. In summary, each p and g neuron in our model has three compartments: a somatic compartment, a basal dendritic compartment and an apical dendritic compartment (Fig. 1b). Only the somatic activation is used for communication between layers (right hand side of Eqns. (2) and (3)) while dendritic compartment activations are variables affecting internal neuronal dynamics and learning as described below (Eqns. (4) and (6)).

2.2 Theta oscillations gate the direction of information flow through the network

The dynamics of the somatic activations p ( t ) and g ( t ) are as follows: the voltage in each soma is either equal to the voltage in the basal compartment or the voltage in the apical compartment depending on the phase of an underlying theta oscillation. This is achieved by a simple theta-gating mechanism (Fig. 1b):

where θ ( t ) is a 5 Hz global theta oscillation variable defined by the square wave function:

for T = 1 /f θ and f θ = 5 Hz, matching the hippocampal theta frequency (5-10 Hz) [50]. According to this model theta-band oscillations in the hippocampal local field potential gate which dendritic compartment drives the soma. Experimental [47, 51, 52] and modelling work [53] gives provisional support for this assumption.

These local theta-dynamics have global consequences: the early phase ( θ ( t ) = 1 ) of each theta cycle can be thought of as a 'wake' phase where information flows upwards through the network from the environment to the hidden layer, sampling the inference model. The latter phase ( θ ( t ) = 0 ) of each theta cycle is a 'sleep' phase where information flows down from the hidden layer to the sensory units, sampling the generative model. These dynamics are displayed in Fig. 1.

2.3 Hebbian-style learning rules train synapses to minimise local prediction errors

In contrast to comparable models which are optimised end-to-end using backpropagation through time our model learns synaptic weights according to a local plasticity rule which is a simplified variant of a rule proposed by Urbanczik and Senn [43]. Incoming synaptic projections are continually adjusted in order to minimize the discrepancy between the somatic activation and the dendritic activation. The full learning rules are described in the supplement but simplified versions are given here:

Notably this learning rule is equivalent for all plastic synapses in the model: p to g , g to p and the recurrent g to g synapses (see Fig. 1b). If a local prediction error is detected, for example the somatic activation is larger than the dendritic activation, then the synaptic strength of inputs into that dendritic compartment which are positive/negative are strengthed/weakened to reduce the error. This model can equivalently be viewed as a type of Hebbian learning - weight change is proportional to the correlation of pre- and post-synaptic activity (the first term) - regularised (by the second term) to prevent unbounded growth.

During the wake phase the weights of the generative model ( w p A and w g A ) are trained and plasticity on the inference weights ( w g B ) falls to zero. This occurs naturally because p = p B so there will be no basal prediction errors to correct. During sleep the reverse occurs; the weights of the inference model are trained and plasticity on the generative model falls to zero. Experimentally, apical activity is known to guide plasticity at basal synapses in CA1 [46]. This alternating, coordinated regime of sampling and learning (sample-inference-train-generative, then sample-generative-train-inference) is a hallmark of the wake-sleep algorithm. It fundamentally differs from the forward and backward sweeps of backpropagation since neurons remain provisionally active at all times so the process of learning minimally perturbs perception. Also, whereas backpropagation sends error signals down through the network to train synaptic weights, here only predictions are sent between layers and error signals are calculated locally at each dendrite.

As discussed in section 1, Bredenberg et al. [41] mathematically derive learning rules similar to these starting from a loss function closely related to the evidence lower bound (ELBO). As such our identification of early- and late-theta phases as 'wake' and 'sleep' cycles can be considered precise: from a Bayesian perspective our hippocampal model is minimising a modified ELBO loss (see supplement) thus learns to find approximately optimal inference and generative models accounting from the temporally varying stimulus stream it is presented.

2.4 Velocity inputs into the hidden layer

For path integration, the hidden state needs access to an idiothetic (internally generated) velocity signal. To satisfy this we endow the hidden layer, g , with conjunctive velocity inputs, henceforth 'conjunctive cells', as shown in Fig. 3a & b. Conjunctive cells are organised into two groups: g v L is responsible for leftward motion and g v R for rightward motion. Each conjunctive cell receives input from the hidden units and either the leftward ( v L = max(0 , -˙ x ) ) or rightward ( v R = max(0 , ˙ x ) ) component of the velocity. For the results shown this connectivity is one-to-one [ w g vL ] ij = [ w g vR ] ij = δ ij but

<latexi sh

1_b

64="I92ETNup

Q

8q

H

yDv3+

m

Fk

>A

B

n

c

V

LSgM

rPU

Z

o

0O

7

X

w

j

z

W

JR

/

f

5

C

G

K

Y

d

<latexi sh

1_b

64="Gk78CLZN

WJ

yD5ApEV

wQ

>

3

c

S

F

r

n

z

TU

uq

K

d

0

+

IO/

H

P

Xfo

M

gR

9

Y2

m

B

j

v

<latexi sh

1_b

64="UfPZjMcWC

p

O

o

zgA

>

3

VDLS

N

FJ

r

7nQ

kT

uqG

EK

d

0

+

I

/

H

X

R

59w

8

Y2

m

B

v

y

<latexi sh

1_b

64="mUSf/

A53Hqd7zK

gYo

OC

Z

>

B8

c

VDL

N

FJ

u

v

Qkp

G

2

T

9

M

E

R

+

0

W

P

j

r

w

y

X

n

I

<latexi sh

1_b

64="+G8

H5d27O

BF

Y

M3

Jf

D

Z

>A

c

V

LS

N

U

u

ovgQkp

RT

K9

zE

/

q

0

W

P

j

r

m

w

y

X

n

I

C

<latexi sh

1_b

64="2r9qD

B

pZ

c

M

JP

gju7A

>

8X

V

LS

N

F

3U

f

d

k

wI

E

m

0

O

G

+

C

v

T

H

5

zn

Q

YW

y

o

R

K

/

<latexi sh

1_b

64="Mwd+

L0/

fN7cA

ZRD39vPmW

>

B8X

V

S

FJ

U

2

5C

g

q

rG

IYy

o

k

uH

z

p

O

Q

jK

T

E

n

〈

p

A

-

p

B

〉

〈

g

A

-

g

B

〉

0

30

10

-1

Prediction error

a

Auto-correlation

Lag / s

4

z

g

Test network in  sleep  mode

d

Time, t  / min

1

0

Stimulus / Hz

p

A

p

B

c

Training / min

Inference model

Generative model

b

Time, t / min

0

1

z

i

(

t

)

Tested during wake

Tested during sleep

10

-0.5

10

-1.5

random connectivity works too, see supplement. Finally, conjunctive cells send return connections back to the apical dendritic compartment of the hidden units via a randomly initialised plastic synaptic weight matrix. This inputs are what drive the hidden units to path integrate.

This model takes inspiration from so-called conjunctive grid cells [54] found in the medial entorhinal cortex (MEC). These cells, though to be an integral component of the mammilian path integration system[27], are jointly tuned to head direction and location much like the conjunctive cells in our model. An important and novel aspect of our model is that synaptic weights between or into the hidden units are learned . This deviates from other models for example that by Burak and Fiete [27] (where all connectivity is predefined and fixed) or Vafidis et al. [35] and Widloski and Fiete [36] (where sensory inputs to the hidden units are pre-defined and fixed). This is not only more realistic but affords the model flexibility to translate path integration abilities between environments without having to relearn them, a form of transfer learning which we demonstrate in section 3.3.

3 Results

3.1 Validation on an artifical latent learning task

We begin by testing the basic model (i.e. without conjunctive inputs, Fig. 1a) on an artificial task. N z = 5 latents, z i ( t ) , are independently sampled from a smooth, random process with an autocorrelation timescale of 1 second (Fig. 2a). The sensory layer, N p = 50 , then receives a high-dimensional random linear mixture of the latents into the basal compartments:

where A ∈ R 50 × 5 and [ A ] ij ∼ N (0 , 1 √ N z ) . The hidden layer, g ( t ) , is matched in size to the latent process, N g = N z = 5 , and all dendritic activation functions are linear. We train the model for 30 minutes of simulated time and track prediction errors, the difference between the basal and apical activations in the sensory and hidden layers, which reliably decreased throughout training (Fig. 2b). We then perform two tests designed to confirm whether the model has learnt accurate inference and generative models.

<latexi sh

1_b

64="DMNqOkU

E0vJ

9c3

WVf

HCo

>A

B+X

LSg

F

2

Zdu

j

P

RQ

7

Y

mT

G

n

w

/

I

5

y

z

p

8

r

K

<latexi sh

1_b

64="A

SF

kM

u

D

pTVQ

>

B8

c

L

N

J

Y

7

+

K

Ir

G

2

CmU

n

d

0

9

Ev

O

5

Z

R

o

g

Pz

/

f

<latexi sh

1_b

64="

wH

CVDKE8

3

zL

G

W

P

/uA

>

B

X

c

SgM

F

U

f

Zd

k

I

Omj

n

2

y

YN

J

oR

q

+

T

7

0v

5

r

p

9

Q

<latexi sh

1_b

64="N

d

HK

S8JuO

C

j

k2TWXw

>A

B

c

VDL

gM

F

3U

f

Z

I

mr

y

n

0o

z

Q

PG

+

v

E

/R

7

p

5

9

q

Y

<latexi sh

1_b

64="w5q/

YWF

n

oyzP

CX

M3

v

I

>A

B8

c

VDLSg

U

2

+

R

Z

p

r

0

E

k

m

J

O

Q

K

9

7

u

f

N

G

T

H

j

d

<latexi sh

1_b

64="UfPZjMcWC

p

O

o

zgA

>

3

VDLS

N

FJ

r

7nQ

kT

uqG

EK

d

0

+

I

/

H

X

R

59w

8

Y2

m

B

v

y

<latexi sh

1_b

64="Gk78CLZN

WJ

yD5ApEV

wQ

>

3

c

S

F

r

n

z

TU

uq

K

d

0

+

IO/

H

P

Xfo

M

gR

9

Y2

m

B

j

v

<latexi sh

1_b

64="8+qO

/

5

cMpU

G

oF7

B

LmE

>A

X

VD

3

2v

Zdu j0

y

n

W

w

TK

R

N

Q

J

z

fk

HI

C

r

Y

P

9

<latexi sh

1_b

64="

GB

J

vO

MC

P

NI

A

>

83

c

VDLS

F

YH7W+qu

KT

rk

2

9g

p

Q

Z

E

o

w

d

RX

y/

z

0

5

f

U

<latexi sh

1_b

64="N

d

HK

S8JuO

C

j

k2TWXw

>A

B

c

VDL

gM

F

3U

f

Z

I

mr

y

n

0o

z

Q

PG

+

v

E

/R

7

p

5

9

q

Y

<latexi sh

1_b

64="9

S2unTGYr

XW

Q

L3

Bf vPjg

>A

/H

c

VD

N

FJ

U

y5CR

K

0

o

M

+

OZ

E

7

w

d

k

I

8p

z

q

m

<latexi sh

1_b

64="

CdAZ

8R

QSJo pTD23v

H

rE

>

c

V

L

gM

F

U

q

N

u

y

+

j

w

W

m

Y

K

5

O

0

kX

9z

P

7

f

G

n

I

<latexi sh

1_b

64="

GB

J

vO

MC

P

NI

A

>

83

c

VDLS

F

YH7W+qu

KT

rk

2

9g

p

Q

Z

E

o

w

d

RX

y/

z

0

5

f

U

<latexi sh

1_b

64="TX

kCJ+UZP

Dv

8pH

3/

w

>A

B

c

V

LS

N

F

f

dug

I

E

2r

7o

m

0

M

O

G

j

R

n

KW

9

5

Qy

Y

<latexi sh

1_b

64="XO

jwTW2Z

noqg

J7rp

LF

cu

>A

VD

S

N

3

d

Ev

U

K

M

f

I

yYmR

0

8

z

5

C

G

Q

/

H

k

9

P

<latexi sh

1_b

64="

5LO0+qBGDT/Ky

R

7A

dwI

>

83

c

V

S

N

FJ

U

f

Z

ugk

Ep2r

X

m

M

Cj

z

n

o

v

H

W

9

P

Q

Y

<latexi sh

1_b

64="2r9qD

B

pZ

c

M

JP

gju7A

>

8X

V

LS

N

F

3U

f

d

k

wI

E

m

0

O

G

+

C

v

T

H

5

zn

Q

YW

y

o

R

K

/

<latexi sh

1_b

64="g

+noI9/2vyBZQU3LM

O

qRk

w

>A

c

VD

S

F

rP

5C

W

7

uX

jJ

0

m

N

z

H

KE

p

T

d

f

G

<latexi sh

1_b

64="AoL+

K/

wFyYv

V

d

rW

8

>

B

X

c

D

SgM

3

PU

5CR

Z

pOm

T

u

7

k0

z

I

n

j

f

J

GN

H

9

Q

2

E

q

<latexi sh

1_b

64="3CIQO9

JF

koB

dPT

uj

Vzqyf0

>A

+X

c

DLSgM

r

U

5

R

Z

W

7

w

2

mn

p

N

H

K

v

E

G

Y

/

8

<latexi sh

1_b

64="2r9qD

B

pZ

c

M

JP

gju7A

>

8X

V

LS

N

F

3U

f

d

k

wI

E

m

0

O

G

+

C

v

T

H

5

zn

Q

YW

y

o

R

K

/

<latexi sh

1_b

64="3CIQO9

JF

koB

dPT

uj

Vzqyf0

>A

+X

c

DLSgM

r

U

5

R

Z

W

7

w

2

mn

p

N

H

K

v

E

G

Y

/

8

<latexi sh

1_b

64="uq

f

0+

cB

kLym

Ujg

>A

X

VD

S

FJ3

W

o

d

Ev

K

2

M

wr

HTp5

/

C

Q9

R

n

YO

z

P

8

G

Z

<latexi sh

1_b

64="3CIQO9

JF

koB

dPT

uj

Vzqyf0

>A

+X

c

DLSgM

r

U

5

R

Z

W

7

w

2

mn

p

N

H

K

v

E

G

Y

/

8

<latexi sh

1_b

64="XO

jwTW2Z

noqg

J7rp

LF

cu

>A

VD

S

N

3

d

Ev

U

K

M

f

I

yYmR

0

8

z

5

C

G

Q

/

H

k

9

P

Position [x,y]

T

g

z

g

g

A

g

B

g

vL

g

vR

v

R

g

vR

w

w

gA

w

gA

from

g

w

gA

from

g

vL

w

gA

from

g

vR

a

b

c

d

Prediction error

Training / min

0

30

0.1

〈

p

A

-

p

B

〉

〈

g

A

-

g

B

〉

e

0

1

Time / s

30

0

0

40

Error / cm

Error

〉

± SEM

Position on track / m

p

=

δ

ij

10

20

sensory lesion

w

gvR

MEC

HPC

p

MEC

conjunctive

v

R

v

L

velocity

velocity

00000000

First, we set the dynamics of the model to 'wake' mode ( θ = 1 ) and measure the basal and apical activations of one of the sensory neurons for 60 seconds. Close correspondence (Fig. 2c) confirms that the network accurately 'autoencodes' the high-dimensional sensory inputs through the compressed hidden layer. Since all activation functions are linear this implies that w g B and w p A are pseudoinverses. Next, we place the network in 'sleep' mode ( θ = 0 ) and allow the generative model to run freely. The autocorrelation of the generated hidden states ( g ( t | θ = 0) , displayed fully in the supplement) match that of the true environmental latents ( z ( t ) ), Fig. 2d, implying the generative model has statistics closely matching those of the true underlying generative process.

3.2 Learnable path integration with a hidden ring attractor

Next we turn our attention to the hippocampal formation's role in spatial navigation, and our central result. The environment consists of an agent randomly moving around a 1 m 1D circular track (motion and cell data is generated using the RatInABox package [55]). The basal compartment of each HPC neuron is spatially tuned to a single different Gaussian input however non-Gaussian

randomly spatially tuned inputs work as well (see supplement Fig. S2b):

x ( t ) is the position of the agent and { x i } N p i =1 are the centres of the Gaussian inputs ( σ = 6 cm), intended to simulate hippocampal place fields, evenly spaced at 1 cm intervals along the track. MEC (i.e. the hidden layer, g ( t ) ) is matched in size N g = N p = 100 with rectified tanh activation functions on both dendritic compartments ( σ g B ( x ) = σ g A ( x ) = max(0 , tanh( x )) ) and HPC (the sensory layer p ( t ) ) is linear ( σ p A ( x ) = x ). Two populations of conjunctive cells (Fig. 3a & b) feed into the apical compartments of the MEC recurrent units. Random initialisation of w g B means that MEC neurons start off with random non-Gaussian spatial tunings. w g A and w p A are also randomly initialised.

The network is trained for 30 minutes with learning plateauing after 15 (Fig. 3d). A lesion study, designed to test path integration, is then performed as follows: First, the network is run for 10 seconds normally (i.e. with theta-oscillating periods of wake and sleep). Since the simulated HPC neurons receive place-tuned inputs uniformly ordered along the track (i.e. x j > x i ∀ i, j > i ) an activity heatmap of HPC reveals a bump of activity accurately tracking agent's position (Fig. 3e, left). The network is then placed into a sleep phase ( θ = 0 ) for 20 seconds. This amounts to a full sensory lesion since top-down MEC inputs, not bottom-up place-tuned sensory inputs, drive HPC. Despite the full sensory lesion, hippocampal activity remains approximately unperturbed and the activity bump continues to accurately track position, slowly accumulating errors (Fig. 3e right). Since our HPC layer has no recurrent connectivity it cannot support this post-lesion activity on its own. Instead feed-forward drive from an MEC ring attractor, which we turn our attention to now, is responsible for maintaining the HPC code.

To find the ring attractor we must first reorder the MEC cells. We do this according to the position of the peak of their receptive fields (defined in the supplement). After reordering, the recurrent connectivity matrix can be seen to have acquired a centre-surround connectivity profile. Nearby MEC cells were, on average, strongly and positively recurrently connected to one another. Those far apart weakly inhibit one another (Fig. 3c, left; band of strong positive weights along diagonal flanked by weak negative weights). This profile matches that of a quasi-continuous ring attractor: local excitatory and long-range inhibitory connections stabilise a bump of activity on the attractor manifold in the absence of sensory input [56]. Weights from the conjunctive cells acquired asymmetric connectivity (Fig. 3c, middle & right) skewed towards the velocity direction for which they are selective. These asymmetric connections enable conjunctive cells to 'push' the activity bump around the manifold, integrating velocity (see supplement for a visualisation of the MEC bump attractor). Theoretical work on ring attractors has demonstrated that for accurate path integration the asymmetric weights must be proportional to the derivative of the symmetric weights [56], approximately observed here. A noteworthy observation is that some MEC neurons become perpetually inactive; this is a consequence of the fact that both top-down and bottom-up synapses into the hidden layer are plastic and can fall to zero (Fig. 3c bands of zero-weights) satisfying a trivial g A = g B = 0 solution for minimising the prediction error. Despite this, not all MEC neurons die and the surviving subset are sufficient for path integration. In supplementary section 5.4.2 we discuss additional results showing when the network learns robust path integrate under a variety of plasticity, initialisation and noise manipulations.

Crucially, what sets this model apart from others [19, 20, 21, 22] is that the network is not optimized using a conventional path-integration objective and backpropagation. Instead, it has been demonstrated how path integration can naturally arise in a biologically constrained network subject to a much simpler (yet more broadly applicable) local objective, in cases where idiothetic velocity signals are available to the hidden layers.

3.3 Remapping: transfer of structural knowledge between environments

Finally, we demonstrate how our trained network can transfer structural knowledge - which here means the ring attractor and thereby path integration - between environments. We start by training the network as in section 3.2; the only diffence is that for simplicity we choose to fix w g B = δ ij giving rise to MEC representations which, like HPC, are unimodal (this constraint can be relaxed and, in the more general case, MEC units typically have multiple receptive fields, Fig S4d, reminiscent of grid cells). We then simulate a hippocampal 'remapping' event by shuffling the sensory inputs to the HPC layer (Fig. 4a & b, top panel) and retraining the network for a further 30 minutes but this time

<latexi sh

1_b

64="DMNqOkU

E0vJ

9c3

WVf

HCo

>A

B+X

LSg

F

2

Zdu

j

P

RQ

7

Y

mT

G

n

w

/

I

5

y

z

p

8

r

K

<latexi sh

1_b

64="

wH

CVDKE8

3

zL

G

W

P

/uA

>

B

X

c

SgM

F

U

f

Zd

k

I

Omj

n

2

y

YN

J

oR

q

+

T

7

0v

5

r

p

9

Q

<latexi sh

1_b

64="N

d

HK

S8JuO

C

j

k2TWXw

>A

B

c

VDL

gM

F

3U

f

Z

I

mr

y

n

0o

z

Q

PG

+

v

E

/R

7

p

5

9

q

Y

<latexi sh

1_b

64="w5q/

YWF

n

oyzP

CX

M3

v

I

>A

B8

c

VDLSg

U

2

+

R

Z

p

r

0

E

k

m

J

O

Q

K

9

7

u

f

N

G

T

H

j

d

<latexi sh

1_b

64="8+qO

/

5

cMpU

G

oF7

B

LmE

>A

X

VD

3

2v

Zdu j0

y

n

W

w

TK

R

N

Q

J

z

fk

HI

C

r

Y

P

9

<latexi sh

1_b

64="AoL+

K/

wFyYv

V

d

rW

8

>

B

X

c

D

SgM

3

PU

5CR

Z

pOm

T

u

7

k0

z

I

n

j

f

J

GN

H

9

Q

2

E

q

<latexi sh

1_b

64="

yYT3dDc+LvJKk

7Q

2GXZ

o0

>A

B

V

S

N

F

rPU

I

gq

u

M

f

On

m

R

H

C

5

E

j

w

9

8

p

z

W

/

<latexi sh

1_b

64="O2

K95y wqSp

cLj

BC

WP

G

>A

+X

VD

gM

F

3

r

U

R

Z

T

u

H

I

Jk

N

0

8

m

f

v

E

z

7

d

n

o

Y

Q

/

<latexi sh

1_b

64="PHNd jpnFq9r

wm

Cg8

>A

c

VDLS

M

3U

2

T

I

k

RZ

O

Qv

W0

Y

J

G

X

/

f

u

7

y5

K

E

o

z

<latexi sh

1_b

64="CR5

E

K

Ju

Bn3

>A

+H

c

VDL

rPX

U

Z

N0

Q

dp

WY

v

8G

T

/wk

7

I

f

z

9

o

y

Spatial receptive fields…

1

100

0

1

100

i

i

b

0

1

0

1

z

a

g

Remapping experiment

retrain

&

2

remap

1

c

p

i

(

x

)

g

i

(

x

)

before

shuffle

immediately

after shuffle

after

retrain

Error

〉

0

10

20

chance decoder

Position, x / m

MEC

HPC

p

g

vR

g

vL

w

w

pA

w

pB

1

2

before remap

just after remap

after retrain

holding weights in the hidden layer, w g A . Only the HPC ↔ MEC synapses ( w g B & w p A ) remain plastic during retraining. Biologically this may be accounted for by the observation that cortical plasticity is substantially slower than hippocampal plasticity [57].

During biological remapping events place cells remap independently whereas grid cells remap en masse with entire modules shifting by the same constant phase [58]. This observation is reproduced in our model: after retraining MEC units regroup with receptive fields as they were before remapping but with a constant phase shift along the track. This re-emergence of structure occurs because the ring attractor seeds a bump of activity on the attractor manifold (during the 'sleep' phases of retraining) onto which the shuffled HPC inputs then bind. Since nothing constrains where on the circularly symmetric attractor manifold this regrouping can initiate, only relative correlations, modulo a phase shift, are preserved.

Decoding error one second after a sensory lesion is tested just before remapping, just after remapping and after retraining (Fig. 4c). After the remapping path integration abilities temporarily disappear because the MEC ring attractor is still tuned to the old and invalid HPC receptive fields. After relearning - and despite no adjustments to the MEC weights, w g A , where the ring attractor is stored -path integration recovers to almost the level before remapping. This differs substantially from other local models of path integration learning [35, 36] which don't consider plasticity on the ring attractor inputs. In these models, adaptation to a new environment necessarily requires complete relearning of the ring attractor. Instead our model exploits the basic fact that movement (path integration) in one environment is fundamentally the same as in another, one must simply learn a new mapping to/from the ring attractor, 'translating' it to fit the new sensory stimuli.

4 Discussion

We propose that the hippocampal formation resembles a Helmholtz machine, simultaneously learning an inference and generative model of sensory stimuli. Like previous models [23] medial entorhinal

cortex (MEC) sits hierarchically above the hippocampus (HPC) to which it sends generative predictions. Our model differs in the learning rules and neural dynamics: local prediction errors are minimised between distinct dendritic compartments receiving bottom-up and top-down signals. Theta oscillations regulate internal neural dynamics, switching the network between wake and sleep phases. In a navigation task our MEC model forms a ring attractor capable of path integration. Despite simple learning rules and dynamics our model retains key cognitive capabilities of the hippocampal formation including the ability to transfer knowledge across different sensory environments.

Local learning rules are commonly recognised as essential in biologically plausible learning algorithms [43]. However, the importance of learning scheduling - how neural systems coordinate or multiplex distinct phases of forward and backward information flow - is often overlooked[59]. Neural oscillations such as theta, hypothesized to temporally coordinate communication between neuronal populations [60], likely play an underexplored role in this regard (neural 'bursting' has also been pointed out as a potential solution to multiplexing [61]). One advantage of the wake-sleep algorithm, which this study suggests neural oscillations can support, compared to forward and backward sweeps is that, during convergence, the two phases become highly similar, allowing learning to proceed without affecting perception.

While our discussion has primarily focused on theta oscillations as a mechanism for learning, they have also been proposed as a mechanism for short-range future prediction via so-called 'mindtravel'[62]. During the latter phase of each theta cycle (i.e. the sleep phase) gain amplified velocity signals might rapidly drive the MEC activity bump along the manifold allowing the agent to assess nearby upcoming locations. This complimentary proposition could neatly integrate into the framework proposed here and emphasizes the need for further investigation into the multifaceted functions of neural rhythms within the hippocampal/entorhinal system.

Beyond theta oscillations, both faster gamma cycles [63] and the slower physiological states of sleep and wake [64] have been associated with learning. Based on our model we suggest a tentative hypothesis that theta oscillations may be favored due to an optimality criterion; whilst faster oscillations could be a mechanism to prevent extreme drift during sleep that might disrupt learning their frequency might by upper bounded biophysically by the neural time constants associated with the biophysical processes supporting dendritic gating the soma. These ideas, their relevance to other brain regions involved in generative learning, 2D spatial dynamics, and offline memory consolidation/replay remain exciting questions for future theoretical and experimental investigation.

References

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/sohl-dickstein15.html .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762 .

Tom M George and Pietro Lio. Unsupervised machine learning for data encoding applied to ovarian cancer transcriptomes. November 2019. doi: 10.1101/855593. URL https: //doi.org/10.1101/855593 .

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. URL https://arxiv.org/abs/2102.12092 .

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.

Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2):127-138, January 2010. doi: 10.1038/nrn2787. URL https://doi.org/10.1038/ nrn2787 .

Samuel J. Gershman. The generative adversarial brain. Frontiers in Artificial Intelligence, 2, September 2019. doi: 10.3389/frai.2019.00018. URL https://doi.org/10.3389/frai. 2019.00018 .

Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January 1999. doi: 10.1038/4580. URL https://doi.org/10.1038/4580 .

John OKeefe. Place units in the hippocampus of the freely moving rat. Experimental Neurology, 51(1):78-109, January 1976. doi: 10.1016/0014-4886(76)90055-8. URL https://doi.org/ 10.1016/0014-4886(76)90055-8 .

Larry R. Squire. "memory and the hippocampus: A synthesis from findings with rats, monkeys, and humans": Correction. Psychological Review, 99(3):582-582, July 1992. doi: 10.1037/ 0033-295x.99.3.582. URL https://doi.org/10.1037/0033-295x.99.3.582 .

Honi Sanders, Matthew A Wilson, and Samuel J Gershman. Hippocampal remapping as hidden state inference. eLife, 9, June 2020. doi: 10.7554/elife.51140. URL https://doi.org/10. 7554/elife.51140 .

Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I. Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436(7052):801-806, June 2005. doi: 10.1038/nature03721. URL https://doi.org/10.1038/nature03721 .

Edvard I Moser, May-Britt Moser, and Bruce L McNaughton. Spatial representation in the hippocampal formation: a history. Nature Neuroscience, 20(11):1448-1464, November 2017. doi: 10.1038/nn.4653. URL https://doi.org/10.1038/nn.4653 .

Hugo J. Spiers and Eleanor A. Maguire. Thoughts, behaviour, and brain dynamics during navigation in the real world. NeuroImage, 31(4):1826-1840, July 2006. doi: 10.1016/j.neuroimage. 2006.01.037. URL https://doi.org/10.1016/j.neuroimage.2006.01.037 .

Margaret F Carr, Shantanu P Jadhav, and Loren M Frank. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nature Neuroscience, 14(2): 147-153, January 2011. doi: 10.1038/nn.2732. URL https://doi.org/10.1038/nn.2732 .

B. L. McNaughton, C. A. Barnes, J. L. Gerrard, K. Gothard, M. W. Jung, J. J. Knierim, H. Kudrimoti, Y. Qin, W. E. Skaggs, M. Suster, and K. L. Weaver. Deciphering the hippocampal polyglot: the hippocampus as a path integration system. Journal of Experimental Biology, 199 (1):173-185, January 1996. doi: 10.1242/jeb.199.1.173. URL https://doi.org/10.1242/ jeb.199.1.173 .

Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization, 2018.

Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):429-433, May 2018. doi: 10.1038/s41586-018-0102-6. URL https://doi.org/10.1038/s41586-018-0102-6 .

Ben Sorscher, Gabriel C. Mel, Samuel A. Ocko, Lisa M. Giocomo, and Surya Ganguli. A unified theory for the computational and mechanistic origins of grid cells. Neuron, 111(1): 121-137.e13, 2023. ISSN 0896-6273. doi: https://doi.org/10.1016/j.neuron.2022.10.003. URL https://www.sciencedirect.com/science/article/pii/S0896627322009072 .

William Dorrell, Peter E. Latham, Timothy E. J. Behrens, and James C. R. Whittington. Actionable neural representations: Grid cells from minimal constraints, 2023.

James C.R. Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy E.J. Behrens. The tolman-eichenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation. Cell, 183(5):12491263.e23, November 2020. doi: 10.1016/j.cell.2020.10.024. URL https://doi.org/10. 1016/j.cell.2020.10.024 .

Dileep George, Rajeev V. Rikhye, Nishad Gothoskar, J. Swaroop Guntupalli, Antoine Dedieu, and Miguel Lázaro-Gredilla. Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps. Nature Communications, 12(1), April 2021. doi: 10.1038/s41467-021-22559-5. URL https://doi.org/10.1038/s41467-021-22559-5 .

W. E. Skaggs, J. J. Knierim, H. S. Kudrimoti, and B. L. McNaughton. A model of the neural basis of the rat's sense of direction. Advances in neural information processing systems, 7: 173-180, 1995. ISSN 1049-5258.

Alexei Samsonovich and Bruce L. McNaughton. Path integration and cognitive mapping in a continuous attractor neural network model. Journal of Neuroscience, 17(15):5900-5920, 1997. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.17-15-05900.1997. URL https://www. jneurosci.org/content/17/15/5900 .

Yoram Burak and Ila R. Fiete. Accurate path integration in continuous attractor network models of grid cells. PLoS Computational Biology, 5(2):e1000291, February 2009. doi: 10.1371/ journal.pcbi.1000291. URL https://doi.org/10.1371/journal.pcbi.1000291 .

Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. arXiv, 2021. doi: 10.48550/arxiv.2112.03978.

Peter Dayan, Geoffrey E. Hinton, Radford M. Neal, and Richard S. Zemel. The helmholtz machine. Neural Computation, 7(5):889-904, September 1995. doi: 10.1162/neco.1995.7.5.889. URL https://doi.org/10.1162/neco.1995.7.5.889 .

Gyorgy Buzsaki. Theta oscillations in the hippocampus. Neuron, 33(3):325-340, January 2002. doi: 10.1016/s0896-6273(02)00586-x. URL https://doi.org/10.1016/s0896-6273(02) 00586-x .

Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214):1158-1161, May 1995. doi: 10.1126/science.7761831. URL https://doi.org/10.1126/science.7761831 .

Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zambaldi, Dharshan Kumaran, Demis Hassabis, Caswell Barry, and Charles Blundell. A model of egocentric to allocentric understanding in mammalian brains. November 2020. doi: 10.1101/2020.11.11.378141. URL https://doi.org/10.1101/2020.11.11.378141 .

Dongqi Han, Kenji Doya, and Jun Tani. Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks. Neural Networks, 129:149-162, September 2020. doi: 10.1016/j.neunet.2020.06.002. URL https://doi.org/10.1016/j. neunet.2020.06.002 .

Sugandha Sharma, Sarthak Chandra, and Ila Fiete. Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19658-19682. PMLR, 17-23 Jul 2022. URL https: //proceedings.mlr.press/v162/sharma22b.html .

Pantelis Vafidis, David Owald, Tiziano D'Albis, and Richard Kempter. Learning accurate path integration in ring attractor models of the head direction system. eLife, 11:e69841, jun 2022. ISSN 2050-084X. doi: 10.7554/eLife.69841. URL https://doi.org/10.7554/ eLife.69841 .

John Widloski and Ila R. Fiete. A Model of Grid Cell Development through Spatial Exploration and Spike Time-Dependent Plasticity. Neuron, 83(2):481-495, 2014. ISSN 0896-6273. doi: 10.1016/j.neuron.2014.06.018.

William E. Skaggs, Bruce L. McNaughton, Matthew A. Wilson, and Carol A. Barnes. Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences. Hippocampus, 6(2):149-172, 1996. doi: 10.1002/(sici)1098-1063(1996)6:2<149:: aid-hipo6>3.0.co;2-k. URL https://doi.org/10.1002/(sici)1098-1063(1996)6: 2<149::aid-hipo6>3.0.co;2-k .

M.R. Mehta, M.C. Quirk, and M.A. Wilson. Experience-Dependent Asymmetric Shape of Hippocampal Receptive Fields. Neuron, 25:707-715, 2000.

Tom M George, William de Cothi, Kimberly L Stachenfeld, and Caswell Barry. Rapid learning of predictive maps with STDP and theta phase precession. eLife, 12, March 2023. doi: 10.7554/elife.80663. URL https://doi.org/10.7554/elife.80663 .

Tom M George. Theta sequences as eligibility traces: A biological solution to credit assignment. In International Conference on Learning Representations 2023 (TinyPapers track), 2023. doi: https://doi.org/10.48550/arXiv.2305.08124. URL https://openreview.net/forum?id= vd16AYbem3Z .

Colin Bredenberg, Eero P Simoncelli, Benjamin S H Lyo, and Cristina Savin. Impression learning: Online representation learning with synaptic plasticity. page 13, 2021.

Konrad P. Kording and Peter Konig. Supervised and unsupervised learning with two sites of synaptic integration. Journal of Computational Neuroscience, 11(3):207-215, 2001. doi: 10.1023/a:1013776130161. URL https://doi.org/10.1023/a:1013776130161 .

Robert Urbanczik and Walter Senn. Learning by the Dendritic Prediction of Somatic Spiking. Neuron, 81(3):521-528, 2014. ISSN 08966273. doi: 10.1016/j.neuron.2013.11.030. URL https://linkinghub.elsevier.com/retrieve/pii/S0896627313011276 .

Joao Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pages 8721-8732, 2018.

Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current Opinion in Neurobiology, 54:28-36, February 2019. doi: 10.1016/j.conb.2018.08.003. URL https://doi.org/10.1016/j.conb.2018.08.003 .

Katie C Bittner, Christine Grienberger, Sachin P Vaidya, Aaron D Milstein, John J Macklin, Junghyup Suh, Susumu Tonegawa, and Jeffrey C Magee. Conjunctive input processing drives feature selectivity in hippocampal CA1 neurons. Nature Neuroscience, 18(8):1133-1142, July 2015. doi: 10.1038/nn.4062. URL https://doi.org/10.1038/nn.4062 .

Jurij Brankack, Mark Stewart, and Steven E. Fox. Current source density analysis of the hippocampal theta rhythm: associated sustained potentials and candidate synaptic generators. Brain Research, 615(2):310-327, July 1993. doi: 10.1016/0006-8993(93)90043-m. URL https://doi.org/10.1016/0006-8993(93)90043-m .

Kenji Mizuseki, Anton Sirota, Eva Pastalkova, and György Buzsáki. Theta oscillations provide temporal windows for local circuit computation in the entorhinal-hippocampal loop. Neuron, 64(2):267-280, October 2009. doi: 10.1016/j.neuron.2009.08.037. URL https://doi.org/ 10.1016/j.neuron.2009.08.037 .

Matthew E. Larkum. Are dendrites conceptually useful? Neuroscience, 489:4-14, May 2022. doi: 10.1016/j.neuroscience.2022.03.008. URL https://doi.org/10.1016/j. neuroscience.2022.03.008 .

David J. Foster and Matthew A. Wilson. Hippocampal theta sequences. Hippocampus, 17 (11):1093-1099, 2007. doi: 10.1002/hipo.20345. URL https://doi.org/10.1002/hipo. 20345 .

Christian Holscher, Roger Anwyl, and Michael J. Rowan. Stimulation on the positive phase of hippocampal theta rhythm induces long-term potentiation that can be depotentiated by stimulation on the negative phase in area ca1. The Journal of Neuroscience, 17(16):6470-6477, August 1997. doi: 10.1523/jneurosci.17-16-06470.1997. URL https://doi.org/10.1523/ jneurosci.17-16-06470.1997 .

Yoko Yamaguchi, Yoshito Aota, Bruce L. McNaughton, and Peter Lipa. Bimodality of theta phase precession in hippocampal place cells in freely running rats. Journal of Neurophysiology, 87(6):2629-2642, June 2002. doi: 10.1152/jn.2002.87.6.2629. URL https://doi.org/10. 1152/jn.2002.87.6.2629 .

Michael E. Hasselmo, Clara Bodelón, and Bradley P. Wyble. A proposed function for hippocampal theta rhythm: Separate phases of encoding and retrieval enhance reversal of prior learning. Neural Computation, 14(4):793-817, April 2002. doi: 10.1162/089976602317318965. URL https://doi.org/10.1162/089976602317318965 .

Francesca Sargolini, Marianne Fyhn, Torkel Hafting, Bruce L. McNaughton, Menno P. Witter, May-Britt Moser, and Edvard I. Moser. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science, 312(5774):758-762, May 2006. doi: 10.1126/science. 1125572. URL https://doi.org/10.1126/science.1125572 .

Tom M George, William de Cothi, Claudia Clopath, Kimberly Stachenfeld, and Caswell Barry. RatInABox: A toolkit for modelling locomotion and neuronal activity in continuous environments. aug 2022. doi: 10.1101/2022.08.10.503541. URL https://doi.org/10. 1101%2F2022.08.10.503541 .

K Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. The Journal of Neuroscience, 16(6):2112-2126, March 1996. doi: 10.1523/jneurosci.16-06-02112.1996. URL https://doi.org/10.1523/jneurosci. 16-06-02112.1996 .

Ceren Ergorul and Howard Eichenbaum. Essential role of the hippocampal formation in rapid learning of higher-order sequential associations. The Journal of Neuroscience, 26(15):41114117, April 2006. doi: 10.1523/jneurosci.0441-06.2006. URL https://doi.org/10.1523/ jneurosci.0441-06.2006 .

Marianne Fyhn, Torkel Hafting, Alessandro Treves, May-Britt Moser, and Edvard I. Moser. Hippocampal remapping and grid realignment in entorhinal cortex. Nature, 446(7132):190-194, February 2007. doi: 10.1038/nature05601. URL https://doi.org/10.1038/nature05601 .

Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. eLife, 6, December 2017. doi: 10.7554/elife.22901. URL https: //doi.org/10.7554/elife.22901 .

Pascal Fries. Rhythms for cognition: Communication through coherence. Neuron, 88(1): 220-235, October 2015. doi: 10.1016/j.neuron.2015.09.034. URL https://doi.org/10. 1016/j.neuron.2015.09.034 .

Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature Neuroscience, 24(7):1010-1019, May 2021. doi: 10.1038/s41593-021-00857-x. URL https://doi.org/10.1038/s41593-021-00857-x .

Honi Sanders, César Rennó-Costa, Marco Idiart, and John Lisman. Grid cells and place cells: An integrated view of their navigational and memory function. Trends in Neurosciences, 38

(12):763-775, December 2015. doi: 10.1016/j.tins.2015.10.004. URL https://doi.org/10. 1016/j.tins.2015.10.004 .

Kwan Tung Li, Junhao Liang, and Changsong Zhou. Gamma oscillations facilitate effective learning in excitatory-inhibitory balanced neural circuits. Neural Plasticity, 2021:1-18, January 2021. doi: 10.1155/2021/6668175. URL https://doi.org/10.1155/2021/6668175 .

William E. Skaggs and Bruce L. McNaughton. Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science, 271(5257):1870-1873, March 1996. doi: 10.1126/science.271.5257.1870. URL https://doi.org/10.1126/science. 271.5257.1870 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is an application of the learning scheme derived in Bredenberg et al. 2021 to representation learning and path integration in the MEC and HPC. By adapting the aforementioned scheme to continuous time, and by relating the proposed oscillatory 'gating' signal to theta oscillations that have been observed to gate dynamics in HPC and MEC, the authors propose to model representation learning in these two brain areas as a form of rapidly alternating 'wake-sleep' learning where brief periods of generative prediction dynamics alternate with more standard position-driven dynamics and deviations between path-integrated predictions and sensory stimulus-driven inputs are used to drive learning at both apical and basal synapses. Subsequently they demonstrate that their MEC model is able to capture two interesting features, learned exclusively through local synaptic plasticity rules:\n\n1. The MEC is able to learn a ring attractor capable of path integration based only on motion cues that is able to accurately predict subsequent dynamics.\n2. This attractor can easily generalize to a new environment without modifications to its recurrent synaptic weights.\n\nSTRENGTHS:\nWhile previous models have successfully captured the features I previously mentioned through various optimization schemes, they have not shown how such learning could occur through local, biophysically-motivated synaptic plasticity which can be cleanly related to any representation learning objective function (in this case, the authors relate their learning to a variant of the ELBO objective). This is the principle success of the paper, and in my mind could prove to be a very important model for learning orchestrated between the MEC and HPC.\n\nThe authors also relax several constraints on the learning derived from Bredenberg et al. 2021:\n1. They generalize learning dynamics to continuous time.\n2. They include self-motion information in the 'generative' part of learning dynamics, which allows for path integration.\n3. They relax constraints on gating of plasticity that are tied to theta cycles, simplifying learning and requiring fewer assumed variables that are locally available to synapses.\n\nFurthermore, the paper is clearly written and logically presented.\n\nWEAKNESSES:\nTo me, the principal weaknesses of the paper come from additional analyses that the authors could have done, but did not do. I will list several below:\n\n1. The authors do not test their learning algorithm in more complex environments. Simple two-dimensional environments such as boxes or circles could be used to test whether or not the learning algorithm is sufficient to develop grid-like cells in MEC. More complex image-based environments could be used to test the scalability of the proposed learning algorithm, and could potentially test whether or not the proposed learning algorithm is also sufficient to develop place cell responses (rather than having them hand-provided for the system). Given that algorithms similar to Wake-Sleep do not scale as well to high-dimensional problems as backpropagation and that the brain is presumably able to learn place cell information indirectly from multimodal sensory input (vision, audio, vestibular inputs, etc.), this is a valid concern.\n\n2. The authors do not discuss testable predictions for their learning algorithm--I can identify several, which I would like to see discussed in more detail. In particular, though the authors suggest that after learning has converged the 'generative' and 'inference' phases of learning 'become indistinguishable,' this is both untested and unlikely to be true for nondeterministic systems and agents operating in partially observable environments with stochastic transitions. In particular, conditioned on the system being in the generative phase, stochasticity in the neural network and uncertainty in the environment should cause the precision of the path integration estimate to decrease throughout time (as the system advances deeper into the theta phase). In fact, the authors' results (Fig. 3e) appear to show exactly this phenomenon. In particular, this is closely related to results from Ujfalussy & Orbán,  Elife, 2022, which use similar phenomena to support the notion that some form of sampling tied to theta phase oscillations is occurring in the hippocampus.\n\nQUESTIONS:\nCould you elaborate on simplifying assumptions made about hippocampal circuitry? It seems to me that the hippocampus plays no role in inference here (it just relays sensory information as an identity function without any plasticity and plays no role in path integration), whereas in your model the 'unlearned' sensory layer that prevents representational collapse could have been an earlier stage of sensory processing like the thalamus, allowing basal synapses in the hippocampus to learn as has been observed in experimental data (e.g. BTSP).\n\nWhat happens in this model during 'sleep'? I.e. if there are no vestibular cues provided to this network, does it still sample along its path integrator in a way that preserves spatial information? How does this relate to existing experimental information on sleep 'rehearsal' in the hippocampus?\n\nWhat happens if more noise is introduced into dynamics of the network during both inference and learning? In particular, I'm concerned about the absence of an explicit gate on plasticity. While it is conceptually an improvement to remove gating of plasticity tied to the theta oscillation, in the model of Bredenberg et. al 2021, the update is not strictly zero for inference parameters during the inference phase--it is only zero in mean (the same is true for the generative phase). As noise grows in the neural output, it is possible for increased variability in these updates, which could longitudinally compound errors or cause some form of representational drift. Have you tested for this possibility? If not, it may be important to note that the gate is only unnecessary in near-deterministic noise regimes.\n\nLIMITATIONS:\nThere are no obvious negative societal impacts of the authors' work, and beyond the questions I've raised above, the authors do an adequate job of addressing the limitations of their work and explicitly relating their contribution to previous studies.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors give a continuous version of the 'impression learning' [1] and use the theta oscillation to modulate wake-sleep phase.\n\n[1] Bredenberg, Colin, et al. \"Impression learning: Online representation learning with synaptic plasticity.\" Advances in Neural Information Processing Systems 34 (2021): 11717-11729.\n\nSTRENGTHS:\nThe authors links the theta oscillations to $\\theta_t$ in 'impression learning', which gives a neural implementation of the wake-sleep algorithm in Helmholtz machine.\n\nWEAKNESSES:\nFrom a computational perspective, the contribution lacks novelty.\n\nIn Seciton 5.2, the authors clearify their relationship to 'impression learning'. They list four differences:\n\n1. continuous version. \n\nThe discrete version is equivalent as long as the time steps are sufficiently small. And the authors did not provide the mathematical form of the Evidence Lower Bound (ELBO) for the continuous-time version.\n\n2. $\\theta_t$ can be ignored.\n3. disregard $1-k_t$.\n\nThese are only technical improvments.\n\n4. Link the theta oscillation (5Hz) to $\\theta_t$\n\nThis is a special case of  'impression learning'.  And the authors did not provide an explanation of how it is implemented from a computational standpoint. They merely established the connection without detailing the computational aspects.\n\nOverall, compared to 'impression learning,' I do not believe this work has enough novelty from a computational perspective. Although it provides a biological explanation of 'impression learning,' I do not think this paper is suitable for the current venue. Perhaps journals like Nature Neuroscience/Communications, or eLife would be more appropriate.\n\nQUESTIONS:\nThe latent variable $z$ is not used in the current model, so just disregard it. Eq.(1) can be ignored. If you want to use $z$, there must be a decoder from the neural activity $r$ to $z$.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a neat model that incorporates aspects of hippocampal function under one umbrella: \n* first, the input from the environment (z) goes into the sensory layer (p) and activates the internal state (g) in a certain way, the model captures this an \"inference\" or \"wake\" stage of the training\n* next, a set of recurrent connections in the \"internal state layer\" g simulates the prediction-making, guessing the next internal state, simulating the predictive mechanism in navigation (or potentially other cognitive functions)\n* finally, a generative pathway goes the opposite way from g down to p and attempts to generate what the sensory input should look like\n\nTraining of the inference and the generative parts of the model is alternative with a 5 Hz square wave function, which reflects existing observations of the role of this oscillation in hippocampus.\n\nThe contribution of the paper is just to propose a model that describes the dynamics of simple dynamical processes, with potential extension to path integration.\n\n------------- Update after the rebuttal period -------------\n\nI would like to thank the authors for a very detailed rebuttal and being engaged in the discussion on both the technical and the ~philosophical levels.\n\nMy main criticism was based on the idea that just training a model that captures something is not enough, because there is an infinite number of models & architectures that will be able to do that. However, after the rebuttal period and skimming the paper again I came to think that the model presented in this paper achieves more that just capturing the dynamics by \"any means necessary\", but actually does so under heavy restrictions, and, while there are still multiple such models possible, the fact that the model still works under said restrictions makes this result interesting.\n\nI am raising my score to \"7: Accept\" as this work, in my estimation achieves exactly the required level of \"high impact on at least one sub-area\" (computational modelling of hippocampal formation) and has \"good-to-excellent evaluation, resources, reproducibility\".\n\nSTRENGTHS:\nThis is a well-written paper, it has a good flow and is quite understandable. The proposed model combines various ideas about brain function in an elegant way and, while remaining simple, does manage to make those ideas to work in unison. The relevant work section gives great context for the work. The experimental work is well explained and documented.\n\nWEAKNESSES:\nAs someone who is not coming from the attractor perspective I fail to understand the significance of an attractor appearing. In my mind (please help me understand why this is not the case) the appearance on an attractor is exactly the goal and purpose of training. Basically, the way I see it, if we have a certain dynamical process, and we have successfully trained a neural network to capture this dynamics (the model can predict s(t+1) from s(t)) then saying \"it formed an attractor\" is equivalent to saying \"the learning has converged\". But isn't that precisely the purpose of training the model? The mere fact of it converging (and forming an attractor that captures the training data trajectory on a manifold) is, of course, a good thing -- the learning was successful, but is not in any way unexpected or remarkable. After all this is precisely what we wanted to happen -- we make an effort to build a model and the learning process that captures the data, and if done correctly, that's exactly what it is going to do.\n\nIn this work the fact that the generative part of the model had formed an attractor (aka \"can generate the dynamical process correctly\") is presented as a significant outcome. But from the machine learning perspective this is a trivial result, that IS what happens when you train something successfully and is not sufficient to imply a special biological significance of the model.\n\nPut it this way: if I would make this model to have not 1 sensory and 1 hidden layer, but, let's say 3 sensory and 4 hidden layers -- it would, of course, also form some sort of an attractor (with a more complex underlying manifold and shape due to the fact that we have more layers), but that would not tell me anything about this model's biological plausibility.\n\nI guess what I would like to discuss with the authors and other reviewers is whether creating A machine learning model has scientific value, or (as I posit) we need to make a step further and offer a model that would satisfy more than just capturing the dynamics, but also make predictions or coincide with biological restrictions that we did not explicitly encode, etc. Then -- yes -- we could say that not only this model captures the dynamics, but it also the only variant (or at least belongs to a small family of variants) that also do X, Y and Z the way the brain does it.\n\nIn the olden days of modeling proposing a set of differential equations that capture the dynamics was impressive because it was not a given than such a set of equations can ever be found. The outcome of such scientific endeavor could fail if the scientist was not able to describe the dynamics. So when a model was found nonetheless it made the contribution significant. But can we really apply the same criteria to ML-driven modeling? Because in the case of ML it is almost a given, that a \"set of equations\" (now represented by an artificial neural network) will be found to satisfy the data. Which tells us nothing more that \"this data can be described\" (which is rather noncathartical) and the is likely to be an infinite amount of such models that would do so (even the same architecture, trained again from a different initialization starting point would likely produce a \"new\" model).\n\nI am looking forward to discussing this with other reviewers and the authors, to help me understand (1) am I wrong about the (in)significance of an attractor appearing and (2) whether it is too much and prohibitive for science to ask more from a model in order to count it as a significant contribution to the field than just it being able to capture the data, especially when this data is simulated?\n\nFor the authors, one way to make this critique constructive and actionable I guess would be to say more in the paper about why this particular model (it's architecture and other characteristic) is especially suited to described hippocampal dynamics and why all the same conditions would not be satisfied by any number of similar models.\n\nOther points:\n\n* It would be great to see some ablation experiments that would help understand that the observed experimental results are unique to the proposed model and would not emerge if some of the critical components of the model are turned off (and help assess their criticality).\n\n* I also ask this as a question below, but I wonder why 2D environments were not explored experimentally? It would add so much to the work, make it relevant to more people and help build connections to existing work on hippocampal function, in particular place/grid cells.\n\nQUESTIONS:\nFig 1: What would be the appropriate intuition for the g layer? If z is \"input\" and p is \"sensory\", then should we think of g as an \"internal state\"?\n\n95: Are there studies that support the idea of the theta rhythm acting as the direction switch? Are there recordings that somehow show the change in the direction of the flow of information?\n\n147: Although in the Introduction (line 102) you mention that there is not discrete time assumption and the whole thing works continuously, I still can't help by wonder how many weight updates \"fit\" into one 5 Hz cycle for \"wake\" and \"sleep\" steps? Is this number driven by the computational power of the simulation hardware, cramming as many updates as possible, or there is a biologically-informed constrain that says something like \"within one theta cycle synaptic connection can be updated only X times\"? How does your implementation of the model handle this? Note: while the formulation is continuous, the act of updating is still a discrete step that takes place at some point, and this is what my question is about, basically how many times within one cycle does the line of the code that does w = w + update is being executed?\n\n156: Is it correct to think of g->g recurrent connection as a step in the internal representation space where the brain tries to predict what the next (t+1) state will look like? And this internal representation is then used as a starting point for the generative part of the model to correctly generate the state of the environment z_t+1?\n\n161: What is the \"ground truth\" for training w_p_A and w_g_A in the wake cycle? How do we know what what we should train toward do? I understand that the training is local and works to minimize the differences between dendritic (apical for the wake cycle, right?) and somatic activation, but how do we know that the dendritic activations represent the \"true\" ones and that bring the the system closer to them will result in successful learning?\n\n161: Same question rephrased: what is considered to be correct output of the inference part of the model?\n\n164: What is the ground truth of the generative training step? Let's say I run 1 pass from g -> p -> z, where does the learning signal come from? Do we compare the generated z_t+1 with the actual one to know whether the generative step has generated the correct stuff? What is considered the \"correct\" output of the generative part of the model and what this output is compared with in order to be able to say \"yep, it has generated correctly\"?\n\n209: When you say that only the \"wake\" part of the model is activated, how do you conclude that is \"correctly autoencodes\"? I guess this question is still stemming from the ones above, but I fail to see what is considered the ground truth? Does it go all the way to z, or it is sufficient at this point to train w_g_B (as per Fig 1) to produce such a g_t, that after g->g step that will create g_t+1 this g_t+1 will correctly be reconstructed via w_p_A back into original p?\n\n209: Another question here is when you \"autoencode\" do you autoencode p_t into p_t or do you \"autoencode over time\" so that p_t autoencodes into p_t+1? If not the latter, then I am confused when does the g->g step happen and what's the intuitive meaning of that step.\n\n(Sorry for the avalanche of questions, I imagine these will be hard to parse, I tried to formulate them the best I could :) )\n\n233: If the hippocampal \"internal\" part of the model is trained to autoencode g->p, then why would we expect sensory lesion to affect that in any way? You say it is remarkable that is keep generating, but isn't it exactly the behavior we would expect? I am trying to understand why the fact that g->p keep working is \"remarkable\" when we turn the z off. Since z is not part of that machinery, isn't it trivial that g->p will keep doing it's thing? Please let me know what I am missing and if the observation is not trivial as I postulate, then would would be the trivial behavior that we would expect but that is remarkably not occurring?\n\n280: It would be interesting to confirm that indeed MEC has learned some basic property of path integrating by running an ablation experiment, where MEC is randomly initialized and for 30 minutes on the MEC <-> HPC is allowed to train. As with echo networks it might the case that learning MEC <-> HPC is always sufficient even is MEC internal state is random. In the ablation experiment you can check whether using pre-trained MEC is any different from using randomly initialized MEC (given the same approximate shape of the value distribution), would it result in longer re-training times, worse performance, etc or not.\n\n301: Would anything change (in terms of the attractor, performance, etc) in instead of 5-10 Hz oscillations between wake and sleep you would use 1 Hz? 20 Hz? Is the 5 Hz crucial to the success of the model, and if yes - what would \"break\" is another rhythm would be implemented?\n\n* What was the main reason for choosing very simple experimental environment for this work? A 2D navigation experiment would help show so much more and build empirical connection to place/grid cells.\n\nLIMITATIONS:\nThere is not discussion of the limitations in the paper, and while I personally do not consider it a must-have, in this case it would be useful to help understand how far-reaching the authors deem their clams and results and which aspects of it rely on strong assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Hippocampus is postulated as a generative model to learn latent state representations and generate sensory predictions to solve spatial and nonspatial tasks. Theta-band oscillations are used to gate information flow into the generative model to modulate learning. A ring attractor develops within the generative model for path integration and flexibly transfer structures between environments. These computations are performed using biologically plausible learning rule.\n\nSTRENGTHS:\n-\tScheduling neural dynamics with theta oscillator: Model neurons are usually described as input summation followed by a nonlinear transformation. Here, the authors expanded the computational dynamics of a neuron by coupling the membrane potential with the theta oscillator (Eq. 4). This affords fine-grained control or better scheduling of the neurons and plasticity using a central gating mechanism.  \n\n-\tBiologically plausible learning rule: The model seems to be an extension of the MESH architecture which uses the Pseduoinverse learning rule. Here, the network is using Hebbian like learning algorithms to learn a ring attractor for path integration and can dynamically relearn a new environment even with some of the weights are fixed, adding to the list of biologically plausible models.\n\nWEAKNESSES:\n-\tNot a completely novel architecture: Although the multi-compartment neuron architecture controlled by the theta oscillator is novel, past works have described similar neural architecture (Sharma et al., 2022; Han et al., 2020) of having two hierarchical layers recurrently connected. I am curious to know how these neural architectures correspond to the author’s proposed architecture and if there can be some overlap in learning performance?\n\n-\tInsufficient analysis: It is not clear what computations the generative model performs using the ring attractor. The inference model seems to resemble place like receptive fields in Fig. 4b but so does the generative model? I anticipated grid like fields instead but this is not the case? It might also be interesting to show the synaptic weights ($Wgb$, $WpA$) evolve through learning and offer some insights to its computation, similar to Fig. 3c.\n\n-\tInclusion of network prediction prior to learning: Could the authors include the y axis information in Fig. 2b to show how much of the prediction error has been reduced through learning, and the activity of the inference (Fig. 2c) and generative (Fig. 2d) model prior to training to determine the difficulty of the task and the effectiveness of learning? \n\n-\tPath integration capability: Fig 3e demonstrates that the mean decoding error increases monotonically beyond 20 cm by 5 seconds. Can this high error still be claimed to be accurate path integration, where prior to the lesion, error was almost 0? What else could be integrated into the model such that the error does not increase as fast as in Fig. 3e?\n\n-\tApplication to a 2D navigation task: Authors demonstrated the model’s application to a 1D navigation task (left and right) which is rather limiting. Learning to path integrate in a 2D environment will be much more convincing. Furthermore, if the authors could demonstrate the agent ability to perform vector-based navigation by estimating its location using path integration and recalling a goal location from a memory system (Kumar et al., 2023; Foster et al., 2000), this will be a complete contribution to the research in biologically plausible spatial navigation using the entorhinal-hippocampal model. \n\n-\tInsufficient ablation studies: Ablation studies of the theta oscillator, multi-compartment neurons and individual layer plasticity will give us a better understanding of the roles played by each of these components for learning a ring attractor for path integration and remapping to a new environment. \n\nReferences:\n\nHan, D., Doya, K., & Tani, J. (2020). Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks. Neural Networks, 129, 149–162. https://doi.org/10.1016/j.neunet.2020.06.002 \n\nSharma, S., Chandra, S., & Fiete, I. R. (2022). Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold. ICML. http://arxiv.org/abs/2202.00159 \n\nKumar, M. G., Tan, C., Libedinsky, C., Yen, S., & Tan, A. Y.-Y. (2023). One-shot learning of paired associations by a reservoir computing model with Hebbian plasticity. ArXiv. http://arxiv.org/abs/2106.03580\n\nFoster, D. J., Morris, R. G., & Dayan, P. (2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1), 1–16. https://doi.org/10.1002/(SICI)1098-1063(2000)10:1<1::AID-HIPO1>3.0.CO;2-1\n\nQUESTIONS:\n-\tWould the plasticity rule still work if it was purely Hebbian plasticity i.e. $dW/dt = p(t)*g(t)$ without the additional normalizing term?\n\n-\tThe authors contrasted the wake-sleep algorithm to backpropagation. Could the authors compare the intuition of the wake-sleep algorithm to contrastive Hebbian learning? This might be a way to describe contrastive Hebbian learning using a biologically plausible alternative.\n\nLIMITATIONS:\nThe authors have sufficiently addressed their future directions. Model limitations and conditions at which the model breaks could have been further explored.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}