Learning grid cells by predictive coding

## Abstract

Abstract Grid cells in the medial entorhinal cortex (MEC) of the mammalian brain exhibit a strikingly regular hexagonal firing field over space. These cells are learned after birth and are thought to support spatial navigation but also more abstract computations. Although various computational models, including those based on artificial neural networks, have been proposed to explain the formation of grid cells, the process through which the MEC circuit learns to develop grid cells remains unclear.
In this study, we argue that predictive coding, a biologically plausible plasticity rule known to learn visual representations, can also train neural networks to develop hexagonal grid representations from spatial inputs. We demonstrate that grid cells emerge robustly through predictive coding in both static and dynamic environments, and we develop an understanding of this grid cell learning capability by analytically comparing predictive coding with existing models. Our work therefore offers a novel and biologically plausible perspective on the learning mechanisms underlying grid cells. Moreover, it extends the predictive coding theory to the hippocampal formation, suggesting a unified learning algorithm for diverse cortical representations.

## 1 Introduction

Our brain contains a rich set of neural representations of space that help us navigate in an ever-changing world. These include hippocampal place cells (O’Keefe, ), which fire when an animal is at a specific spatial position, and grid cells observed in the medial entorhinal cortex (MEC) (Hafting et al., ), which fire when an animal occupies multiple positions on a hexagonal or triangular grid. Grid cells have been observed across various species (Fyhn et al., ; Yartsev et al., ; Doeller et al., ), and their remarkable regularity has raised extensive interest in the computational mechanism underlying their emergence. Earlier models have focused on how mechanisms, such as membrane potential oscillation (O’Keefe & Burgess, ; Hasselmo et al., ) and specialized recurrent connectivity, can generate grid-like firing patterns (Fuhs & Touretzky, ; Burak & Fiete, ). More recently, research has shown that grid cells can emerge in recurrent neural networks (RNNs) trained using backpropagation through time (BPTT) for path integration tasks. The models are trained to predict their current location by integrating velocity inputs (Cueva & Wei, ; Banino et al., ; Whittington et al., ; Sorscher et al., ), providing a normative, task-driven account of the computational problem that the MEC grid cells address. However, the process by which the MEC circuit acquires, or *learns* the grid cells in a biologically plausible way has been largely neglected, despite the fact that grid cells are known to be learned, rather than hardwired at birth (Langston et al., ; Wills et al., ). Existing learning models (e.g. Weber & Sprekeler ()) are highly specialized for grid cells,
and it is unclear whether plasticity rules for only one specific cell type exist in the brain.

In this paper, we directly tackle the learning problem underlying the emergence of grid cells using *predictive coding*, an algorithm modeling the plasticity rules for a variety of cortical functions and representations (Rao & Ballard, ; Friston, ). Our approach to modeling grid cell emergence through predictive coding is motivated by three key factors: Firstly, the predictive coding algorithm can be implemented in predictive coding networks (PCNs) with local computations and Hebbian plasticity (Bogacz, ), making it more biologically plausible than learning rules such as backpropagation. Secondly, PCNs have been successful in replicating representations in other regions of the brain, such as the visual cortex (Rao & Ballard, ; Olshausen & Field, ; Millidge et al., ). Thirdly, PCNs have demonstrated the ability to perform hippocampus-related functions, such as associative and sequential memories (Salvatori et al., ; Tang et al., ; ).

The primary contribution of this work is to demonstrate for the first time that grid cells naturally emerge in PCNs trained to represent spatial inputs with biologically plausible plasticity rules. In this work we:

- •
show that hexagonal grid cells develop as the latent representations of place cells in classical PCNs (Rao & Ballard, ; Olshausen & Field, ) with sparse and non-negative constraints;
- •
train a dynamical extension of classical PCNs, called temporal predictive coding network (tPCN) (Millidge et al., ), in path integration tasks and observe that the latent activities of the tPCN develop hexagonal, grid-like representations, similar to what has been discovered in RNNs;
- •
develop an understanding of grid cell emergence in tPCN, by showing analytically that the Hebbian learning rule of tPCN implicitly approximates truncated BPTT (Williams & Peng, );
- •
show that tPCN can robustly develop grid cells under different architectural choices, and even without velocity inputs in path integration.

Overall, our results present an effective and plausible learning rule for hexagonal grid cells in the MEC based on predictive coding. We offer a novel extension of predictive coding theory, which has traditionally been used to model visual representations (Rao & Ballard, ; Olshausen & Field, ), to encompass spatial representations in the MEC. Our findings therefore offer a novel understanding of how a single, unified learning algorithm can be employed by different brain regions to represent inputs of various levels of abstraction.

## 2 Related work

#### Computational Models of Grid Cells

The periodicity of grid cells inspired early models of grid cells based on membrane potential oscillations, where the periodic firing of grid cells results naturally from the interference between somatic and dendritic oscillators in MEC pyramidal neurons (O’Keefe & Burgess, ; Hasselmo et al., ). These models were subsequently extended to incorporate multiple networks of oscillatory neurons (Zilli & Hasselmo, ). However, these models lack biological plausibility as they require an unrealistically large number of networks (Giocomo et al., ). Another major family of models leverages the recurrent attractor networks and obtains grid firing patterns (Fuhs & Touretzky, ; Burak & Fiete, ; Ocko et al., ) by hand-tuning the recurrent connectivity to form a center-surround structure. These networks perform robust and accurate path integration (Burak & Fiete, ) and can explain experimental observations such as the deformation of grid cells in irregular environments (Ocko et al., ). However, as pointed out by Sorscher et al. (), these models lack an explanation for the underlying spatial task that gives rise to the specific recurrent connectivity.

To address this gap, recent studies have explored the question *‘If grid cell is the answer, what is the question?’*. Dordek et al. () showed that grid cells emerge as the non-negative principal components of place cells, while Stachenfeld et al. () proposed that grid cells form a basis for predicting future observations. Other studies have focused on the multi-modularity of grid cells by optimizing biologically constrained objective functions (Dorrell et al., ; Schaeffer et al., ). Notably, multiple research tracks have found that RNNs trained to perform path integration tasks will develop hexagonal grid representations in their latent states (Cueva & Wei, ; Banino et al., ; Whittington et al., ), suggesting that grid cells emerge as a result of successful navigation. These findings were further reinforced by Sorscher et al. (), who analytically demonstrated that path integration with certain implementation choices, such as non-negativity, is a sufficient condition for the emergence of grid cells, clarifying earlier controversies (Schaeffer et al., ). However, none of these works have addressed how the MEC/hippocampal network learns the grid cells. The RNN models are trained by BPTT, a learning rule unlikely to be employed by the brain (Lillicrap & Santoro, ). Even though the principal component model by Dordek et al. () can be learned with the plausible Sanger’s rule (Sanger, ), it has been shown that principal component analysis (PCA) cannot be applied to other brain regions such as the visual areas (Olshausen & Field, ), and Sanger’s rule cannot be generalized to dynamical tasks such as path integration. Earlier models of the learning process of grid cells have explored plausible learning rules such as spike time-dependent plasticity (Widloski & Fiete, ) and variants of Hebbian learning rules (Kropff & Treves, ) within networks of excitatory and inhibitory neurons (Weber & Sprekeler, ). However, these learning rules are highly specialized, and have not been shown to reproduce representations from other brain regions with non-spatial tasks. Recent works have also modeled the hippocampal formation using generative models with plausible learning rules similar to predictive coding (George et al., ; Bredenberg et al., ), though these studies did not address 2D spatial learning.

#### Predictive Coding

Predictive coding has been an influential theory in understanding cortical computations (Friston, ; Rao & Ballard, ; Bogacz, ) and has been applied to modeling various cortical functions (see Millidge et al. () for a review). Specifically, in the visual cortex, PCNs develop realistic visual representations such as Gabor-like receptive fields in response to both static (Rao & Ballard, ; Olshausen & Field, ) and moving stimuli (Millidge et al., ). Recently, theories have been developed to describe hippocampo-neocortical interactions using predictive coding (Barron et al., ), and PCNs have demonstrated the ability to memorize and retrieve static and dynamic visual patterns, a key function of the hippocampus (Salvatori et al., ; Tang et al., ; ). Our work explores whether the representational learning capabilities of predictive coding can be extended to the hippocampal formation, which has so far only been functionally modeled by PCNs.

The computations of PCNs use only local neural dynamics and Hebbian plasticity, making it biologically more plausible than backpropagation (Whittington & Bogacz, ). It has also been shown that predictive coding approximates backpropagation both in theory and practice (Whittington & Bogacz, ; Song et al., ; Pinchetti et al., ). Unlike many other Hebbian learning rules, predictive coding can be extended to temporal predictive coding networks (tPCNs), which use recurrent connections to process dynamic stimuli (Millidge et al., ). However, while Millidge et al. () demonstrated that tPCNs approximate Kalman filtering, the relationships between tPCNs and RNNs remain unclear. In this work, we train tPCNs for path integration and compare their performance with RNNs both analytically and experimentally in this context.

## 3 Models

Figure: Figure 1: Architecture and circuit implementation of PCNs. A: Sparse, non-negative PCN as a generative model. During learning, $\mathbf{p}$ is given and the latent $\mathbf{g}$ and $\mathbf{W}$ are inferred and learned through a type of EM algorithm. B: Simlar to A, but with dynamic inputs $\mathbf{p}_{t}$ and recurrent weights $\mathbf{W}_{\textnormal{r}}$. The dashed velocity inputs are optional (see Section 4.4). C: Circuit implementation of tPCN, adapted from Tang et al. () with a mapping to MEC and hippocampus.
Refer to caption: x1.png

#### Non-negative Sparse PCN

We first investigate the classical PCN (Rao & Ballard, ) for its ability to form grid representations. Assuming a place cell input $\mathbf{p}\in\mathbb{R}^{N_{p}}$ that represents a location in $2$D space as an $N_{p}$-dimensional vector, a simple $2$-layer PCN generates predictions of $\mathbf{p}$ using its latent activities $\mathbf{g}\in\mathbb{R}^{N_{g}}$ (which will develop grid-like representations) and a weight matrix $\mathbf{W}$ (Fig 1A). The generative model minimizes the following loss function subject to two constraints:

$$ $\mathcal{L}_{\textnormal{PCN}}=\|\mathbf{p}-\mathbf{W}\mathbf{g}\|_{2}^{2}+\| \mathbf{g}\|_{2}^{2}+2\lambda\|\mathbf{g}\|_{1}$ (1) $$

where $\|\mathbf{g}\|_{2}^{2}$ constrains the $l$2 norm of the latent $\mathbf{g}$ and $\lambda\|\mathbf{g}\|_{1}$ enforces sparsity, similar to the sparse coding model (Olshausen & Field, ). This loss function is minimized via an expectation-maximization (EM) algorithm, alternating between the optimization over $\mathbf{g}$ (inference) and $\mathbf{W}$ (learning) (see Appendix A.1 for the training algorithm):

$$ $\Delta\mathbf{g}\propto-\nabla_{\mathbf{g}}\mathcal{L}_{\textnormal{PCN}}=- \mathbf{g}-\lambda\textnormal{sgn}(\mathbf{g})+\mathbf{W}^{\top}\boldsymbol{ \epsilon}^{\mathbf{p}};\quad\mathbf{g}\leftarrow\texttt{ReLU}(\mathbf{g}+ \Delta\mathbf{g})$ (2) $$

$$ $\Delta\mathbf{W}\propto-\nabla_{\mathbf{W}}\mathcal{L}_{\textnormal{PCN}}= \boldsymbol{\epsilon}^{\mathbf{p}}\mathbf{g}^{\top}$ (3) $$

where $\boldsymbol{\epsilon}^{\mathbf{p}}:=\mathbf{p}-\mathbf{W}\mathbf{g}$ and we apply a ReLU to the inference dynamics to constrain the latent activities to be non-negative. The inference and learning dynamics can be implemented in a plausible circuit (Bogacz, ). After convergence, we examine the firing fields of the latent activities $\mathbf{g}$.

#### Path Integrating tPCN

To account for the learning of spatial representations in moving animals, we also investigate tPCN that extends the classical PCNs to the temporal domain (Millidge et al., ; Tang et al., ) in path integration tasks (Fig. 1B). The model receives a series of place cell activities $\mathbf{p}_{1},...,\mathbf{p}_{T}$ and velocity inputs $\mathbf{v}_{1},...,\mathbf{v}_{T}$ that represent the trajectory of an agent moving in a $2$D space, and minimizes the following loss function at each time step $t$:

$$ $\mathcal{L}_{\textnormal{tPCN},t}=\|\mathbf{p}_{t}-f(\mathbf{W}_{\textnormal{ out}}\mathbf{g}_{t})\|_{2}^{2}+\|\mathbf{g}_{t}-h(\mathbf{W}_{\textnormal{r}} \hat{\mathbf{g}}_{t-1}+\mathbf{W}_{\textnormal{in}}\mathbf{v}_{t})\|_{2}^{2}$ (4) $$

where $f$ and $h$ are both nonlinear activation functions, and $\mathbf{W}_{\textnormal{in}}$, $\mathbf{W}_{\textnormal{r}}$ and $\mathbf{W}_{\textnormal{out}}$ are weight matrices projecting the predictions. We define $\boldsymbol{\epsilon}^{\mathbf{p}}_{t}:=\mathbf{p}_{t}-f(\mathbf{W}_{
\textnormal{out}}\mathbf{g}_{t})$, $\boldsymbol{\epsilon}^{\mathbf{g}}_{t}:=\mathbf{g}_{t}-h(\mathbf{W}_{
\textnormal{r}}\hat{\mathbf{g}}_{t-1}+\mathbf{W}_{\textnormal{in}}\mathbf{v}_{
t})$. The model learns by first optimizing the loss function with respect to $\mathbf{g}_{t}$ via gradient descent:

$$ $\Delta\mathbf{g}_{t}\propto-\nabla_{\mathbf{g}_{t}}\mathcal{L}_{\textnormal{ tPCN},t}=-\boldsymbol{\epsilon}^{\mathbf{g}}_{t}+\mathbf{W}_{\textnormal{out}} ^{\top}f^{\prime}(\mathbf{W}_{\textnormal{out}}\mathbf{g}_{t})\boldsymbol{ \epsilon}^{\mathbf{p}}_{t}$ (5) $$

and then optimizing weights by:

$$ $\begin{split}\{\Delta\mathbf{W}_{\textnormal{out}},\Delta\mathbf{W}_{ \textnormal{r}},\Delta\mathbf{W}_{\textnormal{in}}\}&\propto-\nabla_{\{\mathbf {W}_{\textnormal{out}},\mathbf{W}_{\textnormal{r}},\mathbf{W}_{\textnormal{in} }\}}\mathcal{L}_{\textnormal{tPCN},t}\\ &=\{f^{\prime}(\mathbf{W}_{\textnormal{out}}\mathbf{g}_{t})\boldsymbol{ \epsilon}^{\mathbf{p}}_{t}\mathbf{g}_{t}^{\top},h^{\prime}(\tilde{\mathbf{g}}_ {t})\boldsymbol{\epsilon}^{\mathbf{g}}_{t}\hat{\mathbf{g}}_{t-1}^{\top},h^{ \prime}(\tilde{\mathbf{g}}_{t})\boldsymbol{\epsilon}^{\mathbf{g}}_{t}\mathbf{v }_{t}^{\top}\}\end{split}$ (6) $$

where $f^{\prime}$ and $h^{\prime}$ are Jacobians of the nonlinear functions $f$ and $h$, and $\tilde{\mathbf{g}}_{t}:=\mathbf{W}_{\textnormal{r}}\hat{\mathbf{g}}_{t-1}+
\mathbf{W}_{\textnormal{in}}\mathbf{v}_{t}$. After the inference (Equation 5) converges, we set $\hat{\mathbf{g}}_{t}$ to the converged value of $\mathbf{g}_{t}$, which will be used for optimizing the objective function at the next time step i.e., $\mathcal{L}_{\textnormal{tPCN},t+1}$. The model is trained on a large number of trajectories $\{\mathbf{v}_{t},\mathbf{p}_{t}\}$ and after training, a set of velocity inputs from unseen trajectories is presented to the model. The model then performs a forward pass through time and layers to predict the positions encoded by place cells (see Appendix A.1 for the training and testing algorithms of tPCN):

$$ $\mathbf{g}_{t}=h(\mathbf{W}_{\textnormal{r}}\mathbf{g}_{t-1}+\mathbf{W}_{ \textnormal{in}}\mathbf{v}_{t}),\quad\hat{\mathbf{p}}_{t}=f(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})$ (7) $$

The model is evaluated on 1) the accuracy of path integration position prediction $\hat{\mathbf{p}}_{t}$ and 2) the firing fields of the latent $\mathbf{g}$. When both $f$ and $h$ are linear, these computations can be plausibly implemented in a neural circuit shown in Figure 1C, with local inference computations (Equation 5) and Hebbian learning rules (Equation 6) (Millidge et al., ). When the activation functions involve only local nonlinearity, such as tanh or ReLU, the Jacobians are diagonal and the inference and learning rules remain local and Hebbian (Millidge et al., ), and additional circuitry components can be included to plausibly implement the nonlinearities (Whittington & Bogacz, ). Within the context of spatial representation learning, this circuit implementation can be naturally mapped to the circuitry of the hippocampal formation. We discuss the relationship of this circuit implementation to existing and potential experimental evidence in the Discussion section.

#### Input of the Model

In models discussed in this work, we assume that grid cells are inferred as latent representations of place cells. Although previous models have followed the opposite direction of the relationship, several strands of experimental evidence have suggested the emergence of grid cells as a result of place cells, including the earlier development of place cells (Bush et al., ; Langston et al., ; Wills et al., ). In both PCN and tPCN models, the place cell inputs are constructed as $2$D difference-of-softmaxed-Gaussian (DoS) curves flattened into $1$D vectors, which have been shown to yield hexagonal grid representations in RNNs (Schaeffer et al., ; Sorscher et al., ). The firing centers of the place cells are uniformly distributed across a $2$D environment. For PCN, the inputs are $N_{x}$ evenly distributed locations in the environment ($N_{x}$ large enough to cover the whole environment) represented by the $N_{p}$ place cells. For tPCN, the trajectories for the path integration task are obtained by simulating an agent performing a smooth random walk in the square environment. At each point in time, the $N_{p}$ place cells will be uniquely activated, representing the agent’s current location. The velocity inputs $\mathbf{v}_{t}$ are $2$D vectors representing the speed of the simulated agent on the $x$ and $y$ coordinates at time step $t$. The effect of boundaries is simulated by slowing down the agent and reverting its moving direction near the borders of the environment. We sample a large number of trajectories to cover the whole simulated environment for training.

## 4 Results

### 4.1 Sparse non-negative PCN develops latent grid cells

Figure: Figure 2: Grid cells developed in PCN. A: Latent representations of a sparse, non-negative PCN, resembling hexagonal grid cells in the MEC. Numbers in the title reflect the grid scores. B: Grid cells obtained via the pattern formation theory/non-negative PCA discussed in Sorscher et al. (); Dordek et al. (). C, D: Latent representations without sparsity or non-negativity, respectively. E: Distribution of grid scores of the representations in A and B.
Refer to caption: x2.png

Here we examine whether the sparse non-negative PCN can develop hexagonal, grid-like latent representations of the space after training, by plotting each latent neuron’s responses to the $N_{x}=900$ locations in the $2$D space. We use $N_{p}=512$ and $N_{g}=256$. The “gridness” of the 2D latent representations is evaluated using the *grid score* metric, commonly employed in both experimental and computational studies (Sargolini et al., ; Banino et al., ) (see A.3 for grid score calculation). We found that this simple, $2$-layer PCN can develop hexagonal grid cells similar to those observed in the MEC (Figure 2A). For comparison, we reproduce the results from Dordek et al. () and Sorscher et al. (), which show theoretically that performing non-negative PCA on the place cell inputs is guaranteed to produce hexagonal grid representations as the principal components of the $N_{x}\times N_{p}$ place cell input matrix. The visual results of the reproduction are shown in Figure 2B, and we compare the distribution of grid scores of the PCN’s latent neuron firing fields with those of the non-negative principal components in Figure 2E. The grid scores between our sparse non-negative PCN and non-negative PCA are similarly distributed.

Why does the sparse, non-negative PCN develop hexagonal grid cells? While a precise analytical explanation is left for future work, we offer an intuitive hypothesis here. When presented with a batch $N_{x}$ of place cell inputs, the objective of PCN (Equation 1) can be written compactly as:

$$ $\mathcal{L}_{\textnormal{PCN}}=\|\mathbf{P}-\mathbf{G}\mathbf{W}^{\top}\|_{F}^ {2}+\textstyle{\sum_{i=1}^{N_{x}}}\|\mathbf{g}_{i}\|_{2}^{2}+2\lambda\|\mathbf {g}_{i}\|_{1}$ (8) $$

where $\mathbf{P}\in\mathbb{R}^{N_{x}\times N_{p}}$ is the place cell activities across $N_{x}$ locations, and $\mathbf{G}\in\mathbb{R}^{N_{x}\times N_{g}}$ represents grid cell responses. On the other hand, the objective function of PCA is:

$$ $\mathcal{L}_{\textnormal{PCA}}=\|\mathbf{P}-\mathbf{G}\mathbf{M}\|_{F}^{2} \quad\textnormal{s.t.}\quad\mathbf{G}\mathbf{G}^{\top}={\bm{I}}_{N_{x}}$ (9) $$

where $\mathbf{M}$ is the $N_{g}\times N_{p}$ readout matrix. The constraint $\mathbf{G}\mathbf{G}^{\top}={\bm{I}}_{N_{x}}$ in Equation 9 enforces orthonormality of the grid cell matrix $\mathbf{G}$ columns, meaning they are orthogonal and have unit norm. We hypothesize that the constraint $\|\mathbf{g}_{i}\|_{2}^{2}+2\lambda\|\mathbf{g}_{i}\|_{1}$ for our sparse PCN achieves this orthonormality implicitly: while the constraints are imposed on the rows of $\mathbf{G}$, the overall sparsity of entries in $\mathbf{G}$ could induce orthogonality among its columns, with the $l_{2}$ term constraining the norm of the columns to achieve normality implicitly. Indeed, Figure 2C shows that if we remove the sparsity constraint, the latent neurons’ firing fields will no longer be hexagonal. Similarly, without ReLU i.e., non-negativity applied to the inference dynamics, we also could not obtain hexagonal grid cells (Figure 2D). It is worth noting that although (non-negative) PCA can be learned with the biologically plausible Sanger’s rule (Sanger, ), it lacks PCN’s generalizability to different architectures (Salvatori et al., ) and to other brain regions such as the visual cortex (Olshausen & Field, ; Rao & Ballard, ).
However, it can be noticed that the grid cells by PCN lack the multi-modularity of the grid cells by non-negative PCA i.e., grid cells with different firing periods.
We suspect that although sparse PCNs can approximate the orthonormality of latent variables, they lack PCA’s ability to extract latent variables ordered by the amount of explained variance in data, with higher variance naturally corresponding to larger spatial scales and vice versa.

### 4.2 tPCN develops grid cells by path integration

Figure: Figure 3: tPCN in path integration. A: Visual demonstration of the performance of tPCN and RNN in path integration. B: RMSEs between the decoded and ground-truth 2D positions by tPCN and RNN with different agent moving speed. C: Grid score distributions of tPCN and RNN with different agent moving speed. D, E: Firing fields of latent neurons in a tPCN and an RNN respectively, when $dt=0.02$. F, G: Firing fields of latent neurons in a tPCN and an RNN respectively, when $dt=0.1$.
Refer to caption: x3.png

Although training a static PCN with a large number of place cell activations can already give rise to brain-like hexagonal grid cells,
the emergence of grid cells is known to rely on dynamic motion of animals (McNaughton et al., ; Winter et al., ). Therefore, we investigate tPCN in a path integration task, where the simulated agent uses dynamic velocity inputs to determine its current position. As a reference, we compare tPCN with RNNs trained in path integration, which have been shown to develop hexagonal grid cells (Cueva & Wei, ; Banino et al., ; Sorscher et al., ) and share the same graphical structure as tPCN (Figure 1B). However, it is important to note that RNNs are trained with the biologically implausible backpropagation-though-time (BPTT) algorithm, which requires “unrolling” of the network through time, a process unlikely to occur in the brain (Lillicrap & Santoro, ).

We first evaluate whether tPCN can learn to perform the path integration task using local and Hebbian learning rules. We trained a tPCN model with $N_{g}=2048$ latent neurons on trajectories within a $1.4\textnormal{m}\times 1.4$m environment represented by $N_{p}=512$ place cells. After training, we tested the model on a set of unseen trajectories with velocity input $\mathbf{v}_{t}$, and assessed whether the tPCN and RNN models could predict the correct positions using Equation 7. As the output of the networks is the $N_{p}$-dimensional population activity of the place cells, we calculate the predicted $2$D positions by averaging the center positions of the $3$ most active place cells in the output $\hat{\mathbf{p}}_{t}$, and calculate the root mean square error (RMSE) between the decoded and ground-truth $2$D positions. The visual and numerical results are shown in Figure 3A and B, where we also varied a scaling factor $dt$ of the simulated agent’s speed, sampled from a Rayleigh distribution with mean $1$, to test the robustness of the results. Note that we do not intend to model physiologically realistic speed of animals with these values. The performance of tPCN is comparable to that of the RNN, though it slightly deteriorates when the agent moves at higher speeds.

Next, we examine whether the tPCN model develops grid-like representations in its latent layer during path integration. We plot the firing fields of the $2048$ latent neurons given an unseen set of trajectories covering the entire space. The neurons with the highest grid scores are shown in Figure 3C, which reveals a grid-like, hexagonal firing pattern with high grid scores. Visually, these grid cells are similar to those in a trained RNN with the same architecture shown in Figure 3E, replicating the results from (Sorscher et al., ). To systematically compare the grid cells in tPCN and RNN, we plot the distribution of grid scores in both models as a function of the movement speed of the agent in the environment in Figure 3C. When the movement is slow, the grid score distributions are similar between tPCN and RNN. However, as the $dt$ increases to $0.05$ and $0.1$, tPCN tends to have higher grid scores than RNN. This is visually reflected in Figure 3F (tPCN) and G (RNN), which shows the latent representations developed by tPCN largely retain the grid-like pattern whereas firing centers of many of the RNN neurons no longer form a grid when $dt=0.1$. Interestingly, the band-like representations present in both models in this case are observed in MEC (Krupic et al., ), although their existence is controversial (Navratilova et al., ).

### 4.3 tPCN approximates truncated BPTT

Figure: Figure 4: Comparing tPCN and tBPTT. A: Dependencies of latent grid cells in tPCN and RNN trained with $1$-step tBPTT. Black arrows indicate the flow of computations during a forward pass and red arrows indicate the dependency of latent variables. B: Firing fields of the latent neurons of an RNN trained by $1$-step tBPTT. C, D: Path integration RMSE and grid score distributions of $1$-step tBPTT, BPTT and tPCNs with different inference iterations. “tPCNk” indicates tPCN trained with k inference iterations.
Refer to caption: x4.png

Next, we asked why hexagonal grid representations emerge both when training a tPCN using a BPTT-free Hebbian learning rule and when training an RNN using BPTT. We provide an analytical comparison between the learning rules of tPCN and RNN. Assuming a vanilla, sequence-to-sequence RNN with exactly the same graphical structure as in Figure 1A, its dynamics can be recursively described as:

$$ $\mathbf{g}_{t}=h(\mathbf{W}_{\textnormal{r}}\mathbf{g}_{t-1}+\mathbf{W}_{ \textnormal{in}}\mathbf{v}_{t});\quad\hat{\mathbf{p}}_{t}=f(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})$ (10) $$

The loss that this RNN is trained to minimize is the cumulative prediction error:

$$ $\mathcal{L}_{\textnormal{RNN}}=\textstyle{\sum_{t=1}^{T}\mathcal{L}_{ \textnormal{RNN},t}=\sum_{t=1}^{T}\|\mathbf{p}_{t}-\hat{\mathbf{p}}_{t}\|_{2}^ {2}}$ (11) $$

Suppose BPTT is performed at every step $t$ to update weights in this RNN, the learning rule for $\mathbf{W}_{\textnormal{r}}$ at step $t$ can be expressed as (see Appendix A.2 for derivations):

$$ $\Delta\mathbf{W}_{\textnormal{r}}^{\textnormal{RNN}}=\textstyle{\sum_{k=1}^{t} \tfrac{\partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}h^{\prime}(\tilde{ \mathbf{g}}_{t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})\boldsymbol{\epsilon}_{t}^{\mathbf{p}} \underline{\mathbf{g}_{k-1}^{\top}}}$ (12) $$

where $\boldsymbol{\epsilon}_{t}^{\mathbf{p}}$ denotes the prediction error $\mathbf{p}_{t}-\hat{\mathbf{p}}_{t}$ and the $\frac{\partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}$ terms correspond to the unrolling in BPTT, which can be factorized into a chain of partial derivatives (Bellec et al., ). On the other hand, for tPCN, if we assume that the inference dynamics in Equation 5 has fully converged ($\Delta\mathbf{g}_{t}=0$) at the time of weight update, the learning rule of tPCN can be written as (see Appendix A.2 for derivations):

$$ $\Delta\mathbf{W}_{\textnormal{r}}^{\textnormal{tPCN}}=h^{\prime}(\tilde{ \mathbf{g}}_{t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})\boldsymbol{\epsilon}_{t}^{\mathbf{p}} \underline{\hat{\mathbf{g}}_{t-1}^{\top}}$ (13) $$

Two key differences between these learning rules stand out. First, tPCN does not involve the recursive unrolling term, thereby avoiding the need to maintain a perfect memory of all preceding hidden states. Second, instead of using the forward-propagated $\mathbf{g}_{t-1}$ as in Equation 10, tPCN employs the inferred $\hat{\mathbf{g}}_{t-1}$ from Equation 5 (underlined). The first difference suggests an equivalence between tPCN and RNN trained with truncated BPTT (tBPTT) with a truncation window of size 1 (1-step tBPTT) (Williams & Peng, ), where the RNN does not backpropagate *any* hidden states through time when updating the weights. This characteristic could potentially harm the RNN’s performance as it cannot effectively perform temporal credit assignment. However, the second difference partially solves this problem, as $\hat{\mathbf{g}}_{t-1}$ is inferred following Equation 5, which includes the term $\boldsymbol{\epsilon}^{\mathbf{p}}_{t-1}$ that communicates the place cell prediction error at step $t-1$. Therefore, when $\mathbf{W}_{\textnormal{r}}$ is updated at step $t$, the $\hat{\mathbf{g}}_{t-1}^{\top}$ term in $\Delta\mathbf{W}_{\textnormal{r}}^{\textnormal{tPCN}}$ will effectively form an eligibility trace (Bellec et al., ) that allows the model to access historical prediction errors on the place cell level. Figure 4A illustrates this difference between tPCN and RNN trained by $1$-step tBPTT, highlighting the dependency of tPCN hidden states on past place cell activations. In Appendix A.2 we also discuss the relationship between the update rules for $\mathbf{W}_{\textnormal{in}}$ and $\mathbf{W}_{\textnormal{out}}$ in these two models.

To verify this theoretical difference, we compare tPCN with RNNs trained by tBPTT in the path integration task. Since $\hat{\mathbf{g}}_{t}$ in tPCN is initialized by a forward pass $f(\mathbf{W}_{\textnormal{r}}\hat{\mathbf{g}}_{t-1})$ and then updated by the iterative inference (Appendix A.1), the behavior of 1-step tBPTT, which computes its latent states via a forward pass at each time step, should be closer to tPCN with fewer inference iterations. Therefore, we evaluate tPCN with various inference iterations. Figure 4B shows the grid cells learned by an RNN trained with $1$-step tBPTT, which still exhibit hexagonal grid firing fields, though with lower grid scores than those from full BPTT. This suggests that backpropagating the error through all time steps is not entirely necessary for RNNs to generate grid cell-like representations. In Figure 4C we show the path integration performance of RNN by 1-step tBPTT and BPTT, as well as tPCNs with different inference iterations from $1$ to $50$. As can be seen, tPCN with a single inference iteration has identical performance to RNN trained by tBPTT, and its performance will improve as we increase the number of inference iterations but will saturate around $20$ iterations. Overall, this graph suggests that tPCN with $5$ or more inference iterations can effectively perform temporal credit assignment that improves upon tPCN$1$ or $1$-step tBPTT, potentially due to the eligibility trace. However, this eligibility trace arises from local inference dynamics (Equation 5) rather than from unrolling the RNN graph as in Bellec et al. (). This improvement is also reflected in the grid scores (Figure 4D), although increasing the inference iterations does not necessarily result in better grid score representations. We suspect that although the gridness of latent representations is somewhat related to path integration performance, their relationship is not linear. It is also worth noting that to fully evaluate the similarities and differences between BPTT and tPCN, an in-depth comparison is needed across different tasks and versions of tBPTT. We aim to investigate this question in future works as it is beyond the scope of this paper.

### 4.4 Robustness of grid cell representations in tPCN

Figure: Figure 5: Robust emergence of grid cells in tPCN. A, B: Path integration RMSE and grid scores of tPCN in different setups. “Stationary baseline” refers to a model that always predicts the initial position regardless of movement. C-I: Firing fields of latent neurons in tPCNs with C: Gaussian place cells; D: $f=$tanh; E: $h=$tanh; F: $1.8\textnormal{m}\times 1.8\textnormal{m}$ environment; G: $1.2\textnormal{m}\times 1.2\textnormal{m}$ environment; H: $256$ latent neurons; I: tPCN without velocity input.
Refer to caption: x5.png

Inspired by Schaeffer et al. (), we examine the robustness of our results against different architectural choices of the tPCN model, to understand what contributes to the emergence of grid cells within tPCN. Specifically, we vary the following components of the model: 1) Encoding of the place cell activities; 2) Output nonlinearity $f$; 3) Recurrent nonlinearity $h$; 4) Environment sizes; 5) Latent sizes and 6) Velocity input to the model. The baseline model has DoS place cell encodings, $h=\texttt{ReLU}$, $f=\texttt{softmax}$, $1.4\textnormal{m}\times 1.4\textnormal{m}$ environment and latent size $2048$ with velocity inputs.

We first examine whether replacing the place encoding with Gaussian curves affects the model’s performance. As shown in Figure 5A, B and C, the Gaussian place cells do not affect the path integration performance, but the latent representations are no longer hexagonal. This is consistent with earlier findings that the DoS place cell encoding is necessary for hexagonal grid cells (Dordek et al., ; Sorscher et al., ; Schaeffer et al., ).

The choices of $f$ and $h$ are particularly interesting: as discovered by earlier works (Dordek et al., ; Sorscher et al., ), a choice of $h$ that imposes non-negativity constraint on the latent activities, such as ReLU, is necessary for the emergence of hexagonal grid cells. In our tPCN model, the activation functions are also important for biological plausibility: in both Equation 5 and Equation 6, the multiplication with the Jacobians $h^{\prime}$ and $f^{\prime}$ can be reduced to local, element-wise multiplications if $h$ and $f$ are element-wise nonlinearities such as ReLU and tanh. Although it is possible to design a circuit to perform the computations in softmax (Snow & Orchard, ), it is unclear how the Jacobian matrix of softmax can be computed in a biological circuit. Therefore, we first replace $f$ with a tanh function in our tPCN model and evaluate the model’s performance in both path integration and its latent representations. As shown in Figure 5A, replacing $f$ with tanh results in slightly worse path integration performance and lower grid scores than the softmax baseline. However, visually, the latent representations are hexagonal and grid-like (Figure 5D), suggesting that using a biologically more plausible $f$ would not significantly affect the emergence of grid cells within tPCN. On the other hand, replacing the non-negative constraint (ReLU) on the latent activities with $h=\texttt{tanh}$ results in the amorphous latent representations (Figure 5E), which is consistent with Sorscher et al. ().

We next investigate the impact of the size of the environment, by training tPCN within a square environment of size $1.8\textnormal{m}\times 1.8\textnormal{m}$ (big) and an environment of size $1.2\textnormal{m}\times 1.2\textnormal{m}$ (small). Changing environment sizes does not affect the path integration performance, and does not affect tPCN’s capability of developing grid cells either (Figure 5F for big environment and G for small environment). We also vary the number of latent neurons in the model from $256$ to $512$ and $1024$, which does not affect the grid cell representations (Figure 5H shows the latent representations learned by a tPCN with $256$ latent neurons). However, with fewer latent neurons, the performance in path integration becomes worse as the model has fewer number of parameters to perform the task (Figure 5A).

Earlier studies using PCNs to model visual representations have mostly used unsupervised PCNs (Rao & Ballard, ; Olshausen & Field, ; Millidge et al., ), which corresponds to blocking the velocity input $\mathbf{v}_{t}$ into tPCN in Figure 1B. Here we asked how removing velocity input would affect the path integration performance and grid cell emergence of tPCN. Mathematically, this is achieved simply by re-defining $\tilde{\mathbf{g}}_{t}:=\mathbf{W}_{\textnormal{r}}\hat{\mathbf{g}}_{t-1}$ without changing any inference or learning dynamics.
It can be seen from Figure 5A that the path integration performance is significantly affected by the absence of velocity input, with an RMSE even higher than the stationary baseline, where the model does not predict any movement at all. Intriguingly, the latent representations developed by this unsupervised tPCN are still grid cell-like (Figure 5I) with a similar grid score distribution to the baseline model. This result demonstrates that grid cells can still emerge even in a model unable to perform path integration at all. Therefore, our model predicts that path integration is not a sufficient condition for the emergence of grid cells, which resonates with Schaeffer et al. (). In other words, it predicts that animals unable to navigate due to impaired velocity encoding may still develop grid cells as a result of self-motion.

## 5 Discussion

#### Relationship to Experimental Observations

Here, we highlight properties of the biologically plausible circuit in Figure 1C, including those consistent with experimental observations, and those generating prediction about the hippocampal formation. This circuit can be naturally divided into a MEC layer and a hippocampal layer. The MEC layer contains velocity-encoding neurons ($\mathbf{v}$) and grid cells ($\mathbf{g}$), which aligns with experimental findings of the conjunctive representations of velocity and grids in the entorhinal cortex (Sargolini et al., ). In our model, grid cells in the MEC layer are recurrently connected through a specialized circuit involving interneurons $\hat{\mathbf{g}}_{t-1}$ that inhibit the output signal from the grid cells, allowing the error neurons $\boldsymbol{\epsilon}^{\mathbf{g}}_{t}$ to compute the temporal prediction errors. Experimental evidence suggests that lateral interactions in layer II of the MEC are predominantly inhibitory (Witter & Moser, ) and are mediated by interneurons such as basket cells (Jones & Bühl, ). Our model also predicts that these interneurons encode an eligibility trace $\hat{\mathbf{g}}_{t-1}$ from the immediate past. While recent studies have reported grid cells representing prospective locations (Ouchi & Fujisawa, ), it remains to be verified whether these cells are mechanistically supported by such “past” cells. Additionally, neurons in the entorhinal cortex are known to encode errors (Ku et al., ), suggesting a possible error-driven learning mechanism similar to that in tPCN.

In our model, the MEC and hippocampus are bidirectionally connected, a well-documented characteristic of entorhinal-hippocampal connectivity (Canto et al., ). Crucially, the circuit also posits the existence of error neurons $\boldsymbol{\epsilon}^{\mathbf{p}}_{t}$ in the hippocampus, which encode the discrepancy between place cell activities and inputs from MEC grid cells. The CA1 sub-region of the hippocampus has been shown to serve as a mismatch detector between the hippocampus and cortex (Lisman, ; Duncan et al., ). Our model predicts that in spatial navigation, the error neurons $\boldsymbol{\epsilon}^{\mathbf{p}}_{t}$ in the hippocampus, whose existence has been supported by Wirth et al. () and Ku et al. (), can encode exactly this mismatch signal between the two regions.

#### Conclusion

In this work, we have demonstrated a biologically plausible learning rule for grid cells based on predictive coding. We have shown that with sparsity and non-negative constraints, classical PCNs can develop grid cell-like representations of batched place cell inputs. With inputs representing trajectories of moving agents, tPCN can also develop grid cell activations while performing path integration. We have developed a theoretical understanding of this property of tPCN by deriving and comparing its learning dynamics with that of BPTT, showing that unrolling a recurrent network is unnecessary for it to learn grid cells, and a more plausible approach with recursive inference dynamics should suffice. Furthermore, we have examined the robustness of our results by varying hyper-parameters of the model, and found that grid cells can be learned even without velocity inputs. Overall, our work demonstrates that predictive coding can serve as an effective and biologically plausible plasticity rule for neural networks to learn grid cells observed in the MEC. Importantly, compared with earlier learning rules specialized for grid cells, predictive coding is a general learning rule able to reproduce many other cortical functions and representations. Thus, our findings suggest that a single, unified plausible learning rule can be employed by the brain to find the most appropriate representation of cortical inputs in different regions.

## Acknowledgement

This work has been supported by Medical Research Council UK grant MC_UU_00003/1 to RB, an E.P. Abraham Scholarship in the Chemical, Biological/Life and Medical Sciences to MT, and UKRI Future Leaders Fellowship to HB (MR/W008939/1). The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work.
http://dx.doi.org/10.5281/zenodo.22558

## Reproducibility Statement

The code used for the experiments in this paper can be found at [https://github.com/C16Mftang/place-cell-pred-coding](https://github.com/C16Mftang/place-cell-pred-coding). All hyperparameters for training are detailed in the appendix. Additionally, proofs for the theoretical results discussed in the paper are also included in the appendix for verification.

## Appendix A Appendix

### A.1 Algorithms

Below is the training algorithm for a sparse, non-negative PCN given spatial inputs $\mathbf{p}$. We obtain the grid cells shown in the main text directly by taking the converged latent activities $\mathbf{g}$ after training.

Figure: Algorithm 1 Learning latent representations of space with a PCN

Below is the training algorithm for tPCN in path integration tasks. The testing performance and grid cells shown in the main text are obtained by performing a forward pass through the model after training, given an unseen trajectory $\{\mathbf{v}_{t},\mathbf{p}_{t}\}$.

Figure: Algorithm 2 Path integration with tPCN

### A.2 Derivations of learning dynamics

Here we derive the recurrent weight update rules for $\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}$ (Equation 12) and $\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{r}}$ (Equation 13). For RNN, we assume that the weights are updated at each time step and therefore $\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}$ is updated following the chain rule:

$$ $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}=-\frac{d\mathcal{L}_{ \textnormal{RNN},t}}{d\mathbf{W}_{\textnormal{r}}}=-\frac{d\mathcal{L}_{ \textnormal{RNN},t}}{d\mathbf{g}_{t}}\frac{d\mathbf{g}_{t}}{d\mathbf{W}_{ \textnormal{r}}}$ (14) $$

We first look at the term $\frac{d\mathbf{g}_{t}}{d\mathbf{W}_{\textnormal{r}}}$, which, following the rule of partial derivatives, can be written as:

$$ $\begin{split}\frac{d\mathbf{g}_{t}}{d\mathbf{W}_{\textnormal{r}}}&=\frac{ \partial\mathbf{g}_{t}}{\partial\mathbf{W}_{\textnormal{r}}}+\frac{\partial \mathbf{g}_{t}}{\partial\mathbf{g}_{t-1}}\frac{d\mathbf{g}_{t-1}}{d\mathbf{W}_ {\textnormal{r}}}\\ &=\frac{\partial\mathbf{g}_{t}}{\partial\mathbf{W}_{\textnormal{r}}}+\frac{ \partial\mathbf{g}_{t}}{\partial\mathbf{g}_{t-1}}\left(\frac{\partial\mathbf{g }_{t-1}}{\partial\mathbf{W}_{\textnormal{r}}}+\frac{\partial\mathbf{g}_{t-1}}{ \partial\mathbf{g}_{t-2}}\frac{d\mathbf{g}_{t-2}}{d\mathbf{W}_{\textnormal{r}} }\right)\\ &=...\\ &=\sum_{k=1}^{t}\frac{\partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}\frac{ \partial\mathbf{g}_{k}}{\partial\mathbf{W}_{\textnormal{r}}}\end{split}$ (15) $$

due to the recursive and implicit dependency of $\mathbf{g}_{t}$ on $\mathbf{g}_{t-1}$ and $\mathbf{g}_{t-1}$ on $\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}$ for all $t$. Thus, the update rule can be written as:

$$ $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}=-\sum_{k=1}^{t}\frac{d \mathcal{L}_{\textnormal{RNN},t}}{d\mathbf{g}_{t}}\frac{\partial\mathbf{g}_{t} }{\partial\mathbf{g}_{k}}\frac{\partial\mathbf{g}_{k}}{\partial\mathbf{W}_{ \textnormal{r}}}$ (16) $$

Since $\mathbf{g}_{k}=h(\tilde{\mathbf{g}}_{k})=h(\mathbf{W}_{\textnormal{r}}\mathbf{
g}_{k-1}+\mathbf{W}_{\textnormal{in}}\mathbf{v}_{k})$, and $\mathcal{L}_{\textnormal{RNN},t}=\|\mathbf{p}_{t}-f(\mathbf{W}_{\textnormal{
out}}\mathbf{g}_{t})\|_{2}^{2}$ the update rule can be written as:

$$ $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{r}}=\sum_{k=1}^{t}\tfrac{ \partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}h^{\prime}(\tilde{\mathbf{g}}_{ t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{\textnormal{out} }\mathbf{g}_{t})\boldsymbol{\epsilon}_{t}^{\mathbf{p}}\mathbf{g}_{k-1}^{\top},$ (17) $$

concluding our proof for Equation 12. The derivation for $\mathbf{W}^{\textnormal{RNN}}_{\textnormal{in}}$ is similar:

$$ $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{in}}=-\frac{d\mathcal{L}_{ \textnormal{RNN},t}}{d\mathbf{W}_{\textnormal{in}}}=-\frac{d\mathcal{L}_{ \textnormal{RNN},t}}{d\mathbf{g}_{t}}\frac{d\mathbf{g}_{t}}{d\mathbf{W}_{ \textnormal{in}}},$ (18) $$

and

$$ $\begin{split}\frac{d\mathbf{g}_{t}}{d\mathbf{W}_{\textnormal{in}}}&=\frac{ \partial\mathbf{g}_{t}}{\partial\mathbf{W}_{\textnormal{in}}}+\frac{\partial \mathbf{g}_{t}}{\partial\mathbf{g}_{t-1}}\frac{d\mathbf{g}_{t-1}}{d\mathbf{W}_ {\textnormal{in}}}\\ &=\frac{\partial\mathbf{g}_{t}}{\partial\mathbf{W}_{\textnormal{in}}}+\frac{ \partial\mathbf{g}_{t}}{\partial\mathbf{g}_{t-1}}\left(\frac{\partial\mathbf{g }_{t-1}}{\partial\mathbf{W}_{\textnormal{in}}}+\frac{\partial\mathbf{g}_{t-1}} {\partial\mathbf{g}_{t-2}}\frac{d\mathbf{g}_{t-2}}{d\mathbf{W}_{\textnormal{in }}}\right)\\ &=...\\ &=\sum_{k=1}^{t}\frac{\partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}\frac{ \partial\mathbf{g}_{k}}{\partial\mathbf{W}_{\textnormal{in}}}\end{split}$ (19) $$

Therefore, the update rule for $\mathbf{W}_{\textnormal{in}}$ in an RNN can be written as:

$$ $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{in}}=\sum_{k=1}^{t}\tfrac{ \partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}h^{\prime}(\tilde{\mathbf{g}}_{ t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{\textnormal{out} }\mathbf{g}_{t})\boldsymbol{\epsilon}_{t}^{\mathbf{p}}\mathbf{v}_{k}^{\top}$ (20) $$

Finally, the update rule for $\mathbf{W}^{\textnormal{RNN}}_{\textnormal{out}}$ can be straightforwardly expressed as:

$$ $\begin{split}\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{out}}&=-\frac{d \mathcal{L}_{\textnormal{RNN},t}}{d\mathbf{W}_{\textnormal{out}}}\\ &=-\frac{d\mathcal{L}_{\textnormal{RNN},t}}{d\hat{\mathbf{p}}_{t}}\frac{d\hat{ \mathbf{p}}_{t}}{d\mathbf{W}_{\textnormal{out}}}\\ &=f^{\prime}(\mathbf{W}_{\textnormal{out}}\mathbf{g}_{t})\boldsymbol{\epsilon} _{t}^{\mathbf{p}}\mathbf{g}_{t}^{\top}\end{split}$ (21) $$

as there is no recursive dependency.

For tPCN, at each time step $t$ the following loss is minimized with respect to $\mathbf{W}_{\textnormal{r}}$:

$$ $\mathcal{L}_{\textnormal{tPCN},t}=\|\mathbf{p}_{t}-f(\mathbf{W}_{\textnormal{ out}}\mathbf{g}_{t})\|_{2}^{2}+\|\mathbf{g}_{t}-h(\mathbf{W}_{\textnormal{r}} \hat{\mathbf{g}}_{t-1}+\mathbf{W}_{\textnormal{in}}\mathbf{v}_{t})\|_{2}^{2}$ (22) $$

Since $\hat{\mathbf{g}}_{t-1}$ is inferred through Equation 5, rather than forward-propagated by $\mathbf{W}_{\textnormal{r}}$, the recursive dependency on $\mathbf{W}_{\textnormal{r}}$ disappears, and thus the update rule for $\mathbf{W}_{\textnormal{r}}$ can be locally derived as:

$$ $\Delta\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{r}}=-\frac{d\mathcal{L}_{ \textnormal{tPCN},t}}{d\mathbf{W}_{\textnormal{r}}}=h^{\prime}(\tilde{\mathbf{ g}}_{t})\boldsymbol{\epsilon}^{\mathbf{g}}_{t}\hat{\mathbf{g}}_{t-1}^{\top}$ (23) $$

If we also assume that the inference dynamics in Equation 5 have converged when the weights are updated, namely:

$$ $\Delta\mathbf{g}_{t}=0\Rightarrow\boldsymbol{\epsilon}^{\mathbf{g}}_{t}= \mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{\textnormal{out}} \mathbf{g}_{t})\boldsymbol{\epsilon}^{\mathbf{p}}_{t},$ (24) $$

the update rule can be written as:

$$ $\Delta\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{r}}=h^{\prime}(\tilde{ \mathbf{g}}_{t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})\boldsymbol{\epsilon}^{\mathbf{p}}_{t}\hat{ \mathbf{g}}_{t-1}^{\top},$ (25) $$

which concludes our proof for Equation 13. Similarly, following the same assumption of converged inference and Equation 6, the update rule $\Delta\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{in}}$ can be written as:

$$ $\Delta\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{in}}=h^{\prime}(\tilde{ \mathbf{g}}_{t})\mathbf{W}_{\textnormal{out}}^{\top}f^{\prime}(\mathbf{W}_{ \textnormal{out}}\mathbf{g}_{t})\boldsymbol{\epsilon}^{\mathbf{p}}_{t}\mathbf{ v}_{t}^{\top}$ (26) $$

It can be seen that it differs from $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{in}}$ only in the absence of the unrolling term $\tfrac{\partial\mathbf{g}_{t}}{\partial\mathbf{g}_{k}}$. On the other hand, the update rules $\Delta\mathbf{W}^{\textnormal{RNN}}_{\textnormal{out}}$ and $\Delta\mathbf{W}^{\textnormal{tPCN}}_{\textnormal{out}}$ are exactly the same.

### A.3 Experimental setups and hyperparameters

#### Place cell and trajectory parameters

We use DoS place cell encodings throughout most of our experiments. Formally, the activity of the $i$th place cell with this encoding, given a particular location $x$ can be written as:

$$ $K(x,C,\tau):=\exp{\left(-\frac{(x-C)^{2}}{\tau\xi^{2}}\right)}$ (27) $$

$$ $p_{i}=\frac{K(x,C_{i},2)}{\sum_{j=1}^{N_{p}}K(x,C_{j},2)}-\frac{K(x,C_{i},4)}{ \sum_{j=1}^{N_{p}}K(x,C_{j},4)}$ (28) $$

where $C_{i}$ is the center of the place cell’s firing field, and $\tau$ and $\xi$ define the width of the firing field’s center and surround. The table below specifies parameters defining the place cells and trajectories:

| $\xi$ | Path length | Average agent speed | Environment size |
| --- | --- | --- | --- |
| 0.12m | 10 steps | {0.02, 0.05, 0.1}m/s | {$1.4^{2},1.8^{2},2.0^{2}$}m^2 |

Specifically, at time step $t=0$, a $2$D position and a head direction scalar in $[0,2\pi]$ are randomly initialized. At each of the subsequent time steps, a random turn angle is sampled from a normal distribution and a random speed is sampled from a Rayleigh distribution. Both values are then multiplied by $dt$ mentioned in the main text. If the simulated agent hits a border wall at this time step, its speed is slowed and its turn angle is inverted. The position of the agent is updated according to the speed and turn angle at this time step. The trajectories are simulated using parameters adapted from the code provided in Sorscher et al. ().

#### Model and training hyperparameters

In our experiments, we have used three models: sparsity and non-negativity constrained PCN, RNN and tPCN. The table below specifies parameters of model architectures:

| Model | $N_{p}$ | $N_{g}$ | $h$ | $f$ |
| --- | --- | --- | --- | --- |
| sparse, non-neg. PCN | 512 | 256 | N/A | N/A |
| tPCN | 512 | {256,512,1024,2048} | {$\texttt{ReLU},\texttt{tanh}$} | {$\texttt{softmax},\texttt{tanh}$} |
| RNN | 512 | 2048 | ReLU | softmax |

The table below specifies hyperparameters used in training RNN and tPCN. We use Adam optimizer for all weight updates, and plain SGD for inference dynamics in tPCN. We found that in general, RNNs take more epochs to converge in the path integration task.

| Model | $N_{x}$ | batch size | learning rate | inference step size | epochs | inference iters | weight decay |
| --- | --- | --- | --- | --- | --- | --- | --- |
| tPCN | 50000 | 500 | $10^{-4}$ | $10^{-2}$ | $150$ | $20$ | $10^{-4}$ |
| RNN | 50000 | 500 | $10^{-4}$ | N/A | $200$ | N/A | $10^{-4}$ |

The table below specifies hyperparameters used in training the sparse, non-negative PCN. We use Adam optimizer for all weight updates, and plain SGD for inference dynamics.

| $N_{x}$ | batch size | learning rate | inference step size | epochs | inference iters | weight decay | $\lambda$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 900 | 100 | $2\times 10^{-3}$ | $10^{-2}$ | $600$ | $20$ | $10^{-5}$ | 0.05 |

#### Calculation of grid scores

The following grid score calculation process is adapted from Sargolini et al. () and the code of Sorscher et al. (). It is summarized below for completeness and clarity:

- •
Get the rate map of latent neurons (potentially hexagonal grid cells);
- •
Place one copy of the rate map on top of the other, and start moving the top copy by $\delta\in\mathbb{R}^{2}$. If the rate maps are hexagonal grids, for particular $\delta$’s that make the firing peaks overlap, the autocorrelation between the stationary and moved maps will be 1; otherwise, the autocorrelation will be 0. We will then have a hexagonal autocorrelation map if the rate map itself is hexagonal;
- •
We then rotate the autocorrelation map and compute the correlation between each rotated map and the original map. If the rate maps are hexagonal, the correlation as a function of rotated degrees will be sinusoidal, with 60 and 120 degrees as peaks and 30, 90 and 150 degrees as troughs.
- •
Grid score is calculated as the minimum difference between the peak and trough correlation, which in theory is a real value in range $[-2,2]$.

All experiments were performed on a single Tesla V100 GPU.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: - The soundness of the paper is high, but my primary concerns center around the novelty of the algorithm beyond tPCN itself. Simply applying a non-negative constraint and applying to a new task does not seem like a sufficiently novel contribution for ICLR. It is unclear what enhancements of the algorithm could be necessary in the context of spatial navigation.
soundness: 3
strengths: -	The general finding of the tPCN is encouraging, and the generalization to a different task than the Millidge 2024 paper is promising. 
-	The robustness experiments (4.4) show that the emergent grid-like activity is robust to model architectures. This is encouraging, since many experimental neuroscience manipulations show grid cells to be robust to manipulations of the environment of neural activity.
summary: The authors investigate how a temporally-dependent version of predictive coding can extract compact latent spaces in the form of periodic grid activity from temporally structured place cell input. The findings are of general interest to theories of learning in biological settings, and replicate many previous results with a more biologically plausible learning mechanism.
weaknesses: -	Overall, the study seems like an incremental follow on of the tPCN paper applied to a new domain, but which does not require fundamental changes to the original algorithm. 
-	The path integrating tPCN assumes input in the form of place cell activity, but does not account for how place cells and grid cells form from the combination of visual and self-motion information. Combined with the lack of anatomical constraints of direction of connectivity, the study is more about the formation of compressed latent spaces than the medial temporal lobe. Several existing studies, largely cited in the paper, already investigate the formation of such successor representations by predictive coding.
-	The authors dismiss previous examples of learned grid cells (Dordek, Stachenfeld, Schaffer, etc) on the basis that these are not biologically plausible learning methods, but then move to use real-valued activation functions. There is no evidence from the methods presented in this paper that a spike-based temporal predictive coding network would converge.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: I have two questions:

1. In both model architectures presented, grid cell activity depends on input from place cells. However, in biological systems, place cell activity varies significantly across different environments, showing a phenomenon known as global remapping, whereas grid cells maintain a stable 2D toroidal manifold across environments. How does this model account for this discrepancy? If place cell activity, the input source for grid cells, changes substantially across environments, how does the model explain the stability of grid cell activity?

2. In the medial entorhinal cortex (MEC), grid cells are organized into modules with distinct spacings. In the model proposed in this paper, do the network’s grid cells display discrete spacing distributions, and are there any indications of modular independence in their connectivity?
soundness: 2
strengths: To the best of my knowledge, this paper is the first to suggest that a predictive coding network can serve as a biologically plausible model for learning grid cells and perform simulations to validate this hypothesis. Additionally, the paper extends the application of PCN’s locally-based learning method to approximate backpropagation (BP) in temporally processing networks, using tPCN. While not formally proven, the authors draw comparisons between tPCN and 1-step BPTT, indicating that with multi-step inferences, tPCN’s performance could approach that of BPTT.
summary: This paper proposes that the mechanism by which grid cells are learned in biological systems may involve predictive coding. To test this hypothesis, the authors trained both a predictive coding network (PCN) and a temporal predictive coding network (tPCN) on path integration and non-path integration tasks. They observed that hexagonal firing patterns, characteristic of grid cells, emerged in both paradigms. Since PCN and tPCN introduce error cells that enable learning with spatially and temporally local rules, this discovery suggests a biologically plausible mechanism for grid cell formation. The authors also analyze the learning process in tPCN, comparing it analytically with 1-step backpropagation through time (BPTT), to explain the emergence of grid cells. Finally, they assess the robustness of grid cell emergence in their model by testing various activation functions, environments, and network sizes.
weaknesses: The main limitation lies in novelty. First, previous studies have already shown that grid cells can be learned either through non-negative PCA or via a single-layer BP-based network from place cell activity. Likewise, RNNs trained via BPTT for path integration to predict place cell activity have also been reported (see Sorscher et al., 2022). Additionally, the ability of PCN to approximate BP using local learning rules has been demonstrated previously (see Song et al., 2020), and the t-PCN structure’s capacity to approximate BPTT is a straightforward extension of prior work (Millidge et al., 2024). The robustness analysis in this paper largely follows procedures established in earlier RNN studies and does not report new phenomena (Schaeffer et al., 2022). Other biologically plausible learning algorithms, such as those using Oja’s rule, have also achieved grid cell-like activity, suggesting that this paper’s algorithm is not unique in this regard. Overall, the contribution seems to synthesize existing ideas without introducing significant innovation.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: see weakness
soundness: 2
strengths: The paper is clearly written, and the question is well-defined.
summary: This study demonstrates that predictive coding can effectively train neural networks to develop hexagonal grid representations from spatial inputs, providing a biologically plausible explanation for the emergence of grid cells in the medial entorhinal cortex. By analytically comparing predictive coding with existing models, we offer new insights into the learning mechanisms of grid cells and extend predictive coding theory to the hippocampal formation, suggesting a unified learning algorithm for various cortical representations.
weaknesses: My major concern is that the work may lack novelty.

1. The use of non-negative and sparse network designs to produce grid cell-like patterns has been extensively discussed. For example, [1] reported that non-negative and sparse properties can generate grid cell -like patterns and theoretically demonstrated why non-negativity is the main driver of grid cell formation (which the author's  paper does not address) instead of sparsity. Similar findings were also reported in [2]. Earlier, [3] proves that a nonnegativity constraint on firing rates induces a symmetry-breaking mechanism which favors hexagonal firing fields. [4] further explored, through extensive experiments, the conditions necessary for generating grid cells.

2. Prediction tasks, including path integration, that produce grid cell-like patterns have also been widely reported, especially when the input data takes a place cell-like form. For instance, [5] also used place cell like input and path integration tasks to train networks and generate grid cells, while [6] theoretically analyzed the role of predictive learning in forming low-dimensional representations.

3. In my understanding, tPCN is very similar to a one-step RNN (apart from the difference in local learning rules), so the fact that its training process resembles that of one-step tBPTT is not surprising. As previously noted, the key to forming grid cells lies in the predictive task, not the RNN network itself. Therefore, the similarity between tPCN and RNN does not offer significant insight into the generation of grid cells.

For the reasons above, I believe this paper does not offer substantial novelty or make a clear contribution to the field.



[1]Whittington, James CR, et al. "Disentanglement with biological constraints: A theory of functional cell types." *arXiv preprint arXiv:2210.01768* (2022).

[2]Dorrell, William, et al. "Actionable neural representations: Grid cells from minimal constraints." *arXiv preprint arXiv:2209.15563* (2022).

[3]Sorscher, Ben, et al. "A unified theory for the origin of grid cells through the lens of pattern formation." *Advances in neural information processing systems* 32 (2019).

[4]Schaeffer, Rylan, Mikail Khona, and Ila Fiete. "No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit." *Advances in neural information processing systems* 35 (2022): 16052-16067.

[5]Whittington, James CR, et al. "The Tolman-Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation." *Cell* 183.5 (2020): 1249-1263.

[6]Recanatesi, Stefano, et al. "Predictive learning as a network mechanism for extracting low-dimensional latent space representations." *Nature communications* 12.1 (2021): 1417.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: The authors find that grid cells emerge under various configurations and constraints, even in the absence of velocity input. Could they expand on the implications of this finding for the role of predictive coding in spatial learning?

You claim that tPCN approximates tBPTT; however, the RMSE indicates that when the inference has fully converged, the tPCN outperformed tBPTT. Path integration is a Markov process, and it therefore makes sense that tBPTT should work. However, as you show, having the extra inference steps helps. Is it then tPCN that approximates tBPTT or the other way around (tBPTT approximates tPCN)

Moreover, this begs the question: what is the difference between $g_{t-1}$ from RNNs and $\hat{g}_{t-1}$ tPCNs that give this performance boost? 

Is there a qualitative difference in grid cells between the models, or are there other cell types that make $\hat{g}_{t-1}$ "better"? One way to hint at this would be to ablate neurons in $g$ and rank them according to their effect on the loss. Are there any differences between these two populations? Another way would be to perform a detailed analysis of the predictive power of $g$ cells in the two models, for example, according to Ouchi et al.

Related works, such as the work from Giocomo in 2011, are outdated. Whether oscillatory dynamics are important for grid cells started as you point out with the work by [Burgess](https://pmc.ncbi.nlm.nih.gov/articles/PMC2678278/) and Hasselmo, but it was later included in CANNs by [Bush and Burgess](https://pubmed.ncbi.nlm.nih.gov/24695724/). The importance of oscillations in grid cells has been tested experimentally by  [Lepperød et al](https://www.science.org/doi/full/10.1126/sciadv.abd5684), [Schmidt-Hieber et al.](https://www.nature.com/articles/nn.3340), [Robinson et al.](https://www.sciencedirect.com/science/article/pii/S2211124724009197)

**Minor**

 - $\hat{g}$ is used but not introduced as inferred before line 392; this can be nice to point out earlier.
 - Whether grid cells are learned or are there from birth is disputed; I would present this in less certain terms.
soundness: 4
strengths: **Originality:**
This paper provides a new perspective on grid cell formation by applying predictive coding. While previous work has used RNNs trained with BPTT to simulate grid cells, this study introduces predictive coding networks (PCNs) and temporal PCNs (tPCNs) as biologically plausible alternatives. While predictive coding has been addressed in hippocampal formation previously (Stachenfeld et al. ++), the proposed learning rules are novel in this context. 

**Quality:**
The authors demonstrate grid cell emergence in PCNs and perform a comparative analysis with existing RNN models. By analytically showing that tPCNs approximate truncated BPTT, they provide a theoretical solid grounding for their approach. Further, the robustness analysis—exploring different model architectures, non-linearities, and environments—addresses shortcomings proclaimed in recent work (Sorscher vs. Schaeffer). The theoretical and empirical sections are well-integrated.

**Clarity:**
The authors use clear visual representations of presented ideas, making interpretation intuitive.  The derivations are well-presented, especially in demonstrating the correspondence between tPCNs and truncated BPTT. However, some technical details on the inference dynamics of tPCNs might benefit from additional clarity or simplification, especially for readers less familiar with predictive coding. 

**Significance:**
The findings are interesting for neuroscience and machine learning. They suggest that predictive coding may underpin not only perceptual but also spatial and navigational representations. For neuroscience, predictive coding may unify perspectives across cortical functions. For machine learning, it offers an alternative to backpropagation-based learning in dynamic systems.
summary: The paper investigates the emergence of grid cells, known for their hexagonal firing patterns in spatial navigation, using predictive coding—a biologically plausible learning rule. The authors propose that grid cells can be learned by neural networks through predictive coding, which aligns well with the principles of local computations and Hebbian plasticity.

The key contributions are:

 - Demonstrating that predictive coding networks (PCNs) can naturally develop grid cell representations with sparse, non-negative constraints, and a temporal extension (tPCN) achieves similar results in dynamic tasks like path integration.
 - Establishing that tPCNs approximate the truncated backpropagation through time (BPTT), highlighting a biologically plausible alternative to BPTT for learning grid cells.
 - Analyzing the robustness of grid cell emergence in PCNs and tPCNs across varied architectural and environmental conditions, showing grid cells can still form even without velocity input.
weaknesses: Although it is nice to see grid cells emerge in the proposed setup, it is not that surprising given the setup with static place cell readout. The comparison between BPTT and tPCNs is more interesting, in my opinion, than the grid cell results and can have broader implications beyond this particular setting; I would present this as the main result and, therefore, consider moving this result to an earlier stage and presenting the grid cell stuff as a test case.

The model operates under certain assumptions (e.g., reliance on sparsity, non-negative constraints, simplified path integration tasks, and place cell readout) that may not generalize well across different types of neuronal representations or tasks. However, the discussion lacks a critical assessment of these assumptions, specifically regarding where the predictive coding model might fall short compared to other frameworks for grid cells, such as the recent development of self-supervised learning for grid cells ([Schaeffer et al.](https://arxiv.org/abs/2311.02316)), conformal isometry, or distance preservation ([Xu et al.](https://arxiv.org/abs/2210.02684), [Dorell et al.](https://arxiv.org/abs/2209.15563)). For example, the choice of static read-out place cells limits studies of remapping (but can be done; see [Schøyen et al.](https://www.sciencedirect.com/science/article/pii/S258900422302179X), different geometries [Krupic et al.](https://www.nature.com/articles/nature14153) etc.

The proposed predictive coding model successfully generates grid cells, but the mechanistic explanation for how and why grid cells emerge under predictive coding is lacking. Moreover, the field suffers from challenges in comparing representations across studies, barring visual inspection. Grid scores are used to assess grid cell likeness; however, these give little insight beyond 60-degree symmetry. I suggest you use something else to assess the function of the networks, such as ablation studies and studying the full representational setting of the network. For example, do you see border cells, band cells, etc? At least provide examples, preferably representations from the full network, in the supplementary.

All in all, since the title and introduction of the paper highlight grid cells, I would expect more analysis of this finding and a broader comparison with the existing literature. However, I think the more interesting finding is the comparison between BPTT and tPCNs. Therefore, I would recommend lifting this part of the paper and proposing the grid cell story as a potential application motivating further studies on this line of work, although I do see your point on extended analysis on this being out of scope.