<figure><img src="image_1.png" /></figure>

HAL
open science

Semantic and episodic memories in a predictive coding model of the neocortex

Lucie Fontaine, Frédéric Alexandre

<figure><img src="image_2.png" /></figure>

To cite this version:

Lucie Fontaine, Frédéric Alexandre. Semantic and episodic memories in a predictive coding model of the neocortex. IJCNN 2025 - International Joint Conference on Neural Networks, Jun 2025, Rome, Italy. hal-05202850

HAL Id: hal-05202850

https://hal.science/hal-05202850v1

Submitted on 20 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Semantic and episodic memories in a predictive coding model of the neocortex

Lucie Fontaine *, Frédéric Alexandre †

Centre Inria de l'université de Bordeaux

Laboratoire Bordelais de Recherche en Informatique (LaBRI)

Institut des Maladies Neurodégénératives (IMN)

Bordeaux, France

Email: * fontaluc@gmail.com, † frederic.alexandre@inria.fr

*Abstract*—Complementary Learning Systems theory holds that intelligent agents need two learning systems. Semantic memory is encoded in the neocortex with dense, overlapping representations and acquires structured knowledge. Episodic memory is encoded in the hippocampus with sparse, pattern-separated representations and quickly learns the specifics of individual experiences. Recently, this duality between semantic and episodic memories has been challenged by predictive coding, a biologically plausible neural network model of the neocortex which was shown to have hippocampus-like abilities on auto-associative memory tasks. These results raise the question of the episodic capabilities of the neocortex and their relation to semantic memory. In this paper, we present such a predictive coding model of the neocortex and explore its episodic capabilities. We show that this kind of model can indeed recall the specifics of individual examples but only if it is trained on a small number of examples. The model is overfitted to these examples and does not generalize well, suggesting that episodic memory can arise from semantic learning. Indeed, a model trained with many more examples loses its recall capabilities. This work suggests that individual examples can be encoded gradually in the neocortex using dense, overlapping representations but only in a limited number, motivating the need for sparse, pattern-separated representations as found in the hippocampus.

*Index Terms*—episodic memory, semantic memory, neocortex, hippocampus, predictive coding, generative model, auto-associative memory.

## I. INTRODUCTION

Different models of neural networks have been shown to store different representations of information, that can be related in neuroscience to episodic memory and semantic memory [1], [2]. The way these representations of information interact is much discussed both in Neuroscience and AI with tremendous implications in the nature of underlying cognitive functions [3]. This debate motivates the present experimental study.

### A. Neuroscience

1) *Complementary Learning Systems:* According to Complementary Learning Systems (CLS) theory, intelligent agents need episodic and semantic memories [3]. Episodic memory corresponds to an individual's sensory and emotional experience, whereas semantic memory is one's knowledge about the world. For example, the memory of the particular dog you saw yesterday is considered episodic, whereas the knowledge

you have about how dogs in general look like would be semantic. In mammals, the hippocampus and neocortex are respectively thought to be mainly responsible for episodic and semantic memories [3]. In these brain regions, these two kinds of memory differ in their representations and storage. Whereas episodic memory involves sparse, non-overlapping (pattern-separated) representations which are acquired rapidly by the hippocampus, semantic memory involves dense, overlapping representations which are acquired gradually by the neocortex. Following up on our previous example, you can remember yesterday's dog after seeing it only once whereas knowledge about dogs is typically acquired after seeing multiple dogs. In the hippocampus, two dogs you have seen will be represented quite differently despite their similarity to avoid interference, whereas in the neocortex they will be represented similarly to allow generalization.

According to the CLS theory, episodic memories are transformed into semantic ones through hippocampal replay. This division into two learning sytems is assumed to be a solution to the problem of catastrophic forgetting or interference. When an agent learns a task and switches to a new task, replays of the experience of the first task are interleaved with examples of the second task to avoid forgetting the old task.

2) *Consolidation theory:* In contrast, in the standard consolidation theory (SCT), episodic memories are simply transferred to the neocortex through consolidation [4]. It builds upon the observation that patients with hippocampal damage can recall remote memories better than recent ones, a phenomeon called temporally-graded retrograde amnesia [5]–[7]. Thus, contrarily to the CLS theory, the SCT supports the hypothesis that the neocortex stores episodic memory transferred from the hippocampus. That would entail that the neocortex stores both semantic and episodic memories.

However, the theory has been challenged after it has been found that the recall of remote memories is impaired following hippocampal lesions when the memories are truly episodic [8]–[10]. These findings suggest that the hippocampus is needed for recalling detailed episodic memory vividly. Thus, it is likely that in temporally-graded retrograde amnesia, remote memories could be recalled semantically, but not episodically. The multiple trace theory was proposed to account for these findings as an alternative to the SCT [11]. This view aligns
with the CLS theory according to which the neocortex stores only semantic memory, and therefore episodic memories are transformed into semantic ones in the neocortex.

## B. Artificial intelligence

In artificial intelligence, episodic and semantic memories are traditionally implemented respectively by models like Hopfield networks [12] and Boltzmann machines [13]. On the one hand, Hopfield networks are recurrent autoassociative memory (AM) models which store binary data points as attractors, so that they can be recovered by energy minimization when presenting noisy or partial versions. A modern, continuous version, called Modern Hopfield Networks (MHN), extends their memory capacities [14]. On the other hand, Boltzmann machines are energy-based generative models, which are considered as the predecessors of deep learning models, and especially Variational Autoencoders (VAE). The fast learning of individual examples in Hopfield networks and their recurrent structure is reminiscent of the hippocampus (especially the hippocampal subfield CA3), whereas the slow learning of statistical structure in Boltzmann machines are reminiscent of the neocortex.

However, it has been shown that overparameterized autoencoders (AE) are also good AM: an AE trained to generate a particular data point stores it as an attractor [15]. In addition, Salvatori et al. show that predictive coding networks (PCN), which have been proposed as a biologically plausible model of the neocortex [16], outperform AEs and MHN in AM tasks [17].

Therefore, from a neuroscience perspective, it seems that semantic learning systems also have the ability to store individual experiences like episodic memory systems, even though these experiences have dense, overlapped representations which are acquired slowly. This has led researchers to relate them to the hippocampus, despite the differences in representation and storage. Indeed, due to the similarity in architecture and function, autoencoders have been used to model the hippocampus [18], [19]. Following the observation that the hippocampus is at the apex of the neocortical hierarchy [20], Salvatori et al. proposed that the top layer of their PCN could be viewed as the hippocampus, while lower layers would correspond to the neocortex. Therefore, the need for two separate learning systems put forward by the CLS theory seems to be questioned.

In this work, we investigate the supposed episodic nature of the neocortex using a PCN, and asked how it can memorize individual examples through statistical learning. First, we recall the theory of predictive coding. Then, we derive the inference and learning updates for a 3-layer PCN. We also explain how this model can be mapped onto the brain and how replay and recall can be performed. In our experiments, we train the model on a subset of the MNIST dataset until all prediction errors are minimized and examine the learned model. We show that a PCN which is trained on a mini-batch of images is overfitted and can successfully recall them. When the same model is trained on the full dataset, recall performance deteriorates even though the model can successfully

reconstruct the images, and recall seems mostly semantic. This study suggests that a limited number of episodes can be stored in a semantic system, but as this number grows, information about individual episodes is lost to the benefit of statistical knowledge.

## II. PREDICTIVE CODING

Predictive coding is a model and an algorithm for inference and learning which has been mapped onto the cortical microcircuitry. It has its roots in studies of the visual cortex. Olshausen and Field show that learning a sparse code for natural images leads to the emergence of receptive fields which are similar to those found for simple cells in the mammalian primary visual cortex [21]. They assume that an image with pixel intensity $I(x, y)$ at coordinates $x, y$ can be represented by a linear combination of basis functions, $\phi_i(x, y)$

$$I(x, y) = \sum_{i} a_{i} \phi_{i}(x, y). \quad (1)$$

where the coefficients $a_i$ change from one image to the next. The goal of efficient coding is to find a set of basis functions $\phi_i(x, y)$ that forms a complete code (i.e. spans the entire image space) and results in the coefficient values $a_i$ being as statistically independent as possible over an ensemble of natural images. Sparse coding proposes to reduce statistical dependencies by ensuring that the representations are sparse, i.e. distributed among few coefficients.

Considering coefficients $a_i$ as neuron activities and basis functions $\phi_i(x, y)$ as weights, sparse coding could be implemented as a two-layer predictive coding neural network with the image $I(x, y)$ in the input layer. Rao and Ballard make this model hierarchical by adding a third layer predicting the second, similarly to how the second layer predicts the first [22]. Their three-level predictive coding network showed extra-classical receptive field effects, in addition to simple cell receptive fields, suggesting that these effects might emerge in the visual cortex from a hierarchical and predictive strategy for encoding natural images.

Friston goes further by proposing predictive coding as a general theory of cortical computation, given the similarity in architecture throughout the whole neocortex, with its hierarchical structure and reciprocal connections [16]. He adapts Rao and Ballard's model to a variational inference framework, extends it to $n$ layers and shows how it can be implemented in a biologically plausible way in the neocortex. He considers a conditionally independent hierarchical model (also called parametric empirical Bayes) with $n$ layers

$$\forall i \in 1...n, p_{\theta_i}(\nu_i | \nu_{i+1}) = \mathcal{N}(\nu_i : G_i(\nu_{i+1}, \theta_i), \Sigma_i(\lambda_i))$$

where $\nu_1$ is the input variable, equal to the input $u$, and $\nu_2, ..., \nu_n$ are the latent variables. This model is a generative model, where each variable $\nu_i$ is caused by the variable in the level above $\nu_{i+1}$, potentially reflecting the hierarchical causal structure of the world. $\theta_i$ and $\lambda_i$ parameterize the means and covariances of the likelihood at each level $i$, through functions
$G_i$ and $\Sigma_i$. Recognition is assumed to be deterministic, such that

$$q_{\phi}(\nu_2, \dots, \nu_n | u) = \prod_{i=2}^{n} \delta(\nu_i - \phi_i)$$

where $\phi = (\phi_2, \dots, \phi_n)$ is an estimate of the causes of the input $u$.

In a variational inference framework, inference of the latent variables and learning of the parameters correspond to the minimization of a lower bound to the negative likelihood, called variational free energy. The variational free energy for input $u$ is

$$\begin{aligned}
\ell(u) &= - \langle \log p(u, \nu_2, \dots, \nu_n) \rangle_q \\
&= - \log p(u, \phi_2, \dots, \phi_n) \\
&= - \log p(u | \phi_2) + \log p(\phi_2 | \phi_3) + \dots \\
&\qquad + \log p(\phi_{n-1} | \phi_n) + \log p(\phi_n) \\
&= \frac{1}{2} \sum_{i=1}^{n-1} [\xi_i^T \xi_i + \log |\Sigma_i|] + \log p(\phi_n) + \text{constant}
\end{aligned}$$

where $\xi_i = \Sigma_i^{-\frac{1}{2}}(\phi_i - G_i(\phi_{i+1}, \theta_i))$ and the total variational free energy is

$$F = \langle \ell(u) \rangle_u.$$

Implemented in a hierarchical predictive coding network, each layer $i$ consists of two types of neurons: units $\phi_i$ and error units $\xi_i$. Neurons with activity $\phi_i$ are predicted based on neurons in the layer above $i+1$ through backward connections $\theta_i$. Error neurons $\xi_i$ receive this prediction $G_i(\phi_{i+1}, \theta_i)$ from units in the level above and connections from the units $\phi_i$ being predicted to compute the prediction error. Choosing functions $\Sigma_i(\lambda_i) = (1+\lambda_i)^2$ for the covariances, the prediction error becomes

$$\begin{aligned}
\xi_i &= (1 + \lambda_i)^{-1}(\phi_i - G_i(\phi_{i+1}, \theta_i)) \\
&= \phi_i - G_i(\phi_{i+1}, \theta_i) - \lambda_i \xi_i.
\end{aligned}$$

and it can be seen that $\lambda_i$ are the strengths of lateral connections among error units, which could serve to decorrelate them.

The activities of the hidden neurons $\phi$ and connection strengths $\theta$ and $\lambda$ are obtained by performing gradient descent on the variational free energy using a form of the EM algorithm.

1) E-step:

$$\dot{\phi}_{i+1} = - \frac{\delta l(u)}{\delta \phi_{i+1}} = - \frac{\delta \xi_i^T}{\delta \phi_{i+1}} \xi_i - \frac{\delta \xi_{i+1}^T}{\delta \phi_{i+1}} \xi_{i+1}$$

2) M-step:

$$\dot{\theta}_i = - \frac{\delta F}{\delta \theta_i} = - \langle \frac{\delta \xi_i^T}{\delta \theta_i} \xi_i \rangle_u \quad (2)$$

$$\dot{\lambda}_i = - \frac{\delta F}{\delta \lambda_i} = - \langle \frac{\delta \xi_i^T}{\delta \lambda_i} \xi_i \rangle_u - (1 + \lambda_i)^{-1} \quad (3)$$

We refer to the E-step as inference and the M-step as learning. In the E-step, the estimators $\phi_{i+1}$ are updated based on

prediction errors $\xi_i$ carried forward from the layer below, along with lateral interactions with the error units $\xi_{i+1}$. It can be seen that all calculations are local, and weight updates can be seen as Hebbian learning. More details of the proposed neuronal implementation can be found in Friston [16]. In later publications, Friston proposed that precision $\Sigma_i^{-1}$ is predicted by higher layers and mediates attention [23]. Indeed, if precision $\Sigma_i^{-1}$ is low, prediction error $\xi_i = \Sigma_i^{-\frac{1}{2}}(\phi_i - G_i(\phi_{i+1}, \theta_i))$ is low, and will not influence the update of neuron activities and weights.

### III. METHODS

We propose a predictive coding model to investigate how the learning of representations supports memory using the MNIST dataset.

#### A. Model

Our model is a PCN with the 3-level hierarchical model

$$\begin{aligned}
p_{\theta_1}(u | \nu_2) &= \mathcal{N}(u : \theta_1 \nu_2, I) \\
p_{\theta_2}(\nu_2 | \nu_3) &= \mathcal{N}(\nu_2 : f(\theta_2 \nu_3), I) \\
p(\nu_3) &= \mathcal{N}(\nu_3 : 0, I).
\end{aligned}$$

where we use an identity covariance matrix for all layers and an identity activation function for the first layer following equation 1. The variational free energy per input $u$ is

$$l(u) = \frac{1}{2} \xi_1^T \xi_1 + \frac{1}{2} \xi_2^T \xi_2 + \frac{1}{2} \xi_3^T \xi_3 + \text{constant}$$

where

$$\begin{aligned}
\xi_1 &= u - \theta_1 \phi_2 \\
\xi_2 &= \phi_2 - f(\theta_2 \phi_3) \\
\xi_3 &= \phi_3.
\end{aligned}$$

The inference and learning update rules can be derived by calculating the gradient of the variational free energy with respect to the estimators $\phi$ and parameters $\theta, \lambda$ respectively. With an identity function, the calculation is easy for the first layer since

$$\begin{aligned}
\nabla_{\phi_2} l &= -\theta_1^T \xi_1 + \xi_2 \\
\nabla_{\theta_1} l &= -\xi_1 \phi_2^T.
\end{aligned}$$

For the second layer, we define $l_2 = \frac{1}{2}\xi_2^T\xi_2$ and $l_3 = \frac{1}{2}\xi_3^T\xi_3$, so that we can express $\nabla_{\phi_3}l = \nabla_{\phi_3}l_2 + \nabla_{\phi_3}l_3$ where

$$\begin{aligned}
\nabla_{\phi_3} l_3 &= \xi_3 \\
\nabla_{\phi_3} l_2 &= \frac{\partial}{\partial \phi_3} [\phi_2 - f(\theta_2 \phi_3)] \xi_2 \\
&= -\frac{\partial}{\partial \phi_3} [f(\theta_2 \phi_3)] \xi_2 \\
&= -\theta_2^T \text{diag} [f'(\theta_2 \phi_3)] \xi_2
\end{aligned}$$

resulting in

$$\nabla_{\phi_3} l = -\theta_2^T \text{diag} [f'(\theta_2 \phi_3)] \xi_2 + \xi_3.$$
<figure><img src="image_4.png" /><figcaption>Fig. 1. PCN with 3 layers. Each node corresponds to a layer corresponding to multiple neurons. Circle nodes correspond to units $\phi_i$ and triangle nodes correspond to error units $\xi_i$. Each connection between two layers corresponds to a fully connected network, with excitatory connections in the upward direction and inhibitory connections in the downward direction. We consider the upmost prediction to be 0, to have a standard normal distribution as prior.</figcaption></figure>

The gradient with respect to $\theta_2$ can be obtained similarly to the backpropagation update rule using the chain rule, such that

$$ \nabla_{\theta_2} l = -[\xi_2 \odot f'(\theta_2 \phi_3)] \phi_3^T $$

where $\odot$ is the element-wise product.

Therefore, the inference and learning rules are

$$ \phi_2 \leftarrow \phi_2 + \alpha(\theta_1^T \xi_1 - \xi_2) \quad (4) $$

$$ \phi_3 \leftarrow \phi_3 + \alpha(\theta_2^T \text{diag} [f'(\theta_2 \nu_3)] \xi_2 - \xi_3) \quad (5) $$

$$ \theta_1 \leftarrow \theta_1 + \beta < \xi_1 \phi_2^T >_u \quad (6) $$

$$ \theta_2 \leftarrow \theta_2 + \beta < [\xi_2 \odot f'(\theta_2 \phi_3)] \phi_3^T >_u \quad (7) $$

where $\alpha$ and $\beta$ are respectively the inference and learning rates. Figure 1 shows how the model can be seen as a neural network.

## B. Replay and recall in PCN

Mapping our model to the visual pathway of the brain, the three layers would correspond respectively to the lateral geniculate nucleus (LGN), the visual cortex (VC) and the entorhinal cortex (EC). LGN is a structure in the thalamus which connects the retina to the visual cortex and EC is the main interface between the hippocampus and the neocortex. The input layer is mapped to the LGN and not the retina, because there is no backward connection from the LGN to the retina. When the LGN is presented with an image, the VC and EC converge to hierarchical representations of the image following the inference rules in equations 4 and 5. It has been shown that low-level representations in the early

visual cortex encode fine details of images, whereas high-level representations in associative areas, such as inferotemporal (IT) cortex, encode more global information about objects [22]. In our model, representations $\nu_2$ in VC are predictive of the image $\nu_1$ in LGN, and the prediction $\theta_1 \nu_2$ corresponding to the image is called a reconstruction.

Replay of episodic memories is triggered either by cues (during wakeful rest) or noise (during sleep) arriving at EC. Patterns in EC undergo pattern completion in the hippocampus and the recalled pattern in EC is then reinstantiated in the cortex, which is VC in our model. According to Barron [20], reinstated representations are 'protected' from ascending input or prediction errors from lower hierarchical levels. This could be done by setting the precision of the input layer $\Sigma_1^{-1}$ (or equivalently the prediction errors $\xi_1$) to 0. This will have the effect of cancelling the influence of the input on the above layer VC, which will only depend on EC. Therefore, replay of an input $u$ can be implemented with the following steps.

1) Infer $\nu_3$ using the inference rules from equations 4 and 5 until convergence.

2) Set $\nu_3$ to the value obtained in step 1 and $\xi_1$ to 0.

3) Infer $\nu_2$ using equation 4 until convergence.

4) (optional) Update $\theta_2$ using equation 7.

The prediction $\theta_1 \nu_2$ then corresponds to the replayed input. The last step is not necessary here since we do not consider the consolidation of episodic memories.

Recall of episodic memories consists in the pattern completion of a corrupted (noisy or partial) version. We will consider the more challenging recall (or AM) task when presenting a partial version. Following Salvatori et al. [17], we initialize the input layer $\nu_1$ to the corrupted input and let the corrupted part converge with the update rule

$$ \phi_1 \leftarrow \phi_1 - \alpha \xi_1 $$

obtained using $\nabla_{\phi_1} l = \xi_1$ where $\phi_1$ is the estimate of $\nu_1$. The other two layers $\nu_2, \nu_3$ are similarly inferred following equations 4 and 5 until convergence. The converged value for $\phi_1$ would correspond to the recalled input.

## C. Experiments

We implement the above PC model using the inference and learning rules in equations 4 to 7. We use Pytorch to train the model on mini-batches using GPU, but without using the library's automatic differentiation and optimizers. In the inference phase, hidden neurons are initialized randomly and then updated for T iterations according to the above inference rules. The final values are used to perform a single update on the weights $\theta$. We also implement Incremental PC (iPC), a more stable modification of PC where the weights are updated alongside the latent variables at every time step [24].

Two experiments are carried out on a subset of the MNIST dataset with only digits 4 and 7. The dataset is split in a training set of size 10097 and validation and test sets, both of size 2010. Training and validation sets are used during the training, whereas the test set is used after training. In experiment 1, the PC model is trained on a single mini-batch
<table><caption>TABLE I
HYPERPARAMETER VALUES FOR THE TWO EXPERIMENTS</caption><thead><tr><th>Model</th><th>PC</th><th>iPC</th></tr></thead><tbody><tr><td>Dimensions</td><td colspan="2">[784, 35, 2]</td></tr><tr><td>Activation function f</td><td colspan="2">tanh</td></tr><tr><td>Batch size</td><td colspan="2">64</td></tr><tr><td>Number of iterations T</td><td colspan="2">50</td></tr><tr><td>Inference rate α</td><td colspan="2">0.01</td></tr><tr><td>Inference optimizer</td><td colspan="2">SGD</td></tr><tr><td>Learning rate β</td><td>10<sup>-4</sup></td><td>10<sup>-5</sup></td></tr><tr><td>Learning optimizer</td><td colspan="2">Adam</td></tr></tbody></table>

of size 64 from the initial training set. In experiment 2, the iPC model is trained on all images from the training set. In both experiments, the model is trained for multiple epochs, until convergence of all prediction errors. Since convergence on the whole dataset in experiment 2 requires many weight updates, iPC is required for stability. Hyperparameters are chosen based on empirical trials informed by the literature [25]–[27] and summarized in Table I.

Except for the content of the training dataset, the two experiments are performed in the same way. During training, we keep track of the average sum in prediction errors in each layer for the training and validation sets. After training, we examine the learned models by looking at the weights $\theta_1$ and the reconstructions for a mini-batch from the training and test sets. The same training mini-batch is replayed and the replayed inputs are visualized. Then, we plot the representations for the test set in each layer in 2 dimensions using t-SNE. Lastly, we perform an AM task by considering 10 images from the training set as episodic memories and recalling the bottom half of each image when presenting only the top half. Reconstruction, replay and recall are implemented following the methods in the previous subsection.

## IV. RESULTS

### A. Experiment 1: Training on a single minibatch

The prediction errors in the input layer converge to higher values for the validation set compared to the training set (not shown), suggesting that the model is overfitted to the minibatch. Indeed, as shown in Figure 2, images from the test set are not reconstructed as well by the model compared to images from the training set. This is reflected in the weights $\theta_1$ of the model, which are shown in Figure 3 as 35 images of size 28 × 28 that look like noisy combinations of examples of 4 and 7.

The hierarchical representations of the test data are visualized in Figure 4 using t-SNE to reduce the dimensionality of the first two levels to 2. The representations in the first two levels are well separated according to the class, suggesting that the images $u$ (bottom plot) and their causes $v_2$ lie in a 2-dimensional manifold corresponding to the two classes. Indeed, the estimates of the two-dimensional causes of $v_2$, $v_3$, are also disentangled according to the class. However, they overlap at the boundary between the two classes. This means that some digits might not be easily classified by our model.

<figure><img src="image_5.png" /><figcaption>(a) Training set</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Test set</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 3. Experiment 1: Visualization of weights $\theta_1$ as 35 images of size 28 × 28.</figcaption></figure>

As a result, replayed inputs corresponding to these high-level representations might be ambiguous. This is what we see in Figure 5: some digits are misremembered when replayed, and often, the replayed inputs look like a combination of 4 and 7. The other images are replayed as the right class but the details of the original images are lost. This is expected because replayed images are obtained from high-level representations in EC, which are supposed to encode global information about the classes.

Finally, the episodic memories from the training set were successfully recalled, as shown in Figure 6.
<figure><img src="image_8.png" /><figcaption>Fig. 4. Experiment 1: Visualization of the hierarchical representations of the test data. Each image from the test set is represented as one data point in each of the three subplots, colored according to the class. Blue and orange represent digits 4 and 7 respectively. The first two levels (at the bottom of the figure), corresponding to *u* and *φ*₂, are visualized in two dimensions using t-SNE.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 5. Experiment 1: Replayed inputs (right) for a mini-batch of training images (left).</figcaption></figure>

## B. Experiment 2: Training on the full dataset

The hierarchical representations of the test data and the replayed images are similar to those obtained in the previous experiment. However, there are major differences in the prediction errors and the recall.

On the one hand, the training curves of the prediction errors for the training and validation sets overlap (not shown), suggesting a good fit. Low prediction errors in the input layer $\xi_1$ at the end of training for the training and validation sets

<figure><img src="image_10.png" /><figcaption>Fig. 6. Experiment 1: AM task. Ten images from the training set (right column) are taken as episodic memories and the bottom half of the images is recalled (middle column) when presenting only the top half of the images (left column).</figcaption></figure>

suggest that the model is able to reconstruct well images of handwritten digits 4 and 7, whether they were seen during training or not. Indeed, we show in Figure 7 that images from the test set are reconstructed as well as those from the training set. This is due to weights $\theta_1$ being smoother than in the previous experiment, as shown in Figure 8, constituting a complete code for images of digits 4 and 7 from the MNIST dataset.

On the other hand, Figure 9 shows that the episodic memories are not recalled as well as in the previous experiment. Even though the images are recalled in a realistic way, the details of the original images seem to be lost and in some images, a bar appears in the middle of the digit, even though it is absent in the original images. This reflects semantic learning because many images of 4 and 7 in the training set have a bar in the middle.

## V. RELATED WORK

Millidge implemented predictive coding in a similar way, using the same inference and learning rules, and examined the reconstructions and latent space of MNIST images [25].
<figure><img src="image_11.png" /><figcaption>(a) Train set</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Test set</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 8. Experiment 2: Visualization of weights $\theta_1$ as 35 images of size 28 × 28.</figcaption></figure>

However, these rules were not derived in the paper and the results were only shown for a two-layer PCN. The latest implementations used automatic differentiation with Pytorch to perform inference and learning [27], [28]. However, they are not trained until convergence of all prediction errors, but until performance on specific tasks has converged, because their goal is to apply predictive coding to AI tasks, and not to study the learning of representations in the neocortex. Salvatori et al. trained two two-layer PCNs with 1024 and 2048 hidden neurons on 50 images from the Tiny ImageNet dataset and successfully recalled all of them when presenting only 1/8 of the images, similarly to our first experiment [28]. Pinchetti et al. studied the memory capacity of four-

<figure><img src="image_14.png" /><figcaption>Fig. 9. Experiment 2: AM task. Ten images from the training set (right column) are taken as episodic memories and the bottom half of the images is recalled (middle column) when presenting only the top half of the images (left column).</figcaption></figure>

layer PCNs of increasing widths on a subset of the Tiny ImageNet dataset. They reported the MSE between the recalled and original images with model sizes [12288, *d*, *d*, 512] where *d* ∈ {512, 1024, 2048} and training datasets of sizes 50, 100 and 250. They also examine the reconstructions obtained by such a model when trained on an entire dataset of images, like in our second experiment. They compare the reconstruction errors, i.e. the MSE between the reconstructed and original images, of PC, iPC and VAE on different image datasets: MNIST, FashionMNIST, CIFAR-10 and CELEB-A.

## VI. DISCUSSION

In this paper, we proposed a predictive coding model of the neocortex to study the learning of representations and the
formation of memory. We showed that this model can be
implemented as a neural network with local learning rules,
and that it can be mapped onto the neocortical hierarchy.
Using a subset of the MNIST dataset with only digits 4 and
7, we showed that the model can infer the two-dimensional
structure of the dataset thanks to its hierarchical structure, as
well as replay images considered as episodic memories. More
importantly, we showed that our model can memorize a small
training set by overfitting to it, but as the training set becomes
larger, this ability is lost to the benefit of more general
knowledge. As the neocortex is exposed to a vast quantity of
sensory inputs, it likely cannot memorize individual episodes.
Therefore, this work supports the CLS theory according to
which the neocortex only supports semantic memory and
episodic memory is supported by the hippocampus. This is
likely due to the nature of representations learned by these
two systems. On the one hand, dense, overlapping represen-
tations are learned slowly by the neocortex, and do not allow
the recall of the specifics of individual examples. On the
other hand, sparse, pattern-separated representations, which are
learned rapidly by the hippocampus, support recall, but do
not allow the extraction of statistical structure. Overall, this
work contributes to a better understanding of learning in the
hippocampus and the neocortex as well as their interactions
from a neuroscience perspective, and also sheds light on
the nature of generative models in AI. Future work could
further examine the relationship between training set size and
recall performance, with increasing model width and dataset
complexity.

REFERENCES

[1] E. Spens and N. Burgess, "A generative model of memory construction and consolidation," *Nature Human Behaviour*, pp. 1–18, 2024.

[2] S. Káli and P. Dayan, “Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions,” *Nature Neuroscience*, vol. 7, no. 3, pp. 286–294, 2004.

[3] D. Kumaran, D. Hassabis, and J. L. McClelland, “What learning systems do intelligent agents need? complementary learning systems theory updated,” *Trends in cognitive sciences*, vol. 20, no. 7, pp. 512–534, 2016.

[4] L. R. Squire and P. Alvarez, “Retrograde amnesia and memory consolidation: a neurobiological perspective,” *Current Opinion in Neurobiology*, vol. 5, no. 2, pp. 169–177, 1995. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0959438895800239

[5] G. Winocur, “Anterograde and retrograde amnesia in rats with dorsal hippocampal or dorsomedial thalamic lesions,” *Behavioural Brain Research*, vol. 38, no. 2, pp. 145–154, 1990. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0166432890900124

[6] S. M. Zola-Morgan and L. R. Squire, “The primate hippocampal formation: Evidence for a time-limited role in memory storage,” *Science*, vol. 250, no. 4978, pp. 288–290, 1990. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.2218534

[7] N. L. Rempel-Clower, S. M. Zola, L. R. Squire, and D. G. Amaral, “Three cases of enduring memory impairment after bilateral damage limited to the hippocampal formation,” *Journal of Neuroscience*, vol. 16, no. 16, pp. 5233–5255, 1996. [Online]. Available: https://www.jneurosci.org/content/16/16/5233

[8] I. V. Viskontas, M. P. McAndrews, and M. Moscovitch, “Memory for famous people in patients with unilateral temporal lobe epilepsy and excisions,” *Neuropsychology*, vol. 16, no. 4, pp. 472–480, 2002.

[9] R. S. Rosenbaum, S. Priselac, S. Köhler, S. E. Black, F. Gao, L. Nadel, and M. Moscovitch, “Remote spatial memory in an amnesic person with extensive bilateral hippocampal lesions,” *Nat. Neurosci.*, vol. 3, no. 10, pp. 1044–1048, Oct. 2000.

[10] L. Cipolotti, T. Shallice, D. Chan, N. Fox, R. Scahill, G. Harrison, J. Stevens, and P. Rudge, "Long-term retrograde amnesia... the crucial role of the hippocampus," *Neuropsychologia*, vol. 39, no. 2, pp. 151-172, 2001. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0028393200001032

[11] L. Nadel, A. Samsonovich, L. Ryan, and M. Moscovitch, "Multiple trace theory of human memory: Computational, neuroimaging, and neuropsychological results," *Hippocampus*, vol. 10, no. 4, pp. 352-368, 2000. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/1098-1063%282000%2910%3A%3C352%3A%3AAID-HP02%3E3.0.CO%3B2-D

[12] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities." *Proceedings of the National Academy of Sciences*, vol. 79, no. 8, pp. 2554-2558, 1982. [Online]. Available: https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554

[13] G. E. Hinton and T. J. Sejnowski, "Learning and relearning in boltzmann machines," in *Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations*. The MIT Press, 07 1986. [Online]. Available: https://doi.org/10.7551/mitpress/5236.003.0011

[14] D. Krotov, "A new frontier for hopfield networks," *Nature Reviews Physics*, vol. 5, no. 7, p. 366-367, May 2023. [Online]. Available: http://dx.doi.org/10.1038/s42254-023-00595-y

[15] A. Radhakrishnan, M. Belkin, and C. Uhler, "Overparam-
eterized neural networks can implement associative mem-
ory," CoRR, vol. abs/1909.12362, 2019. [Online]. Available:
http://arxiv.org/abs/1909.12362

[16] K. Friston, "Learning and inference in the brain," *Neural Networks*, vol. 16, no. 9, pp. 1325-1352, 2003.

[17] T. Salvatori, Y. Song, Y. Hong, S. Frieder, L. Sha, Z. Xu, R. Bogacz, and T. Lukasiewicz, "Associative memories via predictive coding," 2021. [Online]. Available: https://arxiv.org/abs/2109.08063

[18] M. K. Benna and S. Fusi, “Place cells may simply be memory cells: Memory compression leads to spatial tuning and history dependence,” *Proceedings of the National Academy of Sciences*, vol. 118, no. 51, p. e2018422118, 2021.

[19] D. Santos-Pata, A. F. Emil, I. G. Raikov, C. Rennó-Costa, A. Mura, I. Soltesz, and P. F. Verschure, “Entorhinal mismatch: A model of self-supervised learning in the hippocampus,” *Science*, vol. 24, no. 4, 2021.

[20] H. C. Barron, R. Auksztulewicz, and K. Friston, "Prediction and memory: A predictive coding account," *Progress in neurobiology*, vol. 192, p. 101821, 2020.

[21] B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," *Nature*, vol. 381, no. 6583, pp. 607-609, 1996.

[22] R. P. Rao and D. H. Ballard, "Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects," *Nature neuroscience*, vol. 2, no. 1, pp. 79-87, 1999.

[23] H. Feldman and K. Friston, "Attention, uncertainty, and free-energy," *Frontiers in Human Neuroscience*, vol. 4, 2010. [Online]. Available: https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2010.00215

[24] T. Salvatori, Y. Song, Y. Yordanov, B. Millidge, Z. Xu, L. Sha, C. Emde, R. Bogacz, and T. Lukasiewicz, "A stable, fast, and fully automatic learning algorithm for predictive coding networks," 2024. [Online]. Available: https://arxiv.org/abs/2212.00720

[25] B. Millidge, "Implementing predictive processing and active inference: Preliminary steps and results," Mar. 2019. [Online]. Available: http://dx.doi.org/10.31234/osf.io/4hb58

[26] S. Frieder, L. Pinchetti, and T. Lukasiewicz, “Bad minima of predictive coding energy functions,” in *The Second Tiny Papers Track at ICLR 2024*, 2024. [Online]. Available: https://openreview.net/forum?id=5Vc1ACZ8eC

[27] L. Pinchetti, C. Qi, O. Lokshyn, G. Olivers, C. Emde, M. Tang, A. M'Charrak, S. Frieder, B. Menzat, R. Bogacz, T. Lukasiewicz, and T. Salvatori, "Benchmarking predictive coding networks - made simple," 2024. [Online]. Available: https://arxiv.org/abs/2407.01163

[28] T. Salvatori, Y. Song, Y. Hong, L. Sha, S. Frieder, Z. Xu, R. Bogacz, and T. Lukasiewicz, "Associative memories via predictive coding," *Advances in Neural Information Processing Systems*, vol. 34, pp. 3874–3886, 2021.