Associative Memories via Predictive Coding
Authors: Tommaso Salvatori
Yuhang Song
Yujian Hong
Lei Sha
Simon Frieder
Zhenghua Xu
Rafal Bogacz
Thomas Lukasiewicz
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning
Associative Memory
Cognitive Science

Abstract: Associative memories in the brain receive and store patterns of activity registered by the sensory neurons, and are able to retrieve them when necessary. Due to their importance in human intelligence, computational models of associative memories have been developed for several decades now. In this paper, we present a novel neural model for realizing associative memories, which is based on a hierarchical generative network that receives external stimuli via sensory neurons. It is trained using predictive coding, an error-based learning algorithm inspired by information processing in the cortex. To test the model's capabilities, we perform multiple retrieval experiments from both corrupted and incomplete data points. In an extensive comparison, we show that this new model outperforms in retrieval accuracy and robustness popular associative memory models, such as  autoencoders trained via backpropagation, and modern Hopfield networks. In particular, in completing partial data points, our model achieves remarkable results on natural image datasets, such as ImageNet, with a surprisingly high accuracy, even when only a tiny fraction of pixels of the original images is presented. Our model provides a plausible framework to study learning and retrieval of memories in the brain, as it closely mimics the behavior of the hippocampus as a memory index and generative model.

Associative Memories via Predictive Coding
Tommaso Salvatori', Yuhang Song 13,,, Yujian Hong', Lei Sha', Simon Frieder', Zhenghua Xu², Rafal Bogacz', Thomas Lukasiewicz'
'Department Computer Science, University of Oxford, UK
2 State Key Laboratory of Reliability and Intelligence of Electrical Equipment,
Hebei University of Technology. Tianjin, China
MRC Brain Network Dynamics Unit, University of Oxford, UK
(tommaso.salvv salvatori, yuhang.song yujian. hong. lei.sha, frieder.simon, thomas. thomas.lukasiewi.. Iukasiewicz @ ess.ox.ac.uk, zhenghua.xun @ hebut. edu.en, rafal. offal.bogaczee @ ndcn.ox.ac.u.
Abstract
Associative memories in the brain receive and store patterns of activity registered by the sensory neurons, and are able to retrieve them when necessary. Due to their importance in human intelligence, computational models of associative memories have been developed for several decades now. In this paper, we present a novel neural model for realizing associative memories, which based on a hierarchical generative network that receives external stimuli via sensory neurons. It is trained using predictive coding, an error-based learning algorithm inspired by information processing in the cortex. To test the model's 's capabilities, we perform multiple retrieval experiments from both corrupted and incomplete data points. an exten- sive comparison, we show that this new model outperforms in retrieval accuracy and robustness popular associative memory models, such as autoencoders trained via backpropagation, and modern Hopfield networks. In particular, in completing partial data points, our model achieves remarkable results on natural image datasets, such as ImageNet, with surprisingly high accuracy, even when only a tiny fraction pixels of the original images is presented. Our model provides a plausible framework to study learning and retrieval of memories in the brain, it closely mimics the behavior of the hippocampus as a memory index and generative model.
1 Introduction
Throughout our lives, we learn a huge number of associations between concepts: the taste of particular food, the meaning of gesture, to stop when we see red light. Every time acquire new information of this kind, it gets stored in our long-term memory, situated in distributed networks of brain areas [[] In particular, visual memories are stored in hierarchical network of visual and associative areas (2]. These regions learn progressively more abstract representations of visual stimuli, both perception and memory as each area memorizes relationships present in their inputs [3]. Accordingly, early visual areas learn common regularities present in the stimuli [4], while at the top of this hierarchy, associative areas (such as hippocampus, entorhinal cortex, and perirhinal cortex) store the relationships between extracted features, which encode an entire stimulus or episode [5) The memory system the brain is able to both recall complex memories [ () and use them to generate predictions to guide behavior (7]. Learning in these associative memories shapes our understanding of the world around us and builds the foundations of human intelligence.
so they participate
Building models that are able to store and retrieve information has been an important direction of research in artificial intelligence. Particularly, such models include (asto)assoc memories (AMs), which allow for the storage of data points and their contents- based retrieval, i.c., for retrieving stored
Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure 1: Generative PCN with two layers. Each line linking neurons denotes a pair of connections (excitatory in the left direction, and inhibitory in the right direction). At f = O, the value nodes of the output layer are fixed to the vector representing the pixels of the figure. Then, during the learning phase, both weight parameters and value nodes are updated.
b
layer
Memory Layer
Sensory Layer
data point s from a corrupted or a partial variant of s. One way to realize AMs is to store data points attractors, that they can be easily recovered via energy minimization process when presenting their corrupted variants S 9
Existing AMS include Hopfield networks S and modern Hopfield networks (MHNs) 10] The latter are one-shot leamers, which are able to store exponentially many memories, and to perfectly retrieve them. However, the retrieval process often fails when dealing with complex data, such as natural images. Recent works have shown that overparametrized autoencoders (AEs) are excellent AMs as well. Particularly, when training an AE to generate a specific point when s itself is presented as an input, it gets stored as attractor
In this work, we present a novel AM model that is based on an energy-based generative approach. This AM model differs from Hopfield networks, it is trained using predictive coding (PC), which a biologically plausible learning algorithm inspired by learning in the visual cortex 4. The idea that PC may naturally be related AMs is inspired by recent works showing that the generative neural architecture that connects the hippocampus to the neocortex is based on an error-driven learning algorithm, which be interpreted with PC framework [6] [2]. From machine learning perspective, predictive coding networks (PCNs) are able to perform both supervised and unsupervised tasks with a high accuracy [4. 13, and are completely equivalent to backpropagation when trained with a specific algorithm 14 We show that the new AM model is not only interesting from neuroscience perspective, but it also outperforms popular AM models when it comes to the storage and retrieval of complex data points. Our results can be briefly summarized as follows:
We define generative PCNs and empirically show that they store training data points as attractors of their dynamics by demonstrating that they can restore original data points from corrupted versions. In an extensive comparison of the new AM model against standard AEs, the new model considerably outperforms AEs (in storage capacity, retrieval accuracy, and robustness) when tested on neural networks of the same size.
The reconstruction of incomplete data points is a challenging task for AMs. Our model naturally solves the task of reconstructing complex and colored images with a surprisingly high accuracy. We also test our model on ImageNet, perfectly reconstructing single pictures even after removing all but 1 /8 of the original image. We then show that, to increase the overall capacity and retrieval robustness of the model, it suffices to add additional layers. We also compare our model against MHNs, showing that it significantly outperforms them in the aforrmennioned tasks.
2 Generative predictive coding networks
We now briefly recall predictive coding networks (PCNs), and we introduce generative PCNs, which are the underlying neural model for the novel AMs introduced in the subsequent section.
Deep neural networks have a multi-layer structure, where each layer is formed by a vector of neurons [7]. While in standard deep learning the goal is to minimize the error on a specific layer, PC defines an error in every layer of the network, minimized by gradient descent on global energy function 4. Particularly, let M be PCN with L- fully connected layers of dimension n. followed by a fully connected layer of dimension d. We call the d-dimensional layer sensory layer (indexed as layer 0), which biologically corresponds to sensory neurons (see Fig. D We call the most internal layer (layer L) memory, which is equipped with an n-dimensional memory vector b. Every layer / contains value nodes <t. and every pair of consecutive layers is connected via weight matrices 0', which represent the synaptic weights between neurons of different layers. The value nodes, the

---

Algorithm 1: Learning to generate & with IL
Require: 26 is fixed
1: for = to T do
2: for each neuron i in each level ( do
3: update o,,e to minimize Es via Eq.33
4: end for
5: end for:
6: update each 8 and b, to minimize E, via Eqs. and 5
weight matrices, and the memory vector are all trainable parameters of the model. The signal passed from layer + to layer called prediction ply computed as follows:
Σラ= t faatt) if0 S <<L
Mie
(I)
if L. L.
where f is a non non-linear activation function. To conclude, the difference between the value of and their predictions j is the error ei,, =i, 14, .- We describe how PCNs are trained. To do this, we explain one iteration of a training algorithm, called inference learning (IL), that is divided into an inference phase and a weight update phase.
Inference: Only the value nodes of the network are updated, while both the weight parameters and the memory vector fixed. Particularly, the value nodes are modified via gradient descent to minimize the global error of the network, expressed by the following energy function Eg:
E = EE,e,))
Assume that train generative PCN training point & Rdd To do this, the value nodes of the sensory layer are fixed to the training point 8, and are never updated. Thus, the error on every neuron of the sensory layer is equal to et, = 8; n.,,. The process of minimizing E, by modifying all s/, leads to the following changes in the value nodes:
(-el,, J'd,, 4. (...))
ir I L
1 0,
where 7 is the integration step, which is constant determining by how much the activity changes in each iteration. The computations in Eqs. ( and (3) are biologically plausible, as they have a neural implementation that can be realized in network with value nodes d and error nodes si, [4. as shown in Fig. [ The inference phase works as follows: starting from a given configuration of the value nodes Ty. inference continuously upates the value nodes according to Eq. B) until has converged. We call the configuration of the value nodes at convergence Fy, where T is the number of steps needed to reach convergence (in practice,
a fixed large number).
Weight Update: When the value nodes of the sensory layer are fixed to an input signal 8, inference may not be sufficient to reduce the total energy zero. Hence, to further decrease the total error, single weight update is performed: both the weight matrices and the memory vector are updated by gradient descent minimize the same objective function E,, and behave according to the following equations, where a is the learning rate. Particularly, the derived update rule is the following:
1 = aa - 00-/EE//000000 100 = a /
Ab.- = -a /0b. -ast.--
(5)
The phases of inference and weight update are iterated until the total energy Eg reaches a minimum. This algorithm learns dataset by using only local computations, which minimize the same energy function. Fig. [ gives a graphical representation of generative PCNs, while the pseudocode is shown in Alg. 1 Detailed derivations of Eqs. (3) and (5) are in the supplementary material (and in [13)
3 Predictive coding for associative memories
So far, we have shown how PCNs can perform generative tasks. We now show how generative PCNs can be used as associative memories, i.e., how the model stores the data points that is trained on, and how these data points can be retrieved when presenting corrupted versions to the network, returning
3

---

Inference
Fm (
E2 (z)
F
(m) (e)
Value Nodes
Figure 2: Left: Representation of the function F, used to retrieve stored images. can be decomposed into three steps: (1) The value nodes of the sensory layer are fixed to the pixels of the corrupted image FM (c). (2) Inference runs for T' operations (until convergence). (3) We set (c)+l(() to the prediction of the sensory layer zig. Note that the weight parameters are never updated during the above steps. We have omitted the error nodes for simplicity. Right: Representation of an AM, where an image $ (photo of a dog) is stored as an attractor of the dynamics. A corrupted image that lies in specific neighborhood of 5 converges when minimizing the total energy via running inference.
the most similar stored data point. Let 3 be training data point, and M be the PCN considered above, already trained until convergence to generate s. Moreover, assume that M makes the total energy converge to zero at iteration T. At this point, the energy function defined on the value nodes has local minimum E in which the value nodes of the sensory layer are equal to the entries of 8. Note that T is actually attractor of the dynamics of E,: when given configuration that is not local minimum, inference will update the value nodes until the total energy reaches a minimum. If this configuration lies in a specific neighborhood of æ, inference will converge to T. So, given dataset, we obtain an AM of the dataset if all the training points are stored in the above way as attractors.
The above can be used to retrieve stored data points 8: given a comrupted version ē E Rd of 8, one can retrieve & as follows. First, we set the value nodes of the sensory layer to the corrupted points, i.c., z = ē for the whole process. Then, we inference until convergence and save the prediction po of the sensory layer. If the original data point was stored as an attractor, we expect the prediction ü.g to be a less corrupted version of it. Let F: Tdd -> Rd be the funetion that sends jit just described, and summarized in Fig. 2 Many iterations of this function allow to retrieve the stored data point. Hence, summarizing the above, training points are stored in the memory vector by and the weight parameters, and what the algorithm does to retrieve them is simply the inference phase of PCNs. Since visual memories are stored in hierarchical networks of brain areas, PC could be a highly plausible algorithm to better understand how memory and prediction work in the brain.
To experimentally show that generative PCNs AMs, we trained 2-layer network with ReLU non-linearity on a subset of 100 images of Tiny ImageNet and CIFAR 10. After training, we presented the model with corrupted variant (by adding Gaussian noise) of the training set. We then used the PCNs to reconstruct the original images from the corrupted ones. The experimental results confirm that the model is able to retrieve the original image, given corrupted one. The obtained reconstructions for the Tiny Imagenet dataset (the most complex one, as each data point consists of 3 x 64 x 64 pixels) are shown in Fig. 4 We now provide more comprehensive analysis, which studies the capacity of generative PCNs when changing the number of data points and parameters.
Experiments: We trained 2-layer PCNs with ReLU non-linearit and hidden dimension n E (512, 1024, 2048} on subsets of the aforementioned datasets of cardinality N = {100, 250, 500, 1000). Every model is trained until convergence, and all the images are retrieved described in Section To provide a numerical evaluation, an image is considered recovered when the mean squared error between the original image and the recovered image is less than 0.005.
To compare our results against a standard baseline, we also trained 3-layer autoencoders (AEs) with the same hidden dimension on the same task, and compared the results. Note that the number of parameters of 2-layer PCN smaller than the one of a 3-layer AE with the same hidden dimension. This follows, as the additional layer (input layer, not needed in generative PCNs) almost doubles the number of parameters in some cases. Further details about the experiments and used hyperparameters given in the supplementary material.

---

CIFAR10
Tuny Imagenet
512
1024
2048
512
1024
2048
Model
Nodel
Careety Rebjeced PSa
Comnity Rebjeved
200 150 1020 Data Polnss
220 720
250 730 1003
223 738 1200 Data Foints
250 130 1000 Data Points
302 233 750 1050
Data Pb nts
Data Poiins
Data Foints
Figure 3: Percentage of correctly retrieved images by 2-layer generative PCNs (PC) and 3-layer autoencoders (AE) with hidden-layer dimensions of 512, 1024, and 2048, when presented with corrupted image with Gaussian noise variance 0.2 of CIFARIO and of Tiny ImageNet.
Figure 4: Reconstructions x 64 Tiny ImageNet images with noise levels 0.2, 0.7, and 0.9, respectively (original, noisy, and reconstructed image in first, second, and third row, respectively).
Results: The analysis shows that our model is able store and retrieve data points even when the network is not overparametrized (see Fig. 3). AEs trained with BP did not perform well: AEs with less than 1024 hidden neurons always failed restore even single data point on the Tiny Imagenet dataset, and very few on other ones. The performance of AEs with 2048 hidden neurons were always worse than PCNs with 512 hidden neurons. This shows that overparametrization is essential for AEs to perform AM tasks, and that our proposed method offers a much more network-effictee alternative.
In terms of capacity, l-yerrr PCNs with 512 hidden neurons correctly store datasets of 250 images of both CIFARIO and Tiny ImageNet, and networks with 2048 hidden neurons a always store and retrieve all the presented datasets. As typical for AMs, small models trained large datasets fail to store data points, as the space of parameters is not large enough to store each data point as an independent attractor. Our model is different: PCNs with 512 hidden units are able to store almost 500 Tiny ImageNet images when trained on datasets of that size, but fail to store more than 200 when trained on larger ones.
Retrieval from partial data points
So far, we have shown how the proposed generative model can be used to retrieve stored data points when presented with corrupted variants. We now tackle the different task retrieving data points when presented with partial ones. Let 8 be the stored data point, and assume that a fraction of pixels of 5 are accessible, and the goal is to retrieve the remaining ones. Let be vector of the same dimension of 8, where fraction of pixels are equal to the ones of the stored data point, and assume that the position of the pixels that are equal to the ones of is known. We now show how to retrieve the complete data point by using the same network M, trained as already shown in Section 3
Let M be PCN trained generate Then, given the partial version of the original data point s', is possible to retrieve the full data point using M as follows: first, we only fix the value nodes of the sensory layer øς to the entries of the partial data point s' that know are equal to the ones of the stored data point, leaving the rest free to be updated. Then, we run inference on the whole network until convergence. At this point, we expect the value nodes x... to have converged to the entries of the original data point, stored an attractor. A graphical representation of the above mechanism is described Fig. [5] To show the capabilities of this network, we have performed multiple experiments on the Tiny ImageNet and ImageNet datasets, and compared against existing models in the literature.
5

---

Figure 5: Algorithm used to retrieve stored data point from a fraction of it: (1) The value nodes of the sensory layer that correspond to the available entries of & are fixed to the respective values. (2) Inference runs for T operations (until convergence).
Inference
Nituil
(3) We set our prediction to be equal to the value nodes of the sensory layer zy. The weight parameters are never updated during the above steps. The eπror nodes are omitted for simplicity.
Algorithm 2: Retrieving 5 given a non-corrupted fraction
Require: if s a correct entry of the original memory, then x²,, is fixed to s.. I: for t = 1 to T do
2: for each neuron i each level do
3: update xi, to minimize E,
4: end for
5: end for;
6:
return Z9.
We now start by providing visual evidence on the effectiveness of this method. Note that the geometry of the mask does not influence the final performance, as our model simply memorizes single pixels.
Experiments: We trained two networks with hidden dimensions of 1024 and 2048, to generate 50 images of the first class of Tiny ImageNet (corresponding to goldfishes), and a network of 8192 hidden neurons to reconstruct 25 ImageNet images. Then, we used inference as explained to retrieve the original images. We considered an image to be correctly reconstructed when the error between the original and the retrieved image was smaller than 0.001. Furthermore, we plotted the partial images together with their reconstructions, for visual check. Note that we used the thresholds that provided the fairest comparison: the denoising experiments fail to have perfect retrieval, despite the fact that most of the images look visually good. Hence, we determined the threshold to be equal to 0.005. Then, with the same threshold for the retrieval of partial images, our method always successfully retrieved all the images, and so opted for a smaller threshold, which more informative.
Results: On Tiny ImageNet, a generative PCN with 1024 hidden neurons managed reconstruct all the images when presented with 1/8 of the original image, and more than half for the smallest fraction considered, 1/16. The network with 2048 neurons also failed to reconstruct all the images when presented with 1/16 of the original image. However, even when presented with portion as small as 1/16 of the original image, the reconstruction was clear, although not perfect and hence now aove our threshold. This result is shown in Fig. (left).
Surprisingly, generative PCNs trained on ImageNet correctly stored all the presented training images, and correctly reconstructed them with no visible error. This shows that PCNs can be used to store high-dimensions and high-quality images in practical setups, which can be retrieved using only low-dimensional key, formed by fraction of the original image. Particularly, Fig. [6] (right) shows the perfect reconstruction obtained on ImageNet when the network presented with only 1/8 the original image. Further experiments on ImageNet are shown in the supplementary material, where we present multiple high-quality reconstructions.
5 More training data points and/or deeper PCNs
In the above experiments, a shallow architecture is able to store 100 natural images from Tiny ImageNet, and to reconstruct them with no visible error when provided with 1/8 the original images. We now study how this changes when either the cardinality of the dataset is increased, or the number of provided pixels during reconstruction is updated. study this, we have trained two models with (1024, 2048) hidden units on subsets of Tiny Image of cardinality N = (50, 100, 250, 500), and reconstructed images with only (1/2, 1/8, 1/16) of the original images. According to visualization experiments, have noted that image has visible reconstruction noise when

---

Figure 6: Partial images and their reconstructions (from top bottom: partial image, reconstructed image, original image, and reconstruction error (difference between original and reconstruction)) Left: 64 x 64 Tiny ImageNet images presented with 1/4, 1/8, and 1/16 of the original image, respectively, using a network 1024 hidden neurons. Right: 224 x 224 ImageNet images presented with 1/8 of the original image, using network with 8196 hidden neurons.
1024
4096
0.8
138
0.8
Correctly (ettrnnee
0.-
0.8
A/26
9.0
Corvetty
92
Cara Potnits
Data Paints
Figure 7: Percentage of perfectly retrieved images given different fractions of the original images for PCNs with 512, 1024, 2048, and 4096 hidden neurons trained on datasets of different sizes. Note that here count retrieved only images with non-visible errors. Most of the images that not considered retrieved in this plot, were correctly retrieved, but only slightly noisy.
the error between the original image and the reconstruction is less than 0.001. Hence, consider an image to be correctly retrieved the reconstruction error below this threshold.
Results: Every network with 50 stored memories was able to perfectly reconstruct the original image when provided with 1/4 of and about half when provided with only 1/8. However, this changes when increasing the training reconstruct an image when provided with 1/16 of it, besides a few cases. In terms of capacity, the more images we train on, the harder the reconstruction becomes: retrieved when training on N = 500 images when provided with a fraction smaller than 1/2 of the original image, regardless of the width of the network. A summary of these results is given in Fig. 7
network trained more than 50 images was able to correctly
training image
correctly
These experiments show the limits of shallow generative PCNs on both reconstruction capabilities and capacity. As expected and common in standard AM experiments, increasing the number of training points and reducing the number of available information for reconstruction hurts the performance. We now show how increasing the number of layers solves this capacity problem.
Deep generative PCNs. We now test how increasing the depth of generative PCNs increases their capacity. We then compare the results against overparametrized AEs. Particularly, we trained generative PCNs with images of Tiny ImageNet. Furthermore,
{3, 10} on N 500 reconstructed the stored images using fraction of p {1/2, 1 1/4] of the original images. To further compare against sfate-of-the-ar AEs, we trained equivalent AEs, which is the best best-performing one according to [IT]. Also, all the hyperparameters used for training are the ones reported by the authors. To make the comparison completely fair, we also assume that the correct pixels of the incomplete images are known when testing the AE. Particularly, we fixed these pixels at every iteration of the reconstruction process of the AFs. As above, consider image to be correctly retrieved if the reconstruction error is less than 0.001.
{1024, 2048) hidden neurons, depth
7

---

1.0
Correctly Ratrieved (%)
Cornecly Ratrieved (%)
PCN
Correctly Retrieved (%)
Correcly Retrieved
02-
Hidden Layers
Hidden Lavers
Hidden Lavers
Hidden Lavers
Figure 8: Percentage correctly retrieved images as function of the number of hidden layers, for both AEs and generative PCNs, on a dataset of 500 images. We used networks with (1024, 2048) hidden neurons, and the images are taken from Tiny ImageNet.
N
0
0
3
2
/ 2 3
3 3
10
/ a 3 # 6 8
Original
/ 3 # c 8
2
Generative PCN
Classica Hopfield Network
Generntive
Modern Hopfield Netwock
Figure 9: Left: Recovery of 2, 3, 5, and 10 MNIST images using generative PCN and classical original image as a key. shown, Hopfield 2, but otherwise always failed to retrieve
Hopfield networks [8] when presented with 1/2
networks only recovered the original images when N 2, them when trained on more than two images. Right: Reconstruction of 100 Tiny ImageNet images when provided with 1/2 of the original image, using generative PCN and MHN with 8 =
The results confirm the hypothesis: the deeper the network, the higher the capacity. Partieu- larly, 10-layer network was able to reconstruct more than 98% (against the 72% of the AE) of the images when providing half the original pixels, and 74% (against the 48% of the AE) of the images when providing 1/4 4 of the pixels. This shows that model clearly outperforms state-of-the the-art AEs image reconstruction, even the overparametrized regime. Fig. summarizes these results.
Results:
Comparison with Hopfield networks
In this section, we compare against the most popular AM models, namely, classical and modern Hopfield networks (MHNs) X 10.. Despite their popularity, both types
N
p
1/2
p
1/4
1)
0.2
4
Hopfield networks fail to retrieve complex data such
100
4
3
3
natural images. We now show that the retrieval process model is significantly better, as it always converges plausible solution, even when provided with a tiny amount of information, large amount of corruption.
250
9
500
1
3
Table 1: Number of correctly retrieved
images different tasks by MHN trained on fractions of CIFAR10.
Classical Hopfield networks. We now compare against classical Hopfield networks [8). We have trained Hopfield network on = (2, 3, 5, 10} images of the MNIST dataset, and converted every grey-scale pixel to either 0 or 1, Hopfield networks only work on binary data. However, this model was able to retrieve the original images when presented with 1/2 only when N 2. To test generative PCNs the same task, used a small network with one hidden layer and 32 hidden neurons. Our model was always able to retrieve the original images. Visual results of this experiments are shown in Fig. (left).
Modern Hopfield networks. trained an MHN to memorize 50/100/250/500 CIFARIO images, and then performed three different experiments. In the first two, tried to reconstruct the original

---

Corrupted
Recovered
Original
Error
Compupted
Recovered
Original
Error
Corrupted
Recovered
Original
Error
Figure 10: Left: Our reconstruction 25 ImageNet 224 images and network with 8192 hidden neurons. Centre: Our reconstruction using a generative PCN with 1024 hidden neurons on CIFARIO 32x32 images. Right: Reconstruction of DANN on the same dataset (taken from (T8).
memories when providing 1/2 and 1/4 of the original image key. Then, we corrupted the original images by adding Gaussian noise. Despite the high capacity, MHNs do not perform well in image retrieval. Particularly, they were able retrieve most 9 images when presented with corrupted image, and at most when presented with an incomplete one. The results the experiment are given in Table ! In Fig. (right), provide visual comparison between showing that they are only able to retrieve one image when trained on complex images taken from the Tiny Imagenet dataset. Further details about the experiments are given in the supplementary material.
method and MHNs,
Deep associative neural networks. To improve their performance, MHNs have been augmented with a convolutional multilayer structure for unsupervised feature detection [18]. The resulting architecture, called deep associative neural network (DANN), is able to perform both AM and classification tasks. We now compare the image reconstruction experiments performed in [8] with ones performed by our generative PCN. We replicate the experiment proposed, which consists in presenting the network partial image of the CIFAR10 10 dataset. where squared patch covers the centre each image. Note that this is not a completely fair comparison, as (1) DANNs merge AM with convolutions and unsupervised feature detection layers, and (2) it is trained on the whole dataset. Hence, comparing it against a pure AM model such as ours may be unfair. However, we believe that presenting comparisons against models that perform AM tasks such DANNs is still interesting in understanding how our model compares against existing ones in the literature.
We first ran the same experiment CIFARIO as in 18] Particularly, we used 250 images. The comparison given Fig. 10 Unlike the image in [8,, our reconstruction shows no visible error. Furthermore, we replicated the experiment using 50 images of ImageNet, and showed that our model is again able to provide perfect reconstructions of the original images.
7 Biological relevance
The model described this paper closely related with theories of functioning of the memory system in the brain. It is typically assumed that memories are stored in a network of cortical areas organized hierarchically, where sensory information is processed and encoded by several neo-cortical regions, and then it is integrated across modalities in the hippocampus 12 2 It has been recently proposed that this aapocampo-neocorti
system can be viewed PCN 6 similar the one in this paper. In particular, Barron et al. (6) proposed that the hippocampus could be viewed as top layer in the hierarchical PCN (which would correspond to the memory layer our model), while neo-cortical areas would correspond to lower layers in the network. A contribution of our paper is to implement this idea and demonstrate in simulations that PCNs, which were only theoretically proposed to act as memory system (6), can actually work and effectively store and retrieve memories.
The mechanisms of learning and retrieval proposed in the PC account of the brain memory system in 6 closely resemble those occurring in our model. Barron et al. proposed that during learning, the generative model of the hippocampus updated until the hippocampal predictions correct the prediction errors by suppressing neo-cortical activity [6]. The training phase of our generative model closely resembles this framework, as it also accumulates prediction errors from the sensory neurons to the hidden layers and memory vector, while a continuous update of the value nodes corrects the emrors in the sensory neurons. Then, the parameters of our generative model are updated until its predictions reach low/zero error in the sensory neurons. Learning in PCNs relies on local Hebbian rules, i.c., the

---

weights between co-active neurons are increased, and such form of synaptic plasticity has been first demonstrated experimentally in the hippocampus [9] It has been further suggested that during the reconstructions of past memories, the hippocampus sends descending input to the neocortical neurons to reinstate activity patterns that recapitulate previous sensory experience [6] In the reconstruction phase of our generative model, the memory vector provides descending inputs to the sensory neurons, to generate patterns that recall previous sensory experience, e.g., stored data points.
Our model captures certain high-level features of the brain memory system, but many details of the model will have to be refined to describe biological memory networks. Barron et al. [6] reviewed in detail how the architecture of a PC model could be related to the known anatomy of neo-cortical memory areas, and how different populations of neurons and connections in the model could be mapped on known groups of neurons and projections in the neo-cortex. Interesting future work is to extend the top layer of our memory network include key features of the hippocampus, i.e., recurrent connections, known to exist in the subfield CA3, as well as an additional population of neurons in dentate gyrus and CA1, which have been proposed to play specific roles in learning new memories [20]. It is also interesting to investigate if our model can reproduce experimental data how the hippocampus drives the reinstantiation of stored patterns in the neo-cortex during retrieval 22 22
Related work
Predictive coding 4 and AM models 8 23 have rarely been related in the literature to date. However, they share multiple similarities, the main which is that both are energy- based models that update their parameters using local information.
Associative memories (AMs). In computer science, AMs date back to 1961, with the introduction of the learn matrix [24] a hardware implementation of hetero-associative memories using ferromagnetic properties. However, the milestones of the field are Hopfield networks, presented in the early eighties in their discrete [ and continuous [23] version. Recently, AMs have increasingly gained popularity. Particularly, 2-layer version of Hopfield networks with polynomial activations has been shown to have surprisingly high capacity [IO], which can be increased even more with exponential activations [25][66] (6)). The recent focus of the machine learning community on AM models, however, does not lie in applying them directly to solve practical hetero- associatiee memory tasks. On the contrary, it has been shown that modern architectures are implicitly AMs. Examples of these are standard deep neural networks (7 11 and transformers
(28) Hence, there is growing belief that understanding AM models will help the understanding and improvements of deep learning (2 27 30 31.. In fact, different classification models perform well when augmented with AM capabilities. DANNS (18].
e.g., use deep belief networks to filter information, while (32] improves over standard convolutional models by adding an external memory that memorizes associations among images.
Predictive
coding (PC). This framework includes both a biologically plausible neural architecture and a learning algorithm [4] developed to emulate learning in the visual cortex. Particularly, PC allows to describe multiple phenomena happening in the brain, using a single framework, such as free-energy minimization with local computations (6). From the more practical point of view, PC can be used to train high-performing neural architectures can exactly replicate the weight update of BP [6.
13,37 38, as it has been shown that
9 Conclusion
In this paper, we have shown that predictive coding (which was originally designed to simulate learning in the visual cortex) naturally implements associative memories that have high capacity and a high retrieval accuracy and robustness, and that can be expressed using small and simple fully connected neural architectures. We have shown this empirically in a large number of experiments. Particularly, we have performed denoising and image reconstruction tasks, comparing against standard autoencoders. Furthermore, we have shown that this model is able to reconstruct with no visible error natural high-resolution images of the ImageNet dataset, when provided with a tiny fraction of them. Overall, this work strengthens the connection between the machine learning and the neuroscience community. It underlines the importance of predictive coding in both areas, both as highly plausible algorithm to better understand how memory and prediction work the brain, and as an approach to solve corresponding problems in machine learning and artificial intelligence.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Alan Turing Institute under the EPSRC grant EP/N5 and by the AXA Research Fund. This work was also supported by the China Scholarship Council under the State Scholarship Fund, by the UK Medical Research Council under the grant MC UU_00003/1, by the National Natural Science Foundation of China under the grant 61906063, by the Natural Science Foundation of Hebei Province, China, under the grant F2021202064, by the Natural Science Foundation of Tianjin City, China, under the grant 19JCQNJC00400, and by the "100 Talents Plan" of Hebei Province, China, under the grant E2019050017.
References
[1] L. Squire and S. Zola-Morgan, "The medial temporal lobe memory system," Science, vol. 253, no. 5026, 1991.
[2] J. Felleman and D. C. Van Essen, "Distributed hierarchical processing in the primate cerebral cortex.," Cerebral Cortex, vol. no. pp. 1-47, 1991.
[3] E. A. Murray, T. J. Bussey, and L. M. Saksida, "Visual perception and memory: A new view of medial temporal lobe function in primates and rodents," Annual Reviews Neuroscience, vol. 30, pp. 92-122,22 2007.
[4] R. P. Rao and D. H. Ballard, "Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects," Nature Neuroscience, vol. 2, no. 1, pp. 79-87, 1999.
[5] J. Mcclelland, B. Menaughton, and R. O' O'Reilly, "Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory," Psychological Review, vol. 102, pp. 419-57, 1995.
[6] H. C. Barron, R. Auksztulewicz, and K. Friston, "Prediction and memory: A predictive coding account," Progress in Neurobiology, vol. 192, p. 101821, 2020.
[7] K. L. Stachenfeld, M. M. Botvinick, and S. Gershman, "The hippocampus as a predictive map," Nature Neuroscience, vol. 20, no. 11, p. 1643, 2017.
[8] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy of Sciences, vol. 79, 1982.
[9] D. Krotov and J. Hopfield, "Unsupervised learning by competing hidden units," Proceedings of the National Academy of Sciences, vol. 116, no. 16, pp. 7723-7731, 2019.
[10] D. Krotov and J. J. Hopfield, "Dense associative memory for pattern recognition." in Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, 2016.
[11] A. Radhakrishnan, M. Belkin, and C. Uhler, "Overparameterized neural networks implement associative memory," arXiv: 12362, 2019.
[12] K. Liu, J. Sibille, and G. Dragoi, "Generative predictive codes by multiplexed hippocampal neuronal tuplets," Neuron, vol. 99, no. 6. pp. 1329-1341, 2018.
[13] J. C. Whittington and R. Bogacz, "An approximation of the error backpropagation algorithm in predictive coding network with local Hebbian synaptic plasticity." Neural Computation, vol. 29, no. 5, 2017.
[14] Y. Song, Т. Lukasiewicz, Z. Xu, and R. Bogacz, "Can the brain do backpropagation? - Exact implementation backpropagation in predictive coding networks," in Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, 2020.
[15] Salvatori, Song, T. Lukasiewicz, Z. Xu, and R. Bogacz, "Predictive coding can do exact backpropagation convolutional and recurrent neural networks." arXiv:2103.0324 2021.
11

---

[16] T. Salvatori, Y. Song, T. Lukasiewicz. R. Bogacz, and Z. Xu, "Predictive coding can do exact backpropagation on any neural network," arXiv:2103.04689 2021.
[17] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back- propagating errors," Nature, vol. 323, no. 6088, pp. 533-536, 1986.
[18] J. Liu, M. Gong, and H. He, "Deep associative neural network for associative memory based on unsupervised representation learning," Neural Networks, vol. 113, pp. 41-53, 2019.
[19] T. V. Bliss and T. Lømo, "Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path." The Journal of Physiology, vol. 232, no. 2, pp. 331-356, 1973.
[20] R. C. O'Reell and J. L. McClelland, "Hippocampal conjunctive encoding, storage, and recall: Avoiding a trade-off," Hippocampus, vol. 4, no. 6, pp. 661-682, 1994.
[21] K. K. Cowansage, T. Shuman, B. C. Dillingham, A. Chang, P. Golshani, and M. Mayford, "Direct reactivation of a coherent neocortical memory of context," Neuron, vol. 84, no. 2, pp. 432 441, 2014.
[22] K. Z. Tanaka, A. Pevzner, A. B. Hamidi, Y. Nakazawa, J. Graham, and B. J. Wiltgen, "Cortical representations are reinstated by the hippocampus during memory retrieval," Neuron, vol. 84, no. 2. pp. 347-354, 2014.
[23] J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons," Proceedings of the National Academy of Sciences, vol. 81, 1984.
[24] K. Steinbuch, "Die Lernmatrix," Kybern., vol.
vol. 1, no. 1, pp. 36 45, 1961.
[25] M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet, "On model of associative memory with huge storage capacity," Journal of Statistical Physics, vol. 168, 2017.
[26] D. Krotov and J. Hopfield, "Large associative memory problem in neurobiology and machine learning," in International Conference on Learning Representations, 2021.
[27] T. Poggio, "From associative memories to powerful machines," CBMM Memo No. 114, 01/2021 2021.
[28] H. Ramsauer, B. Schäff, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, T. Adler, D. Kreil, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, "Hopfield networks is all you need," in International Conference on Learning Representations, 2021.
[29] M. Widrich, Schäfl, H. Ramsauer, M. Pavlović, L. Gruber, M. Holzleitner, J. Brandstetter, G. K. Sandve, V. Greiff, S. Hochreiter, et al., "Modern Hopfield networks and attention for immune repertoire classification," arXiv:2007.13500, 2020.
[30] P. Domingos, "Every model learned by gradient descent is approximately a kernel machine," arXiv:2012.00155. 2020.
[31] V. Feldman and C. Zhang, "What neural networks memorize and why: Discovering the long tail via influence estimation." T:XI1:2008.03703. 2020.
[32] G. Yang and F. Ding, "Associative memory optimized method on deep neural networks for image classification." Information Sciences, vol. 533, 2020.
[33] R. Bogacz, "A tutorial on the free-energy framework for modelling perception and learning," " Journal of Mathematical Psychology, vol. 76, pp. 198-211, 2017.
[34] K. Friston, "Learning and inference in the brain," Neural Networks, vol. 16, no. 9, pp. 1325- 1352, 2003.
[35] K. Friston, "A theory cortical responses," Philosophical Transactions of the Royal Society Biological Sciences, vol. 360, no. 1456, 2005.
12

---

[36] J. C. Whittington and R. Bogacz, "Theories of error back propaaatoot in the brain," Trends in Cognitive Sciences, 2019.
[37] B. Millidge, A. Tschantz, and C. Buckley, "Predictive coding approximates backprop along arbitrary computation graphs," arXiv:2006.04182, 2020.
[38] A. Ororbia and D. Kifer, "The neural coding framework for learning generative models," arXiv:2012.03405, 2020.
13


Reviews:
[{"review_id": "senHV3Rn-JZ", "reviewer": "Reviewer_SV1E", "summary": "In this paper, the authors explore the memorization capability of a Predictive Coding (PC) model. Following training on small subsets of images taken from popular datasets (CIFAR, ImageNet), the model is probed on a variety of recall tasks that require to reconstruct images corrupted by noise or to recover entire images from partial evidence. Recall performance is compared against that of a multi-layer autoencoder trained with error backpropagation.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This research work seems mostly incremental: the authors introduce a “novel” neural model for realizing an associative memory, however the proposed architecture seems based on existing predictive coding approaches [14, 15]. Also the tasks explored are based on recently published work.\n\nQuality: The article is well-written, and the results are clearly presented. However, I think the paper has several weaknesses that needs to be addressed:\n \n-\tA potential limitation is given by the relatively small number of data points (images) that can be accurately stored by the model: though it is interesting that the PC model can perfectly encode 500 ImageNet patterns, such number does not prove that the model can scale to realistic memory/recall scenarios.\n-\tI am not sure that [3] represents the state-of-the-art in (auto)associative learning models. The authors should also explore more powerful architectures, such as the PixelRNN /PixelCNN (van den Oord et al., 2016).\n-\tHow were the training images selected? The correlation between patterns could help or hinder the storing/retrieval dynamics. The authors should investigate this issue, and possibly choose training patterns according to different correlational structure. In this respect, it could be useful to measure the “effective dimensionality” of the input, which can be defined as the number of non-zero eigenvalues in the input covariance matrix (also see Olshausen and Field, 1997).\n-\tIt is a bit suspicious that the autoencoder with 256 hidden units is not able to learn to retrieve even just 100 images. The authors should report the trend of the loss function and carefully verify that the AE hyperparameters space was correctly explored.\n-\tWhat happens when the addition of Gaussian noise causes pixel values to go outside the range allowed? Were the values simply clipped within the [min, max] interval?\n-\tIt would be interesting to also explore other noise types, such as salt and pepper noise, which creates less smoothed versions of the corrupted patterns. Also, in the “Retrieval from Partial Data Points” task it would be interesting to explore the possibility that the given evidence (i.e., initial pixels) might not be contiguous.\n-\tFigure 9: it is valuable that the authors try to compare their model with previously published results. However, from my understanding the model presented in [19] was trained to memorize (and thus reconstruct) the entire CIFAR dataset, not just a small subset of images, as in the present model. A fair comparison would thus require testing the PC model on a much more challenging setup, where the entire CIFAR needs to be memorized.\n-\tMNIST is mentioned among the tested datasets (Line 159), but the authors never refer to it in the remaining of the paper.\n\nClarity: Overall, the paper is well written and properly organized. I am not sure the sentence “In AMs, the more training points they are able to memorize, the better, and feature learning is meaningless, as there is no test set that they have to generalize to” is legitimate. In fact, associative memories with latent units could learn higher-order correlations (features) from the visible pattern to increase storing capacity through data compression.\n\nSignificance: The research topic is timely and of broad interest for both the machine learning and the computational neuroscience communities.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors could better stress the limitations of their research approach.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical issues related to the content of this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Idac6SqpE_5", "reviewer": "Reviewer_K5he", "summary": "This paper is novel in that (1) it proposes a new framework to implement predictive coding -- it back-propagates the reconstruction error to optimize the activities at each hidden layer; it then uses an EM-like algorithm to update the weights of the generative model; (2) it uses this generative predictive coding framework to implement associative memory for remembering images    Using a hierarchical generative neural network to encode associative memories, traditionally realized in Hopfield net, is rather unusual and interesting.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "They probably should have cited and compared with current literature better. They only compared the performance or capacity of their model with an auto-encoder proposed in an archived paper by Radhakrishnan et al. (2019).   Their framework has strong connections to auto-decoder, PredNet,  Variational autoencoder, iterative amortization, but none are cited.  They should compare the memory capacity and performance of their model with SOTA and standard Hopfield net.  It is not clear whether the model's memory capacity and performance as associative memory are close to state of the art. Neural field or implicit function (see recent work by Matthew Tancik et al.  NeurIPS 2020 oral) can be made to encode far more images with much higher resolution and with fewer neurons.  While there are many insufficiencies in this paper,  I  nevertheless find it creative and conceptually refreshing and might deserve a forum at NeurIPS.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Umm. Broader and societal impact statement is missing. \nOn the other hand, this is basic scientific work on neural network and associative memory, I don't think that there will be any negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours."}, "extra_text": {}}, {"review_id": "GNhUeHzIjow", "reviewer": "Reviewer_XKUQ", "summary": "The authors tested a generative predictive coding network on associative memory tasks and showed that it compares favorably against autoencoders.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nI am not aware of prior work doing exactly what the authors did.\n\nQuality\nI have some concerns about the quality of this work.\n\nMajor concern:\nInsufficient comparison with alternative models and overblown claims\nLine 224: The author claimed that “This is, to our knowledge, the first AM model able to reconstruct ImageNet images to this level of precision”. I don’t know why the authors would claim that. Many memory networks such as those in NTM, DNC, Modern Hopfield network, etc. can store arbitrary real-valued inputs almost perfectly up to memory capacity. They should be able to store arbitrary ImageNet images as well. The authors mostly compared their networks to autoencoders, which is a weak baseline. Then the authors compared their results with one specific deep associative neural network, but even the authors admit that it’s not a fair comparison (Line 276).\n\nMinor concerns:\nIn the Introduction, the authors talk about the link between associative memory models and overparametrized neural networks (Line 39-40). While True, this statement ignores the fact that many associative memory models can memorize a pattern after a single exposure, while gradient-descent trained networks mostly can’t.\n\nIn the Introduction\nSaying predictive coding replicates learning in the visual cortex is overblown. It gives a sense that we already understand the learning mechanism in visual cortex, which we don’t.\n\nThis sentence is confusing/misleading.\n“we have shown that this model is able to reconstruct with no visible error natural high-resolution images of the ImageNet dataset, when provided with a tiny fraction of them”\nIt reads as if the model can reconstruct all 1M+ ImageNet images when trained on a small number of them.\n\n\nClarity\nThe writing is overall understandable. But there lacks clearer discussion of the limitation of the approach. Some important details are hidden in the appendix. For example, the network has to be run for many time steps before convergence, linearly and substantially increasing the computational complexity. But I think the number of time steps used, which ranges from 12 to 48, is only reported in the appendix.\n\nThe use of predictive coding in this particular setup is not well motivated. Why should it be good here?\n\nSignificance\nI have major doubts about several claims in the paper (see Quality section), so I cannot discuss the significance of these claims.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing beyond what’s mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "63yLrHWmfHk", "reviewer": "Reviewer_EmMS", "summary": "This papers uses predictive coding to train a new class of content addressable/ associative memory systems. It shows that it can perform substantially better than traditional autoencoders, with simpler architectures/fewer neurons. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall evaluation: The paper falls into the niche of building associative memory systems by training generative networks to the point of overfitting single training examples. While the idea is quite straightforward, it still feels odd in the sense of exploiting a pathological regime for feature learning training --which is essentially a bug from the perspective of the original training goal ---as a feature. PCNs seem to do better at this and have the advantage of naturally defining energy-based retrieval dynamics, which is a a small, but useful conceptual improvement. I am not sure how impressed one should be about the performance improvements, given that the comparisons are limited and incomplete.\n\nMajor criticisms: While the quality of some of the reconstructions is visually impressive, the numerical experiments are insufficient (see also itemized list below). For shallow networks, the comparison to VAEs only restricted to pixel noise and the partial cue scenario being evaluated mostly qualitatively. The comparison to DANNs is not convincing and purely qualitative. \n\nItemized questions/issues: \n- Unclear: what is the dimensionality of the 'memory vector' b?\n- Unclear: what is the exact functional form of the loss (not the local errors in eq2, but the actual link-to-data part of the loss)?\n- Unclear: in general, how long does it take for the network to relax to a solution?\n- Unclear and potentially very important: how many times is the exact same data point presented to the network (Hopfield nets are traditionally one-shot) Are training points noise-free?\n-how was the threshold for correct retrieval determined? why does it change across experiments?Can you show the distribution of errors for the same comparisons?\n- Related: can one see example reconstructions of the VAE for the same images? is the difference perceptually noticeable or mostly quantitative?\n- More generally: why does one need a different setup for the noisy vs incomplete cue experiments, in terms of number of neurons/ number of data points?\n-don't understand the statement: 'training points are stored in the memory vector' (line154) isn't b just another hidden layer with dynamic values?\n-for the partial retrieval experiments: does the geometry of the mask matter? i.e. is it important that the partial cue is spatially contiguous?\n- I find several statements in the intro and abstract very odd: for instance comparing energy based attractor networks with issues with single datapoint memorization in (usually feedforward) deep nets (in abstract and intro), or the statement that 'there is not test set that they need to generalize to' when there is a natural notion of the amount of noise that retrieval needs to be invariant to, which corresponds to the size of the attraction basin in a Hopfield net (see e.g. the probabilistic formalization of Hopfield network from Sommer and Dayan and any of the following work from Lengyel and collaborators).  Also 'standard neural networks' can;t generally reconstruct their inputs so their general use as content addresable memory devices is limited.  Overall, there is a sense of a lack of understanding of the conceptual context in the introductory text that negatively impacts the evaluation of the results as a whole. \n-it would be help to add errorbars to the summary plots\n\nOriginality and significance: There is a growing interest in using predictive coding networks in ML context; this paper shows a novel use of PCNs for autoassociative memory. The very small N (number of items stored) limits its applicability in practice and it is unclear how the phenomenology presented generalizes to more complex architectures.  The relevance for neuroscience is minimal.\n\nClarity: The text for the abstract and introduction needs work. Extensive use of acronyms sometimes hurts readability.\nThe description of IL is somewhat hard to parse if one hasn't seen the framework before; intuitively one tends to think of an initial feedforward flow of information, followed by error computations, rather than exclusively feedback 'synapses' with dynamics that relax to a fixed point of the global energy. In this sense the parallel to the brain hinders rather than helps the exposition. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No immediate negative societal impact concerns. LImitationsandcomparisontoexisticngliteraturecouldbeimprovedseemainreviesw)", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The paper tackles an interesting direction of realizing an associative memory via predictive coding, which is quite significant considering the importance of both associative memory and predictive coding in ML and neuroscience. 
- The argument about biological plausibility is controversial and rather subjective. So, I don't take this aspect into account.
- The reconstruction quality seems impressive compared to the past results. 
- I think the rebuttal addressed many concerns well enough including Hopfield Net experiments, baselines, etc. The baseline seems reasonable for this specific line of work (I agree with the argument of the authors in the rebuttal.) although the experiment and discussion can be improved based on the reviewer's feedback.
- The topic matches very well to the interest of NeurIPS. 
- This is a kind of very fundamental work for which we would like to see more interesting ideas even if the experiment results is not comparable to that of very well engineered large-scale systems. 
- The points raised by the reviewers are important and thus should be discussed in depth in the revision. Particularly, it would make the paper stronger by discussing the limitation more clearly. 
- I like the fact that the method is simple!

Decision: Accept (Poster)