On the relationship between variational inference and auto-associative memory
Authors: Louis Annabi
Alexandre Pitti
Mathias Quoy
Venue: NeurIPS 2022 Accept (2022)
Keywords: Associative Memory
Variational Inference
Predictive Coding
Hopfield Networks
Variational Autoencoders

Abstract: In this article, we propose a variational inference formulation of auto-associative memories, allowing us to combine perceptual inference and memory retrieval into the same mathematical framework. In this formulation, the prior probability distribution onto latent representations is made memory dependent, thus pulling the inference process towards previously stored representations. We then study how different neural network approaches to variational inference can be applied in this framework. We compare methods relying on amortized inference such as Variational Auto Encoders and methods relying on iterative inference such as Predictive Coding and suggest combining both approaches to design new auto-associative memory models. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models such as Hopfield Networks, End-to-End Memory Networks and Neural Turing Machines.

On the Relationship Between Variational Inference and Auto-Associative Memory
Louis Annabi
Alexandre Pitti
ETIS UMR 8051
ETIS UMR 8051
CY Cergy Paris Université, ENSEA, CNRS
CY Cergy Paris Université. ENSEA, CNRS
Cergy, France I aww.aiiigaial...mmm
Cergy, France sexxandreepittiieesee..
Mathias Quoy
ETIS UMR 8051
CY Cergy Paris Université, ENSEA, CNRS Cergy, France
mathias, quoy@ensea.ff
mathias..quo@ennee..f fr
Abstract
In this article, we propose a variational inference formulation of auto-associative memories, allowing us to combine perceptual inference and memory retrieval into the same mathematical framework. In this formulation, the prior probability distribution onto latent representations made memory dependent, thus pulling the inference process towards previously stored representations, We then study how different neural network approaches to variational inference can be applied in this framework. We compare methods relying amortized inference such as Variational Auto Encoders and methods relying on iterative inference such as Predictive Coding and suggest combining both approaches to design new auto- associative memory models. We evaluate the obtained algorithms on the CIFARIO and CLEVR image datasets and compare them with other associative memory models such as Hopfield Networks, End-to-End Memory Networks and Neural Turing Machines.
Introduction
In the recent years, many methods have been proposed in order to augment deep learning models with long-term memories. These models provide writing and reading mechanisms allowing to store and retrieve certain patterns inside a memory. Among these models, associative memories are memories that perform content-based addressing, meaning that they retrieve stored patterns based on an approximate incomplete version of those.
While many models consider one-step reading mechanisms to retrieve patterns from the memory, biological memory retrieval is an iterative process that can take a variable amount time. Taking this into consideration, Hopfield networks (15) have been proposed as Associative Memory (AM) implemented by a Recurrent Neural Network (RNN). Hopfield networks perform memory retrieval iterative process where stored patterns constitute local attractors of the RNN dynamics.
In this work, we show how AM models such as Hopfield networks can be formulated as variational inference methods using memory dependent probabilistic model of the observed data. In variational inference algorithms such as Variational Auto-Encoders (VAE) 19 32 and Predictive Coding (PC)
31 12] the representation inferred from the observed data tries to minimize energy function
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

based on the probabilistic model p(æ) generating the observed data. In the Free Energy Principle (FEP) literature [TO] [T] this function is called Variational Free-Energy (VFE), and is equivalent to the negative Evidence Lower Bound (ELBO) more often used in machine learning. The probabilistic model p(a:) can be decomposed according to prior probability distribution over representations p(z), and a likelihood p( ((=l²) 2)) describing the probability of observed data based on the representation. We propose to make p(z) depend on the patterns stored in memory, which allows us to derive a new expression of the VFE. The obtained energy function has minima that should be close to one of the stored patterns while properly encoding the observed input. As such, inference based on this energy function can be seen both as perceptual inference and memory retrieval. While using memorr-deepenent generative models or Gaussian Mixture Models (GMM) with VAEs not novel ideas, the PC approach has never been to this problem.
are
The paper organized as follows: in section we provide a deeper presentation of the concepts connected to our work and review related approaches. In section 3 we present an overview of the FEP mathematical framework and derive an expression of the VFE depending on the patterns stored in memory. In section 4 we design several AM models minimizing this energy function. In section we evaluate the obtained algorithms on the task of memory retrieval on two image datasets, and compare their performance with other AM models.
5
The contributions brought by this work are the following:
We design four AM models based on a variational inference formulation of memory retrieval.
We draw a connection hetween PC and the modern continuous Hopfield network 30
2
Related work
2.1 Auto-associative memory
Several approaches to the long-term storage of information in artificial neural networks have been proposed. Memory networks [38), and End-to-End Memory Networks (MemN2N) [4] propose to store information in a memory matrix that can be addressed using attention coefficients computed based on the content of different memory locations. The Neural Turing Machine (NTM) [13]
and the differentiable neural computer [14] combine this content- based addressing using attention mechanisms with location-baser addressing allowing more computer-like memory accesses.
Standing out from these approaches that consider a feedforward reading mechanism, Hopfield networks [15] and continuous Hopfield networks (16) instead store patterns as local attractors of an RNN. These memories can thus be addressed by initializing the RNN state with the input pattern, and retrieving the stored attractor after convergence. In order to improve the memory capacity, modern Hopfield networks 22 21 g propose several variants of the energy function using polynomial or exponential interactions. Extending these models to the continuous case, [00 proposed the Modern Continuous Hopfield Network (MCHN) with update rules implementing self attention, that they relate to the transformer model [36]. In (26), the authors introduce general Hopfield network framework where the update rules are built using three components: a similarity function, a separation function, and projection function.
has been shown that overparameterized auto-encoders also implement AM 28 33). These methods embed the stored patterns attractors through training. and retrieval is performed by iterating over the auto-encoding loop. In contrast, our methods allow one-shot writing: a new pattern æ can be inserted in memory simply by computing its representation and adding it to the memory matrix M.
Another line of research takes inspiration from the Sparse Distributed Memory model [8] building connections with attention mechanisms 4 and with the variational inference framework 440 41 29].
In particular, the Kanerva Machine 40 41 29 similar to the models we build in many aspects: they use an iterative reading mechanism, a memory depended prior the representation, all within the variational inference framework. Though, the PC toolbox provides methods that set the proposed models apart from these approaches, detailed in the methods section.

---

2.2 Predictive Coding
PC theory of brain function 31 7 extending the idea that neural representations emerge as part of an inference process of the causes of sensory observations, as already suggested by Helmholtz in 1867 B7.. The FEP [O] IT provides a principled derivation of PC networks based on the minimization of variational free-energy (VFE), quantity equivalent to the negative evidence lower bound (ELBO) used variational Bayesian methods, and defined as:
F(x:) = / log ((2) p(xx, 2) ((2)
where p(æ, =) denotes the generative model and q(z) denotes the approximate posterior (also called recognition density) on 2. VAEs [9 32) are a well-known method applying the idea of VFE minimization (equivalently ELBO maximization) to neural networks. In VAEs, this quantity is only optimized during the model training, and perceptual inference is performed as a simple forward pass through the encoder.
In contrast, neural network models based on PC intertwine the prediction (decoder) and inference (encoder) mechanisms within single hierarchical recurrent architecture comprising population of representation neurons and a population of prediction error neurons at each layer. Perceptual inference is then iterative mechanism supported by the dynamics of this RNN. Given observed input æ, the representation z is updated at each time step based on a bottom-up signal pushing z towards values that minimize the reconstruction error, and on top-down signal pushing towards values that maximize its prior probability p(z). While this prior probability on is often ignored, we show that it can bring auto-associative capacities the PC network, and we draw connection between the obtained mechanism and the modern continuous Hopfield network proposed in (30).
While both VAEs and PC networks output estimation of the approximate posterior (=),, they differ the computational mechanisms used for inference: VAEs perform amortized inference via a forward pass through encoder, while PC networks perform iterative inference implemented by its recurrent dynamics. Some works have suggested combining both approaches: the iterative inference can be initialized using the estimation provided by the amortized inference method 2 35
previous work [] explored the use PC techniques together with GMM-based prior probability distributions p(z). However, the obtained models were not used to perform memory retrieval.
3 Memory-based Variational Inference
3.1 Framework definition
Our framework is based on generative model p(x, 2) that can be factored into prior probability p(=) and a likelihood p(æ|z). The prior probability on is defined as a memory dependent distribution:
p(z) = p(2; M)
(2)
where the vectors Mj (the columns of M) constitute repertoire of stored representations.
The probability p(x[2) [z) is defined as a hierarchical generative model featuring several intermediate variables (h1,- h }. where L denotes the number of layers. By extension use the notations hg = x and hr = z. We assume that the generative model a cascade of multivariate Gaussians:
p(hy = Nh; (h141).
(3)
where is the covariance matrix of the Gaussians and is uniform across all layers, and fg are functions (typically neural network layers) parameterized by 0. Note that this can be adapted to arbitrary computation graphs by replacing 1 by the set of parent nodes of by for each node. For simplicity, we assume in the following derivations that each node only has one parent. In our experiments, the functions f correspond the CIFAR1O dataset, and of a MONet [6] decoder on the CLEVR dataset. We denote by fo the composition of all layers f/-
the different layers of Convolutional Neural Network (CNN) on
We can derive the VFE corresponding to the described generative model. The FEP formulation PC uses different approximations that allow us to greatly simplify the expression of the VFE introduced
3

---

in equation expression:
We refer to appendix A for detailed derivations and simply provide the simplified
1-1
F(x,h,,... -
Σ 1=0
Ih,
faitt)) M) + C
(4)
2
where is a quantity independent from {æz, hi...., h 2). The vectors (hi.- h 2) correspond to the means of the approximate posterior q(),,.... h1. = g(z) 11 (h;).
The term corresponding to the prior probability is often omitted, which can be justified as being equivalent to having no prior preferences over different values of Key to our method is the idea that this prior pull the inference process towards values of previously stored in the memory M, turning the patterns Mg into attractors of the PC network. As such, a suitable distribution would be one that associates high probabilities for the patterns stored in memory.
3.2 Classification of related methods in this framework
Here, we show that we can formulate Modern Continuous Hopfield Networks (MCHN) as PC networks derived from this expression of the VFE under some conditions:
There no representation component: 0.
The PC network is initialized with &
The prior distribution p(2; M) is defined as:
(2; = Σ TNN(2; (=; MR. Mk.
(5)
expl 3 M M
with Yk, n. Σ expl 8 MD MM
(6)
This prior distribution is Gaussian Mixture Model (GMM) with mixture means corresponding to the N stored patterns and mixing coefficients Tr depending on the patterns' Euclidean norm. It is parameterized by a coefficient > Based on these assumptions, we can derive the following expression for the VFE:
N 13 M&) +
F(2) 2T. log
which is, up constant C' and factor B, equivalent the MCHN energy function proposed in [00] The complete derivations of this expression are provided in appendix B. We can note that VFE does not depend on the input æ. The input is not part of the energy function, but serves as an initial estimate of the approximate posterior mean &. According to the FEP formulation of PC, applying gradient descent the VFE with regard to the approximate posterior mean á yields the update rules of the PC network. Using our expression of the VFE, we obtain:
t- softmax 2 MT
(8)
This equation is identical to the update rule of MCHN (detailed derivations are provided in appendix). We have shown that our framework can be related to MCHNs, but variational inference methods for perceptual inference also be retrieved in this framework by simply using neutral prior.
For VAEs and variants, this prior probability distribution exploited during learning, but the amortized inference cannot be dynamically adapted to this distribution. Therefore, to take into account change in this prior, we would need retrain the encoder. In the derivations of PC networks 12 5], the term p(z) is often ignored, which is equivalent to having flat prior distribution. However,

---

Table 1: Summary of the related models and proposed models.
Model
Representation
Associative memory
MCHN 30
X
Iterative
VAE 19
Amortized
X
PC [31
Iterative
BP [27
Iterative
X
HPC 135
Amortized and iterative
GMVAE 9
Amortized
Amortized
Overparameterized VAE
28
Amortized
Iterative
Kanerva Machines 40
Amortized
Iterative
VAE-PC-GMM (Ours)
Amortized and iterative
Iterative
VAE-BP-GMM (Ours)
Amortized and iterative
Iterative
VAE-GMM (Ours)
Amortized
Iterative
theoretically the iterative inference could take into account prior distribution, and dynamically adapt changes in this distribution, without retraining the model.
In the family of iterative inference algorithms, we can also mention the methods based on backprop- agation (BP) to estimate the representation. Instead of using neural computations to simulate the gradient descent on the energy function as done in PC, these methods directly use BP to optimize the representation 2. For instance 27 minimize prediction error using BPTT to adjust latent variable in an RNN. This method is fact very similar to PC, since has been shown that under some conditions, PC approximates the update rules entailed by BP applied the reconstruction error [39 (5). We provide in table I simple classification of these approaches depending on whether they representation and memory component, and whether the inference mechanism is amortized or iterative.
Methods
In this section, we present the AM models that we have designed using this framework. All these models are based pre-trained VAE that is used to provide initial estimates ž and predictions of based on ž. The proposed methods are represented in figure [
M
M
M
M
t
K.F
P
Auto-encoder
PC-GMM update (alg. 1)
Z
Z
Z
BP-GMM update (alg. 2)
GMM update (eq. 11)
GMM update (eq. 12)
X
X
X
Figure I: Representation of the memory retrieval process in the different proposed models. From left
to right: VAE-PC-GMM (4.2). VAE-BP-GMM 4.3), VAE-GMM (4.4) and VAE-GMM* 4.5).
4.1 Probabilistic model
The prior probability over the representation crucial to obtain model that infers representations close to stored patterns. We have shown that MCHN can be obtained starting from GMM prior biased towards stored patterns of large Fuclidean norm. Since there no intuitive justification for this bias, choose to instead start with balanced GMM prior distribution:
5

---

1 p(=; (2) - E N Ma,
where E denotes the covariance matrix, uniform across the mixtures. In the following models, we consider the simpler where Σ I except for the last proposed model (section 4.5 where the precision matrix P = Σ is trained.
4.2 PC based inference (VAE-PC-GMM)
Following the FEP formulation of PC, can derive the gradient of F (from eq. 4 according to each layer's representation hj. As such, performing gradient descent F yields system of update rules on each representation, that can interpreted as RNN dynamics. This PC network comprises at each layer two populations of neurons, one encoding the layer's representation, and one encoding the layer prediction error El f4 f (hi+1).
Detailed derivations this model are provided in appendix C, where the forward pass through the model is given in algorithm 1. Since the RNN dynamics implement a gradient descent, the initialization of this network responsible for the local minimum to which converges. We label VAE-PC-GMM the version of this algorithm where & is initialized using the result of the amortized inference via the encoder of the VAE.
4.3
BP based inference (VAE-PP-GMM)
In this second algorithm, we instead use BP the iterative inference mechanism used to optimize é. BP minimizes the following loss function:
llfo 1og
(10)
where y is an hyperparameter weighting the influence of bottom-up and top-down mechanisms, and Se denotes the decoder of the VAE. This gradient descent is parameterized by a learning rate Once again, we can initialize the gradient descent using the estimate obtained with the encoder of the VAE. Derivations of this model are provided in appendix C, where the forward pass through the model is given in algorithm 2.
Investigating the relationship between PC and BP based inference, have found that PC networks needed a larger number of iterations to convey information from the reconstruction error, and that this number of iterations was exponential with regard the depth of the decoder. For this reason, experiments with the VAE-PC-GMM model were prohibitively slow, and we only conducted experiments with the remaining models.
4.4 Restricting the iterative inference to the memory component (VAE-GMM)
Backpropagating the reconstruction error might not be necessary when the estimate provided by the VAE encoder already conveys enough information from the observed input æ. As such, also experiment with simpler version the previous model where iterative inference only considers the top-down update rule coming from the memory. In this simpler version, the update rule for & provided in algorithm becomes (proof in appendix C):
2
1-MM
softmax
MT
(11)
20²
This update mechanism is computationally lighter than the Ba-ba inference model, and has the advantage of being differentiable.
4.5 Training the precision matrix (VAE-GMM+)
For this last model, we suggest optimizing the precision coefficients Σ-1 of the GMM. These coefficients condition the shape of the Gaussian mixtures. Some memory retrieval tasks might need

---

to ignore partial information from the observed imput and as such could benefit from such an adaptation. In our experiments, we design such tasks on the CLEVR dataset, where the objective is to retrieve scenes using as input shifted image of the same scene, or an image of the same scene where the colors of the objects have been modified. To properly accomplish these two tasks, the AM model needs to learn to give less importance to position information in the first case, and to color information in the second case.
The model starts from the estimate 2 inferred by the VAE, and uses the following update rule:
2 t softmax (=- P. (2 (8-M)) M
(12)
During training, P optimized using BP in order to reduce the mean squared error between the inferred representation & and the correct memory pattern M During evaluation, the values of P are fixed.
5 Experiments
In this section, we present the experiments performed to evaluate the proposed models. In all experi- ments, we measure performance using the percentage of properly retrieved patterns from associative memories containing N = 100 patterns. We consider that a pattern is properly retrieved if the distance between the inferred representation & and the correct memory pattern Mg. is lower than a threshold value chosen manually. All the implementation details (benchmark models, hyperparameters, training, dataset splits) are provided in appendix D.
5.1 Datasets
We evaluate the proposed models on two image datasets: CIFARIO and CLEVR. CIFAR10 [20] (MIT License) consists of 32x32 RGB images of 10 classes. We pretrain VAE on the training set using a convolutional architecture, with a latent space of dimension d = 16.
The CLEVR dataset [17] (CC BY 4.0 License) consists of 64x64 RGB images of 3D scenes composed of simple 3D objects. We use a pretrained MONet (6) model as the VAF, with latent space of dimension
16. The MONet encoder infers a representation given an input scene image. In our experiments, we flatten this set representation, which can lead to wrong measures of similarity between information, but once flattened the permuted and original representations might have low similarity score. To limit this issue, we have capped the number of objects present in the scene images 3, and capped the size of the set representation to 4. Examples of images from the CLEVR dataset along with possible transformations are presented figure
representations. Indeed, a permuted set representation still encodes the same
5.2 Benchmark models
We compare our AM models with Modern Continuous Hopfield Networks (MCHN), Neural Turing Machines (NTM) and End-to-End Memory Networks (MemN2N). For fair comparison, all benchmark models, the memory is initialized with the stored patterns (we do not use the writing mechanisms of NTMs). We also experiment with these models in the representation space, where the memory is initialized with representations of the stored images, and the networks receive as input the representation of the corrupted image. We indicate this modification with the VAE prefix.
The NTM and MemN2N models are trained with BP using as loss function the mean squared error between the predicted output è (respectively & when using the VAE) and the correct memory pattern Mt. For a fair comparison, this training is performed with clean inputs x M to avoid giving any prior information about the type of transformation the inputs might be corrupted with during evaluation. Finally, we also experiment with version of the MCHN using a balanced GMM as starting prior distribution, that we denote GMM in our experiments. Equivalently, this the version of our VAE-GMI algorithm without representation component.
7

---

Original
Noise
Noise
Mask
Mask W 48
Shift
rolation
Figure 2: Examples of original, corrupted, and transformed images from the CLEVR dataset.
5.3 Corrupted inputs
First, evaluate whether the associative memory models able to properly retrieve stored patterns when given corrupted versions of these patterns. We apply different level of noise to the images (a (0.2, 0.4, 0.6, .... ,2.0)), , well different sizes of masks (16 and 24 pixels wide the CIFAR1O 10 dataset, and 32 and pixels wide on the CLEVR dataset). The results of these experiments reported in table 2 3 More figures and examples are provided in appendix E.
Table 2: Memory retrieval with corrupted inputs the CIFARIO dataset. Mean percentages of successful retrieval and standard deviations are reported.
Input
Clean input
Noise o=0.2
Noise 0=6.
Mask W=16
Mask W=24
GMM
100 + 0.0
100 + 0.0
94.3 + 6.6
99.7 1.3
48.7 12.7
Hopfield
3.0 3.0
3.9 4.0
1.5 2.4
2.3 4.303 3.6
1.6 + 2.6
NTM
99.3 2.4
98.9 + 2.7
91.6 + 15.0
9
62.7 20.9
MemN2N
81.1 11.3
77.9 + 11.1
65.8 13.4
63.2 13.9
32.3 11.5
VAE-BP-COE
0.0
100 0.0
8.91
9..
39.5 10.7
VAE-GMM
1.0
100
8.1
2.8
39.5 11.8
VAE- VAE-Hopfield
6.2
9.07
10.3
8.1
52.5 10.1
VAE-NTM
100
1.0
98.
9.5
58.7 10.2
VAE-MemN2N
95.0 4.7
92.6
8.72
8.2
44.4 11.7
Table 3: Memory retrieval with corrupted inputs on the CLEVR dataset. Mean percentages of successful retrieval and standard deviations are reported.
Input
Clean input
Noise o=0.2
Noise o=0.6
Mask W=32
Mask W=48
GMM
100 + 0.0
100 + 0.0
73.0 10.1
69.3 + 8.6
30.3 + 10.9
Hopfield
1.3
2.8
1.0 10 2.5
0.8 + 1.8
1.5 2.4
NTM
82.1
84.5 11.9
47.8 + 16.8
56.4 + 13.2
17.6 9.7
MemN2N
17
16 12.3
14.5 + 10.5
9
5.5 6.0
VAE-BP-GMM
100 0.0
49.5
2.5
69.2 +: 10.2
36.5 5.8
VAE-GMM
1.0
11.8
4.4
68.9 1 9.7
31.5611.6
VAE-Hopfiel
9.92
31.5+114
2.7
61.4
3.97
VAE-NTM
0.79
10.0
3.4
75.0
VAE-MemN2N
9.1
31.7+114
2.43.1
66.1 10.4
34.2 10.1

---

We can observe that the two models using dot- tot-product attention, MCHN (Hopfield in the tables) and MemN2N, perform poorly compared to other methods, not even reaching 100% of correctly retrieved patterns when presented with clean inputs. The NTM model, using cosine similarity based attention, works better and often outperforms the proposed models. Among the proposed models, we can see that the VAE-BP-GMM does not perform significantly better than the simpler VAE-GMM model.
An important observation from these results that in most scenarios, the best performing methods do not use a representation component. In particular, the MONet model seems very sensitive to noise, However, our intuition is that memory retrieval on the representation level should be more powerful if the transformation applied on the observed inputs æ has limited effect on the representations.
5.4 Scene transformations
To verify this hypothesis, we perform two additional experiments on the CLEVR dataset. we assume that the representations provided by MONet encode object positions, shapes and colors, then an AM model working on the representation level could perform well when we apply transformations on these specific object features. We propose two such scenarios. In the first scenario, we perform an RGB rotation on the images, which results in identical scenes with the exception of the object colors. In the second scenario, we take shifted crop of the original CLEVR image, which can simulate shifted point of view of the same scene. Examples images are displayed in figure 2
We measure the success of memory retrieval with these transformed inputs using our models as well the NTM and MemN2N models. We also experiment with the VAE-GMM* model performing precision coefficient training. We expect the model to learn that some information conveyed by the encoder (for instance color in the first scenario) is irrelevant for memory retrieval. For fair comparison, we also experiment with the VAE-NTM and we denote with an asterisk. The results are reported in table
models trained in the two scenarios, that
Table 4: Memory retrieval with transformed inputs the CLEVR dataset. Mean percentages of successful retrieval and standard deviations are reported.
Input
Color rotation
Shift
GMM
76 + 12.4 48.8 + 16.9
28 + 8.5 0.7 + 1.9 6.1
NTM
MemN2N
12.6 + 11.5
VAE-GMM
53.6 + 12.0
63.1 + 10.2
VAE-NTM
54.6 + 10.7
61.5 + 11.6
VAE-MemN2N
41.2 + 11.0
56.2 + 12.1
VAE-GMM
98.9 2.5
93.7 5.3
VAE-NTM*
98.2 + 3.2
96.3 + 4.4
M
91.6 6.9
84.2 7.8
Using representation component improves the performance of the models in these scenarios. Additionally, training on the retrieval task where the transformations are applied provides almost perfect retrieval scores for the VAE-GMM and VAE-NTM models.
6 Discussion
We have proposed formulation of auto associiiie memories based on the FEP and the variational inference framework, where the stored patterns condition the prior probability on the representation This framework has allowed to draw connection between MCHNs and the PC theory, as well as to design several AM models with high retrieval accuracy and robustness.
Combining representation and memory allows to retrieve patterns on the representation level, which decreases the size of the memory store, and performs retrieval based on a similarity measured in the latent space, where we can expect more meaningful features to appear. For instance, we have shown that this allows to recognize visual scenes from shifted point of view, while pixel-level AM models failed most of the time. Our results also seem demonstrate that distance based attention as

---

used in our models (see equation I1 outperforms dot-pre product based attention as used in MCHNs, although this performance gup could be mitigated with normalization techniques, as hinted by the results obtained with NTMs, that use cosine similarity instead of dot-product.
The proposed PC and BP based models take into account information from the query a: at each inference iteration, by directly minimizing an energy function that depends on æ (eq. I| and eq. (O). In contrast, the Kanerva Machine [40] [1 or the VAE-GMM and VAE-GMM* models we proposed only use this information to output an initial estimate & that is later optimized in order to minimize an energy function only depending on the memory. According to our experiments, this feature of the PC and BP based models does not seem useful for memory retrieval. However, in other applicative settings, these models could benefit from their ability to weight query pattern information and stored patterns information, performing form of memory-aided iterative perceptual inference.
The proposed formulation is limited content-based reading mechanisms. In contrast, models such as NTMs offer a larger variety of addressing schemes as well as writing mechanisms that allow to only store relevant information the memory. Another limitation of this work that we did not investigate the memory capacity and convergence properties of the proposed models. Future work should focus on these analyses, and possible improvements of the PC-based inference, for instance using precision weighting, or the "learning to optimize" approach of 24
7
Acknowledgements
This work was funded by the CY Cergy-Paris University Foundation (Facebook grant) and partially by Labex MME-DII, France (ANRII-LBX-0023-0),
References
[1] L. Annabi, A. Pitti, and M. Quoy. A predictive coding account for chaotic itinerancy. In I. Farkaš, P. Masulli, S. Otte, and S. Wermter, editors, Artificial Neural Networks and Machine Learning - ICANN 2021, pages 581-592, Cham, 2021. Springer International Publishing. ISBN 978-3-030-86362-3.
[2] Y. Bengio, B. Scellier, O. Bilaniuk, J. Sacramento, and W. Senn. Feedforward initializa- tion for fast inference of deep generative networks is biologically plausible. arXiv preprint 1rXiv:1606.0165) 2016.
[3] J. Bornschein, A. Mnih, D. Zoran, and D. Jimenez Rezende. Variational memory addressing in generative models. Advances in Neural Information Processing Systems, 30, 2017.
[4] T. Bricken and C. Pehlevan. Attention approximates sparse distributed memory. Advances in Neural Information Processing Systems, 34: 53011-131333 2021.
[5] C. L. Buckley, C. S. Kim, S. McGregor, and A. Seth. The free energy principle for
action and perception: A mathematical review. Journal of Mathematical Psychology, 81: 55 - 79, 2017. ISSN 0022-2496. doi: http://www...c.acaccc.ceeeeeeeaa
URL
http://wwer /wwpw/ww..o sciencedirect. www.coo//ccc////
[6] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Ler- chner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901. 11900, 2019.
[7] A. Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 1813)4411-204, 2013. doi: 10.1017/50140525xx 10000477.
[8] M. Demircigil, J. Heusel, M. Löwe, S. Upgang, and F. Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2):288-999, 2017.
[9] N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXin:1611.02644 2016.
10

---

[10] K. Friston. The free-energy principle: a rough guide to the brain? Trends in Cognitive Sciences, 13(7):293-301, 2009. ISSN 1364-6613. doi:
005.. URL
[11] K. Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11 (2)1127-138, 2 2010. ISSN 1471-0048. doi: 10.1038/mm2287. URL https:://oi.org// 1038/nrn2787
[12] K. Friston and S. Kiebel. Predictive coding under the free-energy principle. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 364:1211-21, 2009,
[13] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv: 2014.
[14] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska- S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7226):411-476 2016.
[15] J. Hopfield. Neural networks and physical systems with emergent collectivee computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2558 1982. ISSN 0027- 8424. doi: 10. 10.1073/pnas.79 http://ann.............
URL https: //ww://wwwww... pnas org httg://cntet//9//8000 2554
[16] J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10):3888-3022 1984.
[17] Johnson, Hariharan, Van Der Maaten, Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901- 2910, 2017.
[18] P. Kanerva. Sparse distributed memory. MIT press, 1988.
[19] D. Kingma and M. Welling. Auto-encoding variational bayes. CoRR, abs/1312.61 14, 2014.
[20] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[21] D. Krotov and J. Hopfield. Dense associative memory is robust to adversarial inputs. Neural 2018.
computation, 30(12):3151-33
[22] D. Krotov and J. Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
[23] H. Le, T. Tran, T. Nguyen, and S. Venkatesh. Variational memory encoder-decoder. Advances neurul information processing systems, 31, 2018
[24] J. Marino, Y. Yue, and S. Mandt. Iterative amortized inference. In International Conference on Machine Learning, pages
PMLR, 2018.
[25] B. Millidge, Tschantz, and C. L. Buckley, Predictive coding approximates backprop along arbitrary computation graphs. CoRR, abs/2006.04182,, 2020. httts:///aa: wwwreiiiioor/aaao abs/ 2006. 04182
[26] B. Millidge, T. Salvatori, Y. Song, T. Lukasiewicz, and R. Bogacz. Universal hopfield net- works: A general framework for single-shot associative memory models. arXiv preprint arXiv:2202.04557 2022.
[27] S. Otte, T. Schmitt, K. Friston, and M. V. Butz. Inferring adaptive goal-directed behavior within recurrent neural networks. In International Conference on Artificial Neural Networks, pages 227-235, Springer, 2017.
[28] A. Radhakrishnan, M. Belkin, and C. Uhler. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117(44:27162-27177, 2020.

---

[29] J. Ramapuram, Y. Wu, and A. Kalousis. Kanervat+: Extending the kanerva machine with differentiable, locally block allocated latent memory. International Conference on Learning Representations, 2020.
[30] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleit- ner, M. Pavlović, G. K. Sandve, et al. Hopfield networks all you need. arXiv preprint arXiv:2008.02211 2020.
[31] R. Rao and D. Ballard. Predictive coding in the visual cortex functional interpretation of some extra-classical receptive-field effects. Nat Neurosci, 2:79-87, 1999.
[32] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference deep generative models, 2014.
[33] Salvatori, Y. Song, Y. Hong, I.. Sha, Frieder, 7. Xu, R. Bogacz, and T. Lukasiewiez. Associative memories via predictive coding. Advances in Neural Information Processing Systems, 34, 2021.
[34] S. Sukhbaatar, a. szlam, J. Weston, and R. Fergus. End-to-end memory networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
[35] A. Tschantz, B. Millidge, A. K. Seth, and C. L. Buckley. Hybrid predictive coding: Inferring, fast and slow. arXiv preprint vrXiv:2204002166 2022.
[36] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[37] H. Von Helmholtz. Handbuch der physiologischen Optik: mit 213 in den Text eingedruckten Holzschnitten und 11 Tafeln, volume 9, Voss, 1867.
[38] J. Weston, S. Chopra, and A. Bordes. Memory networks. CoRR, abs/1410.3916, 2015.
[39] C. R. Whittington and R. Bogacz. An Approximation of the Error Backpropagation Algorithm in Predictive Coding Network with Local Hebbian Synaptic Plasticity. Neural Computation, 2269):122-11262, 05 2017. ISSN 0899-7667. doi: 10.1162/NECO. 00949. URL https
[40] Y. Wu, G. Wayne, A. Graves, and T. Lillicrap. The kanerva machine: A generative distributed memory. In International Conference on Learning Representations, 2018.
[41] Y. Wu, G. Wayne, K. Gregor, and T. Lillicrap. Learning attractor dynamics for generative memory, In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 9401 441-9410, 2018
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We believe that the claimed contributions in Introduction
properly reflected in sections B and 4
(b) Did you describe the limitations of your work? [Yes] See the last paragraph of the Discussion.
(c) Did you discuss any potential negative societal impacts of your work? [No] We did not identify any direct potential negative societal impact. any indirect potential impact brought our attention, we would be happy to discuss it.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] The assump- tions and definitions used in our derivations are stated (see sections 3.2 and and appendices A,B and C).
(b) Did you include complete proofs of all theoretical results? [Yes] We provide detailed derivations of our models and theoretical results in appendices A, B and C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See appendix
D.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix D.
(c) Did you report error bars (c.g., with respect to the random seed after running exper-
iments multiple times)? [Yes] See tables 3 and 4
well figures in appendix
E.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix D.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] Datasets were credited
in section code assets are credited in appendix
(b) Did you mention the license of the assets? [Yes] Licences for the datasets were
mentionned in Section the license for used code assets is mentionned in D.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Implementations of our models are linked in appendix D.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The data used are commonly used ML datasets. CLEVR is an automatically generated dataset and to our knowledge, the CIFARIO dataset does contain any personally identifiable offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tnTfVuY2zk", "reviewer": "Reviewer_E281", "summary": "The paper attempts to link between ideas in predictive coding (PC) and associative memory (e.g., modern continuous Hopfield network), showing that it is possible to reformulate associative memory retrieval as a kind of variational inference using latent variables. Several inference algorithms are the proposed based on this framework.", "questions": "- Please clarify the contributions in the context of existing works (e.g., Kanerva machines and variational memory) as commented in Strengths and Weaknesses.\n- The memory M should be introduced with more details. What is the relation between M and M_k in Eq.2? How patterns are stored in the memory? ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=======\n- It is useful to draw connections between separate ideas studied in different communities (variational inference, predictive coding and associative memory).\n- The connection between PC and the continuous modern Hopfield network is interesting.\n\nWeaknesses\n==========\n- The treatment of latent variables conditioned on memory is not entirely new. Several attempts before share similar insights: Kanerva machines [1,2] and variational memory [3].\n- The writing is confusing with multiple versions of Eq (2), between framework (Section 3) and methods (Section 4).\n- The experiments are somewhat limited.\n\n[1] Wu, Y., Wayne, G., Graves, A., & Lillicrap, T. (2018). The Kanerva Machine: A Generative Distributed Memory. In International Conference on Learning Representations.\n\n[2] Wu, Y., Wayne, G., Gregor, K., & Lillicrap, T. (2018). Learning attractor dynamics for generative memory. Advances in Neural Information Processing Systems, 31.\n\n[3] Le, H., Tran, T., Nguyen, T., & Venkatesh, S. (2018). Variational memory encoder-decoder. Advances in neural information processing systems, 31.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i0JfLd5M_b1", "reviewer": "Reviewer_7s9W", "summary": "This paper integrates the Modern Continuous Hopfield Network into the latent space of a VAE and shows the relationships between this, predictive coding, and the free energy principle. It then provides provisional analysis of these different approaches on two baseline datasets. ", "questions": "See above. ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Should cite https://pubmed.ncbi.nlm.nih.gov/33067397/ and have as a benchmark this model that is just a normal VAE. This may also make your classification in Table 1 harder for saying the VAE is not associative. \n\nIs the relationship to FEP really important here? Why not cache everything in terms of ELBO and VAEs that more people should be familiar with? \n\nThere is no no intuitive justification for biasing the GMM prior towards patterns with a larger Euclidian norm? \n\nWhy was MONet used for the CLEVR dataset if it makes vector similarity metrics less meaningful. You say that you try to address this by limiting the number of items etc. but why not just use another pretrained VAE where this problem is removed entirely? \n\nWhen you say representation you mean learning a compressed latent embedding as opposed to working with the raw image pixels? This could be made more clear. \n\nYou say that the VAE embeddings are not necessary for strong performance in Tables 2 and 3 but this is very unsurprising given that you only use n=100 images. Please add analysis where the performance of each model changes as a function of the number of memories being stored. \n\nWhat is the difference between the VAE-Hopfield and the VAE-GMM? In using the softmax update rule for the GMM is this not the same as the MCHN update rule? \n\nThis paper should be another benchmark: http://proceedings.mlr.press/v80/marino18a/marino18a.pdf it also makes the point of a hierarchical model integrating top down and bottom up signals in a PC esque way. \n\nThis work should also be cited: https://arxiv.org/pdf/1709.07116.pdf \n\nHere is associative memory being read from in a probabilistic fashion and iteratively: https://arxiv.org/pdf/1804.01756.pdf while this work https://proceedings.neurips.cc/paper/2021/file/8171ac2c5544a5cb54ac0f38bf477af4-Paper.pdf shows that SDM is closely related to Attention and thus MCHNs. Also see the Kanerva Machine follow up with more of an explicit discussion of convergence to attractor states here: https://proceedings.neurips.cc/paper/2018/file/6e4243f5511fd6ef0f03e9f386d54403-Paper.pdf \n\nYou claim that distance based metrics are better than taking dot products but this seems like a false dichotomy. If you normalize your vectors first then you can use a dot product and have cosine similarity etc. Modern Hopfield Networks use a form of bounded normalization (https://arxiv.org/pdf/1606.01164.pdf, pg.5), MCHNs also discuss normalization extensively and how it is intrinsic to the bipolar bits used in the original Hopfield Networks, Transformers use LayerNorm, it seems unfair to characterize MCHNs as not performing well only because you decided to not L2 norm their vectors. Also the Universal Hopfield Networks paper you cite shows that different distance metrics are heterogenous in their performance. \n\nI'd also like for it to be more clear what the implications or benefits of this work are? In many cases it looks like the NTM does better and there is not much discussion of this while you also acknowledge that the NTM is more flexible with writing and reading mechanisms. If the objective is to be able to store as many things as possible then there should be analysis of memory capacity. If the goal is just to provide a novel insight well fixed points in memory and representing them as GMMs is not novel nor is relating to them to VAEs (see https://jamied157.github.io/HQA/) which cites work from Hinton in 1994 on this topic (see the other references I have also provided above).\n\nI do like Eqs. 7 and 8. But these come from your axioms in 3.2 including setting the MCHN to use the softmax update from the start and so these convergences between the MCHN loss function and the VFE should not be surprising? This brings me back to the lack of an explanation for weighting each gaussian by the Euclidian norm of the memory. And my critique about not having a comparison where these vectors are normalized. \n\nThe discussion of VAE-PC-GMM also felt strange given that you conclude it is computationally intractable. And this is probably the gold standard model that you build towards in the paper. \n\nTo conclude, at a minimum I'd like to see experiments around memory capacity, an explanation for weighting of the Gaussians, a much larger related work section, and normalization of the memories. At a maximum, I'd like to know what makes this work unique in the context of both new insights not made in the context of the papers I reference and or new baseline beating performance on ML tasks. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a0hahGjRIyV", "reviewer": "Reviewer_NGa5", "summary": "The authors explore ways to combine auto-associative memories with variational Bayesian (VB) methods.\n  Building on developments in modern Hopfield networks, the authors analyze iterative, memory-dependent inference through the lense of predictive coding.\n  Specifically, the authors present:\n\n  1. A VB interpretation of auto-associative memories, and the entailed connection between predictive coding and MCHNs\n  2. Four new models that, to a varying degree, combine auto-associative memory and iterative VB inference", "questions": "Please, see above.\n", "limitations": "Not applicable.", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "  Despite the interesting choice of subject matter, I regrettably find the submission to be lacking in many respects.\n\n  1. With respect to the originality, and the scientific context the authors omit several publications.\n  More importantly, this omission highlights that the research questions being addressed are poorly motivated.\n  I presume that there both practical motives, such as the ability to easily \"write\" memories, as well fundamental, neuroscientific research questions.\n  Unfortunately, the empirical evaluation does not provide such a self apparent justification, and the neuroscientific motivation remains unstated.\n\n  2. The latter point, ties into the clarity of the submission.\n  While the writing is clear, I subjectively feel it is somewhat deceptive. Please correct me if I am wrong, but all of the derivations presented in the main text are done under the assumption that the covariances are either fixed or that they are effectively null (e.g Eq. 15 in the appendix).\n  For this reason, I do not believe one should frame the problems being addressed nor present the results in any Bayesian fashion.\n\n  3. Given the preceding, I do not believe the presented results offer a particularly valuable contribution. The limited numerical evaluation further solidified this impression. For instance, using the auto-associative memory to do \"one-shot learning\" subjectively is a natural experiment to conduct. This additionally would present a potential, practical use case for the models introduced.\n\n\n  Potentially relevant literature:\n\n  1. Ramapuram, J., Wu, Y. & Kalousis, A. Kanerva++: Extending the Kanerva Machine With Differentiable, Locally Block Allocated Latent Memory. in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (OpenReview.net, 2021).\n  2. Wu, Y., Wayne, G., Gregor, K. & Lillicrap, T. Learning Attractor Dynamics for Generative Memory. in Advances in Neural Information Processing Systems vol. 31 (Curran Associates, Inc., 2018).\n  3. Wu, Y., Wayne, G., Graves, A. & Lillicrap, T. The Kanerva Machine: A Generative Distributed Memory. in International Conference on Learning Representations (2018).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B7gPG1sOZ02", "reviewer": "Reviewer_NJ3P", "summary": "This work provides a new formalization of associative memory models, merging them with the framework of variational inference. I found the idea interesting: In the standard predictive coding (PC) literature, we aim to infer the 'cause' that generates a specific datapoint. This cause may, or may not, be also influenced by a prior. When modelling the prior distribution according to a dataset of memories M, converging to the 'cause' that minimizes the variational free energy can be seen as a memory retrieval process. In the specific case of the prior having a Gaussian mixture distribution, and when no hierarchy is present, the resulting model is basically equivalent to a modern Hopfield network. Interestingly, in section 4.4 the authors simplify the update rule of PC and use t to derive an update rule that is similar to that of universal Hopfield networks with Euclidean distance as a similarity function [1]. This introductory intuition paves a foundation that can be used to build over associative memory models, able to encode semantic information by using variational auto encoders (VAEs).  The experiments show that (1) scoring memories in a 'semantic' space improves most of the performance of standard memory models, and that (2) it is possible to train the covariance matrix underlying the GMM to further improve the results under two kinds of corruption.\n\n\n[1] Millidge, Beren, et al. \"Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models.\" arXiv preprint arXiv:2202.04557 (2022).", "questions": "Regarding clarity, could the authors:\n\n1) Provide sketches of the proposed models; I believe they would be a nice and needed improvement to the paper. Particularly, could you describe step by step the retrieval process of the VAE-BP-GMM model?\n\n2) Why have you decided to put a lot of emphasis on PC in the preliminaries and introduction, and then remove it from any experiment? This separation gave me the feeling of reading two different works in the same paper: the first, provides a theoretical derivation of AM models based on the theory of variational inference and proposes a new hierarchical model based on PC and VAEs; the second performs multiple experiments without any PC in it. I do get the underlying connection between the two parts.\n\nRandom question that would make an interesting addition to the theoretical side of the paper, but not important in the scoring:\n\nOne of the interesting properties of predictive coding, is that it has a neural implementation, where all the operations are implemented using local information via inhibitory and excitatory connections. Is there a neural implementation for the Hopfield network variation of PC?\n\n\n\n\n", "limitations": "The limitations are correctly addressed in the last page of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is sound, and I like the overall intuition;\n- The experimental results improve the performance over the (well picked) baselines.\n\nCons:\n\n- The writing becomes confusing in the methods section. I have had some troubles understanding the exact functioning and structure of the proposed models. It would be nice to have drawings that explain how these models retrieve stored memories, as this confusion made me re-read sections 4 and 5 several times. \n\n- The authors show multiple experiments that compare standard models that perform storage and retrieval operations directly on the pixel space, against the proposed ones.  I'm puzzled by the choice of the authors on this task: no one of the proposed models is better than the baseline (Tabs. 2 and 3). \n\n- The authors then conclude with an interesting result of the paper: training on the retrieval task improves the performance. Again, however, the proposed method outperforms the competitor only on one task. \n\n- An experimental contribution, as stated by the authors, is that distance-based similarity scores seem to always outperform dot products. This is has been already shown in [1] cited above: Manhattan distance seems even better than Euclidean distance on natural images. \n\nAll in all, I believe this is a nice paper, where the strengths slightly outperform the weaknesses, and that will be eventually accepted in a high level conference with little effort by addressing mine and other reviewers comments. While I believe that with little work this can be made a stronger submission, this paper in my opinion falls slightly above the acceptance rate, and hence I propose weak accept.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept