Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors explore ways to combine auto-associative memories with variational Bayesian (VB) methods.
  Building on developments in modern Hopfield networks, the authors analyze iterative, memory-dependent inference through the lense of predictive coding.
  Specifically, the authors present:

  1. A VB interpretation of auto-associative memories, and the entailed connection between predictive coding and MCHNs
  2. Four new models that, to a varying degree, combine auto-associative memory and iterative VB inference
strengths_and_weaknesses:   Despite the interesting choice of subject matter, I regrettably find the submission to be lacking in many respects.

  1. With respect to the originality, and the scientific context the authors omit several publications.
  More importantly, this omission highlights that the research questions being addressed are poorly motivated.
  I presume that there both practical motives, such as the ability to easily "write" memories, as well fundamental, neuroscientific research questions.
  Unfortunately, the empirical evaluation does not provide such a self apparent justification, and the neuroscientific motivation remains unstated.

  2. The latter point, ties into the clarity of the submission.
  While the writing is clear, I subjectively feel it is somewhat deceptive. Please correct me if I am wrong, but all of the derivations presented in the main text are done under the assumption that the covariances are either fixed or that they are effectively null (e.g Eq. 15 in the appendix).
  For this reason, I do not believe one should frame the problems being addressed nor present the results in any Bayesian fashion.

  3. Given the preceding, I do not believe the presented results offer a particularly valuable contribution. The limited numerical evaluation further solidified this impression. For instance, using the auto-associative memory to do "one-shot learning" subjectively is a natural experiment to conduct. This additionally would present a potential, practical use case for the models introduced.


  Potentially relevant literature:

  1. Ramapuram, J., Wu, Y. & Kalousis, A. Kanerva++: Extending the Kanerva Machine With Differentiable, Locally Block Allocated Latent Memory. in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (OpenReview.net, 2021).
  2. Wu, Y., Wayne, G., Gregor, K. & Lillicrap, T. Learning Attractor Dynamics for Generative Memory. in Advances in Neural Information Processing Systems vol. 31 (Curran Associates, Inc., 2018).
  3. Wu, Y., Wayne, G., Graves, A. & Lillicrap, T. The Kanerva Machine: A Generative Distributed Memory. in International Conference on Learning Representations (2018).
questions: Please, see above.

limitations: Not applicable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper attempts to link between ideas in predictive coding (PC) and associative memory (e.g., modern continuous Hopfield network), showing that it is possible to reformulate associative memory retrieval as a kind of variational inference using latent variables. Several inference algorithms are the proposed based on this framework.
strengths_and_weaknesses: Strengths
=======
- It is useful to draw connections between separate ideas studied in different communities (variational inference, predictive coding and associative memory).
- The connection between PC and the continuous modern Hopfield network is interesting.

Weaknesses
==========
- The treatment of latent variables conditioned on memory is not entirely new. Several attempts before share similar insights: Kanerva machines [1,2] and variational memory [3].
- The writing is confusing with multiple versions of Eq (2), between framework (Section 3) and methods (Section 4).
- The experiments are somewhat limited.

[1] Wu, Y., Wayne, G., Graves, A., & Lillicrap, T. (2018). The Kanerva Machine: A Generative Distributed Memory. In International Conference on Learning Representations.

[2] Wu, Y., Wayne, G., Gregor, K., & Lillicrap, T. (2018). Learning attractor dynamics for generative memory. Advances in Neural Information Processing Systems, 31.

[3] Le, H., Tran, T., Nguyen, T., & Venkatesh, S. (2018). Variational memory encoder-decoder. Advances in neural information processing systems, 31.

questions: - Please clarify the contributions in the context of existing works (e.g., Kanerva machines and variational memory) as commented in Strengths and Weaknesses.
- The memory M should be introduced with more details. What is the relation between M and M_k in Eq.2? How patterns are stored in the memory? 
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper integrates the Modern Continuous Hopfield Network into the latent space of a VAE and shows the relationships between this, predictive coding, and the free energy principle. It then provides provisional analysis of these different approaches on two baseline datasets. 
strengths_and_weaknesses: Should cite https://pubmed.ncbi.nlm.nih.gov/33067397/ and have as a benchmark this model that is just a normal VAE. This may also make your classification in Table 1 harder for saying the VAE is not associative. 

Is the relationship to FEP really important here? Why not cache everything in terms of ELBO and VAEs that more people should be familiar with? 

There is no no intuitive justification for biasing the GMM prior towards patterns with a larger Euclidian norm? 

Why was MONet used for the CLEVR dataset if it makes vector similarity metrics less meaningful. You say that you try to address this by limiting the number of items etc. but why not just use another pretrained VAE where this problem is removed entirely? 

When you say representation you mean learning a compressed latent embedding as opposed to working with the raw image pixels? This could be made more clear. 

You say that the VAE embeddings are not necessary for strong performance in Tables 2 and 3 but this is very unsurprising given that you only use n=100 images. Please add analysis where the performance of each model changes as a function of the number of memories being stored. 

What is the difference between the VAE-Hopfield and the VAE-GMM? In using the softmax update rule for the GMM is this not the same as the MCHN update rule? 

This paper should be another benchmark: http://proceedings.mlr.press/v80/marino18a/marino18a.pdf it also makes the point of a hierarchical model integrating top down and bottom up signals in a PC esque way. 

This work should also be cited: https://arxiv.org/pdf/1709.07116.pdf 

Here is associative memory being read from in a probabilistic fashion and iteratively: https://arxiv.org/pdf/1804.01756.pdf while this work https://proceedings.neurips.cc/paper/2021/file/8171ac2c5544a5cb54ac0f38bf477af4-Paper.pdf shows that SDM is closely related to Attention and thus MCHNs. Also see the Kanerva Machine follow up with more of an explicit discussion of convergence to attractor states here: https://proceedings.neurips.cc/paper/2018/file/6e4243f5511fd6ef0f03e9f386d54403-Paper.pdf 

You claim that distance based metrics are better than taking dot products but this seems like a false dichotomy. If you normalize your vectors first then you can use a dot product and have cosine similarity etc. Modern Hopfield Networks use a form of bounded normalization (https://arxiv.org/pdf/1606.01164.pdf, pg.5), MCHNs also discuss normalization extensively and how it is intrinsic to the bipolar bits used in the original Hopfield Networks, Transformers use LayerNorm, it seems unfair to characterize MCHNs as not performing well only because you decided to not L2 norm their vectors. Also the Universal Hopfield Networks paper you cite shows that different distance metrics are heterogenous in their performance. 

I'd also like for it to be more clear what the implications or benefits of this work are? In many cases it looks like the NTM does better and there is not much discussion of this while you also acknowledge that the NTM is more flexible with writing and reading mechanisms. If the objective is to be able to store as many things as possible then there should be analysis of memory capacity. If the goal is just to provide a novel insight well fixed points in memory and representing them as GMMs is not novel nor is relating to them to VAEs (see https://jamied157.github.io/HQA/) which cites work from Hinton in 1994 on this topic (see the other references I have also provided above).

I do like Eqs. 7 and 8. But these come from your axioms in 3.2 including setting the MCHN to use the softmax update from the start and so these convergences between the MCHN loss function and the VFE should not be surprising? This brings me back to the lack of an explanation for weighting each gaussian by the Euclidian norm of the memory. And my critique about not having a comparison where these vectors are normalized. 

The discussion of VAE-PC-GMM also felt strange given that you conclude it is computationally intractable. And this is probably the gold standard model that you build towards in the paper. 

To conclude, at a minimum I'd like to see experiments around memory capacity, an explanation for weighting of the Gaussians, a much larger related work section, and normalization of the memories. At a maximum, I'd like to know what makes this work unique in the context of both new insights not made in the context of the papers I reference and or new baseline beating performance on ML tasks. 



questions: See above. 
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work provides a new formalization of associative memory models, merging them with the framework of variational inference. I found the idea interesting: In the standard predictive coding (PC) literature, we aim to infer the 'cause' that generates a specific datapoint. This cause may, or may not, be also influenced by a prior. When modelling the prior distribution according to a dataset of memories M, converging to the 'cause' that minimizes the variational free energy can be seen as a memory retrieval process. In the specific case of the prior having a Gaussian mixture distribution, and when no hierarchy is present, the resulting model is basically equivalent to a modern Hopfield network. Interestingly, in section 4.4 the authors simplify the update rule of PC and use t to derive an update rule that is similar to that of universal Hopfield networks with Euclidean distance as a similarity function [1]. This introductory intuition paves a foundation that can be used to build over associative memory models, able to encode semantic information by using variational auto encoders (VAEs).  The experiments show that (1) scoring memories in a 'semantic' space improves most of the performance of standard memory models, and that (2) it is possible to train the covariance matrix underlying the GMM to further improve the results under two kinds of corruption.


[1] Millidge, Beren, et al. "Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models." arXiv preprint arXiv:2202.04557 (2022).
strengths_and_weaknesses: Strengths:

- The paper is sound, and I like the overall intuition;
- The experimental results improve the performance over the (well picked) baselines.

Cons:

- The writing becomes confusing in the methods section. I have had some troubles understanding the exact functioning and structure of the proposed models. It would be nice to have drawings that explain how these models retrieve stored memories, as this confusion made me re-read sections 4 and 5 several times. 

- The authors show multiple experiments that compare standard models that perform storage and retrieval operations directly on the pixel space, against the proposed ones.  I'm puzzled by the choice of the authors on this task: no one of the proposed models is better than the baseline (Tabs. 2 and 3). 

- The authors then conclude with an interesting result of the paper: training on the retrieval task improves the performance. Again, however, the proposed method outperforms the competitor only on one task. 

- An experimental contribution, as stated by the authors, is that distance-based similarity scores seem to always outperform dot products. This is has been already shown in [1] cited above: Manhattan distance seems even better than Euclidean distance on natural images. 

All in all, I believe this is a nice paper, where the strengths slightly outperform the weaknesses, and that will be eventually accepted in a high level conference with little effort by addressing mine and other reviewers comments. While I believe that with little work this can be made a stronger submission, this paper in my opinion falls slightly above the acceptance rate, and hence I propose weak accept.

questions: Regarding clarity, could the authors:

1) Provide sketches of the proposed models; I believe they would be a nice and needed improvement to the paper. Particularly, could you describe step by step the retrieval process of the VAE-BP-GMM model?

2) Why have you decided to put a lot of emphasis on PC in the preliminaries and introduction, and then remove it from any experiment? This separation gave me the feeling of reading two different works in the same paper: the first, provides a theoretical derivation of AM models based on the theory of variational inference and proposes a new hierarchical model based on PC and VAEs; the second performs multiple experiments without any PC in it. I do get the underlying connection between the two parts.

Random question that would make an interesting addition to the theoretical side of the paper, but not important in the scoring:

One of the interesting properties of predictive coding, is that it has a neural implementation, where all the operations are implemented using local information via inhibitory and excitatory connections. Is there a neural implementation for the Hopfield network variation of PC?





limitations: The limitations are correctly addressed in the last page of the paper.
review_title: null

Meta-review: metareview: This submission provides a link between auto-associative memory and variational inference by making priors of representations being memory dependent. This view provides new important insights, as well as new algorithms. 

The reviewers raised concerns mostly about 1) the clarity of the presentation, and 2) connections to previous work. The authors addressed both of these points during the rebuttal stage. Still some questions remain about performance evaluation, but the AC is of the opinion that these concerns are minor compared to the conceptual advances the paper makes.


confidence: Less certain
Decision: Accept