Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This papers uses predictive coding to train a new class of content addressable/ associative memory systems. It shows that it can perform substantially better than traditional autoencoders, with simpler architectures/fewer neurons. 
main_review: Overall evaluation: The paper falls into the niche of building associative memory systems by training generative networks to the point of overfitting single training examples. While the idea is quite straightforward, it still feels odd in the sense of exploiting a pathological regime for feature learning training --which is essentially a bug from the perspective of the original training goal ---as a feature. PCNs seem to do better at this and have the advantage of naturally defining energy-based retrieval dynamics, which is a a small, but useful conceptual improvement. I am not sure how impressed one should be about the performance improvements, given that the comparisons are limited and incomplete.

Major criticisms: While the quality of some of the reconstructions is visually impressive, the numerical experiments are insufficient (see also itemized list below). For shallow networks, the comparison to VAEs only restricted to pixel noise and the partial cue scenario being evaluated mostly qualitatively. The comparison to DANNs is not convincing and purely qualitative. 

Itemized questions/issues: 
- Unclear: what is the dimensionality of the 'memory vector' b?
- Unclear: what is the exact functional form of the loss (not the local errors in eq2, but the actual link-to-data part of the loss)?
- Unclear: in general, how long does it take for the network to relax to a solution?
- Unclear and potentially very important: how many times is the exact same data point presented to the network (Hopfield nets are traditionally one-shot) Are training points noise-free?
-how was the threshold for correct retrieval determined? why does it change across experiments?Can you show the distribution of errors for the same comparisons?
- Related: can one see example reconstructions of the VAE for the same images? is the difference perceptually noticeable or mostly quantitative?
- More generally: why does one need a different setup for the noisy vs incomplete cue experiments, in terms of number of neurons/ number of data points?
-don't understand the statement: 'training points are stored in the memory vector' (line154) isn't b just another hidden layer with dynamic values?
-for the partial retrieval experiments: does the geometry of the mask matter? i.e. is it important that the partial cue is spatially contiguous?
- I find several statements in the intro and abstract very odd: for instance comparing energy based attractor networks with issues with single datapoint memorization in (usually feedforward) deep nets (in abstract and intro), or the statement that 'there is not test set that they need to generalize to' when there is a natural notion of the amount of noise that retrieval needs to be invariant to, which corresponds to the size of the attraction basin in a Hopfield net (see e.g. the probabilistic formalization of Hopfield network from Sommer and Dayan and any of the following work from Lengyel and collaborators).  Also 'standard neural networks' can;t generally reconstruct their inputs so their general use as content addresable memory devices is limited.  Overall, there is a sense of a lack of understanding of the conceptual context in the introductory text that negatively impacts the evaluation of the results as a whole. 
-it would be help to add errorbars to the summary plots

Originality and significance: There is a growing interest in using predictive coding networks in ML context; this paper shows a novel use of PCNs for autoassociative memory. The very small N (number of items stored) limits its applicability in practice and it is unclear how the phenomenology presented generalizes to more complex architectures.  The relevance for neuroscience is minimal.

Clarity: The text for the abstract and introduction needs work. Extensive use of acronyms sometimes hurts readability.
The description of IL is somewhat hard to parse if one hasn't seen the framework before; intuitively one tends to think of an initial feedforward flow of information, followed by error computations, rather than exclusively feedback 'synapses' with dynamics that relax to a fixed point of the global energy. In this sense the parallel to the brain hinders rather than helps the exposition. 
limitations_and_societal_impact: No immediate negative societal impact concerns. LImitationsandcomparisontoexisticngliteraturecouldbeimprovedseemainreviesw)
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors tested a generative predictive coding network on associative memory tasks and showed that it compares favorably against autoencoders.
main_review: Originality
I am not aware of prior work doing exactly what the authors did.

Quality
I have some concerns about the quality of this work.

Major concern:
Insufficient comparison with alternative models and overblown claims
Line 224: The author claimed that “This is, to our knowledge, the first AM model able to reconstruct ImageNet images to this level of precision”. I don’t know why the authors would claim that. Many memory networks such as those in NTM, DNC, Modern Hopfield network, etc. can store arbitrary real-valued inputs almost perfectly up to memory capacity. They should be able to store arbitrary ImageNet images as well. The authors mostly compared their networks to autoencoders, which is a weak baseline. Then the authors compared their results with one specific deep associative neural network, but even the authors admit that it’s not a fair comparison (Line 276).

Minor concerns:
In the Introduction, the authors talk about the link between associative memory models and overparametrized neural networks (Line 39-40). While True, this statement ignores the fact that many associative memory models can memorize a pattern after a single exposure, while gradient-descent trained networks mostly can’t.

In the Introduction
Saying predictive coding replicates learning in the visual cortex is overblown. It gives a sense that we already understand the learning mechanism in visual cortex, which we don’t.

This sentence is confusing/misleading.
“we have shown that this model is able to reconstruct with no visible error natural high-resolution images of the ImageNet dataset, when provided with a tiny fraction of them”
It reads as if the model can reconstruct all 1M+ ImageNet images when trained on a small number of them.


Clarity
The writing is overall understandable. But there lacks clearer discussion of the limitation of the approach. Some important details are hidden in the appendix. For example, the network has to be run for many time steps before convergence, linearly and substantially increasing the computational complexity. But I think the number of time steps used, which ranges from 12 to 48, is only reported in the appendix.

The use of predictive coding in this particular setup is not well motivated. Why should it be good here?

Significance
I have major doubts about several claims in the paper (see Quality section), so I cannot discuss the significance of these claims.

limitations_and_societal_impact: Nothing beyond what’s mentioned above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is novel in that (1) it proposes a new framework to implement predictive coding -- it back-propagates the reconstruction error to optimize the activities at each hidden layer; it then uses an EM-like algorithm to update the weights of the generative model; (2) it uses this generative predictive coding framework to implement associative memory for remembering images    Using a hierarchical generative neural network to encode associative memories, traditionally realized in Hopfield net, is rather unusual and interesting.  
main_review: They probably should have cited and compared with current literature better. They only compared the performance or capacity of their model with an auto-encoder proposed in an archived paper by Radhakrishnan et al. (2019).   Their framework has strong connections to auto-decoder, PredNet,  Variational autoencoder, iterative amortization, but none are cited.  They should compare the memory capacity and performance of their model with SOTA and standard Hopfield net.  It is not clear whether the model's memory capacity and performance as associative memory are close to state of the art. Neural field or implicit function (see recent work by Matthew Tancik et al.  NeurIPS 2020 oral) can be made to encode far more images with much higher resolution and with fewer neurons.  While there are many insufficiencies in this paper,  I  nevertheless find it creative and conceptually refreshing and might deserve a forum at NeurIPS.

limitations_and_societal_impact: Umm. Broader and societal impact statement is missing. 
On the other hand, this is basic scientific work on neural network and associative memory, I don't think that there will be any negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors explore the memorization capability of a Predictive Coding (PC) model. Following training on small subsets of images taken from popular datasets (CIFAR, ImageNet), the model is probed on a variety of recall tasks that require to reconstruct images corrupted by noise or to recover entire images from partial evidence. Recall performance is compared against that of a multi-layer autoencoder trained with error backpropagation.
main_review: Originality: This research work seems mostly incremental: the authors introduce a “novel” neural model for realizing an associative memory, however the proposed architecture seems based on existing predictive coding approaches [14, 15]. Also the tasks explored are based on recently published work.

Quality: The article is well-written, and the results are clearly presented. However, I think the paper has several weaknesses that needs to be addressed:
 
-	A potential limitation is given by the relatively small number of data points (images) that can be accurately stored by the model: though it is interesting that the PC model can perfectly encode 500 ImageNet patterns, such number does not prove that the model can scale to realistic memory/recall scenarios.
-	I am not sure that [3] represents the state-of-the-art in (auto)associative learning models. The authors should also explore more powerful architectures, such as the PixelRNN /PixelCNN (van den Oord et al., 2016).
-	How were the training images selected? The correlation between patterns could help or hinder the storing/retrieval dynamics. The authors should investigate this issue, and possibly choose training patterns according to different correlational structure. In this respect, it could be useful to measure the “effective dimensionality” of the input, which can be defined as the number of non-zero eigenvalues in the input covariance matrix (also see Olshausen and Field, 1997).
-	It is a bit suspicious that the autoencoder with 256 hidden units is not able to learn to retrieve even just 100 images. The authors should report the trend of the loss function and carefully verify that the AE hyperparameters space was correctly explored.
-	What happens when the addition of Gaussian noise causes pixel values to go outside the range allowed? Were the values simply clipped within the [min, max] interval?
-	It would be interesting to also explore other noise types, such as salt and pepper noise, which creates less smoothed versions of the corrupted patterns. Also, in the “Retrieval from Partial Data Points” task it would be interesting to explore the possibility that the given evidence (i.e., initial pixels) might not be contiguous.
-	Figure 9: it is valuable that the authors try to compare their model with previously published results. However, from my understanding the model presented in [19] was trained to memorize (and thus reconstruct) the entire CIFAR dataset, not just a small subset of images, as in the present model. A fair comparison would thus require testing the PC model on a much more challenging setup, where the entire CIFAR needs to be memorized.
-	MNIST is mentioned among the tested datasets (Line 159), but the authors never refer to it in the remaining of the paper.

Clarity: Overall, the paper is well written and properly organized. I am not sure the sentence “In AMs, the more training points they are able to memorize, the better, and feature learning is meaningless, as there is no test set that they have to generalize to” is legitimate. In fact, associative memories with latent units could learn higher-order correlations (features) from the visible pattern to increase storing capacity through data compression.

Significance: The research topic is timely and of broad interest for both the machine learning and the computational neuroscience communities.

limitations_and_societal_impact: The authors could better stress the limitations of their research approach.
review_title: null

Meta-review: - The paper tackles an interesting direction of realizing an associative memory via predictive coding, which is quite significant considering the importance of both associative memory and predictive coding in ML and neuroscience. 
- The argument about biological plausibility is controversial and rather subjective. So, I don't take this aspect into account.
- The reconstruction quality seems impressive compared to the past results. 
- I think the rebuttal addressed many concerns well enough including Hopfield Net experiments, baselines, etc. The baseline seems reasonable for this specific line of work (I agree with the argument of the authors in the rebuttal.) although the experiment and discussion can be improved based on the reviewer's feedback.
- The topic matches very well to the interest of NeurIPS. 
- This is a kind of very fundamental work for which we would like to see more interesting ideas even if the experiment results is not comparable to that of very well engineered large-scale systems. 
- The points raised by the reviewers are important and thus should be discussed in depth in the revision. Particularly, it would make the paper stronger by discussing the limitation more clearly. 
- I like the fact that the method is simple!

Decision: Accept (Poster)