Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper proposes a new type of energy-based models, a class of non-normalized generative models that relies on an energy function to retrieve patterns that correspond to its minima. The goal that is tackled by the authors is to implement an associative memory system, i.e. a mechanism that is able to retrieve any one of a set of patterns, given a distorted copy of these patterns. This task is traditionally carried out using attractor neural networks like the Hopfield model, a recurrent neural network model endowed with a learning rule that allows it to quickly embed a given set of patterns in its weight matrix such that the patterns become stable fix points of its dynamics. As the authors point out though, models like the Hopfield model are limited in their capacity to assimilate attractor patterns and in term of their expressiveness. On the other hand, more complex models based on deep architectures trained with gradient descent are slow at updating their weights to create new attractors.
The authors propose a new method to make up for the weaknesses of these two approaches. Their method is based on meta-learning, and in short consists in meta-training an energy function parametrized as a neural network such that  executing a write dynamics on the weights results in a model whose read dynamics (a gradient descent on the energy function) is able to denoise distorted inputs and retrieve the original ones. In practice, the write dynamics is obtained as a gradient descent procedure on a writing loss (which is itself dependent on the energy function) as a function of the weights. The meta-learning procedure minimizes the discrepancy between the original patterns and the retrieved ones by optimizing end-to-end the learning schedule parameters and initial conditions of the weights, analogously to gradient-based meta-learning methods like MAML.
The authors then carry out a series of experiments to check that their model is competitive with Memory-Augmented Neural Network (MANN) and Memory Networks (MemNets) in retrieving samples from Omniglot, CIFAR and ImageNet, in terms of retrieving abilities for a given memory size. In the supplementary material section they in addition compare their model's performance against the Hopfield model and recurrent networks on the classical toy task of retrieving random binary patterns, also with good results for the new model.

Decision:
This paper is very clearly and compactly written. The idea of training an energy-based model through gradient-based meta-learning seems novel and innovative. 
One thing that the the paper is arguably missing, is a convincing motivation section for focusing on energy-based models. The panorama of generative models has radically changed since attractor neural networks and energy-based models were first introduced. At the time powerful methods like variational autoencoders, normalizing flows and GANs didn't exist. But nowadays, one could arguably expect that energy-based models should be contextualized and motivated in the perspective of comparing them with these new breeds of deep generative models. I am absolutely not suggesting that the authors should providing experimental comparisons between their models and GAN or VAE, but simply that they compare them to their style of generative modeling in terms of advantages, disadvantages, use cases, and potential for applications.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: *Summary:*

The authors propose to tackle the associative memory problem by recasting read/write operations to read/write by optimizing the parameters/input of an energy based model. Writing is reformulated as training a parametric energy model (EBMM) to have local minima of energy w.r.t. the parameters at memorized data points. Reading is performed by performing (projected) gradient descent on the corrupted/incomplete input to minimize the energy. To ensure the operations are fast (read and write with minimal gradient steps), the authors propose to take inspiration from modern meta-learning literature and learn initialization parameters of the energy model (and other hyperparameters for GD during read/write) from which writing is fast while ensuring reading is also fast, since the models are trained to maximize read/write performance within a constrained number of gradient steps.  Experimentally, the authors show that EBMM reading performs similar to baseline methods (but better across many memory sizes) on the standard Omniglot task. On CIFAR-10 and downsized ImageNet, they show much better L2 reconstruction error of corrupted images. They also show that the learnt energy 

*Recommendation:*

I believe this is a very neat idea, and utilizes large parametric models for "smart" overfitting and compression of data for the associative memory task. The proposed meta-learning approach to training the model seems to perform well across multiple simple and challenging datasets, and therefore I would recommend accept. My current recommendation is very borderline (weak accept) because of a lack of some experimental rigour (which I would love clarifications on), and missing related work, which I mention below.

*Discussion Points and Concerns from the Reviewer:*

- Dataset / batching details 
Please mention how the datasets were split for training and testing the models. How much training data is utilized to meta-learn the EBMM initialization? How is batching performed? I believe these details are very important to mention in the paper for reproducibility of results. 

Are there any correlations in the batch selection? Can you evaluate how good the associative memory performs across different correlation levels in the batch (A well learnt algorithm should demonstrate better reconstruction at lower memory levels for correlated batches). 

- Experiments across multiple SNR and generalization on noise patterns
The authors mention at the beginning of Section 4 that a random block is corrupted, but in the end the experiments are done on a constant corruption size on the CIFAR and ImageNet images. How do the models perform across different signal-to-noise ratios? Similarly, the model is trained on simple noise patterns 

- Missing related work
There is related work [1] in learning in Hopfield Networks using the implicit function theorem and finding stationary points of the dynamics. This work is not mentioned in the paper, and is a valid baseline for this paper as well.

- Mentioning Appendix D in the main paper
Appendix D is not mentioned in the main paper and has a short discussion on the mismatch between the reading process and the writing loss during meta-training. It also mentions additional tricks required for the training, and I believe it should be mentioned in the main paper like other sections are appropriately referenced. 

- Large batch sizes for ImageNet
Work from [2] can be utilized to backpropagate through very long optimization sequences and therefore can be utilized to train with larger batch sizes in the ImageNet example. It is important to see how the small model utilized for ImageNet works to compress higher batch sizes, as that is one of the major practical issues with the algorithm.

- Related paper at NeurIPS this year 
[3] is a related paper from Neurips this year, which the authors could consider adding as contemporary work

- Comments on scalability
The associative memory papers have often been criticized for lack of scalability, and I think the authors make progress towards making this better with the use of unconstrained energy models in the learning process. It would be nice to have a discussion of the scalability from the authors, highlighting issues in the current model and future directions

References:
[1] Reviving and Improving Recurrent Back-Propagation, ICML '18
[2] Gradient-based Hyperparameter Optimization through Reversible Learning, Maclaurin et al. ICML '15
[3] Metalearned Neural Memory, Munkhdalai et al. NeurIPS '19
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Thanks for the extensive answers. I updated my rating based on the provided clarification and extra experiments.

=====================
My understanding of eq 1-5 is that the algorithm finds an energy landscape (by modifying \theta) for each dataset (task) such that in this landscape, the inputs from the distribution are reachable by truncated gradient-descent initiating at a query (distorted input with respect to some distortion model).

1- The connection to meta-learning is unclear in your experiments. Can you elaborate on that? 

2- The expectation in eq 5 is over different input patterns, which I assume that a set of input patterns belong to a task. What is that you write in memory?  For each experiment, what are the different input patterns (tasks) that you have written in the memory?

3- What is the \theta that is feed to the read function at the test time? 

4- Are you testing on the tasks that you already trained on?

5- How this approach generalizes to unseen (or relatively close) task? 

6- Can it recover any query that is not constructed with respect to the distortion model that is trained on? or what happens if the distorted image at test times comes from a different distortion model? (image blocking, for example)

7- How many distorted samples are used for training?


8- For the chosen tasks, I am curious to see the experimental comparison to deep image prior (Ulyanov, 2018). Deep image prior would be very similar to the read operator (although the gradient descent is over the parameter of model) without having write operations when you define the energy as MSE.



Typos and writing style:
-- The expectation in eq 2 should independently show what the expectation is taken with respect to. 
-- input patters -> input patterns
-- figure 3 -> Figure 3
-- section 1 -> Section 1
-- 4 random images -> four random images 
-- the Figure 5b -> Figure 5b
-- Models such as (Ba et al., 2016; Miconi et al., 2018) enable ->  Models such as Ba et al. (2016) and Miconi et al. (2018) enable
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
======================================== Update after revisions ============================================

I appreciate the effort the authors have put into the revision and the rebuttal. I'm happy to increase my score and recommend acceptance based on the revised paper. 

However, I have to say that some of my worries still linger. With respect to non-memory baselines, the authors have responded that the memory based models will outperform non-memory models in cases where prior structure is less important than memory and provided a demonstration of this with an extreme example, i.e. a case with no structure (random binary strings example). I understand the point being made here, but this is a rather pedantic and uninteresting example. The authors have provided another (more interesting) example in Appendix A3 and shown that the memory-based model outperforms some simpler baselines such as the DAE even in this case. But no explanation is given for this result. Why is the memory based model outperforming the DAE in this case, given that this is an example where prior *is* very important? I'm a bit worried that the DAE results may perhaps be due to a non-optimized architecture or training setup (and what exactly is the architecture used for the DAE here)? I would appreciate it if the authors could clarify these issues in the final version.

I have also spotted several typos. For the final version, please make sure to go through the paper thoroughly at least once and fix all the typos.

========================================================================================================

This paper proposes a meta-learning approach to learning fast read and write mechanisms in an energy-based model so that a given set of images can be quickly inducted into memory and retrieved from memory with noisy queries. The paper is well-written and the proposed approach seems interesting and novel enough. However, I have some concerns about the paper that need to be addressed. Here are the main issues for me:

1) I am in general not really convinced about the supposed advantages of these attractor memory models (this paper and the earlier Kanerva machine) over more standard and much simpler approaches. For example, for the problem of retrieval from noisy queries, a more standard approach would be a simple autoencoder. Note that in an autoencoder, reading (inference) is already fast. The authors might point out that writing (training) will not be fast, which is correct. However, the meta-learning phase proposed in this paper will also not be fast and perhaps the fair comparison should be between the meta-learning phase of this paper and the standard training phase of an autoencoder. Note that the autoencoder will have additional benefits. For example, with the autoencoder, one is not constrained by memory storage requirements and can make use of a much larger set of images to train the model. This allows the model to learn a richer structure in images. Moreover, with a large enough feedforward net, one can approximate arbitrarily complex dependencies in images. However, in attractor memory models, on the other hand, one necessarily restricts oneself to a particular model class that can be expressed as gradient descent dynamics in an energy landscape both during reading and writing. This seems overly restrictive to me. So, perhaps, the authors can clarify the supposed advantages of these attractor memory models a bit better. For example, I would be interested in seeing some comparative results with, say, a denoising autoencoder model.

2) Currently, the paper only uses a specific type of “block-noise” corruption. One thing that would be nice to see is some results with other noise models. I think this is important to demonstrate that the approach is general enough to handle different kinds of noise. Also, a salt-and-pepper noise will allow the authors to compare their results with the dynamic Kanerva machine (the authors note that the DKM failed to train successfully for the block-noise used here). 

3) It would be good to say something about the meta-learned parameters, theta_bar, r, tau. Is there any meaningful structure in these parameters that distinguishes them from their random initial values? Is one of these parameters more important than the others? For example, what happens if you just use generic step size decay rules for gamma and eta (or perhaps no decay at all)?
review_title: Official Blind Review #3

Meta-review: Four knowledgable reviewers recommend accept. Good job!
Decision: Accept (Poster)