Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper explores the properties of an auto-encoder to behave as an associative memory retrieval mechanism. The authors show really interesting results where they are able to retrieve a small subset of encoded images (mnist) by giving the autoencoder random noise. They also show they can retrieve full videos by giving the autoencoder the output frame from the previous timestep.

The overall problem is a really interesting one which is to try to develop associative memory, retrieval models. 

Decision:
Reject

Reasons:
1. Although the work is interesting, the only related work the authors cover is hopfield networks. A cursory search indicates that this has been done before (k. Niki IEEE, Trischler 2016, M.A.Kramer 1992).

Improvement:
1. A more thorough discussion of related work would be helpful.
2. A direct qualitative comparison to related work would also be helpful.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper empirically demonstrates that DNNs can be trained to be identity mappings for small quantities of samples. It also demonstrates that for many parameterizations, these identity DNNs also have a small number of attractors, iterative fixed points, and can also learn short circular sequences of examples.

The paper is well written and easy to understand, although the presentation could be improved a bit (see comments). Its contents aren't particularly novel in terms of ideas, but they investigate memorization and attractors much further than previous studies. 
In a way, the memorization results are unsurprising. We know that DNNs can memorize perfectly, including sequences, so it is natural that by increasing capacity, at some point they should be able to memorize entire images (in fact this is what Zhang et al. (2019)'s Figure 1 appears to be showing). 
The more novel and surprising aspect of this is that DNNs would learn such strong (and so few) attractor basins. The fact that deep autoencoders could have attractors centered on the training points has been postulated before (see [1]), but this work makes a stronger case for it.

A crucial aspect that is missing from this paper in order for me to give in an accept is that there is very little about how this paper positions itself in the current literature. There could be much more discussion about related work, and much more discussion about the impacts of these findings.

I have given this paper a 'weak reject' mark but I think with some work this paper could be of interest to many. To reiterate, I am unable to see anything wrong with this paper, but at the same time I am unable to see how impactful these findings are.


Detailed comments:
- It's interesting that DNNs can implement associative memory, but what is the cost of doing that? Should we be using that in practice? Since there is no sense of how costly the presented experiments are, it is hard to tell.
- Again, these results are interesting, but after some time pondering about it, I can't really convince myself that knowing the results of this paper will be beneficial to future research. That being said, there are many areas of Machine Learning that I am unfamiliar with. It should be part of the paper to familiarize readers with areas where these results could be impactful.
- "the function interpolates the training images" not sure what this means. Interpolation means making a prediction for a point `u` that is "between" two points `x,y` with known values
- "black and white" should be "grayscale" if values are in [0,1]
- Figure 2b is interesting, but I wonder what happens if e.g. a perturbed version of e.g. Example 6 is fed. Presumably since Example 6 is not an attractor (Jacbian with an eigeinvalue > 1), it should converge to another example.
- Figure 2b's caption numbers, which say you use 1k example, to not correspond to numbers earlier in the text, which say you use 10k examples.
- "Since overparameterized autoencoders interpolate the training data", again this is a fairly important assumption and it needs to be defined very clearly, because it could mean many things.
- "it is essential that we interpolate to numerical precision", I don't think you are using the word "interpolate" correctly, do you mean "inference"? "train"?
- Adam citation should be "Adam: A Method for Stochastic Optimization, Diederik P. Kingma, Jimmy Ba", not Goodfellow et al., RMSprop should also have a citation, Hinton et al. 2012
- ReLU citation should be "Rectified linear units improve restricted Boltzmann machines, Nair & Hinton", Leaky ReLU should be Maas et al 2013, SELU should be Klambauer et al. 2017.
- The combination of section 3.1 and Figure 3 doesn't make it clear if models trained with Adam and RMSprop have weight decay or not. Can you clarify?
- "Note that a minimum width of 100 is needed to allow for interpolation." Again I think you mean "learning" rather than "interpolation".
- You say that you trained black and white images, but all the images of CIFAR10 in the figures are colored, including the inputs and outputs. Can you clarify why?
- You might be interested in [2], which is much older work about perceptrons, but still relevant to what is studied here.
- The linked supplemental material gives a 404 for me. I replicated the MNIST Figure 6 experiment. I was unable to replicate exactly your results but they were similar enough. In particular, the activation function choice seems to be critical.

[1] The Potential Energy of an Autoencoder, Hanna Kamyshanska, Roland Memisevic
[2] Basins of Attraction in a Perceptron-like Neural Network, Werner Krauth, Marc Mezard, Jean-Pierre Nadal


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper studies a phenomenon of unusual memorisation in deep overparametrized neural networks.
Authors observe that, if an auto-encoder overfits to machine precision on a number of images, they can be reliably decoded from random noise and that it is even possible to memorise this way a sequence of images.
Essentially, images from such a training set become attractors for the mapping defined by the auto-encoder.
The impact of network size, nonlinearity and initialization is studied and, quite surprisingly, very unusual trigonometric non-linearities performed the best.

I find the studied phenomenon rather interesting and the analysis well-performed, but I am not sure how practically important is this work.
First, I would argue that to call the overfit auto-encoder a function associative memory, it must be able to retrieve stored images not just from random noise, but from a somehow distorted or partially known version. 
Otherwise we are just left with a ridiculously large network that can only recall a handful of images we could store in the raw format using much less numbers.
Second, training until convergence takes prohibitively long time.

I would be also interested to at least an interesting discussion, if not an answer, to the question of why and how exactly trained images become attractors. 

In terms of novelty, it feels like Zhang et al, 2019 already studied a very similar phenomenon and the submitted paper does not add much to understanding of memorisation in neural networks. However, memorization of sequences was indeed a surprise. 

Overall, I do not have a strong opinion on rejecting the paper, it just feels like more work in this direction will make the paper significantly better. 
review_title: Official Blind Review #2

Meta-review: The paper shows that overparameterized autoencoders can be trained to memorize a small number of training samples, which can be retrieved via fixed point iteration. After rounds of discussion with the authors, the reviewers agree that the idea is interesting and overall quality of writing and experiments is reasonable, but they were skeptical regarding the significance of the finding and impact to the field and thus encourage studying the phenomenon further and resubmitting in a future conference. I thus recommend rejecting this submission for now.
Decision: Reject