Overparameterized Neural Networks Can Implement Associative Memory
Authors: Adityanarayanan Radhakrishnan
Mikhail Belkin
Caroline Uhler
Venue:  (2020)
Keywords: Associative Memory
Memorization and Recall
Attractors
Deep Autoencoders

Abstract: Identifying computational mechanisms for memorization and retrieval is a long-standing problem at the intersection of machine learning and neuroscience.  In this work, we demonstrate empirically that overparameterized deep neural networks trained using standard optimization methods provide a mechanism for memorization and retrieval of real-valued data.  In particular, we show that overparameterized autoencoders store training examples as attractors, and thus, can be viewed as implementations of associative memory with the retrieval mechanism given by iterating the map.  We study this phenomenon under a variety of common architectures and optimization methods and construct a network that can recall 500 real-valued images without any apparent spurious attractor states.  Lastly, we demonstrate how the same mechanism allows encoding sequences, including movies and audio, instead of individual examples.  Interestingly, this appears to provide an even more efficient mechanism for storage and retrieval than autoencoding single instances.


Under review as a conference paper at ICLR 2020
OVERPARAMETERIZED NEURAL NETWORKS CAN IMPLEMENT ASSOCIATIVE MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Identifying computational mechanisms for memorization and retrieval a long- standing problem at the intersection of machine learning and neuroscience. In this work, we demonstrate empirically that overparameterized deep neural networks trained using standard optimization methods provide mechanism for memoriza- tion and retrieval of real-valued data. In particular, we show that overparameter- ized autoencoders store training examples as attractors, and thus, can be viewed as implementations of associative memory with the retrieval mechanism given by iterating the map. We study this phenomenon under variety of common architectures and optimization methods and construct network that can recall 500 real-valued images without any apparent spurious attractor states. Lastly, we demonstrate how the same mechanism allows encoding sequences, including movies and audio, instead of individual examples. Interestingly, this appears to provide more efficient mechanism for storage and retrieval than autoen- coding single instances.
INTRODUCTION
Training neural networks to act as a model of associative memory is problem dating back to the in- troduction of Hopfield networks (Little) 1974 Hophield 1982). Hopfield networks are able to store binary training patterns as attractive fixed points; thus, the network allows for recall of the train- ing patterns given corrupted inputs. In this work, we demonstrate that modern overparameterized deep neural networks trained using standard optimization methods also provide a method for mem- orization and retrieval by storing training instances as attractors. Moreover, we show that through autoencoding and sequence encoding, the training examples can be recovered from random inputs to the trained network. We are not aware of any observation of this phenomenon in the literature, with the exception of Zhang et al. (2019), which studies the case of memorizing a single image.
Autoencoding. We begin with the example of an autoencoder neural network implementing family of continuous functions in the space F { Rd + [R.). Given training examples (200) J= gradient descent is used to minimize the following autoencoder objective:
Σ
()) - 20
arg min JEF
After training in the overparameterized regime, we obtain function f satisfying f(x(()) x a:(0) for all i, i.e., the function interpolates the training images. In Figure Ia] we analyze the function that learned after training fully connected autoencoder (architecture detailed in Appendix B) 10 examples from the CIFARI0 dataset (Krizhessky 2009). Note that inputting random images causes the autoencoder to produce outputs that are visually similar to training examples.
While the ability to recall training examples from random inputs can be inspected visually as in Figure Ta we can also quantify the number training examples that can be recalled by viewing autoencoders from dynamical systems perspective. Namely, since an autoencoder is a map from the feature space to itself, we can iteratively apply the autoencoder to the output starting from any input and investigate whether the sequence of iterates converges example is called an attractor, when there exists a set of instances with non-zero measure for which iterating the trained autoencoder map converges to the training example. This happens if the largest.
training example. A training

---

Under review as a conference paper at ICLR 2020
Outbut
Netrol Inseggts
Outad
(a)
(h)
Figure I: (a) Fully connected autoencoder trained on 10 images from CIFARIO. When fed Gaussian noise test images from CIFARIO the model outputs image that is visually indistinguishable from a training image. (b) A fully connected network trained to encode a sequence 389 frames of size 128 x 128 from the Disney film "Steamboat Willie." Inputting Gaussian noise and iterating the network yields the entire video.
eigenvalue (in absolute value) of the Jacobian of the autoencoding map at the training example is less than (see Section While the network trained Figure allared outputs an image that is visu- ally indistinguishable from training example in iteration, further iterations of the autoencoder result in increasingly more accurate reconstructions of the training examples (see Figure 2a).
As demonstrate throughout this paper, overparameterized autoencoders store training examples as attractors, and so training examples can be recovered through iterating a trained autoencoder on random input. We note that for fixed point be an attractor, all eigenvalues of the Jacobian matrix do not exceed one in absolute value. This is a highly restrictive condition as the number of these eigenvalues is equal to the dimension of the space. Thus, we do not expect fixed point of an arbitrary high- -igh-dimensional map to be an attractor. In that regard, the fact that deep neural networks produce attractors readily rather surprising. Moreover, even if an autoencoder stores training examples as attractors, there could be additional spurious attractors outside the training set. Interestingly, this paper, we present several examples of autoencoders for which we could not discover any attractors outside the training set through iteration from random instances. In particular, we now provide an example of a fully connected autoencoder that stores 500 real-valued training examples attractors with other observable attractors.
Memory and Recall of 500 Images as Attractors of an Autoencoder. We train a fully connected autoencoder (architecture detailed in Appendix B) on 500 black and white CIFAR 10 training images until the training loss is less than 10 We find that all 500 examples are stored as attractors since the top eigenvalue of the Jacobian for all of these examples is less than 1. Lastly, upon iterating 10,000 black and white examples from CIFAR1O and 10,000 examples of Gaussian random noise, are unable find any attractors outside the training set.
Sequence Encoding, Remarkably, this phenomenon extends from encoding single instances (such images) to sequences. Instead of autoencoding, where the loss is the reconstruction loss single images, train f such that f (xx(1) s ((x(((+1) mod w)). In Figure 1b we trained network to encode 389 frames of size 128 x 128 from the Disney film "Steamboat Willie" by mapping frame to frame i ++1 mod 389. As is shown in the attached video and Figure Ib] inputting random noise to the trained network f and iterating the network yields the original video. In fact, after a few iterations from random image, one of the frames of the original video is recovered
numerical
precision, and since the network is trained to map i to frame i+1 1 mod 389, once one of the frames of the original video recovered, the remaining frames are all recovered. Interestingly, we show in Section that encoding a sequence of instances in this manner appears to provide a more efficient mechanism for storage and retrieval than autoencoding each instance in the sequence separately.
We can similarly apply the sequence encoding objective to audio. Given an 8 second sample audio, we trained a fully connected network to map each second i of frequencies (represented as a 22k dimensional vector) to second i+1 mod 8. As demonstrated in the attached audio file, performing iteration starting from random noise yields the entire audio sequence. These two examples demon- strate that both long sequences well as high-dimensional sequences can be stored and recalled using fully connected networks. In Appendix A, we describe these examples in further detail, and also provide example of recurrent network memorizing sequences of text.
2

---

Under review as a conference paper at ICLR 2020
400
00-
730
300
22.5
215
2010
Thiiing
(a) Average log(MSE) between 500 Gaussian Noise inputs iterated 20 times and the closest training example.
(h) Identifying attractors from iterating 1000 test examples from CIFARIO and 1000 examples of Gaussian noise.
Training Example
Top Eigenvalue:
0.9
.77
1.8
0.5
0.9
2.2
0.9
(c) Top eigenvalue of Jacobian at each training image.
Figure 2: Fully connected autoencoder trained on 10 images from CIFARIO.
In the following sections, we will demonstrate that variety of network architectures can be used to implement associative memory when in the overparameterized setting. In Section 2 we provide some background on dynamical systems and describe how to quantify the number of attractors or limit cycles autoencoders and sequence encoders. In Section 3 nonlinearity, and initialization the number of training examples stored attractors for fixed fully connected network topology. In this section, we also show that convolutional autoencoders, similar fully connected autoencoders,
we study the impact of optimizer,
store training examples as attractors. We then conclude, in Section [ with a study of the amount overparameterization (i.e., depth and width) needed for autoencoders store individual examples attractors as compared the amount needed for sequence encoders to store the same data as limit cycles. Interestingly, we show that sequences can be stored limit cycles using smaller networks. Details on all architectures and hyperparameters used are provided in Appendix B.
2 DYNAMICAL SYSTEMS PERSPECTIVE
After training a fully connected autoencoder f on 10 examples from CIFAR10, we observed in Figure Ta inputting random image x the autoencoder yields output f(x) that is close to one of the training examples. Since autoencoders are maps from the feature space to itself, the map can be iterated through compositions (xx) = f(... (f())). Figure 2a shows that iteration of the map started at random image converges to a training example. More precisely, Figure 2a shows that when iterating the convolutional autoencoder on 500 samples of standard Gaussian noise, the distance between the iterated image and the nearest training example decreases monotonically and the reconstruction becomes increasingly accurate. better understand when this iteration converges to training image, we take dynamical systems perspective.
Formally, we let f I IRd denote the function learned by an autoencoder trained on a dataset X ( Jam C IRd We now consider the sequence (x*(z) (()))) where j² denotes composi- tions of and E Rd. When (xx I, then the sequence { trivially converges æ, and we refer to such points x E pd as fixed points of the function f. Since overparameterized autoen- coders interpolate the training data, it holds that (xx(+) ~ f.(() for each training example z(()) E X and hence all training examples are fixed points of f In the following, we formally define what it means for a fixed point to be an attractor and provide a sufficient condition for this property.
As measured by mqaa-squared error (MSE). To ensure (t)( as 200), it is essential that we interpolate to numerical precision i.c. training loss less than 10 when possible.
3

---

Under review as a conference paper at ICLR 2020
Definition. A fixed point x² ER is an attractor if there exists open neighborhood of x², such that for any x in that neighborhood the sequence {f*())) converges to x² as k + cc. The set S of all such points is called the basin of attraction of x².
Proposition 1. A fixed point x² E Ra is an attractor of a differentiable map J if all eigenvalues of the Jacobian of f at x² are strictly less than absolute value. If any of the eigenvalues are greater than 1, x² cannot be an attractor.
Proposition 1 is a well-known condition in the theory of dynamical systems and follows from the Banach Fixed Point theorem (Rudin) 1964). The condition intuitively means that the function f is "flatter" around an attractor x="
Returning to overparameterized autoencoders, where each training example is fixed point, com- puting the top eigenvalue (in absolute value) of the Jacobian for each training example can be used to determine whether the example is an attractor. Figure 2c lists the maximal eigenvalue of the Ja- cobian for each training example in this application; 7 out of the 10 training examples are attractors and can hence be recovered through iteration.
To investigate whether there are additional attractors outside of the training examples, we sample random images and iterate the map until convergence. More precisely, we say that iterative applica- tion of the autoencoder has converged for an input if for some k. (x) then compare the output (x) to each of the training examples and say that has converged one of the training examples if the MSE to the nearest training example is less than 10-2 In Figure 2b we iterated 1000 test examples from CIFAR 10 and 1000 examples of standard Gaussian noise until convergence. The histogram indicates the number of input patterns whose iterations converged to each training example. Note that no instances converged to the training examples 2, 6 and 9, which is consistent with the fact that these training examples are not attractors (the maximal eigenvalue of the Jacobian is larger than 1). Interestingly, every one of the 2000 instances converged to a training example, i.e., attractors were found outside the training set. In addition, while it is conceivable that this iterative map may diverge for some inputs x, empirically the map converged in all instances.
(æ)]l2 < 10 We
We end this section by considering the equivalent of attractor for sequence encoding, namely discrete limit cycles. Since each training example maps to the next example in the sequence, none of the training examples are fixed points after training.
Proposition 2. Let f _2d + IRd be trained to interpolate the training sequence 22)) ...., (")), i.e. f(x!0) ((x((++)) mod n), Then the training sequence {x((1 y forms a discrete limit cycle if the largest eigenvalue of the Jacobian of f" (x(()) is (in absolute value) less than 1 for any i.
This follows directly from Proposition 1 by considering the map f", since ( f" (z). To determine other limit cycles of length n, one can check whether iterating the sequence encoder from random point yields a convergent sequence of period n.
3 IMPACT OF ARCHITECTURE AND OPTIMIZER
We start this section by analyzing how different optimizers, initializations and nonlincarities influ- ence the number of training examples that are stored as attractors for fully connected autoencoders; we discuss convolutional autoencoders at the end of this section. For fully connected networks, we base our analysis of storage and retrieval on a dataset of 100 black and white images from CIFARIC
3.1 MEMORIZATION AND RECALL ACROSS OPTIMIZERS AND NONLINFARITIES
We begin by studying how changing the optimizer and nonlincarity affects the number of training examples stored as attractors. In Figure we provide the number of training examples that become attractors when using popular training algorithms including gradient descent (GD), GD with mo- mentum, GD with momentum and weight decay, RMSprop, and Adam (Ch. 8 of Goodfellow et al.) (2016) with popular nonlinearities including ReLU, Leaky ReLU, SELU, cosid (i.e. cos x), Swish (Xu et al., 2015 Ramachandran et al. 2017 Eger et al. 2018) We used a learning rate of
"e chose to use black and white images in order to make convergence to MSE < 10-8 and the computa- tion of the Jacobian comptationally feasible (for each of the 100 examples the top eigenvalue among all 1024 eigenvalues of the Jacobian needs be computed).
4

---

Under review as a conference paper at ICLR 2020
Act.
ReLU
Leaky ReLU
SELU
Swish
+ 20
Opt
GD
GD
14/100
23/100
10/100
NA
2/100
21/100
Momentum
Momentum
NA'
18/100
22/100
Weight
Decay
RMSprop
97/100
98/100
100/100
49/100
100/100
100/100
Adam
38/100
53/100
30/100
14/100
100/100
100/100
Figure 3: Impact of optimizer and nonlinearity on number of training examples stored attractors. In all experiments, we used a fully connected network with 11 hidden layers, 256 hidden units per layer, and default Py PyTorch initialization. (*) NA indicates that the training error did not decrease below 10 in 1,000,000 epochs.
10 for GD methods and 10 4 for RMSprop and Adam. We used a momentum value of 0.009 and weight decay of 0.0001. Overall, observed that using adaptive methods with learning rate per parameter can significantly alter the number of training examples stored as attractors. Notably, found that RMSprop consistently led to nearly all 100 examples being stored attractors across different nonlinearities. We note that methods using gradient descent were very slow to con- verge; in particular, the loss often did not reduce less than 10-5 in over a million epochs (denoted by NA in Figure
Interestingly, when analyzing whether there are any attractors other than the training examples, we observed that the GD methods tended to have a larger number of attractors outside the training set than adaptive methods. Figures I1a and 11b Appendix show the distribution of attractors found through iteration for SELU networks trained using GD with momenutm and weight decay and RMSprop. Figure ie Appendix provides examples of attractors outside the training set for the network trained with GD momentum weight decay.
With respect to nonlinearities, Figure shows that trigonometrie nonlinearities allow for all 100 training examples to be stored as attractors when using RMSprop or Adam. Figure 12 Appendix C provides the distributions of attractors found through iteration for a subset of these settings. Overall, these plots indicate that the trignometric nonlinearities have fewer attractors outside the training set than the piecewise activation functions (ReLU, Leaky ReLU, SELU). Notably, we did not find any attractors outside the training using the cosid nonlinearity and the Adam optimizer.
3.2 MEMORIZATION AND RECALL ACROSS INITIALIZATIONS
In the following, we study the impact of initialization on the number of training examples that are stored as attractors the learned map. For all experiments in this section, we used a fully connected network with 11 hidden layers and 256 hidden units per layer that is trained using Adam until MSE 10- Thus far, we have been using the default initialization provided from PyTorch (Paszke al. 2017 Under this initialization scheme, each layer's weights are drawn i.i.d. from uniform 1/vh, where h is the number of hidden units in that layer. Since architecture has 256 hidden units per layer, = 0.0625 for but the last layer by default. We note that since use 256 hidden units in each layer, with appropriate constants, this initialization is the same as the Xavier or Kaiming uniform initializers for almost all layers (Glorot & Bengio 2010 He al. 2015).
distribution U[-a, a] with
In Figure we demonstrate the tradeoff between nonlinearity and uniform initialization scheme. We observe that as we increase a, the number of training examples stored as attractors mostly decreases. In fact, for large values of a such as a = .15, not only are fewer training examples attractors for these activation functions, but iteration starting from random examples often diverges. In Figure 13
Unlike our other experiments, we here choose an MSE threshold of 10 5 as almost none non-adaptive optimizers could reduce the loss below 10-8 in under 1, 000, 000 iterations.
We used PyTorch version 0.4; note that the default initialization scheme may be different later versions.
5

---

Under review as a conference paper at ICLR 2020
Act.
ReLU
Leaky
SELU
CO8
sim 10x
Inst
U| U|00.01, 0.001
62/100
78/100
78/100
16/100
26/100
93/100
U| 0.02, 0.02]
43/100
65/100
71/100
20/100
31/100
70/100
U0000.0,.00 0.06
55/100
29/100
U..0
36/100
43/100
13/100
NA"
0|.0.15.0.15 34/100
38/100
13/100
6/100
100/100
NAA
Figure 4: Impact of initialization on number of training examples stored as attractors. all ex- periments, we used fully connected network with 11 hidden layers and 256 hidden units per layer trained using the Adam optimizer (Ir=10-4). (+) NA indicates that the training error did not decrease below 10 in 1,000,000 epochs.
of Appendix C, again provide the distribution of attractors found through iteration for subset of these settings.
3.3 MEMORIZATION AND RECALL IN CONVOLUTIONAL AUTOENCODERS
Thus far, we have provided several examples showing that fully connected autoencoders store training examples as attractors. We now demonstrate that convolutional autoencoders can also ex- hibit this behavior. For this, trained U-Net convolutional autoencoder (detailed in Appendix B) on 10 CIFARIO training examples (Ronneberger et al. 2015 Ulyanov et al. 2017). We performed this analysis only on 10 training images due to the high computational cost of training U-Net con- volutional autoencoder to MSE < 10-8 (roughly 9 hours on 10 training images) and computing the top eigenvalue for each of the 10 examples (~ hour). For this reason, we also focused the analysis in the previous sections on fully connected networks.
Figure 5a shows that, as for fully connected networks, inputting test examples from CIFARIO or Gaussian noise into the trained U-Net convolutional autoencoder leads to images that are visually indistinguishable from the training examples. fact, since the eigenvalues of the Jacobian are less than 1 for each training example, all 10 training examples are attractors. In addition, Figure 5b shows that iterating the map starting in 1000 examples of CIFAR LO test examples or 1000 examples of Gaussian noise did not lead the discovery of any attractors outside the training set.
4 MEMORIZATION AND RECALL OF SEQUENCES
We have thus far considered the impact of optimizer, nonlinearity, and initialization on the number of training examples stored as attractors in autoencoders. In this section, we first study how changing
TV
6.33
Rx
5.22
Ex.7
0.22
Ex.5
5.05
5.00
Dc
604
Es 10 0.00
(a)
(b)
Figure 5: (a) U-Net convolutional autoencoder trained on 10 images from CIFAR1O. When fed Gaussian noise test images from CIFARIO the model outputs images that are visually indistin- guishable from training images. All training images are attractors (EV is eigenvalue). (b) Identifying attractors from iterating the trained autoencoder starting in 1000 test examples from CIFARIO and 1000 examples of Gaussian noise. No attractors are found outside the training set.

---

Under review as a conference paper at ICLR 2020
Width
128
256
512
Depth
0
0
6
12
11
24
(a) Width/Depth 128/1
(b) Width/Depth 128/16
21
56
75
26
83
86
94
31
92
90
99
Figure 6: Impact of width and depth on number of MNIST examples (out of 100) stored attractors. The networks used have SELU activations, default ini- tialization and trained using Adam until MSE < 1
(c) Width//epth 256/16 (d) Width/Depth=512211
Figure 7: Top eigenvalues of Jacobian for all 100 train- ing examples.
the width and depth of a fully connected autoencoder impacts the number of training examples stored attractors. partieular, we observe that the number training examples that stored as attractors increases as depth and width increase. We then study how sequence encoders can store the same data as limit cycles. Interestingly, we find that storing the data as sequence in limit cycle requires less width and depth than storing the images individually. In order to improve computational efficiency when training to a loss of 10- using Adam and computing the eigenvalues of the Jacobian, in this section, we use 100 examples from MNIST as training data. In all of our experiments, we use fully connected SELU networks and fix the random seed prior to training for reproducibility.
Figure 6 we demonstrate how increasing depth and width lead to the storage of almost all training examples as attractors. Note that minimum width of 100 is needed allow for interpolation. Interestingly, when there is sufficient depth for storing at least training example as an attractor, we observe that increasing width leads to the storage of more training examples attractors. Figure 7 shows that the top eigenvalues of the Jacobian for all 100 examples decrease when increasing depth and width.
Instead of training autoencoders to store individual images as attractors, we can instead encode MNIST digits sequences and then use sequence encoding to store the sequences limit cycles. We consider the following example of encoding two 10-digit sequences from MNIST: one counting upwards from digit 0 to 9 and the other counting down from digit digit The training data presented in Figure g Using a fully connected network with depth of 31 layers, 256 hidden units per layer, and SELU activations trained using the Adam optimizer until MSE < 10-8, we find that iterating Gaussian inputs through the encoder leads to recovery of both training sequences. This quite surprising, since even though digits from the two sequences are very similar, iterating the map converges precisely to one of the two sequences (i.e., there are jumps between a digit of one sequence and the other). Applying Proposition we find that the maximal eigenvalues of the
2
3
4
3
6
7
8
9
Training Sequences:
9
8
7
6
5
4
3
2
1
0
O
/
2 3
4
5
6
7
8 1
Random Inpuls:
9
8
7
6
5
3
2
0
Figure 8: MNIST digit training sequences of length 10 stored as limit cycles of an encoder. It- eration from random noise converges to each of the sequences. Our network has 256 hidden units per layer, 31 hidden layers, SELU activations, default initialization, and is trained using the Adam optimizer until MSE < 10-8

---

Under review as a conference paper at ICLR 2020
128
256
512
128
256
512
Depth
Depth
0
6
0
3
10
6
x
/
11
5
10
11
16
9
21
9
10
21
x
/
26
10
26
x
v
(a) Impact width and depth on storing 10 MNIST sequences of length as limit cycles.
(b) Impact width and depth on storing MNIST sequence of length 100 as a limit cycle.
Figure 9: Storing 100 MNIST examples through sequences. The networks used have SELU activa- tions, default initialization and are trained using Adam until MSE < 10-8
Jacobian of the trained encoder composed 10 times is 0.0034 and 0.0033 for the images from the first and second sequence, respectively. Hence, we conclude that these two sequences are indeed limit cycles of the trained encoder.
In Figure 6 we saw that none of the analyzed fully connected networks stored all 100 training examples attractors. Instead of using architecture autoencoder, the 100 examples into sequences and train the network to encode each of the sequences. In Figure 9 investigate whether it easier to store shorter longer sequences limit cycles. In particular, we begin by grouping the 100 training images into 10 sequences of length 10. In Figure Pa we study how width and depth affects the number of sequences that become limit cycles of the network. Interestingly, while none of the analyzed architectures were able to store the 100 training images as attractors the autoencoding setting, already a small network of depth and width 512 can store all training examples as 10 limit cycles of length 10 when encoding the training images as sequences.
instead partition
Given this intriguing observation, we end with the following experiment: Instead of encoding 10 sequences of length 10, we encode all 100 training examples into single sequence of length 100. Consistent with the above findings, Figure 99 shows that an even smaller network can used to store all 100 training images in this manner. Interestingly, the sequence of 100 images is limit cycle a network with only hidden layer and 512 hidden units.
5 CONCLUSION
In this work, we demonstrated experimentally that overparameterized autoencoders and sequence encoders can be used to implement associative memory by storing training examples as attractors or limit cycles. We showed that this remarkable phenomenon is pervasive in range of common architectures using the standard fully connected convolutional layers, across several popular opti- mizers, nonlincarities, and initializations schemes. In fact, by utilizing the architecture and optimizer that provided the most attractors empirically, we were able train network that could store and recover 500 real-valued images with no apparent spurious attractors. We further showed that in fully connected architectures, increasing the depth or width could lead to a greater number of training examples or sequences being stored as attractors or limit cycles. Lastly, we demonstrated that net- works are able to store and retrieve more training examples by training to encode sequences instead of autoencoding individual instances.
While our paper concentrates on the question of implementing associative memory, we employ essentially the same training procedures and neural net architectures as those used in standard su- pervised learning. Thus the surprising fact of the existence and, indeed, the ubiquity of attractors in these maps may shed light on the important question of inductive biases of interpolating neural networks (Belkin et al. 2019).
Finally, another avenue for future exploration is the connection of autoencoding and sequence coding to neural net memory mechanisms in biological systems. fact, this question had been one of the main motivations for the original work Hopfield networks (Hopfilld 1982..
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconeiling modern machine- learning practice and the classical bias-var trade-off. Proceedings of the National Academy 2019.
of Sciences, 156(32):55899113
Steffen Eger, Paul Youssef, and Iryna Gurevych. Is it Time Swish? Comparing Deep Learning Activation Functions Across NLP tasks. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning, volume 1. MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. International Conference on Computer Vision (ICCV), 2015.
John J. Hopfield. Neural Networks and Physical Systems with Emergent Collective Computational Abilities. Proceedings of the National Academy of Sciences, (08)):254-25555 1982.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
W.A. Little. The existence of persistent states in the brain. Mathernatical Biosciences, 19(1):101- 120, 1974.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetio, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. In Inter- national Conference on Learning Representations (ICLR), 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomed- ical Image Segmentation. International Conference on Medical Image Computing and Com- puter Assisted Intervention (MICCAI), 2015.
Walter Rudin. Principles of Mathematical Analysis, volume 3. McGraw-Hill, 1964.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. arXiv:1711.10925.
Deep Image Prior, 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical Evaluation of Rectified Activations in Convolution Network, 2015. arXiv:1505.00s: 550.00085.....
Chiyuan Zhang, Samy Bengio, Moritz Hardt, and Yoram Singer. Identity Crisis: Memorization and Generalization under Extreme Overparametterization,, 2019. 190.49990..04698.

---

Under review as a conference paper at ICLR 2020
APPENDIX A: DATASET DESCRIPTIONS AND MEMORIZATION IN RNNS
Disney "Steamboat Willie" Video Clip. In order to demonstrate that fully connected networks can memorize long sequences, we captured 389 frames of size 128 x 128 from the Disney movie "Steam- boat Wille." The architecture used to memorize and recall the video is provided in Appendix B. We have attached full video titled "disney_sted Litssby_stembbat__ satt__illll__eeeeeeeeeeeeeeeeeee rww_rreeccedddeee from__oise..ne miiieempp" displaying that iteration from random noise leads to recovery of the entire video. Note that iteration from ran- dom noise converges so fast that it is difficult to see that we began the attached video from a random noise image. This is more clearly displayed in Figure lb
Donald Trump Audio Sample. In order to demonstrate that fully connected networks can memorize high dimensional sequences, we captured an audio clip of an 8 second recording from the Donald Trump talking pen. Each second of the audio is roughly 22, 000 frequencies, and we trained a fully connected network to map from the frequencies in second i to second i + 1 mod 8. We demonstrating that
have attached an audio sample titled ttuump_quoth tettttuuuuotteteeeeeeeee recovered_ recovered_froomnoosee iteration from random noise leads to recovery of the entire quote. The full architecture used is provided in Appendix B.
Text Memorization. In order to demonstrate that recurrent neural networks (RNNs) can also mem- orize and recall sequences, we trained a vamilla RNN (whose architecture is detailed in Appendix B) to encode the following sentence from introduction: "Hopfield networks are able to store binary training patterns as attractive fixed points." " When training the RNN, we encode each word using 1-hot representation i.e., since there are 13 words in the sentence, we represent each word with vector of size 13 and place a "1" in the index corresponding to the word. We train such that each word is mapped the next modulo 13 using the Cross Entropy Loss (as is done in practice). Un- like the other settings, RNNs are used to generate new sequences after training by sampling new word from the vector output given a previous word Under our architecture, we find that repeatedly choosing the highest probability word given the previous word consistently outputs the entire train- ing sentence regardless of the number of times this sampling process is repeated. Note that if the number of hidden units per layer is halved our architecture, repeating this sampling process will only output part of the training sequence. Namely, only the sequence "Hopfield patterns as attractive fixed points" will ever be generated.
APPENDIX B: ARCHITECTURES
1. For Figure Ta units per layer, SELU activation, and default PyTorch initialization (random seed 2). We trained using the Adam optimizer until the training loss was less than 10-8
we used a fully connected autoencoder with 31 hidden layers, 256 hidden
2. For Figure 1b we used fully connected architecture with 16 hidden layers, 1024 hidden units per layer, SELU activation, and default Py PyTorch initialization (random seed 2). We trained using the Adam optimizer until the training loss was less than 10-7
3. For the audio example in Section I we used a network with 36 hidden layers, 15 hidden units per layer, SELU activation, and default PyTorch initialization (random seed 2). We trained using the Adam optimizer until the training loss was less than 10-12 12 (we need high precision reconstructions to avoid noisy artifacts the audio file).
4. For the RNN example in Appendix A, we used two fully connected networks with 31 hidden layers and 128 hidden units per layer such that one constructs the next hidden state and the other constructs the next output. We used SELU activations throughout and trained using Adam with learning rate 10 to minimize the Cross Entropy Loss.
5. To encode 500 black and white CIFARIO images as attractors, we used a fully connected architecture with 11 hidden layers. 1024 hidden units per layer, cosid activation, and default PyTorch initialization (random seed 2). We trained using the Adam optimizer until the training loss was less than 10-
6. For the convolutional autoencoder used for Figure 5a we display the architecture in Figure TO
We again used the default PyTorch initialization (random seed 2) and trained using Adam until the MSE was less than 10 All of our filters have a kernel size of 3.
This process is usually started from inputting the all zero vector.
10

---

Under review as a conference paper at ICLR 2020
x5
Conv 256 Stride-2
Conv 256
Conv 256 Stride=1 Leaky
Conv Stride-1 Leaky
Billnear Upsampling
Stride-1
Leaky
Leaky
Figure 10: Convolutional autoencoder architecture used in our experiments.
7. We used random seed of for all experiments in Figures
As these experiments were computationally expensive, we could not run all of them across several random seeds. However, for a few of the entries in each table, note that other random seeds gave similar results.
APPENDIX C: DISTRIBUTION ATTRACTORS DISCOVERED THROUGH ITERATION
In this section, we provide series of histograms indicating the attractors found by iterating test examples from black and white versions of CIFARIO images and from Gaussian random noisc. The histograms also indicate an estimate for the size of the basin of attraction for each attractor.
In Figure [T provide histograms for two different optimization methods indicating the distri- bution of attractors found by iterating 1000 test examples from CIFAR10 and 1000 examples of Gaussian noise. To compare different optimization methods, used architecture with SELU nonlinearity and the default PyTorch initializer. The histograms demonstrate that adapative opti- mizers such RMSprop lead the storage of training examples as attractors. Furthermore, when using RMSProp we did not observe any attractors outside the training set. On the other hand, identified attractors outside the training when using GD momentum + weight decay, and examples of these attractors are presented in Figure Tcc
In Figure 12
provide histograms for four different nonlinearities indicating the distribution of tractors found by iterating 1000 test examples from CIFAR10 10 and 1000 examples of Gaussian noise. To compare nonlinearities, we used an architecture that was trained using the Adam optimizer and that was initialized using the default PyTorch initializer. The histograms indicate that trignometric nonlinearities tend to store more training examples as attractors while also having fewer spurious attractive states. Interestingly, the cosid nonlinearity stored all 100 training examples as attractors and had 0 other observed attractors outside the training set.
provide histograms for four different initializations indicating the distribution of attractors found by iterating 1000 test examples from CIFARIO and 1000 examples of Gaussian noise. To compare different initializations. selected architecture that trained using the Adam optimizer and used the SELU nonlinearity. The histograms indicate that smaller variance
In Figure 13
Cavoia
200
(b)
Figure 11: Identifying attractors from random inputs across optimizers for the SELU nonlincarity. (a) SELU network trained using GD momentum + weight decay; observed attractors outside training set. (b) SELU network trained using R.Sppopp 0 observed attractors outside training set. (c) Examples of attractors outside training set for SELU network trained using GD momentum weight decay.
11

---

Under review as a conference paper at ICLR 2020
Co tieenttee
Goazzann holto
30
200
ep
Tharing
(a)
(h)
200
Manue
200
0P
(c)
(d)
Figure 12: Identifying attractors from random inputs across nonlinearities.Ga Leaky RELU network trained using Adam; 59 observed attractors outside trainset. (b) SELU network trained using Adam; 17 observed attractors outside trainset. (c) Network with x+ 5 outside trainset. (d) Network with cos activations; 0 observed attractors outside trainset.
10z activations; 9 observed attractors
initializations tend to store more training examples as attractors while also having fewer spurious attractive states.
In Figure 14 we provide a histogram indicating the distribution of attractors found by iterating 10,000 test examples from CIFARIO and 10,000 examples of Gaussian noise for a fully connected network trained to store 500 examples. We selected an architecture that was trained using the Adam optimizer, used the cosid nonlinearity, was initialized using the default PyTorch initialization, and that had 11 hidden layers with 1024 hidden units per layer. Notably, we did not observe any other attractor states from this network, even though we iterated 20,000 examples.
12

---

Under review as a conference paper at ICLR 2020
DrRAE 100
0914
100000 www.ooo
(a)
1350
(c)
(d)
Figure 13: Identifying attractors from random inputs across initializations. (a) SELU network trained with Adam with initialization U.00.01 0.01]; observed attractors outside the training set. (b) SELU network trained with Adam with initialization U[-0.02, 0.02] ; 8 observed attractors outside the training set. (c) SELU network trained with Adam with initialization U[-0.05, 0.05]. 12 observed attractors outside the training set. (d) SELU network trained with Adam with initialization U1-0.1, 0.1]; almost no inputs converged to training examples within 1000 iterations.
2030
1000
FX 12.
200
other
Trahiing Formple Inder
Figure 14: Iterating 10,000 test examples and 10,000 examples of random noise does not lead to any observable attractors outside the training set.
13


Reviews:
[{"review_id": "rJxPhC1FYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies a phenomenon of unusual memorisation in deep overparametrized neural networks.\nAuthors observe that, if an auto-encoder overfits to machine precision on a number of images, they can be reliably decoded from random noise and that it is even possible to memorise this way a sequence of images.\nEssentially, images from such a training set become attractors for the mapping defined by the auto-encoder.\nThe impact of network size, nonlinearity and initialization is studied and, quite surprisingly, very unusual trigonometric non-linearities performed the best.\n\nI find the studied phenomenon rather interesting and the analysis well-performed, but I am not sure how practically important is this work.\nFirst, I would argue that to call the overfit auto-encoder a function associative memory, it must be able to retrieve stored images not just from random noise, but from a somehow distorted or partially known version. \nOtherwise we are just left with a ridiculously large network that can only recall a handful of images we could store in the raw format using much less numbers.\nSecond, training until convergence takes prohibitively long time.\n\nI would be also interested to at least an interesting discussion, if not an answer, to the question of why and how exactly trained images become attractors. \n\nIn terms of novelty, it feels like Zhang et al, 2019 already studied a very similar phenomenon and the submitted paper does not add much to understanding of memorisation in neural networks. However, memorization of sequences was indeed a surprise. \n\nOverall, I do not have a strong opinion on rejecting the paper, it just feels like more work in this direction will make the paper significantly better. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJx5RNjiqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper explores the properties of an auto-encoder to behave as an associative memory retrieval mechanism. The authors show really interesting results where they are able to retrieve a small subset of encoded images (mnist) by giving the autoencoder random noise. They also show they can retrieve full videos by giving the autoencoder the output frame from the previous timestep.\n\nThe overall problem is a really interesting one which is to try to develop associative memory, retrieval models. \n\nDecision:\nReject\n\nReasons:\n1. Although the work is interesting, the only related work the authors cover is hopfield networks. A cursory search indicates that this has been done before (k. Niki IEEE, Trischler 2016, M.A.Kramer 1992).\n\nImprovement:\n1. A more thorough discussion of related work would be helpful.\n2. A direct qualitative comparison to related work would also be helpful.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJlgnPFiKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically demonstrates that DNNs can be trained to be identity mappings for small quantities of samples. It also demonstrates that for many parameterizations, these identity DNNs also have a small number of attractors, iterative fixed points, and can also learn short circular sequences of examples.\n\nThe paper is well written and easy to understand, although the presentation could be improved a bit (see comments). Its contents aren't particularly novel in terms of ideas, but they investigate memorization and attractors much further than previous studies. \nIn a way, the memorization results are unsurprising. We know that DNNs can memorize perfectly, including sequences, so it is natural that by increasing capacity, at some point they should be able to memorize entire images (in fact this is what Zhang et al. (2019)'s Figure 1 appears to be showing). \nThe more novel and surprising aspect of this is that DNNs would learn such strong (and so few) attractor basins. The fact that deep autoencoders could have attractors centered on the training points has been postulated before (see [1]), but this work makes a stronger case for it.\n\nA crucial aspect that is missing from this paper in order for me to give in an accept is that there is very little about how this paper positions itself in the current literature. There could be much more discussion about related work, and much more discussion about the impacts of these findings.\n\nI have given this paper a 'weak reject' mark but I think with some work this paper could be of interest to many. To reiterate, I am unable to see anything wrong with this paper, but at the same time I am unable to see how impactful these findings are.\n\n\nDetailed comments:\n- It's interesting that DNNs can implement associative memory, but what is the cost of doing that? Should we be using that in practice? Since there is no sense of how costly the presented experiments are, it is hard to tell.\n- Again, these results are interesting, but after some time pondering about it, I can't really convince myself that knowing the results of this paper will be beneficial to future research. That being said, there are many areas of Machine Learning that I am unfamiliar with. It should be part of the paper to familiarize readers with areas where these results could be impactful.\n- \"the function interpolates the training images\" not sure what this means. Interpolation means making a prediction for a point `u` that is \"between\" two points `x,y` with known values\n- \"black and white\" should be \"grayscale\" if values are in [0,1]\n- Figure 2b is interesting, but I wonder what happens if e.g. a perturbed version of e.g. Example 6 is fed. Presumably since Example 6 is not an attractor (Jacbian with an eigeinvalue > 1), it should converge to another example.\n- Figure 2b's caption numbers, which say you use 1k example, to not correspond to numbers earlier in the text, which say you use 10k examples.\n- \"Since overparameterized autoencoders interpolate the training data\", again this is a fairly important assumption and it needs to be defined very clearly, because it could mean many things.\n- \"it is essential that we interpolate to numerical precision\", I don't think you are using the word \"interpolate\" correctly, do you mean \"inference\"? \"train\"?\n- Adam citation should be \"Adam: A Method for Stochastic Optimization, Diederik P. Kingma, Jimmy Ba\", not Goodfellow et al., RMSprop should also have a citation, Hinton et al. 2012\n- ReLU citation should be \"Rectified linear units improve restricted Boltzmann machines, Nair & Hinton\", Leaky ReLU should be Maas et al 2013, SELU should be Klambauer et al. 2017.\n- The combination of section 3.1 and Figure 3 doesn't make it clear if models trained with Adam and RMSprop have weight decay or not. Can you clarify?\n- \"Note that a minimum width of 100 is needed to allow for interpolation.\" Again I think you mean \"learning\" rather than \"interpolation\".\n- You say that you trained black and white images, but all the images of CIFAR10 in the figures are colored, including the inputs and outputs. Can you clarify why?\n- You might be interested in [2], which is much older work about perceptrons, but still relevant to what is studied here.\n- The linked supplemental material gives a 404 for me. I replicated the MNIST Figure 6 experiment. I was unable to replicate exactly your results but they were similar enough. In particular, the activation function choice seems to be critical.\n\n[1] The Potential Energy of an Autoencoder, Hanna Kamyshanska, Roland Memisevic\n[2] Basins of Attraction in a Perceptron-like Neural Network, Werner Krauth, Marc Mezard, Jean-Pierre Nadal\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that overparameterized autoencoders can be trained to memorize a small number of training samples, which can be retrieved via fixed point iteration. After rounds of discussion with the authors, the reviewers agree that the idea is interesting and overall quality of writing and experiments is reasonable, but they were skeptical regarding the significance of the finding and impact to the field and thus encourage studying the phenomenon further and resubmitting in a future conference. I thus recommend rejecting this submission for now.
Decision: Reject