Overparameterized Neural Networks Can Implement Associative Memory

1 INTRODUCTION . Training neural networks to act as a model of associative memory is a problem dating back to the introduction of Hopfield networks ( Little , 1974 ; Hopfield , 1982 ) . Hopfield networks are able to store binary training patterns as attractive fixed points ; thus , the network allows for recall of the training patterns given corrupted inputs . In this work , we demonstrate that modern overparameterized deep neural networks trained using standard optimization methods also provide a method for memorization and retrieval by storing training instances as attractors . Moreover , we show that through autoencoding and sequence encoding , the training examples can be recovered from random inputs to the trained network . We are not aware of any observation of this phenomenon in the literature , with the exception of Zhang et al . ( 2019 ) , which studies the case of memorizing a single image . Autoencoding . We begin with the example of an autoencoder neural network implementing a family of continuous functions in the space F = { f : Rd → Rd } . Given training examples { x ( i ) } ni=1 , gradient descent is used to minimize the following autoencoder objective : argmin f∈F ∑n i=1 ‖f ( x ( i ) ) − x ( i ) ‖22 After training in the overparameterized regime , we obtain a function f satisfying f ( x ( i ) ) ≈ x ( i ) for all i , i.e. , the function interpolates the training images . In Figure 1a , we analyze the function that is learned after training a fully connected autoencoder ( architecture detailed in Appendix B ) on 10 examples from the CIFAR10 dataset ( Krizhevsky , 2009 ) . Note that inputting random images causes the autoencoder to produce outputs that are visually similar to training examples . While the ability to recall training examples from random inputs can be inspected visually as in Figure 1a , we can also quantify the number of training examples that can be recalled by viewing autoencoders from a dynamical systems perspective . Namely , since an autoencoder is a map from the feature space to itself , we can iteratively apply the autoencoder to the output starting from any input and investigate whether the sequence of iterates converges to a training example . A training example is called an attractor , when there exists a set of instances with non-zero measure for which iterating the trained autoencoder map converges to the training example . This happens if the largest eigenvalue ( in absolute value ) of the Jacobian of the autoencoding map at the training example is less than 1 ( see Section 2 ) . While the network trained in Figure 1a already outputs an image that is visually indistinguishable from a training example in one iteration , further iterations of the autoencoder result in increasingly more accurate reconstructions of the training examples ( see Figure 2a ) . As we demonstrate throughout this paper , overparameterized autoencoders store training examples as attractors , and so training examples can be recovered through iterating a trained autoencoder on random input . We note that for a fixed point to be an attractor , all eigenvalues of the Jacobian matrix do not exceed one in absolute value . This is a highly restrictive condition as the number of these eigenvalues is equal to the dimension of the space . Thus , we do not expect a fixed point of an arbitrary high-dimensional map to be an attractor . In that regard , the fact that deep neural networks produce attractors so readily is rather surprising . Moreover , even if an autoencoder stores training examples as attractors , there could be additional spurious attractors outside the training set . Interestingly , in this paper , we present several examples of autoencoders for which we could not discover any attractors outside the training set through iteration from random instances . In particular , we now provide an example of a fully connected autoencoder that stores 500 real-valued training examples as attractors with no other observable attractors . Memory and Recall of 500 Images as Attractors of an Autoencoder . We train a fully connected autoencoder ( architecture detailed in Appendix B ) on 500 black and white CIFAR10 training images until the training loss is less than 10−8 . We find that all 500 examples are stored as attractors since the top eigenvalue of the Jacobian for all of these examples is less than 1 . Lastly , upon iterating 10,000 black and white examples from CIFAR10 and 10,000 examples of Gaussian random noise , we are unable to find any attractors outside the training set . Sequence Encoding . Remarkably , this phenomenon extends from encoding single instances ( such as images ) to sequences . Instead of autoencoding , where the loss is the reconstruction loss on single images , we train f such that f ( x ( i ) ) ≈ f ( x ( ( i+1 ) mod n ) ) . In Figure 1b we trained a network to encode 389 frames of size 128 × 128 from the Disney film “ Steamboat Willie ” by mapping frame i to frame i + 1 mod 389 . As is shown in the attached video and in Figure 1b , inputting random noise to the trained network f and iterating the network yields the original video . In fact , after a few iterations from a random image , one of the frames of the original video is recovered to numerical precision , and since the network is trained to map i to frame i+1 mod 389 , once one of the frames of the original video is recovered , the remaining frames are all recovered . Interestingly , we show in Section 4 that encoding a sequence of instances in this manner appears to provide a more efficient mechanism for storage and retrieval than autoencoding each instance in the sequence separately . We can similarly apply the sequence encoding objective to audio . Given an 8 second sample audio , we trained a fully connected network to map each second i of frequencies ( represented as a 22k dimensional vector ) to second i+1 mod 8 . As demonstrated in the attached audio file , performing iteration starting from random noise yields the entire audio sequence . These two examples demonstrate that both long sequences as well as high-dimensional sequences can be stored and recalled using fully connected networks . In Appendix A , we describe these examples in further detail , and also provide an example of a recurrent network memorizing sequences of text . In the following sections , we will demonstrate that a variety of network architectures can be used to implement associative memory when in the overparameterized setting . In Section 2 , we provide some background on dynamical systems and describe how to quantify the number of attractors or limit cycles of autoencoders and sequence encoders . In Section 3 , we study the impact of optimizer , nonlinearity , and initialization on the number of training examples stored as attractors for a fixed fully connected network topology . In this section , we also show that convolutional autoencoders , similar to fully connected autoencoders , can store training examples as attractors . We then conclude , in Section 4 , with a study of the amount of overparameterization ( i.e. , depth and width ) needed for autoencoders to store individual examples as attractors as compared to the amount needed for sequence encoders to store the same data as limit cycles . Interestingly , we show that sequences can be stored as limit cycles using smaller networks . Details on all architectures and hyperparameters used are provided in Appendix B . 2 DYNAMICAL SYSTEMS PERSPECTIVE . After training a fully connected autoencoder f on 10 examples from CIFAR10 , we observed in Figure 1a that inputting a random image x to the autoencoder yields an output f ( x ) that is close∗ to one of the training examples . Since autoencoders are maps from the feature space to itself , the map can be iterated through k compositions fk ( x ) = f ( . . . ( f ( x ) ) ) . Figure 2a shows that iteration of the map started at a random image converges to a training example . More precisely , Figure 2a shows that when iterating the convolutional autoencoder on 500 samples of standard Gaussian noise , the distance between the iterated image and the nearest training example decreases monotonically and the reconstruction becomes increasingly accurate . To better understand when this iteration converges to a training image , we take a dynamical systems perspective . Formally , we let f : Rd → Rd denote the function learned by an autoencoder trained on a dataset X = { x ( i ) } ni=1 ⊂ Rd . We now consider the sequence { fk ( x ) } k∈N where fk denotes k compositions of f and x ∈ Rd . When f ( x ) = x , then the sequence { fk ( x ) } trivially converges to x , and we refer to such points x ∈ Rd as fixed points of the function f . Since overparameterized autoencoders interpolate the training data , it holds that f ( x ( i ) ) ≈ x ( i ) for each training example x ( i ) ∈ X and hence all training examples are fixed points of f .† In the following , we formally define what it means for a fixed point to be an attractor and provide a sufficient condition for this property . ∗As measured by mean-squared error ( MSE ) . †To ensure f ( x ( i ) ) ≈ x ( i ) , it is essential that we interpolate to numerical precision i.e . training loss less than 10−8 when possible . Definition . A fixed point x∗ ∈ Rd is an attractor if there exists an open neighborhood of x∗ , such that for any x in that neighborhood the sequence { fk ( x ) } k∈N converges to x∗ as k → ∞ . The set S of all such points is called the basin of attraction of x∗ . Proposition 1 . A fixed point x∗ ∈ Rd is an attractor of a differentiable map f if all eigenvalues of the Jacobian of f at x∗ are strictly less than 1 in absolute value . If any of the eigenvalues are greater than 1 , x∗ can not be an attractor . Proposition 1 is a well-known condition in the theory of dynamical systems and follows from the Banach Fixed Point theorem ( Rudin , 1964 ) . The condition intuitively means that the function f is “ flatter ” around an attractor x∗ . Returning to overparameterized autoencoders , where each training example is a fixed point , computing the top eigenvalue ( in absolute value ) of the Jacobian for each training example can be used to determine whether the example is an attractor . Figure 2c lists the maximal eigenvalue of the Jacobian for each training example in this application ; 7 out of the 10 training examples are attractors and can hence be recovered through iteration . To investigate whether there are additional attractors outside of the training examples , we sample random images and iterate the map until convergence . More precisely , we say that iterative application of the autoencoder has converged for an input x if for some k , ‖fk ( x ) −fk+1 ( x ) ‖2 < 10−8 . We then compare the output fk ( x ) to each of the training examples and say that it has converged to one of the training examples if the MSE to the nearest training example is less than 10−2 . In Figure 2b , we iterated 1000 test examples from CIFAR10 and 1000 examples of standard Gaussian noise until convergence . The histogram indicates the number of input patterns whose iterations converged to each training example . Note that no instances converged to the training examples 2 , 6 and 9 , which is consistent with the fact that these training examples are not attractors ( the maximal eigenvalue of the Jacobian is larger than 1 ) . Interestingly , every one of the 2000 instances converged to a training example , i.e. , no attractors were found outside the training set . In addition , while it is conceivable that this iterative map may diverge for some inputs x , empirically the map converged in all instances . We end this section by considering the equivalent of an attractor for sequence encoding , namely discrete limit cycles . Since each training example maps to the next example in the sequence , none of the training examples are fixed points after training . Proposition 2 . Let f : Rd → Rd be trained to interpolate the training sequence x ( 1 ) , . . . , x ( n ) , i.e. , f ( x ( i ) ) = f ( x ( ( i+1 ) mod n ) . Then the training sequence { x ( i ) } ni=1 forms a discrete limit cycle if the largest eigenvalue of the Jacobian of fn ( x ( i ) ) is ( in absolute value ) less than 1 for any i . This follows directly from Proposition 1 by considering the map fn , since x ( i ) = fn ( x ( i ) ) . To determine other limit cycles of length n , one can check whether iterating the sequence encoder from a random point yields a convergent sequence of period n .

Summary: This paper explores the properties of an auto-encoder to behave as an associative memory retrieval mechanism. The authors show really interesting results where they are able to retrieve a small subset of encoded images (mnist) by giving the autoencoder random noise. They also show they can retrieve full videos by giving the autoencoder the output frame from the previous timestep.