Associative Memories via Predictive Coding

1 Introduction . Throughout our lives , we learn a huge number of associations between concepts : the taste of a particular food , the meaning of a gesture , or to stop when we see a red light . Every time we acquire new information of this kind , it gets stored in our long-term memory , situated in distributed networks of brain areas [ 1 ] . In particular , visual memories are stored in a hierarchical network of visual and associative areas [ 2 ] . These regions learn progressively more abstract representations of visual stimuli , so they participate in both perception and memory as each area memorizes relationships present in their inputs [ 3 ] . Accordingly , early visual areas learn common regularities present in the stimuli [ 4 ] , while at the top of this hierarchy , associative areas ( such as hippocampus , entorhinal cortex , and perirhinal cortex ) store the relationships between extracted features , which encode an entire stimulus or episode [ 5 ] . The memory system of the brain is able to both recall complex memories [ 1 , 6 ] , and use them to generate predictions to guide behavior [ 7 ] . Learning in these associative memories shapes our understanding of the world around us and builds the foundations of human intelligence . Building models that are able to store and retrieve information has been an important direction of research in artificial intelligence . Particularly , such models include ( auto ) associative memories ( AMs ) , which allow for the storage of data points and their contents-based retrieval , i.e. , for retrieving a stored * Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Inference Memory Layer Sensory Layer Figure 1 : Generative PCN with two layers . Each line linking neurons denotes a pair of connections ( excitatory in the left direction , and inhibitory in the right direction ) . At t = 0 , the value nodes of the output layer are fixed to the vector representing the pixels of the figure . Then , during the learning phase , both weight parameters and value nodes are updated . data point s from a corrupted or a partial variant of s. One way to realize AMs is to store data points as attractors , so that they can be easily recovered via an energy minimization process when presenting their corrupted variants [ 8 , 9 ] . Existing AMs include Hopfield networks [ 8 ] and modern Hopfield networks ( MHNs ) [ 10 ] . The latter are one-shot learners , which are able to store exponentially many memories , and to perfectly retrieve them . However , the retrieval process often fails when dealing with complex data , such as natural images . Recent works have shown that overparametrized autoencoders ( AEs ) are excellent AMs as well . Particularly , when training an AE to generate a specific point s when s itself is presented as an input , it gets stored as an attractor [ 11 ] . In this work , we present a novel AM model that is based on an energy-based generative approach . This AM model differs from Hopfield networks , as it is trained using predictive coding ( PC ) , which is a biologically plausible learning algorithm inspired by learning in the visual cortex [ 4 ] . The idea that PC may naturally be related to AMs is inspired by recent works showing that the generative neural architecture that connects the hippocampus to the neocortex is based on an error-driven learning algorithm , which can be interpreted with a PC framework [ 6 , 12 ] . From a machine learning perspective , predictive coding networks ( PCNs ) are able to perform both supervised and unsupervised tasks with a high accuracy [ 4 , 13 ] , and are completely equivalent to backpropagation when trained with a specific algorithm [ 14–16 ] . We show that the new AM model is not only interesting from a neuroscience perspective , but it also outperforms popular AM models when it comes to the storage and retrieval of complex data points . Our results can be briefly summarized as follows : • We define generative PCNs and empirically show that they store training data points as attractors of their dynamics by demonstrating that they can restore original data points from corrupted versions . In an extensive comparison of the new AM model against standard AEs , the new model considerably outperforms AEs ( in storage capacity , retrieval accuracy , and robustness ) when tested on neural networks of the same size . • The reconstruction of incomplete data points is a challenging task for AMs . Our model naturally solves the task of reconstructing complex and colored images with a surprisingly high accuracy . We also test our model on ImageNet , perfectly reconstructing single pictures even after removing all but 1/8 of the original image . We then show that , to increase the overall capacity and retrieval robustness of the model , it suffices to add additional layers . We also compare our model against MHNs , showing that it significantly outperforms them in the aforementioned tasks . 2 Generative predictive coding networks . We now briefly recall predictive coding networks ( PCNs ) , and we introduce generative PCNs , which are the underlying neural model for the novel AMs introduced in the subsequent section . Deep neural networks have a multi-layer structure , where each layer is formed by a vector of neurons [ 17 ] . While in standard deep learning the goal is to minimize the error on a specific layer , PC defines an error in every layer of the network , minimized by gradient descent on a global energy function [ 4 ] . Particularly , let M be a PCN with L−1 fully connected layers of dimension n , followed by a fully connected layer of dimension d. We call the d-dimensional layer sensory layer ( indexed as layer 0 ) , which biologically corresponds to sensory neurons ( see Fig . 1 ) . We call the most internal layer ( layer L ) memory , which is equipped with an n-dimensional memory vector b . Every layer l contains value nodes xli , t , and every pair of consecutive layers is connected via weight matrices θ̄ l , which represent the synaptic weights between neurons of different layers . The value nodes , the Algorithm 1 : Learning to generate s̄ with IL Require : x0 is fixed to s. 1 : for t = 0 to T do 2 : for each neuron i in each level l do 3 : update xli , t to minimize Et via Eq . ( 3 ) 4 : end for 5 : end for ; 6 : update each θl+1i , j and bi to minimize Et via Eqs . ( 4 ) and ( 5 ) . weight matrices , and the memory vector are all trainable parameters of the model . The signal passed from layer l + 1 to layer l , called prediction µ̄lt , is computed as follows : µli , t = { ∑nl+1 j=1 θ l+1 i , j f ( x l+1 j , t ) if 0 ≤ l < L b if l = L , ( 1 ) where f is a non-linear activation function . To conclude , the difference between the value x̄lt and their predictions µ̄lt is the error ε l i , t = x l i , t − µli , t . We now describe how PCNs are trained . To do this , we explain one iteration of a training algorithm , called inference learning ( IL ) , that is divided into an inference phase and a weight update phase . Inference : Only the value nodes of the network are updated , while both the weight parameters and the memory vector are fixed . Particularly , the value nodes are modified via gradient descent to minimize the global error of the network , expressed by the following energy function Et : Et = 1 2 ∑ i , l ( ε l i , t ) 2 . ( 2 ) Assume that we train a generative PCN on a training point s̄ ∈ Rd . To do this , the value nodes of the sensory layer are fixed to the training point s̄ , and are never updated . Thus , the error on every neuron of the sensory layer is equal to ε0i , t = si − µ0i , t . The process of minimizing Et by modifying all xli , t leads to the following changes in the value nodes : ∆xli , t = { γ · ( −εli , t + f ′ ( xli , t ) ∑nl−1 k=1 ε l−1 k , t θ l k , i , t ) if 0 < l ≤ L 0 if l = 0 , ( 3 ) where γ is the integration step , which is a constant determining by how much the activity changes in each iteration . The computations in Eqs . ( 1 ) and ( 3 ) are biologically plausible , as they have a neural implementation that can be realized in a network with value nodes xli , t and error nodes ε l i , t [ 4 ] , as shown in Fig . 1 . The inference phase works as follows : starting from a given configuration of the value nodes x̄0 , inference continuously upates the value nodes according to Eq . ( 3 ) until it has converged . We call the configuration of the value nodes at convergence x̄T , where T is the number of steps needed to reach convergence ( in practice , it is a fixed large number ) . Weight Update : When the value nodes of the sensory layer are fixed to an input signal s̄ , inference may not be sufficient to reduce the total energy to zero . Hence , to further decrease the total error , a single weight update is performed : both the weight matrices and the memory vector are updated by gradient descent to minimize the same objective function Et , and behave according to the following equations , where α is the learning rate . Particularly , the derived update rule is the following : ∆θl+1i , j = −α · ∂ET /∂θ l+1 i , j = α · ε l i , T f ( x l+1 j , T ) , ( 4 ) ∆bi = −α · ∂ET /∂bi = −αεLi , T . ( 5 ) The phases of inference and weight update are iterated until the total energy Et reaches a minimum . This algorithm learns a dataset by using only local computations , which minimize the same energy function . Fig . 1 gives a graphical representation of generative PCNs , while the pseudocode is shown in Alg . 1 . Detailed derivations of Eqs . ( 3 ) and ( 5 ) are in the supplementary material ( and in [ 13 ] ) . 3 Predictive coding for associative memories . So far , we have shown how PCNs can perform generative tasks . We now show how generative PCNs can be used as associative memories , i.e. , how the model stores the data points that it is trained on , and how these data points can be retrieved when presenting corrupted versions to the network , returning the most similar stored data point . Let s̄ be a training data point , and M be the PCN considered above , already trained until convergence to generate s̄ . Moreover , assume that M makes the total energy converge to zero at iteration T . At this point , the energy function defined on the value nodes has a local minimum x̄ in which the value nodes of the sensory layer are equal to the entries of s̄ . Note that x̄ is actually an attractor of the dynamics of Et : when given a configuration that is not a local minimum , inference will update the value nodes until the total energy reaches a minimum . If this configuration lies in a specific neighborhood of x̄ , inference will converge to x̄ . So , given a dataset , we obtain an AM of the dataset if all the training points are stored in the above way as attractors . The above can be used to retrieve stored data points s̄ : given a corrupted version c̄ ∈ Rd of s̄ , one can retrieve s̄ as follows . First , we set the value nodes of the sensory layer to the corrupted points , i.e. , x̄0t = c̄ for the whole process . Then , we run inference until convergence and save the prediction µ̄ 0 T of the sensory layer . If the original data point was stored as an attractor , we expect the prediction µ̄0T to be a less corrupted version of it . Let F : Rd→Rd be the function that sends c̄ to µ̄0T just described , and summarized in Fig . 2 . Many iterations of this function allow to retrieve the stored data point . Hence , summarizing the above , training points are stored in the memory vector b̄T and the weight parameters , and what the algorithm does to retrieve them is simply the inference phase of PCNs . Since visual memories are stored in hierarchical networks of brain areas , PC could be a highly plausible algorithm to better understand how memory and prediction work in the brain . To experimentally show that generative PCNs are AMs , we trained a 2-layer network with ReLU non-linearity on a subset of 100 images of Tiny ImageNet and CIFAR10 . After training , we presented the model with a corrupted variant ( by adding Gaussian noise ) of the training set . We then used the PCNs to reconstruct the original images from the corrupted ones . The experimental results confirm that the model is able to retrieve the original image , given a corrupted one . The obtained reconstructions for the Tiny Imagenet dataset ( the most complex one , as each data point consists of 3 × 64 × 64 pixels ) are shown in Fig . 4 . We now provide a more comprehensive analysis , which studies the capacity of generative PCNs when changing the number of data points and parameters . Experiments : We trained 2-layer PCNs with ReLU non-linearity and hidden dimension n ∈ { 512 , 1024 , 2048 } on subsets of the aforementioned datasets of cardinality N = { 100 , 250 , 500 , 1000 } . Every model is trained until convergence , and all the images are retrieved as described in Section 3 . To provide a numerical evaluation , an image is considered recovered when the mean squared error between the original image and the recovered image is less than 0.005 . To compare our results against a standard baseline , we also trained 3-layer autoencoders ( AEs ) with the same hidden dimension on the same task , and compared the results . Note that the number of parameters of a 2-layer PCN is smaller than the one of a 3-layer AE with the same hidden dimension . This follows , as the additional layer ( input layer , not needed in generative PCNs ) almost doubles the number of parameters in some cases . Further details about the experiments and used hyperparameters are given in the supplementary material . Results : The analysis shows that our model is able to store and retrieve data points even when the network is not overparametrized ( see Fig . 3 ) . AEs trained with BP did not perform well : AEs with less than 1024 hidden neurons always failed to restore even a single data point on the Tiny Imagenet dataset , and very few on other ones . The performance of AEs with 2048 hidden neurons were always worse than PCNs with 512 hidden neurons . This shows that overparametrization is essential for AEs to perform AM tasks , and that our proposed method offers a much more network-efficient alternative . In terms of capacity , 2-layer PCNs with 512 hidden neurons correctly store datasets of 250 images of both CIFAR10 and Tiny ImageNet , and networks with 2048 hidden neurons always store and retrieve all the presented datasets . As typical for AMs , small models trained on large datasets fail to store data points , as the space of parameters is not large enough to store each data point as an independent attractor . Our model is no different : PCNs with 512 hidden units are able to store almost 500 Tiny ImageNet images when trained on datasets of that size , but fail to store more than 200 when trained on larger ones .

Summary: In this paper, the authors explore the memorization capability of a Predictive Coding (PC) model. Following training on small subsets of images taken from popular datasets (CIFAR, ImageNet), the model is probed on a variety of recall tasks that require to reconstruct images corrupted by noise or to recover entire images from partial evidence. Recall performance is compared against that of a multi-layer autoencoder trained with error backpropagation.