Large Associative Memory Problem in Neurobiology and Machine Learning

1 INTRODUCTION . Associative memory is defined in psychology as the ability to remember ( link ) many sets , called memories , of unrelated items . Prompted by a large enough subset of items taken from one memory , an animal or computer with an associative memory can retrieve the rest of the items belonging to that memory . The diverse human cognitive abilities which involve making appropriate responses to stimulus patterns can often be understood as the operation of an associative memory , with the “ memories ” often being distillations and consolidations of multiple experiences rather than merely corresponding to a single event . The intuitive idea of associative memory can be described using a “ feature space ” . In a mathematical model abstracted from neurobiology , the presence ( or absence ) of each particular feature i is denoted by the activity ( or lack of activity ) of a model neuron vi due to being directly driven by a feature signal . If there are Nf possible features , there can be only at most N2f distinct connections ( synapses ) in a neural circuit involving only these neurons . Typical cortical synapses are not highly reliable , and can store only a few bits of information1 . The description of a particular memory requires roughly Nf bits of information . Such a system can therefore store at most ∼ Nf unrelated memories . Artificial neural network models of associative memory ( based on attractor dynamics of feature neurons and understood through an energy function ) exhibit this limitation even with precise synapses , with limits of memory storage to less than ∼ 0.14Nf memories ( Hopfield , 1982 ) . 1For instance , a recent study ( Bromer et al. , 2018 ) reports the information content of individual synapses ranging between 2.7 and 4.7 bits , based on electron microscopy imaging , see also ( Bartol Jr et al. , 2015 ) . These numbers refer to the structural accuracy of synapses . There is also electrical and chemical noise in synaptic currents induced by the biophysical details of vesicle release and neurotransmitter binding . The unreliability of the fusion of pre-synaptic vesicles ( containing neurotransmitter ) with the pre-synaptic neuron membrane is the dominant source of trial-to-trial synaptic current variation ( Allen & Stevens , 1994 ) . This noise decreases the electrical information capacity of individual synapses from the maximal value that the synaptic structure would otherwise provide . Situations arise in which the number Nf is small and the desired number of memories far exceeds ∼ Nf , see some examples from biological and AI systems in Section 4 . In these situations the associative memory model of ( Hopfield , 1982 ) would be insufficient , since it would not be able to memorize the required number of patterns . At the same time , models of associative memory with large storage capacity considered in our paper , can easily solve these problems . The starting point of this paper is a machine learning approach to associative memory based on an energy function and attractor dynamics in the space of Nf variables , called Dense Associative Memory ( Krotov & Hopfield , 2016 ) . This idea has been shown to dramatically increase the memory storage capacity of the corresponding neural network ( Krotov & Hopfield , 2016 ; Demircigil et al. , 2017 ) and was proposed to be useful for increasing robustness of neural networks to adversarial attacks ( Krotov & Hopfield , 2018 ) . Recently , an extension of this idea to continuous variables , called modern Hopfield network , demonstrated remarkably successful results on the immune repertoire classification ( Widrich et al. , 2020 ) , and provided valuable insights into the properties of attention heads in Transformer architectures ( Ramsauer et al. , 2020 ) . Dense Associative Memories or modern Hopfield networks , however , can not describe biological neural networks in terms of true microscopic degrees of freedom , since they contain many-body interaction terms in equations describing their dynamics and the corresponding energy functions . To illustrate this point consider two networks : a conventional Hopfield network ( Hopfield , 1982 ) and a Dense Associative Memory with cubic interaction term in the energy function ( see Fig . 1 ) . In the conventional network the dynamics is encoded in the matrix Tij , which represents the strengths of the synaptic connections between feature neurons i and j . Thus , this network is manifestly describable in terms of only two-body synapses , which is approximately true for many biological synapses . In contrast , a Dense Associative Memory network with cubic energy function naively requires the synaptic connections to be tensors Tijk with three indices , which are harder , although not impossible , to implement biologically . Many-body synapses become even more problematic in situations when the interaction term is described by a more complicated function than a simple power ( in this case the Taylor expansion of that function would generate a series of terms with increasing powers ) . Many-body synapses typically appear in situations when one starts with a microscopic theory described by only two-body synapses and integrates out some of the degrees of freedom ( hidden neurons ) . The argument described above based on counting the information stored in synapses in conjunction with the fact that modern Hopfield nets and Dense Associative Memories can have a huge storage capacity hints at the same solution . The reason why these networks have a storage capacity much greater than Nf is because they do not describe the dynamics of only Nf neurons , but rather involve additional neurons and synapses . Thus , there remains a theoretical question : what does this hidden circuitry look like ? Is it possible to introduce a set of hidden neurons with appropriately chosen interaction terms and activation functions so that the resulting theory has both large memory storage capacity ( significantly bigger than Nf ) , and , at the same time , is manifestly describable in terms on only two-body synapses ? The main contributions of this current paper are the following . First , we extend the model of ( Krotov & Hopfield , 2016 ) to continuous state variables and continuous time , so that the state of the network is described by a system of non-linear differential equations . Second , we couple an additional set of Nh “ complex neurons ” or “ memory neurons ” or hidden neurons to the Nf feature neurons . When the synaptic couplings and neuron activation functions are appropriately chosen , this dynamical system in Nf + Nh variables has an energy function describing its dynamics . The minima ( stable points ) of this dynamics are at the same locations in the Nf - dimensional feature subspace as the minima in the corresponding Dense Associative Memory system . Importantly , the resulting dynamical system has a mathematical structure of a conventional recurrent neural network , in which the neurons interact only in pairs through a two-body matrix of synaptic connections . We study three limiting cases of this new theory , which we call models A , B , and C. In one limit ( model A ) it reduces to Dense Associative Memory model of ( Krotov & Hopfield , 2016 ) or ( Demircigil et al. , 2017 ) depending on the choice of the activation function . In another limit ( model B ) our model reduces to the network of ( Ramsauer et al. , 2020 ) . Finally , we present a third limit ( model C ) which we call Spherical Memory model . To the best of our knowledge this model has not been studied in the literature . However , it has a high degree of symmetry and for this reason might be useful for future explorations of various models of large associative memory and recurrent neural networks in machine learning . For the purposes of this paper we defined “ biological plausiblity ” as the absence of many-body synapses . It is important to note that there other aspects in which our model described by equations ( 1 ) below is biologically implausible . For instance , it assumes that the strengths of two physically different synapses µ → i and i → µ are equal . This assumption is necessary for the existence of the energy function , which makes it easy to prove the convergence to a fixed point . It can be relaxed in equations ( 1 ) , which makes them even more biological , but , at the same time , more difficult to analyse . 2 MATHEMATICAL FORMULATION . In this section , we present a simple mathematical model in continuous time , which , on one hand , permits the storage of a huge number of patterns in the artificial neural network , and , at the same time , involves only pairwise interactions between the neurons through synaptic junctions . Thus , this system has the useful associative memory properties of the AI system , while maintaining conventional neural network dynamics and thus a degree of biological plausibility . The spikes of action potentials in a pre-synaptic cell produce input currents into a postsynaptic neuron . As a result of a single spike in the pre-synaptic cell the current in the post-synaptic neuron rises instantaneously and then falls off exponentially with a time constant τ . In the following the currents of the feature neurons are denoted by vi ( which are enumerated by the latin indices ) , and the currents of the complex memory neurons are denoted by hµ ( h stands for hidden neurons , which are enumerated by the greek indices ) . A simple cartoon of the network that we discuss is shown in Fig.2 . There are no synaptic connections among the feature neurons or the memory neurons . A matrix ξµi denotes the strength of synapses from a feature neuron i to the memory neuron µ . The synapses are assumed to be symmetric , so that the same value ξiµ = ξµi characterizes a different physical synapse from the memory neuron µ to the feature neuron i . The outputs of the memory neurons and the feature neurons are denoted by fµ and gi , which are non-linear functions of the corresponding currents . In some situations ( model A ) these outputs can be interpreted as activation functions for the corresponding neurons , so that fµ = f ( hµ ) and gi = g ( vi ) with some non-linear functions f ( x ) and g ( x ) . In other cases ( models B and C ) these outputs involve contrastive normalization , e.g . a softmax , and can depend on the currents of all the neurons in that layer . In these cases fµ = f ( { hµ } ) and gi = g ( { vi } ) . For the most part of this paper one can think about them as firing rates of the corresponding neurons . In some limiting cases , however , the function g ( vi ) will have both positive and negative signs . Then it should be interpreted as the input current from a pre-synaptic neuron . The functions f ( hµ ) and g ( vi ) are the only nonlinearities that appear in our model . Finally , the time constants for the two groups of neurons are denoted by τf and τh . With these notations our model can be written as τf dvi dt = Nh∑ µ=1 ξiµfµ − vi + Ii τh dhµ dt = Nf∑ i=1 ξµigi − hµ ( 1 ) where Ii denotes the input current into the feature neurons . The connectivity of our network has the structure of a bipartite graph , so that the connections exist between two groups of neurons , but not within each of the two groups . This design of a neural network is inspired by the class of models called Restricted Boltzmann Machines ( RBM ) ( Smolensky , 1986 ) . There is a body of literature studying thermodynamic properties of these systems and learning rules for the synaptic weights . In contrast , the goal of our work is to write down a general dynamical system and an energy function so that the network has useful properties of associative memories with a large memory storage capacity , is describable only in terms of manifestly two-body synapses , and is sufficiently general so that it can be reduced to various models of this class previously discussed in the literature . We also note that although we use the notation vi ( v stands for visible neurons ) , commonly used in the RBM literature , it is more appropriate to think about vi as higher level features . For example the input to our network can be a latent representation produced by a convolutional neural network or a latent representation of a BERT-like system ( Devlin et al. , 2018 ) rather than raw input data . Additionally , our general formulation makes it possible to use a much broader class of activation functions ( e.g . involving contrastive or spherical normalization ) than those typically used in the RBM literature . Also , the relationship between Dense Associative Memories and RBMs has been previously studied in ( Barra et al. , 2018 ; Agliari & De Marzo , 2020 ) . We also note that a Hopfield network with exponential capacity was studied in ( Chaudhuri & Fiete , 2019 ) , but their construction requires specifically engineered memory vectors and can not be applied to general arbitrary memory vectors . Mathematically , equations ( 1 ) describe temporal evolution of two groups of neurons . For each neuron its temporal updates are determined by the inputs from other neurons and its own state ( the decay term on the right hand side of the dynamical equations ) . For this reason , an energy function for this system is expected to be represented as a sum of three terms : two terms describing the neurons in each specific group , and the interaction term between the two groups of neurons . We have chosen the specific mathematical form of these three terms so that the energy function decreases on the dynamical trajectory . With these choices the energy function for the network ( 1 ) can be written as E ( t ) = [ Nf∑ i=1 ( vi − Ii ) gi − Lv ] + [ Nh∑ µ=1 hµfµ − Lh ] − ∑ µ , i fµξµigi ( 2 ) Here we introduced two Lagrangian functionsLv ( { vi } ) andLh ( { hµ } ) for the feature and the hidden neurons . They are defined through the following equations , so that derivatives of the Lagrangian functions correspond to the outputs of neurons fµ = ∂Lh ∂hµ , and gi = ∂Lv ∂vi ( 3 ) With these notations expressions in the square brackets in ( 2 ) have a familiar from classical mechanics structure of the Legendre transform between a Lagrangian and an energy function . By taking time derivative of the energy and using dynamical equations ( 1 ) one can show ( see Appendix A for details ) that the energy monotonically decreases on the dynamical trajectory dE ( t ) dt = −τf Nf∑ i , j=1 dvi dt ∂2Lv ∂vi∂vj dvj dt − τh Nh∑ µ , ν=1 dhµ dt ∂2Lh ∂hµ∂hν dhν dt ≤ 0 ( 4 ) The last inequality sign holds provided that the Hessian matrices of the Lagrangian functions are positive semi-definite . In addition to decrease of the energy function on the dynamical trajectory it is important to check that for a specific choice of the activation functions ( or Lagrangian functions ) the corresponding energy is bounded from below . This can be achieved for example by using bounded activation function for the feature neurons g ( vi ) , e.g . hyperbolic tangent or a sigmoid . Provided that the energy is bounded , the dynamics of the neural network will eventually reach a fixed point , which corresponds to one of the local minima of the energy function2 . The proposed energy function has three terms in it : the first term depends only on the feature neurons , the second term depends only on the hidden neurons , and the third term is the “ interaction ” term between the two groups of neurons . Note , that this third term is manifestly describable by twobody synapses - a function of the activity of the feature neurons is coupled to another function of the activity of the memory neurons , and the strength of this coupling is characterized by the parameters ξµi . The absence of many-body interaction terms in the energy function results in the conventional structure ( with unconventional activation functions ) of the dynamical equations ( 1 ) . Each neuron collects outputs of other neurons , weights them with coefficients ξ and generates its own output . Thus , the network described by equations ( 1 ) is biologically plausible according to our definition ( see Introduction ) . Lastly , note that the memory patterns ξµi of our network ( 1 ) can be interpreted as the strengths of the synapses connecting feature and memory neurons . This interpretation is different from the conventional interpretation , in which the strengths of the synapses is determined by matrices Tij = ∑ µ ξµiξµj ( see Fig . 1 ) , which are outer products of the memory vectors ( or higher order generalizations of the outer products ) .

Summary: The authors proposed a dynamical system that unifies several associative memory models, including the classical Hopfield network and two recently proposed modern Hopfield networks. The dynamical system is described as interactions between two groups of neurons (feature and memory neurons), providing a more biological interpretation of modern Hopfield networks. The proposed system reduces to different associative memory models by choosing different generalized activation functions, each of which maps inputs to a group of neurons into output activity. This manuscript provides sufficient details for understanding, its derivations are correct, and its results are useful in bringing modern Hopfield networks closer to biology.