Hopfield Networks is All You Need

We introduce a modern Hopfield network with continuous states and a corresponding update rule . The new Hopfield network can store exponentially ( with the dimension of the associative space ) many patterns , retrieves the pattern with one update , and has exponentially small retrieval errors . It has three types of energy minima ( fixed points of the update ) : ( 1 ) global fixed point averaging over all patterns , ( 2 ) metastable states averaging over a subset of patterns , and ( 3 ) fixed points which store a single pattern . The new update rule is equivalent to the attention mechanism used in transformers . This equivalence enables a characterization of the heads of transformer models . These heads perform in the first layers preferably global averaging and in higher layers partial averaging via metastable states . The new modern Hopfield network can be integrated into deep learning architectures as layers to allow the storage of and access to raw input data , intermediate results , or learned prototypes . These Hopfield layers enable new ways of deep learning , beyond fully-connected , convolutional , or recurrent networks , and provide pooling , memory , association , and attention mechanisms . We demonstrate the broad applicability of the Hopfield layers across various domains . Hopfield layers improved state-of-the-art on three out of four considered multiple instance learning problems as well as on immune repertoire classification with several hundreds of thousands of instances . On the UCI benchmark collections of small classification tasks , where deep learning methods typically struggle , Hopfield layers yielded a new state-ofthe-art when compared to different machine learning methods . Finally , Hopfield layers achieved state-of-the-art on two drug design datasets . The implementation is available at : https : //github.com/ml-jku/hopfield-layers 1 INTRODUCTION . The deep learning community has been looking for alternatives to recurrent neural networks ( RNNs ) for storing information . For example , linear memory networks use a linear autoencoder for sequences as a memory ( Carta et al. , 2020 ) . Additional memories for RNNs like holographic reduced representations ( Danihelka et al. , 2016 ) , tensor product representations ( Schlag & Schmidhuber , 2018 ; Schlag et al. , 2019 ) and classical associative memories ( extended to fast weight approaches ) ( Schmidhuber , 1992 ; Ba et al. , 2016a ; b ; Zhang & Zhou , 2017 ; Schlag et al. , 2021 ) have been suggested . Most approaches to new memories are based on attention . The neural Turing machine ( NTM ) is equipped with an external memory and an attention process ( Graves et al. , 2014 ) . Memory networks ( Weston et al. , 2014 ) use an arg max attention by first mapping a query and patterns into a space and then retrieving the pattern with the largest dot product . End to end memory networks ( EMN ) make this attention scheme differentiable by replacing arg max through a softmax ( Sukhbaatar et al. , 2015a ; b ) . EMN with dot products became very popular and implement a key-value attention ( Daniluk et al. , 2017 ) for self-attention . An enhancement of EMN is the transformer ( Vaswani et al. , 2017a ; b ) and its extensions ( Dehghani et al. , 2018 ) . The transformer has had a great impact on the natural language processing ( NLP ) community , in particular via the BERT models ( Devlin et al. , 2018 ; 2019 ) . Contribution of this work : ( i ) introducing novel deep learning layers that are equipped with a memory via modern Hopfield networks , ( ii ) introducing a novel energy function and a novel update rule for continuous modern Hopfield networks that are differentiable and typically retrieve patterns after one update . Differentiability is required for gradient descent parameter updates and retrieval with one update is compatible with activating the layers of deep networks . We suggest using modern Hopfield networks to store information or learned prototypes in different layers of neural networks . Binary Hopfield networks were introduced as associative memories that can store and retrieve patterns ( Hopfield , 1982 ) . A query pattern can retrieve the pattern to which it is most similar or an average over similar patterns . Hopfield networks seem to be an ancient technique , however , new energy functions improved their properties . The stability of spurious states or metastable states was sensibly reduced ( Barra et al. , 2018 ) . The largest and most impactful successes are reported on increasing the storage capacity of Hopfield networks . In a d-dimensional space , the standard Hopfield model can store d uncorrelated patterns without errors but only Cd/ log ( d ) random patterns with C < 1/2 for a fixed stable pattern or C < 1/4 if all patterns are stable ( McEliece et al. , 1987 ) . The same bound holds for nonlinear learning rules ( Mazza , 1997 ) . Using tricks-of-trade and allowing small retrieval errors , the storage capacity is about 0.138d ( Crisanti et al. , 1986 ; Hertz et al. , 1991 ; Torres et al. , 2002 ) . If the learning rule is not related to the Hebb rule , then up to d patterns can be stored ( Abu-Mostafa & StJacques , 1985 ) . For Hopfield networks with non-zero diagonal matrices , the storage can be increased to Cd log ( d ) ( Folli et al. , 2017 ) . In contrast to the storage capacity , the number of energy minima ( spurious states , stable states ) of Hopfield networks is exponential in d ( Tanaka & Edwards , 1980 ; Bruck & Roychowdhury , 1990 ; Wainrib & Touboul , 2013 ) . The standard binary Hopfield network has an energy function that can be expressed as the sum of interaction functions F with F ( x ) = x2 . Modern Hopfield networks , also called “ dense associative memory ” ( DAM ) models , use an energy function with interaction functions of the form F ( x ) = xn and , thereby , achieve a storage capacity proportional to dn−1 ( Krotov & Hopfield , 2016 ; 2018 ) . The energy function of modern Hopfield networks makes them robust against adversarial attacks ( Krotov & Hopfield , 2018 ) . Modern binary Hopfield networks with energy functions based on interaction functions of the form F ( x ) = exp ( x ) even lead to storage capacity of 2d/2 , where all stored binary patterns are fixed points but the radius of attraction vanishes ( Demircigil et al. , 2017 ) . However , in order to integrate Hopfield networks into deep learning architectures , it is necessary to make them differentiable , that is , we require continuous Hopfield networks ( Hopfield , 1984 ; Koiran , 1994 ) . Therefore , we generalize the energy function of Demircigil et al . ( 2017 ) that builds on exponential interaction functions to continuous patterns and states and obtain a new modern Hopfield network . We also propose a new update rule which ensures global convergence to stationary points of the energy ( local minima or saddle points ) . We prove that our new modern Hopfield network typically retrieves patterns in one update step ( -close to the fixed point ) with an exponentially low error and has a storage capacity proportional to c d−1 4 ( reasonable settings for c = 1.37 and c = 3.15 are given in Theorem 3 ) . The retrieval of patterns with one update is important to integrate Hopfield networks in deep learning architectures , where layers are activated only once . Surprisingly , our new update rule is also the key-value attention as used in transformer and BERT models ( see Fig . 1 ) . Our modern Hopfield networks can be integrated as a new layer in deep learning architectures for pooling , memory , prototype learning , and attention . We test these new layers on different benchmark datasets and tasks like immune repertoire classification . 2 MODERN HOPFIELD NETS WITH CONTINUOUS STATES . New energy function for continuous state Hopfield networks . In order to integrate modern Hopfield networks into deep learning architectures , we have to make them continuous . To allow for continuous states , we propose a new energy function that is a modification of the energy of modern Hopfield networks ( Demircigil et al. , 2017 ) . We also propose a new update rule which can be proven to converge to stationary points of the energy ( local minima or saddle points ) . We have N stored ( key ) patterns xi ∈ Rd represented by the matrix X = ( x1 , . . . , xN ) with the largest pattern M = maxi ‖xi‖ . The state ( query ) pattern is ξ ∈ Rd . For exponential interaction functions , we need the log-sum-exp function ( lse ) for 0 < β lse ( β , x ) = β−1 log ( N∑ i=1 exp ( βxi ) ) , ( 1 ) which is convex ( see appendix Eq . ( 461 ) , and Lemma A22 ) . The energy function E of the modern Hopfield networks for binary patterns xi and a binary state pattern ξ is E = − ∑N i=1 F ( ξTxi ) ( Krotov & Hopfield , 2016 ) . Here , F ( x ) = xn is the interaction function , where n = 2 gives the classical Hopfield network . The storage capacity is proportional to dn−1 ( Krotov & Hopfield , 2016 ) . This model was generalized by Demircigil et al . ( 2017 ) to exponential interaction functions F ( x ) = exp ( x ) which gives the energy E = − exp ( lse ( 1 , XT ξ ) ) . This energy leads to an exponential storage capacity of N = 2d/2 for binary patterns . Furthermore , with a single update , the fixed point is recovered with high probability for random patterns . However , still this modern Hopfield network has binary states . We generalize this energy function to continuous-valued patterns while keeping the properties of the modern Hopfield networks like the exponential storage capacity and the extremely fast convergence ( see Fig . 1 ) . For the new energy we take the logarithm of the negative energy of modern Hopfield networks and add a quadratic term of the current state . The quadratic term ensures that the norm of the state vector ξ remains finite and the energy is bounded . Classical Hopfield networks do not require to bound the norm of their state vector , since it is binary and has fixed length . We define the novel energy function E as E = − lse ( β , XT ξ ) + 1 2 ξT ξ + β−1 logN + 1 2 M2 . ( 2 ) We have 0 6 E 6 2M2 ( see appendix Lemma A1 ) . Using p = softmax ( βXT ξ ) , we define a novel update rule ( see Fig . 1 ) : ξnew = f ( ξ ) = Xp = Xsoftmax ( βXT ξ ) . ( 3 ) The next theorem states that the update rule Eq . ( 3 ) converges globally . The proof uses the ConcaveConvex Procedure ( CCCP ) ( Yuille & Rangarajan , 2002 ; 2003 ) , which is equivalent to Legendre minimization ( Rangarajan et al. , 1996 ; 1999 ) algorithms ( Yuille & Rangarajan , 2003 ) . Theorem 1 . The update rule Eq . ( 3 ) converges globally : For ξt+1 = f ( ξt ) , the energy E ( ξt ) → E ( ξ∗ ) for t→∞ and a fixed point ξ∗ . Proof . The update rule in Eq . ( 3 ) is the CCCP for minimizing the energy E , which is the sum of the convex 1/2ξT ξ and concave −lse ( see details in appendix Theorem 1 ) . Theorem 2 in Yuille & Rangarajan ( 2002 ) yields the global convergence property . Also , in Theorem 2 in Sriperumbudur & Lanckriet ( 2009 ) the global convergence of CCCP is proven via a rigorous analysis using Zangwill ’ s global convergence theory of iterative algorithms . The global convergence theorem only assures that for the energy E ( ξt ) → E ( ξ∗ ) for t→∞ but not ξt → ξ∗ . The next theorem strengthens Zangwill ’ s global convergence theorem ( Meyer , 1976 ) and gives convergence results similar to those known for expectation maximization ( Wu , 1983 ) . Theorem 2 . For the iteration Eq . ( 3 ) we have E ( ξt ) → E ( ξ∗ ) = E∗ as t → ∞ , for some stationary point ξ∗ . Furthermore , ∥∥ξt+1 − ξt∥∥ → 0 and either { ξt } ∞t=0 converges or , in the other case , the set of limit points of { ξt } ∞t=0 is a connected and compact subset of L ( E∗ ) , where L ( a ) = { ξ ∈ L | E ( ξ ) = a } and L is the set of stationary points of the iteration Eq . ( 3 ) . If L ( E∗ ) is finite , then any sequence { ξt } ∞t=0 generated by the iteration Eq . ( 3 ) converges to some ξ∗ ∈ L ( E∗ ) . For a proof , see appendix Theorem 2 . Therefore , all the limit points of any sequence generated by the iteration Eq . ( 3 ) are stationary points ( local minima or saddle points ) of the energy function E. Either the iteration converges or , otherwise , the set of limit points is a connected and compact set . The next theorem gives the results on the storage capacity of our new continuous state modern Hopfield network . We first define what we mean by storing and retrieving patterns using a modern Hopfield network with continuous states . Definition 1 ( Pattern Stored and Retrieved ) . We assume that around every pattern xi a sphere Si is given . We say xi is stored if there is a single fixed point x∗i ∈ Si to which all points ξ ∈ Si converge , and Si ∩ Sj = ∅ for i 6= j . We say xi is retrieved for a given if iteration ( update rule ) Eq . ( 3 ) gives a point x̃i that is at least -close to the single fixed point x∗i ∈ Si . The retrieval error is ‖x̃i − xi‖ . As with classical Hopfield networks , we consider patterns on the sphere , i.e . patterns with a fixed norm . For randomly chosen patterns , the number of patterns that can be stored is exponential in the dimension d of the space of the patterns ( xi ∈ Rd ) . Theorem 3 . We assume a failure probability 0 < p 6 1 and randomly chosen patterns on the sphere with radius M : = K √ d− 1 . We define a : = 2d−1 ( 1 + ln ( 2βK 2p ( d− 1 ) ) ) , b : = 2K 2β 5 , and c : = bW0 ( exp ( a+ln ( b ) ) , where W0 is the upper branch of the Lambert W function ( Olver et al. , 2010 , ( 4.13 ) ) , and ensure c ≥ ( 2√ p ) 4 d−1 . Then with probability 1− p , the number of random patterns that can be stored is N ≥ √p c d−1 4 . ( 4 ) Therefore it is proven for c ≥ 3.1546 with β = 1 , K = 3 , d = 20 and p = 0.001 ( a+ ln ( b ) > 1.27 ) and proven for c ≥ 1.3718 with β = 1 , K = 1 , d = 75 , and p = 0.001 ( a+ ln ( b ) < −0.94 ) . For a proof , see appendix Theorem A5 . The next theorem states that the update rule typically retrieves patterns after one update . Retrieval of a pattern xi for fixed point x∗i and query ξ is defined via an by ‖f ( ξ ) − x∗i ‖ < , that is , the update is -close to the fixed point . Retrieval with one update is crucial to integrate modern Hopfield networks into deep learning architectures , where layers are activated only once . First we need the concept of separation of a pattern . For pattern xi we define its separation ∆i to other patterns by : ∆i : = min j , j 6=i ( xTi xi − xTi xj ) = xTi xi − max j , j 6=i xTi xj . ( 5 ) The update rule retrieves patterns with one update for well separated patterns , that is , patterns with large ∆i . Theorem 4 . With query ξ , after one update the distance of the new point f ( ξ ) to the fixed point x∗i is exponentially small in the separation ∆i . The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ξ and its value Jm in the mean value theorem are : ‖f ( ξ ) − x∗i ‖ 6 ‖Jm‖2 ‖ξ − x ∗ i ‖ , ( 6 ) ‖Jm‖2 6 2 β N M 2 ( N − 1 ) exp ( − β ( ∆i − 2 max { ‖ξ − xi‖ , ‖x∗i − xi‖ } M ) ) . ( 7 ) For given and sufficient large ∆i , we have ‖f ( ξ ) − x∗i ‖ < , that is , retrieval with one update . See proof in appendix Theorem A8 . At the same time , the retrieval error decreases exponentially with the separation ∆i . Theorem 5 ( Exponentially Small Retrieval Error ) . The retrieval error ‖f ( ξ ) − xi‖ of pattern xi is bounded by ‖f ( ξ ) − xi‖ 6 2 ( N − 1 ) exp ( − β ( ∆i − 2 max { ‖ξ − xi‖ , ‖x∗i − xi‖ } M ) ) M ( 8 ) and for ‖xi − x∗i ‖ 6 12 β M together with ‖xi − ξ‖ 6 1 2 β M by ‖xi − x∗i ‖ 6 2 e ( N − 1 ) M exp ( − β ∆i ) . ( 9 ) See proof in appendix Theorem A9 . Metastable states and one global fixed point . So far , we considered patterns xi that are well separated and the iteration converges to a fixed point which is near a pattern xi . If no pattern xi is well separated from the others , then the iteration converges to a global fixed point close to the arithmetic mean of the vectors . In this case the softmax vector p is close to uniform , that is , pi = 1/N . If some vectors are similar to each other and well separated from all other vectors , then a metastable state near the similar vectors exists . Iterations that start near the metastable state converge to this metastable state , also if initialized by one of the similar patterns . For convergence proofs to one global fixed point and to metastable states see appendix Lemma A7 and Lemma A12 , respectively . Hopfield update rule is attention of the transformer . The Hopfield network update rule is the attention mechanism used in transformer and BERT models ( see Fig . 1 ) . To see this , we assume N stored ( key ) patterns yi and S state ( query ) patterns ri that are mapped to the Hopfield space of dimension dk . We set xi = W TKyi , ξi = W T Q ri , and multiply the result of our update rule withWV . The matrices Y = ( y1 , . . . , yN ) T and R = ( r1 , . . . , rS ) T combine the yi and ri as row vectors . We define the matricesXT = K = YWK , ΞT = Q = RWQ , and V = YWKWV = XTWV , where WK ∈ Rdy×dk , WQ ∈ Rdr×dk , WV ∈ Rdk×dv . If β = 1/ √ dk and softmax ∈ RN is changed to a row vector , we obtain for the update rule Eq . ( 3 ) multiplied byWV : Z = softmax ( 1/ √ dk QK T ) V = softmax ( β RWQW T KY T ) Y WKWV . ( 10 ) The left part of Eq . ( 10 ) is the transformer attention . In the transformer self-attentionR = Y , and WKWV replaced by justWV . Besides the attention mechanism , Hopfield networks allow for other functionalities in deep network architectures , which we introduce via specific layers in the next section . The right part of Eq . ( 10 ) serves to explain these specific layers .

Summary: This work extends the binary Hopfield network (Demircigil et al., 2017) to continuous patterns and states. Connections are drawn between the result model to the attention layers of the transformers, the pooling operation of LSTM, similarity search, and fully connected layers. Experimental results are briefly described for analyzing the attention of Bert models, multiple instance learning, and small UCI classification tasks.