On the mapping between Hopfield networks and Restricted Boltzmann Machines

1 INTRODUCTION . Hopfield networks ( HNs ) ( Hopfield , 1982 ; Amit , 1989 ) are a classical neural network architecture that can store prescribed patterns as fixed-point attractors of a dynamical system . In their standard formulation with binary valued units , HNs can be regarded as spin glasses with pairwise interactions Jij that are fully determined by the patterns to be encoded . HNs have been extensively studied in the statistical mechanics literature ( e.g . ( Kanter & Sompolinsky , 1987 ; Amit et al. , 1985 ) ) , where they can be seen as an interpolation between the ferromagnetic Ising model ( p = 1 pattern ) and the Sherrington-Kirkpatrick spin glass model ( many random patterns ) ( Kirkpatrick & Sherrington , 1978 ; Barra & Guerra , 2008 ) . By encoding patterns as dynamical attractors which are robust to perturbations , HNs provide an elegant solution to pattern recognition and classification tasks . They are considered the prototypical attractor neural network , and are the historical precursor to modern recurrent neural networks . Concurrently , spin glasses have been used extensively in the historical machine learning literature where they comprise a sub-class of “ Boltzmann machines ” ( BMs ) ( Ackley et al. , 1985 ) . Given a collection of data samples drawn from a data distribution , one is generally interested in “ training ” a BM by tuning its weights Jij such that its equilibrium distribution can reproduce the data distribution as closely as possible ( Hinton , 2012 ) . The resulting optimization problem is dramatically simplified when the network has a two-layer structure where each layer has no self-interactions , so that there are only inter-layer connections ( Hinton , 2012 ) ( see Fig . 1 ) . This architecture is known as a Restricted Boltzmann Machine ( RBM ) , and the two layers are sometimes called the visible layer and the hidden layer . The visible layer characteristics ( dimension , type of units ) are determined by the training data , whereas the hidden layer can have binary or continuous units and the dimension is chosen somewhat arbitrarily . In addition to generative modelling , RBMs and their multi-layer extensions have been used for a variety of learning tasks , such as classification , feature extraction , and dimension reduction ( e.g . Salakhutdinov et al . ( 2007 ) ; Hinton & Salakhutdinov ( 2006 ) ) . There has been extensive interest in the relationship between HNs and RBMs , as both are built on the Ising model formalism and fulfill similar roles , with the aim of better understanding RBM behaviour and potentially improving performance . Various results in this area have been recently reviewed ( Marullo & Agliari , 2021 ) . In particular , an exact mapping between HNs and RBMs has been previously noted for the special case of uncorrelated ( orthogonal ) patterns ( Barra et al. , 2012 ) . Several related models have since been studied ( Agliari et al. , 2013 ; Mézard , 2017 ) , which partially relax the uncorrelated pattern constraint . However , the patterns observed in most real datasets exhibit significant correlations , precluding the use of these approaches . In this paper , we demonstrate exact correspondence between HNs and RBMs in the case of correlated pattern HNs . Specifically , we show that any HN with N binary units and p < N arbitrary ( i.e . non-orthogonal ) binary patterns encoded via the projection rule ( Kanter & Sompolinsky , 1987 ; Personnaz et al. , 1986 ) , can be transformed into an RBM with N binary and p gaussian variables . We then characterize when the reverse map from RBMs to HNs can be made . We consider a practical example using the mapping , and discuss the potential importance of this correspondence for the training and interpretability of RBMs . 2 RESULTS . We first introduce the classical solution to the problem of encodingN -dimensional binary { −1 , +1 } vectors { ξµ } pµ=1 , termed “ patterns ” , as global minima of a pairwise spin glass H ( s ) = − 12s TJs . This is often framed as a pattern retrieval problem , where the goal is to specify or learn Jij such that an energy-decreasing update rule for H ( s ) converges to the patterns ( i.e . they are stable fixed points ) . Consider the N × p matrix ξ with the p patterns as its columns . Then the classical prescription known as the projection rule ( or pseudo-inverse rule ) ( Kanter & Sompolinsky , 1987 ; Personnaz et al. , 1986 ) , J = ξ ( ξT ξ ) −1ξT , guarantees that the p patterns will be global minima of H ( s ) . This resulting spin model is commonly called a ( projection ) Hopfield network , and has the Hamiltonian H ( s ) = −1 2 sT ξ ( ξT ξ ) −1ξTs . ( 1 ) Note that ξT ξ invertibility is guaranteed as long as the patterns are linearly independent ( we therefore require p ≤ N ) . Also note that in the special ( rare ) case of orthogonal patterns ξµ · ξν = Nδµν ( also called “ uncorrelated ” ) , studied in the previous work ( Barra et al. , 2012 ) , one has ξT ξ = NI and so the pseudo-inverse interactions reduce to the well-known Hebbian form J = 1N ξξ T ( the properties of which are studied extensively in Amit et al . ( 1985 ) ) . Additional details on the projection HN Eq . ( 1 ) are provided in Appendix A . To make progress in analyzing Eq . ( 1 ) , we first consider a transformation of ξ which eliminates the inverse factor . 2.1 MAPPING A HOPFIELD NETWORK TO A RESTRICTED BOLTZMANN MACHINE . In order to obtain a more useful representation of the quadratic form Eq . ( 1 ) ( for our purposes ) , we utilize the QR-decomposition ( Schott & Stewart , 1999 ) of ξ to “ orthogonalize ” the patterns , ξ = QR , ( 2 ) with Q ∈ RN×p , R ∈ Rp×p . The columns of Q are the orthogonalized patterns , and form an orthonormal basis ( of non-binary vectors ) for the p-dimensional subspace spanned by the binary patterns . R is upper triangular , and if its diagonals are held positive then Q and R are both unique ( Schott & Stewart , 1999 ) . Note both the order and sign of the columns of ξ are irrelevant for HN pattern recall , so there are n = 2p · p ! possibleQ , R pairs . Fixing a pattern ordering , we can use the orthogonality ofQ to re-write the interaction matrix as J = ξ ( ξT ξ ) −1ξT = QR ( RTR ) −1RTQT = QQT ( 3 ) ( the last equality follows from ( RTR ) −1 = R−1 ( RT ) −1 ) . Eq . ( 3 ) resembles the simple Hebbian rule but with non-binary orthogonal patterns . Defining q ≡ QTs in analogy to the classical pattern overlap parameterm ≡ 1N ξ Ts ( Amit et al. , 1985 ) , we have H ( s ) = −1 2 sTQQTs = −1 2 q ( s ) · q ( s ) . ( 4 ) Using a Gaussian integral as in Amit et al . ( 1985 ) ; Barra et al . ( 2012 ) ; Mézard ( 2017 ) to transform ( exactly ) the partition function Z ≡ ∑ { s } e −βH ( s ) of Eq . ( 1 ) , we get Z = ∑ { s } e 1 2 ( βq ) T ( β−1I ) ( βq ) = ∑ { s } ∫ e− β 2 ∑ µ λ 2 µ+β ∑ µ λµ ∑ iQiµsi ∏ µ dλµ√ 2π/β . ( 5 ) The second line can be seen as the partition function of an expanded Hamiltonian for the N ( binary ) original variables { si } and the p ( continuous ) auxiliary variables { λµ } , i.e . HRBM ( { si } , { λµ } ) = 1 2 ∑ µ λ2µ − ∑ µ ∑ i Qiµsiλµ . ( 6 ) Note that this is the Hamiltonian of a binary-continuous RBM with inter-layer weights Qiµ . The original HN is therefore equivalent to an RBM described by Eq . ( 6 ) ( depicted in Fig . 1 ) . As mentioned above , there are many RBMs which correspond to the same HN due to the combinatorics of choosing Q . In fact , instead of QR factorization one can use any decomposition which satisfies J = UUT , with orthogonal U ∈ RN×p ( see Appendix B ) , in which case U acts as the RBM weights . Also note the inclusion of an applied field term − ∑ i bisi in Eq . ( 1 ) trivially carries through the procedure , i.e . H̃RBM ( { si } , { λµ } ) = 12 ∑ µ λ 2 µ − ∑ i bisi − ∑ µ ∑ iQiµsiλµ . Instead of working with the joint form Eq . ( 6 ) , one could take a different direction from Eq . ( 5 ) and sum out the original variables { si } , i.e . Z = ∫ e− β 2 ∑ µ λ 2 µ2N ∏ i cosh ( β ∑ µ Qiµλµ ) ∏ µ dλµ√ 2π/β . ( 7 ) This continuous , p-dimensional representation is useful for numerical estimation of Z ( Section 3.1 ) . We may write Eq . ( 7 ) as Z = ∫ e−F0 ( λ ) dλµ , where F0 ( { λµ } ) = 1 2 ∑ µ λ2µ − 1 β ∑ i ln cosh ( β ∑ µ Qiµλµ ) . ( 8 ) Eq . ( 8 ) is an approximate Lyapunov function for the mean dynamics of { λµ } ; ∇λF0 describes the effective behaviour of the stochastic dynamics of the N binary variables { si } at temperature β−1 . 2.2 COMMENTS ON THE REVERSE MAPPING . With the mapping from HNs ( with correlated patterns ) to RBMs established , we now consider the reverse direction . Consider a binary-continuous RBM with inter-layer weights Wiµ which couple a visible layer of N binary variables { si } to a hidden layer of p continuous variables { λµ } , H ( s , λ ) = 1 2 ∑ µ λ2µ − ∑ i bisi − ∑ µ ∑ i Wiµsiλµ . ( 9 ) Here we use W instead of Q for the RBM weights to emphasize that the RBM is not necessarily an HN . First , following Mehta et al . ( 2019 ) , we transform the RBM to a BM with binary states by integrating out the hidden variables . The corresponding Hamiltonian for the visible units alone is ( see Appendix D.1 for details ) , H̃ ( s ) = − ∑ i bisi − 1 2 ∑ i ∑ j ∑ µ WiµWjµsisj , ( 10 ) a pairwise Ising model with a particular coupling structure Jij = ∑ µWiµWjµ , which in vector form is J = ∑ µ wµw T µ =WW T , ( 11 ) where { wµ } are the p columns ofW . In general , this Ising model Eq . ( 10 ) produced by integrating out the hidden variables need not have Hopfield structure ( discussed below ) . However , it automatically does ( as noted in Barra et al . ( 2012 ) ) , in the very special case whereWiµ ∈ { −1 , +1 } . In that case , the binary patterns are simply { wµ } , so that Eq . ( 11 ) represents a Hopfield network with the Hebbian prescription . This situation is likely rare and may only arise as a by-product of constrained training ; for a generically trained RBM the weights will not be binary . It is therefore interesting to clarify when and how real-valued RBM interactionsW can be associated with HNs . Approximate binary representation of W : In Section 2.1 , we orthogonalized the binary matrix ξ via the QR decomposition ξ = QR , where Q is an orthogonal ( but non-binary ) matrix , which allowed us to map a projection HN ( defined by its patterns ξ , Eq . ( 1 ) ) to an RBM ( defined by its inter-layer weightsQ , Eq . ( 6 ) ) . Here we consider the reverse map . Given a trained RBM with weights W ∈ RN×p , we look for an invertible transformation X ∈ Rp×p which binarizes W . We make the mild assumption that W is rank p. If we find such an X , then B =WX will be the Hopfield pattern matrix ( analogous to ξ ) , with Biµ ∈ { −1 , +1 } . This is a non-trivial problem , and an exact solution is not guaranteed . As a first step to study the problem , we relax it to that of finding a matrix X ∈ GLp ( R ) ( i.e . invertible , p × p , real ) which minimizes the binarization error argmin X∈GLp ( R ) ||WX − sgn ( WX ) ||F . ( 12 ) We denote the approximately binary transformation ofW via a particular solutionX by Bp =WX . ( 13 ) We also define the associated error matrixE ≡ Bp− sgn ( Bp ) . We stress thatBp is non-binary and approximatesB ≡ sgn ( Bp ) , the columns of which will be HN patterns under certain conditions on E. We provide an initial characterization and example in Appendix D .

Summary: This paper shows a relationship between the project rule weights of a Hopfield network (HN) and the interaction weights in a corresponding restricted Boltzmann machine (RBM). The mapping from HN to RBM is facilitated by realising that the partition function of BN can be seen as the partition function of a binary-continuous (Bernoulli-Gaussian) RBM. The authors comments on the mapping from RBM to BN. The experiments show the advantages of training RBM with weights initialised from BN projection weights in generation and classification.