On the mapping between Hopfield networks and Restricted Boltzmann Machines
Authors: Matthew Smart
Anton Zilman
Venue: ICLR 2021 Oral (2021)
Keywords: Hopfield Networks
Restricted Boltzmann Machines
Statistical Physics

Abstract: Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience. Recently, there has been interest in the relationship between HNs and RBMs, due to their similarity under the statistical mechanics formalism. An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (“uncorrelated”) encoded patterns. We present here an exact mapping in the case of correlated pattern HNs, which are more broadly applicable to existing datasets. Specifically, we show that any HN with $N$ binary variables and $p<N$ potentially correlated binary patterns can be transformed into an RBM with $N$ binary visible variables and $p$ gaussian hidden variables. We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights. We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of feature extraction methods which utilize RBMs.

Published as a conference paper at ICLR 2021
ON THE MAPPING BETWEEN HOPFIELD NETWORKS AND RESTRICTED BOLTZMANN MACHINES
Matthew Smart
Anton Zilman
Department of Physics
Department of Physics and Institute for Biomedical Engingeering University of Toronto
University of Toronto
msma rt@prrrrcscccss..mo www.ssttooooooooooooo
z i lmanammpaasiceee.c utw.ento..roooooooooo
ABSTRACT
Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models the interface of statistical physics, machine learning. and neu- roscience. Recently, there has been interest in the relationship between HNs and RBMs, due to their similarity under the statistical mechanics formalism. An exact mapping between HNs and RBMs has been previously noted for the special case orthogonal ("uncorrelated") encoded patterns. We present here an exact map- ping in the case of correlated pattern HNs, which are more broadly applicable existing datasets. Specifically, we show that any HN with binary variables and p < N arbitrary binary patterns can be transformed into an RBM with N binary visible variables and p gaussian hidden variables. We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides useful initialization to the RBM weights. We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.
INTRODUCTION
Hopfield networks (HNs) (Hopfield 1982: Amit 1989) are a classical neural network architecture that can store prescribed patterns as fixed-poim point attractors of a dynamical system. In their standard formulation with binary valued units, HNs can be regarded spin glasses with pairwise interactions Jiy that are fully determined by the patterns to be encoded. HNs have been extensively studied in the statistical mechanics literature (e.g. (Kanter & Sompolinsky 1987 Amit et al. 1985)), where they can be seen as an interpolation between the ferromagnetic Ising model (p pattern) and the Sherrington- -irkpatriic spin glass model (many random patterns) (Kirkpatrick & Sherrington, 1978 Barra Guerra 2008). By encoding patterns as dynamical attractors which are robust to perturbations, HNs provide an elegant solution to pattern recognition and classification tasks. They are considered the prototypical attractor neural network, and are the historical precursor to modern recurrent neural networks.
Concurrently, spin glasses have been used extensively in the historical machine learning literature where they comprise sub-class of "Boltzmann machines" (BMs) (Ackley et al., 1985). Given collection of data samples drawn from a data distribution, one is generally interested in "training" BM by tuning its weights Juy such that its cquilibrium distribution can reproduce the data distri- bution as closely as possible (Hinton 2012). The resulting optimization problem is dramatically simplified when the network has a two-layer structure where each layer has no reff-interactions, so that there are only inter-layer connections (Hinton) 2012) (see Fig. [] This architecture is known as a Restricted Boltzmann Machine (RBM), and the two layers are sometimes called the visible layer and the hidden layer. The visible layer characteristics (dimension, type of units) are determined by the training data, whereas the hidden layer can have binary or continuous units and the dimension is chosen somewhat arbitrarily. In addition to generative modelling, RBMs and their multi-layer extensions have been used for a variety of learning tasks, such as classification, feature extraction, and dimension reduction (e.g. Salakhutdinov al. (2007) Hinton & Salakhutdinov (006)).

---

Published as a conference paper at ICLR 2021
There has been extensive interest in the relationship between HNs and RBMs, as both are built on the Ising model formalism and fulfill similar roles, with the aim of better understanding RBM behaviour and potentially improving performance. Various results in this area have been recently reviewed (Marullo & Agliari (021). In particular, an exact mapping between HNs and RBMs has been previously noted for the special case of uncorrelated (orthogonal) patterns (Barra al. 2012). Several related models have since been studied (Agliari et al. 2013) Mézard (0177), which partially relax the uncorrelated pattern constraint. However, the patterns observed in most real datasets exhibit significant correlations, precluding the use of these approaches.
In this paper, we demonstrate exact correspondence between HNs and RBMs in the case of corre- lated pattern HNs. Specifically, we show that any HN with N binary units and p < N arbitrary (i.e. non-orthogonal) binary patterns encoded via the projection rule (Kanter Sompolinsky 1987 Personnaz et al., 1986), can be transformed into an RBM with N binary and gaussian variables. We then characterize when the reverse map from RBMs to HNs can be made. We consider a practi- cal example using the mapping. and discuss the potential importance of this correspondence for the training and interpretability of RBMs.
RESULTS
We first introduce the classical solution to the problem of encoding N-dimensional binary 1-1,+13 vectors (5"))-1, termed "patterns", as global minima of pairwise spin glass H(s) 3 This is often framed pattern retrieval problem, where the goal is specify learn J4, such that an -eeggyddecra update rule for H (s) converges to the patterns (i.c. they are stable fixed points). Consider the N x p matrix & with the p patterns its columns. Then the classical prescrip- tion known as the projection rule (or pseudd-iiverss rule) (Kanter & Sompolinsky 1987, Personnaz et al. 1986). J = E(E" "-""", guarantees that the patterns will be global minima of H (s). This resulting spin model is commonly called (projection) Hopfield network, and has the Hamiltonian
3/8"" J
(s) = -
(I)
Note that E invertibility is guaranteed as long as the patterns are linearly independent (we there- fore require p S N). Also note that in the special (rare) case of orthogonal patterns ş 5" N8 (also called "uncorrelated"), studied in the previous work (Barra et al. 2012). one has E < NI and the pseudo-inverse interactions reduce to the well-known Hebbian form J = _E (the properties of which are studied extensively in Amit et al. (1985).. Additional details on the pro- jection HN Eq. (T) are provided in Appendix A To make progress in analyzing Eq. (T). we first consider a transformation of which climinates the inverse factor.
2.1 MAPPING A HOPFIELD NETWORK TO A RESTRICTED BOLTZMANN MACHINE
In order to obtain a more useful representation of the quadratic form Eq. ① (for our purposes), we utilize the QR- decomposition (Schott& & Stewart 1999) of & to "orthogonalize" the patterns,
E EQQR,
(2)
with Q E N.P. R E RPX, The columns of Q are the orthogonalized patterns, and form an orthonormal basis (of non-binary vectors) for the p-dimensional subspace spanned by the binary patterns. R is upper triangular, and if its diagonals are held positive then Q and R are both unique (Schott& & Stewart| 1999). Note both the order and sign of the columns of & are irrelevant for HN pattern recall, so there are n = 2". p! possible Q, R pairs. Fixing a pattern ordering, we can use the orthogonality of Q to re-write the interaction matrix
J -(E5"
QR R- R) RR Q" QQ"
resembles the simple Hebbian rule but with non- non-binary orthogonal patterns. Defining q = Q" s in analogy to the classical pattern
(the last equality follows from IRTR) R)-
Eq.
overlap parameter m = 5 s Amit et al. 1985), we have
H(s) = 2
TQQ s= = -=)(())
(()) q(s).
(4)
2

---

Published as a conference paper at ICLR 2021
Using Gaussian integral as in Amit al. (1985); :: Barra et al. (0122)
Mézard (007) to transform
(exactly) the partition function 7 (x))
-P of Eq. we get
7 = Σ
e (8q) (8-+) (3
{x}
(5)
Σ
E (a)
Ap Repss
dd
/2T/B
The second line be seen as the partition function of an expanded Hamiltonian for the N (binary) original variables {$t} and the (continuous) auxiliary variables {Au), i.e.
HRBM ( (sss) (A,))
2 - Σ x
-E-E AusstAp -
(6)
Note that this is the Hamiltonian of a binary-comtinuous RBM with inter-layer weights Que- The original HN is therefore equivalent to an RBM described by Eq. (6) (depicted in Fig. [ As mentioned above, there are many RBMs which correspond the same HN due the combinatories of choosing Q. In fact, instead of QR factorization one can use any decomposition which satisfies = UU with orthogonal weights. Also note the inclusion of an applied field term - >. b,s; in Eq. 1 trivially carries through the procedure, i.c. HRBM ({si), (Ar)) 1 En X E, bis; En, D. Qip
E Nxp (see Appendix B). in which case acts as the RBM
Figure 1: Correspondence between Hop- field Networks (HNs) with correlated pat- terns and binary-gaussian restricted boltz- mann machines (RBMs). The HN has N binary units and pairwise interactions J de- fined by p < N (possibly correlated) pat- terns {E")P=. The patterns are encoded as p==1-
Qip
Ju
1,
= QR
minima Eq. [I through the projection where e" form the columns of &. We orthogonalize the patterns through QR decomposition
rule J -)
QR. The HN is equivalent to an RBM with N binary visible units and gaussian hidden units with inter-layer weights defined the orthogonal- ized patterns Qia. and Hamiltonian Eq. (6). See (Agliari al., (017) for the analogous mapping in the uncorrelated case.
J= 1=
Instead of working with the joint form Eq. sum out the original variables {s:). i.e.
(6) one could take different direction from Eq. 5 and
7 / $ En 2
II cosh (m
II
dAp
2T/3
This continuous, p-dimensional representation is useful for numerical estimation of Z (Section 3.1 We may write Eq. (7) Z f - Fo()) d'y, where
Fo((A,,))
Σ 2
Σ In cosh
3 E Eiidd "
j
Eq. 8 is approximate Lyapunov function for the mean dynamics of (Au); VxFo describes the effective behaviour of the stochastic dynamics of the N binary variables {st} at temperature 3-1
2.2 COMMENTS ON THE REVERSE MAPPING
With the mapping from HNs (with correlated patterns) to RBMs established, we now consider the reverse direction. Consider binary-continuous RBM with inter-layer weights W., which couple
3

---

Published as a conference paper at ICLR 2021
visible layer of N binary variables {sy} hidden layer of p continuous variables {
H(s Σ 2 Σ b,5:
ΣΣ W Wip
Here use W instead of Q for the RBM weights to emphasize that the RBM is not necessarily an HN. First, following Mehta et al. (2019), we transform the RBM to a BM with binary states by integrating the hidden variables. The corresponding Hamiltonian for the visible units alone is (see Appendix D.I for details),
H(s) - --s)=-D
Σ b,s: 2
IEE Wap W Wnss,3j.
(10)
a pairwise Ising model with a particular coupling structure Jg = En Win Wi which in vector form
= Σ
Wp W WW
(11)
where {wy} the p columns of W.
In general, this Ising model Eq. 10 produced by integrating out the hidden variables need not have Hopfield structure (discussed below). However, it automatically does (as noted in |Barra et al. 2012)) ) in the very special case where W E [-1,+1), ). In that case, the binary patterns are simply (w,...s so that Eq. (1 represents Hopfield network with the Hebbian prescription. This situation is likely rare and may only arise as by-product of constrained training; for generically trained RBM the weights will not be binary. It is therefore interesting to clarify when and how real-valued RBM interactions W can be associated with HNs.
Approximate binary representation of W: In Section 2.1 we orthogonalized the binary matrix & via the QR decomposition = QR, where Q is an orthogonal (but non- non-binary) matrix, which allowed us map a projection HN (defined by its patterns g. Eq. (T) to an RBM (defined by its inter-layer weights Q, Eq. (
Here we consider the reverse map. Given a trained RBM with weights W E RNN we look for an invertible transformation X E RPXP which binarizes W. We make the mild assumption that W is rank P. we find such an X, then B WX will be the Hopfield pattern matrix (analogous to E), with Big E +1}.
This is a non-trivial problem, and an exact solution is not guaranteed. As a first step to study the problem, we relax it to that of finding matrix X E GL,(R) (i.e. invertible, p X p, real) which minimizes the binarization error
arg min x (R)
|[ - sgnn W
(12)
We denote the approximately binary transformation of W via particular solution X by
B, BX
(13)
We also define the associated error matrix E B, - sgn( B,)) We stress that B, is non-binary and the columns of which will be HN patterns under certain conditions
approximates B E. We provide an initial characterization and example in Appendix D
EXPERIMENTS ON MNIST DATASET
Next we investigate whether the Hopfield-RBa correspondence can provide an advantage for train- ing binary-gaussian RBMs. We consider the popular MNIST dataset of handwritten digits (LeCun) et al. 1998) which consists of 28 x 28 images of handwritten images, with greyscale pixel values to 255. We treat the sample images as N = 784 dimensional binary vectors of {-1, +1} by setting all non-zer values to +1. The dataset includes M = 60, 000 training images and 10, 000 testing images, well their class labels μa (0....,
4

---

Published as a conference paper at ICLR 2021
3.1 GENERATIVE OBJECTIVE
The primary task for generative models such as RBMs is to reproduce a data distribution. Given data distribution Pdata the generative objective is train model (here an RBM defined by its parameters 0), such that the model distribution pe is as close to Pdata as possible. This is often quan-
tified by the Kullback- Leibler (KL) divergence D (PdaullPe) = E. (s) In
Paus (a) Pa(s)
One
generally does not have access the actual data distribution, instead there is usually representa- tive training set S = { sampled from it. As the data distribution is constant with respect to 0, the generative objective is equivalent maximizing L(0) La In po(.)).
3.1.1 HOPFIELD RBM SPECIFICATION
With labelled classes of training data, specification of an RBM via a one-shot Hopfield rule ("Hop- field RBM") is straightforward. In the simplest approach, we define p = 10 representative patterns via the (binarized) class means
= sgn
(14)
S
where l {0, {O....,9) 9} and Sj is the set of sample images for class μr. These patterns comprise the columns of the N x p pattern matrix &. which is then orthogonalized as in Eq. (2) to obtain the RBM weights W which couple N binary visible units to p gaussian hidden units.
We also consider refining this approach by considering sub-classes within each class, representing, for example, the different ways one might draw a "7" As a proof of principle, we split each digit class into k sab--paterns using hierarchical clustering. We found good results with Agglomera- tive clustering using Ward linkage and Euclidean distance (see Murtagh & Contreras (2012) for an overview of this and related methods). In this way, we can define hierarchy of Hopfield RBMs. At one end, A: = we have our simplest RBM which has p = 10 hidden units and encodes 10 patterns (using Eq. (14)) one for each digit class. At the other end, 10k/N - 1, ingly refined RBMs that encode & sub sub- patterns for each of the 10 digit classes, for a total of p = 10k patterns and hidden units. This approach has additional cost of identifying the sub-classes, but is still typically faster than training the RBM weights directly (discussed below).
specify increas-
The generative performance funetion of is and 3 is shown in Fig. 2 and increases monotonically with k in the range plotted. If 3 is too high (very low temperature) the free energy basins will be very deep directly the patterns, and the model distribution will not capture the diversity of images from the data. If /3 is too low (high temperature), there is a "melting transition" where the original pattern basins disappear entirely, and the data will therefore be poorly modelled. Taking
p ~ 0.1 (roughly & = 8), Fig. of Kanter & Sompolinsky (1987) predicts 8m. z 1.5 for the theoretical melting transition for the pattern basins. Interestingly, this quite close to our observed peak near = 2. Note also as k is increased, the generative performance is sustained at lower temperatures.
-20
Figure 2: Hopfield RBM generative perfor- mance as function of 3 for varying numbers of encoded sub- sattpatterns per digit. The num- ber of hidden units in each RBM is p = 10k, corresponding the total number encoded patterns. In Z is computed using annealed im- portance sampling (AIS) NNal 200I) on the continuous representation of Z. Eq. (7), with 500 chains for 1000 steps (see Appendix Each curve displays the mean of three runs.
400
Inpo pe (c)
-600
k
800
1
2
3
1000
10
1200
8
10
In situations where one already has access to the class labels, this approach to obtain RBM weights is very fast. The class averaging has negligible computational cost O(MN) for the whole training
5

---

Published as a conference paper at ICLR 2021
set (M samples), and the QR decomposition has modest complexity of O(Np²) (Schott & Stewart Conventional RBM training. discussed below, requires significantly more computation.
1999).
3.1.2
CONVENTIONAL RBM TRAINING
RBM training is performed through gradient ascent the log-likelihood of the data, I((0) M E. In pe (s.) (equivalent here to minimizing KL divergence, as mentioned above). We are fo- cused here on the weights W in order compare to the Hopfield RBM weights, and so we neglect the biases on both layers. As is common (Hinton 2012), we approximate the total gradient by split- denoted B. The resulting gradient ascent rule for the
1
ting the training dataset into weights is (see Appendix
W++ W +1 (ZZ W just
(s;
(15)
aEB
where (Sidp) Z E(s) /si2pe- (sAAe-DHiooN) II, d)p is an average over the model distribution.
The first bracketed term of Eq. (15) is simple to calculate at each iteration of the weights. The second
term, however, is intractable as it requires one to calculate the partition function Z. We instead
approximate it using contrastive divergence (CD-K) (Carreira-Perpinan & Hinton (2005] Hinton 2012]. See Appendix for details. Each full step of RBM weight updates involves O(K BNp) operations (Melchior et al. 2017). Training generally involves many mini-batch iterations, such that the entire dataset is iterated over (one epoch) many times. In our experiments we train for 50 epochs with mini-batches of size 100 (3 10 weight updates), the overall training time be extensive compared to the one-shot Hopfield approach presented above. For further details on RBM training Melchior et al. 2017))
e.g. Hinton 2012);
In Fig. we give an example of the Hopfield RBM weights (for A: = 1), as well as how they evolve during conventional RBM training. Note Fig. Ba), (b) appear qualitatively similar, suggesting that
the proposed initialization Q from Eqs. (2) (14) may be near a local optimum of the objective.
0.15
Hopfield initialized (0 epochs)
0.00
-0.5
(b) Hopfield initialized (50 epochs)
Figure 3: Binary-gaussian RBM weights for p = 10 hidden units prior to and during generative
training. (a) Initial values of the columns of W (specified the orthogonalized Hopfield patterns
via Eqs. 14 (b) Same columns of W after 50 epochs of CD-20 training (see Fig. a))
In Fig. 4 a), (b), we compare conventional RBM training on four different weight initializations: (i) random Wip (O, 0.01) (purple), commonly used in the literature; (ii) our proposed weights from the projection rule Hopfield mapping for correlated patterns (blue); (iii) the "Hebbian" Hopfield
mapping described in previous work for uncorrelated patterns, W 1-2 [Baraa et al., 2012)
(green); and (iv) the top PCA components of the training data (pink). In Fig. (c.. (d) we compare generated sample images from two RBMs, each with p = 50 hidden units but different initial
weights (random in (c) and the HN mapping in (d)). The quality of samples in Fig. d) reflect the efficient training of the Hopfield initialized RBM.
Fig. aa) (b) show that the Hopfield initialized weights provide an advantage over other approaches during the early stages of training. The PCA and Hebbian initializations start at much lower values of the objective and require one or more epochs of training to perform similarly (Fig. 4[a) inset), while the randomly initialized RBM takes 25 epochs to reach similar level. All initializations ultimately reach the same value. This is noteworthy because the proposed weight initialization is fast compared conventional RBM training. PCA performs best for intermediate training times.

---

Published as a conference paper at ICLR 2021
-100
-200
330
200
Install RBM weights Wic Hophield (projection) PCAA
(c) Normal initialization, 15 epochs
-400
400
Hopfield (HHsb)aa)) W4 A00. 0.01)
5 5 5
500
2
20
-0
60
30 spach
<goch
(a) 10 hidden units
(b) 50 hidden units
(d) Hopfield initialization, 15 epochs
Figure 4: Generative performance of gaaassiin RBMs trained with (a) p = 10 and (b) p = 50
hidden units. Curves are colored according to the choice of weight initialization (see legend in (b), and further detail in the preceding text). Each curve shows the mean and standard deviation over 5 runs. The inset in (a) details the first two epochs. We compute In po as in Fig. 2 but with 100 AIS chains. The learning rate is 1/0 = 10-4 except the first 25 epochs of the randomly initialized weights in (b), where we used 1) = 5%/0 due to slow training. The mini-batch size is B 100 for all curves in (b) and the purple curve in (a), and B 1000 otherwise. (c), (d) Samples from two RBMs from (b) (projection HN and random) after 15 epochs, generated by initializing the visible state to an example image from the desired class and performing 20 RBM updates with B = 2. Training parameters: 2, and CD-20.
Despite being common choice, the random initialization trains surprisingly slowly, taking roughly 40 epochs in Fig. 4 a). and in Fig. (b) we had to increase the basal learning rate y/o = 10-4 by factor of 5 for the first 25 epochs due to slow training. The non-random initializations, by comparison, arrive at the same maximum value much sooner. The relatively small change over training for the Hopfield initialized weights supports the idea that they may be near local optimum of the objective, and that conventional training may simply be mildly tuning them (Fig. 3
That the HN initialization performs well O epochs suggests that the p Hopfield patterns concisely summarize the dataset. This is intuitive, as the projection rule encodes the patterns (and nearby states) as high probability basins the free energy landscape of Eq. []. As the data itself is clustered near the patterns, these basins should model the true data distribution well. Overall, our results suggest that the HN correspondence provides useful initialization for generative modelling with binary-gauee gaussian RBMs, displaying excellent performance with minimal training.
3.2 CLASSIFICATION OBJECTIVE
As with the generative objective, we find that the Hopfield initialization provides an advantage for classification tasks. Here we consider the closely related MNIST classification problem. The goal is to train model on the MNIST Training dataset which accurately predicts the class of presented images. The key statistic is the number of misclassified images on the MNIST Testing dataset.
We found relatively poor classification results with the single (large) RBM architecture from the preceding Section 3.1 Instead, we use a minimal product-of-experts (PoE) architecture as described in Hinton (2002): the input data is first passed to 10 RBMs, one for each class μ. This "layer of RBMs functions as a pre-processing layer which maps the high dimensional sample s to a feature vector f(s) R 10 This feature vector is then passed to logistic regression layer in order predict the class of 8. The RBM layer and the classification layer are trained separately.
The first step is to train the RBM layer to produce useful features for classification. As in Hinton (2002), each small RBM is trained to model the distribution of samples from a specific digit class μ.. We use CD-20 generative training as in Section 3.1 with the caveat that each expert is trained solely on examples from their respective class. Each RBM connects N binary visible units to l gaussian hidden units, and becomes an "expert" at generating samples from one class. To focus on the effect of interlayer weight initialization, we set the layer biases to 0.
After generative training, each expert should have relatively high probability p6(") (s.) for sample digits& s, of the corresponding class μa and lower probability for digits from other classes. This idea is used define 10 features, one from each expert, based on the log-probability of given sample
7

---

Published as a conference paper at ICLR 2021
under each expert, In Po (s.) -3H (8.) -ln)-Inn Z(o), Note that 3 and In Z(H) are constants with respect the data and thus irrelevant for classification. For a binary-gaussian RBM, /1 (1)) (s.) has the simple form Eq. TO), so the features we use are
(s)
ΣΣΣ
W s,5; =|s² W
(16)
With the feature map defined, we then train standard logistic regression classifier (using scikit- learn (Pedregosa et al. 2011)) on these features. In Fig. [ we report the classification error on the MNIST Testing set of 10,000 images (held out during both generative and classification training). Note the size p = 10 of the feature vector is independent of the hidden dimension each RBM, so the classifier is very efficient.
Product-of-experts classification performance for the various weight initializa- tions. For cach digit model (expert), per- form CD-20 training according to Eq. (15) (as in Fig. + for fixed number of epochs. A given sample image is mapped to the 10- dimensional feature vector with elements Eq.
Figure 5:
Initial RBM weights Winis
Wip ~ 0.0.0.01)
11
PCA
Test error (%)
Hopfield (projoction)
Number of sub-patterus
&
100
(6). These features are used to train a logistic regression classifier, and the average MNIST Test set errors are reported. The initializations considered for each expert's weights w R-Nx are Wi (N00, 0.01) (purple, dash- dot), PCA (pink, dashed), and the orthogo- nalized Hopfield sub-patterns for digit class μ (blue, solid). The error bars show the min/max of three runs.
10
20
30
40
50
epoch
Despite this relatively simple approach, the PoE initialized using the orthogonalized Hopfield pat- terns ("Hopfield PoE") performs fairly well (Fig. 5 blue curves), especially as the number of sub- patterns is increased. We found that generative training beyond 50 epochs did not significantly improve performance for the projection HN or PCA. (in Fig. E we train to 100 epochs and also display the aforementioned "Hebbian" initial condition, which performs much worse for classifi- cation). Intuitively, increasing the number of hidden units l increases classification performance independent of weight initialization (with sufficient training).
For l fixed, the Hopfield initialization provides a significant benefit to classification performance compared the randomly initialized weights (purple curves). For few sub-pattems (circles k 10 and squares A = 20), the Hopfield initialized models perform best without additional training and until epoch, after which PCA (pink) performs better. When each RBM has k 100 hidden features (triangles), the Hopfield and PCA PoE reach 3.0% training eтror, whereas the randomly initialized PoE reaches 4.5%. However, the Hopfield PoE performs much better than PCA with minimal training, and maintains its advantage until 10 epochs, after which they are similar. Inter- estingly, both the Hopfield and PCA initialized PoE with just = 10 encoded patterns performs better than equal to the As 100 randomly initialized PoE at each epoch despite having order of magnitude fewer trainable parameters. Finally, without any generative training (0 epochs), the k 100 Hopfield PoE performs slightly better (4.4%) than the k- = 100 randomly initialized PoE with 50 epochs of training.
4 DISCUSSION
We have presented an explicit, exact mapping from projection rule Hopfield networks to Restricted Boltzmann Machines with binary visible units and gaussian hidden units. This provides gener- alization of previous results which considered uncorrelated patterns (Barra et al. 2012), or special cases of correlated patterns (Agliari et al. 2013 Mézard 2017). We provide an initial characteriza- tion of the reverse map from RBMs to HNs, along with a matrix- factorization approach to construct approximate associated HNs when the exact reverse map is not possible. Importantly, our HN to
8

---

Published as a conference paper at ICLR 2021
RBM mapping can be applied correlated patterns such those found in real world datasets. As a result, we are able to conduct experiments (Section 3 on the MNIST dataset which suggest the mapping provides several advantages.
The conversion an HN to an equivalent RBM has practical utility: it trades simplicity of presen- tation for faster processing. The weight matrix of the RBM is potentially much smaller than the HN (Np elements instead of [ - 2). More importantly, proper sampling of stochastic trajectories in HNs requires asynchronous updates of the units, whereas RBM dynamics can be simulated in parallelizable, layer-wise fashion. We also utilized the mapping to efficiently estimate the partition function of the Hopfield network (Fig. 2 by summing out the spins after representing an RBM.
This mapping also has another practical utility. When used as an RBM weight initialization, the HN correspondence enables efficient training generative models (Section 3.1 Fig. 4. RBMs initialized with random weights and trained for moderate amount of time perform worse than RBMs initialized orthogonalized Hopfield patterns and not trained at all. Further, with mild training of just few epochs, Hopfield RBMs outperform conventionally initialized RBMs trained several times longer. The revealed initialization also shows advantages over alternative non-random initializations (PCA and the "Hebbian" Hopfield mapping) during early training. By leveraging this advantage for generative tasks. we show that the correspondence can also be used to improve classification performance (Section 3.2 Fig. Appendix E.3
Overall, the RBM initialization revealed by the mapping allows for smaller models which perform better despite shorter training time (for instance, using fewer hidden units achieve similar clas- sification performance). Reducing the size and training time of models critical, as more realistic datasets (e-g. gene expression data from single-cell RNA sequencing) may require orders of mag- nitude more visible units. For generative modelling of such high dimensional data, our proposed weight initialization based on orthogonalized Hopfield patterns could be of practical use. Our the- ory and experiments are a proof-of of ofiicciie if they can be extended to the large family of deep architectures which are built upon RBMs, such as deep belief networks (Hinton et al., (206) and deep Boltzmann machines (Salakhutdinov & Hinton 2009), it would be of great benefit. This will be explored future work.
More broadly, exposing the relationship between RBMs and their representative HNs helps to ad- dress the infamous interpretability problem machine learning which criticizes trained models as "black boxes". HNs are relatively transparent models, where the role of the patterns as robust dynamical attractors is theoretically well-understood. We believe this correspondence, along with future work to further characterize the reverse map, will be especially fruitful for explaining the performance of deep architectures constructed from RBMs.
ACKNOWLEDGMENTS
The authors thank Duncan Kirby and Jeremy Rothschild for helpful comments and discussions. This work is supported by the National Science and Engineering Research Council of Canada (NSERC) through Discovery Grant RGPIN 402591 to A.Z. and CGS-D Graduate Fellowship to M.S.
REFERENCES
David Ackley, Geoffrey E Hinton, and Terrence J. Sejnowski. learning algorithm for boltz- mann machines. Cognitive Science, 9():1177-16, 1985. ISSN 03640213. doi: 10.1016/
50364-02133(33800122-4
Elena Agliari, Adriano Barra, Andrea De Antoni, and Andrea Galluzzi. Parallel retrieval of cor- related patterns: From Hopfield networks to Boltzmann machines. Neural Networks, 38:52-63, 2013. ISSN 08936080. doi: new.et..cumm...212....... 1.000. URL http: //tp doi http://w....oo./...oo//o org/10. 1016/1.neun 2012 1..02.11.10
Elena Agliari, Adriano Barra, Chiara Longo, and Daniele Tantari. Neural Networks Retrieving Boolean Patterns in a Sea Gaussian Ones. Journal of Statistical Physics, 16855):0055-1104, 2017. ISSN 00224715. doi: 10. 10.10077/19555-07-1111

---

Published as a conference paper at ICLR 2021
Daniel J. Amit. Modeling Brain Function: The World of Attractor Neural Nenworks. Cambridge University Press, sep 1989. doi: 1L1017/cbo978051622222
Daniel J. Amit, Hanoch Gutfreund, and H. Sompolinsky. Spin-glass models of neural networks.
Physical Review A, 320):1007-1018, 1985. ISSN 10502947. doi: 10.1105/PhysRevA.32....
Adriano Barra and Francesco Guerra. About the ergodic regime in the analogical Hopfield neural networks: Moments of the partition function. Journal of Mathematical Physics, 49(12), 2008.
ISSN 00222488. doi: 10.
Adriano Barra, Alberto Bernacchia, Enrica Santucei, and Pierluigi Contucci. On the equivalence of Hopfield networks and Boltzmann Machines. Neural Networks, 34:1-9, 2012. ISSN 08936080. doi: 10. hww.10/66....nnettcc.. memee.200.00002000000000.
/xx wttp///.o...o//1.001111...1 URL http: /ett::/dx..e
neunet.
201 2. 06.006.66 003]
Miguel A Carreira-Perpinal and Geoffrey E Hinton. On contrastive divergence learning. In Aistats, volume 10, pp. 33-40. Citeseer, 2005.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504 507, 2006. ISSN 0036-8075. doi: 10.1126/science. 127647. URL
https: www.ennceee. www.sciiinnnn.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8)11771-1800, 2002. ISSN 08997667. doi: 10. N0.166220977662220222111
Geoffrey E. Hinton. A practical guide to training restricted boltzmann machines.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture
Notes in Bioinformatics), 7700 LECTU:599-619, 2012. ISSN 03029743. doi: 10.1007/
978-3-642-35289---32.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 2006. ISSN 08997667. doi: 10.1162/neco 2006. 1 18.7.1527.
JJ Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(8):2554- 2558, 1982. ISSN 00278424. doi: 10. 10.10033pnas.79 79...2544.
I. Kanter and H. Sompolinsky. Associative recall of memory without errors. Physical Review A, 35 (1):380-392, 1987. ISSN 10502947. doi: 10. 103/PhysRevA.35.300
Scott Kirkpatrick and David Sherrington. Infinite-ranged models of spin-glasses. Physical Review B, 17(11)4484-4403, 1978. ISSN 01631829. doi: 10.1103/Physeev..1...
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bart based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23) 2323, 1998. ISSN 00189 00189219. doi: 10.1109/5.726791.
Per Olov Löwdin. On the Nonorthogonality Problem. Advances Quantum Chemistry, 5(C): 185-199, 1970. ISSN 00653276. doi: 10.1016/50065-32766 3270000)6326000003399111
Chiara Marullo and Elena Agliari. Boltzmann machines generalized hopfield networks: A review of recent results and outlooks. Entropy, 23(1):1-16, 2021. ISSN 10994300. doi: 10.3390/ e23010034.
Pankaj Mehta, Marin Bukov, Ching Hao Wang. Alexandre G.R. Day, Clint Richardson. Charles K Fisher, and David J Schwab. A high-bias, low-variance introduction to Machine Learning for physicists, 2019. ISSN 03701573.
Jan Melchior, Nan Wang, and Laurenz Wiskott. Gaussian-binary restricted Boltzmann machines for modeling natural image statistics. PLoS ONE, 12(2), 2017. ISSN 19326203. doi: 10.1371/ Nournal.pone.017110 0171015.
Mare Mézard. Mean-field bessaag-paassing equations in the Hopfield model and its generalizations. Physical Review E, 95(2):1-15, 2017. ISSN 24700053. doi: 10. 10.11033 100/PhyyReeE..55022217 17.
10

---

Published as a conference paper at ICLR 2021
Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-97, 2012. ISSN 19424787. doi: 10.1002/widm.53.
Radford M. Neal. Annealed importance sampling, Statistics and Computing, 11(2):125-133) 2001. ISSN 09603174. doi: 10. (0.1023/A110089922155000
Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duch- esnay, Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12: 2825-2830, 2011. ISSN 15324435.
L. Personnaz, I. Guyon, and G. Dreyfus. Collective computational properties of neural networks: New learning mechanisms. Physical Review A, 34(5):2117-4228, 1986. ISSN 10502947. doi: 10.110/PPhysRevA.34.2111
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann machines. In Journal of Machine Learning Research, volume 5, pp. 448 4558 2009.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th International Conference on Machine Learn- ing, ICML '07, pp. 791-798, New York, NY, USA, 2007. Association for Computing Machin- URL http::: wttpo/...........
ery. ISBN 9781595937933. doi: 10.1145/127349.1125 1273596.
1145/12734 1273596
James R. Schott and G. W. Stewart. Matrix Algorithms, Volume Basic Decompositions. Journal of the American Statistical Association, 94(448):1388, 1999. ISSN 01621459. doi: 10.2307/ 2669960.
A HOPFIELD NETWORK DETAILS
Consider p < N-dimensional binary patterns (5") j==1 that are to be "stored". From them, construct the N x x matrix & whose columns are the p patterns. If they are mutually orthogonal (e.g. randomly sampled patterns in the large N - 00 limit), then choosing interactions according to the Hebbian rule, JHsh TST" guarantees that they will all be stable minima H(s) = - Js, provided p/N Oce where a, 0.14 Amit et al. 1985). If they (referred to correlated), then using the "projection rule" Jpmi = - guarantees that they will all be stable minima of H ((s), provided p < N (Kanter & Sompolinsky 1987 Personnaz as
mutually orthogonal
al. 1986). Note JProj + Jebb in the limit of orthogonal patterns. In the main text, shorthand for Jrroj-
We provide some relevant notation from Kanter & Sompolinsky (1887). Define the overlap state with the p patterns as mss) = t s, and define the projection of a state onto the p patterns as as)) = ()" - = A A- ms). Note A = ==" is the overlap matrix, and my a2 E 1-
We can re-write the projection rule Hamiltonian Eq. ① as
NN' H(s)
2 m a(s).
For simplicity, we include the self-interactions rather than keeping track of their omission; the results are the same as N + co. From Eq. (A several quadratic forms can be written depending on which variables one wants work with:
((s) - ---2 Nmm (TTTETTT mm
II. H(s) = - a
These are the starting points for the alternative Boltzmann Machines (L.e. not RBMs) presented in Appendix
11

---

Published as a conference paper at ICLR 2021
B ADDITIONAL HN TO RBM MAPPINGS
We used QR factorization in the main text to establish the HN to RBM mapping. However, one can use any decomposition which satisfies
JProj UU
B
such that U Rp orthogonal (orthogonal for tall matrices means U I). that case, U becomes the RBM weights. We provide two simple alternatives below, and show they are all part of the same family of orthogonal decompositions.
"Square root" decomposition: Define the matrix K = E) Note that K is orthogonal, and that JProj K KKK
Singular value decomposition: More generally, consider the SVD of the pattern matrix E:
& = -UVV
B 2
where EXP, V Rexp store the left and right singular vectors (respectively) of f as orthog- onal columns, and E E R2xp is diagonal and contains the singular values of &. Note in the limit of orthogonal patterns, we have E = /.I. This decomposition gives several relations for quantities of interest:
= ==
VW
Hobb
N EET
1
UT
B B
JPno = <(5" 1 UU.
The last line is simply the diagonalization of JPojj and shows that our RBM mapping is preserved we swap the Q from QR with U from SVD. However, since there are p degenerate eigenvalues
x² = / is not unique any orthogonal basis for the 1-eigenspace can be chosen. Thus U" UO
where O is orthogonal also valid, and the QR decomposition and "square root" decomposition correspond to particular choices of O.
C HN TO BM MAPPINGS USING ALTERNATIVE REPRESENTATIONS OF THE HOPFIELD HAMILTONIAN
In addition the orthogonalized representation in the main text, there natural representations to consider based on the pattern overlaps and projections introduced in Appendix A These lead to generalized Boltzmann Machines (BMs) consisting of N original binary spins and p continuous variables. These representations are not RBMs as the continuous variables interact with each other. We present them for completeness.
"Overlap" Boltzmann Machine: Writing H(s) = - mm m="(E"E) ('m,
'm, we have
7 = Σ exp ( (OVNT)) (NET5) (BVNm))
C
{s)
Applying the gaussian integral identity,
Z - /det (E"
Σ
in (lau 3 E,, Ap E, Si
dd
2
{s}
from which we identify the BM Hamiltonian
H(s = 2N
1
VN ΣΣ App
MAN
This is the analog of Eq. 66 in the main text for the "overlap" representation. Note we can also sum
the binary variables Eq. (C2), which allows for an analogous expression Eq. (8)
Fo((A,))
1
Σ (ETTE)
Σ In cosh
B
Σ
2N
VN
12

---

Published as a conference paper at ICLR 2021
Curiously, note that may perform second Gaussian integral on Eq. auxiliary variables Ty to remove the interactions between the Ap variables:
introducing
Z
V det
82 N
E
Σ (0)
E.Je
x" (E"
II
p 2m
C5)
Eq. C describes three-layer RBM with complex interactions between the A and variables, a representation which could be useful in some contexts.
" "Projection" Boltzmann Machine: Proceeding above but for H(s) --aa" 2g""(""0 E), finds
det (ET&)
1/22
Σ (8)
GAT(ETE) (ETRAT(ETE)) LET ddp
6
/
which corresponds to the BM Hamiltonian
H(s, A) 1 -xxx" XE -E"
7
The analogous expression to Eq.
8 in this case is
Fo(A) { (E Σ In cosh (B(E(E" (((((E"()-11))))) =>_)),))
(88)
D RBM TO HN DETAILS
D.1 INTEGRATING OUT THE HIDDEN VARIABLES
The explanation from Mehta et al. 2019) for integrating out the hidden variables of an RBM is presented here for completeness. For given binary-gaussian RBM defined by HRBM (s, X) (as in
Eq. (9)) we have p(s) z-1 - (e--HaaM(sAA) dX. Consider also that p(s) ---e-996 (e) for some
unknown (s). Equating these expressions gives,
1 Σ In ( (e-AAAEtEE 3
H(s) = Σ b,s;
+ W Muussrdddy
D 1)
I by defining qu (A,.). a gaussian with zero mean and variance 8-1 Writing t, B E, W Sis one observes that the second term (up to constant) is sum of cumulant generating functions, i.e.
Decompose the argument of In(.) in Eq.
(t(()) = In (e Jap In (ʃ @netudnddyy)
D 2)
ak 0t, 10,=0 is the th cumulant of qu. However, since qu (An)) is gaussian, only the second term remains, leaving (t,(t) E, a.) Putting this all together, have
These can be written as a cumulant expansion, Kz(ta)= En=1
(n) (n) Mi n!. where Ki
H(s) = Eb. 2 ΣΣΣ
W W Wissssj
D
Note that in general, (A,) need not be gaussian, in which case the resultant Hamiltonian H(s) can have higher order interactions (expressed via the cumulant expansion).
D.2 APPROXIMATE REVERSE MAPPING
Suppose one has a trial solution B, = WX to the approximate binarization problem Eq. (12 with emror matrix E B, sgn( B,). We consider two cases depending if W is orthogonal.
13

---

Published as a conference paper at ICLR 2021
Case 1: If W is orthogonal, then starting from Eq. II and applying Eq-
13
we have J
WW = (x "B"
BT" Using I = W W. we get
J= B
Thus, the interactions between the visible units is the familiar projection rule used to store the approximately binary patterns pp "Storage" means the patterns stable fixed points of the deter- ministic update rule s sgn(Js").
We cannot initialize the network to a non-binary state. Therefore, the columns of B = sgn(B,, are candidate patterns. To test if they are fixed points, consider
sgn(JB) sgn - J JE)
sgn( -JE).
We need the error E to be such that J E will not alter the sign of Bp. Two sufficient conditions are:
(a) small error: (l < ((Bp) or (b) error with compatible sign: (JE) ()))), < O.
When either these conditions hold for each element, we have (JB( sgn( B, so that the candidate patterns B are fixed points. It remains to show that they are also stable (i.e. minima).
Case 2: If W is not orthogonal but its singular values remain close one. then Löwdin Orthogo- nalization (also known Symmetric Orthogonalization) (Löwdin 1970) provides way preserve the HN mapping from Case above.
Consider the SVD of W: W UEVT. The closest matrix to W (in terms of Frobenius norm) is called the Löwdin Orthogonalization of W. Note the approximation becomes exact when all the singular values are one. We then write WW WT UUT and the orthogonal W approach of Case can then be applied.
with orthogonal columns is
UV and the approximation
On the other hand, W may be strongly not orthogonal (singular values far from onc). If it is still full rank, then its pseudo-inverse W (W" W) W VE ""' well-defined. Repeating the steps from the orthogonal case, note here X x B (W Defining C' (Wi)Twi UE UUT we arrive the corresponding result,
J= B, CB, CB,)
5
This is analogous to the projection rule but with "correction factor" C. However, it is not imme- diately clear how affects pattern storage. Given the resemblance between JPıoj and Eq. (D44 (relative to Eq. (D expect that RBMs trained with an orthogonality constraint the weights may be more readily mapped to HNs.
D.3 EXAMPLE OF THE APPROXIMATE REVERSE MAPPING
(12.. the solutions of which provide approximately binary patterns through Fq. (13). To numerically solve Eq. 12 and
In the main text we introduced the approximate binarization problem Eq.
obtain a candidate solution X", we perform gradient descent on differentiable variant.
Specifically, we replace sgn(u) u) with tanh for large Define E = W tanh(a (X) as the main text. Then the derivative of the "softened" Eq. (12) with respect to X is
G(X) 2W (E a @ sech (WX)))
Given an initial condition Xu, we apply the update rule
X X, Xxxx)(()))
until convergence a local minimum X
In the absence of prior information, we consider randomly initialized X,. Our preliminary experi- ments using Eq. (D77) to binarize arbitrary RBM weights W have generally led to high binarization enror. This is due in part to the difficulty in choosing a good initial condition Xo, which will be explored in future work.
14

---

Published as a conference paper at ICLR 2021
To avoid this issue, consider the case of Hopfield-initializzel RBM following CD-k training. At epoch O, we in fact have the exact binarization solution X R (from the QR decomposition, Eq. (2). which recovers the encoded binary patterns &. We may use Xo R. informed initial condition to Eq. (D77). to approximately binarize the weight at later epochs and monitor how the learned patterns change.
In Fig. D we give an example of this approximate reverse mapping for a Hopfield-initializes RBM following generative training (Fig. 4.. Fig. D[I shows the p 10 encoded binary patterns, denoted below by En. and Fig. bb) shows the approximate reverse mapping applied to the RBM weights at epoch 10. We denote these nearly binary patterns by E10- Interestingly, some of the non-binary regions in Fig. [DJlb) coincide with features that distinguish the respective pattern. For example, the strongly "off" area in the top-right of the "six" pattern.
2,50
0 6
5
7
9
1.50
0.75
(a) Encoded Hopficld patterns
-0.75
1150
(b) Recovered Hopfield patterns (10 epuchs)
2.50
Figure D Reverse mapping example. (a) The p 10 encoded binary patterns used to initialize the RBM in Fig. 4a). (b) The approximate reverse mapping applied to the RBM weights after 10 epochs of CD-k training. Parameters: 200 and y 0.05.
We also considered the associative memory performance of the projection HNs built from the the patterns Eo. É10 from Fig. D Specifically, we test the extent to which images from the MNIST Test dataset are attracted the correct patterns. Ideally, each pattern should attract all test images of the corresponding class. The results are displayed in Fig. 2 and elaborated in the caption.
Net accuracy 76.8%
Net accuracy 73.7%
B83
0
10
14
27
31
772
12
19
7
59
58
2
32
1000
1106
0
3
8
2
14
1112
0
3
0
4
2
0
14
0
31
117
711
17
91
0
39
22
50
22
120
764
14
33
2
33
17
41
5
800
34
793
True labol
6GL
34
39
728
600
28
107
247
465
33
15
115
151
25
169
20
34
28
53
15
2
77
2B
740
18
61
25
1
80
28
730
3
400
80
11
0
16
1
3
8:30
18
61
106
12
18
1
2
779
26
73
20
45
10
88
5
16
14
10
726
14
51
14
129
30
17
12
649
46
200
15
34
5
10
35
11
158
14
2
0
1
2
3
2
3
6
Fixed point
Fixed point
(a) Epoch 0 patterns
(b) Epoch 10 patterns
Associative memory task using (a) Es. the initial Hopfield patterns, and (b) E10- the patterns recovered from reverse mapping after 10 epochs. The patterns are used to construct projection HN as in Eq. (T) Each sample from the MNIST Test set is updated deterministically until reaches local minimum of the HN. If the fixed point is one of the encoded patterns, the sample contributes value I to the table. Otherwise, and ensemble size n = 20 until one of the encoded patterns is reached, defined as an overlap N-1 "T sgn(E") > 0.7, with each trajectory contributing 1/n to the table.
Figure D 2:
perform stochastie updates with 3 2.0
15

---

Published as a conference paper at ICLR 2021
The results in Fig. D2 suggest that 10 patterns may be too crude for the associative memory network to be used as an accurate MNIST classifier (as compared to e.g. Fig. 5p. Notably, the HN constructed from É performs about 3% worse than the HN constructed from Eo. although this per- formance might improve with a more sophisticated method for the associative memory task. There may therefore be cost, in terms of associative memory performance, to increasing the generative functionality of such models (Fig. 4). Our results from Appendix D.2 indicate that incorporating an orthogonality constraint on the weights during CD-k generative training may provide way to preserve increase the associative memory functionality. This will be explored in future work.
E
RBM TRAINING
Consider general binary-gaussian RBM with N visible and hidden units. The energy function is
Hs, A) = 50 cn) -
b,s . ΣΣ
WinssAa
First, we note the Gibbs-block update distributions for sampling one layer of a gaausssnn RBM given the other (see e.g. Melchior et al. (2017).
Visible units: p(s; 1 | 22 where H; = E, Win d; defines input to 8i.
Hidden units: p(> {(s) NN(h, 8-1). where hp E, Whss; St + Cp. defines the input to Ap=
E.1 GENERATIVE TRAINING
For completeness, we re-derive the binary-gaussian RBM weight updates, along the lines of Mel- chior et al. (2017). We want to maximize L = M Ea In po(a).. The contribution for a single datapoint 8, has the form In po In(C-1 f (d In Z with C (2 B))/ The
gradient with respect to the model is
( A) )c 2H II, dip E(a) sidne ( 11, dd
i In (o()s) 00
se-BH A) II ddp
Z
We are focused on the interlayer weights, with 0H(s.A) oW..
sid so
i In po(ss)
s(Ape (( II dyu
E(s) a Ssiuue -H((..
IT, dd
3((80) e- (BH(( II ddp
B
00
Z
=3(ss)i (> ma)
The first term is straightforward to compute: (A 1s- -8a) E, W (sa) Cp- The second
term is intractable and needs to be approximated. We use contrastive divergence
& Hinton 2005 Hinton 2012): (s:A ~ X
(k) Here h denotes CD-k - k steps of Gibbs-
(k) (k) comprise the final state. We evaluate both
block updates (introduced above) - from which 8
terms over mini-batcher of the training data to arrive at the weight update rule Eq. (5).
E.2 GENERATIVE PERFORMANCE
We are interested in estimating the objective function L = M Ea In po(s)) during training. As above, we split into two terms,
I L= In= In (o-1e--
(HH(en.)))
II ddp
Z,
with C' (2T/3)//2 The first term evaluates
B M
bT 8n +
- WT sal))
wT'
B
M
16

---

Published as a conference paper at ICLR 2021
which be computed deterministically. In Z, on the other hand, needs be estimated. For this we perform Annealed Importance Sampling (AIS) (Neal 2001) on its continuous representation
== / c - E, In cosh Ai
II dd
For AIS we need to specify the target distribution's un-normalized log-probabilitte
In(Zp(A)) = In
P
In
2x
Σ (Ap - 22)2+ Σ In cosh
Σ
Windy + bū
2
3
2
well initial proposal distribution anneal from, which fix p-dimensional unit normal distribution (O,, I).
E.3 CLASSIFICATION PERFORMANCE
Here we provide extended data (Fig. on the classification performance shown Fig. 5 As in the main text, color denotes initial condition (introduced in Section [.1) and shape denotes the number of sub-patterns. Here we train for each 100 epochs, which allows convergence of most of the curves. We also include the "Hebbian" initialization (green curves).
Initial RBM weights W Wini
11
Win ~N(00.001)
Hopfield (Hebbian)
10
PCA
9
Hopfield (projection)
Number of sub-patterns k
Test error (%)
8
0
A
10
20
100
7
6
5
4
3
10
20
30
40
50
60
70
80
90
100
epoch
Figure 1: Product-of-experts classification performance (extended data).
Notably, the Hebbian initialization performs quite poorly in the classification task (as compared to direct generative objective, Fig. 4). In particular, for the 100 sub-patterns case, where the projection
17

---

Published as a conference paper at ICLR 2021
HN performs best, the Hebbian curve trains very slowly (still not converged after 100 epochs) and lags behind even the 10 sub-pattern Hebbian curve for most of the training. This emphasizes the benefits of the projection rule HN over the Hebbian HN when the data is composed of correlated patterns, which applies to most real-world datasets.
18


Reviews:
[{"review_id": "qHPewiE3o1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers a mapping between the well known Hopfield Neural Networks and Restricted Boltzmann Machines. In contrast with previous literature that consider the case where the patterns / data features to memorize were uncorrelated, the authors extend the mapping to arbitrarily correlated patterns, which allows to consider much more realistic settings. The mapping is computationally speaking relatively cheap. This mapping is shown to allow for significantly better initialization (than random) of the weights of a RBM, in the sense that the training is then much faster to reach comparable generative and/or generalization performance. In this sense the mapping is not only interesting from a theoretical point of view, but also practically. This paper should be considered as an applied one, as there is no real analytic theory of why this mapping helps the learning, but the experiments are well carried: the boost in learning is demonstrated through experiments in MNIST data, and the results are well explained and convincing. The appendices are also well written and are a good addition to the main part. Overall the paper is well written (the paper can be used by non-specialists also as introduction to Hopfield NNs and RBMs), the results are interesting and relevant to the ML community, the paper can be read without much effort. Even if RBM are not anymore state-ot-the art generative models, the results are encouraging and might lead to future improvements in more modern architectures. I have no specific concern. The paper is overall very well written. The paper is slightly incremental as similar mappings were known, but it remains a relevant contribution, and the aspect of using this mapping as a way to boost learning in RBM seems new, and interesting. I recommend publication after slight corrections, see below.\n\nTypos and corrections:\n_text below (1): J=1/N Xi^T Xi^T ->1/N Xi Xi^T \n_(3): please detail the last equality\n_(8) is true only for the lambda that verify the fixed point / saddle point equations: please mention it\n_below (11): the p the columns of -> the p columns of\n_(12): please explain what is GL_p(R)\n_\"At the other end, 0 ≪ 10k/N < 1, ...\" : Any x > 0 is >> 0, so please be more precise\n_Above Fig 3: \"appear qualitatively similar\" : this is not obvious...\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mkAx8mWK_S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper shows a relationship between the project rule weights of a Hopfield network (HN) and the interaction weights in a corresponding restricted Boltzmann machine (RBM). The mapping from HN to RBM is facilitated by realising that the partition function of BN can be seen as the partition function of a binary-continuous (Bernoulli-Gaussian) RBM. The authors comments on the mapping from RBM to BN. The experiments show the advantages of training RBM with weights initialised from BN projection weights in generation and classification.\n\n## Strong points:\n+ I am not familiar with the literature, but the results seem new to me. \n+ The experiments show advantages of BN initialisation, pointing to new directions of improving RBM training.\n+ The paper is fairly clearly written.\n\n## Weak points\n\n- The HN -> RBM mapping is quite clear, but the reverse RBM -> HN mapping is not very well established, and there are no experiments showing how effective the approximate reverse mapping works on associative memory tasks typical for HNs. I also believe this lowers the impact of this paper, given that the forward mapping is based on a simple revelation.\n- The authors' description of the experimental results are not accurate enough. and the results raise several questions to be addressed.\n\n# Recommendation\n\nI'm in favour of rejection, but some concerns can be addressed fairly easily (with experiments) so I'm open to raising my score if questions are well-addressed.\n\n## Issues and questions to address\n\n* The authors should provide experiments on the reverse mapping as suggested above.\n* I do not agree that figure 3 shows that RBM training \"simply 'fine tunes'\" the weights -- the difference is quite stark. How about increasing the batch size so that there is little SGD noise?\n* Figure 4a: traces are cut-off just when random initialization is catching up with HN initialization. This also applies to Figure 5. \n* There are a few descriptions suggesting \"HN init. appears to train much faster than random init\". However, the rate of increase in of likelihood in Figure 4 is shallower for HN than for rand init. Is the advantage only at the 0'th RBM epoch? \n* The author only compared with purely random initialisation, which is perhaps the most naive baseline. I would suggest comparing to a (slightly) more clever initialisation, perhaps PCA or something better (those mappings in previous work the authors cited and in Appendix B). Or, the authors could also initialise the RBM by first training it on the within-class cluster centres (using a very large number of sleep samples for the sleep-phase) which may also be a more fair comparison?\n* In the classification objective, if I understand correctly, the feature function is essentially quadratic in the input patterns. Should there be an ideal test error that is computed by a quadratic neural network trained with supervision by backpropagation? If the HN classifier (blue in Figure 5) is approaching this idealization, then this will strengthen the claim.\n* The discussion on the extension to more generic, deep architectures is not well supported, and I do not see the extension to be so straightforward given the content of the current paper. In generation, supervised labels and clustering are used to simplify learning. Is the network able to learn just on the MNIST digits, even for the real images within a single class (e.g. \"7\")?\n* Can the authors try to characterise whether the HN initialisation is related to log-likelihood training? I wonder if there is any interesting theory; otherwise, measuring model performance by log-likelihood seems a bit arbitrary (though it makes the comparison to contrastive divergence easier).\n\n# Detailed suggestions (not to affect decision)\n\n* Reference to RBMs should include more historic ones from Hinton (e.g. 2006)\n* I do not see the purpose of (7) and (8), and they are only referred to in the Appendix (the review content in the Appendix is informative by itself though).\n* Eqn (11), should it be $w_\\mu w_\\mu^T$ in the sum?\n* Third line above (16), should $H$ and $Z$ be indexed by $\\mu$?\n* Line above (D.4), $WW^T = \\dots B_p^T$?\n\n\n==== update ====\n\nI thank the authors for providing such detailed response. All my concerns are addressed and reflected in the revision (though some are much better done than the rest). I congratulate the authors on their spirit of maintaining a high standard on the theory, experiments and descriptions, and therefore significantly raise my score. I hope to see this paper accepted. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "lSYPM0mjLSJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper demonstrates a mathematical equivalence between Hopfield nets and RBMs, and it shows how this connection can be leveraged for better training of RBMs.\n\nWhat a great paper - well written, an enlightening mathematical connection between two well-known models that to my knowledge was not previously known.  Hopfield nets and RBM's have been around for decades, and I don't think we've been aware of this connection, so it seems like a pretty important finding.  The paper explores the utility of this connection by applying to an MNIST task.  Interestingly, the connection yields important insights in both directions: stochastic sampling in an RBM is faster than Hopfield due to a smaller matrix and parallel layer wise updates, whereas initializing an RBM with the projection rule from Hopfield allows it to find a better solution faster.\n\nI really enjoyed reading the paper, I learned something new, and I think others will too!  It is an important advance in our understanding of Hopfield nets and RBMs.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two knowledgeable reviewers were positive 7 and very positive 10 about this paper, considering it an important contribution that illuminates previously unknown aspects of two classic models, namely RBMs and Hopfield networks. They considered the work very well developed, theoretically interesting and also of potential practical relevance. A third reviewer initially expressed some reservations in regard to the inverse map from RBMs to HNs and the experiments. Following the authors' responses, which the reviewer found detailed and informative, he/she significantly raised his/her score to 7, also emphasizing that he/she hoped to see the paper accepted. With the unanimously positive feedback, I am recommending the paper to be accepted. 
Decision: Accept (Oral)