Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: The paper demonstrates a mathematical equivalence between Hopfield nets and RBMs, and it shows how this connection can be leveraged for better training of RBMs.

What a great paper - well written, an enlightening mathematical connection between two well-known models that to my knowledge was not previously known.  Hopfield nets and RBM's have been around for decades, and I don't think we've been aware of this connection, so it seems like a pretty important finding.  The paper explores the utility of this connection by applying to an MNIST task.  Interestingly, the connection yields important insights in both directions: stochastic sampling in an RBM is faster than Hopfield due to a smaller matrix and parallel layer wise updates, whereas initializing an RBM with the projection rule from Hopfield allows it to find a better solution faster.

I really enjoyed reading the paper, I learned something new, and I think others will too!  It is an important advance in our understanding of Hopfield nets and RBMs.


review_title: A nice theoretical exposition and result!

final_score: rating: 7: Good paper, accept
review_content: This paper considers a mapping between the well known Hopfield Neural Networks and Restricted Boltzmann Machines. In contrast with previous literature that consider the case where the patterns / data features to memorize were uncorrelated, the authors extend the mapping to arbitrarily correlated patterns, which allows to consider much more realistic settings. The mapping is computationally speaking relatively cheap. This mapping is shown to allow for significantly better initialization (than random) of the weights of a RBM, in the sense that the training is then much faster to reach comparable generative and/or generalization performance. In this sense the mapping is not only interesting from a theoretical point of view, but also practically. This paper should be considered as an applied one, as there is no real analytic theory of why this mapping helps the learning, but the experiments are well carried: the boost in learning is demonstrated through experiments in MNIST data, and the results are well explained and convincing. The appendices are also well written and are a good addition to the main part. Overall the paper is well written (the paper can be used by non-specialists also as introduction to Hopfield NNs and RBMs), the results are interesting and relevant to the ML community, the paper can be read without much effort. Even if RBM are not anymore state-ot-the art generative models, the results are encouraging and might lead to future improvements in more modern architectures. I have no specific concern. The paper is overall very well written. The paper is slightly incremental as similar mappings were known, but it remains a relevant contribution, and the aspect of using this mapping as a way to boost learning in RBM seems new, and interesting. I recommend publication after slight corrections, see below.

Typos and corrections:
_text below (1): J=1/N Xi^T Xi^T ->1/N Xi Xi^T 
_(3): please detail the last equality
_(8) is true only for the lambda that verify the fixed point / saddle point equations: please mention it
_below (11): the p the columns of -> the p columns of
_(12): please explain what is GL_p(R)
_"At the other end, 0 ≪ 10k/N < 1, ..." : Any x > 0 is >> 0, so please be more precise
_Above Fig 3: "appear qualitatively similar" : this is not obvious...

review_title: ICLR review for "On the mapping between Hopfield networks and Restricted Boltzmann Machines"

final_score: rating: 7: Good paper, accept
review_content: # Summary

This paper shows a relationship between the project rule weights of a Hopfield network (HN) and the interaction weights in a corresponding restricted Boltzmann machine (RBM). The mapping from HN to RBM is facilitated by realising that the partition function of BN can be seen as the partition function of a binary-continuous (Bernoulli-Gaussian) RBM. The authors comments on the mapping from RBM to BN. The experiments show the advantages of training RBM with weights initialised from BN projection weights in generation and classification.

## Strong points:
+ I am not familiar with the literature, but the results seem new to me. 
+ The experiments show advantages of BN initialisation, pointing to new directions of improving RBM training.
+ The paper is fairly clearly written.

## Weak points

- The HN -> RBM mapping is quite clear, but the reverse RBM -> HN mapping is not very well established, and there are no experiments showing how effective the approximate reverse mapping works on associative memory tasks typical for HNs. I also believe this lowers the impact of this paper, given that the forward mapping is based on a simple revelation.
- The authors' description of the experimental results are not accurate enough. and the results raise several questions to be addressed.

# Recommendation

I'm in favour of rejection, but some concerns can be addressed fairly easily (with experiments) so I'm open to raising my score if questions are well-addressed.

## Issues and questions to address

* The authors should provide experiments on the reverse mapping as suggested above.
* I do not agree that figure 3 shows that RBM training "simply 'fine tunes'" the weights -- the difference is quite stark. How about increasing the batch size so that there is little SGD noise?
* Figure 4a: traces are cut-off just when random initialization is catching up with HN initialization. This also applies to Figure 5. 
* There are a few descriptions suggesting "HN init. appears to train much faster than random init". However, the rate of increase in of likelihood in Figure 4 is shallower for HN than for rand init. Is the advantage only at the 0'th RBM epoch? 
* The author only compared with purely random initialisation, which is perhaps the most naive baseline. I would suggest comparing to a (slightly) more clever initialisation, perhaps PCA or something better (those mappings in previous work the authors cited and in Appendix B). Or, the authors could also initialise the RBM by first training it on the within-class cluster centres (using a very large number of sleep samples for the sleep-phase) which may also be a more fair comparison?
* In the classification objective, if I understand correctly, the feature function is essentially quadratic in the input patterns. Should there be an ideal test error that is computed by a quadratic neural network trained with supervision by backpropagation? If the HN classifier (blue in Figure 5) is approaching this idealization, then this will strengthen the claim.
* The discussion on the extension to more generic, deep architectures is not well supported, and I do not see the extension to be so straightforward given the content of the current paper. In generation, supervised labels and clustering are used to simplify learning. Is the network able to learn just on the MNIST digits, even for the real images within a single class (e.g. "7")?
* Can the authors try to characterise whether the HN initialisation is related to log-likelihood training? I wonder if there is any interesting theory; otherwise, measuring model performance by log-likelihood seems a bit arbitrary (though it makes the comparison to contrastive divergence easier).

# Detailed suggestions (not to affect decision)

* Reference to RBMs should include more historic ones from Hinton (e.g. 2006)
* I do not see the purpose of (7) and (8), and they are only referred to in the Appendix (the review content in the Appendix is informative by itself though).
* Eqn (11), should it be $w_\mu w_\mu^T$ in the sum?
* Third line above (16), should $H$ and $Z$ be indexed by $\mu$?
* Line above (D.4), $WW^T = \dots B_p^T$?


==== update ====

I thank the authors for providing such detailed response. All my concerns are addressed and reflected in the revision (though some are much better done than the rest). I congratulate the authors on their spirit of maintaining a high standard on the theory, experiments and descriptions, and therefore significantly raise my score. I hope to see this paper accepted. 
review_title: A theoretical link between BN and RBM, but experiments are worth improving

Meta-review: Two knowledgeable reviewers were positive 7 and very positive 10 about this paper, considering it an important contribution that illuminates previously unknown aspects of two classic models, namely RBMs and Hopfield networks. They considered the work very well developed, theoretically interesting and also of potential practical relevance. A third reviewer initially expressed some reservations in regard to the inverse map from RBMs to HNs and the experiments. Following the authors' responses, which the reviewer found detailed and informative, he/she significantly raised his/her score to 7, also emphasizing that he/she hoped to see the paper accepted. With the unanimously positive feedback, I am recommending the paper to be accepted. 
Decision: Accept (Oral)