Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper considers a continuous version of the classical Hopfield network (HN) model.In contrast to well studied discrete models where the patterns (vectors) that are 
stored are discrete, this paper studied continuous vectors and a new continuous energy function.
Convergence results to a fixed point are proven for the new rule, and it is shown that for the case of random patterns, the Hopfield network can memorize exponentially many patterns (with high probability).  Finally several implementations are given showing how incorporating the new Hopfield net in classification tasks can improve classification accuracy in regimes where 
data is scarce and where neural networks do not fare well. 

The paper is rather long and I did not verify all results. The description appears sound.The proofs appear non-trivial and rather technical. While the results here are nontrivial I was left me wondering about the 
added value of this new model. One of the biggest advantages of HN was its simplicity and elegance. More recent results of Hopfield and others with higher degree energy functions managed to maintain this clarity and brevity. The new model however is significantly more involved. It was not clear to me what is gained by this greater complexity and whether the gains 
justify the larger complexity. In actual implementations very limited precision is often necessary.How does this discretization influence the continuous model? How robust is it to rounding errors? Don't we get "old" discrete models in disguise? 

The (impressive) empirical results raise similar questions. Can't we use old discrete HN instead of the new model and achieve similar results? It would be perhaps more informative to compare different HN to the new model presented in this paper. It seems a bit strange that previous uses of HN (discrete ) did not achieve such an improvement in previous studies. It would be beneficial to add more on related work in this area. 

 The authors might consider breaking their long paper to two different sections, one presenting the theoretical advantages of their new model and the other focusing on practical benefits. 

Finally, the nature of convergence to a fixed point wasn't clear to me. It seems likely that if patterns are not random convergence can take a long time as is the case for discrete HN. 
Some recent work about the complexity of finding fixed points of continuous functions may be relevant here:A converse to Banach's fixed point theorem and its CLS-completeness.
More specific comments:
1) The paper starts with a rather lengthy discussion of previous work. 
I would recommend outlining the contributions of this paper earlier on. 
2) "converge in one update step with exponentially low error and have storage capacity proportional to..." It was not clear to me that random patterns are considered here. 
3) "proven for c= 1.37andc= 3.15 in Theorem 3" for what c exactly is the result proven? 
4) "Furthermore, with a single update, the fixed point recovered with high probability"I presume this is true for random patterns? 
5) Is beta>0?
review_title: Interesting results but some questions arise

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a new Hopfield network which have continuous states and propose update rules for optimizing it. It also draws connections between the new model and attention mechanism used in transformers. Small scale empirical study is presented.

Overall I like the technical contribution of the work but feel the paper could be revised to improve clarity about the optimization in the new proposed variant of hopfield networks. Below some specific comments:

Pros:
- connecting hopfield networks to attention mechanism and drawing out the variants in section 3 (as hopfield layers) is useful
- The exposition in section 1 and 2 where the authors describe the hopfield network with continuous states is written well (although I do feel the motivation behind update equations could be explained a bit better)

Cons:
- As I mentioned earlier, I don't fully understand the intuition behind convergence in one update. Can the authors clarify this? Also the paper mentions update rule in eqn (5) converges after one update for well separated patterns. What happens to the updates / optimization when the patterns are not well separated? This should be discussed after equation (5). Maybe present different scenarios to make it clear.

- Empirical study is limited in my opinion and can be improved. Is the trend in Fig 2 observed across more or less across all datasets? Can the authors comment on this? I like the visualization in the figure but it is bit hard to interpret (perhaps a more clearer label for it could help with that). 

Other comments:
- The idea of separated patterns leads me to ask this question: is there any connection of this work to max-margin classifiers / kernel methods?

- Did the authors consider what would happen if non-linear transformations (e.g. activation functions in DNNs) are applied on top of the inputs? How does the existing network change in that case?

- Can the authors comment on the utility / challenges in applying their proposed method on datasets / tasks beyond the small scale UCI datasets used in their experiments? e.g. using them in large scale language modeling tasks where transformers are popular right now.


review_title: Paper makes good technical contribution draws interesting connections between classical Hopfield networks and Attention Mechanism in transformers

final_score: rating: 7: Good paper, accept
review_content: This work extends the binary Hopfield network (Demircigil et al., 2017) to continuous patterns and states. Connections are drawn between the result model to the attention layers of the transformers, the pooling operation of LSTM, similarity search, and fully connected layers. Experimental results are briefly described for analyzing the attention of Bert models, multiple instance learning, and small UCI classification tasks.

The proposed model seems very interesting, and the proposed applications seem reasonable at a very high level. However, there is just not enough detail in this paper for me to understand how the models are implemented or why the model works better than other approaches.

For example, section 3 declared 3 types of Hopfield layers, but without any formal definitions to them, or how they are integrated to the proposed models. The experiment section compares performances with existing models, but lacks any analysis of why the proposed models work better. Similarly, there is a lack of motivation/intuition in the introduction section.

## After author feedback ##
Thanks for the paper update, and now I have a better understanding of the proposed approach. I have updated my review to the following:

Previously Widrich+ (2020) showed that integrating transformer-like attention (or equivalently modern Hopfield networks based on softmax)  into deep learning architectures outperforms existing methods (kNN and logistic regression) for massive MIL such as immune repertoire classification. More specifically a pooling layer can be formed by attending over a repertoire of instances with a fixed (but learnable) query vector.

This work provides theoretical analysis of such a layer for its energy function, convergence of updates, and storage capacity, and points to directions of how such a layer can be understood and controlled. It extends the previous experiment:
1) apply HopfieldPooling (attention with fixed learnable query Q) to more MIL datasets (animal image and breast cancer)  and achieve state of the art results. 
2) apply Hopfield (attention) to 75 small UCI benchmarks replacing feedforward nets. Here Selu units (Klambauer+ 2017) are used to map input to storage Y  and query R. The result is quite positive beating previous approaches including SVM, random forest, and SNN (Klambauer+ 2017)
3) apply HopfieldLayer (attention with fixed training data Y as storage) to 4 drug design tasks  acting as an instance-based learning approach.

The result seems quite interesting indicating that general purpose layers such as  feedforward, pooling and nearest neighbors can be improved (in terms of robustness, learnability, or controllability) by adding attention like operations.

I think the paper can talk less about existing results, and focus more on the new results and their analysis:
- remove [Immune Repertoire Classification] result since it is from previous work.
- move the Drug Design experiment details to the main text, and add some comment about under what condition Hopfield outperforms/underperforms RF.
- for the UCI benchmark experiment the transformer layer (Vaswani+ 2017) seems to be a natural baseline and should be compared to. 

Suggestions for the presentation:
- Should only in the future work section state that Hopfield can potentially substitute LSTMs or GRUs, since it is all hypothetical with no experiment result at this point.
- The word "implemented"  in Section 4 seems misleading as there is nothing changed in the Bert model structure? "Transformer and BERT models can be implemented by the layer Hopfield."  
- Can be more specific in descriptions. For example in the description of (2) Layer HopfieldPooling and (3) Layer HopfieldLayer in Section 3, R and  W_K can be  referenced again for "state (query) patterns " and "The stored (key) patterns" respectively.
- It is probably more informative to replace figure 1 with a table to directly compare the energy function and updating rules of different Hopfield nets--i.e., classical, exponential and attention.
- Avoid using "x" in equation 1, since the symbol has already been used for the stored patterns.
- "HopfieldLayer" seems to be a very strange name.


review_title: Very interesting but missing critical content

Meta-review: The novelty of the paper are:
+ introduces a new Hopfield network with continuous states, hence can be learned end-to-end differentiation and back propagation.
+ derives efficient update rules
+ reveals a connection between the update rules and transformers
+ illustrate how the network can be used as a layer in deep neural network that can perform different functions

The presentation was clear enough for the reviewers to understand and appreciate the novelty, although there were a few points of confusion. I would recommend the authors to address several suggestions that came up in the discussions including:
- additional analysis to highlight when and how the networks is able to outperform other competing models
- intuitions about the proofs for the theorems (okay to leave the detailed derivation in the appendix)



Decision: Accept (Poster)