Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is a theoretical work in which the authors derive a set of normative models that describe the general structure of memory networks that can perform error-free recall of a given set of input patterns. Starting from well-known properties of binary classifiers they show how hetero-associate and auto-associate memories can be formulated as feed-forward and recurrent SVM networks, respectively. They also characterize the storage capacity of these networks in terms of their Vapnik-Chervonenkis dimension. Importantly, they show that previously proposed memory models such as Kanerva's sparse distributed memory and Hopfield networks are special cases of the SVM networks. They also extend their auto-associate memory model to be able to store continuous-valued patterns instead of just binary-valued patterns. Finally, they discuss how the models developed here could potentially be mapped to anatomical properties of biological neurons. 
strengths_and_weaknesses: This paper is a purely theoretical work that develops general expressions for optimal weights in memory networks storing binary or continuous-valued patterns with maximal noise tolerance. It is interesting to think about how the general theory developed here could give rise to better models of memory, and how they could lead to better AI methods. 

Also, the mapping of kernel attractor networks to anatomical properties of biological neurons is potentially an interesting avenue of research. 

The presentation is also very clear and it is easy to follow the main ideas. 


questions: - Are there any ideas on how the theory developed here could give rise to better models of memory? 
limitations: The authors discuss the limitations of their approach.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work provides a nice framework to describe hetero- and auto-associative memory models. This framework can be used to describe dynamics and capacity of different existing models, such as SDMs and Hopfield networks. As a background, the authors provide an explanation to kernel methods, that are then needed to understand the contributions of the paper: they show how a kernel trick for "one hidden layer nets" is equivalent to finding the best set of weights with minimum norm, and a general property of multilevel feedforward kernels.
The first contribution (Section 3), consists in defining auto associative memory models with binary states as recurrent SVMs. Particularly, they state that the larger the attractor basins, the more robust the model. This is then used to draw connections with prowler models in the literature, such as SDMs and Hopfield models. This study is then extended to include recent models, generalized to continuous patterns. 

strengths_and_weaknesses: Pros:

 Very nice paper, well written and straight to the point: it does exactly what is stated in the title.

Cons:

I believe the paper is suffering the 9 pages limit, as the authors use most of the space to derive the results, while having a little more space would allow interesting discussions. Particularly, the introduction and conclusion do not address future works/ possible implication of the work in any way.
questions: You state that your "hope is that a proper understanding of the theoretical overlap of contemporary memory models can facilitate the development" of new models. Generally, I agree that fundamental works that are able to identify underlying common principles of different models are important in the literature. However, in your specific case, how do you think your formulation will help future progress in the field? How will your work influence future applications of AM models?


Minor:

Remove one of the two remarks at lines 82 and 102




limitations: The very few limitations (it is a theoretical paper) are discussed, and I do not see any negative societal impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work pursued a classical yet interesting problem that designing a neural network to store a set of patterns with maximal noise robustness, and presented alternative understandings of this problem by investigating several kernel attractor networks. The claimed contributions can be summarized as follows:
1. proposed two mathematical structures of feedforward and recurrent memory networks for binary patterns.
2. show that the normative models include, as special cases, the classical and modern Hopfield network, as well as the SDM.
3. derived a simple attractor network model for storing an exponential number of continuous-valued patterns with a finite basin of attraction.
strengths_and_weaknesses: Strengths:
1. This paper is organized clearly, and the authors provided many plots for illustrating the structure of the proposed networks.
2. The derivation procedure of the feedforward and recurrent memory networks are easy to follow.
3. The result about storing an exponential number of continuous-valued patterns with a finite basin of attraction seems to be a laudable performance.

Weaknesses:
Limited to my knowledge, this paper is too hard to follow and comment. Here, I list the most concerned points, which may hinder the acceptance.
1. Is there something over-claimed? For example, the discussions about its similarity to attention and new biological interpretations seem only to give some understandings from the perspectives of attention and other fields with slight analysis. It would be better to provide detailed and solid support for this claim.
2. There lacks some clear-out and comparative results between the proposed networks and the previous ones. For example, what are the advantages of this framework that the authors proposed here.
questions: What is the definition or formulation of the maximal noise robustness. It is difficult to get any noise or robustness information from Property 1, 2.1, and 2.2 unless I missed something.
limitations: Nothing mentioned.
review_title: null

Meta-review: metareview: The paper is well written and, in addition to provide a unified perspective on hetero-associative and auto-associative memories via feed-forward and recurrent SVM networks, it shows how in the proposed framework is possible to use conventional kernel methods to design an attractor network that can store an exponential number of continuous-valued patterns with a finite basin of attraction. In the rebuttal/revised version of the paper the authors properly addressed the issues raised by the reviewers. Overall, given the quality of the contribution and no significant negative issue raised by the reviewers, I think the paper can constitute a valid contribution to the NeurIPS program.
confidence: Less certain
Decision: Accept