Nonparametric Modern Hopfield Models
Authors: Jerry Yao-Chieh Hu
Bo-Yu Chen
Dennis Wu
Feng Ruan
Han Liu
Venue: ICML 2025 poster (2025)
Keywords: dense associative memory
modern Hopfield model
Hopfield network
efficient transformer
efficient attention
transformer
attention
foundation model
large language model
Primary area: deep_learning->attention_mechanisms
TL;DR: A nonparametric construction for transformer-compatible modern Hopfield models and utilize this framework to debut efficient variants.

Abstract: We present a nonparametric interpretation for deep learning compatible modern Hopfield models and utilize this new perspective to debut efficient variants. 
Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs.
Interestingly,
our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing *sparse-structured* modern Hopfield models with sub-quadratic complexity.
We establish that this sparse model inherits the appealing theoretical properties of its dense analogue --- connection with transformer attention,  fixed point convergence and exponential memory capacity.
Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-$K$ and positive random feature modern Hopfield models.
Empirically, we validate our framework in both synthetic and realistic settings for memory retrieval and learning tasks.

Nonparametric Modern Hopfield Models
Jerry Yao-Chieh Hu"
Bo-Yu Chen
Dennis Wu
Feng Ruan 3 Han Liu
Abstract
Ramsauer al., 2020) a class of transformer-comppiible associative memories. In short, we present nonparametric framework', and then debuting efficient modern Hopfield models with sub-quadratic complexity and appealing the- oretical properties. Such a construction is of practical im- portance. As in many Hopfield-centric methods (Hu et al., 2024a; Wu et al., 2024a; Xu al., 2024; Wu et al., 2024b; Schimunek et al., 2023: Fürst et al., 2022; Paischer et al., 2022; Seidl al., 2022; Widrich al., 2020), modern Hop- field models (and their derived deep learning layers) serve as powerful alternatives to the attention mechanism with additional functionalities, but lack efficient implementation for gigantic deep models (Hu et al., 2023, Section C.2).
We present a nonparametric interpretation for deep learning compatible modern Hopfield mod- els and utilize this new perspective debut effi- cient variants. Our key contribution stems from interpreting the memory storage and retrieval pro- cesses in modern Hopfield models as a nonpara- metric regression problem subject to a set of query-memory pairs. Interestingly, our frame- work not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding ef- ficient modern Hopfield models, by introducing pparse-structured modern Hopfield models with sub- uub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue connection with transformer attention, fixed point conver- gence and exponential memory capacity. Addi- tionally, we showcase the versatility of our frame- work by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-K and positive random feature mod- em Hopfield models. Empirically. we validate our framework in both synthetic and realistic settings for memory retrieval and learning tasks. Code is available at GitHub; future updates are on arXiv.
This issue becomes more prominent in this era of Large Foundation Models (Bommasani al., 2021). Foundation models are huge transformer-bast based models pretrained on massive datasets, and play central role not only in machine learning but also in a wide range of scientific domains, such as ChatGPT (Brown et al., 2020; Floridi & Chiriatti, 2020) for natural language, BloombergGPT (Wu et al., 2023) for finance, DNABERT (Zhou et al., 2024a;b; Ji et al., 2021) for genomics, and many others. To push toward Hopfield- based large foundation models, this work provides timely efficient solution, back-boned by solid theoretical ground.
Modern Hopfield models (Ramsauer et al., 2020), motivated by the dense associative memory models (Demircigil et al., 2017; Krotov Hopfield, 2016), are (auto-) jauto-)assocciaive memory models that (i) have exponential memory capacity, (ii) retrieve stored patterns based on input queries with only retrieval step, and (iii) are compatible with deep learn- ing architectures. They achieve (i) by adopting highly non- linear energy functions, (ii) by adopting a memory-retrieval dynamics ensuring monotonic minimization of the energy function, and (iii) by the connection between their memory retrieval dynamics and attention mechanism. Deepening (ii) and (iii,, Hu et al. (2023) and Wu al. (2024b) propose a theoretical framework for deriving modern Hopfield models
1 Introduction
We tackle the challenges in computational efficiency of modern Hopfield models (Wu et al., 2024b; Hu al., 2023;
'Equal contribution 'Department of Computer Science, Northwestern University. Evanston, IL. USA 'Department of Physies and Computer Science, National Taiwan University, Taipei, Taiwan 'Department of Statisties and Data Science, Northwestern University, Evanston, II, USA. Correspondence to: Jerry Yao-Cheih Hu www.uh.northeeeeeeeeeeeee www.twwsteee..n.mmm edu'>, Den- Feng Liu
Bo-Yu Chen <b122020230ntu.
After completing the draft, the authors became aware of the independent study by Nguyen et al. (2024) on a nomparametric (primal-dual) formulation for Transformer attention. To our knowl- edge, Nguyen al. (2024) were the first to cast attention an e-SVR solution (Awad et al., 2015; Vapnik, 2013; Kar & Karnick, 2012; Schölkopf Smola, 2002). Our study presents a similar framework but focuses on Hopfield-style associative memory.
nis Wu
northwwsenn..hseer... edu>> edu>,
Ruan <fengruandnorthvestt
Han
<hanliu@horthweeteenn edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Nonparametric Modern Hopfield Models
using various entropic regularizers. In addition, they intro- duce sparse extension of the original modern Hopfield model to handle its computational burden and vulnerability to noise. As a result, their proposal not only connects to sparse attention mechanism (Correia et al., 2019; Martins & Astudillo, 2016) but also offers both provably computational advantages and robust empirical performance.
retrieval error bound of the sparse modern Hopfield with ex- plicit sparsity dependence (Theorem 4.1). This bound offers rigorous characterizations of the sparsity-induced advan- tages of the sparse model compared with its dense counter- part, including higher precision in memory retrieval (Corol- lary 1.1.1 and Corollary 4. 4.1.2), enhanced robustness to noise (Remark 4.2) and exponential-ing capacity (Lemma 4. and Proposition 4.1, refers to pattern size). Interestingly, un- like existing Hopfield models (Hu et al., 2023; Wu et al., 2023; Ramsauer et al., 2020) requiring explicit energy function to guarantee the stability of the model, we show that the sparse modern Hopfield model guarantees the fixed- point convergence even without details of the Hopfield en- ergy function (Corollary 4.1.3).
However, there are still some missing pieces:
(P1) Lack of Efficiency. Computationally, while Hu et al. (2023) and Wu et al. (2024b) indeed introduce sparsity into their model, this sparsity does not implies computa- tional efficiency. In fact, it only increases efficiency at the level of memory retrieval, (i.e. the sparsity in (Hu et al., 2023; Wu et al., 2024b) only leads to faster mem- ory retrieval but not necessarily shorter running time, as discussed in (Hu et al., 2023, Section C.2)). Namely, the sparse modern Hopfield model still suffers by the O(n)) complexity (with the input sequence length n), which hampers its scalability?.
To fill (P3), beyond introducing the sparse modern Hopfield model, our framework supports family of modern Hopfield models that connect with various attention variants. This complements the findings in (Hu et al., 2023; Wu et al., 2024b), pushing us toward a more unified understanding.
(P2) Lack of Rigorous Analysis on Sparsity. Theoreti- cally, because Hu et al. (2023) choose not to make strong assumptions (on the memory and query patterns) in order to maintain their model's s generality, they only offer qual- itative justifications (Hu et al., 2023, Section 3). They do not rigorously characterize how sparsity impacts different aspects of the sparse model, e.g., the retrieval error, the well-separation condition, and the memory capacity,
Contributions. Our contributions are as follows:
We propose nonparametric framework for deep learning compatible modern Hopfield models. Building upon this, we introduce the first efficient sparse modern Hopfield model with sub-quadratic complexity.
We provide rigorous characterizations of the sparsity- induced advantages of the proposed efficient model: tighter retrieval error bound (Corollary 4.1.1 and Corol- lary 4.2)), stronger noise robustness (Remark 4.2) and exponential-d-cappe d-apppcct (Lemma 4. and Proposition 4.
(P3) Incomplete Connection between Attention and Hopfield Models. Methodologically, while numerous variants of the attention module exist (Choromanski et al., 2021; Katharopoulos et al., 2020; Beltagy et al., 2020; Child et al., 2019), Hu et al. (2023) only bridge subset of them to modern Hopfield models. A natural question arises: How can we integrate the advancements of state- of-the-art attention into modern Hopfield models? As noted in (Hu et al., 2024b; 2023; Wu et al., 2024b), this question is far from trivial. Naively substituting the soft- max activation function with other alternatives does not necessarily yield well-defined Hopfield models and might sabotage their desirable properties and functionalities.
Based on the proposed framework, we construct a family of modern Hopfield models connecting to many existing attention variants (Choromanski et al., 2021; Zaheer et al., 2020; Beltagy et al., 2020: Katharopoulos et al., 2020). We also verify their efficacy through thorough numerical experiments in both synthetic and realistic settings (mem- ory retrieval and learning performance in Appendices G. 1 to G.4 and efficiency in Appendix G.S.).
Notations. We denote vectors by lower case bold letters, and matrices by upper case bold letters. We write (a, b) == a a as the inner product for vectors a, b. Let ali denotes the i-th element of vector a. The index set {1,... /) is denoted by [4], where E N4. The spectral norm is denoted by II-II, which is equivalent to the 12 no-mmmm when applied to a vector. We denote the memory patterns by & E RR and the query pattern by x E Rd, and = == [EI. ....... ..M] E RdxM as shorthand for stored memory patterns (Em)eetM) Moreover, we set m == Max [|
Overview of Our Theoretical Results. To fill these gaps, this work presents nonparametric framework for deep learning compatible modern Hopfield models.
To fill (P1), this framework allows us to not only recover the standard dense modern Hopfield model (Ramsauer et al., 2020), but also introduce an efficient modern Hopfield model, termed sparse-structure model (Theorem 3.2).
To fill (P2), our framework facilitates the derivation of
"See Remark 3.6 for the connection between time complexity of attention and of modern Hopfield models.
Organization. Section 2 reviews modern Hopfield mod- els. Section 3 presents nonparametric construction for
2

---

Nonparametric Modern Hopfield Models
Hopfield Energy Eg(x) [Ramsauer et al., 2020]
Minimized via CCCP
Retrieval Dynamics (Equation (2.2))
Retrieved Patterns x new E R
(e.g. Lemma 2.1 of [Hu al., 2023])
Exponential Memory Capacity
(Proposition 4.1)
Memory Patterns E
Query x E Rd
Fixed Point Convergence
(Corollary 4.1.3)
Nonparametric Retrieval
Nonparametric Regression (Theorem 3.1)
Specifying Kerness
Dynamics
Retrieval Accuracy (Thm. 4.1)
[Lemma 3.1, Thm. 3.2 & AAppendix E)
(Lemma 3.1 & Thm. 3.2)
Theoretical Guarantees
Figure 1. High-level Visualization. The upper row formalizes the memorization and retrieval of standard modern Hopfield model (Wu et al., 2024b; Hu et al., 2023; Ramsauer et al., 2020). The lower row conceptualizes our nonparametric interpertation.
modern Hopfield models, and debut the sparse-structured (efficient) modern Hopfield models. Section 4 provides the theoretical analysis on the sparse-structured modern Hop- field models. Appendix E includes a family of modern Hop- field models as possible extensions. We conduct numerical experiments to support our framework in Appendix G.
establish connection between Hopfield models and the vanilla softmax attention. Motivated by this connection, (Hu et al., 2023; Wu et al., 2024b) (and later (Martins et al., 2023)) propose a theoretical framework for modern Hopfield models based on the relationship between entropic regular- izers and finite-domain distributions with varying support sets. Importantly, they not only show that (Ramsauer et al., 2020) is just special case within their framework but also propose a sparse extension with superior properties (e.g.. robust representation learning, fast fixed-point convergence, and exponential memory capacity) and connection to cer- tain types of sparse attention. However, this is not end of the story. As highlighted in (Hu et al., 2023, Section E), their framework only bridges a subset of existing attention variants (with dense quadratic attention score matrix) and hence is not complete. This work fills this theoretical gap by providing a principle construction for the many modern Hopfield models with theoretical guarantees. Moreover, our framework supports a family of modern Hopfield models mirroring many popular structured efficient attention mech- anisms, including Attention with Pre-defined Patterns (each sequence token attends a predetermined subset of tokens instead of the entire sequence, e.g, Big Bird (Zaheer et al., 2020), Longformer (Beltagy ct al., 2020), Blockwise (Qiu et al., 2019), Sparse (Child et al., 2019)), and Kernelized At- tention (e.g.. Performer (Choromanski et al., 2021), Linear (Clevert et al., 2015) and Multi-heate (Vaswani et al., 2017)).
1.1 Related Work
Modern Hopfield Models for Deep Learning. The clas- sical Hopfield models (Amari, 1972; Hopfield, 1984; 1982; Krotov & Hopfield, 2016) are canonical models of the hu- man brain's 's associative memory. Their primary function is the storage and retrieval of specific memory patterns. Re- cently, a resurgence of interest in Hopfield models within the machine learning field is attributed to developments in un- derstanding memory storage capacities (Krotov& & Hopfield, 2016; Demircigil et al., 2017; Wu et al., 2024a), innovative architecture (Hoover et al., 2023; Seidl et al., 2022; Fürst et al., 2022; Ramsauer et al., 2020), and their biological plausibility (Kozachkov et al., 2022; Krotov & Hopfield, 2021). Notably, the modern Hopfield models (Wu et al., 2024b; Hu et al., 2023; Ramsauer et al., 2020; Brandstet- ter, 2021), demonstrate not only a strong connection to the transformer attention mechanisms in deep learning, but also superior performance, and a theoretically guaranteed exponential memory capacity. In this regard, seeing the modern Hopfield models as an advanced extension of atten- tion mechanisms opens up prospects for crafting Hopfield- centric architectural designs. Therefore, their applicability spans diverse areas like drug discovery (Schimunek et al., 2023), immunology (Widrich et al., 2020), tabular learning (Xu et al., 2024), time series forecasting (Wu et al., 2024b; Auer et al., 2024), reinforcement learning (Paischer al., 2022), and large foundation models (Hu al., 2024a; Fürst et al., 2022). This work emphasizes refining this line of research towards efficient models. We posit that this effort is crucial in guiding future research towards Hopfield-driven design paradigms, especially for larger models.
Sparse-Structured Hopfield Models. After completing this work, the authors attended ICML 2024 and learned of study by Santos et al. (2024b) proposing different model, also named the sprrse-structured Hopfield network. Both models emphasize sparse retrieval patterns. However, Santos et al. (2024a;b) differ in (i) introducing sparsity via optimized Fenchel-Young energies and (ii) enhancing ef- ficiency using the SparseMAP transformation and active set algorithm (Niculae et al., 2018) with predefined k-ary relations among stored patterns or top-k operations.
Sparse Modern Hopfield Model. (Ramsauer et al., 2020)
3

---

Nonparametric Modern Hopfield Models
2 Background: Modern Hopfield Models
1. Dense dynamics converge well (T2) and can retrieve pat- terns accurately in just one step (T1).
This section presents the ideas we build on.
2. Modern Hopfield model from (2.1) possesses an expo- nential memory capacity in pattern size d.
Let x Fd be the input query pattern and
[E,, .Ea]] RaxM the M memory patterns.
3. Notably, the one-step approximation of Dense mirrors the attention mechanism in transformers, leading to a novel deep learning architecture design: the Hopfield layers.
Hopfield Models. The aim of Hopfield models (Amari, 1972; Hopfield, 1982; 1984) is to store these memory pat- terns E and retrieve a specific memory S,, when given a query x. They achieve these by embedding the memories in the energy landscape E(x) of a physical system, where local minimum. When query is presented, the model initiates energy minimmiiii retrieval dynamics T at the query, which then navigate the energy landscape to find the nearest local minimum, effec- tively retrieving the memory most similar to the query.
Attention ++ Modern Hopfield Model. To above 3., suppose that X and =\t are embedded from the raw query R and Y memory patterns, respectively, via X = RW == Q, and = = YWx k = K, with some projection matrices Wo and WK. Then, taking the transpose of T in (2.2) and multiplying with Wv such that V = WWv, v. we obtain
each memory E,, corresponds
Z = Q new nwwWWwww Softmax (BQKT) ) V.
(2.3)
These models comprise two primary components: an en- ergy function E(x) that encodes memories into its local minima, and a refrieval dynamics Txx) that fetches mem- ory by iteratively minimizing Exx) starting with a query. Constructing E(x), is straightforward. As outlined in (Kro- tov& & Hopfield, 2016), memories get encoded into E(x) using the overlap-construction: F((x) = F(ETx) x), where :: RRM + RR is a smooth function. This encourages the memories [Ewhpe[M to reside at the stationary points of E(x). i.c., (= x)le. = 0 for all μ E [M]. The choice of results in different Hopfield model types, as demon- strated in (Krotov& & Hopfield, 2021: Ramsauer al., 2020; Demircigil et al., 2017; Krotov& & Hopfield, 2016).
This enables modern Hopfield models to serve as alterna- tives to attention mechanism with extra functionalities.
Given the equivalence (2.3), one might wonder if the quest for efficient modern Hopfield models is equivalent to seek- ing efficient attention mechanisms (Tay et al., 2022), specif- ically in terms of finding efficient implementations of the Softmax matrix computation. We contend that they are not the same. To build a modern Hopfield model, we ex- pect not only its retrieval dynamics to connect to attention mechanism, but also it to serve as an associative memory model (Hu et al., 2024a; Wu et al., 2024b; Hu et al., 2023; Ramsauer et al., 2020) by design. Moreover, we observe that (TI) and (T2) are essentially about encoding memories onto the fixed points of T.
However, determining suitable retrieval dynamics, T, for a given energy Exx is more challenging. For effective memory retrieval, the iterative retrieval dynamics 'T must:
These motivate us to view the construction of T as a learn- ing problem: we aim to learn function T satisfying (T2) from a dataset consisting of query-memory pairs. Thus, rather than using the traditional Hopfield model's learning rule - where the model memorizes memories by defining an energy function, like the overlap-construuctonn (Hu et al., 2023) we interpret the memorization process as learning function that maps queries to memories. This new perspec- tive allows us to construct novel modern Hopfield models that are equivalent to various attention variants.
(T1) Monotonically reduce F(x) when applied iteratively.
(T2) Ensure its fixed points coincide with the stationary points of E(x) for precise retrieval.
Modern Hopfield Models. Ramsauer et al. (2020) pro- pose the modern Hopfield model with a specific set of E and T satisfying above requirements. and integrate it into deep learning architectures via its strong connection with atten- tion mechanism, offering enhanced performance, and theo- retically guaranteed exponential memory capacity. Specifi- cally, they introduce the energy function:
3 Nonparametric Modern Hopfield Models
E(x) = [8, x } x).
High-Levvel Overview.
(2.1)
In Section 3.1, we formulate the memory storage and retrieval of modern Hopfield models nonparametric regression problem. We first align the definition of T (the retrieval dynamics (2.2)) with a nonparametric regression problem subject to a set of query-memory pairs. Then, by solving for optimality, we derive a nonparametric for- mulation of T.
where the retrieval dynamics is given by
x = Teensee = = Softmax (0 x).
(2.2)
The function Ise (3,z) == log (IM= exp( / the sig-sum-expooential for any given vector E RM and 0. Their analysis reveals that:
4

---

Nonparametric Modern Hopfield Models
In Section 3.2, we showcase our framework with two special cases: the standard dense modern Hopfield model (Ramsauer et al., 2020) (Lemma 3.1), and new, efficient sparse-structured modern Hopfield model (Theorem 3.2).
aiming to minimize the emror rate while ensuring the model remains insensitive errors within certain threshold.
We first define the regression model. Given a weight matrix
W E RdxDs, and feature map Φ : E + RR",, denote fw: : Red + Rd to be the mapping
3.1 Retrieval Dynamics
fw: (x).).
(3.1)
The retrieval dynamics (2.2) T=((): : I2 + Rd maps an in- put query x to T=xx;), with the aim of retrieving the memory pattern Eμ closest to x. To formalize this notion of retrieval, we need few definitions and notation.
Denote X2)) == (0(z1). (x2))) This is positive semidefinite kernel, and there is a unique RKHS H associ- ated with this kernel K. (Wainwright, 2019, Theorem 12.11).
To cast T as a SVR problem using (3.1), we now specify the data points that f(x) should fit. Since the goal of 7 is to re- trieve the memory pattern most similar to given query x, we consider the training dataset D {(E, + 66 Ex) ME Namely, the input query x = E, + òb,, is the contami- nated target memory pattern with noise òE,,, and the output fμ is target memory pattern. For convenience, we shorthand [&i + őε1,-.. EM + 5f.l = Es E RdxM as the contaminated memory patterns.
Definition 3.1 (Generalized Fixed Point (Sriperumbudur &
Lanckriet, 2009)). We say set S C R generalized fixed point with respect to Ts if (y) E S for every y E S.
Remark 3.1 (Fixed Point). In contrast, a fixed point of T= is a point y for which T=(y) - y.
In particular, if the retrieval dynamics is initiated at x E S where S is an invariant set', then subsequent iterates such as (x). T= o T= (x), remain in the invariant set S.
Next, we frame the memorization in modern Hopfield mod- els as fitting f to the dataset D, and obtain the following nonparametric (support vector) regression problem. Given a dataset D = {(E + 5L,.: [e))) consider the support vector regression using the feature map &
Now we introduce a neighborhood Sp. a ball of radius R at every memory pattern En:
Sp (E ILE -Eull= S R}.
1 min E5, - Ev
where
R = 2 [M]
M
Min W 2 + + W Wn..i 2
(I, (np + +)))
s.t.
(3.2)
By definition, neighborhoods associated with distinct mem- ory patterns do not overlap: S₁, n S, = 0 for μ * v. To measure the progress of the dynamics in retrieving the mem- ory pattern, we introduce the notion of memory storage and e-retrieval.
1=1
-(21+ 7 S E - (W, PE + ok,)) S e'1 + n
e n, E1// <llvVd
7u 2 0, n. 2 0, 4y E [M)
Definition 3.2 (Storage and Reerrieval. A memory pattern En, is stored if S₁ is a generalized fixed point of T. A memory pattern εn, gets e-retrieved by Ts with an input query if T= (x) - Erul S.
where the constraints are component-wise, é' > 0 is a component-wise error margin, C 2 0 is a penalty coef- ficient, and 6 > 0 is the memory retrieval error. We denote the unique (given the strong convexity of the optimization problem (3.2)) minimizer as (W;, nā 77. and the solu- tion to (3.2) (x). By solving the optimality via the Lagrangian duality, we obtain the following.
In below, when the context is clear, we suppress the notation dependence of T= on the memory patterns 3 for simplicity.
Definition 3.2 states that for an input x around a stored memory pattern &, its corresponding mapping output T(x) should be located in the same sphere S. This motivates to view T as function aiming to map the query onto its nearest memory within an error-tolerance margin R. More precisely, we construct such a function satisfying Def- inition 3.2 as a learning problem, using memory patterns as data. A natural choice for doing this function is through the soft-margin SVR (Awad et al., 2015; Vapnik, 2013; Kar& & Karnick, 2012; Schölkopf & Smola, 2002) (also see Appendix C. for concise overview): it fits the best hy- perplane to the data points within predefined error margin,
Theorem 3.1. Let a., ā denote the Lagrangian multipliers of the dual problem of (3.2). Let W := (wi.... wi)
RdxDs denote the minimizer of (3.2). Then,
W Σ
(all
( (Ep + )E,))
P
where a[i] denotes the i-th element of a vector a.
Proof. See Appendix D.1 for a detailed proof.
'A generalized fixed point S with respect Ts also invariant set with respect to Tg.
For any featurization map L Theorem 3.1 introduces a map TsvR,o == fw; By construction, for any $ TsvR,
5

---

Nonparametric Modern Hopfield Models
obeys the e-retrieval property [ -, any j and x E Sp. Hence, we arrive nonparametric inter- pretation for constructing many modern Hopfield models. Given an input query the i-th component of the retrieved pattern by applying TsvR once is
E.. S e, for
3.2 Nonparametric Dense and Sparse-Structured Modern Hopfield Models
In this section, showcase the nonparametric framework Theorem 3.1 with two special cases. First, we recover the standard dense modern Hopfield model (Ramsauer et al., 2020). Then, we introduce the efficient sparse-structured modern Hopfield models with sub-quadratic complexity.
new ( (w), o(x))
(3.3)
Dense Modern Hopfield Model (Ramsauer et al., 2020).
Remark 3.2. Note that c' is the component-wise SVR error, not the in Hopfield retrieval error defined in Definition 3.2.
Mod- Lemma 3.1 (Nonparametrie Dense em (o) .1 DD........ 01 (1) ..... D..... ......) with the
Hopfield Model). Let (n)
(-)
Remark 3.3. Without any assumption on e, TsvR converges to generalized fixed points, in contrast to the fixed point convergence in (Hu et al., 2023; Ramsauer et al., 2020). Thus, there is no multiple update convergence for Tsvr without specifying & (and thereby proving the fixed point convergence property.) We provide specific T with provably fixed point convergence in Section 3.2 and Remark 4.3.
formulation, for 1 D S Dn with D, == (ata-1).
a )
(VBx1) ... ( (B3a))
(3.4)
D Ens (0(E,, + Skn), ((x)) Vi!!....d! Ed!
where 4 + +.... + la = =n n. By Theorem 3.1, fitting TsvR on 'D following (3.2) gives
Remark 3.4. This regression problem is nonparametric. That is, it does not assume a specific functional form for TsvR and is flexible in the number of parameters, allowing the number of support vectors to adjust based on the data.
Teess (x) = E Softmax (B=]x) E Rd,
(3.5)
where 3, = [ + 851,-. ... E.M + dEML E RdxM M denotes the contaminated memory patterns.
Intuitively, this optimization problem learns a TsvR, to replace T from the training dataset D = {(EN pc [u] Thus, for any given query t5, TsvRR retrieves target memory pattem Eμ with e precision, for all E [M]. Specifically, this precision comes from the up- per bound of the maximum component wiis error e' (and e' + (+) S e/vd, defined in (3.2). This choice of SVR error margin mimics the e-retrieval of modern Hopfield models via the flexibility of soft-margin SVR. As a result, the objective of the SVR problem (3.2) coincides with the memorization and retrieval processes of modern Hopfield models. While T retrieves memory patterns [Ew]ne|M] based on with an error tolerance e, the SVR problem (3.2)
Proof Sketch. We first select & to be the Taylor expansion of the exp function via the exp kernel's ' feature expansion (Nguyen et al., 2024; Hamid et al., 2014; Kar & Karnick, 2012; Schölkopf & Smola, 2002). By solving the optimiza- tion problem (3.2), we arrive a retrieval dynamics resem- bling (2.2). See Appendix D.2 for a detailed proof.
Remark 3.5 (Hetero- v.s. Auto-Associative Memory.). So far, we derive a nonparametric framework for hetero- associative modern Hopfield models, differentiating x and y by incorporating inherent noise óg into D. we elimi- nate noises [E[M] from the training memory patterns, (3.5) reduces that of the standard auto-associative dense modern Hopfield model, shown in (2.2).
(Memorization:) Fits a function TsvR satisfying Defini- tion 3.2, which
(Retrieval:) Maps queries onto memory patterns within a component-wise error-margin e/ /d.
With Remark 3.5, Lemma 3.1 facilitates the replication of known results from the standard dense modern Hopfield model (Ramsauer et al., 2020). The recovery of dense modern Hopfield model provides sanity check for our nonparametrie framework.
Importantly, Theorem 3.1 enables us to derive a family of nonparametric modern Hopfield models through con- structing their retrieval dynamics with various kernel func- tions (-)), including Dense (Ramsauer et al., 2020), Linear (Katharopoulos et al., 2020), Multi-Head (Vaswani et al., 2017), Sparse-Stree Sttrrssrrr (Zaheer et al., 2020; Beltagy et al., 2020; Child et al., 2019) and Generalized Kernelizable or Positive Random Features (Choromanski et al., 2021) mod- em Hopfield models. Appendix E includes constructions of these models as extensions from our framework.
Sparse-Structured Modern Hopfield Models. Next, we present set of efficient modern Hopfield models with sparse-structured patterns via the following mask.
Definition 3.3 SSparse-Structured Mask), Let M = {1,..., M) be the reduced sup- port set for Tsvr of size k S M. Then, for μ E [M]. the
(M(()...,,M(B))
6

---

Nonparametric Modern Hopfield Models
optimization problem in (3.2) reduces to
window size vL, we obtain an efficient modern Hopfield model with sub-quadratic ( complexity. This model connects to the Longformer attention (Beltagy et al., 2020) by design.
Min W.n.
|| + C'
Σ
(1. (7 + n,)) s.t.
(3.6)
PEM
Example (Top-K Modern Hopfield Model), Let the sequence {Pr)pe[M] be the inner products of memories [Ew]ee[M) and query x, i.c., Pp (x, Eu), and let p" be the K-th largest element in {p (Pulne[M]- Then we obtain sparse-structured mask M such that
-c'1 (p) S E. (W, (E, + )(,)) S 6 + T
d1+ S cd/v
n, 2 0, n 2 0, M.
With Definition 3.3, we obtain the following sparse- structured retrieval dynamics (and thereby its corresponding Hopfield mo)el(s)) by fitting TsvR on D masked by M.
μ M, M, pf 2p" 1 E if < p"
(3.8)
Theorem 3.2 (Sparse-Strunl Structured Modern Hopfield Models). (0) (1)
Let (-)) = (00 191 ..... D....... ..... (n) (n)
Da
With (3.8), arrive a top- modern Hopfield model with quadratic complexity, i.e., inefficient. This model connects to the top-K attention (Gupta et al., 2021) by design.
with, for 1 S D' < Dn with Dn = (d-n-1).
(vBa)) ... (vBar)
Remark 3.6 (Time Complexity of Modern Hopfield Models and Attention Mechanism). The time complexity of modern Hopfield models and Hopfield layers is given by:
D ΣREM (E), - ). $(x)) VEI. Ea!
where l +.... + = By Theorem 3.1, fitting TsvR on D masked by .M following (3.6) gives
Time complexity of modern Hopfield model: ((M)).
Sparse (x)
Σ
[Softmax({E]xx)] S..
(3.7)
When used as cross-attention (Hopfield layer) with length- L (query) and length-M (memory) input sequences: (L.M))..
ME
ER
where EM = 1... -2 + σE, ...] RdxlMi with e [[M]].
When used as self-attention with length- L input sequence (set M = L)): O(n2d²).
Proof. See Appendix D.3 for a detailed proof.
Our efficient modern Hopfield models achieve high effi- ciency through two means: a sparse-structured mask and various choices of the kernel d. The sparse-structured mask, with a support set size of < M, reduces the complexity from (Mdd) to Okd?). Additionally, different choices of kernel, such the linear kernel and positive random kernel in Appendix E, lead to efficient implementations.
We emphasize that (3.7) is in fact generic and is able to de- scribe many sparse-structured modern Hopfield models with various support sets. Importantly, it allows us to construct efficient variants with sub-quadratic complexity, and hence fills the void in the literature regarding efficient modern Hopfield models, as discussed in (Hu et al., 2023).
Numerically, we verify their performance in Appendices G. 1 to G.4 and efficiency in Appendix G.5 (e.g., duration time in Figure and Floating point operations in Figure 7.)
We present three efficient variants based on (3.7) below. To analyze efficiency for long query sequences", we first generalize (3.7) from a single query x to a sequence of L query denoted by X- (x,,...> Xr.). Let the binary matrix Ly be the corresponding sparse-sturctured mask.
4 Theoretical Analysis
Example 1 (Random Masked Modern Hopfield Model with (k..) Complexity). By setting M to randomly mask (M - k) entries, we obtain an efficient modern Hopfield model with sub-quadratic OlkL) complexity. This model connects to the random attention of BigBird (Zaheer et al., 2020).
In this section, we characterize how sparsity affects the sparse-structured models defined in (3.7). Our theoretical analysis on these new sparse models consists of the follow- ing two aspects:
1. Derive the prove their fixed point convergence.
retrieval error bound and
Example 2 (Efficient Modern Hopfield Model with (I.VL) Complexity). By setting M for each query in way that IIl reproduces the sliding window pattern of
2. Characterize the fundamental limit of memory capacity.
'Considering long query sequences is crucial, they contribute to inefficiency (see (Hu al., 2023, Section C.2)).
'We plural "models" M in (3.7) is a generic expression for many models with different sparse patterns.
7

---

Nonparametric Modern Hopfield Models
As reminder, we adopt Definition 3.2 for memory storage and retrieval. Additionally, we recall the following defini- tion regarding the separation between memory patterns.
Computationally, Corollary 4.1.1 suggests that Sperse ne- cessitates fewer iterations reach fixed points compared to TDense given the same error tolerance level. In other words, Tsparse retrieves stored memory patterns faster than Dease- Remark 4.3 (Multiple- -upple))) Another important im- plication of Corollary 4.1.1 is that T exhibits similar multiple-update functionality to existing models (Hu et al., 2023; Wu et al., 2024b; Ramsauer et al., 2020).
Definition 4.1 (Separation of Patterns). The separation of a memory pattern Eμ from all other memory patterns E is defined as its minimal inner product difference to any other patterns: A == Mingart [S En) - [.
4.1 Memory Retrieval: Error Bounds and Fixed Point Convergence
To bridge to deep learning methodologies, we show that Tsparse retrieves memory patterns with high accuracy after single activation in the following corollary, akin to (Hu et al., 2023; Wu et al., 2024b).
To analyze the accu- racy of memory retrieval, we derive the upper bound on retrieval error of the sparse-structured models.
Memory Retrieval Error Bounds.
Corollary 4.1.2 (One-Step Retrieval with High Accuracy), For any query x E Sμ and μ E M.. Tsparse retrieve the memory pattern É,, with retrieval error e exponentially sup- pressed by A..
Theorem 4.1 (Sparsity-Depent
Retrieval Error). Let Tsparse be the sparse-structured retrieval dynamics (3.7). For query x E Sp, it holds
Tsspasse (x) - -Eull
(4.1)
Proof. See Appendix D.5 for a detailed proof.
S <m(M M + k k- exp -3( (S(L))
Max rE(M),N*
(Spr Su))
Corollary 4.1.2 indicates that, with sufficiently large Ane Tsparse retrieves memory patterns in a single iteration, al- lowing the integration of models into deep learning architectures similarly to (Xu et al., 2024; Hu et al., 2024a; Wu al., 2024b; Schimunek et al., 2023; Hoover et al., 2023; Seidl et al., 2022).
for all μ E M, where k = [MD E [M] denotes the size of the support set M, and m = Max, ||
modern Hopfield
Proof. See Appendix D.4 for a detailed proof.
Fixed Point Convergence. By design, the retrieval dy- namics constructed via Lemma 3.1 satisfy (T2). We now verify this adherence as samity check. Interestingly, while previous studies (Hu et al., 2023; Wu et al., 2024b; Ram- sauer et al., 2020) rely on the detailed energy functions to show the convergence properties of modern Hopfield mod- cls, we prove them for sparse-structure modern Hopfield models even without knowing E in the next corollary. It affirms that Tsparse satisfies (T2).
Interestingly, the retrieval error bound in Theorem 4.1 is sparsity-dependers which is governed by the size of the support set M, i.e. sparsity dimension & == IM].
Remark 4.1 (Comparing with the Sparse Modern Hopfield Model (Hu et al., 2023)). Compared to the retrieval error bound in (Hu et al., 2023), which lacks explicit dependence on its input (data)-dependen) sparsity, the sparsity (size of M) here is pre-specified. When there are fewer elements in the sparse-structured mask, i.e., when k is small, the retrieval error bound is tighter, and vice versa.
Corollary 4.1.3 (Fixed Point Convergence). Let Tsparse be the sparse- structured retrieval dynamics (3.7). For all M, the query E Sp. converges to a fixed point if it is iteratively applied by Tsparse-
Remark 4.2 (Noise Robustness). By Theorem 4.1, in cases involving contaminated query or memory, i.c. x = + dx (noise in query) or & E + og (noise in memory), the impact of noise on the sparse retrieval error (4.1) is less than that its impact on the dense counterpart due to the smaller coefficient (M- + k - 2).
Proof. See Appendix D.6 for detailed proof.
4.2 Memory Capacity
Corollary 4.1.1. Let Thense and Tsparse be the dense (3.7) and sparse-structured (3.7) retrieval dynamics, respectively. For any query pattern x E S,, and μ E M,, it holds
To characterize the fundamental limit of memory capacity, we ask the following two questions for sparse-structured modern Hopfield models following (Hu et al., 2023):
T (x) - E. < Deusse (x) - Exll-
(A) What is the necessary condition for a pattern e,, being considered well-stored, and correctly retrieved?
(B) What is the expected number of memory patterns such that the above condition is satisfied?
Proof. See Appendix D.5 for a detailed proof.
8

---

Nonparametric Modern Hopfield Models
Well-Separation Condition. To address (A), we identify the necessary condition for pattern being well-stored and retrieved by the sparse-structured modern Hopfield models.
5 Conclusion and Discussion
We introduce nonparametric framework for modern Hop- field models. We use two examples to validate our frame- work: the original dense & the sparse-structured modern Hopfield models. With Lemma 3.1, we replicate the known results of the original modern Hopfield model (Ramsauer et al., 2020). With Theorem 3.2, we introduce the efficient sparse-structured Hopfield models with robust theoretical properties: tighter retrieval error bound (Corollary 4.1.1 1.1 & Corollary 4. 1.2), stronger noise robustness (Remark 4.2) and exponential-ines capacity (Lemma 4.1 & Proposition 4.1).
Lemma 4.1 (Well-Separation Condition). Following Def- inition 3.2, for j E M, suppose every memory pattern (Su)uem is enclosed by sphere Sμ == {x ||x Eull x R), with finite radius R. == 1 w.e.e ||,, E- Then, the retrieved dynamics Tsparse maps Sy to itself if
1. The starting point is inside Sp: E Sp.
2. The well-separation condition:
(M + h 2)m
pa
2 In
R
+ 2mR.
Comparing with Existing Works. Our framework com- plements existing works (Hu et al., 2023; Wu et al., 2024b; Martins et al., 2023) by filling the efficiency gaps and con- necting to various attentions in the following. Notably, when the size of the support set k = M, the results of Theo- rem 4.1, Lemma 4. 1 and Proposition 4.1 reduce to those of the dense modern Hopfield model (Ramsauer et al., 2020).
Proof. See Appendix D.7 for a detailed proof.
Intuitively, the well-separation condition establishes a threshold that ensures any pattern {Ea]ueM is distinguish- able from all others, enabling patterns to be well-stored at a fixed point of Tsparse and retrieved with R precision by Tsparse Notably, Lemma 4.1 reveals that the lower bound on 42 diminishes as k decreases. Consequently, becomes sparser, satisfying the well-separation condition becomes easier, facilitating the storage of patterns and lead- ing to a larger memory capacity lower bound for sparse- structured modern Hopfield models.
Extensions. In Appendix E, present a family of mod- ern Hopfield models connecting to many other existing at- tention mechanisms, including Linear (Katharopoulos et al., 2020), Multi-Head (Vaswani et al., 2017), and Generalized Kernelizable or PRFs (Positive Random Features) (Choro- manski et al., 2021) modern Hopfield models.
M
Hopfield Layers and Numerical Experiments. In line with (Hu et al., 2023; Wu et al., 2024b; Ramsauer et al., 2020), we introduce deep learning layers as competitive attention alternatives with memory-enhanced functionalities (Remark F.1), corresponding to nonparametric modern Hopfield models (sparse-structured and above extensions) in Appendix F. Numerically, we verify their memory retrieval (as associative memory models) and supervised learning (as transformer alternatives) performance in Appendices G.I to G.4 and efficiency in Appendix G.5.
Memory Capacity. To address (B), we derive the lower bound for the maximum number of memory patterns that well-stored and retrievable according to Lemma 4.1:
Proposition 4.1 (Modified from (Hu et al., 2023)). De- fine the probability of storing and retrieving memory pat- tern as p. Memory capacity, the maximum number of patterns randomly sampled from sphere with radius m that the sparse modern Hopfield models can store and retrieve, has an lower bound: MSpprse 2 V where C' is the solution for the identity C' = // (/ with the principal branch of Lambert W function, == (4/-11) (In (mn(/+t-11// 1//]] + 1) and b = 4m 9/s(d-1).
Accuracy-Efficiency Tradeoff. For learning tasks con- ducted in Appendix G, we do not expect generally superior performance from efficient models. Ultimately, there is the provably sccuracy-efficieng tradeoff (Keles et al., 2023; Deng et al., 2023) based on complexity analysis of matrix multiplication (hence, this result is transferable to modern Hopfield models (Hu et al., 2024b)). This work only pro- vides a theoretical framework supporting the derivation of efficient variants of modern Hopfield model, with no strictly superior performance guarantee. However, we do observe that, in many cases, linear and random features modern Hopfield models deliver acceptable results.
Proof. See Appendix D.8 for a detailed proof.
Remark 4.4. Proposition 4.1 gives a memory capacity ponential in the pattern size d (maximum allowed value k). Since t < M,, the scaling behavior of sparse-structured modern Hopfield models is similar to that of (Ramsauer et al., 2020; Hu et al., 2023). This result mirrors findings in (Wu et al., 2024b; Hu et al., 2023; Ramsauer et al., 2020).
Limitations and Future Work. We defer the discussion of limitations and future work to Appendix

---

Nonparametric Modern Hopfield Models
Impact Statement
Brauchart, S., Reznikov, A. B., Saff, E. B., Sloan, I. H., Wang, Y. G., and Womersley, R. S. Random point sets the sphere-hole radii, covering, and separation. Experi- mental Mathematics, 27(1))62-81, 2018.
By the theoretical nature of this work, we expect no negative social impacts.
Acknowledgments
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
JH thanks Thomas F. Burns, Dmitry Krotov, Saul Santos, Andre Martins, Mimi Gallagher, Sara Sanchez, Dino Feng, and Andrew Chen for enlightening discussions; Weimin Wu and Jennifer Zhang for collaborations on related top- ies; Jaiyi Wang for assistance with numerical experiments; and the Red Maple Family for their support. The authors also thank the anonymous reviewers and program chairs for their constructive comments, especially the Area Chair of NeurlPS 2024 for pointing out key typos in appendix.
Carbonneau, M.-A., Cheplygina, V., Granger, E., and Gagnon, G. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recog- nition, 77:329-353. 2018
Child, R., Gray, S., Radford, A., and Sutskever, Gen- erating long sequences with sparse transformers. arXiv preprini arXiv: 1904::9944100 10509, 2019.
JH is partially supported by the Walter P. Murphy Fellow- ship. HL is partially supported by NIH ROILM1372201, Vie and Dolby. This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and North- western University Information Technology. The content is solely the responsibility of the authors and does not neces- sarily represent the official views of the funding agencies.
Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q. Mo- hiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In International Conference on Learning Representations, 2021.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 15/ 1.07289, 2015.
References
Amari, S.-I. Characteristics of random nets of analog neuron-like elements. IEEE Transactions on systems, man. and cybernetics, (5):643-57, 1972.
Correia, G. M., Niculae, V., and Martins, F. Adaptively sparse transformers. arXiv preprint arXiv:1909 00015, 2019.
Auer, A., Gauch, M., Klotz, D., and Hochreiter, S. Con- formal prediction for time series with modern hopfield networks. Advances in Neural Information Processing Systems, 36, 2024.
Demircigil, M., Heusel, J., Löwe, M., Upgang, S., and Vermet, F. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288- 299, 2017.
Awad, M., Khanna, R., Awad, M., and Khanna, R. Support vector regression. Efficient learning machines: Theo- ries, concepts, and applications for engineers and system designers, pp. 67-80, 2015.
Deng. Y., Song, Z., and Zhou, T. Superiority of softmax: Unveiling the performance edge over linear attention. arXiv preprint arXiv:2310. 1685, 2023.
Beltagy. I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.000 05500, 2020.
Floridi, L. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx. S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 2021.
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D., Kopp, M., Klambauer, G., Bitto, A., et al. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in neural information processing systems, 35:20450 23:20450-20468, 2022.
Brandstetter, J. Blog post: Hopfield networks is all you need, 2021. URL https://mlljjuu.ae cessed: April 4, 2023.
Gupta, A., Dar, G., Goodman, S., Ciprut, D., and Berant, J. Memory transformers via top-k attention. arXiv preprint arXiv:2 106.06899, 2021.
-aaeeiiied-lyyerss..... Ac-
10

---

Nonparametric Modern Hopfield Models
Hamid, R., Xiao, Y., Gittens, A., and DeCoste, D. Compact random feature maps. In International conference machine learning, pp. 19-27. PMLR, 2014.
Ji. Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioin- formatics, (1115)2211-21200 2021.
Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D. H., Zaki, M. J., and Krotov, D. Energy trans- former. arXiv preprint arXiv: 07253, 2023.
Kandemir, M., Zhang. C., and Hamprecht, F. A. Empower- ing multiple instance histopathology cancer diagnosis by cell graphs. In Medical Image Computing and Computer- Assisted InterventionMICCC
Hoover, B., Chau, D. H., Strobelt, H., Ram, P., and Krotov, D. Dense associative memory through the lens of random features. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
2014: 17th International Conference, Boston, MA, USA, September 14-18, 2014, Proceedings, Part 11 17, pp. 228 228-235, Springer, 2014.
Kar, P. and Karnick, H. Random feature maps for dot product kernels. In Artificial intelligence and statistics, pp. 583-591, PMLR, 2012.
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 798)):254-22588 1982.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers mnns: Fast autoregressive transformers with linear attention. In International conference chine learning, pp. 5156-5166 PMLR, 2020.
Hopfield, J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10): 3088-3092, 1984.
Keles, F. D.. Wijewardena, P. M., and Hegde, C. On the computational complexity of self-attention. In Interna- tional Conference on Algorithmic Learning Theory, pp. 597-619. PMLR, 2023.
Hu, J. Y.-C., Yang, D., Wu, D., Xu, C.,, Chen, B.-Y., and Liu, H. On sparse modern hopfield model. In Thirty-seventh Conference on Neurul Information Processing Systems, 2023.
Kozachkov, L., Kastanenka, K. V., and Krotov, D. Building transformers from neurons and astrocytes. bioRxiv, pp. 2022-10, 2022.
Hu, J. Y.-C., Chang, P.-H., Luo, R., Chen, H.-Y., Li, W., Wang, W.-P., and Liu, H. Outlier-effcieen hopfield lay- ers for large consformer-based models. In Forty-first International Conference on Machine Learning, 2024a.
Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Hu, J. Y.-C., Lin, T., Song, Z., and Liu, H. On computational limits of modern hopfield models: A fine-grained com- plexity analysis. In Forty-first International Conference on Machine Learning, 2024b.
Krotov, D. and Hopfield, J. J. Large associative memory problem in neurobiology and machine learning. Inter- national Conference on Learning Representations, 2021.
Hu, J. Y.-C., Wu, D., and Liu, Provably optimal mem- ory capacity for modern hopfield models: Transformer- compatible dense associative memories as spherical codes. Advances in Neural Information Processing Sys- tems, 38, 2024c.
Martins, A. and Astudillo, R. From softmax to sparsemax: sparse model of attention and multi-label classification. In International conference on machine learning, pp. 1614 1623. PMLR, 2016.
Martins, A. F. T., Niculae, V., and McNamee, D. Sparse modern hopfield networks. Associative Memory & Hop- field Networks in 2023. NeurIPS 2023 workshop., 2023.
latropoulos, G., Brea, J., and Gerstner, W. Kernel memory networks: A unifying framework for memory modeling. Advances in Neural Information Processing Systems, 35: 35326-35338, 2022.
Nguyen, T. M., Nguyen, T., Ho. N., Bertozzi, A. L., Bara- niuk, R. G., and Osher, S. J. A primal-dual framework for transformers and neural networks. arXiv preprint arXiv:2406. 1378/, 2024.
Ilse, M., Tomezak, J., and Welling, M. Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
Niculac, V., Martins, A., Blondel, M., and Cardie, C. Sparsemap: Differentiable sparse structured inference. International Conference on Machine Learning, pp. 3799-3808 PMLR,, 2018
Jaggi, M. An equivalence between the lasso and support vector machines. Regularization, optimization, kernels, and support vector machines, pp. 1-26, 2014.

---

Nonparametric Modern Hopfield Models
Olver, F. W., Lozier, D. W., Boisvert, R. F., and Clark, C. W. NIST handbook of mathematical functions hardback and CD-ROM. 2010.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Paischer, F., Adler, T., Patil, V., Bitto-Nemling, A., Holzleit- ner, M., Lehner, S., Eghbal-Zadch H., and Hochreiter, S. History compression via language models in reinforce- ment learning. In International Conference on Machine Learning, pp. 17156-17181 17185. PMLR, 2022.
Wainwright, M. Ifigh-dimensional statistics: non- asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Widrich, M., Schäfl, B., Pavlović, M., Ramsauer, H., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, K., Greiff, V., Hochreiter, S., al. Modern hopfield networks and attention for immune repertoire classification. Advances in Neural Information Processing Systems, 33:1 23:18832- 18845, 2020.
Qiu, J., Ma, H., Levy, O., Yih, S. W.-t, Wang, S., and Tang, Blockwise self-attention for long document understand- ing. arXiv preprint urXiv:/91 1.0 02970,9922222 2019.
Ramsauer, H., Schaft, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlovic, M., Sandve, G. K., et al. Hopfield networks is all you need. arXiv preprint arXiv:2008..2217, 2020.
Wu, D., Hu, J. Y.-C,, Hsiao, T.-Y., and Liu, H. Uniform memory retrieval with larger capacity for modern hop- field models. In Forty-first International Conference on Machine Learning, 2024a.
Santos, S., Niculae, V., McNamee, D., and Martins, F. Hopfield-fenchel-yonnn networks: A unified frame- work for associative memory retrieval. arXiv preprint arXiv:24/1.08590 2024a.
Wu, D., Hu, J. Y.C.,, Li, W., Chen, B.-Y., and Liu, H. STanhop: Sparse tandem hopfield model for memory- enhanced time series prediction. In The Twelfth Interna- tional Conference on Learning Representations, 2024b.
Santos, S., Niculae, V., McNamee, D., and Martins, A. F. Sparse and structured hopfield networks. arXiv preprint arXiv:2402.13775, 2024b.
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M.. Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564, 2023.
Schimunek, J., Seidl, P., Friedrich, L., Kuhn, D., Rippmann, F., Hochreiter, S., and Klambauer, G. Context-enriched molecule representations improve few-shot drug discov- ery. In The Eleventh International Conference on Learn- ing Representations, 2023.
Xu, C., Huang, Y.-.., Hu, Y.-C., Li, W., Gilani, A., Goan, H.-S., and Liu, H. Bishop: Bi-directional cellular learning for tabular data with generalized sparse modern hopfield model. In Forty-first International Conference on Ma- chine Learning, 2024.
Schölkopf, B. and Smola, A. J. Learning with kernels:
support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q. Yang, L., et al. Big bird: Transformers for longer se- quences. Advances in neural information processing systems, 33: 33: 13217733-1729,, 2020.
Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, J. K., Segler, M., Hochreiter, S., and Klam- bauer, G. Improving few-and zero-shot reaction template prediction using modern hopfield networks. Journal of chemical information and modeling, 22122221 2022.
Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri. R. V., and Liu, H. DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth International Conference on Learning Representations, 2024a.
Sriperumbudur, B. K. and Lanckriet, G. R. On the con- vergence of the concave-convex procedure. In Advances in neural information processing systems, volume 9, pp. 1759-1767, 2009.
Zhou, 7., Wu, W., Ho, H., Wang. J., Shi, I.,, Davuluri, R. V., Wang, Z., and Liu, H. Dnabert-s: Learning species-aware dna embedding with genome foundation models. arXiv preprint arXiv:2402.08777,, 2024b.
Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey, ACM Computing Surveys, 55(6): 28, 2022.
Vapnik, V. The nature of statistical learning theory. Springer science & business media, 2013.
12

---

Nonparametric Modern Hopfield Models
Supplementary Material
A Table of Notations
14
Limitations and Future Work
15
Supplementary Theoretical Backgrounds
16
C.I Soft-Margin Support Vector Regression
16
D Proofs of Main Text
17
D.1
Theorem 3.1
17
D.2 Lemma 3.1
17
D.3 Theorem 3.2
20
D.4 Theorem 4.1
21
D.5 Corollary 4.1.1 and Corollary 4.
D.6 Corollary 4.1.3
22
D.7 Lemma 4.1
22
D.8
Proposition 4.1
23
Nonparametric Modern Hopfield Family
25
E.I Linear Modern Hopfield Model
25
E.2 Multi-Head Modern Hopfield Models
25
E.3 PRFs (Positive Random Features) Kernel Modern Hopfield Model
26
F Nonparametric Modern Hopfield Layers for Deep Learning
27
G Experimental Studies
28
G.1 Memory Retrieval Task (Figure 2)
28
G.2 Multiple Instance Learning on MNIST (Figure & Figure 4)
30
G.3 Multiple Instance Learning on Real World Datasets
33
G.4 Time Series Prediction
35
G.4.1 Implementation Details
35
G.5 Computational Efficiency
37
13

---

Nonparametric Modern Hopfield Models
A
Table of Notations
Table Mathematical Notations and Symbols
Symbol
Description
a.j
The i-th component of vector a Inner product for vectors a, Rdd Index set (1. ... /) where E N+
(a, b)
7
Spectral norm, equivalent the 22-norm when applied vector
d
Dimension of patterns
M
Number of stored memory patterns
8
Scaling factor of the energy function controlling the learning dynamics. We set 1/vá in practice
x
State/configurationnqqer pattern in I2"
x*
Stationary points of the Hopfield energy function
g
Memory patterns (keys) in Rd
gg
Noises in memory patterns in RRd Training data set [(E, + & [uM)
D
Shorthand for M stored memory (key) patterns [Eu]ue]M] R
E,
Shorthand for M contaminated memory (key) patterns {6g, (eSMa in Rexar M-dimensional overlap vector ((EI,)),.. ... (E, x) ..... (EM. x)) RM
E
4())
Kernelized feature mapping ((.): R2 D
Flement the (b-) = (%) Φ1
(1)
D.......
D...
De
Dimension of the kernel space, i.e., dimension of output of (-)
k(-
Normalization mapping in the regression model defined by (3.1)
W
Weighted matrix in the regression model defined by (3.1) in Bd
i-th row of the weighted matrix W in R DD Kernel function takes the inner product form K(--)) (+()), (b-)) in K.: :R pm Ro + R.
W;
K(G-)
é'
Component-vv wise term error margin in the support vector regression problem Slack variables in the support vector regression
77
C
Penalized coefficient the support vector regression
C
Lagrangian corresponding to (3.2)
a ā. X.
Dual variables in the Lagrangian C
M
Reduced support set for TsvR M =
C II,...,,M
I LM(())
Indicator function corresponding M, where 1 LM(() for μ E M and [M(() 0 μ $ M Size of the support set M, dcfined as k = .M]
k
m
Largest norm of memory patterns, denoted as m = [a ||
R
Minimal Euclidean distance across all possible pairs of memory patterns, denoted as R = } Min 11E, - Eull Sphere centered memory pattern E,, with finite radius Fixed point of T covered by Sy. i.e., x1 E Sμ
S,
X
Separation a memory pattern &, from all other memory patterns 3, defined in (4.1)
Separation E at given x from memory patterns E, defined (4.1)
14

---

Nonparametric Modern Hopfield Models
B Limitations and Future Work
By the theoretical nature of this work, we rely on certain simplifying assumptions. These assumptions limit the generality of our results.
We require specific feature mapping & to guarantee fixed-point convergence (Section 3.2). This requirement imposes structure on the retrieval dynamics. Without it, an unconstrained nonparametrie Hopfield update can converge "generalized" fixed point, not the intended memory (Remark 3.3). Then, multiple iterations may fail to recover the true stored pattern. We emphasize that meeting this requirement is easy (Section 3.2); see also the nonparametric modern Hopfield model family in Appendix E.
Our relative error analysis (Theorem 4.1) assumes that the correct memory item is in the chosen support set. If random- masking step removes that item, retrieval fails (Figure 2). We remark that while this assumption restrictive, necessary for tractable analysis.
Our exponential capacity result in Proposition 4.1 needs strong "well-separation" condition. Each pattern must be distinct enough from the others. Many real datasets have correlated or structured patterns, so this assumption may be hard to satisfy. Still, it is standard in modern Hopfield model literature (Krotov & Hopfield, 2016; Demircigil et al., 2017; Ramsauer et al., 2020; latropoulos et al., 2022; Hu et al., 2023; Wu et al., 2024b; Santos et al., 2024a;b). To mitigate this in practice, (Wu al., 2024a; Hu et al., 2024c) relax this data dependency by optimizing the Hopfield-energy landscape for larger memory capacity.
Lastly, we do not analyze the extensions introduced in Appendix E. We do not prove their convergence or capacity.
Looking ahead, we plan to address these theoretical gaps and to relax strict assumptions like well-separated patterns.
15

---

Nonparametric Modern Hopfield Models
Supplementary Theoretical Backgrounds
C.1 Soft-Margin Support Vector Regression
Soft-margin Support Vector Regression (SVR) (Awad et al., 2015; Jaggi, 2014; Vapnik, 2013; Kar & Karnick, 2012; Schölkopf & Smola, 2002) generalizes Support Vector Machines (SVM) to regression tasks. It finds function
f(x)
W (x) +
that remains within an e'ttube of the target outputs, while allowing limited violations (soft margin) when data points lie outside this tube.
Notation and Setup.
(w)pp) is the training set, where Xp E Rd and Yu E Rd.
④ Fd I2 De is feature map into (possibly high-dimensional) space.
W E Rs and b E Rd are the regression parameters.
To tolerate errors beyond e' while penalizing them, SVR introduces nonnegative slack variables n, and n for each data point. The soft-margin SVR with 1o-loss is formulated as
Primal Formulation.
YP (W, (xx)( b
|I + +C W.n
min
Σ (1, (m mw)) subject to
(W, 4 (x,)) +b- S E'1l
(C.1)
np. n. 2 0, #E [M[M]
where C >> controls the trade-off between model fidelity and penalty large deviations (1,, mn.) Since (C.1) is strongly convex, it admits unique global minimizer.
This formulation follows the standard SVR derivation; see (Schölkopf & Smola, 2002) for a comprehensive treatment.
Lagrangian and Dual Problem.
To solve (C.1), we form the Lagrangian:
C
Ws 2 A
[p [i]
a [t] (c' t -yll + (w,, (xxx)) + bli)) + [e + 7,,11 - (w,, +(x,)) b/i] +y
where Xy, X are multipliers for the slack constraints 7, n, 2 0, and au O₂ are multipliers for the é'tube constraints. By applying the Karush-Kuhn-Tuckke (KKT) conditions to L, one obtains the dual problem. In practical kernelized SVR, one uses a kernel K(X,, x.) = ($(x,), (xx.)), which may bypass explicit construction of +.
Summary. Soft-margin SVR balances a tight é'tube fit with penalty-based tolerance for outliers. Strong convexity guarantees a unique solution in the primal, while the dual formulation reveals sparse, data-driven representation in terms of support vectors. This dependence on the training data for model complexity classifies SVR as a nonparametric method. Remark C.1 (Nonparametric Nature of SVR). Although the primal objective in (C.1) involves a fixed matrix W, the dual solution is data-dependent. In particular, the regressor can be written as
M
f(x) = E (ap a.) (I(x,), ((x)) + +b, p=1
where only the points with nonzero (a,, σ,,) (the support vectors) affect f. The number of such points can grow with M, making SVR nonparametric method whose capacity adapts to the size of the training data.
16

---

Nonparametric Modern Hopfield Models
D Proofs of Main Text
D.1 Theorem 3.1
Proof of Theorem 3.1. The Lagrangian of convex optimization problem defined in (3.2) is
M
L
W:
11 +
Mu + x [i)
t=1
#=1
1=1
M
d
aul1 (e' + n
Ex (w., (EE,, + o(,)))
/=
( (e' + m (w:, (E,, gg,)) + Ex),)
(D.1)
where X [t]. [i] [ and [ are Lagrange multipliers. Next, we solve stationary condition with respect to w., 7) [ and 7. [i] from above Lagrangian and derive corresponding optimal solution. The Lagrangian in (D.1) admits a stationary solution, which is given by:
w; Σp (a [i] C. [i]) ( + gE.)) = 0,
C Ap []] a [i] 0,
(D.2)
C 1 0 [i] 0.
Substitute (D.2) into (3.1) to write
x
new
TTvR = (w7, ((x)),
with the learned weight matrix
M (x) + 6E,)) E Do.
W
Σ
(a
ER
E
The complementary slackness condition and dual feasibility of (D.1) are given by
a (c' + null EnIl]] + (w,, $(En + (g,))) = 0
(e' + [ (w,, (EE, + dE,)) + Ex(t) 0
(D.3)
a, (i), [pli] 2 0.
for all и E [M] and i E [d].
This completes the proof.
D.2 Lemma 3.1
To simplify our proofs, we define
(xx)
(xx)
(D.4)
((x)
where h(-) Rdd IR is some normalization function for later convenience.
To prove Lemma 3.1, we introduce the following three auxiliary lemmas.
17

---

Nonparametric Modern Hopfield Models
Lemma D.1. Let [x1l 2 0, [ following bounds
be solution to (D.2) with KKT conditions (D.3), Then a. xx has the
-C S a [ a C, Vy E [M].e [d].
Proof of Lemma D.I. We prove this lemma by contradiction. Recall that for each fixed values of and
a 2 0, 1 2 0.
Firstly, we assume [, R+ (non-zero), for all [M]] and E [d]. Recall complementary slackness conditions from (D.3)
é' + [ Eu[i] + (w,, (E,, + )Eμ)) = 0 nu [i] (w,, (E,,. (),,) + En
Combine above two equations write
1 + 1 26' S 0.
Since the womponent-wise error e' > O, we have [ + [ 0. This conclusion contradicts the assumption of the
non-negative condition on slack variables [1]. 7 [i] 2 0. Therefore, together with (D.2), at least one of a,,,1i), [ must be 0, for all y and all i. Subsequently, we have
0 0g 1 C and 0 s a.lil SC
which leads to
-C S a a,l SC
This completes the proof.
Lemma D.2 (Multinomial Expansion). Given y E Rdd The identity
y)) )
Ed
Σ
nd!
Ca!
Vil....
81 +lam
holds for all N.
Proof.
yxT
(1191 + ..... + xd
n! n!
1
n!!
(2131)" ) .... + G Cy!
(I.Z1) e, +... (IaMa)"
(E 4 n)
n!
Σ
21 Lal (a:)
t4
f1 ...
Vd 84
G G!...ed Ea!
Σ 44gn
d
Vey!...
Vegl... Cal
This completes the proof.
18

---

Nonparametric Modern Hopfield Models
Our next lemma restates known result that the exponential dot-product kernel admits infinitediimeniional feature
expansion via its power series (Nguyen et al., 2024; Hamid al., 2014; Kar & Karnick, 2012; Schölkopf & Smola, 2002). We include the derivation here for completeness.
Lemma D.3 (Closed-Form Exponential
Kernel), Let (C(.)) be the exponential dot-product kernel:
(x"y)"
K(x, cxp/ = (0(x), k(y))
where x, Id and maps the feature vectors x and y into infinite dimensional space. Then,
(()
(0b 3
PD.
J(n)
DD
7(n)
has closed form solution
fl ba Vhi
(n)
d
D'
where & + ......4. = n. < D' & D and D, = (dta-1).
Proof of Lemma D.3. Applying Lemma D.2 on the exp kernel, we have
[ y)
(0(x), F(y)) = Σ
n!
n n=
I(v.)
l..... La
n=0 41+.+t +o
Vi
ei!...la!!
R=0 tugn
20
I of Vigi..-ad ld!
yi .it...y'h Vid VGll... La!
0= tly=n
From above, we observe that, for each fixed n, there are (dta-1) terms in the summation. Consequently, F(x) has a solution
T(x)
(
(1)
elements
(4n-1
elements
where D,, = (dtn-1) and
la
for 1 < D' < Dn and & +...+ This completes the proof.
Proof of Lemma 3.1. Recall that the learned weight matrix W is composed of
M
T + 8L.))
(a
h(x)
Substitute w into (3.1) to write
M
Cn [] [n(11]
(T(E, + 5gw), (xx))
M [d|d] ap(d
(T + (Ew), F(xx) h(x)
Tensse (x)
h(E 6E,)) h(x)
h(E + 5E.))
#=1
19

---

Nonparametric Modern Hopfield Models
Let ;= := [ulll-aal 0. +0G,+
au [d] Fn Me, t66,)
and h(x) = E (E + dEv), ((x). Then Deuse reduces to
(E, + )En), F(x))
Dense (x) Σ M ( + ). x(x)) En-
v=l
Following Lemma D.3, here define the inner product of ① kernel K: Do x Do R_
(x(x), (E,, + o5,)) = K(x, En + )5,))
Dense is now given by
x(x, Ey + +E
Tense (x) Σ
Eu.-
(D.5)
Evm1 K(x, Ev + ó
Observe that (2.2) Tense takes a Boltzmann form: expt-)/ Σve expl-). By Lemma D.3, we take
(vBor ) (vBxr)
T
Vlll Vl!!...d Etd
with the kernel
(VV3x, V3E, + /366,,))
K(xx, Eu + oEp)
= Σ
(D.6)
n!!
n=
Substitute (D.6) into (D.5) and write
E ((v3x, VBE, + )BBEw))" /n!
Dense (x)
Vom1 Eino ((V3x, VBE, + vB8E.)) /t! Eu--
By Taylor's theorem, Tense takes the form
M Densee (x) Σ
ex Ex jEp)}
E. = SSotmax (S
(D.7)
#=1 DM e (x, E.r + kg,))
where (61. Exs) RdxM and E, (E1 + "6,," Eas + dEvr) 1 denote memories and noises in
memories, respectively. This completes the proof.
D.3 Theorem 3.2
Proof of Theorem 3.2. To take w* for the sparse-structured model, the partial derivatives of L with respect w., mulii] and Nu [i] must satisfy the stationarity condition
w; - EyeN (a - ( (EE,, + +EE,) = 0,
A 0pp [i] = 0,
C Apliy C. [i] 0.
Then, arrive
( + 8E,,)
Wi Σ (a 2,l(i)
h(x))
PEM
Following similar approach as in Appendix D.2, we derive the retrieval dynamics for the sparse sparse-structure structured modern Hopfield model:
Tsparse (x)
Σ
3oftmax(83]Jxx) E..
MEM
where the softmax is computed over JE x with BM = T..,,.; Ej 5E, ..... This completes the proof.
jEllM|]
20

---

Nonparametric Modern Hopfield Models
D.4
Theorem 4.1
Proof of Theorem 4.1. To connect Tsparse with Ayo first derive the bound on Sparss (x) Exil via (Ramsauer al., 2020) for μ M
S (x) Evrl S
E/ Σ
[Softmax (SEnx)] En
NE.
(1
[Softmax (d=]x)lu)E +
Σ
[Sotmmax (5 E
vEM
7
$
E
M
vEM,urAu
S ém +
(k- - 1)m
M
< M + k 2-
M -
mmM k 1-2) 2) exp{ -B (x, Ex)
Max (x, Ev) }
(D.8)
vE[M]
where k = [M), Max, ||5,11, == (M exp( -8 ((x, En)
MaxxElaN (x, Evi)} and the inequality
(Softmax(3E1xx x)] II+ E+ ( (x,, Evrr (x. En,)) S &cxpl -8 (xx,E,)) Max (x, Exi ) wE(M)
expl ((x, Ex) (x, {n))}
}
used in (D.8). This completes the proof.
D.5 Corollary 4.1.1 and Corollary 4.1.2
Proof of Corollary 4.1.1 and Corollary 4.1.2. Since (Ep, 2 (Ev, x) for all v + we have
[Softmax (D=]x)lu 2 [Softmax (3=!x)]u: for v # p.
For the support set M, we have
(E)
[Softmax (3= [3x)))]
(D.9)
Ejem (Ej, x)
2 [Sofmma (S=
(By the smaller denominator in (D.9)
This implies EvEM [Softmax (B=Nx)], is "pulled more strongly" toward e,, than EM [Softmax (B=rx)], Ev. To see this, we write
T (x) Euull
|1 [[Softmax (BE],))], Ep
Σ
[Softmax (8E Evll.
VEM,UAM
:=(
and
TDense (x) - EEu
= =[[ofttaax (BE xx] 1) +
Σ
[Softmax (BE]x)),, Evll.
ME(M).KA
(IT)
21

---

Nonparametric Modern Hopfield Models
By (D.9), (1) (IT). This means the pull toward Eμ is strictly stronger, and hence the softmax-weighted average EvEM [Softmax (1EMx)] &u lies closer E
[[Trpee (x) Eall [TDenss Ep1l S Tsparse (x) Evul S Dense (x) Eall-
(D.10)
This completes the proof of Corollary 4.1.1.
From (Ramsauer et al., 2020, Theorem 4), for any query x, Dense approximately retrieves memory pattern &μ. with retrieval emor exponentially suppressed by u
(x()) E & 2m(M 1) exp{ 3 (4,, 2m Max |||x En
By (D.10), Tsparse also enjoys above retrieval error bound. Therefore, TSpans (x) retrieves memory pattern Sμ with high accuracy after a single activation with sufficiently large A,. This completes the proof.
D.6
Corollary 4.1.3
Proof of Corollary 4.1.3, Recall (Hu et al., 2023, Lemma 2.2) that for initial query Xu E Sp
lim || xx - E O, t-000
(D.11)
where [x:]r=n a sequence generated by Tense from Xo. i.c. TDense = Xt+1-
Moreover, recall that for any query pattern x E Sμ
0 < Ssparse (x) E,ul [ Eall.
(D.12)
By applying squeeze theorem on (D.12) and (D.11), we have
lim Xx - -1ill = O, 100
where {= sequence generated by Tsparse i.c. Tsparse (3t) X1+1 This completes the proof.
D.7
Lemma 4.1
Proof of 4.1. Following (Wu et al., 2024b; Hu et al., 2023), we define the separation of &N given from all memory patterns =\ as
A Min [[x, Ex) - (x, E.)
Plug above into (D.8), and get
sparse (x) Exlll m(M k 2) exp exp(-125 -94
By Cauchy-Schwartz inequality, for all M,
KE Ep) - (x, En)) 5 |1E, - x|| ||E.|| S ||,, - x x||m,
we write A in terms of A,::
Á Ap 21|E, x|Im A, 2mR
(By x E Sw)
where R is radius of the sphere Sy. Since T is a mapping T Sp + Sy, output of the mapping T falls in S, with radius R.
Therefore, R is lower-bounded by
R z (M + 1: 2)
2mR))m [T(x) E.ll,
and thus
1 In
(M + k -2)mm
A,
2mR.
R
This completes the proof.
22

---

Nonparametric Modern Hopfield Models
D.8
Proposition 4.1
We built our proof on top of (Hu et al., 2023, Lemma 2.1), which consists steps:
(Step 1.) We establish a more refined well-separation condition, ensuring that patterns { NE[M]] are well-stored in H and can be retrieved by T with an eтrоr at most R.
(Step 2.) This condition is then related the cosine similarity of memory patterns, from which deduce inequality governing the probability of successful pattern storage and retrieval.
(Step 3.) We pinpoint the conditions for exponential memory capacity and confirm their satisfaction.
Proof of Proposition 4.1. Our proof is built on top of (Hu et al., 2023, Corollary 3.1.1) with different well-separation condition.
Let Amin = [in A and Puv Here we define Aii and Pur be the angle between two patterns 8 and 5"
In order for pattern Ep be well-stored, by Lemma 4.1, need
(M k 2)m
min
2m R.
R
On the other hand, we observe
min
Min ISSUUS
Im² (1 cos(6Hu))) m² cos(@mia))
where 0min Min S Oe E m]. Then, have
1 In (M + k 2)m²
m² [1- cos(mmi)))
2mR.
(D.13)
As result, the probability of successful storage and retrieval, i.e., the minimal separation Amin that satisfies Lemma 4.1, is given by
(M + h: 2)m + 2mR - p.
P
A 2 In
B
R
Inserting (D.13) into above, we obtain
1
(M + h: 2)m
m² [1 cos(mmin) 2
+ 2mR = p.
R
From (Olver et al., 2010, Equation (4.22.2)), for 0 < cos cossi) I 1, cos(6 (fmin) has an upper bound
82
< cos(Omin)
min
5
holds
p
02
min
1 In
(M + k: 2)m -2)m
2mmR p.
B
which leads to
V5M 2
[1
(M + 1: 2)m
7
P
M
In
+ 2mR
=1 =1- p.
77
3
23

---

Nonparametric Modern Hopfield Models
For later convenience, here we introduce extra M M/ on both sides.
Let wa = 2x 1/22 be the area of d-dimensional unit sphere manifold, with r(.) denoting the gamma function.
Following (Brauchart et al., 2018, Lemma 3.5), we have
VBM=
1 In
(M + k - 2)m
$
P M 1 mmnn
+ 2mR
=1 - P
R
2 1
2 Ya-15 15 +
1
M
(d-11)
(d-1)
1 In
(M + k - 2)m
2mR
(D.14)
R.
where aa is the ratio between the surface areas of the unit spheres in (d - 1) and d dimensions:
1 Wid-1
r
Yd d Wd
dv π T (
Recall d, M E N+, P E [0, 1].
Hence, it holds M = VDC بيه for some real values C E R.
Then, by (D.14), we have
(vpc+++ + -
d-1
(vbO+t)² (d-1) (dd-1) In
m
5 5
p S 0,
R.
and thus
(VDC*
+ ki - -1)
1=1
5 +
- (d-1)
In
+ B
1.
(D.15)
3
R
Further, we rewrite (D.15)
VDC x + - m
50
In
+1 1 s 0,
2 B
and identify
ml /p 1: 1)
4m 3 5(d-
a = d
In
+1 +1..
R
By (Hu et al, 2023, Lemam 3.1), takes the form
C Wa(exp(a + Inb))
(D.16)
where Wo(+) is the upper branch of the Lambert W function. Since the domain of the Lambert W function is x (-1/e, and the fact exp(a + In b] >> 0, the solution for (6. exists.
00)
When the inequality (D.15) holds, the lower bound on the exponential storage capacity M can be written as:
M 4
In particular, the above lower bound takes a form similar (Ramsauer al., 2020, Theorem 3). This completes the proof.
24

---

Nonparametric Modern Hopfield Models
E
Nonparametric Modern Hopfield Family
In this section, we derive a family modern Hopfield models as possible extensions based on the proposed framework (Theorem 3.1).
E.1 Linear Modern Hopfield Model
Proposition E.1 (Linear Modern Hopfield Model). Let ((x) = ,.(x) with the component Φb:
elu (0xx +1
(x)
Vi E [d].
(E.1)
M 1= (x(x), (E,, - o6H,)
where elu(-) denotes the exponential linear unit activation function proposed by (Clevert et al., 2015). By Theorem 3.1, fitting TsvR on D following (3.2) gives
Eme 1=1 ( 4 (E,, + 65,,)) Ex
Lincar (x)
EV= (x(x). ( )E,))
By setting the kernel mapping & linear feature map (E.1), obtain linear modern Hopfield model with linear complexity (n). Compared with dense modern Hopfield model, our proposed linear modern Hopfield model has time and memory complexity (n) instead of (n²)) since only need to compute Σø= ((E,, E5, and Σ ( gg,) once and reuse them for the computation of every query pattern. This model is by design connected to the random attention of linear attention (Katharopoulos et al., 2020).
E.2 Multi-Head Modern Hopfield Models
To derive the multi-head Hopfield model, cast Multi as multiple SVR problems such that the memorization of memory patterns E corresponds to training regression model Tuull on datasets { xE(M) with noises {E}. These S training data
sets are given {EE) d pe) {(E + öc </ [EE[M] To handle multiple regression problems, we extend the regression model (3.1) into the following.
Definition E.1 (Multi-Hese Regression Model), Given an input vector x E R.. The output ÿ e Fd of the regression model Tmuti is defined as:
y = Tuulu (x) = Σ w (w*+*(x)) Rd,
where W Radd W [wi.... will Rd for all [H], and (x() = (x) .bh (x)) Rd Rr. denote series of output projection matrices, weighted matrix and kernel mapping. respectively.
Adopting this multi-head regression model, we introduce the following multi modern Hopfield model.
Proposition E.2 (Multi-Head Modern Hopfield Models). Let ((.) ( for 1 < D' < Dn.
..... ...... D..... ..... "i",...... 0 (n)
(n) D. ....) with
(V3x1) (vBra)
( = E (E, + )E,)) (xx)) VEi.... Ex!
where & + ... + la = n, and Dn = (d By Theorem 3.1, fitting TsvR on H training data sets {(E)) + oE El) .., {E S5 E DrE(M following (3.2) gives
Maata (x)
Σ s=1
W (E, (3E)))
HH et al. (2024b) provide theoretical characterization of these possible extensions from the perspective of fine-grained complexity theory.
25

---

Nonparametric Modern Hopfield Models
This model is by design connected to the standard multi-head attention.
8.3
PRFs (Positive Random Features) Kernel Modern Hopfield Model
Proposition E.3 (Positive Random Features Modern Hopfield Model), Let +(-) (01..... )Dn) with
(xx) (1'1 ((I..))), eh ((Pm x)).. ...., ......,,..(...... (1((p1, 1 (mm x)]),
((x) VD+
where D4. = 1. m, v Fd R. $1,,..... Um are functions that map from R + R, and Pi..... Pm itd are vectors from some distribution A (A {p R4 2i Ps = 1f is the (d -)ddimensional unit simplex.). By Theorem 3.1, fitting TsvR on D following (3.2) gives
M
TpRF(x) = Σ
EDID 1 (x(x), (E + 6
where we adopt the normalization map Đ-1 = (EI, x) given by (Choromanski et al., 2021).
Comparing with regular modern Hopfield model, PRF Hopfield model' only has the linear space and time complexity, without any additional treatment such as introducing sparsity or low-rankness. The significance of this representational capability lies in its ability to facilitate precise comparison between softmax and alternative kernels in the context of extensive tasks, surpassing the capabilities of regular modern Hopfield models and enabling a comprehensive exploration of optimal kernels. This model is by design connected to the Performer-type attention (Choromanski al., 2021). In practice, the default option for P is standard Gaussian (Choromanski al., 2021).
"Along the line of research, Hoover al. (2024) also utilizes random feature approximation setting facilitate compressed memory storage for associative memory models.
26

---

Nonparametric Modern Hopfield Models
F Nonparametrie Modern Hopfield Layers for Deep Learning
Building on the link between the nonparametric modern Hopfield models and the attention mechanisms, we introduce the Nonparametric Hopfield (NPH) layers for deep learning.
Following (Hu al., 2023; Ramsauer et al., 2020), say X and 3 are in the associative space (embedded space), as they are embedded from the raw query R and Y memory patterns, respectively, via X RW Q Q, and E YWK K K, with some We and WR x- Taking the transpose of 'I in (3.3) (with given feature map b) and multiplying with Wy such that V = KWv, we have
Z = Q new Wv = TsvR (BQK)) V,
(F.1)
which leads to an attention mechanisms with various TsvR activation functions. Plugging back the raw patterns R and Y, we arrive the Nonparametric Modern Hopfield (NPH) layer(s),
NPH (R, Y) TsvR
YW&WV.
(F.2)
which can be seamlessly integrated into deep learning architectures. Concretely, the NPH layers take matrices R. Y as inputs, with the weight matrices Wo, WK. Wv. Depending on its configuration, it offers several functionalities:
1. Memory Retrieval: In this learning-free setting, weight matrices Wr K. Q. and Wy set as identity matrices. Here, R represents the query input, and Y denotes the stored memory patterns for retrieval.
2. NPH: This configuration takes R and Y as inputs. Intending to substitute the attention mechanism, the weight matrices WK. Wq. and Wv are rendered learnable. Furthermore, R. Y, and Y serve as the sources for query, key, and value respectively. Achieving self-attentiin--lkk mechanism requires setting R equal Y.
3. XPHPooling: With inputs Q and Y, this layer uses Q as a static prototype pattern, while Y contains patterns over which pooling is desired. Given that the query pattern replaced by the static prototype pattern Q, the only learnable weight matrices are WK K and Wy.
4. XPHLayer: The NPHLayer layer takes the query R as its single input. The layer equips with learnable weight matrices Wx x and Wv, which function as our stored patterns and their corresponding projections. This design ensures that our key and value are decoupled from the input. In practice, we set Wa and as identity matrices.
Remark F.1. We emphasize that Hopfield memory models and Hopfield networks/layers are conceptually distinct:
Hopfield memory model: A fixed content- addressalee memory model with no training; retrieval is based on similarity to stored patterns. This focus.
Hopfield networks (e.g., Hopfield Layers (Brandstetter, 2021)): Neural layers integrated into deep learning, trained with backprop. They builds on the "Dense Associative Memory ++ Transformer Attention" correspondence (Ramsauer et al., 2020), later generalized by (Hu et al., 2023; Wu et al., 2024b). These includes architectural innovations such as additional memory-enhanced functionalities studied prior works (e.g., (Ramsauer et al., 2020) for prototype learning, (Schimunek et al., 2023) for template enriching and (Wu et al., 2024b) for fast test-time adaptation.)
Our work extends the prior modern Hopfield memory model by framing it as a nonparametric regression problem. This allows efficient variants (Sections 3 and and connects attention mechanisms (e.g.. Performer (Choromanski et al., 2021)) under rigorous, unified theory.
27

---

Nonparametric Modern Hopfield Models
G Experimental Studies
Tasks. We verify the method proposed the main content with the following experimental sections, These tasks mainly focus on (1) validating the theoretical results, (2) the real-world application and (3) computational efficiency.
Appendix G.I: Memory Retrieval Task (Figure 2).
Appendix G.2: Multiple Instance Learning MNIST (Figure 4).
Appendix G.3: Multiple Instance Learning on Real World Datasets.
Appendix G,4: Time Series Prediction.
Appendix G.5: Computational Efficiency.
Baselines and Considered Models. We consider the following variations of Modern Hopfield Models in this paper:
Dense Modern Hopfield (Ramsauer et al., 2020)
Sparse Modern Hopfield (Hu al., 2023)
Sparse-Structurt Structured Modern Hopfield:
Random Masked Modern Hopfield
Window Modern Hopfield
Top-K Modern Hopfield
Linear Modern Hopfield
Random Feature Modern Hopfield
Experiment Environment. All experiments conducted the platform with NVIDIA GEFORCE RTX 2080 Ti and INTEL XEON SILVER 4214 @ 2.20GHz. We use PyTorch 1.8.0 for all experiments, and use RayTune for hyperparameter search.
Remark G.1 (One-Step Retrieval), For simplicity and as a proof concept, all experiments in this work use single-step (feed-forward) retrieval without iterative updates. This avoids confusion with multi-step or recurrent retrieval.
G.1 Memory Retrieval Task (Figure 2)
In the memory retrieval task, we examine two datasets: MNIST (sparse) and CIFARIO (dense). We employ the sum- of-squares distance between the retrieved image and the ground truth image to measure retrieval error. This experiment encompasses settings:
1. Half-masked image recovery, and
2. Noisy image recovery.
In the half masked image recovery scenario, we obscure half of the pixels the image. The memory set size (M) is varied from 200, and we report the average retrieval error (sum-of-square difference) over 50 runs. In the noisy image recovery scenario, fix the memory set size 100, and introduce varying scales of Gaussian noise to the image, with variance ranging from 0.1 to 1.4.
Implementation Details. The memory set itself is chosen randomly from the dataset each iteration. We adhere to the implementation outlined in (Hu et al., 2023).
28

---

Nonparametric Modern Hopfield Models
Results. We first clear differences in retrieval success among different Hopfield models (Figure 2).
For top-k Hopfield models, retrieval success depends directly on sparsity set by k. Lower means higher sparsity and lower retrieval errors. This aligns our theory (Thcorem 4.1).
The top-k model performs similarly to the Sparse Hopfield model (Hu et al., 2023) on sparse data (MNIST) with smaller & (higer sparsity). This aligns our theoretical result on capacity (Proposition 4.1). Our sparse-structured model maintains exponential memory capacity when the target memory is within support, ensured by the top-k operation.
In contrast, random masked Hopfield models perform poorly, especially MNIST (higher sparsity). This is because random masking may remove the target memory. This violates key assumption (fu E .M) in Theorem 4.1. Thus, their poor performance is expected.
These numerical results (corresponding Theorem 4.1 and Proposition 4.1) confirm that careful sparse strategies is capable of matching exceeding dense Hopfield retrieval in sparse settings.
Memory Capacity (MNIST)
Memory Capacity (CIFAR10)
1.0
Dense Hopfield [Ramsauer al., 2020]
Dense Hopfield [Ramsauer et al., 2020]
Sparse Hopfleld (Hu et al., 2023]
Sparse Hopfield [Hu et al., 2023]
0.8
Top 80% Hoptield
0.8
Top 80% Hopfteld
Retrieval Success Rate
Hopfield
Top 50% Hopfield
20% Hopfield
Top 20% Hopfleld
0.6
Random Masked Hoofield 0.2
Retrieval Success
Random Masked Hopfield 0.2
Random Masked Hoofield 0.5
Random Masked Hopfield
Hoofield
0.4
0.4
0.2
0.2
0.0
20
40
60
100
0
20
40
60
80
100
Number of Memery Patterss
Number Memory Patterns
Memory Robustness (MNIST)
Memory Robustness (CIFAR10)
1.0
Dense Hopheld [Ramsauer 2020]
Dense Hoptteld [Ramsauer et al.. 2020]
Sparse Hoofield (Hu 2023)
Sparse Hopfield [Hu
Top 80% Hopfield
Top 80% Hopfield
Retrivaa Success Rate
Top 50% Hopfield
Success Rate
Top 50% Hopfleid
Top 20% Hopfield
Top 20% Hopfleld
0.6
Hoofield
0.6
Masked
Random Masked Hoofield 0.5
Random Masked Hopfield 0.5
Random Masked Hoofield 0.8
Random Masked Hopfield 0.8
0.4
0.0
Noise Variance (o)
Noise Variance (o)
Figure Numerical Justifications for Theoretical Results: Memory Capacity and Noise Robustness. (Upper): Retrieval success from half-masked queries (Thcorem 4.1 and Proposition 4.1). (Lower): Retrieval success with different levels of Gaussian noise (Remark 4.2). We set 0.01 (MNIST) and (CIFARIO). Lines show averages over runs. Shaded areas show standard deviations. Top-k Hopfield models have retrieval rates tied to their sparsity: smaller gives lower success, aligning theory (Theorem 4.1). Top-k models perform similarly to Sparse Hopfield (Hu et al., 2023) on sparse data (MNIST). This aligns our theory on memory capacity (Proposition 4). Random masked models perform poorly, especially on MNIST: This is expected since random masking can remove the target memory, violating our the y E MM assumption Theorem 3.2. These results confirm our theory: careful sparsity can match or surpass dense models in sparse settings.
29

---

Nonparametric Modern Hopfield Models
G.2
Multiple Instance Learning on MNIST (Figure 3 & Figure 4)
Quoted from (Hu et al., 2023, Section 4.2):
Multiple Instance Learning (MIL) (Ilse et al., 2018; Carbonneau et al., 2018) is variation of supervised learning where the training set consists of labeled bags, each containing multiple instances. The goal of MIL is to predict the bag labels based on the instances they contain, which makes it particularly useful in scenarios where labeling individual instances is difficult or impractical, but bag-level labels are available. Examples of such scenarios include medical imaging (where a bag could be an image, instances could be patches of the image, and the label could indicate the presence or absence of disease) and document classification (where a bag could be document, instances could he the words or sentences in the document, and the label could indicate the topic or sentiment of the document).
In this experiment, we evaluate Dense Hopfield, Sparse Hopfield, and Top-K Hopfield models on Multiple Instance Learning (MIL) task using MNIST bags. This task is standard in modern Hopfield model literture (Ramsauer et al., 2020; Hu et al., 2023; Santos et al., 2024b).
Setup. We designate one digit from MNIST as a negative signal, and the remaining digits as positive signals. The objective is to predict whether a given bag of instances (digits) contains the negative signal. We vary the memory size (number of instances per bag) to study how task difficulty affects each model's performance and convergence. We vary the memory set size (M) from 5 to 100 and report the mean accuracy over 10 runs. We compare the performance of Dense Hopfield, Sparse Hopfield, Top-K Hopfield (with 20%, 50%, and 80%), Random Feature Hopfield, Random Masked Hopfield and Linear Hopfield models. We omit the Window Hopfield model for reasons mentioned earlier.
Implementation Details. We employ an embedding layer to project the flattened MNIST images into the hidden space, followed by a layer of layer normalization. Subsequently, we utilize the Hopfield Pooling layer to pool over all the instances in the bag, followed by second layer normalization layer. Finally, fully connected layer is used to project the hidden representation of the bag into the label space. All models are trained using the Adam optimizer for 1,50 epochs, with a cosine annealing learning rate decay applied to all models. Note that we exclude Window Hopfield in this and the subsequent MIL experiment since Window Hopfield requires both the query and memory pattern numbers be large to perform the sliding window operation. However, in our model structure, the number of query patterns in the pooling layer is set 2. The details of the hyperparameters can be found in Table 2.
Results. We report the results in Figure 3. Additionally, we also conduct convergence analysis in Figure 4 with bag size 50. We plot the loss and accuracy curve on MNIST MIL training and test set. Below, summarize the key findings, connecting them to our theoretical guarantees (e.g. e-sparse convergence from Theorem 3 and the sparse masking assumption):
Robust Performance with Increasing Memory Size (Figure 3): We measure accuracy as the bag size grows (x-axis of Table 2). Top-K Hopfield and Sparse Hopfield keep high accuracy even with large bags. Dense Hopfield struggles: it attends to all instances, so distractors dilute the target signal. As result, its performance drops when the bag is big. In contrast, Sparse and Top- K Hopfield focus on the most relevant entries. They ignore low-similarity memories and avoid noise. Thus, they hold strong performance as bag size increases. This aligns with our theory (Theorem 4. and Corollary 4.1.1), which shows that masking out small similarities enforces clear margin for correct retrieval. Random masked models do poorly if the mask removes the target memory, violating the E . assumption. Hence, selective sparsity helps retrieve the correct instance reliably, even with many distractors.
We compare the training loss and accuracy curves of various Hopfield models on the MNIST MIL task. Sparse and Top-K Hopfield models converge quickly and stably: their loss drops sharply and accuracy climbs rapidly, reaching near-perfect performance within a few epochs. Notable, Random Feature Hopfield model also exhibits relatively fast convergence and competitive performance. This behavior aligns with our e-sparse convergence theorem (Theorem 1),, as restricting updates to the largest memory entries drives the network to retrieve the correct instance in each bag without interference. In contrast, Dense Hopfield model (Ramsauer et al., 2020) improves more gradually and can plateau or oscillate before converging, reflecting weaker theoretical guarantees when many small memory entries compete. Also, Random masked models perform poorly with large masking (e.g. 80%). This expected since random masking can remove the target memory, violating our the μ E M
Convergence and Training Dynamics (Figure 4):
30

---

Nonparametric Modern Hopfield Models
assumption in Theorem 3.2. Ultimately, all models converge, but the sparse variants reach high accuracy faster and more reliably.
Dense Hopfield [Ramsauer at al. 2020]
0.9
Linear Hopfield
Random Feature Hopfield
0.8
Random Masked Hopfiell eld 0.2
Random Masked Hopfi eld 0.5
Random Masked Hopfield
Sparse Hopfield [Hu et al., 2023]
0.6
Top 20% Hopfield
Top 50% Hopfield
Top 80% Hopfield
0.5
100
Memory Size
Figure MIL Accuracy Memory Size MNIST. The y-axis represents the accuracy test set. We compare accuracy for Dense Hopfield (no sparsity), Sparse Hopfield (Hu et al., 2023), and Top-k Hopfield (exactly k active memory slots) models as the number of instances per bag increases. Larger memory size (more instances in bag) makes the task difficult due (blue), Linear (orange), and Random Masked Hopfield models lose accuracy significantly with larger bags. In contrast, Sparse Hopfield (pink) and Top-k models maintain high accuracy. Namely, sparse models filter irrelevant instances effectively. This aligns with our theoretical prediction that sparse retrieval improves robustness to large bag sizes (Theorem 4.1).
distractors. Dense
Table Hyperparameter used the MIL. MNIST experiment.
parameter
values
batch size
256
learning rate
le-3
embedding dimension
256
number of heads
head dimension test size
64
500
train set size scaling
2000
0.1
num of pattern
2
epochs
150
31

---

Nonparametric Modern Hopfield Models
Accuracy
Train
1.0
0.8
Random Feature Hopfield
Dense Hopfield [Ramnsuurr el el. 2020
0.7
Sparse Hopfield [Hu et al., 2023]
0.9
Linear Hopfield
Top 20% Hopfield
Train Accuracy
Random Feature Hopfield
Top 50% Hopfield
Dense Hopfield [Ramsauer et al. 2020]
Top 80% Hopfield
Sparse Hopfield [Hu et el., 2023]
Trann
Masked Hopfield
Hopfield
Random Masked Hopfield 0.5
20% Hopfield
Random Masked Hopfield 0.8
Top 50% Hopfield
Top 80% Hopfield
Random Hopfield
Random Masked Hopfield 0,5
Random Masked Hopfield 0.8
0
20
40
60
80
100
120
140
20
40
60
80
100
120
140
Egoch
Epoch
Validation Accuracy
Validation Loss
0.7
Random Feature Hopfield
Dense Hopfield [Ramsauer al. 2020)
0.9
Sparse Hopfield [Hu at al., 2023]
Lincar Hopfield
Validation Accuracy
Top Hogfield
0.8
Random Hopfield
oalidaton Loss
Top 50% Hopfield
Hopfield [Ramsauer 2020]
Top 80% Hopfield
Sparse Hopfield [Hu et all, 2023]
Random Masien Hopfield 0.2
Linear Hopfield
0.4
Hopfield
Hopfield
Random Masied Hopfield
50% Hopfield
0.6
Top 80% Hopfield
0.3
Random Masked Hopfield 0.2
Hopfield
0.2
0.5
Random Masked Hopfield 0.8
120
100
140
Epoch
Epoch
Figure 4. Convergence Analysis of Hopfield Models MNIST MII.. (Upper): Training loss and accuracy curves for various Hopfield models on the MNIST multiple instance learning task (Theorem 4. (Bottom): Validation loss and accuracy curves on the same task (Theorem 4. 1). All models are trained for 150 epochs with cosine annealing learning rate decay. Each line the mean over runs. Sparse Hopfield model by (Hu et al., 2023) attains the nearly the highest validation accuracy. Random Feature Hopfield model also converges quickly competitive performance, Top-20% Hopfield model converges fast and shows minimal performance drop. Dense Hopfield model converges more slowly, exhibiting occasional plateaus. Again, Random masked models perform poorly with large masking (e.g. 80%). This is expected since random masking can remove the target memory, violating our the E A assumption in Thcorem 3.2. In contrast, sparse updates (e.g. Sparse and Top-K) retrieve relevant entries more reliably, avoid spurious attractors, and ensure stable convergence. Consequently, these models learn faster and reach higher accuracy on sparse data (Theorem 4.1 and Corollary 4.1.1), while Dense Hoptield model faces interference from many memory entries. See Appendix G for more details.
32

---

Nonparametric Modern Hopfield Models
G.3 Multiple Instance Learning on Real World Datasets
For this experiment, we follow (Ramsauer et al., 2020; Hu et al., 2023) to conduct MIL experiments on real world datasets. However, employ simpler model structure and smaller hyperparameter search space, rendering results incomparable. We utilize four datasets: Elephant, Fox, and Tiger for image annotation (Ilse al., 2018), and UCSB breast cancer classification (Kandemir et al., 2014). We compare Dense Hopfield, Sparse Hopfield, TopK Hopfield at 20%, 50%, and 80%, Random Feature Hopfield, and Linear Hopfield. Random Masked Hopfield is excluded due to its non-deterministic inference, and Window Hopfield is omitted as previously mentioned. The results are presented in Table 5.
Dataset Details. The experiment is conducted on four MIL datasets. Elephant, Fox, and Tiger are designed for image annotation and consist preprocessed and segmented colored images. Each image characterized by descriptors for color, texture, and shape. These datasets each contain 100 positive images featuring the specified animal and 100 negative images drawn from set of images depicting other animals. Additionally, we evaluate model on the UCSB breast cancer classification task. In the UCSB dataset, each instance comprises patch of histopathological image depicting either cancerous or normal tissue. The detailed statistics of the datasets are reported in Table 3.
Table Statistics of MIL benchmark datasets
Name
Instances
Features
Bags
+ +bags
bags
Elephant
1391
230
200
100
100
Fox
1302
230
200
100
100
Tiger
1220
230
200
100
100
UCSB
2002
708
58
26
32
Implementation Details. We follow the experimental setting in (Ramsauer et al., 2020) and employ stratified 10-fold cross-validation to evaluate the performance of each baseline Hopfield model. In each fold, we utilize a stratified sampling process to partition the data into training set and validation set, with a split rate of 0.1. Hyperparameters are optimized via random search by maximizing the ROC-AUC score on the validation set. All reported ROC-AUC scores represent the average results over 5 runs with different random seeds. The random search space is delineated Table 4, with the number of trials set to 50 for each fold. The embedding layer, a pre- gereHoffielPoolingg linear network, has its layer width determined by the number of hidden units. A dropout operation, also referred to bag dropout, is applied post the embedding layer and the Hopfield Pooling layer. Notably, to better showease the performance of Top-k Hopfield, dropout is not applied the attention weight. All models are trained using the Adam optimizer over 50 epochs. To mitigate overfitting, an early-stopping mechanism is employed, selecting the best checkpoint based on the validation set.
Results. For real-world MIL datasets, Sparse Hopfield dominates most tasks (except for UCSB). However, other sparse- structured Hopfield models, especially Top-20% Hopfield, show comparable performance with Sparse Hopfield, indicating a potential trade-off between computational efficiency and model performance. For random feature and linear Hopfield, they did not outperform other baselines. However, their retrieval dynamics behave differently than other sparse-structured Hopfield models. Understanding how to fully utilize their potential and identifying the scenarios where they are most suitable is worth studying in the future
33

---

Nonparametric Modern Hopfield Models
Table Hyperparameter random search space on the respective validation sets of the Elephant, Fox, Tiger and UCSB breast cancer datascts.
parameter
values
batch size
84,8,16)
learning rates
(10-3.10 10 10
weight decay
10.10-3.10 10
layer width
{128, 256, 512)
number of heads
84
scaling factors
{0.1, 13 (0.0, 0.5)
dropout
Table 5. Results for MIL benehmark datasets in terms of AUC score. The results suggest that the proposed model achieves performance comparable to existing Dense and Sparse Modern Hopfield models (Hu et al., 2023; Ramsauer al., 2020). Note that, since our aim here is to conduct atomic setting for fair comparison, we employ simpler network structure (with smaller hyperparameter search space) compared to the ones used in (Hu et al., 2023; Ramsauer et al., 2020). Consequently, our results not align with those in (Hu al., 2023) for Dense and Sparse Modern Hopfield Models.
Method
Tiger
Fox
Elephant
UCSB
Dense Hopfield (Ramsauer et al., 2020)
0.813
0.563
0.877
0.524
Sparse Hopfield (Hu al., 2023)
0.830
0.573
0.893
0.585
Top-20 Hopfield
0.824
0.562
0.848
0.586
Top-50% Hopfield
0.812
0.566
0.852
0.572
Top-80% Hopfield
0.812
0.560
0.872
0.551
Random Feature Hopfield
0.802
0.508
0.875
0.566
Linear Hopfield
0.797
0.571
0.869
0.561
34

---

Nonparametric Modern Hopfield Models
G.4 Time Series Prediction
We further showcase the performance (in Table 6) and efficiency (in Figure 5) of the proposed nonparametric modern Hopfield models with multivariate time series prediction tasks.
Table 6. Time series prediction using different Hopfield layers (Appendix F) across five datasets. We evaluate cach datasct with different prediction horizons (showed in the second column). We report the average Mcan Square Error (MSE) and Mean Absolute Error (MAE) metrics of 5 runs. RF denotes the Random Feature Hopfield layer. One notable observation is that the noise level of the datasct significantly influences time series prediction. Therefore, employing Hopfield layers with strong noise-robustness offers performance improvements. Moreover, based our results, the proposed efficient Hopfield models not only offer significant computational efficiency hut also maintain comparable performance. Especially, the Random Feature Hopfield and Linear Hopfield layers (models) not only match but even outperform Dense Hopfield model in several settings. As side note, Window Hopfield exhibits significant performance degradation in most settings. This degradation arises because it solely focuses on local information. Being the only Hopfield model that does not span the entire associative range (i.c,, sequence length), it overlooks a substantial portion of the autoregressive correlation present in time series data. Wc also record the time used for one epoch on ETThl dataset with different prediction horizon (input length as well).
The duration time per epoch was showed in Figure 5.
Models
Dense
Sparse
Top20%
Top50%
Top80%
Window
RF
Linear
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
96
0.137
0.307
0.144
0.314
0.148
0.319
0.153
0.321
0.147
0.318
1.043
0.881
0.147
0.312
0,149
0.320
Erlml ETTTi
192
0,153
0,326
0.152
0.325
0.146
0.318
0.161
0.333
0.150
0.320
1.003
0.870
0.158
0.332
0.141
0.313
336
0.148
0.319
0.146
0.319
0.156
0.327
0.122
0,286
0.160
0.333
0.889
0.767
0.151
0.322
0.138
0.307
720
0.169
0.331
0.148
0,314
0.184
0.345
0.161
0.327
0.123
0.287
0,756
0.761
0.141
0.271
0.171
0.333
96
0.148
0.301
0.147
0.301
0.144
0.311
0.151
0.310
0.142
0.31o
0.943
0.854
0.151
0.314
0.155
0.319
192
0.189
0.350
0.187
0.340
0.191
0.347
0.185
0.338
0.188
0.341
1.054
0.893
0.190
0.347
0,192
0.348
336
0.163
0.320
0.165
0.322
0.168
0.331
0.161
0.312
0.169
0.330
0.873
0.334
0.175
0.333
0.176
0.337
720
0.159
0,300
0.161
0,303
0.165
0.313
0.167
0,313
0.169
0.320
0.764
0.731
0.162
0.309
0.165
0.310
96
0.378
0.371
0.373
0.370
0.384
0.382
0.386
0.386
0,383
0.376
0.989
0.854
0.390
0.403
0.365
0.378
WTH BCL
192
0.486
0.426
0.535
0.507
0.502
0.427
0.501
0,464
0,519
0.481
1.000
0.843
0.543
0.438
0.549
0.464
336
0.748
0.693
0.760
0.688
0.650
0.549
0.674
0,571
0,638
0.545
1.012
0.849
0.767
0.588
0.672
0.578
720
0.961
0.711
0.993
0.758
1.145
0.843
1.166
0,847
1,211
0.872
1.061
0.865
1.362
0.896
1.052
0.770
96
0.347
0.474
0.347
0.477
0.348
0.474
0.348
0.474
0.356
0.479
0.952
0.819
0.345
0.470
0.355
0.476
192
0.399
0.505
0.386
0.497
0.360
0.482
0.370
0.490
0.361
0.482
0.977
0.828
0.368
0.487
0.354
0.478
336
0.407
0,512
0.387
0.501
0,376
0.489
0.397
0.503
0.403
0,505
0,931
0.808
0.342
0.504
0.407
0.514
720
0.669
0.631
0,632
0.623
0.590
0.604
0.569
0.593
0.618
0.618
0.564
0.595
0.564
0.595
0.747
0.676
96
1.466
0.654
1.489
0.638
1,483
0.645
1.517
0.630
1.477
0.638
1,520
0.625
1,515
0.635
1.489
0.644
Traftic
192
1.551
0.654
1.550
0.657
1,557
0.649
1.548
0.657
1.551
0.652
1.570
0.637
1.551
0.654
1.551
0.653
336
1.595
0.663
1.595
0.662
1.599
0.663
1.592
0.665
1.604
0.657
1.612
0.646
1.613
0.646
1.614
0.646
720
1.660
0.681
1.671
0.671
1. 1.664
0.674
1.676
0.663
1.682
0.661
1.683
0.661
1.682
0.661
1.681
0.660
G.4.1
IMPLEMENTATION DETAILS
For ease of comparison, we employ the simplest possible architecture: an embedding layer to project each signal into a hidden space, followed by single Hopfield layer. By doing so, we treat every signal query pattern. Next, we employ a Hopfield Pooling layer to pool over all the signals into a single hidden vector. Finally, we utilize fully connected layer to generate the prediction. For all experiments, we maintain the same input and prediction horizon for simplicity. The results can be found in Table 6 and Figure 5.
Datasets. We conduct the experiments on four multivariate time series real-world datasets: ETThl (Electricity Trans- former Temperature-hourly) ETTml (Flectricity Transformer cemperature-minuedd WTH (Weather), ECL (Electricity Consuming Load), Traffic.
Setup. For each dataset, we use their univariate setting for our time series prediction experiment. We choose Dense, Sparse, Random Feature, Linear, TopK and Window Hopfield as baselines. We select 4 different prediction horizons for demonstration, which are 96, 196, 336, 720. We report the average error of 5 runs, evaluated using Mean Square Error (MSE) and Mean Absolute Error (MAE) metrics. For window Hopfield, we set the window size as 8, 12, 14, 16, w.r.t. 96, 196, 336, 720.
35

---

Nonparametric Modern Hopfield Models
Dense Hopfield [Ramsauer et al. 2020]
Linear Hopfield
16000
Random Feature Hopfield
14000 Sparse Hopfield [Hu al., 2023]
12000 Top 20% Hopfield
Top 50% Hopfield
10000 Top Hopfield
8000
Window Hopfield
6000
4000
2000
100
200
300
400
500
600
700
Memory
Figure The processing time comparison among different Hopfield models utilized in the time series prediction task described in Table We evaluate the efficiency of multivariate time series prediction on ETThl dataset. The findings consistent with the efficiency discussion in Section 3.2, where the Sparse/Dense/Top models (all with (d²) complexity) necessitate more time to complete an epoch. In conjunction with the results in Figure 4, it is evident that the efficient modem Hopfield models (Window, Linear, Random Feature) not only converge in fewer or comparable epochs but also require less time per epoch compared to the less efficient Seaarse/Dense/To--eR Hopfield models.
36

---

Nonparametric Modern Hopfield Models
G.5
Computational Efficiency
Here we demonstrate the computational overhead for different efficient modern Hopfield variants. We focus on the computational time duration and Flops (the number of Floating point operations). The results demonstrate
For random masked Hopfield, the computational time scales up with respect to probability.
Random feature Hopfield, Linear Hopfield and Window Hopfield demonstrates fast computational overhead in practice, In addition, these efficient Hopfield models also enjoy significantly lower floating point operations with only marginal sacrifice in performance.
Under PyTorch (version 1.11.0) framework, random masked Hopfield is not able obtain computational efficiency improvement despite from its searse-structured nature.
duration
duration
25
Hopfield [Ramsauer et 2020]
Random Hopfield 0.1
Hopfield [Hu al., 2023]
250
Random Masked Hopfield
Window Hopfield
Random Masked Hopfield 0.3
Random Feature Hopfield
200
Random Masked Hapfield 0.4
Linear Hopfield
Random Masked Hopfield 0.5
Random Masked Hopficid 0.6
15-
eurazion
curacion
150
Random Masked Hopfield 0.7
Random Masked Hopfield 0.8
Random Hopfield
10
100
1000
2500 3000
500
1000
1500
3000
Memory Size
Memory Size
Figure 6. (LHS:) Comparison of duration (ms) per batch for different Hopfield Models, (RHS:) The scaling behavior of Random Masked Hopfield with different masking ratios. The probability denotes the ratio being masked out. We employ various variants of the Hopf ell layers to process batch of tensors, with a batch size of and hidden dimension of We vary the input memory size (input length). Note that separate the Random Masked Hopfield from other baselines since the sparse matrix operation in PyTorch, still in the beta stage, may not as fully optimized as dense tensor operations.
Finps
Flops
Random Masked Hopfield 0.1
Dense Hopfield [Ramsauer al. 2020]
Random Masked Hopfield 0,2
Sparse Hopfield [Hu et ell, 2023]
Random Masked Hopfield 0.3
ndowww Hopfield
Random Masked Hopfield
Feature Hopfield
Random Masked Hopfield 0,5
0.8
Linear Hopfield
Random Masked Hopfieed 0.6
Random Masked Hopfield 0.7
Random Masked Hosfield
Flops
Random Masked Hopfield 0.9
2
0.4
0.2
0.0
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Memory Size
Memory Size
Figure 7. (LHS:) The FLOPs comparison for Random Masked Hopfield with different probabilities is depicted. The lines for Dense and Sparse Hopfield are overlapped, are the lines Random Feature Hopfield and Linear Hopfield. (RHS:) The FLOPs comparison across different Hopfield Models is shown. We employ the same settings as in the duration figure. Note that the fveore package may count sparse matrix operations as normal floating point operations, which why we might not see difference.
37

---

Nonparametric Modern Hopfield Models
Implementation Details. In this section, we exclusively evaluate the computational efficiency of different Hopfield models with respect varying input lengths using the Hopfield layer. We report the average duration time per batch, shown in Figure 6, and the FLOPs concerning different input lengths (memory sizes), as depicted in Figure 7. It's notable that different code implementation methods could potentially affect computational efficiency. We use randomized batched tensor as input x, where x E memmmrr sizex16 and the batch size is 4 For Random Feature Hopfield and Linear Hopfield, we adhere to the Performer implementation', while for Window Hopfield, we follow the Longformer implementation" 10 For Random Masked Hopfield, we utilize the torch. sparse. sampled addmm"! feature, and for other baselines, we employ standard PyTorch built-in functions for implementation. We report the average forward pass time over 10 runs, alongside the FLOPs, with both metries evaluated different input lengths. FLOPPs are calculated using the fvcore package 2 Note that most publicly available packages for FLOPs profiling are either under development or in beta, hence calculation errors are anticipated. Additionally, the torch. sparse package is also in beta, implying its performance may not be fully optimized, especially regarding FLOPs calculation and operation overhead.
Discussion. Note that, by nature, both Dense and Sparse Hopfield exhibit the same FLOPs. Moreover, it is observed that Random Feature Hopfield and Linear Hopfield also share the same FLOPs, as the only distinction between them lies the kernel function. Regarding Window Hopfield, its FLOPs fall between, demonstrating notable efficiency compared to both Dense and Sparse Hopfield. In terms of duration time per batch, Sparse Hopfield appears slightly faster than its dense counterpart, likely due to the additional zeros generated by sparsemax. Window Hopfield, on the other hand, showcases significant reduction in duration compared to Sparse Hopfield. Lastly, it is noted that the processing time for both Random Feature Hopfield and Linear Hopfield converges as the memory size increases.
"approximately (4 4 16 x memory size) bytes
http:///wwnnaaaoooooemeeeeeeceed addmm.hmmHHorch.ppanee. addmm httpp//ggggia..ooomcccccoocceees
38


Reviews:
[{"review_id": "Un7LI0HHMO", "reviewer": "Reviewer_DjFS", "summary": "In this work, the authors replace the energy minimization step by support vector regression which trains on the pairs of a pattern and its perturbed version, regressing the true pattern, in terms of perturbed ones. They also provide a sparse version which uses a subset of patterns.  In addition, they perform synthetic and realistic memory retrieval and supervised learning tasks with this model.\n\n## Update after rebuttal\n\nI requested that the authors should clarify what they mean by the sparse model inheriting the appealing theoretical properties of the dense model so that the reader does not get a false impression, which they agreed to. I have upgraded the score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The revised draft is in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/81sygn18f4ma3ridm1xlf/AHFYlvzMMlYZnNRhBN9U8mw?rlkey=e1tvpqs6v83kx2rspfvmfgswh&st=z9iuk4iu&dl=0).  All modifications are marked in BLUE color. Thanks!\n\n---\n\n### **Reviewer’s Comment (Claims and Evidence 1)**\n> *In the abstract, they claim the sparse model **inherits the appealing theoretical properties** of its dense analogue...Results like Corollary 4.11 showing the error of sparse recovery being lower than dense recovery **only applies to patterns kept in the sparse model**. Target patterns which are masked out may not get properly recovered. This seems to be borne out by the empirical results.*\n\n---\n\n### **Authors’ Response**  \n\n**When we say the sparse model “inherits” the theoretical properties of the dense version, we mean it retains the key Hopfield guarantees** (e.g., alignment with transformer attention, rapid convergence to fixed-point memories, exponential storage capacity). Our results show that introducing sparsity does not break these fundamental properties.\n\nYes, performance can degrade when target patterns are masked out. **This reflects a general accuracy–efficiency tradeoff** (see lines 421–431). However, our numerical results (Fig. 4, Sec. G.5) show that, in practice, the degradation is mild: sparse masking achieves decent accuracy while significantly improving efficiency (similar to how linear/random feature approximations to softmax perform in other contexts).\n\n---\n\n### **Reviewer’s Comment (Theoretical Claims)**\n> *The theoretical claims seem correct but they are of limited use if the sparse model is only good for a subset of patterns. Am I missing something?*\n\n---\n\n### **Authors’ Response**  \n**We do NOT claim the sparse model always outperforms the dense model in all situations.** Instead, we specify **conditions** where sparse retrieval can yield better accuracy (e.g., when many irrelevant memories introduce noise). Under those conditions, our empirical results (e.g., Fig. 2, p. 30) support the theoretical claim https://imgur.com/a/JjxZwxv.\n\n---\n\n### **Reviewer’s Comment (Claims And Evidence 2)**\n> *From what I can understand, the sparse Hopfield model in Hu et al. (2023) is a very different thing where the regularizer is chosen so that attention is sparse. Sparsity by random masking is not the same thing. Thus, their claims of sparsity-induced advantages ring a bit hollow to me.*\n\n---\n\n### **Authors’ Response: Sparse Hopfield Model – Sparsity Masking vs. Regularization**\nWe appreciate the comparison. To clarify, our model uses **sparse masking** (Definition 3.3), with random masking as one special case (Example 1 in line 275). In **Remark 4.1 (line 368)**, we compared our approach to [Hu et al. 2023]. Here is a brief overview:\n\n- **Different Sparsity Mechanisms**  \n  - **Ours:** Data-***independent***, predefined masking for efficiency.  \n  - **Hu et al. 2023:** Data-***dependent***, regularizer-based sparsity (but still with $O(n^2)$ complexity like [Ramsauer et al. 2020]).\n\n- **Focus on Analysis**  \n  We study how controlling the sparsity dimension $k$ affects retrieval accuracy (Theorem 4.1). Hu et al. (2023) do not provide that level of interpretability.\n\n- **No Claim of Superiority**  \n  We do not assert that random/sparse masking categorically outperforms regularizer-based methods; we emphasize their **analytical tractability** and **efficiency**.\n\n---\n\n### **Reviewer’s Comment (Questions For Authors)**\n> *Why is $\\Phi$ defined in Lemma 3.1 to weigh $\\xi_{\\mu}$ by the attention between $x$ and $\\xi_{\\mu} + \\delta \\xi_{\\mu}$, rather than $\\xi_{\\mu}$ itself (Eq. 3.5)?*\n\n---\n\n### **Authors’ Response**\n1. **SVR-Based Nonparametric Interpretation for MHM** (lines 172–202; 159–169; Appendix C.1)  \n   - We cast Hopfield retrieval as a soft-margin SVR.\n   - Each stored pattern $\\xi_\\mu$ is paired with $\\xi_\\mu + \\delta\\xi_\\mu$.\n   - This enforces the SVR model to map any noisy query near $\\xi_\\mu$ back to $\\xi_\\mu$.\n   -  Equation (3.2) enforces $\\epsilon$-retrieval (Definition 3.2).\n\n\n2. **Aligning with Basin of Attraction (lines 258–274) and $\\epsilon$-Retrieval**\n   - Hopfield models attract queries near $\\xi_{\\mu}$ back to $\\xi_{\\mu}$.\n   - $\\xi_{\\mu} + \\delta\\xi_{\\mu}$ defines a local basin around each pattern.\n   - The SVR constraints ensure $\\lvert T_{\\mathrm{SVR},\\Phi}(x) - \\xi_{\\mu}\\rvert \\le \\epsilon$ within that basin, enabling **noise-robust retrieval**  (from partial or corrupted queries i.e., Definition 3.2.)\n\n3. **Avoiding Trivial SVR Solutions**\n   - Using $(\\xi_{\\mu}, \\xi_{\\mu})$ only would let the system learn the **identity** (no actual denoising).  \n   - Including $\\delta\\xi_{\\mu}$ forces **true associative memory** behavior.\n\nOverall, using $\\xi_{\\mu} + \\delta\\xi_{\\mu}$ ensures the nonparametric retrieval aligns with Hopfield’s attractor dynamics, supports noise-robust memory retrieval, and prevents trivial SVR solutions. \n\n---\n\nThank you for the review. Happy to discuss more :)", "claims_and_evidence": "They claim they propose a nonparametric framework for deep learning compatible modern Hopfield network. Turning approximate retrieval into a regression problem is interesting, but seeing approximate retrieval as denoising makes sense. They say they introduce the first efficient sparse modern Hopfield model with sub-quadratic complexity, which is true. In the abstract, they claim the sparse model inherits the appealing theoretical properties of its dense analog. This is where I begin to get confused. Results like Corollary 4.11 showing the error of sparse recovery being lower than dense recovery only apply to patterns kept in the sparse model. Target patterns that are masked out may not get properly recovered. This seems to be borne out by the empirical results. From what I can understand the sparse Hopfield model in Hu et al 2023 is a very different thing where the regularizer is chosen so that attention is sparse. Sparsity by random masking is not the same thing. Thus their claims of sparsity-induced advantages (the topic of many of their theoretical results) ring a bit hollow to me.\n\nAfter the discussion with the authors, I feel they should clarify this point in the main text.", "methods_and_evaluation_criteria": "The benchmarks and methods are fine.", "theoretical_claims": "The theoretical claims seem correct but they are of limited use, if sparse model only is good for subset of patterns. Am I missing something?", "experimental_designs_or_analyses": "They are sound.", "relation_to_broader_scientific_literature": "Replacing the energy model by a trainable support vector regression is an interesting idea with the basis functions inspired by softmax attention. Perhaps we do not need the crutch of Hopfield models anymore?", "essential_references_not_discussed": "I am not aware of any.", "other_strengths_and_weaknesses": "No comments.", "other_comments_or_suggestions": "There are a few typos, like page 4: $\\mathcal T_\\Xi(x):\\mathbb{R}^d\\to\\mathbb{R}^d$.", "questions_for_authors": "I hope I am misunderstanding something. I keep feeling the general statements made about sparse Hopfield model defined here are too good to be true, as explained above.\n\nAlso why is $\\Phi$ defined in Lemma 3.1 so that we weigh $\\xi_mu$ with the attention between $x$ and $\\xi_mu+\\delta \\xi_mu$, rather than with $\\xi_mu$ (Eq. 3.5)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4D8K6RjlfC", "reviewer": "Reviewer_zGHJ", "summary": "This work leverages the concept of soft-margin Support Vector Regression (SVR) to reformulate modern Hopfield models as a non-parametric regression task, where a noisy target pattern is mapped to a reconstructed target pattern using Support Vector Machines (SVMs). By applying the Lagrange multiplier method, the problem is transformed by incorporating boundary conditions, ultimately deriving the mapping function $T_{SVR,\\Phi}$, which serves as the update rule for the model.\n\nWithin this unified framework, the authors successfully reproduce the standard dense modern Hopfield model while also proposing a generalized Sparse-Structured Modern Hopfield Model. This new formulation significantly reduces computational complexity and introduces three specific efficient variants: Random Masked Modern Hopfield Model, Efficient Modern Hopfield Model, Top-K Modern Hopfield Model.\n\nFurthermore, the authors conduct a rigorous theoretical analysis under this framework, leading to more precise sparsity-dependent retrieval error bounds and proving the fixed-point convergence of the sparse-structured Hopfield model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer’s Comment (Claims and Evidence & Experimental Designs or Analyses)\n> **Concerns**  \n> - Claim2: The theoretical results (noise robustness, tighter retrieval error bounds) do not appear numerically verified in detail.  \n> - Claim3: The experimental section doesn’t explicitly connect to the theory.\n\n> The paper lacks explicit discussion tying the figures/tables in Appendix G.1–G.4 to the theoretical claims.  \n> Empirical validation of the theoretical retrieval error bounds is missing.\n\n### **Authors’ Response**\n\n**In response to Concerns of Claim 2**, numerical verifications of the retrieval error bounds appear in **Sec. G.1** and are summarized in **Fig. 2** (pages 29–30). \n\n\n**In response to Concerns of Claim 3,** we believe there might be an oversight. In the “Results” paragraphs, as well as the captions of **Figs. 2 and 4**, we explicitly linked our numerical findings to their underlying theoretical claims.\n\nWe feel that these “Results” paragraphs and captions are sufficiently informative (also check our concluding remarks, line 411-420, right). \n\nWe apologize for any confusion caused. We have revised the draft with better clarity  (e.g., https://imgur.com/a/OS2q77m and https://imgur.com/a/JjxZwxv and https://imgur.com/a/50mFsX5.)\n\n---\n\n## Reviewer’s Comment (Relation to Broader Scientific Literature)\n> Certain properties, such as tighter error bounds and stronger noise robustness, might be seen as natural extensions of classical Hopfield networks. What’s truly novel here?\n\n### **Authors’ Response**\n1. **Nonparametric Framework**: Prior works do not formulate Hopfield retrieval as a **soft-margin SVR**, which is our key novelty. By bridging Hopfield networks with regression theory, we gain new insights and design flexibility (e.g., kernelized or masked retrieval).  \n2. **Sub-Quadratic Sparse Hopfield**: We extend the dense model to a *computationally efficient* variant while retaining classical benefits (exponential capacity, fixed-point convergence).  \n3. **Sparsity Analysis**: Our results detail *when* sparse retrieval can outperform dense retrieval, complementing prior sparse Hopfield analyses (Hu et al., 2023).\n\n---\n\n## Reviewer’s Comment (Essential References Not Discussed)\n> ... Amari (1972) for earlier foundational work...\n\n### **Authors’ Response**\nThanks for the suggestion. We had updated the draft accordingly (https://imgur.com/a/IYMKyu3).\n\n---\n\n## Reviewer’s Comment (Questions for Authors)\n> 1. Are the theoretical retrieval error bounds actually verified numerically?  \n> 2. How does this approach compare to Hu et al. (2023), which reportedly achieves better capacity and robustness in some cases?  \n> 3. Could you clarify the main takeaways of the figures/tables in Appendix G.1–G.4?  \n> 4. Are these “tighter” bounds and “stronger noise robustness” truly novel beyond standard Hopfield extensions?\n\n### **Authors’ Response**\n\n1. **Numerical Verification (Q1)**  \n   - Yes. **Sec. G.1 (Fig. 2)** (pp. 29–30) shows empirical vs. predicted retrieval error for varied sparsity levels, confirming Theorem 4.1. Please also see https://imgur.com/a/JjxZwxv. We can add further numeric comparisons if requested.\n\n2. **Comparison with Hu et al. (2023) (Q2)**  \n   - **Fundamentally Different Mechanisms**: Hu et al. rely on a data-**dependent** entropic regularizer (still $O(n^2)$). Our approach imposes data-**independent** masks for **sub-quadratic** retrieval.  \n   - **Focus on Efficiency**: We do **not** claim consistent accuracy superiority; rather, we stress efficiency (Thm 3.2).  \n   - **Extended Theory**: Our SVR-based approach and explicit sparsity scaling (Theorem 4.1, Remark 3.6) broaden the theory beyond Hu et al.’s entropic analysis.\n\n3. **Main Takeaways in Appendix G (Q3)**  \n   - **Sec. G.1–G.4**: We evaluate different sparse strategies (top-$k$, random masks, etc.) under noise and large-scale tasks. Each figure corresponds to Thm 4.1 or Coro 4.11 or Prop 4.1, showing that careful sparsity can match or surpass dense retrieval in sparse regimes.\n\n4. **Novel Theoretical Insights (Q4)**  \n    - We recast retrieval as a **soft-margin SVR** problem, yielding **margin-based error bounds** and **stronger noise-robustness** than purely energy-based or entropic methods (e.g., Hu et al., 2023).  \n    - This framework supports a **diverse range of sparsity schemes**—top-$k$, random masking, or learned patterns—yet still guarantees **fixed-point convergence** and **exponential capacity**.  \n    - **Explicit sparsity-dependent bounds** show how selectively ignoring irrelevant patterns can *improve* retrieval, providing concrete criteria for balancing accuracy and efficiency.  \n    - Consequently, these “tighter” bounds and “stronger” robustness are not minor tweaks to classical Hopfield models; rather, they stem from the **SVR-based margin analysis** and support **sub-quadratic** implementations for large-scale memory retrieval.\n\n---\n\nThank you for the review. Happy to discuss more :)", "claims_and_evidence": "The submission makes three main claims, each of which is examined below regarding its supporting evidence and potential concerns.\n\nClaim #1: The paper provides a nonparametric framework and sub-quadratic complexity\n\n* Supported:\nThe paper reformulates modern Hopfield models using soft-margin SVR, demonstrating a nonparametric regression-based approach.\nChapter 3 provides a formal derivation of the Sparse-Structured Modern Hopfield Model, proving its reduced computational complexity. Appendix G.5 provides empirical FLOPs and runtime comparisons.\n\n* Concerns:\nThe computational efficiency improvements rely on asymptotic analysis, but practical benefits are not always consistent across implementations (e.g., PyTorch’s inefficiency with random masked Hopfield).\n\nClaim #2: The paper provides a formal characterization of retrieval error bounds, noise robustness, and memory capacity.\n\n* Supported:\nTighter retrieval error bound (Corollary 4.1.1, 4.1.2)\nStronger noise robustness (Remark 4.2)\nExponential capacity scaling (Lemma 4.1, Proposition 4.1)\n\n* Concerns:\nThese results seem natural extensions of classical Hopfield models, raising the question of whether they provide fundamentally new insights.\nTheoretical error bounds are not numerically validated—while the framework is well-formulated, empirical verification of retrieval error predictions is missing.\n\nClaim #3: The framework bridges modern Hopfield models with attention mechanisms (e.g., BigBird, Longformer, Linear Attention).\n\n* Supported:\nThe theoretical formulation indeed connects the proposed models to various attention mechanisms (such as BigBird, Longformer, Linear Attention, and Kernelized Attention).\nAppendix G contains numerical experiments, showing performance metrics such as loss curves and accuracy trends.\n\n* Concerns:\nThe experimental section lacks clear explanations of what conclusions should be drawn.\nWeak connection between experiments and theoretical findings.", "methods_and_evaluation_criteria": "The main body of the paper primarily focuses on theoretical derivations, while experiments are mainly presented in the appendix. Each experiment includes a corresponding description of the models and parameters used, ensuring clarity in implementation details.\nThe concerns regarding the experimental validation have already been discussed in the previous question.", "theoretical_claims": "I carefully reviewed the theoretical analysis presented in the main text and found the derivations to be well-structured and logically sound. Additionally, I briefly browsed the proofs in appendix C and D and did not notice any apparent issues.", "experimental_designs_or_analyses": "I reviewed the experimental design and analyses, particularly those presented in Appendix G. The experiments are well-structured, with clear descriptions of the models, parameters, and evaluation metrics such as ACC, Loss, FLOPs, and computational time.\n\nThe FLOPs and runtime comparisons in Appendix G.5 provide empirical validation of the efficiency of different sparse Hopfield variants. The results confirm that Random Feature Hopfield, Linear Hopfield, and Window Hopfield achieve reduced FLOPs with minimal performance loss, supporting the claim of computational efficiency.\n\nHowever, some concerns remain:\n\n* Empirical validation of theoretical retrieval error bounds is missing. While the theoretical framework is rigorous, numerical verification comparing predicted retrieval errors with actual retrieval performance would further strengthen the findings.\n* Lack of clear interpretation of experimental results. In Appendix G.1–G.4, the figures and tables present results, but there is no explicit discussion of how these findings support the theoretical claims. A clearer explanation of the key takeaways would improve the clarity and impact of the experimental section.", "relation_to_broader_scientific_literature": "This paper introduces a novel approach by leveraging soft-margin SVR to reformulate modern Hopfield models as a non-parametric regression task. To the best of my knowledge, this specific formulation has not been explored in prior work, making it a notable contribution to the field.\n\nBuilding on this framework, the paper proposes a generalized Sparse-Structured Modern Hopfield Model, which aims to enhance computational efficiency and scalability. However, based on the experimental results in the appendix, the proposed model does not consistently outperform existing methods. Specifically, Hu et al., 2023 appears to achieve better capacity and robustness in many cases.\n\nFurthermore, while the paper derives sparsity-dependent retrieval error bounds and proves fixed-point convergence, some of the additional conclusions—such as tighter retrieval error bounds and stronger noise robustness—are relatively intuitive extensions of classical Hopfield networks. These properties are already well understood in the standard Hopfield model, raising the question of whether the new framework offers fundamentally novel insights beyond a reformulation of known results.", "essential_references_not_discussed": "When discussing Hopfield networks, please also reference Amari’s earlier work (e.g., Amari, 1972), which laid a more detailed mathematical foundation for associative memory models and preceded Hopfield’s 1982 formulation.", "other_strengths_and_weaknesses": "One of the major strengths of this paper is its clear and well-structured presentation. Despite being a theoretical work, the explanations are concise, logically organized, and easy to follow, making the complex mathematical formulations accessible. The clarity of writing ensures that the theoretical contributions are well-communicated without unnecessary complexity.", "other_comments_or_suggestions": "On line 100, right side, the phrase \"We also verify their efficacy through thorough numerical\" is repetitive.", "questions_for_authors": "1. The paper provides rigorous theoretical retrieval error bounds, but there is no numerical verification comparing predicted vs. actual retrieval performance. Could you provide empirical validation to confirm that the theoretical bounds accurately reflect practical performance?\n2. The appendix suggests that Hu et al., 2023 achieves better capacity and robustness in many cases. What are the key advantages of the proposed model over Hu et al., 2023, beyond computational complexity?\n3. The figures and tables in Appendix G.1–G.4 lack clear explanations regarding how they support the theoretical claims. Could you provide a more explicit discussion of the key takeaways from these results?\n4. Some conclusions, such as tighter retrieval error bounds and stronger noise robustness, seem to be natural extensions of classical Hopfield models. What new theoretical insights does your framework provide beyond these expected properties?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EEv8Sjo8NK", "reviewer": "Reviewer_v1Bp", "summary": "This is a theoretical paper that introduces a non-parametric interpretation of Hopfield Nets. The proposed method uses SVR to learn a parameter matrix $W$ mapping from feature space to data space. Additionally, rather than focusing on \"memorizing data\" as the original Hopfield Nets do, the authors propose to frame the AM task as a map from queries to memories, where the queries can be considered corrupted versions of the memories (auto-association), or units that are distinct from the keys (hetero-association).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. We have revised our paper to address each concern in detail.\n\nThe revised draft is in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/81sygn18f4ma3ridm1xlf/AHFYlvzMMlYZnNRhBN9U8mw?rlkey=e1tvpqs6v83kx2rspfvmfgswh&st=z9iuk4iu&dl=0).  All modifications are marked in BLUE color.\n\n---\n\n### 1. Empirical Clarity and Figure Explanations\n\n> **Comment:** The reviewer found our experimental setup unclear regarding retrieval steps, masking strategy, and theoretical connections. They also requested stronger links between figures and the underlying theory.\n\n**Response:**\n\n1. **One-Step Retrieval:** We now explicitly state that *all experiments use a single feed-forward (one-step) retrieval* without iterative updates: https://imgur.com/a/lAJLKR6\n2. **Masking Strategy:** The masking is done on similarity score level following random masked attention. It sometimes removes the target memory if done randomly (e.g., Random Masking). This is what the statement in Fig2 caption is about.  In contrast, top-$k$ masking retains the most relevant entries and preserves the correct memory in the set. \n3. **Linking Theory to Experiments:** We have revised Appendix G.1 and G.2 to include direct references to the theorems and remarks in the main text:\n   - **Figures 2 & 3 (Thm 4.1, Prop 4.1, Coro 4.1.1):** These results highlight memory capacity limits and robustness under sparsity or noise. \n   - **Figure 2 (Remark 4.2):** Explains how different noise levels affect retrieval. \n   - **Figure 4:** Demonstrates quick convergence and stable performance under $\\epsilon$-sparse retrieval. We note that random masking can remove the target memory, violating the $\\mu \\in \\mathcal{M}$ assumption (Theorem 3.2).\n\nIn the revised draft, we have updated the figure captions and added clarifications such that readers can immediately see the theoretical motivations for each experiment.\n\nExamples of these changes appear in the updated Appendix G.1 (\"Results\") and G.2 (\"Captions\" for Figures 2–4). E.g., https://imgur.com/a/OS2q77m and https://imgur.com/a/JjxZwxv and https://imgur.com/a/50mFsX5.\n\n---\n\n### 2. Focus on Hopfield Memory Models & Novelty\n\n> **Comment:** The reviewer questions the distinction between Hopfield networks and Hopfield memory models, and the novelty of our approach.\n\n**Response:**  We emphasize that **Hopfield memory models** and **Hopfield networks/layers** are conceptually distinct:\n\n- **Hopfield memory model**: A fixed content-addressable **memory model** with no training; retrieval is based on similarity to stored patterns. This is our focus.  \n- **Hopfield network (e.g., Hopfield Layer)**: A **neural layer** integrated into deep learning, trained with backprop. This builds on the “Dense Associative Memory–Transformer Attention” correspondence (Ramsauer et al. 2021), later generalized by Hu et al. (2023) and Wu et al. (2024).\n\nOur work **extends the prior modern Hopfield memory model** by framing it as a nonparametric regression problem. This allows efficient variants (Sec 3–4) and connects it to attention mechanisms (e.g., Performer) under a rigorous, unified theory. To our knowledge, this unification has not been done before and constitutes a novel contribution.\n\nWe have added a remark regarding above points https://imgur.com/a/E93HSM5. In the final version, we promise to move this remark to Sec 2 (background) if space allows. \n\n---\n\n### 3. Terminology & Background\n\n> **Comment:** The reviewer recommended refining terminology and providing enough background to ensure clarity.\n\n**Response:**  \n- **Cite Hoover et al. 2024:** https://imgur.com/a/Ix7cp7n\n- **Question: Masked Target?** Yes, if a pattern is masked, it will not be memorized by the SVR. This allows us use different masking strategies to accompany different sparsity patterns in sec 3.2 & E. \n- **Nonparametric:** Yes, it does describe NPH. Despite having a matrix $W$ in the **primal** formulation of SVR, the model parameter is not fixed. This is more obvious in the **dual** form of SVR. In sec 3, we cast the retrieval dynamics $\\mathcal{T}$ as SVR. The number of memories is proportional to to the number of support vectors, and hence proportional to model size. Thus the model size is not fixed. This is nonparametric (Remark 3.4). We had revised the Appendix C to strengthen this: https://imgur.com/a/E3Fnic5.\n- **Background in Main Text:** We agree that the related work section can provide more background to main text . When finalizing the paper, we will try our best to integrate this background if space allows.  \n- **“Memory-Enhanced Functionalities” (L415, right):** We have added a remark to clarify this in Appendix F (https://imgur.com/a/E93HSM5).  In the final version, we promise to move this remark to Sec 2 (background) if space allows. This remark ties them to the benefits of content-addressable memory systems in real-world applications.\n\n---\n\nThank you for the review. Happy to discuss more :)", "claims_and_evidence": "The authors propose a novel \"Nonparametric Hopfield\" (NPH) Models, and claim this technique \n\n1. Is compatible with modern Deep Learning architectures (this is not novel as the [Hopfield Layers](https://github.com/ml-jku/hopfield-layers) integrates similar ideas into modern architectures).\n2. Comes with with strong (sub-quadratic) efficiency advantages (this is not novel since sub-quadratic complexity is also achieved by e.g., Performer) due to sparsity and use of basis functions. \n3. The non-parametric Modern Hopfield framework can be used to formulate many existing methods.\n\nGiven that none of the above claimed contributions are novel on their own, it seems that the fundamental contribution and characterization of their NPH framework itself. Evidence for these contributions are proofs in the appendix.", "methods_and_evaluation_criteria": "See \"Experimental Designs & Analyses\"", "theoretical_claims": "1. SVR can be used to train a matrix $\\mathbf{W}$ that stores (query, memory) pairs\n2. This framework can be extended to several modern Hopfield Networks (e.g., Ramsauer et al. Choromonski et al., Beltagy et al.) as an extension.\n\nI did not check the proofs for these claims in the appendix. The definitions and methods are sound (see \"questions\" for my confusion about sparsity).", "experimental_designs_or_analyses": "The paper offers no experimental validation of their claims in the main paper. However, Appendix G offers a study on memory retrieval and learning. Given my time and because this section has been relegated to the appendix, I will give only brief overview and comments:\n\nI have a difficult time interpreting the results as the plots are noisy (many overlaid lines. Unclear what the takeaway message is for each figure). I have several questions on this section and I believe it can be improved substantially:\n\n1. Are these experiments tested in the 1-step retrieval paradigm or is recurrence involved?\n2. Fig 2 caption says that the masking did not work well, but I don't understand the justification of \"violating the $\\mu \\in \\mathcal{M}$ assumption. How is \"masking\" even applied? Via zero-ing out pixels? Replacing pixels with gaussian noise? Adding gaussian noise? Surely there must be a \"masking\" choice that keeps the correct pattern in the memory set.\n3. How can I understand these results in light of the theoretical guarantees of the paper? Would be nice to see a reference line for what the theory would guarantee, or if the theory does not specify a bound for a particular experiment to explicitly mention that. Additionally, some of the plots are really noisy and its hard to tell what your method is.\n4. Would be nice to see some qualitative pictures to understand what the experiments test.\n\nI will continue my review as though this is a purely theoretical paper since these results do not add additional insights or clearly confirm the proposed theory.", "relation_to_broader_scientific_literature": "Specifically, NPH models store memories in a matrix that maps features back to the data space. This is a very general framework with which we can understand many recent optimizations of the MHN, esp. in the attention operation.", "essential_references_not_discussed": "It is worth mentioning [Hoover et. al 2024](https://arxiv.org/abs/2410.24153) for its use of random features in a *recurrent* framework for Associative Memory that converges to actual (not just generalized) fixed points.", "other_strengths_and_weaknesses": "Strengths:\n- **A general, novel approach**. The generalizability of this framework to existing methods of single-step associative memories is compelling. \n\nWeaknesses:\n- **Unclear empirical results**. It is not clear how the empirical results included in the appendix can be used to make any statements as to the efficacy and potential of this approach. Additionally, because the error in this method depends on how close together the memories are, it is unclear how strong the bounds are on large datasets.\n- **Missing necessary background**. The introduction is difficult to read without strong background on papers Hu et al. and Wu et al. published in the past two years. The background explains this paper's relevance to attention (P3), describing the one-step update rule of Ramsauer et al. and the desiderata for learning an associative memory (i.e., must satisfy a monotonically decreasing energy function (T1) that converges to fixed points (T2)), but it does not elucidate the claims made in the introduction regarding sparsity (P2) and efficiency (P1). It would be nice if the related work section in the appendix (which I hope has a chance to make it into the main paper in some form when accepted) could serve as all the necessary background needed to understand this paper's contributions.", "other_comments_or_suggestions": "Please take my comments with a grain of salt since there were several aspects of this work that I did not understand.\n\n1. The paper would benefit from a formal definition/justification of what \"non-parametric\" means in the context of NPH models. Isn't the matrix $\\mathbf{W}$ the parameter matrix? For example, does the definition used to describe the SVR learning task work for understanding the NPH models?:\n    > one that does not assume a specific functional form, and is flexible in the number of parameters. [L256]\n\n2. The statement \"competitive alternatives with memory-enhanced functionalities\" [L415 right col] requires some experimental justification which should be discussed in the main paper.\n\n3. For clarity of exposition, I would recommend the term \"query\" be used instead of \"contaminated memory patterns\" $\\xi_\\mu + \\delta \\xi_\\mu$ (as currently described in e.g., Lemma 3.1 and [L200 right col]), as used in the beginning of sec 3.1 and again in 4.1. I believe this will help sell the message that this framework applies to both auto- and hetero- associative memories in the important introduction to the method.", "questions_for_authors": "1. I don't understand what's special about the Sparse-Structured formulation of NPH models. Definition 3.3 is simply the SVR optimization problem of Eq. (3.2) on a subset of the data points. If I understand correctly, you will never learn those data points which are not masked?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "nQ22LJwvkf", "reviewer": "Reviewer_g2qn", "summary": "This work proposes a non-parametric procedure to construct retrieval dynamics maps for modern Hopfield networks, based on a supervised support vector regression-like problem where contaminated patterns serve as a training data. The proposed procedure is shown to recover the standard dense retrieval dynamics from Ramsauer et al. (2020) as a particular choice of kernel. Building upon this procedure, the authors show that more efficient dynamics, based on applying a mask to the pattern samples, lead to sparse map with sub-quadratic complexity, while preserving desirable properties such as one-step retrieval of the patterns. Retrieval error upper-bounds for these Sparse-Structured models is also are also discussed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We thank the reviewer for the valuable comments. We have revised our paper to address each point.\n\nThe revised draft is in this [anonymous Dropbox folder](https://www.dropbox.com/scl/fo/81sygn18f4ma3ridm1xlf/AHFYlvzMMlYZnNRhBN9U8mw?rlkey=e1tvpqs6v83kx2rspfvmfgswh&st=z9iuk4iu&dl=0).  All modifications are marked in BLUE color. Thanks!\n\n---\n\n### **1. Critical Discussion on Limitations (Newly Added Appendix B.2 https://imgur.com/a/R4PLZMs)**\n\n  - **Fixed Feature Mapping (Reviewer Concern: Realistic Embedding Scenarios)**  \n    - We base our theoretical analysis on a fixed feature mapping.  \n    - This choice streamlines convergence proofs but restricts the model if features evolve.  \n    - Yet, **this condition is easy to meet. Moreover, in practice, static memory model do not have evolving feature map.**\n\n  - **Robustness to Random Masking (Reviewer Concern: Potential Retrieval Failures)**  \n    - Our retrieval guarantee assumes the correct memory item remains in the support set.  \n    - If the random mask excludes it, retrieval may fail.  \n    - **While this assumption is restrictive, it is necessary for tractable analysis.**\n\n  - **Well-Separated Patterns for Exponential Capacity (Reviewer Concern: Data Correlations)**  \n    - Our exponential capacity analysis requires patterns to be sufficiently distinct.  \n    - This simplifies theoretical arguments, yet real datasets often have correlated patterns.  \n    - Yet, **this setup is standard for characterizing capacity in literature** (Krotov & Hopfield, 2016; Demircigil et al., 2017; Ramsauer et al., 2020; Iatropoulos et al., 2022; Hu et al., 2023; Wu et al., 2024b; Santos et al., 2024a;b). Moreover, it can be relaxed by optimizing  the Hopfield-energy landscape for larger memory capacity.\n\n  - **Unproven Extensions in Section 5 (Reviewer Concern: Additional Convergence and Capacity Proofs)**  \n    - We outline several model extensions but do not provide full proofs (no convergence and capacity guarantees).  \n\nWe see these points as opportunities for growth. Addressing them will extend our framework to broader settings and make retrieval more robust. We also invite the reviewer to check the newly added Appendix B.2 for detail in our latest revision.\n\n---\n\n### **2. Experiments and Sparsity (Methods And Evaluation Criteria)**  \n   - **Our current experiments already simulate the conditions the theory addresses**, especially regarding sparsity of patterns.\n   - Namely, our experiments already simulate both sparse (MNIST) and denser (CIFAR-10) data. This aligns with our theoretical assumptions.  \n   - **Our current numerical results already justify our theory.** Figure 2 in Appendix G.1 shows sparse models perform better on MNIST. This matches our claim that higher sparsity improves retrieval capacity and reduces error (Proposition 4.1, Theorem 4.1).  \n   - In MIL supervised experiments, sparse models converge faster (Corollary 4.1.1). These results match the modern Hopfield literature (e.g., Hu et al. 2023, Wu et al. 2024, Ramsauer et al. 2020).\n\nWe have revise the draft to emphasize these: \n* Revised G.1 \"Results\": https://imgur.com/a/OS2q77m and \n* Revised Fig 2 \"Caption\": https://imgur.com/a/JjxZwxv\n* Revised G.2 \"Results\", \"Captions\" of Fig 3, Fig 4: https://imgur.com/a/50mFsX5\n\n---\n\n### **3. Technical Clarifications**  \n   - **Monotonic Decrease of $E(x)$ (Condition T1)**  \n  Our convergence analysis assumes that each update decreases (or does not increase) the Hopfield energy $E(x)$. **This is a standard property of energy-based Hopfield models** and guarantees deterministic convergence to a fixed point. It does not apply to stochastic or non-monotonic updates. We emphasize that **the Hopfield memory model serves a different purpose than deep learning models**: here, minimizing energy corresponds to pattern retrieval, not parameter learning. Comparing this condition to stochastic optimization in deep learning (e.g., SGD) is not appropriate—the objectives and dynamics differ fundamentally.\n\n   - **Inequalities Below Eq. (3.2)**: We confirm these are component-wise inequalities. We clarify this in the revised text (https://imgur.com/a/wu2eW9A).\n\n---\n\nWe appreciate the reviewer’s time and attention to details. The revisions should address the concern about missing limitations and clarify our theoretical and experimental contributions. We hope the updated manuscript meets your expectations.", "claims_and_evidence": "The core of the manuscript is to propose a new method for designing retrieval dynamics maps for modern Hopfield networks. A few theoretical properties of the method are discussed, and these claims are supported by rigorous mathematical proofs. Numerical evidence for the advantage of the sparse-structured maps is also discussed in the appendix.\n\nOverall, it is not very clear what are the drawbacks of the proposed method. For someone not familiar with the modern Hopfield literature, it would have been nice to have a broader limitation discussion in the paper.", "methods_and_evaluation_criteria": "The bechmarking presented in the appendix uses standard data sets for retrieval tasks. It would have been nice to have experiments with synthetic data satisfying all the assumptions in the theory, to illustrate tasks for which the sparse method can significantly outperform the dense method.", "theoretical_claims": "I skimmed through the proofs, and it looks sound. It mostly leverages standard results in SVR adapted to the setting of the paper.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "From my reading, the related literature on dense and sparse modern Hopfield networks is well acknowledged. However, I am unfortunately not familiar with this literature.", "essential_references_not_discussed": "Unfortunately I am not familiar with this literature.", "other_strengths_and_weaknesses": "Overall, the paper is well-written and easy to follow. I appreciate the authors introduce modern Hopfield networks, as it helps the unfamiliar reader putting the results in perspective.\n\nThe main weakness is the lack of a critical discussion of the proposed method. All the results are seemingly positive, and a lack of a honest \"limitation\" section describing the main drawbacks raises a red flag for the unfamiliar reader.", "other_comments_or_suggestions": "I strongly encourage the authors to discuss the limitations of their work in perspective with the related literature.", "questions_for_authors": "- In condition (T1), do you really need to monotonically decrease $E(x)$? This is strong, in principle rules out stochastic dynamics rules, e.g. SGD.\n- Below eq. (3.2), are the inequalities applied to vectors component-wise? It would be nice to precise this in the text.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper reformulate modern Hopfield models as a non-parametric regression task tackled with soft-margin Support Vector Regression,  where noisy patterns are mapped to reconstructed target patterns.

The reviewers point out as strengths the originality of the approach on the context of Hopfield networls, as well as its generalizability to existing methods of single-step associative memories. The new formulation significantly reduces computational complexity. The main weaknesses are the limited and unclear empirical results (included only in the appendix) which are not sufficient to prove the effectiveness of the proposed approach. There were also some concerns about what "sparse" means in the context of this work in comparison to prior work on sparse Hopfield networks. The paper would benefit also from improving the background and discussing the limitations of the proposed method. The authors' rebuttal addresses several of these concerns, which makes me lean towards acceptance.
Decision: Accept (poster)