Dense Associative Memory with Epanechnikov Energy
Authors: Benjamin Hoover
Zhaoyang Shi
Krishna Balasubramanian
Dmitry Krotov
Parikshit Ram
Venue: NeurIPS 2025 spotlight ()
Keywords: Associative Memory
Dense Associative Memory
Kernel Density Estimation
Epanechnikov kernel
energy-based models
exponential memory capacity
Hopfield Network
spurious minima
Primary area: general_machine_learning
TL;DR: Changing the DenseAM kernel from the standard Gaussian kernel to the KDE-optimal Epanechnikov kernel results in 1) exponential capacity without the exponential and 2) meaningful, emergent memories.

Abstract: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Uniquely, it introduces abundant additional emergent local minima while preserving perfect pattern recovery --- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.

Dense Associative Memory with Epanechnikov Energy
Benjamin Hoover
Zhaoyang Shi Harvard
Krishnakumar Balasubramanian
IBM Research
UC Davis
Georgia Tech
Dmitry Krotov
Parikshit Ram IBM Research
IBM Research
Abstract
We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-Rel.U (L.SR), inspired by optimal kernel density estima- tion. Unlike the common -eg-suuexxponnniall (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential ca- pacity without requiring exponential separation functions. Moreover, it introduces abundant additional emergent local minima while preserving perfect pattern re- covery characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-like likelihood to LSE-d models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and gener- ative tasks.
1 Associative Memories and Energy Landscapes
Energy-based Associative Memory networks or AMs are models parameterized with M "memories" in d dimensions, =\t {E,, E (RC E [M]]. A popular class of models from this family can be described by an energy function defined on the state vector x E X C Rdd
M Σ
Eg(x: Q
( (o(x)
where 9 Rd Rd is vector operation (such as binarization, (layer) normalization), S Rdd Rd - IR is similarity function (e.g., dot-produs negative Euclidean distance), 3 > 0 denotes the inverse temperature, F R - E is rapidly growing separation function (power, exponential) and Q is monotonic scaling function (logarithm, linear) [1, 2, 3]. With as the sign-function, E,, E (-1,++1) +(x,.,(xxxx'))) x') = (x, x") and F as the quadratic function, and Q as linear function, we recover the classical Hopfield model [4]. The output of an associative memory (AM) corresponds to one of the local minima its energy function. A memory &,, is said to retrieved x & corresponds to such local minimum, with exact retrieval occurring when x f.. The memory capacity the AM is defined as the maximum number M of correctly retrieved memories. For classical AMs, the capacity scales as M (dd). With the introduction of power-law separation functions - that is, F(x)= = x" for > 2 modern Dense Associative Memories (DenseAMs) achieve significantly higher capacity of M O(d)) [].
The use of an exponential separation function combined with a logarithmic scaling function F(x) exp(z) and Q(x) = logr leads to the widely studied log-sum-exp (LSSE) energy func- tion [6. 7, 8], yielding exponential memory capacity M ~ exp(d) [9]. Hicrarchical organization of memories has also been explored in Krotov [10] and Hoover et al. [11]. Given that the gradient
39th Conference on Ncural Information Processing Systems (NcurIPS 2025).

---

LSR preserves memories while creating novel ones. LSE can do only one or the other.
1 novel
0 preserved
novel
preserved
Low /
Critical |
High A
Pres veddvvedie stmmalis
LSR
(ours)
B-03
B-06
LSE
B=13
2
preserved
novel
preserved
Figure 1: LSR energy can create more memories than there are stored patterns under critical regimes of 3. Left: ID LSR vs LSE energy landscape. Note that LSE is never capable of having more local minima than the number of stored patterns. Right: 2D LSR energy landscape, where increasing creates novel local minima where basins intersect. Unsupported regions are shaded gray.
of the LSE energy corresponds softmax over all stored patterns, recent works have proposed sparsified variants to improve scalability. In particular, Hu et al. [12] and dos Santos et al. [13] consider sparsified softmax-based gradients, effectively projecting the full gradient onto reduced support.' Alternatively, Wu et al. [14] propose learning new representations for the memories to increase capacity, while continuing to use the LSE energy in the transformed representation space.
In this work, we consider the following motivating question: can we achieve simultaneous perfect memorization and generalizution in associative memory models? While the exponential separation function enables DenseAMs to achieve high memory capacity, capacity alone not the only desider- atum. In standard supervised learning, it long believed that exactly interpolating or memorizing the training data - achieving zero training loss would harm generalization. However, recent advances, particularly in deep learning, have challenged this belief: models that perfectly fit the training data can still generalize well. Although this phenomenon gained prominence with deep networks, it has earlier roots in kernel methods and boosting [15, 16, 17].
An analogous question arises the context of associative memory (AM) models. Traditionally, AMs focus on storing fixed set of patterns. But from a broader machine learning perspective, the goal extends beyond memorization to include the generation of new, meaningful patterns. Prior work using LSE-type energy functions has shown that generating such novel patterns typically requires sacrificing perfect recall of the original patterns. This trade-off highlights a core tension between memorization and generalization. To address this, we explore alternative separation functions that can preserve exact memorization while enabling the emergence of new patterns - pushing toward models that truly unify memory and generalization.
Our approach also motivated by the well-established connection between the energy and prob- ability density function. An energy function E IRd + IR induces probability density function p : Rd - R2 with p(x)= = exp(- f- Ex))dd Conversely, given a density p, we have an energy E(x) x log p(x). ) the negative log- ligelhood Minimizing the energy corresponds to maximizing the log-likelihood (with respect to the corresponding density). Based on this con- nection, with (--) = log(-), the exp(-E 3)] Σn, F(SS(x, E,)) in eq. (1) (assuming g is identity) is the corresponding (unnormalized) density at x. Assuming that the memories E,- ~ p are sampled from an unknown ground truth density p. the exp(-Fa, 1=53:: E)] is an unnormalized kernel density estimate or KDE of p at x with the kernel F and bandwidth 1/ [18]. Thus, the LSE energy with F F(x) = exp(x:) and 5(x,x') kernel.
1/2||x - x21 corresponds to the KDE of p with the Gaussian
KDE is well studied in nonparametric statistics [18, 19], and various forms of kernels have been explored. The quality of the estimates are well characterized in terms of properties on the kernels; we will elaborate on this in the sequel. While the Gaussian kernel is extremely popular for KDE (much like LSE in AM literature), there are various other kernels which have better estimation abilities than the Gaussian kernel. Among the commonly used kernels, the Epanechnikov kernel
'Sparsified softmax bossdd gradients can be interpreted specific projections of the original gradient.

---

has the most favorable estimation quality (see section 2), In our notation, this corresponds to kernel F(x) max(1
z. +,. ReLU (1+ z), shifted ReLU operation (again with S(x, x'x -1/2|xx - x113). This results in novel energy function that we name log-sum-Rel.U or L.RR (see eq. (3)). Surprisingly, we show that this energy function capable of both exactly memorizing all M original patterns (with M ~ exp(d)) and simultaneously generating new, meaningful emergent memories (see Definition 2). This defies the conventional tradeoff seen in prior AM models - where improving generalization (ability to create emergent local minima) typically requires compromising exact memorization - and reveals that precise memory storage and creative pattern generation are not inherently at odds with one another. To summarize, make the following contributions in this work:
Novel ReI.U-based energy function with exponential memory capacity. We propose a LSR energy function for DenseAM using the popular ReLU activation, built upon the connection be- tween energy functions and densities. In Theorems 1 and 2 respectively, we demonstrate exact retrieval and exponential memory capacity of LSR energy, without the use of exp(-) separation function.
Simultaneous memorization and emergence. We show that this LSR energy has a unique prop- erty of simultaneously being able to exactly retrieve all original memories (training data points) while also creating many emergent memories (additional local minima). The total number of memories of LSR can exceed the number of stored patterss, a property absent with LSE (see Proposition 1). When applied to images, LSR can generate novel and seemingly creative memo- ries that not present in the training dataset.
2 Kernel Density Estimation and the Choice of Kernels
We now provide a brief overview of Kernel Density Estimation (KDE) considering the univari- ate setting for simplicity; similar conclusions also hold in higher dimensions. Given a sam- ple E = {Eu E R. μ E [M]] drawn from an unknown density f. the KDE is defined as (.)()) = (MA) m= ($-Ex) where K(-)) is the kernel function and h > is the bandwidth parameter. The kernel function assumed to satisfy: (i) symmetry (i.e., K (-=z) K(x), for all E R), (ii) positivity (i.e., K K(x) 2 0, for all x E R) and (iii) normalization (i.e., [ (z) dr = 1). Note that for the purpose of KDE, the scale of the kernel function is not unique. That is, for a given b- (-/b), for some > 0. Then, one obtains the same KDE by rescaling the choice of h. Hence, the shape of the kernel function plays a more important role in determining the choice of the kernel. We now introduce two parameters associated with the kernel, μк == x² K (Ix) dr and σк = fe (x) da that correspond to the scale and regularity of the kernel. We will discuss below how the generalization error of KDE depends on the aforementioned parameters.
K()). we can define (-))
The generalization error of [C) is measured by the Mean Integrated Squared Error (MISE), given by MISE(h) = E (fx(E) - d(E)) Assuming that the ground-truth density f(E) is twice continuously differentiable, second-order Taylor expansion gives the leading terms of the MISE(h), which decomposes into squared bias and variance terms: MISE(N) x 12 Se (E) + : see Wand and Jones [18, Section 2.5] for details. This result shows that reducing h decreases bias but increases variance, while increasing h smooths the estimate but introduces bias, highlighting the bias variance trade-off. The optimal mean-square is obtained by minimizing MISE(h) with respect to h. We thus obtain the optimal choice of h and the optimal generalization accuracy as
1/5
5 VUROK (E) 4/5 M
σK 4
h,
and MISE(h.) 4
My2 (C)
respectively. From this, we see that the choice of the kernel in the KDE, controls the generaliza- tion error via the term р/кOк.
Thus, a natural question is to find the choice of kernel K(-) that results in the minimum MISE(h..) ). As discussed above, the scale of the kernel function is non-unique. Hence, the problem boils down to minimizing o (which regularity parameter of the kernel, determining the shape), subjected to μд = 1 (without loss of generality), over the class of normalized, symmetrie, and positive kernels.
3

---

This problem is well-studied (see, for example, [20], [21], Wand and Jones [18, Section 2.7D), and,
as it turns out, the Epanechnikov kernel Kepan ( max x²² 0} ReLU (1 (1- x²) achieves
the optimal generalization error. The quantity, Eff(K) σk/ / Kapa is hence referred to as the efficiency of any kernel with respect to the Epanechnikov kernel. A description of choices for kernel functions and their efficiency relative the Epanechnikov kernel is provided in Appendix
The number of modes of the KDE has also been examined in the literature, mostly when the target is unimodal. Assuming that the target is unimodal, direct consequence of Mammen [22, Theorem IJ on the number of modes of the KDE when d = 1 (also see Geshkovski et al. [23, Theorem I.IJ) is that the number of modes of KDE with a Gaussian kernel with bandwidth h is @(1/Vī); see Geshkovski al. [23, Section 1.2] for extensions to dimension d' > 1.
3 A New Energy Function with Emergent Memory Capabilities
So far, we have seen the relationship between the LSE energy and the KDE, i... expl-Ea(X; E) an unnormalized kernel density estimate with the Gaussian kernel and the bandwidth 1/B, and the optimality of using the Epanechnikov kernel in KDE. Given these observations, we will explore the use of the corresponding shifted ReU. separation function ReLU (1 + x²) in the energy function instead of the widely used exponentiation. Before we state the precise energy functions, we compare and contrast the shapes of these separation functions F(3x) in fig. 2 for varying values of the inverse temperature 3. Note that, as the 3 increases, both these separation functions decay faster. However, as expected, the Shifted-ReLU separation linearly decays and then zeroes out.
E. Based on the discussion on separation functions, our proposed LSR ENERGY (which we also refer as Epanechnikov energy) is given by
Recall that LSE ENERGY is given by E E)
1/ log Em= exp(- ||x
M ReLU (1
B ||x 2
$mili²)
E
xx;
3) = 3
log
6 +
21
(3)
where ||-II describes the Euclidean norm and / is inverse temperature.
The factor 0 in the LSR energy is a small non- negative constant, where an > O ensures that every point in the space has finite (albeit extremely large O(log(I/e)) energy for all values of B. Indeed, with c = 0, defining S, fx E X - < /2/²), it is easy to see that Vx E X S D (x) = oo. This is a result of the finite-ness of the ReLU separation function. Regions of infi- nite energy implies zero probability density, which matches the finite support of the density estimate with the Epanechnikov kernel. Based on the intro- duced LSR energy, we next highlight the following favourable properties; see appendix for the proof' and technical details.
F(Ax) exp(
FI(EX) ReLu(1 (Bx)
1.0
E-1.0
F(gki
$-0.5
0.5
0.0
-2
0
-4
-2
Six.
S'x.x')
Figure 2: Visualizing the separation func-
tions F(3x) = exp(3x) (LSE) and
F(3z) ReLU (1 3r) (LSR) with
5(x, x') for varying values of 3. We focus S(x, 11/2|xx x'
||E, - Evlll be the minimum Euclidean distance between any two memories. Let (A) (x E X ||x - E.lll S A} be basin around the 1nth memory for some basin radius A E rO. Then, with 3 == 2/(r A)2, for any p. E [M] and any input E Sy(A), the output of the DenseAM via LSR energy gradient descent is exactly E,.. implying that all memories Eur H E [M] are retrievable. Furthermore, if the learning rate of the energy gradient descent is set appropriately, then for any l E [M] and any x E (,)),, the memory is exactly retrieved with single energy gradient descent step (single step retrieval).
Theorem 1. Let r
The above result states that, given set of memories, and an appropriately selected 8, there is a distinct basin of attraction Sy (A) around each memory Ê.. and any input x from within that basin exactly retrieves the memory as the output of the DenseAM.
Remark 1. For a finite but appropriately large B, the LSR energy gradient V ERL E) any memory En is exactly zero, implying exact retrieval of the memory. The LSE energy gradient v FELSE E) is only approximately zero, and the retrieved point is approximately equal to an

---

original memory [7, Theorem 3). However. if 8 = oo then the LSE energy gradient is exactly zero at the memory.
The striking phenomenon that observe with LSR energy is that the DenseAM can simultaneously create local energy minima around the original memories as well as additional local minima around points that are not part of the set of original memories; see fig. 1. We formalize this concept below through the notion of global emergence.
Definition 1 (Novel local minima). Consider a DenseAM parameterized with M memories E = {EI...., {. h. X, and equipped with an energy function Ex(x; E) at any state x E X for specific inverse temperature 3 >> For some E > O, we define M, as the (possibly empty) set of novel local minima É E X such that V E &Me.
(a) & is a local energy minimum with E E) and (3:
(b) E is novel with respect to the original memories, that is, min (6-Eu 2E.
Definition 2 (Global emergence). For the DenseAM in Definition and for some
let Ms be the (possibly empty) set of novel local minima. For some 3 C (0, xi). we claim that this system exhibits e-global emergence (i) each original memory Eu [M] is local energy minimum with V EE, 3) 0 and 4² Ey(E,, We term MM as the set of e-globally emergent memories.
(positive definite), and (ii) the Me is non-empty.
The notion of E-global emergence specifically refers to those new patterns that arise after all original memories have been exactly stored. Definition 2 characterizes emergence property of the global energy function at specific inverse temperature, requiring simultaneous exact recovery of all orig- inal memories and the presence of at least one novel local minimum (parameterized with s). It is instructive to start by understanding the above definition for DenseAMs equipped with LSE energy. According to Ramsauer et al. [7], any point x* such that V VELSE [x" 3) = 0 is defined via the soft-
max corresponding to the transformer attention as follows: x Σ 2 softmax(8(x* (E)) and the softmax can be highly peaked if all { are well separated and x* is near a stored pattern E.. If no stored pattern É,, is well separated from the others, then x" is close to global fixed point, which is the arithmetic mean of all the stored patterns. Based on this, we can make the following observations:
Case I: All LSE memories are novel. With a large enough but finite 3, there is a minimum close to each of the original memories. However, each of these local minima will be considered novel local minima as these are distinct from the original memories, thus condition (ii) in Definition will be satisfied. However, then the condition (i) in Definition 2 would not be satisfied.
Case II: No LSE memories are novel. If we do consider the case B = co, then the original memories would exactly be the local energy minima, and condition (i) will be satisfied. But then the set of novel local minima Mc for a strictly positive s would be empty, violating condition (ii). Case III: Novel LSE memories form only when basins merge. For moderate LSE can form novel local minima by merging the basins of attractions of the original memories, thereby giving us non-empty Mr and satisfying condition (ii) in Definition 2. However, condition (i) will be violated as the memories whose basins are merged would no longer be local minima.
Thus, we can make this more formal the following:
Proposition 1. Assume (E are i...d. from any density fully supported on X. Note that they are un linearly independent with probability 1, as otherwise they lie in a lower dimensional space. Then,
for any 0, the LSE energy, SE (:), does not satisfy the E-global emergence in Definition 2.
One can argue that global emergence as in Definition 2 too restrictive; we also want to characterize an individual local minimum as "emergent", or not. Thus we present a relaxed local notion of emergence in the following, noting that LSE does not satisfy this weaker form of emergence either:
Definition (Locally emergent memory). Consider the DenseAM in Definition with a non-empty set of novel local minima Me for a E > O. For any & E MM. let S(E) C E be the minimal non- empty subset of E such that, for each En- E S(E), & is no longer a local minimum of the energy Ep(-E \ { that excludes the memory. Then we define & E Me as a e-localyy emergent memory if there is some original memory &n, E SS() which still is local minimum of the energy
5

---

Eg(::), If every original memory E. E S(E) is a local minimum of Ea(: we call & Ma
E-local strongly emergent memory.
While Definition 2 discusses emergence at a global energy level, Definition 3 characterizes emer- gence locally for each of the novel local minima. This distinction is important as we see emergence as a general property of the energy function that can be driven by a subset of the memories. Defini- tion 2 requires all original memories to be retrievable, while in Definition 3 allow for emergence due to the interaction of stored patterns in system even if not all original memories are retrievable, so long as a critical subset of the original memories are. Global emergence implies the existence of at least one local strongly emergent memory; every globally emergent memory is a local strongly emergent one. However, the existence of local strongly emergent memory does not imply global emergence. As with global emergence, we can see that a DenseAM with LSE energy does not also have locally emergent memories. First note that, for finite 3, all local minima are novel local minima, and the minimal set S(E) for novel local minimum g is the whole set 3 given the infinite support of the exponential function, with none of them being local minimum. For , = 00, the set of novel local minima Me is empty. So in both cases, the required conditions are not satisfied and there are no locally emergent memories with LSE energy.
Note that these novel local minima are different from the well-studied spurious memories or parasitic memories [24, 25]. In classical AMs, spurious memories start appearing when the AM is packed with memories beyond its memory capacity. In contrast, the appearance of emergence (novel local minima) does not seem to be related to whether the DenseAM is over or under capacity we show in fig. I, a locally emergent memory can appear even with just 2 stored patterns of any dimension. Spin-glass states [26] do not occur in either the LSE or LSR energy due to our use of Euclidean similarity over the dot product in both energies.
as
The next result provides explicit characterization of the form of novel memories in LSR energy.
Proposition 2. Consider the LSR energy in eq. (3). For any x E interior(X), letting B(x) - {n E [MT ||x - Eull S /2/B}, there is a local minima of the LSR energy which is given by [B(O]
Σ ExE E..
Note that when [B(x) 1, the local minima in Proposition 2 is exactly the stored memory
( With [B(x) > 1, it is not equal to any of the original memories {En. E [M]] (with probability 1). The region {x E X [xx) > 1} C X' is precisely characterized as (Une[M) Sp) \ (4,)(M)S,,4)) where S,. is the region of finite energy around the 1Ith memory and (A) (defined in Theorem 1) is the distinct attracting basin for the 1thh memory. The follow- ing theorem shows that this LSR based DenseAM is capable of simultaneously retrieving all (up to exponentially many) memories while also creating many novel local minima, and quantifies this phenomenon precisely.
Theorem 2. Consider a DenseAM parameterized with M memories = = (EI.... (i....,,M))) sampled uniformly from X with vol(X) = V < co and the LSR energy EySR(; (;; defined in eq. (3). For each novel local minimum x" = [ Exertse) Eu given in Proposition 2, define D Dmax = max, ||x* -Eull.
8mial (xx) == min
3 -1x Eull2)
Yunin (x") = min "gB(x*)
||x [|x*-EEn Ex
2
n. (xx)
B
Then, for all B > O such that maxx* 8min (x*) ) > and minx. (imu(x") > O, there exists an E ==
minx- (/Daaxxx' + min({minn (ōmin (x*), ) Tmin (x*)] - D Dmax (xxxxxx)) > O such that ELSR() satisfies the e-ooobal emergence condition (Definition 2) with high probability, as:
(a) With probability at least E (0,1), and M = O ((11-3exp(ad)) exp(odd) for a positive a, all memories are retrievable as per Theorem I with the value of the minimum pairwise distance
r = DiD IrE, Ev 2 (Va/V)-1/d) -1/d 22 and per-memory basin radius A e (0, (Va/V)-1/dg 1--ee--- 20) with 3 S 2/((VaVV) e 2a A), where Va is the volume of the
unit ball in Rdd.
(b) For each novel local minimum x", there exists a radius
r= / DD (x*) + min ( 8min (x*) 7min (x*)] - -mmxxx'' > 0,
(4)

---

such that Sx- (r*) E - and for any x (r"(r",, the output of the DenseAM via energy gradient descent is exactly x', implying the novel memories are retrievable.
x:
rr"} 5=''=>>"
forms basin around the novel memory x";
Furthermore, with probability at least 1 M the number of s-globally emergent memories is
o exp
Va
eVV Va
3 2
2
M
log
(5)
In particular, for fixed 3 > O and d, the bound grows with M whenever 3 > satisfies [B(x) E
(1, M] due Proposition 2. Here, for any x E X, large B leads to small [ [x)),. and [B(x)] >
is the required condition for a novel local minima; a small 3 leads to a large [(x)), and [B(x] S
M stands for the case (xx) at most covers the entire domain X.
The above result demonstrates that with the LSR energy, it is possible to exactly memorize all original patterns (with high probability) and still generate new patterns - what we term emergent memories (Definition 2). This behavior is surprising in the same way interpolating models in deep learning generalize unexpectedly well: both challenge the classical bias-variance intuition [15, 16]. While LSE-based models also produce novel memories, they typically do so at the expense of perfect recall of the original patterns. In Proposition we show that LSE based DenseAMs do not have the global emergence property. This distinction highlights key contribution of our work: new memory creation need not come at the cost of perfect memorization.
Next, we provide an exact order (i.e., upper and lower bounds) of the number of emergent memories under grid design assumption.
Proposition 3. If } - form a grid over X of equal size with Vol(X)) V co, the number of emergent memories of order & ((Mi/ X2/d + +1)") 1)"). where À = (M such that
and for 0 (8/3)) (8/3)²)
M.
Note that we showed an explicit form of the emergent memories x" 1 B(x*)) EpeB(x)) Eno where B(x") = {p: ||x - Eلul < v c {I...., M.. equation in Theorem 2 is stated under the uniform sampling regime, and Proposition 3 is stated under a fixed grid setting. In general, the number of emergent memories varies according to the specific geometry of the stored patterns MM i.e., whether such subset of f1..... .M can be realized by a ball { [|x- -xull < v2/3]. This can grow much faster than a linear order of M, and is naively bounded by 2.M
4
Experiments
4.1
Quantifying the scaling of emergent memories
How many local minima do we see in practice as we: (a) vary the number of stored patterns, (b) change the dimensionality of those patterns, and (c) vary the inverse temperature (3? We observe that, at critical values of 3, we can create orders of magnitude more emergent memories than stored patterns. These results are shown in fig. 3 (left).
To quantify the number of local minima induced by the LSR energy, we uniformly sample M pat- terns from the d-dimensional unit hypercube to serve as memories E. We enumerate all possible local minima of the LSR energy by computing the centroid εx := AC| E'EE En for every pos- sible subset of stored patterns I. C [M] (there are 2M possible subsets if we allow for singleton sets). For each subset, we first check that its centroid is supported (i.e., that (Ex:: 3) < 00 at
< 0), and then declare that εx: is a local minimum of the LSR energy if small å > 0. 3 values are varied across the "interesting" regime between fully overlapping support regions (a single local minimum in the unit hypereube) to fully disjoint support regions around each memory. See experimental details in appendix D.2.
v D SR
(Ex: 3) ō for
Certain values of 3 yield particularly interesting behavior. For example, we observe that LSR can create orders of magnitude more emergent memories under ranges of 3 where: (i) a majority (>60%) of stored patterns are also recoverable, and (ii) around 20 percent of the unit hypercube is still
7

---

LSR Energy creates novel memories while preserving stored patterns
M = 25
100
dm8
dm32
Volume SuppRree
Log Log-Linilhoodd
Volume sApperted
Nam, unique generations
M=15
Volume susporred
M=
Nam. original memories recovered
B
Figure 3: (Left) Analyzing local minima in LSR energy reveals a number novel memories several orders of magnitude larger than M, the number of stored patterns, critical values of / (note that the y-axes are logscale). These emergent memories occur even while still preserving the stored patterns as memories. Smaller values of have larger region of support on the unit hypercube. (Right) Given samples from some known true density function (in this case, a k = 10 mixture 8- dim Gaussians with means drawn uniformly from the unit hypercube and 0.1), memories from LSR energy have ligeliod comparable to, and occasionally slightly higher than, LSE under the true density function. Note that LSR achieves comparable log-likelihood while having more unique samples than LSE, even when both are seeded with the same N = 500 queries. Regions of where LSR outperforms LSE on a metric are specified by the orange regions. Error bars indicate the standard error across 5 different seeds for sampling stored patterns and initial queries.
supported. Note that in each experiment there are choices of 33 such that the LSR energy does not exhibit global emergence (Definition 2, i.c., at low where novel memories are forming but not all stored patterns are yet retrievable). However, in these regions local emergence (Definition 3) of the novel memories still holds (see fig. 1 for intuition).
4.2 Generative quality of emergent memories
LSR memories are certainly more diverse than those of LSE, but do they represent more "meaning- ful" samples from true, underlying density function p(x) (as measured by their log-likelihood)? The experimental setup is as follows: Let p(x) be a mixture of k Gaussians whose means μ4 ~ 2((0, 1] for [&]] are uniformly sampled from the d-dimensional unit hypercube with scalar (a = 0.1) covariances such that p(x)
> N(x (I,)) We sample M points ( E,, ~ p(x) to serve as the stored patterns E used to parameterize both LSE and LSR energies from eq. (3). Define thin support boundary induced by pattern &, to be E..1l 2 for some small å 0. Then, for ini- tial points X n E [N] sampled from the support boundary around each stored pattern, LSE memories can be found using gradient descent
suppiE,; 8] {x 23-1
< ||x-E,ll1
x(((1))) a OV (1-1):
until convergence memory x We use algorithm to efficiently find the LSS memory corre- sponding to each initial point. Thus we have N "samples" (memories) from both LSE and LSR on which we compare three metrics of interest in fig. 3 (right):
1. Average Lig-Liielihood. Do LSR memories xisR have higher log p(xiss) than LSE memories?
2. Number of Unique Samples. Does high log x occur alongside many emergent memories?
We use the same initial points seed the dynamics of both ESR and LSE See appendix D.3 for details.

---

All LSR memories
46 emergent memories
38
emergent memories
6
5
6
8
0
3
3
2
5
2
6
6
1
3
8
9
9
0
3
9
9
6
0
8
9
3
9
8
3
8
8
8
2
8
9
9
+
22 24
40, preserved memories
preserved memories
9
0
6
8
6
2
6
8
8
0
0
9
3
3
2
5
9
9
/
6
X
8
X
O
All LSE memories (total, at same
02 O 2
Figure 4: LSR's emergent memories appear novel, creative generations when the energy is applied to semantically meaningful latent space. (Left) randomly-selected MNIST images are encoded into 10-dim VAE latents and stored into an LSR- and LSE-energy using carefully chosen 3 (see algorithm 2). Gray boxes indicate which stored patterns were not preserved at the chosen B. (Right) 40 Tmaylmagenet [27] images
encoded into 256-dim latents using a pretrained VAE [28] and stored into an LSR- and LSE-energy using a carefully chosen B. Note that this TinyImagener example the LSR energy is, by definition, globally emergent since all stored patterns are recoverable, while the MNIST example is not. See experiment details in appendix D.4.
Number of Original Memories Recoverable. Does high log p(x(ss) occur when alongside high numbers of preserved memories? How does this trend compare with L.S memory performance?
The results tell a consistent story. Despite LSE energy being a more natural choice to model the underlying Mixture of Gaussians' density p(x) (LSE has Gaussian kernel that makes it ideal for the modeling task), LSR can match LSE log- likeelhodood while simultaneously generating more diverse samples and preserving the stored patterns. See appendix D.3 for more experimental results and extended discussion.
4.3 Emergent memories latent space
What do emergent memories look like when LSR is applied real-world datasets? To study this behavior, we use a VAE to encode MNIST and Tinylmagene [27] images into latent vectors that serve the stored patterns for LSR and LSE energies (see fig. 4). Using carefully chosen 8, compute all memories (both preserved and novel) for each energy. The emergent memories of L.R in principle simply the centroids of small subsets of the stored patterns, yet when decoded they appear as plausible and creative generations.
With the same 3 value, LSR generates order of magnitude more total memories than LSE. While this choice of 3 is somewhat arbitrary and could be tuned separately for each energy, LSE would only ever be able to retrieve up M memories (where M is the total number of stored patterns). See full experiment details in appendix D.4.
Emergent memories are mechanistically simple: they are simply the centroids of small subsets of the stored patterns. The semantic novelty the emergent memories Figure occur because latent space is structured to be semantically meaningful, where averaging two or more stored patterns

---

produces seemingly novel semantics. We conduct a similar experiment in appendix D.5 where we ablate the VAE to show what emergent memories look like in pixel space.
Discussion
Emergent memories are powerful tool for creating novel samples, but the "meaningfulness" of these samples nuanced question that depends heavily on the specific application domain and task requirements. For example, in Figure 3 (right) we show that the LSR energy approximates an unknown p.d.f. better than LSE's energy while simultaneously generating diverse samples. This represents a desirable behavior of emergence, since high log-likelihood samples from the LSE en- engy are quite homogeneous, causing 500 initial queries converge to the same ~0 memories. In this density estimation context, the emergent memories serve a clear functional purpose: they capture meaningful interpolations within the data distribution that improve generalization.
However, consider the novel memories from Tiny Imagenet in Figure 4. Though visually plausi- ble, many of the emergent generations appear blurry and would be considered "undesirable" from the perspective of a high-fidelity image generation model. We discuss the limitations of emergent memories further in appendix A.
We note that there are potential parallels between emergent memories and "hallucinations" as ob- served in LLMs. We discuss the philosophical similarities between emergent memories and hallu- cinations in appendix B.1. It is also interesting to note that the LSR Energy presented in this work have a feasible biological implementation using bipartite neurons. See further discussion in appendix B.2.
Finally, we reiterate that there are many choices for alternative kernels, and not just the Epanech- nikov kernel. We discover that many kernels with compact support are capable of producing emer- gent memories and manifolds, and the energy landscapes they produce can look quite different from each other. To this end, we show the "basin merging" behavior across different kernels in ID in Figure 6 and include extended discussion these other kernels in appendix C.
Conclusion
Our work introduces the LSR energy function which achieves the surprising combination of exact memorization of exponentially many patterns and the emergence of new, meaningful memories, thereby providing a powerful alternative to traditional AM formulations. The properties the LSR energy define novel class of emergent memories in Dense Associative Memory systems phe- nomenon not observed in any previous AM formulations. Unlike conventional models (e.g.. LSE- based energies) where generalization (formation of local minima different from training data) typi- cally comes at the cost of perfect memorization, LSR demonstrates that these objectives can coexist harmoniously in single energy function. We additionally demonstrated that the diverse memories created by LSR achieve log-like oog-likelihood comparable to LSE when sampling from a true density func- tion, while generating an order of magnitude more unique memories. Finally, showed that when applied to latent representations of real-world image datasets, LSR's emergent memories represent plausible and creative generations.
10

---

References
[1] Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Ad- vances in Neural Information Processing Systems, 29, 2016.
[2] Dmitry Krotov. new frontier for hopfield networks. Nature Reviews Physics, 5(7):366-367, 2023.
[3] Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, Parikshit Ram, and Dmitry Krotov. Dense associative memory through the lens of random features. In The Thirty-eighth Anmual Conference Neural Information Processing Systems, 2024.
[4] John J Hopficld. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-255) 2558, 1982.
[5] Dmitry Krotov and John Hopfield. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12):3151-316) 2018.
[6] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:
288 288-299, 2017.
[7] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Scidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael Kopp. Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In Inter- national Conference on Learning Representations, 2021. URL https: ttttts://openreee oppnreview.
net htt.:/ooomnn-ttLLtnn1I11
[8] Dmitry Krotov and John J Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021.
[9] Carlo Lucibello and Mare Mézard. Exponential capacity of dense associative memories. Phys- ical Review Letters, 132(():077301, 2024.
[10] Dmitry Krotov. Hierarchical associative memory. arXiv preprint arXiv:2107.06444,, 2021.
[11] Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov. A universal ab- straction for hierarchical hopfield networks. In The Symbiosis of Deep Learning and Differen- tial Equations II, 2022.
[12] Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Chen, and Han Liu. sparse modern hopfield model. Advances Neural Information Processing Systems, 36, 2023. URL https::/procceeddd www.://orredddn.......o..oo..... neurips cc / httpp/aperrri//// fireerrfilesse /tlpss/ppppr//2233 fiL2://72caa52222200031117rrree2
here--Coooeeeeeeee .df..
[13] Saul José Rodrigues dos Santos, Vlad Niculae, Daniel McNeamee, and Andre Martins. Sparse and structured hopfield networks. In Forty-first International Conference on Machine Learn- ing, 2024. URL wwwphttpp.... http::///ppeeeeeeeeeeeeeeeeeeeee net/ forum? d=OdP 1FWExX1.
[14] Dennis Wu, Jerry Yao-Chieh Hu, Teng-Yup Yun Hsiao, and Han Liu. Uniform memory retrieval with larger capacity for modern hopfield models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Re- search, pages 53471-53514, PMLR, 21-27 Jul 2024. URL pettps://proceedingg. mlr. www.ppressvv222 htm.1.m.1................
[15] Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference Artificial Intelli- gence and Statistics, pages 1611-1619. PMLR, 2019. URL:ttp: httpttp://prooceedinggs mlr. pdf.
press//v8/bblliiil9aa///s
[16] Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism interpolation. Acta Numerica, 30:203-248, 2021.
[17] Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical view- point. Acta numerica, 30:87-201, 2021.

---

[18] Matt P Wand and M Chris Jones. Kernel smoothing. CRC press, 1994.
[19] Luc Devroye and Gábor Lugosi. Combinatorial methods in density estimation. Springer Sci- ence Business Media, 2001.
[20] Vassiliy A Epanechnikov. Non-parametric estimation of multivariate probability density. Theory of Probability & Its Applications, 14(1):153-158, 1969.
[21] Hans-Georg Müller. Smooth optimum kernel estimators of densities, regression curves and modes. The Annals of Statistics, pages 766-774, 1984.
[22] Enno Mammen. On qualitative smoothness of kernel density estimates. Statistics: a journal of theoretical applied statistiis, 26(3):253-263) 1995.
[23] Borjan Geshkovski, Philippe Rigollet, and Yihang Sun. On the number of modes of gaussian kernel density estimators. arXiv preprint arXiv:2412.09080, 2024.
[24] Shun-Ichi Amari and Kenjiro Maginu. Statistical neurodynamics of associative memory. Neu- ral Networks, 1(1):63-33, 1988.
[25] Anthony V Robins and Simon JR McCallum. A robust method for distinguishing between learned and spurious attractors. Neural Networks, 17(3):313-326, 2004.
[26] Adriano Barra, Giuseppe Genovese, Peter Sollich, and Daniele Tantari. Phase diagram of re- stricted boltzmann machines and generalized hopfield networks with arbitrary priors. Physical Review E, 97(2):022310, 2018.
[27] Ya and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https: //app.seaandeeeeeeeeem www.stecccoonoooseeo.oeooocoo org I 61666667700
[28] Ollin Boer Bohan. Tiny autoencoder for stable diffusion.
https://github... com/
madebyollin/tasee 2023.
[29] Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01700 2019.
[30] James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-M and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //githus ccom/jax-nnt maxja.
[31] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
wwwxii.ooggaallaa.aa.aaaoaeoaeaa absg///b///e at//aabss/111/ 11111111122 6114.
[32] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual con- cepts with a constrained variational framework. In International Conference on Learning Rep- resentations, 2017. URL htthttt::: httptttpe///peeeeeiieeee// opwpreveeeeeeeeeeeee/eeeee wwt. for//ooumm/iieee 2ttt://foormmmmd15222222222 2it.iiddssyy2ggyyyg
[33] Martin J Wainwright. dime-diiel statisties: non-asymptotic viewpoint, volume 48. Cambridge university press, 2019,
12

---

A Limitations
To enable both emergence and exact memorization, the LSR energy studied in this work comes with certain limitations. Unlike the LSE energy whose gradient remains nonzero regardless of the query s distance from the stored patterns, the gradient of LSR energy vanishes exactly outside the support of the stored pattern. This makes gradient-basse any memorized pattern (though one can easily introduce a query-dependent temperature parameter 8(x) that dynamically adjusts to ensure the query lies within least one basin of attraction). Alter- natively, one can create a "hybrid" energy function that combines the LSE and LSR energies, taking advantage of LSE's non-zero gradient everywhere outside LSR's support around the stored patterns. Such temperature-tuned DenseAMs and hybrid energies are of independent interest and we leave their systematic study future work.
retrieval ineffective when the query lies far from
Additionally, we want to emphasize that LSR can only create emergent memories precisely at the centroid of overlapping basins between stored patterns. This makes it possible quickly and exactly retrieve memories (see Algorithm 3), but it also means that the "creativity" of this model is limited to predictable subset of the convex hull of the stored patterns. The apparent novelty and creativity of Figure 4 is strongly aided by the semantic structure contained the VAEs' latent space.
B Extended Discussion
B.1 On the relationship between emergent memories and hallucination
In everyday use of LLMs, we call a hallucination when the model confidently produces an in- correct fact, especially when we know the model is capable of producing the correct fact in other settings. To formalize this idea into the context of emergent memories, we must imagine a setting where the model has an explicit energy function and known set of stored "facts."
This is the setting of Associative Memory as studied in this work, where the outputs of our model are always energy minima (memories). A "hallucination" can occur when two or more stored facts interfere, leading the system to settle into an emergent minimum that looks meaningful (i.e., it lies near the "factual manifold" that generated our stored patterns) but does not correspond to any true, stored fact. Emergent memories are precisely these "interference minima" when they coexist with the original stored patterns. By definition, they are not stored facts, yet they can resemble meaningful combinations of them making them the memory-system analog of hallucinations.
B.2 On a biological implementation of emergent memories
Dense AMs permit standard mapping onto "biological" networks by introducing auxiliary hidden neurons following the method of [8]. Thus, the feature neurons in both LSE and LSR models can be augmented with auxiliary hidden neurons, resulting in model with only pair-wise neuron interactions and bipartite connectivity between feature and hidden layers. This augmented model is more biological compared the model considered in the main paper. The proposed model defined by the energy eq. (3) and the corresponding update rule eq. (7) can be obtained by integrating out these auxiliary hidden neurons, which means LSR energy will still have the same emergent behavior. The augmented model will still have some un-biological aspects because of the weight symmetry between forward and backward projections, which necessary to ensure that the network has a global energy function.
C On Kernels
We show different kernels that are typically used for KDE and their efficiency relative to the Epanechnikov kernel in fig. 5. See the explanation optimal kernel density estimation in section 2 for more details.
Though our main submission focuses on the Epanechnikov kernel, we find that the phenomenon of emergence (Definitions 2 and 3) not limited to the Epanechnikov kernel. We state our findings for other popular kernels below and graphically in Figure 6. In summary, all compact support kernels are capable of producing emergent memories except for the TriWeight kernel (meanwhile, the uniform kernel is impractical for associative memory). Details below:
13

---

Gaursian [Mpp-P[[)) Efficiency. 95.1%,
Triangle
Uniform
Cosine
Triwllght
Relu11 jall
11 (xP)
ReLU11 jrje
castmind8)sj.2
Relu11 - Jrjjj
RecUil [opjjp Erficencce
Recott - PPp.
Efficincyy
Emclency:
Efficency: 100.0%
Efficiancy 99.9%
Efficience 99.4%
Effilenccy
Figure 5: Different kernels used in KDE with their expression and KDE efficiency relative to the Epanechnikov kernel (higher is better, text for details). The center of each kernel is marked with a red *. To highlight the shape of the kernel, we have removed any scaling in the kernel expression. Note that all above kernels except Gaussian have finite support. The Epanechnikov kernel has the highest efficiency (100%). While the Gaussian kernel is extremely popular, and it is more efficient (95.1%) 1) than the Uniform kernel (92.9%), there are various other kernels with better efficiency.
Triangle kernel The triangle kernel exhibits very interesting emergence behavior. A perfectly flat energy manifold is formed where two or more basins overlap. If the energy of this manifold lower than the energy of the stored patterns, the stored patterns "merge" and are no longer retrievable. If the energy of the manifold is higher, both original patterns are preserved and we observe a local emergent manifold of memory.
2. Uniform kernel - A uniform kernel produces an energy landscape that is impractical for asso- ciative memory because the gradient is zero everywhere except at the discontinuities. Emergent minima exist according to Definition 3 and have exclusively lower energy than the stored patterns. Triweight kernel We were not able to find emergence at any temperature with the triweight kernel, despite its compact support. Indeed, looking at the / = 0.19 row of Figure 6, we see that the transition of two basins merging results a single, almost flat minimum. This phenomenon of compact support without emergence (or perhaps, emergence that is limited to an extremely narrow range of 3 that we were not able to find) requires further investigation.
4. Quartic kernel The quartic kernel produces smoother energy landscapes than the Epanech- nikov does, and emergence appears within a very narrow range of B. Unlike the Epanechnikov kernel, overlapping basins do not guarantee emergent minima, and the emergent minimum is unlikely to have lower energy than the stored patterns.
5. Tricube kernel The tricube kernel behaves like the Quartic kernel, but emergent memories are likely to have lower energy than the stored patterns. The Tricube kernel has an interesting property where local minima flatten right before basins merge. The resulting energy landscape smoother than that of the Epanechnikov kernel.
6. Cosine kernel- The cosine kernel looks remarkably like the Epanechnikov kernel, and its emer- gence properties are almost identical to that of LSR. However, it appears that Cosine-emergent memories have slightly higher energy than their Epanechnikov counterpart.
D Experimental Details
D.1 Reproducibility & Technical Resources
The codebase is published in a GitHub repository and contains necessary instructions setup the environment and to recreate all results, down to the same seed used for training and sampling. Experiments use both PyTorch [29] and JAX [30]. Each experiment was run on system with access to 8xA40 GPUs each with 40GB of memory. The log-likelihood experimental sweep in appendix D.3 took 12 hours using an optimized scheduler across 45 available CPU cores (memory turned out to be the bottle neck and the CPUs had access to much more RAM than the GPU). Searching for all novel minima at different 3 and M in appendix D.2 took 448 hours using an optimized scheduler across all 8 GPUs. Training the VAE in appendix D.4 took < 30 min on single GPU; enumerating all local minima using the efficient algorithm 2 and algorithm 3 took < 15 min.
D.2 Details: Quantifying Novel Minima
In this experiment we tested across a geometrically spaced range of B E [2d-1, 1.2'. 22rm2l where T'in min |r,, - E. is the minimum pairwise distance between any two stored patterns the current subset K. C [M]. At the largest B, the support regions of the stored patterns are disjoint and the only memories are the M stored patterns themselves; this configuration has a very small support region (shown as the shaded green curve in fig. 3, which is computed by monte carlo sampling le6
14

---

Gaunsian
Beiifomm dAtji sil1
Quartic ROLUK ROLKL-BIiiT -UNIPt
Tricube POUII - -Ale
Cosine congnentilllle
Epanechnikov Reluc - Bl
haLaa - FlAlo
RecUI 10U03330017
p-25
6+4.0
D.0.25
p=0.19
P-0.11
Figure 6: Comparing emergence across different choices of kernel in the DenseAM energy function. Emergent memories are highlighted in red, where manifolds are shown as flat line and single points as larger dots. Interestingly, all compact kernels exhibit some form of emergence except the TriWeight kernel.
points on the unit hypercube and computing the fraction of energies that are finite at e = 0) as a fraction of the unit hypercube. At the smallest tested 3, only single energy minimum is induced at the centroid of all stored patterns with a region of support covering the whole unit hypercube. At the largest tested B, all original memories recoverable and there are spurious memories.
D.3 Details: Generative quality of memories
D.3.1
Additional experiments
We ran the same experiment in section 4.2 under varying dimensions d = [8, 16) and number of mixtures k [5, 10), averaging the results of each run across 5 different random seeds. The results for each of these experiments is shown below in figs. and 8, where fig. 8 (left) is the same as reported in fig. 3 (right) in the main paper.
D.3.2 Aligning I's across random seeds
We used the same setup for choosing an interesting range of 3 as we did in appendix D.2. However, over random seeds the value for Tmin can vary since it is dependent on the random choice of stored patterns. This makes difficult plot error bars over an individual 3 across seeds. fix this, we use the fact that each experiment has the same number of geometrically spaced 8's that start from same Blo and compute statistics averaged across the I's that share an index. The r-axis then represents the 3 value for each index averaged over seeds.
D.3.3 Determining the uniqueness of minima
To sample the LSE minima, 500 initial points are uniformly sampled from the support boundary around each memory and gradient descent eq. (6) is performed for 13000 steps at a cosine-decayed learning rate a from 0.01 -> 0.0001. However, even after this descent process, there are variations in the retrieved memories due to discrete step size a and floating point precision requiring us to be careful when deciding if two samples are distinct. Generally, memory retrieval is said to converge when V Ex)]| < for some small > 0, when the number of iterations T exceeds some threshold small a. Because we know the of the LSE energy, by the properties of the Gaussian kernel we know that two basins merge when their means are within two standard deviations of each
15

---

Mixture of 5 Gaussions 8 dims
Mixture of k 5 Gaussians in 16 dims
=5
M=100
Average
Log- Likelihood
-30
30
# of unique generations
307
# of original memories recovered
Figure 7: Comparing d = 8 and d 16 for = mixture of gaussians at number of stored patterns M 10 and M 100. Error bars computed by averaging the results of different random seeds. Regions of |3 where LSR outperforms LSE on a particular metric (on average) are shaded orange.
Mixture 10 Goussians in d dims
M 25
M 100
25
M 100
Average Log- Likelihood
-30
# of unique generations
300
# of original memories recovered
B
Figure 8: Comparing d = 8 and = 16 for k 10 mixture of gaussians at number of stored patterns M 10 and M 100. Error bars are computed by averaging the results different random seeds. Regions of B where LSR outperforms LSE a particular metric (on average) are shaded orange.
other. Thus, we can say that two distinct samples are generated by the same memory if they are within 2 of each other.
VB
When counting the uniqueness of the samples from the LSR energy, perform the following trick to exactly compute the fixed points of the dynamics. We first compute our "best guess" for the fixed point by performing standard gradient descent according to eq. (6) for T steps, at which point =
x (T) is close (but not exactly equal) to the fixed point x*. We then pass algorithm 1 to compute the fixed point exactly. With a good initial guess z, algorithm converges after a single iteration.
16

---

Algorithm 1: Fixed Point Computation for the LSR Memory Retrieval
Input:
Initial guess 2, stored patterns {Eu { inverse temperature ; Fixed point z
Output:
Initialize previous point Eprey <- + 0c
while Zprev + do
Zprey f-2
Compute supports near z S(z) + (Enr 1| - -2ll S V
Update to mean of support centroids z <- [S(2) ΣENESto) E.p
end
return
Finally, we choose to sample points near the support boundary of each stored pattern because this maximizes the probability that will end up in spurious minimum. The size of spurious basins in high dimension can be very small, and the probability of landing in them decreases rapidly with increasing (see the region of support plot in fig. 3).
D.4 Details: Qualitative reconstructions
The experiments in fig. 4 are conducted in two steps. First, we design energy for each dataset. Then, we discover all memories for each system.
Designing the energy The DenseAM energies studied in this work are described by matrix of stored patterns and an inverse temperature hparam [3.
Choosing E. For MNIST, 3 is obtained as follows: 24 random images from the MNIST training set are normalized be [0, 1]. rasterized into a 784-dim vector, and projected into 10-dim latent space of 3-VAB trained according to the methods laid out below. These latents become our stored patterns E MNIST& E [0. 1]21x 768 that we use to parameterize both the LSR energy and the LSS energy. The procedure for Tinylmagenet [27] is similar. We randomly select 40 images from the dataset, each of shape (C, H, W) = (3, 64, 64). These samples are passed through a small pretrained VAE called TAESD [28] to produce latents that are of shape (4, 8, 8) which are then rasterized into vectors of shape (256.). Thus, our stored pattern matrix for Tinylmagenet is TTinylmgnet E IR 40x266
Choosing 3 Tuning 8 for the LSR energy is challenging. When / is too small, each stored pattern interacts with all other stored patterns induce one minimum the centroid and several minima far from the data distribution; when 3 is too large, each stored pattern will interact with no other stored patterns and will induce only a single minimum at itself. Neither of these regimes are interesting. For large ranges of 33 between these limits, the combinatorial search space of possible memories computationally prohibitive. For this reason, we use a binary search algorithm to choose B that, average, causes each stored pattern to interact with approximately 4 other stored patterns (in the 10-dim MNIST case) and other stored patterns in the 256-dim Tinylmagenet case. This / encourages the LSR energy to exhibit emergence where it is computationally feasible to enumerate all memories, and we use for both the LSR and LSE energy experiments. In pseudocode, the search algorithm is given by algorithm 2
Training the 3-VAE for MNIST. MNIST images are encoded by a 3-VAE [31, 32] with a latent dimension of 10. The VAE takes input the 784-dim rasterized MNIST images. The VAE's encoder and decoder are two layer MIPs configured with LeakyRel.U and BatchNorm activations, with hidden dimension size of 512. Training proceeded for 50 epochs using learning rate of le-3, the Adam optimizer, and a minibatch size of 128. The of the 3-VAE (distinct from the inverse temperature 3 used by the LSR energy) is set to 4.
Discovering all memories Once we have chosen a 3 that results in a feasible number of basin inter- actions K (where the combinatorial search space is tractable), all memories for LSR are discovered by filtering the set of "mmemory candidates" around each stored patterns - to those whose energy gradient is zero. This method is explicitly de- scribed in algorithm 3, which iterates through each stored patterns E,, and only searches for emergent memories formed by near-enough stored patterns.
the set of centroids formed by at overlapping basins
17

---

Algorithm 2: Binary Search over & to achieve desired memory interactions
Input:
Desired avg. interactions per pattern K, stored patterns (En ) max iterations 7mmax Output: Optimal 8 achieving number of basin interactions K Compute
Distance matrix Dio < K 1r Binary bounds (Tmin T'mxx) <- (0.5 min (D), max(D))
Initial basin radius r <- mean( [fmin m'max)
end
Viter 6-
repeat
Compute avg. number of interacting basins per memory
(Σ 1P 2rl)
K' mean,
Update binary search conditions
Tmin f- r K' K
Tmax +- if K' K
r + mean min: f'mxx
Titer Aster
until K' = K or Vläer 2 Tmmax
8 2/r² return 3
Algorithm 3: Discover local minima of the LSR energy at specific B.
Input: Stored patterns ( inverse temperature 3, gradient norm threshold å near Output: Set of LSR memories X Compute
Distance matrix DuV < fllE Er11}
Basin radius r < 2/3
end
Initialize of local minima X² 0
for μ E [M] do
Compute set of interacting neighbors X,, + {E. Wv S 2r]
Compute the set all non-empty subsets +- (S C X, size(S) > 0}
for S E C do
Compute centroid of neighbors Xş +- mean(S)
Compute Xs neighbors I's + {Ev : E xsl rr}
if ELER (&s) < 8 & ÆLSR (8s) < oo then
Update set of local minima X' X {xs}
end
end end return X
All LSE memories are discovered via gradient descent. We initialize queries x = f and perform gradient descent according eq. (6) until convergence (for this experiment, we iterated for 20k steps at small step-size 0.002). The retrieved fixed points are the complete set of LSE memories.
D.5 Additional experiment: Pixel-space emergence
When the stored patterns live a semantically structured latent space, as is done in fig. 4, the energy landscape inherits the structure such that centroids of subsets of stored patterns appear semantically novel. However, the latent space visually obscures the mechanistic simplicity emergent memories. Thus, repeat the experiment in pixel space reinforce how emergent memories work.
We store 8 randomly selected MNIST images as rasterized pixels (normalized between 0 and 1) into the LSR energy. The resulting emergent minima and stored patterns are shown in Figure 9, where the 3 chosen balance the number of emergent minima and the retrievability of the stored patterns (global emergence Definition 2).
18

---

D.6 Additional experiment: Scaling number of stored patterns
The LSR energy function is simple and its properties hold across any scale of stored patterns. To show this, we store all 60,000 MNIST training images into the LSR energy and select B for which ~50% of the stored patterns are still retrievable. We select a "seed" image at random and randomly select 15 images from all images whose basins interact with the seed image at the chosen B. This example is shown Figure 10.
E Proofs
E.1 Proof of Theorem 1
For any x E X, let Bxx) = {u E [M] ||x Eull s 2/33. Then the gradient of the LSR energy in eq. (3) is given by
(x Ew) 1 ||x 111--
Eall -
V F (x) Σ
E ReLU (1
2 ||x E
E +
Σ (x- Ep)
(8)
++ E EveB(x) ReLU (1 8
2 x|x -Ex1()))
With 3 = 2/(r - A)? Bxx) = Eor E [M]] ||x Eull
I (r - A)}- For any x E S,(A),
B(x) [n]. Thus the LSR energy gradient simplifies to
(x (x-E,
Vx E SAAA),
Vx ELSR (x)
(9)
E ReLU
[1 2 ||x Eull
which is exactly zero at x = En. thus giving us the retrieval of the 1uth memory via energy gradient descent.
Furthermore, again for x E (A) with a energy gradient descent learning rate set to + +
ReLU (1 -1- (1-8/2]| E.1 1) the update exactly 17 v) F, (x) = (x Ep)- Thus single step gradient descent update to x with x - V V (x) = x (x - -.,) = Ep results the retrieval of the n memory.
E.2 Proof of Proposition
Proof. (Proposition 2.)
For any X, given the definition of B(x), recall that the gradient of the LSR energy is given in eq. (7). This gradient zero when
Σ ENS
B(x)
((x)
E+H
E+F
F+H
A+F
D+F+H
A+D+F
D+H
A+D
Emergent Memories
4
8
4
All 8 stored patterns are retrievable
A S
B
2
C
0
D
(
E
7
F
8
G
6
H
4
Figure 9: Emergent memories are centroids of subsets of stored patterns, shown clearly when 8 stored images are visualized in pixel space alongside their induced emergent memories. Stored pat- terns (bottom, indexed A-H) merge to form emergent memories (top, labeled by the stored patterns that merged form the emergent memory). 3 is chosen such that the number of emergent memories is approximately the same as the number of stored patterns.
19

---

Seed image
Nearby emergent memories
9
7
9
9
9
7
7
7
7
7
7
7
4
9
7
4
Figure 10: Sampling emergent memories near a seed image when all 60k MNIST training images stored into the LSR energy. Left: Random seed image, which is preserved memory. Right: 16 randomly sampled emergent memories formed by the seed image's interactions with other stored patterns (at patterns, these emergent memories represent tiny sample of the total emergent memories near the seed image.
0.11). Because the seed image interacts with the basins of ~7.3k other stored
the geometric mean of the memories corresponding the set Bxx). Moreover, by standard algebraic computations, we have
[B(x))
(LES((()) = ++ Evs ReLU (1 (1-
la > 0. 8 ||x - -Ev1(²)
E.3
Proof of Theorem 2
Note that once we prove properties (a) and (b), the e-global emergence in Definition 2 follows immediately.
We first focus on (a). Note that for any fixed x E X' and & > 0, vol(B(x, E) n X) is at most vol( ((x,))) = Vae", where Va is the volume the unit ball in Rd Thus, we obtain: for any pair (Enr Ex) with μ + v, have:
vol(B(r,)) dx s V.V-igd vol(2)
Pr [|E, Eall Eoll<E
Jx
Then, we have:
Pr min E| Evll < MvV ISHFUSM
(10)
Now, we let E = (Va/V)-1/dg-a) for a positive a. Thus, with probability at least 1 2 2nd the minimum pairwise Euclidean distance minisiifgM E1 - EE 2 (Va/V) 1ldd 20 Then, with
2 (r A)- ((VaVV)-1/de 2a 2)
0 0<
are able to retrieve any memory f,, with E S,(A), for A E (0, (V More- over, the size M is given by the success probability:
8 = 1 - 2-e 2ad
This gives
M M O
Next, we focus on (b). Recall the gradient is given by
Σ Epe (x - En)
Vx (x)x(xx
+ EpeB(x) ReLU (1- 8 E 11,111
20

---

where it is determined by the set
2
B(x) [e [1=1|x-
1|1 Sull
V
Let Sx- (A) fx ||x xl - A} be basin around emergent memory x". Note that by Schwarz inequality, the change within the basin of the squared distance to any En is at most
|||x 1 1|x 6,1l2
s 2||x*
En + A
We now for activated memories μ E B(x*) let
2 1| - Eull > 0.
miin (x')
min pEB(x*
This is the margin the boundary of activation. Moreover, for inactivated memories & B(x*), let
||x - 2
Tmin (x")
min B(x)
0.
3
M has a density, the probability of either of them This is the margin to being activated. Given (En [ being exact zero is zero.
Finally, we determine the radius A. To ensure that for any x E (x-( VxLiSR(x))
Vx (xxx(x))) it sufficess to make sure (xx) B(x*) for any Sx (A). To this end, we can pick
A such that
2D 2Dmax (x*) + A min (anin(mma(( (xmin(")), "x")n(")), where Dmax fxax((x)= max X Euull-
1
This gives
V Di (x*) + (omin (x*), Yiin (x*)} D maxx (x*) =ir-.
To prove the final claim, without loss of generality, we consider the case when X [0, 1jd with V 1. Since {5n } are i.i.d. uniform, E,2 E { [|x Eاul s / are i.i.d. indicators over
a = ,....,M, M, by the concentration of sub- Gaussian random variables [33, Proposition 2.5], we have: with probability at least M-
A = [B(x)
2
(2)
log M, MV. (2)
+5 MVa (2) log M
E
MV.
-51 MVa -VAV (B)
+
(11)
B
This implies X o (MV (z) (3)+). Note that according to Proposition 2, for each [B(x)
(1, .). there corresponding emergent memory as the average of all stored patterns over E B(x). Then, the number of new emergent memories is bounded as
where A = (Mra 2 3
M (M)
Qa.LSR
By Stirling's 's formula, we have:
Q Q S e.M
Thus, holds that
eM
8
log QO.LSR I Alog
= 0
MVA.
log
Va
This implies
2
Q8,L.R = O exp
MVa
log
3
Va
2
21

---

E.4 Proof of Proposition
Without loss of generality, we consider the case when X [0, 1 with V 1. Now, we divide [0, 1] into equally spaced grid of size M, i.e., divide each dimension [0, 1] into [mM-1/d (m 1)M-1/dj for o,..., M 1. Under our grid setting, (E lie in such grid of equal size.
Now, note that
/
(xx)
pe: 1x:1|x-GE Sull < V3J
Suppose the ball in definition of Bxx contains À of those equally spaced stored patterns such that = -B( = À E (1, M.. We want to put hypercube the grid such that it contains exact out of M equally spaced stored patterns in the grid. Here, for simplicity, it is up constant depending on d if we think of the ball as a hypercube in Rd containing the points in the grid in this case. Hence, in each dimension 1 < i S d, we put an interval containing of order \ points the grid inside the entire interval containing of order M1/d points in the grid. thus gives the number of choice for each dimension 1 < S d as:
M 1/d
Hence, counting all choices over d dimensions, we obtain the number of such (xx) as
(M2/4 (MV/4->1a-1)
Moreover, due the equal spacing of the grid, have:
x= M 8 4
(12)
Hence, combining all bounds above, obtain: the number of emergent memories under uniform sampling is of order
e ((ArVm=>1/++1)) +1)++110 +1)²)
22/d
where À satisfies equation and 1 < A S M.
E,5
Proof of Proposition
Recall the LSE energy:
ELSE (x)
log Σ e f ||
1
Thus, the gradient
M xx)=x- Σ (x) Px(x)
||x- E.ll
V LSE (x)
Σd e- 11x-e,
Suppose & for some y is the zero of this gradient, it yields that
(1 (E,))) = Een(En
This is contradiction of the assumption that M are linearly independent.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction make claims that are clearly backed by theorems and experiments in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings, is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations discussed in the limitations section of the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to viola- tions of these assumptions (e.g., independence assumptions, noiseless settings, model well- specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address prob- lems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof? Answer: [Yes]
Justification: Each theorem provides the full set of assumptions and a complete (and correct) proof the Proofs section of the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement any theorems.
23

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main exper- imental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes]
Justification: The Reproducibility section of the Appendix provides information about the hard- ware and description of the code used to create the results reported in the main paper. For important algorithms, we include succinct and clear pseudocode in the Appendix. For review, the anonymized code is submitted alongside the full supplementary zip file, while the code is made publicly available on GitHub for the final version. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soures source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: We link to repository that contains code to completely reproduce the results ported in the main paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /nips/pp........ cc/
publi / guides/ CodeSuomdm aisionpoicyy for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central the contribution (c.g., for a new open-source benchmark).
24

---

The instructions should contain the exact command and environment needed to run to repro- duce the results. See the NeurIPS code and data submission guidelines Pattps://ning
cc/ cc/publ i с //gui des /CodeSubm as onPo 1 i cy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The attached code specifies all the training and test details necessary reproduce the results. Important hyperparameter configurations for the experiment are provided in the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experiments where the results are strongly affected by random seeds are repeated over multiple seeds. The results are averaged and reported alongside errorbars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates). If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the exper- iments?
Answer: [Yes]
Justification: The Reproducibility section of the Appendix provides information about the hard- ware and resources.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the
experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ////urrrippp wwtppp/wcc/tttcccseeooooccc
Answer: [Yes]
Justification: In publishing this paper we have adhered in full to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consideration
due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work primarily theoretical and considers only toy examples (MNIST, syn- thetic data, Iinylmageeee). It will have no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strate- gies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: There is no risk of misuse of the contributions of this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary
safeguards to allow for controlled use of the model, for example by requiring that users adhere
usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
26

---

We recognize that providing effective safeguards is challenging, and many papers do not re- quire this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All code and data properly credited and the licenses properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswa wwwithcooe tcooe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset. For existing datasets that are re- p--acaaged, both the original license and the license of the de- rived asset (if it has changed) should be provided. available online, the authors are encouraged reach out the asset's
If this information
creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmdell as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc. The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/revieww based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly
state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and cations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if appli- cable), such as the institution conducting the review.
16. Declaration of LLM usage
an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required. Answer: [NA]
Question: Does the paper describe the usage of LLMs
Justification: The paper does not involve LLMs any important, original, or non-standard com- ponents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: :estts://neeeee //tp:///ee...eeeeeeee hctpa//cecceeeece//222222. L2MM)LL for what should should not described.
28


Reviews:
[{"review_id": "kdX2aWq1Ay", "reviewer": "Reviewer_Ua9r", "summary": "The paper explores energy-based associative memory networks. A notable prior work is the LSE energy function, which yields exponential memory capacity. However, it typically faces a trade-off between perfect memorization of stored patterns and the generalization to novel patterns. This paper proposes a new energy function log-sum-relu (LSR) inspired by the connection between energy functions and probability density functions. Through some exploratory experiments, the paper shows promising demonstrations for exact memorization of all original patterns, and simultaneously generates new, meaningful emergent memories.", "questions": "* On the use of ReLU and its relevance to the modern deep networks. ReLU was proposed in part to facilitate gradient flows and avoid the gradient vanishing problem. Does ReLU play a similar role in LSR? How about studying other more advanced activation functions such as GeLU and GeGLU?\n\n* It may be non-trivial to stack multiple LSR to form deep networks of memories. I am wondering the potential of formulating and implementing this.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The paper provides a novel view regarding the connection between energy functions and KDE. \n\n* Theoretical contributions are made especially for generating novel emergent memories. This also lays out a framework on studying emergent capabilities for future works.\n\nWeakness:\n\n* The scale of the current experiment results is too small. For results shown in section 4.3, only a handful of less than 100 images are used for memorization. It remains unclear if the training images scale to thousands or millions, the current conclusion might not be hold.\n\n* The paper now lacks quantitative experimental evaluations. While theoretical contributions and empirical demonstrations are nice, it is beneficial to follow LSE paper for realisitic quantitive evaluations, such as on classification problems.", "was_revised": true, "final_justification": "My concerns have been well addressed. I decide to keep my positive rating.", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper. We are encouraged by the positive scores! Reviewers across the board have highlighted the following strengths of our work:\n\n1. Regarding **novelty**, our work exploring the connections of KDE and Associative Memory is \"novel\" (Ua9r), \"original\" (TqXz), \"elegant\" (4hP4), and \"insightful\" (TqXz, 4hP4). It is a \"conceptual breakthrough for the field of associative memories\" (TqXz)\n2. Regarding **rigor**, our work is \"a principled solution to the memorization-generalization trade-off in DenseAMs\" (4hP4) that \"lays out a framework on studying emergent capabilities for future works\" (Ua9r)\n3. Regarding **quality**, the paper is \"well-written and easy to follow\" (TqXz) and \"already above the acceptance threshold\" (4hP4)\n\nReviewers also noted that several discussions/experiments would further improve our work:\n\n1. **Increase the scale of the current experiments** and confirm that the current conclusions hold (Ua9r)\n2. **Clarify when emergent memory is meaningful** rather than harmful (TqXz)\n3. **Discuss biological/neuroscience implications** (4hP4)\n4. **Elaborate on the experimental details for Fig. 4**, the VAE real-data experiments (4hP4)\n5. **Discuss emergence properties of other kernels** used for KDE (4hP4)\n\nWe believe the reviewer feedback has increased the strength of our submission, whose changes we summarize in our responses to each reviewer. Unfortunately, this year's NeurIPS rebuttal format prevents us from sharing the revised submission and updated figures. We do our best to provide clarification and describe results in plain text.\n\n---\n\n>  It remains unclear if the training images scale to thousands or millions, the current conclusion might not be hold.\n\nIt seems like the reviewer is requesting empirical confirmation of Thm. 2 in the regime of large data (please correct us if our interpretation of \"current conclusion\" is amiss). To satisfy this, we design an experiment to store all 60k rasterized MNIST training images into the LSR DenseAM. In this large data regime, we show multiple examples of novel images forming between two or more stored patterns that are still retrievable. This confirms the primary claim of the paper: that LSR energy can create *emergent memories* --- novel energy minima co-existing with energy minima of the original stored patterns --- at any scale of data.\n\nNote that our current submission focuses on quantifying and studying the complete set of *emergent memories* in regimes that we could feasibly compute. It becomes computationally infeasible to enumerate all the emergent memories in the regime of high-dimensional spaces $d$ and a large number of stored patterns $M$. For instance, consider the largest experimental setup of Fig. 3 (Left), where $d=32$ and $M=25$. To enumerate all possible emergent minima, we need to check the energy gradient of the centroids of all possible subsets of the 25 stored patterns --- a combinatorial search problem over $2^M$ possible centroids per each tested $\\beta$. The real-data experiments in Fig. 4 are even larger, where we store $M=40$ patterns of dimension $d=256$, requiring a search over $2^{40}$ possible centroids. We can optimize this search using e.g., Alg. 3 in Appendix C, but for large $M$ (like the complete MNIST training set), even this algorithm would computationally fail at enumerating all minima for interesting $\\beta$.\n\n> ReLU was proposed in part to facilitate gradient flows and avoid the gradient vanishing problem. Does ReLU play a similar role in LSR? How about studying other more advanced activation functions such as GeLU and GeGLU?\n\nReLU is a particularly interesting function since it is prevalent in many modern architectures, though our inspiration for ReLU in the LSR energy comes primarily from KDE literature. ReLU does indeed help prevent vanishing gradients during memory recall in Associative Memories compared to e.g., the sigmoid function, where gradients are tiny for large positive or negative inputs to the function. Just note that the \"vanishing gradient\" problem in our work refers to gradients vanishing during *inference* (energy minimization w.r.t. the inputs), which is different from the vanishing gradient observed during traditional AI *training* (loss minimization w.r.t. the parameters).\n\nDespite their similarity to the ReLU, neither the GeLU or GeGLU are valid activations for our architecture for the following reasons:\n\n1. **Ge[G]LU violates a core tenet of Associative Memory** because it is not a monotonic function (i.e., its Jacobian is not positive definite), which means the energy is *not* guaranteed to monotonically decrease with gradient descent. We are happy to elaborate on this more if you have additional questions.\n2. **Ge[G]LU is not a proper kernel** since it has regions where it dips negative. This behavior of \"negative probability density\" usually has no clear meaning and application in KDE.\n\n> It is beneficial to follow LSE paper for realisitic quantitive evaluations, such as on classification problems\n\nWe appreciate the reviewer’s suggestion and agree that classification experiments, as in [2], can provide a valuable benchmark. In this work, however, we deliberately focus on a canonical evaluation for the \"memory\" aspect of DenseAMs --- the storage capacity and retrieval quality of stored patterns. Thus, our experiments validate LSR energy's ability to store patterns (Fig. 3 left) and the quality of those stored patterns (quantitatively in Fig. 3, right, and qualitatively in Fig. 4) relative to the LSE baseline. These experiments do not require training the stored patterns themselves. \n\nIt is an exciting research direction to incorporate training procedures such as backpropagation or contrastive learning into the LSR energy, and we expect both techniques to produce interesting research challenges of their own. We believe that training these architectures is an orthogonal research dimension to the contributions of this work, one that would be best explored in a dedicated follow‑up paper.\n\n> It may be non-trivial to stack multiple LSR to form deep networks of memories. I am wondering the potential of formulating and implementing this.\n\nThis is a fantastic observation. Indeed it is possible, though as you pointed out the mathematics are non-trivial. Following the methods of \"Hierarchical Associative Memory\" [1], we can make our energy function deeper by introducing both latent variables and new parameter matrices. Each parameter matrix is coupled to a latent variable and will store patterns observed in those latent variables. To extract information from these matrices, we can use an Epanechnikov kernel (or any other kernel discussed in Appendix B) to compute similarities between a \"query\" latent and its parameter matrix. This results in a new energy that represents a deep, intricately connected network of memories.\n\nHowever, this architecture is a large departure from the fundamental discussion in our submission and we will leave complete evaluation of this \"Hierarchical Epanechnikov AM\" to future work.\n\n[1] Krotov, \"Hierarchical Associative Memory\". ArXiv preprint, 2021\n\n[2] Krotov \\& Hopfield, \"Dense Associative Memory\". NeurIPS, 2016\n\n\n---\n\nWe thank you again for your insightful feedback and questions! If our response above satisfies the reviewer's original concerns, we would greatly appreciate an increase of score to reflect your increased confidence in our work."}, {"review_id": "q8zAP6Ga3H", "reviewer": "Reviewer_TqXz", "summary": "This paper introduces a novel energy function for DenseAMs, termed LSR, which is derived from the principles of optimal Kernel Density Estimation (KDE). By establishing a connection between AM energy functions and KDE, the authors propose using a ReLU-based separation function, analogous to the statistically optimal Epanechnikov kernel, as an alternative to the commonly used LSE formulation (which corresponds to a Gaussian kernel). The primary contribution of this work is to demonstrate that the LSR energy function resolves the fundamental trade-off between perfect memorization and generalization. The authors provide theoretical proofs and empirical evidence showing that LSR can achieve exponential storage capacity with *exact* retrieval of all stored patterns, while simultaneously creating a vast number of novel \"emergent\" local minima. These emergent memories, which arise as centroids of overlapping subsets of stored patterns, are a unique feature not present in LSE-based models under the condition of perfect recall. Experiments on synthetic data and latent spaces of image datasets validate that LSR produces a significantly richer and more diverse set of high-quality memories than LSE, showcasing its potential for both high-capacity storage and generative tasks.", "questions": "Typo: “textvol” in Proposition 3", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThis is a strong paper with great contributions. The central idea of leveraging insights from optimal KDE to design a new energy function for associative memories is original and insightful. The paper's main finding is a conceptual breakthrough for the field of associative memories. The phenomenon of \"global emergence\", where a model can simultaneously achieve exact memorization and generate a vast number of novel patterns, is attractive. It provides a more profound understanding of the memorization-generalization trade-off AMs. The paper is well-written and easy to follow. The motivation is clearly articulated, logically flowing from the limitations of existing AMs to the KDE-inspired solution.\n\n**Weaknesses:**\n\nThe emergent memories are mechanistically simple, being the centroids of subsets of stored patterns. The generative capability is therefore predictable and less expressive. The authors do not justify why the emergent memory is meaningful rather than harmful for real tasks. For example, while the paper regards emergent memories as a desirable and controlled form of generalization, these novel patterns are not explicitly present in the stored data. It would be beneficial to discuss the relationship between the emergent memory and model hallucination.", "was_revised": true, "final_justification": "I have carefully read the author's responses to my comments and the feedback from the other reviewers. The author has addressed all of my questions and concerns effectively. I am now confident in supporting the publication of this manuscript and will maintain my initial high score.", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper. We are encouraged by the positive scores! Reviewers across the board have highlighted the following strengths of our work:\n\n1. Regarding **novelty**, our work exploring the connections of KDE and Associative Memory is \"novel\" (Ua9r), \"original\" (TqXz), \"elegant\" (4hP4), and \"insightful\" (TqXz, 4hP4). It is a \"conceptual breakthrough for the field of associative memories\" (TqXz)\n2. Regarding **rigor**, our work is \"a principled solution to the memorization-generalization trade-off in DenseAMs\" (4hP4) that \"lays out a framework on studying emergent capabilities for future works\" (Ua9r)\n3. Regarding **quality**, the paper is \"well-written and easy to follow\" (TqXz) and \"already above the acceptance threshold\" (4hP4)\n\nReviewers also noted that several discussions/experiments would further improve our work:\n\n1. **Increase the scale of the current experiments** and confirm that the current conclusions hold (Ua9r)\n2. **Clarify when emergent memory is meaningful** rather than harmful (TqXz)\n3. **Discuss biological/neuroscience implications** (4hP4)\n4. **Elaborate on the experimental details for Fig. 4**, the VAE real-data experiments (4hP4)\n5. **Discuss emergence properties of other kernels** used for KDE (4hP4)\n\nWe believe the reviewer feedback has increased the strength of our submission, whose changes we summarize in our responses to each reviewer. Unfortunately, this year's NeurIPS rebuttal format prevents us from sharing the revised submission and updated figures. We do our best to provide clarification and describe results in plain text.\n\n---\n\n---\n\n> ...justify why the emergent memory is meaningful rather than harmful for real tasks.\n\nThis is a really good question, and one that is hard to answer without nuance. Indeed, it is difficult to know if a novel generation is \"good\" or \"bad\" --- it is more helpful to say if it is *useful* or not for a particular task.\n\nFor example, in Fig. 3 (right) we show that the LSR energy approximates an unknown PDF better than LSE's energy while simultaneously generating diverse samples. This is a *desirable* behavior of emergence, since high log-likelihood samples from the LSE energy are quite homogeneous, where 500 initial queries converge to the same ~10 memories. On the other hand, consider the novel memories from Tiny Imagenet in Fig. 4. Though plausible, many of the novel generations are blurry and would be considered *undesirable* from the perspective of an image generation model.\n\nWe include a more fleshed out version of this response regarding meaningful vs. harmful emergent memories in our camera-ready submission.\n\n> ...discuss the relationship between the emergent memory and model hallucination.\n\nThe reviewer raises an intriguing point about the connection between emergent memories and hallucination. In everyday use of LLMs, we call it a *hallucination* when the model confidently produces an incorrect fact, especially when we know the model is capable of producing the correct fact in other settings. But to reason about hallucination in the context of emergent memories, we must imagine a setting where the model has an explicit energy function and a known set of stored “facts.”\n\nThis is the setting of Associative Memory, where the outputs of our model are always energy minima (memories). A \"hallucination\" can occur when two or more stored facts interfere, leading the system to settle into an emergent minimum that looks meaningful (i.e., it lies near the \"factual manifold\" that generated our stored patterns) but does not correspond to any true, stored fact. Emergent memories are precisely these \"interference minima\" when they coexist with the original stored patterns. By definition, they are not stored facts, yet they can resemble meaningful combinations of them --- making them the *memory‑system analog of hallucinations*.\n\nThis said, we can imagine a highly structured latent space where interpolations between facts are facts themselves. For example, if stored patterns are facts representing logical statements, and the local average of two or more facts in a feature space corresponds to an `OR` operation, then that emergent memory would also be factual. But in general, emergent memories illustrate exactly how a system can produce plausible‑looking outputs that are not true to the stored data.\n\nWe believe this is an interesting discussion and provide a brief note of it in the appendix of our submission. Full discussion of \"hallucination from the perspective of emergence\" would require more rigorous definitions and exploration that is outside the scope of this work, but we share your interest in this relationship!\n\n> The emergent memories are mechanistically simple, being the centroids of subsets of stored patterns\n\nIndeed, the emergent memories are mechanistically simple. But the simplicity of emergent minima in the LSR energy is also a benefit. We can take advantage of this simplicity to develop efficient inference that allows us to converge to the *exact* fixed point in as little as a single step (Alg. 1). Meanwhile, one-step retrievals on the LSE energy converge to approximate fixed points [3]. We can also take advantage of the simplicity to efficiently predict candidate emergent memories without needing to explore the entire search space.\n\nWe also point out that, with enough stored patterns at the right $\\beta$, local linear interpolations can approximate globally curved manifolds. That is, by placing memories on the centroids of linear interpolations between any 2 (or 3, 4, etc.) data points, the LSR energy can be used to approximate arbitrary manifolds. Even deep ReLU networks approximate complex functions by learning a many linear approximations to curves. We elaborate on this in our camera-ready submission.\n\n> Typo: “textvol” in Proposition 3\n\nThank you for the catch! We have fixed this in our submission\n\n[3] Ramsauer et al., \"Hopfield Networks is All You Need\". ICLR 2021\n\n---\n\nWe thank you again for your insightful feedback and questions. If our response above satisfies the reviewer's original concerns, we would greatly appreciate an increase of score to reflect your increased confidence in our work."}, {"review_id": "hDJxBpxFXc", "reviewer": "Reviewer_4hP4", "summary": "This paper proposes a new energy function for Dense Associative Memory (DenseAM) models, called the Log-Sum-ReLU (LSR) energy, inspired by kernel density estimation (KDE) and particularly the Epanechnikov kernel. Classical and modern Hopfield networks rely on energy functions like log-sum-exp (LSE) to enable memory retrieval with high capacity. However, such models often trade off memorization for generalization. The authors show that LSR energy defies this trade-off: it enables exact retrieval of all stored memories and also generates emergent local minima—interpretable as novel “creative” memories—without compromising recall.\n\nTheoretically, the authors demonstrate that LSR energy preserves exponential capacity and enables global emergence, where novel attractors appear even while all original patterns are retrievable. This contrasts with LSE, which lacks this capability under any fixed temperature $\\beta$. The LSR energy is derived from a shifted ReLU separation function, corresponding to the optimal KDE kernel under certain assumptions.\n\nEmpirically, the paper validates these findings in toy datasets (uniform points in a hypercube), synthetic mixture-of-Gaussian tasks, and real-world datasets (MNIST and TinyImageNet), with memory retrieval occurring in VAE latent space. Experiments show that LSR generates more unique, semantically plausible memories than LSE, sometimes with better log-likelihood under the ground-truth density.", "questions": "I feel this paper is already above the acceptance threshold. However, I see the following potential opportunities that may promote this paper to a spotlight/oral one if the authors could include them:\n\n- Provide more experimental detail for Figure 4: how images are encoded and retrieved, how the number of preserved/emergent memories is counted, and what defines a preserved memory.\n- Justify or ablate the VAE. Can LSR energy retrieve and generate meaningful memories directly in image space (e.g., for MNIST)? If not, why?\n- Add a brief discussion in the introduction or conclusion about potential biological relevance—e.g., how different similarity kernels might exist across cortical circuits to balance memory and creativity.\n- Evaluate other kernels (from Appendix B) in the DenseAM energy function to test if similar results arise with kernels like quartic, triweight, or cosine.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The analogy between associative memory energies and KDEs is elegant and insightful. I enjoy reading how this paper is motivated by this analogy.\n- The LSR energy allows for perfect memorization and emergence of novel, creative attractors. This provides a principled solution to the memorization–generalization trade-off in DenseAMs.\n- Theorems and propositions throughout the paper clearly articulate properties of LSR energy, including capacity, convergence, and emergence. However, due to personal bandwidth and expertise I wasn't able to go through the whole appendix to examine the correctness of the proof. Therefore my recommendation on the theoretical part of this paper is conditional on confirmation by reviewers with appropriate background.\n\nWeaknesses:\n- The experiment details for figure 4 is insufficient. I'd love to see more information on the experimental pipeline, including how images are sampled, how the model is queried to generate (latent representations of) the images. It is also unclear to me why the VAE is needed here as denseAMs can directly perform memory retrieval in the image space. Maybe a sentence or so to justify the use of the VAE.\n- Only one KDE kernel (Epanechnikov) is tested in the associative memory context. Appendix B lists other well-known KDE kernels—testing a few of these in the DenseAM setting would help clarify whether the observed behavior is unique to Epanechnikov or more general. (i understand this paper focuses on Epanechnikov kernel; however, testing other kernels, at least in the appendix, could strengthen the theoretical findingds of this paper)\n- The biological/neuroscience implication of the findings is missing - the findings imply that the brain utilizes a unified computational principle (kernel/Hopfield networks) to perform both memorization and generalization. I feel this is an important implication biologically.", "was_revised": true, "final_justification": "This original submission was already above the acceptance threshold. However, it lacked clarity for its experimental details and justifications. It also lacked experiments that can enhance the argument. During the rebuttal, the authors have convincingly resolved the problems I have raised in the initial comments (although due to the rebuttal format requirements, I can only imagine the visual results based on the authors' descriptions). I therefore raised the score. However, I still can't assess the mathematical derivations in the appendix due to personal expertise and bandwidth, so retained my confidence score.", "rebuttal": "We thank the reviewer for their time and effort in reviewing our paper. We are encouraged by the positive scores! Reviewers across the board have highlighted the following strengths of our work:\n\n1. Regarding **novelty**, our work exploring the connections of KDE and Associative Memory is \"novel\" (Ua9r), \"original\" (TqXz), \"elegant\" (4hP4), and \"insightful\" (TqXz, 4hP4). It is a \"conceptual breakthrough for the field of associative memories\" (TqXz)\n2. Regarding **rigor**, our work is \"a principled solution to the memorization-generalization trade-off in DenseAMs\" (4hP4) that \"lays out a framework on studying emergent capabilities for future works\" (Ua9r)\n3. Regarding **quality**, the paper is \"well-written and easy to follow\" (TqXz) and \"already above the acceptance threshold\" (4hP4)\n\nReviewers also noted that several discussions/experiments would further improve our work:\n\n1. **Increase the scale of the current experiments** and confirm that the current conclusions hold (Ua9r)\n2. **Clarify when emergent memory is meaningful** rather than harmful (TqXz)\n3. **Discuss biological/neuroscience implications** (4hP4)\n4. **Elaborate on the experimental details for Fig. 4**, the VAE real-data experiments (4hP4)\n5. **Discuss emergence properties of other kernels** used for KDE (4hP4)\n\nWe believe the feedback has increased the strength of our submission --- we summarize our changes in our responses to each reviewer. Unfortunately, this year's rebuttal format prevents us from sharing the revised submission and updated figures. We do our best to provide clarification and describe results in plain text.\n\n---\n\n> I see the following potential opportunities that may promote this paper to a spotlight/oral one if the authors could include them\n\nThank you for your vote of confidence! You have dangled a carrot for us; we hope to satisfy your reasonable requests with our answers below.\n\n> Provide more experimental detail for Figure 4.\n\nWe flesh out the experimental methodology below for Fig. 4 below:\n\n**Designing the energy.** The DenseAM energies studied in this work are described by a matrix of stored patterns $\\Xi$ and an inverse temperature hparam $\\beta$.\n\nFor *MNIST*, the experiment proceeds as follows: 24 random images from the MNIST training set are normalized to be $[0,1]$, rasterized into a vector of shape `(784,)`, and projected into a 10-dim latent space of a $\\beta$-VAE trained according to the methods laid out in Appendix C.4 [L516 of our submission]. These latents become our stored patterns $\\Xi \\in \\mathbb{R}^{24 \\times 10}$ that we use to parameterize both the LSR energy and the LSE energy. We then use Alg. 2 to discover a $\\beta$ value such that these stored patterns interact in the LSR energy to exhibit *emergence* ($\\beta$ values that are too high will have no emergence, whereas $\\beta$ values that are too small will have few emergent/no preserved memories) --- this $\\beta$ value is also used to describe the LSE energy.\n\nFor *TinyImagenet*, we randomly select 40 images from the dataset, each of shape `(C,H,W)=(3,64,64)`. These samples are passed these through a pretrained VAE called TAESD to produce latents that are of shape `(4,8,8)` which are then rasterized into vectors of shape `(256,)`. Thus, our stored pattern matrix $\\Xi \\in \\mathbb{R}^{40 \\times 256}$. Other details remain the same as MNIST.\n\n**Discovering local minima** Fig. 4 shows **all** memories for each system. Local minima are enumerated differently for the LSR and LSE energy. To clarify App. C.4 [L535 of our submission], LSR memories are discovered by filtering the set of \"memory candidates\" --- the set of centroids of stored patterns whose basins overlap --- to those whose energy gradient is zero. This method is explicitly described in Alg. 3. LSE memories are computed via gradient descent, initializing queries at the stored patterns and minimizing the energy until convergence. The set of unique fixed points from this inference is the complete set of LSE memories.\n\nWe have clarified our submission with the details above.\n\n> Justify or ablate the VAE. Can LSR energy retrieve and generate meaningful memories directly in image space (e.g., for MNIST)? If not, why?\n\nThis is an excellent question. We use the VAE for one primary reason that we discuss in the Limitations of our submission and which we clarify further in our response to Reviewer TqXz: *An emergent minima is exactly the centroid of some collection of stored patterns*. \n\nIn MNIST pixel space, this results in memories that look like superpositions of digits: e.g., a \"1\" and a \"7\" displayed on top of each other, or a \"2\" that has been overlayed with a \"5\" and a \"3\" to create something like a blurry \"8\". In the structured latent spaces of deep models, we observe more interesting behavior. A \"5\" and a \"9\" can merge to become a \"7\", or two slanted \"1\"s will average out into a vertical \"1\", behaviors that do not occur when averaging in pixel-space. The \"meaningfulness\" of the emergent memory depends on the structure of the latent space.\n\nWe wish we could communicate the idea of pixel-space superposition through a figure in this response. It is an elegant way to visualize how LSR energy works in high dimensions. We have updated our submission to include a discussion of emergence in pixel-space in the Appendix. We are grateful to the reviewer for raising this question.\n\n> Evaluate other kernels... to test if similar results arise\n\nAnother excellent suggestion. Though our main focus was the Epanechnikov kernel, it turns out that the theoretical results in Prop. 2, Prop. 3, and Thm. 2 are *not* specific to this choice of kernel and indeed generalize to many kernels with *compact support*. This ensures the property that *local* averaging, rather than *global* averaging, can lead to emergence. The Epanechnikov kernel induces a localized interaction structure via the activation set $B(x) := \\left[ \\mu : |x - \\xi_\\mu|^2 < \\frac{2}{\\beta} \\right]$, where emergent memories are centroids of subsets of stored patterns. Other kernels in App. B  will lead to different forms of the activation set $B(x)$ and thus different (and possibly more complex) averaging/emergence. For example, the Triangle kernel leads to a new set $B(x) := \\left[ \\mu : |x - \\xi_\\mu| < \\frac{2}{\\beta} \\right]$. Results in Prop. 2, Prop. 3, and Thm. 2 are expected to hold with corresponding changes. Interestingly, all these kernels exhibit exponential memory capacity when viewed from the perspective of Associative Memory.\n\nTo demonstrate this behavior, we recreate Fig. 1 (left) to show 1D merging of two energy basins for every kernel in App. B (i.e., Triangle, Uniform, Cosine, Quartic, Triweight, and Tricube) at different $\\beta$ (bandwidth). This new figure shows what \"emergence\" looks like for each of them. We describe their behavior below.\n\n1. **Triangle**. The Triangle kernel exhibits very interesting emergence behavior. A perfectly flat energy manifold is formed where two or more basins overlap. If the energy of this manifold is lower than the energy of the stored patterns, the stored patterns \"merge\" and are no longer retrievable. If the energy of the manifold is higher, both original patterns are preserved and we observe a locally emergent *manifold* of memory.\n2. **Uniform**. A Uniform kernel produces an energy landscape that is not continuous and arguably impractical for associative memory. Emergent minima have exclusively lower energy than the stored patterns. Except for discontinuities, the entire energy landscape is flat.\n3. **Cosine**. The Cosine kernel looks remarkably like the Epanechnikov kernel, and its emergence properties are almost identical to that of LSR. It appears that Cosine-emergent memories have slightly higher energy than their Epanechnikov counterpart.\n4. **Quartic**. The Quartic kernel produces smoother energy landscapes than the Epanechnikov does. Overlapping basins does not guarantee emergent minima, but carefully tuning $\\beta$ discovers emergence. This behavior is an interesting departure from the previous kernels that will need to be studied more carefully.\n5. **Triweight**. The Triweight kernel behaves like the quartic kernel in that emergence is difficult to find. The closest we were able to discover is a nearly flat manifold connecting the two stored patterns where everything is at the same, flat energy.\n6. **Tricube**. Behaves like the Cosine kernel, but the energy is noticeably smoother. Tricube's emergent minima have higher energy than cosine's at the same bandwidth.\n\nWe include this discussion alongside a figure describing their behavior in our updated submission.\n\n> Add a brief discussion in the introduction or conclusion about potential biological relevance\n\nDense Associative Memories permit a standard mapping onto \"biological\" networks by introducing auxiliary hidden neurons following the method of (Krotov & Hopfield, ICLR 2021 - Ref [6]). Thus, the $d$ feature neurons in both LSE and LSR models can be augmented with auxiliary $M$ hidden neurons, resulting in a model with only pair-wise neuron interactions and bipartite connectivity between feature and hidden layers. This augmented model is more biological compared to the model considered in the submission. The proposed model defined by the energy Eq. (3) and the corresponding update rule can be obtained by integrating out these auxiliary hidden neurons, which means LSR energy will still have the same emergent behavior. The augmented model will still have some un-biological aspects because of the weight symmetry between forward and backward projections, which is necessary to ensure that the network has a global energy function. This discussion has been added to the revised version of the paper. \n\n---\n\nWe thank you again for your interest in this work and your helpful feedback! If our response above satisfies your original concerns, we would greatly appreciate an increase of score to reflect your increased confidence in our work."}]

Author rebuttal:

Meta-review: The paper proposes a form of associative memory networks represented by a new energy function, log-sum-ReLU (LSR), inspired by the connection between energy functions and probability density functions. All reviewers are positive about the submission. Leveraging insights from optimal KDE to design a new energy function for associative memories is considered both original and insightful, and the ability to achieve exact memorization while generating a large number of novel patterns is particularly attractive. Although some concerns were raised regarding the experimental results, these were satisfactorily addressed during the rebuttal. 

Overall, the paper makes a strong contribution by advancing our understanding of the fundamentals of generalization. It may be able to attract broad interest within the machine learning community. The AC therefore recommends clear acceptance with a spotlight presentation.
Decision: Accept (spotlight)