Title: Dense Associative Memory with Epanechnikov Energy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Associative Memory

Dense Associative Memory

Kernel Density Estimation

Epanechnikov kernel

energy-based models

exponential memory capacity

Hopfield Network

spurious minima
Abstract: We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Uniquely, it introduces abundant additional emergent local minima while preserving perfect pattern recovery --- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new energy function for Dense Associative Memory (DenseAM) models, called the Log-Sum-ReLU (LSR) energy, inspired by kernel density estimation (KDE) and particularly the Epanechnikov kernel. Classical and modern Hopfield networks rely on energy functions like log-sum-exp (LSE) to enable memory retrieval with high capacity. However, such models often trade off memorization for generalization. The authors show that LSR energy defies this trade-off: it enables exact retrieval of all stored memories and also generates emergent local minima—interpretable as novel “creative” memories—without compromising recall.\n\nTheoretically, the authors demonstrate that LSR energy preserves exponential capacity and enables global emergence, where novel attractors appear even while all original patterns are retrievable. This contrasts with LSE, which lacks this capability under any fixed temperature $\\beta$. The LSR energy is derived from a shifted ReLU separation function, corresponding to the optimal KDE kernel under certain assumptions.\n\nEmpirically, the paper validates these findings in toy datasets (uniform points in a hypercube), synthetic mixture-of-Gaussian tasks, and real-world datasets (MNIST and TinyImageNet), with memory retrieval occurring in VAE latent space. Experiments show that LSR generates more unique, semantically plausible memories than LSE, sometimes with better log-likelihood under the ground-truth density.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The analogy between associative memory energies and KDEs is elegant and insightful. I enjoy reading how this paper is motivated by this analogy.\n- The LSR energy allows for perfect memorization and emergence of novel, creative attractors. This provides a principled solution to the memorization–generalization trade-off in DenseAMs.\n- Theorems and propositions throughout the paper clearly articulate properties of LSR energy, including capacity, convergence, and emergence. However, due to personal bandwidth and expertise I wasn't able to go through the whole appendix to examine the correctness of the proof. Therefore my recommendation on the theoretical part of this paper is conditional on confirmation by reviewers with appropriate background.\n\nWeaknesses:\n- The experiment details for figure 4 is insufficient. I'd love to see more information on the experimental pipeline, including how images are sampled, how the model is queried to generate (latent representations of) the images. It is also unclear to me why the VAE is needed here as denseAMs can directly perform memory retrieval in the image space. Maybe a sentence or so to justify the use of the VAE.\n- Only one KDE kernel (Epanechnikov) is tested in the associative memory context. Appendix B lists other well-known KDE kernels—testing a few of these in the DenseAM setting would help clarify whether the observed behavior is unique to Epanechnikov or more general. (i understand this paper focuses on Epanechnikov kernel; however, testing other kernels, at least in the appendix, could strengthen the theoretical findingds of this paper)\n- The biological/neuroscience implication of the findings is missing - the findings imply that the brain utilizes a unified computational principle (kernel/Hopfield networks) to perform both memorization and generalization. I feel this is an important implication biologically.\n\nQUESTIONS:\nI feel this paper is already above the acceptance threshold. However, I see the following potential opportunities that may promote this paper to a spotlight/oral one if the authors could include them:\n\n- Provide more experimental detail for Figure 4: how images are encoded and retrieved, how the number of preserved/emergent memories is counted, and what defines a preserved memory.\n- Justify or ablate the VAE. Can LSR energy retrieve and generate meaningful memories directly in image space (e.g., for MNIST)? If not, why?\n- Add a brief discussion in the introduction or conclusion about potential biological relevance—e.g., how different similarity kernels might exist across cortical circuits to balance memory and creativity.\n- Evaluate other kernels (from Appendix B) in the DenseAM energy function to test if similar results arise with kernels like quartic, triweight, or cosine.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel energy function for DenseAMs, termed LSR, which is derived from the principles of optimal Kernel Density Estimation (KDE). By establishing a connection between AM energy functions and KDE, the authors propose using a ReLU-based separation function, analogous to the statistically optimal Epanechnikov kernel, as an alternative to the commonly used LSE formulation (which corresponds to a Gaussian kernel). The primary contribution of this work is to demonstrate that the LSR energy function resolves the fundamental trade-off between perfect memorization and generalization. The authors provide theoretical proofs and empirical evidence showing that LSR can achieve exponential storage capacity with *exact* retrieval of all stored patterns, while simultaneously creating a vast number of novel \"emergent\" local minima. These emergent memories, which arise as centroids of overlapping subsets of stored patterns, are a unique feature not present in LSE-based models under the condition of perfect recall. Experiments on synthetic data and latent spaces of image datasets validate that LSR produces a significantly richer and more diverse set of high-quality memories than LSE, showcasing its potential for both high-capacity storage and generative tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThis is a strong paper with great contributions. The central idea of leveraging insights from optimal KDE to design a new energy function for associative memories is original and insightful. The paper's main finding is a conceptual breakthrough for the field of associative memories. The phenomenon of \"global emergence\", where a model can simultaneously achieve exact memorization and generate a vast number of novel patterns, is attractive. It provides a more profound understanding of the memorization-generalization trade-off AMs. The paper is well-written and easy to follow. The motivation is clearly articulated, logically flowing from the limitations of existing AMs to the KDE-inspired solution.\n\n**Weaknesses:**\n\nThe emergent memories are mechanistically simple, being the centroids of subsets of stored patterns. The generative capability is therefore predictable and less expressive. The authors do not justify why the emergent memory is meaningful rather than harmful for real tasks. For example, while the paper regards emergent memories as a desirable and controlled form of generalization, these novel patterns are not explicitly present in the stored data. It would be beneficial to discuss the relationship between the emergent memory and model hallucination.\n\nQUESTIONS:\nTypo: “textvol” in Proposition 3\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores energy-based associative memory networks. A notable prior work is the LSE energy function, which yields exponential memory capacity. However, it typically faces a trade-off between perfect memorization of stored patterns and the generalization to novel patterns. This paper proposes a new energy function log-sum-relu (LSR) inspired by the connection between energy functions and probability density functions. Through some exploratory experiments, the paper shows promising demonstrations for exact memorization of all original patterns, and simultaneously generates new, meaningful emergent memories.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The paper provides a novel view regarding the connection between energy functions and KDE. \n\n* Theoretical contributions are made especially for generating novel emergent memories. This also lays out a framework on studying emergent capabilities for future works.\n\nWeakness:\n\n* The scale of the current experiment results is too small. For results shown in section 4.3, only a handful of less than 100 images are used for memorization. It remains unclear if the training images scale to thousands or millions, the current conclusion might not be hold.\n\n* The paper now lacks quantitative experimental evaluations. While theoretical contributions and empirical demonstrations are nice, it is beneficial to follow LSE paper for realisitic quantitive evaluations, such as on classification problems.\n\nQUESTIONS:\n* On the use of ReLU and its relevance to the modern deep networks. ReLU was proposed in part to facilitate gradient flows and avoid the gradient vanishing problem. Does ReLU play a similar role in LSR? How about studying other more advanced activation functions such as GeLU and GeGLU?\n\n* It may be non-trivial to stack multiple LSR to form deep networks of memories. I am wondering the potential of formulating and implementing this.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-17"}