A Biologically Plausible Dense Associative Memory with Exponential Capacity
Authors: Mohadeseh Shafiei Kafraj
Dmitry Krotov
Peter E. Latham
Venue: ICLR 2026 Poster (2026)
Keywords: Dense Associative Memory
Hopfield Network
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce a biologically plausible Dense Associative Memory model with exponential storage capacity.

Abstract: Krotov and Hopfield (2021) proposed a biologically plausible two-layer associative memory network with memory storage capacity exponential in the number of visible neurons. However, the capacity was only linear in the number of hidden neurons. This limitation arose from the choice of nonlinearity between the visible and hidden units, which enforced winner-take-all dynamics in the hidden layer, thereby restricting each hidden unit to encode only a single memory. We overcome this limitation by introducing a novel associative memory network with a threshold nonlinearity that enables distributed representations. In contrast to winner-take-all dynamics, where each hidden neuron is tied to an entire memory, our network allows hidden neurons to encode basic components shared across many memories. Consequently, complex patterns are represented through combinations of hidden neurons. These representations reduce redundancy and allow many correlated memories to be stored compositionally. Thus, we achieve much higher capacity: exponential in the number of hidden units, provided the number of visible units is sufficiently large relative to the number of hidden units. Exponential capacity arises because all binary states of the hidden units can become stable memory patterns. Moreover, the distributed hidden representation, which has much lower dimensionality than the visible layer, preserves class-discriminative structure, supporting efficient nonlinear decoding. These results establish a new regime for associative memory, enabling high-capacity, robust, and scalable architectures consistent with biological constraints.

Published as a conference paper at ICLR 2026
A BIOLOGICALLY PLAUSIBLE DENSE ASSOCIATIVE MEMORY WITH EXPONENTIAL CAPACITY
Mohadeseh Shafiei Kafraj
Dmitry Krotov
Gatsby Computational Neuroscience Unit University College London
IBM Research
krotov dmi try ..cm
London WIT 4JG, UK
mohadeseh. kafrajaraj..... 22uucl... ac uk
Peter E. Latham
Gatsby Computational Neuroscience Unit University College London
London WIT 4JG, UK
pel sellggatby...o www.syy..cc......ooo ac..uk
ABSTRACT
Krotov and Hopfield (2021) proposed a biologically plausible two-layer associative memory network with memory storage capacity exponential in the number of visible neurons. However, the capacity was only linear in the number of hidden neurons. This limitation arose from the choice of nonlinearity between the visible and hidden units, which enforced winner-take- thereby restricting each hidden unit to encode only single memory. We overcome this limitation by introducing novel associative memory network with threshold nonlinearity that enables distributed representations. In contrast to winner-take-all dynamics, where each hidden neuron is tied to an entire memory, our network allows hidden neurons to encode basic components shared across many memorics. Consequently, complex patterns represented through combinations of hidden neurons. These representations reduce redundancy and allow many correlated memories to be stored compositionally. Thus, achieve much higher capacity: exponential in the number of hidden units, provided the number of visible units is sufficiently large relative to the number of hidden units. Exponential capacity arises because all binary states of the hidden units can become stable memory patterns. Moreover, the distributed hidden representation, which has much lower dimen- sionality than the visible layer, preserves class-discrime discriminative structure, supporting efficient nonlinear decoding. These results establish new regime for associative memory, enabling high-capacity, robust, and scalable architectures consistent with biological constraints.
dynamics in the hidden layer,
INTRODUCTION
Associative memory networks are class of attractor models in which the system can recall stored memories from their incomplete or noisy versions via recurrent dynamics (Krotov et al. (2025) ) In such models, memories are conceptualized as the stable fixed points of the network dynamics. The number of fixed points determines the storage capacity of the network, and significant effort has been made construct networks with sufficiently high storage capacity to explain human memory.
The classical Hopfield network, leading model for associative memory, has storage capacity that scales linearly with the number of neurons in the network (Hopfield (1982).. Dense associative memories, sometimes also referred to modern Hopfield networks (Krotov and Hopfield (0166)) are promising modifications of the classical Hopfield model. By incorporating higher-order interactions (e.g., interactions that quadratic, rather than linear, in the input neuron), they achieve storage capacity that scales super-lineane with the number of neurons. There are many possible choices for the energy function in this class of models. For instance, the power interaction vertex leads to the

---

Published as a conference paper at ICLR 2026
power-law scaling of the capacity (Baldi and Venkatesh (1987): Gardner ( (1987); Abbott and Arian (1987); Horn and Usher [1988); Chen et al. (986); Krotov and Hopfield (2016)). More sophisticated shapes of the energy function result in exponential storage capacity in the number of neurons, while maintaining large basins of attraction (Demircigil et al.
2017): Lucibello and Mézard (2024)..
The naive implementation of Dense Associative Memory models, however, relies on synaptic inter- actions that are challenging to implement in biological circuits. In particular, these models require nonlinear interactions among synapses. While several biological mechanisms could in principle sup- port restricted forms of higher-order interactions, such as astrocytic processes, dendritic computations, or distributed neurotransmitter effects, these remain limited in scope and dictate strong constraints on the possible shape of the energy function (Burns and Fukai (023); Kozachkov et al. (2025); Kairaj et al. (202)). The implementation of Dense Associative Memory introduced by Krotov and Hopfield (2021) does not suffer from these limitations, as it relies only on standard synaptic interactions. In this architecture, the visible neurons correspond to features of the patterns, while the hidden neurons serve as auxiliary computational elements that mediate complex interactions. Higher-order interactions among visible neurons emerge by selecting appropriate activation functions for the hidden neurons.
Nevertheless, the implementation of Krotov and Hopfield has two key limitations. First, the storage capacity is at most linear in the number of hidden neurons (Krotov (202I): Krotov and Hopfield (202I). This is unsatisfactory from the perspective of information storage one would like to store as much information as possible while utilizing only a small number of neurons. Second, at inference time, the network demonstrates winner-take-all behavior. This means that the asymptotic fixed point that the network converges to corresponds to single hidden neuron being activated, while the rest of the hidden neurons are inactive. This behavior results in grandmother- representations for hidden neurons, as opposed to distributed representations, which are more efficient at storing information.
Our work tackles these two limitations. Specifically, we present a novel implementation of Dense Associative Memory that achieves exponential storage capacity in the number of hidden neurons. This is accomplished with a simple yet critical change: we use threshold activation function that does not enforce winner-take-all dynamics. The threshold activation enables distributed memory coossentatios-mullipe
hidden neurons can be active for a memory, and each hidden neuron can participate in multiple memories. As result, all possible binary patterns of hidden neuron states become stable fixed points, enabling the network to store exponentially many memories, including highly correlated ones. Beyond high capacity, the hidden layer of the network is low- dimensional compared to the visible layer, yet it produces structured representations that preserve clas-discrimiinatiee information, with memories sharing components represented close together in the hidden activity space. We establish this result through both theoretical analysis and numerical simulations, and show that the resulting fixed points also possess large basins of attraction.
Our model is closely related to the framework recently proposed by Chandra et al. (2025), which combines multiple Dense Associative Memory modules to produce a distributed code for the visible neurons. In that work, each module performs a winner-take-all operation similar to Krotov and Hopfield (202T). so only a single hidden neuron is active per module. By combining several modules, though, they achieve exponential storage capacity. However, we show that multiple modules are unnecessary: exponential capacity can be achieved with a single module, provided the activation function is chosen appropriately.
Beyond its biological motivation, our work also connects to a growing body of research on Dense Associative Memories in machine learning. Notably, it has been shown that Dense Associative Memory closely corresponds to the attention mechanism in transformer architectures (Ramsauer et al.. (202T): Hoover et al. (2023a)), offering a principled framework for viewing the transformer's s attention and feedforward computations as steps in a global energy minimization process. Complementary research has demonstrated that generative diffusion models, widely used in high-quality image generation, also exhibit associative memory behavior (Hoover et al. (023b): Ambrogioni (2024);
Pham al. (0255). Further studies have expanded the model's functionality: for instance, (Chaudhry et al. (2023) examined its ability to store and retrieve long sequences; Dohmatob (2023) and Hoover al. (2025) proposed alternative energy functions that also support exponential storage capacity; and Burns and Fukai (2023) introduced higher-order simplicial interactions. Our results contribute to
this line of work by showing how exponential storage capacity can be achieved within a biologically plausible two-layer framework, thereby bridging theoretical neuroscience with modern machine learning architectures.
2

---

Published as a conference paper at ICLR 2026
In the following sections, first formally define the model and its dynamics and derive the optimal threshold analytically for a network with fixed weights. We then present a theoretical analysis of storage capacity and basins of attraction, showing that the network exhibits large basins of attraction, making recall robust to substantial noise in the visible inputs. Next, we introduce a learning rule for storing real, correlated memories, enabling compositional memory storage, and present numerical experiments on MNIST and CIFAR-I0 that demonstrate high-capacity recall, structured hidden representations, and robustness noise. Finally, we conclude by discussing the biological plausibility of the network and potential directions for extending the model to incorporate additional constraints and more realistic neuronal properties.
MODEL
In this section present model and demonstrate that its storage capacity scales exponentially with the number of hidden neurons, meaning that all possible binary patterns of hidden neurons are stable fixed points.
We first define the dynamics of the system as follows,
dv;
N U/
To
+
Sip O12 0)
(la)
dt
VNn
p=1
dh Th dt
VNA
N.
h4
Sus
(Ib)
N
where ((-) is the standard Hcaviside step function,
[0
=
((2)
(2)
[1 i 0.
The parameter will be chosen to ensure the stability of all binary patterns in the hidden layer.
The network consists of No visible neurons (the v.) and N,, hidden neurons (the h,), arranged in bipartite architecture, i... without lateral connections within either layer.
Synaptic connections between visible neuron i and hidden neuron μ are symmetric and drawn randomly from standard normal distribution,
Eμi Eip (O,
The scaling factors in front of the sums in Eq. T are chosen purely for convenience, as they simplify subsequent expressions. Additionally, for simplicity, theoretical analysis and experiments assume a Heaviside step function; however, Appendix shows that this assumption not required.
2.1 STORAGE CAPACITY
To determine the storage capacity, we'll first focus on the fixed points of the dynamics given in Eq. IT. Defining
sμ (h, 0),
(4)
it is straightforward to show that in steady state, 8p satisfies
8p θ
Σ J &a
(5)
where
J =
EuoSav
(6)
N
3

---

Published as a conference paper at ICLR 2026
Equation with the weight matrix given in Eq. (6). is very close to the classical Hopfield model; the only difference is that in the classical model, the Eus are binary, whereas in our model they re Gaussian. However, the classical Hopfield model works in the regime N, < NN, with memory storage possible only No < O. 1388 (Amit et al. (1985). Here, though, we' II consider a very different regime: N, > Nn. In this limit, J approaches the identity matrix (Marchenko and Pastur (1967)), which decouples the hidden neurons. Assuming the threshold, 0, is chosen correctly, this leads immediately to exponential storage capacity.
Exponential capacity clearly holds in the limit N. - Jc. What happens when N, is finite? We show in Appendix A.1 that
Ju 8nm Sun VNn
(7a)
Suv ~ NNO, 1 6pw))
(7b)
Here and in what follows 6nv is the Kronecker delta: it's 1 if μ v and 0 otherwise. Thus, Eq. can be written as
(5)
84
+
S4
SarSy -8
(8)
Because the Sno are independent, the second term in parentheses, which we denote qir scales as (see Eq. (24) in Appendix A.
No Sumse
Σ $5 (11+)pu)5 6pme
No.
V N
NA +1
qul
N
N₂
where the second inequality follows because S, is either 0 or 1.
we set f 1/2, in the limit N, >> N₁, Eq. (8) typically has two solutions: one Sy and one
at Sy = 1. However, if 9u > 1/2 then the only solution is sp = 1, and if 9₂ < -1/2 then the only
solution is 8, In both cases there can be "bit flip", where 8n is forced to the wrong value. In Appendix D (see in particular Eqs. (40) and 36 but with
0) we show that
NA +
Poo flips N
(10)
N
VT/2
This bound approaches 1 exxoneetiallyy rapidly as the number of visible units, Ne, increases.
There are exponentially many fixed points, but are they stable? To answer that, we need to do stability analysis. Combining Eq. (T) with the definitions of su. Eq. (4), and Juws Eq. (6) we have
hy Σ Juusy.
(11)
Since Juw. is approximately the identity matrix, Because our nonlinearity a step function, its derivative vanishes at equilibrium, guaranteeing the stability of the fixed points (Appendix [A.2). Consequently, when solve Eq. @ we expect to
that at equilibrium h, is close to either 0 or
see 2 stable fixed points in the regime N, > Nn. This prediction is consistent with numerical simulations, as can be seen in Figure Th.
2.2 BASINS OF ATTRACTION
Although the fixed points are stable, that still leaves the question: how big are the basins of attraction? We'll assume that noisy input enters the network via the visible units, and initially all the h are zero. How far from the fixed points can the input be and still be recalled perfectly?
Assuming the noise additive, the initial values of the visible and hidden neurons are,
(0) = 1 N₁
Eup ( - 0) +
(12a)
/ /Nx
(0) = 0
(12b)
4

---

Published as a conference paper at ICLR 2026
where Rug.tage 1 if neuron il encodes the target memory, and 0 otherwise (motivated by the fact that by is close to either 0 or at the fixed points; see Eq. (II) and the noise, ej is independent and normally distributed with variance oy
~~~N(0.03
(13)
To reach the target fixed point in both the hidden and visible unit space, the hidden neurons must evolve to their target values, target before the visible units change much. That requires the visible units to evolve much more slowly than the hidden units, which we can guarantee by setting TA < T
(see Appendix G ). With this condition, a time satisfying 7h & Te. hy reaches equilibrium
while (t) is still approximately equal to (0). Using Eq. Ia with du, dt = 0 along with Eq. (
that equilibrium is given by
VNn
(t)
Ine
Jur O (hr, 0)
E + O(t/T.).
(14)
N₂
Using Eq. (7a). we see that the first term is O (1 scales o. V No. NN.No Thus, so long
0) + O NN.N.,) And the second term
N
(15)
N
ha(t) (t) will be close to its target value when f < Te. Since (t) is close to its target value at that time, will stay close, and asymptotically the target pattern will recovered. Given that N₂ >> N₁, the added noise be very large without affecting recall. Thus, the basin of attraction is very large (see Figure [Tp and Appendix
a
Theery: log, (2%))- -N.
Theory: =2 No
Sim: 8 0.5
loss (Capacitty) S
Sim: a, -1
loqa (Capacity)
Sint: 0,=3
Sim: =
Figure 1: Capacity versus the number hidden units, Nr. with N, 100N, and T, 207 (a) Capacity for different thresholds, 0. The highest storage capacity is achieved when the threshold is set its optimal theoretical value 0.5. (b) The effect of noise in the visible layer (c)' in Eq. (12a)). shown for different noise variances, demonstrates the large basin of attraction of the fixed points.
2.3 BIOLOGICAL PLAUSIBILITY
Compared to Krotov and Hopfield (202T), , our model exhibits greater biological plausibility several respects.
The activation function used here is local and keeps neuron activity within a biologically realistic range. In contrast, in Krotov and Hopfield (2021), Model A is not biologically plausible because the hidden neuron activities can grow to unrealistically large values as a consequence of the power- law activation, which does not reflect realistic neural firing. Models B and C, on the other hand, rely on non-local activation functions, softmax and spherical normalization, respectively, which biologically implausible unless additional mechanisms asee Appendix C for more details).
Although theoretical analysis focuses symmetric weights and global threshold for all neurons, these assumptions are not restrictions of the model. Experimentally, we show that networks with asymmetric weights and heterogeneous thresholds also achieve stable recall. Allowing asymmetric
5

---

Published as a conference paper at ICLR 2026
weights is important because exact symmetry is rarely observed in biological neural circuits. Similarly, heterogeneous thresholds capture the variability in neuron excitability across real neurons, and demonstrate that stable memory dynamics do not require finely tuned, uniform parameters. Together, these features indicate that our model better reflects realistic neural mechanisms while retaining associative memory functionality. Figure[gin Appendix (apeedix[E[hhane shows representative recall examples for networks with asymmetric weights and heterogencous thresholds that stored MNIST and CIFAR-10 images, respectively, using learning rule similar to that discussed in Section 3.1
RESULTS
3.1 LEARNING RULE
So far we have focused on storage capacity with fixed synaptic weights. A natural next step is to understand how these weights can be learned. In this section, we introduce learning rule that reflects compositional learning: small number of simple, reusable components can be combined to form complex patterns, and conversely, complex patterns can be decomposed into simpler components.
In steady-state state, visible activity in Eq. can be expressed as
N
Su5p
(16)
where f,, is the p-th column of E E NoNN,, i.c., (En)s= = Eiu. If only hidden neuron p is active, the visible state equals e,. A visible memory called basic if corresponds to a single active hidden neuron, and complex if it is formed by the activation of multiple hidden neurons, i.e. composition of several basic memories.
The goal of learning is to find synaptic weight matrix and threshold 0 such that set of target memories {vm IR No
} Jm= approximately correspond to stable fixed points the network dynamics, with M >>>>>> Nn (e.g., MNIST or CIFAR-10). This is achieved using the following optimization procedure,
=)(""
(NN E Vm 8)
(5,0) arg min E
Σ
E. O
(17)
V
No
where we have replaced with its target steady-state value. This learning rule is identical to the one proposed in |Radhakiishnan et al. (2020). We used Xavier initialization for the weights and approximated the threshold function é with sharp sigmoid to allow gradient-based bbaeed training.
3.2 EXPERIMENTS
Figure 2 shows recall results after storing 60,000 MNIST digits with N, = 784 and N₁ = 50. Despite the high correlation among patterns, the network learns 57913 unique minima corresponding to the 60,000 stored images. Variants of the same digit produce hidden representations that are distinct yet partially overlapping, and the recalled visible states remain recognizable.
Figure 3] shows the learned basic memories for the MNIST dataset, which correspond to the columns of As shown in Figure Bp, these basic memories are nearly orthogonal, consistent with Eq. (7a).
To evaluate the generality of the proposed learning rule beyond MNIST, we applied the same procedure the CIFAR-10 dataset. In this case, N, 3072 (3 32 32), and compensate for the increased complexity of this dataset, we used network with 500 hidden neurons, compared to 50 hidden neurons for MNIST. Figure presents examples of the cues alongside their recalls. These results show that the network is able to reconstruct interpretable outputs from the learned representations, despite storing a large number complex memories (50,000) and significantly violating the condition N, > N₁. Importantly, these images are highly correlated, yet the network produces 49982 unique stable minima corresponding to the stored memories, with each memory representation being both stable and interpretable.

---

Published as a conference paper at ICLR 2026
Cue 6
Recall
Cue 4
Recall
ceeeeee ooooooo www.0000000ooooooo ooooooo
BOCCRCOO DO OOCOSe Pocossseange cooooo
RO0.00.. 3
CODCEOO GOODD
aoco .
Examples of recall in a network with 50 hidden neurons that memorized 60,000 MNIST images. Hidden neurons are shown on the ring, and visible neurons visualized as two-dimensional images. On the ring, black indicates high activity, and white indicates low activity. Highly correlated images of every digit, for instance, the digit 6 shown here, converge to unique but overlapping hidden representations.
Figure 2:
a
-10
08
06
Hidden Units
Unseen Cue
Recall
6
6
Figure 3: a) 25 (out of 50) columns of the learned weight matrix for MNIST images, which serve as basic memories, shown two-dimensional images. b) Correlation matrix of the basic patterns, which correspond to the hidden units. c) The network generalizes compositionally, associating unseen cues with interpretable fixed points.
The learned basic memories for CIFAR-10 images are shown Figure 5a. They form a more heterogeneous set, yet remain nearly orthogonal, as shown in Figure 5p.
The network learns a global threshold = 0.21 for MNIST and = 0.43 for CIFAR-10. Note that the statistics of the leamed basic memories in MNIST and CIFAR- 10 differ from another

---

Published as a conference paper at ICLR 2026
Cue www.ooooooooooooooooooooooo
Recall
Cue
Recall
Cooooo 2O00000000 DOO000
BOODOO
OODD OOOOO COODO
OOODO OO
COOOOO OOODOO WOODOOOOOOOOOO 1000000000000 ROOCOOooooo GOODOOO DODER
0000000000 Co00 T0 w0w.000000000oooo.oo www.oooooooooooooo
ooo oeoeo 0000 foDDoc OODD
Figure 4: Examples of recall in a network with 500 hidden neurons that memorized 50,000 CIFAR-10 images. Hidden neurons are arranged on ring (50 out 500), while visible neurons are shown as two-dimensional images. On the ring, black indicates high activity, and white indicates low activity.
and from the normal distribution assumed in the theory, which explains the difference between the learned thresholds.
a
Hidden Units
Hidden Units
Unseen Cue
Recall
Figure 5: a) 25 (out of 500) columns of the learned weight matrix for CIFAR- 10 images, which serve as basic memories, are shown as images. b) Correlation matrix the basic patterns, which correspond to the hidden units. c) The network generalizes compositionally. associating unseen cues with interpretable fixed points.
For this system to function as an associative memory, new "unseen" cues should converge toward the approximately correct fixed points. For example, cues related to dogs should end up fixed points associated with dogs, not horses. As shown in Figure B and FigureBp. this behavior is indeed observed. The network converges to the nearest minimum of the energy landscape relative to the cue.
8

---

Published as a conference paper at ICLR 2026
Importantly, when the learned basic memories are expressive enough, this nearest minimum can correspond to a stable representation that is very close to the unseen cue itself rather than to a memorized pattern. In other words, the network not only memorizes but also generalizes: the learned basic memories shape the energy landscape so that unseen inputs are mapped to stable attractors that capture their distinctive features.
For example, in Figure Bp. when two unseen images of the digit "6" are presented, the network converges two distinct attractors that preserve the distinguishing details of each input while still sharing overlapping components in the hidden layer that identify them as class "6". This illustrates an advantage of our model: it supports both memorization and generalization through its learned basic components.
To quantitatively evaluate this behavior, we trained nonlinear classifiers on the recalled hidden and visible representations of the stored images, and tested them on the recalled representations of unseen images (Convolutional neural network (CNN) for visible representations and multilayer perceptron (MLP) for hidden representations). For comparison, we trained CNNs directly on the original stored images and tested them on the original unseen images as well. This allows us to assess the classifiabilityy of the recalled visible representations by the associative memory network relative to the original memories. (see Appendix B for details of the classifiers)
For MNIST, classification accuracy is high for both the hidden and visible representations. This is a desirable property, as it indicates that the lower-dimensional hidden representations still preserve strong class discriminability. In MNIST, the raw pixel space itself carries strong class structure: two images of the same digit are highly correlated and closer to each other than images of different digits. Consequently, the hidden neurons retain this information almost perfectly, having structured and meaningful encodings in which correlated memories are represented close together and remain classifiable.
For CIFAR-10. 10, classification accuracy is higher for the visible representations than for the hidden ones. This difference arises because, in CIFAR-10, two images of the same class (for example, dogs) are not necessarily correlated in raw pixel space, so the hidden layer, which is a nonlinear transformation of those pixels, does not exhibit clear class structure. The classifier used for the visible representations in this analysis is a CNN, which, when trained on raw pixel data, first learns nonlinear transformation that maps images of the same class close together a learned feature space while separating images from different classes. After this transformation, classification is performed using a linear decision boundary in that space. This ability to internally build such class-specific representations explains why classification accuracy remains higher for the visible neurons.
Overall, the high performance on the visible representations for MNIST, and the reasonably high performance for CIFAR- 10, demonstrates that the recalled representations remain dlass-iicriminative. and that the associative memory preserves the essential structure of the data, enabling compositional generalization to unseen examples (for the CIFAR10 dataset, the classification accuracy can be increased by scaling up the associative memory, including N₁. the epoch size, the optimization step, and the number of training samples).
Representation
MNIST Accuracy CIFAR-10 Accuracy 40%
Recalled Hidden Patterns Recalled Visible Patterns
95%
98%
56%
Original Images
99%
88%
Table I: Classification test accuracy of nonlinear classifiers trained and tested on recalled hidden and visible representations, as well as on the original images for comparison, for MNIST and CIFAR- 10. Reasonably high accuracy visible representations for both datasets demonstrates that the recalled representations remain highly class-diseimminatie,, while the lower accuracy on hidden representations for CIFAR-10 reflects the lack of strong class structure in raw pixel space.
4 CONCLUSIONS
This work introduces a novel Dense Associative Memory Krotov and Hopfield (001) that achieves exponential storage capacity in the number of hidden neurons, overcoming the limitations of previous

---

Published as a conference paper at ICLR 2026
two-layer models. By using threshold activation function, with theoretically derived threshold, the network supports distributed hidden representations, allowing each hidden neuron to participate in multiple memories. This enables compositional storage of complex and correlated patterns, reducing redundancy while maintaining robust retrieval.
Notably, the network achieves exponential capacity, 2N using only N, parameters. In contrast, previous two-layer implementations were limited to a maximum capacity of N₂ (Krotov and Hopfilld (202T). As a result, in the proposed network, the number of memories per weight grows as 2
N. R 2N,,, while in previous implementations it is at best J. N, Even for complex datasets such as MNIST and CIFAR-II 10, networks with only and 500 hidden units, respectively, were able to store tens of thousands of highly correlated memories and associate the vast majority of them with unique
minima, whereas previous models could not store more memories than the number of hidden units (see AppendixC).
Beyond storing exponentially many memories, the network is also able to generalize to novel inputs. This behavior arises because the hidden layer encodes set of basic memories that be flexibly composed to represent previously unseen patterns, associating them with distinct minima rather than forcing retrieval toward the nearest stored pattern, while still producing meaningful, class-consistent representations. Our results are consistent with biological principles of feature learning, as embodied in hierarchical predictive coding models, which detect novelty and generalize by recombining features across successive levels of abstraction (Li et al. (2025); Salvaton et al. (202I)). This mechanism further illustrates that learning the underlying compositional structure of naturalistic data enables a biological associative memory to effectively support both memorization and generalization.
The model biologically grounded, relying solely on standard pairwise synapses and a local activation function. We also provide evidence that it achieves stable recall even in the presence of asymmetric weights and heterogeneous neuronal thresholds. Moreover, the hidden layer forms low-dimensional representations that preserve with shared components close together in activity space. This structured organization supports efficient nonlinear decoding.
information, placing memories
Overall, this work establishes a new regime for associative memory that combines high capacity, robust recall, compositional and interpretable representations, and biological plausibility. It provides a theoretical foundation for scalable memory systems that bridge neuroscience models with modern machine learning architectures.
Future work will focus on developing a biologically plausible learning rule and on examining the model's is capacity under additional biological constraints, including sparse connectivity and adherence to Dale's s law.
ACKNOWLEDGMENTS
We thank Brendan A. Bicknell and Reidar Riveland for helpful discussions. We acknowledge the following funding sources: Gatsby Charitable Foundation (GAT3850) to MSK and PEL. The results presented here were obtained while Dmitry Krotov was employed by IBM Research, At the time of the camera-ready submission Dmitry Krotov is no longer employed by IBM Research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Laurence F Abbott and Yair Arian. Storage capacity of generalized networks, Physical review A, 36 (10):5091, 1987.
Luca Ambrogioni. search dispersed memories: Generative diffusion models are associative memory networks. Entropy, 26(5):381, 2024.
Daniel Amit, Hanoch Gutfreund, and Haim Sompolinsky. Storing infinite numbers of patterns in spin-glass model of neural networks. Physical Review Letters, 55(14): 1530, 1985.
Pierre Baldi and Santosh S Venkatesh. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58(9)9113, 1987.
Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023.
Sarthak Chandra, Sugandha Sharma, Rishidev Chaudhuri, and lla Fiete. Episodic and associative memory from spatial scaffolds in the hippocampus. Nature, 2025.
Hamza Chaudhry, Jacob Zavatone- Veth, Dmitry Krotov, and Cengiz Pehlevan. Long sequence hopfield memory. Advances in Neural Information Processing Systems, 10344000-54400 2023.
HH Chen, YC Lee, GZ Sun, HY Lee, Tom Maxwell, and C Lee Giles. High order correlation model for associative memory. In AIP Conference Proceedings, volume 151, pages 86 86-99, 1986.
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On model of associative memory with huge storage capacity, Journal of Statistical Physics, 168, 2017.
Elvis Dohmatob. A different route to exponential storage capacity. In Associative Memory (\&) Hopfield Networks in 2023, 2023.
Elizabeth Gardner. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20:11)334 1987.
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances in neural information processing systems, 36:27532-27559, 2023a.
Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffman, Zsolt Kira, and Duen Horng Chau. Memory in plain sight: survey the uncanny resemblances between diffusion models and associative memories. In Associative Memory { &{&} Hopfield Networks in 2023, 2023b.
Benjamin Hoover, Zhaoyang Shi, Krishnakumar Balasubramanian. Dmitry Krotov, and Parikshit Ram. Dense associative memory with epanechnikov energy. arXiv preprint arXiv:2506.10801, 2025.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79:2554-2558, 1982.
Horn and M Usher. Capacities of multiconnected memory models. Journal de Physique, 49(3):
389-395, 1988.
Mohadeseh Shafiei Kafraj, Dmitry Krotov, Brendan Bicknell, and Peter Latham. A biologically plausible associative memory network. In New Frontiers in Associative Memories, 2025.
Leo Kozachkov, Jean-Jacques Slotine, and Dmitry Krotov. Neuron-astrocyte associative memory. Proceedings of the National Academy of Sciences of the United States of America, 122(21):
e2417788122. 2025.
Dmitry Krotov. Hierarchical associative memory. arXiv preprint arXiv:2107.0644 2021.
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.

---

Published as a conference paper at ICLR 2026
Dmitry Krotov and John Hopfield. Large associative memory problem in neurobiology and machine learning. International Conference on Learning Representations, 2021.
Dmitry Krotov, Benjamin Hoover, Parikshit Ram, and Bao Pham. Modern methods in associative memory. arXiv preprint arXiv:2507.06211 2025.
T Ed Li, Mufeng Tang, and Rafal Bogacz. Predictive coding model detects novelty on different levels of representation hierarchy. Neural computation, 37(8):1373 1373-1408, 2025.
Carlo Lucibello and Mare Mézard. Exponential capacity of dense associative memories. Physical
Review Letters, 137(7),07301, 2024.
VA Marchenko and Leonid A Pastur. Distribution of eigenvalues for some sets of random matrices. Mat. Sb.)N 72(114):4 1967.
Bao Pham, Gabriel Raya, Matteo Negri, Mohammed J Zaki, Luca Ambrogioni, and Dmitry Krotov. Memorization to generalization: Emergence of diffusion models from associative memory. arXiv preprint arXiv:2505.21777, 2025.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterizee neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117, 2020.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, al. Hopfield networks is all you need. In International Conference on Learning Representations, 2021.
Tommaso Salvatori, Yuhang Song, Yujian Hong. Lei Sha, Simon Frieder, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. Associative memories via predictive coding. In Proceedings of the 35th International Conference Neural Information Processing Systems, pages 3874-3886, 2021.
12

---

Published as a conference paper at ICLR 2026
A
TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
A.1
DISTRIBUTIONAL PROPERTIES OF Sur
Combining Eqs. 66) and 7a we see that Sav given by
Saili
NN88ur
(18)
where, recall, the īμr independent, zero mean unit variance Gaussian random variables (see Eq. (3)). Thus, for & p, each product EaaCio is a zero-mean, unit variance random variable. By the central limit theorem, the sum of these N, independent terms converges in distribution Gaussian, so for # D we have
Suv 4, N(0,
(19)
For = D, we write
N Spi - 1).
Spp
(20)
By the central limit theorem, this too is Gaussian, but with slightly larger variance,
Sup 4 NO, Varlg?]) (N,
(21)
Combining Eqs. 19 and (21 we arrive
Spu (0,11+6mu)
(22)
Now consider the random variable qa defined as
4p
Spu8
(23)
This is random variable with respect the index y with sy fixed. Its variance is given by
N₂
1 No
Var
Var(
1 N
Σ (1 + 8pm)
(24)
N
N
where used the fact that the Gov independent random variables with mean O and variance given in Eq. 22
A.2 STABILITY OF THE FIXED POINTS
The stability of fixed points is determined by the Jacobian of the system. Grouping the variables into (v, h), corresponding the visible and hidden units respectively, the Jacobian, denoted A, has the block structure
A nh
A
(25)
A he
A
For the diagonal blocks, consider first the visible units. We have
Dü,
11,
= i
A = -IIN
(26)
0. j
13

---

Published as a conference paper at ICLR 2026
where IN. is the No Ne identity matrix. Similarly, for the hidden units,
Ohy [-1.
1, p.
Am
INN,
(27)
oh, 0,
V 5 p
where IN, is the N₂ x N₁ identity matrix. For the off-diagonal blocks, the derivative of the Heaviside step function in Eq. everywhere,
is zero almost
e'(=)
= 0.
(28)
Therefore, away from threshold crossings (h4 o 0) in Eq. la
Oi:
ph 0.
(29)
The hidden dynamics depend explicitly on the visible variables:
Ohy
Exr:
Abe = (Epz).
(30)
0c,
Putting everything together, the Jacobian is lower-triangular,
IN=
A
(31)
A
INN
The eigenvalues of a triangular matrix are its diagonal entries, which in this case are equal to Hence, all fixed points of the dynamics are stable.
B DETAILS OF THE CLASSIFIERS
Table 2: Architecture and Parameters of the CNN Classifier
Block
Layer Type
Channels Filters
Kernel / Pool
Activation
Conv Block
2 x Conv2D BatchNorm22
- 32
3 x 3, MaxPool(2)
ReLU
Conv Block 2
2 Conv2D BatchNorm22
32 - 64
3 3, MaxPool(2)
ReLU
Conv Block
2 Conv2D BatchNorm22
64 128
3 MaxPool(2)
ReLU
Flatten
Auto- compuoe (f)
Fully Connected 1
Lincar
f - 128
ReLU
Fully Connected 2
Lincar
128 + Nelasacs
Softmax
Table 3: Architecture and Parameters of the MLP Classifier
Layer
Type Dimensions / Units Activation
Input
Linear
Input dimension =d
Hidden Layer
Linear
d 256
ReLU
Hidden Layer 2
Linear
256 + 128
ReLU
Output Layer
Linear
128 Netasess
Softmax
C COMPARISON WITH PREVIOUS DENSE ASSOCIATIVE MEMORY NETWORKS
Krotov and Hopfield (2021) proposed two-layer associative memory defined as
To de: 12 +
Win (hx):
(32a)
dt
dh dl.
The
hp +
(v:)
(32b)
14

---

Published as a conference paper at ICLR 2026
where uei Wip and each a E RN represents stored memory. Three choices for the nonlineari- ties f and were introduced in Krotov and Hopfield (2021), summarized in Table 4
We evaluate their recall performance (Models A, B, and C) together with our proposed model. Model is tested using fh,,,) = h.. (A (i)) and f(,,,) = b (A (i)).
Despite differences, all three models operate under the same effective mechanism: during recall, single hidden neuron becomes strongly active while the others remain suppressed, allowing only one memory per hidden neuron.
In contrast, our nonlincarity produces a fundamentally different recall regime. Multiple hidden neurons remain active simultaneously, enabling each hidden neuron to encode multiple stored patterns. This results a dramatic increase storage capacity: with only fifty hidden neurons, our model successfully stores all MNIST images (60,000) with high recall accuracy. By comparison, Models A, B, and C are limited to 50 memories and often fail to recall reliably (e.g., Model A (i) and Model C), as shown in T
And from a biological perspective, the nonlinearity used in Model A is not plausible, because the power-law activation causes hidden neuron activity to reach unrealistically high values during recall. Models B and C also rely on non-local activation functions, which would require additional circuit mechanisms to implement, In contrast, our model maintains bounded activity, and the nonlinearity fully local.
Model
f(hy)
g(vt)
N
Stored Memories
Recall Performance
A (i)
sign(rg
50
50
12%
h t
(ii)
sign (v:)
50
50
84%
B
50
50
90%
En chu
14
50
50
2%
4
Σ, D
Our model
+ (hp 0)
12
50
60,000
98%
Table 4: Nonlincarities used in the Dense Associative Memory models from Krotov and Hopfieed 2021) and in our model, and a comparison of their recall performance. Recall performance the percentage of recalled digits that are classified correctly.
D NOISE-INDUCED BIT FLIPS
the noise associated either with finite N,, corrections to Juar with perturbations the iniiall value of the visible units, or both-is too large, the target solution can be destabilized via "bit flips" in the hidden units. Here we quantify this, and in particular show that the probability of bit flip falls off rapidly as the number of visible units, N., increases.
As pointed out in Secs. 2.1 and 2.2
From Eqs. 9 and [4g with 0 set to 0.5, we have
No. VN N
(t) = tagget + targor
SueE + O(t/TT) - 0.5
(33)
where lagte is normally distributed random variable with variance given in Eq. 24 but with 8, replaced by Sutarget As discussed the paragraph following Eq. (14). the third term in parentheses has variance of NN, Thus, in the limit Fu > Th. > t,, we can approximate
sp (t) e tu.gr 2ugtt 0.5
(34)
where zaatareet is normally distributed with variance given by, at most,
Na,target + N.,,2
Var tagot)
(35)
N,
15

---

Published as a conference paper at ICLR 2026
Here Na.urger is the number of active target hidden neurons, and the "at most" qualifier is because should include the "+1" term only if sp.taget = 1 (see Eq. (24)).
The probability of single bit flip depends in relatively simple way the variance of z tagett However, the dependence on the number of active target neurons and the "+1" makes it nontrivial to determine the probability that there focusing on the upper bound on the variance of tartrreer which is given by
bit flips at all. We therefore bound that probability by
NA +1+ o oº2
(36)
N
With this definition, the probably of a single bit flip, which is the probability that taget is either greater than 1/2 or less than -1/2, depending on whether sptarget is 0 or I, is bounded by
Pi bit flip S 1 - (11/2oz)
(37)
where & is the cumulative normal function. The probability of no bit flips is, then, bounded by
PRo bit dips (1 Pi ba mi) mp) 2 $1122,))
(38)
To write this in more interpretable form, note that
22 ay
4
1- ((a)
dx
dy
dy
(39)
2n
2TT
1
av 2m
Inserting this into Eq.
40 then gives us the bound
e-1/80?
Nr
e-1/802
Pno dips
1 - 20=
21 NNo,
(40)
22T
VT/2
where the second inequality follows from Jensen's inequality.
This probability rapidly approaches as a₂ becomes small. In this regime, both the stability of the fixed point and correct recall are ensured.
log2 (Capacity)
8
Optimal Capacity
5-
Simulation
10
20
40
Ny/Nn
Figure 6: Capacity versus the ratio between visible and hidden neurons, for a fixed value of N₂, = 10.
To determine how many hidden neurons are required for real world memories with diverse statisties, note that memories from an N.-ddimnnsional space are recalled within an at most Nx-dimeeniional subspace spanned by the N₁, basic memories defined by the hidden to visible weights. this subspace is not expressive enough, the reconstructed images will not be recognizable, particularly for complex datasets such as CIFAR- 10. Consequently, the number of hidden neurons must be sufficiently smaller than the number of visible neurons guarantee stable recall, but it must also be
16

---

Published as a conference paper at ICLR 2026
large enough to represent the statistical structure of the stored memories so that the reconstructions remain recognizable.
Figure 7 and Figure [ show representative recall examples on MNIST and CIFAR-10. 10, for networks with an insufficient number of hidden neurons (N,, 16).
CLM
Taccall
7
3
9
3
Figure 7: Learned basic memories (columns of learned weight matrix) and representative recall examples for MNIST, for network with an insufficient number of hidden neurons (N₂, = 16).
Recall
Figure 8: Learned basic memories (columns of learned weight matrix) and representative recall
examples for CIFAR-10. 10, for network with an insufficient number of hidden neurons (N₂, = 16).
E ASYMMETRIC WEIGHTS AND HETEROGENEOUS THRESHOLDS
shores representative recall examples for networks with asymmetric weights and heteroge- neous neuron thresholds both MNIST and CIFAR-10.
17

---

Published as a conference paper at ICLR 2026
Cue
Recall
Recall
9
POOLO02000
3
6
DIORGO CFODEOOCODDD PORCOOORaOD
Figure 9: Representative recall examples for networks with asymmetric weights and heterogeneous neuron thresholds. On the left are examples of cue and recall for network that stored 60,000 MNIST images with hidden neurons, and on the right are examples for a network that stored 50,000 CIFAR-10 images with 500 hidden neurons.
F SIGMOID ACTIVATION FUNCTIONS
In both the theory and the experiments, used the Heaviside step function for the activation of hidden neurons. A smooth step function was used only during optimization.
However, the Hcaviside function is the only valid choice of activation. In fact, any sigmoid function that has three intersections with the identity line as showin in Figure [Ta where the middle intersection at 1/2 an unstable fixed point of (8), and the intersections near 0 and are stable fixed points, is valid activation function that guarantees stable recall as shown in
We demonstrated this experimentally by showing that recall remains perfect even when a smooth sigmoid is used in place of the Heaviside function, as shown in Figure 10
a
b
sharpness =10
Theory
sharpness
[x)
logz (Capacity)
Sim: sharpness
Sim: sharpness =0 20
0.8
7
6
0.2
2
0.2
1.0
2
3
4
5
7
10
Nn
Figure 10: a) Sigmoid nonlinearities with different sharpness, each having three intersections with identity line. b) Both nonlinearities produce stable recall, and the capacity remains exponential.
18

---

Published as a conference paper at ICLR 2026
G THE TIME CONSTANT OF HIDDEN AND VISIBLE NEURONS
Figure 11 shows the impact of Tv/ The the ratio of the time constant between visible and hidden neurons recall performance. As discussed in Section 3.2 on the basin of attraction, the visible neurons must be sufficiently slower than the hidden neurons because, during the cue, only the visible neurons receive input while the hidden neurons are initialized zero. The visible neurons therefore need to evolve slowly enough to allow the hidden neurons to reach their correct steady state before the visible pattern changes significantly. Somewhat surprisingly, factor of only enough to ensure perfect recall.
10
8-
log> (Capacity)
6-
4-
2.
Optimal Capacity Simulation
0
2
Ty/Th
Figure 11: Capacity versus the ratio between the time constant of visible and hidden neurons for
NA 10 and N. 100N
19


Reviews:
[{"review_id": "iqQelsvPfe", "reviewer": "Reviewer_Gxnx", "summary": "Associative memory models often struggle with a trade-off between biological plausibility and storage capacity. Previous biologically plausible two-layer models, such as Krotov and Hopfield (2021), were limited by winner-takes-all dynamics, resulting in a storage capacity that scaled only linearly with the number of hidden neurons.\nThis paper proposes a modification to this two-layer architecture: replacing the winner-takes-all nonlinearity with a simple threshold Heaviside activation function. The authors claim this allows for distributed, compositional representations in the hidden layer, rather than localized representations that tie memory to neurons.\nThe primary contribution is the theoretical and empirical demonstration that this network achieves an exponential storage capacity, $2^{N_h}$, in the number of hidden neurons ($N_h$), provided the number of visible neurons ($N_v$) is much larger than the hidden neurons ($N_v \\gg N_h$).", "questions": "1, In your classification results table, the nonlinear classifier on original MNIST images achieves 100% accuracy, but on the recalled visible patterns, it only achieves 91%, can you briefly explain this performance drop?\n\n2, What values did the learned threshold $\\theta$ converge to for the MNIST and CIFAR-10 experiments?\n\n3, Does the CIFAR-10 results imply the $N_v \\gg N_h$ condition is sufficient but not necessary, or is a different mechanism (perhaps from the learning rule) responsible for the high capacity in this regime?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper's main idea is simple and effective. It directly addresses a key limitation of the Krotov and Hopfield (2021) model—linear capacity—by changing the activation function. It also provides a simpler alternative to other recent work that required combining multiple modules to achieve exponential capacity.\n\nThe theoretical analysis is also nice. The paper provides a clear derivation for how the $N_v \\gg N_h$ regime leads to a decoupled hidden layer where all $2^{N_h}$ binary states can be stable fixed points. This theory is backed by numerical experiments. The stability of these points is also analyzed. The experiments on MNIST and CIFAR-10 show that the model can store tens of thousands of highly correlated, real-world patterns. The assumptions made are mostly in line with derivations in related literature and so are the experiments.\n\nThe paper is written clearly. It identifies the limitations of prior work and effectively explains how the proposed mechanism overcomes them.", "weaknesses": "In the CIFAR-10 experiment, the assumption used to derive the theories, $N_v \\gg N_h$, seems to not to be the case as $N_v=1024, N_h=500$, it would be nice to discuss more about the implications of this.\n\nAlthough not a fault of the paper per se but rather symptomatic of the computational neuroscience literature at large, biological plausibility is a loaded expression and can mean very different things. Also, although the architecture (pairwise synapses, two layers) avoids the implausible interactions of other models , the learning rule is a standard gradient-based optimization of a global cost function, which, despite various research on the biological basis of backprop, remains unclear whether it is truly biologically plausible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GAtfwkm7zQ", "reviewer": "Reviewer_81Kw", "summary": "This work assess a biologically plausible two-layer associative memory network that overcomes the linear capacity constraint of previous models. The core innovation is replacing the winner-takes-all nonlinearity with a threshold activation function, which enables distributed representations where single hidden neurons encode components shared across many memories. Consequently, the network achieves an exponential storage capacity of $2^{N_h}$, supports compositional storage of complex patterns, and demonstrates robust recall from noise on datasets like MNIST and CIFAR-10.", "questions": "1. The theoretical capacity holds strongest in the limit $N_v \\gg N_h$, but the authors achieved successful recall on CIFAR-10 with a ratio of $N_v/N_h \\approx 2$ ($1024$ visible vs. $500$ hidden units) (see L346-355). What is the rigorous theoretical lower bound on the ratio $N_v/N_h$ that is necessary to maintain near-exponential capacity and robust recall, especially when dealing with real-world, highly correlated memories?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The network is shown to have exponentially higher capacity than the prior two-layer implementations.\n2. The model's fixed points possess large basins of attraction, making the memory retrieval process robust to substantial noise in the visible inputs.", "weaknesses": "1. While I certainly agree pairwise interactions between neurons is biological, I do not agree with the implication that setwise interactions are not. I think the claim of this point should be more nuanced, e.g., see appendix A.2 of Burns & Fukai (2023).\n2. The theoretical exponential capacity relies on the condition that the number of visible units ($N_v$) is much larger than the number of hidden units ($N_h$), i.e., $N_v \\gg N_h$ (see L188-193). This does not seem obviously \"biological\". It would help if the authors could reference some related neurobiological literature on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gq4XrGTlus", "reviewer": "Reviewer_1mum", "summary": "This paper presents a 2-layer associative memory network that extends Krotov and Hopfield (2021). The key innovation is a replacement of the softmax/WTA hidden layer nonlinearity with a Heaviside step function, which effectively enables multiple hidden neurons to represent a memory instead of forming grandmother cells. The paper performed theoretical study of the capacity of the network and found that the network can achieve $2^{N_h}$ capacity with certain assumptions (e.g., $N_v >> N_h$), and it also studied the size of the attractor basin.", "questions": "- After training on MNIST/CIFAR10, how many distinct stable attractors exist beyond the training set, and how close is that to the theoretical $2^{N_h}$? Can't you experiment with a smaller $N_h$ e.g., 16, so the theoretical capacity is closer to the training set size?\n- How sensitive is stability / capacity to smoothing the Heaviside (which you actually use in training) and to adding noise / heterogeneity in thresholds?\n- How crucial is $\\tau_h << \\tau_v$? Can you show recall robustness curves as a function of this ratio?\n- Can you concretely map your hidden code / retrieval dynamics onto a transformer attention head or a feedforward block, and/or show a toy transformer experiment?\n- Can you tone down / clarify the “biological plausibility” claim, or alternatively provide evidence that asymmetric weights or mild threshold heterogeneity don’t affect capacity?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It proposes a simple yet effective fix to a known limit of Krotov and Hopfield 2021; theoretical proof and experiment analysis demonstrates the effectiveness of this proposal\n- The theoretical analysis is correct, thorough and clearly presented", "weaknesses": "Conceptually, I feel this paper has the following weaknesses:\n- The idea, though effective, is incremental to Krotov and Hopfield 2021. This is exacerbated by the fact that Krotov and Hopfield 2021 already discussed different possibilities of hidden/sensory nonlinearities so this paper does not serve as a generalization.\n- Despite the point about, there is still opportunity for this idea to be influential if the authors could extend it along the lines of biological plausibility (e.g., what if the weights are not symmetric?) and ML applications (e.g., how can a transformer benefit from the step function nonlinearity if it can be plugged into attention?). In other words, the paper can go 'wide' as it is incremental from a 'depth' point of view\n- The claimed biological plausibility is a direct inheritance of the biological plausible implmentation of Krotov and Hopfield 2021; no novel plausible implementation is introduced. The theoretically optimal $\\theta=0.5$ may even be too strict as a biological constraint. Therefore I suggest the authors not to sell this point too much (for example in the title).\n\nIn addition, I also feel this paper lacks analytical and experimental depth:\n- It lacks a direct comparison to other nonlinearities (e.g., models in Krotov and Hopfield 2021) - I understand theoretically it should have a bigger capacity, but a complete result section needs explicit experimental results to support the theory\n- Many experimental setting violates assumptions in theory. For example, in experiments, the $\\Theta$ function is a smoothed step-function; in both MNIST and CIFAR experiments $N_v$ is not much bigger than $N_h$. \n- The capacity results do not seem consistent with the theory: for example, 55k unique minima is far less than $2^{N_h}=2^50$. This makes me wonder the importance of the violated theoretical assumptions above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)