Dynamical properties of dense associative memory
Authors: Kazushi Mimura
Junichi Takeuchi
Yuto Sumikawa
Yoshiyuki Kabashima
Anthony CC Coolen
Venue: ICLR 2026 Poster (2026)
Keywords: Hopfield networks
dense associative memory
dynamics
convergence time
attraction basin
generating functional analysis
Primary area: learning theory
TL;DR: We analyze the recalling process using the generating functional analysis and discuss the convergence time and the attraction basins and so on.

Abstract: Dense associative memory, a fundamental instance of modern Hopfield networks, can store a large number of memory patterns as equilibrium states of recurrent networks. While the stationary-state storage capacity has been investigated, its dynamical properties have not yet been discussed. In this paper, we analyze the dynamics using an exact approach based on generating functional analysis. We show results on convergence properties of memory retrieval, such as the convergence time and the size of the attraction basins. Our analysis enables a quantitative evaluation of the convergence time and the storage capacity of dense associative memory, which is useful for model design. Unlike the traditional Hopfield model, the retrieval of a pattern does not act as additional noise to itself, suggesting that the structure of modern networks makes recall more robust. Furthermore, the methodology addressed here can be applied to other energy-based models, and thus has the potential to contribute to the design of future architectures.

DYNAMICAL PROPERTIES OF DENSE ASSOCIATIVE MEMORY
Kazushi Mimura
Jun'ichi Takeuchi
Hiroshima City University, Hiroshima, Japan RIKEN ATP, Tokyo, Japan
Graduate School Info. Sci. and Elect. Eng. Kyushu University, Fukuoka, Japan
mimura@hi roshima-hhhaaauueee jp kazushi www.mmmmmaaa..mmmmmmmmmmm riken.
t @ iff www.iin...uuuuu... kyushu-u. ac.
Yuto Sumikawa
Yoshiyuki Kabashima
Institute for Physics of Intelligence
Institute for Physics of Intelligence
The University of Tokyo, Tokyo, Japan sumi kawa lawtooy ecc. u act
The University of Tokyo, Tokyo, Japan kabaêphys wut.kyoo.oakoo.ooooooo
Anthony C. C. Coolen
Saddle Point Science Europe and DCN Donders Institute, Radboud University, Nijmegen, Netherlands a.cool en@sci ence. ru.nl
ABSTRACT
Dense associative memory, fundamental instance of modern Hopfield networks, can store a large number of memory patterns equilibrium states of recurrent networks. While the stationary-state storage capacity has been investigated, its dynamical properties have not yet been discussed. In this paper, we analyze the dynamics using an exact approach based on generating functional analysis. We show results convergence properties of memory retrieval, such the conver- gence time and the size the attraction basins. Our analysis enables a quantitative evaluation of the convergence time and the storage capacity of dense associative memory, which is useful for model design. Unlike the traditional Hopfield model, the retrieval of pattern does not act additional noise to itself, suggesting that the structure of modern networks makes recall more robust. Furthermore, the methodology addressed here can be applied other energy-based models, and thus has the potential to contribute to the design of future architectures.
INTRODUCTION
1.1 BACKGROUND
Dense associative memory (Krotov Hopheld, 2016,, model for storing binary patterns, was proposed and shown to significantly improve the storage capacity of the traditional Hopfield model (Hophield 1982) While can be regarded
rediscovery of the many-body Hopfield model Gardner. 1987: Abbott & Arian, 1987),, it exhibits slightly different properties. On the other hand, extensions of dense associative memory, such the Hopfield layer, have been actively developed to enable dense associative memory to store real-valued patterns (Demircigil et al, 2017; Ramsauer all 2021).. Hopfield models with such large memory capacities, including these variants, are referred to modern Hopfield networks, which have gained increasing attention and have inspired Transformer architectures (Hoover 2023).
The equilibrium properties of the Hopfield layer have been analytically studied, including evalu- ations of its storage capacity (Lucibello Mezard. (022)). Since the Hopfield layer can reach a near-equilibrium state in almost a single update step, its dynamical properties have not been consid- ered significant issue. contrast, dense associative memory, like the traditional Hopfield model, requires some iterative updates reach stationary state. However, its dynamical behavior has not

---

been investigated so far. As result, fundamental aspects such the attraction basin, namely, how far from a stored pattern can the initial state be for the system to still successfully recall it, still remain unclear.
While the dynamical properties of dense associative memory have not been investigated, those of the traditional Hopfield model have been extensively analyzed. In this paper, we analyze the dynamical properties of dense associative memory using generating functional analysis, an asymptotic theory in the large-system limit, which has been widely used in those studies.
1.2 CONTRIBUTIONS
Our main contributions are as follows:
Asymptotically exact dynamical analysis. We provide, for the first time, an asymptotically exact analysis of the dynamics of dense associative memory in the large-system limit using generating functional analysis (GFA).
Quantitative characterization of convergence. Our analysis yields explicit results con- vergence properties of memory retrieval, including convergence time and the size of attrac- tion basins, thereby enabling quantitative evaluation of stability and storage capacity.
Novel insight into robustness of modern Hopfield networks. We demonstrate that, unlike the traditional Hopfield model, retrieval does not introduce additional self-noise, suggesting that the architecture of modern networks makes recall more robust.
General methodology for energy-based models. - The proposed framework is not limited to dense associative memory. It be applied other energy-based models, providing theoretical tools for the design of robust and scalable architectures.
1.3 RELATED WORKS
Gardner and Abbott independently introduced Hopfield model with many-body interactions, which is essentially equivalent to dense associative memory, and analyzed its equilibrium properties using the replica method evaluate its storage capacity. The difference between their models and the dense associative memory proposed by Krotov and Hopfield lies in the presence or absence of self- coupling terms. While this difference does not affect the order of the storage capacity, does influence the constant factor. Additionally, Lucibello and Mézard analyzed the equilibrium proper- ties of the Hopfield layer using the replica method and obtained its storage capacity (Lucibello Mezard 2014). So far, analysis of the dynamical behavior of the modern Hopfield model has been reported. On the other hand, there has been extensive research on the dynamics of the tradi- tional Hopfield models. For example, the papers (Gardner all [I97: Crisanti & Sompolinsky. 1987: [988; Rieger el al., 1988) and our previous papers provide exact analysis based on GFA.
1.4 RELATED WORKS
Gardner and Abbott independently introduced Hopfield model with many-body interactions, which is essentially equivalent to dense associative memory, and analyzed its equilibrium properties using the replica method evaluate its storage capacity. The difference between their models and the dense associative memory proposed by Krotov and Hopfield lies in the presence or absence of self- coupling terms. While this difference does not affect the order of the storage capacity, does influence the constant factor. Additionally, Lucibello and Mézard analyzed the equilibrium proper- ties of the Hopfield layer using the replica method and obtained its storage capacity (Lucibello Mezard 2022) ). So far, analysis of the dynamical behavior of the modern Hopfield model has been reported. On the other hand, there has been extensive research on the dynamics of the tradi- tional Hopfield models. For example, the papers (Gardner al. 1987: Crisanti & Sompolinsky, 1987: [198% Rieger et all, 1988; Düring et al,, 19986; Coolen, 2001: Minura et all 2004; Minura & Coolen, 2009) provide exact analysis based on GFA.
2

---

2 PRELIMINALIES
2.1 NOTATIONS
Throughout this paper, vectors are denoted by boldface, e.g., and are assumed to be column vectors unless otherwise stated. 244) and (x)(() represent the (-th element of the vector æ. Matrices are denoted by an upper case symbol, e.g., A, and A denotes the transpose matrix A.
2.2 DENSE ASSOCIATIVE MEMORY
The dense associative memory one of the recurrent neural network models store and recall large number of patterns as fixed points of the dynamics. The energy of dense associative memory given by
H =
Σ
(1)
where h; E {+1}} denotes the state of the i-th unit, and & denotes the i-th element of the μ- th pattern. Each 5 independently takes the value #1 with equal probability 1/2. Introducing a nonlinear function F', such power function, makes memory patterns become deeper minima in the energy landscape and reduces interference between different memory patterns. This is because the nonlinearity suppresses weaker overlaps during the recall process. In this paper, restrict ourselves to the case
I
F(x) 2N
(2)
should be noted that since the coefficient 1/(2N"=11 does not affect the performance, so it is equivalent setting (xx) = x". The update rule defined by the difference of energies before and after state transitions. We keep only the leading term in the argument of the sgn function in the update rule, which gives
)
(++1)
sgn
Σ
LeH
+
cH 9
hg
-
Σ
ca
Σ h2
(3)
==1
1=1
j
n-1
sgn
Σ
h2
(small order terms)
jutt
where sgn (x) denotes the sign function that takes 1 if 2 O, and -1 otherwise. In the case of n 2 the network reduces to the parallel dynamics version traditional Hopfield model, i.c., h (++1) = sgn (E)= )iyh)") and J,j = + Em E:Ef By focusing on the leading terms for given function F, we can treat arbitrary activation functions. However, note that in the case of the exponential function, all terms in the power- series expansion have the same order.
2.3 OUTLINE OF GENERATING FUNCTIONAL ANALYSIS
We apply the generating functional analysis (GFA) to investigate dynamical properties of the dense associative memory. GFA has been applied to the model which is described using realizations of random variables (DeDomitess 1978). This method allows to analyze the asymptotic dynamical behavior in the infinitely large system, using the generating functional, which is the dynamical equivalent of the characteristic function statistics.
In GFA formalism, consider the joint probability distribution over the states of all units at all time steps, from the start of the iteration up to some prescribed time, which can be taken sufficiently large. This joint probability is referred to the path probability. From the path probability, we can calculate various expectation values such as the overlap, which the direction cosine between the states of the units and the memory pattern being recalled via the generating functional which can be regarded as an analogue the characteristic function. Table shows the representative analysis for dynamical properties of traditional and modern Hopfield models.
3

---

Paper
Model
Method
Update
Retarded ST
Amar & Magint (1998)
traditional
S/N
parallel
ignored
traditional
hierarchical S/N
parallel
ignored
Kieger (1958)
traditional
generating functional
asyuchronous
treated
Coolen Sherrngtoo
(144
traditional
dynamical replica
asynchronous
treated
Düring et all (19957)
traditional
generating functional
parallel
treated
this paper
modern
generating functional
parallel
treated
Table I: Relationship to existing dynamical analyses for Hopfield models.
3 ANALYSIS
First, the path probability is defined and used describe the generating functional, after which the expectation over the memory patterns appearing in the generating functional is evaluated.
3.1 PATH PROBABILITY
Let vectors (0) (M1)
IT E (+1] be the states of all units at time / and let the initial state be h (0), The updating rule, obtained by retaining only the leading term, is expressed follows:
hg
(t-1)
sgn
(u:
(2)
(5)
M
(0)
+ -0.0)
u
(6)
for all i E (1,..., ,N and (O....,TT 1}. The variable u is referred to as a local field. The parameter 8(t) is called an external field (or threshold). The dynamics of the system characterized by how the output of each unit changes in response to infinitesimal variations in the local field. To evaluate such changes, the external field (0(²) are introduced. The average derivative of the outputs with respect to the external field referred to as the response function, which serves one of fundamental measures for describing the dynamics. After evaluating the response function, all (0()) are set to zero.
In this paper, consider parallel dynamics, in which the states of all units updated simultane- ously. The updating rule of the dense associative memory for the variable h (++-1) at time can be given by the following probability distribution:
N
II 8th 11)1)
sgn (u."),,
where 6/m; ni] denotes the Kronecker's 's delta that takes if m = n and otherwise. This dynamics represents Markovian dynamics. The path probability (0) (T) is given as the products of the probability distribution of the updating rule:
T-1
hh (0)
(T
h
(0
II plh (t+0
where () N p(h (0)} denotes the initial state distribution. Since the same memory patterns
are included at every time step, the states of the units at different times are correlated.
3.2 GENERATING FUNCTIONAL.
The path probability depends on all memory patterns g!.... MM. We define the generating func- tional as follows.
4

---

Definition 1. The generating functional [[pb] is defined as
2[bb] Be...... _ 1110) .. (
plh(o) _1(T) exp
-i E h00 pl0))
where we have introduced the generating variables wb1t) -011 ,.. 2/ = (zb(0),... .... ()(T))fe for shorthand.
(1) N
EN and we write
Here, denotes the imaginary unit. i.c., i= We here assumed that the generating func- tional self-averaging. namely, in the large-system limit, i.e. N is sufficiently large, the generating its average over the memory patterns E < and the typical be-
functional is concentrated haviour of the model only depends on the statistical properties of the memory patterns. In GFA, the expectation values of interest are calculated from derivatives with respect some elements of the generating variables, e.g.,
lim 87[bb] ay!!)
Egg,...
Ser....gull ......M(((ii ((-ihh; (1)
(10)
[
lim
0v. (1) aw!! (t) = Be..... ear 11 (1- 14 n9
(11)
l.......svll 1(-i ah
lim
Eet... sa
00 1
(12)
# 0y.
(1) (0000 (0(0))
where t/ - O denotes (1()) + for i and 4, and the bracket (...)) denotes the average over the path probability, i.e., ((...)) = Eno, ( Mett11) "" """
(..... .... Introducing the
definition of the local field using the Dirac delta function, the generating functional can be rewritten follows:
[ = Egy... ...
Σ
[ du p(h
T-1
II
N TTh (++1),
sgn(u(")) )e e- A
EM
h(0),,.
N
M
N
0.0) )
X
U
(13)
N
#=1
j
Assuming that the pattern g² is being recalled, we separate the local field in the generating func- tional into signal term including the recalling pattern and noise term including other patterns .2......"" .. Using the Fourier integral form of Dirac delta function, the generating functional be- comes
T-1 N
[
Σ
dusu (0)
II 1=
8Th (++))
sgn(u(())) e - -i (9.00000
JRT
T.
N
T-1
N
N
(
x exp
i
ΣΣ
(ut - 0,01
Eer exp
-1 ΣΣ
(1)
cl
Σ
N
1-
1=1
T-1 T-N M ΣΣΣ 1= p=2
N Σ 5
10 de $t
x ger gMr exp
n
(14)
j
Only the last part involves all non-recalled patterns _..... the expectation over these patterns, the generating functional is found to depend on five types of averages. Accordingly, we introduce the following macroscopic parameters:
MM By straightforward calculation of
1 N
N
ml (0) =
Σ
E (1)
R00 = N Ela
N
1=1
5

---

N h 1 1
N Σ N
N h
gll.e)
Qua
C4 K
10
(15)
into the generating functional using the Dirac delta function, where the functions ml)) is referred to as the overlap. The generating functional can be calculated as follows.
Lemma 1. By averaging over the memory patterns, the generating, funciional is given by
Z[bb] /
dimmmhuusckkkggllqqquqdaaa. exp N(W + +2) + 0 N7)
(16)
where
m (m 1100000 1((0 nmm(1)a-11
V
T-1T-1
((tt)g(tt))) + Q(te) k((,) ((0
1
(17)
1=0
N
①
log Σ
dudu p(h (0))
8Th (++1)
sgn(u, (u(2)]
N
h
1=
E cxp
ΣΣΣ
T-1 N
T-1
h m - T-1
+ 1
Σ
{ug
100 0000 -i
-
h 2000 (18)
1=0
t=
T-IT-1
n-
Ω =
2 n
(n - 1)3
Σ
A(n - 2, (()))))
K((' k((.))
~
1= ==
k=0
A(n - 1, (g (11.00) +
(19)
0m
where A(e,k) ( kt B(e k)) and (mm) mmvven (m 1)!!.
A proof is given in Appendix A Here, Lcondition denotes the indicator function that takes 1 if the condition is true, and 0 otherwise. It can be obtained by evaluating the lcading terms after taking the expectation
the memory patterns, using combinatorial arguments. The order of the number of memory patterns is determined by the balance between the magnitude of the signal originating from the retrieved pattern and that of the noise originating from the non-retrieved patterns. From the analysis in Lemma II, the number of memory patterns M is required to scale as M = O(N=~1) for non-trivial analysis. This corresponds to the generating functional being of order e O(N) Therefore, we set
M N
(20)
Further details are given in Appendix A.
The generating functional is dominated by saddle-point in the large-system limit. Averaging over the random variables, we will move to a saddle-poie point problem (Copson, (1963) in the limit N + oc. The saddle point condition gives values the macroscopic parameters. Hereafter, we choose the factorised distribution p|h =I p[4,0) ()))))(h))) } as an initial state distribution, where m denotes initial overlap. The fac- torised initial overlap allows the generating functional to decompose into independent single-unit contributions.
i + m ( + }

---

4 MAIN RESULTS
The behavior of this model differs significantly between the case n = and the case n 2 3. Since the case 2 has already been extensively studied, we focus only the case in this paper. GFA provides an exact solution as an asymptotic analysis in the large-system limit N + co. Using the saddle point method to evaluate the integral in the averaged generating functional, one can obtain the following proposition.
Proposition 1. For given initial state distribution p(h!o] and n 2 3. the overlap m the corre- lation function ( and the response function G(t,t) are given by
a(h())
(1) (Eh C (1 G(iet 1ixe
(21)
00(*))
where (h)) denotes the average defined as
(f(h)) = Es f Do E f(h) (0)
T-1
II 1=
8 1(-11);
; sgn ( -(Th)( + v()) + g(1)
)
h
(22)
which is referred to as the effective path measure. The random vector v follows a multivariate normal distribution with mean and covariance matrix R = (R(t.))), where the (t, t'' eleement is
R(o,t) =naas an Σ A - (((.))) k=0
(23)
The matrix I is given by r D G. The (t, eleemenss of and G are D(t.t") and Giad""
respectively. Each element of the matrix D = (D(t.')) is defined as
D(C =n 1)²an Σ A(n - 2, ()(
(24)
k=0
The operator denotes the Hadamard (elementwise) product.
The proof sketch is given in Appendix B The term (Th)() in the effective path measure represents retarded self- interaction. The retarded self-interaction means the magnitude of the influence that returns to a unit itself after propagating through other units. Due to this retarded self-interaction, the state the next time step depends in complex way on the past states. On the other hand, unlike in the traditional Hopfield model, i.e., the case of n = 2, the noise variance does not depend on the overlap, and it does not increase even when the overlap becomes large. As result, considered that the phenomenon, in which the system begins to recall correctly but eventually fails to complete it, becomes less likely to occur.
DISCUSSION
5.1 NUMERICAL ANALYSIS AND COMPUTER SIMULATIONS
In this paper we considered Krotov's 's dense associative memory. Since the noise variance depends on n, we normalize the constant a, by setting a', (2n 3)llan where a' is referred as the loading rate. The storage cupacity of is defined as the largest loading rate at which the overlap remains positive.
The result of Proposition I can be numerically analyzed using the Monte Carlo method. Figure II shows the numerical analysis for the case n = 3, together with the results of computer simulations. In Fig. M (a) (d), the graphs the left display the simulation results for N 1024 with 100 trials. The vertical axis represents the overlap, while the horizontal axis represents the number of iteration steps. The graphs on the right in Fig. M (a) (d) correspond to the numerical analysis of the overlap based on Proposition MI Although finite-size effects become significant near the basin of attraction, it can be confirmed that the theoretical values agree well with the simulation results
7

---

0.10
0.10
0
0.20
0²3 0.20
10
10
1.
1.0
0.6
0.0
0.5
0.5
E
0.2
0.2
0.2
0.2
10
20
30
20
30
20
30
10
t
(a) 0/3 0.1.
(b) ag
0.2.
a: 0.30
0 0.30
9 0.40
0'3 0.40
1.0
10
1.0
1.0
0.6
0.6
0.5
0.5
E
0.2
0.2
0.2
02-
10
20
30
0
10
20
30
10
20
30
10
20
30
(c) 03 0.3.
(d) 0g 0.4.
Figure 1: Recalling process of Krotov's dense associative memory with F(x) = x. and n = 3. Left: computer simulations, 100 trials, N 1024. Right: theory.
even for relatively small-scale experiments. can be theoretically confirmed that, when retrieval successful, convergence is attained within several tens of iterations.
Figure illlustraees the overlap after 50, 100, and 200 iterations by color while the red solid lines rep- resent the boundary of the basin of attraction assessed by an approximate dynamics discussed below. the dynamics had fully converged, the region where the overlap remains finite would represent basin of attraction, suggesting d',3 = 0.33. However, Fig. II shows gradual decay of the overlap with increasing t, indicating that a',3 is at most about 0.3. Indeed, while the computer simulation results for 2), the DMFT results shown in the right panels of Fig. Ø indicate gradual shrinkage of the region with large m as T increases. Consistently, static analyses based on the replica method (Mezard et all (1977) give smaller values a/33 0.252 under the replica symmetric ansatz, which is con- sistent with the approximate dynamics shown below, and a 0.266 under the 1-step replica symmetry breaking ansatz as shown in Appendix D In similar systems, slow dynamics are known to occur the phase boundary between the crystal and glassy phases (Krzakala Zdeborová 2010), which corresponds here to the retrieval success/failure transition. Therefore, the true basin of attraction narrower than what is shown Fig. but conjecture that accurately determining is challenging due to the presence of slow dynamics. A similar situation arises for 2 4 as well.
512 exhibit almost no dependence on due to finite-size effects (left panels of Fig.
5.2 CONNECTION TO RELATED ANALYSES
From the exact solution obtained via the generating functional analysis, we obtain the following approximated result when self-coupling neglected.
8

---

n= 3 -T=
n= 3 (7= 50)
1.0
88
0.0
0,7
0.6
0.6
0.6
0.5
0,5
0.4
0.4
0.4
C4
0.3
6.2
0.05 0.1
0.15 az
0.25
0.3 0.35
0.
0.05
0.
0.15
0.5
0.3
0.35
04
0g
a3
(a)
n= 3 (T= 100)
n= (T= 100)
100
1.0
0.9
0.3
08
0.6
C.8
0.7-
0.7
0.5-
06
0,6
6.6
0.5
0.4
C.2
01
0.0
0.0
4
(b) T 100.
(0=)00)
(T=200)
1.5
1.0
1.0
1.0
0.3
0.5
0.8
08
004
0.4
0.3
0.3
0.2
02
0.2
G2
0.1
611
0.0
0.0
0.0
0.03 0.1
0.15
0.2 0.25
0.3
0.35
0.4
0.0
0.05
011
0.15
0.2 0.25 0.3 0:35
0.4
0g
(c) T 200.
Figure 2: Overlap with the retrieved pattern m = E h/N after T = 50, 100, 200 iterations for Krotov's dense associative memory with Fx:) x and 3. Left: computer simulations, 100 trials, N = 512. Right: theory. Red lines: attraction basin obtained by the approximate dynamics discussed in Sec. 5.2
Corollary 1. Neglecting the retarded self-interaction tern as an approximation, i.e., setting O, we obtain
(ml1))n-1
m (t+1) erf
((2n (122 3)!! 2an
(25)
where erf (x) = J7 foe -== dz denotes the error function, and (.)) is the initial overlap.
In this approximation, the equilibrium state of the dynamics can be simply obtained by setting (() = m, and the resulting fixed-point equation corresponds to the equilibrium analysis by r--ody Hopfield model. Although the coefficients differ, this is due to the fact that the energy function not the same as that in Krotov's model. The storage capacity that obtained by the fixed-point equation of this approximated dynamics, i.c., m erf (m" (amm"//(((ee) /((((2n - 3)!! 2a,)), gives that of the equilibrium analysis derived by the replica method.

---

We here consider the differences between Krotov's dense associative memory and the n-body Hop- field model independently proposed by Gardner and Abbott. The energy function the n--ody Hopfield model is defined by
M E
H1 V
S -yn hi hag hij.
(26)
Fin
The values of the variables ji, ... IIn are all distinct. This is the main difference from the Krotov's dense associative memory. Using the same way Krotov's method, the corresponding update rule of the n-body Hopfield model is given as
M
N Σ
[++1) sgn 14
S ...
SJ-1
(8) 13
...
(27)
Na-I
We have the exact result in the same way to obtain Proposition II Let M Nu again, Ne- glecting the retarded self-interaction term as an approximation, i.e., setting O, we obtain
m = erf (m(t-1))n-)
(28)
(n 1)! 20
The stationary equation, i.c., setting (t) = m, is equivalent to the result derived by Abbott (Ahhofi Arian 1987). The detail is available in Appendix [].
CONCLUSION
We performed an asymptotically exact analysis of the dynamical behaviour of dense associative memory using generating functional analysis (GFA) in the large-system limit. The analysis revealed the presence of retarded self-coupling term, indicating that the next state of the system depends in complex manner on all past states. We also confirmed that this property cannot be captured by method based on the signal-to-noise analysis. In the traditional Hopfield model, i.e., n = 2. it was found that the system exhibits noise variance that depends intricately on non-recalled patterns. In contrast, for n 2 3, the noise variance due to non-recalled patterns does not depend on the overlap with the recalled pattern. As result, the phenomenon observed in the classical Hopfield model, namely, the increase in noise variance upon successful retrieval, is mitigated. Thus, it arises only from the retarded self-interaction. As result, the recall process becomes simpler than the traditional Hopfield model.
Assuming the existence stationary state, we can also consider a macroscopic fixed-point equa- tion from the GFA equations. Due the presence of the self self-coupling term, this result must differ from that of existing equilibrium analysis. This difference comes from the fact that, for models with 3, the system does not satisfy the detailed balance condition.
In this work, we provided an exact dynamical analysis dense associative memory using the gen- erating functional analysis, and verified the theoretical predictions with numerical experiments. Our results clarify how higher-order interactions, namely, n 2 3, suppress the increasing of crosstalk noise due to the recalling pattern itself, thereby stabilizing recall dynamics and enhancing memory capacity. This contrasts with the classical Hopfield model, where self-retrieval inevitably introduces additional noise. These findings offer quantitative framework to evaluate the stability and storage capacity of associative memory models, which is useful for guiding model design. While our exper- iments were limited to relatively small system sizes and specific interaction orders, the analytical methodology is general and can be applied to broader class of energy-based models. This ap- proach can be extended to modern Hopfield networks, memory-augmentee energy-based formulations will provide further insights into the design of robust and scalable mem- ory systems. In this context, the simplicial Hopfield networks can also be analyzed within the same framework. We currently working analyzing cases where the function F introduced into the energy is exponential, as well as the case where memory patterns are biased (Bielmeier & Friedland. 2025).
architectures, and other
10

---

ACKNOWLEDGMENTS
This work was partially supported by JSPS KAKENHI Grant Nos. 23H05492 (KM, JT) and 23K03841 (KM), and MEXT/JSPS KAKENHI Grant No. 22H05117 (YK).
REFERENCES
L. F. Abbott and Yair Arian. Storage capacity of generalized networks. Physical Review A, 36(10): 5091-5094, 1987.
Shun-ichi Amari and Kenjiro Maginu. Statistical neurodynamics of associative memory. Neural Networks, 1(1):63-33, 1988.
S. Bielmeier and G. Friedland. Effects of feature correlations on associative memory capacity. In Proceedings of New Frontiers in Associative Memory workshop ICLR 2025, 2025.
A. C. C. Coolen and David Sherrington. Order-parameter flow in the fully connected hopfield model near saturation. Physical Review E, 493)::921-19994 1994.
A.C.C. Coolen. Statistical mechanics of recurrent neural networks dynamics. 2000. URL attps:
1/arxiv.
E. T. Copson. Asymptotic Expansions. Cambridge University Press, 1965.
Andrea Crisanti and Haim Sompolinsky. Dynamics of spin systems with random asymmetric bonds: Langevin dynamics and a spherical model. Physical Review A, 36:4922-4933 1987.
Andrea Crisanti and Haim Sompolinsky. Dynamics of spin systems with randomly asymmetric bonds: Ising spins and glauber dynamics. Physical Review A, 37:4865-4495 4874, 1988.
C. DeDominicis. Dynamics substitute for replicas in systems with quenched random impurities. Physical Review B, 18:4913-4919, 1978.
M. Demircigil et al. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-291 299, 2017.
A. Düring, A. C. C. Coolen, and David Sherrington. Phase diagram and storage capacity of sequence processing neural networks. Journal of Physics A: Mathematical and General, 31 31(43):86078861,, 1998a.
A. Düring, A.C.C. Coolen, and D. Sherrington. Phase diagram and storage capacity of sequence pro- cessing neural networks. Journal of Physics A: Mathematical and General, 31 (1(43)88607, 1998b.
Elizabeth Gardner. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20:3453 33644, 1987.
Elizabeth Gardner, Bernard Derrida, and Paul Mottishaw. Zero temperature parallel dynamics for infinite range spin glasses and neural networks. Journal de Physique, 48:741-755, 1987.
Benjamin Hoover, Yiming Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Homg Chau, Mohammed J. Zaki, and Dmitry Krotov. Energy transformer. In Advances in Neural Information Processing Systems 36, 2023.
John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79:2554-2558, 1982.
Dmitry Krotov and John J. Hopfield. Dense associative memory for pattern recognition. In Advances in Neural Information Processing Systems 29, 2016.
Florent Krzakala and Lenka Zdeborová. On melting dynamics and the glass transition. ii glassy dynamics as melting process. The Journal of Chemical Physics, 134(3):034513, 01 2011. ISSN 0021-9606. doi: (0.1063/1.350633 URL
Carlo Lucibello and Mare Mézard. Exponential capacity of dense associative memories. Physical Review Letters, 132:077301, 2024.
11

---

Mare Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An In- troduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
K. Mimura and A. C. C. Coolen. Parallel dynamics of disordered ising spin systems on finitely con- nected directed random graphs with arbitrary degree distributions. Journal of Physics A: Mathe- matical and Theoretical, 42:415001, 2009.
K. Mimura, Masaki Kawamura, and Masato Okada. The path-integral analysis of an associative memory model storing infinite number of finite limit cycles. Journal of Physics A: Mathemat- ical and General, 37(25) 2004.
M. Okada. A hierarchy of macrodynamical equations for associative memory. Neural Networks, (6):833, 1995.
Hubert Ramsauer, Markus Widrich, Milena Pavlovic, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Thomas Adler, Lukas Gruber, Michael Holzleitner, Geir Kjetil Sandve, Vegard Greiff, and Daniel Kreil. Hopfield networks is all you need. In International Conference on Learning Representations, 2021.
Heiko Rieger, Michael Schreckenberg, and Jürgen Zittartz. Glauber dynamics of the little-hopfield model. Zeitschrift für Physik B, 72:523-529, 1988.
12

---

APPENDICES
A PROOF OF LEMMA II
We first calculate the expectation value of the noise term of all non-recalled patterns, which is the last part in (LE). Using the Taylor expansion, we obtain
T-1
M
N
BE exp
T
N
1
(1)
(29)
Egu
1
T-1
Σ
Σ
5f n
Σ
N
n
(30)
N:3(n-11
2=2
(M 1)
12 M
ΣΣΣ
(M) M1
ΣΣΣ>
2 No +0
exp
22
(Na(n-1))
1=0 ~=0
t=0
(31)
where
N
N
N
N
.1 =Eg
En ... Ein-1 Egi ... 1..5%% (1) ... 10
Jiti
in 17. 8(
.22 =1EE
EiEN
En 1in Ey, (
(8) Jn
h
(4)
in iiti
1n-1 Ji
1h 12
Since 2..... ........................... ..M are independent, we can drop the index p. It should be noted that any term in Wi and N2 that contains an odd number of identical index from the same pattern has zero expectation. because all {,, ... En are independent and have zero mean.
We calculate Ng. The leading term in MG can be obtained by calculating the summations in the case where the 2(n 1) variables are grouped into pairs, each pair taking the same value. We have to do this for all possible partitions. We must distinguish three types of pairings: (i) between two primed variables, (ii) between two unprimed variables, and (iii) between a primed and an un- primed variable. Note that depending on the type of pair, the time parameter differs. Therefore, the lcading term can be obtained by counting the number of ways to partition the 2(n indices, i... ".. Jn H1, ... , -n-1, into n 1 pairs in which indices take the same value while cach different pairs takes different values.
We here consider two sets indices: the set unprimed indices J = (jr.... Je and the set of primed indices J' [.'..... -1). First, we consider the number of ways to divide 2 indices, including& l unprimed indices and l primed indices, into l pairs. Let A(E, k) be the number of ways to have exactly unprimed-primee pairs in total pairs, which is given by
( E k
A(e, =
k1 B(e - k)².
(32)
where (mm) is the number of ways where m/2 Hoprimed-unppinee pairs and m/2 primed-primed pairs are made using 2m indices, consisting of m unprimed indices and m primed indices:
B(m) 1m:even (m/2)l
1
m
m - 2
( 2
1 mnvven (m 1)!!. (33)
2
Note that Σ Sizo A(e,)) k)) B(2() holds.
13

---

Using the quantity A(e, k) and the identity )2 obtain
N
N1 = EE
Σ
En
1S1 :M Ex
(1 "n
(4) (5)
(34)
In 31
In-1
"1," Jn-
ng n()))))) +O(N=-3)
=N
A(n 1,
(35)
N
k=0
=O(N9)
Next, we calculate N2. For notational simplicity,let ( (...))....) be an operator to substitute
ji i' into (..)), and (Zj )(...) be an operator for summing (-.-) over ji # i, + V. should be noted that each of j1, ... In-1 can take the value i', and conversely, each of J4, . ,
can take the value i. For all i f1, ,N and i' (1,"), we have
N2 -E ELEP (-
Σ Σ.
Σ
5==
jn-1=1 in-1ti.4
1h-== 1h-171,
En
Ei E5 ... Ex
(1)
2n In- 1 Ve-1
(0) (t)
1t))...1 (t')
(36)
Eeg ELE
in-2ti
Σ
En ... Ejn 1
Eg 1 19 bn
(1)
(0) (6)
1
1...... (C)
+0(M--)
(37)
8n-1
=(n 1)2 n(2n5"" (())) Feg Σ
Σ
j1m" tn
H H Sa
En ... Eim 25 ... Et/ 1100
11())
...).
(t)
+ 0(Nm-1
(38)
jn-z
=(n - 1)² 1 10 ((((())))))))
2 Σ
R-2
A(n - k,
Σ
N
h )
+O(N"-4).
(39)
N
k=0
=0(N9)
Substituting (55) and (39 into (.)). we obtain the expectation value of the noise term of all non-
recalled patterns as follows:
N
M
N
Eea ... &M exp
a
99 n
g
Eenny
(40)
MMM T-17-1
ΣΣ
= =ex
2 N
1= K=
(n 1)2
1
Σ
(1) (((0)
(
n u
2-2
Σ
A(n - 2, 2,,k)
1 N
& Einonne
N
N
14

---

N
200 ((1)
Σ
A(n - 1.
Σ
N
11
+ +
(41)
N
K=
==1
dq (18
Ng(1.t)
Σ (
1=0==0
T
N
(Q
Σ
(=0
==1
dkumn
NK
Σ
h
((1))
1=0-=00
1
M
T-IT-1
(n - 1) ((1,))))))) k(t'00 n-
Σ
A(n - 2, 2,4)) (paee))
x exp
2
Nn-2
P=D
n-1
+ +
+ ((44))
Σ
A(n 1,kb
(42)
k=0
dightt)
(11)
exp (iglt1)
(Ng(1e))
Σ
m (n)((()) (2)
2T
1'===0
dQuu))
dQ((t))
exp sg(tt)
(QQu
Σ A
2x
P=
i=1
T
00
H
x
dK(m))
exp i9
ig(av))
NK(a)))
Σ
1
2T
1=0+==0
==1
1
n M
T-1T-1
(n - 1)=
Σ
A(n 22, -2,0)(p(e))) (glees) b
cxp
2
Nu-2
1=0 P'=0
k=
n-1
(p(tes) k + +
A(n 1,k)
(43)
k=0
by using
The signal term that includes the recalling pattern g¹
can be rearranged as
T-1
N
Ee exp
i
1
N
Σ
11)
t=0
i=
T
N
N
+0(N-1)
= Eeg exp
-
12
t=0
1=1
j=1
Σ (1 ]
18
Nm
1=1
Σ $ (a(t)) )
N
dk(
Nk(())
(1)) + +O(N11) m-1
x Egi exp
-iN Σ
110
(44)
1=0
00
(m)
st H
dm
exp
im Nmm
000
15

---

dk((t)
N Nh:(1 ihlt)
Σ )
dk((t)
expy
a
2T
1=1
T-1 1 m +O(NT))
7-1
X Eg exp
IN
(45)
We here introduced the parameters of (LS) using the Dirac delta function and its Fourier integral form of the Dirac delta function, i.e., 6(x) 2 I d 2 I die We then arrive the generating functional of (IGG). Since the expectation over the non-recalled patterns has already been taken, and only the recalling pattern remains in the expression. The signal term of (H5) is eO(N), On the other hand, the noise term of (H3) is (MM/N"*)3) For non coivaal analysis, the signal term and the noise term must be of the same order, namely, the number of the memory patterns M must be ONN=11).
B PROOF SKETCH OF PROPOSITION II
It should be noted that the normalization relation 2 plays an important role in the elimination of spurious solutions the saddle-point equations. The terms in the averaged generating functional can be split into three related parts. The first one is signal part. The second one is a static noise part due to the random variables within the model. The last one is retarded self-interaction due to the influence of the state at the previous stage, which may be able to affect the present state. The GFA allows us treat the last part. After the analysis, turns out that the system can be described in terms of the following three quantities:
m Ege.... 2d K
).
Σ
(46)
(
c(C,))
= =EeI,.. gM
Σ
(47)
N
ah! (t)
g(t,)) BEe,...
E
0o
(48)
where these are referred to as the overlap, the correlation function, and the response function, respec- tively. One deduce the meaning of macroscopic parameters by differentiating the averaged gen- crating functional with respect to the external field ( and generating functions (i((), The averaged generating functional Z[bb] is dominated by saddle-point for N Using the normalization identity [ = E _M (1) = 1, one can have derivatives the averaged generating functional:
07(μb]
lim
(1)
00
(w:(() (t)
8 7[bb]
lim
- (110) (1 ()) (h!
-0
(t) 0w((
(1)
0 [bb
lim
8. (1(0)))))))))))))))))))))))))))) (1 - 5..0 (1(1)) (((()))))))))))))
(49)
-0
0v (6(!)
0Z(
lim
i(( =0
40+0
(0(0)
8
lim
62,
(2
$ 4
16

---

where (), denotes the average that is defined by
Σ
/
dudu wh, f u, u)
uf u, ú));
(50)
Σ
dudü wy(h, u. i
with
wi (h, u. ú) =p/h (
[(h(t+1;; sgn (u()))
t=0
(g(000)
+ ((.0)))))))))))) + ((06))66))))
x exp
t= ===
T-1
Σ
a00) (ul) 1
h (l)
(51)
saddle
The average ((...)); )); referred to as a single-umit measure. Here, evaluation flsaddle denotes an evaluation of function f at the dominating saddle-poing -pinnt Substituting (H9) into (II) (I2), we then have
Eet... eM (h (h
Eg MM (1) (14()) 6i (h100()))))))))))))))))))))))))) + 52, (1))))0))
(52)
E Egi.... "M @h! (t)
- -dd (1(0)))))
00
In the large-system limit, the averaged generating functional will be evaluated by the dominating saddle-points of the exponent & V S. We can now derive the saddle-point equations by differ- entiation with respect to the integral variables (*) (t) 1t(t), k(0, g(t.t") g(t.t), Qit.e), Qu..t) K(**), and (**²), The saddle-point equations will involve the overlap m!), the correlation (1.))) and the response function G(a,a") It should noted that causality, i.e.,
a(h(a)
0,
(53)
80107
should hold for < t'. Therefore G(t.t") = for S t'. Using causality and the identities (49) and (52),, the straightforward differentiation of + V + Ω2 with respect the integral variables leads us to the following saddle-point equations:
1 N
m (1) =
( ()h ( 0. A(C) = 0. R(00) nmm!0)a-1
(54)
N
i=1
(00)) c(t,t" 1
Σ (n 1 (1)) = 111)) g(ee") 0,
(55)
N
1=1
Q((() = 0, Q((x") - 1
(56)
10
k((,)) (0((00) 1izt
R((e)) D(W'
(57)
00(1)
where
R(t,() = =n²
A(n - 1, (C(A))))
(58)
k=0
17

---

n-
D = nn²(n= 1)²a, on
Σ A(n 2, 2)) k=
(59)
Substituting the solutions of the saddle -oint equation into the single-unit measure, we obtain the effective path measure. We then arrive at Proposition MI
C PROOF SKETCH OF GARDNER'S 'S MODEL
Using similar way to Lemma II the expectation value of the noise term of all non-recalled patterns for the n-body Hopfield model is given by
1
Eg Fd exp
Σ
Ej
...
SJn-1
...h%)
Na-1
z
exp
2
1 M
Nn-
T-I
(n
1)²
N
(2)) (1)
uu
N
(t') Ui
A(n - n2
1=0
P==0
1=1
y=1
1
h
n-
0
A(n 1,
Σ
N
h
+ +
N
==l
j=1
(60)
where
(1)
A(C,
el B(e 0)2 E!
(61)
Applying the same calculation, we arrive at (28)
D SKETCH OF REPLICA COMPUTATION
In a general setting, suppose that the state variable 8 = (s;) is governed by a Hamiltonian (8) ()) that depends on predetermined random variable r. In this case, the thermal average
LT's (e)(()))))
((r) tre -BH(al))
(62)
((r)
becomes random quantity because it varies with the realization of Here, t.x( denotes summation or integration over all possible configurations of
The replica method is technique used to evaluate the moments of the thermal average E, [s,) k] for k = 1,2,.... by means of the replica trick
(=(())) lim E, (r) (s
(63)
[r))
In practice, this reduces the problem computing the y-th moment of the partition function E, (Z" for integers v = 1, 2,.... using the saddle point method, and then analytically contin- uing the resulting expression real values v E R under the assumption of certain symmetry.
For the model defined by (I)) and (2), we analyze its behavior using the replica method under the 8¹ h/N,
assumption that M n-1 and that only the overlap with the first pattern,
is O(1) while the other overlaps, g" h/N for μ 2,..., O, N n-1 remain typically 1/22 To this end, introduce the rescaled variables Uy E" h/N1/2 for μ 2.... a., a=1 and rewrite the Hamiltonian as
N m
N1-n/2 a.
H
(64)
2
2
4
p=2
18

---

The corresponding partition function is thus given by
N mm
N
21-1/2B
anN
Z
=
exp
+
(65)
2
2
h
2
and its v-th moment reads
N
N1-n/23
an
Nm-1
Be Be|z"1
Σ
Be
exp
mm))
x E
Cun An-
exp
(
2
h h
a
a=1
I2
(66)
for natural numbers v =,,2....
Evaluation of Iz.
The quantity Ia is evaluated using the following facts:
The patterns , N are independently drawn from the uniform distribution over
{+1, 11... Thus, Iz is obtained by averaging exp Ni-n//2 Σ== (w²)") with respect to single pattern & (i.e. dropping the subscript μ), and raising the result to the power Ox, N..
For uniformly distributed over {+1. the central limit theorem implies that u' u'....," u² follow a zero-mean multivariate normal distribution with covariance
I [e[[unu]= = ho h =: Gab-
For n 2 3, the factor N 1l-n/2 vanishes as N + co. We therefore apply the Taylor expan- sion
Nl-n/28
(2
exp
2
0=1
N
N1-n/28
+
(u))n
(u²)n
0(N3-33/22
(67)
2
to compute the Gaussian average. Using these observations, we obtain
27
In = 1 B ""!!""
N2-n/23
Σ ((wog))
+ 0(N3-3n12)
2
2
a=1
Na, 32
= exp
End "...
Σ 2u
+ +0(N2-1/33
a.b
Na,
2 exp
Eng u..
Σ (umub22n a.b
(68)
8
which is valid for 2 3.
Evaluation of I₁ and the subshell volume. The contribution I₁ is handled together with the volume of the subshell of configurations h' ......h' h² that satisfy fixed order parameters m and qab (a, v). Specifically, insert the identity
1 x
I dm II dgab II a<b
8(E) n Nm Il 8(h h² - Ngab)
(69)
a=1
acb
19

---

into (66)) This leads to 8(5 he - -Nm*))
N
Eg'
exp
(m))
II a=1
x=1
exp
N3
(m))
x Eg II o h - Nm²)
(70)
2
a=1
Next, the subshell volume
Σ
8(E" ha Nm II a<b
6(h". h² - Ngab)
(71)
h ha a=1
evaluated using the Fourier representations
8(5' h- - -Nm") "mm)== 2Ti 1c0
dma exp maa('' h - Nm*)]
(72)
8(h h -Nqa)) 2ii
dâpa exp 4ab (h. a. h Nga+t)
(73)
100
Combining all contributions and applying the saddle-poine method, finally obtain
B
on
In Eg
extr
(m²))
Eng
N
Σ Has axt
Σ
mmm"
a=1
}}} (74)
+In Be
Σ exp
Σ aab
Gash" 1
Emaahh
magh
htt h!.....""
where extrx (f()) generally stands for extremizing f(X) with respect to X.
To proceed toward the limit v - O, we next impose an appropriate replica-symmetric (or symmetry- broken) ansatz for the saddle-point parameters.
D.1 REPLICA SYMMETRIC SOLUTION
The replica-symmetrcc (RS) solution is obtained by imposing m m. 9ab 9, ma -m qub
q in (72).. Under this ansatz, the Gaussian average becomes
By u...,
(uuubyan a.b
Mn(1) + 1)
(75)
where
[n/2]
(p)
Σ
(n 2r)! ((2r 1)!1) p n-
(76)
2r
Furthermore, using Gaussian integral identity, we obtain
E
Σ
M
h
+
m > Ehe
A
asb
a=1
20

---

v4/3 / Dz Iee (2 (vq= +mE) )
e vgva / D2 (2cosh) (vz=+ (V²++mE) m²))
(77)
where Do dz /2 /22 denotes the standard Gaussian measure.
To characterize the retrieval state that is, a (local) minimum of the Hamiltonian wewe consider the zero-temperature limit B 00. In this limit, introduce the rescaled parameters
we
F F=
K = 3
3 g).
(78)
For the case n = 3, substituting these scalings into (444 yields
1
á
Eg min H] N
lim 8-000
Eg [I 2
lim
lim
In (29
3N
8000
3N
w dv
m²
27
Fx
/ Dz Dz xI.
extr F,K 2
Km
(79)
8
The value of m determined by the extremization means the typical overlap with the retrieved pattern Egle' h//N. After performing the extremization, we obtain the fixed-point equation
m
m 2
m = erf
= erf
(80)
/Ba3s
22
which coincides with the fixed-point condition of the approximate algorithm given in (A for 72
3.
D.2 1-STEP REPLICA SYMMETRY BREAKING SOLUTION
Under the one-step spplic--ymmmerry-bree breaking (IRSB) ansatz, the replica indices 1, 1...., di- vided into v/v groups, each of size x. The order parameters in (74) are set as
Q1
if and belong to the same group,
Gab
(81)
Qo otherwise,
and similarly for dab. For m and mas retain the RS conventions m m and m. m..
Under this ansatz, the Gaussian average becomes
E(urry
z(x - 1) Mn(qi) + 2²² 2-- --1) Mn((0))
Eyy .....
= 1/ Mn(1) + σ
(82)
x
x
a.b
where Mn(p) is defined in (7G).
We also obtain
Eg
Σ
exp
Jah
Σ 1 m > a
h1t ht.....!"
-
e--
f DD
If Dy 2 cosh VVi 9o + v + m)
)
x7 2/2
(83)
As before, to characterize the retrieval state a local minimum of the Hamiltonian H-we consider the rerootemperature limit 3 oo. In this limit, we introduce the rescaled variables 1r Bx.
F 3-2
Fo 3-
K 8-
x B(1 q2),
(84)
21

---

10-
10
0.6
00
0.00
0.30
0:25
0.20
Figure 3: Typical values overlap m EE(s h2/N evaluated under the RS and IRSB ansatzes.
For the case n = 3, substituting these definitions into (ZC yields
1
a In Ee!Z"!
Ee min H lim
Eg
lim 80000 BN
lim
h
8-000
3N
Zv
extr
m²
03
[27x+ 20) +6(1 r11)]
Fix
pl (Fi -Fogo) - Km
(m. Fo, K,)
8
2
1
/ Dz In 1f Dy exp( (olllVEr - Foy Foz + #
(ox2+++))) x1)]} (85)
The RS solution corresponds to special case of the IRSB solution, characterized by the constraints
9o
91 = -0 = -q
Hence, the local stability of the RS solution can be examined by linearizing the IRSB extremum conditions with respect the small perturbations
A
40.
A
40.
around the RS saddle point. This procedure yields the stability condition
/ Dz tanh? (Vē2+ m )
9agB2q
9a39 /
a OK
(((FF++K) 2
Dr
tanh
(86)
However, this condition is never satisfied in the zero-temperature limit. Indeed, one finds
lim tanh tanh(( (B(V/=++++))) -20())=2(V/)))) (2((v==+K)
①
= 22
8-000 OK
which causes the right-hand side of the stability condition (Kf) to diverge.
This demonstrates that the RS solution is always unstable at zero temperature, implying that replica- symmetry breaking must be taken into account in order to obtain correct description of the model defined by (II) and (α).
D.3 SOLUTIONS
We numerically performed the extremization of (79) and (85) for the case n = 3. The resulting values of are plotted versus agg 303 in Fig. The figure shows that the storage capacity is
22

---

estimated as
2,3 ~ 0.252
under the RS ansatz,
2,33 ~ 0.266
under the IRSB ansatz.
23


Reviews:
[{"review_id": "avVCJzZXn0", "reviewer": "Reviewer_f4Gk", "summary": "The paper provides the first asymptotically exact dynamical analysis of dense associative memory (DAM) introduced by Krotov & Hopfield (2016), using the Generating Functional Analysis (GFA) framework.\nWhile previous works mostly studied stationary-state storage capacity, this work studies on dynamical properties like recall convergence time, attraction basin region and self-noise behavior.\nThe main results is DAMs exhibit retarded self-interaction that stabilizes dynamics and prevents the recall process from generating additional self-noise. This makes memory retrieval more robust and expands the basin of attraction.", "questions": "1. Will the argument $M=O(N^{n−1})$ become $M = O(e^N)$, if we switch the activation function to e^x?\n2. Can the authors further discuss the limitations of this work, and potential future directions?\n3. I suggest the authors to provide a table to compare existing results and their result.\n4. Can the authors explain why the existing framework under control theory not work here? Based on my understanding, previous studies shows that the DAM system typically has exponential convergence rate as long as its stable. How does the result from this paper differ from this conclusion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is of high quality, I believe it provides a really good analysis on the whole recall process which previous studies did not fully explore.\n\n1. The motivation is clear\n2. The mathematical derivation looks correct to me\n3. The results are novel to my best knowledge", "weaknesses": "1. The term \"retarded self-interaction\" lacks intuition to me, can the authors provide details on how do we interpret this term biologically?\n2. It would be interesting to see whether the framework used in the paper can be easily expanded to different non-linear activation such as the exponential function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "neVMey8Wkm", "reviewer": "Reviewer_oz6i", "summary": "Using Generating Functional Analysis (GFA), the authors derived an asymptotically exact solution that quantitatively characterises convergence time and the size of attraction basins, aspects previously under-investigated. The analysis provides novel insight into the model's robustness, showing that unlike the traditional model, pattern retrieval does not introduce additional self-noise.", "questions": "1. Could the authors' methods be simply extended to related formulations or models, e.g., Burns & Fukai \"Simplicial Hopfield networks\" ICLR 2023. (Probably it works similarly to the proof sketch of Gardner's model.)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The work provides an asymptotically exact analysis of the dense associative memory's dynamics in the large-system limit using GFA.\n2. The analysis yields explicit and quantitative results on convergence properties, such as the convergence time and the size of the attraction basins.\n3. The modern/dense model is shown to be more robust than the traditional model because the noise variance does not depend on the overlap and does not increase as the overlap grows.", "weaknesses": "1. Rather than providing full proofs, proof sketches are provided. I can see and understand most of the results, however it would be ideal to have full, rigorous proofs provided.\n2. There are some minor typos, e.g., \"confirms\" -> \"confirmed\" in L393. I suggest the authors proofread carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jnd4TAwlOO", "reviewer": "Reviewer_V2XS", "summary": "I feel like I'm slightly cheating here, since I'm somewhat plagiarizing the last paragraph of the paper. But that was an almost perfect summary, and it's more or less what I wanted to say anyway. At least I didn't use an LLM. ;)\n\nThe authors provide an analysis of the time evolution of statistical quantities (means, covariances, etc.) of the dense associative memory. For that they used the generating functional analysis, which is typically exact in the limit N --> infinity. (I'm guessing there are some cases where it breaks down, but I don't know what they are.) They veriﬁed the theoretical predictions with numerical experiments.", "questions": "No questions.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The calculation itself (which I only spot-checked) was highly nontrivial. While the technique itself is not knew, to my knowledge this is the first time it was applied to the dense associattive memory. I can also verify the statement by the author that their results cannot be captured by the standard signal-noise analysis (or at least not in my hands).\n\nThe results shed new light on the dynamics for cubic and higher nonlinearities and higher, and some interesting phenomena emerged -- such as suppression of crosstalk due to the recalled pattern, and a complete characterization of the basin of attraction.\n\nThe authors verified their theoretica predictions with numerical simulations, and agreement was good even though they used only 512 neurons. This is impressive, given that deviations usually scale as 1/sqrt(N), which in their case is 5%.", "weaknesses": "I didn't see any weaknesses. Unless you count a possible typo: it looks to me like theta_i is in a different place in Eq. 13 versus Eq. 6. But maybe that's on purpose?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAsFrILc41", "reviewer": "Reviewer_cF5v", "summary": "The work provides an analysis of the dynamical properties of the Dense Associative Memory model (DAM), focusing on the high-order polynomial function ($n\\geq 3$) from [Krotov and Hopfield (2016)](https://arxiv.org/abs/1606.01164) using generating functional analysis (GFA). The analysis of the work enables a quantitative evaluation of the convergence time and the storage capacity of DAM, shedding further light on why memory retrieval with DAM is more robust than the Classical Hopfield model.", "questions": "Overall, I think the work is interesting. But the presentation can be written much better, and it requires a lot more experimentation to help explain its mathematical contributions. See the weakness section above for my concerns/questions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. An interesting, new work which studies the dynamical aspects of DAM in attempt to provide another perspective on the storage of capacity of these models.\n\n2. The approximated or calculated dynamics are shown to be close to the simulated dynamics, illustrated in Fig. (2).", "weaknesses": "1. Limited experimentation. As far as I am aware, the experimentation is focused on the case of the polynomial order $n = 3$ while the mathematical derivations are focused on the case of $n \\geq 3$. \n\n2. The experiments are not conducted on natural datasets (e.g., CIFAR10, ImageNet, etc.) or a small subset of them. In such datasets, there exists a strong need to understand how the diversity of pattern-features affect the dynamical regimes of DAMs, which are governed by their memories, mathematically.\n\n3. What about asynchronous update? This work focuses on parallel update, which technically had been shown by [Cheung et al. (1987)](https://opg.optica.org/ao/abstract.cfm?uri=ao-26-22-4808) that it does not guarantee convergence for $n = 2$, and it is not the preferred update in [Krotov and Hopfield (2016)].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)