Exponential Dynamic Energy Network for High Capacity Sequence Memory
Authors: Arjun Karuvally
Pichsinee Lertsaroj
Terrence Sejnowski
Hava T Siegelmann
Venue: NeurIPS 2025 poster ()
Keywords: memory models
Hopfield Network
energy networks
Primary area: neuroscience_and_cognitive_science
TL;DR: we propose a continuous hopfield network with exponential capacity and analyze dynamical characteristics in an analytical framework

Abstract: The energy paradigm, exemplified by Hopfield networks, offers a principled framework for memory in neural systems by interpreting dynamics as descent on an energy surface. While powerful for static associative memories, it falls short in modeling sequential memory, where transitions between memories are essential. We introduce the Exponential Dynamic Energy Network (EDEN), a novel architecture that extends the energy paradigm to temporal domains by evolving the energy function over multiple timescales. EDEN combines a static high-capacity energy network with a slow, asymmetrically interacting modulatory population, enabling robust and controlled memory transitions. We formally derive short-timescale energy functions that govern local dynamics and use them to analytically compute memory escape times, revealing a phase transition between static and dynamic regimes. The analysis of capacity, defined as the number of memories that can be stored with minimal error rate as a function of the dimensions of the state space (number of feature neurons), for EDEN shows that it achieves exponential sequence memory capacity $\mathcal{O}(\gamma^N)$, outperforming the linear capacity $\mathcal{O}(N)$ of conventional models. Furthermore, EDEN's dynamics resemble the activity of time and ramping cells observed in the human brain during episodic memory tasks, grounding its biological relevance. By unifying static and sequential memory within a dynamic energy framework, EDEN offers a scalable and interpretable model for high-capacity temporal memory in both artificial and biological systems.

Exponential Dynamic Energy Network for High Capacity Sequence Memory
Arjun Karuvally
Pichsinee Lertsaroj University of Massachusetts Amherst
Salk Institute for Biological Studies akaruvally@salk, edu
Terrence J. Sejnowski Salk Institute for Biological Studies terrySsalk.
Hava T. Siegelmann University of Massachusetts Amherst havaCumass, edu
Abstract
The energy paradigm, exemplified by Hopfield networks, offers a principled frame- work for memory in neural systems by interpreting dynamics descent energy surface. While powerful for static associative memories, falls short modeling sequential memory, where transitions between memories essential. We introduce the Exponential Dynamic Energy Network (EDEN), a novel architec- that extends the energy paradigm to temporal domains by evolving the energy function over multiple timescales. EDEN combines static high-capacity energy network with slow, asymmetrically interacting modulatory population, enabling robust and controlled memory transitions. We formally derive thort-timescale energy functions that govern local dynamics and use them to analytically compute memory escape times, revealing phase transition between static and dynamic regimes. The analysis of capacity, defined the number of memories that can be stored with minimal error rate function of the dimensions of the state space (number of feature neurons), for EDEN shows that it achieves exponential sequence memory capacity O(> outperforming the linear capacity ((N) of conventional models. Furthermore, EDEN's dynamics resemble the activity time and ramping cells observed in the human brain during episodic memory tasks, grounding its biological relevance. By unifying static and sequential memory within a dynamic energy framework. EDEN offers scalable and interpretable model for high nigh-capacity temporal memory in both artificial and biological systems.
Introduction
Memory is crucial element of cognition that is essential for learning, reasoning, and decision-making. Understanding and replicating the human ability to store and recall information is long-term chal- lenge in both biological and artificial intelligence (AI). The energy paradigm, introduced by Hopfield and Amari 40 years ago, revolutionized memory modeling by characterizing the dynamical behavior of neural networks using an energy landscape 21.. According to the energy paradigm, a stimulus instantiates network state that the state travels down the landscape until a minimum is reached. This minimum state is defined as the memory of the network. The energy approach to memory modeling represented significant advancement of our scientific understanding of memory by offering an intuitive understanding of network dynamics, with added theoretical guarantees of stability. The distdvantage was that the number memories that can be reliably stored was only small fraction of the number of neurons and scaled linearly with the increase in neurons. This limited the applicability of energy
energy landscape. The neurons then interact with each other such
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A
network state
Exponential
slos
state
Dynamic
O00
Enengy
Network
ein
ci++1
3
Static Energy Natwork
0
3
Input/
the static energy paradigm, single memory retrieved from collection stored memories
Output
3
3
3
3
Figure I: Schematic Model and Energy Landscape Behavior of Dynamic Energy Networks: A. The dynamic energy network, EDEN, has asymmetrically interacting slow neurons providing information about the next memory in the sequence to the two fast neural populations. B. Static energy-based networks are used as models of human associative memory where single memory associated with provided stimulus is recalled. EDEN, without the slow population, is a static energy network that retrieves single memory from collection of stored memories. The system's 's state (u,), represented by the blue ball, descends the energy surface until stable memory (energy minimum state "2") is reached. After retrieval, the state of the system does change and stays "2" C. Dynamic energy networks enable associative sequence memory, where the associated memory along with its sequential neighbors are recalled. In EDEN, the energy surface changes response the state of the system, causing the minima of the energy surface to change over time (from "2" to "3"), resulting transitions between memories.
networks despite their theoretical advantages. Further, the dynamics on the energy surface guaranteed single final memory, precluding any temporal behavior in the memories. Further research in im- proving these networks proceeded two independent directions. In one direction, researchers sought develop techniques improve the limited memory capacity of the original neural network by proposing modifications to how neurons interact in the network. In the second direction, researchers sought to create alternative formulations energy function such that sequences and temporal aspects of memory can also be modeled with similar theoretical guarantees.
Improving capacity has been central the development of memory models. In the context of Hopfield networks, capacity is defined as the maximum number of memories that can be stored with minimal errors function of the number of dimensions in its state space (the number of visible neurons). Earlier studies revealed that the limited capacity of the classic Hopfield network was due to significant crosstalk between the memories resulting in energy functions with many spurious minima. A major breakthrough in capacity came with the introduction of higher order terms in the energy function that separated the contribution each memory the energy minimum 6 in polynomial capacity scaling and dense networks networks that store more memories than the number of neurons [13]. Further studies introduced exponential terms, greatly increasing memory capacity and enabling practical applications [14] 13 Currently, energy networks are used in AI with applications in sarge-ccale natural language processing [15]
2 resulting
16]. computer vision [17]. and lifelong- learning systems [18 19 as reliable external memory storage. Further, the self-attention mechanism transformer architectures have been shown to be functionally equivalent to the exponential memory capacity network providing insights into their mysterious capabilities [20] 21] The success story of high-capacity static energy networks demonstrates how utilizing the energy paradigm benefits advancement and practical applications.

---

Despite these advancements in the energy paradigm, state-of-the networks are still restricted to retrieving single memories from a collection of stored memories. Reconciling the single stable memory states in the energy paradigm with the dynamic states required for modeling sequences 25 Over the years, there have been several solutions
remains a significant challenge [22 proposed for the challenge. One proposal introduced networks combining symmetric interactions, asymmetric interactions, and delay signals to produce temporal behavior [26] 22]. These proposals succeeded in creating networks that exhibited sequential state transitions, but the energy paradigm could not be applied to these cases as occasionally the network traveled up the energy surface. Another proposal introduced noise into the dynamics for enabling transitions out of memory basin of the energy surface [28] 29
31 32 33]. The energy paradigm applied to these models revealed a lowering of the energy barrier between states more noise is added to the system. Without theoretical insights obtained from the application of the energy paradigm, the modifications needed to improve sequence capacity could not be found. As result, extant sequence networks have capacity much lower than the number of neurons. Developing an energy principle for temporal memory networks will enable memory researchers to develop networks that are capable and aligned with experimental data. It will also enable artificial intelligence researchers to develop capable external memory stores.
30
Our work extends the energy paradigm to temporal memories by allowing the energy surface to change slowly with time. This approach was previously proposed experimentally in [34] and some computational properties studied in [35]. In contrast to the classical energy paradigm, the memories in the dynamic energy networks can lose or gain stability over time, resulting in stability in two timescales. In short timescales, the current memory is always stable, with the energy function guaranteeing convergence and robustness to noise. In longer timescales, the energy surface changes to create a new minimum, destroying the current minimum. The network state changes in response, resulting in stable transitions between memory states. Our analysis of the proposed dynamic energy network shows that (1) The network's s dynamical behavior is well characterized by the short-timescale energy functions assembled piecemeal for long-timescale dynamical behavior, (2) The energy function provides precise analytic computation for the time required escape from a stable memory state and the conditions necessary to exhibit memory transitions, (3) The network capacity scales exponentially in the number of neurons, significantly outperforming existing sequence memory networks, (4) The network populations have biological implications, showing strong behavioral correlations to the activity of cells found in human episodic memory experiments. The new paradigm thus enables the development of biologically relevant sequence memory networks with improved storage capacity.
Our work also provides theoretical insights into current approaches sequence memory modeling. Notably, we extend our earlier work on sequence memory (34] with theoretical analysis about dynamical behavior, and rigorous claims of dense capacity. Another approach used in (36) has similar multiple-timescale dynamics where the sequences are learned from the stimulus and the transitions are governed by successive bifurcations. A more recent work [37] introduced a similar softmax function with asymmetric synapses for dense capacity in a discrete network without using the energy arguments. Our work reveals that the successive bifurcations hypothesized by [36] are due to the
change in stability of the energy landscape, and the capacity increase observed by [7 may be due to separating the memory contributions to the energy functions.
2 Results
2.1
Exponential Dynamic Energy Network (EDEN)
To develop dynamic energy networks with exponential capacity, we incorporated a slow-changing signal that interacts asymmetrically with an exponential capacity static energy network introduced in prior research [21). The resulting model is system of interacting neurons with slow and fast timescale neural populations. The slow timescale population modulates the energy surface for the fast timescale population resulting in system with a temporally varying energy function.
Mathematically, our model is a two-population neural network. The first population consists of a feature layer (input/output layer) represented by the vector v and a hidden layer represented by the vector h. There are N neurons in the feature layer and P neurons in the hidden layer (one for each memory that needs to be stored in the network). This two-layer organization of the fast networks is primarily motivated by a recent general theory of energy- networks [3. The feature and hidden layer make the fast timescale population and are part of the exponential capacity static energy network.
3

---

Global Energy Dynamics of EEDE
Local Energy Dynamics
of EDEN
E4
0.8
0.6
slow feature layer (s)
17
0..4
E3
E2-
30
dolay signili
feasure
50 PC1
minimu
-00
Es
E1
E2
E3
0.5 normalized time
feature layer(v)
Figure Simulation of EDEN reveals robust transitions between memory states and the existence of local energy functions: EDEN is simulated to store and retrieve simple sequence of 5 MNIST digits in numeric order. The global energy surface with both slow and fast populations of EDEN shows the neural state traversing valley of the energy surface with occasional energy- increasing regimes. B. The dynamical behavior the memory overlaps of the fast population (m,, = + ( of EDEN and the analysis of the first principal component (PC1) of the time cvolution of its fixed points show the fast population (blue cross) converging to the instantancous minimum of short-timescall energy functions (red circles). The short-timescale energy minimums are modulated by the slow population. As the slow population approaches the current state of the fast population, the energy minimum switches to the sequentially connected memory. Over time, energy changes slowly so that the fast population has sufficient time to relax its instantaneous minimum. The oong-timescale dynamical behavior of the network can then be assembled from the short-timescale behaviors.
these
In the fast population, the hidden layers instantaneous (very fast) enabling rapid information transfer and follow the state of the art practices in developing energy networks. The interaction between the feature neuron i and the hidden neuron μ is symmetric and is represented by the synaptic weight Eip. The vector obtained by c(") for fixed f and i 2 ... is the μM stored memory (energy minimum) of the system. We analyze the network in the paper under the assumption of
Rademacher distributed memory patterns Pr Pr[s/ [(()) [+ P -11 1/2.
The population of slow neurons represented by the vector is the continuous delay signal from the feature neurons. Therefore, there are N delay neurons. This slow signal retains information about the previous memory state with characteristic dynamical timescale Ta. We consider the when the timescale the slow neurons is higher compared to the feature neurons (To > Ty). This timescale difference enables the existence of short timescale energy functions. The neurons in the slow population interact with the hidden layer neurons through the synapses represented by the vector c(a-1), For simplicity, we assume the memories are arranged c(p-1) + (()) for μ p>> 1 and ((P) + ((1), where P is the number of memories in the sequence to be stored. For exponential memory capacity scaling, the softmax activation function is used for the hidden layer. The evolution of the resultant network given by the following set mathematical
single long circular sequence with
equations with Latin characters indexing the feature and slow neurons, and the Greek characters

---

Escape Time Characteristics of EDEN
escape
0.0
400
250
184
500
300
36
400
Ta
7f
dynamic
static
300
oscapc
200
TalT: 30
150
200
Ta/ 50
100
transttoon
phose
100
50-
0.0
0.2
0.5
0.7
1.0
1.2
0.
Cla
Figure 3: Escape Time Characteristics of EDEN under different parameter regimes: (left) The analysis of the escape times (in Ty units) of EDEN under different parameter settings shows two different dynamic regimes. When the coefficient ratio a,/ac > EDEN has static memories where the dynamic behavior converges to one of the stored memories without any transitions. When the coefficient ratio as/ 0s/00, < 1, EDEN memory transitions. (right) We take sample cross sections of the phase diagram, shown by the colored horizontal lines the left. The average time required to escape a memory state characterized by the timescale (Ta/Ty and the coefficient (ag/ac /a.) ratios. The analytical escape times (the solid lines) computed from the energy function show good agreement with the experimental values (the points) with a mean absolute error of 5.967, units.
indexing the hidden layer neurons.
de; Ty dt
Σ
P
(a)
exp(hp)
E, exp(tr) 1is,
#=1
hy
a2
(H) si
ii + ac Σ
((1-1) 36
(1)
1=1
ds;
Ta
Uj
Sj.
dt
The interaction strength coefficient for the self-memory interaction is a, and for cross-memory inter- action is oee. The self-memory interactions connects memory with itself (g(#) with g(H)), stabilizing the current memory of the network. The cross interactions drive the asymmetric interactions (g(M-1) with <(a) which causes state transitions. This dynamical system of interacting neurons has the following energy function for the fast population (Appendix B
E(u) =
(v:)
log
P Σ 1=11
exp
as
Σ
5
ac
c
e(u-1)
Si
(2)
2
state energy
energy
The first term represents the state energy of the network, and the second term represents the interaction energy from the synapses. The interaction energy now contains additional terms for the slow population compared the energy function of typical Hopfield-type network. The interaction energy from the fast population generates minima near similar memory (defined as the memory with most overlap mW), while the slow population generates minima near the sequentially connected memory. The dynamical behavior of the overall system is characterized by the relative strengths of these two interaction terms. With the network's dynamics defined, we now analyze how its behavior differs from that of static energy networks.
5

---

10"
10
caperity
101
linit
0.7)
1.0)
25
number of neurons
Figure 4: Exponential Sequence Memory Capacity of EDEN:
The plot shows the fixed point
capacity in the log1o scale for EDEN simulated with different a (with a, = 0.999 oi) compared with the reference network when small errors (6 < 10-3) are tolerated. The analytic curves are shown solid lines and experimental values points. The reference network capacity scales linearly with the asymptotic rate of ((N) (dotted orange line), while EDEN scales exponentially with the asymptotic rate ((>^) in the number of feature neurons. The exponent base is higher than the
limit (7 > 2), enabling EDEN to reach the available capacity limits of 2 (dotted blue line) in the asymptotic limit of the number of neurons.
2.2 Energy Dynamics
Conventionally, an energy function precludes any temporal memory behaviors, as the dynamic requirements of temporal memories conflict with the convergent dynamics found in systems with an energy function. However, this argument assumes that the energy function that characterizes the behavior of a system is constant. The theoretical analysis of EDEN reveals that the long- term dynamical behavior of the network can be well explained piecemeal by short-term energy functions.
To analyze how the dynamics of the energy change with the introduction of the slow population, take the time derivative of the energy function from Equation / along the dynamical trajectory of the system. The dynamical evolution of the energy function after separating the two timescales is shown below (see Appendix for the full derivation).
dE Σ
du
exp(I)
c( 1) ds;
dt T
dt
Os
i En exp(hx)
fast timescale (F)
slow timescale (9)
The two terms of Equation 3 which we label by F' and S separate the contributions of the fast and slow timescales. Excluding the term, the fast population will have one of two possible behaviors. When the sign of F is negative, the population converges to single stable state corresponding the minimum the energy function. When the term is O, the system moves an iso-energetic (states that have the same energy) trajectory without convergence. In this paper, we focus only the case of convergent behavior. We find that the case of non-convergence does not arise in the simulations.
The slow population influences the second term, S. When the slow timescale is longer compared to the fast timescale (under the condition that Ta >> Ty). which we assume in the paper, the network exhibits non-increasing energy function and the effect of S is effectively negligible. The analysis reveals two roles the slow population plays in the network dynamics: (I) The slow dynamical nature helps to stabilize the dynamics of the fast population a memory state (2) The asymmetric interactions of the slow population changes the energy surface to create minima and destroy old minima, inducing memory transitions. These two functions result in network with stable transitions between memories.
the energy surface, enabling it to converge to
In our numerical simulations, we consider settings of the slow timescale to be high enough for the slow neurons change sufficiently slowly for the energy function to characterize the dynamics but not so high as to prevent the system from exhibiting state transitions in reasonable time. Figure

---

shows the energy function behavior of EDEN and the dynamic behavior of the feature to memory overlaps m >; c."') 0;. The analysis reveals that although single energy function does not characterize the global temporal behavior of the network, the local behavior is well described by short-timescale energy functions. Analysis of the fixed points of the energy surface predicts when an instability leads to memory transition and governs where each memory transitions to next. A global behavioral characterization can then be obtained piecemeal from these local characterizations.
2.3 Escape Time Characterization of EDEN
To determine the variety of dynamical behaviors exhibited by EDEN, we analyzed how its parameters the timescales (Ta; Ty) and the interaction strength coefficients (aee ag), influence the escape time the time the network spends in a memory state before transitioning to the next. To formalize the average escape time, we define that the network state at some time v(t) is in a memory state μ arg max, Σ, e(')) (t))), that is, if the n memory has the maximum overlap with the network state compared to all other memories. Formally,
te( max {t arg max Σ Eu((E)
when v(0) = -("-1)), s(0) = e(u-2) The average escape time is defined as the time the network stays in memory state fr averaged across all the memories. Computing the escape time for nonlinear dynamical systems like EDEN a significant challenge. However, since we have access to the system's 's energy function, we compute escape time analytically using the time required for the energy function to change minima from memory state g(u-1) to a memory state g(H). The escape time is obtained by evaluating the time taken for the energy contribution of c(w-1) to be lesser than g(a) when the network initially starts at g(u-1) and eventually transitions to g(r). That is, exp as E.E(("u)yr + ae Σ, <((-2))s) s.) < exp (a. E. + a Σ, Eee!-1)) 8; We obtain the following analytic expression for the expected escape time, assuming the effect of transients in the fast population is negligible (details in Appendix (D) and that the transitions are Markovian. These assumptions are reasonable, transients are observed to be almost instantaneous relative to the amount of time spent a memory state (in Figure 2) and the time spent is enough for the network history decay. The average escape time has the analytic expression given below.
in the slow timescale limits we consider in the paper, the memory
Ta T
as
(te)
In
(5)
The phase diagram in Figure constructed from the analytic escape time shows that the ratio of coefficients 2* uniquely determines the emergence of two different regimes in the dynamical behavior of EDEN. In the static memory regime, when ds > 1, the cross- coos-interaatton strength is weaker than the self-interaction strength, resulting in infinite escape time and EDEN exhibiting the dynamical behaviors of a static energy network. For Se < 1, the eross-interaction strength greater, and EDEN enters the dynamic memory regime. The coefficient fraction and the slow-fast timescale ratios define the escape time in the dynamic memory regime. The escape time is sufficiently high for larger timescale ratios to observe stable transitions, making it ideal for storing memory sequences. On the other hand, reducing the slow timescale parameter results in noisy dynamics between memories characterized by short escape times. Ensuring that the coefficients are close to the phase transition boundary enables the resulting network to exhibit stable transitions with a long time spent in memory states.
a.
2.4 EDEN has exponential capacity
Now that we have network that follows an energy function, we evaluate how well the capacity guarantees of the exponential static energy networks translate to the dynamic case. For simplicity, we compute the capacity for networks at the phase change boundary As + 1. The networks at the transition boundary have infinite escape time, resulting in the slow population completely forgetting the previous memory state at the transition point. This enables precisely defining the slow population's state at the transition point. For network at the phase boundary, the fixed point capacity is defined as
7

---

the maximum number of memories that can be stored as function of the size of the state space (N) of the networks. As an added nuance, this ignores the number of hidden neurons in the framework. This follows extant definitions of capacity. Minor errors are allowed in the retrieved memory, with defining how close the fixed point is to the target memory state and defining the rate of tolerable bit emoos. Mathematically. the capacity is defined as
c(N,e, = max Pr((Pl(( (.50) 2 1 of 2 1 =1-
(6)
with,
v(0) g(u-1), and s(0) 0a elu-2)
ac
The factor
02 for the slow population was obtained by solving for its state analytically during state transition (Equation 24 in the Appendix). We then compare the fixed point capacity of EDEN with the following reference network.
de' Ty dt
(as
Σ cسu) (5a0) o(1)) + ac Σ ( 81)
(8)
ds;
Ta
Uj Sj.
dt
where the nonlinearity a is defined as
-1
x x<-1
(
I
-15051
x>1
Minor variations of this reference network have been previously studied multiple timescale models of sequence memory [26]. 36 (38) making it suitable as a proxy for existing multiple timescale sequence networks. The notable difference between the reference network and EDEN is the absence of a hidden layer and the softmax activation function. As a result, the reference network has linear interaction between the neurons in the memory layer.
The analytic form for the capacity of the network is obtained from a given N, e, as (details in Appendix E.2.2
exp((rr exp(a) cosh(ar) cosh(a)
N-1
CEDEN k)(c,)
(10)
where is a constant independent of N in the large N limit. The capacity is exponential in the exp(ar) exp(a)
number of neurons with the asymptotic rate of O(7 where cosh(ar) cosh(o] The maximum capacity possible for a network with neurons is 2 and the exponent > for most choices of a suggests that EDEN reaches the maximum possible capacity in the large N limits. The capacity of the reference network similarly obtained as
e88 Ce -N In(N)
(1I)
The capacity of the reference network is only linear in the number neurons. For large the asymptotic capacity is O(N), which is only linear in the number of neurons. The analytic results are compared against simulations of networks with N E {10, 12...35 in Figure The results show an exponential improvement in the scaling behavior of EDEN when compared to the reference network. Further, EDEN approaches the available limit of 2 memories for higher settings of a. Due to computational constraints, the maximum number of memories to be stored was limited to 10%.

---

A
Time cell like behavior of
B
Ramping behavior of delay
hidden
hidden neuron index
normalizae lized fring rate
0.8-
0.6-
C2
0.4
C3
0.2
0-
C
10-
D.
E
En
0.8
15-
m
0.6
02
0.0
0.2
0.5
0.7
1.0
0.5
normalized time
normalized time
Figure 5: The EDEN neural populations shows behavioral similarity to cells observed in human episodic memory experiments: A The heatmap of the hidden layer neuron activity ordered by time shows time-sensitive behavior analogous the time cells observed in human episodic memory retrieval experiments of [39]. B The slow layer neurons ramp up their activity until it reaches the current memory which in turn induces the transition the next memory. Rather than an instantaneous drop in their activity, the slow layer slowly ramps down to stabilize the feature layer state on the next memory. This ramp up and ramp down activity is analogous to the activity of ramping cells observed in episodic memory experiments [39]
3 Biological Relevance
Episodic memory is the human ability to remember when and what happened during specific events through an autobiographical recall of information [40]. Episodic memory is evaluated in humans using list recall tasks [41 42 43 As an essential component of cognition, the role brain cells in supporting episodic memory is important question. Experimental studies in human episodic memory have identified time cells and ramping cells the hippocampus and entorhinal cortex as playing role in encoding and retrieving episodic memories [39]. Time cells activate in sequence corresponding to the order of the events being recalled and are hypothesized to encode the temporal information of the recalled memory. Ramping cells also activate the timing of memories but show only a gradual increase or decrease activity encoding time in longer timescales. With our theoretical setup for retrieving sequential memories, list recall task. Our findings show that the fast hidden neuron population and the slow population show behavioral characteristies similar those of the time cells and ramping cells observed in neurobiological experiments supporting episodic memory. This indicates that the EDEN architecture may be used develop theories and simulate neurons for evaluating episodic memory the brain.
can analyze the retrieval aspect of the
Figure shows the behavioral correlations between the two populations of neurons in EDEN and the cells found in human episodic memory experiments. Specifically, the dynamic nature of EDEN's hidden neurons lines up sequentially like time cells, reacting to the timing of the stimulus during memory retrieval. The slow neuron population in EDEN shows gradual rise and fall in activity, analogous to the ramping cells, encoding the timing context during the retrieval of memories. Our theoretical analysis of EDEN shows that the slow population helps stabilize the retrieved memory on the energy surface and directs the transition between retrieved memories. Moreover, the time takes for memories to shift from one state another in EDEN influenced by the ramping cells' timing and the strength of their connections to other neurons. The theoretical insights from EDEN suggest that ramping cells may play a role in stabilizing and directing transitions addition to simply encoding temporal information as hypothesized from experiments. The time cells, on the other hand, being the fast population only react to the state of the slow population and play role in identifying and arranging the retrieved memories in time.
4 Discussion
The Hopfield-Ane networks and the energy paradigm have provided foundational knowledge of neural networks. However, addressing the diverse behaviors found in neural networks, is imperative

---

to evolve the energy paradigm beyond its traditional roots of static memory retrieval. We suggest EDEN as model that takes a step in this direction by introducing glow-timescale dynamics with asymmetric memory interactions to the energy function, creating new dynamic energy paradigm. The results point the enhanced capacity and understanding enabled by the new dynamic energy paradigm. With these results, we posit that the network and theory could shed light on other temporal characteristies of human memory experiments. the simulations suggest that dense memory may be used in Al applications requiring robust, high- capacity sequence memory storage and retrieval. The proposed energy paradigm provides universal framework for memory computations in static and dynamic cases. Further, the biological relation of EDEN provides a path for analyzing the episodic memory experiments in a tractable framework that will inform future studies on memory. In future studies, plan to generalize the energy networks further to complex, realistic sequences and dynamic working memory settings.
addition to the potential impact on neuroscience,
Limitations. As theory of dynamical behavior of non-linear system, we make key assumptions that simplify our mathematical analysis. (1) The synaptic strengths of the neuron interactions are fixed and does not vary during training, in actual biological systems synaptic strengths can change due to short and long term potentiation effects and consolidation (2) The timescales of symmetric and asymmetric interactions are separate this allows use to treat the asymmetric part as slowly evolving and change the energy function of the symmetric network is response. In human brains, there are different timescales for information processing but the timescales may not be perfectly separated as distinct slow population of asymmetric connections and fast population of symmetric population, (3)
Binary memory- we assume Rademacher distributed patterns for theoretical exposition following related works in the field, although the theory can be similarly worked out for other distributions (4) Markovian State Transition in deriving our capacity bounds, we assumed that the network spends enough time in a memory state that the historical trajectory information is lost and the state transitions are purely Markovian in nature. Further, the capacity bounds we formulated shows how the maximum number of memories (number of hidden neurons) scales with the number of visual neurons following previous results in the field. The number of hidden neurons required for storage however grows only linearly in the number of hidden neurons.
References
[1] J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(5):2554-2558 1982.
[2] Shun-ichi Amari. Neural theory of association and concept-formation. Biological Cybernetics, 26:175-185, 2004.
[3] Robert J. McEliece, Edward C. Posner, Eugene R. Rodemich, and Santosh S. Venkatesh. The capacity of the hopfield associative memory. IEFE Trans. Inf. Theory, 33:461-482, 1987.
[4] Viola Folli, Marc Leonetti, and Giancarlo Ruoeco. On the maximum storage capacity of the hopfield model. Frontiers in Computational Neuroscience, 10, 2016.
[5] Amit, Gutfreund, and Sompolinsky. Spin-glass models of neural networks. Physical review. A. General physics, 32 2:1007-1018, 1985.
[6] Baldi and Venkatesh. Number of stable points for ppin-glasses and neural networks of higher orders. Physical review letters, 58 9:913-916 1987.
[7] H. H. Chen, Y. C. Lee, G. Z. Sun, H. Y. Lee, T. Maxwell, and C. Lee Giles. High order correlation model for associative memory. 1987.
[8] Kanter and Sompolinsky. Associative recall of memory without error.. Physical review. A, General physics, 35 1:380-392, 1987.
[9] E. Gardner. Multiconnected neural network models. Journal of Physics A, 20:3453-3466 1987.
[10] L. F. Abbott and Yair Arian. Storage capacity of generalized networks. Physical review. A. General physics, 36 10:5091-5094, 1987.
[11] David Hom and Marius Usher. Capacities of multiconnected memory models. Journal De Physique, 49:389-395, 1988.
10

---

[12] Dmitry Krotov and John J. Hopfield. Dense associative memory for pattern recognition. In Proceedings of Thirtieth Conference on Neural Information Processing Systems, 2016.
[13] Dmitry Krotov and John J. Hopfield. Large associative memory problem neurobiology and machine learning. ArXiv, abs/2008.000 069996 2021.
[14] Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, and Günttr Klambauer. Modern hopfield networks and attention for immune repertoire classification. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20,, Red Hook, NY, USA, 2020. Curran Associates Inc.
[15] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training (2018), 2018.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.00405. 2019.
[17] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ArXiv, abs/2005. 12872, 2020.
[18] Tyler L. Hayes, Giri Panamoottil Krishnan, Maxim Bazhenov, Hava T. Siegelmann, Terrence J. Sejnowski, and Christopher Kanan. Replay in deep learning: Current approaches and missing biological elements. Neural Computation, 33:2290 33:2908-29955 2021.
[19] Manoj Acharya, Tyler L. Hayes, and Christopher Kanan. Rodeo: Replay for online object detection. ArXiv, 2et/20088064339 2020.
[20] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
[21] Hubert Ramsauer, Bernhard Schafl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlovi's, Geir Kjetil Sandve, Victor Greiff, David P. Kreil, Michael Kopp. Güinter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. ArXiv, abs/2008.02217, 2021.
[22] Mikhail Rabinovich, Ramón Huerta, Pablo Varona, and Valentin Afraimovich. Transient cognitive dynamics, metastability, and decision making. PLoS Computational Biology, 4, 2008.
[23] Daniel Durstewitz and Gustavo Deco. Computational significance of transient dynamics in cortical networks. European Journal of Neuroscience, 27, 2008.
[24] Nadav Ben-Shushan and Misha Tsodyks. Stabilizing patterns in time: Neural network approach. PLoS Computational Biology, 13, 2017.
[25] Giancarlo La Camera, Alfredo Fontanini, and Luca Mazzucato. Cortical computations via metastable activity. Current Opinion in Neurobiology, 58:37-45, 2019.
[26] David Kleinfeld. Sequential state generation by model neural networks. Proceedings of the National Academy of Sciences of the United States of America, 83 24:9469-73, 1986.
[27] Sompolinsky and Kanter. Temporal association in asymmetric neural networks. Physical review letters, 57 22:2861-2288 -2644, 1986.
[28] Paul Ian Miller and Donald B. Katz. Stochastic transitions between neural states in taste processing and decision-making The Journal of Neuroscience, 30:2559 2570, 2010.
[29] Lauren M Jones, Alfredo Fontanini, Brian Sadacca, Paul Ian Miller, and Donald B. Katz. Natural stimuli evoke dynamic sequences of states in sensory cortical ensembles. Proceedings of the National Academy of Sciences, 104:18772 18777, 2007.
[30] Paul Ian Miller. Itinerancy between attractor states in neural systems. Current Opinion in Neurobiology, 40:14-22. 2016.

---

[31] Jochen Braun and Maurizio Mattia. Attractors and noise: Twin drivers of decisions and multistabilltty. Neurolmage. 52:740-754 2010.
[32] Malbor Asllani, Renaud Lambiotte, and Timotéo Carletti. Structure and dynamical behavior of non-normal networks. Science Advances, 4, 2018.
[33] A. Emin Orhan and Xaq Pitkow. Improved memory in recurrent neural networks with sequential non-normal dynamics. ArXiv, awwp///....1.//// 2020.
[34] Arjun Karuvally, Terrence Sejnowski, and Hava Siegelmann. General sequential episodic memory model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15900-15910. PMLR, 23-29 Jul 2023.
[35] Lukas Herron, Pablo Sartori, and KingKan Xue. Robust retrieval of dynamic sequences through interaction modulation. PRX Life, 1:023012, Dec 2023.
[36] Tomoki Kurikawa and Kunihiko Kaneko. Multiple-timescale neural networks: Generation of history sequences and inference through autonomous bifurcations. Frontiers in Computational Neuroscience, 15, 2021.
[37] Hamza Chaudhry, Jacob A. Zavatone- Veth, Dmitry Krotov, and Cengiz Pehlevan. Long sequence hopfield memory. ArXiv, http/223060053322 2023.
[38] Tomoki Kurikawa. Transitions among metastable states underlie context-dependent working memories in a multiple timescale network. In ICANN, 2021.
[39] Gray S. Umbach, Pranish A. Kantak, Joshua Jacobs, Michael Kahana, Brad E. Pfeiffer, Michael R. Sperling, and Bradley C Lega. Time cells in the human hippocampus and entorhinal cortex support episodic memory. Proceedings of the National Academy of Sciences of the United States of America, 117:28463 28474, 2020.
[40] Endel Tulving. Episodic memory: from mind to brain. Annual review of psychology, 53:1-25,
2002.
[41] David Wechsler. A standardized memory scale for clinical use. The Journal of Psychology, 19:87-95, 1945.
[42] Gordon Chelune, Robert A. Bornstein, and Aurelio Prifitera. The wechsler memory revisedd 1990.
[43] Kathryn L. Cabbage, Shara Brinkley, Shelley I Gray, Mary Alt, Nelson Cowan, Samuel Green, Trudy Y. Kuo, and Tiffany Hogan. Assessing working memory in children: The compre- hensive assessment battery for children working memory (cabe-wm). Journal of Visualized Experiments JoVE, 2017.
[44] David Sussillo and Omri Barak. Opening the black box: Low-dimensional dynamics in high- dimensional recurrent neural networks. Neural Computation, 25:626-641 2013.
[45] Dimitri Petritis.
Thermodynamic formalism of neural computing. 1996.
A
Methods
In the paper, we analyze EDEN using the theoretical framework of non-linear dynamical systems and some new tools obtained by extending the concept of energy functions to the temporal case. The simulations were coded in Python and run in the Unity supercomputing cluster. The github repo for running the capacity experiments can be found at http:://jithhb..cmmmmamuuua/yy// torch
12

---

A.1 Simulations
For numerical simulations of network state dynamics, we used the Euler integration procedure with step size of 0.01. The memories in EDEN are defined as random binary vectors with each dimension of the memory in the model drawn from the Rademacher distribution Pr ([(()) - Pr Pr(s,")
- - 1/2. The similarity between the population activity and each memory is evaluated using the average overlap (Mattis magnetization) of the neural activity with each of the stored memories, defined as m N وj=1, 5)"d) 2, where 5/") is the n* memory in the system and N is the number of feature neurons, æ can be either the state of the feature neurons or the slow population.
These memories are organized as long cyclic sequence episodes: g(1) + g(2) - ...g(P) + g()) The input cue to the system the memory g()), which is initialized as the feature layer state. The slow population is initialized to O.
For Figure 5 the simulations were run with N 100, a, 0.98, a 1.0, 1.0, and Ta = 20.0. The code for the simulations is available in the repository: anonymous repo
A.1.1 Fixed point analysis
We used fixed point finding algorithm to find the fixed points of the energy surface for the fast population at each time step [44]. The algorithm uses an iterative process to find the fixed points of the energy surface evaluated from a given position the energy surface. Starting from the neuron state on the energy landscape, the state is updated to follow the direction of the energy gradient till no more updates possible, indicating convergence fixed point the energy surface.
A.2 Capacity Experiments
To evaluate capacity, we ran simulations estimate the probability of errors in retrieving single bits
Pr (t.)) </-) 2 1 - +)) for the fixed point error rate -1033 For each neuron setting N {10, 323 and the number of memories from e 21.2..2*), the probability is estimated using Monte Carlo simulations. 100 seeds of memory initializations were taken with the memories sampled without replacement to avoid confusion in the retrieved memory sequence. After evaluating the single-bit error probability, the maximum number of memories to be stored is computed for an error rate of 5 10-3 The precise setting of and ō contribute only linearly to the exponential capacity 45 37
B Energy Function Dynamics
The introduction of asymmetric synapses to the symmetric Hopfield network means that the standard energy minimization argument does not hold for EDEN. However, we find here that under sufficiently slow-changing asymmetric interactions the energy argument is valid in short-timescales. To illustrate this, we analyze the derivative of the energy function with respect to time to uncover how the energy function behaves along the dynamic trajectory of the system
dE
dv
1
Σ
(()) de:s + a
(1- ds;
dt U4
dt
Ol
Ev
assi dt
dt
(12)
dE
dt
Ui
dv
dt
Σ
E,
Zy,
20
du;
dt
as
ac Σ
Σn Zx
Ey (a- ds;
St
dt
(13)
i
dE
dup
Vi Σ
Zp
((u)
ac
Zz
(
ds;
(14)
dp
Σ
2v
aaa
Σx xx
dl
13

---

dE
dt =-Σ
T
dv
+ ag
2p
c(u
1)
ds:
(15)
dt
0s
Fx
dt
f.p
The energy function dynamics splits into two terms- one term, which is always negative (analogous the case of standard Hopfield networks), and the other term, which depends on the rate of change of the slow signal. In the adiabatic limit of the slow signal, the negative term dominates and the network dynamics always converge on the energy surface.
C
Slow Population Dynamics
The slow population dynamics is linear ODE, which can be solved exactly analytically under the fast 02 assumptions
Td ds;
= 12
81
(16)
dt
1
1
ds;
8;
dl
04 dl
(17)
Ta
Td
Use integrating factor exp
7.2
1
1
t
1
1
exp
ssp
+
exp
8i
dl
exp
Us dl
(18)
Ta
Td
Ta
Ta
Ta
d d(B5 81 exp
( (
)
Ta
Ta
exp
T
Ui dl
(19)
Integrate both sides
[ d ( (scexp)
(
t.
1
{ exp
8 (1(s)
-
ds
(20)
Ta
Ta to
Ta
t Ta
1
f exp
8 Ta
8, exp
(s(s)
ds
(21)
Ta
to
1
10 Ta
1
/"
(1(1) exp
(1)) exp
+
exp
(z(s) ds
(22)
Ta
Ta
Ta
In 11
/' exp
st(t) = (f(o exp
Ta J
0y (s(s)
ds
(23)
Ta
Ta
Without the input signal s, the network a continuous-time version of exponential static memory 21 and hence has the same capacity guarantees. For analytical simplicity, assume circularly connected memories where g(u-1) + {(r), 4 1 and ((P) {()), where P is the total number of memories. We assume that the transition is instantaneous in the slow timescale Ta, and neglect the effect of transients in the slow population. Without any loss of generality, when the network state starts at state (2),, the slow population state has two components the previous memory state ((1) and the current memory state (2) We assume that Ta > Ty. so the transient states are negligible, a factor that controls what extent the previous state is reflected in the slow population before the transition occurs. The À is computed analytically in Appendix D
exp (
sx(1) xx.") exp
+ (2)
(Tr)
(24)
Ta
14

---

D Escape Time
To ease the computation of the escape time in relation to the parameters of the network, we scale the timescale of the network dynamics by the substitution t' = tTf. This removes Tj from the dynamical equations and replaces its effect as the timescale ratio T = Ta/TT. The slow population dynamics for the rescaled system is
Ta ds; Tj de
14
(25)
and has the following analytic form for the trajectory.
(t) = xe! exp (-E) (-+++ + + (1= (1-exp( exp (-+)
(26)
To compute average escape time, we consider the two memory contributions C22 C3 on the energy
function, for the sequence transition (1) + {(2) + c(3) and analyze for the transition g + g(3)
That is, U; =!²²) and (t) xg." exp ( 4) + 5, (1 exp(-44) where À is the coefficient of
the contribution of (1) to the delayed state before transition to g(x);2
C2+Cs = exp as Σ
c 0i + ae Σ E5(08) 81 + +exp
(2)
as Σ
c
Di + ae Σ 5 Σs:0s
8:
(27)
i
Substituting U, = (2) and st( = (1) exp( exp(-t) + =/00) - (1-cxp(-1) (-1)
C2 + C3
exp
ae
Aae Σ (1)) ci exp + ( + a. Σ
4:
(1) (22)) (1- (1-exp( (1-exp(##)
(--)
+exp
as Σ
(3) (2) { 5.
axa Σ
x! exp
-
+ a Σ 5(22 5(2) (1- expl
(28)
The energy minima is characterized by the competition between the two memory contributions. Now, take the ansatz that the transition occurs when the energy contribution to the minima C2 < C2. Since exp is a monotonic function, this can be written as
+ac ΣE c!") c( exp (-+) + t Eτ50 ((2) (3())((-ccpp)) (1-cxp(##)
as Σ
Σ
(29)
c(T exp (-t) + ac Σ Σe! (2) ((2) (1 (1-cxp( (-csp---)) )
<a. Σ ( (22) a Ee(² (2)))
E < c(t)) NN σ) for μ # can be approximated by a normal
At the large N limit, the terms distributed random variable. Let o(0, and e Σ, el2) (2) Σ, (())
,62 Σ, c(()) .3
a, N + Na. exp ( + tc ( 1-eepp (--exp(-t) a, t (1-exp(-1
(30)
ag + +ta exp (-+) + a N
6 (1-epp(2 (-1) (
E2
6
+ac (1-exp/ (-+) (31)
< a
+ aa
N
N
15

---

exp (-+) (A +
9
(
-1) + ac 1+ |+}
c3
N
a as
N
1
N
(32)
Let r dc
exp - () +
<r (N -1 + (1 + /4 N ;
(33)
N
(1+1- #
exp
(
(34)
-1
Applying in function on both sides
- - in(r(N 2 11) (1+1 2 N 1))
In x m)
(35)
The time to escape written random variable
-1) (1) (0-1)+(1)))) +1+ N 2))
te = In
(A++
1n(r(1)
(36)
N
For large N, the expected escape time using the delta method to approximate the log of random variable as normal distributed is obtained as
(te) In(A + 1 In(1 r))
(37)
Now that we have the time to escape, we compute the slow signal at the transition point:
st) (((
(1) exp
te
=5(2)
(1 exp (-1²
)
(38)
Substituting the equations for escape times,
st(t) = (1)
1-r
+ (2) (1 1-(771)
x
(39)
x+I
s(()= x
1 --T X+1
+ =
(r+r
(40)
x
At transition,
A Arr X
(41)
x+1
a2
=
(42)
ae
Therefore, before transition, the delay signal will be
(t) = - + (1- VT)
(43)
This computation of seem to generate confusions. So. have decided to provide detailed
reasoning. For transition (1) + (2), X is factor quantifying the extend to which the previous
16

---

state g(1) is present in the slow population when the state transition occurs. Using the Markovian assumption, we assume that c(P) is negligible in the slow population when the transition occurred". We perform our analysis just after the transition g()) - g(2) happened where the "old" pattern is indeed g(1), The escape time we compute is for the state transition g(2) + g($), Now, the definition of À is used again as before but on the transition {(2) + {()) similar to above, except now the "old" pattern is g(2)
An alternate way to think about is by imagining a factor corresponding to a memory in the slow variable that increases the network stays in meta-stable memory state. Now, this factor ideally would reach 1 asymptotically over time, while any "old information" exponentially decays to at which point the fast variable escapes the memory state. Instead of exactly 1, compute what this is based on the parameters we have in our model. A sanity check to verify if the factor at escape time in the most ideal Markovian case is very close to which we indeed find in our analysis. For perfect sequential transitions, vr - 1. This guarantees that the old memory is completely lost when the transition occurs and the accurate next state is retrieved. Now, to compute the analytical escape time:
a factor À and
(1e) In In(vF+1) -In(1- r)
(44)
(1e) = In
1nn(1 r)) -
(45)
(te) = Ta
as
In 1
V
(46)
Ti
ac
E
Capacity
There is rich literature analyzing the capacity of energy-based networks like Hopfield networks. The capacity is defined as the scaling relationship between the number of dimensions in the state space of the network (the number of feature neurons) and the maximum number of memories that can be stored. It is typical to assume that minor errors are allowed as long as the eттor does not scale with the number of neurons. We follow the analysis introduced by Petritis (45) and recently used in B7. Recall that capacity is defined as the maximum number of memories that can be stored such that each dimension of the fixed point encounters an error of with a probability 5. Mathematically,
C(N,e, max (PEN: (PEN:Pr(uatt Pr(pa(t)) c!")) 2 1 1- 2 16
(47)
Typically, u; (te) requires solving system of non-linear dynamical equations. Since we have access to the analytic energy function of the system, we compute the fixed point of the energy function at lo and use it as the proxy for the network state that time.
F. Reference Network
The reference network is defined by the following equations:
T du;
as Σ
Euu ((1)) a Σ
Exp Eju- 85 2/.
dt
19
(48)
Ta ds;
02 8;.
dt
The energy function for this network is given as:
E. 1
(na(1)=
(as(6(m),o0o) + ac(glu-1), s))2
(49)
2a.
17

---

Without loss of generality, the fixed point of the energy surface at the point of transition g (+))+0 g( given by
2 = Σ
Σ
(() 1/ sy(te)
Si
+
of
as Σ
o3
ae
We then quantify the probability for the failure of single bit by computing the following probability, where vz(te) = ef:
Pr[ Pr[v:(t)) <<<1
o8 c
2(N 1)
Σ (1) ct (clh), ,0))
Σ
c(4) (5 (
(50)
##3
Let 1
2N-115 simplify the effect of the discontinuity
c! ( (g(()) + (g(u-1), {(2))) _(2)))
= -+ 2(N 1)
1333
Introduce the random variable x
Σ
cl ((0) (16()) + (gla-1), ,(2)))
x 2(N - 1)
##3
Since c!") are Rademacher distributed, the r.v can be simplified
X 2(N -) 1)
Σ
R
R!
Here, R(("), RR(") are Rademacher distributed random variables. The probability of single bit failure is reformulated in the new random variable as:
cr
The moments of x is then computed to find the bounds on the failure probability.
E.1.1 Moments
First Moment (Mean) Note the the distribution is symmetric around the origin, which gives the first moment as
E
4.2 Second Moment (Variance)
Vix] (N 1)
Σ
R(a)
Σ
R!!)
4(N 1)2
Vix =
4(N V Σ (K)
+
Σ
R
v(x 2(P - 1)
4(N 1) V [1((n)]
Vid (P-
2(N 1
18

---

Bounds of chi
Chebyshev's inequality
X
Pr|1x < - 6
(P-1) Pr[xx|2 e 2(N 1)
Using our definition of capacity, obtain
(P - 1)
2(N 1) =
Solving for P, we obtain
P= 28 (N 1)
which is linear in the number of neurons. For constant rates, and 6, the capacity has asymptotic scaling of O(N) in line with prior classical Hopfield Network bounds.
E.2 EDEN
We follow similar approach for EDEN and set a, ra, ro. The fixed point of EDEN is given as
0/ = - (((((et) ,²²) + (g(u-1),g(21)) 11
exp(a(r(g(M) (00(r(g(w,,())++((00 ((())((())))) expla((+r\(N-1))
Let Z = Engs
(w)) c (3) exp (eree)),gt))) + (g(n-1),g(2)) (r 1)(N 1)))
(t = 1+
Σ
-
1 1+2
##3
(11)
There are two random variables in the quantity of interest. The first Z is sum of many terms, and we replace the sum with its mean for easier computation. The mean field approximation becomes valid in large P limits which we consider in the paper.
Σ Enzs I8 exp (ar (z) 2/0))(0)) exp (acle-1) ((8))
Z
exp(a(r + 1)(N - 1)))
introduce an v. ((1))
<luie/" c
Rademacher and y E {,2
Rademacher
Evya 1lins exp (ar exp (a
Z
exp(a(r + 1)( 1)))
E cxp aarp(() (a))
E exp azy (oys (N-1)
E|Z
exp(a(r I) 1)))
cosh(ar) cosh(a) (N-1) exp(ra) exp(a))
=
The Z is then replaced with the mean value. Also define parameter 8r
ccos((a)) exp(x)
x = 1= (3 (P- 1o Enss 5. c(w) 11
exp ar ( exp (/50- explar 11) explalN- - 1))
exp(ar
1 + (P - (N-1)
(52)
19

---

We compute the expectation and variance to characterize the distribution of X. When computing the expectation, the second term does not contribute the expectation due to the symmetry of the distribution.
(P - (.) 1+ (P 2a
E
The independence of dimensions and memories guarantees that the covariance is 0 for the second term, resulting in the variance.
Vxl (1+ B2ar (2a)
11)
The general distribution of is complicated, but it is symmetric around its mean. We, therefore, use moment matching to approximate the distribution of x using Gaussian distribution.
(P - 1). (N-(N
E =
1+ (P () (arBaaa) (N-))
Vix =0
0 V(P - 1) (B2ar B2a)
(8-1)
1+ (P- (arr Sc.
3a) (N-1)
X~ X~NNH,
Pr(x - c Φ 1 5
Here, & is the Gaussian CDF which does not have closed- expression, but it can be approxi- mated analytically by
exp(2kx))
p 0
Φ
k
π x
(1 + (xxp(2kr))
E.2.1 In the large N limit, 5
For given error tolerance é > O, the success rate (given by 5) approaches
(NN 1) + c
- 8 = 1 + exp
2k
VP 32a) ) (221)
Using the property that <
P(BarBaa) (NN 4 VP }2a) (N1
1 + exp
cxx
2k
(N-1))
cosh(ar) cosh(()
-2kVP
exp
(cosh(2ar) c2ah
(53)
(Bear B2a) (N-11//2 (
exp
2k VP
Now, taking the limit N - oo since or, ar, > 0,
(B2ar >
and Baar 3rn - Do when + x
(N-1))
cosh(ar) cos)(a))
exp exp
+
-2kVP
/cosh(2ar) cosh(2a)
also, cash(or) cosh(a) / cooc((2ar) coshi(2a)
> 1, Vo, r > 0 so taking N + oo gives
0
Q.E.D
20

---

E.2.2 EDEN has exponential capacity
(B2arB2aa) and
Bao (Bzerr Pao)
b
Baria
82a
Bar
(N-1)
(22ar32a) -2a)1(N
1
exp
-2VV
exp
2k: vP
(54)
122aBBo
Bea
BanBa
(N-1)
-(N-1) 1
1- 8
-2kv P
+ 2h:
(Bar 32a)
(55)
VB2arB2a
vp
-2
BanBa
(N-1)
+ 2k
(3ear 32a)
32ar-32a)(N-11) -(N-11/2
-n
1-8 8
(56)
V2ariiaa
vP
8
1 1-6 ) (
(B2aar8ea
(N-1)
P VP In
= 0
(57)
2k
BarBa
(aNBB)(N-11)
which is quadratic equation in P and can be solved to obtain
N-1
(B2arB2a)
VP
In
2h
BarBa
(58)
2(N-1)
[nn (
V(22n 32a)
4e
+
2
BarBa
(BarBa) N-1
1N-1)
Let
Ba
32ar
1
N-1
1
8
In
P
In
+
+
(59)
BanBa
2k
vc
1-
2k
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims of the abstract include (1) introduce the dynamic energy surface model (discussed in Reults Exponential Dynamic Energy Network) (2) Analysis of the escape times and the two phases of behavior in Section Results/Escape Time Char- acterization EDEN (3) Analysis of memory capacity in Results/EDEN has exponential capacity (4) Biological Relevance analyzed in Section 3 biological relevance.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper is about quantifying the dynamical behaviors, and the capacities and limitations of EDEN.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are in the main paper, with high level proofs. Detailed proof' are present in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The high level experiment summary is in the main text and details are described in the Methods section of the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The data is generated synthetically and the code is the supplementary materials. it will be publicly available if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcceeeooooooop hutee/ggguudeeeoeeeuuuubiiiiiiii ieaibbmsssionPo ssinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: High level descripitions are in the main text and details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The analytic results are evaluated on the average deviation from predictions.
Error bars are not reported as the theory is primarily about the mean behavior.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experiments are simple and does not require anything more than a regular computing system.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:://eurrpppt http://////.orr.....//////////aa
Answer: [Yes]
Justification: Our research introduces new computational model and this does not use human subjects for the experiments. We do not create any data and use only publicly available datasets or standard synthetic benchmarks. There no societal concerns we are aware of as this is relatively small scale study on computational research question.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
25

---

Justification:
The study is performed and impacts only the academic community interested in conducting further research SSMs. The work is primarily foundational in creating a new computational algorithm for existing models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper uses publicly available data that is identified as risk free and typically used in conducting academic research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code and data are publicly released as open source software. the code bases we used for compiling our code attributed to the respective authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
26

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: release.
README available on how install, test and use the code base we
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
we do not use this experimental protocol.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
27

---

The experiments we do does not use human subjects and do not require IRB approval.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM was not used in formulating the research. Only use LLMs was in editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy auttoe//eearipsscco////orrrrre// for what should or should not be described.
28


Reviews:
[{"review_id": "NaIbsQX7it", "reviewer": "Reviewer_kb6a", "summary": "This work proposes a new kind of dense associative memory (DAM) model for storing and retrieving sequent memory. The proposed model contains two kinds of neurons operating at different time-scales: fast neurons and slow neurons. The slow neurons are organized in a classical dense associative memory network, while the slow neurons modulate the energy landscape the fast neurons are trying to minimize. Thus, the whole system can be seen as a time-evolving DAM. The authors show the following:\n\n- The system spends a number of time steps trapped in a specific memory, and then escapes into another memory, etc. The separation between the time steps (the so-called escape time) is analytically calculated in terms of macrospic parameters that appear in the equations describing the evolution.\n\n- A precise phase-transition is computed as a function of some of the the aforementioned macroscopic parameters, more precisely, in terms of the ratio $r:=\\alpha_s / \\alpha_c$ of energy scales in the model. For $r>1$ the system only contains a fast neurons, and is stuck in the classical static regime. For $r<1$, the system contains a transition into a dynamical regime.\n\n- It is shown that the model has overall exponential memory capacity.\n\n- Finally, a biological connection is made between the proposed model and human episodic memory.", "questions": "See the questions outlined in the \"Weaknesses\" above.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThough this is not the first time DAM's have beeng proposed for handling sequential memory, this work clearly expands the current scope of DAM's by introducing a time-variable. The resulting model neatly captures the concept of sequential memory. Rigorous analysis are conducted to compute key quantities like escape times (between one memory state and the next). Finally, from the biological plausibility angle, strong links to human episodic memory are made.\n\n\n### Limitations\n- ICLR is an ML conference (not a biology conference). What does this model have to say about ML? Does it explain any ML phenomenon or can it be a basis for a new time of ML model? Any experimental results in this direction?\n- What is the motication of assuming that the sequential memories are organized in a circle? How robust are the results to this somewhat adhoc (but technically convenient?) construction?", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments on the manuscript and suggesting that our model **clearly expands the current scope of DAM's by introducing a time-variable**, the theory we developed to compute quantities like escape time is **rigorous**, and the connection to **biological features is strong**. Below we address the concerns raised by the reviewer.\n\n### Limitation 1:  - ICLR is an ML conference (not a biology conference). What does this model have to say about ML? Does it explain any ML phenomenon or can it be a basis for a new time of ML model? Any experimental results in this direction?\n\nIt is our understanding that NeurIPS is an interdisciplinary conference where researchers from ML and neuroscience gather to share ideas. While we have not discussed the merits of our model in the ML domain, identical works in static Dense Associative Memory models have shown promise in both improving and understanding relevant ML ideas. Eg. DAM has strong connections to transformers and can be used as external memory stores. Additionally, recent works have generalization performance of diffusion models can be understood via phase transitions observed in energy function approach. \n\n### Limitation 2 - - What is the motication of assuming that the sequential memories are organized in a circle? How robust are the results to this somewhat adhoc (but technically convenient?) construction?\n\nThe primary motivation for the assumption is to simplify the theoretical exposition. The main assumption we use is that the sequential transitions are Markovian. i.e. the memory to transition to determined purely from the current memory and not the history of memory transitions in the network. eg. when network moves from 1->2->3, the transition 2->3 is independent of whether the network started from 1 then transitioned to 2 or the network initially started from 2. As such, all theoretical results should hold as long as the escape time is long enough for the network to forget this history."}, {"review_id": "yv05Q8UoIm", "reviewer": "Reviewer_rFqE", "summary": "The paper proposes Exponential Dynamic Energy Networks (EDEN), a two-population Hopfield variant in which a slow modulatory population continually reshapes the energy landscape seen by a fast, high-capacity Hopfield core. This drift causes the network to leave one stored attractor and march through a pre-wired cycle of memories. The authors (i) establish a Lyapunov energy for the fast dynamics, (ii) derive a closed-form mean escape-time that marks a phase boundary between “static” and “dynamic” regimes, and (iii) argue that the model can store an exponentially long sequence of memories via a mean-field calculation. Synthetic experiments fit the escape-time law (for binary Rademacher patterns with N ≤ 35) and the authors draw analogies to hippocampal time/ramping cells.", "questions": "1. Validity and Scale of the Capacity Claim: This is my primary concern. Can you provide evidence that exponential capacity holds in the model's actual operating regime? This could be done in two ways: (a) empirically demonstrate the scaling behavior for α_s /α_c < 1 (e.g., 0.9) at a much larger scale, such as N ≥ 256; or (b) provide a theoretical argument that bounds the error of the mean-field approximation and shows that the exponential nature is preserved away from the phase boundary. Convincingly addressing this is critical. If this is not possible for some reason, please explain it to me. I apologize if I missed something basic. \n\n2. Hidden Layer Scaling: Storing γ^N memories naively implies an exponential-size hidden population. Do you instantiate it explicitly? Please spell out memory/time complexity and any tricks that keep things tractable. This is a common criticism for the biologically plausible implementations of the Exponential Dense Associative Memory.\n\n3. Storing Multiple Sequences: The model in its current form seems to store only a single, pre-wired circular sequence. How would the architecture need to be modified to store multiple distinct, non-interfering sequences? How does EDEN cope with real-valued or noisy or correlated patterns (e.g., MNIST with additive noise)? \n\n4. The proposed link to neuroscience is intriguing but qualitative. The model predicts a sharp relationship between the timescale ratio T_d / T_f and the system's dynamics. Could you attempt to fit this relationship to actual neurophysiological data, such as the ramping cell activity from Umbach et al. (2020) or a similar dataset? A quantitative match, even a rough one, would make the biological relevance claim substantially stronger. I understand this is likely difficult in the short rebuttal period, but if you can offer any further backing to the biological plausibility I believe this can improve the strength of the paper. \n\nA convincing response to Question 1 is essential. Addressing that question along with strong answers to any two of the others (2, 3, or 4) would be sufficient for me to raise my score to a borderline or clear accept.", "limitations": "Yes, but the discussion is insufficient. The most critical limitations (the simplified setting for the capacity proof, the lack of a learning rule, the single-sequence architecture, and the scalability concerns) are not consolidated or discussed in the main text. A dedicated \"Limitations\" section is needed to give readers an honest and clear assessment of the work's current scope and boundaries.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper's core theoretical contribution is sound. The extension of the energy paradigm to the temporal domain via a slowly drifting landscape is a clean, powerful concept. The derivation of the escape time in Equation 5 is the key highlight for me, providing a clear and principled link between the model's parameters and its dynamic behavior which lines up well with the toy experiments. \n\nHowever, my enthusiasm for the theory is significantly dampened by the validation of the paper's headline claim: exponential sequence capacity. The claim rests on a fragile foundation for two distinct reasons. First (as noted by the authors), the analytical proof is performed only at the phase transition boundary (α_s / α_c → 1), a special case where escape times are infinite and the model ceases to function as a sequential memory system. It is not at all clear that this result holds in the dynamic regime. Second, the experimental validation leans entirely on a Gaussian mean-field approximation that is only tested on tiny networks (N ≤ 35). Without either a larger-scale experiment or a theoretical bound on the approximation error, the capacity result feels more aspirational than proven. \n\nFurthermore, the work is not well-situated within the broader context. There is no task-level comparison to modern sequential models (RNNs, Transformers) or even recent sequence-Hopfield baselines (e.g., Chaudhry et al., 2023), making it difficult to assess the practical significance. While the use of a drifting energy landscape is a neat twist, the originality is tempered by the fact that asymmetric, multi-timescale Hopfield models have existed for decades (e.g., Kleinfeld ’86; Sompolinsky & Kanter ’86), and a more explicit discussion of what is new versus what is borrowed would strengthen the paper's positioning. Furthermore, the authors should also read and consider citing highly relevant recent work on analysis of dynamic energy landscapes for sequential Hopfield networks (Karuvally, Sejnowski,  Siegelmann 2023).\n\nWith regard to the clarity and writing of the paper, the narrative flows well, the figures are helpful, and the appendix is thorough. However, key assumptions—perfect weight learning, binary memories, strict time-scale separation—are scattered in footnotes and supplements. A concise “Limitations” box in the main text would save readers a scavenger hunt.", "was_revised": true, "final_justification": "They addressed the questions and comments and I see value in their proofs. I believe other reviewers also noted that their citation of previous literature was somewhat sparse and that the idea of storing exponentially long sequences had already been achieved prior in Chaudhry et al or having a dynamic energy function with polynomially long sequences in Karuvalley et al. I think the mention of biological plausibility when discussing exponentially many hidden neurons doesn't make too much sense, but that is a problem of the field rather than this paper alone. \n\nIf the authors properly situate their paper in the broader literature context in the final version, it would serve to highlight the paper's primary strength which is its theoretical arguments.", "rebuttal": "We thank the reviewer for highlighting that our proposed extension of the **energy paradigm to the temporal domain via a slowly drifting landscape is a clean, powerful concept.** and positive comments about our derivation of escape times. We first clarify our key assumption when deriving the capacity formula and address the weaknesses below.\n\n### Clarification of Markovian State Transition Assumption\n\nIn line 195, we make the assumption that the capacity is computed at the phase change boundary, but the key assumption required to obtain the analytic form of Eqn (10) for the capacity is that the state transitions encoded in the network are Markovian and the network stays in a memory state long enough for the previous memory state to dissipate. i.e. Eqn 43 in the appendix is such that $s_i(t)\\approx\\sqrt{r} \\xi^{(2)}_i$, which happens when $r=\\alpha_s/\\alpha_c$ nears 1, but does not necessarily have to be exactly 1, where the escape time is infinite. Note that Eqn (10) for the capacity has $r$ as a parameter so the equation can be used to probe what happens close to the boundary (We chose 0.999 for our simulation). In this limit $(1-\\sqrt{r}) = 5 \\times 10^{-4}$ which is assumed as 0.\n\nIn sequential networks where we observe stable state transitions, this condition needs to be satisfied. Without this condition, the network will easily recall spurious memories .i.e the network may converge to a spurious memory that is a combination of $\\xi^{(2)}$ and $\\xi^{(3)}$ when transitioning from $\\xi^{(2)}$ because some signal of $\\xi^{(1)}$ is left in the state. We have amended our discussion to clarify this better.\n\n### Limitations\n\nWe add a limitations sections to collect the assumptions we make for the formulated theory in our paper.\n\nAs a theory of dynamical behavior of a non-linear system, we make key assumptions that simplify our mathematical analysis. (1) The synaptic strengths of the neuron interactions are fixed and does not vary during training, in actual biological systems synaptic strengths can change due to short and long term potentiation effects and consolidation (2) The timescales of symmetric and asymmetric interactions are separate - this allows use to treat the asymmetric part as slowly evolving and change the energy function of the symmetric network is response. In human brains, there are different timescales for information processing but the timescales may not be perfectly separated as a distinct slow population of asymmetric connections and fast population of symmetric population, (3) Binary memory - we assume Rademacher distributed patterns for theoretical exposition following related works in the field, although the theory can be similarly worked out for other distributions (4) Markovian State Transition - in deriving our capacity bounds, we assumed that the network spends enough time in a memory state that the historical trajectory information is lost and the state transitions are purely Markovian in nature. Further, the capacity bounds we formulated shows how the maximum number of memories (number of hidden neurons) scales with the number of visual neurons ($v$) following previous results in the field. The number of hidden neurons required for storage however grows only linearly in the number of hidden neurons.\n\n### Limitation 1: the analytical proof is performed only at the phase transition boundary (α_s / α_c → 1), a special case where escape times are infinite and the model ceases to function as a sequential memory system.  It is not at all clear that this result holds in the dynamic regime.\n\nSee our clarification above. The key assumption we use to compute the capacity is that Eqn (43) in the appendix for the slow signal can be written as $s_i(t) = \\sqrt{r} \\xi^{(2)}_i + \\xi^{(1)}_i (1 - \\sqrt{r}) \\approx \\xi^{(2)}_i$ which happens when $\\sqrt{r}$ nears $1$ that is **close to the boundary** and not exactly on the boundary. The factor $r$ still occurs in our capacity equation and can be used to probe what happens slightly away from the boundary.\n\nFor example, we do not assume anything about $\\mathcal{T}_d/\\mathcal{T}_f$, so these can be tuned to have the capacity guarantee in a reasonable timeframe. We have amended our discussion to make this nuance clear. In our simulations, we used $r=0.999$ not exactly $1$ and the $\\mathcal{T}_d$ was set at $20$, which produced sequential transitions in a reasonable time in the network **which is the dynamic regime**. For the dynamic regime, the ratio just needs to be less than 1. \n\n### Second, the experimental validation leans entirely on a Gaussian mean-field approximation that is only tested on tiny networks (N ≤ 35). Without either a larger-scale experiment or a theoretical bound on the approximation error, the capacity result feels more aspirational than proven.\n\nThe theory is derived in the limit of large N (asymptotic regime) so the bound becomes more valid as N is increased, The reason we constrain to N<= 35 is because the capacity for these networks are under the 1M (Million) memory limit. The 1M  constraint is the limitation of the simulation hardware available to us. Our network still works for large scale tasks - for example, we simulated a 784 (28x28) unit network to obtain Fig 2 in the paper that uses MNIST digits. The capacity for the 784 network is way higher than 1M memories, MNIST has only 60k samples.\n\n### Furthermore, the work is not well-situated within the broader context. There is no task-level comparison to modern sequential models (RNNs, Transformers) or even recent sequence-Hopfield baselines (e.g., Chaudhry et al., 2023), making it difficult to assess the practical significance.\n\nThe intention of the paper was to describe salient features of an Exponential interacting network using a novel dynamic energy function. Our contributions are primarily in understanding these network mathematically in an analytically tractable way and not to establish any superiority over existing models in practical tasks. This is still an important avenue to pursue, which will be done in future works. We urge the reviewer to consider the merits of the paper within the context of our contributions.\n\n### While the use of a drifting energy landscape is a neat twist, the originality is tempered by the fact that asymmetric, multi-timescale Hopfield models have existed for decades (e.g., Kleinfeld ’86; Sompolinsky & Kanter ’86), and a more explicit discussion of what is new versus what is borrowed would strengthen the paper's positioning.\n\nAs rightly pointed out, asymmetric multi-timescale Hopfield networks have existed for decades, however, to our knowledge computation of the salient features of the network were analytically intractable, and construction of networks with exponential capacity provably has not been conducted. We thank the reviewer for suggesting the citation of Karuvally et. al. (we have added it now with discussion in the introduction). There are two key changes to the above paper we made. (1) While the above paper provided qualitative descriptions of dynamic behavior using the energy approach, we use quantitative approaches that is analytically tractable, (2) The models proposed in the above paper were polynomially interacting and hence had power law capacity with no proofs of the capacity claims, here we make the interactions exponential, and prove that the capacity is exponential in the number of feature neurons. \n\n### Questions\n\n### Question 1: Validity and Scale of the Capacity Claim\n\nSee the above discussion in the Weakness section. To further support the assumption, note that the slow neuron dynamics (in Eqn 26) shows that previous memory state decays exponentially over time. We further analyzed the capacity equation in the presence of one previous memory (non Markovian with history 1), in which case a spurious pattern can be shown to form if the escape time is not long enough. So, setting the operating point close to the boundary is necessary for the stable retrieval of sequential patterns.\n\n### Question 2: Hidden Layer Scaling\n\nWe thank the reviewer for pointing this out. The hidden population size is exponential, which is the current practice in Exponential Dense Associative Memory models. This is one of the reasons why the capacity experiments are constrained to networks <= 35 and upper bounded at 1M memories. To find the capacity, we use a binary search style algorithm where a range of patterns from 1 to 1M is searched to find the upper bound. With this change, the capacity experiment time complexity is only logarithmic in the number of memories which is favourable for such dense networks. Beyond this, there are no additional tricks, so the memory also scales linearly in the number of memories.\n\n### Question 3: Storing Multiple Sequences\n\nThe key assumption needed is Markovian sequences, storing multiple sequences does not need a significant change. circular sequence is only used for convenience in the exposition. We refer the reviewer to Karuvally et. al. where more general connectivity patterns is considered.\n\n### Question 4: Link to Neuroscience\n\nThe focus of the current paper was on understanding salient features of multi-timescale non-linear dynamical system analytically. As of now, the biological connection is only qualitative. The theory and model we developed is in a higher abstraction level compared to biophysical models like Hodgkin-Huxley models. Due to the abstraction, the high level qualitative features may become evident from the model, but not the fine quantitative details needed to fit the parameters. This is a common weakness of models neural models higher in abstraction."}, {"review_id": "NpxP3NK6wl", "reviewer": "Reviewer_q8MD", "summary": "The paper proposes a method to let a Hopfield Network evolve temporally via energy functions of short timescales. Such a formulation is derived by considering two distinct sets of entities that govern the evolution of the energy of the Hopfield system, i.e. fast population neurons (borrowed from the vanilla Hopfield model) and a set of slow population neurons, that attempt to remember previous memory state(s) even after the energy has stabilized over the current state. Accordingly, properties of the system such as energy dynamics (and consequent transitions between memory states) and escape time are analyzed both theoretically and empirically over a simple MNIST dataset. Strong agreement between the theoretical results and empirical findings is noted. Theoretical results on the capacity are also derived. Finally, the paper also discusses a close resemblance of the proposed energy dynamics with biological episodic memory systems.", "questions": "1. What is $z_\\mu$ in Equation 12 (Appendix)? How does Equation (15) follow from Equation (14)?\n\n2. In line 174, why are only energy corresponding to two consecutive memory states considered to compute the exact time? Technically, shouldn't this be summed over all prior states? Have only two consecutive memory states been considered under the assumption (if this isn't an assumption, please show why it holds in general) that contributions from states before the penultimate one are negligible?\n\n3. To derive capacity bounds, I am assuming that the patterns are assumed to be binary (for otherwise Rademacher variables would not be suitable to capture the representation of the patterns). Am I right in assuming this?\n\n4. What is $\\hat{v}$ in Equation 48?\n\n5. Just to confirm, in line 441, $\\alpha$ is indeed $\\alpha_s$, right?\n\n6. With respect to Equation (59), I am guessing $\\beta_{\\alpha r}\\beta_{\\alpha} = \\beta_{\\alpha_s}\\beta_{\\alpha_c}$, based on above. Can the authors add a short comment stating when this term is less than 1, based on their definition of $\\beta_x$?\n\n7. In Equation (23), what is $v_i$ chosen as that enables tractable integration? This may not be true for generic $v_i$, right?\n\n8. In Equation (41), if $\\lambda$ is being derived by equating the coefficients in Equation (40), Equation (41) is not correct. How is it being derived?", "limitations": "I don't see a separate limitations section in the paper. The authors are advised to do this.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper has been presented very well, in terms of readability, appropriate figures, plots and ordering of sections. \n2. The paper proposes a novel Hopfield Energy network formulation for temporal domains, by evolving and piecemealing the energy function over multiple times scales. The paper analyzes a few salient properties of the system in detail, such as its energy dynamics, and escape time.\n3. The above approach is strongly grounded in a number of theoretical results, and is built from first principles of energy dynamics.\n4. The authors also do a good job of justifying each of the components of the proposed formulation with biological counterparts.\n\n\nWeaknesses:\nMy main concerns are with the first two points under the Related Work and first point of the Experiments section below. These are followed by a number of technical questions in the Questions section, followed by other concerns mentioned here. I am eager to increase my score, especially if the first two concerns can be addressed. I thus suggest the authors to prioritize addressing the concerns according to the order I have indicated.\n\nRelated Work:\n1. Line 88 of the main paper suggests that the 'Long Sequence Hopfield Memory' paper does not use energy arguments. While it is unclear what the authors mean by this, it seems to suggest as though they haven't used energy functions rigorously enough, which I think is untrue. They also considering timing delays (albeit not as thoroughly as in this paper) and asymmetric networks. So rather than artificially mellowing it down too much, I suggest the authors spend just a few sentences more concretely describing how their approach differs from the 'Long Sequence Hopfield Memory' paper.\n\n2. This work seems to exhibit a lot of similarities to https://proceedings.mlr.press/v202/karuvally23a.html -- please consider explicitly justifying how your work differs from theirs.\n\n3. I also suggest suggest adding https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011183, https://proceedings.neurips.cc/paper_files/paper/2023/hash/8a8b9c7f979e8819a7986b3ef825c08a-Abstract-Conference.html as a citation and briefly justifying how your work differs from theirs in its core contributions.\n\nExperiments:\n1. In  my understanding, it appears that the method has been tested only on one dataset, i.e. MNIST. While I understand that this is perhaps largely a theoretical work, I strongly suggest showing your methodology and showing results like Figure 2 for at least 2-3 other datasets. You can use datasets from standard papers such as 'Hopfield Networks is All You Need', 'Associative Memories via Predictive Coding,\" or any other such papers.\n\n2. Line 142 of the paper claimed that no non-convergent behavior was observed, but I am led to believe that this was because MNIST is a very simple dataset. Even the 'Hopfield Networks is All You Need' paper suggested that Hopfield Nets could hit unstable local minima or averages of local minima, and this can be observed via some experiments also. In light of this, please either show via datasets chosen above that non-convergence practically never happens regardless of dataset or show some cases where this failure does happen. \n\nTechnical:\n1. In line 443 (Appendix), just after the statement about modelling via Rademacher r.v's, it appears that a minus sign has crept into the exponent of the term in the denominator which wasn't present earlier. This seems to affect the bound critically as well. Please clarify or fix this. \n\n2. The first line of page 15 (Appendix) requires justification. I am not sure how the assumption of modelling via Normal r.v's can be directly made (there is no reference provided for this either).\n\n3. The way Equation (24) follows from Equation (23) does not seem straightforward to me. Please explain more clearly about how Equation (24) is arrived at.\n\nMinor:\n1. Please change \"Appendix 24\" to \"Equation 24 in the Appendix\" or some equivalent of that in line 203 of the main paper.\n2. In line 165 of the main paper, please change $\\mu$ in the superscript of $\\zeta$ with $\\nu$.", "was_revised": true, "final_justification": "This is a theoretically strong, and empirically verified paper in the general field of enery based memory models. The authors did a great job clarifying all my initial concerns. I hope to see this paper accepted at this conference.", "rebuttal": "The thank the reviewer for the comments and highlighting the grounding of our core contributions in rigorous theory. We also thank the reviewer for positive comments on the presentation. We appreciate the level of detail and meticulousness involved in formulating this review and the attention to the math that enabled finding typos in the document. Prior to addressing the concerns, we want to point out a key assumption of Markovian transitions we make in the theory that enables us to obtain analytically tractable results.\n# Markovian Assumption\n\nIn line 195, we make the assumption that the capacity is computed at the phase change boundary, but the key assumption required to obtain the analytic form of Eqn (10) for the capacity is that the state transitions encoded in the network are Markovian and the network stays in a memory state long enough for the previous memory state to dissipate. i.e. Eqn 43 in the appendix is such that $s_i(t) = \\sqrt{r} \\xi^{(2)}_i + \\xi^{(1)}_i (1 - \\sqrt{r} ) \\approx\\sqrt{r} \\xi^{(2)}_i$, which happens when $r=\\alpha_s/\\alpha_c$ nears 1, but does not necessarily have to be exactly 1, where the escape time is infinite. Due to exponential decay, the information is lost in an exponential rate, so the approximation error will be exponentially bounded.\n\nIn sequential networks where we observe *stable* state transitions, this condition seems to be satisfied. Without this condition, the network will easily recall spurious memories .i.e the network may converge to a spurious memory that is a combination of $\\xi^{(2)}$ and $\\xi^{(3)}$ when transitioning from $\\xi^{(2)}$ because some signal of $\\xi^{(1)}$ is left in the state. We have amended our discussion to clarify this assumption better. Most of the questions posed are related to this.\n\n## Line 88 of the main paper suggests that the 'Long Sequence Hopfield Memory' paper does not use energy arguments.\n\nThank you for raising this concern. There are no energy arguments in the \"Long Sequence Hopfield Memory\" paper. i.e there is no energy function defined for their network, the starting point of the paper is an asymmetrically connected Hopfield network. This is because in their case, the authors consider a discrete sequence network where there is no separation of timescales to define an energy function. We have amended our discussion to include the discrete-continuous difference. \n\n## this work seems to exhibit a lot of similarities to...\n\nWe thank the reviewer for pointing out the work. The network and the analysis we performed go beyond the aforementioned work in two ways - (1) compared to the above paper, we provide \"quantitative\" and analytic guarantees on the properties of the network. (2) The networks considered in the Karuvally paper are polynomial activation networks and did not provide mathematical guarantees of capacity, while we provide both empirical and theoretical guarantees of network scalability.\n\n## I also suggest suggest adding...\n\nWe thank the reviewer for the suggestions. We have amended our introduction to include comparisons to these two paper. We identify two differences compared to the referenced works. In the above two papers, the authors consider an abstract network that learns to encode temporal stimuli, where the plasticity rules play a role in encoding temporal information. In our setting, we analyze network dynamics where the temporal information is hardcoded. There are no plasticity rules in our models, but this is an interesting future avenue to pursue. The second difference is that the model we consider is a continuous time model, which enables applying the energy formulation to understanding the network. Discrete models lack this energy formulation for dynamical behavior. The continuous-discrete difference may also be one of the reasons we find qualitative behavior similar to biological networks.\n\n## In my understanding, it appears that the method has been tested ...\n\nThank you for the suggestion, we will add more datasets in the final paper submission in the appendix. As rightly pointed out, the aim of the paper was to obtain analytically tractable features of the network.\n\n## Line 142 of the paper claimed that no non-convergent... \n\nThank you for pointing this out. It is correct that in the regimes we tested, there were no spurious states. However, if the Markovian assumption is violated and the network is forced to transition before the memory of the previous state is lost, then the low energy state becomes a combination of two encoded memories. e.g. for the sequence of transitions $\\xi^{(1)} \\to \\xi^{(2)} \\to \\xi^{(3)}$, during the transition from 2 to 3, if the slow signal still has a trace of 1, then a spurious minima is formed as a combination of $\\xi^{(2)}$ and $\\xi^{(3)}$. Excluding this case is a key assumption we make in the theory when the capacity is analyzed close to the phase change boundary. These spurious states become relevant when the ratio $\\alpha_s/\\alpha_c$ is far from the phase boundary. \n\n## In line 443 (Appendix), ....\n\nWe thank the reviewer for pointing this out. The minus sign is actually an $\\alpha$ that should be corrected in all the denominators of line 443 starting from $Z = \\sum_{\\nu \\neq 3} \\frac{\\exp(\\alpha( r \\langle \\xi^{(\\mu)}, \\xi^{(3)} \\rangle + \\langle \\xi^{(\\mu-1)}, \\xi^{(2)} \\rangle ))}{\\exp(\\alpha(1+r)(N-1))}$. We apologize for the oversight and is corrected in the manuscript. The bound is still correct.\n\n## The first line of page 15 (Appendix)...\n\nThank you for pointing this out, we have amended the discussion to clarify this point now. Since $\\xi^{(\\mu)}_i$ is Rademacher distributed, $\\xi^{(\\mu)}_i \\xi^{(\\nu)}_j, \\mu \\neq \\nu$ is also Rademacher distributed. Next, the sum of Rademacher random variables are binomial distributed, which when $n$ is large, can be approximated by a Normal distribution. These follow from the standard properties of the distributions. \n\n## The way Equation (24) follows from Equation (23) does not seem straightforward...\n\nFor equation 24, we assume that the network transitions from 1->2 almost instantly, so in order to obtain an analytically tractable form for this, we set $t_0=0$, $s_i(t_0)=\\lambda \\xi_i^{(1)}$ and $v_i(s)=\\xi_i^{(2)}$, a constant, for the duration.  This approximation is valid as long as two conditions are satisfied (1) $\\mathcal{T}_d$ is larger than $\\mathcal{T}_f$, that is, there is a separation of timescales and the transient transition between the two memory states are negligible and (2) Markovian state transition, that is the network stays in $\\xi_i^{(1)}$ long enough till $\\xi^{(P)}_i$ is fully lost from the state. The lambda is assumed, and then later found using the energy function dynamics. \n\n## Question 1: What is $z_\\mu$ in Equation 12 (Appendix)? How does Equation (15) follow from Equation (14)?\n\n$z_\\mu = \\exp(h_\\mu)$ is a short hand for the exp terms. I have defined it now in the manuscript. We apologize for the oversight. We appreciate the reviewer pointing out the typo in Equation 14. The summation over $i$ is is not ordered correctly. Eqn 14 is $\\frac{dE}{dt} = \\sum_{i} \\frac{d v_i}{d t} ( v_i  - \\sum_{\\mu} \\frac{z_\\mu}{\\sum_{\\nu} z_\\nu} \\xi^{(\\mu)}_{i} ) + ...$ and $\\frac{d E}{dt} = - \\sum _{i} \\mathcal{T}_f \\left(\\frac{d v_i}{d t}\\right)^2 + ...$. \nThere is no change in the discussion or the results because of these typos.\n\n## Question 2: In line 174, why are only ....\n\nThis is a result of the Markovian assumption. If the network operates farther from the phase change boundary, the influence of prior states become non-negligible and the network starts showing spurious memories.\n## Question 3: To derive capacity bounds ...\n\nYes, the capacity bounds are derived for Rademacher distributed patterns. \n\n## Question 4: What is $\\hat{v}$  in Equation 48?\n\nThank you for pointing out this typo - $\\hat{v}$ is the slow variable $s_i$. \n\n## Question 5: Just to confirm, in line 441, $\\alpha$  is indeed $\\alpha_s$, right?\n\nYes, that is correct. The subscript is removed only for exposition.\n\n## Question 6: With respect to Equation (59), ...?\n\nYes, this is added now. $\\beta_x = \\frac{\\cosh(x)}{\\exp(x)} < 1$ when $x>0$.\n\n## Question 7: In Equation (23), ....\n$v_i(s)=\\xi_i^{(2)}$ a constant, for the duration. The assumption here is that the the slow timescale enables ignoring the transient when transitioning from $\\xi_i^{(1)} \\to \\xi_i^{(2)}$. \n\n## Question 8: In Equation (41),...\n\nTo derive the expression for $\\lambda$ from Eqn 40, we make use of our Markovian assumption. i.e the memory of the old memory state is lost by the time the network reaches the escape time. From there, we take the coefficient of $\\xi_i^{(2)}$ and equate it to $\\lambda$ which is defined in Line 406 as a factor that controls to what extend the previous state is reflected in the slow variable before state transition occurs. This assumption becomes valid closer to the phase change bounday $\\alpha_s/\\alpha_c$ approaches 1, but not exactly 1.\n\n# Minor Typos\n\nWe thank the reviewer for finding these typos. These are corrected in the manuscript."}, {"review_id": "X0o0lXdRyZ", "reviewer": "Reviewer_JBWE", "summary": "The authors introduce a time-varying version of the Hopfield network, where a slowly varying temporal input can change the energy landscape and the resulting attractor the network converges to. The authors also claim *exponential memory* (but see below).", "questions": "1. I would ask the authors to rephrase the abstract and rest of the paper, to more clearly reflect the real bound of $\\mathcal{O}(\\min( \\gamma^N, P))$. (Or explain to me that I am wrong if I am missing something).\n\n2. Regarding Fig.5, is it possible / straightforward to adjust the model time scales to more closely match the observed data?", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe introduced model seems original to me, and the mathematical analyses using separation of time-scales for analysing the memory capacity appear well done. In general the paper reads well, and has nice visualisations (e.g., Figure 2a very nicely shows the changing energy landscape).\n\n**Weaknesses**\n\nI think that, unless I am missing something, the main claim of the paper title and abstract is not entirely reflected the content, as it the authors claim  *exponential memory capacity*, whereas the actual sequence memory capacity is $\\mathcal{O}(\\min( \\gamma^N, P))$, where $P$ is the number of hidden units. In fact the model needs exponentially more hidden units for exponential memory (there is one hidden neuron for every memory to be stored). So arguably the model doesn’t really exhibit the exponential memory capacity that the authors claim. Note also that this bound on the number of memories is, as far as I can see taken from [13], where this additional linear dependence on the hidden units is indeed made very explicit. \n\nThe empirical demonstration is largely proof of concept (which is okay if the theoretical analysis are all sound).", "was_revised": true, "final_justification": "I appreciate the theoretical contributions to time-varying Hopfield networks. My main criticism was that the exponential memory part is very oversold in this paper. For one, the bound is only linear in the total amount of units in the network (which is somewhat hidden in the original manuscript), and second, similar bounds have been shown for very related networks. The authors have convinced me during the rebuttal that the terminology (exponential in feature units) is used commonly in the field. I would have considered raising my score, but statements by the authors such as the following make me hesitant to recommend accepting this paper without being able to have a second look at the manuscript:\n\n>We want to clarify that the term \"Exponential\" in the title refers to the exponential interaction terms in the model\n\nWhich I simply find hard to believe considering the authors write in the manuscript statements like:\n\n> To develop dynamic energy networks with exponential capacity, we incorporated a slow-changing\nsignal that interacts asymmetrically with an exponential capacity static energy network introduced in prior research\n\nAre we really supposed to believe the authors talk about *exponential capacity* static energy network, but EDEN is an *exponential interaction* dynamic energy network?\n\nIf this paper gets accepted I do hope the authors take the suggestions by the reviewers very seriously and do make the fact that the bound is exponential only in the number of feature neurons very obvious.", "rebuttal": "We thank the reviewer for positive comments on the mathematical analyses in our manuscript and the clarity of our figures. We address the weakness raised by the reviewer below.\n\n### I think that, unless I am missing something, the main claim of the paper title and abstract is not entirely reflected the content, as it the authors claim _exponential memory capacity_, whereas...\n\nWe thank the reviewer for the opportunity to clarify this point. The capacity argument is a common criticism of the literature of Dense Associative Memory models and is used in many works. Here, there is a subtlety in the definition of capacity - i.e capacity is defined as the maximum number of memories that can be stored as function of the dimensions in the state space (Line 197 in the paper) and not the actual neurons in the network. This is a subtle point and we have adjusted our discussion to make this clear. We have also introduced a limitation section to enumerate this weakness in the defining capacity in this manner.\n\nThe reason why capacity is defined in this way is because the hidden neurons in Eqn (1) are often not explicitly written out as separate neurons, instead these variables are assumed to be part of dynamics of the visual neurons $(v_i)$ itself (by substituting $h_\\mu$ in the dynamical equation of $v_i$). This requires an assumption that the synapses have many-body interactions (see \\[1\\] below). Even though this definition of capacity may seem to inflate the actual capacity, not all neural networks can accomodate arbitrary high hidden neurons for a fixed number of visual neurons, which is the main problem tackled by dense associative memory models. e.g. Classical hopfield networks can accomodate only a small fraction of the original visual neurons as hidden neurons.\n\n### Regarding Fig.5, is it possible / straightforward to adjust the model time scales to more closely match the observed data?\n\nThank you for the suggestion. It is not straightforward to match the parameters of our network with the biological data. The model we have and the energy-based networks in general are abstract models of memory storage. To our knowledge, there is currently no known setting of the parameters ($\\alpha$'s and the timescales) that are validated from biological data. This is why we claim only that the biological connection is qualitative. If a procedure is devised to obtain these parameters from biological knowledge and use it for prediction, we may be able to predict some high level behavior of biological systems, this is a very interesting avenue to pursue in future work.\n\n\\[1\\] Krotov, Dmitry, and John J. Hopfield. \"Large Associative Memory Problem in Neurobiology and Machine Learning.\" International Conference on Learning Representations. 2021"}, {"review_id": "UguKXH7BHV", "reviewer": "Reviewer_hmVt", "summary": "This paper combines dense associative memory with sequential associative memory by introducing a time-scale-separated slow hidden population that controls the faster dense associative memory network. The authors show that the network can perform sequential memory retrieval with a capacity that grows exponentially with the number of visible neurons N. They analytically calculate the escape time and the memory capacity of the network and corroborate the theory with numerical simulations.", "questions": "The idea of using a slow population to control memory transitions was reported earlier in the literature, as the authors note, but they seem to have omitted another recent addition to the family [1]. In fact, the dynamics considered in Eq. (1) can be viewed as a soft-max generalization of the quadratic interaction reported in earlier literature. However, the present contribution does show that such a sequence Hopfield network can achieve an exponential memory capacity, which is novel.\n\nThe average escape time calculated in Eq. (5), while a good first approximation (as evidenced by the good match in Figure 3), does not consider the more nuanced kinetic effects that are important in energy-based models. In such non-equilibrium systems, the external drive (modeled here as the slow population) modifies the effective energy landscape, but the time it takes the state to escape a local energy minimum is generally different from the time it takes for the energy landscape itself to change (captured in the present calculation). Such a first-passage-time calculation is known as the Kramers problem, and its solution is approximately given by the Arrhenius law. For example, a similar first-passage-time calculation for a neural population under external drive was considered in [2]. Incorporating these kinetic considerations into the calculation would be an interesting next step.\n\nAnother issue is that although the capacity is exponential in N, the model requires one hidden unit per memory, so the total number of hidden units also grows exponentially with N. Consequently, when counting all neurons in the network, the overall capacity scaling is only linear. Having exponential capacity only in the visible neurons is therefore not sufficient; the authors should either devise a way to reduce the number of hidden units required or clarify this important point about scaling. Otherwise, the defined capacity is not the most relevant metric.\n\nFinally, regarding Figure 5: to achieve exponential capacity the model assumes high-order, all-body interactions (obtained by expanding the exponential in the soft-max interaction term). While the figure nicely demonstrates the dynamical traces of the slow neurons, it is a bit of a stretch to call them time cells and ramping cells.\n\nRefs:\n\n[1] Herron, L., Sartori, P., & Xue, B. (2023). Robust retrieval of dynamic sequences through interaction modulation. PRX Life, 1(2), 023012.\n\n[2] Zhong, W., Lu, Z., Schwab, D. J., & Murugan, A. (2020). Nonequilibrium statistical mechanics of continuous attractors. Neural computation, 32(6), 1033-1068.", "limitations": "The authors did not include a discussion of the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Extending the much-discussed Hopfield network to perform sequential memory retrieval is an important problem. In this work, the authors conduct a detailed theoretical analysis and numerical experiments on this topic, and the results are well-presented. However, some of the analysis and claims need further improvement, as I elaborate below.", "was_revised": true, "final_justification": "This topic considered in this paper is not entirely novel: both the sequence memory task and the exponential interaction term (which leads to exponential capacity in the feature space) are already discussed extensively in the literature. However, I do appreciate the theoretical analysis which matches nicely with simulations. Therefore, I can only recommend borderline accept.", "rebuttal": "We thank the reviewer for comments of our work and suggesting related work relevant to the paper. We also thank the reviewer for highlighting that the exponential capacity is novel in the context of sequential memories. We appreciate the thought of the reviewer to suggest incorporating kinetic effects to consider the dynamic properties of the network, which is required for more general settings of sequence models that generalize the Markovian sequence transition assumption we make in the paper.\n\n**Another issue is that although the capacity is exponential in N, the model requires one hidden unit per memory, so the total number of hidden units also grows exponentially with N. Consequently, when counting all neurons in the network, the overall capacity scaling is only linear. Having exponential capacity only in the visible neurons is therefore not sufficient; the authors should either devise a way to reduce the number of hidden units required or clarify this important point about scaling. Otherwise, the defined capacity is not the most relevant metric.**\n\nWe thank the reviewer for the opportunity to clarify this point. The capacity argument is a common criticism of the literature of Dense Memory models and is used in many landmark paper. Here, there is a subtlety in the definition of capacity - i.e capacity is defined as the maximum number of memories that can be stored as function of the dimensions in the state space (Line 197 in the paper) and not the actual neurons in the network. This is a subtle point and we have adjusted our discussion to make this clear. We have also introduced a limitation section to enumerate this weakness in defining capacity in this manner.\n\nThe reason why capacity is defined in this way is because the hidden neurons in Eqn (1) are often not explicitly written out as separate neurons, instead these variables are assumed to be part of dynamics of the visual neurons $(v_i)$ itself (by substituting $h_\\mu$ in the dynamical equation of $v_i$). This requires an assumption that the synapses have many-body interactions (see reference \\[1\\] below). Even though this definition of capacity may seem to inflate the actual capacity, not all neural networks can accommodate arbitrary high hidden neurons for a fixed number of visual neurons, which is the main problem tackled by dense associative memory models. e.g. Classical Hopfield networks can accommodate only a small fraction of the original visual neurons as hidden neurons without introducing spurious patterns.\n\n\\[1\\] Krotov, Dmitry, and John J. Hopfield. \"Large Associative Memory Problem in Neurobiology and Machine Learning.\" _International Conference on Learning Representations_. 2021\n\n**Finally, regarding Figure 5: to achieve exponential capacity the model assumes high-order, all-body interactions (obtained by expanding the exponential in the soft-max interaction term). While the figure nicely demonstrates the dynamical traces of the slow neurons, it is a bit of a stretch to call them time cells and ramping cells.**\n\nThank you for raising this excellent point. The model we have is abstracted away from the biophysically realistic neural models like Hodgkin-Huxley neural networks. As such there are limitations to the extend to which the biological claims can be made. In the section, we make the case that the dynamical behavior is correlated - which is still interesting because this match to biology was not originally intended in the model. We only make the claim that the dynamical characteristics show similarities to biology. The perceived role of ramping/time cells also match some of the predictions we make from the theory. The evidence we show is only correlational and further work may be required to better quantify this connection"}]

Author rebuttal:

Meta-review: Reviewers appreciate the value of the theoretical analysis presented in this paper, which meaningfully extends the results obtained in prior work. In the final camera-ready version authors need to implement the following changes to the manuscript: 

1. Make it crystal clear (in the abstract and introduction) that the capacity is exponential in the number of visible neurons, and linear in the number of hidden units.  

2. Cite the following two papers and adjust the contribution of their paper accordingly: 

     [1] Karuvally, Sejnowski, Siegelmann 2023 

     [2] Herron, L., Sartori, P., & Xue, B. (2023). Robust retrieval of dynamic sequences through interaction modulation. PRX Life, 1(2), 023012.

With these modifications in mind, I recommend acceptance.
Decision: Accept (poster)