Associative memory and dead neurons
Authors: Vladimir Fanaskov
Ivan Oseledets
Venue: ICLR 2025 Poster (2025)
Keywords: associative memory
dead neurone
neural ODE
energy-based model
Primary area: learning theory
TL;DR: Associative memory is sensitive to the presence of dead neurons, we show how to avoid this problem by slightly altering dynamics and introducing new energy functions.

Abstract: In ``Large Associative Memory Problem in Neurobiology and Machine Learning,'' Dmitry Krotov and John Hopfield introduced a general technique for the systematic construction of neural ordinary differential equations with non-increasing energy or Lyapunov function. We study this energy function and identify that it is vulnerable to the problem of dead neurons. Each point in the state space where the neuron dies is contained in a non-compact region with constant energy. In these flat regions, energy function alone does not completely determine all degrees of freedom and, as a consequence, can not be used to analyze stability or find steady states or basins of attraction. We perform a direct analysis of the dynamical system and show how to resolve problems caused by flat directions corresponding to dead neurons: (i) all information about the state vector at a fixed point can be extracted from the energy and Hessian matrix (of Lagrange function), (ii) it is enough to analyze stability in the range of Hessian matrix, (iii) if steady state touching flat region is stable the whole flat region is the basin of attraction. The analysis of the Hessian matrix can be complicated for realistic architectures, so we show that for a slightly altered dynamical system (with the same structure of steady states), one can derive a diverse family of Lyapunov functions that do not have flat regions corresponding to dead neurons. In addition, these energy functions allow one to use Lagrange functions with Hessian matrices that are not necessarily positive definite and even consider architectures with non-symmetric feedforward and feedback connections.

Published as a conference paper at ICLR 2025
ASSOCIATIVE MEMORY AND DEAD NEURONS
Vladimir Fanaskov
Ivan Oseledets AIRI, Skoltech
AIRI, Skoltech
fanaskov. vladimir@gmail com.
ABSTRACT
In "Large Associative Memory Problem in Neurobiology and Machine Learning," " Dmitry Krotov and John Hopfield introduced a general technique for the system- atic construction of neural ordinary differential equations with non non-increasing en- ergy or Lyapunov function. We study this energy function and identify that it is vulnerable to the problem of dead neurons. Each point in the state space where the neuron dies is contained in a non-compact region with constant energy. In these flat regions, energy function alone does not completely determine all degrees of freedom and, as a consequence, can not be used to analyze stability or find steady states or basins of attraction. We perform direct analysis of the dynamical system and show how to resolve problems caused by flat directions corresponding to dead neurons: (i) all information about the state vector fixed point can be extracted from the energy and Hessian matrix (of Lagrange function), (ii) it is enough to analyze stability in the range of Hessian matrix, (iii) if steady state touching flat region is stable the whole flat region is the basin of attraction. The analysis of the Hessian matrix can be complicated for realistic architectures, so we show that for slightly altered dynamical system (with the same structure of steady states), one can derive a diverse family of Lyapunov functions that do not have flat re- gions corresponding to dead neurons. In addition, these energy functions allow one to use Lagrange functions with Hessian matrices that are not necessarily posi- tive definite and even consider architectures with non-symmetric feedforward and feedback connections.
INTRODUCTION
Associative or content-addreessalee memory is system that retrieves the most appropriate stored pattern based on a partially known or distorted input pattern. One particularly influential realization of associative memory was proposed by John Hopfield in [Hopfield (1982) for discrete variables and in HHofield (1984) for continuous variables. Both models are distinguished by their biological plausibility, autonomy, asynchronous operations of constituent parts, robustness to noise, and strong theoretical guarantees. Later in Krotov & Hopfield (2020), it was shown that one could develop a general biologically plausible model that unites many previously known models and allows building novel associative memory systems Krotoy (2023).
The model in Krotov Hopfield (2020) based on the nonlinear dynamical system that evolves in time from a given initial state. Nonlinear dynamical systems show an exceptionally diverse set of behavior Stroggttz (2018), so one needs select an appropriate class of ordinary differential equations suitable to model associative memory. The most crucial requirement is the ability of the system to evolve to single state from many initial conditions that are close according to some problem-specific metrics. This fact suggests one should use dynamical system with many stable steady states. If one selects such a system, each stable steady state corresponds to a particular memory and basin of attraction - all initial conditions that evolve to selected state defines measure of similarity between states.
The technique of choice to study the stability of steady state is to construct energy or Lyapunov func- tion Lyapunov| (1992). This function, defined on the state of a dynamical system, is non- increcccgg on trajectories of dynamical system. If it is possible to find such a function, its isolated local minima will correspond to steady states. Different variants of energy functions with this property available in all previous works associative memory Hopfield (982). (Hopfield (1984. Krotov

---

Published as a conference paper at ICLR 2025
ReLU
softmax
ReLU
softmax
(a) Energy function from Krotov & Hopfield 2020)
(b) Energy function proposed in this article.
Figure I: Vector fields dynamical systems (top row) and level sets of energy functions (bottom row) for energy functions (the one from Krotov & Hopfield (2020) and (TO) (proposed in this article). Vector fields suggest that all steady states are stable, but energy function (2) does not indicate that. The reason energy function (2) has non-compact flat regions touching each point where one or more neurons die (see Proposition for precise statement).
& Hopfield (2016) and recent work Krotov & Hopfield (2020) provides the most general energy function that embraces all previous ones.
Since associative memory from Krotov & Hopfield (2020) has an associated energy function, one can argue that it is a member of broad family of energy- based models LeCun et al. (2006), in- cluding the diffusion models Hoover al. (2023). In addition to that, the dynamical system used clearly related to another subfield of machine learning known as Neural Ordinary Differential Equation Chen et al. (2018. Finally, in Krotov (2021). Hoover al. (2022) it shown that one can select parameters of model Krotov Hopfield (2020) in such way that dynamical variables correspond to the activation pattern of deep neural network with feedback connection. Given that, one can argue that modern Hopfield associative memory Krotov & Hopfield (2200) is uniquely tying
up several major paradigms in deep learning.
In this article, we study the Lyapunov function proposed Krotov & Hopfield (2020) and later adopted and modified in other recent papers, e.g.. .g.,[Hooveeer al. (202e), Hoover al. (0222 Millidge et al. (2022a). Our main observation is that energy function is vulnerable to the problem of dead neurons Lu et al. al. 2019), which result flat energy direction. We illustrate this the left panel of Figure where one can find both the vector field of the dynamical system and isolines of the energy function. Clearly, the energy function from Krotov & Hopfield (2020) does not ensure stability and is not helpful in identifying steady state when dead neurons are present.
As remedy, we propose a slightly modified dynamical system that has no flat direction (see right panel of Figure[] [ but retains other good properties of the Krotov and Hopfield model. In particular, is still an energy-basee model, neural ODE, and has steady states related to deep neural networks.
A more detailed breakdown of our contribution is below:
1. In Section we provide examples of flat energy directions caused by dead neurons (Fig- ure Examples and formally characterize architectures that are vulnerable to that problem in Proposition
Usually, neurons are called dead when activation function saturates, e.g., x < 0 for ReLu or x > for sigmoid. In this article we use an extended definition of dead neurons and say that a neuron is dead when it impossible to reconstruct input to activation function from its output. This is explained in more detail in Section
2

---

Published as a conference paper at ICLR 2025
2. Flat energy directions, show in Section cause several undesirable consequences for sensitivity and stability:
(a) Problems with sensitivity described in Section 3.1 include: (i) energy is completely
independent of degrees of freedom corresponding dead neurons, because of that activation functions are effectively invertible and energy function presented in Krotov & Hopfield (2020) is completely equivalent the old one given in Hopfield (1984); (ii) In the flat regions energy function is not sensitive to the change in bias term (Sec- tion 3.1p.
(b) Stability is discussed in Section 3.2 with the following main takeaways: (i) for steady state with dead neurons, stability condition can not be ensured from the Lyapunov function alone; (ii) independent analysis of dynamical system shows dead neurons do not compromise stability, and only the motion in the range of Hessian of Lagrange function is relevant; (iii) if the steady state is stable, the whole flat direction belongs to the basin of attraction; (iv) if in addition of energy, the range of the Hessian is available (e.g., as the orthogonal projector), this information enough to restore degrees of freedom that energy alone misses steady state.
3. As remedy, in Section we define a slightly modified dynamical system and fam- ily of energy functions with good properties: (i) proposition [ shows that in general one may construct a large family of Lyapunov function with no flat directions; (ii) example [ shows several concrete choices of Lyapunov functions for symmetric weight matrices with no restriction Hessian matrix of Lagrange function; (iii) examples 6 and / further relax restriction on parameters and allow one to consider memory models with non-symmetric weight matrices.
2 DEAD NEURONS
In Krotov & Hopfield 2020) authors proposed to rewrite activation function ( gradient of Lagrange function Lzy). i... g 0L(y), memory models with Lyapunov function on the common grounds. Results that we obtain in the article are applicable to all of them, but for simplicity consider dense model taken from (Krotov) 2021 Equation (2)) (see Appendix Ar further discussion on relations between different models). This model reads:
By This formalism allows to describe a large class of
(() = W Wg(y(() y(() b, y(0) = yo:
(1)
E (y b) ( L(y) (g(z)) Wg(y).
Equation describes temporal dynamies of feature vector y starting from yo- The equation
contains weights W. bias b and activation function g Energy function (2) is non-increasing
on trajectories of (1) and only if W W and Hessian Lagrange function A 8 L is
positive semi-definite (see (Krotov Hopfield 2020 Equation (4), Appendix A) and (Krotov 2021 Equation (4), Appendix A)).
Dynamical system has several favorable properties (see discussion Section [ and Krotov & Hopfield (2020) Krotov (20TT)): (i) steady states can be used as memory vectors, (ii) dynamics in the basins attraction naturally model memory recovery process, (iii) energy function can be used to ensure the existence of steady states and basins of attraction, (iv) memory is related to neural ODEs so can be trained end-to-end, (v) with special choice of W steady states resemble activation pattern of classical deep learning architectures. The example below illustrates the last point (see Section 3)).
Krotov 2021
Example 1 (MLP with feedback connections). For simplicity, we consider four layers, extensions to larger number of layers are straighffowward. Weights and state vectors are partitioned on blocks of conformable size
0
W12 0
111
b1
W
W2
W
W23
0
W
34
y
1/2
y/3
b2
W
0
b4
3

---

Published as a conference paper at ICLR 2025
Note that W is symmetric W12 = W21 With that choice dynamical system becomes
ÿi W., 1.i 19 (Vi-1) + W 1 (3+++) - y: + b,, i i= 2,3
ÿ = Wing (y2) y1 + bi, ÿ4 W439 (3/) - y4 + b4:
so steady-state indeed resembles MLP but with feedback connections that are symmetric.
Besides MLP described in Example 1 associative memory (T) leads to many more interesting and fruitful connections. In Krotov & Hoptield (2020) it allowed the authors to reconstruct dense asso- ciative memory Krotov & Hopfield (2016) and modern Hopfield network Ramsauer al. (2020). In Krotov ( Hoover et al. (2022) it was used to build a memory model with dense hidden lay- ers and convolution neural networks with pooling layers. In Hooorr et al. (2024) authors introduced
an energy transformer using the same formalism. The authors of Tang & Kopp (021)) also noticed
that MLP-mixer Tolstikhiin et al. (202T) related to associative memory LT. Clearly, the technique is valuable and versatile.
As we have seen, steady states of dynamical systems are important to emulate deep learning ar- chitectures and classical memory models. The role of the Lyapunov function is to ensure stability. Unfortunately, if one looks closer at the energy function 2 it becomes clear that it shows some pathological behavior. Before providing formal description, we illustrate this with three examples.
Example (flat energy with ReLU activations). Lagrange function L(y) = =E= 1 (RelU(y))) corresponds to fully-conmected neural network with N neurons and ReLU(x) = 2 (x+1==) (y-b) b)² ReLU(y) - (ReLU(y)))² (ReLU(y)) W ReLU(y). It is easy to see that if neuron i dies, i.e., Di S. O, energy becomes zero for all y = y + ae;, where a S 0. e, is a i-th colunns of identity matrix and y a state vector with ys I 0. In other words, energy is non-discrieenative in non-compact region of state space.
nonlinearity. Energy function becomes E(y)
Example 3 (flat energy with sigmoid activations). For sigmoid activation function a(x)
(1 + +epe(( exp((=)))- Lugrange function reads L(y) Σ, log (1 + e and the energy is Ely) (y (y-b) (yy) - -, log (1 + ex)) 2
(g))) W (yy) Neuron i dies when sigmoid saturates
for some y/ >> 1. After that for all y y aes with 2 0 the energy has flat region, i.e., Ely) = (((())) ) for all udmissible a.
Example 4 (flat energy with softmax activations). Activation softmax (y) = exp(vv/ En exp(v:) corresponds to L(y) log (Eih) exp(vi) and energy E(y) (y b) softmax (3)
log ENN, exp(y:)) 1/2 (sofmax (3)) W softmax (z). Consider variable ÿ(c) = y + c, where is a vector with all components equal R. So constant shift, softmax (y) softmax (ÿ(c)),
Lagrange function shifts on e and ÿ(c) ((smmxx(((c) = yy ssootmxx()) + c, so overall energy
remains the same, i.e., Ely) E(ū(c)) for arbitrary
The examples above demonstrate that energy function fails to distinguish states on large frac- tion of state space. In Examples 2 and 3 happens because the activation function saturates for some inputs, and in Example 4 this is a consequence of invariance. The latter case is typically not considered as a dead neuron, but since the number of degrees of freedom decreases by one we call this "effective neuron" dead all the 3 formal definition of dead neurons appears below (see Maas et al. (0033), Lu et al. (2019). Cui & Fearn 2018) for the discussion of dead neurons in the context of ReLU activation function).
Definition (dead neurons). At a given point y E N activation function g has k dead neurons if possible to find V E RRN xk such that g (y + Vc) = g (1) for all c E li.k
The examples above hint that most of the activation functions will lead dead neurons. The first large class is activation functions with saturation: hyperbolic tangent, sigmoid, ReLU, GELU, SILU,
Formally, the sigmoid function saturates in the limit y =00. However, for computations in floating- point arithmetic it is safe say that sigmoid saturates when the absolute value of the argument is sufficiently large but finite.
3One can also say that softmax is always saturated. If we consider input y in the new basis where 51 Σ, Vi, it is easy to see that after the softmax this component hecomes 1 regardless of the input.
4

---

Published as a conference paper at ICLR 2025
(a)
(b)
Figure 2: Sketch of three problematic energy functions: (a) unbounded from below with two stable states (we analyze this situation Appendix C). . (b) bounded from below but with compact flat re- gion, (c) bounded from below but with non non-compact flat region. According to Proposition [ case (c) realized in models Krotov Hopñlld (2020) when neurons die. In red regions stability proper- ties do not follow from Lyapunov theorems. LaSalle's invariance principle (Haddad & Chellaboina 2008 Theorem 3.3) ensures that in case (b) isolated steady states are stable, but for the case (c)
separate stability analysis is needed (see Section 3.2 for details).
SELU, Gaussian, etc The second class is activation functions with symmetries, e-g., softmax and layer norm. With the definition of dead neurons, we can formalize pathological behavior of energy function (2).
Proposition 1. af given point y activation function g has k dead neuron defined by V RRN energy function 2 has constant value in subspace DD = { + Ve:CE R2
Proof: Appendix
Flat energy directions are illustrated in Figure 1 and Figure 2c Intuitively, it is clear that having large regions with flat energy not good. the next section, we explain in detail what will go awry.
As illustrated in Figure 2 energy function can appear problematic for reasons not directly related to dead neurons. We argue that these two situations are less grim. Energy illustrated in Figure 2a can still be utilized with proper initial conditions, e.g., ||3ol|2 < R for some small R. Besides, we demonstrate in Appendix thtat restricting parameters to ensure energy bounded from below can lead to catastrophic capacity reduction. Energy from Figure 2b is less problematic for stability analysis because invariance principle (Haddad & Chellaboina
2008
Theorem 3.3) can be used.
CONSEQUENCES OF DEAD NEURONS
Proposition I implies that for most architectures used in practice, there are large regions of state space with flat energy functions. This has several negative consequences that somewhat arbitrar- ily classify as problems with sensitivity and stability.
3.1 SENSITIVITY
As evident from Figure I energy is flat in directions V corresponding to dead neurons. This can be formalised if one considers new variables VV y. yo - (I-VVV corresponding to dead and alive neurons. Since y ya Ya. the energy becomes
E(ya, ya) = (ya + yo b) g(ya) L(ya + yo) (g(y))) Wg(ya)
(ya b) T g(ya) I(ya. (g(Ua)) Wg(y.),
where we used gyy + ya)) g(y.) and L(ya ya) L(ya) + +gy ya.
'For sigmoid, hyperbolic tangent and Gaussian function the saturation is precise only in the limit r + too. However, as explained in the 3xample in the floating-pont point arithmetics these functions effectively saturate for finite values of the argument. Moreover, even with arbitrary precision, approximately flat direction can not be used reliably store any information since it will require weights with exponentially large magnitudes.
5

---

Published as a conference paper at ICLR 2025
We see that energy does not depend on variables Ya corresponding to dead neurons, which means the effective number of degrees of freedom is decreased from N to - where l is the number of dead neurons. If only energy is available it is not possible to recover values on ya and steady states can not be found as examples from Figure
Variables Yd from the kernel of which means the activation function becomes effectively invertible. In Hoptield (1984), the energy function for this situation has already been introduced. It is instructive to compare new energy function Krotov & Hopfield (2020) with the old one. Lyapunov function from the 1984 paper, in the notation used in this article, reads sr(w.)
E
! drgg, (T,)
1 (r,) - b0 2 (g(u)) Wg(u),
and dynamical system is precisely the same as Activation functions Ji are invertible and mono- tone by assumption. In Appendix D we explain precisely how adapt the model from [Hopfeld 1984) to our context. The first term of (3) is seemingly distinct from any term of (2). To show that this is illusion we simplify integrals follows a(u)
[ dpg (p)p= =- (0)
dpg(p) + pg(p)] = f dpg(p) + ug(u), (0)
dTg (T)
g 9-1(0)
where in the first step used g(p). Using this simplification and additional definition of
Lagrange function L(u)= g(u) ol(a) we immediately recognize that energy function from
Du
Krotov & Hopfield (2200) is preciscly the same from Hopfield (1984) but without the assumption that g should be invertible. New energy from Krotov & Hopfield (2020) does not formally contain the inverse of the activation function, but we see still implies that g is invertible.
Besides insensitivity to values of dead neurons, there another problematic fact. Proposition implies that, when least one neuron is dead, the energy function has invariant transformation E(y) = Ely + Vc). The steady state of dynamical system (1) does not have this symmetry, so, when one transforms y - y + Ve and preserve energy, this corresponds to a new dynamical system with b + c. In other words, in the regions when dead neurons are present, energy is not sensitive the changes in the bias term.
We summarise arguments made in this section in the proposition below Proposition 2. For energy function (2 the following is true:
1. Energy function does not depend on variables from the kernel of g, so one can always assume that activation function g is invertible.
2. Suppose at given point dead neurons are described by matrix V E Nxk, In this case energy function E(y) is Lyapunov function for dynamical systems 7 with b = b Ve any E R
3.2 STABILITY OF DYNAMICAL SYSTEM
One of the main applications of the Lyapunov function is to perform stability analysis Haddad & Chellaboina (2008). This part relevant for predictive coding and related approaches Xie & Seung (2003), Millidee et al. (022b) to ensure the existence of at least some steady states. Besides, when associative memory is considered, one can only recover stable steady states, so stability directly influences memory capacity. Lyapunov function also characterizes basin of attraction for given steady state. In the context of associative memory, basins of attraction define a measure of similarity between inputs, since inputs from the same basin result in the recovery of the same memory. Below, we will show that in the regions with dead neurons, Lyapunov function (2) alone fails to help in stability analysis.
Stability analysis with the Lyapunov function boils down to the study of the energy landscape near the steady state. Using the Taylor series, we find
Ely² + E(y") TT @E(u)
E2E(u)
1 8
6, t y²) ) E (0, 1),
tu
Ou2
u==y*
4=

---

Published as a conference paper at ICLR 2025
where y² Wg(y") y" +b=0, Computing derivatives, we find Ely" + + Ely") 6t (A A(y")) ||81l, < 1. For stability in the vicinity of a steady state, it is sufficient that the matrix above is positive definite, and for instability, it is sufficient that at least one eigenvalue of this matrix is negative. In situations such as in Figure | and Figure [ when energy has flat direction nothing can he said about the dynamics in this region P It easy show that if dead neurons are present at point u, matrix V lays in zero eigenspace of Au)). To show this we consider the Taylor series g(u u + Vc) g()) + A(u + Ve,
(0, 1), and since g Vc) = g we confirm that AV = 0, i.e., zero eigenspace is always present.
Since energy is insufficient to analyze stability, we need use dynamical system (I) directly. We suppose that in the region of interest V. matrix that describes dead neurons, does not change and use yd: Ya defined in the previous section. With that, one can show that dynamical system becomes
(1) = PaWalya (1) - yo Pab, úa(l)= = PaW (Va - ya + P.b,
(4)
where Pa = VVT and P. I-Pa One can immediately see that dead neurons do not influence stability and, when steady state ya is reached, one can find y - PaW g(v)) + Pab. Moreover, when yo (t) reaches steady state (t(() converges exponentially from the arbitrary starting point, i.e., the whole flat region is a basin of attraction.
The sufficient condition for stability/instabiiity of yo can be obtained from (4 with lincarisation. More specifically, stability is defined by matrix S = VT WVL AV (NN-)))(N--), where Vi E RNNN- is a matrix with orthogonal columns such that Pa = V. When matrix S does not have eigenvalues with positive real part, the steady state is stable: in case there is at least one eigenvalue with a positive real part, the steady state is unstable (see (Haddad& & Chellaboina 2008 Theorem 3.19)..
W_A_ I E
The structure of matrix S suggests that, in fact, dynamics is stable when energy function indicates that. Indeed, A "//"WA'' |A12 has the same spectrum as W A1, sufficient condition for $ is equivalent to sufficient condition for A 1/2 W_AWW
I. Finally since A is full rank we find that if A A A is negatively stable (has non-positive cigenvalues), the dynamics of ya is stable. Since this matrix is restriction of AWA A - A on the range of Hessian we conclude that one can analyze the stability of steady-state using energy if projector on the range of Hessian known. It is important to note that matrix A AWA A A can have other flat directions not corresponding to nullspace of Hessian, these directions will cause instability, so one need to distinguish them from directions corresponding to dead neurons.
Clearly, zero modes of A do not negatively affect the spectrum of matrix S. On the contrary, one may expect stabilizing effect since Cauchy interlacing theorem (Horn & Johnson 2012, Theorem 4.3.28, Corollary 4.3.37) ensures that the spectrum of W1 is in-herween minimal and maximal eigenvalues of W. So when X (WA) 2 1 it is likely that X (W1 A+) the effect, one needs to make some assumptions about the spectrum of W. Given a standard deep learning practice to use random matrices for initialization
To demonstrate
model weights/Glorot Glorot & Bengio (200)), He et al. (2015), it is natural to assume that W is drawn from Gaussian orthogonal ensemble. Under this condition, one may show a stabilization effect, as demonstrated next.
Proposition 3. Suppose W E R N is a matrix from Gaussian orthogonal ensemble. If
neurons are dead W| 1l2 = V $ W|l2 holds for large N.
Proof: Appendix P
We gather the main results of this section in the proposition that follows. Proposition 4. For energy function 2 the following is true:
1. Flat directions of dead neurons characterised by V form nullspace of Hessian A.
Note, however, that for the face given in Figure 2b it is possible construct a positively invariant set based energy levels. After that LaSalle's invariance principle (Haddad & Chellaboina 2008 Theorem 3.3) guarantees that isolated steady states are asymptotically stable. This material is standard and covered in many books, e.g., (Haddad & Chellaboina] 2008 Section 3.3.).
WWe assume for simplicity that has orthonormal columns.
7

---

Published as a conference paper at ICLR 2025
2. Steady state is stable if energy function predicts local stability in the range of Hessian, i.e. if matrix AW A A restricted on the range of Hessian is positive definite.
3. If one performed direct optimization of the energy function and found that y corresponds to the minimum, is possible to recover the steady state of dynamical system 7 as follows: (i) compute the projector Pa on the nullspace of A; (ii) steady state of dynamical system reads y = y + Pa (W gly) + b).
4. When steady state (yar y²) is stable, the whole non-compact flat region (ya V + Vc) is basin of attraction.
4 ASSOCIATIVE MEMORIES WITHOUT DEAD NEURONS
We have seen that dead neurons present certain problems for energy function 2 (i) flat regions make optimization and analysis of energy cumbersome; (ii) one needs to distinguish between harm- less flat direction of Hessian and other flat direction of energy that can compromise stability; (iii) energy do not contain full information and one need to build a projector on the nullspace of Hessian restore the whole state. The last point implies that needs Hessian when stability analysis is not performed and one merely wants to compute a local minimum of energy which can often be done successfully with first-order methods.
To avoid these difficulties we will slightly modify the dynamics of associative memory to minimally alter steady states and get better energy function. In simplified terms, our idea is to modify right-hand side of (I) such that it becomes conservative vector field. If this the case, flat energy directions disappear. For example, the simplest model of this kind is abiu) with energy E(u) = 1 - L(Wu + b). du more general dynamical system along the same lines reads:
(t) = w (W)(()) + b) utt)
i( R(u) (g( W (t) + b) u(()), u(0) uo
where R(u) is matrix-valuee function to be specified later.
If one assumes that is possible select R(u) that has empty nullspace for all u, i.c., that informa- tion is not lost after multiplication by R(u), the condition that we will verify later, steady states of
the newly introduced system (5) follows from the old one with affine transformation y Wa + b.
This means all architectures possible with the old model ( are also possible with the new one (5). More precisely, the structure steady state is the same for memory (T) and memory ( but the structure of basins of attraction is different owing to the matrix R (u) absent in model
Dynamical systems (5) and T have equivalent steady states, but (5) allows to construct large set of Lyapunov functions with good properties in systematic manner.
Proposition 5. For parametric energy function
o(u;o, 8, S) = aF, + 3 + + S),
E1 (u( u. Wg (Wu - - - (Wu+ b)T Wg
1 =u Wu I, (W b), Fa (u: 2 (u o(Wu+b)) S(u g W + b))
E2 E2(u)
2
(6)
the following is true:
1. E(u; a. 3.%, is a Lyapunov function for dynamical system 5 if one can find Q 2 0,
R(u), S S such that for all
R F(u) + F()) R W (aW - 78) + 3W + +7 (7)
2. For arbitrary parameters a. 3,,, , one can always find orthogonal matrix R(u) such that E (1u a., 3,,,, S) is Lyapunov function for dynamical system (5)
3. Fz(w (u) and Ea Pa (u; S) does not have flat directions corresponding dead neurons.
8

---

Published as a conference paper at ICLR 2025
4. Sufficient condition for stability of steady state state u" uu" (Wu" + is
F(u') A (Wu" b) W) + (I- WA (Wu" + +b) (F)(()
(8)
Proof: Appendix
Proposition 5 allows one make several useful conclusions.
First, the construction of the Lyapunov function boils down to the analysis of matrix inequality 77) In general, this inequality nonlinear, but in particular cases, it reduces Sylvester equation which can be systematically analyzed [Simoncin] (2016). Below we will provide several explicit choices of R that result in valid Lyapunov function.
The second point of Proposition 5 shows that suitable R always exist. Moreover, this R does not alter the steady state, since is an orthogonal matrix. Whether this choice of R is practical depends on the parametrization of weights and other details of the model.
The third point shows that unless 3 = = O, the energy function does not have flat direction, and the fourth point gives sufficient condition for the stability of a particular state. This part is harder to analyze general, so we will discuss this condition for selected examples below.
Finally, Proposition 5 does not require Hessian to be positive definite which allows one to use richer set activation functions.
Example 5 (a = = = 0, 8 1). In this case E = E2 condition 7 becomes R W WRR 0. There are many strategies to select R and W:
1. The simplest choice is take R W since in this case condition 7 reduces to W which true for arbitrary symmetric matrix.
2. Unfortunately, if matrix W has low rank and one takes R = W, dynamical system ( can lead false memories whenever f(((() reaches nullspace of W. To exclude this
possibility one can parametrize W OP in terms of its polar decomposition and select
R O. With that choice condition 7 becomes P which automatically follows from polar decomposition.
3. We can restrict W to be positive definite by taking W = KK for some full-rank K. In this case, a large set of positive definite matrices R exists such that RW WR 2 O. More specifically, one needs to restrict the condition number of R as explained in Nicholson (1779).
4. If W is positive definite the other option is to select arbitrary Q >> 0 and consider Lya- punov equation RW + W RR- = Q which is known to have unique solution for arbitrary Q > O. One can explicitly provide it in many forms Lancaster 1970), [imoncini (0166). for example R = = dr exp(- Q exp(-TW))
Condition for the stability of state {8) in this case reduces to W - WA (Wu' + +b) W > 0. Note that nullspace of A does not compromise stability since does not transfer to nullspace of W WAW.
Interestingly, one can also construct associative memory with non-symmetric weights as explained in the next two examples.
Example (a = fined for W o WT If we account for that in Proposition 5 existence condition (7) becomes R [I w A)
0, 7 = 1 and W + WT). Es is the only energy function that is de-
W S S(I AW) 2 0. Two examples of suitable R, S, W are given below:
1. The simplest choice is to take S = I and R = I-WT A. One also may ensure that R is invertible with [ 1|2 < ||
2. Another possible choice is take R = A, and ensure A A (W W A which can be simplified to w w(W) < >mmin (A)/ >max M (A2) where w(W) is numerical abscissa
w(W) = Amax + WT)/2. 3/2. For example, if A 2 O any W with non-positive real part of the spectrum suffices.

---

Published as a conference paper at ICLR 2025
Example 7 (associative memory with smooth Leaky ReLU), Here we build concrete example
of hierarchical associative memory Krotov (2027) with smooth Leaky ReLU Biswas et al. (202) to avoid dealing with ODE having a non non-smooth right-hand side. The activation function reads g u
(5 + 3erf (3u/8)) /8 and its derivative is g (u) = g(u)/u + 9u exp (-9u²) / /64) / (32 where all operations are pointwise and erf is error function. Weights and state vector are selected as Example | but without requirement W Wis++1 = W1,,. After that we select R I-WT A and obtain the Jollowing dynamical system
ú(t)= (I (I-WTo @g (Wu(t) + b)) (o(W + b) -u()), u() u4.
For this system Lyapunov function E2 = 2 (g (( +b) - u(()) T (g( u(() + b) u(t)) is non-increasing trajectories. Matrix W Og (Wu(t) + +b) can in principle have non- trivial nullspace. To avoid this one observes that [| I max; [Wri+lllg + max; W ll2 (the bound is tight) and take W1,++ /i,,ta/ (2 W W..ctal)) (.) some W with the same block structure. Note, that in this construction A is not positive definite, and W is not symmetric.
W
All theoretical statements that we derived are valid only for modified dynamical system []] However, it is easy to see that energy function E's from Proposition S can be adopted for the original temporal dynamics I The resulting energy function and dynamical system reads
ÿ(t) R(y(()) (W) - yt( + b). v(0) y/:
1 Ey(y)
2 ( (Wg - u() + b) S (W (W g(v(t)) y(t) + b)
(10)
We used temporal dynamics p with R = I - W (y) and energy function 10 with S I to generate results given in Figure One can also prove results similar to Proposition function (T0) but since the extension is straightforward we do not pursue this further.
5 for energy
CONCLUSIONS
We describe the effect of dead neurons on the energy landscape of associative memory, analyze the consequences for stability, and provide several remedies, including new dynamical systems with good energy functions. We think it is appropriate to discuss the overall significance of the Lyapunov function for associative memory. Is it necessary to have this function at all? How is this function used in practice?
One may observe that, currently, the Lyapumov function is underutilized. As a rule, one uses several steps or even single step of temporal dynamics of ODE during training and inference Hoover al. (2024), Hoover et al. 2002). ). Millidge et al. 2022a),, Ramsauer et al. (2020). Krootvv & Hopfield 2016 The role of Lyapunov's 's function is merely to provide comfort that some steady states may exist somewhere. As we show in this article, it is often a false comfort.
It is important note, that the Lyapunov function in itself does not ensure stability: (i) it may be the case that no steady state exists, (ii) limit cycles may still be present, (iii) all steady states may be unstable. Besides that, for any steady state, the Lyapunov function can be constructed (by solving the Lyapunov equation) even when the "global" Lyapunov function is unavailable.
The Lyapunov function, as is currently considered, is not of huge help. We can suggest several appropriate use cases: (i) model parameters may be adjusted on the training stage to make the Lya- punov function unstable for particular states something that can not be easily done by integrating the dynamical system alone. This may help to learn from negative and adversarial examples Wang et al. (2024), Goodfellow et al. (2014); (ii) Lyapunov function may help in theoretical understanding of memory capacity, e.g., in the present article we have already shown that whole flat direction cor- responds to the basin of attraction, and can not support more than a single memory; (iii) Lyapunov function may be directly used to compute and manipulate basins of attraction, potentially speeding up learning and making memory more robust to adversarial attacks.
All in all, we think that the Lyapunov function is a powerful tool that can lead to novel theoretical results and practical learning techniques in the field of associative memory. We hope that our results will inspire further research in this direction.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Smu: smooth activation function for deep networks using smoothing maximum technique. arXiv preprint arXiv:2111.04682 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Marco Chiani. Distribution of the largest eigenvalue for real wishart and gaussian random matrices and a simple approximation for the tracy-widom distribution. Journal of Multivariate Analysis, 129:69 81, 2014.
Chenhao Cui and Tom Fearn. Modern practical convolutional neural networks for multivariate regression: Applications to nir calibration. Chemometrics and Intelligent Laboratory Systems, 182:9-20, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.
Wassim M Haddad and VijaySekhar Chellaboina. Nonlinear dynamical systems and control: a Lyapunov-based approach. Princeton university press, 2008
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Benjamin Hoover, Duen Horng Chau, Hendrik Strobelt, and Dmitry Krotov. A universal abstraction for hierarchical hopfield networks. In The Symbiosis of Deep Learning and Differential Equations 11, 2022.
Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffman, Zsolt Kira, and Duen Horng Chau. Memory in plain sight: A survey of the uncanny resemblances between diffusion models and associative memories. arXiv preprint arXiv:2309.16755 2023.
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances Neural Infor- mation Processing Systems, 36, 2024.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10))3088-30 1984.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Dmitry Krotov. Hierarchical associative memory. arXiv preprint arXiv:2107.066446 2021.
Dmitry Krotov, A new frontier for hopfield networks. Nature Reviews Physics, 5(7):366-367, 2023.
Dmitry Krotov and John Hopfield. Large associative memory problem in neurobiology and machine learning. arXiv preprint arXiv:2008. 06996, 2020.
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Peter Lancaster. Explicit solutions of linear matrix equations. SIAM review, 12(4):544-566, 1970.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy- based learning. Predicting structured data, 100) 2006.

---

Published as a conference paper at ICLR 2025
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint mew::1933.073, 2019.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3):531 1992.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Atlanta, GA, 2013.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Uni- versal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pp. 15561-15583, PMLR, 2022a.
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. Back- propagation at the infinitesimal inference limit energy-based models: Unifying predictive cod- ing, equilibrium propagation, and contrastive hebbian learning. arXiv preprint arXiv:2206.02 02629,, 2022b.
D.W. Nicholson. Eigenvalue bounds for ab+ba, with a.b positive definite matrices. Linear Algebra and its Applicutions, 24:173-184, 1979. ISSN 0024-3795. doi: hups://doi.or 10. 0.1011600044-379599999995577--5 URL http://wwwwweeeeeeeeete...ooo
atticle/s11/00225775577000
Richard S Palais. A simple proof of the banach contraction principle. Journal of Fixed Point Theory and Applications, 2:221-223, 2007.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.022: 02217, 2020.
Valeria Simoneini. Computational methods for linear matrix equations. siam REVIEW, 58(3):377-
441, 2016.
Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
Fei Tang and Michael Kopp. A remark on paper of krotov and hopfield [arxiv: 2008.06996.. arXiv preprint arXiv:2105 1..00.55.2.. 15034, 2021.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint 1xXi:24402.11651 2024.
Xiaohui Xie and H Sebastian Seung. Equivalence of backpropagation and contrastive hebbian learn- ing in a layered network. Neural computation, 15(2):441-555 2003.
A RELATIONS BETWEEN MEMORY MODELS
Our model is most directly related to Dense Associative Memory described in Krotov 2021 Equa- tion (2)). This equation reads
dx dt
N
Σ W1J9J II, -I1
where x7, l,..., N are activities of individual neurons, gs which is gradient of Lagrange function I. that depends all activities, and 7y control relaxation
d is activation function d2,
12

---

Published as a conference paper at ICLR 2025
time of individual neurons. Energy function used in this work is available in equation (Krotov 2021
Equation (3)) and is given below:
1
E
rigt
L
Σ
gr W1J93-
2 1.J=1
It is easy to see that there is direct relation between dynamical system ① and the model (2) use in the article and the ones from Krotov (202I): WiJ are the elements of matrix W, activities æy components of y. Two differences between the model from that article and Dense Associative Memory are: bias term that we use is absent from Dense Associative Memory MemoryKrrotoy (2021) time model. The bias
scales тy are absent from since it is not used in the stability analysis, this does not affect our results. Observe, that b = the energy that we use (2) is precisely the same in Krotov (202T). Similarly, time scale 7Ι absent from the definition of energy function in Krotov (2021) so they do not influence flat energy regions. However, they mildly influence stability analysis since after linearisation Jacobian will by multiplied by D where D is a diagonal matrix containing time scales on the diagonal.
can be simply removed by taking b 0 and
Next we discuss the relation to model (Krotov Hopfield, 2020 Equation (1)). This model seems to be quite different from our formulation, but it is specific case of Dense Associative Memory (Krotov] 2021 Equation (2)) with bias term. To see this we describe model given in (?, Equation (1)) In the article Krotov & Hopfield (2020) authors split neurons two parts: activations of memory neurons are 4 μ I..... N₂, and activations feature neurons are U, i = ........y... These
neurons has distinct activations function fp f( and gi g(h,) which are defined derivatives
of corresponding Lagrange functions fp [L, and 9s = aL.((v+.)
(Krotov & Hopfield
Sh
Be;
2020 Equation (3)). Dynamical equations for these activations (Krotov & Hopheld 2020 Equation
(1)) is reproduced below
du:
Tf dt
Euff - 22 1i,
dhp Th dt
Σ Epiti
hpp
where Eur = Eiu are symmetric weights describing "weights of synapses" and IŞ is "the input current into feature neurons." Lyapumov function given in (Krotov & Hopfield 2020, Equation (2)) reads
N
N4 Σ
E
0
1g): Le
+
Ln
SpEuigh p4
To see that this model is related to (rotov 2021 Equation (2)) consider a special weight matrix WIJ, Fı, that have the following block structure
W
0 T
- x ())
-#),==-
T/Cp
Then
where e, and en identity vector of sizes N,, No. With this split and Lagrangian L((zr}) Lo ((v+)) + La ((hy)) we have
Σ W1139 x, W
0L dx
0 E
5
(olv) f(h)
h
so we see that we reproduce (rooovv Hopfield 2020 same way reproduce the energy function but without 1,
Equation (1)) without bias term I,. The
Given that, both models (Krotov Hopfield 2020 Equation (1)), (Krotov 2021 Equation (2)) are directly related to ( The only difference is these models contain 5 variable that speed up slow down temporal dynamics for selected neurons. Importantly, energy functions from models Krotov & Hoptield (2020), Krotov (2021) are equivalent to (2).
Note that other modifications of memory models are available. The main problematic part that lead to flat energy is Lagrange function introduced in Krotov & Hopfield (2020). So, whenever this
13

---

Published as a conference paper at ICLR 2025
function is used, the energy will have non-compact flat regions corresponding dead neurons. To give an example, in Millidge et al. (2022a) authors build a modified model but used a non-trivial Lagrange function Ly in Equation (3). If f(h) sep(h) can produce dead neurons, the energy from Millidge et al. (2022a) will have a non-compact flat region. One such situation occurs when the softmax function is used sep(h) in (Millidge et al. 2022a Equation (5)).
B PROOF OF PROPOSITION I DEAD NEURONS LEAD TO NON-COMPACT FLAT ENERGY REGIONS
We assume that we are at point y in the region where neurons are dead, so there is V such that gy + Vc) g for arbitrary with positive components. We substitute y + Ve into the definition of energy to obtain
E Vc) (y + Ve - b)"
g g Ly Vc) - ( W
First, we use g(v + Vc)= = g to remove shift from activation functions
(ッ) Fly Ve) (y + Ve b) g) - (v Vc)
2 (v)) Wg
1
Next, we use Taylor series with mean-value reminder to expand Lagrange function
[T()))
L + Ve) = (v)
TE (0,
0z
Since by definition g
aL(y) 8y
the expression above can be presented as follows
L (y Vc) (v) (g (y (Ve) Ve, (0,
Now, since 7 > we use g + VVc) = -V) to remove shift and obtain
I (y + Ve) (b) + (v)) Ve We substitute this the last expression for energy which gives
1 Ely + Vej = (y Vc b) g (y) - L ( (g (2)) Ve -0---1000 (1)) (y)
Two remaining terms containing e are (Vc)² (अ) and (3)) Ve cancel each other, obtain
Fly Vc) =
(y (y b) g (v) T. (v) 2 (v)) Wg (v) Fly).
as claimed in the proposition.
C EXAMPLE OF MEMORY CAPACITY DEGRADATION FOR ENERGY FUNCTION BOUNDED FROM BELOW
Figure 2a shows an energy function that is unbounded from below but supports two stable states. In this section we argue that the unbounded energy is not problematic. Moreover, if one is too strict about this property, memory can severely degrade capacity.
It is easy to find papers on associative memory that claim that it is necessary to have energy function bounded from below to ensure stable memory recovery, e.g., see discussion after equation (4) (in both cases) in Krotov (202I) and Krotov & Hopfield (2020). In Xie & Seung (2003) authors even incorrectly claim "Furthermore, with appropriately chosen fe, such as sigmoid functions, E(x) is also bounded below, in which case E(x) is Lyapunov function".. Clearly, the Lyapunov function not required to be bounded.
We certainly agree that if the energy function is bounded and informative (not flat), the memory vector will be recovered eventually from arbitrary initial conditions. However, we have several arguments again the overall importance of this condition:
14

---

Published as a conference paper at ICLR 2025
1. Unbounded at infinity only. Activation functions used for associative memory least continuous, meaning energy can be unbounded only when the activities of neurons reach infinity. This kind of run-away behavior is very easy to detect and prevent when actual implementation of associative memory.
2. Finite number of updates in practice. In practice ODE not integrated for very long time. As we discuss in Section 5 one usually performs a small number of steps for discre- tised dynamics, c.g., 10 or even step. Under these conditions is not possible to diverge using ODEs considered in this article.
3. Stability is a local condition. For associative memory one cares more about local stability in the basins of attraction, and the fact that energy is bounded from below is global information, which is seldom important for local stability (with the important exception discussed below). In practice one can always rescale input vectors (initial conditions) to localize it on sufficiently small sphere with radius R. Memory trained and used this way has little chance to diverge.
4. Extra constraints can compromise capacity. we require energy to be bounded from be- low we will put extra constraints on parameters of associative memory. These constraints, as we will show below, can decrease capacity dramatically.
The last point, the decrease in capacity, is especially interesting. We will illustrate it with ReLU memory model
dy((() WReLU (v(()) y(t b, ((0) yo,
dt
that has energy
Σ (ReI.U(;))) 1 (ReL T W ReLU(yy)
Fly) (y - b) ReLUU()
To analyze this energy function we observe that y ReLUUy) (ReLU(y)) ReLU(y). This fact allows us to simplify energy to
E'(y) (ReLU(y)) (I- W) ReLU(y) beReLU(y).
It is possible to bound this energy function from below using spectral radius p of W
Ely) 2 1 ((W) (ReLU(y)) ReLU(y) ReLU(y),
2
note that this bound is tight unless we restrict allowed weights, since it is saturated for W al for scalar a. From this lower bound we can observe that memory is bounded from below if (W) < 1. Moreover, in light of our previous comment, is a necessary and sufficient condition.
Suppose now we restrict W such that (W) = 1 -e e with arbitrary small but nonzero e. In this case ReLU memory has single memory vector.
We start by showing that a steady state of temporal dynamics exists. For that we consider a discrete iteration
+ bb Wl2 [|(ReLU (cr) ReLU (c2)ll2 < [ ||c c2ll2 = (1 - ||c, c2ll2
(n+1) WReLU
y (n))
Observe that right- right-hand side is Lipschitz function with Lipschitz constant 1
W|RLL (c1) + b (WReLU (c2) + b)ll2 = =|[W (ReLU (c1) - ReLU (c2))ll2
Given that, by Banach fixed point thcorem Palais 2007) these iterations has unique fixed point. This fixed point is steady state since
y WReLU = WReLU (y") 3² -b=
so the right-hand side of the dynamical system is zero.
Now, we will show that this steady state is unique. For that we consider two trajectories starting from (arbitrary) distinct initial conditions y (1), y2((): yn (0) yo. y²( - We will show that
15

---

Published as a conference paper at ICLR 2025
the distance between y/ (t) and (2((t) decreases with time. To do that consider derivative of this distance:
1 d
(dy dyz(t)
dd v1(()) y2(())12 (y)(t) 32(()))
dt di
= (1/((t) (2())) (WReLU (11 (t)) yt(t) WReLU (32(()) )) + y2(()) (ly) =-1ly1 y2 + (t) (t)) (WReLU (t) WReLU (ya(t)))
To bound second term we use that a < |a| and Cauchy Cauchy-Schward Schwarz inequality
1 d (1) y2 (1) - (l))(() 22 (1) (w) 92(1)) (WReLU ())(0)) WReLU (12(())
< - 1 (t) v2(1)l2 + |1 (t) - (1)l R (y (t)) RReLU (y2(t)) 12
Finally, use result that WReLU(-) is Lipschitz with Lipschitz constant to obtain
d
de (1))(() y2 (1) S |1 (1) 3/2 (1) (l - ya (2))) -e) = (1) y2(1)
Using Grönwal inequality obtain
2et d
y1 (1) y2 (1112 Se
d 1 (0) y2(0))l2
dt
In other words, separated trajectories converge exponentially fast. Since there is a fixed point one of the trajectories might as well start from y² which means this fixed point is the only one, and is exponentially stable.
The result that we demonstrate here means one should be careful restricting parameters associative memory, since it may lead to degradation of capacity single memory. The proof above is not working for p( (W) 1, but since can be arbitrary slow one can recover the same result in a limit O.
D MEMORY MODEL FROM HOPFIELD 1984)
In this section we align our notation with the one from Hopfield (9944). The model interest is given in (Hopfield 1984 Equation (5)). We replicated this model below
C dup dt
ΣT.V u; R, + 4,
Uj = 9i (V:),
where u, an instantaneous input to neuron i, Vú is the output or "short term average the firing rate of the cell i", g, is activation function or "the input-output characteristic of nonlinear amplifier with negligible response time". Te are weights and T. "finite impedance between the output Vj and the cell body of cell f", R, is "transmembrane resistance" and C, is "input capacitance", I; is bias or "any other (fixed) input current to neuron t".
If put aside the biological content of [Hopfield] (194), this equation is already very similar to all models we discussed in Appendix A To make the resemblance even more evident observe that R2 be removed without the loss of generality, since can multiply them and redefine C + R,C,, Tig + RTTgg 4 + RyI. After that we also substitute V = g(2:) and obtain the following equation
C. du Tings(u u; m 4t,
di
which is precisely the Dense Associative Memory (Kotovv 2021 Equation (2)) with weights Tijs time variables C, extra bias term I,. Since already discussed the relation of Dense Associative Memory to our model in Appendix A we can be sure that model from Hopfield (1984) is in the same class of memory models.
16

---

Published as a conference paper at ICLR 2025
Dynamical system (Hopfield 1984 Equation (7)):
Equation (5)) comes with the energy function (Hopfield 1984
/ dV (v)+ L.V. _T. Rí
1 V4
E= - IT,,,, + Σ
Using V2 = gi( we obtain
V. E E= - -=n L dV 9i (V) [eg(uu)
In addition to that we can drop R& for the reason explained above
(ug(()))))))) + ΣL, dV g. (V) + Σ (ig:(us)
Σ
= 2 2j
The first and last sums are easily mapped on our notation and the second sum is explained Sec- tion 3.1b.
all, one can say that our model similar to the one from Hopfield (1984) but with C; R.
1. As we argued, R, is irrelevant and C2 does not appear in the energy function.
E PROOF OF PROPOSITION 3 STABILISATION OF DYNAMICS FOR RANDOM MATRICES
Matrix W1 can be written as O WOD where O orthogonal full rank matrix and D is first N - k columns of I. Since GOE is invariant under orthogonal transformations O WO is also matrix from GOE. The effect of D to select a principal submatrix with N rows and columns. Given that W is also from GOE but with N - k rows and columns. For GOE with large N, the distribution of leading eigenvalue quickly converges to -ricy- distribution (see Chiani 2014) equation (50) and numerical results), on the average, spectral radius of matrix with N k rows and columns is 2((N k) (see below) and E Willl [| = 1 N
To obtain an average spectral radius for the GOE matrix M we will use the theory described in Section 4 of CCiani (2014). Let x be a spectral radius of matrix M N We define Gaussian orthogonal ensemble as set of random symmetric matrices with independently identically normally distributed entries (for upper diagonal part) with variances I and 1/2 for entries on and off the diagonal respectively. For sufficiently large N is known that
A1 = UN + PTW, N
(1I)
where is a mean value of spectral radius, μTW, 1.2 is mean value of Tracy-Widom distri- bution and
1 2
1
1 N + 0.
AN
2
N
N
V2N 12
(12)
10
2
V2
So we see that for large N the average spectral radius of the GOE matrix is V2N.
F PROOF OF PROPOSITION 5 ASSOCIATIVE MEMORY WITHOUT DEAD
NEURONS
Observe that aL(Wu+b) Wg (Wu + b) and [g(Wu+b) Ouu
A (Wu+b) W.
Du
Using the identities above we find
OE WA (Wu + +b) W (u g( g(ub+b))
dE2
W (u g(Wu+b))
ru
Iu
0E (I- WA (W 5) S( -u- (ggggg(Wubb)
du
17

---

Published as a conference paper at ICLR 2025
Multiplying gradients a, 8, and adding them gives @E(u) ou
(u) (u). Since i
(a())( where (()(t) = u (Wbbb+ from E =u
2E
we obtain
Du
= f- (R F R f(u).
If one can find positive semidefinite matrix Q such that R F(u) + (F(u)) R Q. energy is non-increasing on trajectories since in this case E =f Qf 0
2. Consider polar decomposition of matrix F(u) = P(u) and take R(u) = O(u).
Since P(u) 2 we have R F(u) + (F(u)) R = ( 2 0 and E < 0.
3. From the proof of Proposition I we know that if Wu + b - Wu + b Vc, Lagrange function shifts on g (Wu b) Vc. The first term in the energy function (6) is quadratic and does not contain so in general energy Eg does not have flat directions. Similarly, Eg has form f(u) Su) where f is not invariant under transformation Wa u + b Wu + b + Ve meaning Ez is not invariant as well.
4. Since BE Pru
F()) and f(u") we find for sufficiently small Ò
Of(u)
E(u" 5) E(u") 5 F(u")
5.
du
u=
For stability needs this quadratic form be positive definite, which gives
18


Reviews:
[{"review_id": "XqwMeIwwK7", "reviewer": "Reviewer_yNbD", "summary": "The work investigates Lyapunov functions and stability in the Krotov and Hopfield models of associative memory. It is found that the conventional formulation suffers from a problem of dead neurons, which arises when some of the neurons’ activation functions saturate. In this situation the energy becomes independent of the state of those neurons. The authors propose a projection of the original class of models on a subspace of non-dead neurons, analyze the stability of those models, and propose a novel class of Lyapunov functions that describe those models.", "questions": "1. I am not sure I understood the derivation in line 284. Why does the Taylor series terminate after the second term? \n2. It seems to me that the model presented in equation 9 can be obtained from the Krotov-Hopfield model by introducing a new variable $\\mathbf{y}=\\mathbf{Wu}+\\mathbf{b}$. Are these two classes of models equivalent under this change of variables, or does the introduction of matrix R make equation 9 more general than Krotov and Hopfield?\n3. In all the derivations $\\mathbf{b}$ is assumed to be time-independent. This is fine (same is true in Krotov-Hopfield). I am wondering though, if anything could be said about situations when $\\mathbf{b}$ depends on time? E.g., could energy function (10) still guarantee convergences for some properties of $\\mathbf{b(t)}$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Dense Associative Memory models have several possible formulations and admit studies of a broad class of AI models used in practice. At the same time, their Lyapunov (energy) functions remain underutilized and the shape of those energy landscapes remains under investigated. This paper tackles this exact question and analyses the stability of the dynamics and the sensitivity of the energy with respect to the state of the neurons from a novel angle. The problem of dead neurons can be a serious obstacle for training these models at scale in interesting AI applications. This work formalizes this problem, and offers a possible solution by identifying non-dead subspaces and proposing a novel class of energy functions. The paper is generally well-written, although there are small issues, which I explain below. The results presented in this work are novel and relevant to the ICLR community.", "weaknesses": "There are two issues with the way how authors introduce the problem of dead neurons (example 2, 3, 4). First, in the Krotov-Hopfield formalism, one needs to check two things: \n1. Positive definiteness of the Hessian of the Lagrangian. \n2. Boundness of the energy function from below. \n\nOnly when both conditions are satisfied the models under consideration have stable fixed point dynamics. The authors rightfully focus on the first requirement in their presentation, but somehow the second one is not discussed enough. For instance, please notice that the model studied in Example 2 does not have an energy bounded from below. This means that one can choose a matrix $\\mathbf{W}$ such that the energy will keep decreasing to minus infinity with time, and the model will never reach a fixed point. The lower-boundness of the energy needs to be checked and discussed in the paper. \n\n\nSecond, Example 4 represents a situation when Lagrangian has a symmetry. Please notice (this is acknowledged in lines 194-195) that in this case there are no dead neurons, all the neurons are alive, it’s just the energy function is insensitive to the states of the variables $\\mathbf{y}$ along the constant shifts of all pre-activations. I am wondering to what extent it makes sense to dub this specific example as a “dead-neuron” problem. The authors rightfully acknowledge this in lines 192-198, but I would invite the authors to consider restructuring the presentation of this example and give this specific phenomenon (Example 4) a different name, e.g. “symmetry”, “flat directions”, etc., not the dead-neuron problem. \n\nSome typos: \n1. Line 507: $E_3(\\mathbf{u})$ -> $E_3(\\mathbf{y})$\n2. Line 509: $\\Lambda(\\mathbf{u})$ -> $\\Lambda(\\mathbf{y})$\n3. Line 156-157 - subscript in $W_{21}$. \n4. Line 535: Lapynov-> Lyapunov", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "011P9c4SQG", "reviewer": "Reviewer_oj8s", "summary": "This work expands on the literature of modern Hopfield networks, and that of Dense associative memories. In detail, it identifies a drawbacks of the current approach developed by Krotov and Hopfield, that is, the presence of ‘dead neurons’ that cause regions of constant energy. The authors then address this problem by defining a new family of Lyapunov functions that is a variation of the original one, but does not suffer from the problem of dead neurons causing flat regions.", "questions": "Are there specific tasks in which you could measure the advantage of the newly proposed energy function? For example, testing how many stored memories can you perfectly retrieve, or the capacity of the model? Similarly to the many provided in Millidge et al.?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work is well structured: it identifies a gap in the literature, and theoretically addresses it. The gap is also a significant one, as flat regions of steady states can negatively affect the retrieval of specific memories.\n\nThe stability analysis is interesting, and it seems well made. Similarly, I share the belief of the authors about the underutilisation of the Lyapunov function in the literature. Actually, I would move the discussion in the conclusion (“*The role of Lyapunov’s function is merely to provide comfort that some steady states may exist somewhere. As we show in this article, it is often a false comfort.*”) in the introduction, to make the claim of the paper clearer. \n\nDespite not being sure I was able to follow the math all the way, I believe the solution proposed by the authors to be correct.", "weaknesses": "The biggest cons of this work is the clarity. I found it hard to follow the computations, theorems, claims, and their consequences. The 10 pages are very dense. I would: \n\n1) Add figures that make some concepts clearer; \n\n2) Move some non-interesting proofs to the appendix, and leave in the main body only the computations needed to understand concepts (you could even provide proof sketches).\n\nAs I did not manage to follow the paper perfectly, I will also provide a low confidence score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ad2Js9GxNe", "reviewer": "Reviewer_WSRe", "summary": "This paper builds on the work of Krotov and Hopfield 2020 looking at associative memories, with particular emphasis on the classical continuous Hopfield network (Hopfield 1984), the Dense Associative Memory (Krotov and Hopfield 2016), the modern continuous Hopfield network (Ramsauer et al. 2020), and the energy transformer (Hoover et al. 2023). More broadly, this paper looks at associative memories as dynamical systems, similar to neural ODEs, and investigates the impact of dead neurons on such systems. A dead neuron in this context refers to a neuron which has no effect on the model behavior due to always taking values that are in some way constant. Dead neurons can arise from many situations, but two such classes discussed in Section 2 are saturated neurons and symmetric activation functions. This paper then analyzes how dead neurons impact the degrees of freedom of a dynamical system and the Lyapunov function of the models. In particular, it is shown that the associative memories in current literature may suffer from steady states that are not stable i.e. there are dimensions of the state space (associated with each dead neuron) that are free to vary without affecting the energy. The paper then introduces new energy functions that alleviates some of the issues relating to dead neurons and shows all architectures explained by the functions in Krotov and Hopfield 2020 are also explainable using the new functions by an affine transformation of the neuron activities.", "questions": "There seems to be a typo on line 044: \"The most cluical requirement\" should be \"The most crucial requirement\".\n\nEquations 1 through 4 --- what do these correspond to in Krotov and Hopfield 2020? Looking through the cited paper, there are no equations of a similar form, have the authors of this paper rewritten something from Krotov and Hopfield 2020 in a more convenient form? If so, perhaps a brief description of what has been changed from Krotov and Hopfield 2020 (a notation translation guide) would benefit this paper. Further, Equations 1 through 4 appear to actually consist of more than four equations, some of which are constraints (such as W = W transpose). Cleaning up the formatting here such that each equation is on one line with a separate number may benefit the paper.\n\nAt the start of Section 2 the paper makes several claims about the properties of Equations 1 through 4, including important properties of the energy function in Equation 4. However, these claims are not obvious to me. Is there a reason that the \"structural assumptions\" of Equation 2 (is that the first or second equation on line two?) ensures the energy function exists? This discussion could be expanded slightly for readers who are not familiar with the equations set forth here.\n\nThe next paragraph continues these discussions and makes claims about the dynamical systems described by Equation 1. However, again I can not immediately see why the steady states of these differential equations can be used as memory vectors --- could this be discussed further or given a citation?\n\nExample 1 seems somewhat ill-defined in the edge cases. The given differential equation for y_i does not apply for y_1; what is the value of the weight matrix W_{10}? Should we treat this as 0? This would line up with my expectation of the MLP with feedback connections, but the formalization does not support it. Perhaps some discussion of the edge cases here would be beneficial.\n\nThere is a slight typesetting error on line 156: The equation is currently rendered like \"$ W_{12} = W_{2}1^\\top $\", rather than \"$ W_{12} = W_{21}^\\top $\" i.e. the \"1\" on the second matrix W is not subscript. \n\nExample 2 is the first time that the term \"dead neuron\" is linked directly to the ReLU activation. Dead neurons are referenced to Lu et al. 2019 in the introduction (line 091) but perhaps a short introduction to the term to ground this work would help a reader understand where the discussion is heading. Especially considering the term is then applied to other activations in Examples 3 and 4. What exactly is meant by a neuron being dead? This could be added to the introduction. Reading on, the definition of a dead neuron appears quite late in Section 2, which is already substantially through the paper. Is there some way to introduce the key concepts of the paper earlier?\n\nThe paper seems to have a significant focus on the energy function being non-discriminative on a non-compact region of state space. It may benefit a reader's understanding to have a short discussion on what this means, especially in the context of associative memories and physics inspired models.\n\nThe proof for Proposition 1 is somewhat difficult to follow. The Taylor expansion of the Lagrangian seems to make the implicit assumption that $ L(y) = 0.5 g(y)^2 $, which perhaps could be stated explicitly in the proof even though this is a common formulation. Past this, it is not clear to me why the second term shifts by the specified quantity, and how this is compensated by a similar shift in the first term. Some more verbose discussion here could help readers follow the proof.\n\nMuch of the body of this paper hinges on the formalization of Equations 1-4. Without having this section very clearly laid out for the reader, the remainder of the paper is often difficult to follow. Is it possible to rework this initial Section 2 to improve clarity? In particular, stating notation clearly early in the paper would be very helpful. Following sections would then be able to reference back to the key statements in Section 2 with minimal confusion.\n\nAgain for Equation 5 it is not clear what Lyapunov function you are referencing in Hopfield 1984. This looks to be most similar to Equation 7 or 11 in the 1984 paper, but the lack of notation formalization in this work makes it difficult to draw parallels.\n\nMuch of Section 3 and 4, (e.g. Section 3.2), may benefit from being written into a longer paper where the authors can be more verbose in explaining each step of the mathematics.  Currently, the paper is very terse in its discussion, which is understandable considering the page limit of the conference, but this does not help a reader understand the key aspects of this mathematically dense manuscript.\n\nNear the start of Section 4, where we discuss the value of R(u), are we assuming that no value of the matrix R(u) is zero, or that the matrix is not equal to the zero matrix (i.e. all values zero)? The manuscript is slightly unclear on this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The work seems rigorous in its discussions, providing proofs of propositions and analyzing the proposed energy functions extensively. The work appears original, although it is mostly building off previous associative memory literature and work on dead neurons. The paper appears to be reasonably significant in the field of associative memories, and if the claims of the impact of dead neurons are true, this could represent a significant step in the analysis of neural ODEs and similar dynamical systems.", "weaknesses": "The basis of the paper, Section 2, is confusing and difficult to match up to other literature on associative memories. In particular, Equations 1 through 4, which are fundamental to the remainder of the paper, are stated to be from Krotov and Hopfield 2020, but do not appear in the form given in this paper. A rewriting of Section 2 would be beneficial to improve the clarity of the work and aid reader's understanding. Rewriting of subsequent sections to reference a clarified Section 2 would also help a reader follow each point made by this work.\n\nThe notation used in the paper is unique to other papers on associative memories, which in itself is not an issue, but there is no explanation of the notation. This is a problem when discussing other literature, as matching the discussions in this paper to other referenced papers becomes nearly impossible. In many sections, steps are taken that are not obvious to the reader and may impact the ability to follow this work. There are numerous grammatical errors throughout the manuscript that can obfuscate the meaning of sentences or entire paragraphs. It is difficult to verify the claims made in this paper due to the difficult-to-follow nature of each section.\n\nThe key ideas of the paper seem sufficient for publication, but would need to be presented in a more understandable format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)