On the Role of Hidden States of Modern Hopfield Network in Transformer
Authors: Tsubasa Masumura
Masato Taki
Venue: NeurIPS 2025 poster ()
Keywords: hopfield networks
self-attention
Transformer
rank collapse
Primary area: deep_learning

Abstract: Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learning. It has been pointed out that the state update rule of the modern Hopfield network (MHN) in the adiabatic approximation is in agreement with the self-attention layer of Transformer. In this paper, we go beyond this approximation and investigate the relationship between MHN and self-attention. Our results show that the correspondence between Hopfield networks and Transformers can be established in a more generalized form by adding a new variable, the hidden state derived from the MHN, to self-attention. This new attention mechanism, modern Hopfield attention (MHA), allows the inheritance of attention scores from the input layer of the Transformer to the output layer, which greatly improves the nature of attention weights. In particular, we show both theoretically and empirically that MHA hidden states significantly improve serious problem of deep Transformers known as rank collapse and token uniformity. We also confirm that MHA can systematically improve accuracy without adding training parameters to the Vision Transformer or GPT. Our results provide a new case in which Hopfield networks can be a useful perspective for improving the Transformer architecture.

On the Role of Hidden States of Modern Hopfield Network in Transformer
Tsubasa Masumura" Masato Taki*
Graduate School of Artificial Intelligence and Science, Rikkyo University, Japan (25wr001m, taki_m) erikkyo............. .ri.kyo.ac.jpp ac.
Abstract
Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learning. It has been pointed out that the state update rule of the modern Hopfield network (MHN) the adiabatic approximation is in agreement with the self-attention layer of Transformer. In this paper, we go beyond this approximation and investigate the relationship between MHN and self- attention. Our results show that the correspondence between Hopfield networks and Transformers can be established in a more generalized form by adding a new variable, the hidden state derived from the MHN, to self-attention. This new attention mechanism, modern Hopfield attention (MHA), allows the inheritance of attention scores from the input layer of the Transformer to the output layer, which greatly improves the nature of attention weights. particular, we show both theoretically and empirically that MHA hidden states significantly improve serious problem of deep Transformers known as rank collapse and token uniformity. We also confirm that MHA systematically improve accuracy without adding training parameters to the Vision Transformer or GPT. Our results provide a new case in which Hopfield networks can be useful perspective for improving the Transformer architecture.
Introduction
The relationship between associative memory in Hopfield networks 22 which has attracted
interest from neuroscientists, and Transformers [46] based on key-value memory that have been
studied in machine learning has attracted interest from the research community [45] 42 48 21). One of the most interesting results is the finding in [40[31]
31 that translating modern Hopfield networks into neural networks yields the Transformer architecture that has been very successful in natural language processing [38 [38]9 and computer vision [2]. What, then, do more general modern Hopfield
networks imply for deep learning? This paper gives concrete answer to this question.
Hopfield networks 22 are a class of models for associative memory. Despite these interesting properties, classical Hopfield networks have the limitation of small storage capacity. Recently, [30 proposed Dense Associative Memory which can achieve storage capacity that scales exponentially power-wiss wise with respect to the number of neurons by introducing high nonlinearity [7]. These models with large storage capacity are collectively referred to as modern Hopfield networks 40 31
Recent advances in Transformer architecture, including its application to language models, have led to significant advances in the study of self-attention mechanisms. These advances have also shed new light on Hopfield networks. In [40]. it was shown that the state update rules of modern continuous Hopfield networks (MCHNs) have a mathematical structure exactly equivalent that of the self- attention mechanism. Furthermore, 3I developed this relationship theoretically, pointing out that
These authors contributed equally to this work
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mintani
K
Lae car
softanck
Matuoul
(b)
Figure 1: (a) The left figure shows the layer structure of Transformer architecture using modern Hopfield attention (MHA). As the hidden state H,, propagates through each attention layer, infor- mation from the upper layer's 's attention scores is reused in the lower layers. Attention score K is accumulated the hidden state each layer, and this value is used for attention calculation. (b) A visualization of the token uniformity in layers 12 and 24 of GPT-2 (Medium) trained on the Wikitext100 dataset, showing a violin plot of the cosine similarity between the tokens. For GPT-2 in the left column, there is strong peak similarity and both layers have mode of 1. On the other hand, in the case of GPT-2 with MHA in the right column, the cosine similarity is kept low and the uniformity of the tokens dramatically improved.
the self-attention layer of Transformer coincides with the adiabatic limit of generic modern Hopfield networks. Thus, it is expected that there is deep relationship between the Transformer's 's architectural design and associative memory. Still the connection, however, lack fundamental understanding.
Therefore, we consider the question: is it possible to interpret modern Hopfield networks without adiabatic approximation in terms of Transformers? The adiabatic limit approximation removes the hidden state dynamics from MCHN. In this paper, we show that maintaining this dynamics introduces a hidden state on the Transformer side, thereby creating mechanism for the propagation of attention score information from the upper to the lower layers.
By implementing this attention mechanism into Transformer architectures, this paper introduces a new type of self-attention layer called modern Hopfield attention (MHA), as shown in Figure MHA does not require additional parameters, and the increase in computational complexity is very small. Nevertheless, simply using MHA instead of the usual attention layer, performance gains can be obtained in various natural language processing and image recognition tasks. Furthermore, we found that MHA effectively solves the problem known as rank collapse, or token uniformity, where Transformer's tokens lose diversity as Figure [bb). These results indicate that ideas derived from the Hopfield network may provide new perspective for Transformer research.
In summary, contributions
follows:
By investigating the relationship between MCHN and Transformer beyond the adiabatic approximation, showed that this correspondence can be further generalized. Based on the correspondence with MCHN, we proposed a new type of attention mechanism with hidden state, MHA. The MHA-based Transformer improves the nature of attention weights by sharing attention score information across layers.
By training Transformers with MHA, we experimentally showed that the MHA mechanism also contributes to the performance of the Transformers. In particular, we investigated image recognition with Vision Transformer and text generation tasks with GPT-2, and confirmed that MHA actually improves their performance. This method does not generate any additional parameters, and thus can lead to performance gains with only a small increase in computational complexity.
Theoretically and experimentally, we showed that the reason why MHA works well as an alternative to self-attention is related to rank collapse. The hidden state MHA is likely to enhance its performance by cleverly improving Transformer's 's rank collapse Figure

---

I b). Our results suggest that the Hopfield Networks can provide guidance for improving the Transformer architectures.
2
Related Works
The relationship between the modern Hopfield network and the Transformer was investigated in
40
BT, and various improvements and extensions have been made to the modern Hopfield network
34
53
26
25
41
5 21
24 23 49 19 17
In 40 the authors demonstrate the fast convergence of the Modern Hopfield Network (MHN) and justify its use as a conventional module Transformer-related architectures. On the other hand, in this study's MHA, we propose dynamic structure that maintains and updates hidden states across layers. More specifically, our MHA naturally incorporates Hopfield recursion into the Transformer layer structure, as "state accumulation and updating" are performed in each layer.
Research on improving the design of Transformer architecture using this relationship has also been conducted in [20]. Unlike these studies, this paper focuses the effect of keeping hidden state dynamics of MCHN.
In this paper, saw that hidden states lead to reuse of attention scores across layers. Attention score reuse has been studied 16 10 from technical perspective, including improvements to the Pre-LN Transformer. These studies are focus only on encoder architecture and do not consider the special combination of moving average with a' and skip connection modification with a as in MHA. On the other hand, this paper showed that the more extended attention mechanisms in MHA can be understood in terms of modern Hopfield networks, and examines its effects including the decoder Transformer. Furthermore, the essential role of MHA is clarified theoretically and experimentally in terms of rank collapse.
Method: Transformers from Hopfield Network
In this chapter, review the methods 40 31 used to derive Transformer from MCHN and give careful treatment of discretization, which has been ignored in previous studies. As a result, we show that hidden state of MCHN leads to significant changes in the mechanisms of self-attention.
3.1 Sell-Attention Mechanism
Let T' be the number of input tokens with dimension d. X, R2'x is the feature obtained by concatenating the input token vectors a, of the n-th attention layer. The attention weight of Transformer is given by the row-wise softmax value of the attention score, which is given by the inner product of the query and the key, and the dot-product self-attention is calculated by weighting and adding the value vectors together X softmax K,, Vn. where the query, key, value given by linear projections of the input X, as Qn = X, Wo. Kn = X, WK and V, = X,, Wy (46). Each token vector is slice x,, (Xa)t, of the feature tensor. Then the formula for attention mechanism for each token is En softmax (,, Vn.
3.2 Modern Continuous Hopfield Network and its Discretization
MCHN is network model with bipartite graph connectivity connecting two dynamic variables and h. The connections are given by the network's weights W, which the memories to be associated are stored. æ is called the visible state or feature neuron, and h is called the hidden state or memory neuron. In the context of the associative memory model, given collapsed x as an initial configuration, the complete is reproduced by association through the time evolution of the state. The time evolution of MCHN given by the following update rule [31]:
dx = (h) W. dt
dh
= (x) Wa h,
(1)
dt
3

---

where Terh are the time constants of the dynamic system given by the Lagrangian functions Li. for h and æ
The activation functions f(-) and g(-) are
aLn
aL
f (h)
g (a) = dx
Oh
In this paper, the vectors h = (ha), f (h) = (fa (h)), vectors. In order to see the correspondence with Transformer below, us derive discrete time counterpart of MCHN. We then discretize this update rule with finite difference Al = bn+1 Ln as follows
(2t) and g (x: = (gi (a)) are all row
Tu A
[I++1 Fp.) (ht) W In.
Th
(ha+1- hh.) - g (2n) W2 hn
Al
where 2n x (t,,) and hn = h(t,). Introducing the ratio between the discretization step width and the time constant Ar 1 and Af - a', we obtain
In+1 = az, + +1 ff (h..) WT,, ha+1 = a'h, + a')g (xx.) W2.
In past studies, the effect of the discretization step was ignored negligible, but the precise derivation here leads to an interesting modification of Transformer. In this paper, we give empirical and theoretical results in which and a's effect is extremely important.
3.3 Adiabatic Limit and Self-Attention
Specific MCHN model is determined by explicitly selecting the Lagrangians. Model B of 31 given by the following choice of Lagrangians
LA = log Σ Eole)
Inp
51100112
(5)
These Lagrangians give the activation functions
fa softmax (ha): 9: Tz.
(6)
The adiabatic limit in 31 Th Rs 0 implies h, = x, W2. The update rule for (6) is then given by #n++1 = az, + (1 - o))ooftmxx (r,, W2) W.T.
Translating 7 by the rule qn = v, We. W, = x, W. = V, W. = Xn KW according to 140 31), we obtain xn+1 az, + (1 a)softmmax (q., V. where X, is the concatenated tensor of the embedding vectors of all tokens Tx1. ... EnT.. When a 0, i.e.. At = Te, this update rule is exactly a usual self-attention mechanism in [46]. In the following, we consider general a and a' investigate the effect of the hidden state dynamics, which are ignored in the adiabatic limit above.
3.4 Hidden State Dynamics and Modern Attention Attention
If the adiabatic limit taken and finite At is kept, the dynamics of the hidden state is
Th.
Th (hn+1 hn) = g (xn+1) W2 hn+1-
(8)
In the following, use the new parameterization At 1-a' obtain simple formula. The dynamics of Model B is then
1x+1 az,, + (1 ao)softmax (h,,) WTT. h = a' h. + +(1 a 2') W2.
(9)
Using the same translation rules before, we get the following novel modification of attention layer
Ba+t1 a. + (1 - a-softmaaxx (hw) Vn, h. = a a'ahn-- + a a')g,K
(10)
II following discussion, we not assume the tying W, and Wz. This breaking of the symmetry the memory matrix violates the assumption of monotonically decreasing energy function in the mathematical discussion of (31). Interpreting the energy function of MHA in asymmetric settings is a very interesting theoretical challenge for future research.

---

Thus, if the dynamics of the hidden state in the MHN is maintained and mapped to the self- eeffaattenioon layer, a new variable h. determined by the value of the attention scores, is added to the self-attention layer. This variable continues to accumulate the value of the attention score in each layer in the form of an exponential moving average across layers. Through this variable, the attention weights of each layer of the Transformer will have coordinated behavior. In the following, we will investigate the effect of adding this hidden state on the attention layer from the Transformer's perspective. this paper, this extended attention mechanism with hidden states will be referred to Modern Hopfield Attention (MHA), Compared to the cost O(dT²) of computing the dot product of self-attention, the computational complexity added by updating the hidden state is about O For Transformer that uses more than several hundred dimensions of d, this is a small increase in computational complexity.
4 Empirical Results
In this chapter, we experimentally investigate how the performance of the model changes when MHA is actually used in place of Transformer's 's self-attention module. We take the Vision Transformer (ViT) as representative example of an encoder Transformer model and the 2PT-2239) architecture as a representative example of a decoder Transformer model, and confirm that MHA does indeed lead to systematic performance improvements in several experiments.
4.1 Architecture with MHA
In the following, we will focus on the simplest case a = =d. It is straightforward to choose both parameter independently, but consider only this case to reduce the hyperparameters. By using our update rule (TO) instead of the attention layer, a new tensor called the hidden state H; propagates across the layers. This tensor accumulates the attention score K,K7T in each layer in the form of an exponential moving average. It is not the original attention score that gives the attention weight, but the softmax of the hidden state. At the same time, a skip connection with the weight (1 - a) linked to the coefficient of the exponential moving average of the hidden state is added according to equation (I0), and the balance between the two effects, controlled by a, is considered to determine the behavior of the MHA. The detailed structure of the architecture corresponding to (T0) is illustrated in Figure Ia).
In the following experiments, we will employ scaled dot-product attention according to the usual in the argument of the softmax function.
Transformer design and introduce the coefficient vdz
4.2 Text Generation: GPT-2
To determine the impact of MHA on Transformer performance, we first trained GPT-2 Small (124M) and Medium(350M) 39 on text generation task and tested their performance. The dataset used was WikiText103 [33]. The following experiments in this paper were conducted using up to eight A100 GPUs. The detailed training settings are described in the supplemental material.
To fairly compare the effectiveness of MHA, we trained the GPT-2 architecture and an architecture in which the self-attention layers of GPT-2 are replaced by MHA in the same setting from scratch and compared their perplexity. Table shows the results. The interest of this paper is not to create a SOTA model with detailed hyperparameter tuning, etc., but to the robustness of the MHA effect, so a was simply set to 0.5 based on rough hyperparameter search.
As Table [ shows, there is a clear improvement in perplexity in both the Small and Medium MHA models. Hopfield networks have often been experimented with in comparison to encoder Transformers 40 but our result shows that such comparisons is also useful for decoders.
Table I: Comparison of the perplexity of GPT-2 and its MHA counterpart trained on the WikiText103 dataset for two cases: GPT-2 Small with 124M parameters and GPT-2 Medium with 350M parameters. In both cases, the introduction of MHA improved the perplexity.
Small( 1 12MM
Medium(350M)
self- attention MHA(a = 0.5)
self-attention
MHA(a = 0.5)
22.87
20.70
20.85
19.61
5

---

4.3 Text Generation: LLaMA Architecture
To evaluate the effectiveness of Modern Hopfield Attention in more practical text generation architec- tures, conducted additional experiments on LLaMA, in addition to GPT-2, using the miniLLaMA implementation. Furthermore, besides WikiText-
we individually examined cases where CNN DailyMail [18] and BookCorpus [56] were used as training datasets. The results summarized in Table 2 Even in practical architectures such as LLaMA, whose refined design aims to enhance performance, MHA was found to exert consistent improvement in perplexity, demonstrating its systematic effectiveness beyond simpler baseline models.
dataset
self-attention
MHA
WikiText- 103
14.49
14.29
DailyMail
19.36
18.97
BoocCorpus
23.76
23.50
Table 2: Comparison of the perplexity of LLaMA and its MHA counterpart (a = 0.5) trained on various datasets, In all cases, the introduction of MHA led improved perplexity.
4.4 Image Recognition: VIT
Next, the Vision Transformer (ViT) was employed as the Transformer decoder model, and again to fairly compare the effect of MHA, two architectures, the ViT architecture and the architecture in which the self-attention layers of ViT are replaced by MHA, were trained in the same configuration. We trained these models in image recognition tasks.
The model used in this study is VIT [12], and the data sets used are CIFARIO/CIFAR 100 29 and ImageNet- 1k 8
The detailed training setup is shown in the supplemental material.
model size
model type
CIFAR10
CIFAR100
ViT-Tiny(5.5M)
self-artention
93.265
73.080
MHA(o = 0.5)
93.015
72.030
MHA(( 0.7)
93.775
72.570
VIT-SmalI(22M)
self-attention
95.450
74.485
MHA(a 0.5)
95.335
75.420
MHA(o 0.7)
95.440
75.590
ViT-Base(86M)
self-attention
96.190
75.360
MHA(a 0.5)
96.175
76.215
MHA(a 0.7)
96.490
75.590
ViT-Large(303M)
self-attention
96.310
72.910
MHA(a 0.5)
96.500
75.775
MHA(a 0.7)
96.690
75.365
Table 3: Experimental results are shown for ViTs and their MHA counterparts. For simple tasks such
as CIFAR10. performance is close saturation and there no clear effect MHA. On the other hand, for CIFAR 100, the performance improvement due to MHA is clear for the larger model. This a common property of a = 0.5 and a = 0.7.
4.4.1 CIFAR10/100
First, as a simple case, we review the results for CIFARIO the left column the Table 3 for CIFAR10 10, the effect of MHA is not clearly visible, partly because the performance is basically close to saturation due to the ease of the task. However, it is interesting to note that the effect of MHA is starting appear in the Base and Large models, which have high learning capacity. In any CIFARIO is not sufficient task for the purpose of observing changes in Transformer performance with scratch training.
So let's 'ss look the results for CIFARIO0 where the task is more difficult: as shown in Table 3 the larger the model, the larger and clearer the improvement compared to the baseline VIT. Interestingly,

---

in both cases of the two choices shown here, the performance improvement relative to VIT can be seen when the model larger than the Small model.
4.4.2 ImageNet-Ik
In the experiments on ImageNet-Ik, due to computational resource constraints, we adopt ViT-B (86M) as model of good enough size to obtain nontrivial training results, The results of 300-epoch training of ViT-B and its MHA counterpart from scratch with ImageNet-Ik are shown in Table Following the standard training setup, AdamW W [32) was used for optimizer and cosine decay for learning rate scheduling. Random erasing(54) mixup[52), cutmix and RandomAugmenn were used for augmentation. For details, please refer to the supplemental material.
Table 4: Classification validation accuracies for VīT-B(86M) and its MHA counterpart.
Data set
sell-attention
MHA(a = 0.5)
MHA(a = 0.7)
ImageNet-Ik
76.074
76.434
77.058
As shown in T the performance improvement in ViT-B was also observed in ImageNet- k.. Although the performance improvement is less than 1%, this performance difference considered a non-trivial result compared to the examples in previous studies ViT improvement. As in previous experiments, this increase in performance is produced by adding only a small amount computation without adding any training parameters. This is interesting result, which suggests that hidden states may help improve attention mechanisms. In the next chapter, we will investigate both theoretically and experimentally regarding how hidden states produce these performance gains.
4.4.3 Downstream Tasks
To evaluate MHA's effectiveness across diverse tasks, measured the transfer performance of pre-trained ImageNet model using linear probing. Using a pre-trained ViT and its MHA counterpart backbones, we conducted transfer learning experiments on four commonly used downstream datasets (Oxford Flowers 102 35), Food-101 [4. Stanford Dogs [27]. and Stanford Cars (28).
Results in Table show that the MHA variants achieve consistently good transfer performance.
dataset
self -elf-attention
MHA
Flower102
81.15
93.85
Food101
74.51
87.99
Stanford dogs
95.00
83.64
Stanford cars
51.54
87.54
Table Comparison transfer accuracy of ImageNet-1K pre-trained ViT and its MHA counter- part (a 0.7) on various downstream datasets.
4.5 Effect of Combining a and a'
Our update rule (10) has two hyperparameters and a', but for simplicity, we have far restricted our discussion to the case where both values are equal o = a'. However, as shown in Figure aa), these two quantities essentially work differently. is quantity that balances the value after attention computation and the strength of the skip connections the attention module. On the other hand, o' the coefficient of the exponential moving average in accumulating the attention scores hidden states.
A nontrivial result (T0) derived from MCHN is that these two independent effects simultancously added to the Transformer. To see whether these two are really both necessary, or whether they work in concert, let try experiment in which and a' are varied independently.
Table 6 shows the change in performance when one of the hyperparameters fixed at 0.5 and the value of the other is varied. When only a is moved from the original performance drops to chance-level accuracy. This is evident from the fact that all values except for the skip connection set to O. On the other hand, when is set to 0, the performance degrades to
0.5 = a' to 1, the
7

---

Table 6: Performance change of ViT-T when two hyperparameters are changed independently
CAIFARIO0 (H
0.5)
a'
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Score
71,16
71.13
72.29
70.77
72.12
72.13
72.06
72.02
71.98
70.72
66.10
CAIFARI00 MIlA(aa
0.5)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
score
69.89
71.20
71.26
71.64
72.02
72.13
72.66
70.52
70.46
67.70
1.00
69.89. Thus, can be seen that further performance improvement realized by adding not only o' but also a. Similarly, when a is fixed, setting a' to 0 also results in poorer performance.
How MHA improves Transformers
5.1 Problem and Improvement of Transformer Layers
Next, let us examine why MHA leads to performance gains in various tasks. It is known that as the depth of Transformer increases, training becomes more difficult and performance tends to saturate rapidly. The phenomenon of rank collapse has been discussed as one cause of this problem. other means.
is possible that our model mitigates the problem without explicit regularization Therefore, we provide below some theoretical and empirical results that support this hypothesis.
5.1.1 Rank Collapse
It has been observed that the depth of the Vision Transformer increases, the patch tokens become extremely similar and rapidly lose diversity [55). This phenomenon is now understood as
token uniformity, rank collapse, or oversmoothing [ 55
50
36
43 15
14 37 47 2 13
Various innovations have been proposed to reduce this problematic phenomenon in order to improve
Transformer performance [44 55 21.
phenomenon in which Transformer feature rapidly collapses into a rank I matrix with increasing depth. Thus, for the feature (L) of the L-th layer, rank collapse is formulated as the property that the residual of deep Transformer feature rapidly converges to zero as follows
Rank collapse [1] defined
Res( X (1) = 0or L 1,
(11)
where Res( X is the residual Res(X) X - 1x for arg min, X - 1x T.. This convergence means the feature is approximately a rank matrix X xs 1x
5.2 Theoretical Implication
For clear theoretical analysis of the causes of rank collapse, we consider a deep network consisting of only the self-attention layers according to []. It is also straightforward to extend the discussion the actual Transformer architecture [1]. Let us consider a self-attention-one only network consisting of L layers without skip connection
AttnNet(X)) = MHSA o ... MHSA(X).
(12)
MHSA(X) (X) is the multi-head self-attention module. The number of heads and embedding dimension of each MHSA are H and dg. In [1I, this attention-only network has been shown to cause very serious rank collapse:
Theorem 5.1 ([II). The norm of the residual of attention-only network (ttNNer(X) X) decays as
|Res (AutNeet( X)) ||,,0 (rC) 3/
||ees (X) 1
(13)
where r
8H and C is certain constant. This suggests the double exponential decay of the rank. Vag
"The rank collapse [ refers to the phenomenon where the tokens corresponding each row feature become perfectly proportional vectors. This means perfect token uniformity. On the other hand, the phenomenon observed in actual Transformers is that many, if not all, tokens are perfectly aligned, forming a group of tokens with mutual cosine similarity of I.

---

11 in this paper is the composite of operator norms The definition of the norm V|X oc.
||
In [T. it was shown that skip connection and the addition of an FFN layer are effective in reducing this serious collapse. Interestingly. however, even though our MHA is not specifically designed to prevent rank collapse, it is able to prevent the decay phenomenon attention-only networks without any skip connection. Even when removing skip connections completely by setting = 0, non-zero a' leads to the following mitigation of rank collapse the attention-only network:
Theorem 5.2. By keeping non-zero o', the upper-bound of inequality evaluation is improved as follows
[ (X) 11,x max m (r(1 (r(1-a')C)) (ra' (ra'C2)""(L"1 ||ees X) 11,eo. (14)
This suggests the avoidance of exponential decay.
Proof. See the supplemental material for detailed proof. The sketch of the proof is as follows: by introducing the hidden state as a' o O, the decaying effect of rank by single attention layer can be evaluated as follows
Res [ReSAMM(A(X)]] & max (r1(1 a') RRes xx 18,
rea' Res (x)Il)
(15)
where 11,2 = rC1,2 2 and the norm here || |l Notice that the second argument the max function significantly reduces the third-order decaying effect in [Л]. By applying this inequality repeatedly over L layers, we obtain the following inequality
|Res (AttmNet(t)))) max (r1(1-a')) (1-1-a)) a()) - |Res X) 11 .. (rza')" ||Res ((ee((X)) (x)11) (16)
where ... means (ri - -0)))(0-111// Th (L- m) (1 (||sss( (x))) for m 1,
... L 1.
On the right hand side of this inequality (14, the L term is the very term that created the double exponential decay of the original self-attention mechanism [Л], but the m = 0 term dominates in 144 and relaxes the rank decay to linear decay as (ra'C2)* ||Res( (X) [|1,00 since
(1 - a0)) (01))(33-1)/((r2o) (32a)) (1--m) ( x) (F2ee))(((
(17)
Note that we assume 1,,2, [[Re((X]] < following the logic of [1]] This decaying factor is controlled by the hidden states of the h-th head of the E-th layer Hi, = a'
M...K KT
and the weight matrix wr(() for the value and output linear projection of attention module as C2 max, max; || Vo.all1.s He,,
In [1] such an effect was created by introducing skip connection, but in the MHA, the hidden state contribution already produces such an effect without using skip connection. Also, setting a' 0 reproduces the double exponential decay results of the original attention-only network (3
5.3 Empirical Results
Using the theoretical analysis setup used in previous studies, we showed that MHA can effectively prevent rank collapse in the previous section. However, since these setups are based on several theoretical simplifications, it is unclear whether the rank collapse reduction also occurs in actual Transformers. In particular, it is not clear whether the introduction of MHA has any further effect in usual architectures with skip connection to reduce rank collapse. In this section, we will confirm that MHA does indeed further reduce rank collapse in a few controlled experiments.
Since the skip-fre network was shown to suffer from rank collapse as it gets deeper, let's examine the effect of MHA on the actual performance degradation with depth. Table (showss the results of trained models from depths to 12 for the skip-free networks and their MHA versions, and evaluating their performance. As can be seen from the results in the left column of the table, when the depth increases beyond layers, the performance drops sharply due to multilayering. On the other hand, for the models in the right column using MHA, it can be seen that the degradation of the model due to multilayering is kept at fairly mild level. Thus, the MHA model can effectively utilize the depth of

---

Table 7: Changes in performance as skip-free networks based on ViT-T are deepened.
self-attention
MHA
depth
CIFARIO
CIFAR100
CIFARIO
CIFAR100
55.08
30.90
65.41
40.08
2
63.72
40.06
79.75 1
56.94 f
57.38
32.25
85.74 1
64.39 f
8
48.59
17.19
80.34
49.90 +
12
10.00
1.00 1
10.00 4
1.00 4
meday BU
HAS 0OD
Hedia's A13
Hed
Hedd LuS
Pack 20
mare 2:20
XXX
-03 55
B3
Made 20
Malh 6:39
Male 3:30
Mask
Meay ALL
Mean
-AA
33
Modaa
GAR
(a)
(b)
Figure The violin plots cosine similarity between tokens in several layers for (a) GPT-2 (Medium) trained Wikitext103 and (b) ViT-B trained CIFAR 100. MHA layers with high average similarity of tokens exist, but tokens with a perfect similarity of 1, as the case of self-attention, disappear, preventing their ranks from dropping.
model than the original model. Therefore, highly likely that MHA can significantly improve rank collapse, which becomes more severe the network becomes deeper, even in real networks.
Next, let examine cases of actual Transformer architectures with skip connections and FFN layers. Figure 2 shows the measured cosine similarity between tokens in several layers for GPT-2 (Medium) and ViT-B trained on CIFAR 100, displayed as violin plots. See the supplementary material for more detailed plots. It is noteworthy that in the cases of normal GPT-2 and ViT-B, the mode of similarity is 1.0 for all layers, while the violin plot shows sharp peak around 1.0. This indicates that even with the addition of the skip connection and FFN layers, there is still non-negligible token uniformity, or partial rank collapse. On the other hand, the results for GPT-2 and ViT models with MHA show that the peaks in the original models have disappeared and the mode values have been reduced to very small values. This indicates that MHA does indeed play a role in dramatically removing token uniformity in GPT-2 and ViT.
6 Conclusion
In this paper, we examine the question of whether new insights can be obtained from the modern Hopfield network for Transformer. The results showed that by introducing the hidden state of MCHN into Transformer, new attention mechanism called MHA, which inherits attention scores from layer layer, has been discovered and can be useful for improving ViT and GPT performance. MHA was also found to play role in solving the rank collapse problem in deep Transformer. The MHA's mechanism to prevent the rank collapse may have contributed Transformer's 's improved performance. We hope that this research will open new possibilities for the systematic design of Transformer architectures using Hopfield networks.
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract reflects the results of Sections 5 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The Supplementary Material describes the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
The Supplementary Material describes proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Codes and settings included in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
12

---

Answer: [Yes]
Justification: The code and other materials included in the supplemental material will be released on GitHub after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ howee//oooSeSuoooossooooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.c//pil wwcppbblc/guuiicuccccccccoe.oopp Detgguddsse/CeSbbbiiiinn htt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: settings and details are included in the Supplementary Material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
Due to the high cost training Transformer models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
13

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational resources are described 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p///..aarr....ooo/oo//////a
Answer: [Yes]
Justification:
The study was conducted in compliance with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This study is a basic study of the Hopfield network and Transformer, and therefore does not directly have a broader impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
14

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Since this is basic research, such risks are minimal.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
A link to the referenced code is posted at the top of the code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
15

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
16

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy htttes:/heerrpp..cc/ooffoorreeee for what should or should not be described.
References
[1] S-I Amari. Learning patterns and pattern sequences by self- self-organizing nets of threshold elements. IEEE Transactions on computers, 100(11):1197-1200 1972.
[2] Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu, Improving vision transformers by revisiting high-frequency components. In European conference on computer vision, pages 1-18. Springer, 2022.
[3] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36:1560-1588, 2023.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. European conference on computer vision, pages 446-461. Springer, 2014.
[5] Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations.
[6] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 702-703, 2020.
[7] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288 2999 2017.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[9] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, 2019.
[10] Anxhelo Diko, Danilo Avola, Marco Cascio, and Luigi Cinque. Revit: Enhancing vision trans- formers feature diversity with attention residual connections. Pattern Recognition, 156: 10853, 2024.
[11] Yihe Dong, Jean- -tan-Btptistt Cordonnier, and Andreas Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning, pages 2793-2880 PMLR, 2021.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
17

---

[13] Gbétondji IS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04300 04300, 2024.
[14] Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: contrastive learning perspective on oversmoothing and beyond. arXiv preprint arXiv:2303.06550 2023.
[15] Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel Smith, and Yee Whye Tch. Deep transformers without shorteuts: Modifying self-attention for faithful signal propagation. arXiv preprint vrXiv:2302.10322 2023.
[16] Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. In Findings of the Association for Computational Linguistics: ACL.- IJCNLP 2021, pages 929-943, 2021.
[17] Zexue He, Leonid Karlinsky, Donghyun Kim, Julian McAuley, Dmitry Krotov, and Rogerio Feris. Camelot: Towards large language models with training- free consolidated associative memory. In First Workshop on Long-Contex, Foundation Models@ ICML 2024.
[18] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
[19] Claus Hofmann, Simon Lucas Schmid, Bernhard Lehner, Daniel Klotz, and Sepp Hochreiter. Energy- based hopfield boosting for dittrf--istriiuution detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[20] Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov, Energy transformer. Advances in neural informa- tion processing systems, 36:27532-27559, 2023.
[21] Benjamin Hoover, Hendrik Strobelt, Dmitry Krotov, Judy Hoffman, Zsolt Kira, and Polo Chau. Memory in plain sight: A survey of the uncanny resemblances between diffusion models and associative memories. In Annual Conference on Neural Information Processing Systems, 2023.
[22] John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences,
1982.
[23] Jerry Yao-Chich Hu, Pei- Pei-Hsuan Chang, Haozheng Luo, Hong-Yt Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu. Outlier-efficient hopfield layers for large transformer- models. In International Conference on Machine Learning, pages 19123-19152. PMLR, 2024.
[24] Jerry Yao-Chieh Hu, Thomas Lin, Zhao Song, and Han Liu. On computational limits of modern hopfield models: a fine-grat fine-grained complexity analysis. In Proceedings of the 41st International Conference on Machine Learning, pages 19327-19343, 2024.
[25] Jerry Yao-Chich Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo- Yu Chen, and Han Liu, On sparse modern hopfield model. Advances in Neural Information Processing Systems, 36:27594- 27608, 2023.
[26] Georgios latropoulos, Johanni Brea, and Wulfram Gerstner. Kernl memory networks: A unifying framework for memory modeling, Advances in neural information processing systems, 35:35326-35338, 2022.
[27] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on grained visual categorization (FGVC), volume 2, 2011.
[28] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[29] Alex Krizhevsky, Learning multiple layers features from tiny images. Technical report, 2009.
[30] Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition, Advances in neural information processing systems, 29, 2016.
18

---

[31] Dmitry Krotov and John J Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations. 2021.
[32] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations.
[33] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[34] Beren Millidge, Tommaso Salvatori, Yuhang Song. Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pages 15561-15583. PMLR, 2022.
[35] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
[36] Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211, 2022.
[37] Namuk Park and Songkuk Kim. arXiv:2202.06709, 2022.
How do vision transformers work? arXiv preprint
[38] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 2019.
[40] Hubert Ramsauer. Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, et al. Hopfield networks is all you need, In International Conference on Learning Representations, 2021.
[41] Bishwajit Saha, Dmitry Krotov, Mohammed J Zaki, and Parikshit Ram. End-to-end differen- tiable clustering with associative memories. In International Conference on Machine Learning. pages 29649-29670. PMLR, 2023.
[42] Tommaso Salvatori, Yuhang Song, Yujian Hong, Lei Sha, Simon Frieder, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. Associative memories via predictive coding, Advances in neural information processing systems, 34:3874-3886, 2021.
[43] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T' Kwok. Revisiting overssmoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08625, 2022.
[44] Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Aug- mented shortcuts for vision transformers. Advances in Neural Information Processing Systems, 34:15316-15327, 2021.
[45] Danil Tyulmankov, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang. Biological learning in key-value memory networks. Advances in Neural Information Processing Systems, 34:22247-22258. 2021.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[47] Peihao Wang, Wenging Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. arXiv preprint arXiv:2203.05966. 2022.
[48] James CR Whittington, Joseph Warren, and Tim EJ Behrens. Relating transformers to models and neural representations of the hippocampal formation. In International Conference on Learning Representations.
19

---

[49] Dennis Wu, Jerry Yao-Chieh Hu, Teng- Yun Hsiao, and Han Liu. Uniform memory retrieval with larger capacity for modern hopfield models. In Proceedings of the 41 International Conference Machine Learning, pages 13377-53311 2024.
[50] Hanqi Yan, Lin Gui, Wenjie Li, and Yulan He. Addressing token uniformity in transformers via singular value transformation. In Uncertainty in artificial intelligence, pages 2181-2191. PMLR, 2022.
[51] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[52] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
[53] Jinsong Zhang. Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, et al. -utto--disttibution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations, 2022.
[54] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[55] Daquan Zhou, Bingyi Kang. Xiaojie Jin, Linjie Yang. Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886, 2021.
[56) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
20


Reviews:
[{"review_id": "x3ktsxupGc", "reviewer": "Reviewer_7EBm", "summary": "This paper introduces a new type of self-attention layer in transformers called Modern Hopfield attention (MHA). Previous work had also noted the relation between Hopfield networks and transformers and  used this relation to develop Hopfield layers in transformers where the Hopfield network served as the memory in the layer.  Earlier it was shown that the state update rule of Modern Hopfield networks in the adiabatic approximation corresponded to self-attention in transformers. This result generalizes beyond the adiabatic approximation to show that the relationship still holds. Specifically, they use MHA to allow the inheritance of attention scores across the layers. The computational cost increase is modest compared to the cost of the computation of full attention in layers. \nThe key result is in the update rule of Equation (10) where instead of the attention layer, a hidden state propagates across the layer accumulating the attention score in each layer as an exponential moving average. The softmax of the hidden state gives the attention score.  Two parameters a and (1-a) control the behavior of the MHA.", "questions": "Since there is a recurrence formulation being implied with the hidden states, it would be good to see a discussion of this approach over the newer state-space models that are offering much more computational advantage. In other words, while the theoretical result is interesting, is it still relevant in the age of state-space models?\n\n\nIt would also be good to see a discussion comparing to the Hopfield layers proposed earlier in reference[35].", "limitations": "Since there is no computational advantage of using the MHA layer and at best a 1% increase in performance in many cases, does the result have any practical implication? This would be good to point out in the discussion. \nThe related work section is bare-boned. An extensive discussion particularly w.r.t to other Hopfield layer formulations would be good to add to the paper. The paper itself did not point to any limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The derivations and the observations made between Modern Hopfield network and transformers are interesting. The paper is largely theoretical in nature and includes a computational complexity analysis that shows that there is a modest cost increase which is a function of the input sequence over and above the attention computation cost. The main novelty is in the modification of the attention layer as specified in Equation 10.  \n\nThe contribution is an elegant theoretical contribution in which the attention weights are accumulated through the hidden Hopfield states in a sense combining recurrent neural networks within the architecture of transformers.  The impact of MHA on transformer performance was analyzed in downstream use cases of text generation and image recognition and in both cases, improvements of over 1% were seen, particularly for larger categories recognition.", "was_revised": true, "final_justification": "I have already indicated acceptance of the paper due to the reasons above mentioned in my original review.", "rebuttal": "We sincerely appreciate your deep understanding of this research, your evaluation of its significance, and your insightful comments from a broad perspective.\n \n**[Question] Relationship with modern architectures (such as state-space models)**\n\nAs you pointed out, the current trend in Transformer research is shifting toward linear structures such as linear memory models and state-space models (SSM). Even within this trend, we would like to explain the significance of this research.\n\nFirst, although not detailed in this paper, the proposed MHA method is also applicable to kernelized attention (linear attention). Linear attention is a framework that integrates RNN-like sequential computation with Transformer parallelism, and by adding gate mechanisms or masks, it can evolve into SSMs such as Mamba, RWKV, and RetNet. By integrating the state accumulation mechanism of MHA into linear attention, we believe it is possible to design SSMs that are both more performant and theoretically consistent.  \n\nFurthermore, mapping the associative mechanism of the Hopfield Network to the Transformer corresponds to the repeated application of weight-shared attention layers. This is deeply related to structures such as the Universal Transformer (Dehghani et al., ICLR 2019) and the recent Looped Transformer (Yang et al., ICLR 2024). The “depth-wise recurrence” introduced in this study through MHA has an essential connection with such prior research.  \n\nGoing forward, we believe that hybrid models integrating layer recurrence via MHA and sequence-wise recurrence in SSM represent a promising direction for next-generation language models. Thank you for your valuable feedback.\n\n**[Question] Comparison with Ramsauer et al. [35]**\n\nI would like to add some additional information regarding the comparison with Ramsauer et al. [35]. In [35], the authors demonstrate the fast convergence of the Modern Hopfield Network (MHN) and justify its use as a non-recurrent layer in neural networks. On the other hand, in this study's MHA, we propose a dynamic structure that maintains and updates hidden states across layers.\n\nMore specifically, the HopfieldPoolLayer and HopfieldLayer in [35] are formally similar to self-attention and do not have recursive properties, as they use the MHN output as a single output. In contrast, our MHA naturally incorporates Hopfield recursion into the Transformer layer structure, as “state accumulation and updating” are performed in each layer.  \n\nWe will clearly describe this difference in the camera-ready version as well. \n\n**[Limitations] Practical Significance and Applicability**\n\nAs you pointed out, it is important to ask how much performance improvement the MHA proposed in this study will bring in practical terms. To clarify this point, additional experimental results in natural language tasks are shown below.\n\n**Additional results in Llama (Perplexity)**\n| Dataset |  Self-attention | MHA(\\alpha=0.5) |\n| ---- | ---- | ---- |\n| Wikitext103(20%)| 29.92| **24.48** |\n| BookCorpus(20%)| 67.05 | **40.68** |\n| CNN_dailymail(20%)| 78.81| **58.62** |\n| xsum(20%)| 103.30| **97.91** |\n\nThese results show that MHA brings about significant improvements in perplexity not only for standard GPT-2 but also for practical Llama-based architectures. Moreover, these improvements were achieved without additional learning parameters, which we believe to be highly practical.\n\nFurthermore, since MHA can be easily integrated into many Transformer-based model, it has great potential to contribute to SOTA improvements in various tasks and models in the future.\n\n**[Related Works] Expansion of Related Research**\n\nThank you for your valuable comments. In addition to [35], we will add a summary and positioning of related research, including further developments of Hopfield-based layers (e.g., [39]), to the camera-ready version. In particular, we plan to discuss this topic in greater detail from the perspective of the relationship between the dynamic state of Hopfield-type models and Transformers."}, {"review_id": "fbtdT7v9lv", "reviewer": "Reviewer_aW1E", "summary": "The paper relooks at the connection between Modern Continuous Hopfield Networks and the Attention Mechanism, as in some relevant prior work (mentioned in the paper). A traditional Hopfield Network consists of two layers, the feature neurons and the hidden neurons (denoted by $v$, $h$ respectively). In that context, it identifies a common assumption made in most contemporary work of discretizing the evolution of the dynamic system under the adiabatic limit and chooses NOT to assume this. Not making this assumption leads to an interesting modified attention style update rule, that performs the softmax operation on the feature neurons, which is in turn updated via a recurrence equation involving its own previous state. Thus a skip connection is implicitly introduced in the proposed attention mechanism. This new rule is rigorously derived from the dynamics of MCHN and tested on a few standard models cum datasets, in both vision and language. Finally, the impactful utility of this attention scheme is demonstrated in the context of token uniformity/rank collapse, a common problem in vanilla Transformers, both theoretically and empirically.", "questions": "1. Are rank collapse and token uniformity different names for the same problem? Or are they different problems that always occur together?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n1. The insight of NOT taking the adiabatic limit of the energy dynamics, then deriving an update rule based on the chosen Lagrangians, and then formulating a new attention mechanism is novel, and particularly so in the context of its connection to Hopfield Networks. \n2a. This formulation involving hidden state recurrence is rigorously derived from the energy dynamics, and not just a variant of mechanisms that combine RNNs with Attention via some heuristics. \n2b. Appendix D provides a case by case analysis for different choices of the discretization step which seems novel, interesting and potentially useful for future work as well. \n3. The utility of this problem in handling a particular issue in training vanilla Transformers known as rank collapse is identified via some experiments. \n4. A theoretical result is also suggested (Theorem 4.2) which suggests how the proposed attention mechanism, MHA, beats vanilla attention at handling rank collapse.\n5. Overall, the presentation makes the paper easy to follow.\n\nWeaknesses:\n\nExperiments:\nThe experiments are overall quite weak to build confidence in the efficacy of the proposed methodology. \n\n1. Especially for language, testing on one dataset alone is insufficient. I strongly suggest the authors to try on atleast 3 more moderate-sized language datasets commonly used in contemporary research such as OpenWebText, MMLU, GLUE, BookCorpus, etc. Based on other statements in the paper, I understand that compute is concern, but please try to choose datasets wisely such that it is not too compute-hungry while at the same time, the point is properly conveyed.\n\n2. Same goes with the language models, currently only GPT-2 small and medium have been experimented. Please experiment with at least 2 more language models, e.g. amongst Llama, Gemma, GPT-3, Mistral, RoBERTa, or anything else convenient but authentic.\n\n3. For images too, it is a similar story. Though three datasets have been experimented with, the improvement in case of CIFAR-10 and ImageNet-10K appears questionable. Further the claim made in lines 185-187 without a proper reference is hard to believe, and it better be backed by some reference or some experiments if that is true. Especially given that the results are not upto the mark for CIFAR-10 and ImageNet-10K, I suggest trying on at least two more datasets, such as TinyImageNet or any other relevant, well-known dataset and showing strong results on them.\n\n4. I do not see any ablations/discussion on the initializations used in training the Transformers, for language and vision alike. This may be important in assessing the efficacy of MHA, as it may affect the results either way. Please try different initializations, including pretrained initializations for both vision and language (for instance, for ViTs, try LION initialization, CLIP based, Xavier-Glorot, etc.) \n\nTechnical:\n\n1. Lines 246 and 254-256 suggest that a doubly-exponential decay is avoided because the case $m=0$ appears in one of the arguments to the max function. However I don't think that directly holds, because it is very well possible that the other terms with $m \\neq 0$ dominate the overall max function and thus becomes doubly-exponential. \n\n2a. Even if $||Res(AttnNet(X))||$ decreases linearly as Theorem 4.2 suggests, it is not necessary that it happens because of MHA. Because ultimately, both Theorems 4.1 (which is not the authors contribution, as far as I understand) and Theorem 4.2 (which is a contribution from the authors, if I understand correctly) provide upper bounds, so it seems possible that even without MHA, $||Res(AttnNet(X))||$ was lower than the (potentially stronger) upper bound that Theorem 4.2 provides, which would make the theorem redundant in that case. \n\n2b. Ideally, to show the efficacy of MHA at avoiding rank collapse, there should be at least one example shown where under standard training of vanilla Transformers with vanilla attention, $||Res(AttnNet(X))||$ is lower bounded by some value, say $\\kappa$, and by using MHA for that case, $||Res(AttnNet(X))||$ can be brought down below $\\kappa$. The more examples this phenomena holds for, the better the efficacy of MHA. \n\n3. Equation 16 in the Appendix looks wrong -- in particular I don't see how the second term in the LHS appears, and also how the terms represented by '...' are constant terms. Please fix this, as this seems to be a crucial equation for the rest of the proof to go through.\n\nPresentation:\n1. Please rigorously define the exact expression for token similarity, as described in lines 280-282. Please also more clearly explain what exactly the violin plots are plotting, as it not is very clear in the current writeup. Also mention how many tokens are used in describing a single such plot.\n\n2. Please move lines 181-183 of the Appendix or some equivalent of it into the main text, as this more clearly describes what token uniformity is, particularly in the context of rank collapse. I don't see that clear explanation given elsewhere in the main text.\n\nMinor:\n\n1. In line 14 of the abstract, please change \"problem\" to \"problem$\\textbf{s}$\".\n2. In line 40 of the main paper, please change \"lack\" to \"lack$\\textbf{s}$\".\n3. Please replace the 3 places in the main text where \"chapter\" is used with \"section\"/\"subsection\" or some more appropriate word like that.\n4. Please use better notation for the max function in the statement for Theorem 4.2.\n\nP.S: I look forward to increasing my score, especially if concerns stated in the Experiments section, first two concerns in the Technical & Presentation sections are addressed. I would thus advise the authors to prioritize addressing the concerns in that order.", "was_revised": true, "final_justification": "The paper presents an interesting modification to self-attention based on a commonly ignored observation in energy based memory models, shows proof-of-concept level experiments and moderate theory to justify the contribution. While the work is probably still not full ripe in this form, I think it has good potential in general. The authors also did a great job clearing practically all my concerns. Thus, I hope to see this paper accepted at NeurIPS.", "rebuttal": "We sincerely appreciate your valuable review and helpful comments. We are particularly grateful for your positive evaluation of the novelty and theoretical contributions of this study.\n\n**[Experiment: W1, W2]**\n\nAs you are probably already well aware, the main purpose of this paper is not to propose a new technical method, but to understand the role of hidden states in Hopfield Networks from the perspective of Transformers. In that context, we conducted empirical verification through performance changes in language and image tasks and mitigation of rank collapse to demonstrate that the corresponding attention mechanism (MHA) has a meaningful structure.\n\nHowever, we fully agree that the reliability and comprehensiveness of the experiments are important issues. Therefore, in response to your comments, we conducted additional experiments within the constraints of computational resources, including a more realistic architecture (Llama) and language datasets from domains other than Wikitext103. The data used were selected from standard datasets available on Hugging Face Datasets, such as BookCorpus, cnn_dailymail, and xsum. These datasets, which have characteristics different from Wikitext103, such as books and news articles, are suitable for verifying language domain generalization. The following table shows the comparison results of the validation perplexity of Llama trained on each dataset. We used the mini-llama codebase as a reliable and convenient implementation. In the following experiments, unless otherwise specified, MHA is set to $\\alpha=0.5$ for all cases.\n\n**Wikitext103(20%) step=40k**\n| Self-attention(Llama)  | MHA(Llama)  |\n| ---- | ---- |\n| 29.92| **24.48** | \n\n**BookCorpus(20%) step=6k**\n| Self-attention(Llama)  | MHA(Llama) |\n| ---- | ---- |\n| 67.05 | **40.68** | \n\n**CNN_dailymail(20%) step=6k**\n| Self-attention(Llama)  | MHA(Llama) |\n| ---- | ---- |\n| 78.81| **58.62** |\n\n**xsum(20%) step=20k**\n| Self-attention(Llama)  | MHA(Llama)  |\n| ---- | ---- |\n| 103.30| **97.91** |\n\n**Training dataset included with the mini-llama implementation**\n| Self-attention(Llama)  | MHA(Llama)  |\n| ---- | ---- |\n| 1.292| **1.28152** |\n\nThese results demonstrate that MHA is effective not only for simple GPT series but also for the latest architectures such as Llama, and that systematic performance improvements can be observed across domains. Thus, we believe that our claims are not limited to results specific to a single model or dataset, but have broader validity.\n\n**[Experiment: W3]**\n\nWe apologize for failing to adequately convey the significance of the results due to the lack of references supplementing our understanding that a 1% improvement in accuracy on ImageNet-1k is non-trivial.\n\nFirst, similar to our research, methods that correct attention maps, such as DeepViT [50], have reported performance improvements of approximately 0.3% for models equivalent to ViT-B ([50], Figure 1, 12-layer). Additionally, the improvement margin for ViT-B in [39] is +0.6%. While both of these methods involve adding learning parameters, our MHA achieves the approximately 1% improvement shown below without adding any parameters. We consider this to be a technically significant result.\n\nFurthermore, improvements of over a few percent on ImageNet typically require distillation, large-scale pre-training, or significant design improvements. For example, the +4% improvement from ViT-B/16 to DeiT-B is due to the distillation effect described in [Touvron et al., ICML 2021]. Such results were obtained through design explicitly aimed at performance improvement, and cannot be directly compared with our results, which are based on minor modifications driven by theoretical motivations.\n\nFurthermore, to confirm the robustness of the performance improvement, we conducted additional ViT-B on ImageNet experiments with the same setting of $\\alpha=\\alpha'=0.7$ on ImageNet-1k as on CIFAR:\n\n| Self-attention | MHA($\\alpha=0.5$) | MHA($\\alpha=0.7$) |\n| ---- | ---- |---- |\n| 76.074 | 76.434 | **77.058** |\n\nAs shown above, the effect of performance improvement becomes even clearer with the appropriate selection of hyperparameters.\n\n**[Experiment: W4]**\n\nThank you very much for your keen observation regarding initialization. In this study, we used Pytorch's nn.Linear to calculate Q, K, and V, and adopted Kaiming-Uniform for initialization. On top of that, we conducted additional comparative experiments with Xavier-Glorot initialization.\n\n**Comparison of initialization: Cifar100, MHA(ViT-tiny), epoch300**\n| Xaiver-Glorot | Kaming-He|\n| ---- | ---- |\n| 68.34| 64.00 |\n\nThe improvement in performance achieved through Xavier initialization is intriguing, and there is potential for further performance gains through initialization. Such technical innovations represent an exciting direction for future research.\n\nFurthermore, initialization using pre-trained weights involves applying weights trained with different model (plain self-attention) to a new MHA, which may affect learning stability. Therefore, the development of various training techniques is necessary (e.g., $\\alpha$ warm-up). In this study, we focused on structural understanding based on theoretical motivation, so we refrained from conducting such experiments this time, but we consider this to be an important direction for future research.  \n\n**[Technical: W1]**\n\nAll terms other than $m=0$ in lines 246, 254-256 are also positive. Therefore, the right side of our inequality evaluation obtained by adding these terms is at least greater than the term $m=0$. Thus, even if the other terms decay exponentially, the term $m=0$ dominates, preventing exponential decay on the right side as a whole. Regarding this point, we will add a clear explanation in the camera-ready version.\n\n**[Technical: W2a, 2b]**\n\nThe theoretical contribution of this study is based on an “upper bound evaluation of rank collapse” in the context of previous studies ([10], [39], etc.). It is extremely difficult to provide a mathematical lower bound, and such an evaluation remains an unsolved problem at this point. Therefore, this study endeavored to complement the theoretical results with empirical evidence.\n\nFurthermore, previous studies have sought architectural innovations to improve the upper bound of the inequality. This approach suggests architectural structures that may mitigate double exponential decay based on theoretical insights (e.g., the effects of shortcut paths and MLP in [10], or modifications to shortcut paths in [39]). This paper also follows this logic.  \n\nFor example, Table 5 shows that rank collapse is mitigated by MHA in the same attention-only network as the theoretical setup. Furthermore, the cosine similarity distribution in Figure 2 shows a similar trend, providing evidence that bridges the gap between theory and actual networks.\n\nFurthermore, stable learning of the attention-only network itself is difficult (Table 5), and it is impossible to make a meaningful comparison of $\\|Res(AttnNet(x))\\|$ with an untrained deep model. Therefore, we supplement the validity with observations of token uniformity improvement not only in the attention-only net but also in the actual Transformer.\n\n**[Technical: W3]**\n\nThere is no error in Appendix (16). The second term you pointed out is the result of applying (18) to $1 b_{Q}^{(\\ell) T} W_{K}^{(\\ell) T} X^{(\\ell) T}$. Furthermore, the last item in Appendix (16) can be written explicitly as\n\n$$\n\\frac{1}{\\sqrt{d_k}}( X^{(\\ell)} W_Q^{(\\ell)} b_K^{(\\ell)} + 1 b_Q^{(\\ell) T} b_K^{(\\ell) })1^T\n$$\n\nThe reason why this last term can be ignored is that the row-wise softmax function is invariant to constant shifts in the logits, as shown by $e^{A_j+c}/\\sum_i e^{A_i+c}=e^{A_j}/\\sum_i e^{A_i}$ (see lines 48-50). In fact, the term described above takes the form of the outer-product of two vectors, $v 1^T$, and thus serves only to add a constant to each row of $A$.\n\n**[Presentation W1]**\n\nThe cosine similarity was calculated using the following procedure:\n\n1. Extract all tokens $x_n(n=1,…,N)$ in a specific layer.\n2. Calculate the cosine between the two vectors for all pairs $(x_n, x_m)$.\n3. Generate a violin plot from these cosine values collected from multiple samples.\n\nNumber of tokens used:\n\n- ViT: 256 tokens $\\times$ 32 samples = 8192 tokens\n-  GPT-2: 1024 tokens $\\times$ 12 samples = 12288 tokens\n\n**[Presentation: W2, Q]**\n\nAs you pointed out, we will move the explanation of token uniformity to the main text.\nRank collapse refers to “a phenomenon in which the rank of the representation matrix decreases significantly,” and narrowly defined token uniformity (all tokens becoming similar vectors) is one form of this. However, even if the rank is small, if there is variation in the direction between vectors (e.g., opposite directions), it cannot be called uniformity. In this study, to avoid confusion between the two, we explicitly observe uniformity using the cosine similarity distribution.\n\n**[Minor Issues]**\n\nAll the description and notation errors you pointed out will be corrected in the camera-ready version.\n\n**Thank you very much for your valuable comments.**"}, {"review_id": "B8s99XFgsp", "reviewer": "Reviewer_WaSH", "summary": "This paper proposes a new continuous modern Hopfield network architecture that introduces dynamics in the hidden state. The number of parameters is kept constant but the computational time is increased linearly as a function of the transformer dimension. The authors show this architecture prevents the rank collapse problem, which may be what modestly improves the ViT and GPT performance.", "questions": "Can the authors identify stronger improvements over baselines using their proposed principle? The improvements seem relatively modest compared to the sophistication of the theory but happy to be convinced otherwise.\n\nI think the figure quality could be significantly improved.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The derivation of the hidden state dynamics seems sound.The authors clearly describe previous and related work. X and Y labels in Figures 1(b) and 2 are missing. Would be important to have confidence intervals in the perplexity results throughout the paper, to check if the improvements in perplexity are statistically significant. \n\nThe paper is clear and well organized. \n\nAddressing the rank collapse problem is important and the theoretical link to MHN is intriguing via the consideration of the dynamics in the hidden state allows for an improvement of the rank collapse problem. However, it would be nice to actually show that the rank is not collapsing in addition to showing the activation of selected layers is more uniform compared to the self attention case. Ultimately, it seems that the performance improvements are modest.\n\t\nIs it clear how this work differs from previous contributions. The proposed algorithm improves slightly the performance compares to the previous state of the art. The paper provides new insights on how to use modern hopfield networks in transformer architectures.", "was_revised": true, "final_justification": "I think the author's rebuttal points are fair in how they contextualize their contribution. As I mention - I am sympathetic to their theoretic perspective (and I really like this paper conceptually) however the empirical results suggest modest benefits (that, ultimately, may not be statistically meaningful) and Im not sure that the authors' have demonstrated the core contribution they propose in terms of performance. Taken together, I think that my rating is appropriate.", "rebuttal": "We would like to express our sincere gratitude for your deep understanding of this paper and your numerous constructive and useful comments.\n\n**[Weakness] Confidence intervals for perplexity and figure labels**\n\nAt present, due to the computational cost of a single training run, we have not obtained sufficient statistics. By the camera-ready version, we will obtain statistical metrics (mean and standard deviation) based on multiple runs and clearly indicate their significance.\n\nOn the other hand, as an alternative to confidence intervals, we conducted the following additional experiments to demonstrate that perplexity improvements in language models can be systematically reproduced in other models and datasets. Using the mini-llama implementation, we evaluated the perplexity of Wikitext103, BookCorpus, CNN/DailyMail, and XSum (20% each), and found significant improvements in all cases.\n\nAdditional results in Llama (Perplexity)\n| Dataset |  Self-attention | MHA(\\alpha=0.5) |\n| ---- | ---- | ---- |\n| Wikitext103(20%)| 29.92| **24.48** |\n| BookCorpus(20%)| 67.05 | **40.68** |\n| CNN_dailymail(20%)| 78.81| **58.62** |\n| xsum(20%)| 103.30| **97.91** |\n\nThese provide strong evidence of reproducible improvements across architectural domains.\n\nWe also appreciate your comments regarding the lack of axis labels in Figures 1 and 2, as well as the overall quality of the figures. In the camera-ready version, we will replace the figures with high-resolution versions that clearly show the axis labels and legends.\n\n**[Weakness] Rank collapse and changes in activity**\n\nRank collapse refers to a decrease in the rank of the representation matrix and is closely related to token uniformity (all tokens becoming similar vectors), but they are not strictly the same. For example, even if the rank is small and the vectors are aligned, if there is variation in the direction between the vectors, it cannot be called uniformity. In this study, to avoid confusion between the two, we use cosine similarity distribution to investigate token uniformity.\n\nTo address your question, we will report not only the angles (similarity) between tokens but also the statistics of their norms. Since norms represent a different degree of freedom from angles, there are various possibilities, such as the cosine approaching 1 and aligning while the norms remain highly variable.  \n\nThe following are the average values (mean) and variances (std) of token norms for each layer of trained ViT-L and MHA. Interestingly, while MHA resolved token uniformity, it was observed that it also uniformly aligned the norms of each token and greatly reduced the variance. Thanks to your comments, we discovered a very interesting phenomenon, which we would like to pursue as an interesting research topic in the future.\n\nCifar100, ViT-Large\n| layer | Mean(Self-attention) | Std(Self-attention)        | Mean(MHA,$ \\alpha=05$) | Std(MHA, $\\alpha=05$) | Mean(MHA, $\\alpha=07$) | Std (MHA,$\\alpha=07$) |\n|-------|---------------|-----------|--------------|-----------|--------------|-----------|\n| 1    | 10.4820      | 0.5994   | 31.9970     | 0.0003455 | 31.9970     | 0.0003455 |\n| 4    | 13.6818      | 0.2291   | 31.9970     | 0.0000852 | 31.9970     | 0.0000852 |\n| 8    | 13.9900      | 0.1034   | 31.9970     | 0.0000563 | 31.9970     | 0.0000563 |\n| 12   | 13.5533      | 0.0630   | 31.9969     | 0.0000698 | 31.9969     | 0.0000698 |\n| 16   | 13.4659      | 0.1685   | 31.9968     | 0.0000487 | 31.9968     | 0.0000487 |\n| 20   | 13.4335      | 0.4192   | 31.9968     | 0.0000751 | 31.9968     | 0.0000751 |\n| 24   | 11.4047      | 0.9176   | 31.9967     | 0.0000690 | 31.9967     | 0.0000690 |\n\nSimilarly, when we evaluated GPT-2, we found no significant differences between GPT and MHA. This is likely due to the fact that the detailed behavior of MHA differs between the bidirectional and autoregressive Transformers.\n\nWikitext103, GPT-2(medium)\n| layer | Mean(Self-attention) | Std(Self-attention) | Mean(MHA, $\\alpha=05$) | Std(MHA, $\\alpha=05$) |\n|-------|---------------|--------|--------------|--------|\n| 1     | 1.3457        | 0.6444 | 1.4607       | 0.4460 |\n| 4     | 10.6074       | 3.0606 | 7.5750       | 1.9212 |\n| 8     | 2.6345        | 0.6598 | 0.2133       | 0.0575 |\n| 12    | 3.1324        | 0.6150 | 3.0704       | 0.6068 |\n| 16    | 13.1777       | 8.8199 | 20.2960      | 8.1086 |\n| 20    | 18.3072       | 6.0310 | 17.8403      | 4.0680 |\n| 24    | 19.1561       | 2.9285 | 17.1602      | 1.8057 |\n\n**[Weakness, Question] The usefulness of performance improvement through MHA**\n\nI understand your question to be about the significance of the proposed algorithm among the numerous technical improvements made to Transformers to date.\n\nAs you understand, the main focus of this paper is not to propose a new practical method, but to understand the role of hidden states in Hopfield Networks from the context of Transformers. To demonstrate that the attention mechanism (MHA) derived from the discussion in this paper has a meaningful structure, we confirmed performance improvements in language and image tasks. These results also suggest practical usefulness, so we agree that the significance of comparing it to existing architectures, as in your question, is an important point.\n\nRegarding your concern about the “relatively modest\" performance improvement, there is potential for further improvement through hyperparameter tuning and technical refinements. As evidence of this, we present the results of additional experiments. Similar to CIFAR experiments, using the setting $\\alpha=\\alpha’=0.7$ on ImageNet-1k further clarified the performance improvement:\n\n| Self-attention | MHA($\\alpha=0.5$) | MHA($\\alpha=0.7$) |\n| ---- | ---- |---- |\n| 76.074 | 76.434 | **77.058** |\n\nIn this way, it is suggested that further performance improvements are possible through appropriate hyperparameter selection.\n\nAnother important result of the additional experiments is the systematic and significant performance improvement over Llama shown above. The fact that applying MHA to an architecture like Llama, which has been technically optimized, still improves performance suggests that simply adding MHA to the various Transformer variants proposed so far for performance optimization could achieve further performance improvements and contribute to raising the state-of-the-art. While a large-scale verification of MHA is beyond the scope of this paper, it is considered an extremely interesting research direction.\n\n**[Quality of Figures]**\n\nWe apologize for any inconvenience caused by the insufficient resolution of the figures. We will create high-quality figures and replace them in the camera-ready version."}, {"review_id": "2aRtrdoM8k", "reviewer": "Reviewer_CUjJ", "summary": "This paper introduces a novel perspective by interpreting the dynamics of hidden states in modern Hopfield networks as a form of transformer attention. The resulting mechanism, termed Modern Hopfield Attention (MHA), is shown—both theoretically and empirically—to mitigate issues such as rank collapse and token uniformity that affect standard self-attention. By replacing the self-attention modules in architectures like GPT-2 and Vision Transformer (ViT) with MHA, the authors demonstrate improved performance on text generation and image recognition tasks.", "questions": "1. The authors refer to the visible variables as $v$ in the text, but use $x$ in Equation (1). Are these variables intended to represent the same quantity? Clarifying the notation would help avoid confusion.\n\n2. Why do the results in Tables 2 and 5 differ significantly, despite both using the same datasets (CIFAR-10 and CIFAR-100)? A more detailed explanation of the experimental settings and differences is needed. In general, the table captions should provide clearer and more comprehensive descriptions to aid interpretation.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper introduces a novel perspective by utilizing the full dynamics of hidden states in modern Hopfield networks, moving beyond the commonly adopted adiabatic approximation. In addition, the authors provide solid theoretical analysis demonstrating that the proposed Modern Hopfield Attention (MHA) mechanism effectively mitigates rank collapse—a known limitation of standard self-attention.\n\nWeaknesses:\nDespite the theoretical contributions, the empirical improvements in text generation and image recognition tasks appear relatively modest compared to models using conventional self-attention. Moreover, the performance is highly sensitive to the hyperparameters $\\alpha$ and $\\alpha'$, as evidenced in Table 4. Given the degree of accuracy fluctuations, it becomes questionable whether the small performance differences reported in Tables 1, 2, and 3 are statistically meaningful.", "was_revised": true, "final_justification": "After reviewing the responses and related discussions, I have decided to maintain my original rating of 3 (borderline reject), as I remain unconvinced that the new formulation has a substantial impact on practical performance.", "rebuttal": "First of all, we would like to express our gratitude for your valuable comments and review. We also appreciate your evaluation of our theoretical contributions.\n \n**[Weakness]**\n\nAs you pointed out, in the experiments in Table 4, we investigated $\\alpha, \\alpha’$ in the range of 0 to 1, and the behavior of the network changes dramatically in this parameter range. Specifically, $\\alpha=1$ corresponds to a pure feed-through layer without any attention, $\\alpha=0$ corresponds to a pure attention layer that removes shortcut paths, $\\alpha’=1$ corresponds to a case where attention is calculated using only information from the previous layer, and $\\alpha’=0$ corresponds to conventional self-attention.\n\nDespite these structurally different behaviors, the performance variation is relatively mild (about 2% in the range of $\\alpha,\\alpha’$ from 0.1 to 0.8), and the key point of this table is that sufficient performance can be obtained with the simple setting of $\\alpha=\\alpha’=0.5$. This demonstrates that our proposed method is robust and does not rely heavily on tuning (The effect of hyperparameter tuning is relatively mild).\n\nFurthermore, we present additional results that suggest the possibility of further performance improvements through technical innovations.\n\nFirst, regarding the Top-1 performance on ImageNet-1k, we tested the hyperparameter $\\alpha=\\alpha'=0.7$ on ImageNet-1k, similar to the CIFAR experiment (Table 2). As a result, the performance difference became even more pronounced:\n\nAdditional results on ImageNet-1k (Top-1 Accuracy)\n| Self-attention | MHA($\\alpha=0.5$) | MHA($\\alpha=0.7$) |\n| ---- | ---- |---- |\n| 76.074 | 76.434 | 77.058 |\n\nThe performance improvement of over 1% without adding any training parameters is sufficiently meaningful when compared to the improvement achieved by modifying ViT-B in existing studies [39][50] (both less than 0.6%). These existing studies differ from this paper in that they achieved accuracy by introducing additional learning parameters, making this performance difference even more notable.\n \nAdditionally, as an additional experiment, we report the results of applying MHA to the Llama architecture (perplexity). In the following experiments, we trained Llama on Wikitext103 as well as various datasets with distinct characteristics, such as books and news articles.\n\nAdditional results in Llama (Perplexity)\n| Dataset |  Self-attention | MHA(\\alpha=0.5) |\n| ---- | ---- | ---- |\n| Wikitext103(20%)| 29.92| **24.48** |\n| BookCorpus(20%)| 67.05 | **40.68** |\n| CNN_dailymail(20%)| 78.81| **58.62** |\n| xsum(20%)| 103.30| **97.91** |\n\nThe above results were obtained by comparing different language domains using the mini-llama implementation. In all cases, MHA outperformed the baseline, demonstrating that our method is effective in terms of both architecture and domain.\n\n**[Question 1]**\n\nThank you for pointing that out. As you mentioned, $v$ is a typo for $x$ and means the same variable. We will correct it in the camera-ready version.\n\n**[Question 2]**\n\nThank you for your important question. Table 2 shows the results of applying MHA to a conventional ViT model, while Table 5 shows a comparison using an attention-only model that excludes shortcut paths and FFNs in order to approximate our theoretical setup.\nAs reported in previous studies [10], rank collapse is particularly prominent in attention-only models, so we adopted this setting to more clearly demonstrate the effectiveness of MHA. This point is explained in lines 270–279, but to make it clearer, we will revise the caption of Table 5 to specify the experimental conditions.\n \n**[Comments on Formatting]**\n\nThank you very much for your suggestions on the structure. As you pointed out, we will consider summarizing the theoretical core results (e.g., Theorem 2) in the introduction to aid the reader's understanding.\nIn addition, we will rearrange the related research (Section 5) immediately after the introduction and reconsider the structure to more clearly convey how our contributions relate to existing literature."}, {"review_id": "2NWS3f1Teh", "reviewer": "Reviewer_BWoL", "summary": "Recently, it was shown that the state update rules of modern continuous Hopfield networks (with softmax-based energy) have a mathematical structure exactly equivalent to that of the self-attention. Building upon this connection, this work shows that the connection between the two systems can be more generalized, in which the authors proposed a new attention mechanism with the hidden state, Modern Hopfield Attention (MHA). The MHA-based Transformer improves the nature of attention weights by sharing attention score information across layers.", "questions": "1. Regarding the inverse temperature $\\beta$, did you simply use the standard value of $\\frac{1}{\\sqrt{d}}$ as $\\beta$? It is worth including $\\beta$ as part of your formulations detailed in section 2.3 for completeness.\n\n2. For GPT-2 (medium), MHA layers exhibit \"softer\" cosine similarity. But for ViT or image domain, this does not seem to be the case for the early layers. Why is that for the early layers the cosine similarity centers around 1, but not for the later layers in the ViT/Image domain?\n\n3. In some sense, I can see that there are two energies operating independently: one for the MLP block and the other for the attention (which is MHN). The hidden state coming from MLP block is evolving following its own energy, and the same applies to the attention. Overall, what is your intuition of this aspect?", "limitations": "There is not a single global energy function controlling the operations of the Transformer like [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf) --- making the model in this work, not a recurrent model. Nonetheless, I think the formulation, presented here, is interesting and novel as it enables exponential moving average of the hidden states (from MLP) and tokens (from attention).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. Unlike [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf), this work does not require training but rather reformulating the operation of Attention to match MHN update and to incorporate the input of the MLP output in Transformer into the MHN update. Hence, there is no parallel update of the token between MLP and Attention as in [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf).\n\n2. The Transformer operation, presented by the work, is now essentially an exponential moving average of the MLP outputs and Attention outputs across a series of layers, which could indicate a more robust representation of the tokens after they are updated.\n\n3. The results illustrate strong performance of the modified Transformer as the model's size becomes larger, see Table 2. Moreover and most importantly, the authors provide an understanding on why the introduced method has a better performance and highlighted the problem of rank collapse in Transformer --- cool!\n\nWeaknesses:\n\n1. The results demonstrated on the text modality are little. It is still unclear if the same trend is applicable for the language modality and multi-modalities domain. It would be nice to see further demonstration of the introduced approach on text, see [He et al. (2024)](https://arxiv.org/pdf/2402.13449).\n\n2. In the MCHN language, both of the key and value matrices, $W_K$ and $W_V$, are treated as the memory matrix $\\xi$, i.e., the memory matrix is shared between the two operations. But in this work, they are treated separately --- following the same line as the standard Transformer operation. \n\n3. The work reformulates the Transformer's operations via the perspective of MHN dynamics, but it is not clear on how these new operations follow a global energy function (which dictates the MHN dynamics). The work should try to highlight and explain this aspect better, since the usage of a well-formulated global energy function is one of the fundamentals of Associative Memory.", "was_revised": true, "final_justification": "I think the work is good and the presented results are significant. My overall score of 4s imply reflects that they need to demonstrate more results, e.g., showing the differences in self-attention and MHA regarding the learned embeddings and text modality. **I trust that the authors will include those final results in their last version.**", "rebuttal": "We sincerely appreciate your valuable review. In particular, we appreciate your positive evaluation of the novelty of this study, its significance in the context of Hopfield Networks, and its theoretical contributions.\n\n**[W1] Verification of effectiveness in text modality**\n\nAs you pointed out, strengthening performance verification in natural language is an important issue. We conducted additional experiments using the Llama architecture (mini-llama implementation) to verify the effectiveness of MHA in other Transformer-based architectures. We trained the model on 20% subsets of Wikitext103, BookCorpus, CNN/DailyMail, and XSum, and confirmed significant improvements in perplexity as follows:\n\n**Additional results in Llama (Perplexity)**\n| Dataset |  Self-attention | MHA($\\alpha=0.5$) |\n| ---- | ---- | ---- |\n| Wikitext103(20%)| 29.92| **24.48** |\n| BookCorpus(20%)| 67.05 | **40.68** |\n| CNN_dailymail(20%)| 78.81| **58.62** |\n| xsum(20%)| 103.30| **97.91** |\n\nThese results demonstrate that MHA is consistently effective across multiple data domains, reinforcing its usefulness in natural language processing. On the other hand, verifying its effectiveness in multimodal domains is an important future direction.\n\n**[W2, W3, Q3] Considerations on the symmetry of memory matrices and energy functions**\n\nThank you for your important questions regarding key matrices and value matrices. As you pointed out, in MCHN, the weight matrix connecting the hidden state and the visible state is given by the same memory matrix $\\xi$ and $\\xi^\\top$. On the other hand, in attempts to relate MCHN to Transformers, such as [35][27], the weight tying of $\\xi$ and $\\xi^\\top$ is resolved, and they are mapped to independent $W_K$ and $W_V$, respectively.\n\nThe violation of symmetry in this memory matrix destroys the assumption of monotonic decrease in the energy function in the mathematical argument of Krotov-Hopfield [27]. Therefore, in such cases, it becomes impossible to prove the existence of a monotonic decrease in the energy function. On the other hand, as a trade-off, we can obtain improved representation and learning capabilities in Transformer models. For example, in [KOT], this issue is investigated in the context of the correspondence between Krotov-Hopfield's MCHN and MLP-Mixer. According to this study, it has been confirmed that breaking symmetry significantly improves image recognition performance.\n\nFurthermore, according to [YM], breaking the symmetry of the memory matrix may cause bifurcations in energy dynamics. The energy function interpretation of MHA in an asymmetric setting of $\\xi$ is an extremely interesting theoretical issue for future research. As you pointed out, the fact that there are unresolved and interesting issues regarding the energy function is extremely important, so we will clearly state this in the camera-ready manuscript.  \n\n**references**\n- [KOT] Karakida-Ota-Taki, Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking\n- [YM]Yampolskaya-Mehta, Controlling the bifurcations of attractors in modern Hopfield networks, NeurIPS 2023 Workshop AMHN\n\n\n**[Q1] Selection of inverse temperature**\n\nAs you pointed out, in this study, we selected the inverse temperature as $\\beta=\\frac{1}{\\sqrt{d_k}}$. In the equations in Section 2, we omitted the inverse temperature for the sake of simplicity. We briefly mentioned this point in line 146, but it was insufficient in terms of clarity for readers. In the camera-ready version, we will revise Section 2 to explicitly state this setting.\n\n**[Q2] Behavior of cosine similarity in the initial layer**\n\nThank you for your interesting comment. In Figure 2, the cosine similarity of a specific layer indicates the cosine similarity between tokens input to that layer. Therefore, the cosine similarity of the first layer is the cosine similarity between the embedding vectors created in the embedding layer. The nature of embedding differs greatly between natural language and images.\n\nIn particular, in the initial layer, while tokens have discrete meanings in natural language, in the image domain, embeddings for each patch have spatial continuity, and the similarity between adjacent image tokens tends to be high. This explains the difference in distribution in the initial layer of ViT.\n\nHowever, this difference in the initial distribution specific to each domain gradually diminishes through MHA's multi-layer processing, and ultimately both converge to a state where token uniformity is improved. We consider this observation to be supporting evidence that MHA consistently performs conversion of representation even across different domains.\n\n**[Comments on formatting]**\n\nThank you for your feedback. As you pointed out, the number of violin plots in the current appendix is excessive and may hinder interpretation. The purpose of these plots was to comprehensively show how the similarity distribution of each token changes across layers and architectures. In the camera-ready version, we will revise the appendix to simplify these plots.\n\nWe appreciate the opportunity to reconsider the theoretical background and future possibilities of our method through the review comments."}]

Author rebuttal:

Meta-review: The paper utilizes the Krotov-Hopfield formulation of modern Hopfiled networks for rewriting the attention mechanism to include hidden units. While conventional attention arises in the adiabatic limit, this novel reformulation makes it possible to depart from the adiabatic limit, and formulate an alternative attention mechanism which benefits from the hidden unit dynamics. The paper argues that this reformulation mitigates the rank collapse problem common in deep transformers. 

The idea is novel and the work received enthusiastic evaluations from reviewers. The reviewers request that the additional results with language models, provided during rebuttal, are included in the camera-ready version.
Decision: Accept (poster)