Title: On the Role of Hidden States of Modern Hopfield Network in Transformer
Venue: NeurIPS 2025
Accepted: yes
Keywords: hopfield networks

self-attention

Transformer

rank collapse
Abstract: Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learning. It has been pointed out that the state update rule of the modern Hopfield network (MHN) in the adiabatic approximation is in agreement with the self-attention layer of Transformer. In this paper, we go beyond this approximation and investigate the relationship between MHN and self-attention. Our results show that the correspondence between Hopfield networks and Transformers can be established in a more generalized form by adding a new variable, the hidden state derived from the MHN, to self-attention. This new attention mechanism, modern Hopfield attention (MHA), allows the inheritance of attention scores from the input layer of the Transformer to the output layer, which greatly improves the nature of attention weights. In particular, we show both theoretically and empirically that MHA hidden states significantly improve serious problem of deep Transformers known as rank collapse and token uniformity. We also confirm that MHA can systematically improve accuracy without adding training parameters to the Vision Transformer or GPT. Our results provide a new case in which Hopfield networks can be a useful perspective for improving the Transformer architecture.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecently, it was shown that the state update rules of modern continuous Hopfield networks (with softmax-based energy) have a mathematical structure exactly equivalent to that of the self-attention. Building upon this connection, this work shows that the connection between the two systems can be more generalized, in which the authors proposed a new attention mechanism with the hidden state, Modern Hopfield Attention (MHA). The MHA-based Transformer improves the nature of attention weights by sharing attention score information across layers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Unlike [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf), this work does not require training but rather reformulating the operation of Attention to match MHN update and to incorporate the input of the MLP output in Transformer into the MHN update. Hence, there is no parallel update of the token between MLP and Attention as in [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf).\n\n2. The Transformer operation, presented by the work, is now essentially an exponential moving average of the MLP outputs and Attention outputs across a series of layers, which could indicate a more robust representation of the tokens after they are updated.\n\n3. The results illustrate strong performance of the modified Transformer as the model's size becomes larger, see Table 2. Moreover and most importantly, the authors provide an understanding on why the introduced method has a better performance and highlighted the problem of rank collapse in Transformer --- cool!\n\nWeaknesses:\n\n1. The results demonstrated on the text modality are little. It is still unclear if the same trend is applicable for the language modality and multi-modalities domain. It would be nice to see further demonstration of the introduced approach on text, see [He et al. (2024)](https://arxiv.org/pdf/2402.13449).\n\n2. In the MCHN language, both of the key and value matrices, $W_K$ and $W_V$, are treated as the memory matrix $\\xi$, i.e., the memory matrix is shared between the two operations. But in this work, they are treated separately --- following the same line as the standard Transformer operation. \n\n3. The work reformulates the Transformer's operations via the perspective of MHN dynamics, but it is not clear on how these new operations follow a global energy function (which dictates the MHN dynamics). The work should try to highlight and explain this aspect better, since the usage of a well-formulated global energy function is one of the fundamentals of Associative Memory.\n\nQUESTIONS:\n1. Regarding the inverse temperature $\\beta$, did you simply use the standard value of $\\frac{1}{\\sqrt{d}}$ as $\\beta$? It is worth including $\\beta$ as part of your formulations detailed in section 2.3 for completeness.\n\n2. For GPT-2 (medium), MHA layers exhibit \"softer\" cosine similarity. But for ViT or image domain, this does not seem to be the case for the early layers. Why is that for the early layers the cosine similarity centers around 1, but not for the later layers in the ViT/Image domain?\n\n3. In some sense, I can see that there are two energies operating independently: one for the MLP block and the other for the attention (which is MHN). The hidden state coming from MLP block is evolving following its own energy, and the same applies to the attention. Overall, what is your intuition of this aspect?\n\nLIMITATIONS:\nThere is not a single global energy function controlling the operations of the Transformer like [Hoover et al., (2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/57a9b97477b67936298489e3c1417b0a-Paper-Conference.pdf) --- making the model in this work, not a recurrent model. Nonetheless, I think the formulation, presented here, is interesting and novel as it enables exponential moving average of the hidden states (from MLP) and tokens (from attention).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel perspective by interpreting the dynamics of hidden states in modern Hopfield networks as a form of transformer attention. The resulting mechanism, termed Modern Hopfield Attention (MHA), is shown—both theoretically and empirically—to mitigate issues such as rank collapse and token uniformity that affect standard self-attention. By replacing the self-attention modules in architectures like GPT-2 and Vision Transformer (ViT) with MHA, the authors demonstrate improved performance on text generation and image recognition tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper introduces a novel perspective by utilizing the full dynamics of hidden states in modern Hopfield networks, moving beyond the commonly adopted adiabatic approximation. In addition, the authors provide solid theoretical analysis demonstrating that the proposed Modern Hopfield Attention (MHA) mechanism effectively mitigates rank collapse—a known limitation of standard self-attention.\n\nWeaknesses:\nDespite the theoretical contributions, the empirical improvements in text generation and image recognition tasks appear relatively modest compared to models using conventional self-attention. Moreover, the performance is highly sensitive to the hyperparameters $\\alpha$ and $\\alpha'$, as evidenced in Table 4. Given the degree of accuracy fluctuations, it becomes questionable whether the small performance differences reported in Tables 1, 2, and 3 are statistically meaningful.\n\nQUESTIONS:\n1. The authors refer to the visible variables as $v$ in the text, but use $x$ in Equation (1). Are these variables intended to represent the same quantity? Clarifying the notation would help avoid confusion.\n\n2. Why do the results in Tables 2 and 5 differ significantly, despite both using the same datasets (CIFAR-10 and CIFAR-100)? A more detailed explanation of the experimental settings and differences is needed. In general, the table captions should provide clearer and more comprehensive descriptions to aid interpretation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new continuous modern Hopfield network architecture that introduces dynamics in the hidden state. The number of parameters is kept constant but the computational time is increased linearly as a function of the transformer dimension. The authors show this architecture prevents the rank collapse problem, which may be what modestly improves the ViT and GPT performance.\n\nSTRENGTHS AND WEAKNESSES:\nThe derivation of the hidden state dynamics seems sound.The authors clearly describe previous and related work. X and Y labels in Figures 1(b) and 2 are missing. Would be important to have confidence intervals in the perplexity results throughout the paper, to check if the improvements in perplexity are statistically significant. \n\nThe paper is clear and well organized. \n\nAddressing the rank collapse problem is important and the theoretical link to MHN is intriguing via the consideration of the dynamics in the hidden state allows for an improvement of the rank collapse problem. However, it would be nice to actually show that the rank is not collapsing in addition to showing the activation of selected layers is more uniform compared to the self attention case. Ultimately, it seems that the performance improvements are modest.\n\t\nIs it clear how this work differs from previous contributions. The proposed algorithm improves slightly the performance compares to the previous state of the art. The paper provides new insights on how to use modern hopfield networks in transformer architectures.\n\nQUESTIONS:\nCan the authors identify stronger improvements over baselines using their proposed principle? The improvements seem relatively modest compared to the sophistication of the theory but happy to be convinced otherwise.\n\nI think the figure quality could be significantly improved.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper relooks at the connection between Modern Continuous Hopfield Networks and the Attention Mechanism, as in some relevant prior work (mentioned in the paper). A traditional Hopfield Network consists of two layers, the feature neurons and the hidden neurons (denoted by $v$, $h$ respectively). In that context, it identifies a common assumption made in most contemporary work of discretizing the evolution of the dynamic system under the adiabatic limit and chooses NOT to assume this. Not making this assumption leads to an interesting modified attention style update rule, that performs the softmax operation on the feature neurons, which is in turn updated via a recurrence equation involving its own previous state. Thus a skip connection is implicitly introduced in the proposed attention mechanism. This new rule is rigorously derived from the dynamics of MCHN and tested on a few standard models cum datasets, in both vision and language. Finally, the impactful utility of this attention scheme is demonstrated in the context of token uniformity/rank collapse, a common problem in vanilla Transformers, both theoretically and empirically.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The insight of NOT taking the adiabatic limit of the energy dynamics, then deriving an update rule based on the chosen Lagrangians, and then formulating a new attention mechanism is novel, and particularly so in the context of its connection to Hopfield Networks. \n2a. This formulation involving hidden state recurrence is rigorously derived from the energy dynamics, and not just a variant of mechanisms that combine RNNs with Attention via some heuristics. \n2b. Appendix D provides a case by case analysis for different choices of the discretization step which seems novel, interesting and potentially useful for future work as well. \n3. The utility of this problem in handling a particular issue in training vanilla Transformers known as rank collapse is identified via some experiments. \n4. A theoretical result is also suggested (Theorem 4.2) which suggests how the proposed attention mechanism, MHA, beats vanilla attention at handling rank collapse.\n5. Overall, the presentation makes the paper easy to follow.\n\nWeaknesses:\n\nExperiments:\nThe experiments are overall quite weak to build confidence in the efficacy of the proposed methodology. \n\n1. Especially for language, testing on one dataset alone is insufficient. I strongly suggest the authors to try on atleast 3 more moderate-sized language datasets commonly used in contemporary research such as OpenWebText, MMLU, GLUE, BookCorpus, etc. Based on other statements in the paper, I understand that compute is concern, but please try to choose datasets wisely such that it is not too compute-hungry while at the same time, the point is properly conveyed.\n\n2. Same goes with the language models, currently only GPT-2 small and medium have been experimented. Please experiment with at least 2 more language models, e.g. amongst Llama, Gemma, GPT-3, Mistral, RoBERTa, or anything else convenient but authentic.\n\n3. For images too, it is a similar story. Though three datasets have been experimented with, the improvement in case of CIFAR-10 and ImageNet-10K appears questionable. Further the claim made in lines 185-187 without a proper reference is hard to believe, and it better be backed by some reference or some experiments if that is true. Especially given that the results are not upto the mark for CIFAR-10 and ImageNet-10K, I suggest trying on at least two more datasets, such as TinyImageNet or any other relevant, well-known dataset and showing strong results on them.\n\n4. I do not see any ablations/discussion on the initializations used in training the Transformers, for language and vision alike. This may be important in assessing the efficacy of MHA, as it may affect the results either way. Please try different initializations, including pretrained initializations for both vision and language (for instance, for ViTs, try LION initialization, CLIP based, Xavier-Glorot, etc.) \n\nTechnical:\n\n1. Lines 246 and 254-256 suggest that a doubly-exponential decay is avoided because the case $m=0$ appears in one of the arguments to the max function. However I don't think that directly holds, because it is very well possible that the other terms with $m \\neq 0$ dominate the overall max function and thus becomes doubly-exponential. \n\n2a. Even if $||Res(AttnNet(X))||$ decreases linearly as Theorem 4.2 suggests, it is not necessary that it happens because of MHA. Because ultimately, both Theorems 4.1 (which is not the authors contribution, as far as I understand) and Theorem 4.2 (which is a contribution from the authors, if I understand correctly) provide upper bounds, so it seems possible that even without MHA, $||Res(AttnNet(X))||$ was lower than the (potentially stronger) upper bound that Theorem 4.2 provides, which would make the theorem redundant in that case. \n\n2b. Ideally, to show the efficacy of MHA at avoiding rank collapse, there should be at least one example shown where under standard training of vanilla Transformers with vanilla attention, $||Res(AttnNet(X))||$ is lower bounded by some value, say $\\kappa$, and by using MHA for that case, $||Res(AttnNet(X))||$ can be brought down below $\\kappa$. The more examples this phenomena holds for, the better the efficacy of MHA. \n\n3. Equation 16 in the Appendix looks wrong -- in particular I don't see how the second term in the LHS appears, and also how the terms represented by '...' are constant terms. Please fix this, as this seems to be a crucial equation for the rest of the proof to go through.\n\nPresentation:\n1. Please rigorously define the exact expression for token similarity, as described in lines 280-282. Please also more clearly explain what exactly the violin plots are plotting, as it not is very clear in the current writeup. Also mention how many tokens are used in describing a single such plot.\n\n2. Please move lines 181-183 of the Appendix or some equivalent of it into the main text, as this more clearly describes what token uniformity is, particularly in the context of rank collapse. I don't see that clear explanation given elsewhere in the main text.\n\nMinor:\n\n1. In line 14 of the abstract, please change \"problem\" to \"problem$\\textbf{s}$\".\n2. In line 40 of the main paper, please change \"lack\" to \"lack$\\textbf{s}$\".\n3. Please replace the 3 places in the main text where \"chapter\" is used with \"section\"/\"subsection\" or some more appropriate word like that.\n4. Please use better notation for the max function in the statement for Theorem 4.2.\n\nP.S: I look forward to increasing my score, especially if concerns stated in the Experiments section, first two concerns in the Technical & Presentation sections are addressed. I would thus advise the authors to prioritize addressing the concerns in that order.\n\nQUESTIONS:\n1. Are rank collapse and token uniformity different names for the same problem? Or are they different problems that always occur together?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new type of self-attention layer in transformers called Modern Hopfield attention (MHA). Previous work had also noted the relation between Hopfield networks and transformers and  used this relation to develop Hopfield layers in transformers where the Hopfield network served as the memory in the layer.  Earlier it was shown that the state update rule of Modern Hopfield networks in the adiabatic approximation corresponded to self-attention in transformers. This result generalizes beyond the adiabatic approximation to show that the relationship still holds. Specifically, they use MHA to allow the inheritance of attention scores across the layers. The computational cost increase is modest compared to the cost of the computation of full attention in layers. \nThe key result is in the update rule of Equation (10) where instead of the attention layer, a hidden state propagates across the layer accumulating the attention score in each layer as an exponential moving average. The softmax of the hidden state gives the attention score.  Two parameters a and (1-a) control the behavior of the MHA.\n\nSTRENGTHS AND WEAKNESSES:\nThe derivations and the observations made between Modern Hopfield network and transformers are interesting. The paper is largely theoretical in nature and includes a computational complexity analysis that shows that there is a modest cost increase which is a function of the input sequence over and above the attention computation cost. The main novelty is in the modification of the attention layer as specified in Equation 10.  \n\nThe contribution is an elegant theoretical contribution in which the attention weights are accumulated through the hidden Hopfield states in a sense combining recurrent neural networks within the architecture of transformers.  The impact of MHA on transformer performance was analyzed in downstream use cases of text generation and image recognition and in both cases, improvements of over 1% were seen, particularly for larger categories recognition.\n\nQUESTIONS:\nSince there is a recurrence formulation being implied with the hidden states, it would be good to see a discussion of this approach over the newer state-space models that are offering much more computational advantage. In other words, while the theoretical result is interesting, is it still relevant in the age of state-space models?\n\n\nIt would also be good to see a discussion comparing to the Hopfield layers proposed earlier in reference[35].\n\nLIMITATIONS:\nSince there is no computational advantage of using the MHA layer and at best a 1% increase in performance in many cases, does the result have any practical implication? This would be good to point out in the discussion. \nThe related work section is bare-boned. An extensive discussion particularly w.r.t to other Hopfield layer formulations would be good to add to the paper. The paper itself did not point to any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}