Title: Is Grokking a Computational Glass Relaxation?
Venue: NeurIPS 2025
Accepted: yes
Keywords: grokking

entropy

generalization

glass
Abstract: Understanding neural network' (NN) generalizability remains a central question in deep learning research.
The special phenomenon of grokking, where NNs abruptly generalize long after the training performance reaches near-perfect level, offers a unique window to investigate the underlying mechanisms of NNs' generalizability.
Here we propose an interpretation for grokking by framing it as a computational glass relaxation: viewing NNs as a physical system where parameters are the degrees of freedom and train loss is the system energy, we find memorization process resembles a rapid cooling of liquid into non-equilibrium glassy state at low temperature and the later generalization is like a slow relaxation towards a more stable configuration. 
This mapping enables us to sample NNs' Boltzmann entropy (states of density) landscape as a function of training loss and test accuracy. 
Our experiments in transformers on arithmetic tasks suggests that there is NO entropy barrier in the memorization-to-generalization transition of grokking, challenging previous theory that defines grokking as a first-order phase transition.
We identify a high-entropy advantage under grokking, an extension of prior work linking entropy to generalizability but much more significant. 
Inspired by grokking's far-from-equilibrium nature, we develop a toy optimizer WanD based on Wang-landau molecular dynamics, which can eliminate grokking without any constraints and find high-norm generalizing solutions. 
This provides strictly-defined counterexamples to theory attributing grokking solely to weight norm evolution towards the Goldilocks zone and also suggests new potential ways for optimizer design.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates, from a theoretical point of view, the grokking phase in NNs. The analysis is recast as a physical system modelled by computational glass relaxation, where the parameters are degrees of freedom, and energy is the corresponding training loss. This approach enables the authors to associate the memorization process with the rapid cooling of a liquid into a non-equilibrium state and the subsequent generalization with the slow relaxation towards stability. In this setup, one can define the NN Bolzman entropy. Experiments are conducted using transformers on modulo arithmetic tasks and reveal the connections between high-entropy minima and generalization ability. A novel toy optimizer that can avoid grokking is presented as a practical outcome of the previous analysis.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces a compelling and original perspective by modeling grokking as a computational analog of glass relaxation. This physical analogy helps explain the delayed generalization phase in neural networks, using entropy landscape analysis to challenge the prevailing interpretation of grokking as a first-order phase transition. The proposed optimizer (WanD) envisions the potential practical impact of the study. More experiments with Wand could be added to support the theoretical findings.\n\nQUESTIONS:\nI list some minor concerns:\n- (pag. 4) Is there any limitation in constraining the NN parameters to the range $3 / \\sqrt m$? \n- (pag. 4) It is stated that in Sec. 4.2  the weight norm is limited to 30. How is this enforced during training without impacting the learning dynamic?\n- In Sect. 5, WanD is tested only on the modular addition $p=67$ task that exhibits the grokking phase. Nonetheless, it would be important to add at least one experiment without grokking to ensure that WanD can be used in a general training setup when the nature of the task to be solved is unknown in advance.\n\nLIMITATIONS:\nThe limitation section is missing. I think it is important to add a discussion on the current limitations of the study worked out on toy modular arithmetic tasks with one-layer transformer.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Authors compare grokking to a slow relaxation of a many-body system into an equilibrium state after a rapid cooling process\n- they argue against a first-order state transition interpretation, as brought forward by previous work, by investigating the boltzmann entropy landscape as a function of train loss and test accuracy and finding no discontinuity in the order parameter during the transition process\n- authors argue for a “high entropy advantage” in the grokking scenario. The training trajectory in the (train loss, test acc) landscape differs vastly from a more beneficial trajectory, the maximum entropy curve, which achieves high test accuracy much quicker. they claim a broad correlation between high (boltzmann) entropy solutions and generalizability in neural networks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- quite clean experiments with nice interpretations\n- interesting hypothesis and theory\n- a refreshing perspective on grokking\n\nWeaknesses\n\n- the binning in (train loss, test acc) seems somewhat arbitrary. I understand that it is necessary for the simulation and the calculation of $S$, but I can imagine that different binning schemes can have vastly different outcomes if the landscape is not very regular. How do you account for that?\n- the impact of this work is not super clear to me. while it is an interesting hypothesis and seems to have some confirming evidence, what should I take away from this for the general community? the entropy to generalization link is certainly broadly interesting, but that contribution belongs to [2]. I am not convinced about the utility of the WanD optimizer.\n- there should be a proof (or at least a link to one) that the WLMD sampling converges to the true $S$ at the places where it is sampled. There are so many possible parameter configurations that it sounds non-trivial to get a grasp of true entropy by iterating this algorithm.\n- the authors claim to overthrow the first-order transition hypothesis from Rubin  et al [1] and attribute the differences to data and training setup. It seems weird that they don’t attempt the same setup as [1] to see whether they reach the same conclusion there, since they claim the disproving of [1] as a main contribution.\n\nQUESTIONS:\n- What the relationship between $U$ and $(L_{train}, A_{test})$ ?\n- I don’t really understand the WL simulation. How exactly is $\\frac{\\partial S}{\\partial L_{train}}$ calculated? You say via numerical differentiation of adjacent bins, but that estimate is very bad at the beginning of the simulation right? Don’t you need to sample a lot of parameter configurations to estimate that? Maybe a proof of how it converges to the right estimate would be helpful.\n- Also a little algorithm box explaining the exact steps to simulate and update NN parameters would be helpful.\n- Why do you need to conserve kinetic energy? Otherwise updates become too large as potential energy is transferred into kinetic?\n- you conclude that grokking is not a first order phase transition because you don’t see a discontinuity in the order parameter you choose (test accuracy). Does that mean that no order parameter that one could choose would be discontinuous?\n- why does the need to cross a free-energy barrier imply a first order phase transition? because crossing wouldn’t happen unless one jumps right to the other side, implying a discontinuity (like quantum tunneling)?\n- How would a discontinuity present itself in Fig 1? Would the yellow line be disconnected?\n    - If that is the case, could one not always reduce the size of the update (i.e. learning rate) to make it continuous? Or would grokking just never appear with such a low learning rate in that view?\n- Why can Fig 1(c) not be learned by training?\n- How much of the training time in Fig 1(b) goes into the vertical ascent of the yellow curve vs the horizontal move along the train loss?\n- how does the conical degree combine weight norm and flatness? it would help to expand a little on that, it’s a little weird to introduce this notion in the conclusion.\n- i don’t understand why the glassy interpretation implies that generalization takes a long time.\n- do you actually check the gradient norms during the relaxation process to corroborate your hypothesis?\n\nNits:\n\n- L177 define A_test and L_train\n- L178: calculates → calculate\n- I was confused by the term “relaxation” until the very end (L354), expecting some relaxation of constraints as is common in optimization. maybe explain in the beginning that you call it relaxation because of the term “structural relaxation”.\n\nLIMITATIONS:\nconsider citing and discussing the relationship to https://arxiv.org/abs/2205.10343v2\n\noverall I like the view this paper takes, but there are some holes in explanation and experimental and theoretical rigor. I am willing to adjust my score if my weaknesses and questions are addressed to my satisfaction.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work draws an analogy between\ngrokking in neural networks (a sudden drop in test error \nwell after the network has memorized the training data) \nand the behavior of particles in a liquid after\na sudden temperature drop. \nBuilding on this connection, the authors construct a phase\ndiagram of the entropy (volume of weight configurations) of a one-layer transformer with a given training loss and test accuracy. \nThey observe that:\n1) in the grokking regime (low training loss), the entropy\nlandscape does not exhibit any barrier, challenging the \nbelief that grokking is a first-order phase transition;\n2) the solutions that maximize the entropy \nfor a fixed training loss generalize well and do not \nexhibit grokking.\n\nBased on these insights, the authors introduce a new \nalgorithm that targets high-entropy solutions and avoids grokking without \nthe need to control weight norms.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. This is a very interesting contribution to the\n    ongoing discussion around the origin of grokking.\n2. Overall, the paper is cleanly written, the story\n    is clear and the experimental results are well presented.\n\n## Weaknesses\n\n1. The presentation of the two core algorithms\n    (for the simulation of the entropy landscape and targeting high-entropy solutions)\n    lack details, both in terms of how and why they\n    work, and how they were implemented. It would\n    be helpful to give more intuition on\n    them, especially for a broader\n    ML audience, which is unlikely to be familiar with\n    Wang-Landau molecular dynamics.\n2. My understanding is that \n    the proposed WandD algorithm is computationally expensive. A discussion about its complexity/convergence rate \n    would strengthen the work.\n\nQUESTIONS:\n1. I found the intuition behind the notion of\n    ''conical degree'' in Section 6 somewhat confusing.\n    Is there a specific reason to believe that the cutoff\n    induced by fixing the training loss admits a simple\n    geometric interpretation?\n2. Figure 1.(b) seems to be the key experimental\n    result, as this is where grokking is most apparent for AdamW. The max-entropy curve in this panel\n    appears to rise sharply towards perfect test accuracy. How did you ensure that the continuity of the \n    max-entropy curve is not a finite-size effect?\n3. Several algorithms have been proposed to explicitly target high-entropy solutions in the traditional loss landscape sense (for example https://iopscience.iop.org/article/10.1088/1742-5468/ab39d9). How do those methods compare empirically to the one proposed in this paper, both in terms of generalization performance and existence of grokking?\n4. In the sampling of the entropy landscape using molecular dynamics (Equation 4), the training loss and test error appear to play symmetric roles, especially in the absence of regularization in the training loss. This makes me wonder, how robust are the resulting phase diagrams to changes in the choice of training loss?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel interpretation of the grokking phenomenon in neural networks through the lens of statistical physics, particularly computational glass relaxation. It employs the Wang-Landau Molecular Dynamics algorithm to estimate entropy landscapes in modular arithmetic tasks and demonstrates that grokking does not correspond to a first-order phase transition, contradicting prior work. Additionally, the authors propose a physics-inspired optimizer, WanD, capable of eliminating grokking without explicit regularization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Novel Theoretical Perspective: The paper provides a fresh and compelling physical analogy for an AI phenomenon. Viewing a neural network as a physical system and grokking as analogous to glass transition/relaxation is an innovative framing. This perspective synthesizes ideas from statistical mechanics with deep learning, potentially opening new avenues for understanding generalization. The analogy is well-grounded in the results: the authors show that pre-grokking fast training is analogous to rapidly quenching a liquid into a glass (a non-equilibrium state), and the grokking phase is akin to a slow structural relaxation toward equilibrium.\n\n2. Entropy Landscape Analysis: Methodologically, the use of the Wang-Landau Molecular Dynamics algorithm to sample the entropy landscape of neural network parameters is a significant strength. This approach goes beyond conventional training-loss landscapes by explicitly estimating the Boltzmann entropy (volume of solutions) at different performance levels. The paper thus moves beyond local geometry (sharpness or flatness of minima) to a global thermodynamic view of the parameter space.\n\n3. Connection to Prior Work: The authors do a good job situating their contribution relative to prior explanations of grokking. They explicitly discuss how their entropy-based view can account for earlier observations.  At the same time, they challenge two prominent prior hypotheses – the phase transition interpretation and the weight-norm-driven explanation.\n\nWeaknesses:\n1. Limited Scope of Experiments: A notable limitation is that all experiments are conducted on relatively small-scale, synthetic tasks (modular arithmetic problems with a small transformer). \n\n2. Computational Complexity and Feasibility: The approach of sampling an entropy landscape with WLMD, while powerful, is computationally intensive and may not be practical beyond toy settings. The paper does not clearly report the computational cost of obtaining the entropy landscape for even these small models.\n\n3. Clarity for Non-Physicists: The paper assumes a certain comfort with physics terminology which might pose a barrier to some readers.\n\n4. Missing or Understated Insights: The paper touches on how its findings relate to earlier hypotheses (like representation learning or the slingshot effect), but these connections remain somewhat abstract. For example, it is asserted that the entropy perspective can explain observations of representation shift or sparse subnetworks, yet the reader must infer the mechanism (presumably, high-entropy solutions correspond to learned sparse structures or improved features). These connections would be more convincing if explicitly demonstrated or at least elaborated.\n\nQUESTIONS:\nImplementation and Stability of WanD Optimizer: The WanD optimizer appears promising, but the manuscript lacks details on its computational overhead, sensitivity to hyperparameters, and stability. Could you provide more information about:\n1. The additional computational cost compared to AdamW and If WanD is shown to be efficient, stable, and generalizable with minimal tuning, that would strengthen the practical contribution and could positively affect the score.\n\nLIMITATIONS:\nI am not very familiar with this area and lack some of the necessary background in physics, which made the paper challenging for me to fully understand.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}