Title: Grokking and Generalization Collapse: Insights from HTSR theory
Venue: NeurIPS 2025
Accepted: no
Keywords: Grokking

Heavy-Tailed Self-Regularization

Random Matrix Theory

Heavy-Tail Exponent

Spectral Analysis

Generalization Dynamics

Catastrophic Generalization Collapse

Implicit Regularization
Abstract: Grokking is a surprising phenomenon in neural network training where test accuracy remains low for an extended period despite near-perfect training accuracy, only to suddenly leap to strong generalization. In this work, we study grokking using a depth-3, width-200 ReLU MLP trained on a subset of MNIST. We investigate it's long-term dynamics under both weight-decay and, critically, no-decay regimes—the latter often characterized by increasing $l^2$ weight norms. Our primary tool is the theory of Heavy-Tailed Self-Regularization **HTSR**, where we track the heavy-tailed exponent $\alpha$. We find that $\alpha$ reliably predicts both the initial grokking transition and subsequent anti-grokking. We benchmark these insights against four prior approaches: progress measures---Activation Sparsity, Absolute Weight Entropy, and Approximate Local Circuit Complexity ---and weight norm ($l^2$) analysis.
Our experiments show that while comparative approaches register significant changes, **in this regime of increasing $l^2$ norm, the heavy-tailed exponent $\alpha$ demonstrates a unique correlation with the ensuing large, long-term dip in test accuracy, a signal not reliably captured by most other measures.**



Extending our zero weight decay experiment significantly beyond typical timescales ($10^{5}$ to approximately $10^{7}$ optimization steps), **we reveal a late-stage catastrophic generalization collapse (``anti-grokking''), characterized by a dramatic drop in test accuracy (over 25 percentage points) while training accuracy remains perfect**; notably, the heavy-tail metric $\alpha$ uniquely provides an early warning of this impending collapse. Our results underscore the utility of Heavy-Tailed Self-Regularization theory for tracking generalization dynamics, even in the challenging regimes without explicit weight decay regularization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the phenomena of Grokking and Generalization Collapse (Anti-Grokking) from the perspective of HTSR theory. The authors find that the parameter $\\alpha$ in the HTSR theory, which characterizes the empirical spectral distribution (ESD) of the parameter matrix’s eigenvalues, can predict both grokking and anti-grokking behavior. They further propose the concept of *Correlation Traps*—abnormally high eigenvalues that lie outside the MP (Marchenko–Pastur) fitting curve after element-wise random shuffling of the parameter matrix—which are considered to be the cause of anti-grokking. The proposed indicators show stronger correlation with grokking than existing metrics such as weight norm, weight entropy, and activation sparsity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The authors introduce the phenomenon of anti-grokking and use the $\\alpha$ parameter and correlation traps for its prediction and explanation, offering a novel perspective to existing work on grokking.\n2. The paper clearly defines the mathematical tools used and explains them with helpful figures, making the presentation easy to follow.\n3. The experimental results support the claim that $\\alpha$ and correlation traps can predict grokking and anti-grokking.\n\n**Weaknesses:**\n1. The paper offers only empirical observations and lacks theoretical depth. Although correlation traps are proposed to explain anti-grokking, the evidence is limited to empirical correlation, without theoretical justification. Please refer to the *Questions* section for further discussion.\n\nQUESTIONS:\n1. Why does a lower $\\alpha$ indicate stronger fitting ability to the training set? Could analyses of gradient or parameter matrix singularity—such as those based on Neural Tangent Kernel (NTK) theory—help in understanding this relationship?\n2. Why do outliers emerge outside the Marchenko–Pastur (MP) bulk after element-wise random shuffling of parameter matrices in the overfitting phase? Could this phenomenon be theoretically justified by making assumptions about the singular values of the original parameter matrix?\n\nI would be happy to increase my score to 4 or 5 if authors could kindly provide more theoretical insights.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the relationship between generalization and the power-law exponent $\\alpha$ in the empirical spectral density (ESD) of network weight matrices in a single setting where grokking occurs. HTSR theory predicts that $\\alpha = 2$ is the ideal value for networks to generalize. When training 3-layer ReLU MLPs on MNIST without weight decay (the authors scale the weights by 8 to induce grokking, as was introduced in (Liu et al. \"Towards Understanding Grokking...\" and \"Omnigrok...\"), the authors find that their network's generalization eventually improves after a prolonged period of training, but then worsens with continued training. They find that the era of training where $\\alpha \\approx 2$ roughly corresponds with the maximum test accuracy, suggesting that HTSR theory can be applied to understand grokking.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe main result of the paper is interesting: the dynamics of $\\alpha$ throughout training correspond with how the network generalizes in the manner that HTSR theory would predict. This is an interesting proof of concept, and warrants further investigation.\n\n### Weaknesses\n* The relationship between $\\alpha$ and generalization is only tested in a single setting. While in Appendix C the authors also test the case where weight decay is nonzero, they do not study the relationship between $\\alpha$ and grokking on any other tasks where grokking occurs, like the algorithmic tasks where grokking was initially observed. While this result is interesting, it is difficult to tell whether this finding is robust.\n* Numerous small misspellings and grammatical and formatting errors that make the paper feel thrown-together. The most surprising is that the caption for Figure 4 includes \"Top\", \"Middle\", and \"Bottom\" descriptions for subfigures that don't correspond with the actual arrangement of the subfigures on that plot.\n\nQUESTIONS:\n* In my view, in order for this article to warrant acceptance to the conference main track, it would have to more thoroughly investigate the relationship between the $\\alpha$ and generalization in grokking. In particular, the authors should conduct experiments where various parameters are varied that affect the timing of generalization in grokking, and then see whether the $\\alpha$ dynamics also vary in the way we'd expect. Some suggestions:\n  - Change the large initialization scale of the parameters in the network to vary grokking timing. Instead of scaling the weights and biases by 8, one could vary this scaling factor, which might change the timing of generalization in the networks. Does $\\alpha$ approach $\\approx 2$ when test accuracy is maximized across each choice of initialization scale?\n  - Sweep the weight decay parameter. In Liu et al. (2022) (Omnigrok), this directly changes the timing of generalization in their MNIST networks. Does the timing at which $\\alpha$ reach the optimal region $\\approx 2$ change accordingly?\n  - Other networks. The easiest would be to study small transformers or MLPs on algorithmic tasks like modular addition.\n\nLIMITATIONS:\nYes. The authors mention the main limitation I identified above, that their experiments are only on a single setup.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the grokking phenomenon, where under training neural networks generalize suddenly after extensive overfitting, on the task of classification with a 3-layer MLP trained on a 1,000-sample subset of MNIST. The authors identify three distinct phases: an initial “pre-grokking” phase, the familiar grokking generalization phase, and a newly discovered “anti-grokking” phase late in training, where test performance degrades again. They analyze these phases through the lens of Heavy–Tailed Self-Regularization (HTSR) theory, revealing how the heavy-tail power-law (PL) exponent $\\alpha$ reflects the transition dynamics between these phases.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The idea of applying HTSR theory to study the grokking phenomena in training neural networks is interesting and new.\n2. The study is able to investigate and signal the grokking phenomenon without weight decay, which previous studies typically relies on, providing new characterizations of such a phenomenon.\n\n**Weaknesses:**\n1. The scope of the dataset and architecture is limited to some toy setups. To demonstrate the universality of the proposed method, more experiments on different tasks, datasets, and architectures are necessary. \n2. The HTSR theory often relies on simplifying assumptions (e.g., high-dimensional asymptotics). The degree to which these align with the experiments is not deeply probed.\n\nQUESTIONS:\n1. To identify the correlation traps, the paper proposes to look into the randomized version $\\mathbf{W}^{\\mathrm{rand}}$ of the weight matrix and compare it to the MP distribution. Can you provide more details on how the randomization is performed? It seems that the corresponding pipeline is not mentioned in the paper.\n2. Inconsistent captions with respect to the figures in Figure 4.\n3. What is the difference between the anti-grokking phase and the standard overfitting phenomenon or double descent phenomenon in training neural networks? Are there any tricks, such as early stopping tailored to HTSR signals, weight decay schedules, or data augmentation, that could mitigate the anti-grokking? \n4. Why does the experiments only use a small subset of the MNIST dataset as the training dataset? (Far smaller than the test split size) This is somehow strange.\n\nLIMITATIONS:\nPlease see the Questions part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors study the **Grokking phenomenon** — a behavior in which training accuracy reaches a high level early, while test accuracy remains low until it suddenly improves after many additional training steps. They investigate this using a ReLU network trained on a subset of the MNIST dataset. By significantly extending the number of training steps, they observe a new behavior, which they refer to as **Anti-Grokking**. They further show that the **HTSR** metric serves as a good indicator of all three phases of Grokking behavior.\n\nSTRENGTHS AND WEAKNESSES:\nI find the relationship between the different training phases and the HTSR value to be very interesting. In particular, showing that other commonly used metrics — such as weight norm or activation sparsity — fail to capture the Anti-Grokking phase is a strong indication that HTSR may be a more informative alternative. That said, since phenomena like Grokking and Anti-Grokking often appear in small or synthetic datasets, there are natural questions about the downstream theoretical and practical relevance of these results. Observing this behavior only on a subset of MNIST makes it difficult to conclude whether the patterns are dataset-independent or more broadly applicable.\n\nQUESTIONS:\n### Questions for the Authors\n\n1. Why did you focus only on the MNIST dataset? Are there computational limitations preventing the investigation of this phenomenon on larger or different datasets, or do Grokking and Anti-Grokking behaviors simply not appear as clearly in other domains?\n\n2. Given that the study is limited to MNIST, how can we be confident that the observed relationship between the HTSR value and the three phases of Grokking is **independent of the dataset**?\n\n3. *(Minor)* In **Figure 2**, please specify what each axis represents. This would make the visualization much easier to interpret on its own.\n\n4. As you briefly mentioned in the limitations section, is there any theoretical insight or justification for **why** HTSR serves as a good indicator of the three phases? Even an informal intuition would help strengthen the argument.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper contains a numerical study of grokking from the point of view of heavy-tailed self-regularisation (HTSR) theory. This theory posits that the spectral density of weights in a deep neural network is key to understanding grokking, and puts the power-law exponent of this spectrum, $\\alpha$, at the centre of these experiments.\n\nThe key result of the paper is a strong correlation between three phases of grokking (pre-grokking, grokking, anti-grokking) and the exponent $\\alpha$. The main result is that $\\alpha$ tracks progress of grokking well and is better than prior progress measures at detecting anti-grokking (i.e. loss of generalisation after extensive training).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The exponent $\\alpha$ correlates quite well with grokking. The existence of a threshold makes for a very straightforward stopping criterion.\n* The concept of correlation traps is interesting, and the algorithm to detect them (page 5) seems simple and effective. The connection with anti-grokking in Table 2 is quite compelling.\n\n**Weaknessess**\n* The results in this paper are exclusively descriptive and correlational. Although HTSR seems to be supported by some prior work (which I admit I'm not familiar with, and is not presented in sufficient depth in the paper), the results presented specifically in this paper do not have any theory behind them. This calls into question where $\\alpha$ plays a causal role in grokking or if there is another, more fundamental factor at play.\n* The experiments are extremely limited. The paper only explores one particular (and very small) architecture, trained just on MNIST (surprisingly with no mention of the dataset or task in the main text), and with one regularisation method.\n* Perhaps the one key claim of the authors (that $\\alpha$ tracks generalisation collapse while other metrics don't) is not supported by enough evidence. Judging visually from Fig. 5, it seems that at least some of the measures (e.g. L2) do detect collapse, in the sense that one could very easily put a threshold (e.g. 125) and declare loss of generalisation when L2 exceeds that threshold.\n* The literature on grokking and generalisation is growing very quickly, and the Related Work does not do justice to it.\n\nQUESTIONS:\n* Why is $\\alpha=2$ the \"ideal value\" that \"corresponds to fully optimized layers\"? This is a very strong claim with no theoretical support in this paper. If there is a theoretical argument in previous work, this should be explained here as well for completeness.\n\n* The results in this paper are mostly correlational, linking fitted $\\alpha$ values with phases of grokking. Is there any evidence that this connection is causal? E.g. can you perturb a given network to have a different $\\alpha$ value and see if its train/test accuracy changes accordingly?\n\n* Does this apply to other architectures and regularisation methods?\n\n* What is the role of the Gaussian baseline in page 3? Forgive me if I'm missing something obvious, but it doesn't seem to play any role elsewhere in the paper.\n\nLIMITATIONS:\nThe paper includes a decently-sized Limitations section that covers the key weaknesses of the work. The paper also has several minor typos, formatting errors, and incomplete words and sentences.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}