Title: Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning

generalization

classification
Abstract: Robust validation metrics remain essential in contemporary deep learning, not only to detect overfitting and poor generalization, but also to monitor training dynamics.
In the supervised classification setting, we investigate whether interactions between training data and model weights can yield such a metric that both tracks generalization during training and attributes performance to individual training samples.
We introduce Gradient-Weight Alignment (GWA), quantifying the coherence between per-sample gradients and model weights.
We show that effective learning corresponds to coherent alignment, while misalignment indicates deteriorating generalization.
GWA is efficiently computable during training and reflects both sample-specific contributions and dataset-wide learning dynamics.
Extensive experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples, providing a validation-set-free approach for model analysis directly from the training data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes gradient-weight alignment (GWA), a metric to quantify generalization. GWA per sample is computed as cosine similarity between model weights and gradients- over datasets it is computed as a kurtosis-corrected average of per-sample GWA. The authors then propose an efficient way to estimate GWA online. Experimentally, the authors demonstrate that using GWA as an early stopping criterion outperforms baselines. The authors also conduct several other analyses showing other properties of GWA, including that it can predict label noise in a dataset.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is overall well-written and presented. Figure 1 is a nice illustration. Experimentally, the authors consider several large-scale datasets and architectures, which is a good choice. Notably, the authors also show several analyses of GWA in Section 4.\n\nOn the negative side, while the proposed method is theoretically motivated, the theory is from prior work; there is no new theory presented in this work. Moreover, the concept of gradient-weight alignment is itself not particularly new, although the particular formulation in equation 2 and the scalable estimator are. Given the limited novelty on the theory or conceptual side, the empirical results are very important. Unfortunately, there are several deficiencies in the experiments in my view:\n\n- Most importantly, GWA is compared to only one other baseline in Table 1 (LabelWave). I encourage the authors to add at least 2 other baselines such as the curvature-based methods mentioned in Section 2. I understand that these may be more computationally expensive (they are often not designed to be online methods), but including these is still critical in my view. There is no need for GWA to outperform these baselines either, especially the non-online ones.\n- Another important empirical concern is that GWA, and even sometimes LabelWave, outperform the validation set. Over many trials, this shouldn't happen on average: assuming iid data, the validation set is as close as one can get to actual generalization performance. This needs an explanation.\n- Relatedly, there are no error bars or standard deviations reported for all the key results in the main paper. This is important to include so the statistical significance of the results is clear.\n- As far as I can tell, there is also no ablation of whether the kurtosis correction in equation 2 is empirically important.\n- There is also no comparison of how the estimated GWA compares to the ground truth GWA.\n\nI don't mean to discourage the authors, but given that the paper is primarily an empirical one, these empirical details are quite important. If they can be addressed, I think this can be a solid contribution.\n\nQUESTIONS:\n- Are there any theoretical results the authors can derive about GWA?\n- How does GWA compare with additional baselines (such as curvature-based methods)?\n- Why do GWA and LabelWave sometimes outperform the validation set\n- What is the statistical significance of the key results in the paper?\n- Empirically, how does removing the kurtosis correction in equation 2 change performance?\n- How does the GWA estimate of equation 3 compare with the ground-truth GWA?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Gradient-Weight Alignment (GWA), a novel metric that measures the coherence between per-sample gradients and model weights. During training, the peak of this metric closely corresponds to the peak of test accuracy. Notably, GWA can serve as an effective early-stopping criterion even in the absence of a validation set. The authors demonstrate that early stopping using GWA outperforms both traditional validation set baseline and the recently proposed LabelWave method. Furthermore, the gradient alignment approach facilitates the assessment of training data quality: higher levels of noise in the data result in lower GWA scores and a broader distribution of alignment values. Individual alignment values also help to detect mislabeled training examples.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper introduces an early-stopping metric that can be computed efficiently during training, without requiring a validation set.\n2. The GWA metric also enables analysis of training data quality, such as detecting mislabeled examples.\n3. Experiments cover both training from scratch and fine-tuning scenarios.\n4. The paper is clearly written and easy to follow.\n\n**Weaknesses:**\n1. The experimental evaluation misses comparisons to several other methods that also explore stopping criteria without validation sets [1, 2, 3, 4, 5, 6]. In particular, [5], which measures disparity between different stochastic gradients, is closely related to GWA. These papers should be at least discussed in the related work section and ideally included as baselines.\n2. The GWA metric appears overly complex, and the authors do not provide enough intuition or explanation for why this complexity is necessary (see questions for more details).\n3. Although GWA seems broadly applicable across data types and training tasks, the experiments are limited to image classification only.\n\n**References:**\n\n[1] Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. Early Stopping without a Validation Set, 2017.\n\n[2] Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does Early Stopping Help Generalization against Label Noise?, 2019.\n\n[3] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Robust Learning by Self-Transition for Handling Noisy Labels, 2021.\n\n[4] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and Improving Early Stopping for Learning with Noisy Labels, 2021.\n\n[5] Mahsa Forouzesh and Patrick Thiran. Disparity Between Batches as a Signal for Early Stopping, 2021. \n\n[6] Ali Vardasbi, Maarten de Rijke, and Mostafa Dehghani. Intersection of Parallels as an Early Stopping Criterion, 2022.\n\nQUESTIONS:\n1. Is GWA actually better than the early stopping criteria mentioned in the p. 1 Weaknesses?  \n2. Is it truly necessary to adjust the expected alignment by the excess kurtosis to obtain a reliable generalization proxy? The paper does not include any comparison plots between the behavior of GWA and the pure expected alignment $\\mathbb{E}_i [\\mathcal{A}_T]$.\n3. Why do the authors measure the alignment of gradients with $w_T$ (the final weights) instead of $w_T - w_0$, where $w_0$ is the initial weight vector? Intuitively, aligning gradients with $w_T - w_0$ reflects whether per-sample gradients are coherent with the shift from the initialization point. A higher alignment would suggest meaningful learning is still occurring, while a lower alignment might indicate that optimization moves along a different direction, possibly signaling overfitting.\n4. Could you please explain how the formula for the gradients of the final layer’s weights (given on line 153) is derived? Also, I understand that calculating gradients only for the final layer reduces the computational cost of GWA, but would the metric behave differently if gradients from all layers were considered? \n5. Is it possible to apply GWA as an early stopping criterion in domains other than images and for tasks beyond classification?\n\nLIMITATIONS:\nAlthough the authors clearly acknowledge the limitations related to the computation of GWA, they do not address what I consider a major shortcoming of the work: the limited number of baseline methods considered (see p. 1, Weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a metric - *Gradient-Weight Alignment* (GWA) that aims to provide an understanding of the training dynamics and its impact on generalization. Specifically, the paper explores utilizing GWA for early stopping. The paper also presents the robustness of GWA by utilizing the early stopped model on a perturbed test set. Experiments are performed on standard computer vision datasets and ViT and ConvNeXt models to asses the metric.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**:\n- The primary strength of this work is that it proposes a validation-set-free method for monitoring training and selecting a model for deployment. This is a significant practical advantage, particularly in data-scarce scenarios where withholding a validation set is costly. The method provides a promising alternative to the standard practice of using a hold-out set.\n- The paper addresses the important and challenging problem of understanding the training dynamics of deep neural networks without relying on computationally expensive second-order methods, such as Hessian-based analysis.\n\n**Weaknesses**:\n\nHowever, it faces a few issues that need to be addressed before being ready for publication. \n- **Insufficient Comparison and Positioning with Relevant Work:**\n    - The paper should be more clearly positioned with respect to Training Data Attribution (TDA) methods and influence functions. While some works are cited , the paper dismisses them as post-training analysis  without a deeper comparison. More recent and relevant methods like Datamodels [1] and approximate unrolling [2] are not discussed.\n    - The methodology, which relies on the alignment of gradients with the final layer weights, is closely related to the well-established phenomenon of Neural Collapse [3]. This connection is a critical omission. The paper should explicitly discuss this relationship and differentiate GWA from other metrics derived from Neural Collapse properties, such as the information-theoretic HDR and MIR metrics [4], which also measure alignment to analyze generalization.\n\n- **Unresolved Issues with the Metric's Formulation:**\n    - The definition of the per-sample alignment score, $\\gamma(x_i, w_T)$, becomes problematic when a sample is perfectly classified. For a cross-entropy loss, a perfect classification results in a gradient that is zero or near-zero, making the cosine similarity term $\\frac{g_T(x_i) \\cdot w_T}{\\Vert g_T(x_i)\\Vert \\Vert w_T \\Vert}$ undefined due to a zero-norm gradient vector. This is a fundamental issue, as many samples will be learned perfectly during training. The paper's claim that alignment approaches 1 (line 106) for perfectly classifiable data  seems to overlook this practical and theoretical problem.\n    - This concern is reinforced by the paper's own efficient gradient calculation, in line 153, which explicitly shows the gradient becomes a zero vector if $\\hat{y}_i=y_i$. \n\n- **Claims Not Substantiated by Experimental Evidence:**\n    - The paper repeatedly claims that GWA is computationally efficient and scalable. However, no empirical evidence (e.g., wall-clock time, FLOPS) is provided to compare its computational cost against the baselines it aims to replace or improve upon, such as validation set evaluation or LabelWave\n    - In Section 4.2, the paper states it will \"investigate whether GWA remains consistent across multiple model initializations\". However, the experiments only report results averaged over three runs, with no analysis or visualization of the variance or consistency across these different initializations.\n\n**References**\n\n[1] *Ilyas, Andrew, et al. \"Datamodels: Predicting Predictions from Training Data.\" Proceedings of the 39th International Conference on Machine Learning. 2022.*\n\n[2] *Bae, Juhan, et al. \"Training data attribution via approximate unrolling.\" Advances in Neural Information Processing Systems 37 (2024): 66647-66686.*\n\n[3] *Papyan, Vardan, X. Y. Han, and David L. Donoho. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" Proceedings of the National Academy of Sciences 117.40 (2020): 24652-24663.*\n\n[4] *Song, Kun, et al. \"Unveiling the Dynamics of Information Interplay in Supervised Learning.\" International Conference on Machine Learning. PMLR, 2024.*\n\nQUESTIONS:\nTo improve the paper's clarity and rigor, please address the following:\n1. How is the per-sample alignment score $\\gamma(x_i, w_T)$ handled numerically when the gradient $g_T(x_i)$ is a zero vector, which should occur for samples that are perfectly classified?\n2. Could you provide a quantitative comparison of the computational complexity and runtime overhead of GWA versus LabelWave and standard validation?\n3. How robust is GWA to different model initializations and variations in model size within the same architecture (e.g., ViT-Small vs. ViT-Base)? The paper claims to investigate this but provides no direct evidence.\n4. What data augmentation transforms were used during training? Could these transforms, especially aggressive ones, affect the stability and interpretation of the GWA metric?\n5. The rationale for Figure 3 is not entirely clear. While it's expected that lower data noise leads to better accuracy and alignment, how does this demonstrate the reproducibility or consistency of the metric itself? Please clarify the intended message.\n6. Could you provide more intuition on why the higher-order moments in GWA (i.e., the kurtosis correction) make it more sensitive to overfitting than traditional validation or methods like LabelWave, as suggested by the results in Figure 2?\n\nLIMITATIONS:\nThe authors state in the checklist that limitations are discussed, but this is not the case. The paper lacks a dedicated discussion of its limitations. Key limitations to address include the potential numerical instability of the GWA metric (the zero-gradient issue), the strong assumption that last-layer gradients are a sufficient proxy for full-model gradients, and potential failure modes of the method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has argued that the alignment of gradients and model parameters is an indicator of convergence of a deep neural network (DNNs). This paper proposed an efficient estimator for this alignment by measuring the alignment per sample, and shows empirically that it can serve as a useful signal when determining when to halt optimization (early stopping) and selecting influential training samples. The estimator is made efficient by focussing on the gradients of the last layer, and computing it in batches (during an epoch).\n\nSTRENGTHS AND WEAKNESSES:\nI believe this paper studies an important research question (how can we measure the generalization of a DNN without a validation set) and makes an clear contribution to it. It contains an extensive set of experiments to empirically support the usefulness of the introduced estimator. I believe the paper is fit for acceptance, although it still has some weaknesses in terms of the theoretical support for the estimator.  Specifically the design choices to focus on the last layer and computing it in batches.\n1. The authors write: *'we exploit the fact that classification fundamentally operates on latent representations. This reduces irrelevant variations'*. While somewhat intuitive, it is not entirely clear to me why the gradient alignment in layers previous to the last layer constitutes of irrelevant variations. It would be good if the authors could either show some connection between alignment in the last layer and alignment in all other layers (which would justify focussing on it), or show in a small-scale experiment that little is gained from measuring alignment on all layers. \n2. The authors write compute the alignment per batch, while in between batches the parameters. I understand that this has an advantage in terms of computational resources, but it would be good to know what the costs are in terms of accurately estimating the alignment. Particularly, it would be good to theoretically analyse how the difference between $E[A_T]$ and $E[\\hat{M}_T^1]$ (the bias of the estimator for the first moment) depends on the learning rate. Intuitively, if there are big steps in between batches, this bias will increase. **To be clear**: I do not think this makes the estimator not useful in practice, but it would be good to have an idea about the conditions under which it yields a good estimate of the alignment.\n\nQUESTIONS:\nThe inclusion of kurtosis in the estimator is motivated by the idea that rare/atypical samples have an outsized influence on the model, and that the kurtosis can account for this. Do the authors also have empirical support for this argument? It would be good to show that if one uses simply the first moment, that the estimator becomes worse for certain objectives (for instance in terms of determining when to halt optimization).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce a new metric called gradient-weight alignment that can be considered as a novel proxy for generalization performance during training, which helps to indicate the most generalized model during training, especially with noisy data. This paper first show per-sample gradient scores aligns well with the generation performance during the training trajectory. The motivation is that the gradient would consistently point in the same direction as the model weights given enough training. However, calculate the per-sample grandient alignment is not cheep given that the neural networks are usually quite heavy and with grate amount of training data. As a result, this paper introduce an estimate that using the alignment with the last linear layer and accumulate the scores during the training epochs. This paper shows that for image classification task, GWA can be a good indicator and achieve better performance on noisy dataset.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is easy to follow and well motivated. The experiment results are good especially for the noisy training dataset and finetuning tasks.  It seems that this method has very strong limitation since it has to be used for classification models due to the size of the weight. However, is it possible to extend this method to other embedding models? For example, self-supervised tasks such as contrastive learning. Also, does this method hold for other optimizer or training strategy such as sharpness-aware-minimization or adversarial training. For classification tasks, we also have many other optimization settings, such as few-shot learning, meta-learning. Or even many NLP tasks are based on the embeddings to get the prediction in the token space. For those cases, will it still hold? The cases this paper shows are quite limited.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}