Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
Authors: Florian A. Hölzl
Daniel Rueckert
Georgios Kaissis
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
generalization
classification
Primary area: general_machine_learning

Abstract: Robust validation metrics remain essential in contemporary deep learning, not only to detect overfitting and poor generalization, but also to monitor training dynamics.
In the supervised classification setting, we investigate whether interactions between training data and model weights can yield such a metric that both tracks generalization during training and attributes performance to individual training samples.
We introduce Gradient-Weight Alignment (GWA), quantifying the coherence between per-sample gradients and model weights.
We show that effective learning corresponds to coherent alignment, while misalignment indicates deteriorating generalization.
GWA is efficiently computable during training and reflects both sample-specific contributions and dataset-wide learning dynamics.
Extensive experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples, providing a validation-set-free approach for model analysis directly from the training data.

Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks
Florian Hölzl, Daniel Rueckert, Georgios Kaissis
Institute for Artifical Intelligence in Medicine
Technical University of Munich
(florian. [llorianhhoezz,, daniel. rueckert, g kaissis lrciiss@tuunn @ttmm de
Abstract
Robust validation metrics remain essential contemporary deep learning, not only to detect overfitting and poor generalization, but also to monitor training dynam- ics. In the supervised classification setting, we investigate whether interactions between training data and model weights can yield such metric that both tracks generalization during training and attributes performance to individual training samples. We introduce Gradient- Weight Alignment (GWA), quantifying the coher- ence between per-sample gradients and model weights. We show that effective learning corresponds to coherent alignment, while misalignment indicates deterio- rating generalization. GWA is efficiently computable during training and reflects both sample-specific contributions and dataset-wide learning dynamics. Extensive experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples, providing a validation set-freee approach for model analysis directly from the training data.
Weiiih
Introduction
Despite the recent surge in self-supervised learning, optimization with cross-entropy loss remains dominant in modern deep learning for both supervised training and fine-tuning, Its enduring popularity stems from its strong label-based learning signal and implicit modeling of predictions as probabilities. However, this very strength relying on maximum likelihood estimation introduces vulnerabilities to overconfidence and sensitivity to noise in both input data and labels. Diagnosing these issues typically relies on hold-out validation sets, which operate under the assumption of independent and identically distributed (i.i.d.) data. While effective, this standard approach necessitates labeled data that is rendered unavailable for training and offers limited insight into how these issues be attributed to training set samples. This motivates a critical question: can we effectively assess model generalization and diagnose potential problems solely using information available during training?
Prior work indicates that robust generalization emerges when all training samples contribute coher- ently towards a shared learning goal, that is, their gradients are directionally well aligned [1, 2]. Conversely, conflicting or misaligned gradient directions indicate a potential failure to generalize. However, computing pairwise gradient alignment is highly memorry-ntennsii distributional quantity; in other words, the average gradient alignment over the dataset provides the
and is not inherently
minimal insight into individual samples' contribution training. In this work, we thus alignment between per-sample gradients and the model weights as a measure for estimating key training dynamics and predicting generalization in training with cross-entropy. This quantity, which we refer to as Gradient- Gradient-Weight Alignment (GWA), as well as its distribution, capture the degree of gradient coherence across the dataset and allow linking model performance directly to the individual underlying input-label pairs (Fig. 1). Moreover, we show that GWA can be efficiently estimated dur- ing training (online), providing insights into training dynamics even during large-scale optimization with negligible overhead. We argue through extensive empirical evaluation that GWA accurately
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Initialization INCOHERENT
Optimal Stopping COHERENT
Overfitting
22
INCOHERENT
Traie
74
44
VC
VL
Tisis
Treie Accuracs
Gap
Gradion Wrigh Aligameet
Figure I: Gradient alignment among individual samples V L; as well the model weights varies during training. with coherent per-sample gradient direction reflecting generalization. Line plots illustrate how GWA captures gradient coherence and model performance at different time points t.
identifies the point in training beyond which further training steps yield diminishing returns in terms of generalization performance, rendering held-out validation sets redundant. Our contributions can be summarized as follows:
We introduce GWA as novel proxy for generalization performance during training effectively replacing the need for withholding separate validation set.
GWA reveals the influence individual training samples on optimization, providing a powerful diagnostic tool for understanding data quality issues like outliers and label errors.
We show GWA scales to large-scale training and finetuning, and remains robust under input/label noise, offering a superior criterion for choosing models deployed in practice.
Related Work
Quantifying the generalization gap understanding how well model performs on unseen data is central challenge in deep learning [3]. Early approaches without hold-out data focused on estimating this gap using the curvature of the loss function, offering insights into sample influence. However, computing second-orded derivatives is computationally expensive [4, 5, 6, 7]. and curvature can be unstable during training and sensitive to hyperparameters [8, 9, 10]. More recently, influence functions[1] ]] and sub-sampling estimators [12. 13] have emerged to compute per-sample influence at end training without second-order derivatives. Our work differs by focusing on an efficient train- time estimator of generalization for online assessment rather than post-traine pest-training influence analysis.
We contend that comprehensively evaluating and characterizing model dynamics is crucial for understanding optimization and its challenges- such as potential overfitting. Prior work has observed that Stochastic Gradient Descent (SGD) exhibits a "simplicity bias". initially learning simple patterns before fitting increasingly complex functions [14, 15, 16, 17, 18. 19]. While we build on these insights, we move beyond characterizing what is learned to quantifying when further learning yields diminishing returns for generalization regime [20, 21J. Most notably from this area, we compare against the recent Labe/Wave [22, 23] which quantifies model prediction changes to determine a suitable early stopping point. Prediction-change- focused methods like Labe/Wave are designed primarily for carly stopping in noisy optimization, but offer limited insight into underlying training dynamics. In contrast, our GWA directly reflects these dynamics via a sample-level measure.
relatively underexplored area outside the noisy label
Quantifying the coherence of per-sample gradients captures training dynamics such as faster con- vergence and improved generalization [24, 1, 25, 26, 27]. As illustrated Fig. 1, this coherence is captured by the direction of the gradients, a signal distinct from their magnitude. However, computing coherence in the aforementioned approaches requires the gradients all training samples to stored in memory impractical limitation for large datasets and often provides only aggregate statistics, obscuring valuable per-sample insights. Gradient Disparity (GD) [28] provides a more efficient approach for k-fold cross validation but has not been shown work in large scale settings. Our

---

approach is scalable and addresses these limitations by leveraging the model weights as a reference vector. Theoretical work demonstrates that under ideal conditions -perfectly classifiable data- weights converge in direction, i.e., maintain specific direction while the gradient aligns with the weights' direction [29, 2]. While the impact of noise in this case is still unexplored, some levels of noise can be beneficial [30, 31], and recent empirical research affirms directional stability even with stochasticity by analyzing batch gradient direction relative to the optimal weights [32]. Our proposed GWA specifically focuses quantifying alignment in the presence of realistic data variance. Moving from pairwise and aggregate gradient statistics to GWA is not only more efficient but allows us to link individual samples to generalization performance in stochastic optimization.
In the following, we propose an efficient computation strategy of per-sample gradient-weight align- ment and leverage the properties of the resulting alignment distribution as a novel measure of generalization and sample-level influence.
3 Gradient-Weight Alignment
We begin by introducing GWA, the key quantity studied in our work. GWA is inspired by theoretical work on the directional convergence of model weights Iearned by gradient flow when minimizing the cross-entropy loss [2]. Intuitively, the fundamental idea of this work is that, for perfectly classifiable data, the weights not only converge in direction, but moreover the corresponding gradients converge in direction to the weights, i.e., the gradient and weights align.
The central hypothesis of our work is therefore that in pructice the alignment between per-sample gradients and model weights can be leveraged capture the convergence and consequent generalization of a model. Differences in per-sample alignment link the model's performance to the individual data samples. This inherently requires studying GWA through two complementary lenses: (1) the per-sample alignment scores, which quantify how well individual samples of a real-life, noisy dataset are represented in the general optimization trajectory and (2) distributional measure of the alignment scores across the dataset, which reflects the degree to which dataset properties (e.g., data diversity, variance, etc.) impact the resulting generalization across training iterations. We first formally define GWA and its related quantities.
Motivation
Definition (Per-Sample Alignment) Lei -V xx) denote the negative gradient of the loss function with respect to the model weights wT al single input-labe pair (1i,yi). where we drop ys for brevity, and at epoch T. Then, the per-sample alignment score is defined as:
gr WT
Mas, w = cos sim (gr(x;). Wr) ||gr(<;) (I:)]l]]w
(1)
The per-sample alignment score 7(x, wr) effectively measures how well the model weights align with the "learning direction" for that specific instance. Intuitively, higher score suggests that the model is more efficiently incorporating information from that sample into its weight updates. The theoretical justification for the per-sample alignment score is derived from the fact that, under ideal conditions of perfectly classifiable data, the predicted probabilities asymptotically approach the target and alignment should satisfy E, [x(, Wr)] + 1 for large enough T [2]: in other words, the gradient would consistently point in the same direction as the model weights. While perfect convergence is rarely observed with real-world datasets due to inherent noise and complexity, this theoretical limit provides a valuable intuition: a model with better generalization performance should exhibit higher average (zz, Wr). Conversely, consistently low alignment scores, i.e., updates orthogonal to the direction of the model weights, can signal issues like noisy labels or learning non-general, sample specific information and potential overfitting.
Definition (Gradient-Ve Weight Alignment) Let :=
= (1(2,, - : (I,, E [-1, be the set of per-sample alignment scores at epoch T. Moreover, let AT denote the empirical distribution of the values of GT and Mf its t moment. Then, the moudient-weight alignment (GWA) at epoch T is defined as the scces--urtosis-ccrreeee expectation of Ar:
E6 [A[[A]] Kurt,[Ar) + B
M (1)
GWAT =
(2)
M (M 3 -3+2
3

---

Above, 3 is added to ensure non-negativiyy of the denominator. We choose 8 = 1.2 to offset the excess kurtosis Kurt of the uniform distribution over [-1, 1], which is limiting case never to be expected in practice.
Unlike theoretical work that assumes perfectly classifiable data, real-world datasets can exhibit high degrees of variance and noise. The distribution of per-sample alignment scores 7(2, wr) provides valuable insights into the quality of learning a coherent alignment distribution (i.e., high GWA) indicates consistent gradient directions across samples and thus effective generalization. At this point, one may wonder why GWA considers not only the mean but also the kurtosis of the alignment distribution. The theoretical motivation for incorporating the kurtosis, a measure of tailedness of the distribution, in other words, of the impact of rare samples, is derived from the long tail theory of deep learning [12, 13]. This line of work demonstrates that, in natural image tasks, rare/atypical samples have an outsized influence on the model. Distributions in which rare samples exert high influence are commonly referred to as "heavy-tailedd" distributions and have high kurtosis. In other words, high kurtosis value in the GWA denominator intuitively indicates a large proportion of samples with disproportionate influence on the overall alignment, thus signaling potentially problematic learning patterns by diminishing the GWA value.
Note that we choose the value of the 3 factor such that the kurtosis has only minimal impact on GWA when the distribution is (truncated) Gaussian (Kurt;[Ay Ps 0). For more concentrated (platykurtic) distributions with lower kurtosis (e.g., approaching a uniform). the GWA value increases, and for distributions with high kurtosis (leptokurtic, e.g., Laplace) the GWA value decreases. We will use the term (highly) coherent synonymously with high GWA. As we show experimentally below, tracking GWA over time reveals critical training dynamics related to overfitting vs. generalization.
Scalable Estimator Capturing the per-sample alignment properties during training on large datasets necessitates scalable GWA estimation approach. However, directly computing Y(2; wr) for all samples at each timestep computationally prohibitive. We address this by leveraging inherent properties of supervised classification models and GWA to design lightweight estimator with minimal overhead that can be run online even for very large models and datasets.
The first primary obstacle to GWA estimation lies in the sensitivity of the cosine similarity term in Eq. (I) to high dimensionality of the arguments. Recall that the expected cosine similarity between vectors with independent noise components diminishes rapidly with increasing dimensions. Moreover, computing full network gradients at every step introduces significant implementation and computational overhead. To mitigate both aforementioned issues, we exploit the fact that classification fundamentally operates on latent representations. A deep classifier's 's primary goal is to learn a representation that is linearly separable by its final layer, with the last layer offering the most direct signal of the learned task [33]. In fact, for our case, including earlier layers degrades the estimator, with gradients in shallower layers being significantly more unstable, as shown in work such as [34]. Consequently, we propose estimating per-sample gradients using only the final layer's weights i.e., without materializing the full model's gradient. This transforms gradient computation into an efficient matrix multiplication gz(,,) = -4 (fi y T with latent representation 2,, logits 9r target yı. and the weights of the linear head, based on the technique proposed in [35] and shown in [36].
A second challenge arises in determining when to measure per-sample alignment. Computing it for every sample at each timestep incurs substantial overhead despite the efficiency improvements outlined above. We address this by proposing computationally efficient estimator of GWA: in- tuitively, instead of recomputing alignment scores across all samples at a fixed gradient update step, we compute them over all gradient update steps of one epoch as follows. Let T denote the current epoch, b the batch size such that K N/6 is the number of minibatches per epoch and N is the total number of samples in the dataset. Within an epoch, let t E 1........ K - 1} index the steps per epoch and let WT, denote the model weights at the beginning of step t of epoch T. Let BT, = ((xi, v:) = tb + 1...., (L + 1)b} denote the batch for step t. Then, we estimate the ath
Note that the term "heavy-tailedd is formally inapplicable to distributions supported on bounded intervals, but the intuition conveyed is still valid and we thus retain this terminology.

---

central moment of Ar, M as follows:
K-1
M (k)
Σ Σ 1= s.EBr.
(T.. -Mo.)-MM..) M1 -M?
N
where M (recursively) estimates the first central moment (mean). Note that above, the model weights are indexed twice because they change after each gradient update. The estimator of GWA then derived by replacing the true central moments of Ar with the estimated central moments in the RHS of Eq. (2). Under the mild assumption of finite empirical moments and a small-enough learning rate, this estimator furnishes extremely computationally efficient technique monitor the aligmment distribution during training even for very large models and datasets, In essence, estimating GWA this way reduces single forward pass through the network's 's linear classifier. Algorithm summarizes our approach for estimating GWA incorporating the aforementioned techniques.
Algorithm Estimation of GWAr
Require: Total per-epoch iterations K, batch size b, learning rate n), classifier weights W7,0
1: for each iteration O..... K 1 do
2: Sample a minibatch BT, of size from the dataset.
3: Compute standard forward pass with BT,--
4:
for each input-label pair (3i, y:) in By, do
5:
Compute per-sample loss [(w,, Ti. M4), softmax logits Ús, and latents 2,- Compute gradients of linear head in closed form: (x;);) -2 (0i ya) end for
6:
7: Compute and store per-sample alignment: y(2,, wT,e) = (x) WT.
[ |wrall
8:
9:
Update model with minibatch gradient based on step
10:
end for
11: Output GWAT estimated according to Eq. (2) on per-sample alignments stored in step 7.
In the following sections, we demonstrate how GWA can used to predict optimal early stopping, compare model performance across runs, and identify influential training samples all without relying on validation sets.
Evaluation
Our primary comparison to empirically evaluate GWA is against standard validation set-based early stopping, the most common practice for evaluating generalization performance during training. To ensure reproducibility and broad applicability, conduct experiments ConvNeXt-Femnl [37, 38] and ViT/S-16 16 [39, 40] architectures both popular choices offering balance between computational efficiency and accuracy. We leverage established public benchmarks to aid reproducibility, including ImageNet-Ik (using the standard validation set for testing), ImageNet-V [41], and ImageNet- ReaL [42] well as CIFAR-10 and its noisy variant CIFAR-100 10-N [43] to systematically assess performance under varying levels of label noise and to detect overfitting. Our fine-tuning experiments conducted VIT/B-16 model pre-trained on ImageNet-21k [44].
Validation sets are created via a standard train/val split of the original validation data (e.g., 90% training. 10% validation). If no test sets exist, the official validation sets are used as hold-out test sets and are referred as such in the following. All models are trained for a fixed number of optimization steps, i.e., with the same compute budget regardless of training set size. Beyond label noise evaluation, we assess the robustness of models selected using different early stopping criteria on CIFAR-C and ImageNet-C [45], employing realistic input perturbations consistent with our other experiments.
When training a ViT/S-16 implemented in JAX on ImageNet- Ik with single NVIDIA RTX A6000, GWA adds z 2.5sec to the per-epoch wall-clock time (on average 1861 images/s with GWA vs. 1867 images/s without GWA for px4 This is more efficient than evaluating 1% validation set (16sec overhead for one iteration). Computing the flosed-form gradients and the cosine similarity requires & 0.003GFI.OP compared to 4.6 GFL.Pss of single forward pass with a ViT/S- 16. Peak GPU memory when using active dcallocation in this setting is 25. with or without GWA (no
5

---

difference). Thus, GWA has minimal overhead. Detailed hyperparameters used for all approaches are provided in Appendix C. An open-source implementation of our approach in JAX and Py Torch can be found under https///gihuub.oo///////g//
In the following, we will demonstrate GWA's capability to predict optimal early stopping, compare models across runs and provide insights into the underlying training dynamics and the corresponding influence of individual samples effectively replacing traditional validation strategies.
4.1 Early Stopping and Training Dynamics
Table 1: GWA matches or outperforms most validation metrics when used as an early-stopping criterion. Top-1 test accuracy achieved by ViT/S-16 and ConvNeXt trained from scratch on CIFAR- 10(-N) (with varying noise percentages), and ImageNet-Ik Ik using different early stopping strategies (averaged across 3 runs, min-max range below in gray). Performances are reported as difference to baseline validation set with 90/10% train/val split and compared to validation sets 99/ split, prediction changes measured by LavelWave, Gradient Disparity (GD), and our proposed GWA without validation set. Best in bold, second best underlined.
Test Accuracy [%] (á) min max)
VIT
ConvNeXi
CIFAR- 10 [label noise %i]
ImageNet-1 1k
CIFAR 10 [label noise %
ImageNet: Ik
Farly Stop
0%
9%
17%
Val
V2
Real.
0%
9%
17%
Val
V2
Real.
Val Set (10%)
81.10
78.31
75.23
73.01
60.01
79.68
85.33
$2.30
71.24
58.38
78.70
(1.14)
(1.32)
(0.27)
(0:25)
(0:51)
(0.36)
(0:71)
(1.09)
(1.44)
(0:25)
(0.31)
(0.41)
Val Set (1%)
79.99
78.70
74.75
73.46
60.52
$0.14
90.62
86.05
83.01
71.60
58.71
79.01
(1.97)
(1.40)
(0.73)
(0.49)
(0.38)
LabelWave
81.00
78.37
75.02
73.02
60.05
79.66
89.84
85.14
81.15
71.23
58.36
78.70
(1.23)
(0.08)
(0.34)
(1.12)
(1.89)
(0:37)
(0.46)
(0.40)
GD
79.22
77.56
74.66
67.22
54.59
74:25
89.25
84.95
81.71
71.23
58.36
78.70
(1.76)
(1:33)
(13.65)
(13.2)
(12.8)
(1:14)
(1,26)
(0:37)
(0:46)
(0.40)
GWA
81.57
78.93
75.70
73.28
60.53
79.95
89.73
86.08
82.55
71.25
58.62
78.74
(0.91)
(0:23)
(0.53)
(0.13)
(0:10)
(0.56)
(1.25)
(1.15)
(1.24)
Our work proposes using GWA as a trustworthy validation metric during optimization. To this end, we evaluate if GWA provides reliable signal to monitor training progress and determine when training should be stopped. Concretely, we use the optimization step during which the maximum GWA value is reached (after a warm-up period of 10% of the total training steps) as an early stopping point. We conduct a large-scale empirical evaluation of various early stopping criteria across diverse datasets and architectures. We train ConvNeXt and ViT/S-16 models from scratch on CIFAR-10. CIFAR- 1O-N (with varying levels of label noise), and ImageNet. For each dataset, we compare the final test accuracy achieved when employing different early stopping strategies: (1) traditional train/validation splits (90/ 10% and 99/1%), 1%). (2) LabelWave (measuring prediction change per sample) and Gradient Disparity (GD, average pairwise gradient fz-distance), and (3) our proposed GWA. GWA is evaluated in fully supervised setting utilizing the entire training dataset with an additional ablation on train set size in Tab. 5.
Results in Tab. 1 indicate that GWA matches or outperforms most other metrics across datasets and model architectures when used as an early stopping criterion. Specifically, on
N, GWA achieves an on average 0.4% higher test accuracy compared to standard GWA validation splits, and 0.67% over LabelWave. In [28] two early stopping criteria are proposed for GD: the fifth inter-epoch increase, or an increase for 5 consecutive epochs. Both fail completely in our case, with the former criteria stopping consistently too early and the latter criteria not being triggered in most of our experiments (see also Fig. 2). This is also the reason why the test accuracies of LavelWave and GD are identical for XonvNeXt on ImageNet- 1kk as both did not stop early. While the 1% validation set slightly outperforms GWA on the Con,NeXt, GWA beats the 10% baseline and provides strong results when dealing with input/label noise (CIFAR-10 10 19%, 17%). V2). On ViT, GWA even outperforms the 99/1% validation split strategy commonly used in literature while eliminating its reliance on held-out data. This performance is particularly strong on the smaller CIFAR- 10 datasct while also scaling to ImageNet.

---

Alignment Distribution To gain deeper understanding of the information captured by GWA, we track its evolution through training compared to validation aceuracy and LabelWave's prediction change on identical experimental runs. The left panel of Fig. 2 demonstrates that both GWA and LabelWave closely track validation accuracy during training with little to no label noise (CIFAR-10), with convergence setting in towards the end. However, the center plot reveals GWA'S 's superior sensitivity to early symptoms of overfitting due to the presence of label noise within CIFAR- IO-N. This heightened sensitivity results in choosing an early stopping point very close to the one determined by validation accuracy, despite not requiring a validation set in the first place. On the other hand, relying on LabelWave in this label noise setting is not helpful, as overfitting is not detected. Since Lave did not outperform the validation set baselines in any of our experiments, we exclude from the further evaluations below.
CIAAR-IO +) CIPAR-IO-N (GWA:-)
Accuracy [X]
Label [+]
GD
GWA [+]
1.00
0.2
0.75
Nomilizdd
WV Wh MM MMW
0.1
0.50
+
0.25
CIFAR-10
CIFAR-IO-
0.00
-0.1
0
20
40 60
80
1.00
2.0
40
80
100
75
90
Iraining kpoch
Training kpoch
Figure 2: GWA tracks validation accuracy and captures subtle training dynamics associated with generalization (left, center) better than LavelWave. Line plots depict normalized values of validation accuracy (10%), LabelWave's prediction change and GWA's 's corrected mean across training. Markers indicate time step for early stopping according to each criterion. The underlying distribution of alignment scores (Iz, WT) (right) at time T can be seen as a cross-section providing further insights into training. Label noise highly influences properties of the CIFAR- 1O-N distribution vs. CIFAR-10.
Focusing on the distributional nature of GWA, the right panel in Fig. illustrates the evolution of the underlying distribution of alignment scores across training. Both CIFAR-10 and CIFAR- 10-N exhibit unimodal distributions approximating Gaussians to different degrees throughout training. However, there are notable difference between the two dataset variants. The "clean" CIFAR-10 dataset displays more concentrated distribution with higher GWA values (right, indicated by +/-) throughout training. Conversely, CIFAR-10-A exhibits consistently lower GWA values, with the substantially larger proportion of negatively aligned samples reflecting the impact of noisy labels on gradient coherence. This analysis confirms that GWA not only captures temporal dynamics but that properties of the underlying per-sample alignment distribution also provide valuable insights into data quality, offering a richer understanding of training behavior than solely considering validation accuracy.
4.2 Model Comparison
Next, we investigate whether GWA remains consistent across multiple model initializations, dataset variants and model architectures. We require such consistency to be able to reproducibly use GWA in practice, for example by monitoring it across hyperparameter sweeps. For this purpose, we analyze the correlation between the maximum alignment achieved during training (maxT R(Ar)) on CIFAR- 10 and its label noise variants CCFAA--1-NM and the final test accuracy, using both ConvNeXi and ViT models. The resulting scatter plot (Fig. 3 left) reveals a strong positive correlation between the test performance of models trained on more or less noisy dataset variants (leading to variations in test accuracy) and GWA. This holds across model families.
To further assess robustness beyond label noise, but also to the effect of domain shifts, we also evalu- ated test accuracy on CIFAR-C version of the standard CIFAR-10 10 test set incorporating realistic input perturbations such as blurring or image noise (Fig. 3 center). Again, strong correlation is observed between the models' test accuracy and GWA across both model families. These observations are quantitatively supported by the correlation statistics presented in the table (Fig. 3 right). which demonstrate high Pearson and Spearman correlation coefficients. These results highlight GWA's capacity to provide consistent measure enabling reliable model quality comparisons.
7

---

100
CIFAR-10
CIFAR-C
Coary
Test Azc. [88]
90
ConvNeX:
ComNNeXi
VI
VIT
80
CIFAR-10
Pearson
Spearman
0.97
0.99
0.98
0.92
CIFAR-C
Pearson
0.99
0.94
0.89
Spearman
0,96
0.92
0.88
0.04
0.06
0.08
0.10
0.04
0.06
0.08
0.10
Correlation Statistics
Max. Aligument B[Ar]
Figure 3: Maximum alignment B[Ar] allows for comparing model performance across runs. Scatter plot (left) shows correlation between E[Ar] and test accuracy on CIFAR-10 10 and with varying performance its label noise variants for ConvNeXt and VīT. Correlation is even stronger when evaluating against robustness benchmark CIFAR-C (center). Pearson and Spearman correlation coefficients for all cases (right) corroborate visual findings (p < 0.001).
Model Robustness As standard test sets often from the same domain even identical initial parent dataset as the training and validation set, we additionally evaluate models selected using the early stopping criteria studied above CIFAR-C and ImageNet-C (Tab. 2), established benchmarks that apply diverse range of image corruptions to standard test sets. This allows us to evaluate if GWA actually detects training dynamics that extend out of strict in-domain learning and detect models that work on corrupted data that mimics real-world perturbations. Our results reveal that models chosen via GWA-based early stopping consistently exhibit improved performance across various corruption types compared to those selected using traditional validation accuracy. Specifically, we observed an average improvement of 0.55% on CIFAR-C and 0.67% on ImageNet-C compared to models trained with 10% validation set. Our findings show that GWA not only correlates with test performance, enabling early stopping, but also enhances model robustness to real sollworld input perturbations, effectively closing the loop from addressing label noise to mitigating input noise.
Table 2: Using GWA to determine early stopping results in models that are more robust to realistic perturbations and suitable for deployment. Test accuracy averaged across 3 runs with ViT/S-16.
Test Accuracy [%}
CIFAR-C
ImageNet-C
Weathar
Weathar
Blar
Digital
Noise
Blu
Digital
Nnise
Model
Val (10%)
79.25
55.78
64.23
62.43
60.06
Val Set (1%)
-0.88
-1.09
-0.68
-1.04
+4.55
+0.44
+0.43
+0.57
GWA
+0.52
+0.53
+0.60
+0.56
+0.57
+0.61
+0.93
+0.60
4.3 Connecting Model Performance to Training Data
Our proposed GWA is inherently linked to the training data itself through the per-sample aligmment scores constituting the distribution. These per-sample alignment scores offer additional insight into the training process which traditional validation metries lack. To demonstrate the value of this distributional quantity, we examined individual per-sample alignment scores during optimization, and the corresponding characteristics of the training images. We established in Sec. 3 that negative alignment scores likely correspond to outliers, examples, or mislabeled samples opposing the overall "training direction",, while positive values indicate that samples are aligned with the currently learned patterns, i.e., representing "general" features/conceets early in training and more specific yet common features/concepts later on; this mirrors the simplicity bias argument of [14] and others.
Figure 4 showcases example images from CIFAR-10 10 and its noisy variant (CIFAR- 1O-N) with highest and lowest per-sample alignment at epochs 5, 50, and for the dog and car classes. A striking observation is that nearly all samples with negative alignment scores when training on CIFAR-10-N are mislabeled major benefit of using GWA achieved without employing an explicit outlier or mislabelling detection technique. Furthermore, analysis of CIFAR- 10 reveals that samples with high
8

---

negative Per-Sample Alignment K. positive
Epoch
car
dog
dog
car
5
90
mislabeled
CIFAR-10
CIFAR-I0-N
Figure Per-sample alignment scores (zt, Wr) reveal insights into data characteristics and learning progression. Example images from CIFAR-10 and CIFAR-I06 R--O with highest and lowest alignment scores epochs 5, 50, and 90 of training. Images displayed for the dog and car classes.
positive alignment scores tend to be visually simpler, while those with negative alignments more cluttered and/or visually challenging. Notably, positively aligned samples in the beginning at epoch predominantly feature easily classifiable instances (e.g., frontal views of cars with white background, dog faces), whereas later epochs showcase increasingly complex yet still representative examples (rear view of cars, dog faces with long ears). This pattern corroborates the aforementioned prior work stating that models initially learn from simpler samples before progressing to more complex features a dynamic directly reflected in the per-sample alignment scores.
4.4 Fine-Tuning
Our previous analyses focused on training models from randomly initialized weights, representing a scenario where the model learns all relevant information during the op- timization process. However, modern deep learning frequently leverages pre-trained mod- els via fine-tuning fundamentally altering the initial conditions and subsequent train- ing dynamics. We next investigate how this impacts our proposed GWA metric. As visu- alized in Fig. 5, GWA exhibits significantly higher values after the first epoch of fine- tuning, reflecting the immediate benefit of pre-trained features for generalization cor- roborated by high initial accuracy. Unlike training from scratch where GWA typically consistently increases, observe an initial
1.0
Nomalizzd Value
0.8
ImageNet
iNat18
0.6
P6aces365
0.4
0.0
7
Training Epoch
Figure 5: Initial GWA decrease during fine-tuning before increasing equivalent to training from scratch. ImageNet-21k pre-trained VIT/B-16 fine-tuned for 10 epochs ImageNet- 1k /Places365, 20 epochs iNat18.

---

dip during the early stages of fine-tuning. This suggests the model must first adapt to dataset-specific details, temporarily disrupting the initially strong alignment. After few epochs, this trend reverses, mirroring the increasing alignment observed during training from scratch. Consequently, when employing GWA for early stopping during fine-tuning, we now prioritize identifying the initial minimum in alignment before taking the maximum GWA to determine early stopping. As shown in Tab. 3, this refined approach yields reliable performance, with test accuracy outperforming the validation-basee metrics on most datasets.
5 Conclusion
We introduced GWA and investigated its ability to capture dataset- and sample-level training dynamics during optimization, as well serve
Table 3: GWA matches or outperforms other early stopping criteria when fine-tuning a ViT/B-16 pre- trained on ImageNet-2 Top-1 test accuracy aver- aged across seeds, min-max range below in gray.
robust early-stopping criterion. We have shown that GWA is a reliable metric that matches surpasses classical validation sets on range of tasks, while offering unique insights into training by connecting performance directly to individual training samples a capability particularly valuable in the presence of noise and low-data regimes. future work, we intend to expand our evaluation beyond supervised image classification, e.g., to the self-supervised setting, where the importance of gradient directions has recently been noted [46], and to other modalities such text, where
Test Accuracy [98] (A min-max)
ImageNet.
iNat88
Places365
Early Stop
Val
Set (10%)
84.04
73.94
72.87
(0.07)
(0:35)
(0.07)
(0.03)
Set (19)
84.11
74.19
89.00
73.65
$8.86
(0.37)
GWA
84.15
74.32
89.05
73.73
58.78
(0.29)
applications to the autoregressive loss are a natural evolution our results. Morcover, seeing as current techniques either too inefficient [1] or ineffective [23] for large-scale applications, hope that our work sparks interest in computationally efficient train-time generalization proxies such as GWA.
10

---

References
[1] Stanislav Fort et al. Stiffness: A New Perspective on Generalization Neural Networks. 2020.
arXiv: 1901. 094919911 [cs LG]. URL: https://aarat btg/abs/1/0 1901 19901094911.
[2] Ziwei Ji and Matus Telgarsky. "Directional convergence and alignment in deep learning". In: Advances in Neural Information Processing Systems 33 (2020), pp. 17176-17 76-17786.
[3] Yiding Jiang et al. "Fantastic Generalization Measures and Where to Find Them". In: ArXiv abs/1912.02177 (2019).
[4] Sepp Hochreiter and Jürgen Schmidhuber. "Simplifying Neural Nets by Discovering Flat Minima". In: Advances in Neural Information Processing Systems. Ed. by G. Tesauro, D. Touretzky, and T. Leen. Vol. 7. MIT Press, 1994, URL: http:: pprccee/eoooeeeediggg heweeeerddnnnnnnnnpieeeee neurips. cc/paper_file files pdf.
1994 ffwee/flle/////eee
[5] James Martens and Roger Grosse. "Optimizing neural networks with Kronecker-factore approximate curvature", In: Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37. ICML'15 15.. Lille, France: JMLR.org, 2015,
pp.
[6] Nitish Shirish Keskar et al. "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima". In: International Conference on Learning Representations. 2017. URL:
https://open ppw.revieee.....ooooo
[7] Garima Pruthi et al. "Estimating training data influence by tracing gradient descent". In: Advances in Neural Information Processing Systems 33 (2020), pp. 19920-19930.
[8] Stanislaw Jastrzebski al. "The Break-Fver Point on Optimization Trajectories of Deep Neural Networks". In: International Conference Learning Representations: 2020. URL.: attps://openreviee net/ wwwoomiidrrreeee 1idmriig7rAAKKA
[9] Jeremy Cohen et al. "Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability". In: International Conference on Learning Representations, 2021. URL: https: // /ppenrevvee net wwwpeeeiiwweeeefouumiiiiieeee.ee
[10] Justin Gilmer et al. "A Loss Curvature Perspective on Training Instabilities of Deep Learning Models". In: International Conference on Learning Representations. 2022. URL: https // /openreview net/ forum? -
[11] Pang Wei Koh and Percy Liang. "Understanding Black-box Predictions via Influence Func- tions". In: Proceedings of the 34th International Conference on Machine Learning. Ed. by Doina Precup and Yee Whye Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, Aug. 2017, pp. 1885-1894. URL: https: /ttttpp:/pproceedingss mlr www..press/voo htt/v////oohaa www./o.oaa..... html.
[12] Vitaly Feldman. "Does learning require memorization? short tale about long tail". In: STOC 2020. Chicago, IL. USA: Association for Computing Machinery. 2020, pp. 954-959, ISBN: 9781450369794, DOr: 10. 1 1011/1155/311 3384290. URL.: https: //doi 10./10. 1145/ 3357713. 035771..3384290. 3384290.
[13] Vitaly Feldman and Chiyuan Zhang. "What neural networks memorize and why: discovering the long tail via influence estimation". : In: Proceedings of the 34th International Conference Neural Information Processing Systems. NIPS' 20. Vancouver, BC, Canada: Curran Associates
Inc., 2020. ISBN: 978171385 13829546..
[14] Devansh Arpit al. "A closer look memorization in deep networks". In: Proceedings of the 34th International Conference on Machine Learning Volume 70. ICML'17. Sydney, NSW, Australia: JMLR.org, 2017, pp. 233-243
[15] Nasim Rahaman et al. "On the Spectral Bias of Neural Networks", In: Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, June 2019, pp. 5301-5310. URL: https:://roceeet http:///pocceeen....oo mwa.g.mlr... wtt.gssssr.rree//9///// wwwppp/aeeeaaaaaaaaeee 1t1p9///raaaaa1199 htt.aammmmmnaanm.mlll...
[16) Dimitris Kalimeris et al. "SGD on Neural Networks Learns Functions of Increasing Complex- ity". In: Advances in Neural Information Processsig Systems. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc., 2019. URL: https: http:///////cceeeeeeeeeeeaeee http://....ttts//ppoeeeeeiigggg www.ips...ppp...m cc/ wccpapprrfilllee /2per/2019 http:r//o...tteeeeeeccccssssrrrs
[17] Karttikeya Mangalam and Vinay Uday Prabhu. "Do deep neural networks learn shallow learnable examples first?" In: ICMI. 2019 Workshop on Identifying and Understanding Deep
Learning Phenomena. 2019. URL: https: wwwp///wwwweee.c////pp/tp wwt.eewwwweeeeeeeeeeee www.eeeetoouaee cerrmmii ind HiddH-HHx4r222

---

[18] Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. "Neural networks trained with SGD leamn distributions of increasing complexity". In: Proceedings of the 40th Interna- tional Conference on Machine Learning. Ed. by Andreas Krause et al. Vol. 202. Proceed- ings of Machine Learning Research. PMIR, July 2023, pp. 28843-28863. URL: https: / //proceedings... mlr........... Setesss/v022/ef...i metmti2a........... html.
[19] Nora Belrose et al. Neural Networks Leurn Statistics of Increasing Complexity. 2024, arXiv: 2402.04366 [ca. LG]. URL: pestps://arxxe org/ 240b.2202....m 04362.
[20] Hwanjun Song et al. How does Early Stopping Help Generalization against Label Noise?
2020. arXiv: 1911 08059 [cs. LG]. URL: https: /ttp:v/.//.vv..//aa org/ ht///bbs/1199 wttp//11..000..33111 08059.
[21] Yingbin Bai al. Understanding and Improving Early Stopping for Learning with Noisy
Labels. 2021. arXiv: 2106. 15853 [Cs.LG] URL: https: //tp:iv///iivvv/ia atttpp://aaxivv.org/ab///////// wtgg 15853.
[22] Suqin Yuan, Lei Feng, and Tongliang Liu. "latte Stopping: Avoiding Confidently Learning from Mislabeled Examples". In: CoRR abs/2308. 13862 (2023). DO1: 10. 48.48855/ARKXA 2308. 13862. arxiv: 2308. 13862. URL: https: //doi/doioiio org/ 10. (8550/arXis 2308. 13862.
[23] Suqin Yuan, Lei Feng, and Tongliang Liu. "Early Stopping Against Label Noise Without Validation Data". In: The Twelfth International Conference on Learning Representations. 2024.
[24] Maren Mahsereci et al. Early Stopping without a Validation Set. 2017. arXiv: 1703. [cs. LG]. URL:
http///xxxvooor//bb/a//.e/ ptrggabs/17000 09580.
[25] Karthik A. Sankararaman et al. "The impact of neural network overparametterization on gradient confusion and stochastic gradient descent". In: Proceedings of the 37th International Conference on Machine Learning. ICML' 20. JMLR.org, 2020.
[26] Jinlong Liu et al. "Understanding Why Neural Networks Generalize Well Through GSNR of Parameters". In: International Conference on Learning Representations. 2020. URL: https: //openreview... wwt.:eee/fofoeeoo hett:/forrmm...ddleeeltttttt iidHHeyiISStti..
[27] Satrajit Chatterjee. "Coherent Gradients: An Approach to Understanding Generalization in Gra- dient Descent- Optimizationn.. In: International Conference on Learning Representations. 2020. URL: https: http:::/tts//ttettee wpwpreeeeeeeeeeeeeeeeee net. https//frummiddrrrr der.undddrreeeoeor
[28] Mahsa Forouzesh and Patrick Thiran. "Disparity between batches as a signal for early stopping". In: Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part II
21. Springer. 2021, pp. 217-232.
[29] Kaifeng Lyu and Jian Li. "Gradient Descent Maximizes the Margin of Homogeneous Neural Networks". In: International Conference on Learning Representations. 2020. URL: https: / //openevvie. www.rrviie..et/forrrre htt///orrumnddSJJLLLeee
[30] Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. "An Alternative View: When Does SGD Escape Local Minima?" In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, July 2018, pp. 2698-2707. v80/kleinberg html.
URL: https: : /ptpppeeeeee...... mlr. press/
[31] Pengfei Chen et al. "Beyond Class-Conditiona Assumption: A Primary Attempt to Combat instance-Dependen Label Noise." " In: Proceedings of the AAAI Conference on Artificial Intelligence. 2021.
[32] Charles Guille-Escuret et al. "No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths". In: Proceedings of the 41st International Conference on Machine Learn- ing. Fd. by Ruslan Salakhutdinov et al. Vol. 235. Proceedings of Machine Learning Research. PMLR, July 2024, pp. 16751-16772 URL: https: / proceedings mlr www.speessseeo v235/ waille-escuret244 html.
[33] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. 2017. URL: https: h/tper//ppenrrvie net/ /tt.//eetuorne/ forum?d-=ry77rTogl.
[34] Georgii Mikriukov et al. "Evaluating the Stability of Semantic Concept Representations in
CNNs for Robust Explainabilityy" In: Explainable Artificial Intelligence. Ed. by Luca Longo. Cham: Springer Nature Switzerland, 2023, pp. 499-524. ISBN: 978-3-031-44067-0,
[35] John S. Bridle. "Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition". In: Neurocomputing. Ed. by Françoise Fogelman Soulié and Jeanny Hérault. Berlin, Heidelberg: Springer Berlin Heidelherg, 1990,
pp. 227-236. ISBN: 978 978-3-642-76153--
12

---

[36] Christopher M. Bishop and Hugh Bishop. Deep Learning: Foundations and Concepts. 1st ed.
Springer Cham, 2023, ISBN: 978-3-031-45468-4 DOI: 10. 00000/9978-001-445--------
[37] Zhuang Liu et al. "A ConvNet for the 2020s". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2022).
[38] Sanghyun Woo et al. "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked
Autoencoders". In: arXiv preprint arXiv:2301 .00808 (2023).
[39] Alexey Dosovitskiy et al. "An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale". In: ICLR (2021).
[40] Hugo Touvron et al. "Training data-efficient image transformers & distillation through at- tention". In: Proceedings of the 38th International Conference Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, July 2021, pp. 10347-10357. URL: https http//pooceee...e///// mlr preespesss /v139/ touvron2ia.. html.
[41] Benjamin Recht et al. "Do ImageNet Classifiers Generalize to ImageNet?" In: Proceedings of the 36th International Conference on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, June 2019, pp. 5389-5400. URL: htttps:/ppooceeee ngs. mlr pressp wttess//////eecht11ee/ html.
[42] Lucas Beyer et al. "Are we done with ImageNer?" In: CoRR abs/2006.07159 (2020). URI.: attps://arxivs wwtp./r///abbbb..bbbb httpaa////00000.0......
[43] Jiaheng Wei al. "Learning with Noisy Labels Revisited: A Study Using Woal-World Human Annotations". In: International Conference on Learning Representations. 2022. URL: https:
///p///////// /oppnreviiew... wwtp:/ooormiiddd///e5P110
[44] Andreas Steiner et al. "How to train your VIT? Data, Augmentation, and Regularization in Vision Transformers", In: arXiv preprint arXiv:2106.1023 (2021).
[45] Dan Hendrycks and Thomas Dietterich. "Benchmarking Neural Network Robustness Com- mon Corruptions and Perturbations". In: Proceedings of the International Conference on Learning Representations (2019).
[46]
Youngwan Lee et al. Exploring The Role of Mean Teachers in Self-supervised Musked Auto- Encoders. 2022. arXiv: 2210.02077.
[47] Gene V. Glass, Barry McGaw, and Mary Lee Smith. Meta-Analysis in Social Research. Beverly Hills, CA Sage Publications, 1981.
[48] Chiyuan Zhang et al. "Understanding deep learning requires rethinking generalization". In: International Conference on Representation Learning. 2017.
[49] Vardan Papyan, X. Y. Han, and David L. Donoho. "Prevalence of neural collapse during the terminal phase of deep learning training". " In: Proceedings of the National Academy
of Sciences 117.40 (2020), pp. 24652-24663. DOI: 10. 1073 pnas 2015509117, eprint: https://wwiz pnas pnas org/ doi /dff/dff//// / /10. 1073 ppnas 2015509117. URL: https: //www.
pnas oatasorrg/oo//aae hwt.://aaaass/oo..aeeee 11t3//............. 1tt..///p......1..1....... 2015509117.
[50] Kun Song et al. Unveiling the Dynamics of Information Interplay in Supervised Learning. 2024. arXiv: 2406 03999 LG]. URL: https: wwwp:/s/////////// 03999.
[51] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain VIT baselines for ImageNet-lk. 2022. arXiv: 2205 01580 [cs.cv]. URL: https arxiv. org abs / 2205. 01580.
[52] Kentaro Yoshioka. vision-transfommer-cfffar Training Vision Transformers (VIT) and related models on CIFAR-1O. https: github com ken/aroy77 vision transf ormers- cifar10. 2024.
[53] Ekin D. Cubuk et al. RandAugment: Practical automated data augmentation with reduced search space. 2019, arXiv: 1909. 13719 [ca. CVs. URL: 13719.
wttp:iiiw.rg/bb///e/eooooo 1909.
[54] Hongyi Zhang al. mixup: Beyond Empirical Risk Minimization. 2018, arXiv: 1710. 09412 [cs. LG]. URL:: https::/ //tps/////xiv.////a org/abs///aooo.oooooooooo..o... 1710. 09412.
13

---

Appendix
Here, we provide further evaluation of our scalable estimator (Appendix A.1), compare against related work not directly usable as a train-time generalization proxy, yet relevant to our approach (Ap- pendix B), and provide more details on our empirical evaluation well the results (Appendix C).
A
Scalable Estimator
A.1 Estimator Bias
The proposed efficient online estimator of GWA is computed continuously during each epoch. This estimator's key difference from pure offtine estimator is that the weights are not fixed; they drift with each mini-batch update. This drift is the source of a systematic bias, and its characterization depends critically the point of comparison.
Online Bias Relative to Fixed Time Point (constant w) Compared to the GWA value at the epoch's starting weights wp, the online estimator exhibits bias directly governed by the learning rate 77. This can be shown by using a first-order Taylor expansion on the expected alignment A(w), where the bias for measurement at step is determined by the change in w, approximately given by V VA(wp) (we- wo). J. Since the weight displacement W/ wo is the result of accumulated gradient steps, each scaled by y). the average bias larger causes greater drift from Wo. resulting in a larger first-order bias. In practice, however, this bias negligible as seen in Fig. 6: both the online and offline estimator have near pear-perfect correlation. Quantifying the bias between online and offline estimators (at the end of each epoch) shows a small effect size [47] for 7) 0.1 and SGD with a mean of 0.04 (maximum 0.12), decreasing further for 0.01 to 0.027 (maximum 0.08), and to 0.020 (maximum 0.05) for 0.001.
the epoch is linearly proportional to the learning rate. A
In addition, the existing bias is better understood not re-frame the comparison against the metric at the epoch's 's midpoint Wmid- the dominant first-order bias cancels out. This is because the weight updates w. are, first approximation, distributed symmetrically around Wmid, causing the average displacement E[w; Wmid] to be near zero. However, the GWA metric defined in the paper (Eq. (2)) is not just the mean alignment but ratio involving the distribution's 's kurtosis. Analyzing higher-order moments like kurtosis requires second-orde -oder Taylor approximation. However, while the online GWA perfectly unbiased with respect the midpoint, its bias is of a higher order.
error, but temporal shift. If
Normalize lized GWA
0.06
0.06
0.073
VIT
ConvNeXt
Offline
0.000
offline
offline
VIT
0.02
online
online
Con/NeXi
Training Epoch
Online GWA
Figure GWA computed online during the forward pass using the efficient scalable estimator detailed in Algorithm I nearly exactly matches the offline (after each epoch) computation of GWA for both ViT and ConvNeXt trained on CIFAR-10 (left, center) with near- near-perfect correlation across the whole training (right).
Online Per-Epoch vs. Step-Wise Offline Estimators An alternative, mini-batch perspective on the estimator's 's bias is through the lens hypothetical step-wise offline estimator, which would perform a full (and computationally prohibitive) offline evaluation on the entire dataset at each actual weight vector w, (i.e., after every single update step). Instead of computing the metric over the full dataset at each time step, our estimator only computes the alignment over smaller set Gr, equal to the batch size b. this case, the bias of the estimator would thus be equivalent to the bias induced by taking the batch representation of the full dataset. For smaller batch sizes, however, this
14

---

can lead to instabilities when computing the corresponding moments of A, for GWA. Our proposed online estimator for GWA can be seen as a special, more efficient case of this step-wise mini-batch estimator that does not have these instability problems. Instead of computing the moments for A based on single batch, we take the alignment scores 7(2;, W)(()) across the entire epoch, but with changing weights w2, to compute A,. This "epoch average" effect inherently results in smoothed version of the step-wise offline estimator, as seen in Fig. 6. It sacrifices temporal accuracy at each step for computationally efficient, stable, and interpretable summary of the epoch's 's overall alignment dynamic. For settings with larger batch sizes, and potentially also no clear definition of an epoch such in NLP, computing A, on batch-level can be reasonable approach, closing the gap the step-wise offline estimator.
To summarize, while the proposed online estimator is neither unbiased nor easily allows for quanti- fying its bias (see Sec. 3 and discussion above), we find that its bias, depending on the properties of the batch and the change in weights across the interval it is measured on, are neglectable in our evaluations. The scalable GWA estimator thus allows for (1) minimal train-time and implementation overhead, and is (2) near-perfectly correlated with the offline estimation of GWA in practice.
A.2
Full Model vs. Linear Classifier
GWA
Mean
Kurtosis
10
10-2
10
0.0
Stugest
Hoadl
0
20
40
60
100
20
40
80
80
100
20
40
80
100
Training Epoch
Epoch
Epoch
Epoch
Full Modul
0.5
Stapel:
0.79
0:77
1.00
0.79
1.00
Stago2:
0.74
0.94
1:00
0.71
0.96
1.00
0.73
0.96
1.00
Stagel::
0.02
0.65
1.00
00.67
0.72
0.77
0.73
1.00
-0.5
Head
0.52
0.78
0.59
0.83
0.45
0.60
0.78
Full
S1:
52:
83:
Read
Full
S1:
52:
83:
Head
PPll
81:
82:
83:
Head
Figure 7: Computing alignment scores using the linear classifier head mitigates dimensionality issues such decreasing mean of the alignment distribution and increasing tailedness (top center and right). Using the full model, or including more layers of the network in addition to the head layer (e.g., from the second residual block onwards in ConvNeXt, denoted as "Stage2:"),, leads to a worse learning signal (top left). This is corroborated by decrease in correlation between these scores and the alignment scores computed with the classifier head (bottom).
To mitigate problems caused by the high dimensionality of modern networks (e.g., noise, computa- tional cost), our approach focuses on the linear classifier head with closed-form per-sample gradient computation as introduced in Sec. 3. The plots in Fig. show that while including more layers still reflects certain patterns during optimization, such as an initial increase in alignment. While this initial increase in dataset alignment visible throughout the model, the carlier layers quickly Icad to an obfuscation of the signal. Gradient updates become increasingly more orthogonal when looking at the full model (top center) while also having relatively more updates near the bounds [-1, 1] of our alignment range (top right).
While leveraging the linear classifier head provides stronger learning signal, alignment score magnitude potentially remains sensitive to latent representation size possibly hindering cross- architectural comparison due to cosine similarity degradation high dimensions. We address this using JLT to reduce dimensionality while preserving pairwise distances. Fig. 8 demonstrates that JLT mitigates this issue, increasing expected alignment scores E[Ar] for models with larger latent spaces, and allowing for more comparable GWA values across architectures despite inherent
15

---

100
Max. Val. Accuracy [%)
85
Normal
JITT
Max. Alignment E[Ar]]
Figure Alignment scores for ConvNeXt models with latent embeddings of increasing size (indicated by circle size) and after applying the Johnson-Lindenntraus, Transform (JLT) to reduce embedding dimensions to a constant size of 192.
differences model design. Notably, even without applying the JLT, the alignment scores correlate well with validation accuracy across embedding sizes.
A.3
Kurtosis Correction
GIFAR-10
CIFAR-10-N
Alignment Distribution
1.0
1.0
2000
Epaci
Val Ame GWA
Val Aue
0.5
0.5
1000
Training Epoch
Alignment Score
Figure 9: Reproduction of Fig. 1 with mean and kurtosis components of GWA plotted individually. In simple cases, when the ConvNeXt learns without problems (left) the mean can be sufficient proxy for generalization. Often, more sophisticated patterns can, however, only be detected by looking not only at the distributions mean but also other properties. Notably, kurtosis seems to be a good correction factor to account for changes in the distributions shape (center, right).
5000
Epech
Epeec
5000
Feueeh 100
0.4
Allgammee by
While the alignment distribution tends to be unimodal in our evaluations, there is no guarantec for such behavior. In some cases, analyzing the alignment distribution directly help to better understand training dynamics and dataset characteristics.
Figure 10:
GWA is based distribution of values for each sample in the dataset. This allows to connect training performance directly to individual samples, but it also introduces a challenge: we need single, representative number to track how this entire distribution changes over time. Describing changing distribution with just one number is difficult, especially when its shape can be complex. We propose quantify the mean shift of the alignment distribution together with a kurtosis correction to create robust summary statistic. This correction is essential because the distribution of alignment scores can not be assumed to be Gaussian. Consistent with the long- and heavy-tail theories of deep learning, we find the kurtosis is suitable correction account for changes in the tails of the
16

---

distribution, which we find to be particularly important for noisy and challenging datasets like CIFAR-IO-N (see Fig. 9, center and right) or ImageNet.
While our corrected GWA metric works well in most cases, it has limitations. These failure cases can often be spotted by looking the plot of the underlying alignment distribution specific epochs (e.g., Fig. 10). One concrete issue is the occurance of the distribution becoming bimodal, which, it happens, tends be late during training. Investigating the learning dynamics that cause these specific distributional shifts is promising direction for future research.
A.4 Interpolation of Noise
80
Align. Mtssa
0.10
Allggn Kurtosta
80
Accuracy
CIVA
0.03
20
0.00
0.00
76 100
100
25
80
Training
Time [normallacd]
Figure 11: Training with random labels forces the model to learn only sample-specific information (label memorization) and random guessing accuracy compared to actual generalization. GWA, capturing all training dynamics, reflects this non ren-eeeerlzzaiin.. While alignment does increase with random labels, the mean and kurtosis show a distinct pattern, different standard smooth generalization. The initial low mean alignment around with corresponding high kurtosis across the dataset flips completely when the network starts memorize.
To fully understand GWA generalization proxy, we train models on CIFAR- 10 with completely randomized labels, following the setup in [48]. While this scenario of training on pure noise is unrealistic for any practical application, it is informative method isolate label memorization and evaluate GWA'S 's response. In this setting, the model cannot learn generalizable patterns and forced memorize the labels, leading validation performance equivalent to random guessing. As shown in Figure 11, GWA exhibits a distinct pattern under these conditions. Initially, the mean alignment is near zero with high kurtosis. As the network begins memorize the random labels, GWA increases sharply, driven by a rapid rise in mean alignment accompanied by a substantial reduction in kurtosis. After this peak, GWA begins to decrease even as the mean alignment stays high. Thus, rather than remaining static near zero as might be expected, GWA characterizes memorization through distinct dynamic pattern: sharp initial rise followed by a decay. This reveals how the metric captures the distinct phases of the model memorizing noisy dataset.
B Related Work
Next, we provide a comparative analysis of GWA to previously introduced techniques for generaliza- tion gap quantification and training dynamic assessment (summarized in Sec. 2). This highlights the benefits of our efficient train-time proxy for generalization and early stopping. Note that the focus of GWA is not exactly the same the other metrics shown here, but believe the comparison be useful nonetheless.
B.1 Gradient Norm and Second Order Information
Gradient norm a common metric used for tasks such approximating second-order information to analyze loss landscapes and sample influence (see e.g., TracIn [7]). Fig. 12 shows, however, that gradient norm is not replacement for GWA to validate generalization. While both the ConvNeXt and ViT models generalize, the average gradient norm develops differently for both architectures (right). While we observe weak overall correlation between the per-sample gradient norms and our alignment scores 7/ (as defined in Algorithm 1), we see discernible patterns: a reduction in gradient norm towards the end of training tends to correlate with an increase alignment, while higher relative gradient norms particularly at the end of training correlate with alignment scores indicating
17

---

Epoch
Epoch 100
Nomal lazedizzzdd Value
Gradient Norn
10
CPWA
-0.2
00.0
0.1
-0.3
0.0
0.3
-0.2
0.0
012
100
200
Alignment Score Y)
Training Epoch
Gradient Norm
Epoch
Bpodh
Nornalized Value
10'
10
Norm
CWWX
10
0.0
-0.2
-0.2
0.0
-0.2
0.0
0.2
50
100
Aligmment Score Y/
Training Epoch
Figure 12: Per-sample gradient norm and alignment scores 7/s for VIT (top) and ConvNeXt (bottom) trained on CIFAR-10 show that both measures quantify distinct characteristics of the training dynam- ics. GWA is consistent across both architectures and better reflects generalization during training (right).
orthogonal gradient updates. An interesting exception observed in our ConyNeXt experiment (bottom), where group of samples exhibits the smallest gradient norm and low alignment at the end of training, suggesting these examples are well-learned with remaining loss reduction being sample-specifie, i.e., inherently orthogonal to general optimization.
Notably, unlike GWA, metries such TracIn are susceptible to the unbounded property of gradient norms and thus sensitive to their volatility and outliers. In the right-bottom panel of Fig. 12 the gradient-norm distribution is heavily skewed, with peak of 7.16 10 at epoch 12. By contrast, GWA's ' distribution is compact, only depending on gradient direction and therefore unaffected by magnitude outliers.
Spearman analysis on CIFAR-10-N from Fig. when overfitting occurs shows that GWA correlates strongly with validation accuracy (R= 0.97) yet poorly with training accuracy (R= 0.56) during overfitting, as desired. contrast, the gradient norm exhibits a high correlation with training accuracy (R= 0.90) but weak one with validation accuracy (R= 0.24). This also holds when only considering the classifier head, similar to GWA, where both gradient norm and also the second-order Hessian trace reach perfect correlation with training accuracy (R= 1.0) yet moderate correlation with validation accuracy (R= 0.48).
In summary, while per-sam per-sample gradient norms do not directly correlate with GWA and thus cannot be used as generalization proxy, analyzing both helps understand which samples have been learned and which remain hard to learn.
B.2 Connection to Loss Landscape Geometry
Limited empirical work has analyzed the alignment between gradients and model weights, with [32] being a notable exception; they analyze cosine similarity between the mini-batch estimate of the gradient g and the difference between the current weights WT and "optimal" weights w*. defined as the final weights of a previous run with the same seed. With this metric they aim to quantiffy the geometric properties of sampled gradients along optimization paths within the loss landscape. Fig. 13 shows that GWA and w* -"-aaignment share similar patterns, converging towards comparable behavior in the later parts of training. However, w -li-gnmmmtnt requires two full training runs, making it computationally expensive or even infeasible for very large model training. GWA offers compelling alternative: capturing gradient alignment from a different perspective yet with similar insights, significantly reduced computational cost, and potentially broader applicability. We regard a combination the efficiency, reliability and validity of GWA with the theoretical insights of [32] promising future research direction.
18

---

1.00
Normalized Value
0.75
0.50
ConvNeXt
VII
0.25
Align.
Align.
GWA
GWA
0.00
20
40
60
80
100
100
150
200
Training Epoch
Figure 13: GWA mirrors the alignment between mini-batch gradients and the direction to the optimal weights w towards the end of training but without requiring the expensive computation of w'.
B.3 Pairwise Gradient Alignment
Relatedly, prior research (discussed in Sec. 2) analyzes gradient direction via pairwise per-sample
gradient alignment proposing metrics such as gradient coherence and stiffness [1, 25, 26. 27]. Most approaches in this area are related and measure very similar quantities, a proxy for which is the pairwise per-sample gradient alignment (i.e., the pairwise gradient cosine similarity). Note that none of these quantities actually use the weights, contrary to GWA. Fig. 14 reveals that the average pairwise gradient alignment exhibits high variance especially early on and likely due initial training randomness while GWA is more stable. After the initial differences, the two measures exhibit similar trends in mid-to-late in training. However, GWA exhibits fewer fluctuations this mid-to-late stage, providing much more consistent signal. Moreover, the substantial computational overhead of pairwise per-sample gradient alignment - requiring full model gradients across multiple timesteps, their storage, and computationally expensive pairwise cosine similarity calculations limits its applicability.
In summary, compared to prior works, GWA offers a compelling complementary approach. It provides comparable insights into gradient alignment but has significantly improved efficiency and thus unlocks new possibilities for large-scale research in this domain. This renders GWA an attractive tool re-evaluate existing work and accelerate progress in understanding neural network training dynamics.
Normalized Value
0.8
0.6
0.4
GWA Paiewise Grad. Align.
0.0
Training Epoch
Figure 14: GWA exhibits substantially more stable relative alignment during the entire training duration compared to pairwise per-sample gradient alignment.
B.4 Neural Collapse
Neural collapse [49] is an important theoretical phenomenon which describes the terminal-phart collapse last-layer features to their structured class-means. GWA offers complementary view based on the per-sample gradient-weight interaction throughout training on realistic, noisy data, where quantifying variance the key signal for generalization. Recent work on neural collapse has introduced matrix information theory analyze training dynamics. [50] use the latent embeddings of
19

---

1.0
Normalized Valun
0.4
Val Ace
GWA
0.2
MIR
HDR
0.0
60
80
100
Training Egoch
Figure 15: GWA exhibits substantially more stable relative alignment during the entire training duration compared to pairwise per-sample gradient alignment.
each samples together with class deas-deepnneent classifier weights to compute matrix mutual information based on Gram matrices. Fig. 15 shows that the two values introduced in [50] based on matrix information theory, matrix mutual information ratio (MIR) and matrix entropy difference ratio (HDR), correlate well with GWA. We believe further connecting these two approaches in future work is not only a promising way to integrate GWA and information-theorette research but also allows for more efficient information theoretic approaches, without full Gram matrices, and providing GWA's per-sample insights.
C Further Details on the Experimental Setup
C.1 Implementation Details
Table 4: Training hyperparameters for respective model-datased pairs. Setting for training ImageNet- Ik from scratch taken from [5]]. ImageNNt-222k pre-trained weights for fine-tuning and settings from [44] with adaptations for datasets ImageNet- 1k, iNat18, and Places365 in parentheses.
CIFAR 10/ CIFAR
ImageNet 1k
Fine Tuning
Hyperparameter
ComNNeXi-P
VIDCIAAR-4
ConvNeXt-F
VIT/S-16
VIT/B-16
Optimizer
Adam
Adam
Adam
Adam
SGD
LR
0.001
0.0001
0.001
0.001
(U.01, 0.05, 0.01)
Weight Decay
Momentum
[0.9,0.999]
[0.9.0.999]
[0.9,0.999]
[0.9,0.999]
0.9
Batch Size
512
256
1024
1024
512
Iterations
9000
35000
112650
112650
(26000, 17500, 37500)
Image Size
32
32
224
224
224
RandAug [53]
(0, (0.0, 0.1. 0.0)
Mixup [54]
0.0
0.0
0.2
0.2
Grad Clip
Scheduler
Cosine
Cosine
WornipCosine 10000
WamnupCosine
WannupCosine 500
Warmup Steps
10000
We provide the training settings for all ConyNeXt models on CIFAR-I0 and its variants as well as for ImageNet in Tab. 4. The settings are used for our main results in Tab. 1, Fig. 2, and Fig. 4 (Sec. 4). Tab. also includes the training and fine-tuning settings for all ViT models used in Tab. 1, Tab. 2, and Tab. 3 (and corresponding Fig. 5). The ImageNet-22k pre-trained weights of the ViT/B-16 are from [44] and are available here. Results in Fig. 3 for both architectures are obtained with the same settings.
C.2 Auxiliary Results
Alignment of Mislabeled Samples Beyond early stopping, GWA provides a novel perspective on training dynamics and generalization. Fig. 16 shows the alignment distribution of CIFAR- 10-N (40% label noise) across three epochs (beginning, close-to-optimal, overfillin). These distributions make up GWA Fig. (center). Initially, most samples have updates that are orthogonal to initial optimization trajectory, with the alignment distribution being compact, around zero, and with
20

---

CIFAR-1O-N (40% label nolse
Correct
Mislabeied
Enuch No0, 0.053
Epoch
Epodi
2000
1500
1000
500
Alignment Scure
Figure 16: Corresponding alignment distribution of Fig. 2 (center) with alignment scores of correctly labeled and mislabeled samples being colored differently. Mislabeled samples tend to be more negatively aligned during training, with the clearest separation between mislabeled and correctly labeled samples around the optimal stopping time step near Epoch 50.
mislabeled examples exhibiting a slightly negative bias. This bias intensifies toward the optimal stopping epoch, differentiating mislabeled and correctly labeled samples. As the model learns, the optimization requires to mislabeled examples further reduce the overall loss, driving corresponding updates towards zero again a directional shift from prior learning. At the same time, correctly labeled samples are also pushed towards zero, potential result of the model overfitting and/or learning samples specific information of these samples. This leads to concentration the distribution. Analyzing these distributional changes and proactively modulating this behavior warrants further investigation.
Results with Fixed Training Set Size Tab. shows the best-case scenario possible with each approach fairly demonstrate the performance of all early-stopping criteria. Using a smaller validation set, validation set at all, allows for using more samples during training, aiding model performance. With a fixed training size, a 10% validation set outperforms both 1% validation set and GWA (Tab. 5). However, despite the artificial restriction of training data size in this ablation, GWA and the 1% valida- tion set remain competitive, indicating that the performance both approaches not due to training set size alone.
Table 5: Artificially restricting the available train- ing data leads to larger validation sets with 10% to better approximate generalization behavior com- pared smaller validation sets (1%) and GWA (val split of 90/0 in Table) in comparison to the opti- mal results achievable with each method in Tab.
CIFAR- 10 [label noise %]
ImageNet
Val
0%
9%
17%
81.10
VIT
90/1
.0.07
-0.01
00.20
-0.08
-0.12
-0.01
-0.27
90/10
89.86
85.33
82.30
71.24
ConvNeXi
90/1
-0.01
-0.02
-1.12
-0.18
90/0
-0.22
-0.17
-0.35
-0.09
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made are within a fixed scope and clearly visible in the introduction. As stated in abstract and introduction, proposed method is theoretically motivated by prior work and empirically evaluated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our approach within the method section and in the discussion and conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: While our work is theoretically motivated and we describe our approach from theoretical perspective, there mathematical proofs, lemmas etc. in the paper. All theory in the methods section is self-contained.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The datasets and models used are described at the beginning of the experimental evaluation. Hyperparameter configurations are given in the appendix. All implementations aims at reproducibility by using publicly available code and settings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
While not publicly available during submission, will release the code when the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
www.iiiiisiiooiiioiooooooissiiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: http:/uudeese//eeeee
/ //niss.
Sotps//oooeSSeesiiieooeee oooooil for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Experimental settings are given in the evaluation section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results are reported in triplicate. Standard deviations reported in the appendix. Statistical correlations are shown using standard measures (Pearson, Spearman).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Resource utilization is reported in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://nmtt /ttpp:::///euipp// wttpcccccccc/ceecc.ccc.ccc.ccltt Bttpa/////hhcscSieeeliiiiee
Answer: [Yes]
Justification:
We foresee no specific negative ethical implications arising from our work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our contribution is fundamental in nature and deals with early stopping and model robustness. We foresee no specific societal impacts arising from our work beyond the general considerations that AI and deep learning entail.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our paper does not include generative or data models and experiments are conducted on public datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appropriate citations are in place for publicly available datasets and licensing information for the datasets available from the sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released at this time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No such research was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Inapplicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
No LLMs were used in any of the aforementioned significant ways.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: wttp://pen.ppeeeeeee 220/://p/////comeeeeee////////// for what should or should not be described.
28


Reviews:
[{"review_id": "y37qwz8ytJ", "reviewer": "Reviewer_RSWb", "summary": "This paper introduce a new metric called gradient-weight alignment that can be considered as a novel proxy for generalization performance during training, which helps to indicate the most generalized model during training, especially with noisy data. This paper first show per-sample gradient scores aligns well with the generation performance during the training trajectory. The motivation is that the gradient would consistently point in the same direction as the model weights given enough training. However, calculate the per-sample grandient alignment is not cheep given that the neural networks are usually quite heavy and with grate amount of training data. As a result, this paper introduce an estimate that using the alignment with the last linear layer and accumulate the scores during the training epochs. This paper shows that for image classification task, GWA can be a good indicator and achieve better performance on noisy dataset.", "questions": "See above", "limitations": "See above", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is easy to follow and well motivated. The experiment results are good especially for the noisy training dataset and finetuning tasks.  It seems that this method has very strong limitation since it has to be used for classification models due to the size of the weight. However, is it possible to extend this method to other embedding models? For example, self-supervised tasks such as contrastive learning. Also, does this method hold for other optimizer or training strategy such as sharpness-aware-minimization or adversarial training. For classification tasks, we also have many other optimization settings, such as few-shot learning, meta-learning. Or even many NLP tasks are based on the embeddings to get the prediction in the token space. For those cases, will it still hold? The cases this paper shows are quite limited.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful questions about the broader applicability of our method.\nYou're right that our current evaluation focuses on classification, but GWA is not limited by model size or task type. In fact, it’s fundamentally optimizer-agnostic and its adaptation to most common tasks is straightforward. We chose the classification setting to rigorously establish our core claims, given its direct link to the motivating theory ([2, 25] in the manuscript).\n\nIn particular, on extending GWA to other tasks and settings:\n* For Self-Supervised Learning: Yes, GWA is directly applicable. The InfoNCE loss is structurally a multi-class cross-entropy loss. The model classifies the correct positive pair against a set of negatives. A closed form computation of the per-sample gradients for the estimator is possible. However, sample-level attribution is more complex due to the inherent dependency on negatives. We thus deferred a detailed analysis to maintain this focus.\n* Other Optimizers and Training Strategies: GWA works without modification. It uses raw gradients, so it’s independent of the optimizer’s update rule. Our results already show robustness across Adam, AdamW, SGD (Appendix, Tab. 4) for common training settings. As GWA is widely applicable, there is a range of different tasks it can be applied to, with few-shot learning being an interesting future direction.\n* For NLP: GWA is directly applicable to autoregressive models, as token prediction is cross-entropy over a vocabulary, so the same framework applies. In preliminary experiments, we found that GWA remains computationally tractable even when the final classifier layer is large (i.e., large vocabulary). However, it’s important to note that interpreting token-level dynamics in NLP over a single pass of the dataset results in intrinsically different training dynamics compared to common computer vision tasks. This warrants a more sophisticated investigation that we have planned in future work (see e.g. memorization in computer vision vs. language [R1, R2]).\n\n[R1] What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. Feldman and Zhang (2020).\n\n[R2] Counterfactual Memorization in Neural Language Models. Zhang et al. (2023)."}, {"review_id": "mmhDUqwhjy", "reviewer": "Reviewer_iv9T", "summary": "Previous work has argued that the alignment of gradients and model parameters is an indicator of convergence of a deep neural network (DNNs). This paper proposed an efficient estimator for this alignment by measuring the alignment per sample, and shows empirically that it can serve as a useful signal when determining when to halt optimization (early stopping) and selecting influential training samples. The estimator is made efficient by focussing on the gradients of the last layer, and computing it in batches (during an epoch).", "questions": "The inclusion of kurtosis in the estimator is motivated by the idea that rare/atypical samples have an outsized influence on the model, and that the kurtosis can account for this. Do the authors also have empirical support for this argument? It would be good to show that if one uses simply the first moment, that the estimator becomes worse for certain objectives (for instance in terms of determining when to halt optimization).", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "I believe this paper studies an important research question (how can we measure the generalization of a DNN without a validation set) and makes an clear contribution to it. It contains an extensive set of experiments to empirically support the usefulness of the introduced estimator. I believe the paper is fit for acceptance, although it still has some weaknesses in terms of the theoretical support for the estimator.  Specifically the design choices to focus on the last layer and computing it in batches.\n1. The authors write: *'we exploit the fact that classification fundamentally operates on latent representations. This reduces irrelevant variations'*. While somewhat intuitive, it is not entirely clear to me why the gradient alignment in layers previous to the last layer constitutes of irrelevant variations. It would be good if the authors could either show some connection between alignment in the last layer and alignment in all other layers (which would justify focussing on it), or show in a small-scale experiment that little is gained from measuring alignment on all layers. \n2. The authors write compute the alignment per batch, while in between batches the parameters. I understand that this has an advantage in terms of computational resources, but it would be good to know what the costs are in terms of accurately estimating the alignment. Particularly, it would be good to theoretically analyse how the difference between $E[A_T]$ and $E[\\hat{M}_T^1]$ (the bias of the estimator for the first moment) depends on the learning rate. Intuitively, if there are big steps in between batches, this bias will increase. **To be clear**: I do not think this makes the estimator not useful in practice, but it would be good to have an idea about the conditions under which it yields a good estimate of the alignment.", "was_revised": true, "final_justification": "The authors convincingly addressed the weaknesses listed as well as my remaining question. While their response improves my assessment of the paper I do not believe this warrants an increase in the rating.", "rebuttal": "Thank you for your positive assessment and your insightful questions. \n\n> [...] it is not entirely clear to me why the gradient alignment in layers previous to the last layer constitutes of irrelevant variations. It would be good if the authors could either show some connection between alignment in the last layer and alignment in all other layers (which would justify focussing on it), or show in a small-scale experiment that little is gained from measuring alignment on all layers.\n\nOur focus on the final layer is a principled design choice, grounded in both theory and practical considerations.\n\nA deep classifier's primary goal is to learn a representation that is linearly separable by its final layer [R3]. Consequently, last-layer gradients provide the most direct, stable signal of the learned task. Gradients in preceding layers, while necessary for building the representation, are an indirect signal; their variations are \"irrelevant\" as the corresponding information is condensed in the final representation [R3].\n\nIn fact, for our case, including earlier layers degrades the estimator. Gradients in shallower layers are significantly more unstable, as shown in work such as [R4]. Together with the high dimensionality of full model gradients, this can lead GWA to go in expectation towards zero, rendering the metric statistically uninformative. On the other hand, the final layer's gradients are lower-dimensional and more directly structured by the classification task, providing a robust signal for evaluating model performance.\n\nOn your recommendation, we have conducted an ablation study showing that: (1) including earlier layers produces a noisier metric, leading to premature and suboptimal early stopping; and (2) the alignment signal from earlier layers is significantly weaker (closer to zero) than the final layer's signal. We will include this in the camera-ready version.\n\n> [...] it would be good to know what the costs are in terms of accurately estimating the alignment.  Particularly, it would be good to theoretically analyse how the difference between $E[\\mathcal{A}_t]$ and $E[\\hat{\\mathcal{A}}_t]$ (the bias of the estimator for the first moment) depends on the learning rate. Intuitively, if there are big steps in between batches, this bias will increase. \n\nYour intuition is absolutely correct.\nThe bias of our estimator is indeed proportional to the learning rate $\\eta$. A first-order Taylor expansion of the alignment function $\\mathcal{A}(w_t)$ around the epoch's initial weights $w_0$ shows the expected bias is:\n$\\text{Bias} \\approx \\frac{1}{T}\\sum_{t=0}^{T-1} \\mathbb{E}\\left[\\nabla_w \\mathcal{A}(w_0)^{\\top}(w_t - w_0)\\right]$\nSince the weight drift $(w_t - w_0)$ is a sum of $t$ gradient updates scaled by $\\eta$, the bias is approximately linear in $\\eta$. For our cosine similarity metric, the bias is driven by the change in the weight vector's direction.\nWe will include this in the manuscript but also already have validated this empirically. In Figure 6 (Appendix), our online estimator and an offline ground-truth calculation yield closely matching curves, validating your intuition that the estimator remains useful.\n\n> [...] inclusion of kurtosis in the estimator is motivated by the idea that rare/atypical samples have an outsized influence on the model, and that the kurtosis can account for this. Do the authors also have empirical support for this argument? It would be good to show that if one uses simply the first moment, that the estimator becomes worse for certain objectives (for instance in terms of determining when to halt optimization).\n\nYes, we can confirm this empirically. On clean datasets like CIFAR-10, both estimators perform comparably. However, on datasets with noisy labels (e.g., CIFAR-10-N), where a subset of atypical samples is memorized late in training, their behavior diverges. In this setting, the atypical samples create a heavy-tailed alignment distribution. The mean-only estimator provides a misleading signal in this scenario, with the mean continuing to change after generalization performance has peaked leading to an incorrect early stopping signal. In contrast, our kurtosis-corrected estimator is robust to these outliers and better tracks the validation accuracy, making it a more reliable indicator for halting optimization.\n\nWe have conducted an ablation directly comparing mean-only and kurtosis-corrected GWA against validation performance over time on CIFAR-10-N. We will add these results, as well as visualizations of the alignment distribution and correlation of both metrics, to illustrate the key advantage of kurtosis correction.\n\n\n[R3] Understanding intermediate layers using linear classifier probes. Alain and Bengio (2016).\n\n[R4] Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability. Mikriukov et al. (2023)."}, {"review_id": "UCnceci2ex", "reviewer": "Reviewer_CWdC", "summary": "This paper proposes a metric - *Gradient-Weight Alignment* (GWA) that aims to provide an understanding of the training dynamics and its impact on generalization. Specifically, the paper explores utilizing GWA for early stopping. The paper also presents the robustness of GWA by utilizing the early stopped model on a perturbed test set. Experiments are performed on standard computer vision datasets and ViT and ConvNeXt models to asses the metric.", "questions": "To improve the paper's clarity and rigor, please address the following:\n1. How is the per-sample alignment score $\\gamma(x_i, w_T)$ handled numerically when the gradient $g_T(x_i)$ is a zero vector, which should occur for samples that are perfectly classified?\n2. Could you provide a quantitative comparison of the computational complexity and runtime overhead of GWA versus LabelWave and standard validation?\n3. How robust is GWA to different model initializations and variations in model size within the same architecture (e.g., ViT-Small vs. ViT-Base)? The paper claims to investigate this but provides no direct evidence.\n4. What data augmentation transforms were used during training? Could these transforms, especially aggressive ones, affect the stability and interpretation of the GWA metric?\n5. The rationale for Figure 3 is not entirely clear. While it's expected that lower data noise leads to better accuracy and alignment, how does this demonstrate the reproducibility or consistency of the metric itself? Please clarify the intended message.\n6. Could you provide more intuition on why the higher-order moments in GWA (i.e., the kurtosis correction) make it more sensitive to overfitting than traditional validation or methods like LabelWave, as suggested by the results in Figure 2?", "limitations": "The authors state in the checklist that limitations are discussed, but this is not the case. The paper lacks a dedicated discussion of its limitations. Key limitations to address include the potential numerical instability of the GWA metric (the zero-gradient issue), the strong assumption that last-layer gradients are a sufficient proxy for full-model gradients, and potential failure modes of the method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strenghts**:\n- The primary strength of this work is that it proposes a validation-set-free method for monitoring training and selecting a model for deployment. This is a significant practical advantage, particularly in data-scarce scenarios where withholding a validation set is costly. The method provides a promising alternative to the standard practice of using a hold-out set.\n- The paper addresses the important and challenging problem of understanding the training dynamics of deep neural networks without relying on computationally expensive second-order methods, such as Hessian-based analysis.\n\n**Weaknesses**:\n\nHowever, it faces a few issues that need to be addressed before being ready for publication. \n- **Insufficient Comparison and Positioning with Relevant Work:**\n    - The paper should be more clearly positioned with respect to Training Data Attribution (TDA) methods and influence functions. While some works are cited , the paper dismisses them as post-training analysis  without a deeper comparison. More recent and relevant methods like Datamodels [1] and approximate unrolling [2] are not discussed.\n    - The methodology, which relies on the alignment of gradients with the final layer weights, is closely related to the well-established phenomenon of Neural Collapse [3]. This connection is a critical omission. The paper should explicitly discuss this relationship and differentiate GWA from other metrics derived from Neural Collapse properties, such as the information-theoretic HDR and MIR metrics [4], which also measure alignment to analyze generalization.\n\n- **Unresolved Issues with the Metric's Formulation:**\n    - The definition of the per-sample alignment score, $\\gamma(x_i, w_T)$, becomes problematic when a sample is perfectly classified. For a cross-entropy loss, a perfect classification results in a gradient that is zero or near-zero, making the cosine similarity term $\\frac{g_T(x_i) \\cdot w_T}{\\Vert g_T(x_i)\\Vert \\Vert w_T \\Vert}$ undefined due to a zero-norm gradient vector. This is a fundamental issue, as many samples will be learned perfectly during training. The paper's claim that alignment approaches 1 (line 106) for perfectly classifiable data  seems to overlook this practical and theoretical problem.\n    - This concern is reinforced by the paper's own efficient gradient calculation, in line 153, which explicitly shows the gradient becomes a zero vector if $\\hat{y}_i=y_i$. \n\n- **Claims Not Substantiated by Experimental Evidence:**\n    - The paper repeatedly claims that GWA is computationally efficient and scalable. However, no empirical evidence (e.g., wall-clock time, FLOPS) is provided to compare its computational cost against the baselines it aims to replace or improve upon, such as validation set evaluation or LabelWave\n    - In Section 4.2, the paper states it will \"investigate whether GWA remains consistent across multiple model initializations\". However, the experiments only report results averaged over three runs, with no analysis or visualization of the variance or consistency across these different initializations.\n\n**References**\n\n[1] *Ilyas, Andrew, et al. \"Datamodels: Predicting Predictions from Training Data.\" Proceedings of the 39th International Conference on Machine Learning. 2022.*\n\n[2] *Bae, Juhan, et al. \"Training data attribution via approximate unrolling.\" Advances in Neural Information Processing Systems 37 (2024): 66647-66686.*\n\n[3] *Papyan, Vardan, X. Y. Han, and David L. Donoho. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" Proceedings of the National Academy of Sciences 117.40 (2020): 24652-24663.*\n\n[4] *Song, Kun, et al. \"Unveiling the Dynamics of Information Interplay in Supervised Learning.\" International Conference on Machine Learning. PMLR, 2024.*", "was_revised": true, "final_justification": "The authors have clarified many of my concerns as well as concerns raised by the other reviewers.", "rebuttal": "> Insufficient Comparison and Positioning with Relevant Work\n\nThank you for your feedback. As you recommended, we will incorporate this discussion into the paper. Concretely:\n\n*Regarding TDA*: Our related work already discusses foundational Training Data Attribution (TDA) and influence function estimation methods, primarily to distinguish GWA as an efficient, online metric. While we agree that the works you recommended ([1], [2] of your references) are valuable in their own right, they do not resolve the core issues stated in our related work, mainly the computational overhead:\n* Datamodels: The framework's core requirement is to train a large number of models from scratch, e.g., **300k models** for a ResNet9 for CIFAR in Table 1 of [1].\n* Approximate Unrolling with SOURCE: The “efficient” version of SOURCE “is L [constant factor of 2 or 3] times **more computationally expensive than influence functions**”, while standard SOURCE requires the computation of the EK-FAC at **every** checkpoint (see Section 3.4 in [2]).\n\nNonetheless we recognise the importance of these methods. Our original submission already included TracIn [R5] (based on gradient norm), an influence function approximation (Appendix A.2). We have now additionally evaluated second-order derivative (curvature-based) influence directly via a block-diagonal Hessian approximation, as is common. We found these first- and second-order metrics to be highly correlated. However, critically, both fail to provide a reliable early-stopping signal, whereas GWA succeeds at this key objective of our work. For instance, TracIn's signal is inconsistent for ConvNeXt and ViT models despite the models having comparable validation accuracy (Figure 8 right). We will add this new analysis to the camera-ready manuscript.\n\n*Regarding Neural Collapse (NC)*: While NC [3] is an important theoretical phenomenon which describes the terminal-phase collapse of last-layer features to their structured class-means, GWA is fundamentally different. GWA analyzes the per-sample gradient-weight interaction throughout training on realistic, noisy data, where quantifying variance is the key signal for generalization.\n\nHowever, regarding the specialized case of MIR/HDR: This is an insightful connection and we thank you for pointing this out. MIR/HDR [4] use features for a global, dataset (or batch)-level perspective on alignment. In contrast, GWA uses per-sample gradients, offering a more granular, complementary view. We have now also evaluated MIR and HDR, but found that the computational requirements of the method (which are also mentioned in Section 5 in [4]) are so high that neither method is a suitable alternative for GWA. In fact, even the small scale batch-approximations proposed in [4] are too computationally expensive to compete with GWA (we had to reduce the batch size on CIFAR-10 on a 48GB GPU). Nonetheless, the values of MIR seem to correlate with GWA in our preliminary investigation. We will add these results to the camera-ready version and believe further connecting these two approaches in future work is a promising way to integrate GWA and information-theoretic research.\n\n> The definition of the per-sample alignment score, $\\gamma$, becomes problematic when a sample is perfectly classified. For a cross-entropy loss, a perfect classification results in a gradient that is zero or near-zero, making the cosine similarity term $\\text{cos sim}$ undefined due to a zero-norm gradient vector. [...] concern is reinforced by the paper's own efficient gradient calculation, in line 153, which explicitly shows the gradient becomes a zero vector if $\\hat{y_i} = y_i$.\n\nThank you for this insightful question. You are correct, that in theory, the gradient $g_T$ becomes zero if $\\hat{y}_i = y_i$. However, the logits will never match the true one-hot class label $y_i$, even for perfectly classifiable data, as the softmax output probabilities only asymptotically approach the true class labels. In practice, the per-sample alignment will never reach 1, as we do not have perfectly classifiable data. Thus, the asymptotic theoretical insight in line 106 only serves as intuition for the range of alignment scores. If you believe this to be confusing for the reader, we are happy to remove it. Please note additionally that we also add a small epsilon to the denominator for numerical stability, as is standard practice.\n\n> The paper repeatedly claims that GWA is computationally efficient and scalable. However, no empirical evidence (e.g., wall-clock time, FLOPS) is provided [...] Could you provide a quantitative comparison of the computational complexity and runtime overhead of GWA.\n\nWhen training a ViT/S-16 implemented in JAX on ImageNet1k with a single NVIDIA RTX A6000, GWA adds $\\sim 2.5$sec to the per-epoch wall-clock time (on average 1861 images/s with GWA vs. 1867 images/s without GWA for $224^2$px). This is more efficient than evaluating a 1% validation set ($\\sim 16$sec overhead for one iteration). Computing the closed-form gradients and the cosine similarity requires $\\sim 0.003$ GFLOPs compared to 4.6 GFLOPs of a single forward pass with a ViT/S-16. Peak GPU memory when using active deallocation in this setting is 25.11GB with or without GWA (no difference). Thus, GWA has minimal overhead. We will add these quantitative results to the paper as you requested.\n\n> [...] experiments only report results averaged over three runs, with no analysis or visualization of the variance or consistency across these different initializations.\n\n> How robust is GWA to different model initializations and variations in model size within the same architecture (e.g., ViT-Small vs. ViT-Base)?\n\n> The rationale for Figure 3 is not entirely clear.\n\nOn Consistency: We already provide run-to-run variance (min-max deviation) for all methods in Appendix, Tables 5 and 6, showing that GWA's stability is comparable to or better than baselines. \n\nOn Robustness: While absolute GWA values can differ across model architectures and sizes (e.g., ViT-S vs. ConvNeXt), its strong positive correlation with final performance remains consistent, which is the key property, as shown in Figure 3 of the manuscript. We will additionally include quantitative results for different model sizes, which are consistent with the results in Figure 3 (i.e. lower performance correlates with lower alignment independent of model size).\n\nOn Figure 3 Rationale: The figure's purpose is to demonstrate GWA's reliability for model comparison. Each point is a distinct run (varied initializations, noise levels, architectures). The strong correlation proves that GWA consistently ranks models: a **higher peak GWA value reliably predicts higher final test accuracy**, validating its use for model selection across different training configurations.\n\n> What data augmentation transforms were used during training? Could these transforms, especially aggressive ones, affect the stability and interpretation of the GWA metric?\n\nAll experimental details, including augmentations (RandAug, Mixup), can be found in Table 4 in the Appendix. Empirically, we find that there is no negative impact on GWA when using augmentations. Rather, the model trains more consistently, learning better generalizing features, and resulting in higher GWA.\n\n> [...] more intuition on why the higher-order moments in GWA (i.e., the kurtosis correction) make it more sensitive to overfitting than traditional validation or methods like LabelWave, as suggested by the results in Figure 2?\n\nRegarding the use of higher order moments: A mean-only estimator is only sufficient for concentrated, unimodal distributions. This assumption is inconsistent with both the long-tailed theory of sample influence ([12, 13] in the paper) and our empirical observations (exemplarily Figure 2 right). Kurtosis provides a robust correction for this deviation that aligns with the long-tail theory. Note that we do not claim that the Kurtosis correction is the main reason for the strong capability of GWA in detecting overfitting, but rather the per-sample granularity that captures potential long-tail dynamics. LabelWave and other methods summarize the contributions of individual samples, e.g. by averaging, while we leverage the entire distributional view.\n\n> Paper Formatting Concerns\n\nAs you recommended, we will elaborate more on the perturbations of the CIFAR-C and ImageNet-C dataset within the paper and will address the other formatting concerns in the updated manuscript.\n\n[R5] Estimating training data influence by tracing gradient descent. Pruthi et al. (2020)."}, {"review_id": "ibF6uS5qDh", "reviewer": "Reviewer_YFoy", "summary": "This paper introduces Gradient-Weight Alignment (GWA), a novel metric that measures the coherence between per-sample gradients and model weights. During training, the peak of this metric closely corresponds to the peak of test accuracy. Notably, GWA can serve as an effective early-stopping criterion even in the absence of a validation set. The authors demonstrate that early stopping using GWA outperforms both traditional validation set baseline and the recently proposed LabelWave method. Furthermore, the gradient alignment approach facilitates the assessment of training data quality: higher levels of noise in the data result in lower GWA scores and a broader distribution of alignment values. Individual alignment values also help to detect mislabeled training examples.", "questions": "1. Is GWA actually better than the early stopping criteria mentioned in the p. 1 Weaknesses?  \n2. Is it truly necessary to adjust the expected alignment by the excess kurtosis to obtain a reliable generalization proxy? The paper does not include any comparison plots between the behavior of GWA and the pure expected alignment $\\mathbb{E}_i [\\mathcal{A}_T]$.\n3. Why do the authors measure the alignment of gradients with $w_T$ (the final weights) instead of $w_T - w_0$, where $w_0$ is the initial weight vector? Intuitively, aligning gradients with $w_T - w_0$ reflects whether per-sample gradients are coherent with the shift from the initialization point. A higher alignment would suggest meaningful learning is still occurring, while a lower alignment might indicate that optimization moves along a different direction, possibly signaling overfitting.\n4. Could you please explain how the formula for the gradients of the final layer’s weights (given on line 153) is derived? Also, I understand that calculating gradients only for the final layer reduces the computational cost of GWA, but would the metric behave differently if gradients from all layers were considered? \n5. Is it possible to apply GWA as an early stopping criterion in domains other than images and for tasks beyond classification?", "limitations": "Although the authors clearly acknowledge the limitations related to the computation of GWA, they do not address what I consider a major shortcoming of the work: the limited number of baseline methods considered (see p. 1, Weaknesses).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The paper introduces an early-stopping metric that can be computed efficiently during training, without requiring a validation set.\n2. The GWA metric also enables analysis of training data quality, such as detecting mislabeled examples.\n3. Experiments cover both training from scratch and fine-tuning scenarios.\n4. The paper is clearly written and easy to follow.\n\n**Weaknesses:**\n1. The experimental evaluation misses comparisons to several other methods that also explore stopping criteria without validation sets [1, 2, 3, 4, 5, 6]. In particular, [5], which measures disparity between different stochastic gradients, is closely related to GWA. These papers should be at least discussed in the related work section and ideally included as baselines.\n2. The GWA metric appears overly complex, and the authors do not provide enough intuition or explanation for why this complexity is necessary (see questions for more details).\n3. Although GWA seems broadly applicable across data types and training tasks, the experiments are limited to image classification only.\n\n**References:**\n\n[1] Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. Early Stopping without a Validation Set, 2017.\n\n[2] Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does Early Stopping Help Generalization against Label Noise?, 2019.\n\n[3] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Robust Learning by Self-Transition for Handling Noisy Labels, 2021.\n\n[4] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and Improving Early Stopping for Learning with Noisy Labels, 2021.\n\n[5] Mahsa Forouzesh and Patrick Thiran. Disparity Between Batches as a Signal for Early Stopping, 2021. \n\n[6] Ali Vardasbi, Maarten de Rijke, and Mostafa Dehghani. Intersection of Parallels as an Early Stopping Criterion, 2022.", "was_revised": true, "final_justification": "I have decided to raise my score by +1 pt, as the authors addressed my concerns during rebuttal. More specifically, they have discussed several other methods I mention in my review, and Gradient Disparity was even included as a baseline. Moreover, several other questions were clarified (kurtosis correction, GWA for non-classification tasks, and allignment with $w_T - w_0$).", "rebuttal": "> [...] evaluation misses comparisons to several other methods that also explore stopping criteria without validation sets [1, 2, 3, 4, 5, 6]. In particular, [5], which measures disparity between different stochastic gradients, is closely related to GWA. These papers should be at least discussed in the related work section and ideally included as baselines.\n\nThank you for these references. We will address your points one-by-one.\n* [1] Gradient Signal: this work is closely connected to the idea of gradient alignment that we also leverage and that we discuss in our related work (see line 66 and following with [23] being most closely related). We will add this paper to the discussion and want to point out that we actually compare GWA against similar work in Appendix A.2. While we found pairwise gradient alignment and gradient SNR to correlate with GWA, the approaches were noisier and more importantly, their associated compute overhead made it prohibitively expensive to run on the larger datasets and models we evaluate in this manuscript.\n* [2, 3, 4]: Learning with Label Noise: While not the primary focus of our work, we consider this line of work important and choose LabelWave as the primary representative baseline due to its recency and strong performance in the field. While effective in their respective regime, these approaches are often not sensitive enough to subtle overfitting, e.g. in robust models such as ConvNeXt (see Figure 2 center). This is a key limitation which GWA addresses.\n* [5] Batch Gradient Comparison: As per your suggestion, we have now run an additional experiment comparing GWA to the gradient disparity method from [5], which computes the L2 distance between successive batch gradients instead of per-sample gradients. We find that gradient disparity fails completely on our benchmarks (note that the most sophisticated experiment in [5] is a CIFAR-100 subset with 1.28k samples), while its computation also incurs substantial overhead (see also Table 4 in [5]) by requiring to store the gradients across multiple steps to compute the metric. Moreover, and unlike GWA, it does not provide any per-sample insights.\n* [6] Parallel Instances of Linear Models: This work proposes to train and compare two parallel instances of a linear model to determine early stopping. While this primarily theoretical work additionally proposes an extension to multi-layer networks, we could not find a suitable way to implement this for any current architecture or dataset evaluated in our work because the work provides no results on e.g. convolution or attention layers.\n\n| CIFAR-10-N             | 0%     | 9%     | 17%    |\n| ---------------------- | ------ | ------ | ------ |\n| Val Set 10%            | 89.86% | 85.33% | 82.30% |\n| GWA                    | 89.73% | 86.08% | 82.55% |\n| Gradient Disparity [5] | 55.85% | 57.93% | 53.85% |\n\n*Table R1: Test accuracy achieved with a ConvNeXt on CIFAR-10-N with different levels of label noise when selecting the optimal model with early stopping based on Gradient Disparity [5], the standard 10% validation set, and our proposed GWA method.*\n\n\nWe will incorporate a discussion of these approaches in the camera ready version.\n\n> The GWA metric appears overly complex, and the authors do not provide enough intuition or explanation for why this complexity is necessary. [...] Is it truly necessary to adjust the expected alignment by the excess kurtosis to obtain a reliable generalization proxy? The paper does not include any comparison plots between the behavior of GWA and the pure expected alignment $\\mathbb{E}_i[\\mathcal{A}_t]$.\n\nWe agree that a more intuitive explanation is beneficial and will add it to the paper. The core idea of GWA is straightforward: measuring the alignment between per-sample gradients and model weights. The perceived complexity in Eq. (2) and (3) stems from the explicit statistical formulation for computing the moments of the alignment distribution, which are essential for a robust and computationally efficient online estimator. Seeing as a rigorous definition is necessary, this is an instance where we chose clarity over simplicity.\n\nYes, the kurtosis adjustment is crucial. Relying solely on the mean alignment, $\\mathbb{E}[\\mathcal{A}_t]$, is insufficient because the mean does not capture the shape of the alignment distribution, specifically its tailedness. This property, measured by kurtosis, reflects the disproportionate influence of atypical samples, a key factor in generalization according to the long-tail theory (see [12, 13] in our submission). While the mean indicates the central tendency of alignment, our kurtosis-based correction allows the metric to better “see” these influential outliers. On datasets with noisy labels (e.g., CIFAR-10-N), where a subset of atypical samples is memorized late in training, the mean-only estimator is sensitive to these tails and provides a misleading early-stopping signal that continues to change after generalization performance has peaked.\nWe will add an ablation plot contrasting GWA with the uncorrected mean $\\mathbb{E}[\\mathcal{A}_t]$ to the final version to clearly demonstrate this necessity for kurtosis correction.\n\n> Although GWA seems broadly applicable across data types and training tasks, the experiments are limited to image classification only. Is it possible to apply GWA as an early stopping criterion in domains other than images and for tasks beyond classification?\n\nYes, GWA is broadly applicable, and its computational efficiency is designed for scalability across domains. GWA extends directly to other settings:\n* Self-Supervised Learning: GWA is applicable to contrastive learning methods where the InfoNCE loss functions can be seen as a multi-class cross-entropy task. We deferred a full analysis to maintain the paper's focus on clean per-sample attribution, which is more complex when gradients depend on the set of negative samples.\n* NLP: GWA's applicability to autoregressive language modeling is more straightforward, as their training objective is a cross-entropy loss over the vocabulary. The challenge is not in applying GWA but in interpreting the resulting token-level dynamics, which we identify as a promising direction for future work.\n\nPlease also refer to our response to reviewer RSWb on a related point.\n\n> Why do the authors measure the alignment of gradients with $w_t$ (the final weights) instead of $w_T - w_0$, where $w_0$ is the initial weight vector?\n\nThe reason for not using the vector $w_t - w_0$ is that this vector measures the displacement from initialization. Thus, aligning gradients with this vector would assess if learning continues along the path from initialization rather than towards convergence. This is not a known indicator of generalization. Conversely, our choice to align with $w_t$ is directly motivated by theoretical work [2, 25] on directional convergence, which shows that gradients of well-generalized samples align with the *current* weight vector $w_t$. This provides a theoretically-grounded measure, which we also show empirically (Appendix, Fig. 9) to be an effective online proxy for the alignment with “optimal” weights $w^*$ (proposed by Guille-Escuret et al. (2024) - see [28] in our submission).\n\n> [...] explain how the formula for the gradients of the final layer’s weights (given on line 153) is derived? [...] would the metric behave differently if gradients from all layers were considered?\n\nThe formula on line 153 is the standard closed-form gradient for a linear layer with a softmax cross-entropy loss. It is derived using the chain rule and represents the outer product of the latent representation $z_i$ and the prediction error $\\hat{y}_i - y_i$, a technique also leveraged by prior work [29].\n\nRegarding the use of full versus final-layer gradients: yes, the metric would behave differently. It would become computationally more complex and noisier. The final layer’s gradient provides the most direct and stable signal of the learned task, as a classifier’s goal is to learn a representation that is linearly separable by this layer [R3]. Conversely, gradients from earlier layers are more unstable [R4] and an indirect signal [R3]. This instability, coupled with the high dimensionality of full gradients, leads to a metric that trends towards zero, obscuring the generalization signal.\nWe have conducted an ablation study showing that: (1) including earlier layers produces a noisier metric, leading to premature and suboptimal early stopping; and (2) the alignment signal from earlier layers is significantly weaker (closer to zero) than the final layer's signal. We will include this in the camera-ready version.\n\n[R3] Understanding intermediate layers using linear classifier probes. Alain and Bengio (2016).\n\n[R4] Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability. Mikriukov et al. (2023)."}, {"review_id": "PoZXExLwru", "reviewer": "Reviewer_tLzt", "summary": "The paper proposes gradient-weight alignment (GWA), a metric to quantify generalization. GWA per sample is computed as cosine similarity between model weights and gradients- over datasets it is computed as a kurtosis-corrected average of per-sample GWA. The authors then propose an efficient way to estimate GWA online. Experimentally, the authors demonstrate that using GWA as an early stopping criterion outperforms baselines. The authors also conduct several other analyses showing other properties of GWA, including that it can predict label noise in a dataset.", "questions": "- Are there any theoretical results the authors can derive about GWA?\n- How does GWA compare with additional baselines (such as curvature-based methods)?\n- Why do GWA and LabelWave sometimes outperform the validation set\n- What is the statistical significance of the key results in the paper?\n- Empirically, how does removing the kurtosis correction in equation 2 change performance?\n- How does the GWA estimate of equation 3 compare with the ground-truth GWA?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is overall well-written and presented. Figure 1 is a nice illustration. Experimentally, the authors consider several large-scale datasets and architectures, which is a good choice. Notably, the authors also show several analyses of GWA in Section 4.\n\nOn the negative side, while the proposed method is theoretically motivated, the theory is from prior work; there is no new theory presented in this work. Moreover, the concept of gradient-weight alignment is itself not particularly new, although the particular formulation in equation 2 and the scalable estimator are. Given the limited novelty on the theory or conceptual side, the empirical results are very important. Unfortunately, there are several deficiencies in the experiments in my view:\n\n- Most importantly, GWA is compared to only one other baseline in Table 1 (LabelWave). I encourage the authors to add at least 2 other baselines such as the curvature-based methods mentioned in Section 2. I understand that these may be more computationally expensive (they are often not designed to be online methods), but including these is still critical in my view. There is no need for GWA to outperform these baselines either, especially the non-online ones.\n- Another important empirical concern is that GWA, and even sometimes LabelWave, outperform the validation set. Over many trials, this shouldn't happen on average: assuming iid data, the validation set is as close as one can get to actual generalization performance. This needs an explanation.\n- Relatedly, there are no error bars or standard deviations reported for all the key results in the main paper. This is important to include so the statistical significance of the results is clear.\n- As far as I can tell, there is also no ablation of whether the kurtosis correction in equation 2 is empirically important.\n- There is also no comparison of how the estimated GWA compares to the ground truth GWA.\n\nI don't mean to discourage the authors, but given that the paper is primarily an empirical one, these empirical details are quite important. If they can be addressed, I think this can be a solid contribution.", "was_revised": true, "final_justification": "One of my main concerns about the statistical significance of the results was addressed by the authors; therefore, I am increasing my score.", "rebuttal": "> [...] GWA is compared to only one other baseline in Table 1 (LabelWave). I encourage the authors to add at least 2 other baselines such as the curvature-based methods mentioned in Section 2. I understand that these may be more computationally expensive (they are often not designed to be online methods), but including these is still critical in my view. There is no need for GWA to outperform these baselines either, especially the non-online ones.\n\nThank you for your constructive feedback. You are correct that a broad comparison is critical given our paper's empirical focus. We would like to point out that the paper already includes comparisons to non-online but relevant methods like gradient norm for TracIn, $w^*$-alignment and pairwise gradient alignment in Appendix A.2 (Figs. 9, 10). Moreover, to address your concern directly, we have now added results for three more baselines: curvature-based influence, Neural Collapse-based MIR/HDR [R6], and Gradient Disparity [R7], as also recommended by other reviewers (CWdC and YFoy). The results from these baselines further corroborate the central conclusions of our work: validation accuracy itself is a formidable SOTA baseline, and the aforementioned baselines consistently underperform it (e.g. Gradient Disparity underperforms GWA by $\\approx 30$%) and/or incur substantial computational overhead. Our proposed GWA is actually the first technique to offer a suitable alternative that can compete with validation accuracy while being efficient enough for large scale applications.\n\n> Another important empirical concern is that GWA, and even sometimes LabelWave, outperform the validation set. Over many trials, this shouldn't happen on average: assuming iid data, the validation set is as close as one can get to actual generalization performance.\n\nYou correctly point out that GWA can outperform validation-based stopping, which would be unexpected under a strict i.i.d. assumption.\nHowever, this assumption is usually violated in practice due to inherent dataset biases. Our own results in Table 1 support this: the performance difference between the 10% and 1% validation splits indicates that small, fixed validation sets suffer from higher sampling bias and do not perfectly mirror the test distribution. GWA is less susceptible to this bias as it derives its signal from the dynamics of the entire training set (which is usually a much larger sample from the distribution), providing a more robust estimate of generalization.\n\n> [...]  there is also no ablation of whether the kurtosis correction in equation 2 is empirically important. [...] Empirically, how does removing the kurtosis correction in equation 2 change performance?\n\n> Relatedly, there are no error bars or standard deviations reported for all the key results in the main paper.\n\n> [...] no comparison of how the estimated GWA compares to the ground truth GWA. [...] How does the GWA estimate of equation 3 compare with the ground-truth GWA?\n\nYou are correct that the kurtosis correction is a key component. Its purpose is to account for the distribution's tailedness, an indicator of outlier influence that the mean alignment score alone cannot express (lines 125-131). We have conducted an ablation showing that GWA without this correction is less robust, often signaling to stop training prematurely, and we will add this to the manuscript.\nPlease also compare to our response to reviewers iv9T and YFoy.\n\nOn error bars: Tables 5 and 6 in Appendix A.3 provide the min-max accuracy ranges across all runs. We will also add these to the main manuscript.\n\nEstimator vs. Ground Truth: Figure 6 in Appendix A.1 directly compares our efficient online estimator with the offline ground truth, showing a near-perfect correlation. This validates our scalable approach.\n\nWe will integrate these into the main paper for the camera-ready version.\n\n> Are there any theoretical results the authors can derive about GWA?\n\nThank you for this feedback. We agree that a formal theoretical analysis of GWA is a compelling direction. Our paper's focus, as stated in the introduction (lines 36-37), is deliberately empirical to investigate GWA's practical utility under realistic, non-ideal conditions.\nWhile GWA is motivated by prior theory on directional convergence ([2, 25] in our manuscript), a formal analysis under realistic data noise, characterizing how noise impacts the alignment distribution, is a highly challenging (possibly intractable) open problem. Our work consciously focuses on empirical evaluation in realistic scenarios and computational efficiency for large-scale models. \n\nWe thank you for suggesting revisions, including the added baselines and ablations, which we believe have further strengthened our manuscript.\n\n[R6] Unveiling the Dynamics of Information Interplay in Supervised Learning. Song et al. (2024).\n\n[R7] Disparity Between Batches as a Signal for Early Stopping. Forouzesh and Thiran (2021)."}]

Author rebuttal:

Meta-review: This paper proposes an efficient per-sample estimator of gradient–parameter alignment, focusing on last-layer gradients computed in batches. The method provides a practical signal for early stopping and identifying influential training samples.

Reviewers raised concerns about broader evaluation across tasks, additional ablations on metric design, comparisons with related work, more experimental evidence, and new theoretical analysis. The authors actively engaged in the discussion and addressed most concerns, leading to one Accept, two borderline accepts, and two borderline rejects. 

The AC carefully considered the two borderline rejects: Reviewer tLzt’s concerns were largely addressed, while Reviewer RSWb remained unresponsive during the discussion. The remaining weaknesses involve evaluation on additional tasks/settings and providing new theoretical justification, which are valuable but largely beyond the empirical scope of this work. Given that the proposed metric is simple, efficient, and effective, with potential impact on future research regarding generalization, the AC recommends acceptance of this paper and encourages the authors to include more evaluation on other tasks/settings in the final version.
Decision: Accept (poster)