Memorization Through the Lens of Sample Gradients
Authors: Deepak Ravikumar
Efstathia Soufleri
Abolfazl Hashemi
Kaushik Roy
Venue: ICLR 2026 Poster (2026)
Keywords: Memorization
Sample Gradients
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Deep neural networks are known to often memorize underrepresented, hard examples, with implications for generalization and privacy.  Feldman & Zhang (2020) defined a rigorous notion of memorization. 
However it is prohibitively expensive to compute at scale because it requires training models both with and without the data point of interest in order to calculate the memorization score.
We observe that samples that are less memorized tend to be learned earlier in training, whereas highly memorized samples are learned later. 
Motivated by this observation, we introduce Cumulative Sample Gradient (CSG), a computationally efficient proxy for memorization. CSG is the gradient of the loss with respect to input samples, accumulated over the course of training.
The advantage of using input gradients is that per-sample gradients can be obtained with negligible overhead during training. The accumulation over training also reduces per-epoch variance and enables a formal link to memorization. Theoretically, we show that CSG is bounded by memorization and by learning time.
Tracking these gradients during training reveals a characteristic rise–peak–decline trajectory whose timing is mirrored by the model’s weight norm. This yields an early-stopping criterion that does not require a validation set: stop at the peak of the weight norm. This early stopping also enables our memorization proxy, CSG, to be up to five orders of magnitude more efficient than the memorization score from  Feldman & Zhang (2020).  It is also approximately 140 $\times$ and 10$\times$ faster than the prior state-of-the-art memorization proxies, input curvature and cumulative sample loss, while still aligning closely with the memorization score, exhibiting high correlation. Further, we develop Sample Gradient Assisted Loss (SGAL), a proxy that further improves alignment with memorization and is highly efficient to compute. Finally, we show that CSG attains state-of-the-art performance on practical dataset diagnostics, such as mislabeled-sample detection and enables bias discovery, providing a  theoretically grounded toolbox for studying memorization in deep networks.

Published as a conference paper at ICLR 2026
MEMORIZATION THROUGH THE LENS OF SAMPLE GRADIENTS
Deepak Ravikumar's Efstathia Soufleri', Abolfazl Hashemi', Kaushik Roy! 'Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA 2 Archhmedes, Athena Research Center, Greece fdravikum, abol fazl, kaushik { edu, soufleereeathheeerr..ggr
ABSTRACT
Deep neural networks known to often memorize underrepresented, hard exam- ples, with implications for generalization and privacy. Feldman & Zhang (2020) defined rigorous notion of memorization. However it is prohibitively expen- sive to compute at scale because it requires training models both with and without the data point of interest in order calculate the memorization score. We ob- serve that samples that are less memorized tend to be learned earlier in training. whereas highly memorized samples are learned later. Motivated by this obser- vation, we introduce Cumulative Sample Gradient (CSG), computationally ef- ficient proxy for memorization. CSG is the gradient of the loss with respect to input samples, accumulated over the course of training. The advantage of us- ing input gradients is that per-sample gradients can be obtained with negligible overhead during training. The accumulation over training also reduces per-epoch variance and enables formal link to memorization. Theoretically, we show that CSG is bounded by memorization and by learning time. Tracking these gradients during training reveals a characteristic rise-peak-decline trajectory whose timing is mirrored by the model's weight norm. This yields an early-stopping criterion that does not require a validation set: stop at the peak of the weight norm. This early stopping also enables our memorization proxy, CSG, to up to five orders of magnitude more efficient than the memorization score from Feldman & Zhang (2020). is also approximately 140x and 10x faster than the prior state-of-the- art memorization proxies, input curvature and cumulative sample loss, while still aligning closely with the memorization score, exhibiting high correlation. Fur- ther, we develop Sample Gradient Assisted Loss (SGAL), proxy that further improves alignment with memorization and is highly efficient to compute. Fi- practical dataset
nally, we show that CSG attains state-of-the-an performance diagnostics, such as mislabeled-ssmple detection and enables bias discovery, pro- viding a theoretically grounded toolbox for studying memorization in deep
INTRODUCTION
Deep learning has become the de facto standard across a wide array of machine learning tasks from generative models (Ho et al. 2020), and classification (Krizhevsky al. 2009 Soufleri al.) 2024a), to reinforcement learning (Shakya et al.) (023). Despite their success, deep neural net- works are prone to memorizing the training data. While some degree of memorization is needed for achieving generalization (Feldman 2020). it has been shown that these models are also capable of memorizing completely random labels (Zhang et al. (017). Understanding the mechanisms of memorization has thus emerged as a key research focus with broad implications for generalization (Bayat et al. 2025 Brown al. 202T) and privacy (Ravikumar al., 2024a).
Several methods have been proposed to measure memorization. [eldman 2020) offers principled definition with strong theoretical and empirical support (Feldman & Zhang 2020). However, it is computationally expensive. Their score quantifies memorization as the change in model's output
"Corresponding Author
ent Memori zati

---

Published as a conference paper at ICLR 2026
0.010
Arg. Train Sumple Gitd.
Arg. Train Sumple Grad
Validation (logscali)
0.008
Validation Loss
0.006
6 10
0.004
0.002
4 10
0.000
10
10'
10
10a
Epochs
(a) Lowest CSG images from CIFAR-100.
(b) Highest CSG images from CIFAR-I00.
(c) Average sample gradient and validation loss.
Figure 1: (a) Lowest CSG images from CIFAR-100 for 6 classes: captures examples prototypical for the class i.e. easy examples. (b) Highest CSG images from CIFAR-I00 for the same 6 classes. (see Figure 16 for ImageNet): captures examples atypical for the class i.e. hard examples, likely memorized. (c) Plot of average train sample gradient for ResNet18 trained on CIFAR-100 and the
corresponding validation set loss. The peak in sample gradient corresponds to the lowest validation loss (first descent of double descent). Thus, one can early stop without needing a validation set!
when a specific training sample is removed, requiring the training of (dataset size) models. To address this, proxies such as model confidence (Carlini et al., 2019b), learning time (Jiang et al. 2021) and adversarial distance (Del Grosso al. 2022) have been proposed. Yet, eritical limita- tions persist among these alternatives. Methods relying on optimizer discrepancies (Agiollo et al. 2024 or influence functions (Pruthi et al. (020) often incur high computational costs due kolold validation and parameter-space single-sample gradients, respectively. Similarly, metrics based on gradient variance, such as VoG (Agarwal et al. 2022), can misidentify consistently hard samples and are ad hoc metrics lacking theory linking them to memorization. Leveraging training dynamics has led to the development of further proxies. including loss sensitivity (Arpit et al. 2017). forgetting frequency (Toneva et al. (2019). the C-score (Jiang et al.] (2021). and cumulative loss (Ravikumar af.) 2025a). Other approaches examine loss landscape sharpness (Krueger et al.) 2017) and input loss curvature (Garg et al., 2024), with RRavikumar et al. (2024a) establishing theoretical links to memorization and privacy. While these proxies offer valuable insights, many fail to capture critical properties of memorization, such as its bi-modality (Lukasik et al. 2023). Developing a stronger theoretical foundation is therefore essential. We observe that less-memorized samples tend to be learned earlier in training, whereas highly memorized samples are learned later (see Fig. 2 Mo- tivated by this, we propose a new proxy, Cumulative Sample Gradient (CSG). CSG is defined as the gradient of the loss with respect to the input data, aggregated across training. We establish theoretical framework linking CSG, memorization, and learning time, and validate it experimentally by showing that CSG is highly correlated with memorization (Feldman & Zhang 2020
Interestingly, the average per-sample gradient follows a rise-peak-decline trajectory (see Figure pattern mirrored by the network's weight norm. We show this arises because the per-iteration weight norm bounds the sample gradient. Crucially, the peak coincides with the minimum validation loss (the first minimum of double descent, Section 4.3). This yields a simple early-stopping criterion: stop at the peak of the weight norm. Using this property, we introduce Sample Gradient Assisted Loss (SGAL. proxy that further improves alignment with memorization. By early stopping. CSG and SGAL achieve up to five orders of magnitude speedup over estimation of memorization by (Feldman & Zhang 2020), and are substantially faster than prior state-of-the -fethrrrrt proxies, including curvature (Garg et al. (224)) (up to 140x) and CSL (Ravikumar et al. (025a) (max 10x, avg. 5x), Finally, we demonstrate that CSG effectively discovers dataset bias and achieves state-of-thh-an performance in identifying mislabeled samples. In summary, our contributions are:
Theoretical foundations: We develop theoretical framework linking CSG to memoriza- tion and learning time, offering up to 5-orders-off of--maniitude speedup over memorization score Feldman & Zhang] (2020), 140x and 10x speedup than previous of-te-ofthhe-a art memorization proxies such as Curvature (Garg et al.] 2024) and CSL (Ravikumar et al, 2025a), respectively.
2

---

Published as a conference paper at ICLR 2026
Practical impact and insights: CSG achieves of-the-rtt performance for identifying mislabeled samples. Our experiments validate that sample gradients enable early stopping without validation set, and reveal biases in training data.
NOTATION
We denote distributions using bold capital letters V. random variables sampled from them italic small letters v for scalars, i for vectors, and capital letters V for matrices. Consider a learning problem, where the task is learning the mapping
x H y where F X E R" and y Y z consisting m samples, where each sample 2 (i) Z. Our analysis also makes use of leave one out dataset denoted by 5% (#1,-.., E Z4+1:-... Em), which is S with the 7th sample removed. In this paper. we consider a deep neural network with layers. Its parameters at iteration t are denoted by vi. [wm(") v .... o W. Here, indexes the optimization iteration, and w²!) collects the parameters of layer k: We use a flattened representation of the parameters. For the k-th layer (with input dimension dx-1 and output dimension d+), the weights represented as the row vector (k) w "i,d,,de.. which lists the entries Wi,eg of the (unflattened) weight (k)
X E R. Let this task be learnt using a dataset S = (2,,,2,..... .e.....,,, Zm)
(<k)) = [ W1,1
(k)
(b)
matrix for i = {1...., [.....,d]] ) and j {1....,de da-1). 1). Let ge Gs to denote the function learned by the neural network. The neural net is trained by the application of a possibly random training algorithm A such SGD and p P denotes the randomness of the algorithm. For example in mini-batch SGD there are two sources of randomness, (a) the choice of mini-batch (b) model initialization. Here, p captures (a). To evaluate model performance, we use the loss sample E, defined as ((g, 5,) = (Tt), ys). Since the function learned at iteration f, St. is fully parameterized by w., we use E(V,, - E(gr 2;) interchangeably. Typically, we interested in the loss of g over the entire data distribution, called the population risk, which is defined as R(gg) = Ez l (g, z)]. In practice we evaluate the empirical risk as follows Remp (g. S) = T E ((g, E S.
RELATED WORK AND BACKGROUND
Understanding memorization has broad implications, ranging from generalization (Brown et al. 2021 Zhang et al. 2017 (021). unlearning (Kodge et al. [025 2024 Kurmanji et al. 2003). and privacy (Dwork et al.] [2066] /Soufleri et al. (024b). to identifying mislabeled and rare subpopula- tion examples (Pleiss et al. 2020; Maini et al.] 2022 Ravikumar et al. 2023). While many notions of memorization exist from exact (Kandpal et al. (022) and k-eidetic (Carlini et al. (022)) to ex- tractable memorization (Nasr et al.) (023) in this paper we focus stability- also referred to as counterfactual memorization, introduced by Feldman (2020). We hereafter refer to this as the memorization score. Formally, the memorization score of the 7h sample ā, = (7, 91) for a network og trained on dataset using algorithm A (with randomness p) is defined as:
memorization,
mem (5, x) = Prig (f.)) vil Pr(ge. (5%) = =y:
The term ceabbllttybbseed comes from its connection to uniform stability. concept central to this work. Formally, an algorithm A is 9-uniform stable (Kearns & Ron 1997) if for some 3 > 0:
[(gg =) - E(gg... =) S B, Vi E 1,........0
An example of such algorithm is Stochastic Gradient Descent (SGD) (Hardt et al. 2016). Proofs of its stability and convergence typically rely on a set of conditions we refer to as the SGD Con- vergence Assumptions: p-Lipschitz continuity (Eq. [8). bu-bounded variance (Eq, 9 unbiased gra- dients, and a decreasing learning rate T) at each iteration t. Under these assumptions, SGD is guaranteed to converge in gradient norm (Ghadimi & Lan 2013) (see Appendix A). Some results additionally require an L-bounded loss satisfying 0 < E < L. While stability-based memorization provides a principled definition, it is prohibitively expensive to compute, as previously discussed. To overcome this, researchers have proposed proxies such model confidence (Carlini et al. 2019b),
learning time (Jiang et al.) 2021). and adversarial distance (Del Grosso et al., (022).. Other methods leverage training dynamics such as loss sensitivity (Arpit al. [201]] forgetting frequency (Toneva ct al. 2019). the C-score (Jian et al., 202I). and cumulative loss (Ravikumar et al. (025a). More
recently approaches study the loss landscape, e.g., sharpness (Krueger al. 2017) and input loss
3

---

Published as a conference paper at ICLR 2026
Hard Example. Learnt Late
Sample Gradient
(w.r.t input)
10-1
10"4
Easy Example, Learnt Early
10-7
Easy
Hard
10"
10'
10
Epochs
Figure 2: Visualizing Input Gradient Norms for Peacock-Class Samples in ImageNet. We plot the input gradient norm (for ResNet18) for one easy and one hard peacock-class training example. Solid lines show individual sample norms; dashed shows the average for the sample over training. The easy example is learned early, while the hard example learned later and memorized more.
curvature Garg al. 2024: Ravikumar et al., 2024a). Altemattve proxies relying on optimizer discrepancies (Agiollo et al. 2024) or influence functions (Pruthi et al. (020) have also been pro- posed; however, these often incur high computational costs due to requirements for k-fold validation or parameter-space gradient calculations. Similarly, metrics based on gradient variance, such as VoG (Agarwal al., (022), focus on fluctuation rather than accumulation, which can misidentify consis- tently hard samples. Finally, while memorization in language models has been analyzed via exact sequence matching (Leybzon Kervadec 2024), such methods often lack theoretical framework linking input gradient dynamics to formal memorization definitions. In contrast, work investi- gates sample gradients (not weight but input) across training as a theoretically grounded proxy for memorization.
4 CUMULATIVE SAMPLE GRADIENT (CSG)
To develop intuition and motivate the introduction of CSG, we analyze how the input loss gradients evolve during training for two samples. Consider an "easy" and "hard" example from the same class (peacock) in the ImageNet dataset (Russakovsky et al., (015). Figure 2 plots the norm of the gradient of the loss w.r.f. input (sample loss gradient) vs epochs. This captures how well the model fits each sample over training. The easy sample, learned early, shows a rapid drop in gradient that stays low. The hard sample, by contrast, maintains high gradient much longer. This contrast highlights how tracking the average sample gradient during training helps distinguishing between easy and hard examples. Traditional metrics like learning and forgetting time rely on thresholds and can be noisy e.g., a sample may appear learned, then forgotten, then relearned. To address this, we introduce cumulative sample gradient (CSG), which smooths such fluctuations and provides more reliable signal. Hard examples tend to be memorized; easy ones are likely generalized which captured by CSG (see Figures
lb 16a and 16b). CSG for a sample z (Z y:) defined as:
CSG( T Tmaz En [[ l (wn)
IV (17
where Tiaax the total number of iterations of SGD For convenience, we introduce the concept of the sample learning condition ER [l] (wR) using the sample learning condition as
< thus learning time Ts, can be defined
TV. min (T: ER V l (wR)13 <T}
(4)
where the gradient is with respect the input Z.. Here, R T is a discrete random variable sampled non-uniformly from the set (1,2...., TT,ar. This borrows from the SGD convergence result of Ghadimi & Lan (011)). We interpret the definition follows: sample is deemed learned when the expected sample gradient drops below a predefined threshold T.
Note that the approximation is justified by the law of large numbers.
4

---

Published as a conference paper at ICLR 2026
4.1 CSG AND LEARNING TIME
Before stating our main result, we recall two preliminaries. First. consider SGD with update rule w = dr - TI V w. f(W7g 2,), where 7h is the learning rate and V.. is an unbiased stochastic gradient estimator. Following Ghadimi & Lan (2033) decreasing step-size schedule required to ensure convergence, such that the sum 1j Lipschitz constant (Eq. [). Second, building on Ravikumar al. (2025a). show that the imput gradient is bounded by the weight norm and the weight-gradient norm. Since the weight-gradient norm is known converge during training (Ghadimi & Lan, 2013), and the input norm remains fixed, the weight norm effectively bounds the input gradient. This a re-framed version from Ravikumar al. (225a), but helps build towards our result. Formally we state the lemma below
Eizo (n)- 5n?)
0o, with p denoting the
Lemma 4.1. For any neural network without a skip connection at the first layer, the Frobenius norm of the gradient of the loss l with respect nonzero input Z. for sample Σạ (Ti, Ui) is bounded by the norm of the network weights Wyy
Nv,e(uus F S
[elll VV 2i)1 /
(5)
The lemma also holds for a mini-batch Xg. In which case, the constant becomes 1/1|ž, becomes [[xx,) }+ ||r/sp where sp is the smallest singular value of P =x,((x) and + denotes pseudo- inverse. Using this general version (see proof in Appendix C.I we define at each iteration t the /sp +lll///ss//////pppp///
factor kl = ||ū]|l=]((x]]""" result.
and f'm max, (N²)
We are now ready to state our main
Theorem 4.2 (CSG is bound by Learning Time). Consider a deep network without skip connec- tions in the first layer, trained by SGD. Suppose the SGD convergence assumptions hold, the loss L-bounded and SGD is B-stable. Then there exist constants C1, C2 such that the expected cumulative sample gradient (CSG) and expected learning time Ts, of a sample 2n satisfy:
(CS. S C1 [T=.] + C22
(6)
Sketch of Proof. The proof establishes bound on input gradient norms by adapting convergence results from randomized SGD but in the input space. It then connects this to the learning condition. By leveraging leave-one-out analysis and the 3-stability of SGD, separate the memorization term. Combining these pieces yields the final result. The proof is provided in Appendix C.2
Understanding the terms. In the result, 7) denotes the maximum learning rate during training. Tiax the maximum number of iterations, P the Lipschitz constant, and T, the variance bound. With these definitions recalled, the constants
..2 DT
K + 7
Kim Ds Tiaax P 12 2 0,
C
C₂
2n
21
the constant C1 depends on the problem-defined threshold т, which can be chosen so that C1 is positive. The absolute value of is not essential; rather, the role of is to help provide theoretical
connection. C2 is determined by parameters related to 3-stability and assumptions from SGD, and therefore positive fixed quantity once the network, and data are specified.
Interpreting the Theory. Theorem 4.2 can be interpreted as follows: for any group or subset of training samples, their CSG scores are bounded by the group's 's learning time. To see this, consider the subset UTT) = { Ts. < T}, which contains all samples learned within a time threshold T. For any randomly chosen subset U drawn from the same distribution as the full dataset, we can define the corresponding threshold as T max(T., 2 U}. Since Equation 6 holds for all such subsets, it follows that the average CSG within subset is bounded proportion to its average learning time. Finally, this can be validated in theory by grouping samples and observing linear relationship between CSG and learning time, which is performed also predicts that large cumulative input gradients must result in either very large learning times or samples that are never learned such as mislabeled examples. This is indeed the case, by thresholding on CSG achieve of-te-ofttheann performance in mislabeled sample detection (see Sec. 5.4
Section 5.T The theory
4.2 CSG AND MEMORIZATION
Theorem 4.3 (CSG is bound by Memorization). Consider a deep network without skip connections in the first layer, trained by SGD. Suppose the SGD convergence assumptions hold, the loss is L--
5

---

Published as a conference paper at ICLR 2026
Easy Sample Inp. Grad.
x10F
Inp. Graa.
Val loes.
4.5
Validation loss
Weight Norm (L2)
6.5
Log Val Loss
Wetpee Norm
-2.5
Log Val. Loss
Costs: Simiarty af w/Mter
0.8
4.0
0.6
6.0
3.0
1/
M
3.5
5.5
0.4
Epoci
Epoci CSG
160
200
240
280
60
120
180
240
300
3.5
0
60
120
180
Epoch
Epuch
Epoch
(b)
(c)
Figure 3: (a) CIFAR-100. The curve closely follows the second descent validation loss (shown in red), highlighting their correlation. (b) Validation loss and weight norm plotted together. The first descent in the loss aligns with the peak in weight norm, reinforcing its connection to the doublc-descent. (c) Similarity between input gradients and memorization scores across epochs (blue), along with the cumulative similarity (green). The CSG similarity (green) plateaus around the first descent point.
norm over training epochs for the 100 easiest examples (lowest CSG)
bounded and SGD is B-stable. Then, memorization bounds cumulative sample gradient as
Es, [CSG(E)) = ((E,, (mem( (2)))
Sketch of Proof. The proof derives both lower and an upper bound on the learning time using convergence properties of SGD and the learning condition. We eliminate learning time using the two bounds, to get an quadratic in terms of the CSG, where the coefficients depend on sample memorization. Analyzing this quadratic reveals that it's roots provide bounds on the CSG, showing CSG is linearly bound to memorization. Full proof is available in Appendix C.3
Interpreting the Theory. The coefficient for O-notation, is proportional to b/2a where and are the second and first order coefficients of the quadratic (See Eq. (2). Similar to Theorem 4.2 Theorem 4.3 applies to every set U(m) & mem( I m)), where all samples have memorization bounded by m. For such subsets, the cumulative sample gradient (CSG) is linearly bounded in memorization. In other words, groups or subsets with lower memorization tend to have lower CSG, while subsets with higher memorization are likely to exhibit higher CSG.
Remark Assumptions. Uniform stability of SGD has been demonstrated in prior work (Hardt ett al. 2016), lending support to our stability assumption (Eq. 2}. The Lipschitz continuity assump- tion is justified by results from Virmaux & Scaman (2018), where they derive general upper bounds on the Lipschitz constant for differentiable deep models. Assumptions on an unbiased gradient estimator and bounded gradient variance are standard in optimization theory (Aketi al., 2024] Ghadimi & Lan 2013). While we assume L-bounded loss, the proof can be potentially extended to cross entropy loss which unbounded using result A.4 from Ravikumar et al. (2024a). Additionally, most architectures such as ViTs, ResNets and VGGs don't have skip connection from the input. Key Takeaways. If the theory bounds are tight, we expect the following: (1) learning time to exhibit linear relationship with CSG, (2) memorization to follow linear relation with CSG.
4.3
SGAL AND LINKS TO DOUBLE DESCENT
Peaky trajectory. Figure Ic shows that the average sample gradient norm rises from a low baseline, peaks sharply, and then declines. The weight norm follows the same trend (Fig. [3b), since it upper bounds the sample gradient (Lemma 4.1 The weight norm is shaped by two opposing forces: performance loss pushes it upward to fit the data, while E2 weight decay and SGD's implicit bias toward minimum-norm solutions (Park et al. 2023) pull it downward. Interestingly we also see this when using the Adam, AdamW, Adagrad, and RMSProp optimizers (see Appendix B.4 and Figures 10 and IT]. Furthermore, we demonstrate consistency across different architectures specifically ViT ViT-M-1662550 and ResNet50 in Figures 12a and 12b respectively. These results provide evidence that this phenomenon is robust to variations in both architecture and optimizer.
Connection to double descent. Double descent refers to the phenomenon where test loss decreases, then increases, and decreases again as a function of iterations, parameters, or dataset size. Here, we focus on iterations. This behavior is evident in Fig. where the test loss first drops, then plateaus,

---

Published as a conference paper at ICLR 2026
CIFAR- 100
CIFAR- 100
ImageNet
ImageNet
160
250
300
CSG
1.0
1.0
CSG
200
CSG
200
150
0.5
0.5
100
100 X
0.0 0.2 0.4 0.6
0.8 1.0
0.0
0.2 0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4 0.6 0.8 1.0
Avg. Mem Score
Learning Time
Avg. Mem. Score
Learning Time
(a)
(h)
(c)
(d)
Figure 4: (a) Plots memorization score vs CSG on CIFAR-100 for Inception model (same arch
as Feldman & Zhang 22000) (b) Plots learning time CSG for CIFAR-100 (Inception) (c) Plots
memorization score vs CSG on ImageNet for ResNet50 (same arch. as Feldman & Zhang (2020)
(d) Plots learning time CSG for ImageNet.
rises, and finally drops again. Classical double-descent studies identify the first validation-loss min- imum as the boundary between interpolation and generalization regimes (Poggio et al.) 2019 Scha- effer et al. 2024). We observe that average CSG gradient peak lands on that same boundary (see Figure Lc). Additionally, the peak, sample gradients maximally aligned with memorization scores Feldman & ZZang 2020) (see Figure 3c) providing more evidence for the link.
Practical takeaway. Let's consider what happens under strong weight penalty. Strong penalty pushes the weight norm down immediately during training. Upon further training the performance loss pushes it back up, and finally SGD-bias drags down. While weight decay is plays an important role, we found that for standard training recipes, can stop peak weight norm. Ideally, moni- toring both sample gradient and weight norm is recommended (see Figures and and 11 for detailed effect of weight decay).
SGAL. In Figure [c] we show the similarity between the input gradients and the memorization score across epochs (in blue), as well as the cumulative similarity up to each epoch (in green). We observe that CSG plateaus around the first descent point, or shortly after. This suggests that we can avoid training on the full dataset and still achieve strong similarity with memorization, significantly lowering compute costs. Motivated by this, we introduce sample Gradient-Assisee early stopping with accumulated Loss (SGAL), a practical proxy for memorization. SGAL accumulates sample loss until the gradient indicates an optimal stopping point, on average allowing training to stop after 10-30% of epochs and yielding a 3-10x efficiency gain. The theory behind SGAL's link to memorization is justified using the results from Ravikumar et al. (2025a). Additionally, input gradients also identify the second descent without validation set, by analyzing gradients of the easiest examples (low CSG). Figure Ba shoss this on CIFAR- 100, where gradient norms of the 100 easiest examples align closely with validation loss.
EXPERIMENTS
5.1 VALIDATING THEORY
Experiment. Here, we examine how well our theory (i.c. Theorem 4.2 and 4.3 holds in prac- tice. We conduct the experiment by training Inception (Szegedy et al. 2) model on CIFAR-100 (Krizhevsky et al., 2009) and ResNet50 (He et al., (0166) on ImageNet (Russakovsky et al.) 2015), same as Feldman & Zhang Feldman & Zhang (2020). For memorization scores, we utilize the pre- computed memorization scores from Feldman & Zhang (2020). Since the theorems apply to groups of samples (i.e. under expectation), we grouped data by the x-axis metric and calculated average scores for each bin. We plot a binned scatter plot of these metrics (see Appendix B.l for more details).
Results. The results visualized in Figures 4a
and 4d Figures 4a and 4c plots the the
memorization score from Feldman & Zhang (2020) against CSG for CIFAR-100 and ImageNet re-
spectively. Figures and 4d plots learning time CSG for CIFAR- 100 and ImageNet respectively.
Takeaways. Theorem predicts a linear bound between learning time and the sample gradient, and Theorem 4.3 also predicts linear relationship with memorization. These predictions are sup-
7

---

Published as a conference paper at ICLR 2026
Table 1: Cosine similarity (CS) and Pearson correlation (Corr.) between CSG and baselines with memorization (Feldman & Zhang 2020), and normalized compute cost. i Computationally infeasi- ble scale to ImageNet.
Method
Compute
CIFAR-100
ImageNet
Corr.
CS
Corr.
Final Sample Loss
0.24
0.17
0.63
0.49
Forget Freg. Toneva et al. 2019
0.76
0.59
0.49
0.04
Loss Sensitivily Arpil d 2017
1x
0.81
0.76
0.49
0.17
CSL. [Ravikumar cl al. 20228
0,87
0.79
0.79
0.64
VoGi Ckpss (Agarwal
0.60
VoG Last Ckpts (Aparwal et
2122
1x
0.33
0.23
0.58
0.32
GraNd (Paul ct al. 2021
7,8x
0.81
0.36
f
Curv. Garg ct al. 2024
14x
0.69
0.49
0.62
SAMIS [Apollo
2024
24x
0.53
0.69
Tracin (Pruthi et al. 2120
26x
0.83
0.71
SGAL (Ours)
0.1
0.3x
0.86
0.77
0.78
0.62
CSG (Ours)
0.71
Table 2: Calibration results (mean + std) for CIFAR-100 ResNet18 checkpoints
Metric
Ours
Val. Loss Stop
Last Epoch
Accuracy
0.6306
0.0103
0.6262 0.0096
0.7492 = 0.0046
ECE |
0.1382 0.0262
0.1260
0.0083
0.1017 0.0031
MCE
0.2721 = 0.0458
0.2352
0.0203
0.2794 + 0.0149
MSCE
0.4208
0.1620
0.3257
0.0355
0.4427 0.0410
UCE
2.0821 + 0.4648
1.8667
0.1424
2.1839 + 0.0840
ported by experimental results shown in Figures 4a 4b
and 4d which empirically confirm these trends. There is a slight deviation from the predicted linear relations at high learning-time and mem- orization respectively. This is likely due the theoretical assumption of uniformly bounded loss. In practice the cross-entropy loss not uniformly bounded. This issue is also discussed in detail by Ravikumar al.] (2024a), where accounting for subpopulation based loss bound improved the match between theory and practice.
4c
5.2 SIMILARITY WITH MEMORIZATION
Experiment. We train Inception CIFAR-100 and ResNet50 on ImageNet, computing memo- rization proxies and comparing them to memorization scores from Feldman & Zhang (2020) us- ing cosine similarity and Pearson correlation. We also evaluate L2 adversarial distance and MIA performance (Appendix I) relating them to both memorization scores and CSG (see Appendix 9bb... Additional setup details in Appendix B.2
Figures
Results. Table shows the similarity with memorization score of each of the proxies: CSG, SGAL, CSL Ravikumar et al., 2025a), curvature (Garg et (2024), final sample loss, loss sensitivity (Arpil
ct al.
2017), forgetting frequency (Toneva et al. | 2019). SAMIS (Agiollo et al. 2024). VoG (Agarwal
al.
2022). TracIn (Pruthi et al.) 2020). and GraNd (Paul et al. (021). It also lists the normalized
compute cost for each method. Figure shows that both CSG and Feldman & Zhang (020)) s
memorization scores follow similar trends with adversarial distance. Additionally, Figures 9a and
show the same using MIA performance (see Appendix B.7 for details on MIA). Additionally, we provide the same set of results when using Adam [Kingaa[201a AdamW (Loshchilov & Hutter]
2019), RMProp (Hinton et al. 2012) and Adagrad (Duchi al. 2011) optimizers in the Appendix Table 6
Takeaways. CSG and SGAL demonstrate strong correlation with memorization and serve as highly efficient proxies. SGAL achieves between 97-99% (Table I of the correlation obtained by the best proxy (CSL) while requiring only 10%-30% of the computational cost. CSG and SGAL are approximately 140x faster than curvature, up to 10x faster than CSL, and five orders of magnitude faster than memorization score (see cost analysis in Appendix B.5
8

---

Published as a conference paper at ICLR 2026
Table Mcan + standard deviation of MIA performance over two runs for different stopping points. Lower values indicate better privacy.
MIA Method
AUROC 1
Balanced Accuracy | Val. Loss Stop
Loss Stop
Epoch
Last Fpooch
Curvature Ravikumar al.
2024b]
59.42 + 5.83
60.67
4,06
85.52
0.45
56.55
4 4,16
57.41
2.84
76.63
0.26
LiRA Carim el Bizzi
55.98 1 5.34
56.98
4,93
85.48
0.18
54.36
1 3,90
$5.09
3.70
78.05
0.24
Cal. Los Warson 7022
67.02
5.52
68.63
5.23
70.94
0.22
61.14
+ 3.12
62.34
3,25
62.36
0.16
MAST Sablavrolles et 9
70.70
+ 6.84
72.03
+ 4.89
78.56
0.25
64.10
+ 4,94
65.06
3,46
68.19
= 0,26
Yeom el
57.57 4.72
58.27
3,31
80.75
0.16
55.91
+ 3.83
$6.47
2.70
74.23 0.23
Table 4: AUROC (mean + std over 3 seeds) for mislabeled-samp sample detection on CIFAR-10 and CIFAR- 100 five noise levels. Best values per column are bold.
Dataset
Method
5% Noise
10% Noise
20% Noise
25% Noise
30% Noise
CL Northoutt et
2021
0,8874 + 0.0019
0.8551
0.0030
0,7169 = 0.1539
0.6060 + 0.1387
0.6794 0.1264
In Conf.
(Carliii et al.
2015a
0,7254 1 0.0214
0.6528
0.0042
0.5978
0.0131
0.5800 1. 0.0051
0.5669
0.0106
CIFAR-ID 10
SSFT
Maini ct al
2022
0.9498 + 0.0042
0,9360 : 0.0020
0.9077 = 0.0023
0,8910 + 0.0050
0,8710
0.0071
Curv.
Gare
2024
0.9800 = 0.0003
0,9819
0.0006
0,9934
0.0002
0.9932 = 0.0001
0,9932 0.0006
CSI.
Ravikumaree
2025a
0.9845
0.0026
0,9864
0.0004
0,9870
0.0003
0.9904
0.0005
0.9906 0.0003
CSG (Ours)
+ 0.0009
0.0011
0.9936
0.000Z
:
0,9935
0.0006
SGAL (Ours)
+ 0.0057
0.8835
0.0128
0.8664
0.0070
+ 0.0145
0.8470 0.0283
CL.
Northoutt al.
202
0,8733 + 0.0010
0.8536 + 0.0006
0,70.30
0.1565
0.6833 + 0. 1427
0.6662 + 0. 2289
In Conl.
Carlmni et al.
2019a
0,7069 1: 0.0069
0.6884 1 0.0053
0.6493
0.0075
0.6324 1 0.0051
0.6257 1 0.0044
CIFAR-IIN 100
SSIT
Maini al.
2032
0.8784 1: 0.0030
0.8664
0,0024
0,8358
0.0008
0.8203 1 0.0016
0.8043
0.0061
Gare
0,9876 + 0.0021
0.9892
0.0012
0,9931
0,9931 + 0.0004
0,9932
0.0002
Ravikummar et
2025a
0.9895
0.0002
0.9902
0.0002
0.9904
0.9903
0.0002
CSG (Ours)
0.98% + 0.0008
0.9904
0.0006
0.9934
0.0003
0,9936 + 0.0002
0.9936 0.0001
SGAL (Ours)
0.9895 + 0.0006
0.9897 0.0008
0.9856
0.0008
D. 0.9863 + 0.0009
0.9861 0.0004
5.3 EARLY STOPPING
Experiment. We examine how sample gradient early stopping affects calibration and privacy in a ResNet18 (He et 2016) on CIFAR-100. Prior work shows early stopping improves both [Ji et al. 202T): well-calibrated models make more reliable predictions, and reduced overfitting lowers privacy risk. We focus on the first descent, which offers stronger privacy with reasonable gener- alization. Calibration is measured with ECE, MCE, MSCE, and UCE (see metric definitions in the Appendix B.3) privacy with LiRA Carllnii et al. 2022). curvature (Ravikumar et al., 2024b), and other attacks (Sablayrolles al. 2019 Watson et al. 2022 Yeom et al. 2018).. Appendix B.3 provides additonal setup details.
Results. Table calibration for all checkpoints, with reliability diagrams in Figure Lower error indicates better calibration. We prioritize MCE and UCE in this context because real-world data distributions often follow long-tail or power-law distribution. As discussed by [Felmann (2020). deep learning models rely on memorizing these rare, atypical examples to achieve gener- alization. ECE is an expectation-based metric; it weights the calibration error of each bin by the number of samples in that bin. Consequently, in long-tailed datasets, low-accuracy bins (the tail) often contain fewer samples and are down weighted by ECE, masking miscalibration on difficult ex- amples. In contrast, MCE and UCE treat calibration across the accuracy range more equally. When the goal is trustworthy AI that performs reliably on both head and tail data, these metrics provide more accurate reflection of model safety. Privacy results (AUROC, balanced accuracy) are in Ta- ble 3 where smaller values imply less training data leakage. Further low-FPR MIA results are in Appendix A.I
The sample gradient checkpoint and has lower calibration errors compared to last epoch checkpoint. ceambership-iffernndd performance drop for early-stopped model, indicating noticeably less privacy risk. In short, sample gradient early stopping reproduces the benefits of conventional validation-based early stopping without needing separate validation set!
Takeaways.
9

---

Published as a conference paper at ICLR 2026
Low CSG
High CSG
Calibration Curves
Perfect
Ginal heaei carly
based early step
0.0
Last epoch
0.2
0.0
0.0
0.2
0.6
0.8
1.
Using the CSG uncovers a bias the FM- NIST dataset: darker clothing with lower contrast is often identified as high CSG (i.e. harder).
Figure 5:
Figure 6: Reliability Diagram for Inp Grad/weight norm based, val. based early stopping and the last checkpoint.
5.4 MISLABEL DETECTION
Experiment. We detect mislabeled samples by thresholding CSG values and compare this method to statc-of-the-art approaches on CIFAR-10/100 with ResNet18 under 5% 30% symmetric label noise (labels flipped uniformly to other classes). Detection is evaluated with AUROC (higher is better). Appendix B.6 provides setup details.
Results. The results are presented in Table 4 showcasing the performance of our method and baselines (see Appendix BB.6. for details the baseline methods) on CIFAR-10 and CIFAR-100 symmetric label noise levels of 5%, 10%, 20%, 25% and 30%.
Takeaways. CSG reliably detects mislabeled examples. It surpasses all baselines CIFAR-100 and, on CIFAR-10, scales with noise matching curvature and CSL at 5-10% and surpassing them at higher levels with significantly lower cost, CSG delivers state-of-the-art performance efficiently.
5.5 BIAS
CSG analysis uncovers hidden biases in training data. In CIFAR-100 (Figures Ta and 1b). images of boys in red shirts or red trucks consistently receive low CSG scores, indicating they are learned earlier. This is likely due
representation of red colored objects. This color-specific ease of learning suggests a bias that may hinder generalization. A similar trend is observed in Fashion- MNIST (Xiao al. 2017). where Figure shows that high CSG samples, learned later training, tend to be darker and lower in contrast. These findings show that irrelevant features (e.g., color, brightness, contrast) can bias learning, and CSG offers a practical way to detect them. Finally, to ensure reproducibility, details of all the experiments are provided in Appendix B and the corre- sponding code is included in the supplementary material.
CONCLUSION
In conclusion, we propose Cumulative Sample Gradient (CSG), a theoretically grounded and effi- cient proxy for memorization. CSG aligns closely with memorization (Feldman Zhang 2020 while being to five orders of magnitude faster, and is 140x faster than curvature (Garg et al. 2024) and 5x faster than CSL (Ravikumar al. 2025a). two prior state-of-the-an proxies. Our analysis establishes theoretical connection between CSG, learning time, and memorization. Ex- perimentally, we show that CSG enables validation-set-Fie early stopping and supports the develop- ment of an additional proxy, SGAL. Further experiments demonstrate state-of-the-art performance mislabeled data and bias detection, positioning CSG as a scalable and interpretable tool for data- centric deep learning.
ACKNOWLEDGMENTS
This work supported in part by the Center for the Co-Design of Cognitive Systems (COCOSYS), a DARPA sponsored JUMP center, Semiconductor Research Corporation (SRC), National Science Foundation (NSF) award DMS-2502560, and Collins Aerospace. Efstathia Soufleri has been par-
10

---

Published as a conference paper at ICLR 2026
tially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.
REFERENCES
Chirag Agarwal, Daniel D' souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10368-10378, 2022.
Andrea Agiollo, Young In Kim, and Rajiv Khanna. Approximating memorization using loss surface geometry for dataset pruning and summarization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 17-28, 2024.
Sai Aparna Aketi, Abolfazl Hashemi, and Kaushik Roy. Global update tracking: A decentralized learning algorithm for heterogeneous data. Advances in neural information processing systems, 36, 2024.
Jason Ansel, Edward Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh. Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Chris- tian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang. William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support f Programming Languages and Operating Systems, Volume (ASPLOS '2).. ACM, April 2024. doi: 041144/56620065336440666 p.
URL
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look memorization in deep networks. In International conference on machine learning, pp- 233-242. PMLR, 2017.
Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Pascal Vincent. The pitfalls of memorization: When memorization hurts generalization. In The Thirteenth Interna- tional Conference Learning Representations, 2025. URL httpss//epenreview.nt
Eorum?id=wWh2h9420
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pp. 123-132, 2021.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv: 1910. 1000000.4... 13427, 2019a.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Prototypical examples in deep learning:
Metrics, characteristics, and utility, 2019b.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp. 2633-2650, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IEFE Symposium on Security and Privacy (SP), pp. 1897-1914, IEEE, 2022.
Ganesh Del Grosso, Hamid Jalalzai, Georg Pichler, Catuscia Palamidessi, and Pablo Piantanida. Leveraging adversarial examples to quantify membership information Icakage. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10399-10400) 2022.
11

---

Published as a conference paper at ICLR 2026
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284. Springer, 2006.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Isha Garg and Kaushik Roy. Samples with low loss curvature improve data efficiency. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20290- 20300, 2023.
Isha Garg, Deepak Ravikumar, and Kaushik Roy. Memorization through the lens of curvature of loss function around samples. Forty-first Internatiional Conference on Machine Learning, 2024.
net Bttee//ooouuunnNNNNNNNSSSSS URL /ttp://s/.oooaoooooooeee///////a
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal Optimization, 23(4):2341 2013.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. International conference on machine learning, pp. 1225-12344 PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Ziwei Ji, Justin Li, and Matus Telgarsky. Early-stopped neural networks are consistent. Advances in Neural Information Processing Systems, 14:1885--8877 2021.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Characterizing structural regularities of labeled data in overparameterized models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 5034-5044. PMLR, 18-24 Jul 2021. URL
htmml
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMLR, 2022.
Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. In Proceedings of the tenth annual conference on Computational learning the- ory, pp. 152-162, 1997.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, and Kaushik Roy. Verifix: Post-training rection to improve label noise robustness with verified samples. arXiv preprint arXiv:2403.0861 2024.
12

---

Published as a conference paper at ICLR 2026
Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, and Kaushik Roy. Sap: Corrective machine unlearning with scaled activation projection for label noise robustness. Proceedings of the AAAI Conference Artificial Intelligence, 39177)177330-17777 Apr. 2025. doi: 10.16699/aai.v3991111.. 33972. UtR.https:////.... aaai org/ wewpg//ndex.. http:///wwwp.p///aaaaaaaaaiaaaaa 33972
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don't learn via mem- orization. In ICLR Workshop, 2017.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances neural information processing systems, 36: 06:1957-1987, 2023.
Danny D Leybzon and Corentin Kervadec, Learning, forgetting, remembering: Insights from track- ing IIm memorization during training. In Proceedings of the 7th BlackboxNLP Workshop: Ana- lyzing and Interpreting Neural Networks for NLP, pp. 43-57, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
formm?? did=
Bkg6RiCqY7
Michal Lukasík, Vaishnavh Nagarajan, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Ku- mar. What do larger image classifiers memorise? arXiv preprint arXiv:2310.05337, 2023.
Pratyush Maini, Saurabh Garg, Zachary Lipton, and Zico Kolter. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044-30057,
2022.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, Feder Cooper, Daphne Ip- polito, Christopher A Ahooquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arX1v:2311.17035, 2023.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
Jiyoung Park, Ian Pelakh, and Stephan Wojtowytsch. Minimum norm interpolation by perceptra: Explicit regularization and implicit bias. Advances in Neural Information Processing Systems, 36, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596 2021.
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances Neural Information Processing Systems, 33:17044-17056, 2020.
Tomaso Poggio, Gil Kur, and Andrzej Banburski. Double descent in the condition number. arXiv preprint arXiv:1912.06199 2019.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930. 2020.
Deepak Ravikumar, Sangamesh Kodge, Isha Garg, and Kaushik Roy. Intra-class mixup for out-of- distribution detection. IEEE Access, 1122966-259811 2023.
Deepak Ravikumar, Efstathia Soufleri. Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. In Forty-first International Conference on Machine Learning, 2024a.
13

---

Published as a conference paper at ICLR 2026
Deepak Ravikumar, Efstathia Soufleri, and Kaushik Roy. Curvature clues: Decoding deep learn- ing privacy with input loss curvature. In The ehirty-eighth Anmual Conference on Neural Information Processing Systems, 2024b. URL https:: http::ss///peeevieee.. http://eneeeeeeetteeeeeeeeeeeeee het/forum?id
GEVDMQ6Mu5
Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Towards memoriza- tion estimation: Fast, formal and free. In Forty-second International Conference on Machine Learning, 2025a.
Deepak Ravikumar, Efstathia Soufleri, and Kaushik Roy. Improved out-of-distributin detection with additive angular margin loss. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2025b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 111322211-222, 2015. doi: 10. 100//07/s1 1 1263-015-0816-y.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. In International Confer- ence on Machine Learning, pp. 5558-5567. PMLR, 2019.
Rylan Schaeffer, Zachary Robertson, Akhilan Boopathy, Mikail Khona, Kateryna Pistunova, Ja- son William Rocks, lla R Fiete, Andrey Gromov, and Sanmi Koyejo. Double descent demystified: Identifying, interpreting & ablating the sources of deep learning puzzle. In The Third Blogpost
Track at ICLR 2024, 2024. URL
opwnrevienereeeeeeeeeeeeeerrreee wwt. forum d=muC 7uLvGHr
Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. Reinforcement learning algo- rithms: A brief survey. Expert Systems with Applications, 231:120495, 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Efstathia Soufleri, Deepak Ravikumar, and Kaushik Roy. Advancing compressed video action recog- nition through progressive knowledge distillation. arXiv preprint urXiv:2407.027153 2024a.
Efstathia Soufleri, Deepak Ravikumar, and Kaushik Roy. Dp-imgsyn: Dataset alignment for ob- fuscated, differentially private image synthesis. Transactions on Machine Learning Research, 2024b.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818 8-28866, 2016.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL https::/ openreview net/ htte//frummiddo SaneuummiddBBB.. BJ1xm30cKm
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222 2023.
Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. On the importance of dif- ficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022. URL httts:::/oppereeeeeess Bet.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 17X8 2017.
14

---

Published as a conference paper at ICLR 2026
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017. URL httpttts/ppeeeeeeeiccce/e net wttp://rrmmmieeeeeeeeeee F
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL BACKGROUND ON STOCHASTIC GRADIENT DESCENT (SGD)
p-Lipschitz Gradient. The gradient of the loss function E said to be p-Lipschitz on Range( W) if, for all Wi, W2 Range( W), there exists constant p such that:
|| un E((ū,) V wa {( pl u7 Will
(8)
Bounded Gradient Variance. Let V w. ((w,) denote the true gradient, and let Vw, unbiased estimator of this gradient. The estimator said to have variance bounded by r
w. ((w)) be an
I 1 ((wx) - V Vm ((x)) S
In SGD, model parameters Wh iteration updated using the gradient of the loss function computed with a mini batch or a single random sample ž, The update rule is
T W7 1)c V ((T,, 2,)
(10)
Here, 1/s is the learning rate, and V w. {1,, 2,) is an unbiased stochastic gradient estimator.
A.1 MEMBERSHIP INFERENCE ATTACKS (MIA)
Membership Inference Attacks (MIA) aim to determine whether a specific data point was part of a machine learning model's training dataset. By analyzing the model's outputs, such as confidence scores or loss values, attackers can infer the membership status of individual data points, posing significant privacy concerns, especially when models are trained on sensitive information (Shokri et al. 2017).
Curvature Clues (Ravikumar al., 2024b). This approach leverages the curvature of the loss function with respect to imput data to distinguish between training and non-training samples. Specif- ically, it examines the trace of the Hessian matrix (input loss curvature) of the loss function. Empir- ical studies have shown that this curvature tends to be higher for training data points compared to non-training ones, enabling effective membership inference even in black-box settings.
Likelihood Ratio Attack (LiRA) (Carlini et al. 2022). LiRA is a black-box membership inference data point being part of the training set. By comparing the loss distributions of shadow models trained with and without specific data points, LiRA computes a likelihood ratio to infer membership. This method has demonstrated superior performance, especially low false positive rates, compared to previous approaches.
technique that utilizes shadow models to estimate the likelihood
Shadow Models are auxiliary models trained to mimic the behavior of the target model. By training these models known datasets, attackers can observe how the model behaves with known members and non-members, creating a reference for inferring membership in the target model.
In our study, we employed 64 shadow models to capture diverse range of behaviors, enhancing the robustness and accuracy of our membership inference attacks.
B EXPERIMENTAL DETAILS
B.1 VALIDATING THEORY SETUP DETAILS
Datasets. We use CIFAR-100 (Krizhessky et al. 2009 and ImageNet (Russakovsky et al., 2015) datasets. For experiments that use memorization scores, we use the pre-computed stability- based memorization scores from Feldman & Zhang (020) which have been made publicly available by the authors.
the precomputed stability- based memorization scores from Feldman & Zhang (2020) we use the same model architectures as Feldman & Zhang (2020), specifically we Inception Szegedy al. (2016) for CIFAR-100 and ResNet50 et al. 2016) for ImageNet.
Architectures. Since
16

---

Published as a conference paper at ICLR 2026
Mem. and CSG vs. Adversarial Distance
2.0
1.5
0.4
Memorization
1.0
0.2
0.5
0.0
0.0
0
10
Adversarial Distance
Figure 7: Comparison of CSG with adversarial distance and memorization.
Training. When training on CIFAR-100 the initial learning rate was set to 0.1 and trained for 200 epochs. The learning rate is decreased by 10 at epochs 120 and 160. The batch size We use stochastic gradient descent for training with momentum set to 0.9 and weight decay set to 1 10-4 For CIFAR-10O, 100, we used the following sequence of data augmentations for training: resize (32 x 32), random crop, and random horizontal flip, this is followed by normalization. For ImageNet we trained a ResNet50 with the resize random crop was to 224 x 224.
128.
B.2 SIMILARITY WITH MEMORIZATION SETUP DETAILS
Setup. The datasets, architectures and training for this experiment is the same as described in section B.1
Similarity Score. For cosine similarity score we use pytorch and for Pearson correlation sklearn's
implementation.
Effect of Architecture. The table below shows the effect using different architecture from that of Feldman & Zhang (2020) and using the proposed CSG proxy to calculate similarity with the precomputed memorization scores from Feldman & Zhang (2020).
Table CSG similarity with FZ Memorization vs Architecture. Cosine Similarity (CS) refers to similarity with Feldman & Zhang 2020) precomputed memorization scores.
Dataset
Architecture
Cosine Similarity (CS)
CIFAR100
Inception (Same FZ)
0.84
CIFAR100
ResNet18
0.77
ImageNet
ResNet50 (Same as FZ)
0.78
ImageNet
ResNet18
0.68
Using MIA to test Memorization. We use the state-of-the-att membership inference attack pro- posed by Ravikumaa et al. (2024b) to evaluate the memorization behavior models. Membership inference attacks measure if a given sample was used to train the model. The success of the attacks reveals privacy risks for training examples. To demonstrate that CSG captures similar properties to the memorization score introduced by Feldman and Zhang [Fllmann & Zhang (020), plot MIA AUROC against both the memorization score and CSG in Figures 99b and 9a respectively. Both plots clearly show monotonic increase in MIA attack success as either the memorization CSG increases, indicating that both metrics effectively reflect model memorization.
or
B.3 EARLY STOPPING DETAILS AND SETUP
Setup. For this experiment we trained ResNet18 on CIFAR- 100. The initial learning rate was set to 0.1 and trained for 300 epochs. The learning rate is decreased by 10 epochs 180 and 240. The
17

---

Published as a conference paper at ICLR 2026
Cosine Similarity w/ Mem.
0.75
0.50
Epoch Wise
Epoch CSG
0.25
0.00
0
100
200
300
Epoch
Figure 8: Epoch-wise similarity for R CIFAR-100 (same model and trajectory) used in Figure Tc We clearly observe the same trend as in Figure 3c FigureB3c was included the main pa- architecture used
per because it obtained using the FZ Inception architecture, which is the to evaluate memorization scores in Feldman & Zhang (202)). However, this result demonstrates that our observations generalize across network architectures, provided there is sufficient parameteriza- tion.
CSG vs Curv MIA AUROC
Mem. Score vs Curv MIA AUROC
1.0
1.0
X X
0.9
0.9
AUROC
AUROC
0.8
0.8
X
0.7
0.7
x
0.6
0.6
500
1000
1500
0.00
0.25
0.50
0.75
1.00
CSG
Mem.
(a) MIA AUROC vs CSG.
(b) MIA AUROC Memorization Score.
Figure 9: Demonstrating that CSG captures similar properties as the memorization score introduced by Feldman and Zhang Feldman & Zhang (2020). we plot MIA AUROC against both the mem- orization score and CSG in Figures Da and 9bl respectively. Both plots show a clear monotonic relationship: as either memorization or CSG increases, the success rate of the MIA attack also in- creases.
18

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of Cosine Similarity (CS) and Pearson Correlation (PC) different methods with precomputed memorization scores from Feldman & Zhang (2200) Experiments are performed on the CIFARIO0 dataset using the Inception model architecture (same as/Feldman & Zhang (2020) across Adam, AdamW, AdaGrad, and RMSProp optimizers. "Top 5k" refers selecting the 5000
most memorized examples based on Feldman & Zhang 2020,, and the cosine similarity and Pearson correlation for these examples is reported.
Adam
AdamW
AdaGGra
RMSPmp
Compute Cost
Dataset
CS
PC
CS
PC
CS
PC
CS
PC
SGAL (Ours)
0,1-0.3x
All
0.87
0.79
0.87
0.78
0.85
0,74
0.87
0.78
CSL (Ravikumar al. 2025b
Ix
All
0.87
0.78
0.85
0.75
0.84
0.73
0.87
0.78
Loss Sensitivity (Arii et al. 2017
1x
All
0.85
0.77
0.84
0.75
0.82
0.70
0.85
0.77
(Ours)
0.1-0.3x
0.84
0.84
0.73
Forgetting Frequenccy TTonvaa et al.
2019
1x
AlI
0.83
0.72
0.79
0.63
0.74
0.54
0.83
0.71
VoG All Ckpts (Agarwal et a) 2022
0.80
0.62
0.65
0.81
0.67
Final Sample Loss
All
0.09
0.06
0.04
0.03
0.45
0.31
0.09
0.06
VoG Last 5 Ckpts (Agarwal ot al. 2022
1x 26x (mensured)
All
0.22
0.14
0.12
0.08
0.47
0,31
0.25
0.17
TracIn (Pruthi
2020
All
0.75
0.70
0.85
0.75
GraNd Paul ata 2021
7.8 (measured)
All
0.81
0.69
0.79
0.65
0,81
0.69
0.69
0.39
Loss Sensitivity Arpit et 2017
1x
Top 5k
0.79
0.69
0.78
0.67
0.75
0.67
0.79
0.69
SGAL
0.1-0.3x
Top 5k
0.83
0.73
0.82
0.72
0.80
0.83
0.73
CSL [Raiiuuar et al. 2025b
1x 0.1-0.3x
Top 5k
0.83
0.73
0.81
0.69
0,80
0.67
0.83
0.73
[Oürs)
Top 5k
0.71
Final Sample Loss
1x
Top 5k
0.08
0.05
0.04
0.03
0.40
0.27
0.08
0.08
Forgelting Frequency Toneva et al. 2019
1x
Top Sk
0.79
0.67
0.93
0.05
0.89
0.07
0.79
0.66
VoG All Ckpts (Agarwal
2029
Top 5k
0.75
0.76
0.60
VoG Last 5 Ckpls Aaaawa e an 2022
1x
Top Sk
0.19
0.12
0.11
0.08
0.42
0.27
0.22
0.15
TracIn (Pruthi
2020
26x (measured)
Top 5k
GraNd PPlli er 2021
7.8 (measured)
Top 5k
0.76
0.62
0.74
0.59
0.76
0.62
0.63
0.32
batch size is set to 128. We use stochastic gradient descent for training with momentum set to 0.9 and weight decay set x 10-4.
Model calibration refers to the extent to which a model's predicted probabilities reflect the true likelihood of outcomes. A well-calibrated model not only makes accurate predictions but also as- signs confidence scores that correspond closely to the observed frequencies of correct classifications. For example, among all predictions assigned a confidence of 70%, approximately 70% should be correct. Proper calibration is crucial in applications where Decision-making depends on reliable uncertainty estimates, such as medical diagnosis or autonomous driving.
To evaluate calibration, reliability diagrams (also known calibration plots) are widely used. These plots compare predicted confidence with empirical accuracy by partitioning predictions into bins based their confidence scores (e.g., intervals like [0.0, 0. [0.1, 0.2), ..., [0.9, 1.0)). For each bin Bm. the accuracy acc( B.)) is the fraction of correct predictions, and the confidence conf (Bm) is the average predicted probability. Discrepancies between these values indicate miscalibration.
Several quantitative metrics are used to summarize calibration performance, see Wang 2023 for a
detailed analysis:
M Σ
Bm |aco(Ba)ccon(()))) conf(Bmm)
Expected Calibration Error (ECE)
(11)
m=
Maximum Calibration Error (MCE)-
max I....,M
| conf(Bm))
(12)
1 M
Mean Squared Calibration Error (MSCE)
Σ
-cnf
(13)
M
m=1
1
Uniform Calibration Error (UCE)
aace conf(
(14)
M
19

---

Published as a conference paper at ICLR 2026
ADAM: Input Gradient and Validation Loss
ADAM: Input Gradient and Weight Norm
1200
1000
Inpul Grailent
0.6
Input Craiient
Weight Norm
800
0.5
Validaatoo
600
0.4
400
0.3
200
Trpel Craalon
0.2
50
100
150
200
50
100
150
200
Epoch
Epoch
(a) Adam: Input Gradient Validation Loss
(b) Adam: Input Gradient Weight Norm
ADAMW: Input Gradicat and Validation Loss
ADAMW: Input Gradient and Weight Norm
0.7
Irper
Valiedor
Wotde
0.6
2500
Input Groiiem:
Input Gradient
Weight Norm
2000
Validation
1500
0.4
1000
0.3
500
50
100
150
200
50
200
Epoch
Epoch
(c) AdamW W: Input Gradient & Validation Loss
(d) AdamW: Input Gradient & Weight Norm
Figure 10: Comparison of input gradients against Validation Loss (Left column) and Weight Norm
(Right column) for Adam and Adam
ECE computes the weighted average of absolute errors across all bins, giving more influence to bins with more samples. MCE captures the worst-case miscalibration observed across bins. MSCE penalizes larger discrepancies more severely by squaring the calibration errors. UCE provides an unweighted average of absolute differences, treating all bins equally regardless of sample size. To- gether, these metrics offer robust framework for assessing and improving the reliability of proba- bilistic predictions.
where
acc(B.n)
1 v.). conf(Bm)
(15)
Bm
1mml
B.....
and n is the total number of samples, M the number of bins, and Bn the set of indices assigned to bin m.
MIA Setup. For the MIA test setup use the code, models and setup from Ravikumar et al. 20246).
B.4 ADDITIONAL RESULTS WITH ADAM OPTIMIZER
We ran additional experiments with the Adam optimizer. For the training procedure, ResNet18 model was trained on CIFAR100 using Adam with an initial learning rate of 10-3 and weight decay of 1 10-4 We employed the standard CrossEntropyLos MultiSteple scheduler, which decayed the learning rate by a factor of 0.1 at 60% and 80% of the 200 total training epochs.
function and
20

---

Published as a conference paper at ICLR 2026
ADAGRAD: Inpu: Grutien: and Validation Loss
ADAGRAD: Inpui Gradient and Weight Norm
0.8
250
0.7
Input Grailent
0.6
Input Cradient
Weight Norm
Valitaaion
150
0.5
wiih
0.4
100
0.3
0.2
50
100
150
200
50
100
150
200
Epoch
Epoch
(a) Adagrad: Input Gradient & Validation Loss
(b) Adagrad: Input Gradient Weight Norm
RMSPROP: Iaput Gradient and Validation Loss
RMSPROP: Input Gradicat and Weight Nomn
0.8
1400
1200
Input Graiten:
0.6
Inpur Grodient
0.5
Valldation
800
Weight
600
0.4
400
0.3
200
50
100
150
200
150
200
Epoch
Egoch
(c) RMSProp: Input Gradient & Validation Loss
(d) RMSProp: Input Gradient & Weight Norm
Figure 11: Comparison of input gradients against Validation Loss (Left column) and Weight Norm (Right column) for Adagrad, and RMSProp.
VIT-M 1-626666 Input Gradients
ResNet50: Input Gradients
Topar Gruterraa
1.0
1400
Toper Grudzzn
1.0
0.8
088
Input Gratiens
6000
Gradionts
1000
0.6
0.6
800
0.4
600
0.4
2000
0.2
20
40
80
20
40
60
Epoch
Epoch
VIT-M-16/256 Input Gradients
(b) ResNet50 Input Gradients
Figure 12: Evolution of input gradients during training ImageNet. Subfigure (a) shows the gradients for ViT-M-16/256, while (b) shows ResNet50.
The results are reported in Table 7 These results show gradient peaking phenomenon similar to that observed with SGD. This potentially hints at a more general behavior neural networks that may be optimizer-agnostic.
21

---

Published as a conference paper at ICLR 2026
Val Loss
Validation Loss
Weight Norm
Weight Norm
120
110
10"
101
10a
Epochs
Figure 13: Weight norm behavior on ImageNet using a modified version of the standard recipe for training ResNet18. The recipe trained longer for 200 epochs (instead of 90) with larger batch size and weight decay. The plot shows an initial decay, followed by an increase and subsequent decrease. The first validation loss minimum roughly coincides with the first weight norm minimum (of by a few epochs but exaggerated here due to the log scale plot and the points are obtained once per epoch).
3.5
140
Validation Loss
3.0
130
2.5
Weight
120
2.0
Val Loss
1.5
Weight Norm
110
10a
10
10a
Epochs
Figure 14: Weight norm behavior on ImageNet using a standard recipe for training ResNet18. The recipe trained for 90 epochs with batch size 128 weight decay Ic-4, with SGD momentum of 0.9. The plot shows the same peaky behavior as CIFAR 100 from Figure 33) where the peak of the weight norm coincides with the minimum validation loss for the first descent.
22

---

Published as a conference paper at ICLR 2026
Table Gradient norm values for ResNet18 trained on CIFARI00 using Adam optimizer.
Epoch
Input Grad Norm
1,2486
10
3.4195
50
3.1032
80
3.1794
100
3,6039
120
1,0826
150
0,2104
180
0.0595
B.5 COMPUTE COST ANALYSTS
In this section, we evaluate the computational overhead associated with our proposed proxy in com- parison existing approaches. Let define the cost of single forward pass through neural network as F. A standard backpropagation step, involving both forward and backward passes, therefore incurs total cost of 3F. With this in mind, and denoting the dataset size by and the number of training epochs by T, we outline the cost incurred by different methods below.
Stability- Based Memorization. As introduced Feldman & Zhang (020)) this approach involves training between 2,000 and 10,000 models to derive memorization scores. Consequently, the compu- tational burden scales with the number of models trained, resulting total cost of approximately:
10,000 3F T T
Cumulative Sample Curvature. The authors of Garg al. (2024) propose estimating memo- rization through average sample curvature computed throughout training. This involves applying Hutchinson's trace estimator, which, for each sample per epoch, requires two forward passes and one backward pass, repeated n times. For favorable comparison, we assume = 2, although bet- ter empirical performance is achieved with n 10. The resulting cost combines standard training and curvature estimation:
Cost 3F T m + 4F .. m =F T + 4F m
(16)
(17)
11F. .m
(18)
If using n 10 to match the best results reported in Tables 4 the cost increases to:
43F T T m
CSG (Ours). Our proposed method, CSG, is designed to introduce minimal additional computa- tional overhead. utilizes gradient information that already computed during training and only needs to be evaluated during the initial iterations. While computing input gradients involves an extra matrix multiplication (multiplying the error from backpropagation by the corresponding weight matrix), this cost is negligible relative to the overall training workload. Thus, the total cost is fraction of standard training:
between 10% and 30% of total training
Cost = 0.1 3F T m 0.3F T m
comparative summary of computational costs is provided in Table g
B.6 MISLABELED DETECTION SETUP DETAILS
Area Under the Receiver Operating Characteristic (AUROC or simply AUC-ROC) is a perfor- mance metric used to evaluate the discriminative ability of binary classification model. It summa- rizes the relationship between the True Positive Rate (TPR) and the False Positive Rate (FPR) across all possible classification thresholds. TPR, also known sensitivity recall, and FPR are defined
23

---

Published as a conference paper at ICLR 2026
Table Summary the compute cost of the proposed metric compared to existing methods.
Method
Absolute Cost
Relative Cost
Stability-Bass (Flldman & 7hang 2020 Input Curvature (Garg et al. 2024
6000
30, 000
2,000 x 30, 000 x
30,000
11FTm 43FTm
3.6 x 14.33 x
CSG (Ours)
0.3FTm
0.1 x
0.3 x
as:
TP TPR TP FN' FP
(19)
FPR FP +N'
(20)
where TP, FN, FP, and TN refer to true positives, false negatives, false positives, and true negatives, respectively. The ROC curve is generated by plotting TPR against FPR the classification threshold varied from 0 to 1. The area under this curve (AUROC) provides a single scalar value representing the model's ability to distinguish between the two classes. An AUROC of 1.0 indicates perfect class separation, while a value of 0.5 suggests the model performs no better than random guessing. Low AUROC values close 0.5 imply poor model performance, whereas values closer to 1.0 indicate strong predictive capability.
Sctup. Setup. We trained ResNet-11 model on the CIFAR-IO and CIFAR-100 datasets. The initial learning rate was set to 0.1, and training was conducted for 300 epochs. The learning rate was reduced by factor of 10 at epochs 180 and 240. batch size of 128 was used. Training was performed using stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 10 The CSG circulations spanned the entire 300 training epochs, yielding speedup of approximately 14x over curvature-based methods. To introduce label noise, specified percentage of the training set was randomly selected and had their labels reassigned to incorrect classes. For instance, in the 5% noise setting, 5% of the 50,000 training examples were randomly chosen and their labels were flipped to another classes. No validation split was used; the model was trained on the full noisy training set, and performance metrics were evaluated the standard test set.
B.6.1 BASELINE METHODS
Next, we elaborate on the experimental setup used for identifying mislabeled data points.
Threshold Based Learning Time. This refers to the first epoch during which train- ing sample is correctly classified. For instance, if the prediction sequence over epochs is [0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1,, 1]. the learning time recorded as epoch the first occurrence of correct prediction.
In Confidence. As defined in Carlini et al. (2019a), the in-confidence score is computed as 1 P. where p is the predicted probability assigned to the true label.
Confident Learning. We implemented confident learning following the approach of Northcutt al. (2021), using the cleanlab package (ht / / com/ c lean l lab/ clean lab). To en- sure reliable predictions, we employed 3-fold cross-validation to estimate out-of-sample class prob- abilities, which were then used with cleanlab to identify likely label errors.
Second Split Forgetting Time (SSFT). (Maini et al., 2022) is computed by training the model sequentially two disjoint subsets, Set I followed by Set 2. The forgetting time corresponds to how quickly samples from the first set become misclassified during fine-tuning on the second. This procedure is reversed to cover the complete dataset, with forgetting times tracked symmetrically for both splits.
Curvature. To measure the curvature of individual training samples, followed the method pro- posed in Garg et al. 2024), using hyperparameters n = 10 and = 0.001, in line with their experimental setup.
Compute Cost. Compared to altermative methods, some approaches introduce considerable compu- tational overhead. For instance, Confidence Learning (CL.) involves training multiple models using
24

---

Published as a conference paper at ICLR 2026
49529 Porcupine
23947
38675 Bridge
10968 Snake
39325 Crocodile
31128 Seal
17688 Seal
16646 Mouse
33638 Otter
31828 Shrew
23175 Snake
41874 Otter
34187 Shark
43843 Shrew
47636 Worm
46752
Worm
30418 Spider
9012 Otter
18461 Snake
25748 Tiger
47806 Crab
8393 Girl
36874 Baby
21899 Seal
46418 Willow
5392 Worm
29704 Train
38817 Couch
26150 Palm
12075 Wolf
45690 Sea
12946 Apple
Figure 15: Visualization of the 32 highest CSG-scoring images from CIFAR- annotated with their class labels and PyTorch image indices. Notably, image indices 23947 and 33638 appear to be identical images but are assigned different labels. All such duplicate pairs in the figure are highlighted. Additional pairs were discovered using this technique, including one for image index 3 1487 (not shown here), which is labeled as 'whale'.
(a) Lowest CSG images from ImageNet.
(b) Highest CSG images from ImageNet.
Figure 16: (a) Lowest CSG images from ImageNet (first few classes). Captures examples prototyp- ical for the class i.e. easy examples. (b) Highest CSG images from ImageNet (for the same first few classes). High CSG captures examples atypical for the class i.c. hard examples, likely memorized.
k-fold strategy (three in this case) which substantially raises the computational burden. Similarly, SSFT necessitates training on at least two distinct subsets of the dataset, effectively doubling the training effort compared to standard setup. Other techniques, such those relying input loss curvature, are even more resource- inttrrceeitnnivvee demanding over an order of magnitude more computa- tional power. CSG offers comparable training costs vanilla training with minimal overhead (one extra matrix multiplication). Overall, this highlights the trade-offs between computational efficiency and performance when selecting among existing methods.
25

---

Published as a conference paper at ICLR 2026
B.7 MIA EXPERIMENTS
Additional Results We present the results of membership inference attack (MIA) detection on early- stopped models, using our proposed technique, validation loss, and the final epoch. These results supplement the main paper by providing true positive rates (TPR) at very low false positive rates (FPR), complementing the previously reported AUROC and balanced accuracy metries.
Table 9: Mean + standard deviation of True Positive Rates (TPR in %) different False Positive Rates (FPR) for various methods and stopping points.
Epoch
0.99
0.77
0.04
1.42= 0.90+
1.00
Yaur
1.15+ 0.05
1.10
0.15=0.000
MAST
0..70
1:30
0.54
2.06
6.85
1.44
8 1.27
18:24
Cul
6.18
0.50
1.00::1..3
7.02
0.76
1368
C PROOFS
C.I
PROOF FOR LEMMA 4.
The proof for this lemma build on the result from Ravikumar al. (2025a), which states the follow- ing.
Lemma For any neural network without a skip connection at the first layer, and a given mini-batch of inputs Zo = (Xo: Yo), the Frobenius norm of the gradient of the loss & with respect to the input is bounded by the norm of the gradient with respect to the network's weights wy Specifically:
V (wū, Z S V l( Zollee
(21)
( IIF I((x))*lle and sp denotes the smallest singular value of P = X (x,))
where kg
SP
where" + denotes pseudo-inverse.
Ravikumar et al. (2025a) provide the proof for the above result. This forms the basis of the general result. To obtain the form stated in Lemma 4.1 consider a single-sample mini-batch, in which case E is column vector. In this case, the norm of the pscudoinverse is the reciprocal of its norm. Since Z is a column vector, the smallest singular value is
To see this, Let X E Rux1 be column vector. Then X E RR..% is a row vector, whose Moore- X
Penrose pseudoinverse given by (XT)+=
Raxl
Therefore,
X
X
X
||
P X
|| = ||
|
Since P is the x matrix [1], its only singular value is 1, and thus the smallest singular value
of P is also 1. Substituting these results into the above lemma, i.e. Zb = 2 (7,, v),,)) X, = E, [[ + ||F = 1/| [|,, we obtain the form of Lemma 4.1
V E(,,, Z)ll I |[wlll || [(w,, Z)llE |7,
C.2 PROOF OF THEOREM
Proof of Theorem 4.2
Assumptions.
Throughout this proof we assume:
The loss function é is p-Lipschitz smooth.
The stochastic gradient estimator unbiased: EE. (o = vir((ut))
The gradient variance is bounded: E || ((v,)) - V {(TT)) < 2.
26

---

Published as a conference paper at ICLR 2026
The gradient noise is uncorrelated with the true gradient: E.(( w. [(t))) (666))))))) 6c)] 0.
Assume B-stability of SGD.
Assume I-bbunned loss.
From the input gradient convergence result from Ravikumar et al. (2025a) (Eq. 57), for any mini- batch X we have
Fim R (((TT) (((Tr))
Kmulpp'
ER [ x, ((xn))
(22)
Tn
2
Please note, T represents some iteration step, while R < T is discrete random variable, sampled non-uniformly from the 21. T (i.e. support of size T). This distinction a key aspect of the proof strategy, adapted from the convergence analysis of SGD (randomized stochastic gradient) from Ghadimi & Lan (2013). The proof establishes guarantee on the expected performance of non-uniformly sampled iterate R. rather than the final iterate T..
Consider the case where the mini-batch X₁ to contains only single training example z;. This choice removes dependence on the SGD batch size (though it affects Km,, but is still constant), yielding
En V n. (
Kem Tn
Er (((T)) - f((ir))) +
18m Dspppp
(23)
2
Let W. represent the global optima, replacing ((WT) by the optimal value ((w.), since by definition ((..) < ((ūT) gives the more interpretable bound
ER V ((
Fimm
E. [e(((Wo)) E(wt)))
Kim Tspp
(24)
TT
Observe that the left-hand side Eq. 24 is exactly the term appearing the learning condition. denote the worst-case bound by t, then for some T we have
Pim Ty
E (b( -((T))))))) + Rim p
(25)
2
For sample E,, this T = T2.. Rearranging yields
Tg
= Eele(10)) - E(w..)
(26)
Km 2
We expand the right-hand hand side to separate the memorization component:
Ts.
TfT
Kmm?spP 2
= Et,,, [C(woo 2;) - e\= (w., 2:] + EE,,,, ( (w))( - f(WT,, =()]. (27)
fim
The second term is exactly the memorization quantity mem(z.) (2,), scaled by T., see Ravikumar et al. 2024a). Taking expectation with respect to the sampling distribution p gives
Km P
[T
Er,.. [e(1o, 21) / (w., 24) + Et,,, [L me)(
(28)
Km
2
Using the definition of 3-stability (Eq, 2 yields
ET.T
1nT
Km
TppI
Ex. (e.p)((uo)) ((TTa)) Er.,, mem(2,)) (2,) B.
(29)
Kmm
2
Now consider Eq. 23
let = T Tmaz. we get
Kim
E. (E(Un) E(T.))) Kym Mspp
ER II, (wR)
(30)
Tmax!!
11 Es, [CSG(2)) & E. [e(w)) - E(1..)]
Kim T)s Tmman pl
(31)
Km
2
1)
E., [CSG(2))]
Ku 1s marpl?
E, [e((T)) - {(WT.)]
(32)
Km
2
27

---

Published as a conference paper at ICLR 2026
Combining Eq. 29 this with the inequality above yields
Kom 7app"
7 Ep.., (CSG(2))
Kim 1/ T max maxpT
Epl
[T[
Ee,e, [L mem (2.)] +
-
Km
2
Kim
(33)
Finally, noting that mem (24) 2 0, and set
122 DJs T Tunax P p
Ci =
C2
+
2 0,
2m
"
21
Since the choice of learning threshold arbitrary, we can choose
Kim "mppsp
21
Thus C1 >> 0. All constants in C2 are positive thus C2 > O. Rearranging the terms, we conclude
Bp. (C)G(E:) S C1 Ey, [T,,] + C2
(34)
C.3 PROOF OF THEOREM 4.3
Assumptions. We assume:
The loss function é is p-Lipschitz smooth.
The stochastic gradient estimator unbiased: E, [. ((T:)) = vil((it))
The gradient variance is bounded: E || ((ū.) - V Vw, (1t) S I
The gradient noise is uncorrelated with the true gradient: Ez and E(wtt) 5,)1 = (Same
Assumption from Ghadimi & Lan (2013).
Assume 3-stability of SGD.
Assume L-bounded loss.
From Equation we have
Km (((w)) - -(T.)) EmmTpl's
ER | [(TR)
(35)
Tn
2
Please note, T represents some iteration step, while R < T is discrete random variable, sampled non-uniformly from the set 21, T)} (i.c. support size T). This distinction a key aspect of the proof strategy, adapted from the convergence analysis of SGD (randomized stochastic gradient) from Ghadimi & Lan (2013). The proof establishes guarantee on the expected performance of non-uniformly sampled iterate R, rather than the final iterate T. Multiplying by T:,, gives
Km (e(wb)) l(T.)) Ro P
Tx En V (())
(36)
17
Using the definition of CSG, (Eq
we have.
Tx (SG
Km (E(18)) - ((w.)) + Km Kmpopl
(37)
Tmax
2
Taking expectation 2, and training randomness p:
Tz. CSG(E)
Km
[e(uio, E1) - e\* 2i)
Ee,.
<
Er,..
Tmax
"
Km Et,z.p
(1 z,) E(1i,, 2i)
Km Mmp IEe. [T,
+
(38)
28

---

Published as a conference paper at ICLR 2026
Next using the memorization result from Ravikumar et al. (2024a) (Eq. 32):
Tx,
(S)
Kmm Bt....P
((Wh, Ei)
(wT. zi)
Tmaar
1
fim
Kum 7mmppl'' Ez,p [Ta,]
+
Er
(emmm()))) +
(39)
11
2
Te. CZG(Z))
Km L
Ez memm (5) + Kum Papp
E8 2.
Ee,
[Te.
(40)
Tmax
7
We need separate the two terms within the expectation, for which the following result [[a(2)6(=)] = R[a(2)] [[b(2)] + Cov(a(z),
(41)
Km Po 2
By choosing 7 Thus, it follows that
C1
0, and by Theorem 4.2 we know Cov (O)(C 2 0.
2n
T
Kin L
Fim Pa
p
Ex,P
Ez.p
[SSG(E)]
<
Ez, [_][mm,)))) 1]
2 [T,.).
E2.
(42)
T
maax
Thus,
Rm L
(Ez, (m.m(2)) +1
E2; [Ta.]
n
(43)
Ep.s, (S)((
Natmmp
Tmax
2
For the division performed in Eq.
43
we require the denominator to be strictly positive
E, (CSG(E))
MoHimp
(44)
maax
which implies
7)s S Ep.s. (C)G(E))
(45)
T Tmaxx
The sign of the denominator is determined by the learning rate, 7t (since T/ is max learning rate over training). A sufficiently small positive learning rate can be chosen to ensure the denominator positive.
Substituting into Equation 33 in Eq. 47 yields
E. T
L E., [mem
[,) +
! Km
Ep, [CSG(Z)) - C..
(46)
Equivalently,
Km T. (1,,
Ez, [memm +
17 Ep.z. [CSG(E)] Ce:
Co Ep.z, [CSG(Z.)
(47)
Makmpl?
Km
Tmax
2
where
TTT
Rim I
KmmTiS max pT
C.
C
2 + +3 C. C. L L2, [mem (ž,)
Km
2
For the above division 47 to hold, we need C, 0 and denominator > 0, we have already shown a sufficiently small positive learning rate can be chosen to ensure the denominator is positive.
Next about C.. since the learning threshold is arbitrary we choose that, C. > 0. This is true k2 pIp
r
2n
29

---

Published as a conference paper at ICLR 2026
Rearranging Eq. 47 as quadratic we have
11 7/s
Bp... [C)
Can
Bp.z, [CSG
+ ? 1) pm Co
(48)
Kim mmaz
Tmax
2
S Ka Km (Ez, [mom [2)]
+1
(49)
Note is a function of Tman- Normally for dominant term analysis we can assume d) x VTmas- Let
X Ep.e. (CSG(E))
(50)
Tmax
Then the inequality takes quadratic form
ax² bx + + 0,
(51)
with
777 Km
b
Cen
17 m
Ma
Catmm
Ez [mem (2:)
1).
T2
2T max
2T max
Tiax
(52)
By the quadratic formula, the inequality holds if
r X 12:
(53)
where rı, r2 are the roots.
Since C. 17, Ce x Ex, [mem {]], and typically n x VT we have
b = 0 Ez, [E))(E)]
V4ac
Ez, (mmmmmm])
(54)
2a
Vmax
2a
VTmax
Thus the dominant contribution is linear, yielding
Ep.as [CSG( = Of Ez, [mem [z]]] Tmax)
(55)
Often Tmax fixed thus we can write the results as
Ep. (SSG(E)) = O Ee, [mem( x,])
(56)
pmf
Conditon for real roots. Note from Ghadimi & Lan (2013) we have 2
2 0.
2
For 122 4ac 2 O we need e S 0. This is true if Ce < 0 or C. 2 0 then, we know from the quadratic
1) Mo
CC
Catimai (Ez, [mem( [i) +
(57)
27
Tman
e 0
(58)
11 11
Km
Km
(Ez, [mem( + 1)
(59)
maaax
Km
2
T maa
Kim kmppTP?
[mem( {2)] [1]++1] 1)
1/ [EZ, [mem (2)) +1)
1)s
(60)
2T 2Tman
27man
Tian
[Kmppl 2C + kmppt pI L(E,. mem (z)) + 1)]
< 2n L LR,, [mem [,) +1)
(61)
Thus we have
Ms [kmmpI + N² pl (Le, [mem [[]] + 1)] < 2 [mem[ [,)] + 1)
(62))
21TT T. 2,,, |nem [ 1)
17. <
(63)
+ x² pI 2 Ez, [mem (z:) + 1)]
2r L mmmm [2,] 1)
n7
(64)
kmpl?
C. + Kmdk (R., [mem [ + +1)
30

---

Published as a conference paper at ICLR 2026
Final Learning Rate condition. For sufficiently small positive learning rate, the condition b² 4ac > 0 will hold true. Therefore, a suitable choice of d/, exists, given by combination of results
from Eq. 45 and Eq 64
LTL memm (z4)] + 1) C. + Kick (Ez, [inem [ii]] + 1)];V
Ep.c [CSG(Z))] T mi pIp
0 S 11 S VT Ts S
min
(65)
Kmply
Conclusion.
This completes the proof. That is,
Ep.a; [CSG(E)) = O Ez, (mem(=))))))
(66)
subject to the step-size constraints above.
D LICENSES FOR ASSETS USED
For each of the assets used we present the licenses below we also have provided the correct citation in the main paper as well as here for convenience.
1. ImageNet (Russakovsky et al. 2015): Terms of access available at https://
image org. org.:///odoooocc.oooooooooooooo. php
2. CIFAR10 (Krizhevsky et al. 2009): Unknown No license provided
3. CIFAR100 KKrizhessky et al. 2009): Unknown No license provided
4. Pre-trained ImageNet models and code: We used pre-trained ImageNet mod-
els from (Fllmann & Zhang 2020 which is licensed under Apache 2.0 https:
//qithub.com///// blob :lob/mastter/LICENN55
reeeereseeeehhhhee dout influence- est estiitiinn..
5. Baseline methods:
We re-implemented the baseline methods hence provided with along with our code which is distributed under the MIT License.
6. Pytorch (Ansel et al. 2024): Custom BSD- style license available at https:
comm/yytorchh/yttrrhh/bbb.
ma LICENSE
7. ResNet Model Architecture (He al. 2016): MIT license available https://
car/b
ob/master/LICNNS
E COMPUTE RESOURCES
All of the experiments were performed on heterogeneous compute cluster consisting of 1080Ti's. 6 2080Ti's and 4 A40 NVIDIA GPUs, with a total of 100 CPU cores and a combined 1.2 TB of main system memory. However, the results can be replicated with single GPU with as minimal as 8GB of VRAM.
F LLM USAGE
The of LLMs was limited grammar and spelling fixes. LLMS not used for any other part of the research project.
G LIMITATIONS
Compared to other memorization proxies, the limitations of CSG are minimal. Its primary con- straint is access to the training process. This is not unique, and is shared by other proxies such curvature Garg & Roy 2023), loss sensitivity Arpit et al. (2017), and forgetting events Toneval et al. (2019). This requirement is also common among methods for identifying mislabeled exam- ples Maini et al. 2022 Northeutt al. (222).
31

---

Published as a conference paper at ICLR 2026
H BROADER IMPACT
This work introduces CSG, an extremely efficient proxy for per-example memorization in deep networks that matches stability-based scores while being up to 5 orders of magnitude cheaper to compute. Because CSG can be gathered during training and aligns with memorization, learning time, and input curvature, it becomes a practical lens through which practitioners and, potentially, regulators can watch how modern models absorb, remember, and sometimes over-fit their data.
Better generalization and safer deployment. By flagging the exact examples that model memorizes, CSG helps teams diagnose over-fitting before deployment and choose early-stopping point without a validation set, cutting training epochs.
Higher-quality, fairer datasets. The method surfaces mislabeled, duplicate. and biased samples achieving state-of-the-art detection accuracy on CIFAR-10/100 and Fashion-MNIST. Removing relabeling such data can reduce spurious correlations, im- proving fairness under-represented sub-populations.
Privacy auditing and mitigation. CSG strongly correlates with membershiipiifference success and adversarial distance. Early-stopping at the CSG peak reduces a suite of neemerrshipinffreen attacks relative to the final checkpoint, making CSG both privacy-risk risk indicator and a knob for mitigating leakage without extra noise injection.
32


Reviews:
[{"review_id": "z3GZqYPWjF", "reviewer": "Reviewer_P7vQ", "summary": "The paper proposes Cumulative Sample Gradient (CSG) as a theoretically motivated and computationally efficient proxy for memorization in deep neural networks. The authors define CSG as the gradient of the loss with respect to input samples, accumulated across training, and claim it correlates strongly with Feldman & Zhang’s formal memorization score while being orders of magnitude cheaper to compute. They theoretically show that CSG is bounded by both learning time and memorization, then empirically validate this relation across CIFAR-100 and ImageNet. Moreover, they propose that the peak of the model’s weight norm corresponds to the optimal early-stopping point, eliminating the need for a validation set. The paper also introduces Sample Gradient Assisted Loss (SGAL) as an efficiency improvement, and reports strong performance on tasks such as mislabeled sample detection and dataset bias discovery", "questions": "1. Since CSG is defined as the accumulated input gradient norm, not the derivative of a loss functional over training trajectories, do we have a rigorous reason to treat it as a “gradient of loss with respect to input samples, accumulated over training”? How does this differ from simply tracking cumulative sensitivity?\n\n2. The paper asserts that stopping at the peak weight norm matches the minimum validation loss. Can this be proven under general conditions? How robust is this correspondence across architectures, datasets, or optimizers (e.g., Adam, Adagrad, adaptive schedulers)?\n\n3. Do you think the CSG–memorization relationship could generalize to regression, contrastive, or generative models, where accuracy or label definitions differ? You don’t need to perform new experiments—rather, please share your intuition on whether and why such generalization might hold.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work links input-space gradients to memorization and learning dynamics through formal theorems, extending prior work that primarily focused on weight gradients or loss-based proxies.\n\n2. The idea of accumulating input gradients incurs minimal additional computation during training and is potentially useful for large-scale data auditing, noisy-label detection, and privacy diagnostics.\n\n3. The observed “rise–peak–decline” trajectory in sample gradients and weight norms provides an intuitive link between optimization dynamics and generalization behavior.", "weaknesses": "1. The main claim that “Cumulative Sample Gradient” represents a gradient of loss with respect to the input, accumulated over training, is conceptually questionable. The proposed CSG is essentially an aggregated gradient norm trajectory rather than a true differentiable functional of the loss. Treating it as a gradient object conflates sensitivity analysis (∇ₓℓ) with memorization, which lacks theoretical grounding in generalization theory. The derivations (Theorems 4.2–4.3) merely establish loose proportionality bounds without proving causality or sufficiency.\n\n2. The assertion that the peak of weight norm universally coincides with the minimum validation loss is overstated. This correspondence may depend on architecture, optimizer, and regularization strength, and may fail under strong augmentation or non-stationary data.\n\n3. While the authors claim that CSG generalizes across tasks, they only test standard supervised image classification. The theoretical link assumes uniform β-stability of SGD and bounded loss, which rarely holds in modern deep nets. It remains unclear whether CSG maintains predictive utility in other regimes such as self-supervised, generative, or multi-label settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dR4LtmlQfC", "reviewer": "Reviewer_gxif", "summary": "The paper proposes Cumulative Sample Gradient (CSG)—the loss gradient w.r.t. the input, accumulated over training—as a computationally cheap proxy for stability-based memorization (Feldman & Zhang, 2020). The authors provide theory that (i) expected CSG is upper-bounded by learning time (Theorem 4.2) and (ii) linearly bounded by memorization (Theorem 4.3). Empirically, they observe a characteristic rise–peak–decline trajectory for average per-sample input gradients that aligns with a peak in weight norm and the first minimum in validation loss (double-descent boundary), enabling validation-free early stopping. They also introduce SGAL (a loss accumulated only until the gradient-based stopping point) for further efficiency. Across CIFAR-100/ImageNet, CSG/SGAL correlates well with memorization scores, is substantially faster than curvature and CSL proxies, supports mislabeled-sample detection, and helps surface dataset biases.", "questions": "1.\tHow sensitive is the weight-norm peak rule to weight decay, label smoothing, data augmentation strength, and optimizer hyperparameters? A small ablation across these knobs would clarify robustness.  \n2.\tPractically, do you compute input gradients every iteration for all samples, or on a schedule/subset? Please quantify wall-clock overhead vs. vanilla training across model sizes.  \n3.\tHave you tested CSG/SGAL on ViTs or transformers for NLP? If not, what obstacles (e.g., tokenization, augmentation) do you anticipate?\n4.\tCan you empirically estimate the constants in Lemma 4.1 (e.g., behavior of κ through training) to illustrate why the linear trends emerge despite potential ill-conditioning?  \n5.\tGiven Table 2 shows mixed results across ECE/MCE/MSCE/UCE, can you reconcile the claim “lower calibration errors than last epoch” and specify which metrics you prioritize and why?  \n6.\tSince you use precomputed F&Z scores, how sensitive are your correlations to training recipe variations (architectures different from F&Z)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tOriginality & clarity: Using input gradients accumulated over training as a memorization proxy is elegant; the rise–peak–decline alignment with the weight norm and double-descent boundary is compelling and clearly presented. \n•\tQuality (theory): Theorems relating CSG to learning time and memorization provide formal grounding absent in many proxies; assumptions and proof sketches are transparent. \n•\tQuality (empirics): Consistent binned linear trends; strong correlation with F&Z scores; broad comparisons (CSL, curvature, forgetting events, loss sensitivity) on CIFAR-100/ImageNet; MIA and adversarial-distance analyses support the privacy link. \n•\tSignificance & practicality: Large speedups (0.1–0.3× of standard training vs. 3.6–14.3× for curvature; orders of magnitude vs. F&Z) lower the barrier to dataset diagnostics at scale; mislabeled-sample AUROCs are SOTA or competitive at all noise levels.", "weaknesses": "•\tAssumption sensitivity and constant opacity. The theoretical bounds rely on β-stability, Lipschitz continuity, L-bounded losses, and learning-rate conditions, and exclude first-layer skip connections. Constants involving the pseudo-inverse of batch matrices (κ terms) may be large/ill-conditioned, making the bounds hard to interpret quantitatively. \n•\tCalibration claim is mixed. Table 2 shows lower ECE for the last epoch (0.1017) than for gradient-based stopping (0.1382), contradicting the blanket statement that early-stopped checkpoints have lower calibration errors; other metrics (MCE/MSCE/UCE) favor earlier stopping, so the narrative should be nuanced. \n•\tScope of datasets/models. Results are primarily on CIFAR-100/ImageNet with ResNet/Inception. It would strengthen generality to include modern architectures (e.g., ViT) and tasks beyond image classification, since input-gradient behavior and training dynamics may differ. (The Adam experiment is a useful first step.) \n•\tComparative coverage. While CSL/curvature/forgetting/loss-sensitivity are included, some adjacent proxies (e.g., EL2N, GraNd / importance-sampling-style difficulty measures) and influence-based approximations (e.g., TracIn) are not compared; these could provide a more complete picture of trade-offs.\n•\tTraining-access requirement. Like many proxies, CSG needs access to per-sample gradients during training; this limits pure post-hoc auditing scenarios (the limitation is acknowledged). \n•\tQualitative bias analysis. The bias discovery examples are informative but largely qualitative; quantitative fairness metrics (e.g., subgroup error rates) would make the case stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JygZB8rteV", "reviewer": "Reviewer_BKpC", "summary": "The paper proposes CSG, a fast, theoretically grounded proxy for measuring memorization in deep networks by accumulating input loss gradients during training. CSG correlates strongly with true memorization scores while being up to five orders of magnitude more efficient, enabling validation-free early stopping, mislabeled-sample detection, and bias discovery.", "questions": "Above", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "CSG offers a theoretically grounded and computationally efficient way to estimate memorization, achieving near-perfect correlation with true scores at a fraction of the cost.\n\nIt enables validation-free early stopping and state-of-the-art mislabeled data detection, making it both practical and interpretable for large-scale deep learning.", "weaknesses": "Novelty:\nMy primary concern lies in the novelty of the work. The authors’ main observation that memorization tends to occur in the later stages of training is well established and has been extensively documented in prior studies [1,2]. Likewise, leveraging gradients to approximate or track memorization has been explored before [3,4], making the core idea appear incremental rather than groundbreaking. Can the authors show how their work performs in relation to [1,3,4].\n\nComputational Cost:\nWhile the proposed approach claims efficiency, computing cumulative sample gradients still requires forward and backward passes for each sample at every epoch, which can be prohibitive for large models and datasets. Prior work [3] already proposes strategies to reduce this overhead.\n\nLimited Optimizer Evaluation:\nThe experiments rely solely on the Adam optimizer. To demonstrate broader applicability, results should be validated across multiple optimizers such as SGD, RMSProp, and AdamW. Can authors provide more info on how their method would behave across optimizers?\n\n\n[1]. Agiollo, Andrea, Young In Kim, and Rajiv Khanna. \"Approximating Memorization Using Loss Surface Geometry for Dataset Pruning and Summarization.\" Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.\n[2] https://aclanthology.org/2024.blackboxnlp-1.4\n[3] https://arxiv.org/abs/2008.11600\n[4] https://arxiv.org/pdf/2002.08484", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcLjSfGWdo", "reviewer": "Reviewer_maBN", "summary": "This work proposes a computationally efficient way to approximate the degree of memorization in deep neural nets. Based on the observation that memorized samples tend to have longer training time, this work proposes cumulative sample gradient (CSG) as a proxy for memorization. Theoretical results show the relation between CSG and learning time & memorization. Empirical evaluations corroborates the findings and shows superior computational performance over previous state-of-the-art.", "questions": "1) What makes the computation of CSG faster than CSL? Seems that both metrics are cumulative and the computation of loss/gradient are not too different in general. Could you tell us more about the source of speedup?\n\n2) The theoretical results show that CSG is upper bounded by learning time and memorization. Does that mean high CSG -> high memorization? What about the opposite direction, does low CSG -> low memorization?\n\n3)  The theories are formulated against pure SGD. Does CSG reliably detect memorization/mislabeled samples for different optimizers? If time allows, could you show some evidence of CSG's success for other optimizer? \n\n4) Are previous SOTA's performance dependent on the choice of optimizer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Overall, this work is well-written and organized. It motivates the problem settings and draws fairly clear connection with previous work. The contribution is pertinent to the current challenge of ML. By providing a more efficient probe for the phenomenon of memorization, this work can accelerate future research in this field. The theoretical formulations are solid and with clear purpose: they do shed light into the construction of the practical proxy. The empirical improvements are encouraging. No major flaw with experiment design.", "weaknesses": "The work doesn't have apparent weaknesses that might lead to clear reject. I do have a few questions about the assumption and the source of computation edge over previous work. Having them clarified can better help the reader understand the contribution and use the tool with confidence.\n\nThere are a few grammatic glitches here and there. For example, \"is plays\" at Line 310 and \"it's roots\" at Line 258. Can be fixed by proof reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)