Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes
Authors: WEI-KAI CHANG
Rajiv Khanna
Venue: NeurIPS 2025 poster ()
Keywords: Coreset Selection
Loss Landscape
Smoothed Loss
Training Efficiency
Robust Optimization
Primary area: optimization

Abstract: As deep learning models continue to scale, the growing computational demands have amplified the need for effective coreset selection techniques. Coreset selection aims to accelerate training by identifying small, representative subsets of data that approximate the performance of the full dataset. Among various approaches, gradient-based methods stand out due to their strong theoretical underpinnings and practical benefits, particularly under limited data budgets. However, these methods face challenges such as naïve stochastic gradient descent (SGD) acting as a surprisingly strong baseline and the breakdown of representativeness due to loss curvature mismatches over time.

In this work, we propose a novel framework that addresses these limitations. First, we establish a connection between posterior sampling and loss landscapes, enabling robust coreset selection even in high-data-corruption scenarios. Second, we introduce a smoothed loss function based on posterior sampling onto the model weights, enhancing stability and generalization while maintaining computational efficiency. We also present a novel convergence analysis for our sampling-based coreset selection method. Finally, through extensive experiments, we demonstrate how our approach achieves faster training and enhanced generalization across diverse datasets than the current state of the art.

Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes
Wei-Kai Chang
Rajiv Khanna
Purdue University chang9860purdue edu
Purdue University rajivakOpurdue.cm edu
Abstract
As deep learning models continue to scale, the growing computational demands have amplified the need for effective coreset selection techniques. Coreset selection aims to accelerate training by identifying small, representative subsets of data that approximate the performance of the full dataset. Among various approaches, gradient-based methods stand out due to their strong theoretical underpinnings and practical benefits, particularly under limited data budgets. However, these methods face challenges such as naïve stochastic gradient descent (SGD) acting as surprisingly strong bascline and the breakdown of representativeness due to loss curvature mismatches over time.
In this work, we propose novel framework that addresses these limitations. First, we establish connection between posterior sampling and loss landscapes, enabling robust coreset selection even in gigh-data-corruppin we introduce a smoothed loss function based on posterior sampling onto the model weights, enhancing stability and generalization while maintaining computational efficiency. We also present novel convergence analysis for our sampling-bas based coreset selection method. Finally, through extensive experiments, how our approach achieves faster training and enhanced generalization across diverse datasets than the current state of the art. (Code are available in: https: //github. com/
scenarios. Second,
demonstrate
Introduction
Modern deep learning thrives massive datasets, but training on all available data can be prohibitively slow. Coreset selection aims to address this issue by selecting a small, representative subset of training data that can be used in lieu of the full dataset, thus accelerating training while preserving model performance. Recent gradient-basee based coreset methods (Mirzasoleiman et al. 2020 Killamsetty et al. 2021a Pooladzandi et al. 2022 Yang al. 2023) have shown promise by approximating the full-data gradient using a subset. In practice, however, we observe a critical failure mode in these approaches: the loss landscape induced by the selected subset is often misaligned with that the full dataset. This misalignment becomes especially pronounced under realistic conditions like label noise, adversarial corruptions, very small subset budgets, leading poor generalization despite the subset's seeming adequacy in matching gradients at selection time.
To illustrate this misalignment problem, consider model selecting coreset noisy dataset. Gradient-based selection methods tend to prioritize examples with large gradient magnitudes. If some training labels corrupted (or the model is under-trained, the coreset size is small), such methods can inadvertently overweight outliers or mislabeled examples simply because they induce large immediate loss gradients. The resulting subset produces distorted loss surface: the model trained on
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(b)
(c)
(d)
Figure 1: (Left) The smoothed surface: The loss landscape generated by coresets selected by (a) Craig (Mirzasoleiman et al., 2020) The loss landscape generated by the coreset of (b) our method. Our method smoothens the induced landscape. The graph is generated with CIFARIO 10 data and ResNet20 using 1 % data budget. (Right) Better match in loss landscape: The black line in both the figures represent the loss landscape of the full training set for MNIST with LeNet using 1% data budget for coreset selection. (d) Our method matches the underlying landscape better than (c) Craig. The mean square error between the two landscapes for Craig and our method are 0.0703 and 0.0662 respectively. (see details in Appendix B.4 and Figure
this subset will descend along directions that differ from those it would follow it saw all data (See Section A.8) In other words, the subset's gradient and curvature information (its geometry) deviate from the full data's. In fact, even without noisy labels, minor aberrations in gradient approximations (e.g. due to small coreset bugdet) can distort the induced loss landscape (see Figure (Left). We term this phenomenon loss landscape misalignment. Over time this discrepancy can magnify the model's update on the subset can lead it into a region of parameter space that is suboptimal for the true objective. Empirically, we observe that with 20%-50% label noise on benchmarks like CIFAR- 10, CIFAR-100, 100, and TinylmageNet, the performance many gradient-matching based methods degrades catastrophically in some cases barely above chance revealing their brittleness in the face of corrupted data.
Prior attempts to address this instability have looked to second-orde information. By incorporating Hessian estimates. one can in principle better align coreset's loss landscape with the full data's landscape. Indeed, recent methods have tried to match not only gradients but also curvatures (c.g. using approximate Hessians) to select more robust subsets YYang et al.) 2023). Unfortunately, these second-order approaches come with serious drawbacks. Computing or approximating Hessians in deep networks is extremely costly and often numerically unstable. The required computations can nullify the very speed-ups ups that coreset selection aims to achieve (as noted by Okanovic et al. (2023); Mahmood et al. (2025)). Further, we empirically also observe these methods are still susceptible to noise in the labels and can even diverge due to sensitivity to noisy curvature estimates.
We propose new perspective: posterior smoothing for coreset selection. Rather than deterministically matching gradients or explicitly calculating Hessians, we sample model weight perturbations from a Gaussian posterior centered at the current parameters, and use these perturbed weights estimate the landscape and evaluate the coreset selection criteria. Intuitively, this strategy smoothens out the loss landscape by marginalizing over local neighborhood in weight space, effectively simulating a Bayesian posterior over models. By averaging gradient information across multiple weight samples, our method obtains a Monte Carlo smoothing of the coreset objective, and yields more stable and poometry--ligned selection criterion. This yields smoothed loss function that closely preserves the underlying structure of the true loss landscape while damping spurious oscillations due to noise or small sample size.
The proposed posterior-sane sampling strategy is both scalable and theoretically grounded. It requires no explicit Hessian computations; the added overhead is merely handful of forward passes for sampling, which is much faster compared full gradient evaluation. We provide a rigorous analysis showing that posterior smoothing provably improves alignment between the induced and full-data loss landscapes. particular, under standard smoothness assumptions, we prove that our posterior sampling leads to tighter approximation of the true gradient and Hessian of the entire dataset (see Theorem 3.2). This result formalizes the notion of "geometry alignment": the subset sees nearly the same curvature the full data (see Figure (Right) for weal-world illustration). Finally, we derive convergence guarantees for training on smoothed coresets: under standard assumptions (e.g. smoothness and bounded curvature), we prove that optimizing the coresct will converge to

---

solution that is e-close to the full data optimum. As a side effect, we improve the best known previous rate under the same set of assumptions by ((1/VM) for multiplicative noise, where M is the number of samples used in Monte Carlo smoothing.
Empirical results on a wide range of datasets strongly support our claims. Our posterior- posterior-based coreset selection consistently outperforms otate-of-the-art methods in both accuracy and convergence speed, while using fraction of the data. The gains are most pronounced in high-noise settings - for instance, with 50% corrupted labels on SNLI, our approach outperforms the next best method by 7% absolute accuracy. Interestingly, for this setting the next best method is Random Sampling - a notoriously strong baseline in data selection. We find that our method handily beats the baselines across almost all settings across multiple datasets (SNLI, TinyImageNet ImageNet- 1kk CIFAR-100, CIFAR- 10, MNIST), across different architectures (LeNet on MNIST to ResNet-50 on VinllmageNe//maggeet Ik and RoBERTa on SNLI), across different subset sizes and different noise corruption levels. Further, our memory footprint is smaller, especially compared to the next most frequently best method (Crest (Yang et al. 2023)) which requires expensive Hessian approximations and sometimes has larger memory footprint (see Figure 2
(Right) with longer run time than our method and sometimes even full data training. To further solidify the supremacy of our method, we also achieve 20% 200% speedup for sea-ttto-ighet-ccccuaaae
compared to Crest across several different datasets. Our ablation studies reveal practical insights well. For example, we find that posterior sampling through normalization layers (rather than into all model weights or final-layer weights) provides an optimal trade-off between perturbation and stability, further boosting performance (See Table 2).
To summarize, we make the following technical contributions in this paper:
Robust coreset selection via posterior sampling. We introduce novel posterior smoothing framework for loss landscape alignment under subset selection. We prove that sampling weights from a local Gaussian posterior yields more faithful gradient and curvature estimates for the subset, resulting in provable improvements in both Hessian alignment and Newton- step similarity between the coreset and full dataset (see Theorem 3.2). This perspective opens up new family of coreset methods that rigorously account for model uncertainty during selection.
Extended convergence theory. We provide comprehensive convergence analysis for sampling-based coreset SGD, strengthening and improving prior theoretical results. Theorem 3.3 Our analysis accounts for the dual sources of randomness subset selection and weight sampling - and characterizes how different noise structures (e.g. spherical vs. Hessian-informed Gaussian) influence convergence rates.
Exhaustive empirical validation. Through extensive experiments on vision and NLP benchmarks, we demonstrate that our approach consistently outperforms existing coreset methods across both clean and corrupted datasets across different noise levels and model architectures. Notably. under severe label noise (20-50% of labels corrupted on CIFAR- 10, CIFAR-100, TinyImage ImageNet- 1K, SNLI), our method remains remarkably robust while prior gradient-based approaches fail. We also highlight key implementation findings for example, the importance of imposing posteriors in solely the normalization layers that further enhance coreset effectiveness. Together, our results establish new state of the art with minimal overhead and memory footprint in both accuracy and stability for coreset selection in deep learning.
For a detailed review of related work, please refer to Appendix A.I
2 Background
The classic training objective is to optimize the Empirical Risk Minimization (ERM) on the training dataset.
w" argmin ((w) = argmin (ww).
(I)
For tractability and better generalization, in practice we use SGD (stochastic gradient descent) for optimization of ( can lead to large memory burden and can be inefficient due to the calculation time.
3

---

Algorithm Ensemble Coreset(r, E, T. B, w/. P)
1: Parameters: Subsample size Rt Ensemble size M, Max epochs T. Number of Batch B, Initial
model parameters wo, Learning rate ', Number of batch selection P. Gaussian Prior 8, minibatch size m
2: for 1 to 7 do
3: po 1 do 4: Select random subset V S. =
5: Select Sp arg mins, Liev, minjes, Es Es V + - V4((w; +8]1,, |sp1 S m end for
6:
6: S. = Upe[p (Sp)
7:
for to B do
8: Sample batch S. C Ss. |&b
9: W1.b+1 = Wrbb Vls, (wbbb)
10: end for
11:
end for
The iterative update of the model parameter can be written as:
Wh-11
Wt 1yV
where is the random subset sampled from the whole training dataset and the gradient (w:) evaluated on the subset. Several studies have investigated the convergence rate of stochastic gradient descent (SGD) under different settings. In particular, Ghadimi & Lan (2013) established while Xiao al. (2015) extended this analysis to the mini-batch setting, Rit ), where R represents the mini-batch size.
convergence rate of O +
obtaining O
Building up on the SGD, the gradient- baseed coreset selections (Mirzasoleiman et al. 2020 Killam
setty et al. 202 la Pooladzandi et al. (2022) aim to find subset of data whose gradient aligns with the gradient direction of whole training dataset. The formulation
follows.
S* argmin SS'| 5'
s.t WLEW
Σ (w:)):
Σ Vw.aill jES'
(3)
Here yi is the weight for the specific sample i. The goal is to jointly optimize for both-the subset S out of the full dataset and the weights "/ while ensuring an error of at most e. Practically, the inner optimization over W is often omitted by setting W {we), the singleton set of the current iterate in the SGD. As pointed out by Mirzasoleiman et al. 2020). we can transform the problem (3) into a submodular cover problem (see Appendix for details):
S argmin SS|| s.t Σ min (l,(w)) - vlj(w:)ll
(4)
CS
JES
3 Posterior-Stable Coreset Selection: New Paradigm for Landscape-Aligned Subsets
The stability in optimization and its importance for generalization is widely studied in the Nguyen ct al.) (2022): Chen et al. (2018): y Lei (2023); Harrison et al. (0022) Attia & Koren (2022). For
example, Bisla et al. (2022): Duchi et al. (2012) suggest making the model more stable by sampling model weights with Gaussian posterior. This leads to smoothening of the loss surface and the induced
stability can help with generalization. Liu et al. (2022) adapted similar idea to sharpness-aware minimization algorithm to help stabilize the training process and gain performance advantage.
We build on these previous works and seek related but different goal. Subset selection can lead to highly unstable and non-smooth loss surfaces (Shin et al. 2023) (see Figure [ How we choose coreset that improves stability of the selection process so that the induced loss landscape matches with the underlying ERM loss landscape Towards this end, we first propose new definition for the stability of the selected coresets:

---

Gredient Dfffrrence Comserison
Hathod
Orec
(Left) Gradient match: The average gradient estimation enoo for method and Craig selection method using LeNet on MNIST. The error is calculated with [S] Eies V4,(w)) 1ST Ejes 7t Vly((,. where S is the training set and S' the subset se- lected. Our method generally produces smaller gradient errors and better gradient estimation com- pared to the Craig method. (Right) Memory buffer: The memory consumption for Tinylmagenet. Our method shows less memory during the training process. Crest( Yang et al. (2023) requires to maintain the information of Hessian during training and the intermediate calculation such as checking the threshold calculation of Hessian norm also require large memory buffer. For the plot, we use running average to average over the time step and show the mean value.
Figure 2:
Definition 3.1. (a, c, w) -taabiiiy subset S' is called (a, e, -table there exists o, such that
(w.N(( Vls (w) - Vl(w)
Sc Sc.
where Vls the weighted gradient over the subset and Vl(w) is the full gradient over entire dataset. This definition captures the essence of stability by quantifying how much the gradient varies when considering perturbations around the model paramcter W. Similar stability frameworks
have been explored in the literature (Bisla et al. 2022, Liu et al. 2022 Duchi al. 2012) highlighting the crucial role of stability in ensuring that machine learning models for generalization and robustness. A key side-effect of the stability is the smoothening of the function (See Theorem 1 in Bisla et al.
)2022))
Algorithm Our goal select the (a, e, st-sbable coresets instead solving the classic coreset problem 3 by replacing the maximization over the domain W with the stability constraint. Similar to the transformed coreset problem (4). we write our coreset selection optimization problem:
5 arrgmnn |S S'CS
s.t
min V3llVi, JES
VL + S
(6)
where 8 (N, ol) is hyperparameter design variable that controls the perturbation of the weights w. By optimizing this selection process, we aim to ensure that the coreset S' retains the essential characteristics of the original dataset while adhering the stability constraints characterized.
Our algorithm is presented in Algorithm T (For time complexity analysis, please refer to appendix () In each epoch I, create pseudo dataset by subsampling stable gradient-matched data points, P times. Instead of solving the constrained selection to match full gradient up to e, we select m data points each time is standard in other implementations of similar algorithms. This creates a shadow dataset S, of size P * m. We then use the shadow dataset S, to compute the gradients and optimize the model parameters using batched stochastic gradient descent S.. This iterative approach allows for refining the model with each epoch, progressively improving its convergence properties and stability. (For more detailed discussion, see Appendix A.7 and Appendix C
For deep neural networks (DNNs), adding stability to the entire wi can be prohibitively expensive. Previous works have tried to mitigate this by focusing on the parameters in the last layer of the neural network (Yang et al. 2023 K Killamsetty et al.) 202 Pooladzandi et al.) 2022). However, more recent research (Mahmood al.) 2025) has shown that such strategies can be detrimental because of implicit regularization properties of the SGD. Motivated by the recent studies (Mueller al.. 2023
Frankle et al. 2021 Xu et al. 2019) importance of normalization layers for stability and improved
5

---

(a) CIFAR-10
(b) CIFAR-I0M
(c) Tinylmagenet
(d) Imagenet- Ik
Figure 3: Time-to-acuurac: We plot the time taken to achieve certain validation accuracy against
the state-of-the art Crest Yang et al. (202)) which uses 2nd order information for coreset selection and show that our proposed method is more efficient and faster in achieving the same validation
accuracies.
predictive performance in DNNs, we focus attention for sampling in the batch normalization layers.
We now discuss theoretical properties of stable coresets. Our algorithm can be viewed as coreset selection on the smoothed loss function that leads to stable and better performance on we shall see in the sequel. Furthermore, loss landscape. Previous works have proposed that in order to successfully select key samples in training dataset, the loss landscape over selected samples must match to the loss landscape of the whole training dataset. To do so, these works relied on calculating or approximating the parameter Hessian matrix which can be prohibitively expensive and using coresets while also requiring additional memory. Instead, our sampling based strategy is a cheaper and faster way match the underlying loss landscape without direct calculation Hessians. To understand the relationship between the Gaussian perturbation and loss landscape matching. We propose a our theory in the following. (The proof is in appendix A.2)
establish the relationship between the posterior sampling and the
offset the speedup obtained by
Theorem 3.2. Suppose subset S' C S is (a, stabla and let the Hessian difference be H we Hss =: &, then,
(I) The Hessian difference matrix & satisfies:
| < ( and tr(E²)
tr(E))= =
(2) The difference between newton step of two subset is bounded.
| Vlg (w) Hi Vig(w)]l S O
Discussion: Theorem 3.2 formalizes the notion that posterior sampling provides an effective means ensure the alignment of the loss landscape of the chosen coreset with that of the underlying data. Specifically, if the gradients of selected subset match those of the full dataset under Gaussian posterior sampling. the discrepancy between their Hessians remains small. This aligns with the findings of Shin et al. (203)), which emphasize the necessity of Hessian similarity across subsets. Furthermore, our approach implicitly ensures that inverse-Hesse Hessian weighted gradients are also aligned, satisfying the selection criteria established by [Pooladzandi et al. (2022). By leveraging posterior sampling, we circumvent the computational challenges associated with direct Hessian comparisons while preserving theoretical rigor. For even greater control over this alignment, more sophisticated posterior distributions can be designed to precisely fine-tune the loss landscape matching between the coreset and the full training dataset (see appendix A.5)
Convergence Analysis Lastly, we also provide convergence analysis for the coresets Mini-batch Stochastic GD along the lines of SGD analysis of Yang et al. (2023) except that we generalize it for the stability setting and improve on their convergence rate.
Theorem 3.3. Say w ~ N (ww, σzI)) at epoch t. Assume the I(-) is 8-smooth. and the expectation in (6) is calculated by taking M samples. We consider noise in the gradients resulting from the random sampling of batches and coresel selection as and &a respectively:
Vi(w) Vlg(w) E1 +
(1) (Absolute noise) If the noise in coreset selection is of the form:
Ell|2211 S

---

JOORY
Drip
Figure 4: Strong against all baseline: Performance at different corrupt ratio with respect MNIST. EMINST, CIFAR-I0 and CIFAR-100. Our method (purple line) consistantly outperform others method different corrupt ratio, and the performance drop is less when raising the corrupt ratio. Note that the Random sampling method still suffer sharp drop in the high corrupt ratio region in CIFAR- 100 dataset.
then by setting the learning rate to be min{ / B rate VT
and oîd =
1
We can have convergence
MVT
Ed V & O( 1 ( - ((lwwww)))))))>)))
B
3e
Bo
+
(7)
2M
M
MR
2. (Multiplicative noise) the noise in estimation of gradients is of the form below
F|l|a2l1| S <||VI(w)]l
(8)
we o2d
and 1) minf V bb we can have convergence rale
MVT
VMRT
1 (2(l(u0)) [(ww(()))))))))>>)>()) + 82 2
[((w))] O0 V VMRT
(9)
Discussion Theorem 3.3 studies the tradeoff between noise levels and corresponding convergence rates. In case 1 of relatively larger absolute noise, we improve certain terms over previously known standard SGD rates by a factor of M, though we have an additional term due to the noise injected into the model weights. These trade-off between different forms of noise actually give us some room for engineering the overall training process. For example, we could avoid the additional term by switching the naive SGD when close to end of the training. Another observation is that the trade-off will be more favorable towards our method when the random sampling noise σ becomes large which happen in the corrupt learning situation (data corruption) making our method more stable in such a situation. In the second case of smaller (proportional to the gradient magnitude)
noise, we improve the convergence rate from O ((Tr) Yang al. 2023 O VMMTT"
Selective Sampling and Stability Batch Normalization (BN) and related normalization techniques
are central to the training and generalization of deep learning models. influencing both the loss
landscape and optimization behavior (Santurkar et al., 2019;; Sun al. [2020] Waang al. 2021).
Recent studies also emphasize the role of noise in normalization layers (Kosson et al. 2023
Liang
et al. 2019), underscoring their importance in stabilizing learning.
Our method involves sampling around model weights to capture curvature information without explicitly computing the Hessian. However, full-model sampling is computationally expensive, and larger variance (e.g., oÿd) leads to smoother loss surfaces but slower convergence (see Theorem 3.3) To balance efficiency and stability, restrict sampling to the batch normalization layers. This choice is supported by recent findings that highlight the unique role of BN layers in controlling sharpness and enabling efficient optimization (Mueller al.) 2023 Frankle al. 2021 Xu et al., 2019). While this design is empirically motivated, developing a full theoretical understanding of BN ''s stabilizing effect remains an open question.
4 Experiments: When Does Posterior Sampling Help Most
To evaluate our method. We test our method on ResNet models (ResNet20, ResNet1 18 and ResNet 50) and transformer based models (VIT, RoBERTa and ELLECTRAA-mlll-diiicimiaanoo with vision datasets (MNIST, EMNIST, CIFARIO, CIFAR100, Tinylmagenet, Imagenet) and language datasets
7

---

(SNLI. REC-50). It achieves state-of- ofate-of-the-asd art accuracy with improved time and memory efficiency, and demonstrates greater robustness label noise than baselines ! All results are averaged over three random seeds for reproducibility. To reduce compute overhead, we apply Gaussian sampling only to batch normalization layers. Each experiment uses sampled models, with a selected via cross-validation from 0.1, 0.01, 0.001. Sampling is performed dynamically during forward passes, eliminating the need for separate model copies and adding at most one extra layer's 's memory overhead. This makes our method scalable to larger networks. (See Appendix B for full setup details and Appendix A.9 for more experiment setting such as training budget and archtectures.)
Gradient Matching: Importance of loss landscape. As noted by YYang et al.] 2023] Shin et al. 2023), naive optimization of the coreset function (3) match the gradients without accounting for the loss landscape as done by Craig (Mizasoleiman et al. (2200) may not actually yield the the best
quality subset. We verify this, and show that even on simple dataset like MNIST, coresets outputted
by Craig as vastly inferior to the ones we generate in matching the gradient of the full dataset. The results are presented in Figure
Dataset
Corruption
Method
Random
0.0
0,9132a0.0013
0,901610000200
0.0022
SNLI
0.1
0.3
0,866420.0012
0.7841+0.0021
0.752940.0031
0.832440.0054
0028
0.5
0.606210.0016
0.531610.0024
0.510420.0055
0.0
0,573210.0011
0.552010.0094
0.560920.0040
TayylmggNN
0.1
0.5526:9.0041
0.5176400.005
0.5150=0.0044
0.3
0.4832+0.0043
0.419340.0063
D.7760+B.0061
0.5
0.3644+000008
0,2857+0.0137
0.3567#0.0069
0.0
0,709110.0004
0.707420.0004
0.7136200.0015
ImageNet-1k
0.6977000016
0.690510.0016
0.694650.00035
0.3
0.6837+0.0007
0.6914+0.0011
D.6606-0000024
0.5
0.6.88=000008
0.593940.0017
0.6051
Table 1: Large scale experiment:
Test accuracy under varying corruption levels on SNLI, Tinylm-
ageNet, and ImageNet- 1k. Our method consistently outperforms Random and Crest. We did not compare to Craig and Glister on TinyImagent as we run into Out- Out-Of-Memory errors while using code base mentioned in Appendix B Note that despite Crest marginally outperform our method in Imagenet-16 1 at zero corruption, the time taken is double of our method (Crest: 42 hours, Ours: 20 hours) along with triple memory consumption.
The role of sampling in batch normalization. Our decision to conduct sampling on BN layers is motivated by both computational efficiency and their fundamental role in shaping model behavior. Empirical studies show that sampling on various layers improves performance, but BN layers consistently yield the best results across different models and datasets. Given this trade-off between computational cost and effectiveness, we focus on BN layer sampling to demonstrate the robustness of our method. The table 2 (Left) performance comparison between sampling at different layers of ResNet20 for CIFAR-10 10 dataset.
Robustness and Accuracy. We perform extensive study across different datasets and label corruption ratios to show the advantage of our method. We summarize the numerical results in Figure 4 For raw numbers, see Table in the appendix. We can observe that our method outperforms established benchmarks across several different settings, especially when the label corruption ratio is high. Most methods suffer a drastic performance drop when the corrupt ratio is increased due to the stability. Naive SGD has been shown to be a strong baseline Okanovic et al. (2023) with little overhead, and our experiments confirm its effectiveness over several traditional baselines. However, its performance also deteriorates significantly with increasing corruption. We attribute this to increased stability of our method compared to other methods. In fact, we also observed that some methods can fail the training as the instability arising out of coreset selections for noisy data can cause the loss to go infinitely large (see caption in Table 4p. To further demonstrate the advantage of our method, we conduct an experiment on even higher corrupt ratio (0.6 - 0.9) in table 2 (Right) and find that our method offers strong performance compared to others. Finally, many methods in coreset selection could not scale to large datasets such as SNLI, Tinylmagenet and Imagenet- 1k (see Table II)
Test performance:
Training dynamics: Memory Footprint and Time-to-Accuracy. As shown in Figure 2 it requires significantly less memory than Crest which performs 2nd best accuracy-wise after our method in
For the comparison with Pooladzandi et al. 2022). see Appendix E
8

---

Dutaset
Corrupt Ratio
Our Method
Random
Crest
0.704 0,0063
0.6122
0.0018
0.334
Layer
2
0.0004
0,8479 (U00)
018079
010064
0.7288 0.0140
CIFAR-10
0.5511 0.0221
0.3701 0.0053
0.5221
0:3011
0.0084
0.1520 0.0276
0.2853
D:0328
RN
088757 a 0JI029
0.8544
0,0834
0 8.065
0.731& a 0.0143
0.9
0.0953 a: 0,0183
0.0650 a 0,0296
0,0938 a 0,0108
PC
0.670S a 0.0019
0.8525 0.0633
0.8099 0.0043
0.7252.00001
0.6
0.3911 a 0.0063
0,2528 + 0.0151
0.3306 + 0.0054
CHAR-100
0.8
0.7
0.1680 0,11107
0,2810 a: 0.0045
0.0863 + 0:0211
0.1504 4 0,0033
0.1613 + 0.0084
0.4422 a 0,0025
0.0761 0.0029
0.0367 0:0087
0.007 0.0061
Table 2: (Lefft) Sampling at different layers: CIFAR-10 10 test accuracy under varying corruption ratios. BN: BatchNorm; FC: Fully Connected; All: All parameters perturbed. (Right) Robust at even higher corruption: Performance comparison on CIFAR-10 and CIFAR-100 under corruption ratios from 0.6 0.9. Our method consistently outperforms Random selection and Crest.
predictive performance on non-corrupted data,. Crest consumes up three times more memory than full training due to tracking additional statistics(their algorithm maintains Hessian and accumulated gradient steps for adaptive coreset updates). While their strategy reduces re-selection frequency for coresets, the computational overhead often negates its theoretical potential benefits (Figure 3) and lead to slower training. Our method consistently outperforms Crest across various settings, from small-scale models (MNIST + LeNet) to large-scale architectures (TinyImagenet / Imagenet / SNLI + ResNet50 RoBERTa).
Design of more effective posteriors The choice of posterior can influence the tradeoff between fidelity to the loss landscape and stability. We test with different posteriors - (a) the spherical Gaussian prior with fixed hyper-parameter a, Gaussian with the Hessian-inverse as the covariance matrix, and an Ensemble of models with 4 different random seed with coresets selected as per Algorithm [ For Ensemble, we average the gradients of the final layer and use for coreset calculation. As shown in T the Spherical-Gaussian posterior achieves the highest final accuracy across various scenarios. We analyze potential failure modes associated with different posteriors, with Figure 5 (in Appendix) illustrating key failure reasons. For CIFAR-10 we observe that Hessian Gaussian posteriors require 5962.6s, 3454.2s (parallel), and 2073.5s, respectively, compared to 3553.5s for full training. Notably, only the Spherical Gaussian posterior offers speed-up. While training multiple models with different random seeds can be parallelized, the increased memory overhead outweighs the benefits, making it a less favorable trade-off.
Ensemble, and
Spherical-Gaussian
Dataset
Corrupt Ratio
Hassian
Ensemble
Spherical-Gaussian
0.0
0,8721 0.0035
0.8738 0.0010
0.8757 : 0.0029
0.1
0,8523 0.0022
0,8506 # 0.0010
0,8544 + 0.0034
CIFAR- 10
0.3
0.8041 0.0240
0.8036 = 0.0010
0.8120 0.0058
0.5
0.5870 0.0156
0.7216 0.0030
0.7318 0.0143
0.0
0.6841 # 0.0045
0.6968 # 0.0070
0.6986 # 0.0025
0.1
0,6438 0.0043
0.6618 = 0.0030
0.6644 0.0034
CIFAR-100
0.3
0.5731 0.0032
0.6066 0.0020
0.6085 0.0044
0.5
0.4340 0.0051
0.4965 0.0060
0.5014 0.0068
Table 3: Sampling with different posterior: Comparison of posterior sampling methods under different corruption ratios. Bold values indicate best performance. We can observe that for method required accurate Hessian estimation, the performance drop sharper compared to the others.
5 Future Directions and limitation
The limitation of work lies in that for more general data type like sound or vidco, the generalization not guarantee as those data type may impose even harder tasks for the models which beyond the scope and guarantce in this work.
Acknowledgments
We thank the Central Indiana Corporate Partnership AnalytiXIN Initiative for their support.

---

References
Allen-Zhie Z. and Li, Y. Neon2: Finding local minima via first-order oracles, 2018, URL https
//arxiv.org abs//711.1..1 06673
Attia, A. and Koren, T. Uniform stability for first-order empirical risk minimization, 2022. URL http:///axxxv...//bbaa//a/a 28827 08257
Barshan, E., Brunet, M.-E., and Dziugaite, G. K. Relatif: Identifying explanttory training examples otggabbs/20..11633333
via relative influence, 2020. URL h
Bisla, D., Wang. J., and Choromanska, Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape, 2022. URL.http:://ariii.or///a..a. 2tt.//2222000005555 08025
Bottou, L.. Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning, 2018. URL
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Carmon, Y., Hinder, O., Duchi, J. C.,, and Sidford, A. "convex until proven guilty": Dimension -feee acceleration of gradient descent on non-convex functions, 2017. URL https:///arrtt htt:://aaxiv.vv.. org/ 02766
Chen, T. Fox, E. B., and Guestrin, C. Stochastic gradient hamiltonian monte carlo, 2014. URL https://arxi>> www.oos/g///eoeoooo/..ooooeeeeet 1402. 4102.
Chen, Y., Welling, M., and Smola, A. Super-samples from kernel herding, 2012. URL https
7/arxiv. htw.b////123..3..
Chen, Y., Jin, C., and Yu, B. Stability and convergence trade-off of iterative optimization algorithms,
2018. Uttpttps: //tpx/a/aivvv///a wttp://...rrr////////000/ htttpp:s//rrra..rrr/bb/////.00. 01619
Cohen, G., Afshar. S., Tapson, J., and van Schaik, A. Emnist: an extension of mnist to handwritten
letters, 2017. URLLittps://axxivv.rrr/// 1732/
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and
Zaharia, M. Selection via proxy: Efficient data selection for deep learning, 2020. URL https 11829
Criscitiello, C. and Boumal, N. An accelerated first-order method for non-convex optimization on 02252
manifolds, 2021. URLLittpp://aazxi.vorr//bb
Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 141-142, 2012.
Ding, H., Yu, H., and Wang, Z. Greedy strategy works for k-center clustering with outliers and coreset construction, 2019. UKl//tt..s//axx...oo////////0...
Duchi, C., Bartlett, P. L.,, and Wainwright, J. Randomized smoothing for stochastic optimization, 2012. https www.aaivv..o//bbbae aalpaaa///1111111 4296
Elenberg, E. R., Khanna, R., Dimakis, A. G., and Negahban, S. Restricted strong convexity implies
weak submodularity, 2017. URL https: //arxii////xvvv arxiv. hbrg/abs/6612...
Fort, S., Hu, H., and Lakshminarayanan, B. Deep ensembles: A loss landscape perspective, 2020. URL https://axxiv.. wwtp://wwwwwraaba/......
Frankle. J.. Schwab, D. J., and Morcos, A. S. Training batchnorm and only batchnorm: On the expressive power of random features in cnns, 2021. URL https://arxiv / 00152
Ghadimi, and Lan, G. Stochastic first- and zeroth-order methods for nonconvex stochastic programming, 2013, URL httpst org/ abs .5499
10

---

Ghorbani, B., Krishman, S., and Xiao, Y. An investigation into neural net optimization via hessian eigenvalue density, 2019. htL.:tpps//aaxx..ooo//////////// 10159.
Guha, S., Khanna, R., and Koyejo, O. O. Distribution preserving bayesian coresets using set con- straints. In NeurIPS 202/ Workshop on Distribution Shifts: Connecting Methods and Applications, 2021. /RRLitttps//opeeeeiien..o htt.://www....tttoooooooooooooca
Harrison, J., Metz, L.., and Sohl-Dickstein, A closer look at learned optimization: Stability, robustness, and inductive biases, 2022. URL htl.httts///axii.orrrgg abs/ 2209. 11208
Hefferon. J. Linear Algebra. Orthogonal Publishing L3C, 2017. URL http:::/tetfterooo///////// net/ linearalgebra/
Jin, C., Ge, R., Netrapalli,, P., Kakade, S. M., and Jordan, M. How to escape saddle points efficiently, 2017a. httphtts:://aree wttpk//..rr//b///005:// 00887
Jin, C., Netrapalli, P., and Jordan, M. I. Accelerated gradient descent escapes saddle points faster than gradient descent, 2017b. URL /tR. abs/ 1711 10456
Jin, C., Netrapalli, P., Ge, R.. Kakade, S. M., and Jordan, M. I. On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points, 2019. URL httpstss///a org/
abs/1902. 04811
Khanna, R., Elenberg, E., Dimakis, A. G., Negahban, S., and Ghosh, Scalable greedy feature selection via weak submodularity, 2017. URL htttpttts:///rxitt http:///xxxvv....aabba/ wwt.a/////...... 1atp://173.002233
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data subset selection for efficient deep model training. 2021a. URL https
//arxiv. org/ abs/ 00123
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning, 2021b. /tttp::////// www.aaiwii....raaaaaaaa abs/
2012. 10630
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan,
J., Ramalho, T., Grabska-Bare Barniiskk,,, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):351133556, March 2017. ISSN 1091-6490. doi: 10. 107310///p 16115 URL http://dx http//xxxiooi...o www.oooogg.iii www.oo.g///.... 10as/pnas. 1611835114
Kiros, R. Training neural networks with stochastic hessian-free optimization, 2013. URL https
//arxiv. Maaxiv.org/abs/1300 hwtrg/abs/3331.0000 3641
Kosson, A., Fan, D., and Jaggi, M. Ghost noise for regularizing deep neural networks, 2023. URL 17205
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https//::
semanticscholaa... wowp//or//r.........o httpr//accccooao.oooooo/ooooaaoa 18268744
Kunin, D., Sagastuy- J., Ganguli, S., Yamins, K., and Tanaka, H. Neural mechanies:
Symmetry and broken conservation laws in deep learning dynamics, 2021. URL httaps org/abs/2012. 204.a////2222..20222.2
Lei, Y. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems, 2023. URL htttp:::/axxiv.orggge//e/// htrrg/abs/22266 22tps/000222220222222222
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets, 2018. htattp:://arri..oog///////// 09913
Li, T., Tao, Q., Yan, W., Wu, Y., Lei, Z., Fang, K., He, M., and Huang, Revisiting random weight perturbation for efficiently improving generalization. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Liang, S., Huang, Z., Liang, M., and Yang, H. Instance enhancemmnttnnnnne batch normalization: an adaptive regulator of batch noise, 2019. URL https: //wxxixxivvv//////////a/////avl/ org/ http:/aaa/11....000 04008

---

Liao, Z., Drummond, T., Reid, I.. and Cameiio, G. Approximate fisher information matrix to characterise the training of deep neural networks, 2018. URL battps://arxiv.org/abbb.. 067677
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.,, and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: AXKv:1907.11692 2019.
Liu, Y., Mai, S., Cheng, M., Chen, X., Hsieh, C.-J,, and You, Y. Random sharpness-aware minimiza- tion. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL Utt.hhttps://oppereeiiee http://wwwwatteeeeeoooooooc/cdba
Liu, Y., Yu, S., and Lin, T. Regularizing deep neural trace, 2023. URL atttp::/aarxiv.org//bbb//// 05924
networks network with stochastic estimators of hessian
MacKay, J. C. A practical bayesian foreessaeeaaaewowooooooooooooonn 4:448-472, 1992. /ttpp//t..///plleeeaaaaaeaaaraaa ottp:///oicacitoaoooooooooo/oaaa 1 16543854,
for
networks. Neural Compulation,
Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., and Wilson, A. G. A simple baseline for bayesian uncertainty in deep learning, 2019. URL httpp://arxxxi http::://wwwww.o.....a//ooo htrrgg/bb/19990 02476
Mahmood, S. H. A., Yin, M., and Khanna, R. On the support vector effect in dons: Rethinking data selection and attribution. In SIGKDD, 2025.
Martens, J. Deep learning via hessian-free optimization. In International Conference on Machine Learning, 2010. URL /tLLAttgss/ppp...aatttcccbobbbbl http:p/p..aaaaaaaaeoooooooororrr :1tp::sss::111:11111 11154521
Mirzasoleiman, B., Bilmes, J., and Leskovec, Coresets for data-efficient training of machine learning models, 2020.
wtwp://wwwww.../a//b//.///. 01827
Mueller, M., Vlaar, T., Rolnick, D., and Hein, M. Normalization layers are all that sharpness-aware minimization needs, 2023. URL httttttss///aiiiva// wttp://www..grab/////baaa 04226
Nguyen, H., Pham, H., Reddi, J., and Póczos, On the algorithmiic stability and generalization of adaptive optimization methods, 2022. URL
Okanovic, P., Waleffe, R., Mageirakos, V., Nikolakakis, K. E., Karbasi, A., Kalogerias, D., Gürel, N. M., and Rekatsinas, T. Repeated random sampling for minimizing the time-to-accuracy of learning, 2023. URL https://arxiv.orrtt http://r/abbbb/...eee00
Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks, 2014. URL https
7/arxiv. org. abs 1//1111111..1111
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training, 2023.
http:////aaaavvoo//a//////////// atgg/abs/2007.00 07075
Pooladzandi, O., Davini, D., and Mirzasoleiman, Adaptive second order coresets for data-efficient machine learning, 2022. URL httpss://aaxiii org/ 2b0/ 13887
Ritter, H., Botev, A., and Barber, D. A scalable laplace approximation for neural networks. In Interna- tional Conference on Learning Representations, 2018a. https://:// http:::://pp..emmmniiccccccool/o www.stiiianinooi.neeeeeeee org Peg/CorpusIDD. : 52366640
Ritter, H., Botev, A., and Barber, D. A scalable laplace approximation for neural networks. In 6th international conference on learning representations, ICI.R 2018-conference track proceedings, volume 6. International Conference on Representation Learning, 2018b.
Russakovsky, O., Deng, J., Su, H.. Krause, J.. Satheesh. S., Ma. S., Huang, Z., Karpathy, A., Khosla,
A., Bernstein, M., Berg, A. C., and Fei- L. Imagenet large scale visual recognition challenge, 2015. URL http:t/ts////aatitv. owwpesiii.rrggaee wwtpaa//........ 0575
Sachdeva, N., Wu, C.-J,, and McAuley, J. Svp-ef: Selection via proxy for collaborative filtering data, 2021. URL httph::// //tp://///xivvaa/ wwwpexxiiioogg/aaa aas///aa/22100 http://aaax.vv..o/////////...../ 04984
12

---

Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. How does batch normalization help optimization?,
2019. URL battps://arxiv. org/
11604
Sen, M., Qin, A. K., G. R. K., Chen, Y.-W,, and Raman, B. Sofim: Stochastic optiniizainoiii using regularized fisher information matrix, 2024. URL httttt:///aaar// http://aaxxv..or//bbbbab/a htgg/bbb/2403.. 02833
Shin, S., Bae, H., Shin, D., Joo, W., and Moon, I.-C. Loss-curvature matching for dataset selection and condensation, 2023.
04449
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. Beyond neural scaling laws: beating power law scaling via data pruning. 2023. URL https: htaps:///arxxii //wxxva//vv////////////a//aaaav/ org/ abs/ 2206 14486
Sun, Y., Wang, X., Liu, Z., Miller, Efros, A. and Hardt, M. Test-time training with self- supervision for generalization under distribution shifts, 2020. URL https:////aax org./.ii.rrr//ooo orgg abs/ 1909. 13231
Toneva, M., Sordoni, A., des Combes, R. T., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning, 2019. URL https //arxiv. harxiv.org// org/ abaaaaa/a1 aa////aaa//1.////////////a/aa//a 05159
Tukan, M., Zhou, S., Maalouf, A., Rus, D., Braverman, V., and Feldman, D. Provable data subset
selection for efficient neural network training, 2023. URL https://arxiv.cm org/ abw////3 05151
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization, 2021. URL https:://xx:xii..orro http://v..rr//bb///////// ab0///6..220....... 10726
Wang, Z. and Mao, Y. On the generalization of models trained with sgd: cetemmation-thhorettee bounds and implications, 2022.
wtt.orgggabb////oo 03128
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Pro- ceedings of the 28th International Conference on International Conference on Machine Learning, ICML'IL, 11, pp. 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Xiao, T., Xia, T., Yang, Y., Huang, C., and Wang, X. Learning from massive noisy labeled data for image classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2691-2699, 2015. doi: 10.
Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. Understanding and improving layer normalization, 2019. URL
org/ /91100000030 07013
Yang, Y., Kang, H., and Mirzasoleiman, B. Towards sustainable learning: Coresets for data-efficient deep learning, 2023. URL htRpsttt: http://.ogg/abaaa/////aa// .1130600124)
Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. Pyhessian: Neural networks through the lens of the hessian, 2020. URL httpltttss///axiii.ttt org/ abs/ 1912.07149
Zhang, J. Y., Khanna, R., Kyrillidis, A., and Koyejo, O. Bayesian coresets: Revisiting the nonconvex optimization perspective, 2021. URL /tttpp:://axiivv wttpa//www..g///////////0.0/
Zhou, M., Liu, T., Li, Y., Lin, D., Zhou, E.. and Zhao, T. Towards understanding the importance of noise training neural networks, 2019. URL https://arxiv.ro arxia.iiii..... org.vio.rrraaaa.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided a proper justification given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: see main context and abstract
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: experiment and limitation at the end
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See proof appendix A.2 and appendix A.3 and theory section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See experiment section and algorithm and details in B
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Will provide code in camera ready version once accpeted
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (Ittps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See experiment details in appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See experiment section and details in appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See experiment section in main content and details in section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: See details in appendix B
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See experiment section in main context and detail in appendix B and related work
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, batasets.paaerssttt paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Will complete if accepted and release in camera ready version
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Appendix
A.1 Related work
Coreset methods To facilitate the training of deep learning models, various methods have been proposed to select informative representative samples based the uncertainty of models toward these samples such as Sachdeva et al. (202T) and Coleman et al. (2020). Another line of work selects samples based on their loss difference or degree of error. Methods such Forgetting Events Toneva et al. (019)) GraNd Paul et al. 2023) have employed this strategy to prioritize samples. These methods aim to reduce the variance of the gradient, thereby improving efficiency of reducing the loss.
Another line of work use Bayesian perspectíve to perform coreset selection Zhang et al. (2021); ); Guha
al. (021) which aims to select subset that equally representative across assigned posteriors.
Additionally, some methods emphasize the centrality of features or embeddings, forming subsets that best represent clusters of samples. Examples include Herding Chen et al. (2012). K-Center Greedy DDng et al. (2019), and Prototypes
et al. (2023). Another category of methods bases their selection strategy observations from a validation set Killamsetty et al. (2021b), leveraging additional information to identify the samples most beneficial for training.
Despite lacking rigorous theoretical proof for the advantages of selection strategies, these approaches have demonstrated empirical improvements in speed or performance. Gradient-based methods Mirzasoleiman et al. (2000); Killamsetty et al. (2.21 PPooadzzandi et al. (2022) on the other hand, select samples that best approximate the gradient of the entire dataset (training or validation sets). These methods offer theoretically sound support and provide guarantees for the training process. Additionally, several works Zhang et al. (2021); Pooladzandi et al. (2022); ; [hhnn et al. (2023) in this direction propose to further match the subset with the loss landscape using information in Hessian matrix and show better convergent result and generalization performance. Our method builds on this direction and addresses the shortcomings of previous work.
Smoothness and Stability The optimization of deep learning models has been active area of studied to understand the reasons behind their superior performance in practice. For instance, the Random Weight Perturbation (RWP) algorithm has been shown to smooth the objective function (Bisla et al.
2022 Li et al. 2024) and improve generalization error (Zhou et al. 2019 Jin et al. 2019: Wang & Mao 2022 despite its simplicity. By perturbing model weights, several studies have demonstrated an increased ability to escape minima with poor generalization and enhance the stability of the optimization process. Our work draws connection to this approach. Instead of explicitly optimizing using smoothed objective, we select samples that capture similar features, thereby achieving comparable effects. This novel perspective enables us to leverage the benefits of smoothness without directly modifying the optimization objective.
Bayesian Methods in Deep Learning Various parameter distributions have been proposed in deep learning to address tasks such as uncertainty estimation, out-of-distribution detection, and classification. Markov chain Monte Carlo (MCMC) methods CCen et al. (2014); Welling & Teh (2011) leverage gradient information for inference, while Laplace approximation- -aateddo--bdddddd approaches MacKay Kirkpaarkk et al. 7): Ritter et al. (2018a) employ Gaussian distributions with the Fisher information matrix or Hessian as the covariance matrix. Other methods explore different strategies: Maddox et al. (2019) average models across different time steps, and Fort et al. (2020) utilize models trained independently with different random seeds. In our work, we evaluate models derived from various posteriors to analyze the trade-offs associated with different sampling strategies.
A.2 Second order proof
Theorem A.I. Suppose a subset S' C S is fa, e, -sttbbee and let the Hessian difference be Hs _ Hs =: and model has d parameters then,
(1) The gradient at difference of subset at w is upper bounded
[Vls(w) (w)) < 2 (c + VelotoR 4 Olet)
(10)
(I) The Hessian difference matrix E satisfies:
||||| S god+ + 1 0 / /Got0 + oo( c302d) and tr(E?) < 6 6- 230
(11)
o(1 201
21

---

(2) The difference between newton step of two subset is bounded. (Amax, Amin) are largest and smallest eigenvalue Hs.w
Amax
V H5 Vls(wi]
Amin 2 (110 v /eto4P + +4c) + (c1d+
1 1
X
1700d12 + ale - -2302d) + Amax
(aodd + v G9oAd2 + (c
2
min
2
v 17 o(c - (302d) 0(1811)
Proof. Let f(w) (l) Vis(w) and V 1s ( Vllg(w) (difference in terms of Hessian)
Assumption Suppose have the ( being 2c1 -ipshiice Hessian i.e., V (w) - ( - 2cq || - T/
(12)
Assumption Suppose we have the V f being bounded below and above
20a I V((w),
Vu.
(13)
Assumption 3 (Symmetric and non-singular) V f(w) is symmetric and non-singular.
With assumption and assumption 2, we can bound the f(w) by the following through Hefferon (017):
callw wll S (w) f( (w will s cillw क| Vu,,
(14)
The assumption is aiming to capture the degree of change of function using polynomial terms. For more discuss about the assumption of the theory, please refer to appendix D
we start with the following
2
f(w)
(((() f(w)-
- f(w)))
(15)
= [(w)² + 25( / (f(w)) f()))p(u)dd + [F(wm) - (())))p(wu)dd
- 2 - aoods( ( (f(vc) f(a)))p(w)dd
For last inequality, we use the following:
f(w) /U(0)) - ((w)))( = f(w) / (f(w) f(0) Vf()) 1) Vf(w)(w - w))p(w)dw
/ ((f(w) f()) Vf(ū) ((())=-))p(wddd
())p()) + f(() / Vf(W)(w w))p(w)dd
f())
((f(w) - f( Vf()) ())p())du
f(w)
< |||((0||| / [I([(w)) f(w) vs(w) w)llpwwdd
< || cill - w| (p wd)
cills())] :||
I c Epl[w m|
= clll
(16)
22

---

We first focus on the first two terms
< -
(17)
By solving the equation, obtain upper bound on the [f(w) as following:
ll(mw)l - 2 Vegodd + 4e) O(e))
(18)
as check, if we use linear approximation (i.e., Cj = we will have
[f(@)] S Ve
(19)
Now, obtain upper bound on the graident differece at w, we continue on the cross terms
[(((w) ((v)))((((u p(w)dd
(U((e) f(w) V/(W)(W + V/(()(( (o)
= ((f(w) f()) Vf(m)(w p())p(w)d
+2 / (F(w) f(w) f x) ()) w)p(w)dw-
//( (w- w)p(w)dw
(20)
continue by noticing that first term is lower bounded in assumption.
< ego'd
+2 (f(w) f(w) - V/ ())(w)dw [(u. (w-
(V-WWVV(()(((m -
(o)p(w)du
(21)
The first term is obtained through following:
J [1f(w) ((0) - v P())dd > callw wll2((wdde
(22)
= =2pppllw - -
= -20
Now address the cross term.
2 1// ([((() (a))) 1 ()((w)d 2 221 / [|w wl|2[(w w)lp(w)du
2 22 max do,5 / (ww- w)²
2 -20 2 AE,
(23)
Now, we address the last term
[(u- ww w)2n det(oT) - exp(- (w (w- (o) - T) dw
(24)
23

---

By change of variable a - (w t) we can obtain the following by assuming that & is not degenerate:
/Idet(e ()(""E
u2 "u det(o!) exp(- 1 fue& (o1) ( uuuu
2
|det(E-))) d))(E()))))) tr(E(a1))))
det((o!))
(25)
tr(alE?)
2 o tr(E²)
max 22.5
We assume the HT1 and Vf(w) are symmetry matrix. Here, utilize identities for the first. equility and second inequility:
tr(ABC)= tr(BCA)
(26)
The second identity follows the following derivation by first noting that symmetry matrix can be decompose into the spectral form = Ei AA.u
tr tr(A Σ (uf
2 min AB, Σ (vz Aws)
(27)
min An Lr(A)
Now, we put everything together,
2 ego 20102= max Ae, + a max X2
(28)
we can solve for the largest difference in eigenvalue
clod + = Goolde +o(E-czoldd 2 max λg,
(29)
Therefore, have that
H a Hs < O(et)
(30)
We can also obtain upper bound on the difference in Hessian terms of trace
E 200d2 -2c 02 max A + a max X
2c -2 tr trte a(1 20 2c tr²)
(31)
and therefore
c 0 2c
2 tr(
(32)
Now, prove the newton step is also similar, write the overall into the following equation
Hy V.((w) H5 (w)(
||(Hs, E) (Vls(((( + +
H51 v(s(w)
(33)
24

---

To obtain the upper bound of the Hessian inverse, we use inverse perturbation theory
(Hs + 8)- H 5 5,0 + +(1E11
(34)
Now, we obtain upper bound on the magnitude of eigenvalue of the difference on the Hessian for the subset.(i.e., max, / & cood + Vi + a(E - a(e-c)pdd) and the gradient difference [V]) - (w)]l S 3(c + Vejo'd' + 4e). We can follow to upper bound the desired quantity | Vlg (w) Hs (lw(w) At this point, we assume that the gradient has bounded magnitude [Vs(w)] I c, vS,w w
Put in the above, and assume that the gradient has bounded magnitude VVs(w)]l S VS, | V.l(w) 3 (w))l
-I. E) - + &2) (1)
< |1H5 2 A. Vls(w) +
< | + H 5, 5 Vls(m)] + H E 3.0 22 +
lao'd eoo + +c) +
Amax
codd 1
v / + o(e - c2g2d))
2l1ood V + +4)((acc
+ 1
Amin
min
- V Viooe oo d + 0 {
s ((E))
(35)
At this point, we can conclude that
H St (.
H Vl, S (c))
(36)
A.3 Proof for Theorem 3K
Next, the assumptions required for the proof are listed below: Assumption 1. Bounded variance and unbiased estimator of the random sampling and coreset selection subrutine:
Vls(w))(w) - v((w))
(37)
E[V]s(w)] Vi(w)
(38)
Assumption 2. a-lipschitz continuity:
I(w) - ((v) s allw 2112
(39)
Assumption 3. 8-smoothness:
[[V((w) - vi(O)|I S B|| 0||2
(40)
Assumption 4. 6 Rdd sampled i.d from diagonal gaussian:
E; ~ (O,
Vi 1...M
(41)
There exist two randomnesses in our formulation. One is the subset obtained through random sampling from the whole dataset. The other randomness results from the noise inject to model weight.
((w++1) < ((w,) + Vl(w:) (w - wi) - 2
3 Elutt11 weel
1
M
3r2
1
M
(42)
S ((w+) - n V((w.))
Val(w, + E;) +
ssl(w, + +
M
2
M
25

---

We rewrite the last term as follows:
M
2 M
sl((ur + +
2 M Σ
(11
sl + +
v((w;)]1 +
Σ
V s + vi)[((w)) [V((w))]]3)
2
M
M
i=1
(43)
Input the above into the original formulation:
M
B1Z
1(++1)= S ((wr)= - - V((w,)
V V We + te)
sl + et)l
M
M
3nk
M
< ((wi)) - (1 Tx (w)))
ssl(w: + +()
+
(1)
V Vsl(w; v(())) ([[[])))))][(((xx)))))) [[V((w))]1
M
2
M
i=1
S (Wi) - (1 ((w))
V s +.e)
Vl [[(
M
M
B
Σ
Vsllu, +
Vi(w; + + 3mh
Vi(w: + Ex) v((2))11
(44)
The last two terms are achieved through the identity ||a - bll - 2(11a cl + cl)) We now want resolve the second term in the formulation.
M
M
vi(w))
V sllw + c2) = v(ww) ((
ssllw, + t2) v((()) +
M
M
M
1
=|VI(w))] + (
V V + v((w:)
M
-IIV((u>)]2++
vi(w,)) 1 M
Σ
(vs(ww +
Vilw + M Σ
V((u; + +;) v((w)))
M
(45)
The term can be simplified by taking the expectation over the randomness in stochastic subset.
M
M
1
Es \w:) T
V sw cx) = ||VI(w))) + V
Vl(wi + - vi(w:))
M
M
< [|VI(w))] Vl(w)))
Vl(w + + v((w:))11)
2
(46)
The last term is achieved through the Cauchy-Suthit Schwarz inequality. Therefore,
M
M Σ
E, (w)) Σ
T
1
Vsl(wi + 4)) < |V((2))) + 2 M
Vi(wi + +4) Vl(w:))
M
(47)
We further simplified the formulation by taking the expectation over the randomness in noise injected to the model weight
1
M
v slw( + +e)) - 1
EEs V((w)))
T
(48)
M
26

---

Next, we first solve the fifth term in the original formulation taking the expectation with respect to for each
M
M M Σ Besll²)
B1g
Vl(w, + 6s) - vI(w))]1
E1
M
I 3 1/2 1 M
(49)
M 1=1
S 8 n/ o.d
Finally, we deal with the term.
B
M Σ
sdlww + 6.) Vl(w, + ex)l
(50)
Here, assume two different errors that arise in practice. The first is the random sampling batch created by sampling from the entire dataset. The second error originates from the coreset approximation. We formulate follows:
Vi(w)= V slw(w) + +
(51)
The Éι is the result of the stochasticity of the random batch generation. The &2 is the error that originates from the selection of the coreset. We consider two different forms of error in &2. One is the absolute error and the other is that the error is propotional to the gradient. We formulate as follows:
Flllelll S e or F11 S <|[V[(w)]l, 6>
(52)
Note: we assume here that these two errors ć1, &2 are independent to each other. Situation We first consider the case where the error is absolute.
M
M
Bnz Es. M
El|
Vsl(w: + (4) -V((wr + c:)1 < Br
Σ
El
V + +G)
Vl(w, + c;)l
M
1=1
1
Bnh M
E11 22,,522 + [1(2,11
1=1
Bri M
(1LE:
221E2 + +1(
oF +
< B1 2
M R
(53)
The R is the batch size for each batch of random sampling. The cross terms are eliminated due the assumption 5.
Integrate those terms into the original formulation.
1(we1)) S ((w))) - 7/s /V((w))) + 1e 11.3)
Buhot MR
Buie² + 2
g
+
(54)
2
Here, pick 7). (fixed step size) and n have follows:
B. Rearrange and sum time step, and will
B0 T-1
Be2 T-1
3oJ T-1
Vl( S ((wo) 1(w") +
71 + 11)) +
Σ 2
Σ n
(55)
2
M
MR
1=0
27

---

Here, we divide on both sides by TT)
B'o T-1
Bot MR
[[w))l1 2 (1(wa) [-ww(ww)) +
(1
1/8
M 11
27
t=
1=
(56)
(I(wn) l(www)) + B Bo3d
o'
3
3
11
Tn
M
MR
As we have control for both y) and akd, we pick oRd
and 1 min vF. 3
1
and, therefore,
MvT
V(( )1 & 1 (((w)) l(w")) max VT
1
3
3e²
Bot
1 g
+
+
VT
2M
M
MR
TM
(57)
If we stop at any specific time step with probability 4. and we observe that the average gradient exist convergent rate V for I large enough which is:
g2
Be
Bot
1
133
E Vll S VT (1(wb) - [(www))
VT
2M
M
MR
T2
(58)
8
Be
Boi
(1(1m) -[(ww +
2M
M
MR
Situation 2 We consider the case where the error propotional to the magnitude of the gradient. i.e,
El <|
<>
(59)
We analyze the term as follows:
M V sl(w, + +&) Vllw, + +]]l < Bाे
1
BnE ElI
Ellk + 25 + 1|6211
M
M
(60)
2 1 of
Bm
+
M
R
Integrate the term previous result.
Bnffe
me(1 1113)
Bnfa?
1(w++1) S I(wx) - (m
)[]
3 30o
+ Noodd (61)
M
2
MR
Set 11 = We rearrange and perform same operation and we get:
T-1 1
31p²²
Fozd T-1
Bot T-1
|||
[[w;)) < ((wa) - 1(ww[lww=>)) +
Σ
m11 + n3) +
Σ n?
M
2
MR
1=1
t=
(62)
8oZd
3 T'p MR
((wo) - ((1wwww)))))))))>))) +
Tn(1 + n18) +
2
Divide by T(n
3 M
on both sides and choose step size such that (1
gp
2
M
T
2 (((wo) 1(ww)) + Po + n13) +
2,30
[VV(())))
(63)
T
MR
28

---

We pick oRd
and
JMR and will have vT
VMIRT
1
Σ
[V
)1 7 T (1(w))) 1(w²))
VMRT B VMR
8
2009
T
VT
VMRT
=
(64)
0 (2(1(w0) - ((1wwww)))) )) + B² + 2
VMRT
Similar to the first situation, we will have convergence rate with VMRT naive SGD.
which is VM
faster than the
29

---

A.4 Data summary
Dataset
Corrupt Ratio
Random
Crest
Glister
Craig
Ours
0.9921=0.0008
0.9892=0.0005
0.9997#0.0001
(*)0.0972+0.0
0.9914=0.0003
0.1
0.98540000009
0.8384200812
0.767640.0271
0.0961#0.0009
0.9876=0.00009
MNIST
0.3
0,9772=000008
0.337420.3385
0.1815#000664
0.519620.2344
0,950900.0025
0.5
0.962#100019
0.0725=0.0109 0.5277=0:2062
0.426420.0131
0.971520.022
0
0.8713=0.00220
0.7955=0.0196
0.8246=0.0044
0000020900000
0.8715=0.000
0.1
0,864920000110
0.7788=0.0132
0.6762=0.0094
0.02T8=000007
0,867920.001
EMNIST
0.3
0.8557=0.0007
0.0004
0.4748=0.0236
0.301
0.8396=0.000 0016
0.5
0.840920.0024
0.7146:0.0051
0.2627=0.0172
0.321:0.0111
0.8100=0.0029
0 0.0600
0.8724400004
0.0007=0.0097
0.761840.008
0.8757
0.1
0.8481#0.00 #0.001
0.844000.003
0.6330=0.025)
0.74900000066
0.0344=0.0034
CIFAR-I0
0.3
0.8063=00010
0.7843#0.004
0.3953=000548
0.7030=0:0112
0.0520
0.5
0.7257+0.015
0.6652#0.013
0.2175=0.0292
0.6320=0.0186
0.731820.014.
0
0.665920.0041
0.6728=0.003
0.600=000005
0.5665=0.0053
0.698620:0025
0.1
0.6268#0.005
0.6391
0.4538=0.0076
0,4629=0.0102
0.664420.0034
CIFAR-I00
0.3
0.5119=0:014
0.5712=0.006
0.2846=000074
0.2968=0.0162
0.6085=0.0044
0.5
0.329320.014
0.4305#0.026
0.1578=0.0218
0.160320.0048
0.501420.0068
Table 4: Performance comparison of different methods across datasets and corruption ratios. Results are reported as mean standard deviation. Each experiments are average over different random seed. The best performance for each setting is highlighted in bold. The * mark the situation in which has diverging behavior during the optimization. The situation usually occurs in Craig method for high coruption scenario.
A.5 Discussion about different posterior
Hessian inverse covariance is the one containing exact information about loss landscape of the models at certain training points and it is also the posterior used in deriving the theory for sampling. However, calculation of Hessian can introduce large memory footprint as will require us to keep track of the Hessian during training large memory footprint, it is also hard to calculate and require steps of approximation. The selection strategy involving calculation of Hessian will inevitably be slowed down it requires at least one forward, backward propogation and the intermediate calculation. Another issue about Hessian inverse is that despite it contains the curvetures information, it easily fail to reflect the true loss landscape from sampling viewpoint(see 5) For convex optimization view point, Hessian indeed captures the global information about the loss landscape, but for non-convex region, it only express the local curvature information and sampling according to the local information can easily lead to sampling of high loss region.
listed in works Yang al. (2023). Not only it cause
Direct training with different random seeds. The posterior consist of models trained with different random seeds is studied in Fort et al. (2020) and shown be simple and strong base line for posterior for uncertainty measurement. The prediction models trained with different random seeds provides strong diversity compared to the sampling methods which explore only the local region. In practice, is easy to implement and applied to various to different leaning scenarios. The shortcoming of the method is that it requires much larger memory than other method as it stores and trains multiple models at the same time. Additionally, it did not necessarily violate the above observation as the models involved in the posteriors are trained simultanously and have low loss guarantee.
Diagonal Gaussian Diagonal Gaussian posteriors is well studied and shown to offer generalization guarantee. It provides easy control for the region to explore. Despite the fact that it did not contain local curvetures information, it can still fit in the observation from our theory as long as we select proper range for the variance. Compared to the two previously mentioned method, it offers better speed and memory consumption as we only need to sample independent variables for the construction of the whole distribution. In addition to the advantages mentioned, there is subtle connection between this posterior and optimization and we will illustrate in the following.
A.6 Details about Greedy selection
Greedy selection method has been studied in many different prior works Khanna et al. (2017); Fleebbrg et al. (2017) to set up basis for its correctness and its applicability for different functions. As pointed out in Mirzasoleiman et al. (2020). one can transform the coreset problem on gradient
30

---

Speed up
CIFARIO
CIFARIO0
Tinylmagenet 1.328621908
Crest
0,46122396
1.220357534
Our
1.71376899
1.227372798
1.448220165
Table 5: Our method obtain better speed up compare to the benchmark method and the results also generalized to different datasets and architectures. The speed up is calculated as (Full training time / Method training time). The results average over 5 different random seeds.
0.-
05
6.3
62-
-20 -30 -2.0 -0.3
0. 3.0
350
Figure 5: (Left) We calculate the Hessian for sampling at red dot and sampling (yellow dot) using the Hessian inverse covariance matrix. The motivation for performing sampling of this kind is that one expect model to be sample lying on the low loss region as it put larger probability density in the small eigenvalue direction. However, when the Hessian loss its ability represent the true loss curvature (non-convex setting), the Hessian posterior can sample models in regions high loss, even when the Hessian is computed correctly. (Right) For ensemble method, we find its performance is competitive to Gaussian posterior. However, we find that there exist performance divergence in the different models trained with different random seed and the coreset selected show different performance gain for different models. The coreset selected through this posterior may not be able offer best performance gain in the ensemble.
into the following submodular cover problem with constant C::
S argmin |S||
Σ JES
min [l((w) vlf(we)ll Se
(65)
S'CS
The y in origin problem will be calculated as the number of times a specific sample j E S' used to achieve minimum distance in the argument right hand side in this transformed problem. Greedy algorithm is used to calculate the sample being selected in which achieve time complexity (nk)
in existing work Mirzasoleiman et al.) (2020); Killamsetty al. (2021a); Pooladzandi et al. (2022). where n is the size the training set and k is the number of samples being selected.
Despite linear complexity in terms of the sample selected, the calculation of the difference norm of the right hand side is still expansive due to the high dimensional properties of deep learning models.
Several works Killamsetty et al. (2021bl b); Mirzasoleiman et al. (2020); [Pooladzandi et al. (2022) demonstrate experimentally and theoretically that one can use the gradient with respect to the last layer for the calculation of gradient difference as it captures the norm of difference properly and greatly speed up the process for practical application, though a recent work has argued against Lastly, we complete the formulation with the formulation with fix sample size selected & as following:
S = argmax C S'CS
Σ
argmin (w:) (1,(w;) ES
JES
(66)
S S
A.7 Details about the algorithm design
In this section, we will briefly discuss the design of the our algorithm. First of all, instead of selecting the entire training set like Mirzasoleiman et al. (2020), we adapt from Yang et al. (023) to select
31

---

14
14
10
0.75
0.75
0.50
0.50
-1.0%.77 20.00.00000000
-1.0%.75..50 2.............. 2,000,2
8.00 0.55 1.00
-0
0.00 0.79 1.00
-0.25
Direction
-0.75
Direction 1
-0.75
-1.00
-1.00
Figure 6: The loss landscape generated by coreset of Craig method and the coreset of our method. The graph is generated with CIFARIO data and ResNet20 using 1 %% data budget.
from mini-batch and union the mini-batch to obtain coreset with specified size. This help to reduce the selection time as mentioned in the Yang al. (2023). For the selection, we calculate the expected version as as66 to ensure the properties obtained check listed in the Yang et al. (2023), we instead perform update on each epoch.
theory remain true. We did not perform threshold
32

---

A.8 Toy model: Trajectory difference
OpLimiationn
0.
Figure 7: The trajectory difference between gradient descent, Crest and gradient descent. To mimic the gradient mismatch resulting from the label noise, we inject high frequency function into the Crest and our method. To mimic the smoothed version of the loss landscape, the gradient in our method. We can find that the injected noise can drastically change the trajectory of different methods while the smoothed version can help better recover the ground truth trajectory. The fluctuation from noise becomes more significant around minima as the gradient around minima becomes smaller under this simulated experiment.
reduce the magnitude of
33

---

A.9 More learning scenarios. (Different training budget, corruption level, and architecture.)
In this section, we perform experiments on even more learning condition such as different corruption level, training budget, and architecture. For architecture, We extend our result to Vision transformer identify whether or not attention based models can work properly with our method and the results show that our method already outperform others under various structures, For other learning setting. we also find our method consistently outperform other benchmark which further justify the robustness of our method.
Budget
Corrupt Ratio
Our Method
Random
Crest
0.2
0.0
0.8969 = 0.0026
0.8968 0.0026
0.8083 0.0125
0.1
0.8795 = 0.0026
0.8788 0.0032
0.8331 0.0454
0.8495 = 0.0053
0.8483 0.0027
0.7632 0.0396
0.5
0.8022 0.0020
0.8011 0.0032
0.5308 0.0103
0.01
0.0
0.6683 0.0044
0.6573+0.0211
0.4370 0.0184
0.1
0.6216 0.0229
0.6147 0.0272
0.3631 0.0496
0.3
0.5667 0.0037
0.5431 0.0272
0.3022 0.0234
0,5
0.4801 + 0,0150
0.4412 0.0172
0.1988 0.0231
Table CIFAR-IO. Performance comparison of different methods across varying corruption ratios and budget levels. The best-performing method for each setting is highlighted in bold.
Budget
Corrupt Ratio
Our Method
Random
Crest
0.2
0.0
0.7337 0.0013
0.7291 0.0010
0.7172 0.0029
0.1
0.6712 0.0022
0.6566 0.0045
0.6357 0.0033
0.3
0.5521 = 0.0021
0.5246 0.0061
0.5081 0.0064
0,5
0.4535 # 0.0014
0.3884 0.0059
0.3500 0.0094
0.01
0,0
0.2660 # 0.0079
0.2521 0.0039
0.1672 = 0.0126
0.1
0.2396 = 0.0029
0,2231
0.0033
0. 0.14669 0.0039
0.3
0.1773 0.0050
0. 1666 0.0025
0.1 0.0038
0.1235 0.0026
0.1134 0.0051
0.0823 0.0021
Table 7: CIFAR-100. Performance comparison of different methods with varying corruption ratios
and budget levels. The best- best-performing method for each setting is highlighted bold.
Corrupt Ratio
0.0
0.1
Our Method
0.8294 + 0.0011
0.8077 # 0.0008
0.7734 + 0.0009
0.7064 0.0026
Crest
0.8274 + 0.0038
0.8002
0.0042
0.7467 0.0137
0.6653 0.0112
Random
0.8200 + 0.0034
0.7980 0.0078
0.7479 0.0040
0.6806 0.0030
Table 8: Performance comparison of different selection methods using pretrained ViT-Base on
CIFAR-100. Our method consistently outperforms both Crest and random sampling. We train with learning rate of 0.0003, weight decay of 0. and warm-up scheduling for 20 epochs. The full training process consists of 100 epochs to fit within the rebuttal time constraints.
34

---

Method
0.0
0.1
0.3
0.5
Ours
0.8773=000046
0.8893=0.0172
0.8787=0.0061
0.8767:t0.0022
Crest
0.8707=0.0163 0.8727=0.0110
0.8593=0.0103 0.7447+1.0042
0.8727+#00055
Random
0.7520=10.0040
0.7373+0.0186 0.7400
Table additional experiment on the TREC-50 language dataset using ELECTRA-Small- discriminator, a compact transformer model with approximately 14 million parameters. Alongside SNLI (used the main paper), this dataset represents a language inference-style task. We finetune the pretrained model for 50 epochs under 10 percent data budget, using Adam W with learning rate of le-4, weight decay of 0.01, and standard default settings. We also apply learning rate warm-up for epoch to stabilize training.
Corruption Ratio
RBFNN et al. 2023)
Craig
CREST
Ours
0.0
0.64491000038
0.566510.0053
0.672810.0030
0.698610.0025
0.1
0.5043=000013
0.46291100102
0.6391+10.0040
0.6644110.0034
0.3
0.3422=#00140
0.2968+0.0162
0.5712+0.0060
0.6085+000044
0.5
0.2019
0.4305=1000260 0.1603+100048
0.501410.0068
Table 10: CIFAR-I00 100 test accuracy (mean + std) under different corruption ratios. We compare our method with sensitivity based method (RBFNN Tukan et al., 2023) on CIFAR100 dataset under different corruption level.
Corruption Ratio
RBENN Tukan et al. 2023)
Craig
CREST
Ours
0.0
0.8255=0.0040
0.7618+0.0080
0.8724+0.0040
0.87574000009
0.1
0.7924=0.0113
0.7490+0.0006
0.8440=10.030
0.85441000034
0.3
0.7280-000118
0.705010.0112
0.7843-10.0040
0.812011000588
0.5
0.6216=0.0041
0.6320+100018
0.6652-1..0130
0.7318
Table 11: CIFAR- 10 test accuracy (mean = std) under different corruption ratios. We compare our method with sensitivity based method (RBFNN(Tukan al., 2023)) on CIFAR10 dataset under different corruption level.
35

---

B Experiment details
B.1
Code base
We develop method base code provided in Crest Yang al. 2023). For Craig and Glis- ter, we use code based from CORD (https://githun cww/deciili team team/cords cords) with training hyperparameter changed to setting.
B.2
Datasets and architectures
In our work, conduct experiments on various image datasets. MNIST (Deng 2012). EMNIST
(Cohen et al. 2017). CIFAR I0, CIFARI00 (Krizhevsky 2009), and Tinyimagenet Russakovsky et al. 2015), SNLI (Bowman et al. 2015) and Imagenet- Ik dataset. For MNIST and EMNIST datasets, use Lenet. For CIFARIO, CIFARTOO, Tinyimagenet and Imagenet- 1k, we use respectively ResNet20, R and ResNet50. For SNLI,
pretrain RoBERTa (Liu et al. (2019) model. To creat data corruption, we pick specified portion of training samples and tlip the corresponding label to other classes ensure the corrupt ratio is rigorous. For all experiments except for Tinyimagenet, on single A10 GPU. For Tinyimagenet, we use single NVIDIA A100 GPU.
B.3 Training hyperparameter
For all experiments, we fix the peak learning rate at 0.1 and total training epoch to 200. The batch size set to 128. For the first 20 epochs, we use linear warm up until learning rate reach 0.1 and decrease the learning rate by factor 0.1 at 120 epoch and 170 epoch. These hyperparameters were consistent with those in Yang al. 2023) and were chosen ensure fair comparisons across methods and datasets.
B.4 Experiment details about loss landscape and its matching
We generate the loss landscape plot using technics in Li al. 2018 We purturb the model weights using
f(a, = 1(w" ad + B3))
(67)
In which the 5 and y randomly initialized vectors with magnitude scaled models parameters. The plots are generated using 20 by 20 grid and for each grid we calculate the loss on the whole training dataset, Craig subset, and our subset with the same parameter. In the plot, our method to the Craig method and use Gaussian noise 0.01. We select all at once as Craig does to verify that the method will bring smoothness the loss surface and we observe that there exist more sharp corner for the loss surface created by Craig and the loss surface generated by our method is smoother than Craig method. The loss for Craig and method scaled loss using the 7 constant obtain through the greedy selection subroutine. Both plot are generated using 1 1% data budget for each selection methods. The 3D plot 6
incorporate
B.5 Evaluation
We evaluated different methods various datasets under different corruption ratios, with fixed data budget of 10%. We recorded the final test accuracy and measured time as the process wall time (i.e., from the start to the end of the process). To ensure reproducibility. all experiments were conducted using 5 different random seeds, and results were averaged across these runs.
C Time complexity analysis
For Craig, there exist the need for forward propogation for each sample, and the corresponding time complexity is (dn.). (d:model parameter, n:size of the dataset) The method to select fraction (0 < y < 1) of the whole dataset with size all at once. The greedy selection strategy for selecting yn samples out n samples (gn2). Therefore, the complete time complexity is Ogn2 dn) for each epoch.
36

---

For Crest, instead of selecting from whole dataset, they select from subset with size M. and the corresponding time complexity for forward propogation (ddR). (d:model parameter, nısize of the dataset) The method is to select m point from the subset. The greedy selection strategy for selecting Rm samples out n samples is (R Therefore, the time complexity is (( + -dR) where the P is the number of subset that is manually chosen. The overall time complexity for Crest method need to multiple by the times they update the coreset within one epoch for fair comparison, but the average number of update for coreset cannot be calculated they utilize adaptive strategy which depend on the curveture of the loss landscape.
For our method, we also select from subset with size R but we need multiple forward propogation (M times) and the corresponding time complexity is O (MdR). We then select m points from the subset and the corresponding time complexity is (m We need to perform multiple time to have q portion of the whole dataset. The overall time complexity is ( (MdR + m.)) for one epoch.
For CIFAR-10 dataset, for each epoch, the running time for forward pass to obtain the gradient is 0.136 second for each batch, and the running time for the greedy selection is 0.000612 second for each batch. Hence, a much larger time is spent on the forward pass instead of the greedy selection part of the algorithm. The current state of the art method CREST requires expensive tracking of the Hessian which results in significantly longer training time and memory footprint. As result, our method remains competitive in terms of the total training time while offering improved selection quality.
D Assumptions in the theory
Theorem 4.2 relies on assumptions of third-order smoothness and Hessian symmetry of the loss function
The assumptions of third-order smoothness and Hessian symmetry are commonly used in deep learning theory to facilitate theoretical analysis. One key observation in deep learning is that the loss landscape often exhibits a degree of continuity, meaning that small changes in the parameter space generally lead to gradual changes in the loss function. This aligns with empirical findings on neural network optimization, where sharp transitions in loss are rare under typical training conditions.
The symmetry of the Hessian follows naturally from the continuity and differentiability of the loss function. There are several important results derived using these assumptions. [A. B, C, D. E, F
2013 Ghorbani et al.) 2019. Kunin et al. 2021 Barshan et al., 2020 Yao ct al. 2020 Bottou et al.
G[(Mateess]] 2010 Kiros
2018) While non-linear activation functions introduce complexities, prior
works suggest that, in practice, the loss function remains smooth enough for such assumptions to be reasonablc.IC. H] GGhorbani et al. 2019 Liu et al. 2023). Additionally, there are lines of research trying to approximate the Hessian using Fisher Information matrix (FIM) such (Pascanu & Bengio 2014 Liao et af., 2018 Sen et al. 2024). This implicitly assume that the Hessian is symmetric as FIM symmetric according its definition. Also, there are works(Kinkpatriie et al. 2017 Ritter
et al. 2018b) using Hessain as a precision matrix in probabilistic models, which implicitly assume symmetry in its structure and receive success in capturing improving the behavior of deep learning.
Similarly, the third-order smoothness assumption extends this notion by ensuring that second-order derivatives do not change abruptly, which aligns with empirical observations about the optimization dynamics of deep networks. These smoothness and regularity conditions are standard in optimization
( et al. 2017a Allen-Zhu & Li 2018 Carmon et al. 2017; Criscitiello & Boumal 1 Jin et al., 2017b] Bottou et al. (018)) and are widely used to analyze generalization and convergence properties of deep learning models.
Thus, while these assumptions may not hold universally in all settings (and we are not aware of any assumptions that hold universally for all models), they are reasonable approximations that enable theoretical insights into the learning dynamies of deep neural networks. We hope the reviewer agrees that our results are novel and useful within the context of current understanding of deep neural networks.
37

---

E
Adacore comparison
AdaCore is related to our work in terms of its motivation capture loss curvature. How- ever, we were unable to include it in our experiments due to the lack of an accessible or func- tional implementation. We explored multiple sources, including the official AdaCore repository Addood//AaCore.git,, which has always been empty ever since it was created, as and DeepCore
well as public coreset libraries such as CORD We did not find a working implementation of AdaCore
wtatttubcooo/Parick////// any these repositories.
We also attempted to reimplement the method based on the paper. but were unable to reproduce the reported performance. The method requires several manually tuned hyperparameters and includes steps involving Hessian computation, which are both computationally intensive and memory demand- ing. This introduces a significant runtime and scalability barrier, particularly problematic for the large-scale or noisy settings we focus on, and undermines the motivation for using coresets to speed up training.
Additionally, we note that AdaCore has not been included in recent coreset benchmarks, such as those by (Yang al. 2023 Okanovic et al. 2023) which includes the authors of Adacore themselves, where efficiency and scalability are prioritized. We believe this omission reflects a broader consensus that AdaCore, while conceptually interesting, is not competitive in practice under modern resource constraints.
38


Reviews:
[{"review_id": "wd35onfN9w", "reviewer": "Reviewer_Pvnw", "summary": "This paper proposes a new coreset selection method: using posterior smoothing for coreset selection.  The idea is to sample model weight perturbations from a Gaussian posterior centered at the current parameters, and use these perturbed weights to estimate the landscape and evaluate the coreset selection criteria.  The proposed approach essentially smooths out the loss landscape by averaging weights in a local neighborhood. The approach is gradient-based, and doesn’t need to compute the Hessians. \n\nThis paper provides convergence analysis and empirical validation for the proposed approach.", "questions": "Have any experiments been performed on transformer based models?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is generally well written. Coreset selection is an interesting and important topic when training with a large amount of data. It is also useful for continual learning. The proposed posterior sampling method is intuitive and solid. The paper also provides convergence analysis for the proposed approach.\nExperiments were conducted on various datasets and model architectures to validate the approach.  The model architectures used include various sizes of ResNet and LeNet.\n\nThe paper results would be stronger if it has results on transformer based model architectures.", "was_revised": true, "final_justification": "This paper proposes a new coreset selection method. The proposed posterior sampling method is intuitive and solid. My questions were adequately answered in the rebuttal. Feedback from other reviewers is also generally positive. I would like to maintain my rating for acceptance.", "rebuttal": "We appreciate Reviewer for the comment and summarize the quesions and responses in the following:\n\n1. Have any experiments been performed on transformer based models?\n\nYes. As shown in Appendix A.9, Table 8, we conduct experiments using ViT-Base on the CIFAR-100 dataset, and our method consistently outperforms other baselines across different corruption ratios. Additionally, we evaluate our method on transformer-based language models, specifically RoBERTa (a variant of BERT utilizing attention-based mechanisms), on the SNLI natural language inference dataset. (in the main context) In both the vision and language domains, our approach demonstrates improved performance over standard baselines, suggesting its general applicability to transformer architectures."}, {"review_id": "xqMgfaTLOx", "reviewer": "Reviewer_qG1y", "summary": "This paper proposes a new framework for coreset selection. A method for choosing representative data subsets to speed up deep learning training. The authors address challenges in existing gradient-based methods, such as strong SGD baselines and issues with representativeness due to changing loss curvature. Their approach builds a connection between posterior sampling and the loss landscape, and introduces a smoothed loss function to improve stability and generalization. They also provide a new convergence analysis and show through experiments that their method outperforms existing techniques in both training speed and generalization.", "questions": "1. Can you provide results on one more language or multimodel dataset on small models? It would make the result more concrete.\n2. Can you theoretically compute time complexity of crest, craig and your method? (I have seen your method's complexity on appendix C but also want to see the difference and comparison of others.) If possible, please provide system runtime for training as well.", "limitations": "1. This work has not been tested on larger models and datasets, which weakens its generality.\n\n2. The paper does not report actual training time and only provides theoretical and data usage analysis. I would encourage the authors to include a more comprehensive evaluation of system performance.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. Amazing efficiency/accuracy improvement on training.\n\n2. Very comprehensive ablation study about corruption ratio and accuracy.\n\nWeakness:\n\n1. Does not prove effectiveness on scalability especially on multimodality and language tasks.\n\n2. Can not prove it is efficienct(in throughputs/latency) in real system.", "was_revised": true, "final_justification": "I would like to keep the original positive score since the authors solved most of my concerns.", "rebuttal": "We appricate the reviewer comment and address them in the following:\n\n1. Can you provide results on one more language or multimodel dataset on small models? It would make the result more concrete.\n\nWe thank the reviewer for the suggestion. To strengthen our claims, we conducted an additional experiment on the TREC-50 language dataset using ELECTRA-small-discriminator, a compact transformer model with approximately 14 million parameters. Alongside SNLI (used in the main paper), this dataset represents a language inference-style task. We finetune the pretrained model for 50 epochs under a 10 percent data budget, using AdamW with a learning rate of 1e-4, weight decay of 0.01, and standard default settings. We also apply learning rate warm-up for 1 epoch to stabilize training. The results below show that our method outperforms both Crest and random selection under various corruption level:\n\n| Method  | 0.0           | 0.1           | 0.3           | 0.5           |\n|---------|---------------|---------------|---------------|---------------|\n| Ours    | 0.8773±0.0046 | 0.8893±0.0172 | 0.8787±0.0061 | 0.8767±0.0042 |\n| Crest   | 0.8707±0.0163 | 0.8727±0.0110 | 0.8593±0.0103 | 0.8727±0.0253 |\n| Random  | 0.7520±0.0040 | 0.7447±0.0142 | 0.7373±0.0186 | 0.7400±0.0243 |\n\n2. Can you theoretically compute time complexity of Crest, Craig and your method? (I have seen your method's complexity on appendix C but also want to see the difference and comparison of others.) If possible, please provide system runtime for training as well.\n\nWe thank the reviewer for asking the question. The main analysis of time complexity lies on the selection of subset and we provide comprehensive comparison here.\n\nFor Craig, there exist the need for forward propogation for each sample, and the corresponding time complexity is $\\mathcal{O}(dn)$. (d:model parameter, n:size of the dataset) The method is to select $q$ fraction ($0 < q < 1$) of the whole dataset with size all at once. The greedy selection strategy for selecting $qn$ samples out n samples is $\\mathcal{O}(qn^2)$. Therefore, the complete time complexity is $\\mathcal{O}(qn^2 + dn)$ for each epoch.\n\nFor Crest, instead of selecting from whole dataset, they select from subset with size M, and the corresponding time complexity for forward propogation is $\\mathcal{O}(dR)$. (d:model parameter, n:size of the dataset) The method is to select $m$ point from the subset. The greedy selection strategy for selecting $Rm$ samples out n samples is $\\mathcal{O}(Rm)$. Therefore, the time complexity is $\\mathcal{O}( P (Rm+ dR))$ where the $P$ is the number of subset that is manually chosen. The overall time complexity for Crest method need to multiple by the times they update the coreset within one epoch for fair comparison, but the average number of update for coreset cannot be calculated as they utilize adaptive strategy which depend on the curveture of the loss landscape.\n\nFor our method, we also select from subset with size R but we need multiple forward propogation (M times) and the corresponding time complexity is $\\mathcal{O}(MdR)$. We then select $m$ points from the subset and the corresponding time complexity is $\\mathcal{O}(mR)$. We need to perform multiple time to have $q$ portion of the whole dataset. The overall time complexity is $\\mathcal{O}(q\\frac{n}{R} (MdR+mR))$ for one epoch.\n\n3. The paper does not report actual training time and only provides theoretical and data usage analysis. I would encourage the authors to include a more comprehensive evaluation of system performance.\n\nWe appreciate the reviewer’s suggestion. We acknowledge the importance of reporting practical system performance in addition to theoretical analysis. We have already included some relevant metrics: Memory usage across the entire training process is presented in Figure 2. Training time for different datasets is provided in Appendix A.5, Table 5. A brief comparison of training time between Crest and our method is discussed in the caption of Table 1 in the main paper. In the updated version, we will include a dedicated table that summarizes training time, memory usage, and coreset selection overhead across methods for a more comprehensive system-level evaluation.\n\n4. This work has not been tested on larger models and datasets, which weakens its generality. The paper does not report actual training time and only provides theoretical and data usage analysis. I would encourage the authors to include a more comprehensive evaluation of system performance.\n\nWe thank the reviewer for raising this concern. To address the issue of generality, we would like to clarify that we have conducted extensive evaluations across a wide range of datasets and model sizes: Datasets: Our experiments include MNIST, CIFAR-10, CIFAR-100, TinyImageNet, ImageNet, and SNLI (a language inference dataset), with dataset sizes ranging from 60,000 to 1.4 million samples. Model scales: We evaluate models from small to large, including LeNet, ResNet-20, ResNet-18, ResNet-50, RoBERTa, and ViT-Base, spanning parameter counts from 10K to over 100M. (See Appendix A.9 for ViT results.) Robustness checks: We test under varying data corruption ratios (0 to 90 percent), data budgets (1, 10, 20 percent), and 3–5 random seeds to ensure consistent performance. As for system performance, we report memory usage in Figure 2 and include training time comparison in Appendix A.5, Table 5. A more detailed runtime and memory comparison table will be added in the updated version for completeness. To our knowledge, little prior work on coreset selection has systematically evaluated across such a broad combination of model sizes, dataset types, corruption levels, and data budgets. We believe this provides a comprehensive demonstration of the method’s robustness and scalability."}, {"review_id": "7as9KsN3Ux", "reviewer": "Reviewer_cZbS", "summary": "The paper conjectures that existing coreset selection methods may not work well since they do not reflect the fact that the data may be noisy. Also, the paper focused the fact that, when we apply a coreset selection method (i.e., reducing the training set), the objective function of the training becomes less smooth. The paper focused on existing gradient-based coreset selection methods that selects a coreset so that the gradient of the objective function at the trained parameter is similar between the coreset and the full dataset. The proposed method altered this formulation that the gradient of the objective function \"around\" the trained parameter is similar between the coreset and the full dataset. They experimentally showed that this produces the objective function becomes smoother as the result of the coreset selection by the proposed method than by an existing method, and that the prediction performance was better than existing methods, especially the case when dataset is highly noisy.", "questions": "### Major comments\n\n- Section 2, definition of $l$: Since the proofs of Theorem A.1 (Appendix A.2) and of Theorem 3.3 (Appendix A.3) require assumptions on $l$, assumptions on $l$ should be described in the main text. If it is not easy to describe in short, it may be good to provide the assumptions in the appendix and declare in the main text that the detailed assumption is not so restrictive but described in the appendix.\n- Section 3, Overall: In this algorithm, if we take $\\sigma\\to 0$ does it imply the conventional method? I wondered to what extent the theoretical properties shown in this section holds for the conventional method.\n- Section 3, Algorithm 1: Can the procedure of taking $E_\\delta$ be described explicitly? Is it computed analytically or numerically? In addition, the paper states that \"we focus on our attention for sampling in the batch normalization layers\" (lines 178-179). How is it implemented in Algorithm 1?\n- Section 3, subsection \"Algorithm\" (from line 158): Although theoretical analysis is done for the problem of equation (6) that finds the smallest coreset whose difference of gradients is $\\epsilon$ or less, in reality Algorithm 1 selects a coreset of predetermined size whose difference of gradients is $\\epsilon$ is smallest. In this sense, can the difference of gradients be represented by $M$, $R$ and/or $T$? In other words, can we derive the convergence analysis or other theoretical properties without $\\epsilon$ (only with parameters controllable in Algorithm 1)?\n- Section 3, Theorem 3.2: The statement (2) presents the difference between newton step of two subset as ${\\cal O}(\\epsilon^{\\frac{1}{2}})$, but is it alright that we ignore $\\sigma$ in this? If so, why?\n- Section 3, Theorem 3.3: In the main text, the effect of $\\xi_1$ (random sampling of batches) is not stated anywhere. As far as reading the appendix, does it mean that $\\xi_1$ can be computed from $M$, $R$ and/or $T$? (If so, defining $\\xi_1$ and $\\xi_2$ as the variables with similar functionality is confusing; it should be clarified that $\\xi_1$ is determined from other variables but we need to give assumptions on $\\xi_2$.)\n- Section 4, Overall: In the experiments $\\sigma$ is selected via cross-validation, but what effect is expected if larger or smaller $\\sigma$ is selected than the optimal one? Also, it is better if the effect of changing $\\sigma$ is examined.\n- Section 4, line 288: What is the \"design of more effective posteriors\"? As far as my understanding, it is not stated in Section 3 or Appendices. What is the mathematical formulation? How can it be implemented?\n\n### Minor comments\n\n- Section 2, lines 121-122: What does the sentence \"For tractability and better generalization, in practice, we use SGD (stochastic gradient descent) for optimization of (1) can lead to large memory burden and can be inefficient due to the calculation time\" intend? As long as I read this sentence, I understood that \"we usually use SGD but it is inefficient\", but it is somewhat unnatural.\n- Section 3, equation (5): Should $|\\ldots|_2^2$ be $\\\\|\\\\ldots\\\\|_2^2$ ?\n\n### Formatting issues / Misspellings\n\n- Overall: Figures and tables are too scattered in many pages even if they are related to experimental results. Please consider using `\\begin{figure}[p]` and `\\begin{figure}[p]` command: It creates pages that contain only figures and/or tables having `[p]`.\n- Section 1, line 37: \"Infact\" --> \"In fact\"\n- Section 3, line 146: \"Shin et al. (2023)\" is good to be replaced with the format \"(Shin et al., 2023)\" since it is not a citation directly read as a text.\n- Section 3, line 205: \"(see A.5)\" --> \"(see Appendix A.5)\" is clearer.\n- Section 4, Table 1: The place of Table 1 is unnatural. (If `\\begin{figure}[p]` above is not applied,) please consider placing at the top of the page, or at the top or the end of the paragraph that refers the table.\n- Section 4, line 279: \"Training dyanamics\" --> \"Training dynamics\"", "limitations": "The paper states that the limitation of the work is that they have examined performances for limited types of datasets (in this paper image datasets are used). I expect that, in the future work, not only the difference of data types but also the effect of model size should be examined.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Theoretical evaluations itself seems to be sound and useful.\n- The algorithm is simple. Experimental results show that it is faster than existing methods.\n\nWeaknesses:\n\n- Several additional analyses are desired. Especially, the effect of $\\sigma$ seems to be examined more (both theoretically and experimentally).\n- Explanation of algorithms and experiments seem to be incomplete (see \"Questions\" section below for details).", "was_revised": true, "final_justification": "First I rated 3 on this paper, but raised to 4 after rebuttals. My questions become clear, and the manuscript is expected to become clearer after fixes described in the rebuttal. Especially, for a parts of the paper with insufficient explicitness, I expect that these points will be improved in the final paper after reading authors' rebuttals.", "rebuttal": "We thank the reviewer for all the comment and address them in the following. If there is still confusion, we will be glad to provide more information or illustration as needed in the discussion stage.\n\n1. Section 2, definition of l: Since the proofs of Theorem A.1 (Appendix A.2) and of Theorem 3.3 (Appendix A.3) require assumptions on loss, assumptions on should be described in the main text. If it is not easy to describe in short, it may be good to provide the assumptions in the appendix and declare in the main text that the detailed assumption is not so restrictive but described in the appendix.\n\nWe thank the reviewer for pointing this out. We will explain the assumption in main context or appendix to make it clear.\n\n2. Section 3, Overall: In this algorithm, if we take $\\sigma$ to zero does it imply the conventional method? I wondered to what extent the theoretical properties shown in this section holds for the conventional method.\n\nYes, when $\\sigma \\to 0$, our method reduces to conventional deterministic approaches such as Craig and Crest. The gradient perturbations vanish, and the selection becomes equivalent to greedy optimization over exact gradients. Importantly, our theoretical results are consistent in this limit. Also, when $\\sigma > 0$, our framework introduces loss landscape matching, which helps align coreset selection with the local geometry of the loss landscape. This provides a principled way to improve robustness over standard methods, offering both a generalization and a deeper interpretation.\n\n3. Section 3, Algorithm 1: Can the procedure of taking expectation $E_\\delta$ be described explicitly? Is it computed analytically or numerically? In addition, the paper states that \"we focus on our attention for sampling in the batch normalization layers\" (lines 178-179). How is it implemented in Algorithm 1?\n\nWe compute the expectation over $\\delta$ numerically by averaging gradients over 4 random perturbations (by sampling $\\delta$ with gaussian distribution and add them to the parameter weight) of the model weights. For efficiency and stability, we restrict perturbations to batch normalization layers. This choice significantly reduces memory cost and avoids destabilizing the gradient scale, while still improving performance. Although perturbing all parameters is theoretically appealing, we found that focusing on batch norm layers offers a good trade-off in practice.\n\n4. Section 3, subsection \"Algorithm\" (from line 158): Although theoretical analysis is done for the problem of equation (6) that finds the smallest coreset whose difference of gradients is $\\epsilon$ or less, in reality Algorithm 1 selects a coreset of predetermined size whose difference of gradients is $\\epsilon$ smallest. In this sense, can the difference of gradients be represented by M, R and/or T? In other words, can we derive the convergence analysis or other theoretical properties without (only with parameters controllable in Algorithm 1)?\n\nWe thank the reviewer for this thoughtful question. The main theoretical tool we rely on is greedy selection for submodular cover problems, where the approximation guarantee is of the form $(1 - \\exp(-\\gamma))$, assuming $\\gamma$-weak submodularity. However, the constant $\\gamma$ depends on properties of the loss landscape and the feature geometry, which are data-dependent and not explicitly controlled by $M$, $R$, or $T$ in Algorithm 1. Therefore, this quantity cannot be determined explicitly through the control parameters presented in the algorithm and remain it in symbolic format in our analysis. Empirically, $\\epsilon$ is small (see details in Craig paper) with greedy algorithm supporting the practical effectiveness of our fixed-size selection strategy. The theoretical gap between $\\epsilon$-based guarantees and fixed-size selection is common in coreset literature (e.g., Craig and Crest), and we adopt a similar pragmatic approach.\n\n5. Section 3, Theorem 3.2: The statement (2) presents the difference between newton step of two subset without dependency of $\\sigma$, but is it alright that we ignore in this? If so, why?\n\nThank you for the insightful question. While the simplified statement in the main text omits $\\sigma$ for clarity, the full derivation in Appendix A.2 (e.g., Eq. 33) shows that certain terms do depend on $\\sigma$, such as $(\\frac{\\epsilon}{\\sigma})^{1/2}$. However, this dependency is entangled with other factors like the eigenvalues of the Hessian. We chose to highlight the dominant structural terms in the main text but agree it's helpful to clarify $\\sigma$'s role. We will revise the paper to make this dependency more explicit.\n\n6. Section 3, Theorem 3.3: In the main text, the effect of $\\xi_1$ (random sampling of batches) is not stated anywhere. As far as reading the appendix, does it mean that $\\xi_1$ can be computed from $M$, $R$ and/or $T$? (If so, defining $\\xi_1$ and $\\xi_2$ as the variables with similar functionality is confusing; it should be clarified that is determined from other variables but we need to give assumptions on $\\xi_2$.)\n\nWe appreciate reviewer for pointing this out. In our analysis, the structure of noise is assumed in equation in assumption part of the proof (see appendix A.2 and equation (49) (50) (57)) (The $\\xi_1$ is isotropic noise with variance $\\sigma_1$) as done in Crest.  We will add additional note for this in the main content to make it clear.\n\n7. Section 4, Overall: In the experiments, $\\sigma$ is selected via cross-validation, but what effect $\\sigma$ is expected if larger or smaller is selected than the optimal one? Also, it is better if the effect of changing is examined.\n\nIn our analysis, we find there exist trade off between matching of gradient and loss landscape. If we increase the magnitude of the noise injected into the model for selection, we will find that there exist better match in loss landscape for current coreset, and decrease of the noise will lead to more accurate estimation of the gradient at current model weight. The balance between the two gives the optimal performance. We also find if we increase the noise by too much, it will start to resemble the random sampling as the gradient becomes too chaotic and cannot capture either the loss landscape or gradient.\n\n8. Section 4, line 288: What is the \"design of more effective posteriors\"? As far as my understanding, it is not stated in Section 3 or Appendices. What is the mathematical formulation? How can it be implemented?\n\nThank you for pointing this out. The \"design of more effective posteriors\" refers to the flexibility in choosing how we perturb the model weights during gradient estimation. The current strategy can be interpreted as drawing samples from an isotropic Gaussian posterior centered at $\\bar{w}$ with variance $\\sigma^2$ (i.e., uniform in all directions). However, one could consider more informed posterior distributions. For instance, perturbing only along sharpest directions in the weight space—akin to using an anisotropic or structured posterior—may better explore sensitive regions of the loss landscape. While Section 4 outlines this idea at a high level, implementation details for our current perturbation strategy are provided in Appendix A.4. We agree that further formalization of alternative posteriors is an exciting direction for future work and will clarify this in the revision.\n\n9. Section 2, lines 121-122: What does the sentence \"For tractability and better generalization, in practice, we use SGD (stochastic gradient descent) for optimization of (1) can lead to large memory burden and can be inefficient due to the calculation time\" intend? As long as I read this sentence, I understood that \"we usually use SGD but it is inefficient\", but it is somewhat unnatural.\n\nWe appreciate that the reviewer pointing this out. There should be gradient descent (GD) in the front for comparison."}, {"review_id": "s65c6xb7Eq", "reviewer": "Reviewer_Qgh2", "summary": "The paper provides a methodology for submodular coreset generation that leverages gradients such that the loss of training on the coreset has lower discrepancy towards the loss of training on the entire dataset than previous submodular coreset generation methodologies. This is done by establishing a connection between the posterior sampling and loss landscapes. To that end, the authors then provide a loss function that takes into account a smoothed posterior sampling as a function of the model's weights.\n\nFinally, experiments are provided showcasing the caveats of using previous submodular coreset generation that leverages gradients when handling corrupted data as well as the robustness of the provided coreset generation technique in the paper.", "questions": "Please address the following questions and edits:\n\n1) Page 2, Line 71: replace \"an\" with \"a\".\n2) Page 4, Algorithm 1: Wouldn't it be more correct to ensure that the union of $V_p$ would be equal to  $S$?\n3) Page 5, Line 166: Remove second \"instead\".\n4) Page 6, Theorem 3.3: What is $\\sigma_2$?\n5) Page 8, Line 264: Add \".\" at the end of the paragraph name.\n6) Supplementary, Page 22: What is $w^\\ast$?\n7) Supplementary, Page 22, equation 15, last equality: There is a missing p(w).\n8) Supplementary, Page 22, equation 15, last inequality: How was the negative term obtained? \n9) Supplementary, Page 23, equation 20: How was the first term obtained? \n10) Please ensure to update the proofs and ensure that every pass is justified correctly, since the theoretical contribution of the paper revolves around these proofs.\n11) Why did the authors not compare against GradMatch?\n12) How does your approach fare against sensitivity-based coreset, e.g., [1]? I am asking since this approach outperformed that of GradMatch.\n\n---------------------------------\n[1] Tukan, M., Zhou, S., Maalouf, A., Rus, D., Braverman, V., & Feldman, D. (2023, July). Provable data subset selection for efficient neural networks training. In International Conference on Machine Learning (pp. 34533-34555). PMLR.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In what follows, the strengths of the paper are given:\n  * The idea of leveraging gradients and ensuring lower discrepancy towards the gradients of the full model during the training process is novel and of high interest in this field.\n  * The idea to use the BN layer for the gradients used for the coreset construction is smart and led to lower memory.\n  * The experimental settings and practical results showcase the importance of the approach and its effect on the quality of the trained model.\n\nTo this end, below are the weaknesses of the paper:\n   * The writing needs some polishing, as it is hard to understand some of the things in the paper -- mainly concerning the proofs (see questions section)", "was_revised": true, "final_justification": "I have increased my score as a result of the rebuttal and discussion with the authors here. The authors have addressed the lingering concerns I had with their submission, and as a result, the score has been updated.", "rebuttal": "1. Page 4, Algorithm 1: Wouldn't it be more correct to ensure that the union $V_p$ of would be equal to $S$?\n\nWe thank the reviewer for raising this point. It is not necessary in our setting for the union of $V_p$ to exactly equal $S$. For example, under a 10 percent data budget with a batch size of 128 and 100 total batches per epoch, we may set $P=10$ and construct each $V_p$ from $R = 500$ examples, selecting $m = 128$ samples from each for training. In this way, $S$ (the final coreset) is formed without requiring full dataset access. This design avoids the high computational cost and selection bias that may arise when scanning the full dataset, as in methods like Craig.\n\n2. Page 6, Theorem 3.3: What is $\\sigma_2$ ?\n\nWe thank the reviewer for the question. In Theorem 3.3, $\\sigma_2$ refers to the variance of the noise used to perturb the model weights as stated in assumption 4 in appendix A.3, while $\\sigma_1$ denotes the variance of the stochastic gradient noise due to data sampling. Unlike $\\sigma_1$, which arises inherently from minibatch training, $\\sigma_2$ is a tunable parameter under our control and can be adjusted to optimize performance. We will clarify this distinction in the main text and reference the relevant derivations in the appendix for completeness.\n\n3. Supplementary, Page 22: What is w*?\n\nWe apologize for the typo. The $w*$ is indeed $\\bar{w}$.\n\n4. Supplementary, Page 22, equation 15, last inequality: How was the negative term obtained?\n\nwe provide the detailed derivation for it below:\n\n\\begin{equation}\n\\begin{split}\n    f(\\bar{w}) \\int (f(w) - f(\\bar{w}))p(w)dw &= f(\\bar{w}) \\int (f(w) - f(\\bar{w}) -\\nabla f(\\bar{w})(w-\\bar{w}) + \\nabla f(\\bar{w})(w-\\bar{w}))p(w)dw \\\\\n    &= f(\\bar{w}) \\int ((f(w) - f(\\bar{w}) -\\nabla f(\\bar{w})(w-\\bar{w}))p(w)dw + f(\\bar{w})\\int \\nabla f(\\bar{w})(w-\\bar{w}))p(w)dw \\\\\n    &= f(\\bar{w}) \\int ((f(w) - f(\\bar{w}) -\\nabla f(\\bar{w})(w-\\bar{w}))p(w)dw \\\\\n    &\\leq ||f(\\bar{w})|| \\int ||(f(w) - f(\\bar{w}) -\\nabla f(\\bar{w})(w-\\bar{w})||p(w)dw \\\\\n    &\\leq ||f(\\bar{w})|| \\int c_1||w-\\bar{w}||p(w)dw \\\\\n    & = c_1 ||f(\\bar{w})|| E_p[||w-\\bar{w}||] \\\\\n    &\\leq c_1 ||f(\\bar{w})|| \\sqrt{E_p||w-\\bar{w}||^2} \\\\\n    &= c_1 ||f(\\bar{w})|| \\sigma \\sqrt{d}\n\\end{split}\n\\end{equation}\n\nThe second term at the second line of the derivation is canceled as $\\int w p(w) dw = \\bar{w}$ and the first inequality is due to the Cauchy-Schwarz inequality and the second inequality is due to the equation 14 in appendix A.1. The final inequality uses Jensen’s inequality due to the concavity of the square root function. Since we obtain the upper bound of the magnitude, we can lower bound the term by switching the sign of the term to be negative as claim in equation 15 and 16 in the appendix A.2. We acknowledge minor issues in the previous version (e.g., a missing constant or square root), but these do not affect the overall dependencies in the result. We will correct these technical details in the updated version.\n\n5. Supplementary, Page 23, equation 20: How was the first term obtained?\n\nwe provide the detailed derivation for it below:\n\n\\begin{equation}\n\\begin{split}\n    \\int ||f(w) - f(\\bar{w}) - \\nabla f(\\bar{w})(w - \\bar{w})||^2 p(w)dw \n    &\\geq \\int c_2 ||w-\\bar{w}||^2p(w)dw \\\\\n    &=c_2 E_p[||w-\\bar{w}||^2] \\\\\n    &=c_2 \\sigma^2d\n\\end{split}\n\\end{equation}\n\nThe first inequality is due to the equation 14 in appendix A.2.\n\n6. Please ensure to update the proofs and ensure that every pass is justified correctly, since the theoretical contribution of the paper revolves around these proofs.\n\nWe thank the reviewer for raising the problem and help make the paper better. We have double checked the proofs and the result remains intact. We will update the proof in the updated version.\n\n7. Why did the authors not compare against GradMatch?\n\nWe thank the reviewer for raising this point. The short answer is that GradMatch is already outclassed by other methods we compare against e.g. CREST extensively evaluated against GradMatch and showed the former is superior.\n\nIn designing our experimental comparison, we aimed to evaluate a diverse range of methods that reflect fundamentally different principles of subset selection. Specifically, we compared against: (1) \\textbf{Craig}, which uses submodular greedy selection based on gradient similarity; (2) \\textbf{Crest}, which adaptively selects points to match loss landscape curvature; (3) \\textbf{Glister}, which employs bi-level optimization to maximize validation performance; and (4) random selection as a baseline. \\textbf{GradMatch} is indeed another gradient matching method, but its core idea—greedy selection based on gradient alignment—is conceptually similar to Craig, with the main difference being the use of Orthogonal Matching Pursuit (OMP) rather than submodular cover. Since Craig already serves as a strong representative of this class, and other methods like Crest have shown strong empirical performance over both, we chose to focus our comparisons on methods with greater conceptual diversity. We will clarify this rationale in the updated version.\n\n8. How does your approach fare against sensitivity-based coreset, e.g., [1]? I am asking since this approach outperformed that of GradMatch.\n\nWe thank the reviewer for suggesting another baseline. The sensitivity-based baseline takes a more `global' perspective - constructing a non-uniform distribution to optimize over the worst case bound for an RBFNN on the original data. this in turn acts as a proxy for gradient matching, assuming the gradients can be closely approximated by an RBFNN. However, since the coreset is selected using worst case bounds to be able to fit /any/ RBFNN, it may not capture nuances of data distributions, especially in presence of noise. This is exactly what we observe too, as the following tables show. Note that we were not able to reproduce the results presented in their paper, may be due to different data setups even the random subsampling reported in that paper is way worse than what we observed, and what many other coreset papers have reported. (potentially due to update frequency, learning rate, architecture and other hyperparameters.) Nevertheless we use the code they made publicly available for our evaluations. We see that the method performs better than Craig but worse than crest which is worse than our method. This is especially apparent in the presence of noise. \n\nCifar-100 \n\n| Corruption Ratio | RBFNN [A]| Craig (mean ± std) | CREST (mean ± std) | Ours (mean ± std) |\n|------------------|------------|--------------------|---------------------|-------------------|\n| 0.0              |    0.6449  ± 0.0038       | 0.5665 ± 0.0053    | 0.6728 ± 0.0030     | **0.6986 ± 0.0025** |\n| 0.1              |    0.5248 ± 0.0133        | 0.4629 ± 0.0102    | 0.6391 ± 0.0040     | **0.6644 ± 0.0034** |\n| 0.3              |    0.3422 ± 0.014        | 0.2968 ± 0.0162    | 0.5712 ± 0.0060     | **0.6085 ± 0.0044** |\n| 0.5              |      0.2868 ± 0.019      | 0.1603 ± 0.0048    | 0.4305 ± 0.0260     | **0.5014 ± 0.0068** |\n\nCifar-10\n\n\n| Corruption Ratio | RBFNN [A] | Craig (mean ± std) | CREST (mean ± std) | Ours (mean ± std) |\n|------------------|------------|--------------------|---------------------|-------------------|\n| 0.0              |   0.8255  ± 0.0040        | 0.7618 ± 0.0080    | 0.8724 ± 0.0040     | **0.8757 ± 0.0029** |\n| 0.1              |     0.7924 ± 0.0113       | 0.7490 ± 0.0066    | 0.8440 ± 0.0030     | **0.8544 ± 0.0034** |\n| 0.3              |     0.7280 ± 0.0118       | 0.7050 ± 0.0112    | 0.7843 ± 0.0040     | **0.8120 ± 0.0058** |\n| 0.5              |     0.6216 ± 0.0241       | 0.6320 ± 0.0186    | 0.6652 ± 0.0130     | **0.7318 ± 0.0143** |\n\n\n[A] Tukan, M., Zhou, S., Maalouf, A., Rus, D., Braverman, V., & Feldman, D. (2023, July). Provable data subset selection for efficient neural networks training. In International Conference on Machine Learning (pp. 34533-34555). PMLR."}]

Author rebuttal:

Meta-review: This paper proposes an approach based on posterior sampling for coreset selection. This approach introduces a smoothed loss function on model weights to improve the stability and generalization. The authors also analyze its convergence behavior and give the convergence rate. Extensive experiments are carried out on a broad variety of datasets and model architectures of various sizes. The authors show that the proposed coreset selection algorithm outperforms some of the SOTA methods. All reviewers consider the work sufficiently novel with a good impact to the machine learning community. There are numerous concerns raised by the reviewers including additional evaluation on more datasets/models and detailed questions on mathematical treatment. The authors' rebuttal has cleared up these concerns. Overall, this is an interesting work that is theoretically sound and practically well evaluated. All reviewers are supportive of accepting it.  The authors need to improve the exposition to make the paper clearer and more rigorous in the revised version.
Decision: Accept (poster)