Towards Accelerated Model Training via Bayesian Data Selection
Authors: Zhijie Deng
Peng Cui
Jun Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: data selection
training acceleration
probabilistic modeling
Bayesian methods

Abstract: Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.

Towards Accelerated Model Training via Bayesian Data Selection
Zhijie Deng' Peng Cui²r Jun Zhu
'Qing Yuan Research Institute, Shanghai Jiao Tong University Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua-Boog Bosch Joint MI. Center, THBI Lab, Tsinghua University, Beijing, 100084 China zhi jiedds www.edd..c xpeng cui @gmail. com, desz (eszjsttinghua edu cn
Abstract
Mislabeled, duplicated, biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility handle such variety simultancously. Recent work has proposed a more reasonable data selection principle by examin- ing the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging lightweight Bayesian treatment and incorporating off-the-sse shelf zero-shot predictors built on large-scale pre- rreitrined models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging Web- Vision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.
Introduction
The past year has witnessed significant breakthroughs deep learning research and applications, with Stable Diffusion (38), ChatGPT [32). and SAM [22) as representative examples. Practitioners have realized that the quality data used to fuel AI systems is critical in unlocking their full potential. Unfortunately, real-world scenarios often present mislabeled, duplicated, or biased data. As result, is paramount to develop methods that can prioritize valuable training data to enable more efficient model training and even improved model convergence.
Data selection for accelerating the training of deep models is gaining increasing interest. Some studies, such curriculum learning, advocate prioritizing easy samples in the early training stages [.) these samples quickly become redundant once they have been learned, making continued training on them waste of time. On the other hand, online batch selection methods [9 [7] prioritize hard samples with high training loss or gradient norm to avoid duplicate training. Nevertheless, in practice, the hardness of samples often arises from pathologics such improper annotations, inherent ambiguity, or unusual patterns, rendering it problematic to prioritize such samples [I].
The reducible hold-out loss selection (RHO-LOSS) approach 31 addresses these issues by quantify- ing the usefulness of sample based on its marginal influence on the model's generalization loss, forming theoretically grounded and universal objective for data selection. However, the estimation of this objective is non-trivial, and RHO-LOSS has to rely on less principled approximations for
"quual contribution.
1 Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

practical adoption. Besides, RHO-LOSS hinges on considerable number of holdout data to train an auxiliary validation model, which can be costly and should be performed repeatedly for new tasks.
This paper aims to bridge this gap to make the generalization loss-based data selection principle more accessible broader audience. We establish a more reasonable approximation of the original objective than RHO-LOSS while eliminating the need for holdout data. To achieve this, we derive a lower bound of the objective to separate the posterior predictive defined on the training data from that defined on the holdout data. Afterward, we propose to use off-the-sheif zero-shot predictors, built upon large-scale pre- pre-trained models B5 41 as a proxy for the latter, since these models often contain generally applicable information for solving specific downstream tasks.
We maintain Bayesian treatment of the training model to ensure accurate estimation of the original objective. Bearing in mind that our original goal is to accelerate the training of deterministic model, we adopt the simple and effective Laplace approximation
g for Bayesian inference. effortlessly converts point-estimate parameters to a Gaussian posterior. We further introduce Kronecker-factoree (KFAC) [30] and last-layer [23] approximations accelerate the processing of modern neural networks (NNs), resulting in an efficient and easy-to-implement algorithm.
[9 37
We conduct comprehensive empirical studies various benchmarks to evaluate the effectiveness of our method. The experiments on standard image recognition tasks demonstrate that our approach can outperform various baselines in aspects of training speed and final accuracy. This conclusion also holds for learning with label noise and class imbalance. On the challenging Web Visvision which contains plenty of noisy labels and ambiguous images collected from the internet, our method significantly reduces the number of training steps needed to reach the target accuracy and achieves up to 19% higher final accuracy than prior arts (see Table 3 These results highlight the practical value of our approach. In addition, we conduct informative ablation studies to gain better understanding of the behavior of our method.
(25 dataset,
2 Background
In this section, we briefly revisit the concept of online batch selection 226 and the data selection principle defined with the model's generalization loss in [31].
Consider training a 0-parameterized deep model fe : X + RR* on a dataset D (1 (+)), using stochastic gradient descent (SGD). The model likelihood is formally p(y zr 0) f=)) At each training step f, we can access a data batch B, of size ng from D. In online batch selection, we need to compute statisties of the samples in B, and select only those that meet certain requirements to update the model. This filtering process aims to remove samples that are deemed less valuable.
Let 1 denote the set of data observed before t and (x', 3/) sample from B, If we accept selecting (a', (a',)'), the updated predictive distribution, in a Bayesian view, will be p(y|æ, D.-1, {(x', v'))) (pt0fp.-1,(1= w ()));,(y/(z 0) The question arises how to estimate the quality of this distribution to determine which sample (z',8')= E B, should be selected. A natural tactic is to compare this distribution to the ground truth data-generating distribution p(a, That said, the typical KL. divergence can be introduced, and our goal becomes solving the following problem:
min Exca) DKi. [Kal[p(vla))))))))> [llllz) D.-1, (('''())))))_) = const. - Eggeoo [ogplyyx 2 Dr-1, ((x', . v'))), (r 8,
(1)
where const. denotes constant agnostic the object to optimize. By applying Monte Carlo (MC)
estimation using extra holdout samples D* = (w,, (1) }mm from p(x, v). we arrive at the following optimization problem:
1 77 [lo poep(illi,. Dr-1, (())
max Logp(D"(Dr-a (/)).//))... (2)
max (e' JEB; m
(B
This objective corresponds to the model's generalization loss instead of the fitness of training data.
By Bayes' rule, there is:
p(D" DD. ((x' ((''s p(y 1'', D', Di-1)p(a', D', D',D,-1) p(y' lo', D', D'',,,DD-. Dz-1) p(y'>'' D.- Pp-1)p(z' D.-1) p(u122' D.
"D(DD (D:"()),,,'), (3)
We assume selecting single sample per time for simplicity. Multi- Multi-sample selection is viable.

---

where the item (D actually equals to the constant p(D"(Pt-1) because x' alone cannot incur model update. Plugging this back into Equation 2 we arrive at the final objective for data selection:
max logplyla, D", Dz-1) po,
(x,y)eB,
where we omit constants and abuse (x. v) to represent (x', hereinafter if there is misleading.
Although the above selection principle is theoretically sound and universally applicable, estimating accurately is challenging. In particular, it is difficult estimate the posterior predictive distribution defined with the combination of training and holdout data a computationally efficient manner. To address this issue, RHO-LOSS proposes approximating log p2y|2: D Dz-1) with log p(y/ D") and approximating the posterior predictive with the point-estimate model' predictions [1]. However, these approximations compromise the method's theoretical groundness, and access to holdout data often be infeasible in practice. Our approach overcomes these limitations by utilizing a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models, as detailed below.
Methodology
In this section, we demonstrate that the data selection principle discussed carlier can be by more easily computable quantities. We then leverage these insights to develop Bayesian data selection approach accelerating the training of deterministic deep models.
3.1 The Lower Bound
As previously discussed, we need to estimate two log- pee--obbbblitiss logp(y/a D", Dz-1) and log p(y|2, D,-1), for each sample in B, to identify the most useful one and update the model accordingly. However, as mentioned, evaluating the former is challenging. To address this issue, we derive a lower bound that allows for more feasible estimation.
We first unfold log p(ylæ, D", Dr-1) as the combination of a posterior DO D:-1) and the model likelihood p(y 0), detailed below (we defer the derivation to Appendix):
log p(ulæ, D", Dz-1) = log p(D"10))((1D):))))): 0)d0 - (DD
(5)
Then, by Jensen's inequality, there is (derivation is deferred to Appendix)
logply|æ, D*, D.-1) 2 pp log p(ylz, 0) + ) log log (og
Notably, the last two terms are independent (x, v) and can be excluded from the optimization.
We similarly derive another lower bound:
log p(ylz, D", Di-1) log 1(D==11)))(000'))aa 1og
(7)
2 pp(0p-=>
+ Ep(o log (D:,((D))))))) - po
where the last two terms also agnostic to (ar, v) and hence can be omitted.
Combining Equation (6) and Equation 77. there is
logp(y/a (y]aa ar, D", D.-1) 2 c<Bpoo/D._.) logp(y) 0) + (1 -1--a (1-a))ppeer) log x. 0) + const., (8)
where E [0, 1] represents trade-off coefficient. This way, disentangle the posterior associated with the training data D.-1 from that associated with the holdout data D*.
Rewriting p(y|æ, D.-1) as (p (Py|e,( we can subsequently convert Equation (4) to the following maximization problem:
max o<Ep(orp._.) logp(y) x, 0) + - ax) Ep(rrr) logp(y/æ, 0) log Eptorp.- 0). (9) E,
The presented objective is intriguing for several reasons. Firstly, the first term and a times the third term perform expectation and logarithm operations in reverse order, resulting in quantity similar to
3

---

the uncertainty estimates in Bayesian deep learning [39 (nonetheless, our objective involves data labels). Additionally, the third term helps to prevent the selection redundant samples. And, the second term prioritizes points with high semantic alignment with their annotations. These three forces are integrated adaptively to accelerate model training across various stages.
3.2 Zero-shot Predictor as the Validation Model
Collecting extra holdout data to train an auxiliary validation model can be costly and should be performed repeatedly for new tasks. To address this issue, we propose to zero-shot predictors built upon large-scale pre-trained models [35 41 as a proxy for the validation model, based on the observation that they usually exhibit promising transfer performance across broad range of downstream applications.
Formally, make the following approximation:
Eppoy-> log plyl(,,,,0) x log (w))
(10)
where f X + R& denotes the zero-shot predictor used.
We can think of the pre-trained model universal validation model trained extensive dataset, leading to the Bayesian posterior collapsing to a point estimate. Although its training data may not precisely follow the data-geneet gata-generating distribution for the current task, they share fundamental patterns with the data in our problem, making the above approximation reasonable. Notably, as shown in Section 4 our trained model eventually performs much better than the zero-shot predictor.
3.3 Lightweight Bayesian Treatment of the Training Model
The first and third terms in Equation raise the requirement of maintaining Bayesian posterior p(0) during training. Due to the high nonlinearity of deep NNs, the analytical posterior is intractable. By convention, we can introduce an approximate posterior ((0)) and tune by approximate Bayesian inference methods like MCMC [42 5 45), variational inference 15 28 44]
211, Laplace approximation (29 37, etc. Recalling that, despite relying on a Bayesian treatment, our final goal is to accelerate and improve the training of deterministic model, Laplace approximation is well suited to our setting- it can convert point-estimate parameters to a Gaussian posterior effortlessly, thus lifting the unnecessary burden of learning and maintaining posterior explicitly. We clarify that, technically, other Bayesian inference methods are also compatible with our approach.
Although Laplace approximation is typically used for maximum posteriori estimation, it can be effectively adapted the online learning cases [36]. Specifically, consider an isotropic Gaussian prior with precision To over parameters 0. Let D == by U b2 U ... U by b denote all selected training samples before time t. The parameters of our deterministic model at time - 1, dubbed as 02-1. are likely to approach a mode of the true posterior p(0)D;-) in the presence of proper weight decay regularizer in stochastic optimization [T]. The online Laplace approximation then deploys the following approximate posterior:
1-1
8 Logp(y)z 0)]
((01D=-1) -N(0. H = Tol + Σ
Σ
(II)
ag2
z.Ebb
The Gaussian covariance should be positive semi-definite, but 0,-1 cannot be guaranteed to be exactly the posterior mode, we would better replace the Hessian matrix with the generalized Gauss-Newton (GGN) matrix to avoid ill-posed approximation, resulting in:
= Grl),, G = Tol +
Σ 1=1
Σ SUEE
Jo, (x) Do, (o))
(12)
where Jo,(x) Vofo(z))ooo, and Ag, (I. - p(y) p(l
Practical acceleration. Modern neural networks usually contain millions of parameters, making the matrix G,-1 too large to fit into CPU/GPU memories. To address this, a common practice is to sparsify the matrix using diagonal or Kronecker-factorad (KFAC) approximations [30]. This work
bb C B, represents the set of selected samples at time step i.

---

Algorithm Bayesian data selection to accelerate the training of deterministic deep models.
I: Input: Number of iterations T, datasct D, prior precision To, number of effective data Ye, batch size n. number of selections yau, zero-shot predictor deterministic model with parameters 0. Intialize 0u. Ag <- 0, Ga <- 0:
3: for in I,..., do
4: Draw mini-batch B, from D:
5: Ve-1 Ane /TOL, Ut-1 Vn.G.-1 vTol;
6: Estimate the objective in Equation (16) for every sample in B, and select the top-ns, ones to form by;
fo, Perform back back- proppaaaanntiion with E Exyeb, fo, (a));;
8: Apply weight decay regularization and do gradient ascent obtain 8;;
Use last-layer features and softmax gradients update Ae and G. with exponential moving average;
10: end for
prefers KFAC preserves the correlations between parameters within the same layers. Besides, recent surprising finding is that we can even apply Laplace approximation to only the last layer of a deep NN, leaving the other parameters point-estimate, conjoin efficiency and calibrated uncertainty As result, we consider combining last-layer and KFAC approximations to reduce the burden caused by the Bayesian treatment, with the details presented below.
estimates [23 8
Let us decompose fe. as a feature extractor he, and linear layer with weights 8.0² E Rd i... fo. (x) he, T(()) We adapt the GGN matrix associated with the last-layer weights f.!) derived in 37 23 to the following formula:
G ~ Vi-1 U--1 (VPP-1lA + voll @ (VID--IGG.11 VTol),
(13)
where @ denotes the Kronecker product and
At-1: =
he (z))1o
D
F.yEb
(14)
G--1 Di-
Σ Σ hogpg//uuhnlaaaa..aooo/////////a
1=1 2.E0;
Of note that the matrices 4-1 Raxd and G R raise only minimal extra storage cost.
The approximate posterior can be formulated as matrix- Gaussian [12] q(((0) (D_1)
(0 1, V= Given the linear nature, it is straightforward to get the distribution over
the model output fo for input x (derivation is deferred to Appendix):
((&1DD-1) = - (
(x) (
(15)
The final data selection objective. With the above equation, the selection objective boils down
1 Σ log pog((ll/))) +1
1 log [s Σ p(ulss
max (a,g)EB, B,
L
(16)
where ((0) (f.n D.-1). 1,.., S are MC samples. Compared to the las-last-layer Laplace approximation, which involves sampling over the parameter space and performing MC integration, the last-layer one enjoys much faster evaluation procedure. It also enables the use of a large S. Our method can also abandon the KFAC approximation when the linear head is small.
The algorithm. We present the training procedure in Algorithm [ To avoid too large |D_-1 i| and hence too sharpened approximate posterior, we use a tunable hyper-parameter n, replace [D,-1] in the computation of V and U,-1. For implemental simplicity, we use the exponential moving average technique to update A, and Ge. For models equipped with batch normalization layers [16], we perform an extra forward propagation for the selected data b, to obtain proper batch statistics for model update, following (1). Our method takes a similar amount of time per iteration as [31] The primary difference is that we use a zero-shot predictor based on pre-trained models to compute the second term Equation (16), whereas [31 uses validation model. Computing the Gaussian covariance in Equation
[5 MC sampling, and updating A, and Ge consume neglectable resources.
5

---

Table 1: Epochs needed reach target test accuracy on clean and noisy data (final accuracy is reported in parentheses). CIFAR-10*/100* denotes adding 10% symmetric label noise to the dataset. Best performance is highlighted in bold. "_" indicates that the target accuracy was not reached. For all methods, only half of the original training set is used for training. The target accuracies are set following RHO-LOSS 31.
Mataset
CIFAR-10
CIFAR-10*
CIFAR-100
CIFAR- 100
CLIP Ace
75.6%
75.6%
41.6%
41.6%
Target Ace
80.0%
87.5%
75.0%
85.0%
40.0%
52.5%
40.0%
47.5%
Train Loss
81 129 (90%)
(28%)
138
(42%)
(4%)
Grad Norm
(61 %)
(23%)
139
(42%)
(4%)
Grad Norm IS
57 139 (89%)
57
(84%)
71 132 (55%)
94 142 (48%)
SVP
(55%)
(48%)
(18%)
(14%)
Irred Loss
(60%)
(62%)
93
(43%)
89
(43%)
Uniform
79
(87%)
62
(85%)
65
133 (54%)
79
116 (50%)
RHO-LOSS
39 65 (91%)
27
49 (91%)
48
77 (61%)
49
65 (60%)
Proposed
33 61 (91%)
25 47 (91%)
32
53 (63%)
53 (61%)
4
Experiment
We compare the proposed method to the prior state-of-the-art (SOTA) data selection methods on various image classification benchmarks, including standard datasets (e.g., CIFAR-10 10 and CIFAR- 100 (24). their variants with controllable degrees of label noise and class-imbalance, and real-world, large-scale, noisy, and imbalanced dataset, Web Vision (25] We also diagnose our selection strategy through elaborate ablation studies.
Datasets. We first evaluate the proposed method on clean CIFAR-10/I00 and noisy CIFAR- with 10% symmetric label noise. Furthermore, we experiment in the context of imbalanced datasets, specifically CIFAR-10/10 datasets with varying degrees of class-imbalance ratio. Last, we investigate web-scraped and large-scale dataset Web Vision, which consists of over 2.5 million images in 1000 categories and suffers from severe label noise and class-imbalance issues. For a fair comparison with RHO-LOSS [31. only half of the training set is used for model training.
Baselines. We compare the proposed method to variety of baselines that define selection principles with various metrics, including the (training) loss [20], gradient norm [19], and gradient norm with importance sampling (gradient norm IS) [99] We also compare uniform sampling, Selection-via- Proxy (SVP)
6 and the latest SOTA method, RHO-LOSS
31
Implementation details. In experiments under label noise, introduce 10% symmetric noise caused by randomly flipping the ground -ruth label to other labels. For the inassimmalaance setting, consider the long-tailed imbalance with two levels of imbalance intensity (i.e., 10 and 100), where an exponential decay on sample size is introduced across classes 33 We use the same optimizer (AdamW (27) and hyperparameters (e.g., learning rate 0.001, weight decay of 0.01, n2 32 and Tia 0.1) as RHO-LOSS. Unless specified otherwise, we use ResNet18 (RN18) [4] as the deterministic model and specify the zero-shot predictor with CL5P- We select the trade-off coefficient from {0.1, 0.2, 0.3, 0.4) and the number of effective data n. from {100, 200, 500, 1000). In most cases, we set the prior precision TD to 1. We report average results over 3 random runs.
Evaluation. Following 311, we evaluate speedup by comparing the number of epochs required to reach a given target test accuracy, which implicitly reflects the quality of the selection strategy.
4.1 Speedup on Clean Data and Noisy Data
We first empirically evaluate the proposed method on CIFAR- 10 and 10FAR-100 with/without label noise. reports the results. We can observe that the proposed method achieves considerably improved training speed compared to competing baselines in both scenarios with and without label noise. This evidences the efficacy our Bayesian selection method in filtering out redundant and noisy data points in the training set. Despite leveraging extra holdout data, RHO-LOSS still

---

Table Epochs required to reach a target test accuracy on imbalanced CIFAR-10 and CIFAR-100 (final accuracy is reported in parentheses).
Dataset
Imbalance Ratio
Target Acc
Uniform
RHO-LOSS
Proposed
60%
68
36
34
10
70%
98 (75%)
56 (80%)
52 (83%)
CIFAR-10
50%
50
44
100
60%
(56%)
87 (62%)
(68%)
20%
34
32
30%
110 (33%)
62 (39%)
58 (45%)
CIFAR-100
15%
70
43
41
100
20%
(19%)
71 (24%)
66 (28%)
Table 3: Epochs required to reach target test accuracy on Web dataset (final accuracy is reported in parentheses).
Dataset
Validation set
Target Ace
Uniform
RHO-LOSS
Proposed
30%
29
25 40 (60%)
WebVision
40%
-(27%)
47 (42%)
VebVision-100
30%
40
33
ILSVRC12
40%
-(23%)
(37%)
50 (55%)
30%
28
22
WebVision
40%
(26%)
48 (42%)
(61%) %)
Web Visionnn20000
30%
35
29
ILSVRC12
40%
(19%)
(39%)
46 (56%)
underperforms our approach, potentially due to the less reliable approximations. Surprisingly, the superiority of our method is even more significant on the more challenging CIFAR-100. We also point out that our method eventually reaches higher final accuracy than the zero-shot CLIP. In other words, our method does not hinge performant zero-shot predictor.
4.2 Speedup on Class-imbalance Data
We further evaluate the proposed method on class-imbalane data, a typical pattern in the real world while probably raising non-trivial difficulties for model training 37. Specifically, we experiment on the aforementioned imbalanced CIFAR-10 10 and CIFAR- 100 using similar protocols to previous studies. The test datasets remain unchanged. Given that RHO-LOSS is superior other baselines, we primarily compare with it in the following studies. We also add uniform sampling into comparison due to its implementation simplicity. Table presents the results.
As shown, the proposed method still consistently outperforms uniform sampling and RHO-LOSS. In particular, the final accuracy achieved by our method is higher than RHO-LOSS. We also note that the gain in training efficiency becomes more significant as the imbalance ratio increases. The failure of RHO-LOSS is partially attributed the less principled approximations. Another reason is that the holdout data are still class- imbalaneedd rendering the trained validation models biased.
4.3 Speedup on Large Web-scraped Data
Learning algorithms face significant challenges when dealing with noisy and imbalanced real-word datasets. To evaluate our proposed method's efficacy in addressing these challenges, we assess it on a large-scale web-scraped dataset, Vision. We defer the details of Web Since the dataset is large, comprising 2.4 million data points, we train classification models only on the first 100/200 classes (known as Visionn 100/200) of the entire dataset, following [1 for an efficient evaluation. We test the trained models the
Appendix.
WebVision validation set

---

Proposed
Proposed
50
RHO-LOSS
Uniform
atready classiffed correclly (%%
Uniform
Proportion of selected dats
Proportion of sdlectdd with label inli (%c
40
30
20
2-
10
0
CIFAR10
CIFAR100
CIFAR10
CIFARI00
(a) Proportion of label noise in selection.
(b) Proportion of redundant points in selection.
Figure I: Properties of the data selected by our method and baselines on CIFAR-I0 and CIFAR 100 with 10% label noise. Redundant points represent the data that have already been classified correctly. The reported values are averaged over 150 epochs of model training and five random runs.
Proposed
9.6
Uniform
Test accuracy (%)
0.4
accuracy
0.3
0.3
RN50
VIT-B/16
0.0
200
4.00
600
500
1000
125
150
Training iterations
Training epochs
Figure 2: Training curves corresponding to us- ing pre-trained ViT-B/16 as the model backbone. Wiiioniii--000 epoch=344 iterations)
Figure 3: Training curves corresponding to using zero-shot predictors defined with various CLIP backbones. (CIFAR-100)
and the ILSVRC12 validation set [9] We report the results in Table 3 We select 30% and 40% as target accuracies according to the final accuracies of RHO-LOSS (~40%) and approach (~60%%).
Still, the proposed method outperforms uniform sampling and RHO-LOSS aspects both speed to reach the target accuracy and the final accuracy both WebVision- 100 and WebVision-22 200.. Notably, our method achieves up to 19% higher final accuracy than RHO-LOSS, suggesting the superiority of our method in coping with real-world datasets. We also report results for method on the entire training set in the Appendix, which shows more significant speedups and higher final accuracy due more training data.
4.4 Analysis of the Properties of the Selected Data
In this section, we analyze the characteristics of the data selected by our method and compare it with existing works. Figure Ta displays the proportion of selected data points with label noise, and it is evident that data selected by our method has the lowest label noise. We also investigate whether our method prioritizes redundant data, defined as those having already been correctly classified by the training model (31]. As depicted Figure [b both our method and RHO-LOSS select significantly fewer redundant data points than uniform sampling, and method is slightly better.
4.5
Ablation Studies
In this section, we conduct ablation studies on model architecture, zero-shot predictor, and some critical hyper-parane parametrs (c.g., a, ne).

---

Table 5: The efficacy of variant of our method (referred to as Proposedt) where the zero-shot validation model is replaced by that used by RHO-LOSS [T]. Experiment on CIFAR-100.
RHO-LOSS
Proposed
Proposed
Epochs reach 40.0% test accuracy Epochs to reach 52.5% test accuracy Final accuracy
48
30
32
77
52
53
61% %
63%
63%
Tol SINAIRA (56)
Teet cnctarcrryyy
Tea 22
--0 0.0
Rg - 100
22
T 10
= 0.2
=200
100
Rg 5500
1000
0.1
Training
Training specis
Training spochs
(a) The trade-off coefficient a.
(b) The number of effective data ne.
(c) The temperature 7.
Figure 4: Ablation studies on some critical haper-parameters including a, The and 7 on CIFAR-100.
Model architecture. We test method using the powerful Vision Transformer (ViT) ar- chitecture on WebVision-200. Specifically, we experiment with a ViT-B/16 model pre-trained unsupervisedly by Masked Auto-encoder [13], considering that fine-tuning pre-trained model is preferred over training model from scratch in practice. We provide the training curve of our method in Figure 2 where uniform sampling is included for comparison. The huge performance gap between our method and the baseline validates the efficacy of our method in this setting.
Zero-shot predictor. We next assess the impact of the zero-shot predictor on method. We specify the zero-shot predictor with various CLIP variants gc.g., RN50 and ViT-B/16) and draw the training curves in Figure 3 As shown, although the zero-shot accuracy of the RN50 variant is significantly lower than that of ViT-B/16, the speedup effect of the two backbones is similar. This finding suggests the robustness of our method against the selection of the zero-shot predictor.
In the above experiments, we use CLIP-based zero-shot predictors without further tuning. Here, we perform linear probing using the CLIP models to demonstrate we can achieve good convergence trivially. We simply adopt the uniform sampling strategy and report the results in Table 4 We do not report convergence speed because the baseline uses pre-trained weights while method trains models from scratch. As shown, our method still bypasses the baseline with clear margins.
Table 4: Comparison of final accuracy (%) be- tween the proposed method and the linear prob- ing with CLIP and uniform sampling strategy.
Dataset
CLIP linear probing Proposed
CIFAR-10
84.5
91.4
CIFAR-10-
84.1
91.3
100
58.5
63.3
CIFAR-100
57.8
61.4
We replace the zero-shot predictor used in our method with the validation model in RHO-LOSS [31 and present the corresponding results in Table The comparison proves the superiority of our method over RHO-LOSS. These results also reflect that our method is robust against the specification of the validation model.
Sxpee--parameters. We then analyze the effects of three crucial papar-parameters, Re, and the temperature 7 used by the softmax operation in the CLIP zero-shot predictor. Figure 4a shows training curves associated with various a. We see that too large a (i.e., lightening the second term in Equation ( can lead a significant drop in training speed and final performance, which emphasizes the importance of the zero-shot predictor. Nevertheless, using only the zero-shot predictor (i.e., a = 0) is also suboptimal, leading to degraded final accuracy. Hence, we conclude that both terms in the lower bound of the original data selection principle are beneficial, where the first term accounts more for the later training stages while the second term accounts more for the earlier ones. Figure 4b shows the ablations study ne, and we witness the robustness of our method against Rg. Figure 4c

---

shows training curves corresponding to different 7. We observe that an appropriate temperature boosts our method considerably.
Wall-clock time. We empirically observe that the per-epoch training time for Uniform, RHO-LOSS, and our method on CIFAR-100 is 14s, 18s, and 21s, respectively. Namely, our approach has slightly increased per-epoch time over RHO-LOSS. It arises from that we use a CLIP-RN50 zero-shot predictor to compute the second term Equation (16), whereas [31] uses RNI8 validation model. In fact, this gap can be reduced by simple implementation trick prec-pig- the CLIP-RN50 predictions for each sample in the training set before training.
As shown in Table compared to RHO-LOSS, we can reduce the number of epochs required to reach 40% test accuracy from 48 to 32 and 52.5% from 77 to 53. According the per-epoch training time, we achieve a 48 * 18/32/2 22 x 1.29 or 77 18 18/53/2 0/53/21 21 R 1.25 times practical acceleration.
5 Related Works
Extensive methods have been proposed to accelerate model training through techniques such as data pruning, coreset selection, eurriculum learning. online batch selection, etc. Data pruning explores various metrics, such as EL2N score [3], forgetting score [40], and classification margin [34), to measure individual differences among data points and retains only the hardest examples for model training. However, data pruning still exhibits limitations when dealing with noisy labels, and some of these metrics are computationally expensive. Coreset selection methods also partially address the problem of accelerating model training. In particular, [3 contributes data scoring mechanism that robust the change of scenarios for coreset selection, and (46 makes an in-depth understanding of the catastrophic accuracy drop issue of one-shot coreset selection and contributes a novel solution to it. However, these methods lack the flexibility to prioritize samples with different properties at various training stages. Curriculum learning. advocated by [1] prioritizes easy points with low label noise before uniformly training on all data points. While this strategy enhances convergence, fails to address the issue of skipping redundant points already learned.
tackle the training acceleration problem by selecting hard data identified by high loss or gradient norm. However, they also suffer from a common drawback-hight loss can be caused by label noise or ambiguous labels, so prioritizing such data can result in a decline in predictive performance. Compared to the prior art, our method establishes Bayesian data selection metric and exploits zero-shot predictors prioritize valuable training data for addressing these issues.
Online batch selection methods 226
19 17]
6 Conclusion
This paper addresses the challenges posed by noisy and biased data in real-world scenarios. Our main contribution is to make the generalization loss-based data selection principle more accessible to accelerate the training of deep models. To achieve this, we first derive a lower bound of this objective to improve its tractability. We then propose leveraging Bayesian treatment and off-the-shelf zero- shot predictors to estimate the data selection objective reliably. The resulting algorithm does not require extra holdout data. Our extensive empirical studies demonstrate the superiority of our method in accelerating model training over competitive basclines.
Limitation. Our method may fail when the zero-shot predictor performs poorly on specific tasks. Future work could explore adapting the zero-shot predictor to a few-shot one using a few clean validation data address this limitation.
Acknowledgments
Z.J. Deng was supported by Natural Science Foundation of Shanghai (No. 23ZR1428700 and the Key Research and Development Program of Shandong Province, China (No. 2023CXGC0101 12). J. Zhu and P. Cui were supported by NSF of China Projects (Nos. 62061 1 136001, 616201 61620106010, 62076145, U19B2034, U181 1461, U19A2081, 6197222); grant from Tsinghua Institute for Guo Qiang: and the High Performance Computing Center, Tsinghua University. J.Z. was also supported by the XPlorer Prize.
10

---

References
[1] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[2] Charles Blundell, Julien Cornebise, Koray Kavukcueglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613 1622, 2015.
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datascts with labcl-distribution aware margin loss. Advances in neural information processing systems, 32, 2019.
[4] Pengfei Chen, Ben Ben Liao. Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, pages 1062-1070. PMLR, 2019.
[5] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference on Machine Learning, pages 1683-1691. PMLR, 2014.
[6] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskoves, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. International Conference Learning Representations. 2020.
[7] Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongic. Class-balancee, loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9268-9277, 2019.
[8] Erik Daxberger, Agustinus Kristiadi, Alexander Immer. Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortleess bayesian deep learning. Advances Neural Information Processing Systems, 20220089-20000 2021.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255.
leee, 2009.
[10] Zhijie Deng and Jun Zhu. Bayesadapter: Being bayesian, inexpensively and reliably, via bayesian fine-tuning. Asian Conference on Machine Learning, pages 280 290.295.20 PMLR, 2023.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010.1 11999, 2020.
[12] Arjun K Cupta and Daya K Nagar. Matrix variate distributions. Chapman and Hal/CRC, 2018
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[15] José Miguel Hernández- Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning, pages 1861-1869, 2015.
[16] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448 456.. pmlr, 2015.
[17] Angela H Jiang. Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv: 00706,,, 2019.
[18] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304 2313, PMLR, 2018
[19] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning. pages
PMLR. 2018.
[20] Kenji Kawaguchi and Haihao Lu, Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pages PMLR, 2020.
11

---

[21] Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weiigtt-ertuurationn in adam. In International Conference on Machine Learning, pages 2616-2625, 2018.
[22] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao. Spencer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. Segment anything. arXiv preprint (rXiv:2304.02643 2023.
[23] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overcon- fidence in relu networks. In Internationall conference on machine learning, pages 5436-5446. PMLR, 2020.
[24] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[25] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data, arXiv preprint arXiv: 1xXiv:708.002880 2017.
[26] Ilya I. Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks, arXiv preprint 1=Xiv:1511.06333.. 2015.
[27] Tlya Loshchilov and Frank Hutter. arXiv:111.051001, 2017.
Decoupled weight decay regularization. arXiv preprint
[28] Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, pages 7708-1716 2016.
[29] David John Cameron Mackay. Bayesian methods for adaptive models. California Institute of Technology. 1992.
[30] James Martens and Roger Grosse. Optimizing neural networks with fromecker-factoree approximate curvature. In International conference machine learning, pages 2408-2417. PMLR. 2015.
[31] Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnic Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 1.6630-16699. PMLR, 2022.
[32] OpenAL. Introducing chatgpt, 2022. https://openai. httpt/ai...oooo/g///aaa// chatgpt Last accessed on 2023-05- 09.
[33] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples carly in training. Advances in Neural Information Processing Systems, 34:20596- 20607, 2021.
[34] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 13:17044-17056 2020.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages 8748 846338763.66 PMLR. 2021.
[36] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
[37] Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018 Coff-Cofeerence Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition, pages 10684-10695, 2022.
[39] Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint Xivi:: 1833:0880300833 2018
[40] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations.
12

---

[41] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models zero-shot learners, In International Conference Learning Reprexentations, 2022.
[42] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, pages 681-688, 2011.
[43] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
[44] Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient variational inference. In International Conference on Machine Leurning, pages 5847-5856, 2018.
[45] Ruqi Zhang, Chunyuan Jianyi Zhang. Changyou Chen, Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. arXiv preprint arXiv:1902.03932, 2019.
[46] Haizhong Zheng, Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection high pruning rates. In The Eleventh International Conference on Learning Reprexentations, 2022.
13

---

A Proofs
A.1 Derivation of Equation
log plyply,æ, , D) = log
p(0 Dz- De-1"",,))p( 0)d0
-og / (1D))))(0(D(--1)
p(D"(D(-1) p p(ylz,0)d
log
A.2
Derivation of Equation (6
logp(y/z, D", D.-1) = log
2 /
(D.-1).
A.3 Derivation of Equation
15
Given that the posterior of the weights of the last layer (="V and the input the last layer is he, (Ir), there is
aL DD-1) = = =MN (x)
(x) ( (f (ho,_1(2) ho (x)) (z))
U,1: ho. (x)) V,
The second equation stems from that (ho, (1)) VV (I)) is scalar.
B Experiment Details
WabVision (all)
WabVislon (Filcke)
Number of Images
WabVision (Googie)
8000
ILSVRC 2012
6000
4000
2000
0
100
200
300
400
500
600
700
800
900
1000
Category
Figure 5: Number of images per category of the Web dataset 225
B.1 Preprocessing
All images of each dataset are normalized and augmented by random horizontal flipping. For CIFAR- 10/100, we use the standard 32 32 random cropping after pro-ppddinng with pixels on each side. In order to adapt to the input size of CLIP, we upsample images of CIFAR using crrchhnnnfunciiona. interpooaaattteeolaatteee in PyTorch. For WebVision, the images are initially resized uniform size of 256. Subsequently, standard data augmentation techniques are applied, involving the random cropping of patches with dimensions of 224 x 224 from each image, followed by the application of horizontal random flipping.
14

---

Table Epochs required reach a target test accuracy on the 100% Web training set (final accuracy is reported in parentheses).
Dataset
Validation set
Target Acc
50% data
100% data
30%
25
12
WebVision
40%
40 (60%)
22 (68%)
Web Vibioonn-
30%
33
18
ILSVRC12
40%
50 (55%)
27 (64%)
30%
22
11
WesVision
40%
36 (61%)
18 (68%)
WebVision-222
30%
29
16
ILSVRC12
40%
46 (56%)
22 (64%)
Table 7: The results of RHO-LOSS 31 using zero-shot predictor as the validation model on CIFAR-100.
Target Acc
Method
Final Ace (%)
40.0%
52.5%
RHO-LOSS w/ zero-shot predictor (CLIP-RN50)
59
92
58
RHO-LOSS w/ zero-shot predictor (CLIP-ViT-BAL
53
86
60
Proposed
32
53
63
B.2
Hyper-parameters The uning
We split the original training set into training and validation sets, where the latter remains clean and balanced for hyper-parameters tuning. In fact, as shown in Figure the trade-off coefficient a in the selection objective is the primary factor that impacts the training curve and should be carefully selected. In particular, we select from {0.1, 0.2, 0.3, 0.4] using a small validation set (of size 500 on CIFAR). We reuse the selected Web VisiVision- 100 without tuning.
B.3 Network
For all experiments, ResNet18 models are trained from scratch using PyTorch 2.0.0. Notably, in the case of CIFAR-10/100, employ downsampling layer with small convolution with 3 x 3 kernel. Additionally, the average pooling at the end of the network removed. Following the set-up of BatchNorm in 31, we compute the BatchNorm statistics on large batch B, for data selection. For model parameters update, we compute the statistics on the selected batch b,.
C More Results
Table reports the results of method the entire training of WisVis (as mentioned in Section E we use only half of the training set for training in the main experiments for a fair comparison with RHO-LOSS 31). We see more significant speedups and higher final accuracy due to more training data.
We establish extra baseline that combines RHO-LOSS with zero-shot predictors. The corre- sponding results based on our codebase are listed Table 7 These indicate that the Bayesian treatment introduced by method plays important role in accelerating and improving the model convergence.
15


Reviews:
[{"review_id": "vptc23ohVF", "reviewer": "Reviewer_agRJ", "summary": "This works is situated in the field of **robust generalization**; in particular it studies the problem of how models that were trained on noisy or imbalanced data perform on clean data. They achieve this via **online batch selection**, and in particular they contribute to the development of a **Bayesian framework** for searching for the best datapoints to use in each mini-batch among some canditate points sampled from the training set. In more details, they extend a SotA method, called **RHO-LOSS** [1]. The authors pinpoint some approximation choices that [1] makes in order to make the principled Bayesian approach to data selection practical. As a result, they design an algorithm which, in contrast to [1], they claim 1) to make **less crude approximations about predictive posteriors**, by using variational Laplacian approximations, MC sampling (and other) instead of pure point estimates, and 2) **not to rely on the existence of oracle clean data** during the training of their model. Their experiments demonstrate **improvements in robust generalization and training acceleration for image classification tasks** over other methods in the literature (incl. [1]).\n\n[1] Mindermann, Sören, et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" International Conference on Machine Learning. PMLR, 2022.", "questions": "### Questions\n\n* (related to weakness 2) How was model selection performed?\n* How were target accuracies chosen for training acceleration results?\n* Lines 151-152: What do we miss by replacing the Hessian matrix in the Laplacian approximation by the Gauss-Newton?\n* Lines 131-132 about “the recent trend of exploring the potential of pretrained models” (in robust generalization) needs some citations.\n* What does Table 4 in the Appendix refer to? Please complete the appendix to explain.\n\n### Typos\n\n* y-axis on Fig.2,3 and 4 are not in % of test accuracy as indicated.", "limitations": "The authors derive a novel approximation scheme to Bayesian data selection for training robust models (Strength 2), with favorable results over past literature in image classifiacation tasks (Strength 3) and the writing is overall great (Strength 1). There is a concern about the applicability of the method in cases where a pretrained model does not exist for the task at hand, in which case we need to find another proxy or probably pretrain it (Weakness 1), and how much actually the method makes no use of priviledged clean/balanced datasets for training (Weakness 1) and validation (Weakness 2). The limitation about the pretrained model is mentioned briefly, but it would be appreciated if it is expanded and experimented on with the ablation studies suggested in the Weakness section above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors contribute to an important direction for robust training methods, in particular developping a bit further the Bayesian data selection framework.\n1.  The paper is well-written and motivated. The derivations are easy to follow and the deferrals to the appendix are used appropriately. Presentation can further be improved if there was somewhere a paragraph (perhaps in the appendix) with the list of approximation steps taken (for future reference), e.g. MC sample of a variational posterior, based on Laplace approximation of the last layer of a neural network (for which the backbone is taken as a point estimate), and the Laplace approximation further does not use the true Gaussian, but a Gauss-Newton matrix + potentially a KFAC approximation of it.\n2.   Authors identify correctly potential improvement points of [1], and proceed in mitigating them. In this way, they introduce a novel approximation scheme to the Bayesian framework of data selection, which is effective and computationally-lightweight.\n3.   Experiments seem to advocate in favor of using their method over [1] or other methods, they are extensive and on various dataset scales in image classification tasks.", "weaknesses": "1.   Proposed method overly claims that it does not use oracle clean data, however it might still depend on clean data via the unsupervised pre-trained zero-shot classification proxy that they use. In particular, CLIP-R50 might have been trained in a superset of datasets (like ImageNet) which contains CIFAR10/100. In that case, information from oracle clean data has been stored in the pretrained model. While I believe that this is a weaker assumption, there needs to be an explicit statement of this assumption and a quantitative assessment of this potential pitfall. To which extent would a pretrained model on the same noisy/imbalanced dataset as the benchmark task be useful? Nonetheless, the authors acknowledge this potential limitation of an underperforming pretrained model at the last section. The following (lightweight) experiments can be further performed to augment their arguments or awareness of the limitation:  \n\n -  A zero-shot CLIP baseline is provided, I think they should also provide with a finetuned version using linear probing and training on imbalanced/noisy CIFAR10/100 with uniform sampling. This way it will be more clear that the improvement are more due to their method and not because of an overpowered pretrained model.\n -  Consider CIFAR10/100 experiments using an unsupervised pretrained model, with linear probing or kNN (so that it is zero-shot if you want that) for the oracle model, via for instance some simple SSL method like MoCov2 [2] on the same noisy/imbalanced training set.\n\nI will increase the assessed score if such experiments are performed and reported.\n\n2.   Ablation study reveals sensitivity to some hyperparameters. As a result, model selection and hyperparameter tuning are important for the success of the method. How was model selection performed? Was the validation split iid to the training set or was it a clean/balanced dataset? This needs to be reported clearly, but not addressed in this paper.\n\n3.   It would be nice to have a more extensive ablation study on the effects of the approximation choices in terms of final test accuracy, spanning from crude approximation schemes to the one finally used by the authors.\n\n[2] Chen, Xinlei, et al. \"Improved baselines with momentum contrastive learning.\" arXiv preprint arXiv:2003.04297 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive review and recognition of the presentation, novelty, and effectiveness of this work. Below, we provide answers to the specific questions raised.\n\n**Q1: To what extent does the effectiveness of this work stem from the effectiveness of the CLIP-based zero-shot predictor that was used?**\n\n**A1:** Thanks for the comments and suggestions. We make the following clarification.\n\nFirstly, we clarify that the zero-shot predictor used in our work contains valuable information regarding CIFAR while *not* being trained on a superset of CIFAR like ImageNet. Specifically, CLIP has been trained on a web-scale collection of image-text pairs using an *aligning loss*. Secondly, we emphasize that large-scale pre-trained models like CLIP, BLIP, SWAG, and others have become essential infrastructure for modern AI. These models are openly available, freely accessible, and generally applicable. In this regard, our method is more favorable than RHO-Loss [31].\n\nAs mentioned in our paper and noticed by the reviewer, our method does not hinge on a performant zero-shot predictor. For example, in Tab. 1, a CLIP zero-shot predictor with 75.6% accuracy on CIFAR-10 leads to a final accuracy of 91% for our method. As shown in Fig. 3, the speedup effect of CLIP-RN50 is similar to that of CLIP-ViT-B/16 on CIFAR-100, while the former is significantly weaker. \n\nWe comment that we use the zero-shot predictors directly without tuning in all the experiments reported in our paper. We conduct a set of experiments following the reviewer’s suggestion—linear probing using CLIP models on the clean/noisy CIFAR-10/100 with uniform sampling. The table below displays the comparison of the final accuracy (the comparison on convergence speed is not sensible because this baseline uses pre-trained weights while the methods listed in Table 1&2 have not):\n\n|   Method/dataset                                   | CIFAR-10 | CIFAR-10* | CIFAR-100 | CIFAR-100* |\n| ------------------------------------ | -------- | --------- | --------- | ---------- |\n| Linear probing with uniform sampling | 84.5     | 84.1      | 58.5      | 57.8       |\n| Proposed                             | 91.4     | 91.3      | 63.3      | 61.4       |\n\n We can see that the proposed method outperforms linear probing using CLIP on both clean (CIFAR-10/100) and data with 10% symmetric label noise (CIFAR-10*/100*).\n\nWe also add a new experiment where we replace the zero-shot predictor in our method with the validation model used in RHO-Loss. The results on CIFAR-100 are listed below:\n\n|Method          | Epochs to reach  40.0% |Epochs to reach  52.5% |Final acc. |\n| -------- | -------- |-------- |-------- |\n| RHO-Loss|    48  |     77|      61|\n| Proposed - zero-shot predictor + validation model from RHO-Loss  |   30    |  52    |   63   |\n| Proposed |    32   |     53|      63|\n\nThe above comparison confirms the necessity of introducing the Bayesian treatment and highlights the superiority of our method over RHO-Loss. \n\n**Q2: How was model selection performed?**\n\n**A2:** We split the original training set into training and validation sets, where the latter remains clean and balanced for model selection. \nIn fact, as shown in Figure 4, the trade-off coefficient $\\alpha$ in the selection objective is the primary factor that impacts the training curve and should be carefully selected. In particular, we select it from $\\{0.1, 0.2, 0.3, 0.4\\}$ (see L207) using a small validation set (of size 500 on CIFAR). We reuse the selected $\\alpha$ on WebVision-100 without tuning.  We’ll make these points clearer.  \n\n**Q3: A more extensive ablation study on the effects of the approximation choices in terms of final test accuracy**\n\n**A3:** Thanks. We will add a more in-depth empirical analysis of our method in the revision. The core of our method lies in the computation of $\\log p(y|x, D^∗, D_{t−1}) - \\log p(y|x, D_{t−1})$. In particular, we approximate $\\log p(y|x, D^∗, D_{t−1})$ with its lower bound for tractability, and approximate the predictive built on validation data with zero-shot predictors. These two choices are our major technical contributions and differences from RHO-LOSS [31]. The former cannot be trivially ablated and the latter has been well studied by us. For Bayesian treatment, we deploy online Laplace approximation for the posterior update, GGN approximation to avoid ill-posed Hessian, and last-layer KFAC approximation for tractability. These choices are essential. For example, introducing variational inference or MCMC for posterior inference will cause substantially higher costs and implementation challenges. \n\nIn fact, RHO-LOSS [31] is a counterpart of our approach that uses less principled and less reliable approximations. This is verified by our existing experiments.\n\n**Q4: How were target accuracies chosen for training acceleration results?**\n\n**A4:** Those in regular and noisy-label experiments follow RHO-LOSS [31]. Those in imbalanced experiments are selected according to the final accuracies achieved by our method and the baselines. \n\n**Q5: What do we miss by replacing the Hessian with Gauss-Newton?**\n\n**A5:** While the theory advocates using Hessian for Laplace approximation, practical experience suggests substituting it with the Gauss-Newton matrix [8, 37]. Dosing so, the information in the probability curvature mostly remains. Besides, the Gauss-Newton matrix is positive semi-definite and can be more easily manipulated in practice.\n\n**Q6: Missed citation in L131**\n\n**A6:** Thanks for the suggestion. We’ll fix this issue in the next version.\n\n**Q7: What does Table 4 in the Appendix refer to?**\n\n**A7:** It reports the results of our method on the *entire* training set of WebVision (see L195: we use only half of the training set for training in the main experiments for a fair comparison with RHO-LOSS [31]). We see more significant speedups and higher final accuracy due to more training data. \n\n"}, {"review_id": "UehWWLLuQq", "reviewer": "Reviewer_GG6S", "summary": "This paper studies data selection methods because training examples may be of different importance/quality. By selecting a subset of high-quality/high-usefulness examples, the model’s performance can be improved when training on this subset. In Particular, the paper proposed a method by leveraging a light-weight Bayesian treatment and incorporating off-the-shelf zero-shot predictors.", "questions": "Please see my comments in Weaknesses", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly motivated.\n2. The proposed method is well explained.\n3. The experiments cover different baselines, models and datasets.\n", "weaknesses": "The problem of data selection has been studied extensively in the ML community, which verifies the importance of this problem. Some recent works ([1,2] from reference list below) proposed data selection methods that do not only focus on hard/easy examples, but also consider data distribution and its impact on the overall loss (e.g., see Theorem 1 from [2]). It looks like these methods may already have partially addressed the problem studied in this paper. It would be better if the authors could include some discussion about these methods and possibly compare against them in the experiments. \n\nReference:    \n[1] Xia, Xiaobo, et al. \"Moderate coreset: A universal method of data selection for real-world data-efficient deep learning.\" ICLR 2023.    \n[2] Zheng, Haizhong, et al. \"Coverage-centric Coreset Selection for High Pruning Rates.\" ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your supportive reviews and for finding our work clearly motivated and well-explained. We address the detailed concerns below. \n\n**Q1: Some discussion and comparison with recent works**\n\n**A1:** Thanks for the constructive suggestion. The problem of data selection is indeed important and well-studied in the ML community, but we clarify that our method makes novel, valuable contributions. In particular, the mentioned papers operate in a different setting from our work: they focus on coreset selection while we perform the online batch selection. More specifically, reference [1] contributes a data scoring mechanism that is robust to the change of scenarios for coreset selection, and reference [2] makes an in-depth understanding of the catastrophic accuracy drop issue of one-shot coreset selection and contributes a novel solution to it. Compared to them, our method selects valuable samples based on their marginal influence on the model’s generalization loss during training, so samples with different properties can be selected at various training stages. As a result, we can boost not only the final accuracy but also the training speed. This can be a major difference. Moreover, our selection is dependent on the model in training, while the mentioned two papers are not, so these approaches are expected to handle various scenarios in practice. Anyway, we will add more thorough discussions of these works in the revision and attempt to include direct empirical comparisons.\n"}, {"review_id": "WDpIKPVtJw", "reviewer": "Reviewer_sAWs", "summary": "The paper builds on recently proposed work that accelerates training through online batch selection using generalisation loss as selection criterion, by using LaPlace approximation for a stronger Bayesian approximation and using off-the-shelf pre-trained models. The paper presents theory deriving their selection function and then demonstrating its efficacy on a number of datasets in the vision setting.", "questions": "As per above.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty on top of RHO-Loss: (1) using pre-trained models as validation model, (2) LaPlace approximation for stronger Bayesian approximation than point-estimate models.\n- Well-written paper, that is clean and easy to read.\n- Good ablations for how to tune the introduced zero-shot predictor as a validation model.\n- Decent performance gains over baselines.", "weaknesses": "- The novelty over RHO-Loss is appreciated, but it should be corrected in the narrative that RHO-Loss does not require access to clean holdout data. This is incorrect as far as I can tell, and Mindermann et al. note this in their paper.\n- Would be nice to see additional datasets and settings as per the original paper (Language and Clothing1M, however WebVision seems to be a bigger/better dataset)\n- In the practical acceleration/algorithm section they note that it's not much slower than [31]. I think a more empirical measurement of the difference would be appreciated, further more how much time does this method add over normal training? Does it really accelerate training in terms of time?\n- It's not clear whether the zero-shot predictor or the laplace approximation is providing the gain? An ablation here seems necessary. If it's just the zero-shot predictor, it's seems like there's not much of a difference here besides slapping a pre-trained model on as a validation model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the acknowledgment of the novelty of our method in comparison to RHO-Loss, as well as its good performance. Below, we provide detailed responses to the specific comments, hoping that you find them satisfactory and raise your score accordingly.\n\n**Q1: RHO-Loss does not require access to clean holdout data**\n\n**A1:** We apologize for the lack of rigor in our previous statements. We have incorrectly understood that the validation model in RHO-Loss relies on *clean* holdout data (especially in the scenarios with label noise). We sincerely appreciate the reviewer for bringing this to our attention, and we will make the necessary revisions to our paper accordingly.\n\nThe main text of [31] presents that RHO-Loss requires training validation models on a separate set of holdout data. We also note that RHO-Loss can work by splitting the training set into two halves and training a validation model on each half to score samples in another half. However, we argue that training additional validation models can be costly and should be performed repeatedly for new tasks.\n\nWe assure you that the majority of the arguments presented in our original manuscript remain valid, and will add the above clarifications in the next version.\n\n**Q2: Additional datasets and settings as per the original paper ((Language and Clothing1M)**\n\n**A2:** As mentioned by the reviewer, the used WebVision dataset is more challenging than Clothing1M, so the current empirical study in this regard is convincing. On the other hand, we acknowledge that tasks such as CoLA and SST-2 are relatively straightforward in the field of NLP. Considering recent advancements, such as the GPT and LLaMA series of models for zero-shot recognition, we have not yet conducted experiments on these tasks. We understand that including empirical studies on these tasks is important, so we plan to include them in our final version due to the time constraints during the rebuttal period. Furthermore, we want to emphasize that our current empirical evaluations strongly support the superiority of our method over RHO-Loss.\n\n**Q3: How much time does this method add over normal training? Does it really accelerate training in terms of time?**\n\n**A3:** Thanks for the advice. The wall clock time is indeed an important evaluation metric. Here we provide a direct comparison of per-epoch training time between RHO-LOSS and our method on CIFAR-100:\n\n|Method          | Per-epoch training time |\n| -------- | -------- |\n| Uniform  | 14s       |\n| RHO-LOSS  | 18s       |\n| Proposed | 21s       |\n\n\nAs shown, our approach has only slightly increased per-epoch time over RHO-Loss. It arises from that we use a CLIP-ResNet50 zero-shot predictor to compute the second term in Eq 16, whereas [31] uses a ResNet18 validation model. In fact, this gap can be reduced by a simple implementation trick—pre-computing the CLIP-ResNet50/ResNet18 predictions for each sample in the training set before training (this is done in [31] but not employed in our reproduction). \n\nAccording to Table 1, compared to RHO-Loss, we can reduce the number of epochs required to reach 40% accuracy from 48 to 32, and for 52.5% from 77 to 53. Combining these results, we see a $48 * 18 / 32 / 21 = 1.29$ or $77 * 18 / 53 / 21=1.25$ times practical acceleration.\n\n**Q4: It's not clear whether the zero-shot predictor or the Laplace approximation is providing the gain**\n\n**A4:** Thanks for the constructive comment. We clarify that both the zero-shot predictor and the Bayesian treatment contribute to the success of our method. These factors are integrated into Eq 9 and are traded off by the coefficient $\\alpha$. Comparing the curves corresponding to $\\alpha=0$ and $\\alpha=0.2$ in Figure 4a, we witness that using only the zero-shot predictor is suboptimal and would lead to degraded final accuracy (as stated in L275). On the other hand, too large $\\alpha$ (i.e., lightening the second term in Eq 9) can lead to a significant drop in training speed and final performance, which emphasizes the importance of the zero-shot predictor. \n\nWe have also ablated on the zero-shot predictor in Figure 3, which proves the robustness of our method against it. \n\nWe further try to replace the zero-shot predictor in our method with the validation model in RHO-Loss. The results on CIFAR-100 are listed below:\n\n|Method          | Epochs to reach  40.0% |Epochs to reach  52.5% |Final acc. |\n| -------- | -------- |-------- |-------- |\n| RHO-Loss|    48  |     77|      61|\n| Proposed - zero-shot predictor + validation model from RHO-Loss  |   30    |  52    |   63   |\n| Proposed |    32   |     53|      63|\n\nThe above comparisons with RHO-Loss confirm the necessity of introducing the Bayesian treatment. \n"}, {"review_id": "j0yL6xxlHu", "reviewer": "Reviewer_UrxY", "summary": "This paper followed up reducible hold-out loss selection (RHO-LOSS) method for data selection and improved it by proposing a reasonable approximation for the non-trivial objective function and eliminating the need for extra hold-out data.", "questions": "In addition to questions addressed in weakness section, I have following questions for the authors:\n1. How is target accuracy selected? Specifically for table 3, why select 30% and 40% as target accuracy?\n2. I am curious if each training epoch for RHO-LOSS and your proposed method take same amount of time considering they have different data selection steps?\n3. It seems the proposed method performs well on classification datasets with large number of classes (CIFAR 100, ILSVRC12 and WebVision datasets) compared with dataset with smaller number of classes (CIFAR 10). I wonder authors can share some ideas from the perspective of the designed algorithm about this.\n4. I wonder if the authors have look into how zero-shot predictor affects the method performance. In other word, is the proposed method robust to poorly-performed predictor? If not, how much will the model be impacted?\n", "limitations": "Yes, authors have addressed their limitation, namely the method performance hugely depends on effectiveness of the zero-shot predictor.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors present both theoretical results (the lower bound for hard-to-estimate objective) and empirical results (algorithm and experiment results).", "weaknesses": "1. I am not fully convinced by the paper that their proposed method out-performs existed SOTA, namely RHO-LOSS much, specifically from the results in table 1 and figure 1. I wonder if authors can provide more intuitive comparisons between the proposed method the RHO-LOSS, either empirically show effectiveness and efficiency of the proposed method, or theoretically show the proposed lower bound is more solid than approximation used by RHO-LOSS.\n2. The ablation study needs more experiment result (more datasets and more model structure) to get valid conclusion. I also wonder why compare with baseline instead of RHO-LOSS.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for providing valuable comments. Below, we address each concern in detail, and we sincerely hope that our response proves satisfactory and leads to a higher score.\n\n**Q1: More intuitive comparisons between the proposed method and RHO-LOSS**\n\n**A1:** We first clarify that theoretically, our method builds a more reliable approximation to the selection objective in Eq 4 than RHO-Loss. In particular, RHO-LOSS proposes approximating $\\log p(y|x, D^∗, D_{t−1})$ with $\\log p(y|x, D^∗)$ and approximating the posterior predictive with the point-estimate models’ predictions, which are both less principled. In comparison, we maintain a Bayesian perspective and develop a lower bound, i.e., Eq 9. Our selection objective is intriguing for several reasons: 1) the first term and $\\alpha$ times the third term perform expectation and logarithm operations in reverse order, resulting in a quantity similar to the uncertainty estimates in Bayesian deep learning [39]; 2) the third term helps to prevent the selection of redundant samples; 3) the second term prioritizes points with high semantic alignment with their annotations since the validation data follows the ground truth data generating distribution. These three forces are integrated adaptively to accelerate model training across various stages.\n\nIn the aspect of empirical comparison, apart from the results reported in Tab. 1-3, the results in Fig. 1 are insightful. Specifically, Fig. 1a evidences that data selected by our method has a lower label noise. Fig. 1b shows that our method selects slightly fewer redundant data points than RHO-LOSS.\n\nWe will add these clarifications to the revision and respectfully ask the reviewer to re-evaluate our paper. \n\n**Q2: Regarding ablation study**\n\n**A2:** We provide clarification to the raised concerns. Firstly, our experiments in sections 4.1-4.4 have already demonstrated the effectiveness of our method across various datasets, including normal, noised, imbalanced, and web-scraped data. Therefore, our ablation studies do not specifically focus on this aspect. Secondly, we have conducted ablation studies on both the architecture of the trained model (Fig. 2) and the architecture of the zero-shot predictor (Fig. 3). These ablation studies have already established the generality and versatility of our method.\n\nSince we have already validated the superiority of our method over RHO-Loss in the main experiments and the purpose of the ablation studies is to gain a better understanding of the behavior of our approach, there is no need to compare with RHO-Loss in the ablation. \n\nTo summarize, our empirical evaluations and ablation studies provide substantial evidence to support the effectiveness of our method, so we respectfully disagree with the assessment that our current conclusion lacks proper support. \n\n**Q3: How is target accuracy selected?**\n\n**A3:** We follow RHO-LOSS [31] to select the target accuracy on CIFAR. In the case of WebVision, which is not covered by [31], we select 30% and 40% according to the final accuracies of RHO-Loss ($\\approx$ 40%) and those of our approach ($\\approx$ 60%).\n\n**Q4: Per-epoch training time**\n\n**A4:** Thanks for the suggestion. Here we provide a direct comparison of per-epoch training time between RHO-LOSS and our method on CIFAR-100:\n\n|Method          | Per-epoch training time |\n| -------- | -------- |\n| Uniform  | 14s       |\n| RHO-LOSS  | 18s       |\n| Proposed | 21s       |\n\nAs shown, our approach has only slightly increased per-epoch time over RHO-Loss. It arises from that we use a CLIP-ResNet50 zero-shot predictor to compute the second term in Eq 16, whereas [31] uses a ResNet18 validation model. In fact, this gap can be reduced by a simple implementation trick—pre-computing the CLIP-ResNet50/ResNet18 predictions for each sample in the training set before training (this is done in [31] but not employed in our reproduction). \n\nAccording to Tab. 1, compared to RHO-Loss, we can reduce the number of epochs required to reach 40% accuracy from 48 to 32, and for 52.5% from 77 to 53. Combining these results, we see a $48 * 18 / 32 / 21 = 1.29$ or $77 * 18 / 53 / 21=1.25$ times practical acceleration.\n\n**Q5: Clearer performance gain on datasets with a larger number of classes**\n\n**A5:** This is because the difficulty of the classification problem is directly correlated with the number of classes involved. This is evident in the final accuracy, where achieving approximately 90% accuracy on CIFAR-10 is relatively straightforward while obtaining 60-70% accuracy on the other datasets poses a greater challenge. So, the conclusion is that our method surpasses the baselines more obviously on more difficult problems. The WebVision dataset setting closely resembles real-world scenarios, and the substantial performance improvement demonstrated by our method over the baselines on it validates our practical value.\n\n**Q6: Is the proposed method robust to poorly-performed predictor?**\n\n**A6:** Yes. As validated in Fig. 3 and stated in L267-269, although the zero-shot accuracy of the CLIP-RN50 variant is significantly lower than that of CLIP-ViT-B/16, the speedup effect of the two zero-shot predictors is similar. Note that CLIP-RN50 is the smallest model in the CLIP family, while CLIP-ViT-B/16 is rather large. This finding suggests the robustness of our method against the selection of the zero-shot predictor. \n\nWe further try to replace the zero-shot predictor in our method with the validation model in RHO-Loss. The results on CIFAR-100 are listed below:\n\n|Method          | Epochs to reach  40.0% |Epochs to reach  52.5% |Final acc. |\n| -------- | -------- |-------- |-------- |\n| RHO-Loss|    48  |     77|      61|\n| Proposed - zero-shot predictor + validation model from RHO-Loss  |   30    |  52    |   63   |\n| Proposed |    32   |     53|      63|\n\nThe above comparison double confirms the robustness of our method and directly proves the Bayesian treatment in our method is beneficial.\n\n"}]

Author rebuttal:

Meta-review: The work provides a Bayesian framework for online batch selection optimizing for generalization loss. The authors improve upon previous works by using a more precise approximation and using less assumptions. The reviews agree that the problem is interesting, and that the provided technique is convincing both in terms of having theoretical backing and in terms of having thorough experiments showcasing its advantage over previous art. Some issues around clarity were raised by the reviews but they were resolved in the discussion phase. I think the paper will be a great addition to Neurips and urge the authors to carefully go through the reviewer comments and add the necessary changes towards the paper’s final version.
Decision: Accept (poster)