Title: Towards Accelerated Model Training via Bayesian Data Selection
Venue: NeurIPS 2023
Accepted: yes
Keywords: data selection

training acceleration

probabilistic modeling

Bayesian methods
Abstract: Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.

Towards Accelerated Model Training via Bayesian Data Selection

Abstract

Mislabeled, duplicated, or biased data in real-world scenarios can lead to prolonged training and even hinder model convergence. Traditional solutions prioritizing easy or hard samples lack the flexibility to handle such a variety simultaneously. Recent work has proposed a more reasonable data selection principle by examining the data's impact on the model's generalization loss. However, its practical adoption relies on less principled approximations and additional holdout data. This work solves these problems by leveraging a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models. The resulting algorithm is efficient and easy to implement. We perform extensive empirical studies on challenging benchmarks with considerable data noise and imbalance in the online batch selection scenario, and observe superior training efficiency over competitive baselines. Notably, on the challenging WebVision benchmark, our method can achieve similar predictive performance with significantly fewer training iterations than leading data selection methods.

1 Introduction

The past year has witnessed significant breakthroughs in deep learning research and applications, with Stable Diffusion [38], ChatGPT [32], and SAM [22] as representative examples. Practitioners have realized that the quality of data used to fuel AI systems is critical in unlocking their full potential. Unfortunately, real-world scenarios often present mislabeled, duplicated, or biased data. As a result, it is paramount to develop methods that can prioritize valuable training data to enable more efficient model training and even improved model convergence.

Data selection for accelerating the training of deep models is gaining increasing interest. Some studies, such as curriculum learning, advocate prioritizing easy samples in the early training stages [1], but these samples quickly become redundant once they have been learned, making continued training on them a waste of time. On the other hand, online batch selection methods [26, 19, 17] prioritize hard samples with high training loss or gradient norm to avoid duplicate training. Nevertheless, in practice, the hardness of samples often arises from pathologies such as improper annotations, inherent ambiguity, or unusual patterns, rendering it problematic to prioritize such samples [31].

The reducible hold-out loss selection (RHO-LOSS) approach [31] addresses these issues by quantifying the usefulness of a sample based on its marginal influence on the model's generalization loss, forming a theoretically grounded and universal objective for data selection. However, the estimation of this objective is non-trivial, and RHO-LOSS has to rely on less principled approximations for

practical adoption. Besides, RHO-LOSS hinges on a considerable number of holdout data to train an auxiliary validation model, which can be costly and should be performed repeatedly for new tasks.

This paper aims to bridge this gap to make the generalization loss-based data selection principle more accessible to a broader audience. We establish a more reasonable approximation of the original objective than RHO-LOSS while eliminating the need for holdout data. To achieve this, we derive a lower bound of the objective to separate the posterior predictive defined on the training data from that defined on the holdout data. Afterward, we propose to use off-the-shelf zero-shot predictors, built upon large-scale pre-trained models [35, 41], as a proxy for the latter, since these models often contain generally applicable information for solving specific downstream tasks.

We maintain a Bayesian treatment of the training model to ensure an accurate estimation of the original objective. Bearing in mind that our original goal is to accelerate the training of a deterministic model, we adopt the simple and effective Laplace approximation [29, 37, 8] for Bayesian inference. It effortlessly converts point-estimate parameters to a Gaussian posterior. We further introduce Kronecker-factored (KFAC) [30] and last-layer [23] approximations to accelerate the processing of modern neural networks (NNs), resulting in an efficient and easy-to-implement algorithm.

We conduct comprehensive empirical studies on various benchmarks to evaluate the effectiveness of our method. The experiments on standard image recognition tasks demonstrate that our approach can outperform various baselines in aspects of training speed and final accuracy. This conclusion also holds for learning with label noise and class imbalance. On the challenging WebVision [25] dataset, which contains plenty of noisy labels and ambiguous images collected from the internet, our method significantly reduces the number of training steps needed to reach the target accuracy and achieves up to 19% higher final accuracy than prior arts (see Table 3). These results highlight the practical value of our approach. In addition, we conduct informative ablation studies to gain a better understanding of the behavior of our method.

2 Background

In this section, we briefly revisit the concept of online batch selection [26] and the data selection principle defined with the model's generalization loss in [31].

Consider training a θ -parameterized deep model f θ : X → R k on a dataset D = { ( x i , y i ) } n i =1 using stochastic gradient descent (SGD). The model likelihood is formally p ( y | x, θ ) = p ( y | f θ ( x )) . At each training step t , we can access a data batch B t of size n B from D . In online batch selection, we need to compute statistics of the samples in B t and select only those that meet certain requirements to update the model. This filtering process aims to remove samples that are deemed less valuable.

Let D t -1 denote the set of data observed before t and ( x ′ , y ′ ) a sample from B t . If we accept selecting ( x ′ , y ′ ) , the updated predictive distribution, in a Bayesian view, will be p ( y | x, D t -1 , { ( x ′ , y ′ ) } ) = E p ( θ |D t -1 , { ( x ′ ,y ′ ) } ) p ( y | x, θ ) . 2 The question arises how to estimate the quality of this distribution to determine which sample ( x ′ , y ′ ) ∈ B t should be selected. A natural tactic is to compare this distribution to the ground truth data-generating distribution ˙ p ( x, y ) . That said, the typical KL divergence can be introduced, and our goal becomes solving the following problem:

where const . denotes a constant agnostic to the object to optimize. By applying Monte Carlo (MC) estimation using extra holdout samples D ∗ = { (˜ x i , ˜ y i ) } i m =1 from ˙ p ( x, y ) , we arrive at the following optimization problem:

This objective corresponds to the model's generalization loss instead of the fitness of training data. By Bayes' rule, there is:

where the item p ( D ∗ |D t -1 , x ′ ) actually equals to the constant p ( D ∗ |D t -1 ) because x ′ alone cannot incur model update. Plugging this back into Equation (2), we arrive at the final objective for data selection:

where we omit constants and abuse ( x, y ) to represent ( x ′ , y ′ ) hereinafter if there is no misleading.

Although the above selection principle is theoretically sound and universally applicable, estimating it accurately is challenging. In particular, it is difficult to estimate the posterior predictive distribution defined with the combination of training and holdout data in a computationally efficient manner. To address this issue, RHO-LOSS proposes approximating log p ( y | x, D ∗ , D t -1 ) with log p ( y | x, D ∗ ) and approximating the posterior predictive with the point-estimate model' predictions [31]. However, these approximations compromise the method's theoretical groundness, and access to holdout data can often be infeasible in practice. Our approach overcomes these limitations by utilizing a lightweight Bayesian treatment and incorporating off-the-shelf zero-shot predictors built on large-scale pre-trained models, as detailed below.

3 Methodology

In this section, we demonstrate that the data selection principle discussed earlier can be lower-bounded by more easily computable quantities. We then leverage these insights to develop a Bayesian data selection approach to accelerating the training of deterministic deep models.

3.1 The Lower Bound

As previously discussed, we need to estimate two log-probabilities, log p ( y | x, D ∗ , D t -1 ) and log p ( y | x, D t -1 ) , for each sample in B t to identify the most useful one and update the model accordingly. However, as mentioned, evaluating the former is challenging. To address this issue, we derive a lower bound that allows for a more feasible estimation.

We first unfold log p ( y | x, D ∗ , D t -1 ) as the combination of a posterior p ( θ |D ∗ , D t -1 ) and the model likelihood p ( y | x, θ ) , detailed below (we defer the derivation to Appendix):

Then, by Jensen's inequality, there is (derivation is deferred to Appendix)

Notably, the last two terms are independent of ( x, y ) and can be excluded from the optimization.

We can similarly derive another lower bound:

where the last two terms are also agnostic to ( x, y ) and hence can be omitted.

Combining Equation (6) and Equation (7), there is

where α ∈ [0 , 1] represents a trade-off coefficient. This way, we disentangle the posterior associated with the training data D t -1 from that associated with the holdout data D ∗ .

Rewriting p ( y | x, D t -1 ) as E p ( θ |D t -1 ) p ( y | x, θ ) , we can subsequently convert Equation (4) to the following maximization problem:

The presented objective is intriguing for several reasons. Firstly, the first term and α times the third term perform expectation and logarithm operations in reverse order, resulting in a quantity similar to

the uncertainty estimates in Bayesian deep learning [39] (nonetheless, our objective involves data labels). Additionally, the third term helps to prevent the selection of redundant samples. And, the second term prioritizes points with high semantic alignment with their annotations. These three forces are integrated adaptively to accelerate model training across various stages.

3.2 Zero-shot Predictor as the Validation Model

Collecting extra holdout data to train an auxiliary validation model can be costly and should be performed repeatedly for new tasks. To address this issue, we propose to use zero-shot predictors built upon large-scale pre-trained models [35, 41] as a proxy for the validation model, based on the observation that they usually exhibit promising transfer performance across a broad range of downstream applications.

Formally, we make the following approximation:

where ˜ f : X → R k denotes the zero-shot predictor used.

We can think of the pre-trained model as a universal validation model trained on an extensive dataset, leading to the Bayesian posterior collapsing to a point estimate. Although its training data may not precisely follow the data-generating distribution for the current task, they share fundamental patterns with the data in our problem, making the above approximation reasonable. Notably, as shown in Section 4, our trained model eventually performs much better than the zero-shot predictor.

3.3 Lightweight Bayesian Treatment of the Training Model

The first and third terms in Equation (9) raise the requirement of maintaining a Bayesian posterior p ( θ |D t -1 ) during training. Due to the high nonlinearity of deep NNs, the analytical posterior is intractable. By convention, we can introduce an approximate posterior q ( θ |D t -1 ) and tune it by approximate Bayesian inference methods like MCMC [42, 5, 45], variational inference [2, 15, 28, 44, 21], Laplace approximation [29, 37], etc. Recalling that, despite relying on a Bayesian treatment, our final goal is to accelerate and improve the training of a deterministic model, Laplace approximation is well suited to our setting-it can convert point-estimate parameters to a Gaussian posterior effortlessly, thus lifting the unnecessary burden of learning and maintaining a posterior explicitly. We clarify that, technically, other Bayesian inference methods are also compatible with our approach.

Although Laplace approximation is typically used for maximum a posteriori estimation, it can be effectively adapted to the online learning cases [36]. Specifically, consider an isotropic Gaussian prior with precision τ 0 over parameters θ . Let D t -1 := b 1 ∪ b 2 ∪ · · · ∪ b t -1 3 denote all selected training samples before time t . The parameters of our deterministic model at time t -1 , dubbed as θ t -1 , are likely to approach a mode of the true posterior p ( θ |D t -1 ) in the presence of a proper weight decay regularizer in stochastic optimization [10]. The online Laplace approximation then deploys the following approximate posterior:

The Gaussian covariance should be positive semi-definite, but as θ t -1 cannot be guaranteed to be exactly the posterior mode, we would better replace the Hessian matrix with the generalized Gauss-Newton (GGN) matrix to avoid ill-posed approximation, resulting in:

Practical acceleration. Modern neural networks usually contain millions of parameters, making the matrix G t -1 too large to fit into CPU/GPU memories. To address this, a common practice is to sparsify the matrix using diagonal or Kronecker-factored (KFAC) approximations [30]. This work

Algorithm 1 Bayesian data selection to accelerate the training of deterministic deep models.

1: Input: Number of iterations T , dataset D , prior precision τ 0 , number of effective data n e , batch size n B , number of selections n b , zero-shot predictor ˜ f , deterministic model with parameters θ .

2: Intialize θ 0 , A 0 ← 0 , G 0 ← 0 ;

3: for t in 1 , . . . , T do

4: Draw a mini-batch B t from D ; √

5: V t -1 ← √ n e A t -1 + τ 0 I , U t -1 ← √ n e G t -1 + √ τ 0 I ;

6: Estimate the objective in Equation (16) for every sample in B t and select the topn b ones to form b t ;

7: Perform back-propagation with ∑ x,y ∈ b t log p ( y | f θ t -1 ( x )) ;

8: Apply weight decay regularization and do gradient ascent to obtain θ t ;

9: Use the last-layer features and softmax gradients to update A t and G t with exponential moving average;

10: end for

prefers KFAC as it preserves the correlations between parameters within the same layers. Besides, a recent surprising finding is that we can even apply Laplace approximation to only the last layer of a deep NN, leaving the other parameters point-estimate, to conjoin efficiency and calibrated uncertainty estimates [23, 8]. As a result, we consider combining last-layer and KFAC approximations to reduce the burden caused by the Bayesian treatment, with the details presented below.

Let us decompose f θ i as a feature extractor h θ i and a linear layer with weights θ ( l ) i ∈ R d × k , i.e., f θ i ( x ) := h θ i ( x ) ⊤ θ ( l ) i . We adapt the GGN matrix associated with the last-layer weights θ ( l ) i derived in [37, 23] to the following formula:

where ⊗ denotes the Kronecker product and

Of note that the matrices A t -1 ∈ R d × d and G t -1 ∈ R k × k raise only minimal extra storage cost. The approximate posterior can be formulated as a matrix-variate Gaussian [12]: q ( θ ( l ) |D t -1 ) = MN ( θ ( l ) | θ ( l ) t -1 , U -1 t -1 , V -1 t -1 ) . Given the linear nature, it is straightforward to get the distribution over the model output f x for input x (derivation is deferred to Appendix):

The final data selection objective. With the above equation, the selection objective boils down to

where f ( s ) x ∼ q ( f x |D t -1 ) , s = 1 , . . . , S are MC samples. Compared to the non-last-layer Laplace approximation, which involves sampling over the parameter space and performing MC integration, the last-layer one enjoys a much faster evaluation procedure. It also enables the use of a large S . Our method can also abandon the KFAC approximation when the linear head is small.

The algorithm. We present the training procedure in Algorithm 1. To avoid too large |D t -1 | and hence too sharpened approximate posterior, we use a tunable hyper-parameter n e to replace |D t -1 | in the computation of V t -1 and U t -1 . For implemental simplicity, we use the exponential moving average technique to update A t and G t . For models equipped with batch normalization layers [16], we perform an extra forward propagation for the selected data b t to obtain proper batch statistics for model update, following [31]. Our method takes a similar amount of time per iteration as [31]. The primary difference is that we use a zero-shot predictor based on pre-trained models to compute the second term in Equation (16), whereas [31] uses a validation model. Computing the Gaussian covariance in Equation (15), MC sampling, and updating A t and G t consume neglectable resources.

4 Experiment

We compare the proposed method to the prior state-of-the-art (SOTA) data selection methods on various image classification benchmarks, including standard datasets (e.g., CIFAR-10 and CIFAR100 [24]), their variants with controllable degrees of label noise and class-imbalance, and a real-world, large-scale, noisy, and imbalanced dataset, WebVision [25]. We also diagnose our selection strategy through elaborate ablation studies.

Datasets. We first evaluate the proposed method on clean CIFAR-10/100 and noisy CIFAR-10/100 with 10% symmetric label noise. Furthermore, we experiment in the context of imbalanced datasets, specifically CIFAR-10/100 datasets with varying degrees of class-imbalance ratio. Last, we investigate a web-scraped and large-scale dataset - WebVision, which consists of over 2 . 5 million images in 1000 categories and suffers from severe label noise and class-imbalance issues. For a fair comparison with RHO-LOSS [31], only half of the training set is used for model training.

Baselines. We compare the proposed method to a variety of baselines that define selection principles with various metrics, including the (training) loss [20], gradient norm [19], and gradient norm with importance sampling (gradient norm IS) [19]. We also compare to uniform sampling, Selection-viaProxy (SVP) [6] and the latest SOTA method, RHO-LOSS [31].

Implementation details. In experiments under label noise, we introduce 10% symmetric noise caused by randomly flipping the ground-truth label to other labels. For the class-imbalance setting, we consider the long-tailed imbalance with two levels of imbalance intensity (i.e., 10 and 100), where an exponential decay on sample size is introduced across classes [3]. We use the same optimizer (AdamW [27]) and hyperparameters (e.g., learning rate 0.001, weight decay of 0.01, n b = 32 and n b n B = 0 . 1 ) as RHO-LOSS. Unless specified otherwise, we use ResNet18 (RN18) [14] as the deterministic model and specify the zero-shot predictor with CLIP-RN50. We select the trade-off coefficient α from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } and the number of effective data n e from { 100 , 200 , 500 , 1000 } . In most cases, we set the prior precision τ 0 to 1 . We report average results over 3 random runs.

Evaluation. Following [31], we evaluate speedup by comparing the number of epochs required to reach a given target test accuracy, which implicitly reflects the quality of the selection strategy.

4.1 Speedup on Clean Data and Noisy Data

We first empirically evaluate the proposed method on CIFAR-10 and CIFAR-100 with/without label noise. Table 1 reports the results. We can observe that the proposed method achieves considerably improved training speed compared to competing baselines in both scenarios with and without label noise. This evidences the efficacy of our Bayesian selection method in filtering out redundant and noisy data points in the training set. Despite leveraging extra holdout data, RHO-LOSS still

underperforms our approach, potentially due to the less reliable approximations. Surprisingly, the superiority of our method is even more significant on the more challenging CIFAR-100. We also point out that our method eventually reaches higher final accuracy than the zero-shot CLIP. In other words, our method does not hinge on a performant zero-shot predictor.

4.2 Speedup on Class-imbalance Data

We further evaluate the proposed method on class-imbalanced data, a typical pattern in the real world while probably raising non-trivial difficulties for model training [7, 3]. Specifically, we experiment on the aforementioned imbalanced CIFAR-10 and CIFAR-100 using similar protocols to previous studies. The test datasets remain unchanged. Given that RHO-LOSS is superior to other baselines, we primarily compare with it in the following studies. We also add uniform sampling into comparison due to its implementation simplicity. Table 2 presents the results.

As shown, the proposed method still consistently outperforms uniform sampling and RHO-LOSS. In particular, the final accuracy achieved by our method is higher than RHO-LOSS. We also note that the gain in training efficiency becomes more significant as the imbalance ratio increases. The failure of RHO-LOSS is partially attributed to the less principled approximations. Another reason is that the holdout data are still class-imbalanced, rendering the trained validation models biased.

4.3 Speedup on Large Web-scraped Data

Learning algorithms face significant challenges when dealing with noisy and imbalanced real-world datasets. To evaluate our proposed method's efficacy in addressing these challenges, we assess it on a large-scale web-scraped dataset, WebVision. We defer the details of WebVision to the Appendix. Since the dataset is large, comprising 2.4 million data points, we train classification models only on the first 100/200 classes (known as WebVision-100/200) of the entire dataset, following [18, 4], for an efficient evaluation. We test the trained models on the human-annotated WebVision validation set

CIFAR10

CIFAR100

0

2

4

6

8

10

Proportion of selected data

with label noise (%)

Proposed

RHO-LOSS

Uniform

(a) Proportion of label noise in selection.

CIFAR10

CIFAR100

0

10

20

30

40

50

Proportion of selected data

already classified correctly (%)

Proposed

RHO-LOSS

Uniform

(b) Proportion of redundant points in selection.

0

200

400

600

800

1000

Training iterations

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Test accuracy (%)

Proposed

Uniform

and the ILSVRC12 validation set [9]. We report the results in Table 3. We select 30% and 40% as target accuracies according to the final accuracies of RHO-LOSS ( ∼ 40%) and our approach ( ∼ 60%).

Still, the proposed method outperforms uniform sampling and RHO-LOSS in aspects of both the speed to reach the target accuracy and the final accuracy on both WebVision-100 and WebVision-200. Notably, our method achieves up to 19% higher final accuracy than RHO-LOSS, suggesting the superiority of our method in coping with real-world datasets. We also report results for our method on the entire training set in the Appendix, which shows more significant speedups and higher final accuracy due to more training data.

4.4 Analysis of the Properties of the Selected Data

In this section, we analyze the characteristics of the data selected by our method and compare it with existing works. Figure 1a displays the proportion of selected data points with label noise, and it is evident that data selected by our method has the lowest label noise. We also investigate whether our method prioritizes redundant data, defined as those having already been correctly classified by the training model [31]. As depicted in Figure 1b, both our method and RHO-LOSS select significantly fewer redundant data points than uniform sampling, and our method is slightly better.

4.5 Ablation Studies

In this section, we conduct ablation studies on model architecture, zero-shot predictor, and some critical hyper-parameters (e.g., α , n e ).

(a) The trade-off coefficient

α

.

0

25

50

75

100

125

150

175

200

Training epochs

0.1

0.2

0.3

0.4

0.5

0.6

Test accuracy (%)

ne

=100

ne

=200

ne

=500

ne

=1000

(b) The number of effective data

n

e

.

0

25

50

75

100

125

150

175

200

Training epochs

0.1

0.2

0.3

0.4

0.5

0.6

Test accuracy (%)

=10

=100

=1

=0.1

(c) The temperature

τ

.

Model architecture. We test our method using the powerful Vision Transformer (ViT) [11] architecture on WebVision-200. Specifically, we experiment with a ViT-B/16 model pre-trained unsupervisedly by Masked Auto-encoder [13], considering that fine-tuning a pre-trained model is preferred over training a model from scratch in practice. We provide the training curve of our method in Figure 2, where uniform sampling is included for comparison. The huge performance gap between our method and the baseline validates the efficacy of our method in this setting.

Zero-shot predictor. We next assess the impact of the zero-shot predictor on our method. We specify the zero-shot predictor with various CLIP variants (e.g., RN50 and ViT-B/16) and draw the training curves in Figure 3. As shown, although the zero-shot accuracy of the RN50 variant is significantly lower than that of ViT-B/16, the speedup effect of the two backbones is similar. This finding suggests the robustness of our method against the selection of the zero-shot predictor.

In the above experiments, we use CLIP-based zero-shot predictors without further tuning. Here, we perform linear probing using the CLIP models to demonstrate if we can achieve good convergence trivially. We simply adopt the uniform sampling strategy and report the results in Table 4. We do not report convergence speed because the baseline uses pre-trained weights while our method trains models from scratch. As shown, our method still bypasses the baseline with clear margins.

We replace the zero-shot predictor used in our method with the validation model in RHO-LOSS [31] and present the corresponding results in Table 5. The comparison proves the superiority of our method over RHO-LOSS. These results also reflect that our method is robust against the specification of the validation model.

Hyper-parameters. We then analyze the effects of three crucial hyper-parameters, α , n e , and the temperature τ used by the softmax operation in the CLIP zero-shot predictor. Figure 4a shows training curves associated with various α . We see that too large α (i.e., lightening the second term in Equation (9)) can lead to a significant drop in training speed and final performance, which emphasizes the importance of the zero-shot predictor. Nevertheless, using only the zero-shot predictor (i.e., α = 0 ) is also suboptimal, leading to degraded final accuracy. Hence, we conclude that both terms in the lower bound of the original data selection principle are beneficial, where the first term accounts more for the later training stages while the second term accounts more for the earlier ones. Figure 4b shows the ablations study on n e , and we witness the robustness of our method against n e . Figure 4c

shows training curves corresponding to different τ . We observe that an appropriate temperature boosts our method considerably.

Wall-clock time. We empirically observe that the per-epoch training time for Uniform, RHO-LOSS, and our method on CIFAR-100 is 14s, 18s, and 21s, respectively. Namely, our approach has slightly increased per-epoch time over RHO-LOSS. It arises from that we use a CLIP-RN50 zero-shot predictor to compute the second term in Equation (16), whereas [31] uses a RN18 validation model. In fact, this gap can be reduced by a simple implementation trick-pre-computing the CLIP-RN50 predictions for each sample in the training set before training.

As shown in Table 1, compared to RHO-LOSS, we can reduce the number of epochs required to reach 40% test accuracy from 48 to 32 and 52.5% from 77 to 53. According to the per-epoch training time, we achieve a 48 ∗ 18 / 32 / 21 ≈ 1 . 29 or 77 ∗ 18 / 53 / 21 ≈ 1 . 25 times practical acceleration.

5 Related Works

Extensive methods have been proposed to accelerate model training through techniques such as data pruning, coreset selection, curriculum learning, online batch selection, etc. Data pruning explores various metrics, such as EL2N score [33], forgetting score [40], and classification margin [34], to measure individual differences among data points and retains only the hardest examples for model training. However, data pruning still exhibits limitations when dealing with noisy labels, and some of these metrics are computationally expensive. Coreset selection methods also partially address the problem of accelerating model training. In particular, [43] contributes a data scoring mechanism that is robust to the change of scenarios for coreset selection, and [46] makes an in-depth understanding of the catastrophic accuracy drop issue of one-shot coreset selection and contributes a novel solution to it. However, these methods lack the flexibility to prioritize samples with different properties at various training stages. Curriculum learning, as advocated by [1], prioritizes easy points with low label noise before uniformly training on all data points. While this strategy enhances convergence, it fails to address the issue of skipping redundant points already learned.

Online batch selection methods [26, 19, 17] tackle the training acceleration problem by selecting hard data identified by high loss or gradient norm. However, they also suffer from a common drawback-high loss can be caused by label noise or ambiguous labels, so prioritizing such data can result in a decline in predictive performance. Compared to the prior art, our method establishes a Bayesian data selection metric and exploits zero-shot predictors to prioritize valuable training data for addressing these issues.

6 Conclusion

This paper addresses the challenges posed by noisy and biased data in real-world scenarios. Our main contribution is to make the generalization loss-based data selection principle more accessible to accelerate the training of deep models. To achieve this, we first derive a lower bound of this objective to improve its tractability. We then propose leveraging a Bayesian treatment and off-the-shelf zeroshot predictors to estimate the data selection objective reliably. The resulting algorithm does not require extra holdout data. Our extensive empirical studies demonstrate the superiority of our method in accelerating model training over competitive baselines.

Limitation. Our method may fail when the zero-shot predictor performs poorly on specific tasks. Future work could explore adapting the zero-shot predictor to a few-shot one using a few clean validation data to address this limitation.

Acknowledgments

Z.J. Deng was supported by Natural Science Foundation of Shanghai (No. 23ZR1428700) and the Key Research and Development Program of Shandong Province, China (No. 2023CXGC010112). J. Zhu and P. Cui were supported by NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); a grant from Tsinghua Institute for Guo Qiang; and the High Performance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.

References

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning , pages 1613-1622, 2015.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning , pages 1062-1070. PMLR, 2019.

Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference on Machine Learning , pages 1683-1691. PMLR, 2014.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations , 2020.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems , 34:20089-20103, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Zhijie Deng and Jun Zhu. Bayesadapter: Being bayesian, inexpensively and reliably, via bayesian fine-tuning. In Asian Conference on Machine Learning , pages 280-295. PMLR, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Arjun K Gupta and Daya K Nagar. Matrix variate distributions . Chapman and Hall/CRC, 2018.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning , pages 1861-1869, 2015.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762 , 2019.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning , pages 2304-2313. PMLR, 2018.

Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning , pages 2525-2534. PMLR, 2018.

Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics , pages 669-679. PMLR, 2020.

Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning , pages 2616-2625, 2018.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. In International conference on machine learning , pages 5436-5446. PMLR, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 , 2017.

Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning , pages 1708-1716, 2016.

David John Cameron Mackay. Bayesian methods for adaptive models . California Institute of Technology, 1992.

James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning , pages 2408-2417. PMLR, 2015.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning , pages 15630-15649. PMLR, 2022.

OpenAI. Introducing chatgpt, 2022. https://openai.com/blog/chatgpt , Last accessed on 2023-0509.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:2059620607, 2021.

Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems , 33:17044-17056, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems , 31, 2018.

Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings , volume 6. International Conference on Representation Learning, 2018.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533 , 2018.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations .

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning , pages 681-688, 2011.

Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations , 2022.

Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference on Machine Learning , pages 5847-5856, 2018.

Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient mcmc for bayesian deep learning. arXiv preprint arXiv:1902.03932 , 2019.

Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations , 2022.

A Proofs

A.1 Derivation of Equation (5)

A.2 Derivation of Equation (6)

A.3 Derivation of Equation (15)

Given that the posterior of the weights of the last layer is q ( θ ( l ) |D t -1 ) = MN ( θ ( l ) | θ ( l ) t -1 , U -1 t -1 , V -1 t -1 ) and the input to the last layer is h θ t -1 ( x ) , there is

The second equation stems from that ( h θ t -1 ( x ) ⊤ V -1 t -1 h θ t -1 ( x )) is a scalar.

B Experiment Details

0

100

200

300

400

500

600

700

800

900

1000

Category

0

2000

4000

6000

8000

10000

12000

Number of Images

WebVision (all)

WebVision (Flickr)

WebVision (Google)

ILSVRC 2012

B.1 Preprocessing

All images of each dataset are normalized and augmented by random horizontal flipping. For CIFAR-10/100, we use the standard 32 × 32 random cropping after zero-padding with 4 pixels on each side. In order to adapt to the input size of CLIP, we upsample images of CIFAR using torch.nn.functional.interpolate in PyTorch. For WebVision, the images are initially resized to a uniform size of 256. Subsequently, standard data augmentation techniques are applied, involving the random cropping of patches with dimensions of 224 × 224 from each image, followed by the application of horizontal random flipping.

B.2 Hyper-parameters The uning

We split the original training set into training and validation sets, where the latter remains clean and balanced for hyper-parameters tuning. In fact, as shown in Figure 4, the trade-off coefficient α in the selection objective is the primary factor that impacts the training curve and should be carefully selected. In particular, we select it from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } using a small validation set (of size 500 on CIFAR). We reuse the selected α on WebVision-100 without tuning.

B.3 Network

For all experiments, ResNet18 models are trained from scratch using PyTorch 2.0.0. Notably, in the case of CIFAR-10/100, we employ a downsampling layer with a small convolution with 3 × 3 kernel. Additionally, the average pooling at the end of the network is removed. Following the set-up of BatchNorm in [31], we compute the BatchNorm statistics on large batch B t for data selection. For model parameters update, we compute the statistics on the selected batch b t .

C More Results

Table 6 reports the results of our method on the entire training set of WebVision (as mentioned in Section 4, we use only half of the training set for training in the main experiments for a fair comparison with RHO-LOSS [31]). We see more significant speedups and higher final accuracy due to more training data.

We establish an extra baseline that combines RHO-LOSS with zero-shot predictors. The corresponding results based on our codebase are listed in Table 7. These indicate that the Bayesian treatment introduced by our method plays an important role in accelerating and improving the model convergence.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper followed up reducible hold-out loss selection (RHO-LOSS) method for data selection and improved it by proposing a reasonable approximation for the non-trivial objective function and eliminating the need for extra hold-out data.\n\nSTRENGTHS:\n1. Authors present both theoretical results (the lower bound for hard-to-estimate objective) and empirical results (algorithm and experiment results).\n\nWEAKNESSES:\n1. I am not fully convinced by the paper that their proposed method out-performs existed SOTA, namely RHO-LOSS much, specifically from the results in table 1 and figure 1. I wonder if authors can provide more intuitive comparisons between the proposed method the RHO-LOSS, either empirically show effectiveness and efficiency of the proposed method, or theoretically show the proposed lower bound is more solid than approximation used by RHO-LOSS.\n2. The ablation study needs more experiment result (more datasets and more model structure) to get valid conclusion. I also wonder why compare with baseline instead of RHO-LOSS.\n\nQUESTIONS:\nIn addition to questions addressed in weakness section, I have following questions for the authors:\n1. How is target accuracy selected? Specifically for table 3, why select 30% and 40% as target accuracy?\n2. I am curious if each training epoch for RHO-LOSS and your proposed method take same amount of time considering they have different data selection steps?\n3. It seems the proposed method performs well on classification datasets with large number of classes (CIFAR 100, ILSVRC12 and WebVision datasets) compared with dataset with smaller number of classes (CIFAR 10). I wonder authors can share some ideas from the perspective of the designed algorithm about this.\n4. I wonder if the authors have look into how zero-shot predictor affects the method performance. In other word, is the proposed method robust to poorly-performed predictor? If not, how much will the model be impacted?\n\nLIMITATIONS:\nYes, authors have addressed their limitation, namely the method performance hugely depends on effectiveness of the zero-shot predictor.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds on recently proposed work that accelerates training through online batch selection using generalisation loss as selection criterion, by using LaPlace approximation for a stronger Bayesian approximation and using off-the-shelf pre-trained models. The paper presents theory deriving their selection function and then demonstrating its efficacy on a number of datasets in the vision setting.\n\nSTRENGTHS:\n- Novelty on top of RHO-Loss: (1) using pre-trained models as validation model, (2) LaPlace approximation for stronger Bayesian approximation than point-estimate models.\n- Well-written paper, that is clean and easy to read.\n- Good ablations for how to tune the introduced zero-shot predictor as a validation model.\n- Decent performance gains over baselines.\n\nWEAKNESSES:\n- The novelty over RHO-Loss is appreciated, but it should be corrected in the narrative that RHO-Loss does not require access to clean holdout data. This is incorrect as far as I can tell, and Mindermann et al. note this in their paper.\n- Would be nice to see additional datasets and settings as per the original paper (Language and Clothing1M, however WebVision seems to be a bigger/better dataset)\n- In the practical acceleration/algorithm section they note that it's not much slower than [31]. I think a more empirical measurement of the difference would be appreciated, further more how much time does this method add over normal training? Does it really accelerate training in terms of time?\n- It's not clear whether the zero-shot predictor or the laplace approximation is providing the gain? An ablation here seems necessary. If it's just the zero-shot predictor, it's seems like there's not much of a difference here besides slapping a pre-trained model on as a validation model?\n\nQUESTIONS:\nAs per above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper studies data selection methods because training examples may be of different importance/quality. By selecting a subset of high-quality/high-usefulness examples, the model’s performance can be improved when training on this subset. In Particular, the paper proposed a method by leveraging a light-weight Bayesian treatment and incorporating off-the-shelf zero-shot predictors.

STRENGTHS:
1. The paper is clearly motivated.
2. The proposed method is well explained.
3. The experiments cover different baselines, models and datasets.

WEAKNESSES:
The problem of data selection has been studied extensively in the ML community, which verifies the importance of this problem. Some recent works ([1,2] from reference list below) proposed data selection methods that do not only focus on hard/easy examples, but also consider data distribution and its impact on the overall loss (e.g., see Theorem 1 from [2]). It looks like these methods may already have partially addressed the problem studied in this paper. It would be better if the authors could include some discussion about these methods and possibly compare against them in the experiments. 

Reference:    
[1] Xia, Xiaobo, et al. "Moderate coreset: A universal method of data selection for real-world data-efficient deep learning." ICLR 2023.    
[2] Zheng, Haizhong, et al. "Coverage-centric Coreset Selection for High Pruning Rates." ICLR 2023.

QUESTIONS:
Please see my comments in Weaknesses

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works is situated in the field of **robust generalization**; in particular it studies the problem of how models that were trained on noisy or imbalanced data perform on clean data. They achieve this via **online batch selection**, and in particular they contribute to the development of a **Bayesian framework** for searching for the best datapoints to use in each mini-batch among some canditate points sampled from the training set. In more details, they extend a SotA method, called **RHO-LOSS** [1]. The authors pinpoint some approximation choices that [1] makes in order to make the principled Bayesian approach to data selection practical. As a result, they design an algorithm which, in contrast to [1], they claim 1) to make **less crude approximations about predictive posteriors**, by using variational Laplacian approximations, MC sampling (and other) instead of pure point estimates, and 2) **not to rely on the existence of oracle clean data** during the training of their model. Their experiments demonstrate **improvements in robust generalization and training acceleration for image classification tasks** over other methods in the literature (incl. [1]).\n\n[1] Mindermann, Sören, et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" International Conference on Machine Learning. PMLR, 2022.\n\nSTRENGTHS:\nThe authors contribute to an important direction for robust training methods, in particular developping a bit further the Bayesian data selection framework.\n1.  The paper is well-written and motivated. The derivations are easy to follow and the deferrals to the appendix are used appropriately. Presentation can further be improved if there was somewhere a paragraph (perhaps in the appendix) with the list of approximation steps taken (for future reference), e.g. MC sample of a variational posterior, based on Laplace approximation of the last layer of a neural network (for which the backbone is taken as a point estimate), and the Laplace approximation further does not use the true Gaussian, but a Gauss-Newton matrix + potentially a KFAC approximation of it.\n2.   Authors identify correctly potential improvement points of [1], and proceed in mitigating them. In this way, they introduce a novel approximation scheme to the Bayesian framework of data selection, which is effective and computationally-lightweight.\n3.   Experiments seem to advocate in favor of using their method over [1] or other methods, they are extensive and on various dataset scales in image classification tasks.\n\nWEAKNESSES:\n1.   Proposed method overly claims that it does not use oracle clean data, however it might still depend on clean data via the unsupervised pre-trained zero-shot classification proxy that they use. In particular, CLIP-R50 might have been trained in a superset of datasets (like ImageNet) which contains CIFAR10/100. In that case, information from oracle clean data has been stored in the pretrained model. While I believe that this is a weaker assumption, there needs to be an explicit statement of this assumption and a quantitative assessment of this potential pitfall. To which extent would a pretrained model on the same noisy/imbalanced dataset as the benchmark task be useful? Nonetheless, the authors acknowledge this potential limitation of an underperforming pretrained model at the last section. The following (lightweight) experiments can be further performed to augment their arguments or awareness of the limitation:  \n\n -  A zero-shot CLIP baseline is provided, I think they should also provide with a finetuned version using linear probing and training on imbalanced/noisy CIFAR10/100 with uniform sampling. This way it will be more clear that the improvement are more due to their method and not because of an overpowered pretrained model.\n -  Consider CIFAR10/100 experiments using an unsupervised pretrained model, with linear probing or kNN (so that it is zero-shot if you want that) for the oracle model, via for instance some simple SSL method like MoCov2 [2] on the same noisy/imbalanced training set.\n\nI will increase the assessed score if such experiments are performed and reported.\n\n2.   Ablation study reveals sensitivity to some hyperparameters. As a result, model selection and hyperparameter tuning are important for the success of the method. How was model selection performed? Was the validation split iid to the training set or was it a clean/balanced dataset? This needs to be reported clearly, but not addressed in this paper.\n\n3.   It would be nice to have a more extensive ablation study on the effects of the approximation choices in terms of final test accuracy, spanning from crude approximation schemes to the one finally used by the authors.\n\n[2] Chen, Xinlei, et al. \"Improved baselines with momentum contrastive learning.\" arXiv preprint arXiv:2003.04297 (2020).\n\nQUESTIONS:\n### Questions\n\n* (related to weakness 2) How was model selection performed?\n* How were target accuracies chosen for training acceleration results?\n* Lines 151-152: What do we miss by replacing the Hessian matrix in the Laplacian approximation by the Gauss-Newton?\n* Lines 131-132 about “the recent trend of exploring the potential of pretrained models” (in robust generalization) needs some citations.\n* What does Table 4 in the Appendix refer to? Please complete the appendix to explain.\n\n### Typos\n\n* y-axis on Fig.2,3 and 4 are not in % of test accuracy as indicated.\n\nLIMITATIONS:\nThe authors derive a novel approximation scheme to Bayesian data selection for training robust models (Strength 2), with favorable results over past literature in image classifiacation tasks (Strength 3) and the writing is overall great (Strength 1). There is a concern about the applicability of the method in cases where a pretrained model does not exist for the task at hand, in which case we need to find another proxy or probably pretrain it (Weakness 1), and how much actually the method makes no use of priviledged clean/balanced datasets for training (Weakness 1) and validation (Weakness 2). The limitation about the pretrained model is mentioned briefly, but it would be appreciated if it is expanded and experimented on with the ablation studies suggested in the Weakness section above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}