SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Epochs needed to reach a target test accuracy on clean and noisy data (final accuracy is reported in parentheses). CIFAR-  $10^{*} / 100^{*}$  denotes adding  $10\%$  symmetric label noise to the dataset. Best performance is highlighted in bold. “-” indicates that the target accuracy was not reached. For all methods, only half of the original training set is used for training. The target accuracies are set following RHO-LOSS [31].
Method\Dataset	CIFAR-10	CIFAR-10*	CIFAR-100	CIFAR-100*
CLIP Acc	75.6%	75.6%	41.6%	41.6%
Target Acc	80.0%	87.5%	75.0%	85.0%	40.0%	52.5%	40.0%	47.5%
Train Loss	81	129 (90%)	-	- (28%)	138	- (42%)	-	- (4%)
Grad Norm	-	- (61%)	-	- (23%)	139	- (42%)	-	- (4%)
Grad Norm IS	57	139 (89%)	57	- (84%)	71	132 (55%)	94	142 (48%)
SVP	-	- (55%)	-	- (48%)	-	- (18%)	-	- (14%)
Irred Loss	-	- (60%)	-	- (62%)	93	- (43%)	89	- (43%)
Uniform	79	- (87%)	62	- (85%)	65	133 (54%)	79	116 (50%)
RHO-LOSS	39	65 (91%)	27	49 (91%)	48	77 (61%)	49	65 (60%)
Proposed	33	61 (91%)	25	47 (91%)	32	53 (63%)	39	53 (61%)
We compare the proposed method to the prior state-of-the-art (SOTA) data selection methods on various image classification benchmarks, including standard datasets (e.g., CIFAR-10 and CIFAR-100 [24]), their variants with controllable degrees of label noise and class-imbalance, and a real-world, large-scale, noisy, and imbalanced dataset, WebVision [25]. We also diagnose our selection strategy through elaborate ablation studies.
Datasets. We first evaluate the proposed method on clean CIFAR-10/100 and noisy CIFAR-10/100 with  $10\%$  symmetric label noise. Furthermore, we experiment in the context of imbalanced datasets, specifically CIFAR-10/100 datasets with varying degrees of class-imbalance ratio. Last, we investigate a web-scraped and large-scale dataset - WebVision, which consists of over 2.5 million images in 1000 categories and suffers from severe label noise and class-imbalance issues. For a fair comparison with RHO-LOSS [31], only half of the training set is used for model training.
Baselines. We compare the proposed method to a variety of baselines that define selection principles with various metrics, including the (training) loss [20], gradient norm [19], and gradient norm with importance sampling (gradient norm IS) [19]. We also compare to uniform sampling, Selection via Proxy (SVP) [6] and the latest SOTA method, RHO-LOSS [31].
Implementation details. In experiments under label noise, we introduce  $10\%$  symmetric noise caused by randomly flipping the ground-truth label to other labels. For the class-imbalance setting, we consider the long-tailed imbalance with two levels of imbalance intensity (i.e., 10 and 100), where an exponential decay on sample size is introduced across classes [3]. We use the same optimizer (AdamW [27]) and hyperparameters (e.g., learning rate 0.001, weight decay of 0.01,  $n_b = 32$  and  $\frac{n_b}{n_B} = 0.1$ ) as RHO-LOSS. Unless specified otherwise, we use ResNet18 (RN18) [14] as the deterministic model and specify the zero-shot predictor with CLIP-RN50. We select the trade-off coefficient  $\alpha$  from  $\{0.1, 0.2, 0.3, 0.4\}$  and the number of effective data  $n_e$  from  $\{100, 200, 500, 1000\}$ . In most cases, we set the prior precision  $\tau_0$  to 1. We report average results over 3 random runs.
Evaluation. Following [31], we evaluate speedup by comparing the number of epochs required to reach a given target test accuracy, which implicitly reflects the quality of the selection strategy.
4.1 Speedup on Clean Data and Noisy Data
We first empirically evaluate the proposed method on CIFAR-10 and CIFAR-100 with/without label noise. Table 1 reports the results. We can observe that the proposed method achieves considerably improved training speed compared to competing baselines in both scenarios with and without label noise. This evidences the efficacy of our Bayesian selection method in filtering out redundant and noisy data points in the training set. Despite leveraging extra holdout data, RHO-LOSS still
Table 2: Epochs required to reach a target test accuracy on imbalanced CIFAR-10 and CIFAR-100 (final accuracy is reported in parentheses).
Dataset	Imbalance Ratio	Target Acc	Uniform	RHO-LOSS	Proposed
CIFAR-10	10	60%	68	36	34
70%	98 (75%)	56 (80%)	52 (83%)
100	50%	81	50	44
60%	119 (56%)	87 (62%)	79 (68%)
CIFAR-100	10	20%	69	34	32
30%	110 (33%)	62 (39%)	58 (45%)
100	15%	70	43	41
20%	- (19%)	71 (24%)	66 (28%)
Table 3: Epochs required to reach a target test accuracy on WebVision dataset (final accuracy is reported in parentheses).
Dataset	Validation set	Target Acc	Uniform	RHO-LOSS	Proposed
WebVision-100	WebVision	30%	-	29	25
40%	- (27%)	47 (42%)	40 (60%)
ILSVRC12	30%	-	40	33
40%	- (23%)	- (37%)	50 (55%)
WebVision-200	WebVision	30%	-	28	22
40%	- (26%)	48 (42%)	36 (61%)
ILSVRC12	30%	-	35	29
40%	- (19%)	- (39%)	46 (56%)
underperforms our approach, potentially due to the less reliable approximations. Surprisingly, the superiority of our method is even more significant on the more challenging CIFAR-100. We also point out that our method eventually reaches higher final accuracy than the zero-shot CLIP. In other words, our method does not hinge on a performant zero-shot predictor.
4.2 Speedup on Class-imbalance Data
We further evaluate the proposed method on class-imbalanced data, a typical pattern in the real world while probably raising non-trivial difficulties for model training [7, 3]. Specifically, we experiment on the aforementioned imbalanced CIFAR-10 and CIFAR-100 using similar protocols to previous studies. The test datasets remain unchanged. Given that RHO-LOSS is superior to other baselines, we primarily compare with it in the following studies. We also add uniform sampling into comparison due to its implementation simplicity. Table 2 presents the results.
As shown, the proposed method still consistently outperforms uniform sampling and RHO-LOSS. In particular, the final accuracy achieved by our method is higher than RHO-LOSS. We also note that the gain in training efficiency becomes more significant as the imbalance ratio increases. The failure of RHO-LOSS is partially attributed to the less principled approximations. Another reason is that the holdout data are still class-imbalanced, rendering the trained validation models biased.
4.3 Speedup on Large Web-scraped Data
Learning algorithms face significant challenges when dealing with noisy and imbalanced real-world datasets. To evaluate our proposed method's efficacy in addressing these challenges, we assess it on a large-scale web-scraped dataset, WebVision. We defer the details of WebVision to the Appendix. Since the dataset is large, comprising 2.4 million data points, we train classification models only on the first 100/200 classes (known as WebVision-100/200) of the entire dataset, following [18, 4], for an efficient evaluation. We test the trained models on the human-annotated WebVision validation set
(a) Proportion of label noise in selection.
(b) Proportion of redundant points in selection.
Figure 2: Training curves corresponding to using pre-trained ViT-B/16 as the model backbone. (WebVision-200; 1 epoch=344 iterations)
Figure 1: Properties of the data selected by our method and baselines on CIFAR-10 and CIFAR-100 with  $10\%$  label noise. Redundant points represent the data that have already been classified correctly. The reported values are averaged over 150 epochs of model training and five random runs.
Figure 3: Training curves corresponding to using zero-shot predictors defined with various CLIP backbones. (CIFAR-100)
and the ILSVRC12 validation set [9]. We report the results in Table 3. We select  $30\%$  and  $40\%$  as target accuracies according to the final accuracies of RHO-LOSS ( $\sim 40\%$ ) and our approach ( $\sim 60\%$ ).
Still, the proposed method outperforms uniform sampling and RHO-LOSS in aspects of both the speed to reach the target accuracy and the final accuracy on both WebVision-100 and WebVision-200. Notably, our method achieves up to  $19\%$  higher final accuracy than RHO-LOSS, suggesting the superiority of our method in coping with real-world datasets. We also report results for our method on the entire training set in the Appendix, which shows more significant speedups and higher final accuracy due to more training data.
4.4 Analysis of the Properties of the Selected Data
In this section, we analyze the characteristics of the data selected by our method and compare it with existing works. Figure 1a displays the proportion of selected data points with label noise, and it is evident that data selected by our method has the lowest label noise. We also investigate whether our method prioritizes redundant data, defined as those having already been correctly classified by the training model [31]. As depicted in Figure 1b, both our method and RHO-LOSS select significantly fewer redundant data points than uniform sampling, and our method is slightly better.
In this section, we conduct ablation studies on model architecture, zero-shot predictor, and some critical hyper-parameters (e.g.,  $\alpha$ ,  $n_e$ ).
Table 5: The efficacy of a variant of our method (referred to as Proposed†) where the zero-shot validation model is replaced by that used by RHO-LOSS [31]. Experiment on CIFAR-100.
RHO-LOSS	Proposed†	Proposed
Epochs to reach 40.0% test accuracy	48	30	32
Epochs to reach 52.5% test accuracy	77	52	53
Final accuracy	61%	63%	63%
(a) The trade-off coefficient  $\alpha$
Figure 4: Ablation studies on some critical hyper-parameters including  $\alpha$ ,  $n_e$ , and  $\tau$  on CIFAR-100.
(b) The number of effective data  $n_e$ .
(c) The temperature  $\tau$
Model architecture. We test our method using the powerful Vision Transformer (ViT) [11] architecture on WebVision-200. Specifically, we experiment with a ViT-B/16 model pre-trained unsupervisedly by Masked Auto-encoder [13], considering that fine-tuning a pre-trained model is preferred over training a model from scratch in practice. We provide the training curve of our method in Figure 2, where uniform sampling is included for comparison. The huge performance gap between our method and the baseline validates the efficacy of our method in this setting.
Zero-shot predictor. We next assess the impact of the zero-shot predictor on our method. We specify the zero-shot predictor with various CLIP variants (e.g., RN50 and ViT-B/16) and draw the training curves in Figure 3. As shown, although the zero-shot accuracy of the RN50 variant is significantly lower than that of ViT-B/16, the speedup effect of the two backbones is similar. This finding suggests the robustness of our method against the selection of the zero-shot predictor.
In the above experiments, we use CLIP-based zero-shot predictors without further tuning. Here, we perform linear probing using the CLIP models to demonstrate if we can achieve good convergence trivially. We simply adopt the uniform sampling strategy and report the results in Table 4. We do not report convergence speed because the baseline uses pre-trained weights while our method trains models from scratch. As shown, our method still bypasses the baseline with clear margins.
Table 4: Comparison of final accuracy  $(\%)$  between the proposed method and the linear probing with CLIP and uniform sampling strategy.
Dataset	CLIP linear probing	Proposed
CIFAR-10	84.5	91.4
CIFAR-10*	84.1	91.3
CIFAR-100	58.5	63.3
CIFAR-100*	57.8	61.4
We replace the zero-shot predictor used in our method with the validation model in RHO-LOSS [31] and present the corresponding results in Table 5. The comparison proves the superiority of our method over RHO-LOSS. These results also reflect that our method is robust against the specification of the validation model.
Hyper-parameters. We then analyze the effects of three crucial hyper-parameters,  $\alpha$ ,  $n_e$ , and the temperature  $\tau$  used by the softmax operation in the CLIP zero-shot predictor. Figure 4a shows training curves associated with various  $\alpha$ . We see that too large  $\alpha$  (i.e., lightening the second term in Equation (9)) can lead to a significant drop in training speed and final performance, which emphasizes the importance of the zero-shot predictor. Nevertheless, using only the zero-shot predictor (i.e.,  $\alpha = 0$ ) is also suboptimal, leading to degraded final accuracy. Hence, we conclude that both terms in the lower bound of the original data selection principle are beneficial, where the first term accounts more for the later training stages while the second term accounts more for the earlier ones. Figure 4b shows the ablations study on  $n_e$ , and we witness the robustness of our method against  $n_e$ . Figure 4c
shows training curves corresponding to different  $\tau$ . We observe that an appropriate temperature boosts our method considerably.
Wall-clock time. We empirically observe that the per-epoch training time for Uniform, RHO-LOSS, and our method on CIFAR-100 is 14s, 18s, and 21s, respectively. Namely, our approach has slightly increased per-epoch time over RHO-LOSS. It arises from that we use a CLIP-RN50 zero-shot predictor to compute the second term in Equation (16), whereas [31] uses a RN18 validation model. In fact, this gap can be reduced by a simple implementation trick—pre-computing the CLIP-RN50 predictions for each sample in the training set before training.
As shown in Table 1, compared to RHO-LOSS, we can reduce the number of epochs required to reach  $40\%$  test accuracy from 48 to 32 and  $52.5\%$  from 77 to 53. According to the per-epoch training time, we achieve a  $48*18/32/21 \approx 1.29$  or  $77*18/53/21 \approx 1.25$  times practical acceleration.
Extensive methods have been proposed to accelerate model training through techniques such as data pruning, coreset selection, curriculum learning, online batch selection, etc. Data pruning explores various metrics, such as EL2N score [33], forgetting score [40], and classification margin [34], to measure individual differences among data points and retains only the hardest examples for model training. However, data pruning still exhibits limitations when dealing with noisy labels, and some of these metrics are computationally expensive. Coreset selection methods also partially address the problem of accelerating model training. In particular, [43] contributes a data scoring mechanism that is robust to the change of scenarios for coreset selection, and [46] makes an in-depth understanding of the catastrophic accuracy drop issue of one-shot coreset selection and contributes a novel solution to it. However, these methods lack the flexibility to prioritize samples with different properties at various training stages. Curriculum learning, as advocated by [1], prioritizes easy points with low label noise before uniformly training on all data points. While this strategy enhances convergence, it fails to address the issue of skipping redundant points already learned.
Online batch selection methods [26, 19, 17] tackle the training acceleration problem by selecting hard data identified by high loss or gradient norm. However, they also suffer from a common drawback—high loss can be caused by label noise or ambiguous labels, so prioritizing such data can result in a decline in predictive performance. Compared to the prior art, our method establishes a Bayesian data selection metric and exploits zero-shot predictors to prioritize valuable training data for addressing these issues.
This paper addresses the challenges posed by noisy and biased data in real-world scenarios. Our main contribution is to make the generalization loss-based data selection principle more accessible to accelerate the training of deep models. To achieve this, we first derive a lower bound of this objective to improve its tractability. We then propose leveraging a Bayesian treatment and off-the-shelf zero-shot predictors to estimate the data selection objective reliably. The resulting algorithm does not require extra holdout data. Our extensive empirical studies demonstrate the superiority of our method in accelerating model training over competitive baselines.
**Limitation.** Our method may fail when the zero-shot predictor performs poorly on specific tasks. Future work could explore adapting the zero-shot predictor to a few-shot one using a few clean validation data to address this limitation.
Z.J. Deng was supported by Natural Science Foundation of Shanghai (No. 23ZR1428700) and the Key Research and Development Program of Shandong Province, China (No. 2023CXGC010112). J. Zhu and P. Cui were supported by NSF of China Projects (Nos. 62061136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); a grant from Tsinghua Institute for Guo Qiang; and the High Performance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.
[41] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[42] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, pages 681-688, 2011.
[43] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
[44] Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference on Machine Learning, pages 5847-5856, 2018.
[45] Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyllical stochastic gradient mcmc for bayesian deep learning. arXiv preprint arXiv:1902.03932, 2019.
[46] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representations, 2022.
A.1 Derivation of Equation (5)
$$
\begin{array}{l} \log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) = \log \int p (\theta | \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) p (y | x, \theta) d \theta \\ = \log \int \frac {p \left(\mathcal {D} ^ {*} | \theta\right) p \left(\theta \mid \mathcal {D} _ {t - 1}\right)}{p \left(\mathcal {D} ^ {*} \mid \mathcal {D} _ {t - 1}\right)} p (y | x, \theta) d \theta \\ = \log \int p \left(\mathcal {D} ^ {*} | \theta\right) p \left(\theta \mid \mathcal {D} _ {t - 1}\right) p (y | x, \theta) d \theta - \log p \left(\mathcal {D} ^ {*} \mid \mathcal {D} _ {t - 1}\right). \\ \end{array}
$$
A.2 Derivation of Equation (6)
$$
\begin{array}{l} \log p (y | x, \mathcal {D} ^ {*}, \mathcal {D} _ {t - 1}) = \log \int p (\mathcal {D} ^ {*} | \theta) p (\theta | \mathcal {D} _ {t - 1}) p (y | x, \theta) d \theta - \log p (\mathcal {D} ^ {*} | \mathcal {D} _ {t - 1}) \\ \geq \int p (\theta | \mathcal {D} _ {t - 1}) \log [ p (\mathcal {D} ^ {*} | \theta) p (y | x, \theta) ] d \theta - \log p (\mathcal {D} ^ {*} | \mathcal {D} _ {t - 1}) \\ = \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} \log p (\mathcal {D} ^ {*} | \theta) + \mathbb {E} _ {p (\theta | \mathcal {D} _ {t - 1})} \log p (y | x, \theta) - \log p (\mathcal {D} ^ {*} | \mathcal {D} _ {t - 1}). \\ \end{array}
$$
A.3 Derivation of Equation (15)
Given that the posterior of the weights of the last layer is  $q(\theta^{(l)}|\mathcal{D}_{t - 1}) = \mathcal{MN}(\theta^{(l)}|\theta_{t - 1}^{(l)},U_{t - 1}^{-1},V_{t - 1}^{-1})$  and the input to the last layer is  $h_{\theta_{t - 1}}(x)$ , there is
$$
\begin{array}{l} q \left(f _ {x} \mid \mathcal {D} _ {t - 1}\right) = \mathcal {M N} \left(h _ {\theta_ {t - 1}} (x) ^ {\top} \theta_ {t - 1} ^ {(l)}, U _ {t - 1} ^ {- 1}, h _ {\theta_ {t - 1}} (x) ^ {\top} V _ {t - 1} ^ {- 1} h _ {\theta_ {t - 1}} (x)\right) \\ = \mathcal {N} \Big (f _ {\theta_ {t - 1}} (x), \big (h _ {\theta_ {t - 1}} (x) ^ {\top} V _ {t - 1} ^ {- 1} h _ {\theta_ {t - 1}} (x) \big) U _ {t - 1} ^ {- 1} \Big). \\ \end{array}
$$
The second equation stems from that  $(h_{\theta_{t-1}}(x)^\top V_{t-1}^{-1} h_{\theta_{t-1}}(x))$  is a scalar.
Figure 5: Number of images per category of the WebVison dataset [25].
B.1 Preprocessing
All images of each dataset are normalized and augmented by random horizontal flipping. For CIFAR-10/100, we use the standard  $32 \times 32$  random cropping after zero-padding with 4 pixels on each side. In order to adapt to the input size of CLIP, we upsample images of CIFAR using torch.nn.Functional.interpolate in PyTorch. For WebVision, the images are initially resized to a uniform size of 256. Subsequently, standard data augmentation techniques are applied, involving the random cropping of patches with dimensions of  $224 \times 224$  from each image, followed by the application of horizontal random flipping.
Table 6: Epochs required to reach a target test accuracy on the  $100\%$  WebVision training set (final accuracy is reported in parentheses).
Dataset	Validation set	Target Acc	50% data	100% data
WebVision-100	WebVision	30%	25	12
40%	40 (60%)	22 (68%)
ILSVRC12	30%	33	18
40%	50 (55%)	27 (64%)
WebVision-200	WebVision	30%	22	11
40%	36 (61%)	18 (68%)
ILSVRC12	30%	29	16
40%	46 (56%)	22 (64%)
Table 7: The results of RHO-LOSS [31] using zero-shot predictor as the validation model on CIFAR-100.
Method	Target Acc	Final Acc (%)
40.0%	52.5%
RHO-LOSS w/ zero-shot predictor (CLIP-RN50)	59	92	58
RHO-LOSS w/ zero-shot predictor (CLIP-ViT-B/16)	53	86	60
Proposed	32	53	63
B.2 Hyper-parameters The uning
We split the original training set into training and validation sets, where the latter remains clean and balanced for hyper-parameters tuning. In fact, as shown in Figure 4, the trade-off coefficient  $\alpha$  in the selection objective is the primary factor that impacts the training curve and should be carefully selected. In particular, we select it from  $\{0.1, 0.2, 0.3, 0.4\}$  using a small validation set (of size 500 on CIFAR). We reuse the selected  $\alpha$  on WebVision-100 without tuning.
B.3 Network
For all experiments, ResNet18 models are trained from scratch using PyTorch 2.0.0. Notably, in the case of CIFAR-10/100, we employ a downsampling layer with a small convolution with  $3 \times 3$  kernel. Additionally, the average pooling at the end of the network is removed. Following the set-up of BatchNorm in [31], we compute the BatchNorm statistics on large batch  $B_{t}$  for data selection. For model parameters update, we compute the statistics on the selected batch  $b_{t}$ .
Table 6 reports the results of our method on the entire training set of WebVision (as mentioned in Section 4, we use only half of the training set for training in the main experiments for a fair comparison with RHO-LOSS [31]). We see more significant speedups and higher final accuracy due to more training data.
We establish an extra baseline that combines RHO-LOSS with zero-shot predictors. The corresponding results based on our codebase are listed in Table 7. These indicate that the Bayesian treatment introduced by our method plays an important role in accelerating and improving the model convergence.