SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Time Complexity Analysis. We analyze the time complexity of our greedy approximation in Algorithm 1. At each step, Prune4ReLU takes the time complexity of  $O(m \log m) + O(md)$ , where  $m$  is the training set size and  $d$  is the embedding dimension size of the warm-up classifier. Specifically, in Line 3, sampling an example with the largest marginal benefit of confidence takes  $O(m \log m)$ , and in Lines 5-6, updating the reduced neighborhood confidence of all training examples takes  $O(md)$ . In addition, with Prune4ReLU, the time complexity is reduced to  $O(m \log (m / c)) + O(md)$  because it iteratively selects the example with the largest marginal benefit within each class subset, which is lower than the time complexity of a similar distance-based data pruning work, kCenterGreedy [38] aiming to maximize the distance coverage of a selected subset to the entire training set by a greedy approximation. At each iteration, kCenterGreedy's runtime is  $O(mk_{t})$ , where  $k_{t}$  is the size of the selected set at iteration  $t$  [46]. Note that, its time complexity increases as the subset size grows, which hinders its usability on a large-scale dataset. In Section 4.2, we empirically show that Prune4ReLU is scalable to prune Clothing-1M, a large dataset with 1M examples, whereas kCenterGreedy is not.
Datasets. We first perform the data pruning task on four real noisy datasets, CIFAR-10N, CIFAR-100N, Webvision, and Clothing-1M. CIFAR-10N and CIFAR-100N [7] contain human re-annotations of 50K training images in the original CIFAR-10 and CIFAR-100 [47]. Specifically, each training image in CIFAR-10N contains three noisy labels, called Random 1,2,3, which are further transformed into the Worst-case label. Each image in CIFAR-100N contains one noisy label. WebVision [8] and Clothing-1M [9] are two large-scale noisy datasets. WebVision contains 2.4M images crawled from the Web using the 1,000 concepts in ImageNet-1K [48]. Following prior work [49], we use mini-WebVision consisting of the first 50 classes of the Google image subset with approximately 66K training images. Clothing-1M consists of 1M training images with noisy labels and 10K clean test images collected from online shopping websites. Additionally, a large-scale synthetic noisy dataset, which we call ImageNet-N, is included in our experiments. It consists of 1.2M training images, which are the training images of ImageNet-1K [48] with asymmetric label noise. See Appendix D for details of the noise injection.
Algorithms. We compare Prune4ReLU with a random selection from a uniform distribution, Uniform, a clean sample selection algorithm, SmallLoss [26], and six data pruning algorithms including Margin [50],  $k$ -CenterGreedy [38], Forgetting [14], GraNd [15], SSP [6], and Moderate [43]. SmallLoss favors examples with a small loss. For data pruning algorithms, (1) Margin selects examples in the
Table 1: Performance comparison of sample selection baselines and Prune4ReL on CIFAR-10N and CIFAR-100N. The best results are in bold.
Re-labelModels	SelectionMethods	CIFAR-10N	CIFAR-100N
Random	Worst	Noisy
0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8
DivMix	Uniform	87.5±0.2	91.9±0.2	93.7±0.1	94.8±0.1	83.2±0.2	88.5±0.1	90.2±0.0	91.4±0.0	30.5±1.0	55.3±0.5	57.5±1.9	58.6±0.9
SmallL	68.1±4.0	82.4±0.8	89.0±0.3	93.1±0.1	70.3±0.6	80.3±0.2	89.1±0.0	92.1±0.1	33.3±3.2	47.4±1.1	59.4±0.7	62.0±1.2
Margin	68.5±2.9	88.5±0.3	93.2±0.2	94.7±0.1	61.3±0.8	75.1±0.7	85.3±0.2	90.2±0.1	17.1±0.8	30.8±1.0	46.3±2.4	61.2±1.3
kCenter	87.4±0.5	93.0±0.1	94.4±0.1	95.0±0.0	82.7±0.8	88.4±0.1	90.6±0.1	92.2±0.0	38.0±1.0	50.0±1.8	59.7±1.3	63.0±0.9
Forget	85.2±0.5	93.0±0.2	94.5±0.1	95.1±0.1	78.3±0.6	88.3±0.2	90.4±0.1	92.0±0.2	26.4±1.3	54.3±0.8	63.1±1.3	66.6±1.1
GraNd	21.8±0.6	60.9±4.5	92.5±1.8	94.8±0.1	18.5±1.7	25.5±0.9	49.3±0.9	88.0±0.5	15.5±1.2	26.0±1.9	44.7±1.5	60.4±1.6
SSP	85.8±1.8	92.2±1.5	93.0±1.1	94.5±0.2	81.4±2.5	86.5±1.9	89.6±1.2	91.9±0.4	30.1±2.2	52.4±1.3	58.7±0.5	63.4±0.5
Moderate	86.4±0.8	91.4±0.3	93.8±0.5	94.8±0.2	81.4±1.2	86.5±0.6	90.0±0.6	91.6±0.2	34.2±1.4	54.5±1.3	56.1±0.5	59.9±0.6
Pr4ReLU	87.6±0.3	92.4±0.4	94.5±0.2	95.1±0.1	83.7±0.5	88.8±0.3	90.2±0.3	92.0±0.3	37.2±1.0	55.3±0.7	61.2±0.5	65.5±0.8
Pr4ReLUB	88.1±0.3	93.0±0.2	94.5±0.2	95.1±0.1	83.7±0.4	88.6±0.4	90.8±0.2	92.4±0.2	39.4±0.8	56.3±0.5	63.5±0.3	67.4±0.7
SOP+	Uniform	87.5±0.3	91.5±0.1	93.4±0.0	94.8±0.2	81.9±0.1	87.5±0.1	90.8±0.1	91.8±0.1	46.5±0.0	55.7±0.2	60.8±0.3	64.4±0.2
SmallL	77.6±2.5	86.2±0.1	90.7±0.6	94.3±0.2	78.8±0.2	84.1±0.1	89.3±0.1	92.3±0.2	48.5±0.8	59.8±0.4	63.9±0.2	66.1±0.6
Margin	52.1±5.0	79.6±8.6	92.6±3.9	95.1±1.3	45.7±1.1	61.8±0.7	84.6±0.3	92.5±0.0	20.0±1.2	34.4±0.3	50.4±0.6	63.3±0.1
kCenter	86.3±0.4	92.2±0.3	94.1±0.2	95.3±0.1	81.9±0.0	88.0±0.0	91.3±0.1	92.3±0.0	44.8±0.6	55.9±0.4	61.6±0.3	65.2±0.6
Forget	82.4±1.0	93.0±0.2	94.2±0.3	95.0±0.1	71.1±0.4	87.7±0.1	90.6±0.3	92.2±0.0	38.0±0.5	55.3±0.2	63.2±0.1	65.8±0.4
GraNd	24.2±5.5	51.6±3.2	85.9±1.2	94.9±0.2	15.4±1.6	25.7±0.8	51.0±0.5	86.8±0.5	11.0±0.1	19.0±0.6	38.7±0.5	62.1±0.5
SSP	80.5±2.6	91.7±1.5	93.8±1.0	95.0±0.2	70.8±2.7	86.6±1.9	89.2±0.9	92.3±0.4	39.2±2.2	54.9±1.5	62.7±0.7	65.0±0.3
Moderate	87.8±1.0	92.8±0.5	94.0±0.3	94.9±0.2	75.2±1.5	81.9±1.2	87.7±0.7	91.8±0.3	46.4±1.8	54.6±1.7	60.2±0.4	64.6±0.4
Pr4ReLU	87.8±1.2	92.7±0.3	94.4±0.2	95.1±0.1	82.7±0.5	88.1±0.4	91.3±0.3	92.5±0.2	50.2±0.2	59.1±0.5	63.9±0.3	65.7±0.5
Pr4ReLUB	88.5±0.3	93.1±0.2	94.4±0.1	95.3±0.1	84.9±0.6	89.2±0.6	91.3±0.3	92.9±0.1	52.9±0.8	60.1±0.6	64.1±0.4	66.2±0.3
increasing order of the difference between the highest and the second highest softmax probability; (2)  $k$ -CenterGreedy selects  $k$  examples that maximize the distance coverage to the entire training set; (3) Forgetting selects examples that are easy to be forgotten by the classifier throughout the warm-up training epochs; (4) GraNd uses the average norm of the gradient vectors to measure the contribution of each example to minimizing the training loss; (5) SSP leverages a self-supervised pre-trained model to select the most prototypical examples; and (6) Moderate aims to select moderately hard examples using the distances to the median.
Implementation Details. We train two representative Re-labeling models, DivideMix [13] and  $\mathrm{SOP + }$  [33] for our experiments. The hyperparameters for DivideMix and  $\mathrm{SOP + }$  are favorably configured following the original papers. Following the prior Re-labeling work [13, 33], for CIFAR-10N and CIFAR-100N, PreAct Resnet-18 [51] is trained for 300 epochs using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The initial learning rate is 0.02, and it is decayed with a cosine annealing scheduler. For WebVision, InceptionResNetV2 [52] is trained for 100 epochs with a batch size of 32. For Clothing-1M, we use ResNet-50 [53] pre-trained on ImageNet and fine-tune it for 10 epochs with a batch size of 32. The initial learning rates of WebVision and Clothing-1M are 0.02 and 0.002, which are dropped by a factor of 10 at the halfway point of the training epochs. For ImageNet-N, ResNet-50 [53] is trained for 50 epochs with a batch size of 64 and an initial learning rate of 0.02 decayed with a cosine annealing scheduler.
For data pruning algorithms, following prior work [16], we perform sample selection after 10 warm-up training epochs for CIFAR-10N, WebVision, and ImageNet-N, and 30 warm-up epochs for CIFAR-100N. For Clothing-1M, we perform the sample selection after 1 warm-up training epoch from the ImageNet pre-trained ResNet-50. The hyperparameters for all data pruning methods are favorably configured following the original papers. For Prune4ReLU, we set its hyperparameter  $\tau$  to 0.975 for CIFAR-10N, to 0.95 for CIFAR-100N, WebVision, and ImageNet-N, and to 0.8 for Clothing-1M. More implementation details can be found in Appendix E. All methods are implemented with PyTorch 1.8.0 and executed on NVIDIA RTX 3080 GPUs. The code is available at https://github.com/kaist-dmlab/Prune4Rel.
Evaluation. For CIFAR-10N, CIFAR-100N, and WebVision, we select the subset with the selection ratios  $\{0.2, 0.4, 0.6, 0.8\}$ . For Clothing-1M and ImageNet-N, we construct the subset with  $\{0.01, 0.05, 0.1, 0.2\}$  and  $\{0.05, 0.1, 0.2, 0.4\}$  selection ratios, respectively. We measure the test accuracy of the Re-labeling models trained from scratch on the selected subset. Every experiment is run three times, and the average of the last accuracy is reported. For CIFAR-10N with the Random noise, we average the test accuracy of the models trained using the three noisy labels.
(a) WebVision.
(b) Clothing-1M.
Figure 3: Data pruning performance comparison: (a) test accuracy of  $\mathrm{SOP + }$  trained on each selected subset of WebVision; (b) test accuracy of DivideMix trained on each selected subset of Clothing-1M; (c) elapsed GPU time for selecting a subset on WebVision with a selection ratio of 0.8.
(c) GPU Time for Selection.
Table 2: Performance comparison of the standard cross-entropy model and Re-labeling models when combined with data pruning methods on CIFAR-10N and CIFAR-100N.
Learning Models	Selection Methods	CIFAR-10N	CIFAR-100N
Random	Worst	Noisy
0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8
Standard	Uniform	75.6±0.1	81.0±0.1	83.0±0.1	84.3±0.5	58.6±0.3	63.9±0.2	66.4±0.1	67.5±0.5	37.4±0.2	46.1±0.2	50.1±0.0	52.3±0.1
SmallL	75.0±1.5	83.4±0.6	87.5±0.3	90.1±0.2	70.1±0.3	77.5±1.6	80.6±0.5	76.4±0.1	42.2±0.4	54.7±0.5	57.7±0.1	57.4±0.2
Forget	82.2±0.5	86.2±0.1	86.1±0.0	85.4±0.2	71.2±0.3	73.3±0.2	71.4±0.2	69.6±0.1	43.5±0.4	54.5±0.1	57.5±0.4	56.6±0.3
DivMix	Pr4ReLU	88.1±0.3	93.0±0.2	94.5±0.2	95.1±0.1	83.7±0.4	88.6±0.4	90.8±0.2	92.4±0.2	39.4±0.8	56.3±0.5	63.5±0.3	67.4±0.7
SOP+	Pr4ReLU	88.5±0.3	93.1±0.2	94.4±0.1	95.3±0.1	84.9±0.6	89.2±0.6	91.3±0.3	92.9±0.1	52.9±0.8	60.1±0.6	64.1±0.4	66.2±0.3
4.2 Main Results on Real Noisy Datasets
Test Accuracy. Table 1 summarizes the test accuracy of eight baselines and Prune4ReLU on CIFAR-10N and CIFAR-100N trained with two Re-labeling models. Overall, Prune4ReLU consistently achieves the best performance for all datasets across varying selection ratios. Numerically, Prune4ReLU improves DivideMix and  $\mathrm{SOP + }$  by up to  $3.7\%$  and  $9.1\%$  , respectively. Compared with six data pruning baselines, they show rapid performance degradation as the size of the subset decreases; most of them, which are designed to favor hard examples, tend to select a large number of noisy examples, resulting in unreliable re-labeling. While Moderate selects moderately hard examples, it is still worse than Prune4ReLU since it is not designed for noise-robust learning scenarios with Re-labeling models. On the other hand, SmallLoss, a clean sample selection baseline, shows poor performance in CIFAR-10N (Random), because this dataset contains a relatively low noise ratio, and selecting clean examples is less critical. Although SmallLoss shows robust performance in CIFAR-100N, it is worse than Prune4ReLU because it loses many informative noisy examples that help generalization if re-labeled correctly. Meanwhile, Uniform is a fairly robust baseline as it selects easy (clean) and hard (noisy) examples in a balanced way; many selected noisy examples may be relabeled correctly by other selected clean neighbors, resulting in satisfactory test accuracy.
Similarly, Figures 3(a) and 3(b) visualize the efficacy of the baselines and Prune4ReLU on the WebVision and Clothing-1M datasets. We train  $\mathrm{SOP + }$  on WebVision and DivideMix on Clothing-1M. Similar to CIFAR-N datasets, Prune4ReLU achieves better performance than existing baselines on two datasets. Quantitatively, Prune4ReLU outperforms the existing sample selection methods by up to  $2.7\%$  on WebVision with a selection ratio of 0.4. This result confirms that the subset selected by Prune4ReLU, which maximizes the total neighborhood confidence of the training set, successfully maintains the performance of Re-labeling models and is effective for model generalization.
Efficiency. In Figure 3(c), we further show the GPU time taken for selecting subsets within the warm-up training. We train  $\mathrm{SOP + }$  on WebVision with a selection ratio of 0.8. Powered by our efficient greedy approximation, Prune4ReL fastly prunes the dataset in a reasonable time. GraNd takes almost 10 times longer than SmallLoss or Margin, since it trains the warm-up classifier multiple times for the ensemble. kCenterGreedy is infeasible to run in our GPU configuration due to its huge computation and memory costs.
Table 3: Effect of the confidence metrics on Prune4ReL.
Re-label Model	Dataset	Conf. Metric	Selection Ratio
0.2	0.4	0.6	0.8
SOP+	CIFAR-10N (Worst)	MaxProb	82.7	88.1	91.3	92.5
DiffProb	82.5	88.5	91.2	92.5
CIFAR-100N	MaxProb	50.2	59.1	63.9	65.7
DiffProb	49.2	59.3	64.1	66.0
Figure 4: Effect of the neighborhood threshold  $\tau$  on Prune4ReL $_B$ .
4.3 Necessity of Data Pruning with Re-labeling under Label Noise
Table 2 shows the superiority of the Re-labeling models over the standard learning model, i.e., only with the cross-entropy loss, for data pruning under label noise. When combined with the data pruning methods, the performance of the Re-labeling models such as DivideMix and SOP+ significantly surpasses those of the standard models on CIFAR-10N and CIFAR-100N by up to  $21.6\%$ . That is, the re-labeling capacity can be well preserved by a proper data pruning strategy. This result demonstrates the necessity of data pruning for the Re-labeling models in the presence of noisy labels.
Effect of Confidence Metrics. Prune4ReL can be integrated with various metrics for the confidence of predictions in Eq. (6). In our investigation, we consider two widely-used metrics: (1) MaxProb, which represents the maximum value of softmax probability and (2) DiffProb, which measures the difference between the highest and second highest softmax probabilities. Table 3 shows the effect of the two confidence metrics on the test accuracy of  $\mathrm{SOP + }$  on CIFAR-10N (Worst) and CIFAR100N. The result indicates that both metrics perform similarly and yield higher accuracy compared to existing data pruning baselines, which demonstrates Prune4ReL is open to the choice of the confidence metric.
Effect of Neighborhood Size. Prune4ReL involves a hyperparameter  $\tau$  in Eq. (6) to determine the neighborhood of each example, where a larger (smaller) value introduces a fewer (more) number of neighbors. Figure 4 shows the effect of  $\tau \in \{0.93, 0.95, 0.99\}$  on the test accuracy of  $\mathrm{SOP}+$  trained on CIFAR-100N with varying selection ratios. In general, Prune4ReL shows better or comparable performance than other sample selection baselines. Among them, Prune4ReL with  $\tau = 0.95$  shows a satisfactory accuracy across varying selection ratios. With a smaller value of  $\tau = 0.93$ , it shows the best performance in a high selection ratio of 0.8, but it becomes less effective in low selection ratios, due to the increasing influence of noisy examples caused by a relatively large neighborhood size. On the contrary, with a large value of  $\tau = 0.99$ , it primarily selects clean yet easy examples due to a small neighborhood size, leading to a relatively less improvement in performance.
4.5 In-depth Analysis of Noisy Examples in Selected Subset
Noise Ratio of Selected Subset. Table 4 shows the ratio of noisy examples in the subset selected by each sample selection method. SmallLoss shows a very low ratio of noisy examples in the subset because it prefers clean examples. Many data pruning methods, including Margin, GraNd, SSP, and Moderate (for CIFAR-100N), tend to select a higher ratio of noisy examples compared with that of each original dataset since they prefer to select hard examples. On the other hand, Prune4ReLU selects a low ratio of noisy examples when the subset size is small and gradually increases the noise ratio as the subset size increases. This result indicates that Prune4ReLU expands the confident subset through Algorithm 1—i.e., selecting the most confident (clean) examples first and then trying to select less confident (hard or noisy) neighbors to ensure accurate re-labeling. While some baselines, such as kCenterGreedy, Forget, and Moderate (for CIFAR-10N), also select a somewhat low ratio of noisy examples, their data pruning performances are worse than Prune4ReLU because the quality (or self-correctability) of noisy examples is not considered when selecting the subset, which is further investigated in Table 5.
Self-correctability of Selected Noisy Examples. Table 5 shows the self-correctability of selected subsets, which indicates the ratio of correctly re-labeled noisy examples out of all selected noisy
Table 4: Ratio (%) of noisy examples in the selected subset.
Re-label Model	Selection Methods	CIFAR-10N (Random, ≈ 18%)	CIFAR-10N (Worst, ≈ 40%)	CIFAR-100N (Noisy, ≈ 40%)
0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8	0.2	0.4	0.6	0.8
SOP+	SmallL	0.1	0.2	1.0	4.0	0.8	3.6	11.5	26.2	3.5	8.7	16.3	27.4
Margin	29.7	25.7	22.5	19.7	54.6	52.1	48.5	44.6	61.5	56.8	51.6	46.2
kCenter	19.0	18.8	19.1	18.6	40.0	41.1	41.6	42.1	37.5	38.7	39.9	40.4
Forget	17.0	17.7	17.5	17.8	37.7	38.8	39.2	40.2	37.9	34.6	33.0	36.8
GraNd	67.5	41.7	28.6	21.5	91.5	79.4	64.2	50.1	93.9	57.3	61.2	49.3
SSP	25.2	23.7	21.6	19.5	48.5	46.4	43.2	42.1	52.7	52.3	46.8	43.0
Moderate	6.6	7.1	8.4	13.5	31.7	33.4	34.7	40.0	33.2	54.6	60.2	64.6
Pr4ReLU	17.0	18.7	19.3	22.5	38.7	42.7	43.5	46.5	28.3	29.1	33.3	37.2
Table 5: Ratio of correctly re-labeled noisy examples in the selected subset (denoted as % Correct).
Re-label Model	Selection Methods	CIFAR-10N (Random)
Test Acc.	% Noisy	% Correct
SOP+	kCenter	86.3	19.0	75.2
Forget	82.4	17.0	61.7
Pr4ReLU	88.1	17.0	90.3
Table 6: Data pruning performance on ImageNet with a  $20\%$  synthetic label noise.
Re-label Model	Selection Methods	ImageNet-1K (Syn, ≈20%)
0.05	0.1	0.2	0.4
SOP+	Uniform	27.8	42.5	52.7	59.2
SmallL	22.8	31.4	42.7	54.4
Forget	4.1	8.3	50.6	57.2
Pr4ReLU_B	30.2	44.3	53.5	60.0
examples. Here, we compare Prune4ReLU with kCenterGreedy and Forgetting on CIFAR-10N (Random) with the selection ratio of 0.2, where the ratio of noisy examples (i.e., %Noisy) in the selected subset of each method is similar (i.e., from  $17\%$  to  $19\%$ ). Although these methods select almost equal amounts of noisy examples, there were differences in the self-correctability (i.e., %Correct) of the selected subsets. Noisy examples selected by Prune4ReLU are mostly self-correctable as it maximizes the total neighborhood confidence of the training set. In contrast, those selected by existing data pruning methods such as kCenter and Forget are not guaranteed to be self-correctable. This result confirms that Prune4ReLU not only selects a low ratio of noisy examples but also considers the quality of the selected subset in terms of maximizing re-labeling accuracy. Therefore, Prune4ReLU fully takes advantage of the Re-labeling methods.
4.6 Results on ImageNet-N with Synthetic Label Noise
We further validate the efficacy of Prune4ReLU on ImageNet-N by injecting synthetic label noise of  $20\%$  to the commonly-used benchmark dataset ImageNet-1K (see Appendix D for details). Table 6 shows the test accuracy of Prune4ReLU and three representative sample selection baselines with varying selection ratios of  $\{0.05, 0.1, 0.2, 0.4\}$ . Similar to the result in Section 4.2, Prune4ReLU consistently outperforms the baselines by up to  $8.6\%$ , thereby adding more evidence of the superiority of Prune4ReLU. In addition, owing to its great computation efficiency, Prune4ReLU is able to scale to ImageNet-N, a large-scale dataset with approximately 1.2M training examples.
We present a noise-robust data pruning method for Re-labeling, called Prune4ReL, that finds a subset that maximizes the total neighborhood confidence of the training examples, thereby maximizing the re-labeling accuracy and generalization performance. To identify a subset that maximizes the re-labeling accuracy, Prune4ReL introduces a novel metric, the reduced neighborhood confidence, which is the prediction confidence of each neighbor example in the selected subset, and the effectiveness of this metric in estimating the re-labeling capacity of a subset is theoretically and empirically validated. Furthermore, we optimize Prune4ReL with an efficient greedy algorithm that expands the subset by selecting the example that contributes the most to increasing the total reduced neighborhood confidence. Experimental results demonstrate the substantial superiority of Prune4ReL compared to existing data pruning methods in the presence of label noise.
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00862, DB4DL: High-Usability and Performance In-Memory Distributed DBMS for Deep Learning and No. 2022-0-00157, Robust, Fair, Extensible Data-Centric Continual Learning).
[1] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409, 2017.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-shot Learners. In NeurIPS, pages 1877–1901, 2020.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, pages 8748-8763, 2021.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929, 2020.
[6] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. In NeurIPS, pages 19523-19536, 2022.
[7] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with Noisy Labels Revisited: A Study using Real-world Human Annotations. arXiv preprint arXiv:2110.12088, 2021.
[8] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision Database: Visual Learning and Understanding from Web Data. arXiv preprint arXiv:1708.02862, 2017.
[9] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from Massive Noisy Labeled Data for Image Classification. In CVPR, pages 2691-2699, 2015.
[10] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from Noisy Labels with Deep Neural Networks: A Survey. TNNLS, pages 1-19, 2022.
[11] Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing Unclean Samples for Robust Deep Learning. In ICML, pages 5907-5915, 2019.
[12] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised Data Augmentation for Consistency Training. In NeurIPS, pages 6256-6268, 2020.
[13] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with Noisy Labels as Semi-supervised Learning. arXiv preprint arXiv:2002.07394, 2020.
[14] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. arXiv preprint arXiv:1812.05159, 2018.
[15] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In NeurIPS, pages 20596-20607, 2021.