Title: Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data Pruning

Data Subset Selection

Noisy Labels

Relabeling

Self-training
Abstract: Data pruning, which aims to downsize a large training set into a small informative subset, is crucial for reducing the enormous computational costs of modern deep learning. Though large-scale data collections invariably contain annotation noise and numerous robust learning methods have been developed, data pruning for the noise-robust learning scenario has received little attention. With state-of-the-art Re-labeling methods that self-correct erroneous labels while training, it is challenging to identify which subset induces the most accurate re-labeling of erroneous labels in the entire training set. In this paper, we formalize the problem of data pruning with re-labeling. We first show that the likelihood of a training example being correctly re-labeled is proportional to the prediction confidence of its neighborhood in the subset. Therefore, we propose a novel data pruning algorithm, Prune4Rel, that finds a subset maximizing the total neighborhood confidence of all training examples, thereby maximizing the re-labeling accuracy and generalization performance. Extensive experiments on four real and one synthetic noisy datasets show that Prune4Rel outperforms the baselines with Re-labeling models by up to 9.1% as well as those with a standard model by up to 21.6%.

Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracy

Abstract

Data pruning , which aims to downsize a large training set into a small informative subset, is crucial for reducing the enormous computational costs of modern deep learning. Though large-scale data collections invariably contain annotation noise and numerous robust learning methods have been developed, data pruning for the noise-robust learning scenario has received little attention. With state-ofthe-art Re-labeling methods that self-correct erroneous labels while training, it is challenging to identify which subset induces the most accurate re-labeling of erroneous labels in the entire training set. In this paper, we formalize the problem of data pruning with re-labeling . We first show that the likelihood of a training example being correctly re-labeled is proportional to the prediction confidence of its neighborhood in the subset. Therefore, we propose a novel data pruning algorithm, Prune4ReL, that finds a subset maximizing the total neighborhood confidence of all training examples, thereby maximizing the re-labeling accuracy and generalization performance. Extensive experiments on four real and one synthetic noisy datasets show that Prune4ReL outperforms the baselines with Re-labeling models by up to 9.1 % as well as those with a standard model by up to 21.6 % .

1 Introduction

By virtue of ever-growing datasets and the neural scaling law [1, 2], where the model accuracy often increases as a power of the training set size, modern deep learning has achieved unprecedented success in many domains, e.g. , GPT[3], CLIP [4], and ViT [5]. With such massive datasets, however, practitioners often suffer from enormous computational costs for training models, tuning their hyper-parameters, and searching for the best architectures, which become the main bottleneck of development cycles. One popular framework to reduce these costs is data pruning , which reduces a huge training set into a small subset while preserving model accuracy. Notably, Sorscher et al. [6] have shown that popular data pruning approaches can break down the neural scaling law from power-law to exponential scaling, meaning that one can reach a desired model accuracy with much fewer data. Despite their great success, the impact of label noise on data pruning has received little attention, which is unavoidable in real-world data collection [7, 8, 9].

Noisy labels are widely known to severely degrade the generalization capability of deep learning, and thus numerous robust learning strategies have been developed to overcome their negative effect in deep learning [10]. Among them, Re-labeling [11], a family of methods that identify wrongly labeled examples and correct their labels during training by a self-correction module such as self-consistency regularization [12], has shown state-of-the-art performance. For example, the performance of DivideMix [13] trained on the CIFAR-10N [7] dataset containing real human annotation noise is nearly identical to that of a standard model trained on the clean CIFAR-10 dataset. Consequently, it is evident that this excellent performance of re-labeling must be carefully considered when designing a framework for data pruning under label noise.

Re-labeling Coverage

82

0.2

0.4

Incorrect

re-labeling

Correct

re-labeling

Confidence

0.6

0.8

High

Low

84

86

88

90

92

Neighborhood

{   }Selected Set

(a) Pruning Performance.

(b) Re-labeling by Neighborhood.

(c) Goal of Prune4ReL.

In this paper, we formulate a new problem of data pruning with re-labeling for a training set with noisy labels, which aims to maximize the generalization power of the selected subset with expecting that a large proportion of erroneous labels are self-corrected ( i.e. , re-labeled). Unfortunately, prior data pruning and sample selection algorithms are not suitable for our problem because the re-labeling capability is not taken into account, and have much room for improvement as shown in Figure 1(a). Popular data pruning approaches (denoted as Forgetting [14] and GraNd [15] in blue and yellow, respectively) favor hard ( i.e. , uncertain) examples because they are considered more beneficial for generalization [16]; however, because it is very difficult to distinguish between hard examples and incorrectly-labeled examples [17], many of the incorrectly-labeled examples can be included in the subset causing unreliable re-labeling. In addition, the small-loss trick [18] (denoted as SmallLoss in green) for sample selection favors easy examples because they are likely to be correctly labeled; however, they are not beneficial for generalization at a later stage of training. Therefore, this new problem necessitates the development of a new data pruning approach.

Accordingly, we suggest a completely novel approach of finding a subset of the training set such that the re-labeling accuracy of all training examples is preserved as much as possible with the model trained on the subset . The first challenge in this direction is how to estimate whether each example can be re-labeled correctly even before fully training models on the candidate subset. The second challenge is how to find the subset that maximizes the overall re-labeling accuracy of the entire training set in an efficient manner.

Addressing these two challenges, we develop a novel framework, called Prune4ReL . For the first challenge, we define the concept of the neighborhood confidence which is the sum of the prediction confidence of each neighbor example in the selected subset. We show that, as in Figure 1(b), an example with high neighborhood confidence is likely to be corrected by Re-labeling methods. We further provide theoretical and empirical evidence of this argument. For the second challenge, we show that the overall re-labeling accuracy is maximized by selecting a subset that maximizes the sum of its reachable neighborhood confidence for all training examples, as shown in Figure 1(c). Furthermore, because enumerating all possible subsets is a combinatorial optimization problem which is NP-hard [19], we provide an efficient greedy selection algorithm that expands the subset one by one by choosing the example that most increases the overall neighborhood confidence.

Extensive experiments on four real noisy datasets, CIFAR-10N, CIFAR-100N, WebVision, and Clothing-1M, and one synthetic noisy dataset on ImageNet-1K show that Prune4ReL consistently outperforms the eight data pruning baselines by up to 9.1%. Moreover, Prune4ReL with Re-labeling models significantly outperforms the data pruning baselines with a standard model by up to 21.6 % , which reaffirms the necessity of data pruning with re-labeling.

2 Preliminary and Related Work

2.1 Robust Learning under Noisy Labels

A long line of literature has been proposed to improve the robustness of DNNs against label noise, and refer to [10] for a detailed survey for deep learning with noisy labels. While some studies have

focused on modifying the architectures [20, 21, 22] and the loss functions [23, 24, 25], others have opted for sample selection approaches [26, 18, 27, 28] that select as many clean examples as possible while discarding noisy examples based on a certain cleanness criterion, e.g. , small-loss [18]. Note that, these works do not consider the compactness or efficiency of the selected subset. Meanwhile, to further exploit even noisy examples for training, Re-labeling [11, 29] approaches try to correct noisy labels and reuse them for training with a re-labeling module, e.g. , a heuristic rule [11]. Notably, according to recent benchmark studies on real-world noisy datasets [7], a family of Re-labeling methods with self-consistency regularization [12] has shown state-of-the-art performance.

In general, Re-labeling methods with self-consistency regularization are based on

where ˜ D = { ( x i , ˜ y i ) } i m =1 is a given noisy training set obtained from a noisy joint distribution X × ˜ Y , θ is a classifier, A is a strong data augmentation, C θ ( · ) is a prediction confidence score, and δ is a threshold to identify confident (clean) examples for the supervised loss L ce ( x, ˜ y ; θ ) , i.e. , crossentropy. The noisy labels are implicitly corrected by the self-consistency loss L reg ( x ; θ ) leveraging the power of strong augmentations [30]. DivideMix [13], ELR+ [31], CORES [32], and SOP+ [33] are popular approaches belonging to this family. DivideMix uses a co-training framework to further improve the re-labeling accuracy, and SOP+ introduces additional learnable variables combined with a self-consistency loss. For simplicity, we call this Re-labeling family with self-consistency regularization as 'Re-labeling' throughout the paper. Despite their effectiveness, Re-labeling models tend to require more computation time due to additional data augmentations, multiple backbones, and longer training epochs, which raises a need to enhance its efficiency.

2.2 Data Pruning

In order to achieve high generalization performance with a selected subset, most data pruning approaches often prioritize the selection of hard or uncertain examples. Specifically, uncertaintybased methods [34, 35, 36] favor selecting less confident examples over more confident ones, as the former is assumed to be more informative than the latter. Similarly, geometry-based methods [37, 38] focus on removing redundant examples that are close to each other in the feature space, and loss-based methods [14, 15, 39, 40] favor selecting the examples with a high loss or gradient measured during training. However, these existing methods may not be effective in realistic scenarios under label noise because noisy examples also exhibit high uncertainty and could be wrongly considered informative for training [14]. Meanwhile, some recent works reported that existing data pruning methods do not work well at high pruning ratios [41, 42]. To alleviate this drawback, AL4DP [41] shows that mixing various levels of uncertain examples is better for data scarcity, Moderate [43] aims to select examples with the distances close to the median, and CCS [42] proposes a coverage-based method that jointly considers data coverage with sample importance. While a few works attempted to improve the robustness of sample selection against label noise by filtering the noise [19], no work yet considers the effect of data pruning on noise-robust learners such as Re-labeling.

3 Methodology

We formalize a problem of data pruning with re-labeling such that it finds the most informative subset S , where a model θ S trained on S maximizes the re-labeling accuracy of the entire noisy training set ˜ D = { ( x i , ˜ y i ) } i m =1 2 . Formally, we aim to find an optimal subset S ∗ such that

where y ∗ is the ground-truth label of a noisy example x , f ( x ; θ S ) ∈ R c is a c -way class prediction of the example x from the Re-labeling model θ S , and s is the target subset size.

Finding the optimal subset S ∗ through direct optimization of Eq. (2) is infeasible because the groundtruth label y ∗ is unknown in practice. In addition, the subset should be found at the early stage of

training, i.e. , in a warm-up period, to reduce the computational cost [16]. To achieve these goals in an accurate and efficient way, in Section 3.1, we first introduce a new metric, the reduced neighborhood confidence , that enables estimating the re-labeling capacity of a subset even in the warm-up period. Then, in Section 3.2, we propose a new data pruning algorithm Prune4ReL using this reduced neighborhood confidence to find a subset that maximizes the re-labeling accuracy.

3.1 Reduced Neighborhood Confidence

As a measurement of estimating the re-labeling accuracy, we use the confidence of neighbor examples for each target noisy example x , because the noisy examples are known to be corrected by their clean neighbor examples with self-consistency regularization [44]. Specifically, once an augmentation of a noisy example has a similar embedding to those of other clean neighbors in the representation space, the self-consistency loss can force the prediction of the noisy example to be similar to those of other clean neighbors as a way of re-labeling. This property is also evidenced by a theory of re-labeling with a generalization bound [45]. Thus, the neighboring relationship among examples can be a clear clue to estimate the re-labeling accuracy even in the early stage of training.

We define a neighborhood and its reduced neighborhood confidence to utilize the relationship of neighboring examples in Definitions 3.1 and 3.2.

̸

Definition 3.1. (NEIGHBORHOOD). Let B ( x i ) = { x : ||A ( x i ) -x || ≤ ϵ } be a set of all possible augmentations from the original example x i using an augmentation function A . Then, given a noisy training set ˜ D , a neighborhood of x i is defined as N ( x i ) = { x ∈ ˜ D : B ( x i ) ∩ B ( x ) = ∅} , which is the set of examples that are reachable by the augmentation A .

Definition 3.2. (REDUCED NEIGHBORHOOD CONFIDENCE). The reduced neighborhood confidence C N ( x i ; S ) of an example x i is the sum of the prediction confidence C ( · ) of its neighbors x j ∈ N ( x i ) in a given reduced ( i.e. , selected) subset S , which is formalized as

and its empirical reduced neighborhood confidence is computed by using the cosine similarity among the augmentations of all possible pairs of example embeddings,

where sim ( · ) is the cosine similarity between the augmentations A ( x ) of two different examples in the embedding space, and τ is a threshold to determine whether the two examples belong to the same neighborhood. Unlike Eq. (3), Eq. (4) is calculated as a weighted sum of prediction confidences with cosine similarity to approximate the likelihood of belonging to the neighborhood.

Based on these definitions, we investigate the theoretical evidence of employing the reduced neighborhood confidence as a means to estimate the re-labeling capacity of a subset.

Theoretical Evidence. A subset S with a high value of the total reduced neighborhood confidence, the sum of the reduced neighborhood confidence of each example in S , allows a Re-labeling model to maximize its re-labeling accuracy in the entire training set. We formally support this optimization by providing a theoretical analysis that extends the generalization bound in the prior re-labeling theory [45] to data pruning.

Assumption 3.3. (EXPANSION AND SEPARATION). Following the assumption in [45], the α -expansion and β -separation assumptions hold for the training set ˜ D . The α -expansion means that an example is reachable to the α number of augmentation neighbors on average, i.e. , E x ∈ ˜ D [ |N ( x ) | ] = α . The β -separation means that data distributions with different ground-truth classes are highly separable, such that the average proportion of the neighbors from different classes is as small as β .

Under these assumptions, we can obtain a training accuracy (error) bound of a Re-labeling model trained on a subset S as in Theorem 3.4.

Theorem 3.4. Assume that a subset S ∈ ˜ D follows α S -expansion and β S -separation, where α S ≤ α . Then, the training error of a Re-labeling model θ S trained on S is bounded by the inverse of the total reduced neighborhood confidence ∑ x ∈ ˜ D C N ( x ; S ) such that,

where θ M is a model trained with the supervised loss in Eq. (1) on a given clean set M⊂S .

Proof. We extend the label denoising theorem in [45] by incorporating the influence of the subset to the expansion factor α S . The complete proof is available in Appendix A.

Since β S is usually very small, its effect on the error bound is negligible. Then, the bound highly depends on the total reduced neighborhood confidence. That is, as the total reduced neighborhood confidence increases, the error bound becomes tighter. This theorem supports that we can utilize the reduced neighborhood confidence for the purpose of maximizing the re-labeling accuracy.

Empirical Evidence. To empirically support Theorem 3.4, we validate the correlation between the empirical reduced neighborhood confidence 3 and the re-labeling accuracy using CIFAR-10N, which is a real-world noisy benchmark dataset.

Specifically, we train DivideMix [13] on the 20 % randomly selected subset S for a warm-up training epoch of 10 and calculate the empirical reduced neighborhood confidence ˆ C N ( x ; S ) for the entire training set. Next, we fully train DivideMix [13] on the random subset S . Last, we divide the entire training set into 15 bins according to the obtained ˆ C N ( x ; S ) and verify the average re-labeling accuracy for each bin.

Empirical Neighborhood Conf.

Re-labeling Accuracy

0.60

0.70

0.80

0.90

1.00

4

6

8

10

12

∆

∆

Converge

∆

Utility

Decrease

መ

𝐶𝑁(𝑥; 𝑆

)

Figure 2 shows how the re-labeling accuracy changes according to the empirical reduced neighborhood confidence in Eq. (4). (The term 'empirical' is simply omitted hereafter.) The re-labeling accuracy shows a strong positive correlation with the reduced neighborhood confidence. Interestingly, as the neighborhood confidence increases, its utility in improving the re-labeling accuracy decreases, eventually reaching a convergence point after surpassing a certain threshold.

3.2 Data Pruning by Maximizing Neighborhood Confidence Coverage

We present a new data pruning algorithm called Prune4ReL which optimizes the total reduced neighborhood confidence defined in Eq. (4). This objective is equivalent to identifying a subset that maximizes the re-labeling accuracy on the entire training set, as justified in Theorem 3.4. Therefore, the objective of Prune4ReL is to find the subset S ∗ , which is formulated as

where σ ( z ) is a utility function of the reduced neighborhood confidence ˆ C N ( x i ; S ) in improving the re-labeling accuracy. By the observation in Figure 2, we define σ ( z ) as a non-decreasing and concave function where σ (0) = 0 . In our implementation, we use the positive of the tanh function as the utility function, i.e. , σ ( z ) = tanh ( z ) . However, directly solving Eq. (6) is computationally expensive and impractical due to its NP-hard nature as a Set-Cover problem [19]. Accordingly, we employ an approximation solution to efficiently address this combinatorial optimization.

Optimization with Greedy Approximation. We present a practical solution for solving the optimization problem stated in Eq. (6) using a greedy approximation. The objective function satisfies both the monotonicity and submodularity conditions, indicating that the return of the objective function monotonically increases and the marginal benefit of adding an example decreases as the subset grows. Thus, a greedy sample selection can be employed as in Algorithm 1.

In detail, we begin with an empty set S and initialize the reduced neighborhood confidence ˆ C N to 0 (lowest confidence) for all training examples (Line 1). Next, at every step, we select an example x that maximizes the marginal benefit σ ( ˆ C N ( x ) + C ( x )) -σ ( ˆ C N ( x )) of Eq. (6), and update the reduced neighborhood confidence ˆ C N based on the similarity scores (Lines 3-7). To further improve robustness and efficiency, we introduce a class-balanced version, Prune4ReL B , of which the detailed process is elaborated in Appendix B.

Algorithm 1 Greedy Neighborhood Confidence

INPUT: ˜ D : training set, s : target subset size, and C ( x ) : confidence from warm-up classifier 1: Initialize S ← ∅ ; ∀ x ∈ ˜ D , ˆ C N ( x ) = 0 2: repeat 3: x =argmax x ∈ ˜ D\S σ ( ˆ C N ( x )+ C ( x )) -σ ( ˆ C N ( x )) 4: S = S ∪ { x } 5: for all v ∈ ˜ D do 6: ˆ C N ( v ) += ✶ [ sim ( x,v ) ≥ τ ] · sim ( x, v ) · C ( x ) 7: until |S| = s OUTPUT: Final selected subset S

In Theorem 3.5, we guarantee the selected subset S obtained by our greedy solution achieves a (1 -1 /e ) -approximation of the optimum.

Theorem 3.5. Since Eq. (6) , denoted as OBJ , is a monotone, submodular, and non-negative function on x , the greedy solution provides a set with a (1 -1 /e ) -approximation of the optimum. Formally,

Proof. We prove the monotonicity and submodularity of Eq. (6). If the two conditions are satisfied, Eq. (7) naturally holds. See Appendix C for the complete proof.

Time Complexity Analysis. We analyze the time complexity of our greedy approximation in Algorithm 1. At each step, Prune4ReL takes the time complexity of O ( m log m ) + O ( md ) , where m is the training set size and d is the embedding dimension size of the warm-up classifier. Specifically, in Line 3, sampling an example with the largest marginal benefit of confidence takes O ( m log m ) , and in Lines 5-6, updating the reduced neighborhood confidence of all training examples takes O ( md ) . In addition, with Prune4ReL B , the time complexity is reduced to O ( m log( m/c )) + O ( md ) because it iteratively selects the example with the largest marginal benefit within each class subset, which is lower than the time complexity of a similar distance-based data pruning work, kCenterGreedy [38] aiming to maximize the distance coverage of a selected subset to the entire training set by a greedy approximation. At each iteration, kCenterGreedy's runtime is O ( mk t ) , where k t is the size of the selected set at iteration t [46]. Note that, its time complexity increases as the subset size grows, which hinders its usability on a large-scale dataset. In Section 4.2, we empirically show that Prune4ReL is scalable to prune Clothing-1M, a large dataset with 1M examples, whereas kCenterGreedy is not.

4 Experiments

4.1 Experiment Setting

Datasets. We first perform the data pruning task on four real noisy datasets, CIFAR-10N, CIFAR100N, Webvision, and Clothing-1M. CIFAR-10N and CIFAR-100N [7] contain human re-annotations of 50K training images in the original CIFAR-10 and CIFAR-100 [47]. Specifically, each training image in CIFAR-10N contains three noisy labels, called Random 1,2,3, which are further transformed into the Worst-case label. Each image in CIFAR-100N contains one noisy label. WebVision [8] and Clothing-1M [9] are two large-scale noisy datasets. WebVision contains 2.4M images crawled from the Web using the 1,000 concepts in ImageNet-1K [48]. Following prior work [49], we use mini-WebVision consisting of the first 50 classes of the Google image subset with approximately 66K training images. Clothing-1M consists of 1M training images with noisy labels and 10K clean test images collected from online shopping websites. Additionally, a large-scale synthetic noisy dataset, which we call ImageNet-N, is included in our experiments. It consists of 1.2M training images, which are the training images of ImageNet-1K [48] with asymmetric label noise. See Appendix D for details of the noise injection.

Algorithms. We compare Prune4ReL with a random selection from a uniform distribution, Uniform, a clean sample selection algorithm, SmallLoss [26], and six data pruning algorithms including Margin [50], k -CenterGreedy [38], Forgetting [14], GraNd [15], SSP [6], and Moderate [43]. SmallLoss favors examples with a small loss. For data pruning algorithms, (1) Margin selects examples in the

increasing order of the difference between the highest and the second highest softmax probability; (2) k -CenterGreedy selects k examples that maximize the distance coverage to the entire training set; (3) Forgetting selects examples that are easy to be forgotten by the classifier throughout the warm-up training epochs; (4) GraNd uses the average norm of the gradient vectors to measure the contribution of each example to minimizing the training loss; (5) SSP leverages a self-supervised pre-trained model to select the most prototypical examples; and (6) Moderate aims to select moderately hard examples using the distances to the median.

Implementation Details. We train two representative Re-labeling models, DivideMix [13] and SOP+[33] for our experiments. The hyperparameters for DivideMix and SOP+ are favorably configured following the original papers. Following the prior Re-labeling work [13, 33], for CIFAR10N and CIFAR-100N, PreAct Resnet-18 [51] is trained for 300 epochs using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The initial learning rate is 0.02, and it is decayed with a cosine annealing scheduler. For WebVision, InceptionResNetV2 [52] is trained for 100 epochs with a batch size of 32. For Clothing-1M, we use ResNet-50 [53] pre-trained on ImageNet and fine-tune it for 10 epochs with a batch size of 32. The initial learning rates of WebVision and Clothing-1M are 0.02 and 0.002, which are dropped by a factor of 10 at the halfway point of the training epochs. For ImageNet-N, ResNet-50 [53] is trained for 50 epochs with a batch size of 64 and an initial learning rate of 0.02 decayed with a cosine annealing scheduler.

For data pruning algorithms, following prior work [16], we perform sample selection after 10 warmup training epochs for CIFAR-10N, WebVision, and ImageNet-N, and 30 warm-up epochs for CIFAR-100N. For Clothing-1M, we perform the sample selection after 1 warm-up training epoch from the ImageNet pre-trained ResNet-50. The hyperparameters for all data pruning methods are favorably configured following the original papers. For Prune4ReL, we set its hyperparameter τ to 0.975 for CIFAR-10N, to 0.95 for CIFAR-100N, WebVision, and ImageNet-N, and to 0.8 for Clothing-1M. More implementation details can be found in Appendix E. All methods are implemented with PyTorch 1.8.0 and executed on NVIDIA RTX 3080 GPUs. The code is available at https: //github.com/kaist-dmlab/Prune4Rel .

Evaluation. For CIFAR-10N, CIFAR-100N, and WebVision, we select the subset with the selection ratios { 0.2, 0.4, 0.6, 0.8 } . For Clothing-1M and ImageNet-N, we construct the subset with { 0.01, 0.05, 0.1, 0.2 } and { 0.05, 0.1, 0.2, 0.4 } selection ratios, respectively. We measure the test accuracy of the Re-labeling models trained from scratch on the selected subset. Every experiment is run three times, and the average of the last accuracy is reported. For CIFAR-10N with the Random noise, we average the test accuracy of the models trained using the three noisy labels.

Selection Method

Elapsed Time (min)

Selection Ratio

Selection Ratio

Test Accuracy (%)

70

65

60

65

45

55

35

75

0.2

0.4

0.6

0.8

0.01

0.05

0.1

0.2

60

20

40

0

80

100

inf.

(a) WebVision.

(b) Clothing-1M.

(c) GPU Time for Selection.

Uniform

Margin

Forget

Moderate

SmallL

kCenter

GraNd

SmallL

Margin

Forget

Moderate

GraNd

kCenter

Prune4ReLg

4.2 Main Results on Real Noisy Datasets

Test Accuracy. Table 1 summarizes the test accuracy of eight baselines and Prune4ReL on CIFAR10N and CIFAR-100N trained with two Re-labeling models. Overall, Prune4ReL consistently achieves the best performance for all datasets across varying selection ratios. Numerically, Prune4ReL improves DivideMix and SOP+ by up to 3.7% and 9.1%, respectively. Compared with six data pruning baselines, they show rapid performance degradation as the size of the subset decreases; most of them, which are designed to favor hard examples, tend to select a large number of noisy examples, resulting in unreliable re-labeling. While Moderate selects moderately hard examples, it is still worse than Prune4ReL since it is not designed for noise-robust learning scenarios with Re-labeling models. On the other hand, SmallLoss, a clean sample selection baseline, shows poor performance in CIFAR-10N (Random), because this dataset contains a relatively low noise ratio, and selecting clean examples is less critical. Although SmallLoss shows robust performance in CIFAR-100N, it is worse than Prune4ReL because it loses many informative noisy examples that help generalization if re-labeled correctly. Meanwhile, Uniform is a fairly robust baseline as it selects easy (clean) and hard (noisy) examples in a balanced way; many selected noisy examples may be relabeled correctly by other selected clean neighbors, resulting in satisfactory test accuracy.

Similarly, Figures 3(a) and 3(b) visualize the efficacy of the baselines and Prune4ReL on the WebVision and Clothing-1M datasets. We train SOP+ on WebVision and DivideMix on Clothing-1M. Similar to CIFAR-N datasets, Prune4ReL achieves better performance than existing baselines on two datasets. Quantitatively, Prune4ReL outperforms the existing sample selection methods by up to 2.7% on WebVision with a selection ratio of 0.4. This result confirms that the subset selected by Prune4ReL, which maximizes the total neighborhood confidence of the training set, successfully maintains the performance of Re-labeling models and is effective for model generalization.

Efficiency. In Figure 3(c), we further show the GPU time taken for selecting subsets within the warm-up training. We train SOP+ on WebVision with a selection ratio of 0.8. Powered by our efficient greedy approximation, Prune4ReL fastly prunes the dataset in a reasonable time. GraNd takes almost 10 times longer than SmallLoss or Margin, since it trains the warm-up classifier multiple times for the ensemble. kCenterGreedy is infeasible to run in our GPU configuration due to its huge computation and memory costs.

Selection Ratio

Test Accuracy (%)

0.2

0.4

0.6

0.8

50

55

60

65

4.3 Necessity of Data Pruning with Re-labeling under Label Noise

Table 2 shows the superiority of the Re-labeling models over the standard learning model, i.e. , only with the cross-entropy loss, for data pruning under label noise. When combined with the data pruning methods, the performance of the Re-labeling models such as DivideMix and SOP+ significantly surpasses those of the standard models on CIFAR-10N and CIFAR-100N by up to 21.6 % . That is, the re-labeling capacity can be well preserved by a proper data pruning strategy. This result demonstrates the necessity of data pruning for the Re-labeling models in the presence of noisy labels.

4.4 Ablation Studies

Effect of Confidence Metrics. Prune4ReL can be integrated with various metrics for the confidence of predictions in Eq. (6). In our investigation, we consider two widely-used metrics: (1) MaxProb, which represents the maximum value of softmax probability and (2) DiffProb, which measures the difference between the highest and second highest softmax probabilities. Table 3 shows the effect of the two confidence metrics on the test accuracy of SOP+ on CIFAR-10N (Worst) and CIFAR100N. The result indicates that both metrics perform similarly and yield higher accuracy compared to existing data pruning baselines, which demonstrates Prune4ReL is open to the choice of the confidence metric.

Effect of Neighborhood Size. Prune4ReL involves a hyperparameter τ in Eq. (6) to determine the neighborhood of each example, where a larger (smaller) value introduces a fewer (more) number of neighbors. Figure 4 shows the effect of τ ∈ { 0 . 93 , 0 . 95 , 0 . 99 } on the test accuracy of SOP+ trained on CIFAR-100N with varying selection ratios. In general, Prune4ReL shows better or comparable performance than other sample selection baselines. Among them, Prune4ReL with τ = 0 . 95 shows a satisfactory accuracy across varying selection ratios. With a smaller value of τ = 0 . 93 , it shows the best performance in a high selection ratio of 0.8, but it becomes less effective in low selection ratios, due to the increasing influence of noisy examples caused by a relatively large neighborhood size. On the contrary, with a large value of τ = 0 . 99 , it primarily selects clean yet easy examples due to a small neighborhood size, leading to a relatively less improvement in performance.

4.5 In-depth Analysis of Noisy Examples in Selected Subset

Noise Ratio of Selected Subset. Table 4 shows the ratio of noisy examples in the subset selected by each sample selection method. SmallLoss shows a very low ratio of noisy examples in the subset because it prefers clean examples. Many data pruning methods, including Margin, GraNd, SSP, and Moderate (for CIFAR-100N), tend to select a higher ratio of noisy examples compared with that of each original dataset since they prefer to select hard examples. On the other hand, Prune4ReL selects a low ratio of noisy examples when the subset size is small and gradually increases the noise ratio as the subset size increases. This result indicates that Prune4ReL expands the confident subset through Algorithm 1i.e. , selecting the most confident (clean) examples first and then trying to select less confident (hard or noisy) neighbors to ensure accurate re-labeling. While some baselines, such as kCenterGreedy, Forget, and Moderate (for CIFAR-10N), also select a somewhat low ratio of noisy examples, their data pruning performances are worse than Prune4ReL because the quality (or self-correctability) of noisy examples is not considered when selecting the subset, which is further investigated in Table 5.

Self-correctability of Selected Noisy Examples. Table 5 shows the self-correctability of selected subsets, which indicates the ratio of correctly re-labeled noisy examples out of all selected noisy

examples. Here, we compare Prune4ReL with kCenterGreedy and Forgetting on CIFAR-10N (Random) with the selection ratio of 0.2, where the ratio of noisy examples ( i.e. , % Noisy ) in the selected subset of each method is similar ( i.e. , from 17 % to 19 % ). Although these methods select almost equal amounts of noisy examples, there were differences in the self-correctability ( i.e. , % Correct ) of the selected subsets. Noisy examples selected by Prune4ReL are mostly selfcorrectable as it maximizes the total neighborhood confidence of the training set. In contrast, those selected by existing data pruning methods such as kCenter and Forget are not guaranteed to be self-correctable. This result confirms that Prune4ReL not only selects a low ratio of noisy examples but also considers the quality of the selected subset in terms of maximizing re-labeling accuracy. Therefore, Prune4ReL fully takes advantage of the Re-labeling methods.

4.6 Results on ImageNet-N with Synthetic Label Noise

We further validate the efficacy of Prune4ReL on ImageNet-N by injecting synthetic label noise of 20 % to the commonly-used benchmark dataset ImageNet-1K (see Appendix D for details). Table 6 shows the test accuracy of Prune4ReL and three representative sample selection baselines with varying selection ratios of { 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 } . Similar to the result in Section 4.2, Prune4ReL consistently outperforms the baselines by up to 8.6 % , thereby adding more evidence of the superiority of Prune4ReL. In addition, owing to its great computation efficiency, Prune4ReL is able to scale to ImageNet-N, a large-scale dataset with approximately 1.2M training examples.

5 Conclusion

We present a noise-robust data pruning method for Re-labeling, called Prune4ReL, that finds a subset that maximizes the total neighborhood confidence of the training examples, thereby maximizing the re-labeling accuracy and generalization performance. To identify a subset that maximizes the relabeling accuracy, Prune4ReL introduces a novel metric, the reduced neighborhood confidence , which is the prediction confidence of each neighbor example in the selected subset, and the effectiveness of this metric in estimating the re-labeling capacity of a subset is theoretically and empirically validated. Furthermore, we optimize Prune4ReL with an efficient greedy algorithm that expands the subset by selecting the example that contributes the most to increasing the total reduced neighborhood confidence. Experimental results demonstrate the substantial superiority of Prune4ReL compared to existing data pruning methods in the presence of label noise.

References

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409 , 2017.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08361 , 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-shot Learners. In NeurIPS , pages 1877-1901, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML , pages 8748-8763, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929 , 2020.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. In NeurIPS , pages 19523-19536, 2022.

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with Noisy Labels Revisited: A Study using Real-world Human Annotations. arXiv preprint arXiv:2110.12088 , 2021.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision Database: Visual Learning and Understanding from Web Data. arXiv preprint arXiv:1708.02862 , 2017.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from Massive Noisy Labeled Data for Image Classification. In CVPR , pages 2691-2699, 2015.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from Noisy Labels with Deep Neural Networks: A Survey. TNNLS , pages 1-19, 2022.

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing Unclean Samples for Robust Deep Learning. In ICML , pages 5907-5915, 2019.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised Data Augmentation for Consistency Training. In NeurIPS , pages 6256-6268, 2020.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with Noisy Labels as Semi-supervised Learning. arXiv preprint arXiv:2002.07394 , 2020.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. arXiv preprint arXiv:1812.05159 , 2018.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In NeurIPS , pages 20596-20607, 2021.

Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A Comprehensive Library for Coreset Selection in Deep Learning. In DEXA , pages 181-195, 2022.

Dongmin Park, Yooju Shin, Jihwan Bang, Youngjun Lee, Hwanjun Song, and Jae-Gil Lee. Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning. In NeurIPS , pages 31416-31429, 2022.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels. In NeurIPS , pages 8536-8546, 2018.

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based Data Subset Selection for Efficient and Robust Learning. In AAAI , pages 8110-8118, 2021.

Jacob Goldberger and Ehud Ben-Reuven. Training Deep Neural Networks Using a Noise Adaptation Layer. In ICLR , 2017.

Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A New Perspective of Noisy Supervision. In NeurIPS , pages 5841-5851, 2018.

Jiangchao Yao, Jiajie Wang, Ivor W Tsang, Ya Zhang, Jun Sun, Chengqi Zhang, and Rui Zhang. Deep Learning from Noisy Image Labels with Quality Embedding. TIP , 28:1909-1922, 2018.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust Loss Functions under Label Noise for Deep Neural Networks. In AAAI , pages 1919-1925, 2017.

Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise. In NeurIPS , pages 10477-10486, 2018.

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized Loss Functions for Deep Learning with Noisy Labels. In ICML , pages 6543-6553, 2020.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning Data-driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In ICML , pages 2304-2313, 2018.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating Noisy Labels by Agreement: A Joint Training Method with Co-regularization. In CVPR , pages 13726-13735, 2020.

Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A Topological Filter for Learning with Label Noise. In NeurIPS , pages 21382-21393, 2020.

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust Curriculum Learning: From Clean Label Detection to Noisy Label Self-correction. In ICLR , 2021.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical Automated Data Augmentation with a Reduced Search Space. In NeurIPS , pages 18613-18624, 2020.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Earlylearning Regularization Prevents Memorization of Noisy Labels. In NeurIPS , pages 2033120342, 2020.

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with Instance-dependent Label Noise: A Sample Sieve Approach. arXiv preprint arXiv:2010.02347 , 2020.

Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust Training under Label Noise by Over-parameterization. In ICML , pages 14153-14172, 2022.

Dan Wang and Yi Shang. A New Active Labeling Method for Deep Learning. In IJCNN , pages 112-119, 2014.

Dan Roth and Kevin Small. Margin-based Active Learning for Structured Output Spaces. In ECML , pages 413-424, 2006.

Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class Active Learning for Image Classification. In CVPR , pages 2372-2379, 2009.

Yutian Chen, Max Welling, and Alex Smola. Super-samples from Kernel Herding. arXiv preprint arXiv:1203.3472 , 2012.

Ozan Sener and Silvio Savarese. Active Learning for Convolutional Neural Networks: A Core-Set Approach. In ICLR , 2018.

Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Gradmatch: Gradient Matching based Data Subset Selection for Efficient Deep Model Training. In ICML , pages 5464-5474, 2021.

Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for Data-efficient Training of Machine Learning Models. In ICML , pages 6950-6960, 2020.

Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active Learning is A Strong Baseline for Data Subset Selection. In NeurIPS Workshop , 2022.

Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric Coreset Selection for High Pruning Rates. ICLR , 2022.

Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning. In ICLR , 2022.

Erik Englesson and Hossein Azizpour. Consistency Regularization Can Improve Robustness to Label Noise. arXiv preprint arXiv:2110.01242 , 2021.

Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical Analysis of Self-training with Deep Networks on Unlabeled Data. arXiv preprint arXiv:2010.03622 , 2020.

Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch Active Learning at Scale. In NeurIPS , pages 1193311944, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-scale Hierarchical Image Database. In CVPR , pages 248-255, 2009.

Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels. In ICML , pages 1062-1070, 2019.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via Proxy: Efficient Data Selection for Deep Learning. arXiv preprint arXiv:1906.11829 , 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In ECCV , pages 630-645, 2016.

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the Impact of Residual Connections on Learning. In AAAI , pages 42784284, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR , pages 770-778, 2016.

Uriel Feige. A Threshold of ln n for Approximating Set Cover. Journal of the ACM (JACM) , 45(4):634-652, 1998.

Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task Curriculum Framework for Open-set Semi-supervised Learning. In ECCV , pages 438-454, 2020.

Dongmin Park, Hwanjun Song, MinSeok Kim, and Jae-Gil Lee. Task-Agnostic Undesirable Feature Deactivation Using Out-of-Distribution Data. In NeurIPS , pages 4040-4052, 2021.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A Continual Learning Survey: Defying Forgetting in Classification Tasks. PAMI , 44(7):3366-3385, 2021.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural Architecture Search: A Survey. JMLR , 20(1):1997-2017, 2019.

Peer reviews:
comments: SUMMARY:
This paper studies data pruning in noisy label setting. Built on re-labeling models, it proposes Prune4ReL that finds a subset to maximize the re-labeling accuracy. In particular, it introduces neighborhood confidence as the criteria for selection, as well as a greedy algorithm to select the subset. Evaluation show its superiority over prior data pruning methods.

STRENGTHS:
The target setting is interesting and well motivated. The proposed neighborhood confidence and Prune4ReL is sound with theoretical guarantee. In addition, the evaluations are comprehensive and experimental results look promising.

WEAKNESSES:
I don't see any major weakness, but I do have some comments:

1. The proposed data pruning method does require model training as many sample selection methods for robust learning do; I think these methods should also be considered as baselines and compared, even though they are not specifically designed for data pruning. Currently, the authors only compare SmallLoss, which is quite outdated.

2. There are some related work in robust learning field that also leverage neighborhood information, but the authors did not include and discuss them, eg, "Learning with Neighbor Consistency for Noisy Labels"

QUESTIONS:
See weakness above

LIMITATIONS:
The authors clearly stated the limitations in appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the task of data pruning, specifically in the setting of label noise. The authors propose a method to perform data pruning by maximizes the total neighborhood confidence of the training examples (which is equivalent to maximizing the relabeling accuracy). \n\nThe authors theoretically analyze this particular setting, bounding the error of a model trained on the subset that satisfies the expansion and separation assumptions from prior work. Their resulting bound is inversely proportional to the neighborhood confidence.\n\nTheir empirical results show mixed results in the comparison against existing methods; namely, in Table 1, the results are slightly better than existing baselines on a small number of tasks but predominantly match the performance of existing methods. In table 2, it also seems that the existing method Forget and kCenter seem to select similar or fewer noisy examples than the proposed method (except in the case of CIFAR-100N with 0.2 or 0.4).\n\nSTRENGTHS:\nThe authors present a theorem that shows that the error of a model trained on a subset of data is inversely related to the neighborhood confidence. They propose a method that maximizes the neighborhood confidence, which in turn minimizes this bound.\n\nThe authors show in Figure 3 that the test accuracy with SOP increases when using a subset produced by the proposed method. However, it is unclear why this is necessarily the case, since there are similar or fewer noisy examples selected by the Forget baseline method.\n\nThe authors provide ablation studies that study the effect of neighborhood threshold $\\tau$ and with different confidence metrics.\n\nWEAKNESSES:\nThe bolding strategies in Table 1 are a bit misleading; in many scenarios, a baseline achieves the same performance as the proposed method (and sometimes achieves a smaller variance), but the proposed method is still listed in bold. In fact, the reported method achieves the same performance across many of the tasks.\n\nSimilarly, the results in Table 2 show that the Forget baseline seems to select subsets with a smaller ratio of noisy examples when compared to the proposed method. However, in subsection 4.3, the authors claim that this method selects a high ratio of noisy examples.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel data pruning algorithm, Prune4ReL, that maximizes the neighborhood confidence of the entire training examples, which is proportional to the likelihood of correct re-labeling. The paper demonstrates the effectiveness of Prune4ReL on four noisy datasets, where it outperforms baselines by a large margin.\n\nSTRENGTHS:\nThe writing of this article is very clear and easy to follow up. Moreover, the methodology in this paper is also reasonable with necessary theoretical analysis. I enjoy this work very much! My only concern about this work is the experimental part.\n\nWEAKNESSES:\n1. Recent works like [1,2,3] are missing in those selected baselines. \n\n2. Lack of results on ImageNet-1K, which is the most convincing part for us.\n\n[1] Moderate: Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. ICLR-2023 \n\n[2] SSP: “Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS-2022 \n\n[3] CCS: Coverage-centric Coreset Selection for High Pruning Rates. ICLR-2023\n\nQUESTIONS:\nIn addition, the current dataset-pruning experiment mainly focuses on image classification, but this kind of task is too simple to reflect the real needs. For example, for CIFAR or even ImageNet, we don't need dataset-pruning at all, and the cost of training on these datasets is very affordable for almost all scientific teams. I think the real application scenario of dataset-pruning should be the most popular tasks such as large language model (LLM) training and multi-modal training. So, what do you think is the main difficulty of using dataset-pruning for these tasks? How is these tasks different from image classification?\n\nAt present, the performance of dataset-pruning algorithm in image classification tasks is generally not very good, for example, the performance degrades very seriously under high pruning-ratio Settings. Moreover, all existing pruning algorithms cannot outperform the ramdom selection baseline with a significant margin. However, in the field of LLM, as a contrast, some manually filtered datasets [1,2] can achieve a data set size reduction of tens of times, and performance remains unchanged or even significantly improved. So, can we say that manual filtering is still far better than algorithmic automated filtering? Or, what in the world is causing such a big gap.\n\n[1]. Textbooks Are All You Need. Arxiv. [2306.11644] \n[2]. LIMA: Less Is More for Alignment. Arxiv. [2305.11206]\n\nLIMITATIONS:\nPlease refer to Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Prune4ReL, which prunes a noisy training dataset such that the performance of a Re-labeling trained downstream model is maximized. \nUnlike previous work, the paper targets pruning a *noisy* dataset and explicitly considers the learning algorithm of the downstream model. The proposed method is specifically tailored to Re-labeling, automatically cleaning the noisy training dataset.\nThe utility function in Prune4ReL is inspired by the theory proposed in [1] and gracefully incorporated into the context of data pruning. \nThe experimental results show that most prior work suffers under noisy datasets, while Prune4ReL remains robust to the noise in the training dataset, resulting in substantial improvements over baselines.\n\n[1] Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data, ICLR2021\n\nSTRENGTHS:\nThe strength of the paper include\n- Clear presentation and easy-to-follow writing\n- The proposed method is theoretically-inspired and, maybe more importantly, easy to implement\n- The evaluation, together with the analysis, is convincing.\n- Include the analysis on why and how the baseline fail\n\nThe paper is well-organized, and the method and evaluation are solid.\n\nWEAKNESSES:\nThe paper conducts a complete study on the proposed Prune4ReL, and the following weakness is relatively minor.\n\n- Prune4ReL outperforms the baselines by a large margin, but the gap between PruneReL and uniform sampling is small.\n- Some notations are not clear.\n  - Def. 3.1: the $x$ is not in the pixel space. Instead, it's in the embedding space. The author should state the dimension of $x$ at the beginning\n  - in Thm 3.4, it should be $\\mathcal{S} \\subseteq \\tilde{D}$\n\nQUESTIONS:\nExperiment\n- Uniform sampling is usually the second-best baseline. It performs especially well in CIFAR-10N Random/Worst and Clothing-1M. Can the authors elaborate more on this? \n  - Do the authors believe the gap between uniform sampling and Prune4ReL is small due to the greedy sampling? Or what are the other hypothesis?\n- The derivation from the reduced neighbourhood confidence (Eq. 3) to the empirical reduced neighbour confidence (Eq.4) is confusing. What motivates using the cosine distance to perform a weighted sum? Is this motivated by importance sampling?\n- WebVision and Clothing-1M are crawled from the web, but Prune4ReL seems to work better in Clothing-1M. Can the authors elaborate on this further? \n  - It seems that kCenter and GraNd are missing in Figure 3b. The authors have to find a better way to visualize this plot (Figure 3b). Try to zoom in a little bit.\n- Section 4.5 is like the motivation of the paper. Consider re-order them.\n\nLIMITATIONS:\nAs mentioned in the #weakness, there are some minor limitations.\n- Empirical results: The proposed method still requires some work on improving over uniform sampling baseline, and the authors did not justify the strong uniform baseline.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}