Noisy Data Pruning by Label Distribution Discrimination

Authors: Xiao Wei Wei, Ma Yi, Qiben Shan, Shaocong Wu, Jingyong Su
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: data pruning; coreset selection; noise label learning; data centric-ai

Abstract:
Data pruning aims to prune large-scale datasets into concise subsets, thereby reducing computational costs during model training.
While a variety of data pruning methods have been proposed, most focus on meticulously curated datasets, and relatively few studies address real-world datasets containing noisy labels. In this paper, we empirically analyze the shortcomings of previous gradient-based methods, revealing that geometry-based methods exhibit greater resilience to noisy labels. Consequently, we propose a novel two-stage noisy data pruning method that incorporates selection and re-labeling processes, which takes into account geometric neighboring information. Specifically, we utilize the distribution divergence between a given label and the predictions of its neighboring samples as an importance metric for data pruning. To ensure reliable neighboring predictions, we employ feature propagation and label propagation to refine these predictions effectively. Furthermore, we utilize re-labeling methods to correct selected subsets and consider the coverage of both easy and hard samples at different pruning rates. Extensive experiments demonstrate the effectiveness of the proposed method, not only on real-world benchmarks but also on synthetic datasets, highlighting its suitability for practical applications with noisy label scenarios.

Decision: reject
Reviewer scores: 5, 3, 3, 1

Reviews and discussion:
<thread>
creation date:
1730098027088

thread_type:
official_review

<thread_object>
creator: reviewer_oddr

<content>
summary:
This paper introduces a novel two-stage robust data pruning method (RoP) aimed at datasets with noisy labels. The first stage identifies clean samples using a Neighborhood Label Inconsistency Score (NLI-Score), followed by a second stage that re-labels the selected samples. RoP employs feature and label propagation to enhance the accuracy of neighboring predictions and uses density-based coverage sampling to balance the number of easy and hard samples across different pruning rates. Extensive experiments demonstrate the effectiveness of RoP on both synthetic noisy datasets and real-world benchmarks.

strengths:
1. The paper is well-organized, with clear presentations of methodology, experiments, and conclusions that effectively guide the reader through the research.

2. The authors conduct extensive experiments across various datasets, including real-world noisy datasets and synthetic noise datasets, which helps to validate the robustness and applicability of the method.

weaknesses:
1. I find this paper completely unoriginal; it merely applies existing techniques from the noisy label learning task to the domain of data pruning. The "FEATURE PROPAGATION" proposed by the authors is present in many works on noisy label learning [1-4]. The authors additionally utilize Equation 5, which involves fusing a sample's own features with those of surrounding samples to improve its own features; however, this may not be very meaningful. Many studies have shown that, in noisy label learning, while model predictions may be misled by noisy labels, the learned features tend to remain reliable. The "LABEL PROPAGATION" proposed by the authors is also found in many works on noisy label learning [5-6]. For the re-labeling part, the authors even explicitly mention using the existing state-of-the-art method, SOP+. 

2. Why are methods related to noisy label learning not compared in the experiments? Many existing methods can be easily adapted to the scenarios presented in this paper.

[1] Multi-Objective Interpolation Training for Robustness to Label Noise. CVPR 2021

[2] Selective-Supervised Contrastive Learning with Noisy Labels. CVPR 2022

[3] RankMatch: Fostering Confidence and Consistency in Learning with Noisy Labels. ICCV 2023

[4] Learning with Neighbor Consistency for Noisy Labels. CVPR 2024

[5] Jo-SRC: A Contrastive Approach for Combating Noisy Labels. CVPR 2021

[6] UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning. CVPR 2022

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644075334

thread_type:
official_review

<thread_object>
creator: reviewer_g36c

<content>
summary:
This paper studies the problem of noisy data pruning which aims to prune noisy large-scale datasets into concise subsets. The authors first reveal that geometry-based methods exhibit greater resilience to noisy labels compared to gradient-based methods. Then, a discrimination, pruning, and re-labeling method is proposed to conduct noisy data pruning. Specifically, noisy label discrimination is achieved by neighborhood label inconsistency estimation, after feature and label propagation. Then, the pruned set is selected by ensuring coverage on both easy and hard samples. Finally, re-labeling is achieved by SOTA noisy label learning methods. Experiments show the effectiveness of the proposed method.

strengths:
- This paper addresses the issue of noisy data pruning, which is crucial in real-world applications.
- The proposed method achieves SOTA results against existing baselines.

weaknesses:
- The presentation of this paper is poor according to the following aspects:
  - Lacking in-depth analysis of the difference between the proposed method and previous SOTA Pr4ReL, since Pr4ReL also follows a selection and relabeling paradigm and uses neighborhood information. It is important to explain the superiority of the proposed method. Also, authors are encouraged to add noisy data pruning methods in the section of related work.
  - The motivation for feature propagation and label propagation is unclear. Besides, what if directly applying label propagation without feature propagation?
- Line 237 mentions that a model is required to be trained on the noisy label datasets. Since the purpose of dataset pruning is to reduce the training cost, is it reasonable and fair to access the trained model? If so, please refer to some related works.
- The process of pruning and re-labeling actually follows existing works, i.e., a SOTA noisy label learning method SOP+ and Coverage Coreset Sampling strategy, limiting the novelty of the proposed method.
- Typo: In line 309 "retaining retaining"

questions:
In line 250, how to get the ground-truth label of the sample?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701524832

thread_type:
official_review

<thread_object>
creator: reviewer_y5ly

<content>
summary:
In this paper, the authors propose a novel geometry-based noisy data pruning method. It consists of two stages and uses feature propagation and label propagation for reliable neighboring predictions. Experiments demonstrated quite good results.

strengths:
- It proposes a novel two-stage noisy data pruning method.
- It employs novel feature propagation and label propagation to refine neighboring predictions.

weaknesses:
- The motivation is weak. First, the loss-based method GraNd is not designed and optimized, especially for noisy samples. Thus it is not suitable to choose GradNd to represent the method. Second, it is not convincing to get the conclusion that the geometry-based approach is better than the loss-based approach by simply comparing the two methods. More methods and more noise settings are needed. Moreover, the statement in line 68 applies to all methods, and also can’t lead to the conclusion.
- I recommend the discussion about the noisy data pruning method and sample selection method. It is an important topic that when the pruning rate is limited, the approximation of the noise rate is neglected. Therefore, the result is destined to be sub-optimal.
- From Table 1, Table 2 and Table 6, it seems that the relabeling method is more dominant than selection. I slightly question the motivation to use pruning for mitigating noisy samples since relabeling has solved this problem well.
- The presentation needs to be polished, e.g. the suddenly appeared GraNd in line 52, and some typos, such as 87.9±-1.3 in Table 1 and brackets in line 6.
- The types of noise are insufficient. Symmetric label noise should be taken into experiments.

questions:
- It’s hard to understand the so-called “intuitive method” in line 70: Since the clean samples are found in the first stage, why do you need to relabel them?
- Are the gradient-based method in line 15 and the loss-based method in line 51 the same thing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703111867

thread_type:
official_review

<thread_object>
creator: reviewer_wddu

<content>
summary:
This paper studied a task combination of label noise and core-set selection. The proposed method, RoP, is a graph-like sample selection strategy that integrates feature propagation and label propagation. The experiment is extensive.

strengths:
- Introduces an innovative NLI-Score for identifying noisy samples by leveraging neighboring sample consistency.  
- Combines feature and label propagation effectively to reduce selection bias in noisy data scenarios.

weaknesses:
- The discussion on a closely related work, Pr4ReL, requires expansion. (1) Pr4ReL also employs a neighborhood-based strategy for selection and relabeling. Further insights into why RoP outperforms Pr4ReL, along with a more detailed comparison, would strengthen the analysis. (2) Additionally, RoP's performance on CIFAR-10N/100N, as shown in Table 1, does not surpass that of Pr4ReL, which merits further examination.

-  The authors employ CCS for coverage pruning; however, the adaptation of CCS to NLI-Scores is not clearly explained. Additional clarification would help readers understand this approach.

- Experimentation. (1) Additional experimentation on large-scale datasets is recommended, as Mini-WebVision contains only 66k training images. An experiment on Clothing-1M, comparable to the setup in Pr4ReL, would provide more robust validation. (2) The ablation study is limited. In Table 6, the authors should include separate analyses of using feature and label propagation independently.

- Minor typos, such as "retaining" on Line 309, should be corrected for clarity.

questions:
see weaknesses.
</content>
</thread_object>

</thread>


Metareview: