Exploring Learning Complexity for Efficient Downstream Dataset Pruning

Authors: Wenyu Jiang, Zhenlong Liu, Zejian Xie, Songxin Zhang, Bingyi Jing, Hongxin Wei
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: data efficiency

Abstract:
The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruning, which aims to reduce dataset size while maintaining task performance.
However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained models.
In this paper, we propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC), to identify informative images and instructions from the downstream dataset efficiently.
Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
Specifically, we define the Learning Complexity to quantify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization.
Based on DLC, we further design a flexible under-sampling strategy with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift.
Extensive experiments with downstream image and instructions dataset pruning benchmarks demonstrate the effectiveness and efficiency of the proposed approach.
In the images pruning benchmark, DLC significantly reduces the pruning time by 35$\times$ while establishing state-of-the-art performance with FlexRand.

Decision: accept (poster)
Reviewer scores: 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730136951400

thread_type:
official_review

<thread_object>
creator: reviewer_azeo

<content>
summary:
This work describes a novel dataset pruning method without the need of pre-training on the target dataset. Given models pre-trained on large scale datasets, this work proposes a Distorting-based Learning Complexity score to identify informative images and instructions. Sample hardness is estimated by randomly masked neural networks, representing networks with different capabilities. Then samples are randomly sampled from the easy and hard groups, respectively. The proposed method achieves effective dataset pruning with 35x less pruning time.

strengths:
1. The design of using random masks to produce classifiers with different capabilities is interesting and practical. With the averaged feature serving as the prediction head, there is no more need to fine-tune the classifier on downstream tasks. 
2. Detailed experiments are conducted to illustrate the effetiveness of the proposed method. The method can be applied to both image and instruction datasets, both demonstrating performance improvement. 
3. The writing is generally fluent and easy to follow.

weaknesses:
1. The authors claim that easy samples are more likely to be correctly classified by a weak classifier in the front part of the learning path. However, the overall Learning Complexity score is acquired by averaging classification loss of multiple randomly sampled networks. The definition of learning path seems not to be utilized in the method design. 
2. Can the utilization also be applied to some of previous methods? For example, the Herding method uses parameter influence as scores for each sample. Here the fine-tuned model can also be substituted by a pre-trained model with averaged features as the prediction head. Although the direct employment of pre-trained models is practical, it is not a unique design. And it will be interesting to see if applying the strategy to previous methods also leads to performance improvement. 
3. The strategy of dividing datasets into different groups and randomly sampling from each group is similar to the idea in Dataset Quantization [1]. Dataset Quantization first iteratively separate the data into multiple bins with coreset selection methods. Normally the early groups tend to cluster around the distribution center, while the later groups show more diversity. By sampling from each bin, the overall distribution will be kept similar to the original one. This paper has a similar claim that FlexRand avoids severe distribution shift. Please discuss the difference of the proposed strategy from Dataset Quantization and the advantages of it. 
4. Section 5 discusses the quality of pre-trained models. The authors claim that the method is not sensitive to the quality of pre-trained models. But weakly supervised models are not always worse than fully supervised models. Please also show the original performance comparison between these two groups of models. 
5. Minor:
    - The use of pretrain and pre-train need to be unified in the paper. 
    - The sample number is represented both by N (line 105) and |D| (line 129). Please unify the use. 

[1] Zhou, Daquan, et al. "Dataset quantization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
1. How is the loss integration implemented? In the integration figures, the upper bound of loss is 1.0. Is it normalized to the range of (0, 1)?
2. How is the masking applied to the neural network? 
3. How is the splitting hyper-parameter $\gamma$ determined in the actual use? If multiple values need to be tested, the tuning time should also be counted towards the pruning time in Figure 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706756028

thread_type:
official_review

<thread_object>
creator: reviewer_fdcj

<content>
summary:
A novel training-free hardness score, Distorting-based Learning Complexity, is proposed to identify informative images and instructions from downstream dataset. Also, a flexible under-sampling method with randomness named FlexRand is proposed to alleviate the severe subset distribution shift. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach.

strengths:
The proposed scoring function, Distorting based Learning Complexity, is an efficient training-free score for dataset pruning. A under-sampling strategy with randomness, FlexRand, is designed to adapt to different data regimes and avoid distribution shift. Extensive experiments demonstrate the effectiveness of the proposed approach. DLC significantly reduces the pruning time by 35× in images pruning benchmark.

weaknesses:
Some typo: Line 021, "a flexible under-sampling with randomness" -> "a flexible under-sampling strategy with randomness"
In Figure 4(a), the MMD value of Random is missing.

questions:
When referring masking the pre-training weights, what specific operation is performed on the network parameters?
What's the meaning of dotted line in blue(10%), green(20%) and orange(30%) in Figure(d)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712212690

thread_type:
official_review

<thread_object>
creator: reviewer_n2di

<content>
summary:
The paper introduces Distorting-based Learning Complexity (DLC), a novel training-free hardness score for efficient downstream dataset pruning. DLC quantifies sample hardness by masking pre-trained weights and approximating loss integration via the Monte Carlo method. The authors also propose FlexRand, a flexible under-sampling strategy to adapt to different data regimes and avoid distribution shift.

strengths:
1.) The significance lies in its potential to reduce the computational burden of fine-tuning large pre-trained models while maintaining performance.
2.) The paper is well-structured, and easy to follow.
3.) The introduction of FlexRand adds another layer of adaptability to the pruning process, making it more robust across different data regimes, a valuable contribution to data pruning strategies.

weaknesses:
1.) The paper suggests that DLC is not sensitive to the quality of pre-trained models, but this claim could be further experimented on different size level pre-trained models.
2.) The method requires storing multiple masked models, which could be a limitation in environments with constrained memory resources, potentially affecting the practicality of the approach.
3.) The paper could benefit from a more detailed discussion on scenarios where DLC might underperform or fail, providing a more comprehensive understanding of its limitations.

questions:
see the Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732025549064

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We are glad that the reviewer (N2di) recognizes the **significance** of downstream dataset pruning. We are also encouraged that reviewers find the method is **effective** (FdCJ,aZeo) on the **extensive** and **detailed** (FdCJ,aZeo) benchmarks, the DLC score is **efficient** (FdCJ), **interesting and practical** (aZeo), and the FlexRand strategy is **robust** and **valuable** (N2di). Besides, reviewers appreciate that the writing is **well-structured**, **fluent**, and **easy to follow** (N2di,aZeo).

In the following responses, we have addressed the reviewers' comments and concerns point by point. The reviews allow us to strengthen our manuscript and the changes$^1$ are summarized below:
- Removed the statement about memory for multiple masked models storage in **Limitations**. [N2di]
- Added discussion of potential failure cases in **Limitations**. [N2di]
- Added formulation of the pre-training weights masking in **Line 199** and **Appendix B**. [FdCJ,aZeo]
- Added comparison of different under-sampling strategies in **Line 457-458** and **Appendix C.2.4**. [aZeo]
- Fixed discussion of pre-trained model quality in **Line 509-517**. [aZeo]
- Added performance of pre-trained models by linear probing in **Line 513** and **Appendix C.2.5**. [aZeo]
- Fixed typos in **Line 021,129,365,487**, **Figure 2**, **Figure 4(a)**, and **Figure 6**.
---
$^1$ For clarity, we highlight the revised part of the manuscript in **blue** color.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a dataset pruning method based on a training-free hardness score called Distorting-based Learning Complexity (DLC) and a flexible under-sampling strategy. Extensive experimental results demonstrate the effectiveness and efficiency of this approach. Overall, the idea is novel, and the performance is impressive. However, the presentation of the motivation can be improved to better articulate the rationale behind the proposed method. Additionally, further clarification would enhance the overall understanding and impact of the work. Based on the overall quality of the work, the novelty of the approach, and the positive feedback from the reviewers, the decision is to recommend the paper for acceptance. We encourage the authors to address the noted shortcomings in the presentation of the motivation in future revisions.