Medium-Difficulty Samples Constitute Smoothed Decision Boundary for Knowledge Distillation on Pruned Datasets

Authors: Yudong Chen, Xuwei Xu, Frank de Hoog, Jiajun Liu, Sen Wang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Knowledge distillation, dataset pruning, image recognition

Abstract:
This paper tackles a new problem of dataset pruning for Knowledge Distillation (KD), from a fresh perspective of Decision Boundary (DB) preservation and drifts. Existing dataset pruning methods generally assume that the post-pruning DB formed by the selected samples can be well-captured by future networks that use those samples for training. Therefore, they tend to preserve hard samples since hard samples are closer to the DB and better characterize the nuances in the distribution of the entire dataset. However, in KD, the limited learning capacity from the student network leads to imperfect preservation of the teacher's feature distribution, resulting in the drift of DB in the student space. Specifically, hard samples worsen such drifts as they are difficult for the student to learn, creating a situation where the student's DB can drift deeper into other classes and make incorrect classifications. Motivated by these findings, our method selects medium-difficulty samples for KD-based dataset pruning. We show that these samples constitute a smoothed version of the teacher's DB and are easier for the student to learn, obtaining a general feature distribution preservation for a class of samples and reasonable DB between different classes for the student. In addition, to reduce the distributional shift due to dataset pruning, we leverage the class-wise distributional information of the teacher's outputs to reshape the logits of the preserved samples. Experiments show that the proposed static pruning method can even perform better than the state-of-the-art dynamic pruning method which needs access to the entire dataset. In addition, our method halves the training times of KD and improves the student's accuracy by 0.4% on ImageNet with a 50% keep ratio. When the ratio further increases to 70%, our method achieves higher accuracy over the vanilla KD while reducing the training times by 30%. Code is available at https://github.com/chenyd7/MDSLR.

Decision: accept (poster)
Reviewer scores: 6, 6, 3, 8

Reviews and discussion:
<thread>
creation date:
1730404752256

thread_type:
official_review

<thread_object>
creator: reviewer_egek

<content>
summary:
The paper deals in data pruning for knowledge distillation. While previous observations show that random pruning is superior over other pruning methods, the authors proposed to retain medium difficulty samples for distillation. The motivation is that these samples are closer to the median distances between class centers.

strengths:
- The explanation the authors suggest for the question of why a student model is sub-optimal when performing hard-sample based pruning methods is interesting and looks like it is the main novelty of the paper.  Also, The visualization in Figure 3 is clear and insightful.

- Experimental results are encouraging and look promising.

weaknesses:
- In Figure  2 the authors show “a large accuracy gap” between the teacher and the student. The figure which indeed shows “Large accuracy gap” does not clearly demonstrate the specific degradation of the student’s accuracy due to the dataset pruning used. An accuracy gap between teacher and student is what we typically see in knowledge distillation in general. What does it mean “large accuracy gap”? Maybe the authors better show accuracy plots for easy, medium and hard pruning. Currently, Figure 2 is out of context.   


- Equation (3), the M_DIF is computed to score hard samples. It is not quit clear why the M_DIF is defined here and what is the context? Is it another form of data pruning algorithm? (in figure 5, various data pruning are compared by the authors). Or it severs only the discussion in section 3.2? I would clarify the motivation.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577100011

thread_type:
official_review

<thread_object>
creator: reviewer_s3pu

<content>
summary:
The paper aims to investigate experimentally how to select a balanced subset of a training set, based on a pretrained teacher, so as to reduce the training time needed to train students in the knowledge distillation (KD) framework while controlling the performance degradation of distilled students as much as possible, where the distilled students are trained on the balanced set. The proposed method consists of two phases. In phase 1, all samples in the original whole training set are sorted in the decreasing order of their corresponding cross entropy losses given by the pretrained teacher, those located in the middle range are selected, and the average of the standard deviation of a logit vector over all logit vectors within each class is computed and recorded. In phase 2 (corresponding to the training stage in KD), the logit vector of each selected sample from the pretrained teacher is first reshaped by the recorded average standard deviation of the class that the selected sample belongs to and  then passed to the KD objective function. To maintain balance, the sorting and sample selection can be carried out on a per class basis, as implied in the first paragraph of Section 4.3. Experiments were carried out on CIFAR 100 and ImageNet for several teacher-student pairs to verify the effectiveness and generalizability of the proposed method.

strengths:
1. The paper is generally well written. 

2. The methodology is easy to follow and simple to implement.

weaknesses:
1. The value proposition in terms of the accuracy vs time savings is kind of weak in this line of research. First, the intended time savings is for the KD training phase, not for the inference phase. Time savings for the inference phase is more important. Second, there is always accuracy degradation when the training dataset is pruned. When the ratio of kept sample is high, the training time savings is small. To have a meaningful training time savings, the ratio of kept sample has to be much smaller, which, however, results in significant, unacceptable accuracy performance degradation. 

2. In comparison with Moderate Dataset Selection (MoDS) and reshaping techniques proposed in the literature, the novelty of the proposed method is kind of limited. 

3. Experiments are insufficient to demonstrate the universality of the proposed method with respect to different students with variety of learning capability. The proposed method is mainly motivated by the learning gap between the pretrained teacher and students. What would happen if the learning capability of the student is close to, or very far away from that of the pretrained teacher? If the pruned training dataset works only for certain students, but not for others, the method won't fly since finding which student fits by experiments also takes significant amounts of training time.

4. Experiments are insufficient to demonstrate its competitiveness with another technique called data distillation.

questions:
1. How are the shaping methods in the literature applied in the setup for Table 2? The comparison in Table 2 seems  like apple vs orange, and may be not fair.

2. More experiments are needed to show the value of the proposed method. 

3. The few shot cases considered in the paper ``BAYES CONDITIONAL DISTRIBUTION ESTIMATION FOR KNOWLEDGE DISTILLATION BASED ON CONDITIONAL MUTUAL INFORMATION'' by L. Ye (ICLR 2024) can also be used to reduce the KD training time. Please compare the performance of the proposed method with the few shot case in that paper in terms of accuracy vs training time to see if the proposed method in this paper is orthogonal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582655641

thread_type:
official_review

<thread_object>
creator: reviewer_rwdy

<content>
summary:
In the paper, the authors addresses the challenge of efficient knowledge distillation (KD) on pruned datasets. The authors propose a static pruning method that focuses on selecting medium-difficulty samples, as opposed to traditional methods that prefer hard or easy samples. The method aims to mitigate decision boundary drift caused by the student's limited capacity to mimic the teacher's feature distribution. Additionally, the paper introduces a logit reshaping technique to reduce distributional shift in the pruned dataset by leveraging the teacher's global logit information. Experimental results demonstrate that the proposed method outperforms existing dynamic and static pruning techniques across various benchmarks, achieving notable efficiency and performance gains.

strengths:
1. **Clear Motivation**: The authors clearly articulate the problem of decision boundary drift and how hard or easy samples influence KD differently. This sets a strong foundation for their proposed solution.
2. **Methodical Innovation**: The selection of medium-difficulty samples for KD and the use of global logit reshaping are novel concepts. The visualizations of decision boundaries and empirical evidence support the validity of these innovations.
3. **Comprehensive Evaluation**: The paper presents results from multiple datasets (CIFAR-100, ImageNet) and various teacher-student configurations. Ablation studies and comparisons with other pruning methods provide a holistic understanding of the approach’s effectiveness.
4. **Efficiency Gains**: By achieving better student performance while significantly reducing training times, the proposed method demonstrates practical utility, especially for resource-constrained settings.

weaknesses:
1. **Theoretical Justification of Medium-Difficulty Samples**: 
Despite the novelty of using medium-difficulty samples, the overall methodological advances feel incremental. The authors should need more rigorous theoretical analysis to make the proposed methodology convince to the reader of the underlying mechanisms driving performance improvements.

   - **Explicit Analysis**: The authors should provide a rigorous mathematical analysis that formalizes how medium-difficulty samples contribute to smoother decision boundaries. This could involve deriving relationships between sample difficulty, gradient stability, and boundary smoothness.
   - **Statistical Modeling**: Conduct an empirical risk analysis or use statistical measures to model the impact of medium-difficulty samples on decision boundary drift. For example, the authors could analyze the variance of gradients during training for different difficulty levels and quantify how medium-difficulty samples stabilize learning.
   - **Geometric Insights**: The authors can use tools from computational geometry to offer insights into how medium-difficulty samples influence class separability. Visualizing the feature space evolution with respect to class overlap during training could also strengthen their argument.

2. **Deeper Analysis of Logit Reshaping**:
   - **Impact Study on Logit Distribution**: Perform a sensitivity analysis on the logit reshaping process. Specifically, the authors can quantify how reshaping logits using the teacher’s global information affects the student’s performance across classes, especially in cases of class imbalance or noisy labels.
   - **Ablation of Reshaping Components**: Break down the logit reshaping mechanism and assess the impact of each component (e.g., the standard deviation adjustment and the mean logit alignment) to isolate what contributes most to performance gains.
   - **Bias and Overfitting Analysis**: Evaluate whether the logit reshaping method introduces bias or overfitting. This could involve testing the method on datasets with varying distributions and comparing generalization performance on unseen data.

3. **Quantitative Analysis of Decision Boundary Dynamics**:
   - **Boundary Drift Measurement**: The authors can introduce or provide a metric or framework for quantifying decision boundary drift in the student network. This could involve computing the alignment of decision boundaries between the teacher and student models using tools like mutual information or boundary discrepancy measures.
   - **Difficulty Spectrum Analysis**: Assess how the performance changes as samples are divided across a spectrum of difficulties, rather than just hard, medium, or easy categories. This can help establish a more granular understanding of why medium-difficulty samples are optimal.

4. **Generalizability and Scalability Experiments to Other KD frameworks on other modalities?**:
The method’s effectiveness on more diverse datasets or in non-vision tasks remains unexplored. The authors should test out the method on a range of datasets beyond vision (e.g., NLP or time-series data) to demonstrate the versatility of the approach. This would help clarify if the medium-difficulty sample selection has domain-specific benefits or if it generalizes well.

questions:
1. **Logit Reshaping Justification**: How sensitive is the logit reshaping method to variations in the teacher’s predictions? Could this technique inadvertently introduce bias in scenarios with highly imbalanced classes?
2. **Generalizability Across Domains**: What challenges do the authors anticipate in adapting the propsoed method to other domains of say NLP and time-series?
3. **Efficiency vs. Performance Trade-off**: Can you elaborate on the practical trade-offs between training time and accuracy? Are there specific scenarios where maximizing efficiency might not justify the associated performance loss?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731070900734

thread_type:
official_review

<thread_object>
creator: reviewer_fjln

<content>
summary:
This manuscript introduces a dataset pruning method to enhance the efficiency of knowledge distillation (KD) by retaining a strategically selected subset of samples and reshaping logits to better align with the teacher’s knowledge. 

The authors propose that selecting medium-difficulty samples strikes a good balance and helps in preserving samples that are neither too easy nor too difficult. This in turn helps to increase the efficiency and accuracy of models.

Further, they introduce a logit reshaping technique that uses global distribution information to counteract biases that dataset pruning might introduce.

The proposed method is evaluated on two benchmark datasets CIFAR-100 and ImageNet with multiple teacher-student architectures, demonstrating consistent improvements over existing pruning techniques.

strengths:
1. The paper introduces a compelling approach to dataset pruning. This method looks interesting focusing on selecting medium-difficulty samples for distillation, which strikes a balance between easy and hard examples and leads to better generalization in the student model.

2. The proposed logit reshaping method effectively mitigates the distributional shift caused by pruning. This can be a key to enhancing distillation performance on pruned datasets.

3. The technique is thoroughly evaluated across multiple benchmark datasets (CIFAR-100, ImageNet) and different teacher-student architectures.

4. The proposed pruning method significantly reduces training time without sacrificing performance. This can give us with some really good solutions for real world use and even we can use this method for time consuming computationally large datasets.

5. The proposed method outperforms existing static pruning techniques (e.g., Herding, Forgetting, MoDS) and achieves comparable or superior performance to dynamic methods like InfoBatch.

weaknesses:
1. The article does not provide a detailed analysis of how sensitive the method is to hyperparameter tuning. The authors should try out with different keep ratios or distillation losses.

2. This article compares with static pruning techniques but a more in-depth comparison with other knowledge distillation methods, especially self-knowledge distillation, would strengthen its position and give us a more in-depth analysis.

3. The approach is evaluated on CIFAR-100 and ImageNet, but it remains unclear how well it would scale with more complex models like ResNet family and Wide ResNets (WRN).

questions:
I have some queries regarding the article----

1. How does the proposed logit reshaping method handle potential conflicts between preserving class-specific distributions and mitigating the variance of logits in a highly imbalanced dataset? Could the method be further optimized for such cases? Can I get some explanations for this?

2. In the context of medium-difficulty sample selection, how does the method ensure that the pruned dataset retains enough diversity to avoid overfitting, especially when dealing with high-dimensional feature spaces in complex models?

3. Given that the pruned dataset relies on teacher feedback to rank sample difficulty, could this process introduce potential biases in class relationships, and how might this be addressed if the teacher's knowledge is imperfect or misaligned with the pruned subset?

4. Can the authors provide some tSNE figures for better understandings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732766642908

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript

comment:
Following the reviewers' insightful comments and valuable suggestions, we have submitted a new version of the manuscript with more analysis and experiments. The newly added contents are highlighted in blue text. In summary, we incorporate the following modifications:

**We thank the Reviewer FjLn for the support and the valuable suggestions about conducting experiments on highly imbalanced datasets to complement the evaluation.** 

* Results on a highly imbalanced dataset are included in Appendix A.6, demonstrating the generalizability of the proposed logit reshaping method.

* Analysis of the trade-off between underfitting and overfitting is provided in Section 3.2.

**We thank the Reviewer RWDY for the recognition of our work and the insightful comments about using mutual information and gradient variation to evaluate the proposed method.** 

* More justification for using medium-difficulty samples for distillation from the perspective of gradient variation and mutual information is provided in Section 4.2.

* More visualizations on the evolution of feature space during the training process are provided in Appendix A.5.

* Results on a highly imbalanced dataset are included in Appendix A.6, demonstrating the generalizability of the proposed logit reshaping method.

**We thank the Reviewer s3Pu for recommending the relevant paper.**

* We introduce the recommended paper in Section 3 and discuss the difference with our method. In addition, we further clarify the focus of this paper at the beginning of Section 3.

* The technical details of different logit reshaping methods are described in Appendix A.3.

**We thank the Reviewer EGEk for their support and valuable suggestions to improve the readability of the manuscript.**

* Following the suggestion of the reviewer, we update Figure 2 (Figure 3 in the revised manuscript) by comparing the training accuracies of different types of samples.

* The motivation for using $M_{DIF}$ is elaborated in Section 3.1.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission proposes a dataset pruning strategy for knowledge distillation (KD), focusing on the selection of medium-difficulty samples and introducing a logit reshaping method to mitigate distributional shifts caused by pruning. The authors present a clear motivation for focusing on medium-difficulty samples. The authors conducted experiments on CIFAR-100 and ImageNet, demonstrating the method’s potential benefits.

The authors have provided detailed responses to all reviewer comments, including additional experiments, updated visualizations, and clarifications in the revised manuscript:

-Concerns about imbalanced datasets, logit reshaping sensitivity, and scalability have been thoroughly addressed, with results incorporated into the manuscript.

-Reviewer FjLn raised their score after satisfactory clarifications. Reviewer RWDY acknowledged improvements and requested additional inclusions in the revised manuscript.

-Reviewer s3Pu's Concerns: While the authors made significant efforts to address s3Pu’s concerns, the reviewer maintained their position, citing fundamental disagreements about the trade-offs inherent in dataset pruning. However, the area chair thinks it is okay for a machine-learning paper to give more attention to its methodological and theoretical parts.