Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels

Authors: Hyeonsu Jeong, Hye Won Chung
Primary area: learning theory
Keywords: self-distillation, partial label learning, label noise correction, training with soft labels, multi-class classification

Abstract:
We investigate the mechanisms of self-distillation in multi-class classification, particularly in the context of linear probing with fixed feature extractors where traditional feature learning explanations do not apply. Our theoretical analysis reveals that multi-round self-distillation effectively performs label averaging among instances with high feature correlations, governed by the eigenvectors of the Gram matrix derived from input features. This process leads to clustered predictions and improved generalization, mitigating the impact of label noise by reducing the model's reliance on potentially corrupted labels. We establish conditions under which multi-round self-distillation achieves 100\% population accuracy despite label noise. Furthermore, we introduce a novel, efficient single-round self-distillation method using refined partial labels from the teacher's top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multi-round distillation in a single round, achieving comparable or superior performance--especially in high-noise scenarios--while significantly reducing computational cost.

Decision: accept (poster)
Reviewer scores: 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730210249005

thread_type:
official_review

<thread_object>
creator: reviewer_cz26

<content>
summary:
In this paper, the authors proposed introduce a single-round self-distillation method using refined partial labels from the teacher’s top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multiround distillation in a single round. Experiments on several datasets demonstrate the proposed method is effective.

strengths:
The analysis of self-distillation in linear probing reveals that self-distillation effectively performs label averaging among instances with high feature correlations when generating predictions on the training data.

weaknesses:
Here are a few questions I would like to ask the authors:

1. What are the clear insights or significances of Sections 2.2, 3, and 4 in relation to the proposed method in Section 5? Could the authors summarize the roles and conclusions of Sections 2.2, 3, and 4, and their implications for Section 5? These sections are not sufficiently clear.

2. In Section 5, the authors state, "selecting the top two labels with the highest values and assigning a weight of 1/2 to each, setting all the other entries to zero." What is the rationale behind this approach?

3. The experiments use ResNet34 as the backbone for linear probe experiments; however, training ResNet34 with all parameters is not particularly time-consuming or resource-intensive. Should a backbone with a larger parameter count be chosen for the experiments?

4. When comparing existing PLL methods, the authors claim, "Our method differs by directly employing the refined partial labels derived from the teacher’s outputs, achieving the same benefits as multi-round distillation in just one round." Does this imply that other methods incorporating the teacher's output could achieve similar or even better results? Are there corresponding experiments to support this?

questions:
Please see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536493587

thread_type:
official_review

<thread_object>
creator: reviewer_g4hh

<content>
summary:
This paper examines the multi-round self-distillation for multi-class classification in the context of linear probing. By approximating the softmax function as a linear function and considering the feature correlation matrix as a low-rank structure, this paper derives a quantified closed-form solution for the output of the $t$-th student model, showing the effect of self-distillation can be interpreted as label averaging among highly correlated instances. The authors then derive the conditions of the label corruption matrix achieving 100% population accuracy for multi-round self-distillation, in the context of balance and superclass corruption. The authors further prove that distilling teacher's top-2 outputs enjoys better theoretical properties. Extensive experiments are conducted and show consistency with the proposed theories.

strengths:
- The proposed theory is solid and valuable, especially providing insight into the effectiveness of distilling with partial labels.
- The authors provide numerical and visual analysis of the approximation of the softmax function and the feature correlation matrix, which is reasonable and convincing.
- This paper is well-organized and clearly written.

weaknesses:
In the experiments, Generalized Cross Entropy loss is applied for PLL which differs from the setting of CE loss in the main theory. The authors need to explain whether applying GCE loss is fair for comparison or provide experimental results of CE loss for PLL.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711416772

thread_type:
official_review

<thread_object>
creator: reviewer_xush

<content>
summary:
This paper presents a theoretical analysis of the mechanisms behind self-distillation in a linear probing setting. The analysis reveals that after $t$ rounds of self-distillation, the model's predictions converge to a weighted average of the provided labels, with the weights determined by the Gram matrix. Building on this finding, the authors investigate the effects of label noise and the efficiency of the self-distillation method. Experiments demonstrate the effectiveness of proposed single-round self-distillation method.

strengths:
* This paper is well-written, particularly in Section 2, which formulates self-distillation and provides a clear overview of the results.
* This paper presents an interesting result in Theorem 2.1, establishing a connection between the predictions of the $t$-th distilled model and the given (possibly noisy) labels.

weaknesses:
The main weakness about this paper is the significance of research problem.

* This paper focuses on self-distillation with linear probing and provides a theoretical analysis in this context. I believe that both self-distillation and linear probing are valuable techniques, but I am unclear about the purpose of combining self-distillation with linear probing. As far as I know, linear probing is widely used in self-supervised learning as a method to evaluate learned features. Why should we combine linear probing with self-distillation, especially in scenarios involving label noise?

* The proposed theory assumes a fixed feature extractor and therefore cannot be applied to trainable feature extractors, which are more commonly used in task adaptation and transfer learning.

questions:
See my questions in weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730982745796

thread_type:
official_review

<thread_object>
creator: reviewer_ukvb

<content>
summary:
This work aims to analyze self-distillation in linear probing with neural network feature extractors. The contributions of this work are threefold: (1) the analysis reveals that self-distillation effectively performs label averaging among instances with highly correlated features when generating predictions on the training data; (2) the analysis quantifies the number of distillation rounds needed to achieve 100% population accuracy in the presence of label noise; and (3) based on the theoretical analysis, the authors introduce a novel self-distillation approach that achievers similar benefits of multi-round distillation in a single round.

strengths:
1. Solid Presentation
Overall, this manuscript is well-written and polished. The introduction provides a satisfying overview of the rich theoretical analysis, which is a strong aspect of the work. Some parts of the analysis are somewhat difficult to follow, but experts in this specific field likely won’t struggle to understand the details.

2. Interesting theoretical analysis
One of the main theoretical contributions of this paper is demonstrating that the effect of self-distillation can be interpreted as label averaging among highly correlated instances. This interpretation offers a new perspective on the mechanism of self-distillation, even in the absence of feature evolution. Initially, I thought this was merely a special case of Allen-Zhu and Li’s study. However, after reviewing previous work, I discovered a notable difference in how the effects of self-distillation are analyzed, which I believe is worth sharing with the community.

3. A New Approach to Converting Multi-Round Self-Distillation into a Single Round
The proposed method takes a different approach by directly using the refined partial labels derived from the teacher's outputs, achieving the same benefits as multi-round distillation in just one round. This is especially appealing, as one of the main drawbacks of self-distillation is the need for multiple rounds, which makes training very inefficient. A single-shot method is highly desirable, and we hope this approach will be widely adopted for training classifiers in noisy label settings.

weaknesses:
1. Most sections are well-written, but some parts would benefit from clearer descriptions or revision. Please refer to the section below.

2. The experiments are somewhat weak, but this is understandable since this work is more theoretical.

questions:
1. The second contribution of this work is to determine the number of distillation rounds required to achieve 100% population accuracy in the presence of label noise. On first reading, the term "100% population accuracy" may not be immediately clear. Introducing a brief explanation of this concept in the first place would be beneficial for readers who may be less familiar with the subject.

2. Could you explain the intuition behind the condition (Eq. 10) in Theorem 4.1? At first glance, it seems almost implausible to reduce training losses to zero in the presence of label noise, even with multiple rounds of self-distillation. 

3. It seems that the terms “true label” and “given label” are not properly defined. As I understand it, the “given label” refers to the target label, which may include noise, while the “true label” represents the oracle.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732167252141

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The revised paper is uploaded

comment:
We sincerely thank the reviewers for their constructive feedback. Based on their comments, we have revised our paper accordingly, marking the changes in blue. The modifications are as follows: 

* **(Sec. 1 L77-78, Sec. 2 L246-250)**: Clarified the explanation regarding 100% population accuracy. (Reviewer UkvB, Q1)
* **(Sec. 2 L130-131, 134-135 / App.J.1 L1578, L1581, 1682 / App.K L1866, 1885, 1922)**: Clearly defined “ground-truth label” and “given label”, and replaced "provided label" with "given label" throughout the manuscript. (Reviewer UkvB, Q3)
* **(App.C)**: Elaborated on extending our work to more general settings of self-distillation, involving trainable feature extractors during multi-round self-distillations. (Reviewer XuSH, W2)
* **(App.E.3)**: Provided a more detailed explanation for the use of the GCE loss in our experiments with the PLL student model. (Reviewer g4HH, W1)
* **(App.H.2)**: Added experiments using a larger feature extractor backbone (ViT-B) to the ablation section. (Reviewer CZ26, W3)

We have also provided detailed responses to all the reviewers' questions. We hope that these responses address the reviewer’s concerns and questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper provides a theoretical analysis of the mechanisms underlying self-distillation in a linear probing setting. The analysis demonstrates that after several rounds of self-distillation, the model's predictions converge to a weighted average of the provided labels, with the weights determined by the Gram matrix. Leveraging this insight, the authors examine the impact of label noise and evaluate the efficiency of the self-distillation method.

Experiments validate the effectiveness of the proposed single-round self-distillation method. A key theoretical contribution of the paper is the interpretation of self-distillation as label averaging among highly correlated instances, offering a fresh perspective on its mechanism even in the absence of feature evolution. Additionally, the authors present numerical and visual analyses of the approximation of the softmax function and the feature correlation matrix, providing reasonable and convincing evidence to support their findings.