Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an instance-specific label smoothing (self-distillation) technique, Pseudo-KD, formulated as a two-stage optimization problem. On popular CV and NLU benchmarks, authors conducted experiments to compare Pseudo-KD against LS, KD, and various teacher-free methods, demonstrating the usefulness and efficiency of the proposed technique.
main_review: This paper presents an online self-distillation method. The paper is well-written and I enjoy reading it. However, the proposed link between label smoothing and knowledge distillation, in my opinion, is still based on previous research findings (e.g., Yuan et al., 2020). When compared to previous research, it would be interesting to see what new insights can be gained from the newly proposed loss function (7) (other than adding the smoothing kl term to the original ce loss). Or, to put it another way, I believe it is nearly equivalent if we use the formulation proposed in previous work (Yuan et al., 2020). However, in section 3.2, this paper presents a closed-form solution to the two-stage optimization problem.

In the evaluation part, it would be great if authors could compare Pseudo-KD to the recent efforts on self-distillation (instance-specific label smoothing), for example:
   Zhang, Zhilu, and Mert R. Sabuncu. "Self-distillation as instance-specific label smoothing."
   Yuan, Li, et al. "Revisiting knowledge distillation via label smoothing regularization."








summary_of_the_review: 1. novelty is limited compared to the recent work on self distillation and instance-specific label smoothing
2. missing relevant baselines in the evaluation section
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel teacher-free label regularization method named Pseudo-KD which bridges the gap between label smoothing and knowledge distillation. The proposed bilevel programming framework is implemented by an alternating two-stage process. The optimal smoothed labels are generated in the upper optimization, and the backbone network is optimized by the inner loop. The overall framework is equivalent to self-distillation without the need for extra computational cost. The authors claim that the extensive experiments on both image classification and natural language understanding tasks validate the effectiveness of the proposed method. 
main_review: Pros:
1. The paper is well-written and well-organized. Codes are provided.
2. The paper proposes an adaptive label smoothing method, which is novel to some extent.
3. The proposed bi-level framework and its implementation look reasonable.

Cons:

1. The motivation of the paper is my main concern. The method is called Pseudo Knowledge Distillation and the authors claim the method bridges the gap between label smoothing and knowledge distillation, however it seems to have no connection to Knowledge Distillation. The generated smoothed labels don't consider the relevance of classes which is defined as the dark knowledge in [1]. The motivation of adding a KL divergence term into Eq.7 is not elaborated clearly. 
2. The effectiveness of the proposed method is not convincing. The improvement of Pseudo-KD in practice is not significant. 
3. Some baselines are missing, such as Deep Mutual Learning [2] and Knowledge Refinery [3].

[1] Geoffrey E. Hinton, Oriol Vinyals, Jeffrey Dean. Distilling the Knowledge in a Neural Network
[2] Zhang, Ying, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. CVPR2018
[3] Qianggang Ding, Sifan Wu, Tao Dai, Hao Sun, Jiadong Guo, Zhang-Hua Fu, Shutao Xia. Knowledge Refinery: Learning from Decoupled Label. AAAI 2021
summary_of_the_review: The idea is interesting and novel enough. However, considering the motivation and the experimental results, I vote for "5: marginally below the acceptance threshold".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new formulation of an objective function for label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD). The objective function learns an instance-specific label smoothing regularization while training a model towards its prediction target. When the instance-specific label smoothing regularization uses a simple uniform distribution for all training samples, the objective function falls back to traditional LS. When the instance-specific label smoothing regularization uses the output distribution of a teacher model, the objective function learns a KD model. The paper formulates the learning of the optimal instance-specific label smoothing regularization as a bi-level optimization problem and derived a closed-form solution for the inner level of the problem. Experimental results on both image classification and natural language understanding tasks show that the proposed objective and learning algorithm are effective and yield results comparable to the baseline LS and KD models. 
main_review: Strength:
1. The paper proposes a new formulation of an objective function for instance-specific label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD), which is quite interesting. 

2. It is good to find a closed-form solution for the inner level of the bi-level optimization problem of the proposed objective function.

3. Experiments have been done with data from different domains (images and texts) and have confirmed the effectiveness of the proposed objective function. 

Weakness:
Presentation of the experimental results can be improved:
In Tables 4.2 and 4.4, it would be interesting to also show the standard deviation of the test accuracy/GLUE scores. The results of the propose Pseudo-KD model does not seem particularly attractive in these two tables. 

Minor presentation issues:
Page 5: "in the second phase, we fix the" => "in the second stage, we fix"
Page 6: It would be good to add references for the KD-Shuffle and KD-29 models. 

summary_of_the_review: The paper proposes a new formulation of an objective function for instance-specific label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD), which is quite interesting but is also largely based on the CVPR 2020 work by Yuan et al. The paper further formulates the proposed objective function as a bi-level optimization problem and finds a closed-form solution for the inner level of the problem. Experiments confirmed the effectiveness of the proposed objective function on both image classification and natural language understanding tasks, while the improvements over the baselines are somewhat limited. Overall this is a solid study with reasonable contributions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a self-distillation approach that is derived from a series of observation made regarding how LS and KD work, linking in the two together in the process. The paper then experimentally shows the superiority of the proposed approach against KD on CIFAR-10/100 and languge datasets. Important comparisons against closely-related methods are however missing. 
main_review: The authors propose a self-distillation approach that is derived from a series of observation made regarding how LS and KD work, linking in the two together in the process.

Pro:
+ The approach and the mathematical proofs provided are sound
+ The two step training process is fast and matches or outperform the vanilla KD baseline.
+ The performance is tested on both vision and language data showing that the approach generalizes well.

Cons:
- There is a relatively large body of work on self-distillation however there is no comparison against them, neither on explaining how they differ nor empirically. A non-exhaustive list:
[A] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation, Zhang et al, ICCV 2019
[B] Self-Knowledge Distillation with Progressive Refinement of Targets, Kim et al, ICCV 2021
[C] MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation, Liu et al, ECCV 2020
[D] Regularizing Class-Wise Predictions via Self-Knowledge Distillation, Yun et al, CVPR 2020
- How does this method work when combined with other distillation losses (such as feature matching)? KD generally can boost their performance, is the same true here?
- Little comparison with state-of-the-art on knowledge distillation and while surpassing the KD baseline is important in this context its unclear if applying such method in practice will suffice. As such the authors should offer a comparison with the latest work on KD too.
- How well does this scale to larger datasets, such as ImageNet?

Minor:
Table 2, some values are wrongly in bold (ex: KD outperforms P-KD on CIFAR-10)


summary_of_the_review: While the paper is interesting and the idea of performing "distillation" (pseudo) is appealing the experimental section needs to be improved.
review_title: null

Meta-review: This work proposes an instance-specific label smoothing method, which is formulated as a two-stage optimization problem for finding the optimal label smoothing. The authors show that the proposed approach can be equivalent to an efficient variant of self-distillation techniques (i.e. no need to store the parameters or the output of a trained model). Experiments on image classification (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) demonstrate that our method is competitive against strong baselines.

The reviewers find the proposed approach reasonable, and the presentation clear. However, they all rated the paper as borderline, due to some concerns that the submission has in its current form. These include limited novelty (by CUiW) [the link between label smoothing and knowledge distillation is largely based on previous research findings (e.g., Yuan et al., 2020)], nonconvincing results on the effectiveness of the proposed method (by bwuZ) [Improvement of Pseudo-KD in practice is not significant in terms of test accuracy gains], and lack of comparison with some recent related methods (by JBRd as well as other reviewers). The authors responded to these (and other concerns), but this did not convince the reviewers about the concerns listed above. I recommend the authors to resubmit after addressing these issues.
Decision: Reject