Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a variant of self distillation which incorporates both the model output and the ground truth targets. The author provided a detailed theoretical analysis about how fixed weighted ground truth restrain the regularization amplified by self distillation.
main_review: The paper clearly stated the motivation of introducing a weighted ground truth target while self distillation. The theoretical results (Theorem 4.3, 4.5) supported the argument. 
The author also provided a close form solution for setting the step wise weight other than a fixed weight for all steps.
The estimation of the optimal weight provided practical usefulness for this method and guidance on how to adjust this hyperparameter efficiently other than grid search.

However, my minor concern about this work are: 

the main theoretical results are all derived under the setting of fixed weight for all distillation steps. While in practice the optimal weights are calculated at each step. 

In section 4.2, the author argued by introducing the weight parameter, the self distillation process can potentially do not sparsify the basis. According to theorem 4.3, this potential heavily depends on the value of the weights and the weights are derived in a closed form manner at each step. It would be more rigorous if the author could provide some guarantees with probability of theorem 4.3 to show how likely the basis will not sparsify.

Overall, I think this work proposed an inspiring potential method on addressing the evolution of basis problem carried out by Mobahi et al.(2020), along with some practical usage.
limitations_and_societal_impact: There is no negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presented a theoretical and empirical study of weighting the teacher outputs and the target outputs during self distillation. For an iterative self distillation process, a solution is shown to compute the fit at any step of the iteration given the first model. This was further related to classical kernel ridge regression with regularization.
main_review: The existing theory behind distillation is thin, and this paper makes a step toward a better grasp of it. I found the claims well supported by experiments and well explained in the text. I have not thoroughly checked the proofs in the supplementary material.

I found the writing quality high and the exposition clear.

The findings and theory presented is likely significant in the community's effort to deepen our understanding of distillation.
limitations_and_societal_impact: There is adequate mention of the limitations of this work. The authors did not address the societal impact, and I agree with the assessment that it is not necessary for this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the theory of self-distillation in the kernel ridge regression setting. The authors extend the theoretical results in Mobahi et al. (2020) to further incorporate the ground-truth labels in the distillation objective and highlight that the ground-truth labels serve to dampen the sparsification and regularization of the self-distilled solutions.
main_review: This paper provides closed-form solutions of self-distillation in the kernel ridge regression setting. Overall, the entire paper is mostly well-written and easy to follow. It is a natural extension of the theoretical results in Mobahi et al. (2020), though the theoretical novelty is limited given the prior works (the theorems are not hard to derive based on the results of Mobahi et al. (2020)). See detailed comments below.


1. After reading the paper, I am not convinced that incorporating the ground-truth label is essential during the self-distillation step to overcome underfitting. As you show in Theorem 4.5, the weighting parameter $\alpha$ turns out to amplify the regularization parameter $\lambda$. In this sense, we can also smartly tune $\lambda$ on the validation set to achieve the same effect as tuning $\alpha$. Therefore, I don't think that incorporating the ground-truth labels of training data are necessary. 


2. The way to estimate the value of $\alpha$ in section 4.3 is not globally optimal, as it does not account for the future steps of self-distillation. I would instead call it a greedy estimation of $\alpha$.


3. For  ResNet-50 on CIFAR-10, why are the best training accuracy only 88% and the best test accuracy only 94%? In my experience, the training accuracy can always reach very close to 100%, and the test accuracy is around ~94%. Please check out this GitHub: https://github.com/kuangliu/pytorch-cifar. 


4. For Line 239-240, you mentioned that "As illustrated in Figure 2a for case (a), the regularization imposed by self-distillation initially improves the quality of the solution". To me, I don't see such a trend in Figure 2(a). The curves corresponding to $f_1, ..., f_{\infty}$ become more underfitting as the index value increases. To better illustrate your argument, I would suggest the authors provide a figure of "$\tau$ vs. training loss" (similar to Figure 5).


5. It would be better to state the contribution in the introduction rather than in related works (Line 84-93). Also, the introductions to the notation (Line 96-101) are well-suited to be at the beginning of Section 3 Problem Setup.


6. One minor correction to footnote 4, for some cases (e.g., using Adam optimizer), the equivalence between L2 regularization and weight decay does not hold. See Loshchilov et al. (2017) and Zhang et al. (2018).


Overall, I think this paper is well-written and studies a critical problem in machine learning, considering that there are still relatively few works exploring the theories of knowledge distillation.  However, the theoretical novelty is very limited as most of the theorems are very natural extensions from the results in Mobahi et al. (2020). Also, I am not very convinced that incorporating the ground-truth label is necessary for self-distillation. Therefore, I vote for a weak rejection at the current stage. I am happy to increase my ratings if the authors can address my concerns properly.

References:

1. Loshchilov, Ilya, and Frank Hutter. "Decoupled weight decay regularization." arXiv preprint arXiv:1711.05101 (2017).

2. Zhang, Guodong, et al. "Three mechanisms of weight decay regularization." ICLR 2018.


====Updates=====

The authors' response resolved my concern. I will increase my score from 5 to 6. 
limitations_and_societal_impact: The authors discussed the limitation of their work but did not discuss the negative societal impact. To discuss the negative societal impact, the authors can consider some cases where self-distillation can be applied to achieve some malicious goals.
review_title: null

Meta-review: This paper provides an in-depth analysis of self-distillation in kernel regression setting, and studies the effect of using a weighted combination of ground truth labels and predictions made by the model, to define the next set of target values. It provides a closed form solution for the optimal choice of weighting parameter at each step, and shows how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search.

All reviewers find this paper interesting, and rate it in the accept zone. Some of the initial scores were lower at first, but after author's reply, reviewers found them convincing and increased the rating. In concordance with the reviewers, I find distillation an important tool with increasing popularity, and providing a solid understanding about this technique is of great interest to the community.

Decision: Accept (Poster)