Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Working in the teacher-student setting, the authors propose and study a possible method of improving the efficacy of the teacher network by manipulating the temperature parameter of its probability density over labels. They are motivated by the practical need to distill the ‘knowledge’ of the teacher into smaller networks that can be run locally on small devices.
strengths_and_weaknesses: This paper treats an interesting and relevant problem, namely, how to increase the information content of the output of the teacher network. They propose a novel method they call asymmetric temperature scaling (ATS) as one potential solution to this problem. However, I think there are two kinds of issues that significantly weaken the work to the point where I really cannot recommend acceptance: (1) the mathematical basis of the proposed method is ill-defined and shaky, and (2) there is a marked lack of clarity in the writing. Core mathematical details on which the analysis of the paper rests are not properly laid out, making it difficult to assess the contribution of the paper. I suspect the issues are structural and cannot be fixed over the course of a review period. While I am happy to be convinced otherwise, for now I am afraid I must recommend that the paper be rejected.
questions: There is a serious lack of clarity in the exposition. For instance, the core technical motivation of the paper seems to be that larger/more complex/more accurate networks ought to be better teachers (than smaller ones performing the same task), but this is not at all an obvious point. Why do the authors have this expectation?

In Section 3, the teacher-student framework within which the authors are working is not carefully set up. The reader is able to infer that the input to the student network is a probability density over labels generated by the teacher network, but this is not explicitly stated, and no further details of the training problem to be solved by the student (or the details of the teacher network) are provided. The authors might consider rewriting this section to be more informative. Once the mathematical framework is established, it will be easier to demonstrate what the problem is with the output of a large teacher network that prevents it from teaching as effectively as a smaller teacher network, which is what the authors try to do in Section 4. It will also make it easier to precisely characterize the relevance of the temperature parameter to this problem.

In Section 4.1, a decomposition of the loss function of the teacher is made. Per my understanding, this decomposition is then used to motivate ATS. Unfortunately, the decomposition itself is ill-defined - the quantity $E_{j\neq y}$ is not defined at all. This makes it difficult to understand and interpret. If the authors could specify the meaning of this quantity, that would be helpful.

The central quantities the authors ultimately work with are $\mathbb{E}(p)$ and $\mathbb{V}(p)$. (These are introduced in Section 4.2.) However, $p$ is a normalized probability density over labels, and not a random variable. Therefore it is not meaningful to talk about its ‘mean’ and ‘variance’. I’d appreciate it if the authors could clarify exactly the meaning of these quantities. There is a similar problem for the quantities $\mathbb{E}(q)$ and $\mathbb{V}(q)$, where $q$ is a vector of probabilities constructed from $p$ by simply removing the entry corresponding to the correct label. For this reason, I am unable to understand the content of Section 4.2 and therefore much of the rest of the paper.

If the authors are able to edit the paper for clarity, addressing the various points I have mentioned, it will make it easier to try to assess their contribution.

Lastly, I wonder how the idea of asymmetric temperatures affects the input-output map of the teacher network? In particular, if the probability of the correct label corresponding to some input example has one associated 'temperature', and all the other labels have another, then the equilibrium distribution of the resulting map between inputs and label probabilities is no longer the Boltzmann distribution. I imagine this has profound consequences for the map between (teacher) inputs and (teacher) labels. I'd be grateful if the authors could please comment on this point.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use asymmetric temperature scaling to modify the correct class / wrong class logits of teacher networks, to achieve a more efficient distillation. The logic behind ATS is that logits from complex teachers are typically over-confident and less informative, thus by applying high temperature on correct class and low temperature on wrong classes, the logits could become both confident and discriminative. Both theoretical analysis and experimental results are provided to support the effectiveness of the proposed ATS strategy. 
strengths_and_weaknesses: Strength:

The idea of ATS is novel. I also like the presentation of this paper. Easy & clear to follow. 

Weakness:

Theoretical part is a bit confusing. Reading the decomposition of KD losses, while the proposed ATS improves the latter two terms, wouldn't it hurt the first term, the correct guidance term? How would the trade-off look like? 

I also believe that the experimental part could be much improved. The authors did provide a lot of details and analysis on their results already, but I would like to see the comparisons to more SOTA methods on KD, e.g.,

1) https://arxiv.org/pdf/2107.00181.pdf
2) https://arxiv.org/pdf/2104.09044.pdf


questions: As listed in the strength and weakness part, my question is primarily regarding if the improvement on smooth regularization and class discrimination comes at a cost of reducing correct guidance, as well as some improvement on the experimental part. 
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper revisits the phenomenon in knowledge distillation that a teacher model with better performance doesn’t necessarily transfer information better to the student model. By revealing that the scaling temperature value has a different influence on three terms, correct guidance, smooth regularization and class discriminability, the authors propose to set asymmetric temperature value for the distillation loss function, particularly encouraging more derived variance to improve the distillation better. Experimental results on commonly used vision models have been shown to prove the effectiveness of the proposed model.
strengths_and_weaknesses: Strength
1 The paper studies a commonly ignored problem in knowledge distillation, with a relatively simple yet effective solution to solve it.

2 The figures to illustrate the relation between different terms and concept are clearly displayed.


Weaknees

1 Some introduction about the compared methods should be put in the main content rather than the supplementary materials. It is difficult to relate the shorname in Table 2 to corresponding methods and tell the significance of the contribution for the proposed method. 

2 For marginal improvements, there lacks of standard deviation to tell the significance. For example, the result in Table2 for RNX29, the last two rows under SFC1 column (75.47 vs 75.43)

questions: 1  When repeating the “KD+ATS” two times and making predictions via ensemble, do the tau1 and tau2 keep the same or not?

2 Compared with ResKD, the proposed method generally performs worse when student model is SFV1 and the only better one is 75.47 over 75.43 which is very insignificant. Can the authors provide any explanation for this?

limitations: Yes
review_title: null

Meta-review: metareview: This paper studies distillation consequences of carefully tuning the softmax temperature.  Reviewers found a variety of weaknesses but were positive overall.  I recommend acceptance, but that the authors carefully address the concerns raised during review, which will greatly improve the strength of the paper's message.
confidence: Less certain
Decision: Accept