Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Authors: Xin-Chun Li
Wen-shu Fan
Shaoming Song
Yinchuan Li
bingshuai Li
yunfeng shao
De-Chuan Zhan
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowldege distillation
larger teacher
temperature scaling

Abstract: Knowledge Distillation (KD) aims at transferring the knowledge of a well-performed neural network (the {\it teacher}) to a weaker one (the {\it student}). A peculiar phenomenon is that a more accurate model doesn't necessarily teach better, and temperature adjustment can neither alleviate the mismatched capacity. To explain this, we decompose the efficacy of KD into three parts: {\it correct guidance}, {\it smooth regularization}, and {\it class discriminability}. The last term describes the distinctness of {\it wrong class probabilities} that the teacher provides in KD. Complex teachers tend to be over-confident and traditional temperature scaling limits the efficacy of {\it class discriminability}, resulting in less discriminative wrong class probabilities. Therefore, we propose {\it Asymmetric Temperature Scaling (ATS)}, which separately applies a higher/lower temperature to the correct/wrong class. ATS enlarges the variance of wrong class probabilities in the teacher's label and makes the students grasp the absolute affinities of wrong classes to the target class as discriminative as possible. Both theoretical analysis and extensive experimental results demonstrate the effectiveness of ATS. The demo developed in Mindspore is available at \url{https://gitee.com/lxcnju/ats-mindspore} and will be available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/ats}.

Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again
Xin-Chun Li', Wen-Shu Fan', Shaoming Song", Yinchuan Li
Bingshuai Li Yunfeng Shao", De-Chuan Zhan
1 State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
2 Huawei Noah's Ark Lab, Beijing. China
(lixc, fanws nnju edu. en, zhandc@nju. edu.
(shaoming.som song, liyinchuan, libingshuai, shaoyunfeng) @huawei. com
Abstract
Knowledge Distillation (KD) aims at transferring the knowledge of a well- performed neural network (the teacher) weaker one (the student). peculiar phenomenon is that more accurate model doesn't necessarily teach better, and temperature adjustment this, we decompose the efficacy of KD into three parts: correct guidance, smooth regularization, and class discriminability. The last term describes the distinctness of wrong class probabilities that the teacher provides in KD. Complex teachers tend be over-confident and traditional temperature scaling limits the efficacy of class discriminability, resulting in less discriminative wrong class probabilities. Therefore, we propose Asymmetric Temperature Scaling (ATS), which separately applies a higher/lower temperature to the correct/ class. ATS enlarges the variance of wrong class probabilities in the teacher's label and makes the students grasp the absolute affinities of wrong classes to the target class as discriminative as possible. Both theoretical analysis and extensive experimental results demon- strate the effectivvneess of ATS. The demo developed in Mindspore is available at and will be available https httpaaaww.....aoooo///////////ee aat/ts
neither alleviate the mismatched capacity. To explain
https: http::///ggieee.ccc////c///////p
1 Introduction
Although large-scale deep neural networks have achieved overwhelming successes in many real- world applications [22 [1 660, the vast capacity hinders them from being deployed portable devices with limited computation and storage resources [3] Some efficient architectures, e.g.,
MobileNets [14 37 and ShuffleNets [59 29) have been proposed for lightweight deployment, while their performances are usually constrained. Fortunately, knowledge distillation (KD) [66] 13] could transfer the knowledge of more complex and rellorerffrmed network (i.e., the teacher) them.
The original KD 13 forces the student to mimic the teacher's behavior via minimizing the Kullback- Leibler (KL) divergence between their output probabilities. Recent studies generalize KD to various types of knowledge [36]
[2 33
52 27
54
45,
50,
26
23] or various distillation
schemes [61 58 20 An intuitive sense after the proposal of KD [3] that larger teachers could teach students better because their accuracies are higher. A recent work (6] first points out that the teacher accuracy is poor predictor of the student's performance. That is, more accurate neural networks don't necessarily teach better. Until now, this phenomenon is still counter-intuitive [51], surprising (31] and unexplored 24 Different from some existing empirical studies and theoretical
analysis 44 18 30 35
55
63
28 15), we investigate the miraculous phenomenon detail and
aim answer the following questions: What's 's the real reason that more complex teachers can't 't teach
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

d
6
Smaller Texcher
SuB
-Bus
-0.4
H0.2
-1.B
Logits
Teacher
Decomposition ofXD
Student
Teucher
Teucher's
Correct Guidanee
Smooth Regularization
Class Diseriminabilit
Lurger Torget Zugel
Smsaller Ankeront Variance
Softmax with Temporature 4.0
0.731
Rubss
auem
runco
Probs
Tencher
Dertved Farisnce: & 8.8857
Teacher
Higher
Derived Variiace: 0.0021
Derived Variance: 0.0030
Figure 1: Left: Decomposition of teacher's label. The first class is the target. As temperature increases, correct guidance is weaker, smooth regularization stronger, while class discriminability (measured by the variance of wrong class probabilities) will first increase and then decrease. Right: Larger/Smaller teachers' logits are consistent in relative class affinities, i.e., logit values of the four wrong classes are in the same order of magnitude. However, larger teachers are over-confident and give larger target logit or smaller inherent variance, leading to a smaller derived variance under traditional temperature scaling, i.e., less distinct wrong class probabilities after softmax.
well? Is it really impossible to make larger teachers teach better through simple operations, such as temperature scaling?
To answer the first question, we focus on analyzing the distinctness of wrong class probabilities
that a teacher provides KD. We decompose the teacher's label into three parts (see Sect. 4.T): (I)
Correct Guidance: the correct class's probability; (II) Smooth Regularization: the average probability of wrong classes; (III) Class Discriminability: the variance of wrong class probabilities (defined as derived variance). The commonly utilized temperature scaling could control the efficacy of these three terms (the left of Fig. D}. More complex teachers are over-confident and assign a larger score for the correct class or less varied scores for the wrong classes. If we use uniform lemperature to scale their logits, the class discriminability of the larger teacher less effective (theoretically analyzed in Sect. i.e., the probabilities of wrong classes are less distinct (the right of Fig. 1
As the second question, we focus on enlarging the variance of wrong class probabilities (i.e.. derived variance) that a teacher provides to make the distillation process more discriminative. To specifically enhance the distinctness of wrong class probabilities, we separately apply a higher/lower temperature to the correct/wrong class's 's logit instead of uniform temperature (see Sect. 4.3). We name our method Asymmetric Temperature Scaling (ATS), and abundant experimental studies verify that utilizing this simple operation could make larger teachers teach well again.
2 Related Works
KD with Larger Teacher: Although KD has been a general technique for knowledge transfer in various applications [13 61 25}, could any student learn from any teacher? (6] first studies the KD's dependence on student and teacher architectures. They find that larger models do not often make better teachers and propose the eurly-stopped teacher as a solution.
[31 introduces a multi-step KD process, employing an intermediate-sized network (the teacher assistant) to bridge the capacity gap. [51] formulates KD as multi-task learning problem with several knowledge transfer losses. The transfer loss will be utilized only when its gradient direction is consistent with the cross-entropy loss.
10. 24 define the knowledge gap as residual, which is utilized to teach the residual student, and then they take the ensemble of the student and residual student for inference. These works attribute the worse teaching performance to capacity mismatch, i.e., weaker students can't t completely mimic the excellent teachers. However, they don't explain this peculiar phenomenon in detail.
Understanding KD: Quite few works focus on understanding the advantages of KD from a principled perspective. (28) unifies KD and privileged information into generalized distillation. 35 18 utilize gradient flow and neural tangent kernel analyze the convergence property KD under deep linear networks and infinitely wide networks. [ explains KD via quantifying the task-

---

Table 1: The used notations in this paper. The definitions of Derived Average, Derived Variance and
Inherent Variance are only for wrong classes (Sect. 4.1 and Sect. 4.2
All Classes
Wrong Classes
Logit
g
Probability
p SF(f)
9 = Pelcty
Derived Average of Probabilities
e(g) = Σ, 9,/(C-1)
Derived Variance of Probabilities
c ((()) (g -(()))((C))
Inherent Variance of Probabilities
q = SF(g), v( = [9, -C9))(()))
relevant and task tusk-irrelevant visual concepts. 7 casts KD as a semiparametric inference problem and proposes corresponding enhancements. Our work is more related to KD decompositions. treats the teacher's s correct/wrong outputs differently, respectively explaining them as importance weighting and class similarities. [40] further decomposes the "dark knowledge" into universal knowledge,
(30] establishes a bias-variance tradeoff to quantify the divergence a teacher with the Bayes teacher. [63] utilizes bias-variance decomposition to analyze
domain knowledge, and gradient rescaling.
KD and discovers regularization samples that could increase bias and decrease variance. Our work is also related to Label smoothing (LS). (55) points out that the regularization effect in KD similar to LS. [32] finds that training teacher with LS could degrade its teaching quality, and attributes this to the fact that LS crases relative information between teacher logits. Recently, (38] further studies this problem and proposes a metric to measure the degree of erased information quantitatively. Our work also decomposes KD into several effects to study why more complex teachers can't 't teach well. Detailed relatedness these works is presented in Sect. 4.1
3 Background
We consider a C-class classification problem with y =[C] = -1.22....C) Given neural network and sample pair (x, y). we could obtain the "logits" f(x) e RC We denote the softmax function with temperature 5 as SF(:: i.e. Pe(T) = exp (f(((x)7))/(() and Z(T) Σ5n exp (f, (x)/r). where p(7) is the softened probability vector that network outputs and is the index of class. Later, we may omit the dependence on x and 7 if without any ambiguity. We use fg and Py to denote the correct class's logit and probability, while we use g and q to represent the vector of wrong classes' logits and probabilities, i.e., g [fe]ory and q [P-]oey- The notations could be found in Tab. [
The most standard KD 13] contains two stages of training. The first stage trains complex teachers, and then the second stage transfers the knowledge from teachers to a smaller student via minimizing the KL divergence between softened probabilities. Usually, the loss function during the second stage (i.e., the student's 's learning objective) is combination of cross-entrops loss and distillation loss:
C Σ P.. (o)
= -1 - A) lo)
(1)
CE Loss
KD Loss
where the upper script "T"'"S" denotes "Teacher" ""Suudent respectively. Commonly, temperature of 1 is utilized for the CE loss, and the student could also take temperature of for the KD loss, e.g., P
a default
1) 13
Suppose we have two teachers, denoted as Tinge and Temall. and the larger teacher performs better on both training and test data. If we use them to teach the same student S. we could find that the student's performance is worse when mimicking the larger teacher's 's outputs. Adjusting the temperature could neither make the larger teacher teach well. The details of this phenomenon could be found in 6 31 and Fig. 9 Obviously, Tine could differ lot from Tinil which is the only difference in the loss function when teaching the student. Hence, we focus on analyzing what probability distributions are tended to be provided by teachers with different capacities.
3

---

4 Proposed Methods
This section first decomposes KD into three parts and defines several quantitative metries. Then, present theoretical analysis to demonstrate why larger networks can't It teach well. Finally, we propose a more appropriate temperature scaling approach an alternative.
4.1 KD Decomposition
We omit the coefficient of Ar in Eq. 1 and define e (q²(+)) = 0-I
1 E Zj-1 Df (T), where qT(+)=
[P" (T) (e)y. Then, we have the following decomposition:
lyd -P' log
Σe(a" (T))
El e (+)) log
(2)
CFY
cPy
Correct Guidance
Smooth Regularization
Class Discriminability
(I) Correct Guidance: this term guarantees correctness during teaching. The decomposition in [9] also contains this term, which is explained as importance weighting. This term works similarly to the cross-entropy loss, which could be dealt with separately when applying temperature scaling.
attribute the success of KD to the efficacy of regularization and study its relation to label smoothing (LS). The combination of this term with correct guidance works similarly to LS. Notably, e (q" (T)) differs across samples, implying that the strength of smoothing is instance-specific, which is similar to the analysis in (62].
(II) Smooth Regularization: some previous works 55
62 631
(III) Class Discriminability: this term tells the student the affinity of wrong classes to the correct class. Transferring the knowledge of class similarities to students has been the mainstream guess of the "dark knowledge" in KD 13 38). Ideally, good teacher should be as discriminating as possible in telling students which classes are more related the correct class.
Illustrations of the decomposition are presented in the left of Fig. [ Obviously, an appropriate temperature should simultaneously contain the efficacy of the three terms, e-g., the shaded row in Fig. 1 A too high or too low temperature could lead to smaller class discriminability, making the guidance less different among wrong classes, which weakens the distillation performance in practical. Among these three terms, we advocate that class discriminability is more fundamental in KD and present more discussions in Appendix [
(verified in Fig. 2 and Fig. 3
To measure these three terms quantitatively, we use the target class probability (i.e., Py), the average of wrong class probabilities (i.c., e(q) c.i Estu Pj), and the variance of wrong class probabilities (i.e., v(q)= टं. Ejen (D)- e(q))²) estimators. e(-) and v(-) calculates the mean and variance of the elements in a vector. In some cases, we use the standard deviation as an estimator for the third term, i.c.. (q))= = 22 (q). Because the latter two terms are calculated after applying softmax to the complete logit vector, we define them as Derived Average (DA) and Derived Variance (DV), respectively. In experiments, we calculate these metries for all training samples and sometimes report the average or standard deviation across these samples.
4.2 Theoretical Analysis
We analyze the mean and variance of the softened probability vector, i.c.. the teacher's label p" (r) used in KD. We defer the proofs of Lemma 4.1 and Proposition 4.3 to Appendix B
Lemma 4.1 (Variance of Softened Probabilities). Given logit vector f E BCC and the softened probability vector = SF(F;T), T (0, oo), (p) monotonically decreases as T increases.
As increases, p(T) becomes more uniform, i.e., it's entropy increases. However, especially focus on the wrong classes, where the mean and variance are more intuitive to calculate and analyze.
Assumption 4.2. The target logit is higher than other classes' logits, i.c., fy 2 f.. Vc y.
Assumption rational because werl-performed teachers could almost achieve higher accuracy (e.g., 995% on the training set, and most training samples meet this requirement.
Proposition 4.3. Under Assumption 4.2 Py monotonically decreases as т increases, and e(g) monotonically increases as T increases. As co, e(q) 1/C.

---

Proposition 4.3 implies that increasing temperature could lead to a higher derived average (empirically see Fig. / and strengthen the smooth regularization term in Eq. 2
Before we analyze the class discriminability term, we define ((T) as the result of applying softmax only to the wrong logits with temperature т, i.e., q(T) SF(g:T), For the element index of q, have
(= - exp((x exp(B)/T) (
Notably, q differs from q a lot. Specifically, the former satisfies Σ. 9. = 1, while the summation of the latter is Eesy Pe 1 Py. The former does not depend on the correct class's 's logit while the latter does. We name v q()) Inherent Variance (IV) because it only depends on wrong classes' logits.
Proposition 4.4 (Derived Variance vs. Inherent Variance). The derived variance is determined by the square of derived average and the inherent variance via:
v(q) (C- e²
v(q)
DV
DA
IV
With т increases, e(q) increases (Proposition 4.3) while v(q) decreases (Lemma 4.1 and hence, is not so easy to judge the specific monotonicity of v(q) w.r.t. т. Empirically, we observe that the derived variance first increases and then decreases (see Fig. which conforms the change of the class discriminability as illustrated in Fig. |
We could use Proposition 4.4 to clearly analyze why larger teacher networks can't 't teach well. Before this, we present another two properties and a corollary without detailed proof. Remark 4.5. Fixing g and т, a higher target logit fy leads to a higher Py, i.e., a smaller derived average e( e(q).
Remark 4.6. Fixing T, less varied wrong logits g leads to less varied q, i.e., a smaller inherent
variance vÂ)).
Corollary 4.7. Suppose we have two teachers T1 and T2, and their logit vectors for a same sample are f and fT:
ff 2 f.J² while T. and Ta are nearly the same, then p 2 DT (Remark 4.5) while v(²")) x v() Hence, v(q S u(q")).
18c = f while v(ght) s (ggh),, then p Ta while o(q2t) s o(q²t) (Remark 4.6 Hence, v(q"t) s u(q")).
This corollary explains why larger teacher can't teach better. Because the larger teacher tends to be over-confident, the target logit f, may be larger or the variance of wrong logits v(g) may be smaller. These are illustrated in Fig. [ and empirically verified Fig. 4 Then the derived variance v v(q) may
be smaller, limiting the efficacy of class discriminability in Eq. 2 Empirical results are in Fig. 7
Notably, we focus on analyzing the variance of wrong class probabilities instead of all classes. Maximizing the variance of all classes' probabilities does not mean maximizing the variance of wrong classes'. For example, although a very low temperature can maximize the variance of all classes' probabilities, the generated teacher's ' label is one-hot that shows no distinctness between wrong classes. In other words, the effectiveness of KD should be more related to the distinctness between wrong classes rather than all classes. However, traditional temperature scaling applies uniform temperature for all classes, which cannot separately handle the wrong classes.
4.3 Asymmetric Temperature Scaling
We conclude the above analysis: if a larger teacher makes an over-confident prediction, the wrong class probabilities it provides could be not discriminative enough. Utilizing uniform temperature could not enlarge the derived variance as much as possible with the interference of the target class's logit (see the middle of Fig. 77. Thanks to the decomposition in Eq, 2 the correct guidance term works similarly to the cross-entropy loss and allows us to deal with it separately. Hence, we propose a novel temperature scaling approach:
(T( 72) exp (f./T.) Σ cxp (E5TTT)) ]c[[]
7 IIi vtT I(i
(5)
5

---

ResNet-=SFV1
WRN-MW
Smooth Regularization
Class Discriminability
76-
66.5
55
08.
71
64
8130
4r40
47463
8130
Derived Average
Derived Variance
4:210
Figure 2: Student's test accuracies without KD ("NoKD"), with KD ("-KD"), and only with the first terms in Eq. 2 ("IISS") "ST"LTT" refers to "small/large teacher".
Figure 3: Correlations of smooth regularization (measured by derived average) and class discrim- inability (measured by derived variance) w.r.t. KD improvement ratio.
where we take Ti > 72 0. This approach is named Asymmetric Temperature Scaling (ATS) because we apply different temperatures to the logits of correct and wrong classes. According to Eq. 4 ATS could bring such benefits when the teacher is over-confident:
If the teacher outputs larger logit fy for the correct class, relatively larger could decrease it to a reasonable magnitude, i.e., decreasing Py and increasing e(q). and finally increasing the derived variance v(g):
If the teacher outputs less varied logits for wrong classes, a relatively smaller temperature T2 could make them more diverse, i.c., increasing v()), finally increasing the derived variance (()).
ATS is more flexible in enlarging the derived variance (see the right of Fig. 7 i.e., it could generate more discriminative distillation guidance during teaching. Take the demo in Fig. ass an example, the smaller and larger teacher captures the same relative class affinities, e.g., they both know that the fourth/fifth class (shaded cells) is the most/least relevant to the target class. However, with uniform temperature 4.0, the smaller teacher provides probabilities (0.073, 0.060) for these two classes, while over-confident larger teachers provide (0.040, 0.033) or (0.074, 0.067). Clearly, the absolute affinities of the larger teachers are not so discriminative as the smaller teacher's. s. Utilizing ATS, we could respectively apply (7 4.67, 72 4.0) (ri 4.0, T2 2.0) the over-confident teacher's logits, generating the same probability vector as the smaller teacher's. ATS utilizes two temperatures and creates the wiggle room to make the distribution over wrong classes more discriminative.
Experiments
We use
Stanford Dogs [19, and Google Speech Commands [88 as the datasets. For teacher networks, we use different versions
100 211. TinyImageNet 43 CUB 47
of ResNet (T]] WideResNet [56]. ResNeXt [49]. For student networks, use VGG [39], Shuf-
IIV2 (5 29], AlexNet 222, MobileNetV2
37 and DSCNN [60]
We majorly follow the training settings in [4] Except that the Google Speech Commands takes 50 epochs, we train networks on other datasets with 240 epochs. We use the SGD optimizer with 0.9 momentum. For VGG, AlexNet, ResNet, WideResNet, and ResNeXt, we set the learning rate as 0.05 (recommended by 44)). For ShuffleNet and MobileNet, we use a smaller learning rate of 0.01 (recommended by 44) We use the pre-trained models provided in PyTorch for CUB and Stanford
Dogs, and correspondingly, their learning rates are scaled by 0.1x. During training, we decay the learning rate by 0.1 every 30 epochs after the first 150 epochs (recommended by [44)). For Google Speech Commands, we decay the learning rate via the cosine annealing. We set the batch size as 128 for CIFAR data, 64 for other datasets. Other dataset, network and training details are in Appendix C
5.1 Observations
Class discriminability matters a lot in KD and correlates with the KD improvement. To show the importance of class discriminability in KD, omit the distinctness of wrong class probabilities during distillation. Specifically, we only keep the first two terms in Eq. 2 which works similarly to

---

OIAAL100 ORARLIE O
DRN8.9 mig)
Derived Variance: olg)
Inherent Variance: a(i)
3004
408
e0005..l
100t 2240
WR
Figure 5: The derived variance and inherent vari- ance on CIFAR- 100 using ResNeXt. Each bar shows the mean and standard deviation across 50K training samples. The x-axis shows networks with various capacity.
Figure 4: The distributions of the larget logit (f,)
and the standard deviation of wrong logits (o(g) of the 50K training samples CIFAR-10/I0 Rows show networks with various capacity.
Specmron Comborico
Set Ouetop
Keedail's Tax
Dartibee Dettore
RES14:TS
RES1L0:TS
RES1L0:ATS
Den Wernnce mo)
De Avcrpe
S8009 61-
6.00
7300
0.70
-.755
-0.13
400
24
300
05:
7.5
4.0
1.4
10.3
2.5
4.0
160
1100254444
110025-444
Temorrsbure
Terpparstire
Terrperature
Figure 6: Several metrics among teachers with various capacities on CIFAR-10. The first three metrics are related to the relative magnitudes of teachers' label, while distillation distance is re- lated to the absolute magnitudes.
Figure 7: The change of derived average (c(q)) and derived variance (v(q)) as T increases from 0.1 to 10.0 on CIFAR-10. The third one shows the results of ResNetl 10 with the proposed ATS. DV under TS is limited while ATS enlarges it.
the instance-specific label smoothing (abbreviated as "ILS"). Fig. 2 shows students' performances on CIFAR-100, Without the third term in Eq, E both small teachers ("ST"" and larger teachers ("LT"" teach worse significantly. Then, investigate the correlations of KD performance improvement (i.e., the test accuracy change ratio with KD w.r.t. without KD) with smooth regularization and class discriminabiliit under 270 pairs of "(teacher, student, temperature)". Details in Appendix C.4 Fig. 3 plots the scatters, where dots show pairs whose improvement is higher than 2%. Clearly, teachers with a larger derived variance tend guide better. These observations show the rationality of the proposed KD decomposition and imply that enhancing derived variance is beneficial
Larger teachers provide larger target logit or less varied wrong logits. We first compare the logit distributions provided by larger and smaller teachers on the training set. Fig. 4 plots the histograms (200 bins) of the target logit (i.e., fg) and the standard deviation of wrong logits (i.c.,
(g)). The left and right, respectivcly, show the results on CIFAR-100 and CIFAR- 10. Clearly, the first column shows that ResNetl 10 tends to generate a larger target logit (i.e., Ex [f,]] 15.0) than
ResNet14 (i.e., [f, & 10.0). On CIFAR 10, the smallest f, given by WRN28-8 is larger than WRN28-1. 1. Furthermore, WRN28-8 gives smaller variance (the fourth column), i.e., smaller inherent variance. These reveal specific manifestations of complex networks' over-confidence.
Larger and smaller teachers have similar inherent variance while different derived variance under traditional temperature scaling. We use 7 = 1.0 to soften the complete logits and only the wrong class logits, respectively, and then show the mean and standard deviation of derived variance and inherent variance across training samples in Fig. Although the larger models' inherent variance is smaller, the difference between RNX29-4-44 and RNX29-64 949--64-4d4-- is only up 1.3x. However, the derived variance differs a lot, where the smaller teacher's 's variance is approximately 9.7x as the larger teacher's. These observations imply that traditional temperature scaling could seriously decrease the derived variance of larger teachers though they have appreciable inherent variance.
Complex teachers know approximately the same as smaller teachers on relative class affinities. Only if the relative magnitudes of the wrong class probabilities are correct, valid to enlarge the discrimination between them. Otherwise, if teacher himself misunderstands the knowledge's principles, it will be counterproductive to reinforce this knowledge to students. Given two teachers Ti and T2, we first calculate the Spearman correlation between p" and TT, Second, the set overlap
7

---

2
indtr 2014
CS0 0000
1o40
0210
Caill q
ryua 4, 22
5006
0030
Figure Probability vector visualization of randomly selected training sample from CIFAR- 100. The target class is y = 1. The bottom row shows applying ATS to the larger teacher.
Figure 9: Distillation results via TS (solid curves) and ATS (dashed curves) CIFAR-100. The x-axis of each figure shows teachers with various capacities.
Table 2: Comparisons with SOTA methods on CIFAR-100. ResNetl 10, WRN28-8, and RNX29- 64-4d are teachers. VGG8, SFV1, and MV2 students. The area in gray shows the results of the ensemble. "KD+ATS" and "KD+ATS+Ens" are our methods.
Teacher VGG8
ResNet (74.09)
WRN28 8 (79.73)
RNX29
64-4d (79.91)
Avg
Student
SFV1
MV2
VGG8
SFVI
MV2
VGG8
SFVI
MV2
NoKD
69.92
70.04
64.75
69.92
70.04
64.75
69.92
70.04
64.75
68.24
ST-KD
72.30
73.22
66.56
71.85
72.85
66.52
71.61
72.18
65.82
70.32
KD
71.35
71.86
65.49
70.46
70.87
64.97
71.13
71.80
64,99
69.21
ESKD
71.88
72.02
65.92
71.13
71.32
65.09
71.09
71.27
64.83
69.39
TAKD
72.71
72.86
66.98
71.20
71.62
65.11
71.46
71.44
65.36
69.86
SCKD
70.38
70.61
64.59
70.83
70.52
65.19
70.33
70.92
64.86
68.69
KD+ATS
72.31
73.44
67.18
72.72
73.58
66.47
72.93
73.03
66.80
70.94
Ens
72.77
73.61
67.76
72.77
73.61
67.76
72.77
73.61
67.76
71.38
ResKD
73.89
76.03
69.00
73.84
75.14
67.69
74.64
75.43
68.10
72.64
KD+ATS+EnS
74,86
75.05
69.50
74.60
75.04
68.79
75.34
75.47
69.82
73.16
between the top-5 predictions is calculated, i.e., |C nC CC C denotes the set of top-5 predicted classes. Third, calculate Kendall's between p and p²2, which directly shows the rank correlation of two teachers. These metrics only depend on classes' relative magnitudes. The results are Fig. 6 Excitingly. these metries among teachers with different capacities do not vary a lot, and the Spearman correlations are almost all larger than 0.85. According to the interpretation of Kendall's s [33, the smaller teacher predicts that class is more related to the target class than that of class j. then the larger teacher has probability of (0.75 + 1)/2 = 0.875 to give the same relative affinity. As a comparison, we also calculate the distillation distance defined in [15) which utilizes L1 distance and depends on the absolute magnitudes of probabilities. Using this metric, the distance increases quickly as the capacity gap increases. These observations demonstrate that teachers know approximately the same about relative class affinities while their absolute values differ significantly under traditional temperature scaling.
The proposed ATS could enlarge the derived variance of larger teachers. The above analysis verifies that the over-confident teachers experience lower derived variance under traditional tempera- ture scaling although they grasp the relative class affinities well. For an intuitive visualization, we plot the probabilities for a randomly sampled instance whose correct class is y = from CIFAR- 100. The top two rows in Fig. show the results of RNX29-4-46 and RNX29-644 4d under traditional temperature scaling (TS), where the latter really experiences smaller derived variance. Using ATS could enhance the derived variance as shown at the bottom row (the bars are more jagged). Then, we study the change of derived average (DA) and derived variance (DV) as temperature increases. Given a r, we obtain the DA and DV for all training samples via softmax and then calculate the average. For ATS, we use 71 = 1.257 and T2 = 0.757. The curves are shown in Fig. 7 According to Proposition 4.3 E(q) increases increases, which enhances the efficacy of smooth regularization.
8

---

Table 3: Comparisons with SOTA methods on TinylmageNet, CUB, and Stanford Dogs. WRN50-2 and RNX101-32-88 are teachers. AlexNet, SFV2, and MV2 are students.
TinyImageNet
CUB
Stanford Dogs
Avg
Teacher
WRN50-2
(66.28)
RNX101-32-8d
(79.50)
RNX101-32-88 8d (73.98)
Student
ANet
SFV2
MV2
ANet
SFV2
MV2
ANet
SFV2
MV2
NoKD
34.62
45.79
52.03
55.66
71.24
74.49
50.20
68.72
68.67
57.94
ST-KD
36.16
49.59
52.93
56.39
72.15
76.80
51.95
69.92
72.06
59.77
KD
35.83
48.48
52.33
55.10
71.89
76.45
50.22
68.48
71.25
58.89
ESKD
34.97
48.34
52.15
55.64
72.15
76.87
50.39
69.02
71.56
59.01
TAKD
36.20
48.71
52.44
54.82
71.53
76.25
50.36
68.94
70.61
58.87
SCKD
36.16
48.76
51.83
56.78
71.99
75.13
51.78
68.80
70.13
59.04
KD+ATS
37.42
50.03
54.11
58.32
73.15
77.83
52.96
70.92
73.16
60.88
Ens
39.37
50.69
56.40
59.84
74,43
77.47
54.04
71.65
72.53
61.82
ResKD
38.66
51.93
57.32
62.60
75.29
76.27
54.68
70.73
72.85
62.26
KD+ATS+EnS
40.42
52.14
58.47
62.00
76.26
78.97
55.69
73.22
74.67
63.54
Notably, this term changes nearly the same between teachers with various capacities. However, the derived variance v(() differs a lot. Empirically, v(q) first increases and then decreases, and the maximal of the larger teacher's DV is smaller, which verifies the Corollary 4.7 Because derived variance corresponds to the efficacy of class discriminability, this shows why larger teachers can't t teach well. Using the proposed ATS could enhance the derived variance, which equivalently improves the efficacy of class discriminability in KD. We conclude that traditional temperature scaling leads to distillation labels with less discriminative information among wrong class probabilities; our proposed ATS could enhance the discrimination among wrong classes and benefit the distillation process.
5.2 Performances
ATS makes larger teachers teach well again. Previous studies find that more accurate teach- ers can't necessarily teach well 6 31]. As shown in Fig. 9 although we tune temperatures in {1.0, 2.0, 4.0, 8.0, 12.0, 16.0), larger teachers still teach worse under traditional temperature scaling (the solid curves). However, using ATS (the dashed curves) could make larger teachers teach well or better again. The details are in Appendix C.4
ATS surpasses previous methods with advanced techniques. We compare with SOTA methods and list the results on CIFAR-100 TinyImage CUB, and Dogs in Tab. 2 and Tab. 3 The sota compared methods include ESKD (6). TAKD [31], SCKD (51), and ResKD 10 24
NoKD trains
students without the teacher's s supervision. ST-KD trains students under the guidance of a smaller teacher. KD trains students under the guidance of the larger teacher. More details of compared methods are in Appendix C.3 The last column of these tables shows the average performance of corresponding rows. The larger teacher could slightly improve the students' performances via traditional KD, i.e., 68.24% + 69.21% and 57.94 + 58.89%. However, using smaller teacher (the row of "ST-KD"" could obtain about 2% improvement on average, i.e., 68.244...... 28.2%%%--- - 70.32% and 37.94% + 59.77%. This again verifies that larger teachers really teach worse on various datasets. Taking advantage of the early-stopper teacher (ESKD), the teacher assistant (TAKD), and the student customized teacher (SCKD) only improves the larger teacher slightly. e.g., 69.21% 6 69.86% and 58.89% 59.04%. These results do not even surpass the small teacher. ResKD improves the students' performances a lot via introducing the residual student and taking the two students' ensemble, which surpasses the ensemble performances of two separately trained students under different initializations (the "Ens" row). For fair comparison, we also test the performances of our methods via repeating the "KD+ATS" two times and making predictions via the ensemble. The results are almost 1% higher than ResKD. Results on CIFAR-I0 10 and speech data are in Appendix D
Ablation Studies One might argue that the validity of ATS due to the tuning from a larger hyper- parameter space. We vary from 1.0 to 6.0, T2 from 1.0 to 5.0, and record the performances in better, and especially, we recommend the setting of 72 2. -1). Although we introduce one more hyper-parameter, ATS is simple to implement and surpasses SOTA methods that take advanced techniques. We achieve the goal of only utilizing simple operations make larger teachers teach well again. Then, study the trade off of the KD
Fig. 10 Obviously, setting Ti > T2 could

---

CEALE BLITAAA
WRN28 6-2
KD
A 20
-4
1=03
3w0
3-07
A.09
Figure 10: Ablation studies on asymmetric Figure Ablation studies on the weighting of KD loss and CE loss on CIFAR-100 (A in Eq.
temperatures on CIFAR-100 and TinylmageNet (T1, T2 5q.5)
loss and CE loss under different X. The results of "WRN28-8 - MV2" on CIFAR-100 (Fig. 11 verifies that ATS could improve the performances under various X. We also compare ATS with other types of KD under various scenes, and the results are in Appendix DD3
Conclusion
We study the miraculous phenomenon in KD that more accurate model doesn't necessarily teach better. The proposed KD decomposition attributes the success of better teacher to three factors, including correct guidance, smooth regularization, and class discriminability. Through theoretical analysis, over-confident teachers could not release their potential abilities of the class discriminability under traditional temperature scaling. As a simple yet effective solution, we propose Asymmetric Temperature Scaling (ATS) to enhance the derived variance of larger teachers, making their distilla- tion labels more discriminative when teaching students. Extensive experimental results verify the superiorities of our proposed methods.
7 Broader Impact
We focus on the variance wrong class probabilities to analyze why larger teachers cannot teach well and hope that research could bring a perspective to the KD field. Our work has potential negative societal impacts.
Acknowledgements
This work is partially supported by National Natural Science Foundation of China (Grant No. 61921006, 41901270), and Natural Science Foundation of Jiangsu Province (Grant No. BK20190296). Thanks Huawei Noah's Ark Lab NetMIND Research Team. We gratefully acknowledge the support of Mindspore used for this reasearch. Professor De-Chuan Zhan is the corresponding author. A
10

---

References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In IEEE Conference on Computer
Vision and Pattern Recognition, pages 9163-9171, 2019.
[2] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geof- frey E. Hinton. Large scale distributed neural network training through online distillation. In The 6th International Conference on Learning Representations, 2018.
[3] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizz Model compression. In Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541, 2006.
[4] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In JEEE Conference on Computer Vision and Pattern Recognition, pages 5008-5017, 2021.
[5] Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12922-12932, 2020.
[6] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In IEEE/CVF International Conference Computer Vision, pages 4793-4801, 2019.
[7] Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. In The 9th International Conference on Learning Representations,
2021.
[8] Ronald Fagin, Ravi Kumar, and D. Sivakumar. Comparing top k lists. SIAM Journal on Discrete Mathematics, 17(1):134-160, 2003.
[9] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages (602-1611, 2018.
[10] Mengya Gao, Yujun Wang, and Liang Wan. Residual error based knowledge distillation. Neurocomputing, 433:154-166 2021.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[12] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In The Thirty-Thirl AAAI Conference on Artificial Intelligence, pages 3779-3787, 2019.
[13] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.0253: 2015.
[14] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
[15] Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation.
In The 9th International Conference on Learning Representations, 2021.
[16] Zhen Huang, Xu Shen, Jun Xing. Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and exploration framework. In IEEE Conference on Computer Vision and Pullern Recognition,
Hua. Revisiting knowledge distillation: An inheritance and
pages 3579-3588, 2021.
[17] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. CRRR, abs/ 177.701215 2017.
[18] Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Advances in Neural Information Processing Systems 33, 2020.
[19] Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Fei fei Li. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, CVPR (2011).
[20] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6567-6576, 2021.
[21] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2012.
11

---

[22] Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25,
pages 1106-1144 14, 2012.
[23] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, pages 2830-2839, 2018.
[24] Xuewei Li, Songyuan L., Bourahla Omar, Fei Wu, and Xi Li. Reskd: Residual-guides knowledge distillation. JEEE Transactions on Image Processing, 30:4735-4746, 2021.
[25] Ruofan Liang. Tianlin Li, Longfei Li, Jing Wang. and Quanshi Zhang. Knowledge consistency between neural networks and beyond. In The 8th International Conference Learning Representations, 2020.
[26] Junjie Liu, Dongchao Wen, Hongxing Gao, Wei Tao, Tse-Wei Chen, Kinya Osa, and Masami Kato. Knowledge representing: Efficient, sparse representation of prior knowledge for knowl- edge distillation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops,
pages 638-646, 2019.
[27] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu,, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7096-71 7104, 2019.
[28] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. In The 4th International Conference on Learning Representations, 2016.
[29] Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Computer Vision ECCV 2018- 15th European Conference, pages 122-138, 2018.
[30] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Proceedings of the 38th International Conference on Machine Learning, pages 7632-7642, 2021.
[31] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, pages 5191-5198, 2020.
[32] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, pages 4696-4705, 2019.
[33] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Computer Vision ECCV 2018 15th European Conference, volume
11215, pages 283-299, 2018.
[34] Baoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou, Yichao Wu, Jiaheng Liu, Zhaoning Zhang, and Yu Liu. Correlation congruence for knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, pages 5006-5015, 2019.
[35] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning, pages 5142-5151 2019.
[36] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitncts: Hints for thin deep nets. In The 3rd International Conference on Learning Representations, 2015.
[37] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer
Vision and Pattern Recognition, pages 4510-4520, 2018.
[38] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Ting Cheng, and Marios Savvides. label smoothing truly incompatible with knowledge distillation: An empirical study. In The 9th International Conference on Learning Representations, 2021.
[39] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale
image recognition. In 3rd International Conference on Learning Representations, 2015.
[40] Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR, abs/2002.03532. 2020.
[41] Raphael Tang and Jimmy Lin. Deep residual learning for small-footprint keyword spotting. In IFFE International Conference on Acoustics, Speech and Signal Processing, pages 5484-5488. 2018.
12

---

[42] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: consistency targets improve semi-supervised deep learning results. In The 5th International Conference on Learning Representations, 2017.
[43] Amirhossein Tavanaci. Embedded encoder-decoder in convolutional networks towards explain- able AI. CoRR, abs/2007. 067122, 2020.
[44] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In The 8th International Conference on Learning Representations, 2020.
[45] Frederick Tung and Greg Mori.
knowledge distillation. In 2019 IEEE/CVF International Conference Computer Vision, pages 1365-1374, 2019.
[46] Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Özlem Aslan, Shengjie Wang. Abdelrahman Mohamed, Matthai Philiposse, Matthew Richardson, and Rich Caruana. Do deep convolutional nets really need to be deep and convolutional? In The 5th International Conference on Learning Representations, 2017.
[47] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011
Dataset. Technical report, 2011.
[48] Pete Warden. Speech commands: A dataset for umited-vocabulaat speech recognition. CoRR, 2018.
[49] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5987-5995, 2017.
[50] Han-Jia Ye, Su Lu, and De-Chuan Zhan. Distilling cross-task knowledge via relationship matching. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12393-12402, 2020.
[51] Yi Wang Yichen Zhu. Student customized knowledge distillation: Bridging the gap between student and teacher. In IEEE/CVF International Conference on Computer Vision, 2021. [52] Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 7130-7138, 2017
[53] Kaichao You. Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In Proceedings of the 38th International Conference on Machine Learning, pages 12133-12143 2021.
[54] Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Armau Ramisa. Learning metrics from teachers: Compact networks for image embedding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2907-2916, 16. 2019.
[55] Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 3902-3910, 2020.
[56] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016
[57] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In The 5th International Conference on Learning Representations, 2017.
[58] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference on Computer Vision, pages 3712-3721. 2019. [59] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE Conference on Computer Vision
and Pattern Recognition, pages 6848-6856 2018
[60] Yundong Zhang, Naveen Suda, Liangzhen Lai, and Vikas Chandra. Hello edge: Keyword spotting on microcomtrollers. CoRR, abs/1711.07 128, 2017.
[61] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In 2018 IEFE Conference on Computer Vision and Pattern Recognition, pages 4320-4328, 2018. [62] Zhilu Zhang and Mert R. Sabuncu. Self-distillation as instance-specific label smoothing. In Advances in Neural Information Processing Systems 33, 2020.
[63] Helong Zhou, Lianghhnn Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In The 9th International Conference on Learning Representations, 2021.
13


Reviews:
[{"review_id": "fq1VIL4_Fk_", "reviewer": "Reviewer_iTrh", "summary": "This paper proposes to use asymmetric temperature scaling to modify the correct class / wrong class logits of teacher networks, to achieve a more efficient distillation. The logic behind ATS is that logits from complex teachers are typically over-confident and less informative, thus by applying high temperature on correct class and low temperature on wrong classes, the logits could become both confident and discriminative. Both theoretical analysis and experimental results are provided to support the effectiveness of the proposed ATS strategy. ", "questions": "As listed in the strength and weakness part, my question is primarily regarding if the improvement on smooth regularization and class discrimination comes at a cost of reducing correct guidance, as well as some improvement on the experimental part. ", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe idea of ATS is novel. I also like the presentation of this paper. Easy & clear to follow. \n\nWeakness:\n\nTheoretical part is a bit confusing. Reading the decomposition of KD losses, while the proposed ATS improves the latter two terms, wouldn't it hurt the first term, the correct guidance term? How would the trade-off look like? \n\nI also believe that the experimental part could be much improved. The authors did provide a lot of details and analysis on their results already, but I would like to see the comparisons to more SOTA methods on KD, e.g.,\n\n1) https://arxiv.org/pdf/2107.00181.pdf\n2) https://arxiv.org/pdf/2104.09044.pdf\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TOznOIDqAi", "reviewer": "Reviewer_HVhD", "summary": "This paper revisits the phenomenon in knowledge distillation that a teacher model with better performance doesn’t necessarily transfer information better to the student model. By revealing that the scaling temperature value has a different influence on three terms, correct guidance, smooth regularization and class discriminability, the authors propose to set asymmetric temperature value for the distillation loss function, particularly encouraging more derived variance to improve the distillation better. Experimental results on commonly used vision models have been shown to prove the effectiveness of the proposed model.", "questions": "1  When repeating the “KD+ATS” two times and making predictions via ensemble, do the tau1 and tau2 keep the same or not?\n\n2 Compared with ResKD, the proposed method generally performs worse when student model is SFV1 and the only better one is 75.47 over 75.43 which is very insignificant. Can the authors provide any explanation for this?\n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1 The paper studies a commonly ignored problem in knowledge distillation, with a relatively simple yet effective solution to solve it.\n\n2 The figures to illustrate the relation between different terms and concept are clearly displayed.\n\n\nWeaknees\n\n1 Some introduction about the compared methods should be put in the main content rather than the supplementary materials. It is difficult to relate the shorname in Table 2 to corresponding methods and tell the significance of the contribution for the proposed method. \n\n2 For marginal improvements, there lacks of standard deviation to tell the significance. For example, the result in Table2 for RNX29, the last two rows under SFC1 column (75.47 vs 75.43)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IfsfxRE5Nx", "reviewer": "Reviewer_RAjp", "summary": "Working in the teacher-student setting, the authors propose and study a possible method of improving the efficacy of the teacher network by manipulating the temperature parameter of its probability density over labels. They are motivated by the practical need to distill the ‘knowledge’ of the teacher into smaller networks that can be run locally on small devices.", "questions": "There is a serious lack of clarity in the exposition. For instance, the core technical motivation of the paper seems to be that larger/more complex/more accurate networks ought to be better teachers (than smaller ones performing the same task), but this is not at all an obvious point. Why do the authors have this expectation?\n\nIn Section 3, the teacher-student framework within which the authors are working is not carefully set up. The reader is able to infer that the input to the student network is a probability density over labels generated by the teacher network, but this is not explicitly stated, and no further details of the training problem to be solved by the student (or the details of the teacher network) are provided. The authors might consider rewriting this section to be more informative. Once the mathematical framework is established, it will be easier to demonstrate what the problem is with the output of a large teacher network that prevents it from teaching as effectively as a smaller teacher network, which is what the authors try to do in Section 4. It will also make it easier to precisely characterize the relevance of the temperature parameter to this problem.\n\nIn Section 4.1, a decomposition of the loss function of the teacher is made. Per my understanding, this decomposition is then used to motivate ATS. Unfortunately, the decomposition itself is ill-defined - the quantity $E_{j\\neq y}$ is not defined at all. This makes it difficult to understand and interpret. If the authors could specify the meaning of this quantity, that would be helpful.\n\nThe central quantities the authors ultimately work with are $\\mathbb{E}(p)$ and $\\mathbb{V}(p)$. (These are introduced in Section 4.2.) However, $p$ is a normalized probability density over labels, and not a random variable. Therefore it is not meaningful to talk about its ‘mean’ and ‘variance’. I’d appreciate it if the authors could clarify exactly the meaning of these quantities. There is a similar problem for the quantities $\\mathbb{E}(q)$ and $\\mathbb{V}(q)$, where $q$ is a vector of probabilities constructed from $p$ by simply removing the entry corresponding to the correct label. For this reason, I am unable to understand the content of Section 4.2 and therefore much of the rest of the paper.\n\nIf the authors are able to edit the paper for clarity, addressing the various points I have mentioned, it will make it easier to try to assess their contribution.\n\nLastly, I wonder how the idea of asymmetric temperatures affects the input-output map of the teacher network? In particular, if the probability of the correct label corresponding to some input example has one associated 'temperature', and all the other labels have another, then the equilibrium distribution of the resulting map between inputs and label probabilities is no longer the Boltzmann distribution. I imagine this has profound consequences for the map between (teacher) inputs and (teacher) labels. I'd be grateful if the authors could please comment on this point.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper treats an interesting and relevant problem, namely, how to increase the information content of the output of the teacher network. They propose a novel method they call asymmetric temperature scaling (ATS) as one potential solution to this problem. However, I think there are two kinds of issues that significantly weaken the work to the point where I really cannot recommend acceptance: (1) the mathematical basis of the proposed method is ill-defined and shaky, and (2) there is a marked lack of clarity in the writing. Core mathematical details on which the analysis of the paper rests are not properly laid out, making it difficult to assess the contribution of the paper. I suspect the issues are structural and cannot be fixed over the course of a review period. While I am happy to be convinced otherwise, for now I am afraid I must recommend that the paper be rejected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept