Knowledge Distillation Based on Transformed Teacher Matching
Authors: Kaixiang Zheng
EN-HUI YANG
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge distillation
temperature scaling
regularization
image classification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: The paper shows that dropping temperature scaling on the student side will greatly improve the performance of KD both theoretically and experimentally, and a novel distillation approach is proposed based on that.

Abstract: As a technique to bridge logit matching and probability distribution matching, temperature scaling plays a pivotal role in knowledge distillation (KD).  Conventionally, temperature scaling is applied to both teacher's logits and student's logits in KD. Motivated by some recent works, in this paper, we drop instead temperature scaling on the student side, and systematically study the resulting variant of KD, dubbed transformed teacher matching (TTM). By reinterpreting temperature scaling as a power transform of probability distribution, we show that in comparison with the original KD, TTM has an inherent Rényi entropy term in its objective function, which serves as an extra regularization term.  Extensive experiment results demonstrate that thanks to this inherent regularization, TTM leads to trained students with better generalization than the original KD. To further enhance student's capability to match teacher's power transformed probability distribution, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). It is shown, by comprehensive experiments, that although WTTM is simple, it is effective, improves upon TTM, and achieves state-of-the-art accuracy performance. Our source code is available at https://github.com/zkxufo/TTM.

Published as a conference paper at ICLR 2024
KNOWLEDGE DISTILLATION BASED ON TRANS- FORMED TEACHER MATCHING
Kaixiang Zheng & En-Hui Yang
Department of Electrical and Computer Engineering, University of Waterloo (k562heng, ehyang ( @wwaterloo.co ca
ABSTRACT
As technique bridge logit matching and probability distribution matching, temperature scaling plays a pivotal role in knowledge distillation (KD). Conven- tionally, temperature scaling is applied both teacher's logits and student's logits in KD. Motivated by some recent works, in this paper, we drop instead tempera- ture scaling on the student side, and systematically study the resulting variant of KD, dubbed transformed teacher matching (TTM). By reinterpreting temperature scaling as a power transform of probability distribution, we show that in compari- son with the original KD, TTM has an inherent Rényi entropy term in its objective function, which serves as an extra regularization term. Extensive experiment re- sults demonstrate that thanks to this inherent regularization, TTM leads to trained students with better generalization than the original KD. To further enhance stu- dent's capability to match teacher's power transformed probability distribution, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). It is shown, by comprehen- sive experiments, that although WTTM is simple, it is effective, improves upon TTM, and achieves offte-of-the-art accuracy performance. Our source code is available at httpp:://ithuue com/ zkxufo/TT3
INTRODUCTION
Knowledge distillation (KD) has achieved great success and drawn a lot of attention ever since was proposed. The original form of KD was proposed by Buciluu et al. (2006), where small model (student) was trained match the logits of large model (teacher). Later, a generalized version now known as KD was proposed by Hiintnn et al. (2055). where the small student model was trained to match the class probability distribution of the large teacher model. Compared to the student model trained with standard empirical risk minimization (ERM). the student model trained via KD has better performance in terms of accuracy, to the extent that this light-weight KD-trained student model is able take the place of some larger and more complex models with little performance degradation, achieving the goal of model compression.
In the literature, KD is generally formulated as minimizing the following loss
LKD = - ( 4) + D(rrler)))
(I)
where Log = H (31, q) is the cross entropy loss between the one-hot probability distribution corre- sponding to label and the student output probability distribution q, which is the canonical loss of ERM. D(ptrllqr) is the Kullback-Leibe bility distribution p of the teacher and the temperature scaled output probability distribution ạr of the student, T is the temperature of distillation, and is balancing weight. Note that pf = (u/TT) and Φr oz/TT), given logits of the teacher and logits of the student, where denotes the soft- max function.
divergence between the temperature scaled output proba-
The use of the temperature T above is a pivotal characteristic of KD. On one hand, it provides way to build a bridge between class probability distribution matching and logits matching. Indeed, it was shown in Hinton et al. (2015) that T goes to co, KD is equivalent to its matiii- predecessor. On the other hand, also distinguishes KD from the logits logits- matching approach, since in practice, empirically optimal values of the temperature T are often quite modest. Beyond these, there

---

Published as a conference paper at ICLR 2024
is little understanding about the role of the temperature T and in general why KD in its formulation helps the student learns better. In particular, the following questions naturally arise:
Q1 Why does the temperature T have to be applied to both the teacher and student?
Q2 Would be better off to apply the temperature T to the teacher only, but not to the student?
So far, answers to the above questions remain elusive at the best.
The purpose of this paper is to address the above questions. First, we demonstrate both theoretically and experimentally that the answer to the question Q2 above is affirmative, and it is better off to drop the temperature T entirely on the student side-the resulting variant of KD is referred to as transformed teacher matching (TTM) and formulated as minimizing the following objective:
CTTM [1. q) (D(r2ll4)
where 3 is a balancing weight. Specifically, we show that (1) temperature scaling of logits is equiv- alent to power transform of probability distribution, and (2) in comparison with KD, TTM has an inherent Rényi entropy term in its objective function 2 It is this inherent Rényi entropy that serves an extra regularization term and hence improves upon KD. This theoretic analysis is further confirmed by extensive experiment results. It is shown by extensive experiments that thanks to this inherent regularization, TTM leads to trained students with better generalization than KD. Second, to further enhance student's 's capability to match teacher's 's power transformed probability distribu- tion, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). WTTM is simple and has almost the same computational complexity as KD. And yet is very effective; it is shown, by comprehensive experiments, that is significantly better than KD in terms of accuracy, improves upon TTM, and achieves state-of-the-an accuracy performance. For example, WTTM can reach 72. 19% classification accuracy on ImageNet for ResNet-18 18 distilled from ResNet-34, outperforming most highly complex feature-based distilla- tion methods.
With the temperature T dropped entirely the student side, TTM and WTTM, along with the sta- tistical perspective of KD (Menon et al. 2021) and the newly established upper bound on error rate in term of the cross entropy H Hp 9) between the true, but often unknown conditional probabil- ity distribution p of label given an input sample æ and the output probability distribution of a model in response the input x. Yang al. (2023a) offer new explanation of why KD helps. First, the purpose of the teacher in KD is to provide a proper estimate for the unknown true conditional probability distribution Pt. which is a linear combination of the one-hot vector corresponding to the label and the power transformed teacher's 's probability distribution prr. Second, the role of the temperature T on the teacher side is to improve this estimate. Third, replacing p'c by its estimate from the transformed teacher, the learning process in KD is to simply minimize the cross entropy upper bound on error rate, which improves upon the standard deep learning process where pz in the cross entropy upper bound is rudimentarily approximated by the one-hot vector corresponding to the label y.
BACKGROUND AND RELATED WORK
2.1 CONFIDENCE PENALTY
In a multi-class classification setting, an output of a neural network in response to an input sample is a probability vector or distribution g with K entries, where K is the number of all possible classes, and the class with the highest probability is the prediction made by the neural network for this particular sample. Conventionally. prediction is said to be confident if the corresponding g concentrates most of its probability mass on the predicted class. [Szegedy et al. (2016) points out that a model is too confident about its predictions, then it tends to suffer from overfitting To avoid overfitting and improve generalization, Pereyra et al. (017)) proposed to penalize confident predictions. Since confident prediction generally corresponds q with low entropy, they enforced confidence penalty (CP) by introducing negative entropy regularizer into the objective function of the learning process, which formulated as
(3) where 77 controls the strength of the confidence penalty. Thanks to the entropy regularization, the learned model is encouraged to output smoother distributions with larger entropy, leading to less confident predictions, and most importantly, better generalization.
Lep = H H g) - (4)
2

---

Published as a conference paper at ICLR 2024
2.2 RÉNYI ENTROPY
Rényi entropy (Rényi, 1961) a generalized version of Shannon entropy, which has been success- fully applied in many machine learning topics, such as differential privacy (Mironov 2017). under-
standing neural networks Yu et al. 2020), and representation distillation (Miles et al. 202T). Given
discrete random variable X with alphabet A {I1, 22, ..,,,........................... Fn and corresponding probabilities p: for i 1,2,...,,ee n, its Rényi entropy is defined as
1
12 log
(x(X)=
P.
(4)
1
where is called the order of Rényi entropy. The limit of Rényi entropy when + 1 is the well-known Shannon entropy.
2.3 LABEL SMOOTHING PERSPECTIVE TOWARDS KD
In the literature, different perspectives have been developed to understand KD. One of them is the label smoothing (LS) perspective advocated by Yuan et al. (2020) and Zhang & Sabuncu (2020).
LS (Szegedy et al. 2016) is technique encourage model to make less confident predictions by minimizing the following objective function in the learning process
(5) where is uniform distribution over all K possible classes, and controls the strength of the smoothing effect. The model trained with LS tends to have significantly less contident predictions and output probability distributions with larger Shannon entropy compared to its counterpart in the case of ERM (visualized in A.
LLS = (1 < 9) - cH g)
If replace with the teacher output p' in 5 then we have LLS (1 c). 9) + (p', (p'.g), which is equivalent to CDD with T = 1, since the entropy H(x') does not depends on the student. Therefore, when T = 1, KD can indeed be regarded sample-adaptive I.S. However, when 7 1, such a perspective no longer holds since temperature scaling is also applied to the student model. This is confirmed by the empirical analysis shown in A.I Although KD with T 1 is able to increase the Shannon entropy of output probability distribution & compared to ERM, KD with 7 4 actually leads to decreased Shannon entropy compared to ERM, showing an opposite effect of LS.
The sample-adaptive LS perspective was also advocated in self-distillation Zhang & Sabuncu (2020), where the temperature T was dropped for convenience on the student side. However, no systematic treatment was provided to justify the drop-out of the temperature T for the student side. In fact, in terms of prediction accuracy, mixed results were demonstrated: dropping out the temperature for the student can either decrease or increase the accuracy.
2.4 STATISTICAL PERSPECTIVE AND CROSS ENTROPY UPPER BOUND
Another perspective to understand KD is the statistical perspective advocated by Menon et al. (2021) A key observation therein is that the Bayes-distilled risk has smaller variance than the standard empirical risk, which is actually the direct consequence of the law of total probability for variance (Ross 2 Since the Bayes class-probability distribution over the labels, i.c., the conditional probability distribution pr [P()])]] of label given an input sample x, is unknown in practice, the role of the teacher in KD was believed to use output probability distribution p' or temperature scaled output probability distribution pf to estimate P for the student. This, in turn, offers some explanation of why improving teacher accuracy can sometimes harm distillation performance, since improving teacher accuracy and providing better estimates for P are two different tasks. In this perspective, the temperature T is also dropped for the student. Again, no justification was provided for dropping T' on the student side. In addition, the question of why minimizing the Bayes-distilled risk or teacher-distilled risk could improve the student's accuracy performance was not answered either.
Recently, it was shown in Yang et al. 2023a) that for any classification neural network, its error rate is upper bounded by E, H pp- 9) Thus, to reduce its error rate, the neural network be trained by minimizing E, (p 9) Since the true conditional distribution pt is generally unavailable in practice, KD with the temperature T' dropped for the student can be essentially regarded one way
3

---

Published as a conference paper at ICLR 2024
to solve approximately the problem of minimizing E, (p 9)}, where pa is first approximated by a linear combination of the one-hot probability distribution corresponding to label y and the temper- ature scaled output probability distribution pir of the teacher. This perspective, when applied to KD, does provide justifications for dropping the temperature T entirely on the student side and also for minimizing the Bayes-distilled risk or teacher-distilled risk. Of course, KD with the temperature T dropped for the student may not be necessarily an effective way to minimize Ex [H(H( (p2, q) Other recent related works are reviewed in Appendix A.7
In contrast, in this paper, we show more directly that it is better off to drop entirely the temperature on the student side in KD by comparing TTM with KD both theoretically and experimentally.
TRANSFORMED TEACHER MATCHING
In this section, compare TTM with KD theoretically by showing that TTM is equivalent KD plus Rényi entropy regularization. To this end, we first come with general concept of power transform of output distributions. Then, we show the equivalence between temperature scaling and power transform. Based on this, simple derivation is provided to decompose TTM into KD plus Rényi entropy regularizer. In view of CP, it's clear that TTM can lead better generalization than KD because of the penalty over confident output distributions.
3.1 POWER TRANSFORM OF PROBABILITY DISTRIBUTIONS
In KD, model output distributions are transformed by temperature scaling to improve their smooth- ness. However, such transform is not unique. There are many other transforms which can smooth out peaked probability distributions as well. Below we will introduce a generalized transform.
Consider point-wise mapping f [0, 1] + [0, 1]. For any probability distribution = -Pi..... .Pk). we can apply f to each component of p to define generalized transform p p. where [pi.....Pk: .K.]. and
[(p)) VISiSk. ΣØ=1 f(,))
D:
(6)
In this above, ΣØ=1 (P;) is used to normalize the vector [ back to a probability sim- plex. With this generalized framework, any specific transform can be described by its associated mapping f. Among all possible mappings f. the most interesting one to us is the power function with exponent 7. If f selected to be the power function with exponent y, the resulting probabil- ity distribution transform p - á is referred to as the power transform of probability distribution. Accordingly, the power transformed distribution given by
K
p 7 ΣEEP
P
Next, we will show that power transform is equivalent temperature scaling. Indeed, suppose that p is the softmax of logits [4,42, Ik]:
di p EK eb Vigisk.
(8)
Then
et Em
)r
Em clm (ml)))) )" ct7l
pr
2 Eecle -E, enly
P:
Σ,
cl
E,er
Li' e
Thus p is the softmax of the scaled logits 72,, Nxx) with temperature T 1/7.
3.2 FROM KD TO TTM
Based on the equivalence between power transform and temperature scaling, we can now reveal the connection between KD and TTM.
4

---

Published as a conference paper at ICLR 2024
Let y 1/T. Go back to 1 and 2). In view of ( we have
pf = p' and qr q.
(10)
Then we can decompose ((o4) as follows:
D(p (4r) D(p 100
Σ 1², log p'z F
pl log q. H(²))
Σn i. log q." E,9²
1T²)))
(11)
Σ p. log q: + log Σ 93 (T())
TH( 9) (1-7) HH - (
(12)
>D()')l)) (1-)))H((1) 11 (1-7)H(p)
(13)
>D(r 114) (1-111 -1)))(())( - (1- H)
(14)
where 11
follows the power transform 7 (9) in (12) the Rényi entropy of q of order y, and
is due to (1O). Rearranging (4). we get
D(p = (D(pill (T 1)H (g) (T 1 1))H(r))
(15)
Plugging
15 into
yields
LTTM-
H(, 4) + STD(P>) 3(T 1)H + (1) B(T (p')
H 2) + (p - -8 - +)H+(0)
(16)
I () +x (pr XT 1)H4 (0)
(17)
[KD XT 1)H (q)
(18)
X
whenever / is selected to be
A
8
T.
(19)
where (6) is due to the fact that the Shannon entropy H (p+) does not depend on the student model, is attributable to (1]
follows (99.. and 18
Thus we have shown that TTM can indeed be decomposed into KD plus a Rényi entropy regularizer. Since Rényi entropy is generalized version of Shannon entropy, plays a role in TTM similar to that of Shannon entropy in CP. With this, we have reasons to believe that can lead to better generalization, which is indeed confirmed later by extensive experiments in Section
is also instructive to compare TTM and KD from the perspective of their respective gradients. The gradients of the distillation component in LTTM with respect to the logits are:
@D(ph Oz.
0H (pr. 9; 0z
2/T
(20)
EjK, (pg) 'T
where and gi are the ith logit and ith class probability of the student model, respectively. In comparison, the corresponding gradients for KD
@D lar) (H vr) 0z,
- (a -
-.)==
q
1/T
(
(21)
0z
>K 1/7
Σ (z))
From Eq. (20). we see that the gradient descent learning process would push g to move towards the power transformed teacher probability distribution, thus encouraging the student to behave like the power transformed teacher, from which the name TTM (transformed teacher matching) is coined. Since the power transformed teacher distribution ア with T' > is smoother, the student trained by TTM will output distribution q with similar smoothness, leading to low confidence and high entropy. On the other hand, in Eq. (2T), it is the transformed student distribution qr that is pushed towards the transformed teacher distribution p Even when fr has similar smoothness as ptr. the original student distribution g can still be quite peaked, thus having high confidence and low entropy.
5

---

Published as a conference paper at ICLR 2024
4 SAMPLE-ADAPTIV MATCHING TO THE TRANSFORMED TEACHER
We can further improve TTM by introducing sample-adaptive weighting coefficient into TTM. This is explored in this section.
In TTM, the soft target we use is linear combination of the one-hot probability distribution cor- responding to y and the power transformed teacher distribution pt., where the same coefficient / is applied to all samples. As discussed in Subsection 2.4 the role of the teacher in KD is to provide pi and use it as an estimate for P Assume this estimate is good. It is reasonable to believe that one-hot target even which p have more intrinsic confusion and is away from the one-hot probability distribution. After all, when pfr is close the corresponding one-hot probability distribution, minimizing H (pr: q) has little difference from minimizing H (y, q), and as result, it's no longer meaningful to do distillation on these types of samples. This motivates us to discriminate among soft targets in TTM based on their smoothness. Concretely, large / should be assigned to smooth p/,, while small 3 should be assigned a peaked prr.
would be better off favor a soft target
for those samples for
To implement the above idea, need quantity to quantify the smoothness of a soft target p4- In view of 7 and the definition of Rényi entropy (4). the following power sum defined for any distribution p and any 0 <
(p(p) = D
comes handy. Given 0 < < 1, we can use the power sum (p) to quantify the smoothness of p, since it is related to both the power transform and Rényi entropy. It is clear that the power sum (p) attains its minimum 1 when p is one-hot and maximum K2-7 when p is uniform. Using (p' to discriminate among different samples, we modify TTM to minimize the following objective function
LWTTM (v, + 3U (D') D(pilla).
(22)
The resulting variant of KD is referred to as weighted TTM (WTTM), Note that other sample- adaptive weights such as (p4r) may also be effective. Nonetheless, systematic study regarding how to select sample-adaptive weights and which one is optimal, is left for future work.
Compared to TTM where the student is trained match all soft targets uniformly, WTTM trains the student to match more closely to smooth soft targets and less closely to peaked soft targets. Thus, students resulting from WTTM would output smoother q than those distilled from TTM, which is further confirmed in the next section by experiments.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
We benchmark TTM and WTTM on two prevailing image classification datasets, namely CIFAR- 100 and ImageNet (Deng et al., 2009).
CIFAR-100 contains 60k 32x32 color images of 100 classes, with 600 images per class, and it's further split into 50k training images and 10k test images. For fair comparison, we adopt the same training strategy and teacher models as CRD (Tian et al.) 2019). Also, following CRD, we gener- ate comprehensive experiment results for 13 teacher-student pairs including both same-architecture distillation and different-architeccure distillation, and the tested model architectures are VGG (Si- monyan & Zisserman, 2014). ResNet (He et al.) (2166), WideResNet (Zagoruyko & Komodakis 2016b), MobileNetV2 (andler et al. 2018), and ShuffleNet (Zhang et 21.1 Ma et al. 2018).
ImageNet is large-scale image dataset consisting of over 1.2 million training images and 50k vali- dation images from 1000 classes. For experiments on ImageNet, we employ torchdistill (Matsubara) 2021) library and follow all the standard settings. The tested model architectures are ResNet and MobileNet (Howard et al. 2017).
Note that we list T and 33 values of all experiments in A.4 for reproducibility.

---

Published as a conference paper at ICLR 2024
5.2 MAIN RESULTS
Results on CIFAR-100.
The pure performances of TTM and WTTM are shown in Table and Table We compare them with feature-based methods FitNet (Romero al. 2014). AT (Zagoruyko &
3
Komodakis 2016al VID (Ahn et al. 2019), RKD (Park et al. 2019). PKT (Passalis & Tefas 2018),
CRD (Tian al., 2019), and logits-based methods such as KD. DIST (Huang al.) 2022) and DKD Zhao et al. 2022). In general. TTM and WTTM provide outstanding performance among
all the compared methods, and WTTM is better than TTM in most cases. Note that TTM always
outperforms KD, confirming our theoretic analysis in Section 3
To further improve the performance, we combine WTTM loss with existing distillation losses respectively, namely CRD and ITRD (Miles et al. 202T), and the resulting performance is shown Table 2 and Table 4 For the combined methods, we directly adopt the optimal hyperparameters that the performance the combined loss is always better than the pure performances of both ingredient losses, meaning that our proposed WTTM loss is orthogonal More importantly, the performance WTTM aided by CRD and ITRD is consistently better than all other methods over all teacher-student pairs, achieving the state-of-the-ar accuracy.
specified in the original papers without tuning (sce|A.3]for details). From the tables,
other losses like CRD and ITRD.
Top-1 accuracy (%) CIFAR-100 of student models trained with various distillation methods, including both seiture-based methods and logits- based methods. Each teacher-studen pair has the same architecture. We highlight the best results in bold, and the second best results with underscores. Note that some results of DIST (for the models excluded in their paper) are produced by our reimplementation. Average over runs.
Table 1:
Teacher Student
WRN-4033
WRN 40-2
resnet56
resnetl 10 resne220
renneti 10 resnet32
restetllu resterdo
vgz13 Vg26
WRN-16-2
WRN 40-1
resne120
Tearher Student
75.61
75.61
72.34
74.3]
74.31
79.42
74.64
73.26
71.98
69.06
69:06
71.14
72.50
70.36
Fxature-haset based
PitNet
73.58
72.24
69.21
68 59
71.06
73.50
71.02
VID
74.11
73.30
70.38
16
72.61
73.09
71.23
RKD
73.35
72.22
69.61
69.25
71.82
71.90
71.48
PKT
74.54
73:45
70.34
70.25
72.61
73.64
72.88
CRD
Logits-baxze
DIST
75.51
74.73
71.75
21.65
73.69
76.31
73.89
DKD
76.24
74,81
71.97
74.11
76.32
74.68
TTM
76.25
74.32
71.83
71.46
73.97
76.17
74.33
76.37
74.58
71.92
71.67
Table 2: Top-1 accuracy (%) on CIFAR-100. Each teacher-student pair has the same architecture. Average over 5 runs (3 runs for ITRD and WTTM+ITRD following the original paper of ITRD).
Toacher Stadent
WRN-40022 WRN-15-7
WRN--302
mosed56
reasti 10
resect 10 meenet
reseetSQuu resenthel
vga13 1924
WRN--0-11
mener30
CRD
75.48
74.14
71.16
71.46
73.48
75.51
73.94
ITRD
75.18
71.99
74.26
76.19
71.67
74.13
WTTM+CRD WITM
72.13
22.16
74.36
77.36
Results on ImageNet. In Table 5 demonstrate the performance of WTTM compared to many competitive distillation methods such as KD, CRD, SRRL (Yang et al. 2020). ReviewKD KD (Chen) al. 202T), ITRD (Miles et al. 22021) DKD (Zhao et al. 2022). DIST (Huang al. 2022). KD++ (Wagg et al. 2023). NKD (Yang et al. 2023b), CTKD (Li et al. 2023c). and KD Zero (Li et al. 2023a). It's shown that WTTM achieves outstanding performance on both teacher-student pairs.
5.3 EXTENSIONS
To provide more comprehensive understanding and deeper insight about TTM and WTTM, we in- clude 4 points of extension this subsection, demonstrating some promising properties WTTM and supporting our methodology with some analysis.

---

Published as a conference paper at ICLR 2024
Table Top-1 accuracy (%) on CIFAR-100. Each teacher-student pair has different architectures. Note that some results of DIST (for the models excluded in their paper) are produced by our reim- plementation. Average over runs.
Teocher Studkni
vgp15 MobilcNaiV2
ResNectu MobileNatV22
ResNecto 1288
ressett2ei Shufl:NNrV11
resnectivd StuMl:NrrV2
WRN-40-2
StuMl:NarVi
Tescher Student
74.64
79,30 34
79. 34
79.42
79.42
75.61
64.6
70.36
71.82
70.5
Feature
FiiNet
64.14
63.16
70.69
73.59
73.54
73.73
AI
59.40
50.58
71.54
71.23
72.75
73.32
VID
63.56
67.57
70.
73, 13.38
73.40
73.61
RKD
64.52
64.43
71.50
72.26
73.21
72.21
74.69
76.05
Logits
KD
67.37
67.35
73.81
74.07
74.45
74.83
DIST
68.50
68.66
74.11
76.34
77.35
76.40
DKD
69.71
70.35
n/a
76.45
77.07
76.70
TIM
08.98
74.87
WTTM
76.55
Table 4: Top-1 accuracy (%) on CIFAR-100. Each teacher-student pair has different architectures. Average over 3 runs.
Studint
MctriiNeeV22
MunmkNNei22
StafiteNetii
StafflcNerV
WITM
69.16
69.59
75.42
WTIM+CRD WTIM-ITRD
10.30
20.86
70.20
71.56
76.00
77.03
77.48
77.44
Table 5: Top-1 accuracy (%) on ImageNet. The adopted teacher models are released by PyTorch Paszke et al. 2019).
Teacher
Student
CRD
SRKL
ReviewKD
ITD
DKD
DIST
KD++
NKD
CTKD
KD-Zero
WTTM
ResNet-34 (73.31)
ResNer-18 18 (69 76)
70.66
71.17
71.73
71.61
71.68
71.70
72.07
71.98
71.96
71.51
72.17
72.19
RosNet-50 (76.16)
MobileNer (68.87)
70.50
71.37
72.49
72.56
72.05
73.34
72.77
72.58
A/a
73,02
73.00
Distill without LoE- In Table 6 we compare the performance of WTTM without Lce to the performance of KD with LCE. We find that even in this unfair setting, WTTM can still outperform KD in most cases. This is of great value in the scenario where the ground-truth labels of the transfer set are not available.
Table 6: Comparison between WTTM without Loe and KD with Loe on CIFAR- 100. Accuracy averaged over 5 runs.
Taahus Studest
WENL-0022
WRN-40-2
receu56
moeeil0
neaci1
roenst32ed
vagl3
WRN-16-2
WRN-40-1
resss220
ccsne120
soso332
reauttie
1258
WTTM CE
75.11
73.16
70.95
70.71
73,21
72.94
74.04
Distill from better teachers. Results in Table 7 show that the student benefit more from better teacher when distilling with WTTM. We observe that as the teacher model grows better, other distillation methods like KD and DIST cannot guarantee consistent improvement side. In contrast, when we apply WTTM, the performance of the student is strictly increasing and consistently better than other distillation methods as the teacher becomes better and better.
the student
Table 7: Performance of ResNet- 18 on ImageNet distilled from different teachers.
Teacher
Student
Teacher
Student
KD
WTTM DIST
ResNes-34 ResNes-50
ResNes- 101
ResNas-18
77.37
76.13
69.76
71.09
71.35
72.08
72.12
72.26
72.34
ResNet: 152
78.31
71.12
72.24
72.39
8

---

Published as a conference paper at ICLR 2024
Regularization effect of TTM and WTTM. Following our methodology, TTM and WTTM able to embed strong regularization into the distillation process, so it's expected that student's 's output probability distributions q resulting from TTM and WTTM should be much smoother than those resulting from KD. To validate this, we track the behavior of the average Shanmon entropy of q for KD, TTM and WTTM respectively during training over 3 teacher-student pairs used in CIFAR-100 experiments, shown in Fig. [ larger entropy than those trained with KD. This is attributable introduced in TTM when we remove the temperature scaling on the student side from KD. Moreover, students trained with WTTM always have slightly larger entropy than those trained with TTM, owing to the sample-adaptive weighting coefficient U+ (p").
Comparatively, students trained with TTM always have significantly
the Rényi entropy regularizer
VTTT
Entrogy
MITTH
Entrogy 1 +
50
300
150
200
250
0
50
100
150
200
250
50
160
150
200
250
Epoch
Epoch
Egach
(a) WRN-40-2-
(b) "ggl3~vvge"
(c)
Figure I: Average H (q) of teacher-student pairs during training. For fair comparison, we use the same temperature T = 4 for KD, TTM and WTTM. The for KD is 0.9, so the / for TTM is 36, computed by Eq. 119 in order to maintain the same ratio between H (yy q) and H Hpbr,9r) KD. As for WTTM, 3 = 36/U, where Ü is the average of Ug (p") over all samples.
WTTM facilitates more accurate teacher matching. A closer look at TTM and WTTM is fa- vorable to shed light on why WTTM generally performs better than TTM. To this end, we track the behavior of the average D(prrllq) for TTM and WTTM during training over the same teacher- student pairs as above, shown in Fig. In order to reflect the behavior of pure distillation, we remove Loe from both WTTM and TTM. It's clear from the plots that WTTM always leads smaller gap between prp and than TTM, demonstrating more accurate transformed teacher matching, which is the reason behind performance improvement.
0.28
NET WAL
DE CN
2-4
KL. Divergence
genegeenc
0.28
2.03
0.04
3.00
KL
02
180
150
2003
250
50
100
350
200
250
Fpach
Fpoch
Epoch
(a) 40-
(b) 1ggg3-11vggg
(c)
Figure 2: Average D ll) of 3 teacher-student pairs during training. For each pair, the same T is adopted in TTM and WTTM.
6 CONCLUSION
The paper systematically studies a variant of KD without temperature scaling on the student side, dubbed TTM. This slight modification gives rise to Rényi entropy regularizer which improves the performance of the standard KD. Furthermore, we propose a sample-adaptive version of TTM, dubbed WTTM, to achieve more significant improvement. Extensive experimental results are pre- sented to show the superiority of TTM and WTTM over other distillation methods on two image classification datasets. With almost the same training cost KD, WTTM demonstrates state-of- the-art performance, better than most crature-based distillation methods with high computational complexity.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN203035-22, and by the Canada Research Chairs Program.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9163-9171, 2019.
Cristian Buciluă, Rich Caruana, and Alexandru Nieulescu-Mizie Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535 535-541, 2006.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge re- view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5-.550000177 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-248222 255. Icve, 2009.
Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 11898-11 1908, 2023.
Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Vanillakd: Re- visit the power of vanilla knowledge distillation from small scale to large scale, arXiv preprint arXiv:2305.1577 15781, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0461, 2017.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. Advances in Neural Information Processing Systems, 1113716--777,,, 2022.
Lujun Li. Self-regulated feature learning via teacher-free feature distillation. In European Confer- ence on Computer Vision pp. 347-363. Springer, 2022.
Lujun Li and Zhe Jin. Shadow knowledge distillation: fer. Advances in Neural Information Processing Systems, 35:635-641 2022.
Bridging offline and online knowledge trans-
Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Yang Ya. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. In Thirth-seventh Conference on Neural Information Pro- cessing Systems, 2023a.
Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp- 17413-17424, 2023b.
Zheng Li, Xiang Li, Lingfeng Yang. Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504-1512, 2023c.
10

---

Published as a conference paper at ICLR 2024
Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching, arXiv preprint arXiv:2305. 2023.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Yoshitomo Matsubara. torchdistill: A modular, configuration-driven framework for knowledge dis- tillation. In International Workshop on Reproducible Research in Pattern Recognition, pp. 24 44. Springer, 2021.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statisti- cal perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Roy Miles, Adrian Lopez Rodriguez, and Krystian Mikolajezyk. Information theoretic representa- tion distillation. arXiv preprint arXiv:2112.00455 2021.
llya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer foundation sympo- sium (CSF), pp. 263-273 275. IEEE, 2017.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976,
2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268 268-284, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 17X1v:1701.06548. 2017.
Alfréd Rényi On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, volume 4, pp. 547-562. University of California Press, 1961.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Sheldon Ross. A First Course in Probability. Pearson Higher Ed, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint 1rXiv:1109.113 2014.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.

---

Published as a conference paper at ICLR 2024
Yuzhu Wang, Lechao Cheng, Manni Duan, Yongheng Wang, Zunlei Feng, and Shu Kong. Im- proving knowledge distillation via regularizing feature norm and direction. arXiv preprint arXiv:2305.17007, 2023.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification, 2023a.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. International Conference on Learning Represen- lations, 2020.
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Practical guidelines for vit feature knowledge distillation. arXiv preprint arXiv:2209.02432 2022.
Zhendong Yang. Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge dis- tillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. arXiv preprint arXiv:2303. 13005, 2023b.
Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and Jose C Principe. Understanding convolu- tional neural networks with information theory: initial exploration. IEEE transactions on neural networks and learning systems, 32(1):435-442 2020.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.08:: 2016a.
Sergey Zagoruyko and Nikos Komodakis. 1rXY::6605.07146. 2016b.
Wide residual networks.
arXiv preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018
Zhilu Zhang and Ment Sabuneu. Self-distillation as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33:2184 092218-2119 2195, 2020.
Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953-1 1962, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 EMPIRICAL ANALYSIS ON THE LS PERSPECTIVE OF KD
In support of our claims in Subsection 2.3 Specifically, we train four resnet20 models on CIFAR-I00 dataset with different objectives and demonstrate their Shannon entropy histograms of the output probability distributions g in Figure 3
carry simple empirical analysis in this section.
From Figures b)) and 3 a). it is clear that the Shannon entropy of q in the case of LS is significantly larger than counterpart the case of ERM, which shows the regularization effect of LS.
In comparison of Figure 3 c) with Figure aa), it is also clear that the Shannon entropy of in the case of KD with T' confirms that KD can indeed be regarded as sample-adaptive LS when T = 1.
also significantly larger than its counterpart the case of ERM, which
However, when 7 > 1, such perspective doesn't hold anymore. To demonstrate this, we also trained resnet20 on CIFAR- 1-00000 dataset with KD setting T 4, corresponding to Figure ().). Comparing digureBdd with Figure we see that the average Shannon entropy in the case of KD with 7 = 4 is even reduced over the ERM case significantly, showing an exactly opposite effect of L.S. This confirms that when T 1, KD can no longer be regarded as sample-adaptive I.S.
Sammp
of
Ertrpyy
Enerpy
Eneropy
Encropy
(a) LCE
(b) LLS
(c) LKD (T
(d) CKD (T = 4)
Figure 3: Entropy histograms for resnet20 trained with LCF, LLS with e = 0.5, LKD with 1. and LKD with T = 4. For fair comparison, the same X 0.9 is adopted in both KD experiments with different temperatures.
A.2 DISCUSSION ON THE GENERALIZED TRANSFORM
In this section, provide more discussion the generalized transform proposed in Subsection 3.1 As mentioned Subsection 3.1 any specific transform can be described by its associated mapping f. For visualization, demonstrate some examples of mapping f in Fig. a). Also, the power function with exponent y E (0, 1) used in TTM and WTTM is visualized in Fig. (b).
0.8
#o)
0.8
Ap) t0
0.5
Rpd moomseog 13
Apd
0.6
Rod -A+..
0.4
inear function
0.4
y=08
identily
y=02
y=01
0.0
08
y=005
0.0
0.2
0.4
0.6
0.0
1.0
Pr
pi
(a)
(b)
Figure 4: (a) Various woint-wise mappings. (b) Power functions with different exponents 7-
The reason why we only consider the power function in the main text is that the resulting power transform is equivalent temperature scaling, which helps to reveal the Rényi entropy regularizer in the subsequent derivation. However, it's worth mentioning that the generalized transform is much more than tool used in derivations.
13

---

Published as a conference paper at ICLR 2024
Currently, the power transform (temperature scaling) to smooth teacher's output distribu- tions in TTM and WTTM, following the convention in standard KD. However, it's possible that some other transforms could lead to better distillation compared to the power transform, Intuitively, mappings f associated to such transforms should satisfy properties:
f (0) = 0 and f(1) = A deterministic prediction shouldn't be modified by the transform.
Non-decreasing. non-decreasing mapping avoids ruining the order information p.
f(p:) > Pt- To improve the smoothness of p. we need mapping above the identity, since it expands the dynamic range of low probability values and compress the dynamic range high probability values. As a result, after the normalization Eq. (6). small probability values will be increased while large probability values will be decreased, achieving the goal smoothing a distribution.
Following these suggested properties, some potential transforms can developed place of power transform, while we leave this topic for future work.
A.3 IMPLEMENTATION TTM AND WTTM
In this section, we provide the pseudo-code for TTM and WTTM in Pytorch-like style, shown in Algorithm |
It's clear that both TTM and WTTM are quite easy to implement.
Algorithm PyTorch-style pseudo-code for TTM and WTTM.
y_s: student output logits
y_t: teacher output logits
r: the exponent for power transform
P_s F.log_softmax (y_s, dim=1)
p_t torch.powtie softmax (y_t, dim=1), r]
U = torch. sum dim=1) $ power sumt
P_t p_t / U.unsqueeze (1) # power transformed teacher
Kl. torch.sum k1
ki__kkkk_iiiiiii dddivvvi____ (p_s, p_t, reduction )onbe)) dim-1)
TTM
ttm loss = torch.ment mean (KL)
WTTM
wttm loss torch. (J+KL)
A.4 HYPERPARAMETERS
We list fine-tuned y and 3 in Tables 8 9 and 10 covering all experiments, where y L/T'. Because we implement the temperature scaling with the equivalent power transform, the tuning is carried out over the exponent y instead of the temperature T.
Table 8: Hyperparameters for same-architecture distillation on CIFAR- -000
Teacher
WRN-48-1
reantt% sema120
reazi110 reme:20
monett 10I resuet32
resnttSae remetted
vga
WRN 16.2
Yaps
TIM WITM
0--0113-- 101
--1.3-- 76
7-03.
2-62,8-
7-0.1. 33
--0.1 8-11,-110 100
0-0.1,0
7-0.13-3
7-03,8-15
7-02,3-2
7-0118
7-0.1,33
7-0.18- -.1.8-2-22
WTTM-CRD
7-0.12-4 --0,8,-6
7-0.1822
7-03.806
0-02,5-14
7-02,8-1
7-02,3-4
7-02.8-4
0-0,5-
--0
--03,--15
1-05,3
7-011,
7-01
0-04
0-05
4-02
A.5 COMBINATION OF DISTILLATION LOSSES
In this section, we clarify how we combine LWTTM with other distillation losses in our experiments. Actually, simply add another distillation component LWTTM with multiplier. The total
14

---

Published as a conference paper at ICLR 2024
Table 9: Hyperparameters for differren-arrchieccture distillation on CIFAR-100.
Teacher Student
vgel3 MobileNetV2
ResNethO MobileNerV2
ResNet50
resnet SbuffiNNerV
resnet32cc SauffieNerV2
WRN-40-2
ShufBcNetV1
TTM
0:2,2,0
7=0.2,3
1=
0 70
0 7=0.1
x=03,8=8
WIIM
7 y= 002,,333
0.2,6,6 =
3=0.,,0 =
7= 14
16
-=03,9=3
WTTM+CRD
1 0.3,
-=03,8=
-=01
7=02,3=02
1=02.3=016
WTTM-ITRD
033,9 0.03
0.2,8 = 0.02
0.1, =1
0.6
0.4,9
0.8
0.1,8 0.2
Table 10: Hyparameters for ImageNet experiments.
Teacher
Student
WTTM
ResNet-34
ResNet-50
ResNet-18
7 0.8, 1.6
ResNet- 101
ResNer-152
ResNet-50
MobileNet
= 0.7, = 3.5
objective is
Liot (v, + 3Ug (p') D(prllq) + pLdist
(23)
where μ balancing weight, and Laist is the additional distillation component, which can be CRD or ITRD in our experiments.
In the case where we combine WTTM with CRD, 1l always set to be 0.8, which the optimal value used in the original paper.
In the case where we combine WTTM with ITRD, a always set be 1. However, ITRD distilla- tion loss itself is a combination of two components shown as follow
Ldist BBorr heorr + Beis Lems
(24)
where Beorr and Bin are two balaneing weights within ITRD distillation loss. our experiments, we always select the optimal Sear and Ami values specified in the original paper. Specifically, Beorr = and B.m = for 3 teacher-student pairs, namely ResNet50 + MobileNetV2, ResNe150 - vgg8 and WRN-40-2 + ShuffleNetV1, while Bcorr 2 and B.i for all the other 10 teacher- student pairs. Note that there is another inherent hyperparameter a; within ITRD, which is selected as 1.01 for same-arch arehterchieeeeure distillation and 1.5 for different-archhe arrchitccturre distillation, following the suggestion in the original paper.
A.6 FUTURE WORK
This work provides multiple directions for our future research:
From Eq. 15 know that the ratio between the distillation term D(p) |lgr) and the regularizer H. (q) in TTM is determined by T. Also, the order of Rényi entropy is bound to be 1/T. However, these constraints are not necessary. In future work, we can directly combine the standard KD with a Rényi entropy regularizer while setting the balancing weight and the order of Rényi entropy as tunable hyperparameters.
Given the generalized transform framework and related discussion in A.2 other transforms can be proposed in place of the power transform (temperature scaling) used in TTM and WTTM.
Systematically analyze the selection of the sample-adaptive weight in WTTM, in order to find the optimal one.
15

---

Published as a conference paper at ICLR 2024
A.7 RELATED WORK
In recent years, a variety of works have been proposed to advance the methodology of KD and its application to related fields. [Huang et al. (2022) proposed correlation-based loss capturing the inter-class and intra-class relations from the teacher explicitly. Yang et al. (2023b) unified KD and self distillation by decomposing and reorganizing the vanilla KD loss into a normalized KD (NKD) loss and proposed a novel self distillation method based on it. Li et al. (2023c) proposed novel distillation method based on dynamic and learnable distillation temperature. HHao et al. (2023) claimed that the power of vanilla KD was underestimated due to small data pitfall, and observed that the performance gap between vanilla KD and other meticulously designed KD variants could be greatly reduced by employing stronger training strategy. Li 2022 proposed novel feature- based self distillation approach, reusing channel- wise and layer-wiss wise features within the student to provide regularization. [iu et al. (2023) presented a two-stage KD method dubbed NORM based on a feature transform module. Li & Jin (2022) proposed a Shadow Knowledge Distillation framework bridge offiin and online distillation in efficient way. Dong al. (2023) presented training- free framework to search for the optimal student architectures given a teacher architecture. Also, following the trend of Automated Machine Learning (AutoML), several recent works (Li et al., 2023a b) focused on automating distiller design using techniques like evolutionary algorithm and Monte Carlo tree search.
A.8 STANDARD DEVIATION FOR RESULTS ON CIFAR-100
Below, we report the standard deviation for results CIFAR-100 dataset in Table 11 and 12
Table 11: Top-1 accuracy (%) CIFAR-100. Each teacher-student pair has the same architecture. Standard deviation is provided (the standard deviation is missing for DKD since it's not available in the literature).
Teacher Student
WRN-10-2
WRN-40-2 WRN-40-1
resnet56 resnet20
resectl 10
resneti resnet32
resnct32x4
vgg13 vgg8
Teacher Student
75.61
75.61
72.34
74.31 69.06
74.31
79.42
74.64
73.26
71.98
69.06
71.14
72.50
70.36
Feature haved
FitNet
73.58 = 0.32
72.24
+: 0.24
69.21
0.36
68.99 = 0.27
71.06
+ 0.13
73.50 0.28
71.02 + 0.31
AT
74.08
0.25
0.10
70.55
72.31
73.44
71.43
VID
74.11 0 0.24
73.30
+ 0.13
70.38 0.14
70.16 = 0.39
72.61
+ 0.28
13.09 =
71.23 + 0.06
RKD
73.35 = 0.09
72.22
+: 0.20
69.61
1.06
69.25 0.05
71.82 + 0,34
7.19
71.48 0.05
PKT
74.54
73.45
0.04
70.25
0.04
72.61
0.16
72.88
0.09
CRD
0.09 75.48
74.14 - 0.22
71.16 - 0.17
71.46 = 0.09
73.48
+ 0.13
75.51 0.18
73.94 + 0.22
Lagits-based
KD
74.92 0.28
73.54 +: 0.20
70.66 + 0.24
70.67 0.27
+
0.25
72.98 +:
DIST
75.51 = 0.04
74.73 +: 0.24
0.30 71.75
71.65 = 0.21
73.69 + 0.23
76.31 + 0.19
73.89 + 0.19
DKD
76.24
74.81
71.97
n/a
74.11
76.32
74.68
TTM
76,23 0.15
74.32
0.16
71.46
0.16
16.17=0.0
74.33
WTTM
76.37
0.10
74.58 + 0.26
71.92
0.40
0.28 71.67
74.13 + 0.37
0.27 76.06
74.44 + 0.19
WTTM-CRD
76.61 = 0.24
74.94 +: 0.35
72.20
72.13 0.26
74.52 + 0.29
76.65
74.71 0.14
WTIMHTRD
76.65 0.33
75.34
0.28
72.20 0.27
74.36
77.36
0.13
75.13
A.9 RESULTS ON TRANSFORMER -BASED MODELS
To verify the effectiveness of our proposed distillation method WTTM on transformer-based models. apply it to a vision transformer model DeiT-Tiny (Touvron al. (021), 1), results shown in Table We conduct experiments following the settings in Yang et al. (203bb) and Yang et al. 20022). and compare our results with the vanilla KD and two distillation methods proposed in the above two papers, namely NKD and VITKD. It's shown that the performance of WTTTM is better than all the three benchmark methods. Moreover, combined with ViTKD, WTTM can improve the Top-1 accuracy of DeiT-Tiny to 78.04%, which is also higher than the performance of NKD combined with ViTKD.
13
16

---

Published as a conference paper at ICLR 2024
Table 12: Top-1 accuracy (%) on CIFAR-I00. Each teacher-student pair has different architectures. Standard deviation is provided (the standard deviation missing for DKD since it's not available in the literature).
Teacher Student
Veg MobileNetV2
ResNer50 MobileNetV2
ResNet50 vgg8
resnet32x4 ShuffleNetV
resnet32x4 ShufflaNetV2
WRN-10-2 ShuffleNerVi
Teacher Student
74.64
79.34
79.34 70.36
79.42
79.42
75.61
64.6
64.6
70.5
71.82
70.5
Feature-based
FitNet
64.14 0.50
63.16 1: 0.47
70.69
0.22
73.59
0.15
73.54 0.22
73.73 1: 0.32
AT
0.20
58.58 + 0.54
71.84 0.28
71.73
0.31
72.73 + 0.09
73.32 - 0.35
VID
6..56
67.57 0.28
70.30 0.31
73.38
0.17
73.61 + 0.12 0.12
RKD
64.52
64.43 0.42
=
0.39
73.21 0.28
72.21
= 0.16
PKT
0.30
66.52 0.33
73.01 0.14
74.10
73.89
0.16
CRD
69.73 + 0.42
69.11 0.28
74.30 0.14
75.11
0.32
75.65
0.10
76.05 0.14
Logits-based
KD
67.37 + 0.32
67.35
0.32
73.81 0.13
74.07
0.19
0.27
74.83 + 0.17
DIST
68.50 - 0.26
68.66 + 0.23
74.11 0.07
76.34
76.40 0.03 0.255
DKD
69.71
70.35
n/a
76.45
77.07
76.70
68.98 + 0.85
69.24 0.28
74.87 0.31
76.57 0.26
7 0.33
WTTM
69.16 - 0.20
69.59 + 0.58
74,82 0.28
74.37 0.39
0.08 76.55
75.42 + 0.34
WTTM+CRD
70.30 1 0.68
70.84 + 0.56
75.30 0.42
75.82 0.16
77.04 0.19
76.86 0.37
WTTM+ITRD
70.70 = 0.45
71.56 + 0.15
76.00
77.03 = 0.26
77.68 0.26
77.44 + 0.27
Table 13: Top-1 accuracy (%) on ImageNet.
Teacher
Student
KD
VITKD
NKD
WTTM
NKD+VITKD
WTTM+ViTKD
DeiT III-Small (82.76)
DeiT-Tiny (74.42)
76.01
76.06
76,68
77.03
77.78
78.04
17


Reviews:
[{"review_id": "bYuBD4MNLm", "reviewer": "Reviewer_LeDY", "summary": "The paper systematically analyzed the effect of dropping the temperature scaling on the student side in knowledge distillation (KD). The theoretical analysis shows that such a transformation leads to a general KD loss and a Renyi entropy regularization that improves the generalization of the student. Further, To further enhance student’s capability to match teacher’s power transformed probability distribution, the paper introduces a sample-adaptive coefficient to the method. Experiments are conducted to validate the effectiveness of both modules. Experiments are evaluated with different model architectures and teacher quality.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I think overall the paper provides new findings to understand the role of temperature in knowledge distillation. And the evaluation experiments are extensive.\n\n1. The theoretical derivation and analysis for the general KD, Renyi entropy, and transformed teacher matching is precise and solid.\n\n2. Extensive experiments confirm the theoretical analysis and show the effectiveness of each proposed module.", "weaknesses": "1. It's better to provide a detailed summary and comparison of the latest related works.\n\n2. It's also more convincing to show results on transformer models such as ViT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ypOw6Us1KF", "reviewer": "Reviewer_DiN5", "summary": "The paper systematically studies a variant of KD without temperature scaling on the student side, dubbed TTM. Temperature scaling is crucial in knowledge distillation (KD). This paper introduces transformed teacher matching (TTM), a variant of KD that omits temperature scaling on the student side. TTM includes an inherent regularization term and produces better generalization compared to the original KD. Weighted TTM (WTTM) further enhances the student's ability to match the teacher's probability distribution, achieving state-of-the-art accuracy.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Fruitful discussion about related works to engage the readers.\n- Theoretical derivation from KD to the proposed TTM.", "weaknesses": "The results are completely dependent on the list T and β values of all experiments (see Table 8 and 9), which makes the method impractical. Furthermore, the optimal value may even vary from task to task, dataset to dataset and backbone to backbone. These are my main concerns. Based on the marginal gain compared to the baselines, these empirical results actually weaken the claimed contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NtJdqPheLy", "reviewer": "Reviewer_2Z26", "summary": "This paper investigates the temperature for Knowledge Distillation, proposing Transformed Teacher Matching (TTM), which drops the temperature scaling on the student side. TTM has an inherent Renyi entropy term in its objective function, and this regularization leads to better performance with KD.", "questions": "above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method that rethinking KD via temperature scaling is interesting.\n2. The final TTM does not introduce extra hyper-parameters. Also, the training speed keeps the same.\n3. The results on various datasets and models prove its effectiveness.", "weaknesses": "1. Some references and comparisons are missing:\n\n    [1] Knowledge distillation from a stronger teacher.\n\n    [2] From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels.\n\n    [3] Curriculum Temperature for Knowledge Distillation.\n\n    [4] VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale.\n2. When temperature=1, is TTM the same as the original KD? In some papers, the temperature on ImageNet is actually 1.0.\n3. Could TTM still achieve better performance for larger models (e.g. DeiT-T or DeiT-S)?  VanillaKD shows under strong training settings, the original KD also performs well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k1uorNj0LV", "reviewer": "Reviewer_fD3s", "summary": "This paper proposes a new variant of knowledge distillation called Transformed Teacher Matching (TTM) that drops temperature scaling on the student side and introduces an inherent regularization term. The paper shows that TTM leads to better generalization and achieves state-of-the-art accuracy performance. The paper also introduces a weighted version of TTM called Weighted Transformed Teacher Matching (WTTM) that enhances the student's capability to match the teacher's power transformed probability distribution. The experiments conducted in the paper demonstrate the effectiveness of TTM and WTTM on various datasets and architectures.", "questions": "The only concern to me is the novelty of the work and I hope the authors could discuss some of the related work I mentioned in the revised version.\n\n\n---------------------------------\n\nThe author's response addressed my concerns well, so I'm improving my score to acceptance, thanks!", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper introduces a new variant of knowledge distillation that drops temperature scaling on the student side and introduces an inherent regularization term. This approach is motivated by recent works and is a departure from conventional knowledge distillation. The paper also introduces a weighted version of TTM that enhances the student's capability to match the teacher's power transformed probability distribution. These contributions are novel and have not been explored in previous works.\n\n2. The paper is well-written and presents a clear and concise description of the proposed methods. The authors provide a thorough analysis of the experimental results and compare their approach with state-of-the-art methods. The experiments are conducted on various datasets and architectures, which demonstrates the effectiveness and robustness of the proposed methods.\n\n3. The proposed methods have the potential to improve the performance of knowledge distillation and have practical applications in various domains. The paper demonstrates that TTM and WTTM achieve state-of-the-art accuracy performance on various datasets and architectures. The inherent regularization term in TTM also provides a new perspective on knowledge distillation and has the potential to inspire further research in this area. Overall, the paper makes a significant contribution to the field of knowledge distillation", "weaknesses": "1. The paper could benefit from addressing the lack of novelty by acknowledging that techniques such as R´enyi or f divergence, temperature scaling, and logits normalization have already been widely used in knowledge distillation. For example, Information Theoretic Representation Distillation (BMVC) employed R´enyi divergence for standard distillation, and AlphaNet (ICML2021) utilized the f divergence to distill different sub-networks. Moreover, this method is likely already considered in the distiller's search work (KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs, NeurIPS-2023). \n\n2. To strengthen the paper's findings, it is important to validate the proposed method on downstream tasks such as object detection and segmentation. Including evaluation results on these tasks will demonstrate the practical effectiveness and applicability of the proposed method. Additionally, providing more examples and visualizations will enhance the readers' understanding of how the method works and its impact on the learning process.\n\n3. Furthermore, it is essential to incorporate a thorough discussion of relevant KD-related studies, including Self-Regulated Feature Learning via Teacher-free Feature Distillation (ECCV2022), NORM: Knowledge Distillation via N-to-One Representation Matching (ICLR2023), Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer (NIPS2022), DisWOT: Student Architecture Search for Distillation Without Training (CVPR2023), and Automated Knowledge Distillation via Monte Carlo Tree Search (ICCV2023). This discussion will help position the proposed approach within the existing literature, establish connections, and provide valuable insights for potential comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)