Respecting Transfer Gap in Knowledge Distillation
Authors: Yulei Niu
Long Chen
Chang Zhou
Hanwang Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowledge distillation
transfer gap
inverse probability weighting

Abstract: Knowledge distillation (KD) is essentially a process of transferring a teacher model's behavior, e.g., network response, to a student model. The network response serves as additional supervision to formulate the machine domain, which uses the data collected from the human domain as a transfer set. Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID). We point out that this naive assumption is unrealistic and there is indeed a transfer gap between the two domains. Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-IID transfer set. To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity of a training sample belonging to the machine domain, and assigns its inverse amount to compensate for under-represented samples. Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of \ours~for both two-stage distillation and one-stage self-distillation.

Respecting Transfer Gap in Knowledge Distillation
Yulei N Long Chen'
Chang Zhou? Hanwang Zhang?
'Columbia University Daao Academy, Alibaba Group Nanyang Technological University yuleiiiin 2 zjuchenlong) @gmail
@gmaaiiilnn@@@mmlllllmam com zhouchang. aaccalibabaaiine com
hanvangzhangêntu sur
Abstract
Knowledge distillation (KD) is essentially a process of transferring a teacher model's behavior, e.g., network response, student model. The network response serves as additional supervision to formulate the machine domain (machine for short), which uses the data collected from the human domain (human for short) as a transfer set. Traditional KD methods hold an underlying assumption that the data collected in both human domain and machine domain are both independent and identically distributed (IID). We point out that this naïve assumption is unrealistic and there indeed a transfer gap between the two domains. Although the gap offers the student model external knowledge from the machine domain, the imbalanced teacher knowledge would make us incorrectly estimate how much to transfer from teacher to student per sample on the non-lID transfer set. To tackle this challenge, we propose Inverse Probability Weighting Distillation (IPWD) that estimates the propensity of training sample belonging the machine domain, and assigns its inverse amount to compensate for represeepeeenned samples. Experiments CIFAR-100 and ImageNet demonstrate the effectiveness of IPWD for both two-stage distillation and one-stage self-distillation.
Introduction
Knowledge distillation (KD) (00 transfers knowledge from teacher model, e.g., a big, cumbersome, and energy-inefficient network, to student model, e.g., small, light, and energy-efficient network, to improve the performance of the student model. A common intuition that teacher with better performance will teach stronger student. However, recent studies find that the teacher's 's accuracy is not a good indicator the resultant student performance (8). For example, a poorly-traines teacher with early stopping can still teach better student 11]]6 or, teacher with a smaller model size than the student is also a good teacher [67]; or, a teacher with the same architecture as the student helps to improve the student-self self-distillation
13 70 25
Should we view KD in the perspective of domain transfer [2 55 we would better understand the above counter-intuitive findings. From Figure we can see that teacher predictions and ground-truth labels indeed behave differently. Although the teacher trained on the balanced dataset, its predicted probability distribution over the dataset is imbalanced. Even the same training set with the same model parameter, teachers with different temperature yield different "soft label" distributions from the ground- -tuth ones. This implies that human and teacher knowledge is from different domains, and there is a transfer gap that drives the "dark knowledge" (20) transferring from teacher to student- regardless of "strong"
"weak"" teachers, it is valid transfer long there is a gap. However, the transfer gap affects the distillation performance of the represreeeesented classes, i.e., classes on the tail of teacher predictions, which is overlooked in recent studies. Take CIFAR- 100 as example. We rank and divide the 100 classes into four groups according to the ranks of predicted
Work done when Yulei was at Nanyang Technological University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Iabel
bbdl
Avg. Prob. # TTtal Samples
600
x Sumples
1600
1600
580
1400
500
1200
450
1000
800
400-
800
40
60
100
1000
Ranked Class Index
Ranked Class Index
Dataset: CIFAR-100; Teacher: ResNet-110
Dataset: ImageNet; Teacher: ResNet-50
Figure 1: Illustration of the distribution discrepancies among ground truth annotations and teacher predictions. Although the teacher model is trained on the balanced data (blue dashed), its prediction distributions are imbalanced according to various temperatures.
probability. As shown in Table [ compared vanilla training, KD achieves better performance in all the subgroups. However, the increase in the top 25 classes is much higher than that in the last 25 classes, i.e., averagely 5.14% vs. 0.85%. We ask: what causes the gap in the first place; more specifically, why does the teacher's non-uniform distributed predictions implies the gap? We answer from an invariance vs. equivariance learning point of view 4 60]:
Human domain: context invari- ance. The discriminative gener- alization is the ability to learn both context-invariant and class- equivariant information from the diverse training samples per class. The human domain only provides invarittnnvariant class-specific information, i.e., hard targets. We normally collect a balanced dataset to formulate human domain.
Table Improvement of KD over vanilla student for different classes. The metric is macro-average recall.
Arch. style
Top 1-25
Top 26-50
Top 51-75
Top 76-100
ResNet50 MobileNerV2
+4.96
+5.92
+1.76
+1.20
resmet32x4 ShuffteNeVV
+5.80
+2.68
+2.52
+0.84
resmet32s4
ShuflleNet22
+4.72
+1.92
+2.24
+0.76
WRN > ShuflleNetVI
+5.08
+7.20
+4.48
+0.60
temperature variable to preserve the context. The temperature allows the teacher to represent a sample not only by its context- invariant class-specifes information, but also its context-equivariant information. For example, dog image with a soft label 0.8-dog 0.2-wolf may imply that the dog has wolf-like contextual attributes such "fluffy coat" and "upright ears". Although the context-invariance (i.e., class) is balanced in (i.e., context) is imbalanced because the context balance is
Machine domain: context equivariance. Teacher models often
the training data, the context-equivarian not considered in class-specific data collection (58). To construct the transfer set for the machine domain, the teacher model annotates each sample after seeing others, i.e., being pre-trained on the whole set. Interestingly, the diverse context results in long-tailed imbalanced distribution, which is exactly reflected in Figure II In other words, the teacher's knowledge is imbalanced even though the teacher trained on a Mlss-balanced dataset.
Now we are ready to point out how the transfer gap is not properly addressed in conventional KD methods. Conventional KD calculates the Cross-Entropy (CE) loss between the ground-truth label and the student's prediction, and the Kullback-Leibler (KL) divergence (30) loss between the teacher's and student's predictions, where constant weight is assigned for the two losses. This is essentially based on the underlying assumption that the data in both the human and machine domains are IID. Based on the analysis of context equivariance, i.e., the teacher's knowledge is imbalanced. Therefore, a constant sample weight for the KL loss would be a bottleneck. In this paper, we propose a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), which compensates for the training samples that are under- weighted the machine domain. For each training sample x, we first estimate its machine-domain propensity score ([Immiieeee) by comparing class-aware and context-aware predictions. A sample with a low propensity score would have high confidence from class-aware predictions and a low confidence from context-aware predictions. Then, IPWD assigns the inverse probability m/azz/maciin))) as the sample weight for the KI. loss to highlight the under-represented samples. In this way, IPWD generates pseudo-poppe popullation [34]]]]]] 24 to deal with imbalanced knowledge.
argue that the assumption unrealistic.
machine)

---

We evaluate our proposed IPWD on two typical knowledge distillation settings: two-stage teacher- student distillation and one-stage self-distillation. Experiments conducted on CIFAR-100 29 and ImageNet [0 demonstrate the effectiveness and generality of our IPWD.
Our contributions are three-fold:
We formulate KD as a domain transfer problem and argue that the naïve IID assumption on machine domain neglects the imbalanced knowledge due to the transfer gap.
We propose Inverse Probability Weighting Distillation (IPWD) which compensate for the samples that are overlooked in the machine domain to tackle the imbalanced knowledge in transfer gap.
Experiments on CIFAR-100 and ImageNet for both two-stage distillation and one-stage self- distillation show that the proper handling of the transfer gap is a promising direction in KD.
2
Related Work
Knowledge distillation (KD) was first introduced to transfer knowledge from an effective but
cumbersome model to smaller and more efficient model
(00]. The knowledge can be formulated
in either output space 20 26 32
68
67
38
53 73 28 or representation space [48 23
69 27
44
18 57
25 KD has attracted wide interest in theory, methodology, and applications [14
For applications, KD has shown its great potential in various areas, including but not limited to classification B3
47 36 21], detection 31
12 61, segmentation [17 39
35
for visual recognition
43 (2),, and text-to-image synthesis [56] for vision-language tasks. Recent studies further discussed how and why KI) works. Specifically, Müller et al. [40] and Shen et al. [51] empirically analyzed the effect of label smoothing on KD. Cho er al. (8). Dong al. [II] and Yuan et al. [67] pointed out that early stopping is good regularization for better teacher. Yuan et al. [67] further found that a poorly trained teacher, even model smaller than the student, can improve the performance of the student. Besides, Memon er al. [37] and Zhou et al. 773 proposed a bias variance trade-off perspective for KD. In this paper, we point out that existing KD methods hold an underlying assumption that the IID training samples are also IID in the machine domain, which overlooks the transfer gap.
tasks, and visual question answering 41
video captioning
Self-distillation is special case of KD, which uses the student network itself as the teacher instead of the cumbersome model, i.e., the teacher and student models have the same architecture 13 7 77 25)
This process can be executed in iterations and produce a stronger ensemble model [3] Similar to KD, traditional self-distillation follows a two-stage process: first pre-training a student model as the teacher, and then distilling the knowledge from the pre-trained model to a new student model. In order to perform the teacher-student optimization in one generation, recent studies [65] 28 proposed one-stage self- dif-distillation that adopts student models earlier epochs as teacher models. These one-stage self- aif-distillation methods outperform vanilla students by large margins. In this paper, we also evaluate the effectiveness of our IPWD as a plug-in in one-stage self-distillation.
Inverse Probability Weighting (IPW) 49 34 24 5) also known as inverse probability of treatment weighting or inverse propensity weighting, was proposed correct the selection bias when the observations are non-IID. IPW uses the inverse of the probability (i.e., propensity score) that the
individual would be assigned to the treatment group reweight the samples. Propensity-weighting techniques have been widely applied and studied in many areas [50], such as causal inference (24]. complete-case analysis [+] machine learning 9 6 54 and recommendation systems [50] 62
In this paper, we view the distillation process as a domain transfer problem and adopt IPW to
dynamically assign the weight each training sample for the distillation loss.
3 Analysis
3.1
Knowledge Distillation (KD)
We view knowledge distillation from a perspective of domain transfer, and take the image classification task as the case study. Suppose that the training data D= () ((x,)) contains x: as the input (e.g., image) and y E TRC as its ground-truth annotation (e.g., one-hot label), where C denotes the number of classes. A standard solution to train the classifier 0 uses the cross-entropy loss as the
3

---

objective:
Lcts (himaan: 0) = E( Prona Reta (z, y: v: =
Σ (x.0)ED
(T 0) - Lei
(1)
D
where (a, v) H(v", y) is the classification loss for sample v. (p, Ei= 4i log denotes the cross entropy between p and q, y² = f(x; f) denotes the model's output probability given I, i.e., y Σi= exp( = where 2" is the output logits of the model. The hard targets provide
exp(=g)
context-invariant class-specific information from the human domain. An assumption held behind Eq. is that the samples are independent and identically distributed (IID) in the training and test set.
KD adopts a teacher model 0 to generate soft targets as extra supervisions, i.e., context-equivariant information. To formulate the machine domain, traditional KD methods commonly use the training set D to construct the transfer set DD' using the same copy of X, i.e. D = { (z, y²)} where y' y f(;; 6*) and z E X. Traditional KD approaches use the KL divergence 30 loss for knowledge transfer:
Edist (machine; 0) (x.,)))) Porst diise 2; 000
(a, & Ldist
DC
(x.
where (xr, 00; = 72 [H vt) Hly vf denotes the distillation loss for sample x. Normally, the outputs of the student and teacher are softened using a temperature T, ie., Df,K
exp(st/r) and V'.A Ee (+/T)) The overall objective combines Lets and Edist as: (xp(/T)
exp(<i/7)
Lid== a Lets + B Ldist;
where a and 3 are the hyper-parameters, The underlying assumption of traditional KD behind Eq. ( is that the transfer set D' is an unbiased approximation of the machine domain. However, the observed long-tailed and semperature-sensitive distributions of teacher's predictions in Figure [ rationally challenge this assumption. As result, samples with lower P(x) (mchiin) are under-representee during the distillation process, which affects the unbiasedness of knowledge transfer. This analysis indicates that Eq. 2 is not optimal to utilize the teacher's ' imbalanced knowledge.
3.2 Transfer Gap in KD
D
D
We interpret the transfer gap and its confounding effect from the perspective of causal inference. Figure 2 illustraaes the causal relations between the image X, training data D = {2x v).. teacher's parameters 6t and teacher's 's output Y in KD. Overall, DD and 6* jointly act as the confounder of X and Y in the transfer set. First, the training set D and transfer set of teacher model DD' (xx y'}} share the same image set, and X = is sampled from the image set of D. i... D serves the cause of X. Second, the teacher 0f trained on D, and y² is calculated based on 0' and i.e., y² fa; 0²). Therefore, X and 0f are the cause of Y Note that the transfer set is constructed based on the images on and teacher model 0', Therefore, we regard the transfer set Dt. the joint of D and 8¹, as the confounder of X and Y
θ
X
Figure 2: Causal graph for KD..
Although D is balanced when considering the context- context-invariant class- specific information, the context information (e.g., attributes) is overlooked, which makes the D imbalanced on context. As shown in Figure D such an imbalanced context leads an imbalanced transfer set D' and further affects the distillation performance of teacher's 's knowledge.
To overcome such confounding effect, a commonly used technique is intervention via ((y"))((=)) in- Pla.y' (D") stead of P(y [z), which is formulated as P(y))do(z)) = ED P(ylæ, D') = ED Piz This transformation suggests that we can use the inverse of propensity score, 1/ 1/P(a as sample weight to implement the intervention and overcome the confounding effect. Thanks to the causality- based theory [49 [5) we can use the Inverse Probability Weighting (IPW) technique to overcome the
confounding effect brought by the transfer gap.

---

Dietil head
Eq.(9)
student brokbone
student bactbone
DistMl heod
Ingar
Jesture
leput
feature
hoad
Bq, (3)
Fq (1)
(a) Traditional
(b) IPWD
Figure 3: The comparison of training pipelines between traditional KD and our IPWD. The module and outputs in grey are not used the delivered student model.
4 Method
We propose simple yet effective method, Inverse Probability Weighting Distillation (IPWD), to respect the transfer gap and imbalance knowledge in KD. this section, we first introduce the overall framework of IPWD, then present the implementation details.
4.1 Inverse Probability Weighting for KD
As analyzed in Section 3 the IID training samples in the human domain are longer IID in the machine domain. Simply assuming the training set as the perfect transfer set may lead to the selection bias: samples that match "head" knowledge over-represented and easy to be observed, while samples that match "tail" knowledge are under-tepresenned and hard to be observed. This would suppress the transfer of "tail" knowledge. The analysis from the perspective of causal inference in Section 3.2 suggests that we can use Inverse Probability Weighting (IPW) for debiased distillation. In short, IPW generates large weights and over-represented samples are assigned with small weights. The weight for sample determined the inverse of its probability, also known propensity score, to the domain E {human, machine), }, i.e., Wald = 1/p(xld). We adopt IPW to KD and obtain the following objective for sample x:
seeudo-population where under-nepresented samples assigned with
(x:; Σ Wold 1² P(æ|human)
lats (2 0) + P(ximachine) ldist (2, v'; (4)
4.2 Implementation
Since the training and test data are normally IID in the human domain, we safely and rationally use the empirical risk. Therefore, we assign a constant weight to each sample when calculating the classification loss. For sure, the training and test samples can be both non-III, egg., long-tailed recognition tasks, which out of the scope of this paper.
As analyzed in Section [ the assumption held by traditional KD, i.e., both P(X human)) and (machinee) are IID, unrealistic in practice. Therefore, should consider the propensity score P(r machine) as sample-specific value for the distillation loss to improve the generalization. Traditional IPW estimates the propensity score using logistic regression, i.e., P( ((arimcch = ex11+exp(Zz)) where 2g is the logit for æ. Since the ground-trute -rund annotation of P( machine) is not available, it is not practicable to directly train the regression model in fully-supervised manner. Therefore, we estimate the propensity score in an unsupervised way.
Recall that the samples with high propensity are over-represented in the transfer set. As result, the student model would learn less from the under-represented group via distillation. Therefore, we use classification-traiie -trainee (CLS-trained) classifier for the human domain as reference, and assume that KD-trained classifier for the machine domain is more confident for over-represented group than the
classifier. We compare the outputs of two classifiers to identify whether sample under-eepresented the machine domain. Suppose that the KD-trained output is ydd and the CLS S-trained output is ele The assumption implies that the logit 2y is negatively correlated with
5

---

you
y
you
Pop
Ronk
Rack
epery
growwe
yM
yo
y
ydb
Baok
Book
Saasen 4u
Abricon wild dog
hyono
ohole
race cot
sports cor
convertbie
ent
ground beatle
liger beetle
you
you
yolk
your
duck
block
ocourtic
(b)
(c)
Figure 4: Illustration of the sample weights. Bold underlined class denotes the ground-truth label. The areas in yd and y represent the proportion of the predicted probability. A high rank indicates large sample weight, while low rank indicates small sample weight.
H (1 kd
and positively correlated with H (y v). where H:,-) the cross-entropy defined in Considering the range of logit, estimate 2g 2z log Hgye
Section
H(y
Figure illustrates the comparison between traditional KD and our IPWD. We take the KD-trained student's output 1/ as yd, and train an extra classifier head (i.e., "CLS head" Figure 3)) to calculate you which is optimized using the cross-entropy loss Lats with the ground-truth labels. As shown in Table 5 we empirically found that directly using y² and y would lead to high variance. For example, wrongly classified sample may have extremely large loss and heavily suppress the distillation of other samples. Therefore, we normalize the logits by dividing them by the standard exp(= (xp(=(le/ /0
deviation of and dits, i.e., T E exp( (xp(=//o*) and jists Σ exp(= /ool)) In this way, the
exp(=t/a")
outputs at the same scale with standard deviation equal to I, which helps reduce the variance. We finally take ÿ as ykd and gels as yels. Combining 2n into the propensity score, we have:
H
|aa|cincine) H Hy v) H(ykd
(5)
Hiy Figure illustrates "y) some examples of training samples and their assigned weights. Under-represented samples, for which the KD-trained classifier less confident than the CLS-trained classifier, are assigned with large weight (Figure [a). Over-represented samples, for which the KD-trained classifier is more confident than the CLS -taiied classifier, are assigned with a small weight (Figure 4c)). Samples for which the two classifiers behave similarly are assigned with a balanced weight (Figure 4b)). The weighted distillation loss formulated as:
and the estimated weight u. for sample is ú. P(x (onachine)
Cipu dist
Σ
(6)
Dt
(D
Our final IPWD objective is formulated as:
Lipud = Lets + CLiiprodist
(7)
where a is a trade-off
between classification and distillation.
Limitations and negative societal impacts.. As introduced in Section 4.2 we estimate the propensity score by comparing the heads of the student model. Therefore, the estimation relies on the quality of the student model. A poor student may not correctly estimate the propensity, which may further suppress the effectiveness IPWD. Also, we assume that the training and test samples are IID in the human domain, which may not be valid for long-tailed tasks. To the best of our knowledge, as our work is purely an algorithm for knowledge distillation, we haven't found any negative societal impact.

---

Table 2: Top-1 accuracies (%) CIFAR-100 for two-stage distillation. results using the same teacher model.
denotes reproduced
Same architecture style
Different architecture style
Teacher Student
resnet32x4 ShuffleNrrVi
WRN-40-2
WRN-40-1
resner20
resnet32
resmet8x4
ShuffleNerV
veg8
MohileNetV2
Teacher
75.61
72.34
74.31
79.42
79.42
75.61
79.34
79.34
Student
71.98
69.06
71.14
72.50
70.50
70.50
70.36
64.60
Nit
72.24
69.06
71.06
73.50
73.59
73.73
70.69
63.16
AT
69
72.77
69.21
72.31
73.44
71.73
73.32
71.84
58.58
SP
59
72.43
69.67
72.69
72.94
73.48
74.52
73.34
68.08
CC 46
72.21
69.63
71.48
72.97
71.14
71.38
70.25
65.43
73.30
70.38
72.61
73.09
73.38
73.61
70.30
67.57
RKD 44
72.22
69.61
71.82
71.90
72.28
72.21
71.50
64.43
PKT 45
73,45
70.34
72.61
73.64
74.10
73.89
73.01
66.52
AB 19
72.38
69.47
70.98
73.17
73.55
73.34
70.65
67.20
FT 27
71.59
69.84
72.37
72.86
71.75
72.03
70.29
60.99
NST 23
72.24
69.60
71.96
73.30
74.12
74.89
71.28
64.96
KD 20
73.54
70.66
73.08
73.33
74.07
74.83
73.81
67.35
CRD [57
74.14
71.16
73.48
75.51
75.11
76.05
74,30
69.11
WSLD
73
73.74
71.53
73.36
74.79
75.09
75.23
73,80
68.79
IPWD
74.64
71.32
73.91
76.03
76.03
76.44
74.97
70.25
Experiments
We take the image classification task as case study evaluate the effectiveness and generalizability of our IPWD. Following previous works [57 73 28 we conduct experiments with two settings,
two-stage distillation and one-stage self-distillation.
5.1 Datasets and Settings
Datasets. We conducted experiments on CIFAR-100 29 and ImageNet [10] CIFAR-100 contains 50K images in the training and 10K images in the
from 100 classes. ImageNet provides 1.2M images in the training set and 50K images in the validation set from IK classes.
Settings. Two-stage distillation is the conventional setting that pre-trains teacher model at the first stage and transfers the knowledge to a student model at the second stage. Commonly, the teacher is a larger model, and the student is smaller model. For self-distillation, the teacher and student have the same architecture. One-stage self-distillation aims to complete the teacher-student optimization simultaneously
655 28 i.e., the pre- pre-training and transfer processes are reduced to one.
5.2 Two-stage Distillation
73 we considered the following methods baselines: KD 20), FitNet [48], AT 699 SP 59 CC 46),
Baseline methods. For two-stage distillation, following Tian al. [57] and Zhou et
VID 2 RKD 44), PKT [45), FSP (66), AB [99, FT 27), NST 233, CRD [57, SSKD and WSLD 73 In particular, WSLD 73 is the most related work to us, which proposed variinariance trade-off perspective for KD and also assigns different weights to each training sample. Similarly, the weight is positive related to the cross-entropy loss of student's output. The main differences between our IPWD and WSLD are as follows. First, our formulation of samples weights is thcoretically guaranteed by the causal theory behind Inverse Probability Weighting (IPW) 49 WSLD estimates the sample weight using both the student model and the teacher model. As a comparison,
34 24 5 Second,
the student model with two different classifier heads to guarantee that the capacities of the compared models are close.
Implementation. For experiments CIFAR- 100, we followed CRD 57 based the open-sourced code. We set the trade-off hyper-parameter a 5 Eq. (7) and the temperature 7 10. Other training details were the same as CRD [57] and provided in the appendix. For ImageNet, followed

---

Table 3: Top-1 accuracies (%) on CIFAR- 1100R-10000 test as a plug-in on SSKD 64 results of SSKD using the same teacher model.
We reproduced the
Same architecture style
Different architecture style
Teacher Student
WRN-40-2
WRN-40-2
mesnet66
resee13244 resnet8u4
ReaNer50
mone ShufleNaaa
WRN-40-2 ShuMleNNrVV
vgz13
WRN- 16-2
WRN-40-1
resne120
MobileNee
MobvleNeeV2
Teacher Student
76.46
76.46
73.44
79.63
79.10
79.63
76.46
75.38
73.64
72.24
69.63
72.51
65.79
70.77
70.77
65.79
SSKD
75.74
75.59
70.61
75.80
72.22
77.71
78.49
77.32
+ IPWD
76.39
76.09
71.69
76.74
72.85
78.30
79.17
77.95
Zhou et al. 173 to conduct experiments based on their open-sourced code. We used the same hyper parameters as WSLD i.e., a as 2.5 and T as 2.
Comparison with baseline methods. Table Shows the results of student models on CIFAR-100 with different teacher-student architectures, which can be grouped into same architecture style and different architecture style. Note that the results of WSLD reported in [73 used different pre-trained teacher model. Since some training techniques like early-stopping 8 67 may improve the distillation performance, we reimplemented WSLD using the same teacher model for fair comparison. Overall, our IPWD outperforms KD by large margins and outperforms other baseline methods on most of the architectures, which demonstrates the effectiveness of our IPWD. In particular, the improvement with the same architecture style is smaller than the different style. The reason that the different architecture style reflects the bigger gap between the human domain and machine domain. Since our IPWD weights the training samples address the non-IID problem, IPWD successfully outperforms KD and other state-of-the thr methods by large margins when the transfer gap is significant.
Note that SSKD 64 achieves higher per- formance because of (1) a better teacher model, and (2) data augmentation for structured knowledge distillation. We further apply IPWD SSKD as a plug- in by weighting the logit distillation ob- jective and keeping the structured knowl- edge distillation terms unchanged. Ta- ble 3 shows that our IPWD can consis- tently improve SSKD by 0.5-1.0% for different architectures. These results in- dicate that our IPWD is a good comple- mentary to distillation methods.
Table 4: Acc. (%) on ImageNet for two-stage distillation.
Same arch. style
Diff. arch. style
Teacher Student
ResNet-34
ResNet-50 MobileNet-vi
ResNet-18
Top-1
Top-5
Top-1
Top-5
Teacher Student
73.31
91.42
76.16
92.87
69.75
89.07
68.87
88.76
AT
69
71.03
90.04
70.18
89.68
NST 23
70.29
89.53
FT 27
69.88
89.50
FSP 66
70.58
89.61
Table 4 further shows the comparison on ImageNet. Following CRD (57) and WSLD [73), we used two teacher- student architectures as the representa- tives. For the same architecture style, our IPWD improves KD by 1.21%, %, and achieves competitive performance com- pared to WSLDD For the different archi- tecture style, the improvement of WSLD over KD drops from 1.37% to 1.03%. As comparison, our IPWD improves KD by 2.16% and outperforms WSLD by 1.13%. This improvement on the large-scale dataset further demonstrates the effectiveness of our IPWD in bridging the transfer gap when the student and teacher model have different architecture styles, which more practical in real-world applications.
AB 19
68.89
88.71
RKD 44
70.40
89.78
68.50
88.32
KD 20
70.67
90.04
70.49
89.92
Overhaul
18
71.03
90.15
71.33
90.33
CRD 57
71.17
90.13
69.07
88.94
SSKD
64
71.62
90.67
DGKD
53
71.73
90.82
WSLD 73
72.04
90.70
71.52
90.34
IPWD
71.88
90.50
72.65
91.08
we used an extra classifier head to produce CLS-trained output, and normalized the logits to reduce the variance for propensity estimation. Note that WSLD 7X uses the teacher model to estimate the sample weight, and the teacher model is also trained with the cross-entropy loss. Therefore, we considered an alternative which replaces the classification head with the teacher model to produce the classification-awall
A Ablation study: technical designs. As introduced in Section
8

---

Table 5: Ablation study of technical designs for weight estimation CIFAR-100. "CLS head" denotes the usage of an extra classification head. "logits norm." denotes that the logits are normalized before calculating the propensity.
Same architecture style
Different architecture style
WRN-10-2
resnct 10
resmet22el
resnet32ad
WRN-10-2
4
+
+
CLS head
logits norm.
WRN-40-1
resner
resnen8x4
ShuffleNerV1
ShuftleNer
ShaffleNetVi
training diverges
52.81
57.99
53.31
74.01
73,41
75.89
75.49
76.48
76.34
74.42
73.48
75.97
75.80
76.45
75.96
IPWD
74.64
73.91
76.03
76.03
76.61
76.61
output. To evaluate the contribution of logits normalization, considered alternative that the logits are not normalized by the standard deviation. Results in Table [ verify the contribution cach design. Without the classification head and logit normalization, the training hard to converge the performance is much worse. As a comparison, either the classification head or logit normalization helps with stable training. Besides, combination of both further improves the performance and achieves the best results. The crash of training is due to the high variance of sample weights. Since the teacher model well pre-trained and has more parameters, it has larger capacity than the student model. Differently, an extra head with a shared backbone guarantees a similar capacity. Also, the normalization will avoid an extremely large or small CE loss, which further reduces the variance.
Teacher trained with label smoothing. Recent works [40] [5] observed that KD performs poorly with label smoothing. Similar to KD, the performance of IPWD drops when the teacher model is trained with label smoothing, but still outperforms KD. However, we found that the improvement of IPWD compared to KD also decreases with label smoothing. For example, CIFAR- given ResNet50 as teacher and MobileNetV2 as student, IPWD outperforms KD by 1.12% (69.67% vs. 68.55%) without label smoothing, but the improvement drops to 0.56% (66. (66.79% vs. 66.23%) with label smoothing. Given resnet32x4 as teacher and ShuffleNetVI as student, IPWD outperforms KD by 1.52% (75.79% vs. 74.27%) without label smoothing, but the improvement drops to 0.53% (73.27% vs. 72.74% with label smoothing. We observed that teacher trained with label smoothing produces more balanced predictions compared teacher trained without label smoothing. Therefore, the results are consistent with our hypothesis that IPWD helps to bridge the transfer gap especially when the context information of teacher is imbalanced.
5.3 One-stage Self-Distillation
Baseline methods and metrics. For one-stage seff-distillation, we apply our method to the state- of-the-art PS-KD (28) method as a plug-in, and consider label smoothing (LSS) method and two self-distillation methods, CS-KI) [68] and TF-KD (67] baselines. PS-KD proposed one-stage framework that progressively distills the knowledge of a model itself to soften the one-hot supervisions regularization. The knowledge is transferred using a conventional distillation loss. As for metrics, besides top-1 and top-5 accuracy, we follow Kim et al. [28) to report expected calibration error (ECE, %) and the area under the risk-coverage curve (AURC, x10s). low ECE indicates well-calibrated predictions, and low AURC represents the well-separation of correct and incorrect predictions.
Implementation. We follow all the training details of PS-KD for fair comparison. Specifically, the architectures we considered are ResNet-18 18 [16], ResNet- 101 [5], ResNeXt-29 [63] (cardinality width=64), and DenseNet- 121 [22] (growth rate=32). During training, PS-KD gradually determines how much the student learns from the teacher's 's knowledge. The formulation is:
Cpe-ke ar) Lele + aq Ldists
(8)
where the trade-off parameter a aT x t/T, T is the number of total epochs (e.g., 300), t is the current epoch, and ar is a hyperparameter. Compared to Eq. (8), our IPWD applied on PS-KD is formulated as
Epe-ke + ipw (1 -ar) Lote + a Lipn-diist
(9)

---

Table 6: Results on CIFAR- 100 test for the one-stage self distillation setting over four architectures.
Top-1 and Top-S indicate the accuracy.
Method
Top-1
Top-5
ECE
AURC
Method
Top-1
Top-5
ECE
AURC
ResNet-18
75.82
93.10
11.84
67.65
DenseNet-121
79.95
95.01
7.34
52.21
+ -LS
79.06
93.98
10.79
57.74
+ +LS
80.20
94.54
0.92
91.06
+ CS-KD
78.70
94.30
+ CS-KD
79.53
93.79
13.80
73.37
+ -T-KD
77.12
93.99
61.77
+ +TE-KD
67
80.12
94.90
7.33
69.23
PS-KD 28
79.18
94.90
1.77
52.10
+PS-KD
28
81.27
96.10
3.71
45.55
+ PS-KD Ours
79.82
95.15
49.71
+ PS-KD + Ours
$1.60
96.04
3.48
45.33
RcsNet-101
79.25
94.72
10.02
$5.45
ResNeX1-29
81.35
95.53
4.17
44.27
+ -.S
80.16
94.93
3.4.
95.76
+I.S
82.40
95.77
22.14
41.92
+ CS-KD 68
79.24
94.38
12.18
64.44
+ CS-KD
68
81.74
95.63
5.95
42.11
+ TE-KD 67
79.90
94.90
6.14
58.80
+ TT-KK
67
82.67
96.13
6.73
40.34
+ PS- KD (28
80.57
95.70
6.92
49.01
+ +PS-KD
28
82.72
96.40
9.15
39.78
+ PS-KD + Ours
81.39
95.91
3.19
43.82
+ PS-KD + Ours
83.30
96,60
4.93
37.49
Since both the student and teacher models are poor at early epochs, the weight estimation is not accurate at early epochs, which may lead to worse self-teacher. Therefore, we apply IPWD at the last 75 epochs over the total 300 epochs.
Comparison with baseline methods. Table shows the results of one-stage self-distillation methods over four architectures. Our IPWD can effectively and constantly improve the top-1 accuracy of PS-KD by 0.33%~~0.88 0.82% with different architectures. Besides, our IPWD significantly lowers the ECE and AURC of PS-KD. These results demonstrate the effectiveness our IPWD.
Ablation study: IPWD stage. We conduct ablation study amalyze whether IPWD should be started from early stage (e-g., the beginning of
Table 7: Ablation study the start of applying IPWD.
Method
Top-1 Ace
Top-5 Ace
ECE
AURC
PS-KD 28)
82.72
96.40
9.15
39.78
training) or a late stage (e.g., . last 1/4 of the epochs). We take ResNeXt-29 as example. As shown in Table 77, apply- ing IPWD from the beginning slightly outperforms PS-KD and under-performs the student modal that applies IPWD only at the late stage by large margins. As the student model poorly trained at the early stage, the weight estimation is inaccurate and hurts the performance of self-teacher, These results indicate that the quality of estimated weight and distillation performance relies on the student model and self-teacher.
IPWD (early)
82.86
96.35
8.56
38.16
IPWD (late)
83.30
96.60
4.93
37.49
6 Conclusion
In this paper, we point out that conventional KD methods hold an invalid IID assumption and does not properly address the transfer gap between the context-invarian human domain and the context-equivariant machine domain, especially the imbalanced knowledge of the teacher model on the transfer set. We further proposed a simple yet effective method, Inverse Probability Weighting Distillation (IPWD), deal with the imbalanced knowledge caused by transfer gap. In the future, we will extend our IPWD to (1) tasks beyond classification, like detection and segmentation, and (2) long-tailed tasks where the training samples in the human domain are also non-lID.
Acknowledgment
thank anonymous ACs and reviewers for their valuable discussion and insightful suggestions. This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-RP-2021-02) and Alibaba-NTU Singapore Joint Research Institute (JRI).
10

---

References
[I] Somak Aditya, Rudra Saha, Yezhou Yang, and Chitta Baral. Spatial knowledge distillation to aid visual reasoning. 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 227-235. IEEE, 2019.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-999999 1171-9171,,001 2019.
[3] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 385-394, 2018.
[4] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization, arXiv preprint arXiv:1907.02880 2019.
[5] Peter C Austin. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research, 46(3):399-424, 2011.
[6] Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research, 10(9). 2009.
[7] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowl- edge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008-5017, 2021.
[8] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4794-4802, 2019.
[9] Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In International conference algorithmic learning theory, pages 38-53. Springer, 2008
[10] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[11] Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation s early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv preprint 190.0155, 2019.
[12] Lixin Duan, Ivor W Tsang, and Dong Xu. Domain transfer multiple kernel learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 34(3):465-479, 2012
[13] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2018
[14] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision. pages
Springer, 2016.
[17] Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 578-587, 2019.
11

---

[18] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1921-1930, 2019.
[19] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillattio of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3779-3787 3787, 2019.
[20] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini 1003:00503.02555 2015.
[21] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-ineremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3957-3966, 2021.
[22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[23] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint /00:://000////0 2017.
[24] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[25] Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and Il-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10664-10673, 2021.
[26] Xiao Jin, Baoyun Peng, Yichao WU, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1345-1354, 2019.
[27] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: network compression via factor transfer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2765-2774, 2018.
[28] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEÉ/CVF International Conference Computer Vision, pages 6567-6576, 2021.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[31] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 6356 0664-664,777 2017.
[32] Xiaojie Li, Jianlong Wu, Hongyu Fang, Yue Liao, Fei Wang, and Chen Qian. Local correlation consistency for knowledge distillation. In European Conference Computer Vision, pages 18-33. Springer, 2020.
[33] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 2017.
[34] Roderick JA Little and Donald B Rubin. Statistical analysis with missing data, volume 793. John Wiley & Sons, 2019.
[35] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2604-2613, 2019.
12

---

[36] Zelun Luo, Jun-Ting Hsieh, Lu Jiang, Juan Carlos Niebles, and Li Fei-Fei. Graph distillation for action detection with privileged modalities. In Proceedings of the European Conference on Computer Vision (ECCV), pages 166-183, 2018.
[37] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pages 7632-7642. PMLR, 2021.
[38] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198, 2020.
[39] Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3573-3582, 2019.
[40] Rafael Müller, Simon Komblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint 1906.02299, 2019.
[41] Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to specialize with knowledge distillation for visual question answering. In NeurIPS, pages 8092-8102, 2018
[42] Yulei Niu and Hanwang Zhang. Introspective distillation for robust question answering. Ad- vances in Neural Information Processing Systems, 34:16292-16304, 2021.
[43] Boxiao Pan, Haoye Cai. De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-1087) 2020.
[44] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[45] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pages 268-284, 2018
[46] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li. Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007-5016, 2019.
[47] Zhimao Peng. Zechao Li, Junge Zhang, Yan Li, Guo-Jun Qi, and Jinhui Tang. Few-shot image recognition with knowledge transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 441 449, 2019.
[48] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 2014.
[49] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[50] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning, pages 1670-1679. PMLR, 2016.
[51] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In International Conference on Learning Representations, 2020.
[52] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. Proceedings of the IEEE international conference on computer vision, pages 3400-3409, 2017.
13

---

[53] Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9395-9404, 2021.
[54] Masashi Sugiyama and Motoaki Kawanabe, Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT press, 2012.
[55] Ben Tan, Yu Zhang, Sinno Pan, and Qiang Yang. Distant domain transfer learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[56] Hongchen Tan, Xiuping Liu, Meng Liu, Baocai Yin, and Xin Li. Kt-gan: knowledge-transfer generative adversarial network for text-to-image synthesis. IEEE Transactions on Image Processing. 30: 50:1275-1290) 2020.
[57] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint 19XIv:9910.109999 2019.
[58] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[59] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
[60] Tan Wang. Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. arXiv preprint arXiv:2110. 15255, 2021.
[61] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933-4942, 2019.
[62] Yixin Wang, Dawen Liang, Laurent Charlin, and David M Blei. The deconfounded recom- mender: A causal inference approach to recommendation. arXiv preprint arXiv: / /808.06581, 2018.
[63] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[64] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In European Conference on Computer Vision, pages 588-604 Springer, 2020.
[65] Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 2859-2868, 2019.
[66] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133 4141, 2017.
[67] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[68] Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13876-13885, 2020.
[69] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016.
[70] Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
14

---

[71] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[72] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with seacher-reoomment learning for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pages 11278-13288 2020.
[73] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A biai-variance tradeoff perspective. arXiv preprint arXiv: (02.::20020005500, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] To the best of our knowledge,
work is purely algorithm for knowledge distillation, we haven't I found any negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Please refer to the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The details are in Section
(c) Did you report error bars (e.g., with respect to the random seed after rumning exper- iments multiple times)? [Yes] We run the experiments with five different random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All the used existing assets open-sourced for research. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] The benchmark dataset does not contain personally identifiable information offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "z-bNO_kiBJJ", "reviewer": "Reviewer_ADzo", "summary": "This paper proposes a new loss function for teacher-student learning by observing the fact that the label predictions from a teacher model has a distribution shift compared with ground-truth labels. It is pointed out that this distribution mismatch can cause a gap during transfer due to over-representation and under-representation of examples, e.g. the existence of context-equivariance and context-invariance information in some image datasets. Specifically, inverse probability weighting is used in the loss function to assign larger weights to the KL divergence loss of under-represented examples. The weight is implemented by comparing the cross-entropies of both classification outputs and knowledge distillation outputs to the ground-truth label. Finally, empirical evaluations are conducted on some image datasets and results show that the proposed method can achieve improved performance over some baselines.", "questions": "Q1: It is claimed several times in the paper that the dataset generated by the teacher model is no longer IID (e.g. L142). I don't understand why this is the case. Once we have a teacher model, the dataset is generated from the distribution induced by this teacher model. Given the parameter $\\theta^t$, individual examples $(x,y^t)$ are also independent from each other. As introduced in Section 2, IPW is practically suitable to non-IID observations. The resulting distribution is likely to be long tailed but the examples still seem to be independent, which weakens the justification of the use of IPW. Could the authors please explain more on this?\n\nQ2: L146: \"Thanks to the causality-based theory, we can use the IPW technique to overcome the confounding effect brought by the transfer gap\". In my opinion, this claim is not well supported by current writing. Which part of the causality-based theory is relevant to the problem described in this paper? In this paragraph only references are mentioned but detailed explanation is missing. For the second half of the sentence, I don't understand what is the confounding effect brought by the transfer gap. How does the confounding effect occur in the dataset of teacher model?\n\nQ3: In Fig.1, the blue dashed line refers to the histogram of different labels. The ImageNet dataset (ILSVRC 2012) has 1,000 classes with varying number of images per class ranging from around 732 to 1300. Its training set is imbalanced. I don't see why in the right subfigure the label distribution is a straight line. How is the average probability on the y-axis calculated for labels or predictions from a teacher model?\n\nQ4: In experimental comparisons, the proposed method is compared with some baselines including CRD and WSLD. I'm not sure whether the reported results are SOTA on knowledge distillation benchmark. Are there any other published baselines?\n", "limitations": "See weakness and questions.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The motivation of solving the distribution mismatch problem of the teacher model is reasonable and novel. This observation is of practical importance in knowledge distillation as this problem is expected to occur in many teacher student learning datasets. The use of inverse probability weighting (IPW) for KD in the proposed method seems to be helpful for this problem according to the results of experiments and ablation study.\n\nWeaknesses: Although the aforementioned problem is well clarified, the use of IPW is not well motivated or explained. Many explanations of IPW and its motivation relate to the problem of non-IID in the model dataset. However, my concern is that this does not exactly fit or target the problem of distribution mismatch problem. Please see the questions below for more details. Moreover, the method of IPW has been well-established before. The technical contribution is limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "clergAPJ-04", "reviewer": "Reviewer_T8CL", "summary": "This paper investigates the knowledge distillation problem by formulating it as a transfer learning problem. It finds that even if the teacher is trained with balanced training data, the transferred information could be imbalanced and it hurts the performance of KD. Therefore, this paper proposes Inverse Probability Weighting Distillation (IPWD) as an improvement of typical knowledge distillation loss, and extensive experimental results show the effectiveness of the proposed IPWD.", "questions": "Please refer to the \"Weaknesses\". ", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper investigates KD from a transfer learning perspective, which is interesting compared to the typical dark knowledge. \n+ The structure and writing are good. \n+ The experimental results are comprehensive. \n\n- Figure 1 shows that there seems long-tailed property of teacher predictions. I wonder whether it could be fixed by some techniques from long-tailed classification. \n\n- Formulation of propensity in Section 4.2 seems sort of heuristic. I do not really get how to understand it in a principal and mathematical way. \n\n- IPDW is a logits-based KD method. Would it be better if combined with the feature-based method? Necessary discussions and experimental comparisons are needed.  \n\n- This paper claims that knowledge distillation will work if there is a valid transfer gap. However, recent work argues that if the teacher becomes way too strong, the KD seems to degrade. Will the proposed IPDW handle this issue? If so, experiments about a strong teacher should be included. A recent work DIST investigates a similar setting, which should be helpful if the authors want to verify the transfer gap issue in terms of a huge or strong teacher. \n\nKnowledge Distillation from A Stronger Teacher, https://arxiv.org/abs/2205.10536\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "USK2rArPAaD", "reviewer": "Reviewer_6yEk", "summary": "This paper proposes a new KD method named IPWD. The authors state that the teacher's knowledge is imbalanced due to the imbalanced soft labels in teacher's predictions, and propose to use inverse probability weighting (IPW) to balance the weight of each sample in KD loss. Experiments on CIFAR-100 and ImageNet are provided to show the superiority of IPWD.", "questions": "1. The distribution of teacher's soft labels would be varied a lot according to different training strategies. Can IPWD adapt better to these distribution shifts? For example, some works [1, 2] observed that KD performs poorly when with label smoothing.\n\nI will raise my rating if the authors can well address my concerns.\n\n\n[1] Müller, R., Kornblith, S. and Hinton, G.E., 2019. **When does label smoothing help?**. Advances in neural information processing systems, 32.  \n[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K.T. and Savvides, M., 2020, September. **Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study**. In International Conference on Learning Representations.", "limitations": "Both limitations and potential negative societal impact were discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper gives an interesting domain transfer perspective of KD, in which the distributions of ground-truth labels and teacher's soft labels are different, this conflict makes KD less effective. Based on this, the authors propose a simple KD loss to balance the teacher knowledge using IPW, which gains significant improvements over the KD baseline.\n\nWeaknesses:\n1. The authors should discuss the impact of imbalanced teacher knowledge in conventional KD methods, to show the necessity of balanced knowledge. For example, compare the predictions of under-represented class of student trained with or without KD.\n2. In Table 3, IPWD performs worse than WSLD on ResNet-18 student (71.88% vs. 72.04%).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept