Knowledge Distillation from A Stronger Teacher
Authors: Tao Huang
Shan You
Fei Wang
Chen Qian
Chang Xu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Unlike existing knowledge distillation methods focus on the baseline settings, where the teacher models and training strategies are not that strong and competing as state-of-the-art approaches, this paper presents a method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predictions between the student and a stronger teacher may tend to be fairly severer. As a result, the exact match of predictions in KL divergence would disturb the training and make existing methods perform poorly. In this paper, we show that simply preserving the relations between the predictions of teacher and student would suffice, and propose a correlation-based loss to capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities to each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well to various architectures, model sizes and training strategies, and can achieve state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https://github.com/hunto/DIST_KD.

Knowledge Distillation from A Stronger Teacher
Tao Huang .22
Shan You
Fei Wang²
Chen Qian
Chang Xu²
Semse'Time Research
School of Computer Science, Faculty of Engineering, The University of Sydney University of Science and Technology of China
Abstract
Unlike existing knowledge distillation methods focus the baseline settings, where the teacher models and training strategies are not that strong and competing state-of-the- art approaches, this paper presents method dubbed DIST to distill better from a stronger teacher. We empirically find that the discrepancy of predic- tions between the student and stronger teacher may tend to be fairly severer. As result, the exact match of predictions KI. divergence would disturb the training and make existing methods perform poorly. In this paper, preserving the relations between the predictions of teacher and student would suffice, and propose correlation-based loss capture the intrinsic inter-class relations from the teacher explicitly. Besides, considering that different instances have different semantic similarities each class, we also extend this relational match to the intra-class level. Our method is simple yet practical, and extensive experiments demonstrate that it adapts well various architectures, model sizes and training strategies, and can achieve state-of-ther offthhe-rt performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at: https//gituub.coo/nnnt///// _DD
show that simply
Introduction
The advent of automatic feature engineering fuels deep neural networks to achieve remarkable success in plethora of computer vision tasks, such image classification [17, 19, 38, 48, 53), object detection [2, 23]. and semantic segmentation [5, 54]. In the path of pursuing better performance, current deep learning models generally grow deeper and wider [13, 45). However, such heavy models are clumsy to deploy in practice due to the limitations of computational and memory resources. For efficient model with competitive performance those larger models, knowledge distillation (KD) [16] has been proposed to boost the performance of the efficient model (student) by distilling the knowledge larger model (teacher) during training.
The essence knowledge distillation relies on how to formulate and transfer the knowledge from teacher to student. The most intuitive yet effective approach is to match the probabilistic prediction (response) scores between the teacher and student via Kullback Leibler (KL) divergence [16]. In this way, the student can be guided with more informative signals during training, and is thus expected to have more promising performance than that being trained stand-alone. Besides this vanilla prediction match, other works [11, 14, 34, 41] also investigate the knowledge within intermediate representations to further boost the distillation performance, but this usually induces additional training cost as a consequence. For example, OFD [14] proposes to distill the information via multiple intermediate layers, but requires additional convolutions for feature alignments; CRD [41] introduces a contrastive loss transfer pair-wise relationships, but it needs to hold memory bank for all 128-d features of ImageNet images, and produces additional 260M FLOPs of computation cost.
"Corresonddence to: Shan You <youshannSsensetiine. com.>.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

78.0
Tocter
800
Techer
XD
76.0
w/oKD
760
7600
ACC
72.2
ACC
72.0
72.0
BesNeet34
ResNet-30
ResNes-131 BosNNt-112
81
B1-15
B2
Teacher
Straligy
(a) Stronger model sizes
(b) Stronger strategies
Figure 1: Comparisons of KD and our proposed DIST on ImageNet with different teachers. (a) The ResNet-18 students are trained using baseline strategy with different model sizes of the teacher. (b) The ResNet-18 18 students are trained using different strategies with ResNet-50 teachers.
Recently, a few studies [8, 29, 39] have been performed to address the poor learning issue of the student network when the student and teacher model sizes significantly differ. For example, TAKD [29] proposes to reduce the discrepancy of teacher and student by resorting to an additional teaching assistant of moderate model size; DGKD [39] further improves TAKD by densely gathering all the assistant models to guide the student. However, increasing the model size is only one of the popular approaches to have a stronger teacher. There lacks a thorough analysis on the training strategies to derive stronger teacher and their effect KD. Most importantly, generic enough solution is preferred to address the difficulty of KD brought by stronger teachers, rather than struggling to deal with different types of stronger teachers (with larger model size stronger training strategy) individually.
To understand what makes a stronger teacher and their effect on KD, we systematically study the prevalent strategies for designing and training deep neural networks, and show that:
Beyond scaling up the model size, stronger teacher also be derived through advanced training strategies, e.g., label smoothing and data augmentation [51]. However, given a stronger teacher, the student's performance the vanilla KD could be dropped, even worse than training from scratch without KD, as shown in Figure
The discrepancy between teacher and student tends to get fairly larger when we switch their training strategy stronger (see Figure 2). In this case, an exact recovery of predictions via KI. divergence could be challenging and lead to the failure vanilla KD.
Preserving the relation of predictions between teacher and student sufficient and effective. When transferring the knowledge from teacher to student, what we really care about is preserving the preference (relative ranks of predictions) by the teacher, instead of recovering the absolute values accurately. Correlation between teacher and student predictions could be favored to relax the exact match of KL divergence and distill the intrinsic relations.
In this paper, we thus leverage the Pearson correlation coefficient [33] as new match manner to replace the KL divergence. In addition, besides the inter-class relations in prediction vector (see Figure 3), with the intuition that different instances have different spectrum similarities with respect to each class, we also propose to distill the intra-class relations for further boosting the performance as Figure 3. Concretely, for each class, we gather its corresponding predicted probabilities of all instances in batch, then transfer this relation from teacher to student. Our proposed method (dubbed DIST) super simple, efficient, and practical, which can be implemented with only several lines of code (see Appendix A. and has almost the same training cost the vanilla KD. As result, the student can be liberated from the burden of matching the exact output of strong teacher, but only be guided appropriately to distill those truly informative relations.
Extensive experiments are conducted on benchmark datasets to verify our effectiveness on various tasks, including image classification, object detection, and semantic segmentation. Experimental results show that our DIST significantly outperforms vanilla KD and those cophistiaatedly-esiigne state-of-te th- KD methods. For example, with the same baseline settings ImageNet, our DIST

---

K.Dhvegenstt (c-l)
KL. Kivneegeena (v-d)
100 020g
(a) KL. div.
(b) KI. div. (T
Figure 2: Discrepancy between the predictions of models trained standalone with different strategies on ImageNet validation set. R/8B/ represents ResNet-18 trained with strategy B1 for instance. Details of training strategies Bl and B2 refer Table
achieves the highest 72.07% accuracy on ResNet-18. With the stronger strategy, our method obtains 82.3% accuracy on the recent transformer Swin-T [27], improving KD by 1 %.
2 Revisiting Prediction Match of KD
In vanilla knowledge distillation [16], the knowledge is transferred from a pre-trained teacher model to student model by minimizing the discrepancy between the prediction scores of the teacher and student models.
Formally, with the logits (a) E BxC and ~ E BxC of student and teacher networks, where B and C' denote batch size and the number of classes, respectively, the vanilla KD loss [16] is represented as
LKD = 72 M K
B
log
(1)
E
i=1
1=
1.j
where KL refers Kullback-Leine divergence with
softmar (o,aaarZZ/))/ /T), y = so ( /T),
(2)
being the probabilistic prediction vectors, and 7 is the temperature factor to control the softness of logits.
In addition to the teacher's soft targets in Eq.(1), KD [16] stated that it is beneficial to train the student together with ground-truth labels, and the overall training loss is composed of the original classification loss Lets and KD loss LKD. i.e.,
Lur = aL + BLKD,
where Cels is usually the cross- loss between the predictions of student network and ground- truth labels, and 3 are factors for balancing the losses.
2.1 Catastrophic discrepancy with stronger teacher
As illustrated Section 1, the effect of a teacher on KD has not been sufficiently investigated, especially when the performance of pre-trained teacher grows stronger, such as with larger model size or being trained with more advanced and competing strategies, e.g., label smoothing, mix-up [51], auto augmentations [9], etc. With this regard, as Figure 2, we train ResNet- 18 and ResNet-50 standalone with strategy and strategy B22, and obtain trained models (RI8BI, R18B2, R50BI, and R50B2 with accuracies 69. 79%76% 73.4%, 13%, %, and 78.5%, respectively), then compare their discrepancy using KL divergence (T and T = 4) on the predicted probabilities Y. We have the following observations:
Training with B2 obtains higher accuracy compared to B1, e.g., 73.4% (B2) 69.8% (B1) ResNet-18.

---

Existing methods
Input
Teacher
occ
Student
Conventional KD
Instance Relation
Output
DIST
t1
81
82
classes
Comventional KD
0.5
sulation
Tescher
Student
intra-class Relation
Figure 3: Difference between our DIST and existing KD methods. Conventional KD matches outputs of student (s E ") to teacher (L E R²) point-wisely; instance relation methods operate on the feature level and measure the internal correlations (corr.) between instances in student and teacher separately, then transfer the teacher's correlations to student. Our DIST proposes to maintain the inter-class and intra-class relations between student and teacher. Inter-class relation: correlation between the predicted probabilistic distributions on each instance of teacher and student. Intra-class relation: correlation of the probabilities of all the instances each class.
The outputs of ResNet- 18 do not change much with the stronger strategy compared to ResNet-50. This implies that the representational capacity limits the student's performance, and it tends to be fairly challenging for the student to exactly match the teacher's 's outputs as their discrepancy becomes larger.
When the teacher and student models are trained with a stronger strategy, the discrepancy between teacher and student would be larger. This indicates that when adopt KDD with stronger training strategy, the misalignment between KD loss and classification loss would be severer, thus disturbing the student's 's training.
As result, the exact match (i.e., the loss reaches the minimal if and only if the teacher and student outputs are exactly identical) with KL divergence seems way too overambitious and demanding since the discrepancy between student and teacher can be considerably huge. Since the exact match can be detrimental with a stronger teacher, our intuition is to develop a relaxed manner for matching the predictions between the teacher and student.
DIST: Distillation from A Stronger Teacher
3.1 Relaxed match with relations
The prediction scores indicate the teacher's confidence (or preference) all classes. For relaxed match of predictions between the teacher and student, we are motivated to consider what we really care about for the teacher's output. Instead of the exact probabilistic values, actually, during inference, we are only concerned about their relations, i.e., relative ranks of predictions of teacher.
In this way, for some metric d(:,.) with RC RC - RR+, the exact match can be formulated that d(a, b) = 0 a for any two prediction vector as f,: Then as a relaxed match, we can introduce additional mappings o(-) and (-)-) with EC + RC such that
(( and y((1) in the KL divergence of Eq,
d()(a), (b)) = d(a, b), Va, ta,b
Therefore, d(a, b) = 0 does not necessarily require a and b should be exactly the same. Nevertheless, since we care about the relation within or b, the mappings Φ and vi should be isotone and do not affect the semantic information and inference result of the prediction vector.

---

With this regard, simple yet effective choice for the isotone mapping is the positive linear transfor- mation, namely,
d(m1a a + n1, m2b + n2) d(a.b) b),
(5)
where m,, m2, R1, and 72 are constants with my X mg > 0. As a result, this match could be invariant under separate changes in scale and shift for the predictions. Actually, to satisfy the property Eq.(5), we can thus adopt the widely-used Pearson's 's distance as the metric, Le.,
dp(u, v) == Pp(u, v).
(6)
(p(u, v) the Pearson correlation coefficient between two random variables and v.
Cov(u, v)
E (w - () - 8) E (u: 8)2 Eim 5)
Pp(u, w) = Std(u))td(v))
where Cov(u, v) is the covariance of and v, ù and Std(u) denote the mean and standard derivation of u, respectively.
In this way, we can define the relation as correlation. More specifically, and the original exact match in vanilla KD [16] can thus relaxed and replaced by maximizing the linear correlation preserve the relation of teacher and student on the probabilistic distribution of each instance, which we call
inter-class relation. Formally, for each pair of prediction vector Y and Y (t). the inter-relation loss can be formulated as
Linter
E dp
B
Some isotone mappings metrics can also be used to relax the match as Eq.(4), such as cosine similarity investigated empirically in Section 4.5; other more advanced and delicate choices could be left as future work.
3.2 Better distillation with intra-relations
Besides the inter-class relation, where we transfer the relation of multiple classes in each instance, the prediction scores of multiple instances in each class are also informative and useful. This scores indicate the similarities of multiple instances one class. For instance, suppose we have three images containing "cat"" "dog". and "plane", respectively, and they have three prediction scores on the 'cat' class, denoted as e, f. and g. Generally, the picture "cat" should have the largest score the "cat" class, while the "plane" should have the smallest score since it is inanimate. This relation of "e > g" could also be transferred the student. Besides, even for the images from the same class, the intrinsic intra-class variance of the semantic similarities is actually also informative. It indicates the prior from the teacher that which one is more reliable to cast this class.
Therefore, we also encourage to distill this intra-relation for better performance. Actually, define prediction matrix (+) and (t) with each row Y:(5) and s. then the above inter-relation is to maximize the correlation row-wisely (see Figure 3). In contrast, for intra-relation, the corresponding loss is thus maximize the correlation column-wisefe i.e.,
1 C E440 Y.
Lintra
As a result, the overall training loss Ler can be composed of the classification loss, inter-class KD loss, and intra-class KD loss, Le.,
Cu = aLola + 3Litee + Y
(10)
where 3, and are factors for balancing the losses. In this way, via the relation loss, we have endowed the student with freedom more or less to match the teacher network's output adaptively, thus boosting the distillation performance to a great extent.
5

---

Table 1: Training strategies image classification tasks. BS: batch size; LR: learning rate; WD: weight decay; LS: label smoothing; EMA: model exponential moving average; RA: RandAugment [9]; RE: random erasing; CJ: color jitter.
Total Tnitial
Stralegy
Dataset
Epochs
BS
LR
Optimizer
WD
LS EMA LR scheduler
Data sugmentation
A1
CIFAR-
100
240
64
0.05
SGD
x 10
x0.1 150, epochs crop+ flip
ImugeNet
100
256
0.1
SGD
1 x 10
x0.1 every 30 epochs
crop + flip
B2
ImagoNet
450
768
0,048
RMSProp
1 10 0.1 0.9999 x0.97 every 2.4 epochs
(BI) RA RE
ImageNet
AdamW
10 D.I. cosine
[82) + Mixup CutMix
Table 2: Evaluation results of baseline seftings on ImageNet. We use ResNet-34 and ResNet-50
released by Torchvision [28] as our teacher networks, and follow the standard training strategy (B)
Student (teacher)
Teacher Student
KD [16] OFD [14] CRD [41] SRRL [47] Review [7] DIST
Top-1
73.31
69.76
70.66
71.08
71.17
71.73
71.61
72.07
ResNet-18 18 (ResNet-34)
Top-5
91.42
89.08
89.88
90.07
90.13
90.60
90.51
90.42
Top-1
76.16
70.13
70.68
71.25
71.37
72.49
72.56
73.24
MobileNet (ResNet-50) 50)
T
92.86
89.49
90.30
90.34
90.41
90.92
91.00
91.12
Experiments
4.1 Experimental settings
Training strategies. The training strategies of image classification task are summarized in Table 1. CIFAR-100. For fair comparisons, we use the same training strategies (referred to Al in Table 1) and pretrained models following CRD [41]. ImageNet. B1: for comparisons with previous KD
methods, we train our baselines with the same simple training strategy as CRD [41]. B2: validate
the effectiveness of KD methods on modern training strategies, we follow EfficientNet [40] and design training strategy B2, which can significantly improve the performance compared B1. B3: the strategy B3 is used for training teiin-Tranffommrss [27], and contains even more stronger data augmentations and regularization.
Loss weights. On CIFAR-100 and ImageNet, we set a = 1, / = 2, and 7 = 2 in Eq.(10). On object detection and semantic segmentation, these three factors all equal to 1. For KD [16], we set a = 0.9, B = in E3)( and use a default temperature r = 4. Specifically, instead of using T = 1 on IetgeNet, we choose a larger temperature 4 CIFAR-100, is easy get overfit and the learned probabilistic distribution is sharp on CIFAR-100.
4.2 Image Classification
Baseline results on ImageNet. We first compare method with prior works using the baseline settings. As shown in Table 2, our DIST significantly outperforms prior KD methods. Note that method is only conducted on the outputs of models, and has similar computational cost as KD [16]. Nevertheless, even achieves better performance compared to those sophisticatedly- designed methods. For example, CRD [41] needs preserve a memory bank for all 128-d features of ImageNet images, and produces additional 260M FLOPPs of computation cost; SRRL [47] and Review [7] require additional convolutions for feature alignments. The implementation of DIST can be found in Appendix 1, which quite simple compared to these methods.
Distillation from stronger teacher models. As the stronger teachers come from larger model sizes and stronger strategies, we here first conduct experiments to compare our DIST with the vanilla KD on different scales (model sizes) of ResNets with baseline strategy B1. As shown in Table 3, when the teacher goes larger, the ResNet-18 18 students perform even worse than that with a medium ResNet-50 teacher. Nevertheless, our DIST shows an upward trend with larger teachers, and the improvements compared to KD also become more significant, indicating that our DIST tackles better on the large discrepancy between the student and larger teacher.
Distillation from stronger training strategies. Recently, the performance models on ImageNet has been significantly improved by the sophisticated training strategies and strong data augmentations

---

Table 3: Performance of ResNet-18 and ResNet-34 on ImageNet with different sizes of teachers.
Top-1
ACC (%)
Student
Teacher
student
teacher
KD
DIST
ResNet-34
73.31
71.21
72.07 (+0.86)
ResNet--50
76.13
71.35
72.12 (+0.77)
ResNet-18
69.76
ResNet-100
77.37
71.09
72.08 (+0.99)
ResNet-112
78.31
71.12
72.24 (+1.12)
ResNet-50
76.13
74.73
75.06 (+0.33)
ResNet-34
ResNet-101
73.31
77.37
74.89
75.36 (+0.47)
ResNet-155
78.31
74.87
75.42 (+0.55)
Table 4: Performance of students trained with strong strategies on ImageNet. The Swin-T is trained with strategy B3 in Table 1, others are trained with B2. f: trained by [44]. f: Pretrained on ImageNet-22K.
Top-1 ACC (%)
Teacher
Student
teacher student
KD [16]
RKD [30]
SRRL [47]
DIST
ResNet-18
73.4
72.6
72.9
71.2
74.5
ResNet-34
76.8
77.2
76.6
76.7
77.8
ResNet-500
80.1
MobileNetV2
73.6
71.7
73.1
69.2
74.4
Efficien(Net-BO
78.0
77.4
77.5
77.3
78.6
ResNet-50
78.5
80.0
78.9
78.6
80.2
Swin-L
86.3
Swin-T
81.3
81.5
81.2
81.5
82.3
(e.g., TIMM [44] achieves 80.4% accuracy on ResNet-50 while the baseline strategy BI only obtains 76. 1%). However, most of the KD methods still conduct experiments with simple training settings. It seldomly investigated whether the KD methods are suitable to the advanced strategies. this way, we conduct experiments with advanced training strategies and compare method with vanilla KD, instance relation-basee RKD [30], and SRRL [47].
We first train traditional CNNs with strong strategies, and also use strong ResNet-50 with 80. 1% accuracy trained by [44] the teacher. results shown Table 4, on both similar architectures (ResNet-18, ResNet-34) and dissimilar architectures (MobileNetV2, EfficientNet-Bol can achieve the best performance. Note that RKD and SRRL can perform worse than training from scratch, especially when the students are small (ResNet- 18 and MobileNet) or the architectures of teacher and student are fairly different (ResNet-50 and Swin-L), this might be because they focus on the intermediate features, which can be more challenging for the student to recover teacher's features compared to predictions.
our DIST
Furthermore, we experiment on the recent state-of-the-an art Swin- Toin-Trraffrmerr [27]. The results show that our DIST gains improvements on even more stronger models and strategies. For example, with Swin-L teacher, our method improves ResNet-50 and Swin-T by 1.7% and 1.0%, respectively.
CIFAR-100.
The results CIFAR-100 dataset in Table show that, by distilling the predicted logits, our method even outperforms those sophisticatedly--dsiieee feature distillation methods.
4.3 Object Detection
We further investigate the effectiveness of DIST on downstream tasks. We conduct experiments MS COCO object detection dataset [25], and simply leverage our DIST as an additional supervision on the final predictions of classes. Following [37, 52], use the same standard training strategies and utilize Cascade Mask R-CNN [2] with ResNeXt- 101 backbone as the teacher for two-stage student of Faster R-CNN [23] with ResNet-50 backbone; while for one-stage RetinaNet [24] with ResNet-50 backbone, the RetinaNet with ResNeXt- 101 backbone is utilized as the teacher.
As shown in Table 6, our DIST achieves competitive results on COCO validation set. For comparisons, we train the vanilla KD under the same settings as our DIST, the results show that our DIST significantly outperforms vanilla KD by simply replacing the loss functions. Morcover, by combining
7

---

Table 5: Evaluation results CIFAR-100 dataset. The upper and lower models denote teacher and student, respectively.
Same architecture style
Different architecture style
Method
WRN-40-2
ResNet-56
ResNet-32x4 ResNet-8x4
ResNet-50
ResNet-32x4 ShuffleNetVi
ResNet-32x4
WRN-40-1
ResNet-20
MobileNetV2
ShuffleNetV2
Teacher
75.61
72.34
79.42
79.34
79.42
79,42
Student
71.98
69.06
72.50
64.6
70.5
71.82
Feature-
methods
FitNet [35]
72.24+0.24
69.21=0.36
73.50=0.28
63.16::0.47
73.59+0.15
73.54::0.22
VID [1]
73.30+0..13
70.38+0.14
73.09+0.21
67.57:4.28
73.08
73.40
RKD [30]
72.22:10.20
69.61
71.90=0.11
64.43:10.42
72.28:10.39
73.21
PKT [31]
73,45:00.19
70.34::.00
73.64-0.18
66.52:10.33
74.10:10.25
74.6910..4
CRD [41]
74.14+0.22 71.16=0.17
75.51
69.11
75.11
75.65+0.100
Logits-based methods
KD [16]
73.54+0.20
70.66:0.22
73.33+0.25
67.35+0.32
74.07=0.19
74.45=0.27
DIST
74.73+0.24 71.75+00.0 76.31+0.19
68.66++0.23
76.34+0.18 77.35++.25
Table 6: Results on COCO validation set. T: teacher; S: Table 7: Results on Cityscapes val
student. *: We implement KD using and other settings dataset. All models are pretrained on are the same as DIST.
ImageNet.
Method
AP APso APTo
APs
APM
APL
Method
mloU (%)
Two-stage detectors
DeepLabV3-R101
78.07
T: Cascado Mask RCNN-X101
45.6 64.1
49.7
26.2
49.6
60.0
S: Deepl.abV3-R118
74.21
S: Faster RCNN-R50
38.4 59.0
42.0
21.5
42.1
50.3
SKD [26]
75.42
KD [16]*
39.7 61.2
43.0
23.2
43.3
51.7
IFVD [43]
75.59
FKD [52]
41.5 62.2
45.1
23.5
45.0
55.3
CWD [37]
75.55
CWD [37]
41.7 62.0
45.5
23.3
45.5
55.5
DIST
43.8
44.6
CIRKD [46]
76.38
DIST mimic
41.8 62.4 45.6
23.4
46.1
55.0
DIST
77.10
One-stage detectors
S: PSPNet-R18
72.55
T: RetinaNet-X101
41.0 60.9 44.0
23.9
45.2
54.0
SKD [26]
73.29
S: RetinaNettR500
37.4 56.7
39.6
20.0
40.7
49.7
IFVD [43]
73.71
KD [16]"
37.2 56.5
39.3
20.4
40.4
49.5
CWD [37]
74.36
FKD [52]
39.6 58.8
42.1
22.7
43.3
52.5
CIRKD [46]
74.73
CWD [37]
40.8 60.4
43.4
22.7
44.5
55.3
DIST
76.31
DIST
42.5
22.0
43.7
53.0
DIST mimic
40.1 59.4
43.0
23.2
44.0
53.6
DIST with mimic, which minimizes the mean square error between FPN features of teacher and student, we can even outperform the state-of-the-art KD methods designed for object detection.
4.4 Semantic Segmentation
We also perform experiments on semantic segmentation, challenging dense prediction task. Follow- ing [37,4] 43, 46], we train DeepLabV3 [6] and PSPNet [54] with ResNet- 18 backbone on Cityscapes dataset, and adopt our DIST on the predictions of classification head using teacher with ResNet- 101 backbone of D33 As results summarized in Table 7, with only the supervision of class predictions, our DIST can significantly outperform existing knowledge distillation methods on semantic segmentation task. For example, our DIST outperforms recent state-of state-of-the-art method CIRKD [46] by 1.58% on PSPNet-R18. This demonstrate our effectiveness on relation modeling.

---

4.5
Ablation studies
Effects of inter-class and intra-class correlations. This paper proposes two types relations: inter- class and intra-class relations. To validate the effectiveness of each relation, we conduct experiments train students with these relations separately. The results on Table 8 verify that, both inter-class and intra-class relations can outperform the vanilla KD; also, the performance could be further boosted by combining them together.
Table 8: Ablation of inter-class and intra-class relations on ImageNet. The student and teacher models ResNet-18 and ResNet-34, respectively.
Method
Inter
Intra
ACC (%)
KD
71.21
DIST (KL div.)
x
70.61
DIST (KL div.)
V
V
71.62
DIST
V
X
71.63
DIST
x
71.55
DIST
72.07
Effect of intra-class relation in vanilla KD To investigate the effectiveness of intra-class relation in vanilla KD, adopt experiments train DIST using KL divergence the relation metric, denoted as DIST (KL div.)". As the results summarized Table 8, adding intra-class relation in the vanilla KD also improve the performance (from 71.21% % 71.62%). However, when the student is trained with intra-class relation only, the improvement of using KL divergence is less significant than using Pearson correlation (70.61 % vs. 71.55%), since the means and variances of intra-class distributions could be varied.
Effect of training students with KD loss only. Training student with only the KD loss can better reflect the distillation ability and the information richness of supervision signals. As results in Table show that, when the student is trained with only the KD loss, our DIST significantly outperforms the vanilla KD. Without using the ground-truth labels, it can even outperform the standalone training accuracy, which indicates the effectiveness of our DIST in distilling those bell-beenfiical relations.
Table Comparisons of training KD with or without the classification loss on ImageNet. The student and teacher models are ResNet-18 and ResNet-34, respectively. The original accuracy of ResNet- 18 without KD is 69. 69.76%.
Method
w/ cls. loss
w/o cls. loss
KD
71.21
68.12
DIST
72.07
70.65
More ablation studies can be found in Section A.3.
5
Conclusion
This paper presents a new knowledge distillation (KD) method named DIST to implement better distillation from stronger teacher. We empirically study the catastrophic discrepancy problem between the student and stronger teacher, and propose relation-based loss to relax the exact match of KL divergence in a linear sense. Our method DIST is simple yet effective in handling strong teachers. Extensive experiments show our superiority in various benchmark tasks. For example, DIST even outperforms state-of-the-ee KD methods designed specifically for object detection and semantic segmentation.
Acknowledgements
This work was supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize.
SSpecifically, the vanilla KD is the same DIST (KL div.) with inter-class relation only.

---

References
[1] S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
[2] Z. Cai and N. Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. JEEE Transactions on Pattern Analysis and Machine Intelligence, page 1-1, 2019. 1.7
[3] K. Chandrasegaran, N.-T. Tran, Y. ZHAO, and N. man Cheung. To smooth or not to smooth? on compatibility between label smoothing and knowledge distillation, 2022. 14
[4] H. Chen, Y. Wang. C. Xu, C. Xu, and D. Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems, 32(1):25-335 33, 2020. 13
[5] L.-C. Chen, Papandreou. F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 1766.00000000 05587, 2017.
[6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[7] P. Chen, S. Liu, H. Zhao, and J. Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5008-5017 2021.
[8] J. H. Cho and B. Hariharan, On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF
international conference computer vision, pages 4794 4802, 2019. 2
[9] E. D. Cubuk, B. Zoph. J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition Workshops, pages 702-703, 2020. 6,6
[10] Y. Dodge. The concise encyclopedia of statistics. Springer Science Business Media, 2008. 13
[11] S. Du, S. You, X. Li. J. Wu, F. Wang. C. Qian, and C. Zhang. Agree disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33:12345- 12355, 2020.
[12] J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6):1789 1819, 2021.
[13] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7700-77. 2016. 1
[14] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 1921-1930 1930, 2019.
1,
[15] B. Heo, M. Lec, S. Yun, and J. Y. Choi. Knowledge transfer via distillation activation boundarics formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779 3787, 2019.
[16] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255 1503.02533 2015. 1, 6, 7,8, 14
[17] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
arXiv: 170400/70.04666 2017. 1
[18] T. Huang. Z. Li, H. Lu, Y. Shan, S. Yang, Y. Feng, F. Wang. S. You, and Xu. Relational surrogate loss learning. International Conference on Learning Representations, 2022.
[19] T. Huang, S. You, B. Zhang, Y. Du, F. Wang, C. Qian, and C. Xu. Dyrep: Bootstrapping training with dynamic re-parameterization. arXiv preprint arXiv:2203.1 /2268 2022.
[20] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938. 13
[21] Kim, S. Park, and N. Kwak. Paraphrasing complex network: Network compression via factor transfer. arXiv preprint 1rXiv:1802.04999 04977. 2018.
[22] S. Kong, T. Guo, S. You, and C. Xu. Learning student networks with few data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4469-4476, 2020.
[23] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2117-2125, 2017. 7,7
[24] T.-Y. Lin. P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980 2988, 2017." 7
10

---

[25] T.-Y. Lin. M. Maire, S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 7
[26] Y. Liu, C. Shu, J. Wang, and C. Shen. Structured knowledge distillation for dense prediction. IEEE transactions pattern analysis and machine intelligence, 2020. 8
127] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint ar arXiv:2103. /4030, 2021. 3, 6, 7
[28] S. Marcel and Y. Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages
2010. 6
[29] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh. Improved knowl- edge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence.
volume 34, pages 5191 5191-5198, 2020. 2
[30] W. Park, D. Kim, Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 3967-3976, 2019. 7, 8, 13,
[31] N. Passalis, M. Tzelepi, and A. Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEEE Transactions on Neurul Networks and Learning Systems, 32(5):2030-200 2039, 2020. 8
[32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury. G. Chanan, Killeen, Z. Lin, N. Gimelshein, I.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information proessing systems, 32:8026-8037, 2019. 13
[33] K. Pearson. Vii. mathematical contributions to the theory of evolution.-iii regression, heredity, and panmixia. Philosophical Transactions of the Royal Sociery of London. Series A, containing papers of mathematical or physical character, (187):253-315 1896. 2
[34] B. Peng. X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 5007-5016, 2019. 13
[35] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep
nets. arXiv preprint arXiv:1412.6555 2014. 8
[36] Z. Shen, Z. Liu, D. Xu, Z. Chen, K.-T. Cheng, and M. Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In International Conference on Learning Representations. 2020. 14
[37] C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5311-5320, 2021. 7,8
[38] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409: 1556, 2014.
[39] W. Son, J. Na, J. Choi, and W. Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9395-
9404, 2021. 2
[40] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Interna- tional Conference on Machine Learning, pages 6105-61 6611.1111 PMLR, 2019. 6
[41] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. International Conference on
Learning Representations, 2019. 1, 6, 8, 14
[42] F. Tung and G. Mori.
knowledge distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
[43] Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Computer Vision, pages 346-362. Springer, 2020. 8
[44] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.0044
2021. 7
[45] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[46] C. Yang, H. Zhou, 7. An, X. Jiang, Y. Xu, and Q. Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint 2rXiv:2204.069M6 2022. 8
[47] J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos. Knowledge distillation via softmax regression
representation learning. International Conference on Learning Representations, 2020. 6, 7, 14
[48] S. You, T. Huang, M. Yang. F. Wang. C. Qian, and C, Zhang. Greedynas: Towards fast one-shot nas with greedy supemet.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 1999- 2009920088220 2020.
11

---

[49] S. You, C. Xu, C. Xu, and D. Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1285-1294, 2017.
[50] S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:/6/2.003928, 2016.
[51] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 3,
[52] L. Zhang and K. Ma. Improve object detection with reature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations. 2020. 7,
[53] X. Zhang. X. Zhou, M. Lin, and J. Sun. Shufflenct: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6848-6856, 2018. 1
[54] H. Zhao, Shi, X. Qi, X. Wang, and J. Jia, Pyramid scene parsing network. Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 2881-2890, 2017. 1,8
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Training details are provided in the paper. Training code and logs are released at GitHub: https://gttuu..com// Stwpriuubb.commuunoooDSS.......T KD.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Standard deviations on CIFAR-I00 are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code and training logs are included.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "p_jsNkiK56z", "reviewer": "Reviewer_FMUS", "summary": "This paper proposes a new knowledge distillation method that can use a stronger teacher to make a better student. It proposes a correlation-based loss at the inter-class level and intra- class level. The distance between the teacher predictions and the student predictions is computed using the Pearson correlation coefficient instead of KL divergence. Experiments show the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "The authors have discussed the limitations and societal impacts of their works. The proposed method cannot fully address the binary classification tasks.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1)\tExisting KD methods perform worse when the teacher becomes stronger, while the proposed method can use a stronger teacher to train a better student.\n2)\tExperimental results verify the effectiveness.\n\n\nWeaknesses:\n1)\tOne of the key components is the matching metric, namely, the Pearson correlation coefficient (PCC). However, the assumption that PCC is a more relaxed constraint compared with KL divergence because of its invariance to scale and shift is not convincing enough. The constraint strength of a loss function is defined via its gradient distribution. For example, KL divergence and MSE loss have the same optimal solution while MSE loss is stricter than KL because of stricter punishment according to its gradient distribution. From this perspective, it is necessary to provide the gradient comparison between KL and PCC.\n2)\tThe experiments are not sufficient enough.\n2-1) There are limited types of teacher architectures.\n2-2) Most compared methods are proposed before 2019 (see Tab. 5).\n2-3) The compared methods are not sufficient in Tab. 3 and 4.\n2-4) The overall performance comparisons are only conducted on the small-scale dataset (i.e., CIFAR100). Large datasets (e.g., ImageNet) should also be evaluated.\n2-5) The performance improvement compared with SOTAs is marginal (see Tab. 5). Some students only have a 0.06% gain compared with CRD.\n3)\tThere are some typos and some improper presentations. The texts of the figure are too small, especially the texts in Fig.2. Some typos, such as “on each classes” in the caption of Fig. 3, should be corrected.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZhqLZBa7Nv", "reviewer": "Reviewer_RvKW", "summary": "The authors come up with a loss for KD that encourages not only the correlation of relative differences in scores of the class predictions for an image, but also the correlation of the relative differences of scores of the images for a class. They claim that this sort of loss improves KD from strong teacher models and they show improvements on ImageNet, COCO, Cityscapes etc. ", "questions": "Some intuition behind why their row and column wise correlation based loss works better to distill knowledge would be nice, It was a bit hard to follow the dog, cat, airplane example. ", "limitations": "Some qualitative examples and quantitative analysis on what fails to be distilled from the strong teachers would also be nice to have. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- They show improvements on various benchmarks. \n- Their proposed loss is straight forward to understand. \n\nWeaknesses/Concerns:\n- It is not clear to me how their described intuition in lines 142 to 148 relate to the column-wise loss where try to correlate the relative difference in scores per class. \n- Some relevant literature references missing: https://arxiv.org/abs/1910.01348, https://arxiv.org/abs/1902.03393  \n- The title seems overly generic and doesn't describe the gist of the paper. KD is obviously done from a stronger teacher, that's not novel. The main novelty is the row and column wise loss. \n- The teachers are also not that strong by current standards. They use a Resnet 18 and 50. Nowadays we have ViT's for image recongnition and other stronger models than resnet 18. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8MpHlTusmme", "reviewer": "Reviewer_xqDf", "summary": "This paper raise a overlooked question in knowledge distillation: how to distill from a strong teacher, i.e., how to conduct knowledge distillation when the discrepancy between the teacher and student models is large. This paper conducted an empirical study and found that existing KD methods may fail when distilling from a strong teacher. This paper further proposed to use the Pearson correlation coefficient as a new match manner to replace the KL divergence.", "questions": "Please see weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n\\+ The proposed research question, how to distill from a strong teacher, is important but overlooked problem for knowledge distillation, which has a great potential in real-world applications. The research problem is novel and practical.\n\n\\+ This paper conducted a comprehensive empirical study on how previous KD methods fail when the discrepancy between the teacher and student models is large. \n\n\\+ This paper proposed to use the Pearson correlation coefficient as a new match manner to replace the KL divergence. The proposed method is simple, efficient, and practical.\n\n\\+ The paper conducted extensive experiments on various benchmark datasets, including image classification, object detection, and semantic segmentation. \n\n\\+ The paper is well written and organized.\n\nWeaknesses:\n\n\\- The compared methods in Table 5 are not state-of-the-art. Some recent works (see References) are not compared.\n\n\\- The font size in Figures 1 and 2 is too small. It would be great if these two figures can be reorganized in the future version.\n\nReferences:\n\n[R1] Knowledge Distillation meets Self-supervision, ECCV 2020.\n[R2] Densely Guided Knowledge Distillation using Multiple Teacher Assistants, ICCV 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4oqLcxHDOv-", "reviewer": "Reviewer_kJN1", "summary": "This paper investigates the topic of learning from a stronger teacher in KD. The authors show that using KL divergence in KD may not perform well when distilling from stronger teachers, and propose a new KD method called DIST to only preserve the relations between the teacher and student outputs instead of exactly matching in KL divergence. Numerical results on different tasks are provided to show the superiority of DIST on baseline and stronger teacher settings.", "questions": "1. The authors adopt Pearson correlation as the \"relation\" metric in DIST, will the performance also be significant when having other relation metrics?", "limitations": "Yes, the authors adequately discussed the limitations of this paper. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem of learning from a stronger teacher is interesting and worth to investigate. Many previous works have shown that the KD performs poorly when the capacity gap between teacher and student is large. This paper extends this topic to \"stronger teacher\", where the \"stronger\" denotes larger capacity or stronger training strategy, and provides a unified solution.\n\n2. The proposed method is simple and effective. The authors empirically find that the discrepancy between the student and a stronger teacher becomes larger, and therefore adopt correlation coefficient to alleviate this discrepancy. The method is intuitively sound and well supported by the experiments.\n\n3. The improvements compared to KD is significant according to the numerous experiments on various tasks.\n\nWeaknesses:\n1. DIST is less effective on larger student networks. For example, in Table 4, DIST achieves 1.9% improvement on ResNet18 compared to KD, but only improves ResNet34 and ResNet50 by 0.6% and 0.2%.\n\n2. The performance of prediction-based distillation method is limited on dense prediction tasks (e.g., detection) compared to feature-based methods, as the feature contains more localization information.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept