Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer
Authors: Lujun Li
ZHE JIN
Venue: NeurIPS 2022 Accept (2022)
Keywords: Knowledge Distillation

Abstract: Knowledge distillation can be generally divided into offline and online categories according to whether teacher model is pre-trained and persistent during the distillation process.  Offline distillation can employ existing models yet always demonstrates inferior performance than online ones.  In this paper, we first empirically show that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion.  Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher to adapt student with such reversed distillation.  However, this fine-tuning process still costs lots of training budgets.  To alleviate this dilemma, we propose SHAKE, a simple yet effective SHAdow KnowlEdge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency.  Specifically, we build an extra shadow head on the backbone to mimic the predictions of pre-trained teacher as its shadow.  Then, this shadow head is leveraged as a proxy teacher to perform bidirectional distillation with student on the fly.  In this way, SHAKE not only updates this student-aware proxy teacher with the knowledge of pre-trained model, but also greatly optimizes costs of augmented reversed distillation.  Extensive experiments on classification and object detection tasks demonstrate that our technique achieves state-of-the-art results with different CNNs and Vision Transformer models.  Additionally, our method shows strong compatibility with multi-teacher and augmentation strategies by gaining additional performance improvement.  Code is made publicly available at https://lilujunai.github.io/SHAKE/.

Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer
Lujun Li Jin Zhe
School of Artificial Intelligence, Anhui University, China Chinese Academy of Science, China
1i Ailujunai@@gaa 1. com; jinzhe@ahu. edu
Abstract
Knowledge distillation can be generally divided into offline and online categories cording to whether teacher model is pre-trained and persistent during the distillation process. Offline distillation inferior performance than online ones. In this paper, we first empirically show that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion. Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher adapt student with such reversed distillation. However, this fine-tuning process still costs lots of training budgets. To alleviate this dilemma, we propose SHAKE, simple yet effective SHAdow KnowlEdge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency. Specifically, we build an extra shadow head on the backbone to mimic the predictions of pre-trained teacher as its shadow. Then, this shadow head is leveraged as a proxy teacher to perform bidirectional distillation with student on the fly. In this way. SHAKE not only updates this student-aware proxy teacher with the knowledge of pre-trained model, but also greatly optimizes costs of augmented reversed distillation. Extensive experiments on classification and object detection tasks demonstrate that our tech- nique achieves state-of-ther -fht--rt- results with different CNNs and Vision Transformer models. Additionally, our method shows strong compatibility with multi-teacher and augmentation strategies by gaining additional performance improvement. Code is made publicly available at https://liljjunai gittub.io/H...K...
employ existing models yet always demonstrates
Introduction
Even though Deep Neural Networks (DNNs) have great success in tackling a variety of challenges
over computer vision 27 13 and natural language processing (2), they usually have large numbers of
parameters, bringing heavy computation costs. To alleviate this problem, many network compression methods [29 46 6 9 have been proposed, among which Knowledge Distillation (KD) [19 58 14 has recently attracted increased attention.
KD aims to transfer knowledge from high- gigh-capacity large model (i.e., teacher) to a low- cww-capactty lightweight model (i.e,, student). Numerous offline methods [9] use two-stage training process that begins with training teacher model and then keeping it fixed to distill student model (see KD in (a)) Besides offline fashion, recent online methods (88] 5 adopt a one-stage training process, jointly training the student and teacher/pper models using bidirectional distillation like DML. [66] in Figure (b)) These online distillations always surpass the offline distillations under the same teacher model. However, some large teacher models trained from scratch would bring some difficulties (e.g., high computational resources and unstable optimizations) for distillation, especially for tasks that rely on large transformers (e.g., BERT [57), GPT-3 2 and ViT-MoE (8)) Therefore, two questions
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Frozen
Tuned
Tusk head
Forward
Backward
KD
CE
cat
Cut
T
S
(D) Figure 1: Illustration of (a) KD, (b) DML, (c) i without K Dss+r, and (d) KD; with DDs-rr
(A)
(B)
(C)
cut
KDs-T denotes reversed distillation from student to teacher.
Table 1: Left: comparison of training time, Top-1 accuracy (%), and teacher-student gap (T-S
gap) among the (a) KD, (b) DMI., (c) DML. without K KDs--.: (d) KDj with K DD5-T, and our SHAKE for ResNet-20 (6900%)) via pre-trained teacher (Pre-T) ResNet-110 (74.31%) on CIFAR- 100. Training time is measured on single 2080Ti GPU, and x represents the improving ratios than KD. The teacher-student gap 8 is defined KL divergence between their outputs (lower is better). Right: training time and accuracy of different settings for KD and SHAKE. KD} (head) means only updating head, and KDj (adapter) performed by adding adaptation layers (Conv-BNY (env-BN--eLU) for output of each stage of teacher and fine-tuning adaptation layers. SHAKE (share) denotes the proxy 1 settings (1]
teacher shares the backbone. SHAKE (multi) denotes SHAKE via
Method
Pre-T KD
KD
Time
Top-1
T-S gap
Method
Top 1
T.S gap
(A) KD
x1.00
70.66
1.12
(F) KDJ (head)
71.05
(B) DMI.
/
/
x4.32
71.52
0.38
(G) KDf (adapter)
71.34
0.92
(C) DMLf
x4.41
70.55
0.82
(H) SHAKE (alone)
71.82
(D) KDf
V
x4.29
71.76
0.66
(I) SHAKE (share)
72.02
0.51
(G) SHAKE
x1.28
72.02
0.51
(J) SHAKE (multi)
72.35
naturally arise: (1) Why is there performance gap between offline and online distillation? (2) How the performance of offline distillation can be advanced with design techniques of online KD methods?
To clarify the first question, we compare KD and DML regarding the training fashion of teacher and distillation loss in T Contrary to the common belief, we empirically observe that training fashion may not affect the distillation performance since DMLÍ obtains similar performance KI) (70.55% vs. 70.66%). Instead, the reversed distillation from the student model yields significant accuracy gains for KDj than KD (71.76% vs. 70.66%) and DML than DML (71.52% vs. 70.55%).
Then, we analyze the output discrepancy of the teacher-student for different methods, and the reversed distillation reduces the gap from 1.12 of KD to 0.66 of KDt. Therefore, the main reasons behind the performance gap lie in two aspects: (a) Teacher models in conventional offline pipelines are optimized for the student model. Thus, they could only provide general knowledge, which may be suboptimal for the particular student. (b) The reversed distillation changes the universal teacher model into a student-aware one and bridges the reacher-student gap, which clearly justifies:
Teachers should teach students in accordance with their aptitude and should not follow the same pattern.
For the second question, fine-tuning the pre- pre-trained teacher model with reversed distillation is straightforward way to bridge the two training fashions. However, in most scenarios. fine-tuning the whole network still needs lots of training budges (more 4x costs than KD in Table Some trade-offss in reducing the fine-tuning parameters or adapter-based methods (see Figure 2 (G)) involve performance loss. Thus, how to augment reversed distillation without much extra overhead is an important issue for the application. Besides fine-tuning, building proxy teacher model (see Figure 2 (H)) to inherit the knowledge of pre-trained models and receive reversed distillation from students also enjoys the same benefits. As shown in Table (H), this proxy teacher model (SHAKE (alone)) can achieve competitive gains than fine-tuning the whole teacher model (KD+). Recent weight-
sharing strategies in AutoML [62 43 21 9 effectively save training overhead and preserve logit

---

Frozen
Tuned
Fi
Praxy Tencher
Adupter
Shadoww
Forward
Backward
KD
CE
T
T
T
PT
S
S
S
S
cut
cat
(G)
(II)
cat
(J)
cat
Figure 2: Evolution of our SHAdow KnowlEdge distillation (SHAKE). (G) KDf with an additional adaptation layer for teacher fine-tuning. (H) We build a proxy teacher model to inherit knowledge from pre- trained models an alternative costly fine-tuning teacher models. (I) This proxy teacher model could reuse the backbone to reduce training budgets and uses the shadow head to perform bidirectional KD with the student. (J) Our SHAKE for multiple teachers: SHAKE leverages multiple shadow heads to individually follow various teacher models.
consistency. This encourages to adopt the same student architecture and weight-sharing strategy to generate the proxy teacher model. Thus. allow this proxy teacher share the backbone but use an individual shadow head to preserve the diversity of logits representations (see Figure(((()) (I)). As shown Table (I), this sharing strategy in SHAKE (share) presents four benefits: (1) more than 3x training acceleration than DML and fine-tuning the whole teacher model. (2) No need for architecture selection cost for proxy teacher. (3) Additional accuracy gains than individual proxy teacher because knowledge inheritance also directly improves the representation in the weight-sharing process.
Based on the above observations, propose SHAdow KnowlEdge distillation (SHAKE), novel and effective logits distillation framework. Our SHAKE builds proxy teacher model and updates its weights via the original teacher model predictions. In this way, SHAKE enjoys the same benefits with teacher fine-tuning to mine the knowledge of the pre-trained model and can perform bidirectional supervision with the student. To optimize training costs, this proxy teacher shares the backbone with the pre-trained teacher but leverages an individual head to preserve the diversity of logits representations. This head is named shadow head since it imitates the original teacher model just like its shadow. During training, we only need to additionally train this shadow head with small training budgets (only 1.28x costs than KD). After training, all teacher models and shadow module can be deprecated, and student model can be separately applied for inference without any overhead. Moreover, we extend SHAKE to multi-teacher model scenarios using multiple shadow heads to inherit diverse knowledge.
In principle. SHAKE alters the chain of knowledge transfer from pre-trained teacher-sstudens in KD to pre-trained teacher-proxxy teacher=student. Other adaptive KD 40) only employs middle networks to distill as large for students and requires multi-step training. The merits of SHAKE lie in three-fold. First, it effectively reduces the mncher-studem. capability gap with reversed distillation, bringing significant gains when pre-trained teacher models are available. Second, for the scenario without pre-trained teacher models, SHAKE also enables the offline KD methods to be more effective in alleviating unstable optimization issues of online KD methods. Thus, SHAKE bridges the offline and online KD methods and enjoys the advantages of both methods. Third, SHAKE achieves favorable trade-offs between accuracy and training budget. By contrast, other adaptive KD 440 needs to sequentially train multiple models with lots of additional training time and resources. We hope that these intriguing observations in SHAKE would expand the application of KD and facilitate future research for KD work to some extent.
teacher +student, which not optimized
We conduct extensive experiments on multiple tasks classiiasiiiiotoion and detection) and datasets (e.g., CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and MS-COCO) to verify the superiority of the proposed method. SHAKE achieves consistent and significant accuracy boost in various neural networks and data augmentations, which outperforms other methods by large margins. For example, SHAKE obtains 2.92% 6.75% accuracy gains than baseline and more than 3x

---

training acceleration than DML on CIFAR-100. On the challenging ImageNet dataset, SHAKE can augment the performance ResNet-18 from 69.75% 72.07% and MobileNet from 70.13% to 72.66%, which are state-of-the-arr results for KD techniques. On vision transformer architecture [55] our approach achieves 75.22% Top-1 accuracy and 2.76% gain for training VIT-T model. the object detection task, SHAKE improves the AP by 1.02 for Faster R-CNN on the MS-COCO, demonstrating its generality.
In summary, we make the following principle contributions in this paper:
By analyzing and exploring the difference between offline and online KD methods, we empirically show that the reversed distillation hinders the performance gain, which fixes the discrepancy between teacher-student capability. This motivates us to propose new SHAdow KnowlEdge distillation (SHAKE) framework to bridge two training fashions, which, to the best of our knowledge, not achieved throughout the area of knowledge distillation.
SHAKE achieves remarkable trade-offs between accuracy and training efficiency with extra shadow head. The shadow head inherits knowledge from pre-trained models, introduces reversed distillation with students, and accelerates the training process.
We perform thorough evaluations on classification and detection. SHAKE achieves state-of- the-art performances in various datasets and architectures (e.g., CNN and vision transformer). Specifically, ResNet-18, MobileNet, and VIT-T with SHAKE achieve 72.07%, 72.66% and 75.22% Top-1 accuracy on ImageNet, outperforming KD by 1.41%, 1.98% and 3.02%, respectively.
Shadow Knowledge Distillation
In this section, we first revisit the offline and online KD methods. Then, present the formulation of SHAKE and understand why SHAKE work from the perspective optimization properties and teacher-student similarity. Finally, present the formulation of SHAKE for multi-teacher models.
The evolution of our approach is shown Figure
2.1 Revisit offline and online KD methods
We first review the formulations of offline and online KD. For simplicity, We choose two typical frameworks (i.e., original KD [9 and DML (66)) for analysis. Regarding training data (X, containing training samples X and labels Y. Let fr be the output logits of the fixed teacher 7 and let fs be the output of student S, respectively. KD equips the student network fs via minimizing:
Cs LCr(sc Y) (Ssr
where À is used to balance these two terms. LCE the regular cross-entropy objective:
Loelss, Y) = o
where is the softmax function and CE(,,) is the cross-entropy loss. LxL in Eq. Iis the distillation goal for transmitting knowledge from a teacher to a student:
Lxt (fs. fT) = KK ( (0( (
(3)
where T is the temperature to generate soft labels and KL represents Kullback-Leibler (KL) diver- gence. The teacher's 's probabilistic outputs can be viewed as the soft labels in this distillation loss rather than the one-hot ground-true -tuth labels, making it modified cross-entropy loss.
DML provides a two-way knowledge transfer technique in contrast to KD's 's one-way distillation. The probabilistic outputs from both teacher and student networks be used to direct one another's training. DML interleaves two aims to jointly train the teacher and student networks in an end-to-end manner:
L Lce(fe, Y) Ffx
Cs = Lorlss Y) ACRI(fs.fT).

---

where the default value of X is 1 in DML, DML consistently outperforms KD under the same teacher- student pair. The common belief argues that the accuracy gap between DML and KD comes from the additional boost of teacher models under mutual training fashion in DML than fixed teachers in KD. However, the significant gains of KDf than KD challenge this belief because the teacher's accuracy of KDj does not obviously improve, but its teacher-student gap reduces, shown in Table [ In summary, introducing reversed distillation a promising way to realize student-adapted KD methods. We illustrate how SHAKE implements it in cost-friendly way in the following section.
2.2 Formulation of SHAKE
As shown in Figure 2 (I), SHAKE proposes a proxy teacher model with output fr via shadow head to augment reverse distillation in the offline framework. In particular, the hierarchical features (HT; .H.,....Hrr of the pre-trained teacher model are transformed by feature adaptation layer A and then performed via a bottom-up feature fusion. For example, for the three-layer feature fusion, the proxy teacher's output fr can be defined as:
(5) where M shadoww is the fully connected layer with the same shape as the pre- pre-trained teacher. The shadow feature fusion scheme consists of directly feature addition @ and Conv-BN-Relle adaptation layer A. In the backpropagation, pre-tra pre-trained teachers are fixed, and only shadow modules updated. Such shadow modules simple to implement, and its additional costs are similar to other feature distillations [25] and parameter efficient fine-tuning strategies (23]. To achieve better
ST = IT [| shadow (H)) Ao(HTz)) @ As(HTs)) global global-poolee poolinggoolillll mnrmm
feature
logit alignment, we perform normalization Iloorm and scaling for the output of the pre-trained
teacher, proxy teacher, and student model, which follows advanced distillation [52 28 31.
During the training process, ST is updated with the output of the pre-trained teacher model fr and performs mutual distillation with the student models fs as:
CT (kI(fr, fr) + ACKI(fr,fs) Ls = LCE(fs, Y) + (CKL(fs.fr)),
(6)
Each loss item has the same form as KD in Eq. D and the effects of X are explored in the ablation study. The inherit distillation loss term D( (T) is presented to imitate the probabilistic outputs of pre-trained teacher, which can be defined as KL or MSE distance. The reverse distillation term from student feedback to the proxy teacher and the direct distillation from the proxy teacher to the student play key roles in our framework, which is analyzed in detail from the perspective of training dynamics and teacher-studen similarity in the following.
Comparison of SHAKE with other KDs in training dynam- ics. Eq. 6. Eq. 4 and Eq. cllearly illustrate the difference of SHAKE and KD/DML. Compared KD, SHAKE introduces reversed distillation so that the teacher can be optimized by the student. In contrast DML, SHAKE leverages the knowledge of pre-trained models, resulting in additional accuracy gains. As shown Figure 3 SHAKE achieves a robust boost than KD and baseline during the training process. Meanwhile, the training curve of DML shows a highly dynamic oscillation due to unreliable predictions of its teacher trained from scratch. Other adaptive KD methods use the same knowledge transfer way as KD and do not optimize teacher with feedback super- vision of student. Thus, these methods, including ATKD [40] and ESKD [8] do not essentially address the teacher-student capacity disparity. and our SHAKE surpasses them with sig- nificant margins (1.18% 1.29% on ImageNet in Figure 3).
Figure 3: Comparison of train- ing curves of baseline (69.75%) KD (70.66%) ATKD (70.78%).
ESKD (70.89%). DML (71.13%) and our SHAKE (72.07%) for ResNet-18 with single R4sNet-34 as teacher on ImageNet.
In-depth analysis about teacher-student similarity in SHAKE, The similarity between student and teacher network is an important measure for KD tasks. Reversed distillation optimizes the teacher model to adapt to the student, reducing the teacher-student gap and improving distillation efficiency. With reversed KD loss, SHAKE surpasses KD and DML with small teacher-student discrepancy. To verify this insight, we employ Kl.-divergence as metrics of similarity [8) where lower -i-diveree implics higher similarity. Figure 4 presents the similarities and performances between the outputs of
5

---

Twedine
SHAKE
Top- mocan (ed)) axcersty (%6
11.5
KRMAT
PMB SHAKE
KI Diverrrrece
6.5
831
R44
R56
R109
Tencher midells
Teacher models
Figure 4: KI-divergence and Top-1 accuracy (%)
Figure 5: Grad-CAM++ visualization.
Figure 6: The penultimate layer visualization of Re (student) with KD (left), SHAKE (middle) and the teacher (right) CIFAR-100.
Figure 7: Logits correlation visualization of teacher-student for the student (ResNet-20) with KD (right) and SHAKE (left) CIFAR-100.
ResNet with different depths teachers (e.g.. ResNet-32, ResNet-110) and student (e.g., ResNet-20), The results show that the distillation from SHAKE always gives a higher similarity than KD and DML, resulting in significant gains under teacher models of different depths. Figure S presents the Grad-CAM visualization of ResNet-18 18 with different methods on ImageNet. The Grad-CAM maps of SHAKE pay
attention the important regions than KD and baseline, which have high similarity with the teacher. In summary, SHAKE bridges KD and DML. to obtain student-friendly distillations, improving teacher-student similarity. In addition, Figure 6 illustrates that comparing with KD, applying SHAKE training helps learn more scattered embeddings logits correlation visualization in Figure clearly demonstrate that teacher model in SHAKE owns student-aware logits.
41 [3 39 3 and
2.3 Extension multiple teacher models
As shown in Figure 2 (J), we build several proxy teachers with outputs (ST; JT; .r..ffr to follow multiple teacher models with various shadow heads. Mutual distillation also exists between the task head of the student and these multiple shadow heads. Similar to Eq. the total optimization function for the multi-teacher scenario can be defined as:
outputs -T, JT;
N
N
Lr E CKL(fT;
(KI(TT;
1=1
Ls Lorlss. Y) + E LxulJs, STy).
where N is the sum of the total teachers. These multiple teacher models and shadows are present only the training phase and do not bring extra overhead for inference. As shown Setting (J) of T this multi-shadow head strategy yields significant accuracy gains with small additional overhead. Furthermore, SHAKE can combined with advanced multi-teacher methods (e.g..
adaptive weights [5 for different teachers in Table
3 Experiments
In this part, we first assess our SHAKE for classification and object detection. After that, extensive ablations conducted to investigate the key our design in SHAKE. As a novel offline logit approach,

---

the main competitor of SHAKE is the original KD [9)] Thus, we perform detailed experimental comparisons between them. Moreover, we compare the performance with recent advanced KD methods. We employ the open-source versions of these methods with the same training settings for fair comparisons. We choose and be and 4 in SHAKE, respectively.
Table 2: Comparison of performances with powerful distillation techniques using the same 240 training epochs. Most results of other methods reference the CRD [54]. SHAKE1 denotes SHAKE via multi-teacher settings [ST]. W40-2, R32x4, R8x4, R50, MV2, SV1 and SV2 stand for WRN-40-2, ResNet32x4, ResNet8x4, ResNet50, MobileNetV2, ShuffleNetVi and ShuffleNetV2.
Same architectural style
Different architectural style
Teacher
W40-2
R56
R110
R32x4
VGG13
VGG13
R50
W40-2
Student
W16-2
R20
R20
R8x4
VGG8
MV2
VGG8
SV1
Teacher
75.61
72.34
74.31
79.42
74.64
74.64
79.34
75.61
Student
73.26
69.06
69.06
72.50
70.36
64.60
70.36
70.50
FitNets 49
73.58
69.21
68.99
73.50
71.02
64.14
70.69
73.73
SP |66
73.83
69.67
70.04
72.94
72.68
66.30
73.34
74.52
RKD
73.35
69.61
69.25
71.90
71.48
64.52
71.50
72.21
CRD
75.48
71.16
71.46
75.51
73.94
69.73
74.30
76.05
Review WilewKD
76.12
71.89
N/A
75.63
N/A
N/A
N/A
77.14
NORM 66]
75.65
71.35
71.55
76.49
73.95
68.94
N/A
77.06
SRRI. 22
75.49
70.86
70.78
75.71
73.31
N/A
N/A
74.18
Tf-FD 30
74.33
N/A
70.62
73.62
71.62
N/A
N/A
N/A
ONE 228
74.68
70.43
70.77
N/A
72.01
66.26
74.35
KDCL [6
74.25
70.58
70.36
74.03
71.26
65.76
73.03
74.79
MetaDistil 68
N/A
71.25
71.40
N/A
73.65
N/A
74.42
77.06
DML [66
75.33
71.48
71.52
74.30
73.64
68.52
74.22
75.58
DML
74.83
70.24
70.55
73.15
72.86
66.30
73.34
74.52
KD [19
74.92
70.66
70.66
73.33
72.98
67.37
73.81
74.83
(ours)))
75.78
71.62
71.76
74.91
73.85
68.81
74.10
76.42
(HAKE(ours)
76.62
72.04
72.02
77.35
74.84
70.03
74.76
77.25
Std & Gain
0.34/3.36
0.12/2.98
0.08/2.96
0.28/4.85
0,38/4.48
0,28/5.43
0,28/4.40
0,28/6.75
KD+FitNets
75.12
71.12
71.24
74.66
73.49
67.73
73.91
77.42
SHAKE+FitNets
76.91
72.00
72.15
78.06
74.78
70.38
75.27
78.04
KD+CRD
75.89
70.90
71.60
75.46
74.08
69.94
74.22
76.27
SHAKE+CRD
77.17
72.29
71.87
76.57
74.65
70.04
75.22
77.61
KD+Mixup
75.28
71.66
71.33
75.20
74.07
67.31
73.91
76.49
SHAKE+Mixup
76.91
71.82
72.07
77.39
75.53
70.25
75.66
78.17
KD+CutMix
75.66
70.90
70.69
75.39
74.78
66.39
75.04
77.44
SHAKE+CutMix
76.29
70.92
70.90
78.28
75.11
69.44
75.98
78.27
KD+AVER
75.22
71.08
71.24
74.99
74.90
68.91
73.26
76.30
SHAKE+AVER
76.82
72.28
72.22
78.59
75.60
70.35
75.51
77.52
KD+AEKD
75.68
71.25
71.36
74.75
74.75
68.39
73.11
76.34
SHAKE+AEKD
76.88
72.32
72.35
78.90
76.26
70.42
75.67
77.60
3.1 Experiments CIFAR-100
Implementation. With CRD's settings [54]. whose training epochs are 240, we perform experiments several teacher-student models CIFAR-100, either using the same architecture style different one. We employ conventional SGD optimizer with a weight decay 0.0005 and a mini-batch size of 64. Initialized 0.05, the multi-step learning rate decrements by 0.1 at 150, 180, and 210 epochs.
Comparison with offline and online KD methods. In Table 2 we compare our approach to some advanced offline/online methods with the same training settings. of the teacher-students, SHAKE obtains 3.35% 5.45% absolute accuracy gains and outperforms KD with 0.26% 2.44% margins and Review with 1.72 margins. Besides, SHAKE achieves more significant gains cross-ares aros-trchiee teacher-student pairs with 4.70% 6.75% margins than
same architecture style

---

Table 3: Top-1 and Top-5 std accuracies (%) on ImageNet dataset. Results of Teacher (T)
Student (S), KD 19). ESKD 8 ATKD [40]. ONE 28
DML [66]. CRD
AT
64 RKD
and OFD 18
reproduced refers to CRD
54
S
Acc
KD
ESKD
ATKD
ONE
DML
CRD
SHAKE
SHAKET
Top-1
73,40
69.75
70.66
70.89
70.78
70.55
71.03
71.17
72.07
72153
R34
RI8
Top-5
91.42
89.07
89.88
90.06
89.99
89.59
90.28
90.32
91.05:011
91.26:+a25
T
S
Acc
T
S
KD
AT
RKD
OFD
DML
CRD
SHAKE
SHAKET
Top-1
76.16
70.13
70.68
70.72
71.32
71.25
71.13
71.40
72.66
73.02
R50
MV1
Top-5
92.86
89.49
90.30
90.03
90.62
90.34
90.22
90.42
91.35
91.62
Table 4: Top-1 mtd accuracies (%) for VIT-T with SHAKE on ImageNet dataset.
Teacher
Student
TY-KD
KD
Hurd KD
DeiT
55
SHAKE
SHAKE!
RegNetY-16GF (82.90)
VIT-T (72.20) 72.35
72.20
74.30
74.50
75.22
75.72
baseline. Compared to other KD methods, SHAKE outperforms KD with 0.66% ~ 1.22% margins, which illustrates the effectiveness of SHAKE in reducing the teacher-student network gap. Compared to DML and DMLI under the same teacher-student pair in Table 2 SHAKE obtains 1.79% ~ 4.20%
relative gains. Compare to online KDs (e.g.. KDCL [16]. ONE (28) with multiple branches as teacher models, SHAKE also obtains 0.71% ~ 3.77% relative accuracy gains. These significant
improvements demonstrate the superiority of SHAKE by leveraging the knowledge of pre-trained models.
Orthogonal to other KDs and data augmentations. As acting the output logits, SHAKE is orthogonal to feature and relation KD methods because of transferring different knowledge. As shown in Table the combination of FitNets with SHAKE surpasses KD with 0.91% 2.65% margins. In addition, for CRD, its combination with SHAKE yields more dramatic gains than KD. For distillation training under strong data augmentation (e.g., Mixup and CutMix (50). SHAKE obtains 0.21% % ~ 3.05% relative accuracy gains than KD.
Extension to multiple teacher models. In Table 2 compare SHAKE and KD with average and adaptive weighting in multiple teacher settings. The results (i.e., 0.75% ~ 4.05% gains than KD) illustrate that the design of SHAKE with the multi-proxy proxy model can effectively inherit the knowledge of different teachers bringing significant performance gains.
3.2 Experiments on ImageNet
Implementation. For the standard ResNet-18 [17] and MobileNet [20] models, we employ the same training configurations, whose the training epochs are 100, most distillation techniques. The learning rate is initialized at 0.1, decreasing by 0.1 every 30 epochs. Recent vision transformers achieve great success on different vision tasks [I0 (5). We also extend SHAKE to VIT-T (55) with the
same training settings (e.g., data augmentation and distillation token) on ImageNet. Implementation details are available in supplementary materials.
Comparison results. Table 3 reports the performance of our approach on ImageNet. SHAKE improves the baseline models of ResNet-18 with 2.32% gains Top-1 accuracy (see Figure 3 for detailed accuracy curves) and MobileNet with 2.53% gains. Compared to other KD methods, SHAKE outperforms KD with 1.41% 1.98% margins and CRD with 0.90% 1.26% margins, which supports the superiority of SHAKE on the large-scale dataset. Equipped with the distillation of two teacher models, SHAKEt obtains 2.78% gain for ResNet-18 and 2.89% gain for MobileNet than baseline. As shown in Table SHAKE obtains 2.76% accuracy gains than baseline. It surpasses KD under the soft or hard label of the CNN teacher (RegNet (55). verifying its effectiveness on different architectures.
8

---

Bisee SEaSE
Figure 8: Left: results for ResNet-22 as Student [S] with ResNet- 10 as Teacher [T] on CIFAR- 10. Middle: results for ResNet-20 [Student] with ResNet- [Teacher] on Tiny-ImageNet. Right: results of Faster R-CNN-R50 [Student] via Faster R-CNN-R101 [Teacher] on MS-COCO.
41%
enc
34%
37%
405
WRN-1662
ResNet-20
ResNet-8x4
VGG-8
Figure 9: Relative gain ratios in SHAKE of reversed distillation (blue part), shadow designs (yellow part), inherited distillation from pre-trained teacher (green part) and distillation to student (red part) for different models in Table 2 on CIFAR-100.
3.3 Extensive experiments in diverse datasets and tasks
Experiments on CIFAR-10 and Tmag-ImageNet. In addition to the commonly evaluated classifica- tion datasets (i.e., CIFAR-100 and ImageNet) in knowledge distillation methods, we also evaluated SHAKE in CIFAR-I0 and Tmny-lmageNet, which have smaller categories and data volumes. As shown in Figure & our intriguing observation about Reversed distillation is also present on different data sets: KDt with reversed KD outperforms KD, DML. and KDt. In addition. with a carefully designed shadow module and distillation strategy, our SHAKE outperforms KDf and achieves best results.
Experiments on object detection. Besides classification tasks, we also extend SHAKE for object detection applications. In particular, we evaluate SHAKE on MS-COCO dataset 35 and use the most popular open-source Detectron2 [34] as the strong baseline. We apply SHAKE to two-stage detector (i.e., Faster R-CNN (47). which are widely used object detection frameworks. Following common practice 34 7. all models are trained with a 2x learning schedule (24 epochs). As shown Figure 8 our SHAKE improves the AP 1.02 on Faster R-CNN, which outperforms KD [19] The performance of object detection greatly depends on the quality of deep features to locate interesting objects, while logits are not capable of providing knowledge for object localization. Thus. SHAKE is naturally weaker than Review (7] (recent excellent feature-based KDs), and we further introduce Review obtain satisfactory results. can be observed that SHAKE can obtain new best results with feature-basse KDs. The success of challenging object detection tasks indicates the generality of our approach and orthogonality for the det-distillers [59 65
3.4 Ablation study
Detailed ablation study of SHAKE As shown in Figure 9 an ablation study has been conducted to demonstrate the individual effectiveness of different components in SHAKE. It is observed that (a) Reversed distillation optimizes the teacher model to adapt the student, improving distillation efficiency. SHAKE, without reversed distillation, only obtains marginal gains. (b) Shadow modules effectively preserve the diversity of knowledge, and their backbone-shand sharing settings can accelerate the student model's 's convergence and result in performance gains. In addition, the stronger shadow module design for storing and teaching more pre-trained knowledge when the capacity gap enlarges. (c) Inherited knowledge from pre-trained teacher via distillation optimization beyond sharing weights important. Proxy teachers in our SHAKE transfer the knowledge the original pre-trained teacher model more efficiently. These results demonstrate that our proxy teacher design can indeed serve as

---

TWP Laved (ND) (ND 05)
ToP mon ont ACCEAN
Top- DUER (MD IN SECEEAC
seee (MR) CECEECEEE
14.5
18.5
76
13
SHARE
SHAKE
SMARE
140
0.5
0.4
1L5
6.2
0.1
0.6
0.5
0.4
0.3
0.2
18] WRN-16-2 (T) WRN-40-2
[S] VGG-S & (T) VGG-13
[S] WRN-16-2
& WIN-44-2
[S]
VGG- & (T) VGG-13
Figure 10: Top-1 mean (std) accuracies (%) of varying for WRN-16-2 & VGG-8 (left) and varying À for WRN-16-2 & VGG-8 (right) CIFAR-100.
good bridge between the original teacher and student. Reverse KD reduces the teacher-student gap to facilitate knowledge transfer.
Sensitivity study for temperature and weight A. Eq. 3 controls the softness teacher's supervision. As increases, the output of the softmax function becomes smoother. As shown in Figure 10 SHAKE presents superior gains and robustness than the original KD in different т. balances the implications of KD loss. The small À limits distillation gains, and À 0.5 is the best option for SHAKE.
4 Related Work
Offiine distillations. In the offline framework [19), the teacher pre-trained and non-updated, and then its soft logits are used as extra supervision to distill students. Although subsequent methods explored to transfer feature knowledges [49 64] and relation knowledges [42 54)), the effective original KD still outperforms most distillations and is widely used for different tasks 12 26 R1 [2] However, the pre-trained teacher model is not optimized for the student, limiting its gains. address this issue, our SHAKE enhances KD using shadow head to introduce optimization from the student.
Online distillations. Online KD methods simplify the KD process by training all models simultane- ously. DMI. (66] performs bidirectional distillation for the peer networks and ONE (2 presents an on-the-fly ensemble distillation among multiple branches. Subsequent studies focus on how to balance multiple teacher [5] or construct the online teacher [60
33
38
33 Online KD methods always
obtain better performance than offline ones. However, large teacher models trained from scratch sometimes perform unstable predictions and bring lots of training costs during distillation. Therefore, we propose that SHAKE combine the advantages of the two pipelines to facilitate application.
Adaptive distillations. Capacity gaps between teacher-student models for their different architectures would limit distillation gains 33.. There are two types of existing works to alleviate this gap terms of training paradigms [II] and architectural adaptation [24] [5]. For instance, ESKD [ proposes stopping training of the teacher early, and ATKD 40 uses a medium-sizz teacher assistant to
perform sequence distillation with large overheads. However, these methods do not optimize the
teacher model for the student, resulting in minor benefits. Recent Meta-KD KD [68 [77 36 implement
quite complex meta-optimization for whole teacher networks with lot of extra overhead. In sharp contrast to these methods, SHAKE is a student-aware offline KD method and opens new direction for adaptive distillation design.
5 Conclusion
In this paper, we present SHAKE, new student-aware logit distillation that is easy to use and effective for bridging offline and online knowledge transfer. Based on our insight into online KD methods success, SHAKE achieves this goal by building an extra shadow head as a proxy teacher model to perform mutual distillation with the student model. Thorough evaluations are performed on classification and detection, and SHAKE achieves significant performance gains in various neural networks without extra inference overheads. future work, we would make the best effort to explore the application of SHAKE for different specific tasks [4]) We hope that this elegant and practical approach would inspire future research on knowle snowledge distillation design and understanding.
10

---

References
[1] Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NeurIPS, 2014.
[2] Tom B. Brown, Benjamin Mann, Nick Ryderand Melanie Subbiah. Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M, Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint, arXiv:2005. 14165, 2020.
[3] Keshigeyan Chandrasegaran, Ngoe-Trung Tran, Yunqing Zhao, and Ngai-Man Cheung. Revisit- ing label smoothing and knowledge distillation compatibility: What was missing? In ICML, 2022.
[4] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam+++: Generalized gradient-basee based visual explanations for deep convolutional networks. In WACV, 2018.
[5] Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In AAAI, 2020.
[6] Kunlong Chen, Liu Yang, Yitian Chen, Kunjin Chen, Yidan Xu, and Lujun Li. Gp-nas-ensemble: a model for the nas performance prediction. 2022.
[7] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021.
[8] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[9] Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan. Prior-guided one-shot neural architecture search. arXiv preprint vrXiv:2206.13222 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 1xXiv:0010.119299 2020.
[11) Mengya Gao, Yujun Shen, Quanquan Li, Junjie Yan, Liang Wan, Dahua Lin, Chen Change Loy, and Xiaoou Tang. An embarrassingly simple approach for knowledge distillation. arXiv preprint
2018.
[12] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In ECCV, 2018.
[13] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014.
[14] Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. wwXiv:2006.05555. 2020.
[15] Jindong Gu and Volker Tresp. Search for better students to learn distilled knowledge. arXiv preprint arXiv:2001.11611 2020.
[16] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. CVPR, 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehensive overhaul of feature distillation. In ICCV, 2019.
11

---

[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1OX3::5503.025:: 2015.
[20] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint, arXiv: 04861, 2017.
[21] Yiming Hu, Xingang Wang. Lujun Li, and Qingyi Gu. Improving one-shot nas with shrinking- and-expanding supernet. Pattern Recognition, 2021.
[22] Adrian Bulat Georgios Tzimiropoulos Jing Yang, Brais Martinez. Knowledge distillation via softmax regression representation learning. ICLR2021, 2021.
[23] Rao Jun, Meng Xv, Ding Liang. Qi Shuhan, and Tao Dacheng. Parameter-effiiiet and student- friendly knowledge distillation. Stti:2205515306 2022.
[24] Minsoo Kang, Jonghwan Mun, and Bohyung Han. Towards oracle knowledge distillation with neural architecture search. In AAAI, 2020.
[25] Jangho Kim, ScoungUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NeurlPS, 2018.
[26] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP, 2016.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[28] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In NeurlPS, 2018.
[29] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
[30] Lujun Li. Self-regulated feature learning via teacher-free feature distillation. In European Conference on Computer Vision (ECCV), 2022.
[31] Lujun Li. Liang Shiuan-Ni NNi, Ya Yang, and Zhe Jin. Boosting online feature transfer via separable feature fusion. In IJCNN, 2022.
[32] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin. Teacher-free distillation via regularizing intermediate representation. In IJCNN, 2022.
[33] Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He. Explicit connection distillation. 2020.
[34] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He,, and Piotr Dollar. Focal loss for dense object detection. In ICCV, 2017.
[35] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[36] Yitao Liu, Tianxiang Sun, Xipeng Qiu, and Xuanjing Huang. Learning to teach with student feedback. arXiv:2109.0444 046444 2022.
[37] Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In CVPR, 2020.
[38] Zhiqiang Liu, Yanxia Liu, and Chengkai Huang. arXiv:2111.11747 2022.
Semi-online knowledge distillation.
[39] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In ICML, 2020.
[40] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
12

---

[41] Rafael Muller, Simon Korbliihh, and Geoffrey E Hinton. When does label smoothing help? NeurIPS, 2019.
[42] Wonpyo Park, Yan Lu, Minsu Cho, and Dongju Kim. Relational knowledge distillation. In CVPR, 2019.
[43] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In JCML, 2018.
[44] Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, and Xingang Wang. Activation modulation and recalibration scheme for weakly supervised semantic segmentation. In AAAI, 2022.
[45] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Design- ing network design spaces. In CVPR, 2020.
[46] Mohammad Rastegari, Vicente Ordonez. Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
[47] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-tima time object detection with region proposal networks. arXiv preprint, arXiv:1506.01499 2015.
[48] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, An- dre Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. arXiv preprint
2021.
[49] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[50] Seong Joon Oh Sanghyuk Chun Junsuk Choe Youngjoon Yoo Sangdoo Yun, Dongyoon Han. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
[51] Xiaojie Li Jianlong Wu Fei Wang Chen Qian Shangchen Du, Shan You and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. In NeurIPS, 2020.
[52] Chengchao Shen, Xinchao Wang. Youtan Yin, Jie Song. Sihui Luo, and Mingli Song. Progressive network grafting for few-shot knowledge distillation. In AAAI, 2021.
[53] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study, arXiv preprint 2xXiv:2/0400061: 2021.
[54] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[55] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp: distillation through attention. In ICML, 2021.
[56] Frederick Tung and Greg Mori. Similarity-ppeservvin knowledge distillation. In ICCV, 2019.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit. Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[58] Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: review and new outlooks. 7-PAMI, 2021.
[59] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In CVPR, 2019.
[60] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. arXiv preprint arXiv:2006.044 04147, 2020.
[61] Liu Xiaolong, Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. 2020.
13

---

[62] Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. CVPR, 2019.
[63] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020.
[64] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
[65] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In ICIR, 2020.
[66] Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018.
[67] Wangchunshu Zhou, Canwen Xu, and Julian McAuley. BERT learns to teach: Knowledge distillation with meta learning. In ACI., 2022.
[68] Wangchunshu Zhou, Canwen Xu, and Julian McAuley, Meta learning for knowledge distillation. In ACL, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. Our method will add additional training overhead.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our approach is new knowledge distillation application technique, which helps lightweight models to achieve better performance without additional teacher architectures. It does not bring negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No]
(b) Did you include complete proofs of all theoretical results? [No]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material implementation details in our experiment Section 3.22 and supplemental materials.
URL.)? [Yes] See
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Table and more results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Table
4. If you using existing assets (c.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We evaluate our approach in generic public dataset, following lots of previous research works.
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [N/A] The datasets we use does not contain personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Our approach is generic training framework, which does not concern human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "Zxp1HuzvRU", "reviewer": "Reviewer_Xeu9", "summary": "The major contribution of this paper is:\n1) This paper proposes a new Knowledge Distillation / Transfer scheme called **SHAKE** (**SH**adow **K**nowl**E**dge) to bridge offline and online knowledge distillation / transfer schemes. The proposed method achieves noticeable improvements in image classification (CIFAR-100, ImageNet) tasks.", "questions": "Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:\n\n1) Can the authors include Top-5 accuracies for Tables 2, 3?\n2) I believe this is not a theoretical paper, not sure why the answer is [Yes] for questions in Checklist (2). \n", "limitations": "Briefly mentioned in Checklist 1(C).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1) This paper is written clearly. It is easy to follow.\n2) The proposed SHAKE framework obtains good improvements on image classification (CIFAR-100, ImageNet) benchmarks.\n\n**Weaknesses:**\n1) My major concern is that this paper lacks analysis. This paper talks about student-aware logits, but no analysis / visualization / quantitative results are provided to add merit to “student-aware” logits claim except the final KD accuracy. Given that this is purely an empirical paper, I’m not convinced regarding the student-aware logits claim. I’m happy to see that the proposed method obtains improvement, but this paper can significantly benefit from good analysis (There are ablation experiments done for CIFAR-100 which I feel is not about student-aware logits). Some useful analysis for “student-aware” logits:\n- Given the proposed method shares the backbone and uses different heads for teacher and student, penultimate layer visualization / analysis could be useful (See [1, 2, 3, 4]) to understand the logits learnt by the teacher/ student using SHAKE. \n\n2) Object detection procedure lacks technical details: \n- For Object detection, classification is treated as a multi-label classification problem rather than a multi-class classification problem. Is KL divergence used for multi-label distillation for Object detection?\n- How did the authors combine ground truth labels for bounding box, objectness score with soft-label predictions from the teacher? I.e.: If the teacher predicts 10% probability for an incorrect class (so no bounding box information in ground-truth), how did the authors perform KD subsequently? Can the authors provide loss function details?\n\n\n**Minor concerns:**\n1) Figure 6 is not an attention map: Given a sample and target class (usually ground truth), it shows pixel-level visual explanations. Showing the probabilities for the correct class across all 5 methods for these 3 samples could be more informative to understand Figure 6 (you can still get pixel-level explanations for wrong predictions, See [5]).\n\nEmpirical results of this paper are interesting. But given the lack of analysis, I feel that the weaknesses of this paper outweigh the strengths. I’m happy to change my opinion based on the rebuttal. \n\n\nReferences\n\n[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. \"When does label smoothing help?.\" Advances in neural information processing systems 32 (2019).\n\n[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR\n\n\n[3] Lukasik, M., Bhojanapalli, S., Menon, A., & Kumar, S. (2020, November). Does label smoothing mitigate label noise?. In International Conference on Machine Learning (pp. 6448-6458). PMLR.\n\n[4] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. In International Conference on Machine Learning\n\n[5] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TqXuobTIlta", "reviewer": "Reviewer_txDD", "summary": "This paper shows that the merits of online knowledge distillation (DML) over the offline knowledge distillation come from the fact that the teacher model are optimized for the student model. Then, to reduce the training overhead of fine-tuning the teacher in online knowledge distillation, they propose SHAKE which distills knowledge from the teacher to the proxy teacher offline, and then perform DML between the proxy teacher and the student. Experiments on both detection and image classification are provided to demonstrate their effectiveness. ", "questions": "Please refer to the weakness.", "limitations": "n/a", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n1. I like the study in Line 37-49, which gives a surprising explanation of the effectiveness of online knowledge distillation.\n2. The authors have given very sufficient studies on both classification and object detection, with the combination of the other KD methods, and different data augmentation settings.\n3. Good paper writing.\n\nWeakness.\n1. About the motivation. In Line44-49, the authors claim that the teacher model in offline knowledge distillation does not work well because it has not been optimized for the student model. I can understand that this conclusion is drawn from the observations in Line41-43. However, even in if DML and the other offline knowledge distillation methods, the \"teacher model\" is trained to minimize the cross-entropy loss between teacher prediction and the labels, and the KL (of feat loss) between teacher prediction and student predictions. In these situations, the teacher model is also not optimized for the student model. To make it easy to be understood, when the teacher in DML minimizes the knowledge distillation loss, it does not optimize the loss for the student, it just optimizes it for itself. Hence, the problem that why an offline teacher works worse is still not clear.\n\n2. In the proposed SHAKE, the teacher knowledge is firstly transferred to the proxy teacher, and then transferred to the student. Thus, it is similar to the teacher-assistant knowledge distillation (TAKD, AAAI2020)[1].  It will be better to discuss the difference and connection between SHAKE and TAKD.\n\n3. One of the motivations of SHAKE is to reduce the fine-tuning overhead of the pretrained teacher. Recently, adapter-based methods have been proposed to efficiently fine-tune a pretrained model. It will be better to compare SHAKE with using these adapter methods for teacher fine-tuning.\n\n4. In the detection experiments in Table 4., SHAKE+Review leads to 0.31 AP improvements than Review, which is not significant.  Besides, both KD (logit) and Review are not knowledge distillation methods for object detection. It will be better if results with detection-oriented knowledge distillation methods, such as [2-3].\n\n\n[1] Improved knowledge distillation via teacher assistant. AAAI2020\n\n[2] Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.ICLR2020.\n\n[3] General instance distillation for object detection. CVPR2021.\n\n\nI am happy to increase my rating if my questions are answered in the rebuttal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6gkFITzBada", "reviewer": "Reviewer_zGhB", "summary": "Offline distillation can employ existing models yet demonstrates inferior performance than online ones. This paper first empirically shows that the essential factor for their performance gap lies in the reversed distillation from student to teacher, rather than the training fashion. Offline distillation can achieve competitive performance gain by fine-tuning pre-trained teacher to adapt student with such reversed distillation. However, this fine-tuning process still costs lots of training budgets. To alleviate this dilemma, this paper proposes SHAKE, a simple yet effective SHAdow KnowlEdge transfer framework to bridge offline and online distillation, which trades the accuracy with efficiency. ", "questions": "The conclusion from Table 1 is only based on one dataset. But the performance margin is not large. Could this be generalized?\n\nWhat is the exact meaning of shadow head? More explaination and why this name is used should be given.", "limitations": "Statistical tests to show the performance difference of different approaches are needed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method allows the proxy teacher to share the student’s backbone but use an individual shadow head to preserve the diversity of logits representations.\n\nSHAKE (share) presents three benefits: (1) more than 3× training acceleration than DML and fine-tuning whole teacher model. (2) No need for architecture selection costs for proxy teacher. (3) Additional accuracy gains than individual proxy teacher because knowledge inherited by proxy teachers also directly improves the representation of students’ backbone in the weight-sharing process.\n\n\nIn principle, SHAKE alters the chain of knowledge transfer from pre-trained teacher→student in\nKD to pre-trained teacher→proxy teacherstudent.\n\nI think the idea is interesting though there are some related works.\n\nThe conclusion from Table 1 is only based on one dataset. But the performance margin is not large. Could this be generalized?\n\nWhat is the exact meaning of shadow head? More explaination and why this name is used should be given.\n\nStatistical tests to show the performance difference of different approaches are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept