Adaptive Block-wise Learning for Knowledge Distillation
Authors: Tianyi Lei
Junyu Xie
Wang qian
Dezhong Peng
Xu Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge distillation
Local error signals
Bilevel optimization

Abstract: Knowledge distillation allows the student network to improve its performance under the supervision of transferred knowledge. Existing knowledge distillation methods are implemented under the  implicit hypothesis that knowledge from teacher and student contributes to each layer of the student network to the same extent. In this work, we argue that there should be different contributions of knowledge from the teacher and the student during training for each layer. Experimental results evidence this argument. To the end, we propose a novel Adaptive Block-wise Learning~(ABL) for Knowledge Distillation to automatically balance teacher-guided knowledge between self-knowledge in each block. Specifically, to solve the problem that the error backpropagation algorithm cannot assign weights to each block of the student network independently, we leverage the local error signals to approximate the global error signals on student objectives. Moreover, we utilize a set of meta variables to control the contribution of the student knowledge and teacher knowledge to each block during the training process. Finally, the extensive experiments prove the effectiveness of our method. Meanwhile, ABL provides an insightful view that in the shallow blocks, the weight of teacher guidance is greater, while in the deep blocks, student knowledge has more influence.

Under review as a conference paper at ICLR 2023
ADAPTIVE BLOCK-WISE LEARNING FOR KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation allows the student network to improve its performance under the supervision of transferred knowledge. Existing knowledge distilla- tion methods are implemented under the implicit hypothesis that knowledge from teacher and student contributes to each layer of the student network to the same extent. In this work, argue that there should be different contributions of knowledge from the teacher and the student during training for each layer. Exper- imental results evidence this argument. To the end, we propose novel Adaptive Block-wiss wise Learning (ABL) for Knowledge Distillation to automatically balance teacher-guides knowledge between self knowledge in each block. Specifically, solve the problem that the error backpropagation algorithm cannot assign weights each block of the student network independently, signals to approximate the global error signals on student objectives. Moreover, we utilize a of meta variables to control the contribution of the student knowl- edge and teacher knowledge to each block during the training process. Finally, the extensive experiments prove the effectiveness of our method. Meanwhile, ABL provides an insightful view that in the shallow blocks, the weight of teacher guid- ance is greater, while in the deep blocks, student knowledge has more influence.
leverage the local error
INTRODUCTION
Knowledge distillation (KD) in deep learning imitates the pattern of human learning. Hinton et al. (2015) proposes the original concept of KD, which minimizes the KI. divergence between the logits of teacher (soft label) and student. This allows KD to be expressed as a mode in which a complex pre-trained model is used as teacher to guide a lightweight student model learning. Following such teacher-student framework, a series KD methods are mainly developed in the following directions: what, where and how to distill. No matter what kind of directions, these existing KD methods are based on the same implicit hypothesis that in the training process of distillation, the contribution of student's 's and teacher's knowledge each layer of the student network is fixed, whether it is the last layer or the first layer. This is because after the error backpropagation (BP) (Rumelhart et al., 1986), the weight of the error signals each layer is determined by the same hyper-parameterss which is shown in Figure [aa Intuitively, this limits the flexibility of balancing the knowledge of teacher and student, which harms excavating the potentialities of the student model.
Therefore, we argue that in the representation learning process of student network guided by teacher knowledge, different layers of the student network have different emphases on the knowledge leamed through the one-hot labels and the knowledge distilled by the teacher. This means that some levels are more inclined to utilize student knowledge to learn while others tend to leverage teacher knowledge. Furthermore, we also argue that the contribution of student and teacher knowl- edge to representation learning should be adaptive at each level. However, the existing KD methods obtain the global error signal from the last layer, which is hard to allocate the hierarchical weights.
To explore the student network hicrarchically in the training process, we modify the backward com- putation graphs and leverage the local error signals by the family of local objectives (Jaderberg) et al. 2017 Nøkland & Eidnes] 2019 [elilovssy et al. 2020; Pyeon et al. (02T) to approximate the global error signals generated by the last layer. These local loss functions focus the local error signals and decoupled learning. Here, by leveraging auxiliary networks, we adopt the above
local strategies to acquire the approximation of the global emror signal created by the student loss

---

Under review as a conference paper at ICLR 2023
Logto
Legitehons
Uhe hott blbd
Stadent appls
Sent abd
One-bot thet
Testert
Soblzbel
Aucz
Block
Tinsage
Fustar wosehaaee
(b) Adaptive block wise learning for knowledge distillation
(a) Backpropagation process
of knowledge distillation
Comparison between the Backpropagation process of KD and adaptive block-wise learn- ing for KD, from the perspective of gradient. The distilled knowledge can be based logits or based on features. (a) The contributions of the knowledge from student and teacher are fixed and equal for different blocks. (b) The gradient flows are different for different blocks, and can be adaptively modified by meta variables
Figure 1:
objective corresponding to the one-hot labels. This provides the possibility to independently assign different weights to teacher knowledge and student knowledge at different layers.
After allowing the student error signals of each layer to obtain independently, the current issue is on exploring the balance between student knowledge and teacher knowledge at each layer. We model this issue bilevel optimization problem (Anandalingam & Friesz 1992) by adding set of meta variables on the error signals corresponding to two types of knowledge. These meta variables represent the influence of which target knowledge is preferred by the update of the corresponding layer. addition, after the bilevel optimization based on the gradient descent solution, we obtain the optimal meta variables of the target network under the target KID method and utilize them for the final evaluation.
To the end, propose novel paradigm dubbed Adaptive Block-wise Learning (ABL) for Knowl- edge Distillation, which allows conventional teacher-student architecture to explore the influence of knowledge from teacher and student in blocks. As shown in Figure [(b] the proposed method
changes the error signals acquisition path of student objective function from global to local and adds a group meta variables measure the contribution of knowledge from students and teach- ers. Furthermore, we acquire the balance between knowledge from the student and the teacher on the validation set. Besides, we leverage the optimized meta variables train the corresponding distillation method. Our main contributions are as follows:
1. We propose novel paradigm named adaptive block-wise learning for knowledge distil- lation, which automatically balances the contribution of knowledge from the student and teacher for each block.
2. We discover that the deep and abstract representation inclines to learn from student knowl- edge, while the shallow and less abstract representation tends be guided by teacher knowledge. We hope this discovery could provide another learning view for KD.
3. We conduct extensive experiments under eleven recent distillation benchmarks. Experi- mental results demonstrate the effectiveness of the proposed framework in improving the performance of existing distillation methods.
2 RELATED WORK
Knowledge distillation. Knowledge distillation usually transfers knowledge from large models to small models, which completed under the teacher-student framework. The vanilla KD is firstly proposed by Hinton al. (2015), which allows the student model to mimic the final prediction of the teacher model. Zhao et al. (2022) decouple the knowledge (binary probabilities) by the target knowledge and non-target knowledge. In addition the above logits-based KD, there
2

---

Under review as a conference paper at ICLR 2023
also many methods based features (Romero al. 2013 Komodakis & Zagoruyko 2017 Tung & Mori 2019 Peng et al. 2019 Park et al. 2019 Ahn et al. 2019 Tian et al.) 2020), which mainly aimed transferring the knowledge of intermediate features. In addition the above consideration of what kind of knowledge to transfer, SSng et al. (022) considers where to transfer knowledge, MMizadee al. (2020), Son al. (202T) and et al. (202T) explore how transfer knowledge. However, none of these focuses on an important question, i.e. how to balance the transferred knowledge and the knowledge from the student model, which is more directly reflected in the effect of the target student model.
Local error signals. In the field of decoupled neural networks, many researches are looking for an efficient way to replace the global error signals with local error signals. DNI (Jaderberg et al. 2017) utilizes the linear layer for each layer learn synthetic gradient by approximating the gradient of backpropagation. Nøkland & Eidnes 2019) obtains the local error signals with similarity matching and prediction local loss unlock the backward updating. DGL (Belilovsky at. 2020) designs the more efficient auxiliary networks MLP-SR to create the local error. SEDONA (Pyeon et al. 2021) adds the neural architecture search technique into greedy learning to obtain the more appropriate local signals meliorated by the global signals. Following these work, we approximate the global error signals of student model with the local error signals.
METHODS
We begin by describing the knowledge distillation from the perspective of gradient (Section We then introduce our proposed adaptive block-wise learning for knowledge distillation (Section B.2), where the global eπror signals are approximated with the local signals to realize changeable weight allocation of the knowledge from teacher and student. Finally, we optimize simultaneously the global error and the local error signals by adopting a bilevel optimization (Section 3.3
3.1 KNOWLEDGE DISTILLATION FROM THE PERSPECTIVE OF GRADIENT
Existing knowledge distillation methods can be divided into two parts: 1) One part is to learn by oneself through the student, 2) and the other is learn by distilling the teacher's knowledge, which can be expressed as:
(1) where Ls usually cross-entropy loss (De Boer et al. 2005) between the predicted probabilities from only the student model and the label, LKD can be the Kullback-Leibler divergence (Hershey) & Olsen 2007) vanilla knowledge distillation, fn-morm distance for the intermediate representa- tions in Fitnets (Romero et al. 2015), or any feature- distillation loses. a and 3 balance the contribution of Ls and LKD-
L = aLs BLKD,
With the above analysis, we rethink knowledge distillation from the perspective of gradient during the training of the student model. We now consider student networks consisting of L layers f()) e {1, 2,...... L}. each outputting h0O) through the parameters g(), Thus, the updating of the parameters with the gradient-based based update rule can be formulated
Oc (h
g(1) + 0(0)
1] ah 000)
OC is the backpropagated error gradients, which is usually Oh
where v represents the learning rate and
represented by 8 By Eq. []. the training process of KD which reflects in the error gradients of each layer in the student network can be written as:
80)) σ + 35 KD
(1)
(3)
d and ( denote the contributions of the student's knowledge and teacher's knowledge in the 1-th layer of the student network, respectively.
where
To facilitate our statements, we define (() and d as student error signals and teacher error signals respectively. Intuitively, matter how and 8 are set. the error gradients of each layer can be expressed by Eq. (3). This means that the contribution allocation of the two parts of each layer is fixed, which is shown Figure ((a)
3

---

Under review as a conference paper at ICLR 2023
Algorithm Bilevel optimization for Adaptive Block-wise Learning for Knowledge Distillation Initialize meta variables 7 as 0
Warm-start input network weights (0, (p) with Eq. and Eq.
8
1: while not converged do
2: Calculate (0", :p*) ) on training mini- batch with Eq. 7 and Eq.
8
3: Updata y on validation mini-batch with Eq.
9
4: Updata (0, sp) on training mini-batch with Eq. and Eq.
8
5: end while
Obtain weight allocation variables y of the global teacher error and the local student error
3.2 APPROXIMATING WITH LOCAL ERROR SIGNALS
Our goal is to realize the adaptive blick-wise weight allocation of the knowledge from the student model and the teacher model in KD. However, it is impossible to assign weights in blocks by using the error backpropagation algorithm in conventional teacher-student frameworks, which is shown in Eq. 3 Thus, we make approximate estimation of the global error signal of the student objective generated by backpropagation, leveraging the block-wise local error signal. Based on the transformation of this gradient flow, our main task is equivalent to the following two steps: (1) generating local gradient flow that approximates the gradient flow created by the last layer, and (2) simultaneously optimizing the gradient error signals generated by the local gradient flow and by the backward propagation, achieving adaptive weight assignment for each block. These two steps will be explored in Section 3.2 and Section 3.3
For step (1), we imitate supervised greedy learning to assign the local error signals which are used to approximate the error obtained by backpropagation. Following previous works (Nøkland & Eidnes) 2019 Beeiilossy et al. [2200] Pyeon et al. 2021). we utilize the auxiliary networks and target vectors to create the student error (local error signals) independently of the blocks, which is shown in Figure l(b) We usually identify single block as an input network consisting of convolutional layers, normalization and pooling layers. As for the input networks with residual connections, we regard block. Each block corresponds to auxiliary network to predict the target and compute the local objective. Meanwhile, this method is used for obtaining the local student error of each block. Note that, for the last block, the auxiliary network is the last fully- connected (FC) layer of the student.
each residual connection
Let (!) and denote the output representation for block I and the labels, E {0, L...., 1}. (() means the input data, ( represents the parameters of the auxiliary networks corresponding to the I-th block. We define the local objective function as (0()) (00); x()) (r())), Y). To apply our paradigm to the training process of the existing KD framework shown in Eq. ( we maintain both sensitive hyper-parameters a and B3. In addition, we propose to adopt a set of meta variables (()) 2 in a two-dimensional continuous domain. is used to modify the contribution of student error and teacher етror to the total eтror of different network layers. To be specific, we utilize a softmax over all possible values, namely, 71 .27 maf((2x)) as the weight allocation variables. Finally, we can formulate the loss objective function for block I as follows:
- (0, pp y; X (0) Y, ".Y") (1) ( (0, (() Y) + 20 @ (0%); ( Y"),
(4)
where Y the distilled signals of the teacher model. Finally, the error gradient of non-last block / can be re-defined as:
( 71 ad
(()) + 7/2
(1) 26
(5)
Oh and the teacher error KD = ack
where the local student error 80 block, we use 8(4)
As for the last
Oh(1)
Ø (1)
3.3 OPTIMIZING META VARIABLES WITH BILEVEL OPTIMIZATION
As mentioned Section 3.2 we need to optimize simultaneously both the global teacher error and the local student error ensure competitive performance of the student network. The objective
4

---

Under review as a conference paper at ICLR 2023
is noted by Eq. and Eq. (5) which is influenced not only by the network parameters (8, yp)
but also by the meta variables >. This implies that this optimization problem can be solved as a
bilevel optimization problem (Anandaligamm & Friesz 1992. Liu et al. 2019 Pyeon et al., 2021)
with (0, sp) as the inner-loop variables and as the outer-loop variables, which can be defined as:
min Lvll (6".""""" .7),)) s.t. (6", x0")== arg min Cerain (0. 4. 4.
(6)
6,0
where Ltrain and Let denote the loss function on training and validation set. By Eq. (6). we adopt the nested method which uses the training set to update the model parameters (0, +p) with global
error signals and local error signals of Eq. (5) and the validation set to optimize the meta parameters
y. Specifically, we fix and update (0, (p) with mini-batch training data in the inner loop. The error gradients updating can be written as:
ah
(7) - g0) (7) - (7)
O0()*
p (7) (7) - 17 V V (0 (0)
(8)
where Lirasn is the local objective of the training set and {0, 1..... ... L}. In the outer loop, we optimize the meta variables with mini-bated of validation set based on (6, sp) updated by Eq.
and Eq. (8) as follows:
p) Fou
A V. ((0
(6, p, p))),,),
where the learning rate of the meta variable 7. Then we apply the chain rule and the finite difference approximation (Liu al. 2019) the last item of Eq. (9)
V-Coal = Voa ,,,,)))))))))))))))))) - V Lerainn (0+ 4 7) V Lerain ,0,4 ,,,77)
(10)
2
where Leal is the validation loss, (8", .4") is the updated results of Eq. and Eq. (0,4p) + ( (o) (0", :9", 7)), & is scale
(ot x-) =
Finally, we fix the reasonable optimize the network parameters (0, p) on the training set and use the updated network parameters as the initial value of the next iteration. The bilevel optimization training process is shown in Algorithm
Warm start We adopt the warm start technique (Pyeon al. 2021 on the network weights (0, yp with fixed meta variables (7 0), which makes bilevel optimizttin stably and avoids the bad results of y caused by inappropriate initialization of (0, sp). In our experiments, we pre-train the student with 40000 iterations to obtain satisfactory initial value for the optimization of 7
Two-stage training Our proposed method adopts a two-stage training strategy, including meta variables optimization and final evaluation. According to the optimized meta variables, we evaluate the network weights (8, up) with the training loss to obtain the final comparable results, by following Eq. and Eq.
4 EXPERIMENTS
We first describe the main experimental settings (Section 4.1 our work. Then, we provide the results of baseline comparisons on several datasets (Section 4.2 Finally, we construct ablation experiments (Section 4.3) and some further discussions (Section 4.4 explore the necessity and effectiveness of the components of our framework.
which are necessary to understand
4.1 EXPERIMENTAL SETTINGS
Datasets. CIFAR- 100 Krizhevsky et al., 2009) is a image classification dataset which consists of 60000 32 colour images 100 classes. ImageNet (Deng al. 2009) is large-scale classification dataset with 1.2 million 256 256 images from 1000 classes. As for the setting of dataset split, we divide the official training set into our training set and validation set, which are 1:1 in size. This is to satisfy the conditions of bilevel optimization of meta variables. However, our warm start and final evaluation are still under the official setting.
(or 'The same setting with Liu al. 2019 & 0.01 (I Eoail(( ,22
5

---

Under review as a conference paper at ICLR 2023
ResNer66
ResNer32xx4
WRN-40-2
VGG13
teacher
72.35
71.31
79-12
75.61
74.61
ResNec20
ResNet32
ResNet8
WRN 10-1
VGG8
student
71.14
72.50
73.26
Stan.
Adap.
2
Stan.
Adap.
A
Ston.
Adap.
A
Stan.
Adap.
Stan.
Adap.
A
KD
70.66
71.01
+0.35
73.08
73.85
+0.77
73.33
73.57
40.34
73.54
74.14
+0.60
72.98
73,95
40.97
lngits
DKD
71.97
72.23
+0.26
74.11
74.25
+0.14
76.32
76.15
-0.17
74.81
75.07
+0.26
74.68
75.37
+0.69
Pimets
69.21
69
+0.39
71.05
71,20
-0.14
73.50
73.70
+0.30
72.24
72,54
+0.30
71.02
71,74
40.72
AT
70.55
71.04
73:44
71.38
+0.91
72.77
+0.70
SP
69.67
71.06
+1.30
72.69
73.07
+0.38
72.94
74.20
+1.26
72.43
73.27
+0.84
72.68
73.52
+0.84
CC
69.63
09.91
+0.28
71.48
72.19
-0.71
72.97
73.46
+0.49
72.21
72,04
-0.15
70.71
71,67
40.96
features
RKD
69.61
69.83
72.16
+0.34
71.90
73.22
+1.32
72:22
72:15
71:48
71.69
+0.21
VID
70.38
70.75
+0.37
72.61
72,85
+0.21
73.09
73.59
+0.10
73.30
72.95
-0.34
71.23
71.74
+0.51
PKT
70.34
70.96
+0.62
72.61
73.02
+0.41
73.64
74.64
+0.98
73.45
73,40
-0.05
72.88
73.06
40.88
CRD
71.16
71.10
+0.61
75.51
75.69
+0.18
74.14
+0.01
ReviewKD
71.89
71.94
+0.05
73.89
74.27
+0.38
75.63
75.86
+0.23
75.09
75.33
+0.21
74.84
74,97
+0.13
Table 1: Test accuracy (%) of homogeneous distillation on the CIFAR-100. Stan., Adap., A denote the standard KD method, the standard KD method within our adaptive block-wise framework and the performance improvement over the corresponding standard KD method, respectively.
Backbone and Auxiliary Networks We adopt several backbone architectures as our main net- works, including ResNet (He et al., 2016). VGG (Simonyan & Zisserman 2015). Wide ResNet (Zagoruyko & Komodakis; 2016), ShuffleNet (Zhang et 2018). and MobieNet (Howard et al.) 2017). Moreover, the selection of auxiliary networks should not only utilize lightweight auxiliary networks, but also ensure the parallel training of the auxiliary networks and main networks. Thus, build an auxiliary block with point-wise convolutional layer, depth-wise convolutional layer, an average pooling layer, and a fully-connected layer.
Baselines We compare the performance of the existing KD benchmarks themselves with those that use ABL. We divide these baselines into two categories: logits-based and feature- based. Logits- based distillation methods include KD (Hinton et al.] 2015) and DKD (Zhao et al.) 2022). The Ceature-based distillation methods are Fitnets (Romero et al. 2015), AT (Komodakis & Zagoruyko 2017. SP (Tung& & Mori 2019). CC (Peng et al. 2019), RKD (Park et 2019), VID (Ahn al.,
2019
PKT (Passalis & Tefas 2018). CRD (Tian et al. 2020), ReviewKD Chen et al. (2021).
More details of these experimental settings are shown Appendix A.l
4.2
MAIN RESULTS
Results on CIFAR-100 We evaluate our framework on CIFAR- Table 1 and Table 2 show the performance of the standard KD baselines and them in our proposed framework. In Table [ we adopt various -edenttteacher combinations in same-style architectures. Table 2 shows the results on heterogeneous distillation of several student-teacher frameworks.
For both types of distillation, we evaluate with five teacher-student combinations and eleven existing (including the state-of-the-art) knowledge distillation methods. After using our framework, despite no tuning for each task, the performance of most methods have improved by 0.5%-2%% which proves the effecttivnesss and wide applicability of our framework. In particular, AT, a feature-based method that plays a negative role in heterogeneous distillation, has a significant 8.1% improvement after our adaptive block-wise training. The extensive experimental results on both homogeneous and heterogeneous distillations demonstrate the effectiveness and portability of the proposed adaptive block-wise learning for KD.
We also evaluate the performance of several methods in our proposed frame- work on the large-scale ImageNet dataset, which is shown in Table 3 and Table These experimen- tal methods include KD and DKD based on logits and AT and ReviewKD based on features. As can be seen from both tables, the performance of the four methods on the accuracy of top-1 1 and top-5 by an average of 0.3-0.5% after using our adaptive framework. These results on ImageNet verify the scalability of our proposed method.
Results on ImageNet
selected has steadily improved

---

Under review as a conference paper at ICLR 2023
VGG11
ResNert0
ResNer32xx4
ResNer32iv4
WRN-40-2
teacher
74.61
79.34
79-12
79:42
75.61
Mo
MobilaNetV2
ShufBeNNrtV
ShutlleNat22
SbuffleNetV]
student
64.60
64.60
70,50
71.28
70.50
Stan.
Adap.
Stan.
Adap.
A
Ston.
Adap.
A
Stam.
Adap.
Stan.
Adap.
A
KD
67:37
68 68.85
+1.448
67.35
68.81
+1.46
74.07
75. 16.36
41.29
74.45
76.22
+1.77
74.83
75.69
40.86
lngits
DKD
69.71
69.58
+0.17
70.35
70.81
+0.16
76.45
77.03
+0.58
77.07
77,89
+0.82
76.70
76.87
+0.17
Pimets
64.14
64.15
+0.01
63.16
63.31
+0.15
73.59
74.49
+0,90
73.54
74,95
+1.41
73.73
74.28
40.55
AT
59:10
65.76
+8.10
71.73
75.07
+3.34
72.73
76.39
75.55
+2.23
SP
66.30
67.59
+1.29
68.08
67.68
-0.40
73.48
75.77
+2.29
74.56
76.62
+2.06
74.52
76.10
+1.58
CC
64,86
65.20
10.34
65.43
65.47
+0.04
71.14
71.84
+0,70
71.29
73.83
+2.54
71.38
72.14
40.76
features
RKD
64.52
61.97
+0.97
72.28
73.01
+0.73
73.21
74.85
+2.07
VID
65.56
66.42
+0.86
67.57
65.87
-1.70
73.38
73.01
-0.37
73:10
73,68
+0.28
73.61
74,11
+0.83
PKT
67.13
67.44
+0.31
66.52
67.07
+0.55
74.10
74.23
40.33
74.69
75,86
+1.17
73.89
74.74
40.85
CRD
69.73
69.88
70.02
+0.91
75.11
75.71
+0.60
75.65
76.50
+0.15
ReviewKD
70.37
70.68
+0.31
69.89
70.70
+0.81
77.45
77.98
+0.53
77.78
77.94
+0.16
77.14
77.41
+0.25
Table 2: Test accuracy (%) of heterogeneous distillation the CIFAR-100. Stan., Adap., A denote the standard KD method, the standard KD method within our adaptive block-wise framework and the performance improvement over the corresponding standard KD method, respectively.
Setting
Teacher
Student
KD
DKD
AT
ReviewKD
ResNar5D
MobileNatV2
Stan.
Adap.
A
Stan.
Adaa.
A
Stan.
Adap.
A
Stan.
Adap.
A
txp-1 1
76.16
68.89
70.47
70.37
-0.10
72.05
72.49
+0.44
69.56
68.79
-0.77
72.56
+0.05 92.15
72.61
top-5
92.86
88.76
89.46
89.98
+0.52
91.05
91.84
+0.79
89.33
89.51
+0.18
91.00
+1.15
Table 3: Test accuracy (%) of KD on the ImageNet between the different-style architecture.
Setting
Teacher
Student
KD
DKD
ReviewKD
ResNer34
ResNet18
Stan.
Adap.
A
Stan.
Adap.
A
Stan,
Adap.
A
Stan.
Adap.
A
1
73.31
69.75
70.66
70.97
+0.31
71.70
72.24
+0.54
70.69
71.09
+0.40
71.61
71.86
+0.25
top-5
91.42
89.07
89.88
90.01
+0.13
90.41
90.95
+0.44
90.01
90.37
+0.36
90.51
90.62
40.11
Table 4: Test accuracy (%) of KD the ImageNet between the same-style architecture.
4.3 ABLATION STUDIES
In this section, we conduct several ablation experiments to verify the effectiveness of our proposed framework. Specially, we explore the framework deeply by answering the following questions.
1) Are the local error signals good approximations of the global error signals? If the local error signals are not representative of the global ones, then there will be significant gap between the final results and the comparison results. To explore the effectiveness of local error signals from the local loss objective, we set the value of meta variables as zero. This means that the effect on any parameters the input neural network the same as the student error and the teacher error, which is similar to the conventional backpropagation KD framework. We build the experiments with several distillation methods under both same-style and different-style architecture setting and the results on CIFAR- 100 are shown in Table 5 From the results, for the KD, AT, RKD, ReviewKD method, the model updated with local error signals performs better, while in the DKD, CRD method, the model updated with global error signals performs better. Overall, the error between the two is less than I 1%. Thus, most of the teacher-student architectures we compared can be updated with local error signals to achieve the performance of conventional training strategies.
2) Are the adaptive meta variables reasonable? To validate the effectiveness of meta vari- ables used for balancing the student eиror and teacher error, we conduct three different strategies of ABL for KD via meta variables: Fixed-distillation, Random-distillation and No-distillation. is based on the assumption that the contribution of student's knowledge is the same as that of teacher's knowledge, and both the corresponding meta variables are 0.5. Candon-distillation randomly decides the balance between knowledge from teacher and student.

---

Under review as a conference paper at ICLR 2023
Tocher Stadent
ResNetl
ReiNer322d
VOG13
ReaNerSO
RoxNer32 x4 ShuffleNeeVi
Re
ResNetfxd4
VGG8
MobileNetV2
Lacal
Global
Lacal
Global
Local
73.28
73.33
67.35
75.14
DKD
74.11
73.62
76.32
76.02
74.68
74.82
70.35
69.31
76.45
75.73
AT 72.51
73.18
73.44
74.21
71.43
72.44
58.55
64.61
71.73
75,04
72.73
71,90
72.37
73.21
CRD
73:45
73.39
73.51
75.71
73,94
73.82
69.11
68.51
75,11
74.62
ReviewKD
73.89
74,06
75,63
75.28
74.84
73.95
69.89
66.32
77.45
77.15
Table 5: Ablation studies local error signals. Test accuracy (%) of knowledge distillation on the CIFAR-100 100. Global and Local denote student error from the global signals and the local signals.
Serting
RocNet LID MelDD--ReeNNii R2
VGGIE--VEEE
ResNer32 See.12x4--Shhiin SSuufllNaaa
Adagtive
Fised
Random
No
Adaepiive
Fixed
Random
No
Adappivo
Fixed
Random
No
Adeptive
Fixed
Rendom
No
KD
73.85
73:28
73.39
71.89
73.95
73.93
73.36
71.82
68.81
68.74
68.16
65.15
75.36
75.14
75,02
72.81
73.62
75.37
75,60
AT
73.30
73.18
72.50
71.89
73.13
72:44
72.71
71.82
66.68
64.61
64.15
65.15
75.07
75.04
74.83
72,81
RKD
72.16
72:73
71.67
71.89
71,69
71.37
71.15
71.82
65.40
64.98
64.00
65.15
73.01
73.21
72.88
72.81
CRD
73.48
73.39
73.03
71.89
73.94
73.82
73.43
71.82
69.11
08.51
67.23
65.15
75.11
74,62
75.01
72.81
ReviewKD
74.06
72.81
Table 6: Ablation studies on the block-wise distillation strategy. Test accuracy (%) of distillation methods the CIFAR-100. Adaptive, Fixed, Random and No represent the adaptive-type, fixed- type, random-type and no-type distillation, respectively.
No-distillation means the student trains using ABL without the teacher's signal. Several ex- periments are conducted on CIFAR-100 100 under the setting of both homogeneous and hetero- geneous distillation. The results are shown in Table [6] which demonstrates that the perfor- mance of our adaptive-distillate strategy has consistent improvement above other compa- rable baselines.
Standart
Accurncy (%)
120
170
Epoch
We also visualize the final evaluation pro- cess corresponding to the meta variables with ResNet20 student and ResNet56 as teacher based on DKD in Figure 2 We compare the training curves of three strategies including Standard- type, Fixed-type and Adaptive-type distillation. These three processes are similar the initial train- ing stage, while in the final stage. our proposed ABL has relatively stable improvement. This result shows the effectiveness of the adaptive-distillatio strategy with reasonable meta variables.
Figure 2: Comparison of learning curves
4.4 FURTHER INVESTIGATION ON ADAPTIVE BLOCK- WISE LEARNING
To further analyze ABL, we deeply explore the optimization process of meta variables and the results from the block-wise views.
Optimization process of meta variables We draw the meta variables training process in Figure 3(a) and ((b In these figures, with the increase in the number of network blocks, the network parameters prefer their own error signals (student error), while the influence of teacher error total gradient error is reduced. In particular, the shallow layers (the first three blocks), the parameter update of ResNet20 is more affected by teacher error signals. This means that from the perspective of gradient flow, the shallow representation of the student model is more likely to be guided by the distilled teacher knowledge, while the deeper and more abstract representation of the student model is more inclined to update from self-knowledge.
Block-wise results The results in Figure demonstrate shallow representation does not have the ability to classify with high accuracy and fit teacher's deep representation. In terms of the specific approach in our framework, DKD achieves better results than other methods the accuracy of each block, which is also proved in Figure ((g and 4bb The difference of the correlation matrices
8

---

Under review as a conference paper at ICLR 2023
16
V4
461
17
V1
15
42
45
Breatiass
(a) Meta variables for student error
(b) Meta variables for teacher error
Figure 3: All experiments are conducted on CIFAR- 100 based on DKD with ResNet20 as student and ResNet56 teacher. Figure (a) and (b) show the meta variables y optimization process for student error and teacher error, respectively. The r-axis and y-axis represent the number iterations and the softmax values (range from 0 to of the meta variables, respectively. Note that the values of the meta variables are set to O at iteration 0, thus all the initial points in the figure are 0.5.
VB
(a) DKD with ABL
Becheek
5-th Monec
15
(c) Test accuracy (%) of different blocks
(b) ReviewKD with ABL
Figure 4: All the experiments are implemented under the setting of ResNet20 as student ResNet56 teacher. (a) and (b) represent the correlations matrices between class logits of student and teacher in different blocks. For ResNer20, a residual block consists of three blocks that we define. (c) draws the test accuracy of nine blocks of ResNet20 based ABL for six types of distillation methods.
between the logits of the student and teacher after DKD distillation is less than ReviewKD. In Figure 4[c) the results show DKD with ABL. achieves better performance on each block. Interestingly, although the test accuracy of Fitnets distillation method is worse than other results in overall results, it is in the lead in the first six blocks. The reason is that, in the experimental setting, the hint-layer the second residual block, i.e., in the sixth block we define, which results in that only the first six blocks have teacher knowledge guidance. These observations present block-wise perspective, which provide more detailed view of the distillation procedures.
in Fitnets method is
CONCLUSION
In this paper, we provide a novel viewpoint on knowledge distillation, which discusses the balance between the teacher's and student's knowledge at different levels. Existing distillation methods are built on an implicit hypothesis that teacher knowledge and student knowledge make the same con- tribution the learning of shallow and deep representations. However, we consider that for shallow representations, the student network is easy to train under the guidance of teacher representation, while for deep representation, the student model is difficult to imitate directly from teacher's repre- sentation knowledge. Thus, propose Adaptive Block-wise Learning for Knowledge Distillation, which leverages a set of meta variables to control the balance between the student's local error sig- nals and the teacher's is global proposed method. More importantly, we confirm the hypothesis that the guidance of the teacher's knowledge the student network does decrease with the increase of blocks.
signals. The experimental results prove the effectiveness of the

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9163-9177 2019.
G. Anandalingam and Terry L. Friesz. Hicrarchical optimization: An introduction. Ann. Oper. Res., 34:1-11, 1992.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In International Conference on Machine Learning, pp. 736-745. PMLR, 2020.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiyy Jia. Distilling knowledge via knowledge re- view. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp.
2021.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaus- sian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSPP07 volume 4, pp. IV-317. IEEE, 2007.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 2(7), 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0067 2017.
Max Jaderberg. Wojeiech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Inter- national conference machine learning, pp. 1627-1635. PMLR, 2017.
Nikos Komodakis and Sergey Zagoruyko. Paying more attention attention: improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICLR (Poster), 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Arild Nøkland and Lars Hiller Fidnes. Training neural networks with local error signals. In Inter- national conference on machine learning, pp. 4839-4850. PMLR, 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), 2018
10

---

Under review as a conference paper at ICLR 2023
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Viston. pp. 5007-5016, 2019.
Myeongjang Pyeon, Jihwan Moon, Taeyoung Hahn, and Gunhee Kim. SEDONA: search for decou- pled neural networks toward greedy block-wise learning. ICLR, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6088):533-536 1986.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Wonchul Son, Jaemin Na. Junyong Choi, and Wonjun Hwang. Densely guided knowledge distilla- tion using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Jie Song, Ying Chen, Jingwen Ye, and Mingli Song. Spot-adaptive knowledge distillation. 1EEE Transactions on Image Processing, 31:3 31:3359-3370, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Frederick Tung and Greg Mori. Similarity-preservivn knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1365-1374, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1848-6856, 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11953-11961 2022.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We train our models on CIFAR-100 100 for 240 epochs with SGD that the weight decay 5e-4 and the momentum is 0.9. Due to the batch size being 64, the initial learning rate is set to 0.01 for ShuffleNet and MobieNet and 0.05 for others, and the learning rate is divided by 10 at 150, 180 and 210 epochs. For ImageNet, train the models for epochs 100. We the batch size to 128, and initialize the learning rate to O.1, which is divided by 10 at 30, 60, and 90 epochs.
Method
KD Hinton
2015
DKD
Fitncts Romero et
2015
AT Komedakis Zagoruyko 2017
0.9
100
1000
Method
SP Tung Mori
2019
CC Peng
2019
RKD
2019
Ahn
3000
Method
Passalis
2018
et 2020
Review Chen et
Table 7: Setting of distillation loss factor B.
11

---

Under review as a conference paper at ICLR 2023
By following Zhao et al. (2022), Chen et al. (2021) and Chen et al. (2021), we train the student model with the following objective:
aLCF BLKD,
(11)
For the setting of the hyper-paramettrs a, all methods are 1 except that KD is 0.1. The setting of distillation loss factor 3 is shown in Table 7 Specially, for the logits- base distillation methods including KD and DKD, set the temperature T 4.
All experiments are conducted with 2 NVIDIA TESLA V100S GPU cards.
A.2 STANDARD DEVIATION FOR RESULTS ON CIFAR-100
We conduct our all experiments CIFAR- 100 over 5 trials. In Table & and Table we provide the standard deviation of 5 trials on benchmarks.
ResNet56
ResNetl
ResNe
WRN--002
VGGI3
74.31
79.42
ResNex211
ResNet32
ResNetlick k4
WRR--001
VGGG
69.06
71.14
72,50
73.26
20.36
Stan.
Star.
Adap
Stan.
Adda.
Stan.
kn
70.00gnas
71
73.08 18
73. 88 is
73 73.33
79.67
73.54ga.0
74.14.a.5
72 72.98
73 18.9
loglts
25.0
24.8
75.07 .0.35
14.08
10237,029
Flinets
89.2 aa
09 60....2
71.20 7100
70,50
72 22.24
72.54.
7.02
71.7 20.0
71.04.0.45
78.28 o0.0
78.44.0.1
74.88, 10.2
72.77.8.0
71.43
SP
00.67.0
71.00.a
72.00
73.07
72,94
n4
72.43ge
73.27
78. 78.52 72.68
69 69
69 20.
71,48 m
72. 19
72 27.772mr1
73. 2
72 21
7204gg e1
70.7
37
frotures
RKD
09:01goss
09 83
71,82
72.16
71. 71.00g
75. nm
72.22
72.45_n54
71.48gnas
60
VID
10.381034
70.75.0.2
22.6
18.88...
73 1o.z
79. 79.59...2
73.30,0.13
12.90.0.15
71.2510.33
10.7
PKT
70.310044
70.96.023
72.61.017
73.02.009
70.641..1
74.64.0.7
13:40
72.8810a
73.06.0.77
CRD
71.164uu
71.40....s
70.48
74.09
75.51
35.09
74.14
74.25....m
73,944022
13.95
ReviswKD
71.89guss
74.27.02z
76.63
76.86,.a1
76.09
74884
Table 8: Test accuracy (%) of homogeneous distillation on the CIFAR-100. Stan., Adap. denote the standard KD and the standard KD within our adaptive block-wise framework respectively.
VOG13
ResNet30
ResNet32
RexNer32x44
WRN-41-22
79.34
79.42
Mi-bilaNerV2
MobileNerV2
ShuMaaNarh1
ShoMlNNerV2
ShaffleNetVi
60
Stan.
Stan.
Adap.
Suan.
Adap.
Ston.
Stan.
Adap.
KD
07. 77.
68 20
67 07
68 81 81,001
74.7 as
76. 20
74.46ua
70.22....v
74884
76 20.6
legits
DKD
00.71
20.88.0.4m an
70.35gn 17
10.81.01e
76
77.07 as
77.80_a.1
76.70
36.87
Flintts
61144
64 04.15.0.4
631111..7
63.3 11.31.0
74 a.
78.54.0
74.96...4z
7878
74 74.28.0.8
10.4
20.2
58.58
.0.2
11.74
76.0 10.0
72.70,a33 31
75.55.020
SP
66.300.
67.50
68.08
67. 28.
T8.48
70. 77
74.56
76.02
74.22
76. 70.10
61 804g0a
20
05.43 is
47
71 71.20 24
73.83
71.38
14
fcatures
RKD
61.52
64.97
64.43
65 40
72:08
73 01
73.21
74.86...30
72.21
74 28
VID
63.561n:4
06.42:0aa
67.57.024
63.87.1.16
73.38.0.0
70.01
73.40.a117
73.08.0.20
73.61an22
74 a4.4
PKT
67.10ana
07.44.037
68.22a83
74.10ama
74.23.017
74.80.a
75.86.n31
73,89
74.74.0a7
CRD
89.33a
09.88 .0.1
60.11
70 00.02.aa3
75.11
76. 70.71,0.1
75.65
76.80....
76.05a0s
70.50,021
ReviewKD
70:77ama
77.46unas
77.9810.15
77.78uazi
77.14am.9
Table 9: Test accuracy (%) of heterogeneous distillation on the CIFAR-100.
A.3 MORE ABLATION STUDIES
1) Is there a problem of gradient vanishing? Gradient vanishing usually exists in networks with too many layers. To verify whether such problem exists in the standard KD scheme, if there is
12

---

Under review as a conference paper at ICLR 2023
V9p13-Vgoe
Scar
044
Smad
Tooch
Epocs
Epech
(a) Loss between standard and ABL(fixed)
Mean gradiet
Mean gradient
Mean gradint
Lust Syaa
-0.2
(b) Mcan gradient of standard KD Scheme
0.42
Fryt Syne
Plat oyer
Free tyer
441
445
-44
-442
Epoch
Epech
Eacch
(c) Mean gradient of KD scheme with (Bi((ixed)
Figure 5: All the experiments are implemented under the same KD setting. The first column takes ResNet32 as the student and ResNetl 10 as the teacher. The second column takes Vgg8 as the student, Vgg13
the teacher. The third column sets ShuffleV I as the student and ResNet32x4 as the teacher. (a) represents the learning curves between the standard and ABL. (b) figures the mean gradient of standard KD scheme. (c) draws the mean gradient of the KD scheme with ABL.
Senting
ResNet110 -ResNetS2
VGGI3-
ResNiet0 -MobiilNNeV2
ResNetSZio4
-ShttleNaaV
Adaa
(kap(
Adaa
Stas/ttas)
(dap((
Arhe.
Ston.
Stan/Jess)
Adap
Adap.
Ston.
Staniles)
Adap (tos)
KD
73,85
73.08
73.02
73334
73.95
72.98
73.48
73.91
68.81
67.35
68.64
68.55
75.36
74.07
74.46
7454
DKD
74.25
74.11
73.55
73.16
75.37
74,88
74.00
74.14
70.81
70.35
70.49
70.26
77.03
76.45
76.75
75.70
Table 10: Ablation studies on the block-wise distillation strategy. Test accuracy (%) of distillation methods on the CIFAR-100, Adap.. Stan., (tan. and (taa) represent the adaptive-type, standard-type, standard(only tcacher)-type and fix(only teacher)-type distillation, respectively.
problem, whether proposed local objectives has been improved by solving the problem, we figure the training process of the standard KD framework and our proposed KD framework, as well as the mean value of the gradient of the first and last layers under the same setting, which is shown in Figure The results of Figure 5(a) show that the training process of the standard KD framework very stable. The loss is always changing, which proves that neither of the two frameworks can be trained because gradient vanishing. In Figure 55b) and Figure 5(c) there is no difference of magnitude between the mean gradient the first layer and the last layer, and there is stagnant training. This proves that there is gradient vanishing in both ABL scheme and the standard KD scheme. The more important thing is the similarity the both schemes images verify that local error signals are good approximations of the global error signals.
and our proposed KD framework
2) Are the adaptive meta variables effective? We construct more experiments to verify the effec- tive of our proposed adaptive meta variables, which shown in Table 10. We create two additional
13

---

Under review as a conference paper at ICLR 2023
strategies: only using teacher knowledge standard KD with supervision (Stan.(tea.),, and the proposed ABL strategy under only teacher guidance (teap. In the Table T0 the standard KD scheme utilized teacher knowledge only performs better than that supervised by both teacher knowledge and the ground truth knowledge. This proves that standard KD framework limits the potential of teacher knowledge. However, the performance of ABI. framework with only teacher knowledge is poorer than that under both teacher's and student's knowledge. This shows that the meta variables effectively help the model to learn under the guidance both student knowledge and teacher knowledge.
3) How to design an appropriate auxiliary network? The only goal of the auxiliary network in ABL is to achieve the possibility of local eπror instead of global error to create a later inquiry into the meta weight allocation of student knowledge contribution and teacher knowledge contribution. Thus, the performance of an appropriate auxiliary network need to approach to standard KD scheme (with the only supervision of last layer). This means that the auxiliary network should not be too simple (shallow) to achieve the effect of using local error instead of global error, nor should it be so complex as to greatly increase the difficulty and duration of training.
Inspired by the greedy learning (Jaderberg et al. 2017 Nøkland & Eidnes 2019 Belilovsky et al. 2020 Pyeon et al. 202I), we conduct several types of auxiliary network as follow.
Auxl: AvgPool FC layer,
Aux2: AvgPool + point-wise convolutional layers + AvgPool + 3-layer MLP,
Aux3: point-wise convolutional layer + depth-wise convolutional layer AvgPool + FC layer,
Aux4: woint-wise convolutional layer + cepplt-wisee AvgPool + FC layer
convolutional layer + inverted residual block
The Table II figures the results on testing accuracy (%) and the training time (GPU hours).
Setting
ResNer 10
VGGI3
Global
Aux
Aux2
Aux3
Aux4
Global
Aux
Aux2
Aux3
Aux4
Testing accuracy (%)
73.08
71.83
72.97
73.28
73.63
72.98
72.61
74.01
73.93
74.33
Training time (GPU hours)
3.42
3.04
5.31
4.52
5.95
1.92
1.54
3.21
2.62
3.59
Table 1 11: Ablation studies on the auxiliary networks. All experiments are conducted under the same KD distillation setting.
A.4 TRAINING EFFICIENCY
Due to the existence auxiliary networks for computing local error and bi-level optimization for meta variables, ABL needs more training costs. The results of training efficiency are shown in Table 12 The total training time of ABL about 1.25-1.35 times of that of the standard KD, but the performance of ABL improves by 0.5%-2% over most distillation methods, which proves these costs are desirable.
Setting(GPU hours) Bi-level optimization
Final evaluation
Total(ours)
Standard
MobileNetV2
1.04
4.81 1
5,85
4.65
VGG8
0.57
2.05
2.62
1.92
ResNet32
0.71
3.81
4.52
3.42
ShuffleV1
1.07
4.98
6.05
4.67
Table 12: Training COsts(GPU hours) of different models evaluate under KD distillation methods the C0FAR-100. The total training process contains a bi-level optimization and a final evaluation.
A.5 MORE INVESTIGATION ON ADAPTIVE BLOCK- WISE LEARNING
To explore the two types of knowledge in different blocks, we plot the test accuracy of each block of different student networks under the DKD distillation setting, which is shown in Figure
14

---

Under review as a conference paper at ICLR 2023
6 In the Figure 6 the results demonstrate that each block of the student networks trained by the proposed ABL scheme outperforms that under the fixed ABL setting. the shallow blocks, the teachers knowledge contributes more than student knowledge, and the accuracy is higher than the fixed one. While in the deep blocks, the student knowledge contributes more, and the accuracy is higher than the fixed one. Thus, teacher knowledge is more suitable to guide shallow representation, while student knowledge is more appropriate to guide deep representation.
15
(a)
(C)
Figure 6: All the experiments are implemented under the same DKD setting. Fixed represents the model trained with the fixed contribution of student's 's knowledge and teacher's knowledge. Adaptive means the model under ABL training.
15


Reviews:
[{"review_id": "yRNWIiAtvki", "reviewer": "Reviewer_Jk2C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors argue that contributions of knowledge from the teacher to the student network should be layer dependent.  Adaptive block-wise learning automatically balances the contribution of knowledge between the student and teacher for each block.\n", "summary_of_the_review": "The paper is incrementally novel, but it has extensive experiments.\nThe paper talks in generalities without any technical or experimental details.  A below average ML paper.", "strength_and_weaknesses": "The paper talks in generalities and employs a general abstract notation. There is no experimental specificity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VsfhfQTzCG", "reviewer": "Reviewer_BeZK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript observes the problem of fixed contributions of ground truth knowledge and teacher knowledge at different blocks of the student networks during knowledge distillation training. The author proposes a bi-level optimization scheme to balance the knowledge on the lower level and update the network based on optimized balance at the higher level.", "summary_of_the_review": "The paper might be helpful to the research community. However, the concern about training time might reduce the chances of its practical applications. While considering about the fundamental contributions, the authors should justify the problem clearly as well as put more analyses on the proposed schemes. ", "strength_and_weaknesses": "Strength:\n+ Paper is well written\n+ Experiments on common benchmark datasets for KD\n+ A number of KD schemes are tested with the proposed scheme in the experimental results\n\nWeaknesses:\n1. The problem might not be well justified. The main argument of the paper is to use the performance of the proposed scheme to claim the solving of the fixed contributions of the two types of knowledge in different blocks. It would be more interesting to see what part of the network goes wrong with the fixed contributions scheme. \n\n2. Whether the improvement of the network comes from the auxiliary network? Please consider adding an experiment by removing the bi-level optimization but still keeping all the auxiliary networks. Consider tuning the fixed parameters between ground truth knowledge and teacher knowledge in this scheme. \n\n3. The increase in performance might be the result of solving the gradient vanishing not solving the balance between ground truth information and the teacher at the different blocks. The addition of an auxiliary could also be interpreted as a shortcut to the last layer. Consider checking the gradient vanishing between the standard KD scheme and the proposed KD scheme. Or further, can we just add a shortcut from each layer to the final layer and have different losses corresponding to each shortcut?\n\n4. For some datasets, the utilization of teacher knowledge only could also achieve comparable or even better results compared to the utilization of both teacher knowledge and ground truth knowledge. The authors should consider adding a KD scheme with only teacher knowledge guidance as one of the baselines. So that the scheme of balancing between teacher knowledge and ground truth knowledge would be more meaningful.\n\n5. How does the proposed scheme training time increase compared to the standard KD schemes? The main concern about the bi-level optimization is that it always takes too much time, while the achieved improvement in the manuscript seems to be not too significant (only around 0.5%, and poorer in some cases). This might be the strongest challenge to apply the proposed scheme to practical applications.\n\n6. In table 3, Is that normal where the standard KD performs poorer than supervised training?\n\n7. The authors should details out how each parameter is fixed in standard KD schemes in all experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UbxbTCYcII", "reviewer": "Reviewer_Tp5S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to improve knowledge distillation (KD) research from the perspective of allocating adaptive weighting factors to update layer-wise learnable parameters of the student model during back-propagation. The authors conjecture that the knowledge from the teacher  should be different to the student layers from shallow to deep. Based on this, they propose a new KD method called adaptive block-wise learning (ABL), which uses auxiliary networks for learning to dynamically generate meta weighting factors for gradients propagation at different blocks of the student network. In designs, these auxiliary networks are added to intermediate student layers, and leverage the local error signals to approximate the global error signals on student objectives. In implementation, ABL uses a greedy bi-level (two-stage) optimization. The effectiveness of ABL is validated on image classification datasets CIFAR100 and ImageNet.     ", "summary_of_the_review": "This paper is below the acceptance bar of ICLR.\n\nPlease refer to my comments in 'Strength And Weaknesses' for details.\n\n", "strength_and_weaknesses": "Strengths.\n\n+ Making the knowledge from teacher to layers of the student network to be different is interesting.\n\n+ The proposed ABL sometimes shows improvements (but actually are not fair) to different KD methods.  \n\n+ Experimental comparisons are conducted on both CIFAR100 and ImageNet datasets with different teacher-student network pairs.\n\nWeaknesses.\n\n- The method.\n\nThe core ideas of the proposed ABL are in two aspects: 1) adding auxiliary networks to some intermediate student layers to ease the training; 2) in optimization, auxiliary networks are used to approximate the global error signals from the pre-trained teacher, learning to dynamically generate meta weighting factors for gradients propagation at different blocks of the student network. However, there already exist many KD works that explore the use of auxiliary networks to improve KD process. Some representative works are DKS [1], BYOT [2], DCM [3] , MetaDistiller [4], to name a few. Besides, MetaDistiller and some other works also explore the use of meta learning. Unfortunately, these works are completely missed by the authors. A comprehensive comparison of ABL with them, both in methodology and performance is necessary. \n\n[1] Deeply-Supervised Knowledge Synergy, CVPR 2019.\n\n[2] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation, ICCV 2019.\n\n[3] Knowledge Transfer via Dense Cross-layer Mutual-distillation, ECCV 2020. \n\n[4] MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation, ECCV 2020.\n\n- The motivation.\n\nIn the paper, the authors argue that different layers of the student network have different emphases on the knowledge learned through the one-hot labels and the knowledge distilled by the teacher. Furthermore, the authors claim a new contribution that the deep and abstract representation inclines to learn from student knowledge, while the shallow and less abstract representation tends to be guided by teacher knowledge. However, the analysis and experiments provided in the paper are not convincing enough. Could the authors provide more evidence or richer experiments to support these claims? E.g., how about the role of auxiliary networks? how to design them? the effects of  different auxiliary classifiers? how about the performance of adding low-quality/shallow auxiliary networks? what will happen if smoothing the predication of the pre-trained teacher model?\n\n- The experiments.\n\nExperimental comparisons are problematic and misleading: (1) For experimental comparisons (Table 1 and Table 2) on CIFAR100, it seems that all results for counterpart baseline methods (denoted as 'Stan.') are directly copied from the paper of CRD (mostly), and the paper of DKD. As a result, the authors merely tested the combination of ABL and each of them, but not tested each corresponding counterpart with the same training machine and training code settings. As a result, all reported \\delta values are totally misleading; (2) For experimental comparisons (Table 3 and Table 4) on ImageNet, the authors also use such an unfair comparison; (3) This even applies to some ablations. In a nutshell, the authors did not actually run any experiments for counterpart baseline methods (denoted as 'Stan.') at all. Therefore, main experiments need to re-design and re-implement. Even w.r.t. the current results, the improvement from ABL is mostly marginal.\n\nA comprehensive comparison of ABL with closely related methods such as DKS [1], BYOT [2], DCM [3] and MetaDistiller [4] is necessary. \n\nAs auxiliary classifiers play a key role in ABL, a deep analysis of them is also necessary, please see my comments in 'The motivation' for details. \n\nHow about the training cost of ABL compared to counterpart baselines methods?\n\n**----Update----**\n\nI keep my original score as my major concerns are not well addressed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-_k8rkUhkj", "reviewer": "Reviewer_XFGM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Adaptive Block-wise Learning, a method that adapts the amount of knowledge distillation from a teacher to a student at each layer of the network. A set of auxiliary networks is used for this purpose. The method uses local error signals to control how much teacher or student knowledge should each layer use. This is accompanied by a set of metavariables to control these contributions, which are optimized using bilevel optimization. The authors report experimental results on several SOTA knowledge distillation methods, showing that this more selective knowledge distillation strategy can benefit both homogeneous and heterogeneous knowledge distillation problems.", "summary_of_the_review": "The paper presents a method for knowledge distillation that considers the contribution of teacher and students distinctly at different layers of the network. This approach seems novel and useful for the area. The conclusions around the different degrees of contribution of teacher and students at different depths of the network seem to be an important contribution (although I am not fully aware of the knowledge distillation literature, so I may be convinced otherwise based on other reviews). Based on this, my recommendation at this stage is for acceptance of the paper.", "strength_and_weaknesses": "Strengths:\n- The paper presents a novel idea of using local errors to determine the contribution of teacher and student networks during knowledge distillation.\n- The paper is well written and well organised. \n- The paper reports relevant findings for the knowledge distillation area, in particular that deep blocks of the network benefit more from student knowledge, while shallow blocks of the network benefit from more teacher knowledge.\n\nWeaknesses:\n- Some terms are used that are never defined. This makes the paper difficult to understand for a researcher not in this area. For example, what are homogeneous and heterogeneous knowledge distillation? Please provide a definition or cite a relevant definition.\n- In the experiments, it is never mentioned how big is the teacher and how big is the student for each of the datasets. Furthermore, how relevant is this for the proposed method? \n- I would like to see how the findings regarding the contribution of teacher and student at different layers in the knowledge distillation setting are connected to previous similar findings in transfer learning in general (see for example [1])\n\nReferences: \n[1] Neyshabur, B., Sedghi, H., & Zhang, C. (2020). What is being transferred in transfer learning?. Advances in neural information processing systems, 33, 512-523.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject