Learning from deep model via exploring local targets
Authors: Wenxian Shi
Yuxuan Song
Hao Zhou
Bohan Li
Lei Li
Venue:  (2021)

Abstract: Deep neural networks often have huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. 
However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima.  In this paper, we propose proKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective  into  local intermediate targets with approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optima. Experiments on both image and text datasets show that our proposed proKT consistently achieves the state-of-the-art performance comparing to all existing knowledge distillation methods.  

Under review as a conference paper at ICLR 2021
LEARNING FROM DEEP MODEL VIA EXPLORING LOCAL TARGETS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks often have huge number of parameters, which posts chal- lenges in deployment in application scenarios with limited memory and computa- tion capacity. Knowledge distillation one approach derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose ProKT, new model-agnostic method by projecting the supervision signals of a teacher model into the student's 's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with approxi- mate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result a better local optima. Experi- ments on both image and text datasets show that our proposed ProKT consistently achieves the state-of-the thet--rrr performance comparing to all existing knowledge distillation methods.
INTRODUCTION
Advanced deep learning models have shown impressive abilities in solving numerous machine learning tasks (Devlin al. 2018b Radford et al. 2018 He et al., 2016). However, the advanced heavy models are not compatibee with many real-world application scenarios due to the low inference efficiency and high energy consumption. Hence preserving the model capacity using fewer parameters has been an active research direction during recent years (Polino et al. 2018 Wu et al. 2016 Hinton jet al. (013). Knowledge distillation (Hinton et al., 2015) is an essential way in the field which refers to model-agnostic method where model with fewer parameters (student) is optimized to minimize some statistical discrepancy between its predictions distribution and the predictions of a higher capacity model (teacher).
Recently, it has been observed that employing a static target as the distillation objective would leash the effectiveness of the knowledge distillation method (Tin et al. 2019 Mirzadeh al.) 2019) when the capacity gap between student and teacher model is large. The underlying reason lies in common sense that optimizing deep learning models with gradient descent is favorable to the target which is close to their model family (Phuong & Lampert 2019). To counter the above issues, designing the intermediate target has been a popular solution: Teacher-Assistan learning (Jin al. 2019)
shows that within the same architecture setting, gradually increasing the teacher size will promote the distillation performance; Route-Constrrr Constrained Optimization (RCO) (Mirzadeh al. 2019) uses the intermediate model during the teacher's 's training process as the anchor to constrain the optimization path of the student, which could close the performance gap between student and teacher model.
One reasonable explanation beyond the above facts could be derived from the perspective of curricu- lum learning (Bengio et al. 2009): the learning process will boosted if the goal is set suitable to the underlying learning preference (bias). The most common arrangement for the tasks is to gradually increase the difficulties during the learning procedures such as pre-training (Sutskever et al. 2009). Correspondingly. TA-learning views the model with more similar capacity/moodel-szze as the easier tasks while RCO views the model with more similar performance as the easier tasks, etc.
In this paper, argue that the utility of the teacher is not necessarily fully explored in previous approaches. First, the intermediate targets usually discretize the training process as several periods and the unsmoothness of target changes in optimization procedure will hurt the very property of

---

Under review as a conference paper at ICLR 2021
introducing intermediate goals. Second, manual design of the learning procedure is needed which is hard to control and adapt among different tasks. Finally, the statistical dependency between the student and intermediate target is never explicitly constrained.
To counter the above obstacles, we propose ProKT, new knowledge distillation method, which better leverages the supervision signal of the teacher improve the optimization path of student. Our method is mainly inspired by the guided policy search in reinforcement learning (Levine & Koltun 2013), where the intermediate target constructed by the teacher should be approximately projected on the student parameter space. More intuitively, the key motivation is to make the teacher model aware of the optimization progress of student model hence the student could get the "hand-on" supervision to get out of the poor minimal or bypass the barrier in the optimization landscape.
The main contribution of this paper that we propose a simple yet effective model-agnostic method for knowledge distillation, where intermediate targets are constructed by model with the same architecture of teacher and trained by approximate mirror descent. We empirically evaluate our methods on a variety of challenging knowledge distillation setting on both image data and text data. We find that our method outperforms the vanilla knowledge distillation approach consistently with a large margin, which even leads to significant improvements compared to several strong baselines and achieves state-of the-art on several knowledge distillation benchmark settings.
2 RELATED WORK
In this section, we discuss several most related literature in model miniaturization and knowledge distillation.
Model Miniaturization. There has been fruitful line of research dedicated to modifyyng the model structure to achieve fast inference during the test time. For instance, MobileNet (Howard et al. 2017) and ShuffleNet (Zhang et al. 2018a) modify the convolution operator to reduce the computational burden. And the method of model pruning tries to compress the large network by removing the removed either based on the weight magnitude or the impact on the loss function. One important hyperparameter of the model pruning is the compression ratio of each layer. He et al. (0188) proposes the automatical tuning strategy instead of setting the ratio manually which are proved to promote the performance.
redundant connection in the large networks. The connections
Knowledge Distillation. Knowledge distillation focuses on boosting the performance while the small network architecture is fixed. Hinton et al. )2015); Buciluu et al. (2006) introduced the idea of distilling knowledge from a heavy model with a relatively smaller and faster model which could preserve the generalization power. To this end, Buciluã et al. (2006) proposes to match the logits of the student and teacher model, and Hinton et al. (2015) tends to decrease the statistical dependency between the output probability distributions of the student model and the teacher model. And Zhang eet al. 20188) proposes the deep mutual learning which demonstrates that bi-jective learning process could boost the distillation performance. Orthogonal to output matching, many works have been conducted on matching the student model and teacher by enforcing the alignment on the latent representation (Yim et al. 2017 Jiao et al., 2019a] Sun et al.) (019). This branch of works typically involves prior knowledge towards the network architectures of student and teacher model which is more favorable distill from the model with the same architecture. In the context of knowledge distillation, our method is mostly related to TA-learning (Mirzadeh et al.) 2019) and the Route- Constraint Optimization(RCO) (Jin et al. 2019) which improved the optimization of student model by designing a sequence of intermediate targets to impose constraint on the optimization path. Both of the above methods could be well motivated in the context of curriculum learning, while the underlying assumption indeed varies: TA-learning views the increasing order of the model capacity implied suitable learning trajectory; while RCO considers the increasing order of the model performance forms favorable learning curriculum for student. However, there have been several limitations. For example, the sequence of learning targets that are set before the training process needs to be manually designed. Besides, targets are also independent of the states of the student which does not enjoy all the merits of curriculum learning.
Connections to Other Fields. Introducing a local target within the training procedure is a widely applied spirit in many fields of machine learning. Montgomery & Levine (2016) introduce the guided policy search where local policy is then introduced to provide the local improved trajectory, which
2

---

Under review as a conference paper at ICLR 2021
has been proved to be useful towards bypassing the bad local minima. He et al. (2012) augmented the training trajectories by introducing the so called "coaching" distribution to ease the training burden and similarity. Le et al. (2016) introduce family of smooth policy classes to reduce smooth imitation learning to a regression problem. [u et al. (2018) introduce an intermediate target so-called mediator during the training of the auto-regressive language model, while the information diserepancy between the intermediate target and the model is constrained through the Bullbacck-eibler(fe Moreover, Chen et al. (2017) utilized the interpolation between the generator's 's output and target as the bridge to alleviate data sparsity and overfitting problems of MLE training. Expect from the distinct research communities and objectives, our method also differs from their methods in both the selection of intermediate targets, i.e. learned online versus designed by hands, and the theoretical motivation, i.e. the explicit constrain in mirror descent guarantee the good property on improvement.
divergence.
3
METHODOLOGY
In this section, we first introduce the background of knowledge distillation and notations in Sec- tion 3.1 Then, in Section we generalize and formalize the knowledge distillation methods with intermediate targets. Section 33. we propose our method ProKT, and in Section 3.4 the connection between ProKT and mirror descent is demonstrated.
3.1 BACKGROUND ON KNOWLEDGE DISTILLATION
To start with, we introduce the necessary notations and backgrounds which are most related to our work. Taking an K-class classification task example, the inputs and label tuple is denoted as (x. y) E X x y and the label y usually in the format of one-hot vector with dimension K. The objective in this setting is to learn parameterized function approximator: f(x;0) 0) X - V. Typically, the function could be characterized as the deep neural networks. With the logits output as u, the output distribution g of the neural network [2; 8) could be acquired by applying the softmax exp(u;/T) function over the logits output u: q ΣØ_ exp(a,/T)' where T corresponds to the temperature. The objective of knowledge distillation could be then written as:
(k) = (1 - H. (y, (6 (0)) + aT H(p, (0))
Here H denotes the cross entropy objective, i.e., H(p.q) = Σ -p; log qi which is the KL divergence between p and q minus the entropy of p (usually constant when p y). P: is the output distribution of given teacher model and is the balanced weight between the standard cross entropy loss and the knowledge distillation loss from teacher. T is the temperature. In the following formulations, omit the T by setting T 1.
3.2 KNOWLEEDGE DISTILLATION WITH DYNAMIC TARGET
In this section, we gen- eralize and formalize the knowledge distillation methods with intermedi- ate targets. We propose that previous knowledge distillation methods, either with a static target (i.e., the vanilla KD) or with hand-crafted gets (i.e., discrete tar-
Figure 1: Mseacher and Mstudent refer to the output manifolds of
student model and teacher model. The lines between circles to squares (m. imply the learning trajectories in the distribution level.
Optimization (RCO) (Jin) al. 2019)), cannot make full use of the knowledge from teacher. Instead, dynamic and continuous sequence of targets is a better choice, and then we propose our method in the next section.
The intuition of ProKT is avoid bad local optimas (triangles (a)) by conducting supervision signal projection.
3

---

Under review as a conference paper at ICLR 2021
Firstly, generalize and formalize the knowledge distillation methods with intermediate targets, named as sequential optimization knowledge distillation (SOKD) methods. Instead of conducting a static teacher model in vanilla KD, the targets to the student model of SOKD methods are changed during the training time. Without loss of generality, we denote the sequence of intermediate target distributions as P. = [pt,pe, ... .Pi ...... Starting from random initialized parameters / the student model is optimized by gradient descent methods to mimic its intermediate target p?":
4T pmh1 88 (8m-1))
(2)
(0) = (1 - a) (v: 9s (0)) + aH (
(3)
One choice to organize the intermediate targets is to split the training process into intervals and adopt fixed target in each intervals, named as discrete targets. For example, the Koute-Constrain Optimization (RCO) (Jin et al. (219)) saves the un-convergent checkpoints of teacher during the teacher's 's training to construct the target sequence. The learning target of student is changed every few epochs.
However, the targets are changed discontinuously in the turning points between discrete intervals, which would incur negative effects on the dynamic knowledge distillation. Firstly, switching to a target that is too difficult for the student model would undermine the advantages of curriculum learning. If the target is changed sharply to model with large complexity improvement, it is hard for student learn. Besides, the ineligible gap between adjacent targets would make the training process unstable and hurt the convergence property during the optimization (Zhou et al. 2018).
Therefore, we propose to replace the discrete target sequence with a continuous and dynamic one, whose targets are adjusted smoothly and dynamically according to the status of student model. In continuous target sequence, targets in each step are changed smoothly with ascending performance. In that case, if the student learns the target well in current step, the target of the next step is easier to learn because of the slight performance gap. The training process is stable as well, because the training targets
improved smoothly. Specifically, the optimization trajectories of the teacher model naturally offer continuous supervision signals for the student. In our work, we propose to conduct the optimization trajectories of teacher model the continuous targets. Besides, to ensure that intermediate teachers are kept easy to learn for students, we introduce an explicit constraint in the objective of the teacher. This constraint dynamically adjusts the updating path of the teacher according to learning progress of the student. The key motivation of our method is illustrated in Fig.
3.3 PROXIMAL KNOWLEDGE TEACHING
In this section, we firstly propose the SOKD adopting the optimization trajectories of teacher as the continuous targets. The learning process is that every time the teacher model updates one step towards the ground-truth, the student model updates one step towards the new teacher. Then based on this, we propose the Proximal Knowledge Teaching (ProKT), which modifies the updating objective of the teacher by explicitly constraining it in the neighbourhood of student model.
To construct the target sequence with continuous ascending target distributions, a natural selection is the gradient flow of the optimization procedure of the teacher distribution. With the student qo, and teacher model Pa, initialized at the same starting point (c.g., gop (z/z²) = (t/z)) = Uniform form(1, K)), we iteratively update the teacher model and the student model according to the following loss functions:
0m1+1 = 0Th 7/, (L,) L.(0,) = H(y )o,): gm+1 8TH 17 VL. (L.. (..) = H Hpo., 9
(4)
(5)
Here, the Tt and 7) are learning rates of student and teacher models, respectively. Starting with the same initialized distribution, the teacher model is updated firstly by running step of stochastic gradient descent. Then, the student model learns from the updated teacher model. In this process, the student could learn from the optimization trajectories of the teacher model, which provides the knowledge of how the teacher model is optimized from a random classifier to a good approximator. Compared with the discrete case such as RCO, the targets are improved progressively and smoothly.
4

---

Under review as a conference paper at ICLR 2021
Algorithm ProKT
1: Input: Initialized student model qo, and teacher model qo,- Data set D.
2: while not converged do
3: Sample batch of input (x, v) from the dataset D.
1)t V Lo.. 4: update teacher by 0g 0:
5: update student by 0g <- 0. 77 oo (())).
6: end while
However, simply conducting iterative optimization following Eq. with gradient descent could not guarantee the teacher would stay close to the student model even with a small update step. The gradient descent step of teacher in Eq. 4 is equivalent to solving the following formulation:
0 arg minC (0.T)) + Ve[(6) 0:"h) + 2 1,"
which only seeks the solution in the neighborhood current parameter 0T terms of the Euclidean distance. Unfortunately, there is explicit constraint that the target distribution Po.++1 (y/z²) stays close to Par (y]z)). Besides, because the learning process of teacher model ignorant of how the student model has been trained, it is probably that the gap between student model and teacher model grows cumulatively.
Therefore, in order to constrain the target distribution to be easy-to-learn for the student, we modify the training objective of teacher model in Eq. 4 by explicitly bounding the KL divergence between the teacher distribution and student distribution:
gm++1 min (y. po.)
DKL Po.) S.
(6)
6,
The e controls the how close the teacher model for the next step to the student model. this case, we make approximation that if the KL divergence of target distribution and the current student distribution is small, this target is easy for student to learn. By optimizing the Eq. the teacher is chosen the best approximator of the teacher model's family in the neighbour of student distribution.
With slight variant of the Lagrangian formula of Eq. 6 the learning objective of teacher model in ProKT is
C. = (1 X)H Po.) + (H(g..
in which the hyper-parameter À controls the difficulty of teacher model compared with student model. The overall algorithm is summarized in Algorithm
3.4 PROKT AS APPROXIMATE MIRROR DESCENT
Following the assumption that supervised learning could globally solve convex optimization problem, could be shown the proposed method corresponds to a special case of mirror descent (Beck & Teboulle 2003) with the objective H (3, qa, ). Note the optimization procedure is conducted on the output distribution space, the constraint is the solution must lie on the manifold of output distributions which could be characterized in the same way as the student model. We use Qo, to denote the possible output distribution family with the same parameterization as the student model.
Proposition 1
The proposed ProKT solves the optimization problem:
qa + arg min H H EQa,
with mirror descent by iteratively conducting the following two step optimization al step
q0 +- arg min H (y. qo.) s.t. DEL (gø), g6.) Se. 48, arg min
DRL (qø) ge.)
(8)
qo, EQo,
The first step is to find a better output distribution which minimizes the classification task and is close to the previous student distribution og" under the KL divergence. While the second step projects the distribution the distribution family Qo, in terms the KL divergence. The monotonic property directly follows the monotonic improvement in miror descent (Beck & Tehoulle 2003).
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
4.1 SETUP
In order to evaluate the performance of ProKT under different knowledge distillation settings, implement the ProKT in different tasks (image recognition and text classification), different network architectures, and different training objectives.
4.1.1 IMAGE RECOGNITION
The image classification experiments are conducted in CIFAR-100 (Krizhevsky al. ing Tian et al. (2019).
2009) follow-
2019), we compare the performance of knowledge distillation methods under various architecture of teacher and student models. We use the following models
Settings. Following the Tian et al.
as teacher or student models: vgg (Simonyan & Zisserman, 2014), MobilcNetV2 (Sandler et al. 2018) (with a width multiplier of 0.5). ShuffleNetVi (Zhang et al. (0118)) ShuffleNet (Smonyyn & Zisserman 2014), Wide Residual Network (WRN-d-w) (Zagoruyko & Komodakis (016) (with depth d and width factor w) and ResNet (He et al., 2016). To evaluate the ProKTT' under different dis- tillation loss, conduct the ProKT with standard KL divergence loss and contrastive representation distillation loss proposed by CRD (Tian et al. 2019).
Baselines. We compare model with the following baselines: vanilla KD (Hinton al., (015)) CRD (Tian et al.) (2119) and RCO (Jin et al. 2019). Results of baselines are from the report of Tian
al. 2019), except for the RCO (Jin 2019). which is implemented by ourselves.
4.1.2
TEXT CLASSIFICATION
Text classification experiments are conducted following the setting of (Ture et al. 2019 and Jiao et al. 2019b) on the GLUE (Wang et al. 2018) benchmark.
Datasets. We evaluate our method for sentiment classification on SST-2 (Socher et al. 2013).
natural language inference on MNLI (Williams et al. 2017) and QNLI (Rajpurkar et al. 2016), and paraphrase similarity matching MRPC (Dolan & Brockett 2005) and QQI[]
2018a) fine-tuned in the training set, which is 12-layer Transformers (Vaswani al. (017) 7) with 768 hidden units. Following the setting of Ture et al. (219)) and Diao et al. (2019b), a BERT of 6 layer Transformers and 786 hidden units is conducted the student model. For distillation between heterogeneous architectures, we use single-layer bi-LSTM with 300 embedding size and 300 hidden size as student model. We implement the basic ProKT with standard KL divergence loss, and combine our method with the TinyBERT (Jiao et al. 2019b) by replacing the second stage of fine-tuning TinyBERT with our ProKT. More experimental details are listed in the supplementary materials.
Settings. The teacher model the BERT-base (Devlin et al.
Baselines. We compare our method with following baselines: (1) BERT + Finetune, fine-tune the BERT student on training set; (2) BERT/bi-LSTM: KD, fine-tune the BERT student or train the bi-LSTM on training set using the vanilla knowledge distillation loss (Hinton et al., 2015); (3) Route Constrained Optimization (RCO) (Jin et al., 2019), use un-convergent teacher checkpoints as intermediate training targets; (4) bi- train Si-LSTM in training set; (5) TinyBERT (Jiao et al.) 2019b); match the attentions and representations of student model with teacher model on the first stage and then fine-tune by the vanilla KD loss the second stage. For vanilla KD methods, we set the temperature as 1.0 and only use the KL divergence with teacher outputs as loss. We also compare method with the results reported by Sunn et al. (2019) and Ture et al. 2019).
4.2 RESULTS
Results of image classification on CIFARI00 100 are shown in Tab. 1 The performance is evaluated by top-1 accuracy. Results of text classification are shown in Tab. 2 The accuracy fl-score on test set are obtained by submitting to the GLUE (Wang et al. 2018) website. Results on both text and image classification tasks show that ProKT achieves the best performance under almost all model settings.
Ruleeea-eeeeeuueon--Piiiiiii Question-Pairs.

---

Under review as a conference paper at ICLR 2021
Table 1: Top-1 test accuracy (%) of student networks distilled from teacher with different network architectures CIFAR100. Results except the RCO, ProKT and CRD+ProKTT are from Tian et al. (0019).
Teacher Student
vgg 13
ResNet50 MobileNetV2
ResNe150 vgg8
resne(32x4
resnet32x4 ShuffleNetV2
WRN-40-2
MobileNetV2
ShuffleNetV1
ShuffleNetVi
Teacher
74.64
79.34
79.34
79.42
79.42
75.61
Student
64.6
64.6
70.36
70.5
71.82
70.5
KD)
67.37
67.35
73.81
74.07
74.45
74.83
RCO
68.42
68.95
73.85
75.62
76.26
75.53
ProKT
68.79
69.32
73.88
75.79
75.59
76.02
CRD
69.73
69.11
74.30
75.11
75.65
76.05
CRD+KD
69.94
69.54
74.58
75.12
76.05
76.27
CRD+ProKT
69.59
69.93
75.14
76.0
76.86
76.76
Table 2: Test results of different knowledge distillation methods in GLUE.
Model
SST-2
MRPC
QQP (fl/ace)
MNLI (acc m/mm)
QNLI
(acc)
(fl/acc)
(acc)
BERT 12 (teacher)
93.4
88,0/83.2
71.4/89.2
84.3/83.4
PF Ture
2019
91.8
86.8/81.7
70.4/88.9
82.8/82.2
88.9
Sun ct al. 2019)
92.0
85.0/79.9
70.7/88.9
81,5/81
89.0
BERT. Finetune
92.6
86.3/81.4
70.4/88.9
82.0/80.4
89.3
BERT; KD
90.8
86.7/81.4
70.5/88.9
81.6/80.8
88.9
BERTs + RCO
92.6
86.8/81.4
70.4/88.7
82.3/8 1.2
89.3
BERT, ProKT (A 0)
92.9
87.1/82.3
70.7/88.9
82.5/81.3
89.4
BERT, ProKT
93.3
87.0/82.3
70.9/88.9
82.9/82.2
89.7
TinyBERT, Jiao al.
2019b
93.1
87.3/82.6
71.6/89.1
84.6/83.2
90.4
TinyBERT, ProKT
93.6
88.1/83.8
71.2/89.2
84.2/83.4
90.9
bi-LSTM
86.3
76.2/67.4 O
60.1/80.7
66.9/66.6
73.2
bi-LSTM KD
86.4
77.7/68.1
60.7/81.2
68.1/67.6
72.7
bi-LSTM + RCO
86.7
76.0/67.3
60.1/80.4
66.9/67.6
72.5
bi-LSTM + ProKT (A =
86.2
80.1/71.8
59.7/79.7
68.4/68.3
73.5
bi-LSTM + ProKT
88.3
80.3/71.0
60.2/80.4
68.8/69.1
76.1
Results show that the continuous and dynamic targets are helpful to take advantage of the knowledge from the teacher. Although adopting discrete targets in RCO could improve the performance to vanilla KD, our ProKT with continuous and dynamic targets is more effective in teaching student. To further show the effectiveness continuity and adaptiveness (i.e., the KL divergence term student in the update of teacher) in ProKT respectively, we test the results of ProKT with = 0, in which the targets are improved smoothly but without the adjustment towards the student. As shown in Tab, 2 the continuous targets are better than discrete targets (i.e., RCO), while incorporating the constraint from student when updating teacher could further improve performance.
ProKT is effective as well when it is combined with different objective of knowledge distillation. When combined with contrastive representation learning loss in CRD, as shown in Tab. [ and combined with TiRyBERT in Tab. 2 ProKT could further boost the performance and achieves the state-of-the-art results in almost all settings.
ProKT is especially effective when the student is of different structure with teacher. As shown in Tab. 2] when the student is bi-LSTM, directly distilling knowledge from a are-trained BERT has a minor effect. ProKT could improve larger margin for bi-LSTM than small BERT when distilled from BERT-base. Since learning from a heterogencous teacher is more difficult, exposing teacher's training process to student could offer better guidance to the student.
7

---

Under review as a conference paper at ICLR 2021
train loss
train acc
6.
40
50
150 epoch
100
200
250
20
0
50
100
150 epoch
200
250
(a) Train loss
(b) Train accuracy
Figure 2: Training loss and accuracy for MobileNetV2 distilled from ResNet50 CIFAR 100.
ProKT
test acc
valid acc
83
68-
67
ProKT
82
0.0
0.2
0.0
0.2
0.4
À
À
(a) VGG13 to MobileNetV2 CIFAR 100. (b) BERT BERT, in MNLI-mm.
Figure 3: Test/valid accuracy with different value of X for teacher and student model in ProKT.
4.3 DISCUSSION
4.3.1 TRAINING DYNAMICS
To visualize the training dynamics of teacher model and student model, show the training loss of student model and the training accuracy of teacher model in Fig. 2 The training losses are calculated by the KL divergence between the student model and their intermediate targets. Fig. shows that the divergence between student and teacher in ProKT (i.e., the training loss for ProKT) is smooth and well bounded the beginning of training. However, at the target switching points, there are impulses in the training curve and then the loss is kept to a relative larger value.
relative small value. For discrete targets in RCO, the divergence is bounded well in
Then, we examine the performance of teacher model in Fig. 2D ResNet50 refers to the teacher model which is trained by vanilla loss. While the ProKT-T denotes teacher model which updated by the ProKT loss. It could be found that the performance of teacher model in ProKT deteriorates because of the "local" constraint from student. However, the lower training accuracy for teacher model does not affect the training performance of the student model as illustrated in the Tab, I which further justified our intuition of involing local targets.
4.3.2 ABLATION STUDY
To test the impact of the constraint from student in Eq. test and valid accuracy with respect to different À for image and text classification tasks are shown in Fig. 3 It is illustrated that the performance is improved in appropriate range of X, which means that the constraint term helpful to provide appropriate targets. However, when the À is too large, the regularization from student will heavily damage the training teacher and the performance of student will drop.
5 CONCLUSION
We propose a novel model agnostic knowledge distillation method, ProKT. The method projects the step-by-step supervision signal on the optimization procedure of student with an approximate mirror descent fashion, i.e,, student model learns from dynamic teacher sequence while the progressive teacher is aware the learning process of student. Experimental results show that ProKT achieves good performance in knowledge distillation for both image and text classification tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizll Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Wenhu Chen, Guanlin Li, Shuo Ren, Shujie Liu, Zhirui Zhang. Mu Li, and Ming Zhou, Generative bridging network in neural sequence prediction. arXiv preprint arXiv: 1706 09/52, 2017.
Jacob Devlin, Ming- Min Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018a.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xX.v:8110.04800 2018b.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
He He, Jason Eisner, and Hal Daume. Imitation learning by coaching. In Advances in Neural Information Processing Systems, pp. 11497,157, 2012.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration mobile devices. In Proceedings the European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02533 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1909. 2019a.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1909. 10351, 2019b.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1345-1354, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hoang M Le, Andrew Kang, Yisong Yue, and Peter Carr. Smooth imitation learning for online sequence prediction. arXiv preprint arXiv: 1660600066 2016.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pp. 1-9, 2013.

---

Under review as a conference paper at ICLR 2021
Sidi Lu, Lantao Yu, Siyuan Feng, Yaoming Zhu, Weinan Zhang, and Yong Yu. Cot: Cooperative training for generative modeling of discrete data, arXiv preprint FrXiv:180.003755 2018.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv:1902.03393, 2019.
William H Montgomery and Sergey Levine. Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems, pp. 4008-44800 4016, 2016.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Internationa Conference on Machine Learning, pp. 5142-5151, 2019.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint arXiv: 100.X0::000005666 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-ss-west amazonaws. com/openai- aasets/reseearccooeeeiee hharr//eearrooovev/uggggggguuuer
understanding paper. pdf. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprchension of text. arXiv preprint arXiv: 05250, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.15566 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:/908.099 2019.
llya Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted boltzmann machine. In Advances in neural information processing systems, pp. 1601-1608, 2009.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv 2019.
preprint 1910.
Iulia Ture, Ming-We Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint 1908 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008. 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. 2018
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXivv 2017..
Jaxiaag Wu, Cong Leng, Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4820 4828, 2016.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4133-4141 2017.
10

---

Under review as a conference paper at ICLR 2021
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018a.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320-43333 4320-43328, 2018b.
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang. Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence adaptive learning rate methods. arXiv preprint arXiv: /10.00300, 2018.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. EXPERIMENTAL DETAILS FOR TEXT CLASSIFICATION
We use the pre-trained BERTs released by Ture al. (201) except for TinyBERTs. For TinyBERTs, we use the pre-trained model released by Jiao et al. (2019h
We fine-tune 4 epoch for non-distillation
training and epoch for distillation training. Adam (Kingma & Ba 2014) optimizer with learning rate 0.001 is used for biLSTM and with learning rate from {3e-5, 5e-5, le-4) is used for BERTs. The hyperr-aramette of Eq. is chosen according to the performance in the validation set. For ProKT in BERyBERT,
the data argumentation following Jiao et al. (2019b).
A.2 FULL COMPARISON OF KD IMAGE RECOGNITION SEC EXPERIMENT RESULTS OF HOMOGENEOUS ARCHITECTURE KD IN IMAGE RECOGNITION
We provide the full comparison of method with respect several additional knowledge distillation methods as extension in the Table. 3
Table Top-1 test accuracy (%) of student networks distilled from teacher with different network architectures on CIFAR100, Results except the RCO, ProKT and CRD+ProKT are from Tian et al. (2019).
Teacher Student
vgg 13
ResNet50 MobileNetV2
ResNe150 vgg8
resmet32x4 ShuffleNerVi
resnet32x4 ShuffleNetV2
WRN-40-2
MobileNetV2
ShuffleNerVi
Teacher Student
74.64
79.34
79.34
79.42
79.42
75.61
64.6
64.6
70.36
70.5
71.82
70.5
KD
67.37
67.35
73.81
74.07
74,45
74,83
FitNet
64.14
63.16
70.69
73.59
73.54
73.73
AT
59.40
58.58
71.84
71.73
72.73
73.32
68.08
73.34
73.48
74.56
74.52
CC
64.86
65.43
70.25
71.14
71.29
71.38
VID
65.56
67.57
70.30
73.38
73.40
73.61
RKD
64.52
64.43
71.50
72.28
73.21
72.21
PKT
67.13
73.01
74.10
74.69
73.89
AB
66.06
67.20
70.65
73.55
74.31
73.34
FT
61.78
60.99
70.29
71.75
72.50
72.03
NST
58.16
64.96
71.28
74.12
74.68
74.89
RCO
68.42
68.95
73.85
75.62
76.26
75.53
ProKT
68.79
73.88
75.79
75.59
76.02
CRD
69.73
69.11
74.30
75.11
75.65
76.05
CRD+KD
69.94
69.54
74.58
75.12
76.05
76.27
CRD+ProKT
69.59
69.93
75.14
76.0
76.86
76.76
12


Reviews:
[{"review_id": "rUjhkaW900", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**  \n\nThis paper presents a method of knowledge distillation, which showed better results than KD and RCO in experimental results. The difference between this method, ProKT and RCO is in the teacher model. In this method, the snapshot interval of the teacher model targeted by the student is more frequent (every training step) than the RCO (authors called this property as *continuous*), and the parameters of the teacher model are also updated during *student* training (*dynamic*).\n\n**Strong points**\n\n- The proposed method was evaluated for two modalities (visual and textual). As far as I know, most KD papers evaluate only one modality (image classifier or language model). Experimenting with both modalities increases the experimental robustness of ProKT.\n\n**Weak points**\n\n- This method is not traditional KD, and the way the teacher model learns simultaneously with the student is called mutual learning, as the author briefly mentioned in the paper. There is a terminological problem that arises because of this: if the teacher model is also trained with the student, then neither model can be called a \"teacher\" or \"student\". Thus I think that most of the terms used in the paper need to be rewritten based on mutual learning terminology.\n- In advance to the previous point, sharing knowledge between the big models and small models during the training is not new. To name a few, see [1, 2, 3].\n- Aside from everything, the performance improvement that ProKT alone brings seems limited compared to CRD, and it is difficult to trust because there is no information such as the number of runs or standard deviation in the results of the experiment.\n\n[1] Zhang, Ying, et al. \"Deep mutual learning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  \n[2] Anil, Rohan, et al. \"Large scale distributed neural network training through online distillation.\" arXiv preprint arXiv:1804.03235 (2018).  \n[3] Park, Wonpyo, et al. \"Diversified Mutual Learning for Deep Metric Learning.\" arXiv preprint arXiv:2009.04170 (2020).  \n\n\n**Recommendation**\n\nI vote for rejection as the paper lacks in novelty and results in small performance improvement.\n\n**Supporting arguments for the recommendation**\n\nThe recommendation is based on what was said in the weakness section. In addition, RCO does not provide any information from the student to the teacher during the training (just as Hinton KD or CRD does), so for a fair comparison, it seems necessary to compare the performance with RCO by separating only *continuous* facet among the two properties of ProKT (*continuous* and *dynamic*) claimed by the authors.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ol-UgnIGof", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview summary: \n\nThis submission studies the knowledge distillation approach for deep learning models. The aspect is focused on the intermediate target models when learning for the student models. The authors claimed the previous teacher targets are poor to make student model to local optima or be unstable. The authors propose a new \"model-agnostic\" method, named ProKT, proximal knowledge teaching, which is implemented by decomposing the training objective of the student model into several local intermediate targets with approximate mirror descent technique. Specifically, the training of the teacher model and student model is step-wise, which means the teacher model updates and distills the knowledge after one step, and the student model then takes the teacher model at the current step as a target, and train the student model itself. To make the training more stable or less sensitive, the teacher model is constrained by a gap bounding between the teacher output distribution and student output distribution (KL divergence). The authors then evaluate their method on image classification task (CIFAR-100) and several text classification tasks, under different model structures, different training objectives. The results show that ProKT is able to achieve effective performance compared to previous knowledge distillation methods. \n\nGeneral comments:\nOverall speaking, the motivation of this submission is clearly described and the idea is easy to follow. I understand the authors are making a better job towards the distillation approach. But after carefully reading the method and the experimental results, I do have several concerns, which make me feel things can be better improved or better modeled. Please refer to my following detailed comments on the pros and cons. \n\nPros:\n1. First, I personally like the idea of better studying knowledge distillation problem, since kd is a very important and effective method in current deep learning models. This submission studies the internal learning path of the student model from the teacher model, with the viewpoint of how to modify the teacher model instead of the student model. The main claim is about the teacher targets (multiple intermediate targets) should be continuous so that the student model can be more stable to follow the guided signals. This is a good point.\n2. The authors present a clear description of the motivation and the proposed approach, which is very easy to follow. As for the implementation, it seems easy to do, though the authors do not open source their implementation. \n3. The authors conduct experiments on two different domains, which are the image and text classifications. The verification is performed under different settings (models, training objectives) so to demonstrate the universal usefulness of the approach. \n\nCons:\nThough the interesting points and the general well modeling. I do feel several concerns as follows:\n1. To be honest, the proposed method mainly do a step-wise distillation for the student model. To make a better constraint and stable training, the teacher model is constrained with the learning of the student model through a KL divergence between the two output distributions. This contribution is a little bit limited. First, the step-wise distillation is straight forward, the authors mainly criticize the discrete targets from previous methods, but the continuous setting modeled in this paper is not a very elegant way or surprising way. This modeling would cost several problems (as I listed in the below comments). This method is also very similar to mutual training, instead of the teacher and student models are different architectures. Second, the constraint between the student model and teacher model is somehow hard to make a clear convincing claim. In my opinion, the teacher model aims to make a strong performance by learning from the supervised target $y$, the goal mainly targets at the performance, this constraint may somehow limit the performance of the strong teacher model (e.g., figure 2(b)). Though the teacher model wants to provide a continuous signal for the student model (as the author claimed), the \"step-wise learning/distillation\" can guarantee this assumption in most cases for the student model. The probable large gap is not very possible (if the author can give several strong examples). Therefore, this is not so convincing.\n2. The continuous targets distillation have some problems from my side. For example, the training cost is increased for the student model. Previous knowledge distillation methods usually do several times distillation or only one-time distillation, which should be efficient to train since the student model is smaller than the teacher model, therefore the training step is less and the convergence speed is faster. With the step-wise distillation, the update steps are increased to be the same as the teacher model.  Besides, for ProKT, the memory (e.g., internal backpropagation parameters) request is more than previous methods, since the approach should update the teacher model and student model at the same time. While for previous works, only the update for the student model parameters required. \n3. As for the experiments, in Table 1, the gap between RCO and ProKT is really limited, which makes it hard to say that ProKT is stronger or more effective than RCO. The performances are similar, only slightly better from ProKT. A more confused setting is in Table 2, the BERT$_6$ + ProKT ($\\lambda=0$) is similar to the next line, which means the constraint is not so important (though in bi-LSTM, it shows better performance), or at least not stable/universally work (somehow this verifies my point of the small gap between teacher and student model distributions of my point 2). This seems to be a negative point for the verification. The differences between the bi-LSTM and BERT experiments also should be better claimed. \n\nSeveral other points and questions:\n1. The detailed values for $\\lambda$ and $\\alpha$ should be reported, in order to give a clear setting. \n2. In figure 2(a), the bleu line fo the student model training is interesting, it first increases the loss, do the authors have more explanations?\n3. In figure 3(a), the better way is to show the valid acc instead of test acc, this would be fairer.\n4. Minor errors, in 4.3.1, the third line of the second phrase, \"It could be found hat xxx\" -> \"It could be found that xxx\". \n5. A suggestion is that the detailed parameter number should be reported for the teacher and student models. \n\n-------------------\nupdated comments:\nI thank the authors to provide responses before the last time of rebuttal. Some of the questions are addressed, but it seems many points still keep disagreement. For example, the contribution is still not clear to me The training cost is increased a lot, this can not be ignored, though I understand in the industry it may be less important. However, for academic research, we should care about this. Besides, it seems that all reviewers agree with the limited contribution and unsatisfied experiments. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nJ6JuczOdB1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper follows the work of RCO[1], where knowledge distillation is conducted by learning from the optimization trajectories of the teacher rather than the converged teacher solely. The main difference is that in the proposed method ProKT, the student model learns from the teacher model step by step, while RCO downsamples the teacher's training trajectory with some sampling strategies, such as equal epoch interval strategy and greedy search strategy. The authors argue that smoothly changed targets can help the student out of poor local optima.\n\nPros:\n1. Although some few works have made attempts to explore the training route of teacher for distillation, this research direction is still under-studied and of great potentials in my view.\n\n2. The idea of making the teacher aware of the student for distillation is interesting.\n\nCons:\n1. My main concern is that the proposed ProKT is highly similar to some prior works such as deep mutual learning [2], KDCL [3], and PCL [4] from the perspective of methodology, although their motivations are somewhat different. This paper lacks comparisons and clarifications of the differences between the proposed method and these prior methods.\n\n2. The experiments are not sufficient to validate the proposed method. On image recognition tasks, comparisons are only conducted under heterogeneous architectures. However, distillation results under the same-style architecture should also be provided to give a more comprehensive view. \n\n3. The proposed ProKT actually demands several times more computation resources than other methods like KD. Strictly speaking, evaluating the proposed method in accuracy only is not fair. How to make the proposed method more efficient, or how to make more fair comparisons should be carefully considered.\n\n[1] Knowledge Distillation via Route Constrained Optimization, ICCV 2019.\n[2] Deep mutual learning, CVPR 2018\n[3] Online Knowledge Distillation via Collaborative Learning, CVPR 2020.\n[4] Peer Collaborative Learning for Online Knowledge Distillation.\n\n==============\npost-rebuttal:\nI have read all the comments from other reviewers and replies from the authors. All the reviewers are leaning to reject this paper due to the limited novelty and unfair and incomplete experimental comparisons. The authors' reply does not address my concerns, so I keep my initial attitude towards this work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OvZFUDriAZL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method is simple yet effective and achieves state-of-the-art results on both image and text classification tasks. Compared to the existing approach (RCO and TA-learning), the main advantage is that ProKT avoids manual control of the learning process, which makes the ProKT more generally applicable. Important related works such as RCO and TA-learning are cited and compared. The paper does not contain a theory part, but wherever possible, equations are provided to illustrate how the method works.\n\nConcerns: \nFrom my understanding, ProKT is a continuous version of RCO with an extra constraint on the KL divergence of the probabilistic distributions of the teacher and the student models. As listed in Table 1, the accuracy improvement over RCO is marginal. The computational cost of ProKT is much higher than RCO as it requires training the teacher and student simultaneously. Therefore, I think the authors need to justify the benefits over RCO other than avoiding selecting the discrete learning objectives manually. From my understanding, ProKT is a continuous version of RCO with an extra constraint on the KL divergence of the probabilistic distributions of the teacher and the student models. As listed in Table 1, the accuracy improvement over RCO is marginal. The computational cost of ProKT is much higher than RCO as it requires training the teacher and student simultaneously. Therefore, I think the authors need to justify the benefits over RCO other than avoiding selecting the discrete learning objectives manually. \n\nMoreover, this paper only provides an empirical evaluation on the CIFAR-100 dataset. It will be useful to see whether ProKT can achieve higher student accuracy on a more complex dataset such as ImageNet. The variability of results is also missing since there are no error bars for the results in Tables 1 and 2. Are the results averaged over multiple trials (if yes how many?), and is there a difference in variance between the methods?\n\nIn the proposed ProKT algorithm, both the student and teacher models are trained with the same number of iterations, and also the student model learns from an updated teacher model in each training step. Although the continuous and smooth learning targets might be desired,  I am curious about the performance of the possible variants of the ProKT algorithm. For example, let the teacher be several iterations ahead of the student or keep the same learning objective (teacher model) for several iterations, which can be viewed as something between ProKT and RCO.\n\nMinor detail:\nIn Algorithm 1, it shows that the student model is updated before the teacher model. Should it be the teacher gets updated first?\n\nReasons for score: In general, I like the idea of turning the discrete learning objective into a continuous one. However, as the proposed algorithm is largely inspired by RCO, the novelty of the paper is incremental. Besides, the performance improvement of the proposed ProKT scheme over RCO also is marginal. I would consider raising my score if the authors could better demonstrate the advantage of ProKT over RCO.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new variant of knowledge distillation. The basic idea is interesting although similar ideas have more or less appeared in the literature as pointed out by the reviewers. Our main concern on this work is that the real empirical improvements are too limited such that it is hard to conclude that the proposed method can really perform better than the baseline. In the meantime, the proposed method is much more computationally expensive. 
Decision: Reject