Explicit Connection Distillation
Authors: Lujun Li
Yikai Wang
Anbang Yao
Yi Qian
Xiao Zhou
Ke He
Venue:  (2021)

Abstract: One effective way to ease the deployment of deep neural networks on resource constrained devices is Knowledge Distillation (KD), which boosts the accuracy of a low-capacity student model by mimicking the learnt information of a high-capacity teacher (either a single model or a multi-model ensemble). Although great progress has been attained on KD research, existing efforts are primarily invested to design better distillation losses by using soft logits or intermediate feature representations of the teacher as the extra supervision. In this paper, we present Explicit Connection Distillation (ECD), a new KD framework, which addresses the knowledge distillation problem in a novel perspective of bridging dense intermediate feature connections between a student network and its corresponding teacher generated automatically in the training, achieving knowledge transfer goal via direct cross-network layer-to-layer gradients propagation. ECD has two interdependent modules. In the first module, given a student network, an auxiliary teacher architecture is temporarily generated conditioned on strengthening feature representations of basic convolutions of the student network via replacing them with dynamic additive convolutions and keeping the other layers unchanged in structure. The teacher generated in this way guarantees its superior capacity and makes a perfect feature alignment (both in input and output dimensions) to the student at every convolutional layer. In the second module, dense feature connections between the aligned convolutional layers from the student to its auxiliary teacher are introduced, which allows explicit layer-to-layer gradients propagation from the teacher to the student via the merged model training from scratch. Intriguingly, as feature connection direction is one-way, all feature connections together with the auxiliary teacher merely exist during training phase. Experiments on popular image classification tasks validate the effectiveness of our method. Code will be made publicly available.

Under review as a conference paper at ICLR 2021
EXPLICIT CONNECTION DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
One effective way to ease the deployment of deep neural networks on resource constrained devices is Knowledge Distillation (KD), which boosts the accuracy of low-capacity student model by mimicking the learnt information of a high- capacity teacher (either a single model or multi-model ensemble). Although great progress has been attained on KD research, existing efforts are primarily invested to design better distillation losses by using soft logits or intermediate feature representations of the teacher as the extra supervision. In this paper, we present Explicit Connection Distillation (ECD), new KD framework, which ad- dresses the knowledge distillation problem in novel perspective of bridging dense intermediate feature connections between a student network and its correspond- ing teacher generated automatically in the training, achieving knowledge transfer goal via direct cross-networt layer-to-layer gradients propagation. ECD has two interdependent modules. In the first module, given student network, auxiliary teacher architecture is temporarily generated conditioned on strengthening feature representations of basic convolutions of the student network via replacing them with dynamic additive convolutions and keeping the other layers unchanged in structure. The teacher generated in this way guarantees its superior capacity and makes perfect feature alignment (both in input and output dimensions) to the stu- dent every convolutional layer. the second module, dense feature connections between the aligned convolutional layers from the student to its auxiliary teacher are introduced, which allows explicit layer-to-layer gradients propagation from the teacher to the student via the merged model training from scratch. Intriguingly, as feature connection direction is one-way, all feature connections together with the auxiliary teacher merely exist during training phase. Experiments popular image classification tasks validate the effectiveness of our method. Code will be made publicly available.
INTRODUCTION
Deep Neural Networks (DNNs) have achieved great success in tackling variety of visual recognition tasks (Krizhevsky et al. 2012 Girshick et al. 2014 Long et al. 2013). Despite the appealing perfor- mance, the prevailing DNN models usually have large numbers of parameters, leading to heavy costs of memory and computation. Conventional techniques such as pruning weights from networks (Han) 2017) and quantizing networks to use low-bit parameters (Courbariaux et al. 2015 Rastegari et al. 2016 Zhou et al. 2016) have proven to be effective for mitigating this compu- lational burden. More recently, Knowledge Distillation (KD), another promising solution family to get compact yet accurate models, has attracted increasing attention.
al. 2015 Li al.
The goal of KD is to transfer the learnt information (knowledge) of high-capacity DNN model or an ensemble of multiple DNN models (teacher) to a low-capacity target DNN model (student). striking better securac--ffficience tradeoff's at runtime. There exist numerous KD methods address the knowledge transfer from the teacher to the student. Many of them use two-stage training process which begins with training à teacher model and keeping it fixed, and then learns a target student model by forcing it to match the outputted knowledge from the pre-trained teacher model. Various types of knowledge have been explored, such as outputted logits (Ba & Caruanal 2014 Hinton et al. 2015). intermediate feature representations (Romero al. Zagoruuoo & Komodakis 2017). and relational information model outputs or representations (Park et al.) 2019[911]]]aa et al. 2020). Instead of defining new types of knowledge, the other methods adopt one-stage training process,

---

Under review as a conference paper at ICLR 2021
jointly training the student and teacher/peer models using bidirectional knowledge distillation (Zhagg et al. 80 Yao & Sun 2020) or on-the-fly ensemble distillation (Lan et al.] 2018: Anil et al. 2018 or multi-exit distillation (Phuong & Lampert 2019). Both two-stage and one-stage KD methods as above typically treat the knowledge transfer as an optimization problem of formulating robust distillation loss functions via introducing more informative knowledge supervisions and more
effective knowledge matching strategies.
Instead of extracting informative knowledge and designing alternative distillation loss functions to fit a desired knowledge transfer optimization objective as existing KD methods, we investigate a new technical perspective in this paper: casting the knowledge distillation problem into designing auxiliary connection paths between the student and teacher networks to enable explicit layer-to to loyer gradients distillation from the teacher to the student via training them from scratch simultaneously. We are partially inspired by recent advances on DNN architecture engineering, which show that designing sophisticated feature connection paths such as residual connections (He et al. 2016) and dense connections (Huang et al. 2017) across neighboring layers can allow better information and gradients flow throughout a single network, making the training easy to have significantly improved performance in model accuracy and convergence. We conjecture this simple principle would also be crucial to open the door to develop a totally new knowledge distillation framework if we can merge the student and teacher into single network temporarily during training phase, and can also separate them easily after training. To explore this hypothesis, we present Explicit Connection Distillation (ECD), very simple knowledge distillation framework. Specifically, we decompose the design of ECD into two interdependent modules, namely auxiliary teacher generation and dense feature connection distillation. Recent works (Liu et al. 2020 Yue al. 2020) show that searching a good alignment of structural feature channels between the student and teacher networks can bring improved knowledge distillation performance under the premise that a pre-trained teacher model available. Regarding the first module of ECD, we hope the generated teacher can make perfect structure alignment (both in input and output feature dimensions of every convolutional layer) to the student network in an easier manner (no need of consuminn searching procedure as used in (Liu et al. 2020] Yue et al.] (0200) while can attain superior model capacity. To this goal, we retain all structural units of the student network in constructing the teacher architecture, except for replacing original convolutions by dynamic additive convolutions (Yang et al. (019) which have proven to be very effective for enhancing model capacity in network architecture engineering research. Regarding the second module of ECD, we hope knowledge distillation can be realized through the explicit layer-to-layer flows of gradients from the teacher the student instead of the conventional mimicking procedure. To this goal, we add dense feature connections from the convolutional layers of the student to those aligned layers of the auxiliary teacher, and train the merged model from scratch. After training, all feature connections can be naturally removed as their connection direction is only from the student to its auxiliary teacher which also exists merely in training phase.
Beyond the common wisdom of knowledge distillation that requires the knowledge mimicking process between the student and teacher networks, our ECD sheds new insight: by considering knowledge distillation from novel student-to-teacher merging, co-training and splitting viewpoint, direct feature connections from the student to its well-aligned auxiliary teacher (generated from the student in automatic manner) can also achieve competitive performance improve low-capacity student models, as validated by extensive experiments on CIFAR- 100 and ImageNet datasets.
2 RELATED WORK
In this section, we make a brief summary of existing knowledge distillation works.
Two-stage KD methods. The idea of training a compact model to mimic the functions learnt by larger ensemble of models is firstly proposed in (Bucilà et al., 2006). Ba & Caruana (2014) extends this idea, showing that shallower yet wider neural networks can also approximate the functions previously learnt by deep ones. Hinton et al. (2015) presents the famous Knowledge Distillation (KD) method, which adopts teacher-student framework for transferring learnt soft knowledge from a nigh-capacity teacher to a low- cow-cipyyyty target student network. In this framework, the teacher is pre- mreitraine and fixed, and then its soft logits on the training data are used as the extra supervision to guide the training of the student besides the ground truth labels. FitNets (Romero et al.) 2015) shows the intermediate feature representations learnt by the teacher can be used as the complementary
2

---

Under review as a conference paper at ICLR 2021
knowledge to soft logits, enhancing knowledge distillation performance to some extent. Zagoruyko & Komodakis (2017) proposes to use spatial attention activations instead of intermediate feature maps. Recent works further show the relations (Park et al. 2019) and higher order dependencies (Tian et al. 2020) of learnt logits or intermediate feature representations by the teacher capture important more effective knowledge augment the knowledge distillation process. Following the teacher-student framework as above, many other works such as (Yim et al.
structural information, which
2017 Lee al. 2018 Kim et al. 2018) attempt to further improve knowledge representations and matching losses.
One-stage KD methods. Unlike two-stage KD methods that rely on a pre- pre-trained teacher model and one-way knowledge transfer from the teacher to the student, one-stage KD methods simplifying the knowledge distillation process by training all models simultaneously. Deep mutual learning (DML) (Zhang et al. 201 8a) considers the teacher as peer of the target student, showing that the outputted logits of the student can be used to assist the training of the teacher by a peer-teaching strategy. In Yao & Sun 2020), the authors further show that deep mutual learning can also be boosted by intermediate soft outputs. just like that in two-stage KD methods. Guo et al. (2020) follows the basic framework of DML but replaces the mutual distillation sttutent-t--tuudent)) by the ensemble (over students)- taresss--ttotttttt tossudeentss distillation. ONE (Lan et al.) 2018) presents an on-the-fly fly ensemble distillation method in which stronger ensemble teacher learnt over multi-branch network is used to enhance the training of every branch. Aniil et al. (2018) extends this idea to accelerate large-scale distributed neural network training applications. [Wu & Gong (2020) follows the basic framework of ONE but jointly uses the mutual distillation (student-to-student and the ensemble- to-sttudees-ss distillation. Other improved variants include but not limited to (Phuong Lampert 2019 Malinin
et al.
2020
Other KD variants. Besides model compression, there also exist a lot of works using knowledge distillation methodology to handle other applications. celf-distillation methods (Furlanello et al. 2018 [Bagherinezhad et al., 2015) explore the benefits of KD techniques to improve the model training, assuming only single network is available. [Lopes et al. (2015) and Chen et al. (2019) address data-free knowledge distillation where the original training data are no longer accessible due to safety or privacy concerns, Gollblum et al. (2020) and Chung ct al. (2020) extend the idea of KD to study how adversarial robustness of the teacher model can be transferred to the student. Li & Hoiem (2016) and Hou et al. (2018) combine KD with fine-tuning and retrospection to handle life-long learning scenarios. Adapting KD to other tasks such as multi-modal visual recognition and
natural language processing is explored in (Garcia et al. 2018 Kim & Rush 2016).
3 EXPLICIT CONNECTION DISTILLATION
In this section, we describe the formulation of the proposed Explicit Connection Distillation (ECD) framework, and detail how to design and implement its two key components: auxiliary teacher generation and dense feature connection distillation.
3.1 CONVENTIONAL KD FORMULATION
For better understanding of our method, we start with the formulation of conventional KD methods. Let S, T and 0s, 0r denote a target student model, its teacher (either a single model or an ensemble of multiple models) and their parameters correspondingly, and let denote training data, and let Q be a set of layer location pairs where knowledge mimicking process between two networks is introduced. Regarding one-stage KD methods, where the student and teacher models are trained jointly, the overall objective function to be optimized can be defined as
LKD Lcr(Os, Len(Or, x)) À Σ {((())
QEQ
where LCu Lcu(fs, .)) and (er(Or, .)) are the standard cross-entropy loss functions of the student model and the teacher model, respectively. is the loss for knowledge mimicking, which measures the distance of learnt knowledge (z)) (x) at all specific layer location pairs between the student and teacher models. As we discussed in introduction section, intermediate feature representations,
3

---

Under review as a conference paper at ICLR 2021
Student
Forward
Backward gradient
Ausomatic geseration
Ground truth
of S
Student
92
dt
off
RC
Training (generate, merge, co-train)
Inference (split)
Figure I: A schematic overview the training stage and inference stage of our ECD, the training phase, the teacher network is automatically generated by taking the student the reference, and then both networks are merged by adding dense feature connections and co-trained from scratch. In the inference phase, in order to obtain the prediction (p), only the student network is needed which is simply split from the teacher.
soft logits, and their relations from learnt models popularly used the knowledge. is tunable weighting factor to balance loss terms, which is usually initialized to a relatively large value and decays during training. Simply removing the loss (cE(Or, x) from Equation (1) gets the objective function of two-stage KD methods, as in them the teacher model is pre-trained and fixed.
3.2 FORMULATION OF ECD
The goal of our ECD is to realize knowledge transfer from the teacher to the student without need of the common knowledge mimicking process described above, making KD design simple as possible. That is, for ECD, the optimization objective is
LECD (cn(Os, (cr(Or,
As illustrated in Figure merely given student network and training data, our ECD achieves such goal by smart teacher- gebbbrrignnr,,nng, student-to-feacher merging, co-training and splitting framework, consisting of two interdependent modules:
Auxiliary teacher generation module (in $ 3.3): A teacher model, which structurally aligns with the given student any network depth, is generated automatically from the student. Thus there is no longer any need to prepare large teacher architectures.
Dense feature connection distillation module (in $3.4): The student model is temporarily merged with the teacher by adding explicit layer-to-layed connections to the teacher. The knowledge is transferred to the student during co-training of the merged model. Thus there is no longer any need to design distillation losses and tune À to balance different losses.
3.3 AUXILIARY TEACHER GENERATION
Unlike existing KD methods which usually need to prepare different teacher architectures for their corresponding student networks, in method, a teacher model is generated automatically from the student by the auxiliary teacher generation module. To guarantee superior capacity of the generated teacher network and make a perfect structure alignment the student network simultaneously, this module retains all structural units of any given student network in generating auxiliary teacher architecture, but replaces original convolutions by dynamic additive convolutions. This new type of convolution is recently proposed in (Yang et al. 2019), which has proven to be very effective for enhancing model capacity network architecture engineering research. We extend to construct the desired teacher which can naturally align with the student structure, and thus is compatible to our overall design goal. Denote convolutional kernels in the student network as K's = [W')..
1, L, where L is the number of convolutional layers. The generated kernels in the teacher network can be written as Kyy = {En a W,), I E 1, 1,.... L, where for the 1th layer and the 5th kernel W it has the same shape with W n the kernel number, and we set it to 8/16 as suggested in (Yang al. 2019), which will be also discussed in Table 13 and of is learnable weight conditioned
4

---

Under review as a conference paper at ICLR 2021
on the input feature maps. Note that the set of a values automatically learnt with softmax function or a sigmoid function conditioned on the transformed features of the input channels. We experimentally used softmax function, thus each af is within [0, 1J and their sum is equal to 1. Thanks to additive property, by replacing normal convolutions with dynamic additive convolutions, both input and output feature dimensions of every convolutional layer between the teacher and student networks are the same completely. You are referred to the block tagged as "teacher" in Figure for clear understanding of this generation process.
3.4 DENSE FEATURE CONNECTION DISTILLATION
Once the teacher and student networks are perfectly aligned at every convolutional layer, then effective knowledge distillation process of our method is realized by dense feature connections from the student to the teacher which merges two networks temporarily and enables co-training from scratch. As illustrated in Figure I we add skip connections to bridge both models, formulated as / = / + &s: I E 1, ..... L, where 0's and Φ/7 are the feature maps after the 1 convolutional layer of the student and teacher models, respectively, and Φ is the new feature maps of the teacher after addition. We call this module dense feature connection distillation which is simple and neat, yet can well enhances the learning performance of the student model due the backward gradient flows from the teacher. After training, all skip connections can be naturally removed as their connection direction is only from the student to its auxiliary teacher which also exists only in training phase.
3.5 UNDERSTANDING ECD
From the optimization objective of Equation common understanding of our method is the joint optimization will improve the training of the teacher as the student is merged into the teacher by the dense layer-to-layer feature aggregation progressively. It is true the teacher model is consistently improved as can be seen from the results in Table 16 but comparatively the improvement to the student model is usually larger. We can explain this reverse thinking: during the joint training. note that the student and the teacher are merged into one single network by dense layer-to-layer feature connections from the student to the teacher, that means during the inference the teacher will depend on the student but the student does not depend on the teacher (stripping away the teacher from the student). In such a perspective, by Equation 2 the teacher can be naturally treated as the auxiliary supervision to the student, which is well in line with the Deep Supervision (DS) methodology (Lee) et al. (015)) in terms of both the mathematical formulation and the inference execution. In the DS methodology, it directly uses individual auxiliary supervisions added to several intermediate layers of a CNN model to ease gradients propagation, and they are discarded during inference. However, they usually bring marginal improvement modern CNNs reported (Huang et al. 2018 Zhang et al.) 2018b [un & Yao 2019). In a sharp contrast to existing DS methods, in our ECD, the teacher acting as the auxiliary supervision is not only based on its own structure generated with a perfect feature alignment (both in input and output dimensions) to the student at every convolutional layers, but also based on the dense layer-to-layer feature aggregation from the student the teacher progressively. enabling dense backward mnyer-to-lyee gradients propagation from the teacher to the student and boosting the training the student. Therefore, our method extends the deep supervision methodology in new perspective on developing KD research.
4 EXPERIMENTS
In this section, we evaluate our method on CIFAR-100 and ImageNet classification datasets, and compare the performance against existing knowledge distillation methods. For fair comparisons, we use the public codes of different KD methods, and adopt the same training and data preprocessing settings throughout the experiments. All experiments are implemented with Py PoTch.. To our ECD, we add Convl x on each connection path as we find it slightly improves the distillation performance (see ablative study part). Based on ECD, also apply another ensemble distillation the outputted logits from the two heads of the merged model to further boost the performance of each head, which is named as ECD* Full implementation details are referred to Appendix.
5

---

Under review as a conference paper at ICLR 2021
Table Main experimental results on the CIFAR-100 dataset. means using learning rate warm-up to smooth the training process. We report top-1 1 "mean (std)" accuracies (%) over runs.
Model
Student
Teacher
ECD
Gain
ECD*
Gain
RexNet20
68.78 (0.22)
71.05 (0.35)
70.75 (0.29)
1.97
71.07 (0.13)
2.29
ResNet32
70,80 (0.15)
72.78 (0.44)
72.40 (0.14)
1.60
73.09 (0.14)
2.29
ResNet44
71.88 (0.13)
73.48 (0.48)
73.53 (0.16)
1.65
74.08 (0.51)
2.20
ResNet56
72.29 (0.17)
73.79
73.94 (0.08)
1.65
74.88 (0.05)
2.59
ResNetl
73.15 (0.45)
75.32 (0.64)
75.11 (0.14)
1.96
75.47 (0.56)
2.32
ResNet164
76.64 (0.61)
78.78 (0.73)
77.94 (0.39)
1.30
78.34 (0.79)
1.70
ResNet1101
74.41 (0.09)
75.68 (0.21)
75.59 (0.10)
1.18
76.46 (0.20)
2.05
ResNet1641
77.15 (0.10)
78.47 (0.11)
78.33 (0.12)
1.18
78.64 (0.19)
1.49
WRN-40-1
71.44 (0.14)
72.48 (0.23)
72.55 (0.17)
1.11
72.83 (0.05)
1.39
WRN-40-2
75.96 (0.12)
77.23 (0.19)
76.55 (0.15)
0.59
76.64 (0.07)
0.68
4.1 EXPERIMENTS ON CIFAR-100
Dataset. CIFAR-100 Krizhevsky & Hinton 2009), containing 50,000 training images and 10,000 test images with 100 classes, is the most popular classification dataset for evaluating the performance of knowledge distillation methods.
Implementation. We performed experiments on prevalent ResNets (He et al. 2016) of different depths and WRNs (Zagoruyko & Komodakis, 2016) of different widths with the typical training settings (see Appendix for details). For auxiliary teacher models, we replace the standard convolutions in BasicBlock or Bottleneck of the student with dynamic additive convolutions (with following (Yang et al., 2019)). For the experiments of each setting, we run each method times and report top-1 "mean (std)" accuracies.
16
Main results. In Table we provide average results for baselines, teachers and our method ECD. Note that results of baselines are slightly higher than those reported in the original papers. For our ECD experiments, we uniformly apply feature connections the first two stages of ResNets or WRNs (we also provide ablative experiments to analyze where to add connection paths). Regarding to ResNet backbones, we observe 1.2% ~ 2.0% absolute accuracy gains for ECD, and 1.5% ~ 2.6% for ECD*, This shows that our design effective for BasicBlock and Bottleneck structures with different depths. Besides, on WRN backbones, ECD and ECD* outperform baselines with 0.6% ~ 1.1% and 0.7% ~ 1.4% margins, respectively. In summary, the proposed ECD noticeably improves the performance of each student network, and ECD* further provides additional improvements. In most cases, the students trained by our method show even better performance than the corresponding teachers. These results verify the effectiveness of our proposed method. The performance is not that strong for WRN-40-2, as this architecture has already widened the network channels.
Table 2: Results comparison with state-of-the-art two-stage and one-stage KD methods. See Appendix for specific settings of these methods. We report top-1 "mean (std)" accuracies (%) over runs.
Two-stage
One-stage
68.78 (0.22)
70.80 (0.15)
71.44 (0)
68.78 (0.22)
70.80 (0.15)
71.44 (0. 14)
Student
Student
(ResNet20)
(ResNet32)
(WRN-40-1)
(ResNet20)
(ResNet32)
(WRN-40-I)
73.15
73.15
73.23
74.43 (0.21)
75.27 (0.11)
73.99 (0.12)
Teacher
Teacher
(ResNet110)
(ResNet110)
(WRN-40-22
(ResNet110)
(ResNet110)
(WRN-40-2)
KD
70.42 (0.16)
72.08 (0.18)
73.23 (0.23)
DML
70.47 (0.25)
72.57 (0.23)
72.44 (0.21)
FitNet
70.36 (6.
72.09 (0.36)
71.98 (0.19)
AT
70.22 (0.03)
71.64 (0.32)
(0.100
72.55 (0.18)
74.82 (0.08)
N/A
FSP
69.95 (0.11)
(0.06)
71.61 (0.03)
Teacher
(Ensemble)
(Ensemble)
(Ensemble)
SP
70.35 (0.07)
72.11 (0.32)
72.47 (0.30)
VID
(0.11)
71.64 (0.30)
71.68 (0.33)
ONE
70.55 (0.07)
72.29 (0.12)
N/A
PKT
70.06 (0.08)
71.66 (0.65)
(0.18)
FT
71.17
(0.10)
71.65 (0.099
71.05 (0.35)
72.78 (0.44)
72.48 (0.23)
Teacher
NST
70.1660.06)
11.7660066
71.31 (0.40)
(Generated)
(Generated)
(Generated)
RKD
70.24 (0.09)
(0.19)
(0.21)
CC
70.37 (0.19)
71.67 (0.18)
71.44
ECD
70.75 (0.18)
72.40 (0.14)
72.55 (0.17)
CRD
70.82 (0.12)
72.84 (0.65)
72.73 (0.18(
ECD*
71.07 (0. 12)
73.09 (0.14)
72.83 (0.05)

---

Under review as a conference paper at ICLR 2021
Table Results on the ImagNet dataset, We report top-1 accuracies (%).
Model
Student
Teacher
ECD
Gain
ECD*
Gain
ResNet18
69.53
72.86
70.33
0.80
70.78
1,25
ResNet50 (0.5x)
71.41
75.27
72.19
0.78
72.48
1.07
Comparison with SOTAs. In Table 2] we compare our method with Kate-offtte--an art knowledge distillation methods under the same training settings with ours, For ResNet20 and WRN-40-1, the automatically generated teachers our method reach 71.05% and 72.48% accuracies (as previously shown in Table D. which are much lower than the teachers used for other methods (73.15% and 76.23%), yet our method is able to achieve very competitive performance. For ResNet32, under the comparable capacity of the teach model (ours 72.78% others 73.15%), our ECD* outperforms all the other methods with obvious margins.
4.2 EXPERIMENTS ON IMAGENET
Dataset. We additionally performed experiments on the ImageNet dataset (ILSVRCI2) (Russakovsky et al. 2015), which is known as the most challenging image classification dataset containing about 1.2 million training images and 50 thousand validation images, and each image belongs to one of 1000 categories.
Experiments are also conducted on prevalent ResNets. For auxiliary teacher models, we replace the standard convolutions in BasicBlock or Bottleneck of the student with dynamic additive convolutions (with n = following (anng et al., 2019)). We chose ResNet18 and Gesnet50-hall models, because they are often evaluated in knowledge distillation and the auxiliary teacher model and the student model have a suitable performance gap. Implementation details are available in Appendix A.1 A.1
Implementation.
Results. Table 3 reports the performance of our method on ImageNet. ECD and ECD* improve baseline models by 0.8% and over 1.0% absolute gains in top-1 accuracy respectively, which support the benefit of our method on the large-scale dataset.
4.3 ABLATION STUDY
In this section, we isolate the influence of each element of our method, as well as compare with possible variants. All experiments conducted CIFAR- 100 dataset. We use ECD but not ECD*, and do not use learning rate warm-up throughout all experiments for a better ablation study. For the experiments of each setting, run our method times and report top-1 "mean (std)" accuracies.
Location of connections. We explore the influence of the locations to add dense feature connections. This is very important because the semantics and robustness of different locations different. We consider different settings by adding our dense feature connections to at most three blocks (including the block Conv2_r, Conv3_r, and Conv4_az denoted as C2, C3 and C4 respectively) on CIFARI00. Detailed results are shown in Table 4 We observe that adding connections in both Conv2_a x and Conv3 brings the best performance improvement, followed by adding connections in Conv2 alone. These results indicate that adding connections in shallow layers such as Conv2_ar __ and Conv3_a can migrate information well, while Conv4_a extracts higher-level semantics and thus may be less
Table 4: Comparison of ECD with dense feature connections added to different blocks. In the table, C2, C3 and C4 refer to Conv2_r, Conv3_a and Conv4_a, respectively. We report top-] "mean (std)" accuracies (%) over runs.
Connection
ResNet20
1e
WRN-40-1
Baseline
68.78 (0.22)
73.15 (0.33)
71.44 (0.14)
C2
70.36 (0.03)
75.00 (0.35)
72.00 (0.39)
C3
70.12 (0.12)
74.41 (0.12)
(0.18)
C4
69.75 (0.12)
73.85 (0.60)
71.32 (0.37)
C2+C3
70.75 (0.29)
75.11 (0.14)
72.55 (0.17)
C2+C3+C4
69.90 (0.10)
73.75 (0.21)
70.93 (0.18)
7

---

Under review as a conference paper at ICLR 2021
Table 5: Results comparison of our ECD and using feature losses. "stage-wise" means feature supervision after each stage, which is consistent with many typical KD methods, and the distillation loss factor here is set to 10-2; "block-wise refers to supervision on the same blocks where we add connections in our ECD, and the distillation loss factor here is set to 10- We report top-1 "mean (std)" accuracies (%) over runs.
Model
ResNer20
ResNetl 10
WRN-40-1
Baseline
68.78 (0.22)
73.15 (0.45)
71.44 (0.14)
12-loss (stage wise)
70.42 (0.16)
74.64 (0.34)
72.10 (0.22)
12-loss (block wise)
70.12 (0.10)
74.32 (0.26)
71.64 (0.15)
ECD
70.75 (0.18)
75.11 (0.12)
72.55 (0.17)
Table 6: Comparison with adding Conv1 x each connection path. We report top-1 "mean (std)"
accuracies (%) over runs.
Transformation
ResNet20
ResNet1
WRN-40-1
Baseline
68.78 (0.22)
73.15 (0.45)
71.44 (0.14)
None
70.42 (0.17)
75.07 (0.21)
72.13 (0.06)
Conv x1
70.75 (0.18)
75.11 (0.12)
72.55 (0.17)
robust, which limits the effect of connection supervision. To some extent, this is consistent with multi-task learning and multi-branch network designs (Lan et al. 2018) Anil et al. (018)) where the shallow network shared and the higher level network is divided into separate branches.
Comparison with using feature loss. Recall that the distance of intermediate features between the student and teacher models is popularly used in existing KD methods to formulate feature loss (Romero et al.] 2015] Zagoruyko & Komodakis, 2017). Instead of using feature loss, we simply use dense feature connections. In Table 5 we compare the performance of feature loss and dense feature connections under the same generated teacher. The results show that in our proposed framework, using dense connections is superior to using feature loss.
Impact of the connection transformation. In Table O we compare the performance when using direct connection and adding a Convl x layer to each connection for improved alignment in feature semantics. We observe that adding Convl x achieves slightly higher results than direct connection. That means direct connections without any transformation like Conv 1 x I still get promising knowledge distillation performance. This is mainly due to our first module for auxiliary teacher generation. By replacing normal convolutions with dynamic additive convolutions, the teacher is well aligned to the student in network depth, and the input and output feature dimensions of every convolutional layer between the teacher and student networks are the same completely.
Auxiliary teacher structure design. As described in 3.3 the teacher is generated automatically through replacing the convolutional kernels in the student by linear combination of several kernels. As the knowledge is transferred from the generated teacher the student, the capability of the teacher could affect the improvements. In Table 7 we compare our default setting (dynamic additive convolution, abbreviated as DAConv) with two other attention based alternatives SE (Hu al. 2018) and CBAM (Woo et al. 2018). We find that under our proposed framework, generating teacher using attention modules such SE and CBAM can also bring noticeable improvements over the student networks. Comparatively, our default setting achieves higher performance than these two alternatives.
Embedding visualization. To verify whether the student learns useful features during the ECD training, we provide visualization of T-SNE object embeddings in Figure 2 The figure illustrates that comparing with the baseline student model, applying ECD training helps learn more scattered embeddings, which provably affirms the ad advantage of our ECD.
Combination experiment with DML and ED. In this part, we compare our ECD with two typical response-based distillation methods including mutual distillation (DML) ZZang et al. 2018a) and Ensemble Distillation (ED) (Lan et al. 2018). As shown in Table 8 our ECD with response- based distillation methods like DML and ED achieve additional performance improvements. At the same time, we observe that the performance of the teacher model during ECD training is greatly improved it is merged with student features. Under such stronger supervision of the teacher model,
8

---

Under review as a conference paper at ICLR 2021
Table Comparison of auxiliary teacher generation with different methods. DAConv refers to our default setting. We report top-1 "mean (std)" accuracies (%) over 3 runs.
Hyper-Cony
Method
ResNet20
ResNet1
WRN-40-1
Baseline
68.78 (0.22)
73.15 (0.45)
71.44 14)
Teacher
70.23 (0.33)
74.98 (0.31)
72.06 (0.23)
ECD
70.05 (0.17)
74.96 (0.79)
72.05 (0.17)
Teacher
70.45 (0.13)
75.50 (0.1
72.26 (0.16)
CBAM
ECD
70.08 (0.13)
74.94 (0.59)
72.26 (0.07)
Teacher
71.05 (0.35)
75.32 (0.64)
72.48 (0.23)
DAConv
ECD
70.75 (0.29)
75.11 (0.14)
72.55 (0.17)
Table 8: Combination experiment with DML and ED, where S represents the student model, represents the teacher model and T+ represents the teacher model connected with the features of the student model under ECD training. For the training time ResNet20 in the experiment, DML cost 5.40 gpu-hours and ECD cost 5.27 gpu-hours teacher-student model, our ECD will be slightly faster than DML because ECD does not need to calculate additional loss.
single NVIDIA TITAN RTX. Under the same
Vanilla
DML
Model
$
T
Ensemble
ResNer20
68.78 (0.22)
71.05 (0.35)
70.21 (0.08)
72.96 (0.13)
70.82 (0.19)
73.14 (0.14)
73.44 (0.18)
ResNer32
70.80 (0.15)
72.78 (0.44)
72.11 (0.07)
74.91 (0.11)
72.31 (0.13)
75.16 (0.12)
75.53 (0.12)
BCD
ECD+DML
ECD+ED
Model
S
T+
S
T+
S
T+
Ensemble
ResNet20
70.75 (0.29)
72,40 (0.24)
70.87 (0.14)
73.01 (0.15)
71.07 (0.13)
73.41 (0. 12)
73.50 (0.16)
ResNet32
72.40 (0.14)
74.83 (0.13) 72.78 (0.16)
75.08 (0.11)
73.09 (0.14)
75.28 (0.16)
75.71 (0.07)
the performance of student model is improved greatly, which is one of the reasons why our ECD can achieve better performance. In previous experiments, we mainly report the performance of the combination of ECD and ED, namely ECD*
Student
Stadent (ECD)
Teacher
Figure 2: Visualization of object embeddings for the student, the student with ECD connections, and the teacher, by using T-SNE. Results are obtained based on CIFAR-10 with ResNet44.
In Appendix, we provide additional experiments, e.g. combining our method with labeling smoothing. More comparisons and discussions also be found in Appendix.
5 CONCLUSION
In this paper, we present Explicit Connection Distillation (ECD), new knowledge distillation framework, which addresses the knowledge distillation problem in a novel perspective of bridging dense intermediate feature connections between student network and its corresponding teacher which is generated automatically in the training. ECD achieves knowledge transfer goal via direct cross-network sayer-to-layer gradients propagation, without need to define distillation losses and assume a pre-trained teacher model to be available. We hope our work can inspire the future research on knowledge distillation designs.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, 2019.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. In ICLR, 2018.
Lei Jimmy Ba and Rich Caruana. Do deep nets really need be deep? In NIPS, 2014.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint 180.::8805.0244,,,
2015.
Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In KDD, 2006.
A. Chattopadhyay, Anirban Sarkar, Prantik Howlader, and V. Balasubramanian. Grad-cam++: Improved visual explanations for deep convolutional networks. 2018.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In ICCV,
Inseop Chung, SkongUk Park, Jangho Kim, and Nojun Kwak. Feature-map-level online adversarial knowledge distillation. In ICML, 2020.
Matthieu Courbariaux, Yoshua Bengio, and Cean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. NIPS, 2015.
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In ICML, 2018.
Nuno C Garcia, Pictro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In ECCV, 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In AAAI, 2020.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020.
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In NIPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /503..550000000
2015.
Saihui Hou, Xinyu Pan, Chen Chang Loy, Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In ECCV, 2018.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018.
Gao Huang, Zhuang Liu, Laurens der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: 7707 207.0/219, 2017.
10

---

Under review as a conference paper at ICLR 2021
Jangho Kim, SeoungUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In NeurIPS, 2018. 2018.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
Alex Krizhevsky, Πya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In NeurlPS, 2018.
Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In AISTATS, 2015.
Seung Hyun Lee, Ha Dae Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In NeurIPS, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In ECCV, 2016.
Yu Liu, Xuhui Jía, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In CVPR, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint 17Xiv:1710.07535, 2015.
Andrey Malinin, Bruno Mlodozeniee, and Mark Gales, Ensemble distribution distillation. In ICLR. 2020.
Wonpyo Park, Yan Lu, Minsu Cho, and Dongju Kim. Relational knowledge distillation. In CVPR, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
Baoyun Peng. Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dong- Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019.
Mary Phuong and Christoph H Lampert. Distillation-bast based training for multi-exit architectures. In ICCV, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Dawei Sun and Anbang Yao. Deeply-supervised knowledge synergy. In CVPR, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ICLR, 2020.
Frederick Tung and Greg Mori. Similarity knowledge distillation. In ICCV, 2019.
11

---

Under review as a conference paper at ICLR 2021
Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018.
Joon-Young
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. arXiv preprint arXiv:2006. 0e:.:0000004447,, 2020.
Brandon Yang. Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In NeurlPS, 2019.
Anbang Yao and Dawei Sun. Knowledge transfer via dense cross-layer mutual-distillation. In ECCV, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. CVPR, 2017.
Kaiyu Yue, Jiangfan Deng, and Feng Zhou. Matching guided distillation. In ECCV, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018a.
Zhenli Zhang, Xiangyu Zhang, Chao Peng. Dazhi Cheng, and Jian Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In ECCV, 2018b.
Shuchang Zhou, Wu Yuxin, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprini 1rXiv:1606.06100 2016.
A APPENDIX
A.1 1 MAIN EXPERIMENTAL SETTINGS
In this section, we provide detailed settings of the experiments conducted on the CIFAR-100 and ImageNet datasets.
A.1.1 DETAILED EXPERIMENTAL SETTINGS ON CIFAR-100
The experiment was carried out on the CIFAR- 100 dataset without additional data augmentation. The network is trained for 200 epochs, the batch size is 128, the weight decay is 5 x 10 10-4, and the optimizer is SGD. For ResNets,
set the initial learning rate to 0.1, which decays by 0.1 epochs 100 and 150. For WRNs, we do not adopt dropout, and the initial learning rate is set to 0.1, which decays by 0.1 epochs 60, 120, and 160. The auxiliary teacher is expanded from 3x3 convolutional kernels (in the block of the original network) to dynamic additive convolutional kernels, and the number of multiplexing weight is 16. The Conv2 x and Conv3 x blocks of models have dense feature connections as shown in Table 9 Features after 3x3 convolution and after the block will be connected the same position on the teacher network as shown Figure 3 The features of the connected parts can be aligned better with the channel semantically with a little gain brought by x 1 convolutions. For ECD*, a learnable ensemble is applied on both outputted logits of the student and the teacher.
A.1.2 DETAILED EXPERIMENTAL SETTINGS ON IMAGENET
In the ImageNet experiments, the student model is trained with 100 training epochs, which is common setting The batch size is set to 256 and the multi-step learning rate initialized to 0.1, which decays by 0.1 at 30, 60, and 90 epochs. The auxiliary teacher has the same ResNet structure except that
thub.
12

---

Under review as a conference paper at ICLR 2021
the ordinary convolution is replaced with the dynamic additive convolution with the multiplexing weight 8. the training process, the features of Conv2_s 2, Conv3 x and Conv4_ x blocks of the student network are connected to the alignment positions of the auxiliary teacher enhanced by 1 convolution layers and BN layers. The output logits of the two output to supervise the training of the two branch heads. In the experiment, chose ResNet18 and Resnet50-half as shown in Table to verify the effectiveness of our method.
Su3 DWCoor
Core
1xt Cora
BN
BN
3x3 DACom
3K3 Conv
ReUU
ReLUU
3xx3 Co
3x1 Cona
ReL.U
Ret
1xt DACom
1xl Cora
RN
BN
IiN
ReLU
Rel U
ReLU
Rell
BaskBBlokk
Bottlenek
Figure 3: The detailed structure diagram of dense feature connections in BasicBlock and Bottleneck of ResNets, where DAConv refers to the dynamic additive convolution.
Table 9;
Details of the convolutional blocks of the ResNetl 10, ResNet1 64 and WRN-40-2 backbones evaluated on the CIFAR- dataset.
layer name
output size
ResNet110
ResNet164
WRN-40-2
conv
32x32
3x3,
3x3, 16
3x3,
3 x 167
[ 1 167
3 3, 32
conv2_x
32x32
18
3 x 16
18
3 x 3, 16
3 x 3, 32
1, 64
3 32
1 x 1, 32
3 64
conv3
16x 16
X 18
3 3, 32
x 18
6
3 x 32
3 3 3, 64
1 1, 128
x 1,
3 x 3, 64
3 128 128
conv4
x
x 18
3 x 3, 64
x 18
x
3 x 64
x 1, 256
classifier
1
average pool, 100-d fc, softmax
A.2 COMPARISON EXPERIMENTS ON CIFAR- 100
In order to make a fair comparison with existing knowledge distillation methods, we refer to the original settings and CRD code zoo to implement various knowledge methods. The setting details, including settings of the factor (A) in Equation (1). are shown in Table 11 We adopt the same standard training settings as our CIAFR- experiments in Table T
A.3
MORE DISCUSSIONS ABOUT ECD
ECD label smoothing. Label smoothing is an effective regularization method. Noise added through soften one-hot labels, which reduces the weight of the real sample label category when calculating the loss function, and suppresses over-fitting. Table [2 indicates that ECD can be combined with label smoothing to provide additional improvements as well.
13

---

Under review as a conference paper at ICLR 2021
Table 10: Structures of convolutional blocks of the ResNer backbones for the ImageNet dataset.
layer name
output size
ResNet18
ResNet50 (0.5x)
7x7, 64, stride
7x7, 32, stride
conv1
112x 112
3x3 max pool, stride
T 3 64
x 1, 32
conv2_x
56x56
x
3 x 3, 32
X 3
3 3x33, 64
x 1, 128
1 x 1, 64
3 128
conv3_x X
28x28
x 2
x 3, 64
x x
3 3, 128
[1 x 1, 256
x 1, 1287
3 3, 256
conv4_x
16x 16
X
3 x 3, 128
x
3 256
x 1, 512]
x 1,
3 3. 512
conv5_x x
8x8
3 512
x
3 3, 256
x 3
x 1, 2048
classifier
average pool, 1000-d softmax
Table 11: Brief implementation details of KD methods and settings of the distillation loss factor (A).
Method
factor (A)
Brief notes of Implementation details.
KD (Hinton al.
2015)
0.1
The temperature factor T is 4. One stage without hint layer.
FitNet [Romero et al.
2015)
0.1
The sum of absolute values with power p=2 is used as the attention.
AT
Zagoruyko & Komodakis
2017
1000
FSP
Yim al.
2017)
SP
Lung & Mon
2019
3000
Hidden channel number same as output channel, and remove BN in μ.
VID
Ahn et al.
2019
PKT
Passalis & Tefas
2018
1000
FT
Kim et al.
2018)
200
NST (Huang Wang
2017)
10
The polynomial kernel 2 and c The distance is 50 and the angle is 25.
RKD
Park al.
2019
(25,50)
CC
(Peng et |
2019)
100
CRD
TTan
2020
0.8
DML Zhhag el al.
20
8a)
ONE
Lan et 2018
Ensemble distillation of three branches.
Table Experiment results of applying Label Smoothing (I.S) to ECD.
Method
ResNet20
ResNetl 10
WRN-40-1
Bascline
68.78 (0.22)
73.15 (0.45)
71.44 (0.14)
Label smoothing
69.89 (0.16)
74.59 (0.31)
71.56 (0.08)
ECD
70.75 (0.29)
75.11
72.55 (0.17)
ECD LS
71.03 (0.08)
75.13 (0.43)
72.66 (0.07)
14

---

Under review as a conference paper at ICLR 2021
Multiplexing weight settings. In Table 13 the multiplexing weight for DAConv. We observe that the number 16 seems to be the best choice, but the distillation performance is not sensitive to the multiplexing weight (e.g. 72.12% v.s. 72.40% by doubling the number from 8 to 16). Besides, by simply setting the multiplexing weight to or 4, we still able to obtain obvious improvements.
explore different settings of our ECD by changing
Table 13:
Experiment results of ResNet32 CIFAR100 in different number of multiplexing weight for DAConv. We report top-1 "mean (std)" accuracies (%) over runs
Number of multiplexing weight
Model
Baseline
4
8
16
32
ResNet32 70.80 (0.15) 71.47 (0.09) 71.87 (0.16)
72.12 (0.07)
72.40 (0.14)
71.86 (0.20)
More comparison experiments with other KD methods. As shown in Table 14 the performance of our ECD and ECD* is quite good compared to other methods especially when combining with the original KD. Furthermore, we experiment with more dense cress-network feature connections in ECD, called ECD (dens), and the corresponding method combined with ED, called ECD (dens)*. Note that "multi-layer" indicates the student layer (at the same depth with the teacher layer) as well its all previous layers. As shown in Table [4] ECD (dens) and ECD (dens)* achieve the best performance among these KD methods, indicating that the feature connection our ECD can effectively transmit feature distillation information. Also, we evaluate the performance of our ECD when combining with FitNet (Romero et al. (2015) or CRD (Tian et al. (2020)) by adding FitNet loss CRD loss during ECD training. We observe such combinations bring additional improvements, which means method is orthogonal to other distillation methods such as FitNet and CRD. In Table 15 we refer to the results reported in CRD for two-stage KD methods, and implement one-stage KD methods under the same training setting with CRD. As shown in Table [5] our ECD and ECD* achieve good performance, ECD (dens) and ECD (dens)* obtain the best performance for ResNet20 and ResNet32. Besides, observe that the combination of ECD and other methods can achieve additional performance improvements.
taalii-lyyr--t--inngleee
Table 14: Results comparison with sate-of-the-an two-stage KD methods and results of combining two-stage KD methods with the original KD. Bold and underline denote the best and the second best results, respectively. We report top-1 "mean (std)" accuracies (%) over 3 runs.
Teacher Student
ResNetl 10 (73.15) ResNet20 (68.78)
ResNet1 (73.15) ResNe132 (70.80)
WRN-40-2 (76.23) WRN-40-1 (71.53)
Method
w/o KD
w/ KD
w/o KD
w/ KD
w/o KD
w/ KD
FitNet
70.36 (0.16)
70.46 (0.11)
72.09 (0.36)
72.70 (0.17)
71.98 (0.19)
73.14 (0.19)
70.22 (0.03)
70.90 (0.21)
71.64 (0.32)
72.51 (0.19)
71.74 (0.10)
72.79 (0.38)
SP
69.95 (0.11)
70.87 (0.15)
71.69 (0.06)
72.92 (0.24)
71.61 (0.03)
72.77 (0.15)
CC
70.35 (0.07)
71.04 (0.31)
72.11 (0.32)
72.92 (0.19)
72.47 (0.30)
72.73 (0.37)
VID
70.14 (0.11)
70.71 (0.39)
71.64 (0.30)
72.81 (0.24)
11.6800.333
72.68 (0.27)
RKD
70.06 (0.08)
70.71 (0.09)
71.66 (0.05)
73.51 (0.31)
(0.18)
72.51 (0.35)
PKT
71.14
71.01 (0.05)
72.82 (0.10)
72.88 (0.10)
71.65
72.93 (0.19)
FT
70.12 (0.06)
71.49 (0.05)
71.76 (0.36)
72.92 (0.24)
(0.41
72.69 (0.03)
FSP
70.24 (0.09)
70.60 (0.07)
71.93 (0.19)
72.88 (0.15)
71.63 (0.21)
72.71 (0.13)
NST
70.37 (0.19)
70.94 (0.06)
71.67 (0.18)
72.96 (0.24)
71.44 (0.09)
72.79 (0.23)
CRD
70.82 (0.12)
71.26 (0.09)
72.84 (0.65)
73.18 (0.13)
72.73 (0.18)
73.54 (0.22)
ECD
70.75 (0.18)
72.40 (0.14)
72.55 (0.17)
ECD*
71.07 (0.12)
73.09 (0.14)
72.83 (0.05)
ECD+FitNet
70,91 (0.16)
72.79 (0.21)
72,96 (0.12)
ECD+CRD
71.33 (0.09)
73.39 (0.11)
73.36 (0.16)
ECD (dens)
71.25 (0.13)
73.11 (0.17)
73.11 (0.21)
ECD (dens)*
72.44 (0.21)
74.19 (0.12)
73.76 (0.14)
15

---

Under review as a conference paper at ICLR 2021
Table 15:
Results comparison with state-of-the-art two-stage KD methods reported in CRD (Tian et al. (2020)) under the training setting of the 240 epochs. Bold and underline denote the best and the second best results, respectively.
Two-stage
One-stage
69.06
71.14
71.98
69.06
71.14
71,98
Student
Student
(ResNet20)
(ResNe132)
(WRN-40-1)
(ResNe120)
(ResNet32)
(WRN-40-I)
74.31
74.31
75,61
74.68 (0.11)
75.53 (0.13)
75.74 (0.14)
Teacher
Teacher
(ResNetl (ResNet110)
10)
(WRN-40-2)
(ResNet110)
(ResNetl 10)
(WRN-40-2)
KD
70.67 (0.27)
73.08 (0.18)
73.54 (0.20)
DML
70.77 (0.20)
72.87 (0.17)
72.63 (0.15)
FitNet
68.99 (0.27)
71.06 (0.13)
72.24 (0.24)
72.85 (0.16)
75.21 (0.17)
N/A
AT
70.22 (0.16)
72.31 (0.08)
72.77 (0.10)
Teacher
(Ensemble)
(Ensemble)
(Ensemble)
FSP
70.11 (0.16)
71.89 (0.11)
N/A
ONE
71.41 (0.22)
73.25 (0.13)
N/A
SP
70.04 (0.21)
72.69 (0.41)
72.43 (0.27)
VID
70.16 (0.39)
72.61 (0.28)
73.30 (0.31)
Teacher
71.35 (0.28)
73.11 (0.36)
72.78 (0.12)
(Generated)
(Generated)
(Generated)
PKT
70.25 (0.04)
72.61 (0.17)
73.45 (0.19)
(0.10)
72.64 (0.15)
72.75 (0.07)
FT
(0.10)
72.37 (0.31)
71.59 (0.15)
ECD*
71.56 (0.14)
73.48 (0.18)
73.78 (0.17)
NST (0.15)
71.96 (0.07)
72.24 (0.22)
ECD+FaNet
71.08 (0.16)
72.91 (0.21)
73.07 (0.12)
RKD
69.25 (0.05)
71.82 (0.34)
72.22 (0.20)
ECD+CRD
71.46 (0.09)
73.52 (0.11)
73,49 (0.16)
CC
69.48 (0.19)
71.48 (0.21)
72.21 (0.25)
ECD (dens)
(0.21)
73.43 (0.18)
73.24 (0.11)
CRD
71.46 (0.09)
73.48 (0.13)
74.14 (0.22)
ECD (dens)*
72.23 (0.19)
74.35 (0.16)
73.99 (0.15)
More discussion for ECD training. As in Table 16 we record the best performance of each model during ECD and ECD* training. We can find that the performance of the teacher model is significantly improved by connected with the features of the student model under ECD training. So the student model gets more improvements. And the performance each model is further improved in ECD* training.
Table 16: Detailed results on the CIFAR-100 dataset. indicates using learning rate warm-up to smooth the training process. S represents the student model, T represents the teacher model and T+ represents the teacher model connected with the features of the student model under ECD training. We report top-1
"mean (std)" accuracies (%) over 3 runs.
Vanilla
ECD
ECD*
Model
S
T
$
T+
S
T+
Ensemble
ResNet20
68.78 (0.22)
71.05 (0.35)
70.75 (0.29)
72.40 (0.24)
71.07 (0.13)
73.41 (0.12)
73.50 (0.16)
RcsNet32
70.80 (0.15)
72.78 (0.44)
72.40 (0.14)
74.83 (0.13)
73.09 (0.14)
75.28 (0.16)
75.71 (0.07)
ResNerd4
71.88 (0.13)
73.48 (0.48)
73.53 (0.16)
75.34 (0.06)
74.08 (0.51)
76.13 (0.12)
76.60 (0.17)
ResNet56
72.29 (0.17)
(0.59)
73.94 (0.04)
75.64
74.88 (0.05)
(0.15)
77.53
ResNet110
73.15 (0.45)
75.32 (0.64)
75.11 (0.14)
75.89 (0.10)
75.47 (0.56)
(0.16)
78.41 (0.64)
ResNet161
76.64 (0.61)
78.78 (0.73)
77.94 (0.39)
79.31 (0.21)
78.34 (0.79)
79.73 (0.10)
80.99 (0.17)
ResNet1101
74.41 (0.09)
75.68 (0.21)
75.59 (0.10)
(0.16)
76.46 (0.20)
(0.65)
78.89 (0.12)
ResNe11641
77.15 (0.10)
78.47 (0.11)
78.33 (0.12)
80.14 (0.20)
78.64 (0.19)
80.07 (0.20)
81.12 (0.13)
WRN-40-1
71.44 (0.14)
72.48 (0.23)
72.55 (0.17)
72.87 (0.09)
72.83 (0.05)
73.52 (0.12)
74.25 (0.15)
WRN-40-2
75.96 (0.12)
77.23 (0.19)
76.55 (0.15)
78.57 (0.19)
76.64 (0.07)
79.12 (0.14)
79.45 (0.17)
Attention map visualization. In ECD, the feature information of the teacher model can be transmitted to the student model through the feature connection. As shown in Figure the attention map of the student network more similar to that of the teacher network under ECD training.
16

---

Under review as a conference paper at ICLR 2021
Input
Student
Student (FCD)
Teacher
Figure 4: Comparison on the Brad-CAM++ Chattopadhyyy et al. 2018)) visualization results between the features of the student, the student with ECD connections, and the teacher network. Results are obtained based on ImageNet with ResNet18. Target objects in the upper two images are misclassified by the student network while correctly classified by the student with ECD connections.
17


Reviews:
[{"review_id": "nxjS8cFWSBM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests an interesting approach to knowledge distillation, which uses architectural properties rather than the loss function to encourage knowledge transfer between a teacher and a student. A student and teacher net are jointly trained on a prediction task, with forward connections from layers in the student net to layers in the teacher net. At test time, the teacher net can be stripped away. The method results in improvements in student performance compared to training the student without the teacher.\n\nThe main positives I see in this paper are:\n+ modest improvements over baselines in the one-stage KD setting\n+ provides a different perspective on KD compared to recent works that are based on loss functions\n\nAlthough the results look promising, I think the paper is not yet ready for publication. The main issues I see are:\n- no comparisons against network pruning and compression methods, which are arguably more relevant than KD baselines\n- little insight or analysis of why the method works\n- concerns with the experiments, detailed below\n\nThis paper positions itself as a KD method, and argues that a novelty is in doing KD via architectural tricks rather than via a loss function. This may indeed be a new perspective for the KD literature, but it isn’t without precedent. Methods that use architectural scaffolding at training time, which is removed at test time, do exist, simply under other names such as weight pruning or model compression (many of these papers are cited in the intro of the current submission). The current paper needs to make it clearer how the proposed method relates to those methods, and how it goes beyond them. Ideally this should include quantitative comparisons, or an explanation for why the other methods are not applicable.\n\nMy second major concern is that this paper provides very little in the way of an explanation for why the method works. There is an intriguing statement that the method “enhances the learning performance of the student model due to the backward gradient flows from the teacher,”  but nothing to back this statement up. Some analysis of how these gradients achieve desirable effects would greatly strengthen the paper.\n\nMy third concern is with the experiments. First, the numbers in Table 2 are somewhat lower than those reported in prior papers (see Table 1 of Tian et al. 2020). Appendix A2 states that the code from Tian et al. 2020 was used to run the comparisons. Why then the discrepancy in performance? Second, many of the prior KD methods perform best when their objective is combined with the original Hinton KD objective (see Table 7 of Tian et al. 2020), but this comparison is not provided in the current paper. These two concerns mean that I’m not sure the proposed method is really outperforming competitive baselines from prior work. Lastly, I did not find enough details about ECD* to be able to really evaluate if those results are strong or interesting.\n\nStylistically, I think the paper would be improved by adopting a more even-handed tone. Statements like “compared to existing KD methods, ECD is more friendly to end users thanks to its simplicity” or that the method is “simple and neat” come across more as advertisement rather than as scientific analysis. The intro argues that the one-stage nature of the proposed approach makes it more applicable than two-stage approaches, but I would say one and two-stage methods are simply targeting qualitatively different applications. Two stage approaches are useful when you are _given_ a big model, which maybe you do not have the resources or data to train, and want to compress it or adapt it, e.g. for mobile deployment. One stage approaches are useful when you are able to train the big model yourself. There are interesting tradeoffs between these two paradigms and one is not better than the other. The current paper should acknowledge this. In general, the advantages and disadvantages of the method should be discussed equally. I also think the paper overemphasizes how simple the method is. I don’t personally feel this method is any simpler than methods that use loss functions, and in fact I find the proposed method more conceptually complex since I don’t know why it works.\n\nDespite these criticisms, something interesting does seem to be going on with this method, and I encourage the authors to pursue it further.\n\nMinor comments:\n1. Abstract: “temporally” —> “temporarily”?\n2. Table 1: what is the \"Baseline\"? The student?\n3. Table 2: citations should be added for each method", "extra_scores": {}, "extra_text": {}}, {"review_id": "my3h4BMpsb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I vote for marginally below the acceptance threshold. I think the idea is somewhat novel for the Explicit Connection Distillation, especially for cross-network layer-to-layer gradients propagation. This paper proposes a new strategy of Knowledge distillation called  Explicit Connection Distillation (ECD), and the ECD achieving knowledge transfer via cross-network layer-to-layer gradients propagation without considering conventional knowledge distillation losses. Experiments on popular image classification tasks show the proposed method is effective.\nHowever, several concerns including the clarity of the paper and some additional ablation studies ( see cons below) result in the decision.\n\n##########################################################################\nPros: \n1) The knowledge distillation by cross-network layer-to-layer gradients propagation is somewhat novel to me.\n2) This paper is easy to follow, the methodology part is clear to me.\n3) The experiments part show detail ablation study of each component and the supplementary lists almost detail of experiments which help the community to reproduce the proposed methods.\n\n##########################################################################\nCons: \n1) The first concern is about motivation. (1): The author claims conventional KD methods leads to complex optimization objectives since they introduce two extra hyper-parameters. To my best of the knowledge, these two parameters have not too much search space. e.p Temperature is from 3-5 and the weight is T^2 from Hilton's paper and following paper.  (2): The drawback of one-stage KD methods is a little bit overclaim,  Both ONE and DML can be applied to a variety of architecture. In my opinion, the teacher design of  ECD  follows a similar strategy with ONE and its variants, which is the teacher is wider than the student.  Overall, I think the motivation of this paper needs to be very careful to clear.\n2) The fairness of comparison. Is the Dynamic additive convolution component used to in student network, does this will influence the comparison of  Table2, Does ONE and DML also use that?\n3) Why the automatically generated teachers of ECD is much lower than other methods in Table2 in term of performance and results in higher student performance. Is there any explanation here, like such as [1][2]?\n4) Could you provide the computation cost comparison of the proposed method and other one-stage methods in Table2.\n5) Some recently SOTA work is missed [3][4], although I know the performance of this paper is outperformed. I think they need to be discussed.\n\n\nReference:\n[1]: Improved Knowledge Distillation via Teacher Assistant\n\n[2]: Search to distill: pearls are everywhere but not the eyes\n\n[3]: Online Knowledge Distillation via Collaborative Learning\n\n[4]: Peer Collaborative Learning for Online Knowledge Distillation\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzNeo0wQJ6y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes new KD framework, i.e., Explicit Connection Distillation (ECD), which unlike existing methods, designs teacher network that is well aligned with the student architecture and trains both the networks simultaneously using explicit dense feature connections. The proposed method is evaluated on CIFAR-100 and ImageNet datasets.\n\nStrengths:\n\n- The proposed method neither requires any explicit pre-trained teacher network, nor any distillation loss. So, the method overcomes the problem of selecting teacher network or alternatives of distillation losses for the task at hand.\n- By design, the generated teacher network has features aligned with the student network at every layer.\n\nConcerns:\n\n- Though existing works involves complex optimization in terms of losses but the hyperparameters involved in distillation like the weight on distillation loss or the temperature value is not so sensitive like learning rate. Even without careful tuning, decent distillation performance can be achieved with moderate temperature, high weight on distillation loss and low weight on cross entropy loss. So, this is not a major limitation in existing methods.\n- In the proposed ECD framework, both the teacher and student networks are trained simultaneously, so number of trainable parameters (teacher parameters + student parameters) would be large. So, the method may not work well in case of limited amount of training samples.\n- Selecting an optimal value of kernel number ‘n’ is a concern.\n- The gain in performance in Table 1 for WRN-40-2 is marginal. So, it seems the proposed method may not be effective on some architectures like wide ResNets where the network channels are widened.\n- In Table 5, marginal improvement using ECD over stage wise feature supervision.\n- Table 4 shows shallow layers migrate more information than higher layers and dense connections are preferred on shallow layers only to get optimal performance. But identifying the layer from which high level semantics would be captured is non-trivial.\n\nQueries for authors:\n\n- Any restriction or range of values that alpha can take?\n- All the experiments are done with n=16, how the performance changes by varying ‘n’?\n- Is the performance of the teacher reported in Table 1, obtained through auxiliary teacher involving feature connections with the student network?\n- While training using the proposed ECD, how to decide number of epochs for training (based on either teacher or student performance on validation data)?\n- Details about ECD* and how learnable ensemble is applied is not mentioned in detail even in Appendix.\n\nGeneral Remarks:\n\nWhile the creation of auxiliary teacher directly from the student network removes its dependencies from pre-trained teacher but dependency on several design choices like the number of dynamic additive convolutions for the first module and appropriate places for adding connection paths in the second module for explicit flow of layer to layer gradients remain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "A8MbY6y0luz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new knowledge distillation method applicable to convolutional neural networks. Given the architecture of a student network, the teacher network is constructed by replacing each convolutional layer with a dynamic additive convolutional layer. In this way, the teacher network is guaranteed to be more capable than the student network. Then, the teacher and student models are trained together, minimizing their own training losses. In order to \"distill\" the student model, the student and teacher model are inter-connected in a way such that the student model receives gradient flow from the teacher network.\n\nPros:\n1. The proposed method is straightforward and easy to implement.\n2. By adding an additional distillation loss on the logits (the ECD* method), it achieved competitive distillation results.\n\nCons:\n1. The authors claim that the gradient flow (from the teacher model to the student model) helps improving the student model. But I do not see why. The gradient is from the teacher's loss. So by applying the gradient on the student model, only the teacher's loss can be improved. This is my greatest concern. Similarly, from Equation (2), I do not see why optimizing \\theta_T would help the generalization ability of the student model.\n2. The authors made several claims about existing knowledge distillation methods. For me, many of the claims are not meaningful. For example: \"..in real applications, for two-stage KD methods well pre-trained teacher models are usually not available, ...\" This is true. But what is the drawback of \"well pre-trained teacher models are usually not available\"? All distillation methods require a teacher model. The proposed method still need to train a teacher model. In fact, I might claim that two-stage KD methods can utilize existing pre-trained teacher models, whereas the proposed method always has to train a teacher model during distillation. Similar claim was made on \"designing complex distillation losses\" as a drawback. But a user does not have to design new forms of losses if he/she stick to existing methods. On the other hand, if the proposed method became popular, researchers may add auxiliary losses on top of the proposed method, and that would not seem like a drawback for the proposed method. \n\nSome confusions:\n1. When comparing to different distillation methods in Table 2, are all teacher models have the same architecture?\n2. When generating the teacher model, do the authors initialize the weights of the teacher model randomly, or according to the student model?\n\nAfter reading the author feedback, I upgrade my rating to 6. See responses in this thread for reasons.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Knowledge distillation (KD) has been widely used in practice for deployment.  In this paper, a variant of KD is proposed: given a student network, an auxiliary teacher architecture is temporarily generated via dynamic additive convolutions; dense feature connections are introduced to co-train the teacher and student models. The proposed method is novel and interesting. Empirical results showed that the proposed method can perform better than several KD variants.  However, it is unclear why the proposed method works, although the authors tried to address this issue in their rebuttal.   Besides this,  a bigger concern on this work is that it missed a comparison with a recent approach in [1] which looks much simpler and performs significantly better on similar experiments.  In [1], their ResNet50 (0.5x) is smaller than the student model in this paper (which used more filters on the top) but showed much stronger performance on both relative and absolute improvements over the same baseline (training from scratch) for the ImageNet classification task. On the technical side, the method in [1] simply uses the original ResNet50 as the teacher model,  and the student model ResNet50 (0.5x) progressively mimics the intermediate outputs of the teacher model from layer to layer. [1] also contains a  theoretic analysis  (mean-field analysis based) to support their method. Comparing with the method in [1], the proposed method here is more complicated, less motivated, and less efficient. 

[1] D. Zhou, M. Ye, C. Chen, T. Meng, M. Tan, X. Song, Q. Le, Q. Liu and D. Schuurmans. Go Wide, Then Narrow: Efficient Training of Deep Thin Networks. ICML 2020.
Decision: Reject