Explicit Connection Distillation

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved great success in tackling a variety of visual recognition tasks ( Krizhevsky et al. , 2012 ; Girshick et al. , 2014 ; Long et al. , 2015 ) . Despite the appealing performance , the prevailing DNN models usually have large numbers of parameters , leading to heavy costs of memory and computation . Conventional techniques such as pruning weights from networks ( Han et al. , 2015 ; Li et al. , 2017 ) and quantizing networks to use low-bit parameters ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ) have proven to be effective for mitigating this computational burden . More recently , Knowledge Distillation ( KD ) , another promising solution family to get compact yet accurate models , has attracted increasing attention . The goal of KD is to transfer the learnt information ( knowledge ) of a high-capacity DNN model or an ensemble of multiple DNN models ( teacher ) to a low-capacity target DNN model ( student ) , striking better accuracy-efficiency tradeoffs at runtime . There exist numerous KD methods to address the knowledge transfer from the teacher to the student . Many of them use a two-stage training process which begins with training a teacher model and keeping it fixed , and then learns a target student model by forcing it to match the outputted knowledge from the pre-trained teacher model . Various types of knowledge have been explored , such as outputted logits ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) , intermediate feature representations ( Romero et al. , 2015 ; Zagoruyko & Komodakis , 2017 ) , and relational information of model outputs or representations ( Park et al. , 2019 ; Tian et al. , 2020 ) . Instead of defining new types of knowledge , the other methods adopt one-stage training process , jointly training the student and teacher/peer models using bidirectional knowledge distillation ( Zhang et al. , 2018a ; Yao & Sun , 2020 ) or on-the-fly ensemble distillation ( Lan et al. , 2018 ; Anil et al. , 2018 ) or multi-exit distillation ( Phuong & Lampert , 2019 ) . Both two-stage and one-stage KD methods as above typically treat the knowledge transfer as an optimization problem of formulating robust distillation loss functions via introducing more informative knowledge supervisions and more effective knowledge matching strategies . Instead of extracting informative knowledge and designing alternative distillation loss functions to fit a desired knowledge transfer optimization objective as existing KD methods , we investigate a new technical perspective in this paper : casting the knowledge distillation problem into designing auxiliary connection paths between the student and teacher networks to enable explicit layer-to-layer gradients distillation from the teacher to the student via training them from scratch simultaneously . We are partially inspired by recent advances on DNN architecture engineering , which show that designing sophisticated feature connection paths such as residual connections ( He et al. , 2016 ) and dense connections ( Huang et al. , 2017 ) across neighboring layers can allow better information and gradients flow throughout a single network , making the training easy to have significantly improved performance in model accuracy and convergence . We conjecture this simple principle would also be crucial to open the door to develop a totally new knowledge distillation framework if we can merge the student and teacher into a single network temporarily during training phase , and can also separate them easily after training . To explore this hypothesis , we present Explicit Connection Distillation ( ECD ) , a very simple knowledge distillation framework . Specifically , we decompose the design of ECD into two interdependent modules , namely auxiliary teacher generation and dense feature connection distillation . Recent works ( Liu et al. , 2020 ; Yue et al. , 2020 ) show that searching a good alignment of structural feature channels between the student and teacher networks can bring improved knowledge distillation performance under the premise that a pre-trained teacher model is available . Regarding the first module of ECD , we hope the generated teacher can make a perfect structure alignment ( both in input and output feature dimensions of every convolutional layer ) to the student network in an easier manner ( no need of time-consuming searching procedure as used in ( Liu et al. , 2020 ; Yue et al. , 2020 ) ) while can attain superior model capacity . To this goal , we retain all structural units of the student network in constructing the teacher architecture , except for replacing original convolutions by dynamic additive convolutions ( Yang et al. , 2019 ) which have proven to be very effective for enhancing model capacity in network architecture engineering research . Regarding the second module of ECD , we hope knowledge distillation can be realized through the explicit layer-to-layer flows of gradients from the teacher to the student instead of the conventional mimicking procedure . To this goal , we add dense feature connections from the convolutional layers of the student to those aligned layers of the auxiliary teacher , and train the merged model from scratch . After training , all feature connections can be naturally removed as their connection direction is only from the student to its auxiliary teacher which also exists merely in training phase . Beyond the common wisdom of knowledge distillation that requires the knowledge mimicking process between the student and teacher networks , our ECD sheds new insight : by considering knowledge distillation from a novel student-to-teacher merging , co-training and splitting viewpoint , direct feature connections from the student to its well-aligned auxiliary teacher ( generated from the student in an automatic manner ) can also achieve competitive performance to improve low-capacity student models , as validated by extensive experiments on CIFAR-100 and ImageNet datasets . 2 RELATED WORK . In this section , we make a brief summary of existing knowledge distillation works . Two-stage KD methods . The idea of training a compact model to mimic the functions learnt by a larger ensemble of models is firstly proposed in ( Bucilǎ et al. , 2006 ) . Ba & Caruana ( 2014 ) extends this idea , showing that shallower yet wider neural networks can also approximate the functions previously learnt by deep ones . Hinton et al . ( 2015 ) presents the famous Knowledge Distillation ( KD ) method , which adopts a teacher-student framework for transferring learnt soft knowledge from a high-capacity teacher to a low-capacity target student network . In this framework , the teacher is pre-trained and fixed , and then its soft logits on the training data are used as the extra supervision to guide the training of the student besides the ground truth labels . FitNets ( Romero et al. , 2015 ) shows the intermediate feature representations learnt by the teacher can be used as the complementary knowledge to soft logits , enhancing knowledge distillation performance to some extent . Zagoruyko & Komodakis ( 2017 ) proposes to use spatial attention activations instead of intermediate feature maps . Recent works further show the relations ( Park et al. , 2019 ) and higher order dependencies ( Tian et al. , 2020 ) of learnt logits or intermediate feature representations by the teacher capture important structural information , which are more effective knowledge to augment the knowledge distillation process . Following the teacher-student framework as above , many other works such as ( Yim et al. , 2017 ; Lee et al. , 2018 ; Kim et al. , 2018 ) attempt to further improve knowledge representations and matching losses . One-stage KD methods . Unlike two-stage KD methods that rely on a pre-trained teacher model and one-way knowledge transfer from the teacher to the student , one-stage KD methods simplifying the knowledge distillation process by training all models simultaneously . Deep mutual learning ( DML ) ( Zhang et al. , 2018a ) considers the teacher as a peer of the target student , showing that the outputted logits of the student can be used to assist the training of the teacher by a peer-teaching strategy . In ( Yao & Sun , 2020 ) , the authors further show that deep mutual learning can also be boosted by intermediate soft outputs , just like that in two-stage KD methods . Guo et al . ( 2020 ) follows the basic framework of DML but replaces the mutual distillation ( student-to-student ) by the ensemble ( over students ) -to-students distillation . ONE ( Lan et al. , 2018 ) presents an on-the-fly ensemble distillation method in which a stronger ensemble teacher learnt over a multi-branch network is used to enhance the training of every branch . Anil et al . ( 2018 ) extends this idea to accelerate large-scale distributed neural network training applications . Wu & Gong ( 2020 ) follows the basic framework of ONE but jointly uses the mutual distillation ( student-to-student ) and the ensemble-to-students distillation . Other improved variants include but are not limited to ( Phuong & Lampert , 2019 ; Malinin et al. , 2020 ) . Other KD variants . Besides model compression , there also exist a lot of works using knowledge distillation methodology to handle other applications . Self-distillation methods ( Furlanello et al. , 2018 ; Bagherinezhad et al. , 2015 ) explore the benefits of KD techniques to improve the model training , assuming only a single network is available . Lopes et al . ( 2015 ) and Chen et al . ( 2019 ) address data-free knowledge distillation where the original training data are no longer accessible due to safety or privacy concerns . Goldblum et al . ( 2020 ) and Chung et al . ( 2020 ) extend the idea of KD to study how adversarial robustness of the teacher model can be transferred to the student . Li & Hoiem ( 2016 ) and Hou et al . ( 2018 ) combine KD with fine-tuning and retrospection to handle life-long learning scenarios . Adapting KD to other tasks such as multi-modal visual recognition and natural language processing is explored in ( Garcia et al. , 2018 ; Kim & Rush , 2016 ) . 3 EXPLICIT CONNECTION DISTILLATION . In this section , we describe the formulation of the proposed Explicit Connection Distillation ( ECD ) framework , and detail how to design and implement its two key components : auxiliary teacher generation and dense feature connection distillation . 3.1 CONVENTIONAL KD FORMULATION . For a better understanding of our method , we start with the formulation of conventional KD methods . Let S , T and θS , θT denote a target student model , its teacher ( either a single model or an ensemble of multiple models ) and their parameters correspondingly , and let x denote training data , and let Q be a set of layer location pairs where knowledge mimicking process between two networks is introduced . Regarding one-stage KD methods , where the student and teacher models are trained jointly , the overall objective function to be optimized can be defined as LKD = LCE ( θS , x ) + LCE ( θT , x ) + λ ∑ q∈Q d ( fqS ( x ) , f q T ( x ) ) , ( 1 ) where LCE ( θS , x ) and LCE ( θT , x ) are the standard cross-entropy loss functions of the student model and the teacher model , respectively . d is the loss for knowledge mimicking , which measures the distance of learnt knowledge fqS ( x ) , f q T ( x ) at all specific layer location pairs between the student and teacher models . As we discussed in introduction section , intermediate feature representations , soft logits , and their relations from learnt models are popularly used as the knowledge . λ is a tunable weighting factor to balance loss terms , which is usually initialized to a relatively large value and decays during training . Simply removing the loss term LCE ( θT , x ) from Equation ( 1 ) gets the objective function of two-stage KD methods , as in them the teacher model is pre-trained and fixed .

Summary: This paper suggests an interesting approach to knowledge distillation, which uses architectural properties rather than the loss function to encourage knowledge transfer between a teacher and a student. A student and teacher net are jointly trained on a prediction task, with forward connections from layers in the student net to layers in the teacher net. At test time, the teacher net can be stripped away. The method results in improvements in student performance compared to training the student without the teacher.