Feature-map-level Online Adversarial Knowledge Distillation

Feature maps contain rich information about image intensity and spatial correlation . However , previous online knowledge distillation methods only utilize the class probabilities . Thus in this paper , we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework . We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks . Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real . By training a network to fool the corresponding discriminator , it can learn the other network ’ s feature map distribution . Discriminators and networks are trained concurrently in a minimax twoplayer game . Also , we propose a novel cyclic learning scheme for training more than two networks together . We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one . 1 INTRODUCTION . With the advent of Alexnet ( Krizhevsky et al. , 2012 ) , deep convolution neural networks have achieved remarkable success in a variety of computer vision tasks . However , high-performance of deep neural network is often gained by increasing the depth or the width of a network . Deep and wide networks cost a large number of computation as well as memory storage which is not suitable for a resource-limited environment such as mobile or embedded systems . To overcome this issue , many researches have been conducted to develop smaller but accurate neural networks . Some of the well-known methods in this line of research are parameter quantization or binarization ( Rastegari et al. , 2016 ) , pruning ( Li et al. , 2016 ) and knowledge distillation ( KD ) ( Hinton et al. , 2015 ) . KD has been an active area of research as a solution to improve the performance of a light-weight network by transferring the knowledge of a large pre-trained network ( or an ensemble of small networks ) as a teacher network . KD sets the teacher network ’ s class probabilities as a target which a small student network tries to mimic . By aligning the student ’ s predictions to those of the teacher , the student can improve its performance . Recently , some studies have shown that rather than using a pretrained teacher , simultaneously training networks to learn from each other in a peer-teaching manner is also possible . This approach is called online distillation . Deep mutual learning ( DML ) ( Zhang et al. , 2018 ) and on-the-fly native ensemble ( ONE ) ( Lan et al. , 2018 ) are the representative online distillation methods that show appealing results in the image classification tasks . Conventional distillation method requires pre-training a powerful teacher network and performs an one-way transfer to a relatively small and untrained student network . On the other hand , in online mutual distillation , there is no specific teacher-student role . All the student networks learn simultaneously by teaching each other from the start of training . It trains with the conventional cross-entropy loss from the ground truth label along with the mimicry loss to learn from its peers . Networks trained in such an online distillation way achieve results superior not only to the networks trained with the cross-entropy loss alone but also to those trained in conventional offline distillation manner from a pre-trained teacher network . However , aforementioned online distillation methods make use of only the logit information . While the logit contains the probabilistic information over classes , the feature map , the output of convolution layer , has more meaningful and abundant feature information on image intensity and spatial correlation . In offline distillation which utilizes a pre-trained model as a teacher network , many methods such as FitNet ( Romero et al. , 2014 ) , attention transfer ( AT ) ( Zagoruyko & Komodakis , 2016a ) and factor transfer ( FT ) ( Kim et al. , 2018 ) make use of this intermediate feature representation as a target to learn for the student network , but in online distillation , to the best of our knowledge , no feature map-based knowledge distillation method has been proposed . This is due to some challenges . Unlike the offline methods that have a clear target to mimic , there is no static target to follow in an online method . At every training iteration , the feature maps of the co-trained network change , thus in online feature map-level distillation , the problem turns into mimicking the moving target properly . While each node of the logit is confined to represent its assigned class probability which does not change drastically over iterations , at the feature map-level , much more flexibility comes into play , which makes the problem more challenging . Therefore , the direct aligning method such as using L1 or L2 distance is not suitable for online mutual feature map distillation because it updates the network parameters to generate a feature map that tries to mimic the current output feature map of the other network . In other words , the direct alignment method only tries to minimize the distance between the two feature map points ( one for each network ) , hence it ignores the distributional difference between the two feature maps ( Fig . 1 ( a ) ) . To alleviate this problem , in this paper , we propose a novel online distillation method that transfers the knowledge of feature maps adversarially as well as a cyclic learning framework for training more than two networks simultaneously . Unlike the direct aligning method , our adversarial distillation method enables a network to learn the overall feature map distribution of the co-trained network ( Fig . 1 ( b ) ) . Since the discriminator is trained to distinguish the difference between the networks ’ feature map distributions ( containing the history of feature maps for different input images ) at every training iteration , by fooling the discriminator , the network learns the co-trained network ’ s changing feature map distribution . Exchanging the knowledge of feature map distribution facilitates the networks to converge to a better feature map manifold that generalizes better and yields more accurate results . Our method consists of two major losses : 1 ) logit-based loss and 2 ) feature map-based loss . Logitbased loss is defined by two different loss terms which are conventional cross-entropy ( CE ) loss and the mutual distillation loss using the Kullback-Leibler divergence ( KLD ) . Our newly proposed feature map-based loss is to distill the feature map indirectly via discriminators . We use the feature map from the last convolution layer since deeper convolution layer generates more meaningful features with a high-level abstraction ( Kim et al. , 2018 ) . The adversarial training scheme of generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) is utilized to transfer the knowledge at feature map-level . The contributions of this paper can be summarized as follows : 1 ) we propose an online knowledge distillation method that utilizes not only the logit but also the feature map from the convolution layer . 2 ) Our method transfers the knowledge of feature maps not by directly aligning them using the distance loss but by learning their distributions using the adversarial training via discriminators . 3 ) We propose a novel cyclic learning scheme for training more than two networks simultaneously . 2 RELATED WORK . The idea of model compression by transferring the knowledge of a high performing model to a smaller model was originally proposed by Buciluǎ et al . ( 2006 ) . Then in recent years , this research area got invigorated due to the work of knowledge distillation ( KD ) by Hinton et al . ( 2015 ) . The main contribution of KD is to use the softened logit of pre-trained teacher network that has higher entropy as an extra supervision to train a student network . KD trains a compact student network to learn not only by the conventional CE loss subjected to the labeled data but also by the final outputs of the teacher network . While KD only utilizes the logit , method such as FitNet ( Romero et al. , 2014 ) , AT ( Zagoruyko & Komodakis , 2016a ) , FT ( Kim et al. , 2018 ) and KTAN ( Liu et al. , 2018 ) use the intermediate feature representation to transfer the knowledge of a teacher network . Online Knowledge Distillation : Conventional offline methods require training a teacher model in advance while online methods do not require any pre-trained model . Instead , the networks teach each other mutually by sharing their knowledge throughout the training process . Some examples of recent online methods are DML ( Zhang et al. , 2018 ) and ONE ( Lan et al. , 2018 ) which demonstrate promising results . DML simply applies KD losses mutually , treating each other as teachers , and it achieves results that is even better than the offline KD method . The drawback of DML is that it lacks an appropriate teacher role , hence provides only limited information to each network . ONE pointed out this defect of DML . Rather than mutually distilling between the networks , ONE generates a gated ensemble logit of the training networks and uses it as a target to align for each network . ONE tries to create a powerful teacher logit that can provide more generalized information . The flaw of ONE is that it can not train different network architectures at the same time due to its architecture of sharing the low-level layers for the gating module . The common limitation of existing online methods is that they are dependent only on the logit and do not make any use of the feature map information . Considering that KD loss term is only applicable to the classification task , transferring knowledge at feature map-level can enlarge the applicability to other tasks . Therefore , our method proposes a distillation method that utilizes not only the logit but also the feature map via adversarial training , moreover , our method can be applied in case where the co-trained networks have different architectures . Generative Adversarial Network ( GAN ) : GAN ( Goodfellow et al. , 2014 ) is a generative model framework that is proposed with an adversarial training scheme , using a generator network G and a discriminator network D. G learns to generate the real data distribution while D is trained to distinguish the real samples of the dataset from the fake results generated by G. The goal of G is to trickD to make a mistake of determining the fake results as the real samples . Though it was initially proposed for generative models , its adversarial training scheme is not limited to data generation . Adversarial training has been adapted to various tasks such as image translation ( Isola et al. , 2017 ; Zhu et al. , 2017 ) , captioning ( Dai et al. , 2017 ) , semi-supervised learning ( Miyato et al. , 2016 ; Springenberg , 2015 ) , reinforcement learning ( Pfau & Vinyals , 2016 ) , and many others . In this paper , we utilize GAN ’ s adversarial training strategy to transfer the knowledge at feature map-level in an online manner . The networks learn the other networks ’ feature map distributions by trying to deceive the discriminators while the discriminators are trained to distinguish the different distributions of each network .

Summary: This paper presents a new deep mutual learning (i.e., online peer-teaching) method based on Knowledge Distillation (KD) in a feature map level. The target task is similar with the original KD in the sense that the a network is taught by another network as well as groundtruth labels, but different with the KD in the sense that the networks are not a (frozen) teacher and a student but teaching each other in an online manner. Most approaches in this relatively new line of research rely on logit-based KD for transferring knowledges between networks, and the paper demonstrates that by an additional feature map level KD the performance can be further improved.