Feature-map-level Online Adversarial Knowledge Distillation

Feature maps contain rich information about image intensity and spatial correlation . However , previous online knowledge distillation methods only utilize the class probabilities . Thus in this paper , we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework . We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks . Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real . By training a network to fool the corresponding discriminator , it can learn the other network ’ s feature map distribution . Discriminators and networks are trained concurrently in a minimax twoplayer game . Also , we propose a novel cyclic learning scheme for training more than two networks together . We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one . 1 INTRODUCTION . With the advent of Alexnet ( Krizhevsky et al. , 2012 ) , deep convolution neural networks have achieved remarkable success in a variety of computer vision tasks . However , high-performance of deep neural network is often gained by increasing the depth or the width of a network . Deep and wide networks cost a large number of computation as well as memory storage which is not suitable for a resource-limited environment such as mobile or embedded systems . To overcome this issue , many researches have been conducted to develop smaller but accurate neural networks . Some of the well-known methods in this line of research are parameter quantization or binarization ( Rastegari et al. , 2016 ) , pruning ( Li et al. , 2016 ) and knowledge distillation ( KD ) ( Hinton et al. , 2015 ) . KD has been an active area of research as a solution to improve the performance of a light-weight network by transferring the knowledge of a large pre-trained network ( or an ensemble of small networks ) as a teacher network . KD sets the teacher network ’ s class probabilities as a target which a small student network tries to mimic . By aligning the student ’ s predictions to those of the teacher , the student can improve its performance . Recently , some studies have shown that rather than using a pretrained teacher , simultaneously training networks to learn from each other in a peer-teaching manner is also possible . This approach is called online distillation . Deep mutual learning ( DML ) ( Zhang et al. , 2018 ) and on-the-fly native ensemble ( ONE ) ( Lan et al. , 2018 ) are the representative online distillation methods that show appealing results in the image classification tasks . Conventional distillation method requires pre-training a powerful teacher network and performs an one-way transfer to a relatively small and untrained student network . On the other hand , in online mutual distillation , there is no specific teacher-student role . All the student networks learn simultaneously by teaching each other from the start of training . It trains with the conventional cross-entropy loss from the ground truth label along with the mimicry loss to learn from its peers . Networks trained in such an online distillation way achieve results superior not only to the networks trained with the cross-entropy loss alone but also to those trained in conventional offline distillation manner from a pre-trained teacher network . However , aforementioned online distillation methods make use of only the logit information . While the logit contains the probabilistic information over classes , the feature map , the output of convolution layer , has more meaningful and abundant feature information on image intensity and spatial correlation . In offline distillation which utilizes a pre-trained model as a teacher network , many methods such as FitNet ( Romero et al. , 2014 ) , attention transfer ( AT ) ( Zagoruyko & Komodakis , 2016a ) and factor transfer ( FT ) ( Kim et al. , 2018 ) make use of this intermediate feature representation as a target to learn for the student network , but in online distillation , to the best of our knowledge , no feature map-based knowledge distillation method has been proposed . This is due to some challenges . Unlike the offline methods that have a clear target to mimic , there is no static target to follow in an online method . At every training iteration , the feature maps of the co-trained network change , thus in online feature map-level distillation , the problem turns into mimicking the moving target properly . While each node of the logit is confined to represent its assigned class probability which does not change drastically over iterations , at the feature map-level , much more flexibility comes into play , which makes the problem more challenging . Therefore , the direct aligning method such as using L1 or L2 distance is not suitable for online mutual feature map distillation because it updates the network parameters to generate a feature map that tries to mimic the current output feature map of the other network . In other words , the direct alignment method only tries to minimize the distance between the two feature map points ( one for each network ) , hence it ignores the distributional difference between the two feature maps ( Fig . 1 ( a ) ) . To alleviate this problem , in this paper , we propose a novel online distillation method that transfers the knowledge of feature maps adversarially as well as a cyclic learning framework for training more than two networks simultaneously . Unlike the direct aligning method , our adversarial distillation method enables a network to learn the overall feature map distribution of the co-trained network ( Fig . 1 ( b ) ) . Since the discriminator is trained to distinguish the difference between the networks ’ feature map distributions ( containing the history of feature maps for different input images ) at every training iteration , by fooling the discriminator , the network learns the co-trained network ’ s changing feature map distribution . Exchanging the knowledge of feature map distribution facilitates the networks to converge to a better feature map manifold that generalizes better and yields more accurate results . Our method consists of two major losses : 1 ) logit-based loss and 2 ) feature map-based loss . Logitbased loss is defined by two different loss terms which are conventional cross-entropy ( CE ) loss and the mutual distillation loss using the Kullback-Leibler divergence ( KLD ) . Our newly proposed feature map-based loss is to distill the feature map indirectly via discriminators . We use the feature map from the last convolution layer since deeper convolution layer generates more meaningful features with a high-level abstraction ( Kim et al. , 2018 ) . The adversarial training scheme of generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) is utilized to transfer the knowledge at feature map-level . The contributions of this paper can be summarized as follows : 1 ) we propose an online knowledge distillation method that utilizes not only the logit but also the feature map from the convolution layer . 2 ) Our method transfers the knowledge of feature maps not by directly aligning them using the distance loss but by learning their distributions using the adversarial training via discriminators . 3 ) We propose a novel cyclic learning scheme for training more than two networks simultaneously . 2 RELATED WORK . The idea of model compression by transferring the knowledge of a high performing model to a smaller model was originally proposed by Buciluǎ et al . ( 2006 ) . Then in recent years , this research area got invigorated due to the work of knowledge distillation ( KD ) by Hinton et al . ( 2015 ) . The main contribution of KD is to use the softened logit of pre-trained teacher network that has higher entropy as an extra supervision to train a student network . KD trains a compact student network to learn not only by the conventional CE loss subjected to the labeled data but also by the final outputs of the teacher network . While KD only utilizes the logit , method such as FitNet ( Romero et al. , 2014 ) , AT ( Zagoruyko & Komodakis , 2016a ) , FT ( Kim et al. , 2018 ) and KTAN ( Liu et al. , 2018 ) use the intermediate feature representation to transfer the knowledge of a teacher network . Online Knowledge Distillation : Conventional offline methods require training a teacher model in advance while online methods do not require any pre-trained model . Instead , the networks teach each other mutually by sharing their knowledge throughout the training process . Some examples of recent online methods are DML ( Zhang et al. , 2018 ) and ONE ( Lan et al. , 2018 ) which demonstrate promising results . DML simply applies KD losses mutually , treating each other as teachers , and it achieves results that is even better than the offline KD method . The drawback of DML is that it lacks an appropriate teacher role , hence provides only limited information to each network . ONE pointed out this defect of DML . Rather than mutually distilling between the networks , ONE generates a gated ensemble logit of the training networks and uses it as a target to align for each network . ONE tries to create a powerful teacher logit that can provide more generalized information . The flaw of ONE is that it can not train different network architectures at the same time due to its architecture of sharing the low-level layers for the gating module . The common limitation of existing online methods is that they are dependent only on the logit and do not make any use of the feature map information . Considering that KD loss term is only applicable to the classification task , transferring knowledge at feature map-level can enlarge the applicability to other tasks . Therefore , our method proposes a distillation method that utilizes not only the logit but also the feature map via adversarial training , moreover , our method can be applied in case where the co-trained networks have different architectures . Generative Adversarial Network ( GAN ) : GAN ( Goodfellow et al. , 2014 ) is a generative model framework that is proposed with an adversarial training scheme , using a generator network G and a discriminator network D. G learns to generate the real data distribution while D is trained to distinguish the real samples of the dataset from the fake results generated by G. The goal of G is to trickD to make a mistake of determining the fake results as the real samples . Though it was initially proposed for generative models , its adversarial training scheme is not limited to data generation . Adversarial training has been adapted to various tasks such as image translation ( Isola et al. , 2017 ; Zhu et al. , 2017 ) , captioning ( Dai et al. , 2017 ) , semi-supervised learning ( Miyato et al. , 2016 ; Springenberg , 2015 ) , reinforcement learning ( Pfau & Vinyals , 2016 ) , and many others . In this paper , we utilize GAN ’ s adversarial training strategy to transfer the knowledge at feature map-level in an online manner . The networks learn the other networks ’ feature map distributions by trying to deceive the discriminators while the discriminators are trained to distinguish the different distributions of each network .

Summary: A new online knowledge distillation is investigated by utilizing feature map information next to the logits via GAN. Instead of direct feature map alignment, the algorithm tries to transfer the distribution of the feature maps. There is no teacher per se, but the big and small nets are trained via an adversarial game where 2 discriminators try to minimize the distributions of the two nets. The idea is understandable but some issues remain: