Feature-map-level Online Adversarial Knowledge Distillation
Authors: Inseop Chung
SeongUk Park
Jangho Kim
Nojun Kwak
Venue:  (2020)
Keywords: Computer vision
Image classification
Knowledge distillation
Deep Learning

Abstract: Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network’s feature map distribution. Discriminators and networks are trained concurrently in a minimax two-player game. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.

Under review as a conference paper at ICLR 2020
FEATURE-MAP-LEVEL ONLINE ADVERSARIAL KNOWLEDGE DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Feature maps contain rich information about image intensity and spatial corre- lation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose online knowledge distilla- tion method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its correspond- ing discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distri- bution. Discriminators and networks are trained concurrently in a minimax two- player game. Also, propose novel cyclic learning scheme for training more than two networks together. We have applied our method to various network ar- chitectures the classification task and discovered significant improvement of performance especially in the case of training pair of a small network and a large one.
INTRODUCTION
With the advent of Alexnet (Krizhevsky et al., 2012). deep convolution neural networks have achieved remarkable success in a variety of computer vision tasks. However, high-performance of deep neural network is often gained by increasing the depth or the width of a network. Deep and wide networks cost a large number of computation as well as memory storage which is not suitable for a lomuree environment such as mobile or embedded systems. To overcome this issue, many researches have been conducted to develop smaller but accurate neural networks. Some of the well-known methods in this line of research are purameter quantization or binarization (Rastegari et al. 2016), pruning (Li et al. 2016) and knowledge distillation (KD) (Hinton et al. 2015).
KD has been an active area of research as a solution to improve the performance of a ligght network by transferring the knowledge of large pre- pre-trained network (or ensemble of small net- works) as a teacher network. KD sets the teacher network's s class probabilities as a target which a small student network tries mimic. By aligning the student's predictions those of the teacher, the student can improve its performance. Recently, some studies have shown that rather than using a pre- trained teacher, simultaneously training networks learn from each other in a peer-teaching manner is also possible. This approach is called online distillation. Deep mutual learning (DML) (Zhang al. 2018) and on-the-fly native ensemble (ONE) (Lan et al. 2018) are the representative online distillation methods that show appealing results in the image classification tasks. Conventional dis- tillation method requires pre-training powerful teacher network and performs an one-way transfer to relatively small and untrained student network. On the other hand, in online mutual distillation, there is no specific teacher-student role. All the student networks learn simultaneously by teach- ing each other from the start of training. It trains with the conventional cross-entropy loss from the ground truth label along with the mimicry loss to learn from its peers. Networks trained in such an online distillation way achieve results superior not only to the networks trained with the cross-entropy loss alone but also to those trained in conventional offline distillation manner from a pre-trained teacher network.

---

Under review as a conference paper at ICLR 2020
A
Not
Net
Net
(a) Direct Feature Map Alignment
(b) Adversarial Feature Map Distillation
Figure 1: The concept of online Adversarial Feature map Distillation (AFD) Each point represents a feature map for the corresponding input denoted by different colors. The thin line arrow indicates the evolvement of feature map data points as iteration goes on and the broader arrow indicates the way each method compares the feature maps from different networks. (a) direct feature map alignment, networks are trained such that the distance between each pair of points with the same color is minimized. (b) In AFD, the discriminators contain information on feature map distributions and thus the networks are trained such that the distributions match. (best viewed in color)
However, aforementioned online distillation methods make use of only the logit information. While the logit contains the probabilistic information over classes, the feature map, the output of convo- lution layer, has more meaningful and abundant feature information on image intensity and spatial correlation. In offline distillation which utilizes a pre-trained model as a teacher network, many methods such FitNet (Romero et al. 2014), attention transfer (AT) (Zagoruyko & Komodakis 2016a) and factor transfer (FT) (Kim et al., 2018) make use of this intermediate feature repre- sentation target to learn for the student network, but in online distillation, the best of our knowledge, no feature map-based knowledge distillation method has been proposed.
This is due to some challenges. Unlike the offline methods that have clear target to mimic, there is no static target to follow in an online method. At every training iteration, the feature maps of the co-trained network change, thus in online feature map-level distillation, the problem turns into mimicking the moving target properly. While each node of the logit is confined to represent its assigned class probability which does not change drastically over iterations, at the feature map-level, much more flexibility comes into play, which makes the problem more challenging. Therefore, the direct aligning method such as using L1 or L2 distance not suitable for online mutual feature map distillation because it updates the network parameters to generate a feature map that tries to mimic the current output feature map of the other network. other words, the direct alignment method only tries to minimize the distance between the two feature map points (one for each network), hence it ignores the distributional difference between the two feature maps (Fig. I()).
To alleviate this problem, in this paper, propose novel online distillation method that transfers the knowledge of feature maps adversarially as well as a cyclic learning framework for training more than two networks simultaneously. Unlike the direct aligning method, our adversarial distillation method enables a network to learn the overall feature map distribution of the co-trained network (Fig. [h)). Since the discriminator is trained distinguish the difference between the networks' feature map distributions (containing the history of feature maps for different input images) at every training iteration, by fooling the discriminator, the network learns the co-trained network's changing feature map distribution. Exchanging the knowledge of feature map distribution facilitates the networks to converge to a better feature map manifold that generalizes better and yields more accurate results.
Our method consists of two major losses: 1) logit-based based loss and 2) feature mas-based loss. Logit- based loss is defined by two different loss terms which are conventional cross-entropy (CE) loss and the mutual distillation loss using the Kullback-Leebbee divergence (KLD). Our newly proposed fea- ture map- loss to distill the feature map indirectly via discriminators. We use the feature map from the last convolution layer since deeper convolution layer generates more meaningful features with a high-level abstraction (Kim et al.) 2018). The adversarial training scheme generative ad- versarial networks (GAN) (Goodfellow et al. 2014) is utilized to transfer the knowledge feature map-level.
The contributions of this paper be summarized follows: 1) propose an online knowledge distillation method that utilizes not only the logit but also the feature map from the convolution layer. 2) Our method transfers the knowledge of feature maps not by directly aligning them using
2

---

Under review as a conference paper at ICLR 2020
the distance loss but by learning their distributions using the adversarial training via discriminators. 3) We propose a novel cyclic learning scheme for training more than two networks simultaneously.
2 RELATED WORK
The idea of model compression by transferring the knowledge of a high performing model to a smaller model was originally proposed by Buciluă et al. (2006). Then in recent years, this research area got invigorated due to the work of knowledge distillation (KD) by Hinton et al.] (2015). The main contribution of KD is to use the softened logit of pre-trained teacher network that has higher entropy as an extra supervision to train a student network. KD trains a compact student network to learn not only by the conventional CE loss subjected to the labeled data but also by the final outputs of the teacher network. While KD only utilizes the logit, method such as FitNet (Romero et al. 2014), AT (Zagoruyko & Komodakis 2016a). FT (Kim et al., 2018) and KTAN (Liu et al.) 2018 use the intermediate feature representation to transfer the knowledge of a teacher network.
Online Knowledge Distillation: Conventional offline methods require training a teacher model in advance while online methods do not require any pre-trained model. Instead, the networks teach each other mutually by sharing their knowledge throughout the training process. Some examples of recent online methods are DMI. /Zhang et al. (018)) and ONE ([.an et al. (018)) which demonstrate promising results. DML simply applies KD losses mutually, treating each other as teachers, and achieves results that is even better than the offline KD method. The drawback of DML is that it lacks an appropriate teacher role, hence provides only limited information to each network. ONE pointed out this defect of DML. Rather than mutually distilling between the networks, ONE generates gated ensemble logit of the training networks and uses it as a target to align for each network. ONE tries to create a powerful teacher logit that can provide more generalized information. The flaw of ONE is that it can not train different network architectures at the same time due to its architecture of sharing the low-level layers for the gating module. The common limitation of existing online methods is that they are dependent only on the logit and do not make any use of the feature map information. Considering that KD loss term only applicable to the classification task, transferring knowledge at feature map-level can enlarge the applicability to other tasks. Therefore, our method proposes distillation method that utilizes not only the logit but also the feature map via adversarial training, moreover, our method can be applied in case where the co-trained networks have different architectures.
Generative Adversarial Network (GAN): GAN (Goodfellow et al., 2014) is generative model framework that is proposed with an adversarial training scheme, using a generator network and discriminator network D. G' learns to generate the real data distribution while D is trained to dis- tinguish the real samples of the dataset from the fake results generated by G. The goal of G is to trick D to make mistake of determining the fake results as the real samples. Though it was initially proposed for generative models, its adversarial training scheme is not limited to data generation. Adversarial training has been adapted various tasks such image translation (Isola et al. 2017]]
Zhu et al. (217)), captioning (Dai et al. (017). semi-supervised learning (Miyato et al. [016]] Sprin- genberg 2015), reinforcement learning (Paau& & Vinyals 2016), and many others. In this paper, we utilize GAN's s adversarial training strategy to transter the knowledge at feature map-level in an on- line manner. The networks learn the other networks' feature map distributions by trying to deceive the discriminators while the discriminators are trained to distinguish the different distributions of each network.
3 PROPOSED METHOD
In this section, we describe the overall process of our proposed Online Adversarial Feature map Distillation (AFD). As can be seen in F when training two different networks, O₁ and Oz in an online manner, we employ two discriminators, D, and Dz. We train Dj such that the feature map of O2 is regarded nator D2. Then, each network O and θ₂ are trained to fool its corresponding discriminator so that it can generate feature map that mimics the other network's 's feature map. Throughout this adver- sarial training, each network learns the feature map distribution of the other network. By exploiting both logit-based distillation loss and feature map-based adversarial loss together, we could observe
real and that of θ is classified as a fake and do vice versa for discrimi-
3

---

Under review as a conference paper at ICLR 2020
Logit
Prediction
C
NetworkO,
Feature Mapi
GT Labels
Fake
D1
0-11
Real
Cu(P1,
Cu P1)
Real
D2
0~1
Fake
Feature
Network0,
Map;
Logitz
Predictions
Figure 2: Overall schematic of online adversarial feature map distillation (AFD). At feature map- level, each network is trained to deceive the corresponding discriminator so that it can mimic the other network's feature map distribution. While at logit-level, KL loss to learn the peer network's logit is applied as well as the conventional CE loss.
significant improvement of performance in various pairs of network architectures especially when training small and large networks together. Also we introduce a cyclic learning scheme for train- ing more than two networks simultaneously. reduces the number of required discriminators from 2 X2 Cxx(hhe (when employing discriminators bidirectionally between every network pairs.) to K where K the number of networks participating. This cyclic learning framework not only requires less computation than the bidirectional way but also achieves better results compared to other online training schemes for multiple networks.
First, we explain the conventional mutual knowledge distillation method conducted among the net- works at the logit-level. Then we introduce our novel online feature map distillation method using the adversarial training scheme in addition to the cyclic learning framework for training than two networks at the same time.
3.1 LOGIT-BASED MUTUAL KNOWLEDGE DISTILLATION
We use two loss terms for logit-based learning, one is the conventional cross-entropy(CE) loss and the other is mutual distillation loss between networks based on Kullback Leibler(KI.) divergence. We formulate our proposed method assuming training two networks. Training scheme for more than two networks will be explained in See 3.3 Below is the overall logit-based loss for two networks:
Clogit (v) + 2 x Culla(z//), Cingit Lee (2,) 2 Culo((a/T))
(2)
Here, o(.) refers to softmax function and E RC is the logit produced from network for C- class classification problem. The temperature term T' is used to control the level of smoothness in probabilities. As the temperature term 7' goes up, creates We use T = 3 for every experiment. Lee is the CE loss between the ground truth label y and the softmax output (2x) that is commonly used in image classification. Lsi the KL loss between the softened logit of each network. We multiply the KL loss term with 2 because the gradients produced by the soft targets are scaled by 1/T2. While the CE loss is between the correct labels and the outputs of the model, the KI. loss is the KI. distance between the outputs of two training networks. The KL loss provides an extra information from the peer network so that the network can improve its generalization performance. The difference with DML is that while DML updates asynchronously which means that it updates one network first and then the other network, our AFD updates the networks synchronously, not alternatingly. The CE loss trains the networks to predict the correct truth label while the mutual distillation loss tries to match the outputs of the peer-netweee neer-networks, enabling the networks to share the knowledge logit-level.
softened probability distribution.
4

---

Under review as a conference paper at ICLR 2020
3.2 FEATURE
LEARNING VIA ADVERSARIAL TRAINING
Our AFD uses adversarial training to transfer knowledge at feature map-level. We formulate our ad- versarial feature map distillation for two networks which will be extended for more networks later. We divide network into two parts, one is the feature extractor part that generates feature map and the other is the classifier part that transforms the feature map into a logit. Each network also has a cor- responding discriminator which distinguishes different feature map distributions. The architecture of the discriminator is simply a series of Cony-Batch. Cony-Batch_Normallle Normalization-LLakk Ree_U-Covv----- -oniigoodd takes feature map of the last layer and it reduces the spatial size and the number of channel of the feature map as it goes through the convolution operation so that it can produce single scalar value. Then we apply the sigmoid function of the value normalize it between 0 and 1.
We utilize the feature extractor part to enable feature map-level distillation. For the convenience of mathematical notation, we name the feature extractor part as G₁. and its discriminator as DR. indicates the network number. As depicted in Figure 2 each network has to fool its discriminator to mimic the peer network's 's feature map and the discriminator has to discriminate from which network the feature map is originated. Following LSGAN (Mao et al. 2017)), our overall adversarial loss for discriminator and the feature extractor can be written as below:
Cp, = [1 - (=(G) (Dr(C(())))
(3)
La. = - DG
(4)
The feature extractors G1 and G2 take input æ and generate feature maps. The discriminator D1 takes feature map and yields scalar between 0 (fake) and (real). It is trained to output 1 if the feature map came from the co-trained network (in this case, G2) or O if the feature map is produced from the network it belongs to (G₁ this case). The goal of D2 is to minimize the discriminator loss term LDI by correctly distinguishing the two different feature map distributions while Gi's goal is to minimize the loss term Lo, by fooling D to make mistake of determining Gi 's feature map as real and yield 1. Each training network's 's object is to minimize La, mimic the peer network's s feature map distribution. This adversarial scheme works exactly the same by changing the role of two networks.
In case where the two networks' feature map outputs have different channel sizes, for example a pair like (WRN-16-2, WRN-16-4) (Zagoruyko & Komodakis 2016b), we use transfer layer that is composed of a convolution layer, a batch normalization and ReLU which converts the number of channels to that of peer network. The above loss terms change as Ln, [1 Di (T2(G2 (Gi Optimization: Combining both logit-based loss and the adversarial feature map-based loss, the overall loss for each network Q and O2 are as follows:
[Di (GT (+))))² and La, = [1 (GI when using the transfer layer Tu.
Le, = Clogit + LG Le, Clogit + LG,
(5)
However, the logit-hased loss term Stagit and the feature map-based loss term La, are not optimized by the same optimizer. In fact, they are optimized alternatingly in a same mini-batch. At every mini- batch iteration, calculate the two loss terms and optimize the networks based on the two losses separately, meaning that we update the parameters by the logit-based loss once and then update again by the feature map-based loss. The reason we optimize separately for each loss term is because they use different learning rates. The adversarial loss requires much slower learning rate thus if we use the same optimizer with the same learning rate, the networks would not be optimized. Note that we do not infer for each loss term, inference conducted only once, only the optimization is conducted twice, one for each loss term.
infer image into model and it computes a logit and feature map. Then
3.3 CYCLIC LEARNING FRAMEWORK
Our method proposes a novel cyclic peer-learning scheme for training more than two networks simultaneously. As can be seen in Figure 3, each network transfers its knowledge to its next peer network in an one-way cyclic manner. If we train number of networks together, each network distills its knowledge to its next network except the last network transfers its knowledge the first.
5

---

Under review as a conference paper at ICLR 2020
0
D2
Real
Fake
CPI P2)
Feature Map
L
Feature
N
Networkk
M op:
Fake
CurlP PL)
Cet P3)
Real
0-1e 1~14
D2
D.
0 ~1
Feature Maps
NetworkO3
Real
Fake
Figure 3: Schematic of cyclic-learning framework for training 3 networks simultancously.
network, creating cyclic knowledge transfer flow as - -
The main contribution of using this cyclic learning framework to avoid employing too many number of discriminators. If we apply our adversarial loss for every pair of networks, it would demand two times the amount of every possible pair of K networks which would cost a lot of computation. Also in Sec 4.5 empirically show that our cyclic training scheme is better than other online methods' training scheme for multiple networks.
EXPERIMENT
In this section, to show the adequacy of our method, we first present comparison experiment with distance method and ablation study to analyze our method. Then compare approach with existing online knowledge distillation methods under different settings. First of all, we demonstrate results using the same sub-network architectures in Sec 4.3) Then, apply method sub- networks with different architectures in Sec (.4]. In Sec 4.5 we also show the results of training more than networks demonstrate that method generalizes well when the number of networks increases.
In most of the experiments, training images and 10K test images over 100 classes, accordingly it has 600 images per each class. All the reported results on CIFAR-I00 are average of experiments. Since our method uses loss terms, basit-based loss and feature map-based loss, we use different learning details for each loss term. For overall learning schedule, we follow the learning schedule of ONE(Lan et al.) 2018) to conduct fair comparison which is 300 epochs of training. In terms of logit-based loss, the learning rate starts at 0.1 and multiplied by 0.1 at 150, 225 epoch. We optimize the logit-based loss using SGD with mini-bated size of 128, momentum 0.9 and weight decay of lc-4. This learning details for logit-based loss is equally applied to other compared online distillation methods. For feature map-based loss, the learning rate starts at 2c-5 for both discriminators and feature extractors and is decayed by 0.1 at 75, 150 epoch. The feature map-based loss is optimized by ADAM(Kingma & Ba 2014) with the same mini-batch size and weight decay of 1e-1.
the CIFAR- (Krizhevsky et al.) dataset. It consists of 50K
In tables, '2 Net Avg' and 'Ens' represents the average accuracy the two sub-ne nebborsssrks and the ensemble accuracy respectively. The average ensemble is used for AFD, DML and KD while ONE uses gated ensemble of sub-networks according to its methodology.
4.1 COMPARISON WITH DIRECT FEATURE MAP ALIGNMENT METHODS
Since our goal to distill feature map information that suits for mutual online distillation, we briefly compare method with conventional direct alignment method in Table We train two networks together, in one setting, we use the same architecture (ResNet-32 (He et al. 2016)) and in the other, different types (WRN-16-2, WRN-28-2 (Zagoruyko & Komodakis 2016b)). For I,, each

---

Under review as a conference paper at ICLR 2020
Table 1: Top-1 accuracy(%) ()) comparison with direct alignment methods using CIFAR-100 dataset.
Model Type
L KD
2,1
KD (cilline)
AFD
Vanilla
Same
Ave
Avg
Student
Ens
Net Ave
Net
ResNet-32
66.82
70.69
70.16
72.44
71.91
69.79
12.07
74.03
75.64
69.38
Dillereni Arch.
Nez2
Ens
Necl
Net2
Studeat
Teacher
Ens
Net2
Netl
Net2
WRN-116-2228-2
69.84
73.41
74.63
12.35
74.82
75.10
73.34
73.62
76.56
75,88
77.08
77,82
71.07
73.30
Table 2: Ablation study of AFD. Top-1 accuracy(%) on CIFAR-100 dataset.
Model Type
w/o KD (Adv only)
w/o Adv (KD only)
Full model (AFD)
Same Arch.
2 Net Avg
Ens
2 Net Avg
Ens
2 Net Avg
Ens
ResNet-32
70.09
74.77
73.38
75.21
74.03
75.64
WRN-
71.94
75.92
74.81
76.20
75.33
76.34
Different Arch.
Netl
Net2
Ens
Netl
Net2
Ens
Netl
Net2
Ens
WRN-(16-2,28-22)
72.05
73.80
76.82
74.99
76.64
77.28
75,88
77.08
77.82
network is trained not only follow the ground-truth label by CE loss, but also to mimic the other network's feature map using the In distance loss. For + KD, KD (Hinton et al. 2015) loss is applied mutually along with the L1 loss between the feature maps. We also compare our results with offline method, L1+ KD (offline) employs a pre-trained network as a teacher network and distills its feature map knowledge to an untrained student network by L1 loss as well as the KD loss at logit level. ResNet-32 and WRN-28-2 that shows 69. 99.79% and 73.62% accuracy are used as the teacher networks in the two settings respectively. The results clearly show that learning the distributions of feature maps with adversarial loss performs better than direct alignment method in both mutual online distillation and offline distillation. We could observe that using I1 distance loss actually disturbs the networks to learn good features in online environment. The accuracy of ResNet-32 has dropped more than 2% compared to its vanilla version accuracy (69.38%) and the accuracy of WRN-16-2 is also lower than its vanilla network (71.07%). Even when combined with KD loss(L1 + KD), direct aligmment method shows poor performance compared to ours in both online and offline manner. Though distance loss is used in many conventional offline methods, they suffer when it comes to online environment. In case of different architecture types, our method also outperforms the direct alignment method. It indicates that when it comes to online feature map distillation, transferring feature map information with direct alignment method such as L1 distance worse than indirect distillation that uses feature map distribution via adversarial loss.
4.2 ABLATION STUDY
Table [] the ablation study of our proposed method. We conduct experiments using the same and different sub-network architectures. We run three experiments with different training settings for each model case. The three settings are full model, without mutual knowledge distillation at logit- level and without adversarial feature map distillation. When trained without the adversarial feature map distillation, the accuracy decreases in all three model cases. The accuracy of both ResNet-32 and WRN-16-2 dropped by 0.65% and 0.52% respectively, and those of (WRN-16-2, WRN-28-2) pair declined by 0.89% and 0.44% compared to the full model. Ensemble results are also lower than those of the full models. When only the adversarial feature map distillation is applied, the accuracy has increased by 0.71% and 0.87% compared to the vanilla versions of ResNet-32 and WRN-16-2 respectively. Especially in case of different sub-network architecture, the accuracy of WRN-16-2 has increased by almost 1%. Based on these experiments, we could confirm that adversarial feature map distillation has some efficacy of improving the performance in online environment.
4.3 SAME ARCHITECTURE
We compare our method with DML and ONE for training two sub-networks with the same architec- ture. The vanilla network refers to the original network trained without any distillation method. As shown in Table in both ResNet and WRN serises, DML, ONE and AFD all improves the networks' accuracy compared to the vanilla networks. However, AFD shows the highest improvement of per- formance in both sub-network and ensemble accuracy among the compared distillation methods. Especially in case of Res ResNet-32 and WRN-16-2, method significantly improves the
7

---

Under review as a conference paper at ICLR 2020
Table 3: Top-1 accuracy( ()) comparison with other online distillation methods for training same architecture networks as pair on the CIFAR- 100 dataset. The numbers in parentheses refer to the amount of increase in accuracy compared to the vanilla network.
DML
ONE
AFD
Model Type
Vanila
Net Avg
Ens
Net Avg
Ens
Net Avg
Ens
ResNet-20
70.90(+3.42%)
72.08
70.56(+3.08%
72.26
71.72(+4.24%%
72.98
67.48
ResNct-32
73,40(+4.02%)
74.89
72.61(+3.23%)
74.07
74.03(+4.65%; )
75.64
69.38
ResNet-556
75.48(+1.6498)
76.73
76.45(42.61%)
77.16
77.25(+3.41%%
78.35
73.84
WRN-16-2
74.68(+3.61%)
75.81
73.85(42.78%)
74.84
75.33(+4.226%)
76.34
71.07
WRN-16-4
78.17(+).79%)
79.06
77.32(+1.94%)
77.79
78.55(+3.17%%
79.28
75.38
WRN-28-2
77.02(43.52%)
78.64
76.67(33.17%)
77.40
77.22(+3.72%)
78.72
73.50
WRN-28-4
79.16(+2.56%)
80.56
79.25(42.65%)
79.73
79.46(+2.86%% %)
80.65
76.60
Table Top-1 accuracy(%) comparison with other online distillation methods for training two dif- ferent architectures pair on CIFAR-100 dataset.
Model Types
KD
DML
AFD
Netl
Net2
Netl
Net2
Ens
Netl
Net2
Ens
Netl
Ne12
Ens
ResNet-32
ResNet-50 56
72.92
76.27
76.71
73.48
76.35
76.74
74.13
76.69
77.11
ResNet-32
WRN-16-4
72.67
77.26
76.94
73.48
77.43
77.01
74.43
77.82
77.67
ResNet-56
WRN-28
75.48
78.91
79.23
76.03
79.32
79.38
77.95
79.21
80.01
ResNet-20
WRN-28-
70.08
78.17
76.12
71.03
77.70
75.78
72.62
77.83
76.70
WRN-16-2
WRN-16-4
74.87
77.42
77.30
74.87
77.17
76.96
75.81
78.00
77.84
WRN-16-2
WRN-28-2
74.86
76.45
77.29
75.11
76.91
77.24
75.88
77.08
77.82
WRN-16-2
WRN-28-4
74.51
78.18
77.60
74.95
78.23
77.67
76.23
78.26
78.28
Average
73.63
77.52
77.31
74.14
77.59
77.25
75.29
77.84
77.92
accuracy by more than 4% compared to the vanilla version while other distillation methods improve around 3% on average except the ResNet-32 of DML.
4.4 DIFFERENT ARCHITECTURE
In this section, compare method with DML and KD using different network architectures. We set Net2 as the higher capacity network. For KD, we use the ensemble the two uut-netwrrss teacher mimic every iteration. The difference with original KD (Hinton et al.) (015)) is that it is an online learning method, not offline. We did not include ONE because ONE can not be applied in case where the sub-networks have different model types due its architecture of sharing the low- level layers. In table 4 we could observe that our method shows better performance improvement than other methods in both Netl and Net2 except for a couple of cases. The interesting result is that when AFD is applied, the performance of Net1 (smaller network) is improved significantly compared to other online distillation methods. This is because AFD can transfer the higher capacity network's meaningful knowledge (feature map distribution) to the lower capacity one better than other online methods. When compared with KD and DML, AFD's 's Netl accuracy higher by 1.66% and 1.15% and the ensemble accuracy is better by 0.61% 61% and 0.67% average respectively. In case of (WRN-166 16-2, WRN-28-4) pair, the Netl's 's parameter size (0.70M) is more than 8 times smaller method improves both networks' accuracy,
than Net2 (5.8 (5.87M). Despite the large size difference, particularly our Netl performance is better than KD and DML by 1.72% and 1 1.28% respectively. The performance of KD and DML seems decline as the difference between the two model sizes gets larger. Throughout this experiment, we have shown that our method also works properly for different architectures of sub-networks even when two networks have large difference in their model sizes. Using our method, smaller network considerably benefits from the large network.
4.5 EXPANSION TO NETWORKS
To show method's expandability for training more than two networks, we conduct experiment of training 3 networks in this section. As proposed in See 3.3] our method uses a cyclic learning framework rather than employing adversarial loss between every network pairs in order reduce
8

---

Under review as a conference paper at ICLR 2020
Table 5: Topp1 accuracy($) ()) comparison with other online distillation methods using 3 networks CIFAR-100 dataset. '3 Net Avg' represents the average accuracy of the 3 networks.
DML
ONE
AFD
Model Type
Vanilla
3 Net Avg
Ens
3 Net Avg
Ens
3 Net Avg
Ens
ResNer-32
73.43
76.11
73.25
74.94
74.14
76.64
69.38
ResNet-556
76.11
77.83
76,49
77.38
77.37
79.18
73.84
WRN-16-2
75.15
76.93
73.87
75.26
75.65
77.54
71.07
WRN-28-2
77.12
79.41
76.66
77.53
77.20
79.78
73.50
Table Top-1 accuracy(%) comparison with DML on ImageNet dataset.
Model Types
DML
AFD
Vanilla
Netl
Net2
Netl
Net2
Ens
Netl
Net2
Ens
Netl 69.76
Net2
ResNet-18
ResNet-34
70.19
73.57
73.33
70.39
74.00
74.47
73.27
the amount of computation and memory. DML calculates the mutual knowledge distillation loss between every network pairs and uses the average of the losses. ONE generates gated ensemble of the sub-networks and transfers the knowledge of the ensemble logit to each network. As it can be seen in Table 5 AFD outperforms the compared online distillation methods on both 3 Net average and ensemble accuracy in every model types. Comparing the results of Table [ to that of Table the overall tendency of performance gains compared to DML and ONE is maintained.
4.6 IMAGENET EXPERIMENT
We evaluate our method on ImageNet dataset to show that our method can also be applicable to a large scale image dataset. We use ImageNet LSVRC 2015 (Russakovsky et al., (2015) which has 1.2M training images and 50K validation images over 1,000 classes. We compare our method with DML using two pre-trained networks ResNet-18 and ResNet-34 pair. The results after 30 epochs training. As shown in Table 6 our method improves the networks better than DML.
CONCLUSION
We proposed an online knowledge distillation method that transfers the knowledge not only logit- level but also at feature map-level using the adversarial training scheme. Unlike existing online distillation methods, our method utilizes the feature map information and showed that knowledge transfer at feature map-level is possible even in an online environment. Through extensive experi- ments, we demonstrated the adequacy of adopting the distribution learning via adversarial training for online feature map distillation and could achieve better performance than existing online meth- ods. We also introduced novel cyclic learning framework for training multiple networks concur- rently and presented its efficacy by comparing with existing approaches. We also confirmed that method is broadly suitable to various architecture types from a very small network (ResNet-20) to large (WRN-28-44 network. We hope that due to the work of research, the area of knowledge distillation can be further advanced and studied by many researchers.
REFERENCES
Cristian Buciluã, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541. ACM, 2006.
Bo Dai, Sanja Fidler. Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descrip- tions via conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2970- 2979, 2017.
Ian Goodfellow, Jean Pbadie- Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-11344 2017.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In Advances in Neural Information Processing Systems, pp. 2760-2769, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). URL http:://www es. torotto.ooo..o......... wwwp://tto.ddd//i/iiieeeeeeeeeee
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2238-. Curran Associates Inc., 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for
efficient convnets. arXiv preprint arXiv:1608.087100 2016.
Peiye Liu, Wu Liu, Huadong Ma, Tao Mci, and Mingoo Scok. Ktan: knowledge transfer adversarial network. arXiv preprint 1810.086,, 2018.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- on Computer Vision, pp. 2794-2802, 2017.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. arXiv preprint arXiv:1605.07725, 2016.
David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. arXiv preprint arXiv:1610.01945. 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xeor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 542. Springer, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65550 2014.
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Im- ageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV, 115(3):211-222 2015. doi: 10. 1233-015-0816.y.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint 1rXiv:1511.06390 2015.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03:: 1612.03928, 2016a.
10

---

Under review as a conference paper at ICLR 2020
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016b.
Wide residual networks.
arXiv preprint
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.


Reviews:
[{"review_id": "SJxW-yvHFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the online knowledge distillation problem and propose a method called AFD (Online Adversarial Feature map Distillation), which aims to transfers the knowledge of intermediate feature map (first propose) using adversarial training. Then, a cyclic learning scheme is proposed to train more than two networks simultaneously and efficiently. Ablation study on CIFAR100 shows that the adversarial training in AFD can improve the accuracy significantly, while the direct method such as using L1 distance is worse. The comparison experiments with several online distillation methods also show the effectiveness of proposed method.\n\nSome comments or suggestions:\n(i) The theoretical analysis is lacking. For example, some formulas proofs can be added to illustrate that the adversarial feature map distillation is more advantageous than the direct feature map alignment.\n(ii) The details of the experiments such as parameter configurations are missing, which makes the results not easy to be reproduced.\n(iii) Tab.1 and Tab.2 can be combined. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xs8081qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A new online knowledge distillation is investigated by utilizing feature map information next to the logits via GAN. Instead of direct feature map alignment, the algorithm tries to transfer the distribution of the feature maps. There is no teacher per se, but the big and small nets are trained via an adversarial game where 2 discriminators try to minimize the distributions of the two nets. The idea is understandable but some issues remain:\n1-\tTraining GAN is by itself an expensive task and optimization is difficult, so how computationally expensive is this online KD compared to the offline one?\n2-\tIt is not clear form the paper feature maps from which layers are being used? If multiple layers are considered, how did you choose which ones are better. Also the smaller model has a different structure, h ow did you choose to pair feature maps in the big model and the small model?\n3-\tWhat would be the performance difference compared to offline knowledge distillation? For example in Table 1 can you please add a column with offline KD?\n4-\tMutual training brings some generalization, and when you compare the results in Table 3 with vanilla model, I am wondering if you made sure this is not only due to a better generalization.\n5-\tIn cyclic learning framework, it is not well-motivated why someone wants to train multiple networks that mimic each others’ behavior; also the complexity increases in that case which makes me wondering wouldn’t it be better to do it offline then?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgVgUR-cH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Summary\nThis paper presents a new deep mutual learning (i.e., online peer-teaching) method based on Knowledge Distillation (KD) in a feature map level. The target task is similar with the original KD in the sense that the a network is taught by another network as well as groundtruth labels, but different with the KD in the sense that the networks are not a (frozen) teacher and a student but teaching each other in an online manner. Most approaches in this relatively new line of research rely on logit-based KD for transferring knowledges between networks, and the paper demonstrates that by an additional feature map level KD the performance can be further improved.\n\n\n= Decision\nThe current decision is borderline in my mind, but officially weak accept. Although the proposed method is simple and consists of known ideas, it is designed convincingly and enhances performance practically. Also, I believe the target task itself is worth to be introduced as a next direction of KD. However, the submission is weak in terms of novelty and the manuscript should be polished carefully.\n\n\n= Comments\n1) Weak clarity\n- The main motivation and advantages of the deep mutual learning is not well introduced in Section 1. Although the two papers (i.e., ONE and DML) are cited here, it would be much better to explicitly describe the main idea and motivation of peer-teaching, what is the difference between the task and the original KD, and the achievements in the previous work. Without these, readers, including me, may get confused why the online KD is required and why there is no clear teacher-student relationship between networks. \n- In a similar context, the motivation of introducing more than two student networks should be given.\n- The architecture of the discriminator seems not described even in the appendix. \n- The meaning of various arrow types in Figure 1 is not clearly described.\n\n2) Insufficient experiments\nIt would be better to report the performance of vanilla and (offline) KD in Table 1 to show more clearly that the feature map alignment is useful and that online KD is better than its offline counterpart.\n\n3) Limited novelty and performance improvement\n- The main idea is already introduced in previous work on the task and the feature map level KD has been studied widely for various applications, their combination is somewhat new though.\n- The performance improvement by the proposed feature map level KD seems marginal as shown in Table 2.\n- The performance gap between DML and the proposed model seems also marginal.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received scores of WR (R1) WR (R2) WA (R3), although R3 stated that they were borderline. The main issues were (i) lack of novelty and (ii) insufficient experiments. The AC has closely look at the reviews/comments/rebuttal and examined the paper. Unfortunately, the AC feels that with no-one strongly advocating for acceptance, the paper cannot be accepted at this time. The authors should use the feedback from reviewers to improve their paper. 
Decision: Reject