Knowledge distillation via softmax regression representation learning
Authors: Jing Yang
Brais Martinez
Adrian Bulat
Georgios Tzimiropoulos
Venue: ICLR 2021 Poster (2021)

Abstract: This paper addresses the problem of model compression via knowledge distillation. We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning. Previous distillation methods which typically impose direct feature matching between the student and the teacher do not take into account the classification problem at hand. On the contrary, our distillation method decouples representation learning and classification and utilizes the teacher's pre-trained classifier to train the student's penultimate layer feature. In particular, for the same input image, we wish the teacher's and student's feature to produce the same output when passed through the teacher's classifier which is achieved with a simple $L_2$ loss. Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains. The code will be available at \url{https://github.com/jingyang2017/KD_SRRL}.

Published as a conference paper at ICLR 2021
KNOWLEDGE DISTILLATION VIA SOFTMAX REGRES- SION REPRESENTATION LEARNING
Jing Yang
Brais Marinez
University of Nottingham Nottingham, UK j jing. wanggyyng228n wangg wiw.aatttiiiignmmm .ukuk
Samsung AI Center Cambridge, UK
brais wa www.trrrrataa.a.mmmmmmmmmm i com
Adrian Bulat
Georgios Tzimiropoulos
Samsung AI Center Cambridge, UK adrian@adri innbullt.. com,
Samsung AI Center
Cambridge, UK
Queen Mary University of London London, UK
g tz ttzim/rroooosseemmuu ac
ABSTRACT
This paper addresses the problem of model compression via knowledge distilla- tion. We advocate for method that optimizes the output feature of the penulti- mate layer of the student network and hence is directly related to representation learning. To this end, we firstly propose direct feature matching approach which focuses on optimizing the student's penultimate layer only. Secondly and more importantly, because feature matching does not take into account the classifica- tion problem at hand, we propose second approach that decouples represen- tation learning and classification and utilizes the teacher's pre-trained classifier to train the student's penultimate layer feature. In particular, for the same input image, we wish the teacher's and student's feature to produce the same output when passed through the teacher's classifier, which is achieved with simple L2 loss. Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-are methods large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains. The code is available httpp: /tttps:/ggithue com www comm/jnggangg00 201ryaanngn0n0oo00000 7/XD_SRRL SRRL
INTRODUCTION
Recently, there has been great amount of research effort to make Convolutional Neural Networks (CNNs) lightweight so that they can be deployed in devices with limited resources. To this end, several approaches for model compression have been proposed, including network pruning (Han) al. 2016; Lebedev & Lempitsky 20066), network quantization (Rastegari al.) 2016 Wu al. Zagoruyko & Komodakis] 2017). ) and
2016
knowledge transfer/distilation (Hinton et al., 2015
neural architecture search (Zoph Le 2017: al.) 2018 Knowledge distillation (Buciluà al.
2006
Hinton et al. 2015) aims to transfer knowledge from one network (the so-called "teacher") another (the so-called "student"). Typically, the teacher achieving high accuracy, while the student is a compact model with much fewer parameters, thus also requiring much less computation. The goal of knowledge distillation to use the teacher to improve the training of the student and push its accuracy closer to that of the teacher.
high-capacity model capable of
The rationale behind knowledge distillation can be explained from an optimization perspective: there is evidence that high capacity models (i.e. the teacher) find good local minima due to pver-parameterization (Du & Lee 2018 Soltanolkotabi et al.) 2018). In knowledge distillation, such models are used to facilitate the optimization of lower capacity models (i.e. the student) during training. For example, in the seminal work of (Hinton et al.] 2015), the softmax outputs of the teacher provide extra supervisory signals of inter-class similarities which facilitate the training of

---

Published as a conference paper at ICLR 2021
hr
T
CT
LSR loss
-
Input
FF loss
hg
S
Cs
Lc loss
Figure 1: Our method performs knowledge distillation by minimizing the discrepancy between the penultimate feature representations hy and hş of the teacher and the student, respectively. To this end, we propose
losses: (a) the Feature Matching loss LrM, and (b) the so-called Soft- max Regression loss LsR- contrary to LeM, our main contribution, Lgn- is designed to take into account the classification task hand. To this end, LsR imposes that for the same input im- age, the teacher's s and student's feature produce the same output when passed through the teacher's pre-trained and frozen classifier. Note that, for simplicity, the function for making the feature di- mensionality of fu and hs the same is not shown.
the student. In other influential works, intermediate representations extracted from the teacher such as feature tensors (Romero et al.) 2015) or attention maps (Zagoruyko & Komodakis (017) have been used define auxiliary loss functions used in the optimization of the student.
Training a network whose output feature representation is rich and powerful has been shown crucial for achieving high accuracy for the subsequent classification task in recent works in both unsuper- vised and supervised learning, see for example (Chen et al. 2020 He et al. (0200) and (Kang et al.) 2020). Hence, in this paper, we are advocating for representation learning- based knowledge distilla- tion by optimizing the student's 's penultimate layer output feature. If we are able to do this effectively, we expect (and show experimentally) to end up with a student network which can generalize better than one trained with logit matching in the KD paper of (Hinton al. 2015).
To accomplish the aforementioned goal we propose two loss functions: The first loss function, akin to (Romero et al. 2015 Zagoruyko & Komodakis 2017), is based direct feature matching focuses on optimizing the student's penultimate layer feature only. Because direct feature matching might be difficult due to the lower representation capacity of the student and, more importantly, is detached from the classification task at hand, we also propose a second loss function: we propose to decouple representation learning and classification and utilize the teacher's pre-trained classifier to train the student's penultimate layer feature. In particular, for the same input image, we wish the teacher's and student's feature to produce the same output when passed through the teacher's 's classifier, which is achieved with a simple L2 loss (see Fig. D This softmax regression projection is used to retain from the student's 's feature the information that is relevant to classification, but since the projection matrix is pre-trained (learned during the teacher's 's training phase) this does compromise the representational power of the student's 's feature.
Main contributions:
Main results: Our method has two advantages: (1) It is simple and straightforward to implement. (2) consistently outperforms state-of-the-and methods over a large set of experimental settings including different (a) network architectures (WiseResNets, ResNets, MobileNets), (b) teacher- student capacities, (c) datasets (CIFAR-10/100, ImageNet), and (d) domains (real-to-binary).
RELATED WORK
Knowledge transfer: In the work of (Hinton et al., 2015), knowledge is defined the teacher's outputs after the final softmax layer. The softmax outputs carry richer information than one-hot labels because they provide extra supervision signals in terms of the inter-class similarities learned
2

---

Published as a conference paper at ICLR 2021
by the teacher. In similar fashion (Hinton et al. 2015), intermediate representations extracted from the teacher such as feature tensors (Romero et al. 2013) or attention maps Zagoruyko & Komodakis 2017) have been used to define loss functions used to facilitate the optimization of the student. Trying to match the whole feature tensor, as in FitNets (Romero et al. 2013), is hard and, in certain cireumstances, such approach may adversely affect the performance and convergence of the student. To relax the assumption of FitNet, Attention Transfer (AT) was proposed in (Zagoruyko) & Komodakis 2017) where knowledge takes the form of attention maps which are summaries of the energies of the feature tensors over the channel dimension. An extension of (Zagoruyko & Komodakis 2017) using Maximum Mean Discrepancy of the network activations as loss term for distillation was proposed in (Huang & Wang 2017). Cho & Hariharan (2019) showed that very accurate networks are "too good" to be good teachers and proposed to mitigate this with early stopping of the teacher's training. Recently, the work of (Heo et al., 20T9a) studied the location within the network at which feature distillation should be applied and proposed margin ReLU and specifically designed distance function that transfers only the useful (positive) information from the teacher to the student. More recently, Li et al. (Li et al., 2020a) proposed to supervise the block- wise architecture search by the architecture knowledge distilled from teacher model. Another NAS based method was proposed in (Guan et al. (020). in which a student-to-teacher loss is used to find the aggregation weights that match the learning ability of the student. Passalis et al. (2020) claimed that traditional KD ignores information plasticity during the training process, and proposed to model the information flow through the various layers of the teacher.
Feature relationship transfer: Another line of knowledge distillation methods focus on exploring transferring the relationship between features, rather than the actual features themselves. In (Yim et al., 2017). feature correlations are captured by computing the Gram matrix of features across layers for both teacher and student and then applying an L2 loss on pairs of seacher-student Gram matrices. The limitation of this work is the high computational cost, which is addressed to some extent in (Lee et al. 2018) by compressing the feature maps by singular value decomposition. Park jet al.] (2019) proposed a relational knowledge distillation method which computes distance- wise and angle-wise relations of each embedded feature vector. This idea is further explored in (Peng let al. 2019) and (Liu et al., 2019). In (Peng al. 2019), Taylor series expansion is proposed to better capture the correlation between multiple instances. In (Liu et al.) 2019), the instance feature and relationships are considered vertexes and edges respectively in graph and instance rela- tionship graph is proposed to model the feature space transformation across layers. Inspired by the observation that semantically similar inputs should have similar activation patterns, (Tung & Morij uimilarity-preserving knowledge distillation method which guides the student to mimic the teacher with respect to generating similar or dissimilar activations. More recently, (Jain) et al. 2020) proposed to matching the student output with the teacher's ' by distilling the knowledge through a quantized visual words space. [Li et al. (2020b) proposed the local correlation exploration framework to represent the relationships of local regions in the feature space which contains more details and discriminative patterns.
2019) proposed
Finally, similar connection between distillation and representation learning was very recently made in (Tian et al.) 2020) which uses contrastive learning for knowledge distillation. We note that our loss is not related to the used in (Tian et al.) 2020), is simpler, and shown in Section 5 outperforms it for all of our experiments, often by a significant margin.
METHOD
We denote by T and S the teacher and student networks respectively. We split these networks into two parts: (i) A convolutional feature extractor fNet, Net = ST, the output of which at the i-th layer is feature tensor Fi E R RC X x W where Cives is the output feature dimensionality,
and H', Wi the output spatial dimensions. We also denote by hNet = E Iw4, Fi E CR the last layer feature representation learned by fNa. (ii) A projection matrix W Weee= E RK which projects the feature representation liee into K class logits =/eet& 1,-.., K, followed by exp(sjyu//T) the softmax function s(=/xet) Σ, exp(=kea/T) with temperature (r 1 for Cross Entropy loss) which put together form softmax regression classifier into K classes
3

---

Published as a conference paper at ICLR 2021
Knowledge Distillation (KD) (Hinton et al. 2015) trains the student with the following loss:
K
LKD
s( s(e)) log s(z))
(1)
so that the discrepancy between the teacher's and student's classifiers is directly minimized.
FitNets (Romero et al., 2015) match intermediate feature representations, For the i-th layer, the following loss is defined:
LFFt -
(2)
where r(.) is function for matching the feature tensor dimensions.
In our work, we propose to minimize the discrepancy between the representations ht and his To accomplish this goal, we propose two losses. The first one is an Lg feature matching loss:
LFM = | hsll
where for notational simplicity we dropped the dependency on r(.). Hence, LFM loss is a simplified FitNet loss which focuses only the final representation learned. The intuition for this is that this feature is directly connected to the classifier and hence imposing the student's feature to be similar to that of the teacher could have more impact classification accuracy. Moreover, it might be questionable why one should optimize for other intermediate representations as (Romero et al, 2015) especially when the student is network of lower representational capacity. In Section Where should the losses be applied?, we confirm that LrM alone has a positive impact but feature matching other layers not helpful.
We found LPM to be effective but only to limited extent. One disadvantage of LEM and, in general, of all feature matching losses e.g. (Romero et al. 2013, Zagoruyko & Komodakis 2017). that it treats each channel dimension in the feature space independently. and ignores the inter-channel dependencies of the feature representations hy and hr for the final classification. This is in contrast to the original logit matching loss proposed by Hinton et al. in (Hinton al.) 2015) which directly targets classification accuracy. To alleviate the aforementioned problem, in this work, we propose a second loss for optimizing hs which is directly linked with classification accuracy. To this end, will use the teacher's 's pre- pre-trained Softmax Regression (SR) classifier.
Let us denote by p the output of the teacher network when fed with some input image æ. Let us also feed the same image through the student network to obtain feature hs(x). Finally let us pass (z( through the teacher's SR classifier obtain output q. See also Fig. 1 Our loss is defined as:
Lsn -p log 4-
(4)
At this point, make the following two observations: If) q (and since the teacher's classifier is frozen), then this implies that hs(a) = lr((#) which shows that indeed Eq. (4) optimizes the student's feature representation hs (hv is also frozen). (2) The loss of Eq.44) can be written as:
LSR = ((W,rr) log 8 (Wths))
(5)
Now let us now write KD loss in a similar way:
LKD = -s(W, log s hWss).
(6)
that the only difference in method is that the frozen, pre- pre-trained teacher's s classifier is used for both teacher and the student. On the contrary, in KD, Ws also optimized. This gives more degrees of freedom the optimization algorithm. in particular, to adjust the weights of both the student's 's feature extractor fs and the student's 's classifier Ws in order to minimize the loss. This has an impact on the learning of the student's 's feature representation hş which, in turn, hinders the generalization capability of the student on the test set. We confirm this hypothesis with the experiment of Section 4 Transferability of representations.
By comparing Eq. (5) with Eq. 6
Finally, we note that we found that, in practice, an La loss between the logits:
LKD W Whhhsl I| = ||h hsllwr.
works slightly better than the cross-entropy loss. The comparison between different types of losses for Lsn is given in the appendix.
4

---

Published as a conference paper at ICLR 2021
Table 1: Effect of proposed losses (IeM and LSR) and position of distillation the test set of CIFAR-100.
Method
Layer
Top-I (%)
Top-5 (%)
Student (WRN-16-4)
76.97
93,89
Teacher (WRN-40-4)
79.50
94.57
LFM
conv4
78.05
94,45
LSR
conv4
79.10
94.99
LFM LSR
conv4
79.58
95.21
LFM +PM++LSR LSR
conv2
77.03
93,94
LFM+LSK
conv3
77.34
94.22
LFM+LSR
conv2+3+4
79.43
94.80
Overall, in our method, we train the student network using three losses:
L LOE + aLFM + L
where a and 3 are the weights used to scale the losses. The teacher network is pretrained and fixed during training the student. ICE is the standard loss based on ground truth labels for the task in hand (e.g. cross-entropy loss for image classification). Note that this results in a very simple algorithm for training the student, summarized Algorithm
Algorithm Knowledge distillation via Softmax Regression Representation Learning
Input: Teacher network T, Student network S, input image x, ground truth label y, weights a, S. Input x to S to obtain feature hs and class prediction ÿ. Calculate cross entropy loss
LeE H(ū. y):
2. Input x to T to obtain feature hy. Calculate distillation losses from Eqs. and
3. Update 5 by optimizing Eq. (
Output: the updated S.
ABLATION STUDIES
We conducted a set of ablation studies on CIFAR-100 (see Section (5T) using Wide ResNet (WRN) for both teacher (WRN-40-4) and student (WRN-16-4) (for network definitions, see Section 5)
Are both LFM and LSR useful? To answer this question, we ran 3 experiments: using LFM alone, LSR alone, and combining them together LFM LSR- The results of Table (first rows) clearly show that all proposed variants offer significant gains: when using LEas and LşR alone, 1% and ~ 2% improvements in Top-1 accuracy obtained. Moreover, when combining them together, an additional ~ 0.4% improvement was gained. Importantly, the results show that LsR is significantly
effective than LFM- We further note at this point that found that LFM offers diminishing gains on ImageNet experiments.
Where should the losses be applied? The proposed losses can be applied at other layers of the networks too. This is straightforward for LFM- We can also extend LSR to more layers, by transfer- ring the mean feature of the student each layer the corresponding layer of the teacher using an AdaIN layer (Huang & Belongie 2017). On one hand, applying the losses early in the network could ensure that the subsequent layers receive "better" features. On the other hand, features produced by early layers are not specialised to particular class. Thus, applying the distillation losses towards the end of the network, where the activations encode discriminative, task-related features should lead to potentially stronger models. The results from Table I (last 3 rows) confirm hypothesis: Applying the loss at multiple points the network actually rather hurts accuracy.
Teacher-studen similarity: The overall aim of knowledge distillation is to make the student mimic the teacher's 's output, so that the student is able to obtain similar performance to that of the teacher. Therefore, see how well the student mimics the teacher, measured the similarity between the teacher's 's and student's 's outputs using (a) the KL divergence between the teacher's 's and student's outputs, and (b) the cross-entropy loss between the student's predictions and the ground truth labels.
5

---

Published as a conference paper at ICLR 2021
Table 2: KI. divergence between teacher and student, and cross-entropy between student and ground truth on the test set of CIFAR-I00 100. Teacher's top-1 accuracy is 79.50%.
Method
KI div. with teacher Cross-entropy with label
Topp1 (%)
Student
0.5964
0,9383
76.97
KD
0.5818
0,9492
78.35
AT
0.5406
0,9049
78.06
LFM
0.5701
0.8980
78.05
LSR
0.4828
0.8418
79.10
LSM+LSR
0.4597
0.8247
79.58
Table 3: L2 Distance ||/t
hg hgll2. and NMI calculated on the test set of CIFAR-100.
Method
Student
LFM
LsR
LFM+LLR
1.2 Distance
1.48
1.33
1.07
1.01
NMI (%)
77.20
78.31
79.35
79.85
T%)- 76.97
78.05
79.10
79.58
From Table 2 it can be observed that KD (Hinton et al., 2015) reduces the KL divergence with the teacher's output offering 1.5% accuracy gain. AT (Zagoruyko & Komodakis 2017) also decreases the KL divergence with the teacher's output offering smaller accuracy gain of ~ 1.0%. Moreover, both proposed losses Lfas and LsR and their combination LFM+LSR show considerably high similarity compared to the KD and AT. This similarity is one of the main reasons for the improved student's accuracy offered by our method.
(a) Stadent
(b) LFM
(c) LyR
(d) LpM+LgR
(c) Teacher
Figure Visualization of hy and hr on the test set CIFAR-100. 100. Better viewed color.
Representations distance: Jsoos the L2 distance between the teacher and student represen- tations hT and hg. The results, presented in Table 3 clearly show that both LFM and LSR narrow the distance, with their combination being the closest to the teacher.
Normalized Mutual Information (NMT): Moreover, we calculated the NMI (Manning al. 2008 which is balanced metric that can be used to determine the quality of fcature clustering. The results, presented in Table 3 show that LFM + Lsn has the highest NMI score, meaning that the features are better clustered. Qualitative results are shown in Figure 2 which visualizes the features hş and able learn more discriminative features, which also
hr. It can be observed that LFM++Lg correlates with quantitative accuracy gains.
Transferability of representations: Following (Tian al., 2020), this section aims to compare the representational power of the learned student's ' representation hgs To this end, we trained the student on CIFAR 100, and then used it as a frozen feature extractor on top of which we train a linear classifier for 2 datasets: STL 10 Coates et al. (2011) and CIFAR 100. We compare the transfer ability of KD, CRD, LFM- LSR- and L The superiority of the proposed losses over KD on STL is evident. Importantly, Lsn largely outperforms KD which confirms analysis of Eqs. (5) and (6). The best results on STL are obtained by CRD. However, on CIFAR100, which the target distillation dataset our method outperforms CRD.
COMPARISON WITH STATE-OF-THEE ART
We thoroughly evaluated the effectiveness of our method across multiple (a) network architectures (ResNet (He et al. 2016), Wide ResNet (Zagoruyko & Komodakis 2016), MobileNetV2 (Sandler et al. 2018), MobileNet (Howard et al. 2017)) with different teacher-student capacities, (b) datasets

---

Published as a conference paper at ICLR 2021
Table 4: Transferability of representations from CIFARI00 S0L-110
10 and CIFAR100 by freezing f and training a linear classifier on top. Top (%) accuracy is provided.
Student
Dataset
KD
CRD
LFM
LSR
LFM+LSSR
WRNI644
STLIO
68.75
72.45
69.3
71.44
72.17
WRNI6-4
CIFARI00
78.28
78.46
77.95
79.03
79.34
MobileNerV2
STLIO
62.17
69.74
66.12
68.23
68.95
MobileNetV2
CIFARI00
69.17
70.68
70.66
71.00
71.63
Table 5: Top-1 accuracy (%) of various knowledge distillation methods on CIFAR-10.
Student(Paramms)
Teacher(Params)
Student
KD
OFD
RKD
Ours
Teacher
WRN-16-1 (0.18M)
WRN-16-2 (0.69M)
91.04
92.57
92.15
92.28
92.51
92.95
93.98
WRN-16-2 (0.69M)
WRN-40-2 (2.2M)
93.98
94.46
94.39
94.30
94.41
94.66
95.07
ResNet-8 (0.08M)
ResNet-26 (0.37M)
87.78
88.75
88.15
87.49
88.50
89.02
93.58
ResNet-14 (0. (0.17M)
ResNet-26 (0.37M)
91.59
92.57
92.11
92.51
92.36
92.70
93.58
ResNet-18 (0.7M)
ResNet-34 (1.4M)
93.35
93.74
93.52
93.80
92.95
93.92
94.11
WRN-16-1 (RN-16-1(0.18M)
ResNet-26 (0.37M)
91.04
92.42
91.32
92.47
92.08
92.94
93.58
(CIFARI0/100, ImageNet), and (c) domains (real-valued and binary networks). The training details for all experiments are provided in the appendix. We denote with ResNet-N a Residual Network with N convolutional layers (He et al. 2016). We denote with WRN-D-k: a WRN architecture with D layers and an expansion rate of (Zagoruyko Komodakis 2017).
For the above mentioned settings, we compare our method with KD (Hinton et al. 2015) and (Zagoruyko & Komodakis 2017), and the more recent methods of OFD (Heo et al. 201 9a), RKD (Park et 2019), CRD (Tian et al., 2020).
Overview of results: From our experiments, we conclude that our approach offers consistent gains across all of the above scenarios, outperforming all methods considered for all settings. Notably, our method is particularly effective for the most difficult datasets (i.e. CIFAR-100 and ImageNet).
5.1 CIFAR-10/100
For CIFAR-IO. Top-1 performance of our method is shown in Table We tested three cases repre- senting different network architectures for student and teacher networks: the first two experiments with ResNets. In the last experiment, teacher
with WRNs. The following three experiments and student have different network architectures. Overall, our method achieves the best results for all cases, with KD (Hinton et al. 2015) closely following.
For CIFAR-106 100 (Krizhevsky & Hinton 2009). we experimented with several student-teacher net- work pairs using different structures. Experiments are grouped in three sets. The first shows per- formance for different teacher and student capacities using WRNs: poor student good teacher (WRN-16-2; WRN-40-4), descent student good teacher (WRN-10-10 WRN-16-10);; good student good teacher WWRN-16-4; WRN-40-4). In the second set, we show that these results hold when using a different architecture, ResNet in this case. The final set is designed to show performance when teacher and student have different architectures (MobileNetV2, ResNet and WRN)
Top-1 performance of our method is shown in Table IT We observe that for almost all configura- tions, our method achieves consistent and significant accuracy gains over prior work. Furthermore, hard to tell which is the second best method as the remaining methods have their own ad- vantages for different configurations. For WRN experiments, OFD ranks second. For ResNet and mixed structure experiments, CRD ranks second. More comparisons with other methods and results obtained by combining our method with KD and AT are provided in the supplementary material. Further improvements could be obtained by combining our method with others but this requires a comprehensive investigation which goes beyond the scope of this paper.
7

---

Published as a conference paper at ICLR 2021
Table Top-1 accuracy (%) of various knowledge distillation methods on CIFAR-100.
Student (Params)
Teacher (Params)
Student
KD
OLD
RKD
CRD
Ours
Teacher
WRN-16-2 (0.70M)
WRN-40-4 (8.97M)
72.70
74.52
74.33
75.37
74.23
75.27
75.96
79.50
WRN-16-4 (2.77M)
WRN-40-4 (8.97M)
76.97
78.35
78.06
79.29
78.38
78.83
79.58
79.50
WRN-10-10 (7.49M)
WRN-16-10 (17.2M)
76.27
78.20
76.44
78.72
77.84
78.35
79.17
79.77
CesNct-10
ResNet-34 (1.39M))
68.42
69.18
68.49
68.94
68.70
70.24
69.91
72.05
R (0.75M)
ResNet-50 (1.99M)
71.07
73.41
71.90
72.79
70.93
73.23
73.47
73.31
ResNet-10 (4.95M)
ResNet-34 (21.33M)
75.01
77.35
76.87
77.35
77.46
77.37
77.90
78.44
WRN-16-2 (0.70M)
ResNet-34 (21.33M)
72.70
73.95
72.32
74.78
73.91
74.88
75.38
78.44
MobileNetV2 (2.37M)
ResNet-34 (21.33M)
68.42
69.36
68.60
69.45
68.75
71.36
71.58
78.44
MobileNerV2 (2.37M)
WRN-40-4 (8.97M)
68.42
69.15
68.95
70.08
68.19
71.46
71.82
79.50
Table Comparison with state-of-the-art on ImageNet.
Student (Params)
Teacher (Params)
Student
KD
AT
OFD
RKD
CRD
Ours
Teacher
Top-I
70.04
70.68
70.59
71.08
71.34
71.17 71.73
73.31
ResNet18 (11.69M)
ResNet34 (21.80M)
Top-5
89.48
90.16
89.73
90.07
90.37
90.13 90.60
91.42
Top-1
70.13
70.72
71.25
71.32
71.40
72.49
76.16
MobileNet (4.23M))
ResNet50 (25.56M)
Top-5
89.49
90.30
90.03
90.34
90.62
90.42
90.92
92.86
5.2 IMAGENET-IK
Our experiments include two pairs of networks which are popular settings for ImageNet (Rus-) sakovsky et al. 2015). The first is distillation from ResNet-34 to ResNet-18 and the second one distillation from ResNet-50 to MobileNet (Howard et al. 2017). Note that, following (Tian al.) 2020) ImageNet, for KD, set the weight for the KL loss to 0.9, the weight for cross-entropy loss to 0.5 which helps to obtain better accuracy.
Our results are presented in Table Again, observe that method achieves significant im- provements over all competing methods. Moreover, there is no method which is consistently second: for 34 to ResNet-18 experiment, RKD is the second best while for ResNet-50 to MobileNet, CRD is the second best. Notably, for the latter experiment, CRD reduces the gap between the teacher and the student by 1.27%, while our method narrows it by 2.36%. Overall, validate the scalability of our method, and show that, when applied to a large-scale dataset, we achieve an even more favourable performance compared against competing methods.
results ImageNet
Table 8: Real-to-binary distillation results on CIFAR-100: real-valued teacher ResNet-34 is used distill binary student.
distillation results on ImageNet- IK: a real-valued ResNet- -8 used distill binary student. OFD result might be suboptimal.
Dataset
Method
Binary
KD
AT
OFD
RKD
CRD
Ours
Real
CIFAR- 100
ResNet34
65.34
68.65
68.54
66.84
68.61
68.78
70.50
75.08
ImageNet-IK IK
ResNet18
56.70
57.39
58.45
55.74
58.84
58,25
59.57
70.20
5.3 BINARY NETWORKS DISTILLATION
Training highly accurate binary neural networks (i.e. the most extreme case of quantization) is very challenging task (Rastegari et al. 2016 Bulat & Tzimiropoulos) 2019), and to this end, knowledge distillation appears to a promising direction. In this section, we present results by applying distillation for the task of training binary student networks guided by real-valued teacher networks. The network architecture is kept the same for both the student and the teacher in this case: specifically used ResNet using the modifications described in (Bulat & Tzimiropoulos 2019).
Table & presents our results. Again, we observe that our method outperforms all methods consider- ably, showing that can effectively transfer knowledge between different domains. Note that it was not clear to us where to place the distillation position for OFD, so although we included our result for this method, emphasize that this result might be suboptimal.
8

---

Published as a conference paper at ICLR 2021
5.4 FACIAL LANDMARK DETECTION
Given face image, the task is to localise set of facial landmarks in terms of their (x.y) coordinates. This is often solved by using CNN to directly regress the (x.y) coordinates of the facial landmarks. In order to show the suitability of our method for this problem, we use the WFLW Wu et al. (2018) dataset, which one of the hardest benchmarks for this task. Performance is measured in terms of Normalised Mean Error (lower is hetter), which is the standard metric for the problem. In our experiment, we use a ResNet50 as the teacher and a ResNet8 as the student. The results, shown in Table 2 confirm the superior performance of our method when compared to other state-of-the-art methods.
Table 9: Facial landmark detection with ResNet50 as teacher and ResNet8 as student. KD is adapted by using an 1.2 loss instead of a KL loss to measure discrepancy between the teach and student predictions.
(tuden)( Teacher(Params)
Student
KD
RKD
PKT
IFM
AT
Ours
Teacher
ResNet8(7.25M) ResNet50126255M NME
7.43
7.32
6.94
7.09
7.14
6.96
6.81
6.38
CONCLUSION
We presented method for knowledge distillation that optimizes the output feature of the penulti- mate layer of the student network and hence is directly related to representation learning. A key our method is the newly proposed Softmax Regression Loss which was found necessary for effective representation learning. We showed that our method consistently outperforms other state- of-the-art distillation methods for a wide range of experimental settings including multiple network architectures (ResNet, Wide ResNet, MobileNet) with different teacher-student capacities, datasets (CIFAR 10/00, ImageNet), and domains (real-valued and binary networks).
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006.
Adrian Bulat and Georgios Tzimiropoulos. XNOR-Net++: Improved binary neural networks. In BMVC, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In International conference artificial intelligence and statistics, 2011.
Simon S Du and Jason D Lee. On the power of over-parametrizzaion in neural networks with quadratic activation. In ICML, 2018.
Yushuo Guan, Pengyu Zhao, Bingxuan Wang, Yuanxing Zhang, Cong Yao, Kaigui Bian, and Jian Tang. Differentiable feature aggregation search for knowledge distillation. In ECCV, 2020.
Song Han, Huizi Mao, and William J Dally, Deep compression: Compressing deep neural networks with pruning, trained guantization and Huffman coding. ICLR. 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.

---

Published as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019a.
Byeongho Heo, Minsik Lee. Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. wwXv::/503.02533 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.048b0 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv: arXiv:170..01219, 2017.
Himalaya Jain, Spyros Gidaris, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. QUEST:
Quantized embedding space for transferring knowledge. ECCV, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In NeurIPS, 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, 2009.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016.
Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. ECCV, 2018.
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In CVPR, 2020a.
Xiaojie Li, Jianlong Wu, Hongyu Fang, Yue Liao, Fei Wang, and Chen Qian. Local correlation consistency for knowledge distillation. In ECCV, 2020b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv, 2018
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to information retrieval (chapter 16). Cambridge university press, 2008
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. In CVPR, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
10

---

Published as a conference paper at ICLR 2021
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. In ECCV, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of pereparaamterized shallow neural networks. TIT, 2018
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Frederick Tung and Greg Mori. preseerity-preervingg knowledge distillation. In ICCV, 2019.
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, 2016.
Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look at boundary: A boundary-aware face alignment algorithm. In CVPR, 2018.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. ICLR, 2017.
A APPENDIX
A. STUDY OF THE HYPER-PARS PAPEMEEES a AND 3
We only performed very basic search find the best hyper-parameters. First, we fix a and search for the best 3. Then, we used the best 3, and search for the best a. This is sub-optimal compared to full grid search over a and B. Furthermore, after some preliminary experimentation, we considered only for 2 values: 1 and 5. Notably, we found that for all teacher-student pairs but (T:WRN40 4, S:MV2) alpha=1 the optimal value. Furthermore, on ImageNet, the optimal values were a 1, B for all seacher-student pairs considered.
A.2 DATASETS AND TRAINING DETAILS
CIFAR-10 CIFAR-10 is popular image classification dataset consisting of 50,000 training and 10,000 testing images equally distributed across 10 classes. All images are resolution 32 x 32px. Following (Zaoruykoo & Komodakis 2017), during training, we randomly cropped and horizontally flipped the images. The ResNet models were trained for 350 epochs using SGD. The initial learning rate set 0.1, and then it reduced by factor of 10 at epochs 150, 250 and 320. Similarly, the WRN models were trained for 200 epochs with a learning rate of 0.1 that was subsequently reduced by 5 at epochs 60, 120 and 160. In all experiments, we the dropout rate 0.

---

Published as a conference paper at ICLR 2021
For traditional KD Hinton et al. 2015), set 0.9 and T 4. For AT Zagoruyko & Komodakis 2017). . as in Zagoruyko & Komodakis 2017 Tung & Mori 2019). we set the weight of distillation loss to 1000. We note that, in experiments, the AT loss added after each layer group
for WRN and the last two groups for ResNet as in (Zagoruyko & Komodakis 2017). Following
OFD (Heo et al. 2019a), we the weight of distillation loss 10 For RKD (Park al. 2019),
we set 31 = 25 for distance, and B3 = 50 for angle, as described in (Park et al. 2019] Tian et al,, 2020). We did not compare with CRD (Tian et al., (0200 on CIFAR-10 because, in our experiments, we found that their parameter setting (used in their paper for CIFAR-100 100 and ImageNet- does
not obtain good performance on CIFAR-10.
CIFAR-100 For CIFAR- 110 (Krizhevsky & Hinton 2009). we used a standard data augmentation scheme Zagoruyko & Komodakis] 2017) including padding 4 pixels prior to random cropping and horizontal flipping. We used SGD with weight decay 5e-4 and momentum 0.9. Batch size was set 128. Learning rate was set to 0.1; then decayed by 0.1 at epochs 100, 150, until training reached 200 epochs (Heo et al. 2019a).
ImageNet-IK Images are cropped to 224 x 224 pixels for both training and evaluation. used SGD with Nesterov momentum 0.9, weight decay le 4, initial learning rate 0.2 which was then dropped by a factor of 10 every 30 epochs, training in total for 100 epochs (for CRD we trained with 10 more epochs suggested by the authors). Batch size set to 512. For simplicity and to enable a fair comparison, we used pretrained PyTorch models Paszke et al. (2017) 7) as teacher networks Heo et al. (2019a); TTia al.. (2020). For binary experiments, used Adam the optimizer with initial learning 0.002 which was then reduced by a factor of 10 every 30 epochs, training in total for 100 epochs.
A.3 ADDITIONAL ABLATION STUDIES
Different losses for Lsnt This part expands Section of our paper by evaluating different losses for LsR- The following loss functions are compared:
1. L2 loss: LSR-L2(P. 1p al1 This is the loss used in Section of our paper
2. Cross Entropy loss (CE) with label y: LSR-CE y) H(g. y).
3. KL loss with temperature Hinton et 20155: L9R-kL(P.0 KL(g/T.p/T)
The results, presented in Table 10, show that all loss functions offer significant improvement gains while LPM+LSR-BE L2 achieves the best accuracy. Therefore, in our paper, LSR- L2 is used in all cases.
Table 10: Evaluation of different loss functions for LsR in terms of Top-1 accuracy on CIFAR-100.
Method
Top-
Top-5(%)
Student: WRN-16-4
76.97
93.89
Teacher: WRN-40-4
79.50
94.57
LFM +
79.58
95,21
LFM LSR-CE
78.80
95.13
LFM +
79.04
95.12
Combining our method with KD and AT: Table II shows additional comparisons on CIFARIOS by combining our method with Zagoruo & Komodakis (2017) and KD|Hinton et al. (2015). spectively. The results show that a straightforward combination did not provide satisfactory results, however it could be possible that more comprehensive investigation might prove to be beneficial.
A.4 ADDITIONAL COMPARISONS
This section provides additional comparisons using the evaluation framework of CRD Tian et al. (002)). Comparisons include distillation between models with the same architecture (c.g. ResNet56 ResNet20) and between different architectures (e.g. ResNet50 to MobileNerV2). In order to
12

---

Published as a conference paper at ICLR 2021
Table 11: Top-1 accuracy (%) combining our method with KD and AT on CIFAR-100.
Student (Params)
Teacher (Params)
Student
KD
A1
KI+OOurs
AT+Ours
Ours
Teacher
WRN-16-2 (0.70M)
W8N-40-4 (8.97M)
72.70
74.52
74.33
74.97
75.01
75.96
79.50
WRN-16-4 (2.77M)
WRN-40-4 (8.97M)
76.97
78.35
78.06
79.00
79.09
79.58
79.50
WRN-10-10 (7.49M)
(17.
76.27
78.20
76.44
78.84
77.79
79.17
79.77
ResNet-10 (0.344)))
ResNet-34 (1.39M)
68.42
69.18
68.49
70.41
69.41
69.91
72.05
ResNet-18 (0.75M)
ResNet-50 (1.99M)
71.07
73.41
71.90
73.46
73.17
73.47
72.83
ResNet-10 (4.995M) (4.95M)
ResNet-34 (21.33M)
75.01
77.35
76.87
77.64
77.48
77.90
78.44
WRN-16-2 (0.70M)
ResNet- (21.33M)
72.70
73.95
72.32
74.90
74.71
75.38
78, 78.44
MohileNetV2 (2.37M)
ResNet-34(21.33M
68.42
69.36
68.60
71.08
70.70
71.58
78.44
MobileNetV2 (2.37M)
WRN-40-4 (8.97M)
68.42
69.15
68.95
70.85
70.63
71.82
79.50
maximize the fairness of the comparison, we followed their experimental setting. Thus, we did choose the training parameters, teacher-student architecture pairs or methods to compare against. The competing methods included are:
Classic: Knowledge Distillation (KD) Hinton
(0015). FitNet Romero al. 2015 Attention Transfer (AT) Zagoruyko & Komodakis (2017).
Most recent:
Similarity Periitt-i-reeeiiggggge KD (SP) (Tung & Mori 2019), Correlation Congruence
(CC) (Peng al., 2019). Variational Information Distillation (VID) (Ahn al. 2019),
Relational Knowledge Distillation (RKD) (Park et al.] 2019), Distillation of Activation
Boundaries (AB) (Heo al.] (019b), Factor Transfer (FT) (Kim et al. 2018). Flow of
Solution (FSP) (Yim et al. 2017) and Contrastive Representation Distillation (CRD)
Tan
ct 2020
Table 12: Distillation experiment with the same architectures (Tian al. 2020): Top- accuracy (%) on CIFAR-100. The student models were trained with teacher of the same architecture. We report average over rums as (Tian et al. 2020).
wrn-40-2
wrn-40-2
resnet56
resnet 10
resnet 10
resnet32x4 resnet8x4
vgg
wrn-16-2
wrn-40-1
resnet20
resnet20
resnet32
vgg8
Teacher 75.61
75.61
72.34
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
69.06
71.14
72.50
70.36
KD
74.92
73.54
70.66
70.67
73.08
73.33
72.98
FitNet
73.58
72.24
69.21
68.99
71.06
73.50
71.02
AT
74.08
72.77
70.55
70.22
72.31
73.44
71.43
SP
73.83
72.43
69.67
70.04
72.69
72.94
72.68
CC
73.56
72.21
69.63
69.48
71.48
72.97
70.71
VID
74.11
73.30
70.38
70.16
72.61
73.09
71.23
RKD
73.35
72.22
69.61
69.25
71.82
71.90
71,48
PKT
74.54
73.45
70.34
70.25
72.61
73.64
72.88
AB
72.50
72.38
69.47
69.53
70.98
73.17
70.94
FT
73.25
71.59
69.84
70.22
72.37
72.86
70.58
FSP
72.91
0.00
69.95
70.11
71.89
72.62
70.23
NST
73.68
72.24
69.60
69.53
71.96
73.30
71.53
CRD
75.48
74.14
71.16
71.46
73.48
75.51
73.94
Ours
75.96
74.75
71.44
71.51
73.80
75.92
74.40
13

---

Published as a conference paper at ICLR 2021
Table 13:
Distillation experiment with different architectures (Tian et al. 2020): Top- accuracy
(%) on CIFAR-100. The student models were trained with a teacher different architecture. We report average over 3 runs as in (Tian et al.) 2020).
vgg13
ResNet50 MobileNetV2
ResNet50
resmet32x4
resnet32x4
wrn-40-2
MobileNetV2
vgg8
ShuffleNetVi
ShuffleNetV2
ShuffleNetVi
Teacher
74.64
79.34
79.34
79.42
79.42
75.61
Student
64.60
64.60
70.36
70.50
71.82
70.50
KD
67.37
67.35
73.81
74.07
74.45
74.83
FitNet
64.14
63.16
70.69
73.59
73.54
73.73
AT
59.40
58.58
71.84
71.73
72.73
73.32
66.30
68.08
73.34
73.48
74.56
74.52
CC
64.86
65.43
70.25
71.14
71.29
71.38
VID
65.56
67.57
70.30
73.38
73.40
73.61
RKD
64.52
64.43
71.50
72.28
73.21
72.21
PKT
67.13
66.52
73.01
74.10
74.69
73,89
AB
66.06
67.20
70.65
73.55
74.31
73.34
FT
61.78
60.99
70.29
71.75
72.50
72.03
NST
58.16
64.96
71.28
74.12
74.68
74.89
CRD
69.73
69.11
74.30
75.11
75.65
76.05
Ours
69.14
69.45
74.46
75.66
76.40
76.61
14


Reviews:
[{"review_id": "hcGNUdl-0Jj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####################################################################################\nSummary:\n\nThis paper proposes a new formulation of knowledge distillation (KD) for model compression. Different from the classic formulation that matches the logits between student and teacher models, this paper suggests to match the output features of the penultimate layers between student and teacher models, based on L2 distance. Two complementary variants are introduced, with one directly minimizing the distance of the original feature vectors and with the other minimizing the distance of the feature vectors projected on the teacher classifiers. The approach is evaluated in a variety of scenarios, such as different network architectures, teacher-student capacities, datasets, and domains, and compared with state-of-the-art results.\n\n####################################################################################\nPros:\n\nThe proposed knowledge distillation formulation is simple. The paper is well written. Experimental evaluations clearly demonstrate the effect by directly matching the output features of the penultimate layers.\n\n####################################################################################\nCons: \n\n- While the proposed approach is interesting, its novelty seems limited, with formulations being special cases of existing methods. In particular, as the authors also mentioned, the proposed L_FM loss is a simplified FitNet loss in Romero et al., which focuses only on matching the final representation without considering the intermediate representations. The proposed L_SR loss is similar to the standard KD formulation in Hinton et al., with the only difference that the pre-trained teacher’s classifier is used for both teacher and student models.\n\n- From the result tables, the performance improvement of the proposed approach is marginal, which is on par with existing works.\n\n- As the authors mentioned, the L_SR loss is inspired by that only using the L_FM loss ignores the inter-channel dependencies of the feature representations h^S and h^T. So L_SR-CE is introduced. However, empirically, L_SR-CE is worse than L_SR-L2 which directly minimizes the distance between projected features. It seems that the empirical formulation is inconsistent with the motivation.\n\n- In Eq 8, I assume that a separate classifier is also trained for the student model using the L_CE loss. How is the performance if we completely remove the student classifier? That is, the student model only consists of a feature extractor, and during inference time, we directly insert the pre-trained teacher classifier on top of the student model.\n\n- It would be interesting to show the results when h^T and h^S have different dimensionality.\n\n- In the ablation studies, the authors investigated “where should be losses be applied”. How is the classifier generated for the intermediate layers?\n\n- How does the performance change with respect to different settings of the hyper-parameters alpha and beta?\n\n- The notation is inconsistent throughout the paper. For example, h^T and h^S are used in the first half of the method section, while h_T and h_S are used later.\n\n#################################################################################### Updated:\n\nThe authors' rebuttal addressed my concerns and I lean toward acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dmw6KydW9KI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel method for knowledge distillation. The idea is to utilize the teacher’s pre-trained classifier to train the student’s penultimate layer feature by adopting two losses: (a) the Feature Matching loss LFM and (b) the Softmax Regression loss LSR. The latter is designed to take into account the classification task at hand, which imposes that for the same input image, the teacher’s and student’s feature produce the same output when passed through the teacher’s pre-trained and frozen classifier.\n\nThe whole idea is easy and clear. The authors also provided necessary experiments to rationalize the approach, for example, why the penultimate layer? why student classifier not included? The final model is thoroughly compared with state-of-the-arts knowledge distillation methods. Despite in several tasks, the proposed model fails to succeed the CRD method, given the simplicity of the proposed method and its superiority over most of the tasks, I think the results are satisfactory.\n\nStill, I would like to give a suggestion:\n1.\tAfter CRD, proposed in 2019, there appears many new methods with refreshing performance. For example, [1] Xu G , Liu Z , Li X , et al. Knowledge Distillation Meets Self-Supervision[C]. in ECCV 2020. It would be more persuasive if comparisons with more recent methods could be provided.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hb67yJYl50", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to leverage the frozen classifier from the teacher model for training better representations for the student model. By further combining with another feature matching loss, the proposed methods outperforms previous methods (such as KD, AT, CRD) on many benchmarks.\n\nPros:\n- The proposed method is very simple, and effective.\n- Authors demonstrate the effectiveness of this method, on CIFAR-10, CIFAR-100, ImageNet, as well as transferring to STL-10. They also apply real-to-binary distillation, which is interesting.\n- The paper is well-written and very easy to follow.\n\nOverall, I did not see much disadvantages of this proposed method. I am a bit surprised that such simple method has been ignored before. But some parts that can be further enhanced is:\n- Limitation: despite its simplicity, this method, in its current format, assumes that the teacher classifier is available, which is not always true in may applications. Besides, it's not clear that whether this method is still effective, if the teacher model is not a standard classification model, e.g., if you want to compress a regression model, whether you can still use teacher's frozen regressor to guide student representation is still unknown. So I encourage the authors to add a section to discuss it.\n\nOverall, I like this simple and surprisingly effective method.\n\n=== update ===\n\nThe provided response is reasonable and helps address some of my concerns. I would keep my rating as acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-x0bdLSSSZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\nThis paper proposes a new knowledge distillation method by enhancing the student network's representation learning. The proposed framework is very simple: use the teacher's final fully-connected layer (or, 'projection matrix $W$' in the paper) to obtain both the teacher's and the student's logit output and optimize the student network by minimizing the gap between two logits. Experiments cover various network architectures on CIFAR and ImageNet datasets and show improvements over the other competitive knowledge distillation methods. \n\n**Strengths**\n1. The main idea to share the teacher's projection matrix $W$ for both the student and teacher's penultimate features seems new. \n2. Proposed method is simple and easy to implement. \n3. Experiments include a wide range of network architectures and datasets. \n\n**Weaknesses**\n1. Definition of representation learning is unclear. \nThe paper claims that using the pre-trained and frozen projection matrix brings decoupling of \"representation learning\" and \"classification\". Also, the paper says \"Because direct feature matching might be difficult due to the lower representation capacity of the student ...\" on page 2. \nAfter reading this section, I got the following questions: 1) According to the paper's solution, the teacher's projection matrix is used to solve the lack of representation capacity of the student. Does the \"representation capacity\" of a network only exist in the final fully-connected layer (projection matrix)? If true, references or pieces of evidence for this should be provided. 2) Why the method of utilizing the teacher's projection matrix is called representation learning? \n\n2. On page 4, by comparing eq (5) and eq (6), the paper claims that the KD loss $L_{KD}$ might have worse generalization ability than $L_{SR}$ since the KD loss simultaneously updates the feature extractor and classifier. The paper only shows empirical results where $L_{SR}$ works better than $L_{KD}$. This argument is not easily accepted for me, so more theoretical backup is necessary. \n\n3. Study on the hyper-parameter $\\alpha$ and $\\beta$ on eq (8) is missing\n\n4. The main idea (\"softmax representation learning\", or $L_{SR}$) is very simple and seems to be easily applied with other KD methods beyond FitNets method. Combining $L_{SR}$ with other KD methods (e.g, AT, OFD, etc) will verify the generalization and applicability of the proposed method. \n\n\n**Post-comments to the author's response**\n- Replies for Q3 and Q4 are good.\n- However, the responses for Q1 and Q2 did not address my concern well. \n- For Q1, the authors claim that the representation capacity is in the CNN features not in the fully-connected layer. Then what is the meaning of utilizing the teacher's projection matrix for better representation learning? \n- Overall, I still this paper has more strengths than weaknesses, so I will keep my rating (6)\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new idea for performing knowledge distillation by leveraging teacher’s classifier to train student’s penultimate layer feature via proposing suitable loss functions. Reviewers appreciate the simultaneous simplicity and effectiveness of the method. A comprehensive set of studies are performed to empirically show the effectiveness of the method. Specifically, the proposed distillation method is shown to outperform state-of-the-art across various network architectures, teacher-student capacities, datasets, and domains. The paper is well-written and is easy to follow. All reviewers rate the paper on the accept side (after the rebuttal) and believe the new perspective this work provides on distillation and its simplicity to implement can lead it to gain high impact. I concur with the reviewers and find this submission a convincing empirical work, and thus recommend for accept.

Decision: Accept (Poster)