Year: 2021

Abstract:
This paper addresses the problem of model compression via knowledge distillation. We advocate for a method that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning. To this end, we firstly propose a direct feature matching approach which focuses on optimizing the student’s penultimate layer only. Secondly and more importantly, because feature matching does not take into account the classification problem at hand, we propose a second approach that decouples representation learning and classification and utilizes the teacher’s pre-trained classifier to train the student’s penultimate layer feature. In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classifier, which is achieved with a simple L2 loss. Our method is extremely simple to implement and straightforward to train and is shown to consistently outperform previous state-of-the-art methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains. The code is available at https://github.com/jingyang2017/KD_SRRL.

heading: 1 INTRODUCTION
text: Recently, there has been a great amount of research effort to make Convolutional Neural Networks (CNNs) lightweight so that they can be deployed in devices with limited resources. To this end, several approaches for model compression have been proposed, including network pruning (Han et al., 2016; Lebedev & Lempitsky, 2016), network quantization (Rastegari et al., 2016; Wu et al., 2016), knowledge transfer/distillation (Hinton et al., 2015; Zagoruyko & Komodakis, 2017), and neural architecture search (Zoph & Le, 2017; Liu et al., 2018). Knowledge distillation (Buciluǎ et al., 2006; Hinton et al., 2015) aims to transfer knowledge from one network (the so-called “teacher”) to another (the so-called “student”). Typically, the teacher is a high-capacity model capable of achieving high accuracy, while the student is a compact model with much fewer parameters, thus also requiring much less computation. The goal of knowledge distillation is to use the teacher to improve the training of the student and push its accuracy closer to that of the teacher.
The rationale behind knowledge distillation can be explained from an optimization perspective: there is evidence that high capacity models (i.e. the teacher) can find good local minima due to over-parameterization (Du & Lee, 2018; Soltanolkotabi et al., 2018). In knowledge distillation, such models are used to facilitate the optimization of lower capacity models (i.e. the student) during training. For example, in the seminal work of (Hinton et al., 2015), the softmax outputs of the teacher provide extra supervisory signals of inter-class similarities which facilitate the training of
the student. In other influential works, intermediate representations extracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to define auxiliary loss functions used in the optimization of the student.
Training a network whose output feature representation is rich and powerful has been shown crucial for achieving high accuracy for the subsequent classification task in recent works in both unsupervised and supervised learning, see for example (Chen et al., 2020; He et al., 2020) and (Kang et al., 2020). Hence, in this paper, we are advocating for representation learning-based knowledge distillation by optimizing the student’s penultimate layer output feature. If we are able to do this effectively, we expect (and show experimentally) to end up with a student network which can generalize better than one trained with logit matching as in the KD paper of (Hinton et al., 2015).
Main contributions: To accomplish the aforementioned goal we propose two loss functions: The first loss function, akin to (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is based on direct feature matching but focuses on optimizing the student’s penultimate layer feature only. Because direct feature matching might be difficult due to the lower representation capacity of the student and, more importantly, is detached from the classification task at hand, we also propose a second loss function: we propose to decouple representation learning and classification and utilize the teacher’s pre-trained classifier to train the student’s penultimate layer feature. In particular, for the same input image, we wish the teacher’s and student’s feature to produce the same output when passed through the teacher’s classifier, which is achieved with a simple L2 loss (see Fig. 1). This softmax regression projection is used to retain from the student’s feature the information that is relevant to classification, but since the projection matrix is pre-trained (learned during the teacher’s training phase) this does not compromise the representational power of the student’s feature.
Main results: Our method has two advantages: (1) It is simple and straightforward to implement. (2) It consistently outperforms state-of-the-art methods over a large set of experimental settings including different (a) network architectures (WideResNets, ResNets, MobileNets), (b) teacherstudent capacities, (c) datasets (CIFAR-10/100, ImageNet), and (d) domains (real-to-binary).

heading: 2 RELATED WORK
text: Knowledge transfer: In the work of (Hinton et al., 2015), knowledge is defined as the teacher’s outputs after the final softmax layer. The softmax outputs carry richer information than one-hot labels because they provide extra supervision signals in terms of the inter-class similarities learned
by the teacher. In a similar fashion to (Hinton et al., 2015), intermediate representations extracted from the teacher such as feature tensors (Romero et al., 2015) or attention maps (Zagoruyko & Komodakis, 2017) have been used to define loss functions used to facilitate the optimization of the student. Trying to match the whole feature tensor, as in FitNets (Romero et al., 2015), is hard and, in certain circumstances, such an approach may adversely affect the performance and convergence of the student. To relax the assumption of FitNet, Attention Transfer (AT) was proposed in (Zagoruyko & Komodakis, 2017) where knowledge takes the form of attention maps which are summaries of the energies of the feature tensors over the channel dimension. An extension of (Zagoruyko & Komodakis, 2017) using Maximum Mean Discrepancy of the network activations as a loss term for distillation was proposed in (Huang & Wang, 2017). Cho & Hariharan (2019) showed that very accurate networks are “too good” to be good teachers and proposed to mitigate this with early stopping of the teacher’s training. Recently, the work of (Heo et al., 2019a) studied the location within the network at which feature distillation should be applied and proposed margin ReLU and a specifically designed distance function that transfers only the useful (positive) information from the teacher to the student. More recently, Li et al. (Li et al., 2020a) proposed to supervise the blockwise architecture search by the architecture knowledge distilled from a teacher model. Another NAS based method was proposed in (Guan et al., 2020), in which a student-to-teacher loss is used to find the aggregation weights that match the learning ability of the student. Passalis et al. (2020) claimed that traditional KD ignores information plasticity during the training process, and proposed to model the information flow through the various layers of the teacher.
Feature relationship transfer: Another line of knowledge distillation methods focus on exploring transferring the relationship between features, rather than the actual features themselves. In (Yim et al., 2017), feature correlations are captured by computing the Gram matrix of features across layers for both teacher and student and then applying an L2 loss on pairs of teacher-student Gram matrices. The limitation of this work is the high computational cost, which is addressed to some extent in (Lee et al., 2018) by compressing the feature maps by singular value decomposition. Park et al. (2019) proposed a relational knowledge distillation method which computes distance-wise and angle-wise relations of each embedded feature vector. This idea is further explored in (Peng et al., 2019) and (Liu et al., 2019). In (Peng et al., 2019), Taylor series expansion is proposed to better capture the correlation between multiple instances. In (Liu et al., 2019), the instance feature and relationships are considered as vertexes and edges respectively in a graph and instance relationship graph is proposed to model the feature space transformation across layers. Inspired by the observation that semantically similar inputs should have similar activation patterns, (Tung & Mori, 2019) proposed a similarity-preserving knowledge distillation method which guides the student to mimic the teacher with respect to generating similar or dissimilar activations. More recently, (Jain et al., 2020) proposed to matching the student output with the teacher’s by distilling the knowledge through a quantized visual words space. Li et al. (2020b) proposed the local correlation exploration framework to represent the relationships of local regions in the feature space which contains more details and discriminative patterns.
Finally, a similar connection between distillation and representation learning was very recently made in (Tian et al., 2020) which uses contrastive learning for knowledge distillation. We note that our loss is not related to the one used in (Tian et al., 2020), is simpler, and as shown in Section 5, outperforms it for all of our experiments, often by a significant margin.

heading: 3 METHOD
text: We denote by T and S the teacher and student networks respectively. We split these networks into two parts: (i) A convolutional feature extractor fNet, Net = {T, S}, the output of which at the i-th layer is a feature tensor F iNet ∈ RC i Net×H i×W i , where CiNet is the output feature dimensionality,
and Hi,W i the output spatial dimensions. We also denote by hNet = ∑HL h=1 ∑WL w=1 F L Net ∈ RC L Net the last layer feature representation learned by fNet. (ii) A projection matrix WNet ∈ RC L Net×K which projects the feature representation hNet into K class logits ziNet, i = 1, . . . ,K, followed by the softmax function s(ziNet) = exp(ziNet/τ)∑ j exp(z j Net/τ) with temperature τ (τ = 1 for Cross Entropy loss) which put together form a softmax regression classifier into K classes.
Knowledge Distillation (KD) (Hinton et al., 2015) trains the student with the following loss:
LKD = − K∑ k=1 s(zkT ) log s(z k S), (1)
so that the discrepancy between the teacher’s and student’s classifiers is directly minimized.
FitNets (Romero et al., 2015) match intermediate feature representations. For the i-th layer, the following loss is defined:
LFit = ∥∥F iT − r(F iS)∥∥2 , (2)
where r(.) is a function for matching the feature tensor dimensions.
In our work, we propose to minimize the discrepancy between the representations hT and hS . To accomplish this goal, we propose to use two losses. The first one is an L2 feature matching loss:
LFM = ‖hT − hS‖2 , (3) where for notational simplicity we dropped the dependency on r(.). Hence, LFM loss is a simplified FitNet loss which focuses only on the final representation learned. The intuition for this is that this feature is directly connected to the classifier and hence imposing the student’s feature to be similar to that of the teacher could have more impact on classification accuracy. Moreover, it might be questionable why one should optimize for other intermediate representations as in (Romero et al., 2015) especially when the student is a network of lower representational capacity. In Section 4: Where should the losses be applied?, we confirm that LFM alone has a positive impact but feature matching in other layers is not helpful.
We found LFM to be effective but only to limited extent. One disadvantage of LFM and, in general, of all feature matching losses e.g. (Romero et al., 2015; Zagoruyko & Komodakis, 2017), is that it treats each channel dimension in the feature space independently, and ignores the inter-channel dependencies of the feature representations hS and hT for the final classification. This is in contrast to the original logit matching loss proposed by Hinton et al. in (Hinton et al., 2015) which directly targets classification accuracy. To alleviate the aforementioned problem, in this work, we propose a second loss for optimizing hS which is directly linked with classification accuracy. To this end, we will use the teacher’s pre-trained Softmax Regression (SR) classifier.
Let us denote by p the output of the teacher network when fed with some input image x. Let us also feed the same image through the student network to obtain feature hS(x). Finally let us pass hS(x) through the teacher’s SR classifier to obtain output q. See also Fig. 1. Our loss is defined as:
LSR = −p log q. (4) At this point, we make the following two observations: (1) If p = q (and since the teacher’s classifier is frozen), then this implies that hS(x) = hT (x) which shows that indeed Eq. (4) optimizes the student’s feature representation hS (hT is also frozen). (2) The loss of Eq.(4) can be written as:
LSR = −s(W ′ThT ) log s(W ′ThS). (5) Now let us now write KD loss in a similar way:
LKD = −s(W ′ThT ) log s(W ′shS). (6) By comparing Eq. (5) with Eq. (6), we see that the only difference in our method is that the frozen, pre-trained teacher’s classifier is used for both teacher and the student. On the contrary, in KD, WS is also optimized. This gives more degrees of freedom to the optimization algorithm, in particular, to adjust the weights of both the student’s feature extractor fS and the student’s classifier WS in order to minimize the loss. This has an impact on the learning of the student’s feature representation hS which, in turn, hinders the generalization capability of the student on the test set. We confirm this hypothesis with the experiment of Section 4: Transferability of representations.
Finally, we note that we found that, in practice, an L2 loss between the logits:
LKD = ‖W ′ThT −W ′ThS‖ 2 = ‖hT − hS‖2WT , (7)
works slightly better than the cross-entropy loss. The comparison between different types of losses for LSR is given in the appendix.
Overall, in our method, we train the student network using three losses:
L = LCE + αLFM + βLSR, (8)
where α and β are the weights used to scale the losses. The teacher network is pretrained and fixed during training the student. LCE is the standard loss based on ground truth labels for the task in hand (e.g. cross-entropy loss for image classification). Note that this results in a very simple algorithm for training the student, summarized in Algorithm 1.
Algorithm 1 Knowledge distillation via Softmax Regression Representation Learning Input: Teacher network T , Student network S, input image x, ground truth label y, weights α, β.
1. Input x to S to obtain feature hS and class prediction ŷ. Calculate cross entropy loss LCE = H(ŷ, y); 2. Input x to T to obtain feature hT . Calculate distillation losses from Eqs. (3) and (7); 3. Update S by optimizing Eq. (8)
Output: the updated S.

heading: 4 ABLATION STUDIES
text: We conducted a set of ablation studies on CIFAR-100 (see Section 5.1) using a Wide ResNet (WRN) for both teacher (WRN-40-4) and student (WRN-16-4) (for network definitions, see Section 5).
Are both LFM and LSR useful? To answer this question, we ran 3 experiments: using LFM alone, LSR alone, and combining them together LFM + LSR. The results of Table 1 (first 3 rows) clearly show that all proposed variants offer significant gains: when using LFM and LSR alone, ∼ 1% and ∼ 2% improvements in Top-1 accuracy were obtained. Moreover, when combining them together, an additional ∼ 0.4% improvement was gained. Importantly, the results show that LSR is significantly more effective than LFM . We further note at this point that we found that LFM offers diminishing gains on ImageNet experiments.
Where should the losses be applied? The proposed losses can be applied at other layers of the networks too. This is straightforward for LFM . We can also extend LSR to more layers, by transferring the mean feature of the student at each layer to the corresponding layer of the teacher using an AdaIN layer (Huang & Belongie, 2017). On one hand, applying the losses early in the network could ensure that the subsequent layers receive “better” features. On the other hand, features produced by early layers are not specialised to a particular class. Thus, applying the distillation losses towards the end of the network, where the activations encode discriminative, task-related features should lead to potentially stronger models. The results from Table 1 (last 3 rows) confirm our hypothesis: Applying the loss at multiple points in the network actually rather hurts accuracy.
Teacher-student similarity: The overall aim of knowledge distillation is to make the student mimic the teacher’s output, so that the student is able to obtain similar performance to that of the teacher. Therefore, to see how well the student mimics the teacher, we measured the similarity between the teacher’s and student’s outputs using (a) the KL divergence between the teacher’s and student’s outputs, and (b) the cross-entropy loss between the student’s predictions and the ground truth labels.
From Table 2, it can be observed that KD (Hinton et al., 2015) reduces the KL divergence with the teacher’s output offering∼ 1.5% accuracy gain. AT (Zagoruyko & Komodakis, 2017) also decreases the KL divergence with the teacher’s output offering a smaller accuracy gain of ∼ 1.0%. Moreover, both proposed losses LFM and LSR and their combination LFM+LSR show considerably high similarity compared to the KD and AT. This similarity is one of the main reasons for the improved student’s accuracy offered by our method.
Representations distance: Table 3 shows the L2 distance between the teacher and student representations hT and hS . The results, presented in Table 3, clearly show that both LFM and LSR narrow the distance, with their combination being the closest to the teacher.
Normalized Mutual Information (NMI): Moreover, we calculated the NMI (Manning et al., 2008) which is a balanced metric that can be used to determine the quality of feature clustering. The results, presented in Table 3, show that LFM + LSR has the highest NMI score, meaning that the features are better clustered. Qualitative results are shown in Figure 2, which visualizes the features hS and hT . It can be observed that LFM+LSR is able to learn more discriminative features, which also correlates with quantitative accuracy gains.
Transferability of representations: Following (Tian et al., 2020), this section aims to compare the representational power of the learned student’s representation hS . To this end, we trained the student on CIFAR100, and then used it as a frozen feature extractor on top of which we train a linear classifier for 2 datasets: STL10 Coates et al. (2011) and CIFAR100. We compare the transfer ability of KD, CRD, LFM , LSR, and LFM+LSR. The superiority of the proposed losses over KD on STL is evident. Importantly, LSR largely outperforms KD which confirms our analysis of Eqs. (5) and (6). The best results on STL are obtained by CRD. However, on CIFAR100, which is the target distillation dataset our method outperforms CRD.

heading: 5 COMPARISON WITH STATE-OF-THE-ART
text: We thoroughly evaluated the effectiveness of our method across multiple (a) network architectures (ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016), MobileNetV2 (Sandler et al., 2018), MobileNet (Howard et al., 2017)) with different teacher-student capacities, (b) datasets
(CIFAR10/100, ImageNet), and (c) domains (real-valued and binary networks). The training details for all experiments are provided in the appendix. We denote with ResNet-N a Residual Network with N convolutional layers (He et al., 2016). We denote with WRN-D-k a WRN architecture with D layers and an expansion rate of k (Zagoruyko & Komodakis, 2017).
For the above mentioned settings, we compare our method with KD (Hinton et al., 2015) and AT (Zagoruyko & Komodakis, 2017), and the more recent methods of OFD (Heo et al., 2019a), RKD (Park et al., 2019), CRD (Tian et al., 2020).
Overview of results: From our experiments, we conclude that our approach offers consistent gains across all of the above scenarios, outperforming all methods considered for all settings. Notably, our method is particularly effective for the most difficult datasets (i.e. CIFAR-100 and ImageNet).

heading: 5.1 CIFAR-10/100
text: For CIFAR-10, Top-1 performance of our method is shown in Table 5. We tested three cases representing different network architectures for student and teacher networks: the first two experiments are with WRNs. The following three experiments are with ResNets. In the last experiment, teacher and student have different network architectures. Overall, our method achieves the best results for all cases, with KD (Hinton et al., 2015) closely following.
For CIFAR-100 (Krizhevsky & Hinton, 2009), we experimented with several student-teacher network pairs using different structures. Experiments are grouped in three sets. The first shows performance for different teacher and student capacities using WRNs: poor student - good teacher (WRN-16-2; WRN-40-4), descent student - good teacher (WRN-10-10; WRN-16-10); good student - good teacher (WRN-16-4; WRN-40-4). In the second set, we show that these results hold when using a different architecture, ResNet in this case. The final set is designed to show performance when teacher and student have different architectures (MobileNetV2, ResNet and WRN).
Top-1 performance of our method is shown in Table 11. We observe that for almost all configurations, our method achieves consistent and significant accuracy gains over prior work. Furthermore, it is hard to tell which is the second best method as the remaining methods have their own advantages for different configurations. For WRN experiments, OFD ranks second. For ResNet and mixed structure experiments, CRD ranks second. More comparisons with other methods and results obtained by combining our method with KD and AT are provided in the supplementary material. Further improvements could be obtained by combining our method with others but this requires a comprehensive investigation which goes beyond the scope of this paper.

heading: 5.2 IMAGENET-1K
text: Our experiments include two pairs of networks which are popular settings for ImageNet (Russakovsky et al., 2015). The first is distillation from ResNet-34 to ResNet-18 and the second one is distillation from ResNet-50 to MobileNet (Howard et al., 2017). Note that, following (Tian et al., 2020) on ImageNet, for KD, we set the weight for the KL loss to 0.9, the weight for cross-entropy loss to 0.5 which helps to obtain better accuracy.
Our results are presented in Table 7. Again, we observe that our method achieves significant improvements over all competing methods. Moreover, there is no method which is consistently second: for ResNet-34 to ResNet-18 experiment, RKD is the second best while for ResNet-50 to MobileNet, CRD is the second best. Notably, for the latter experiment, CRD reduces the gap between the teacher and the student by 1.27%, while our method narrows it by 2.36%. Overall, our results on ImageNet validate the scalability of our method, and show that, when applied to a large-scale dataset, we achieve an even more favourable performance compared against competing methods.

heading: 5.3 BINARY NETWORKS DISTILLATION
text: Training highly accurate binary neural networks (i.e. the most extreme case of quantization) is a very challenging task (Rastegari et al., 2016; Bulat & Tzimiropoulos, 2019), and to this end, knowledge distillation appears to be a promising direction. In this section, we present results by applying distillation for the task of training binary student networks guided by real-valued teacher networks. The network architecture is kept the same for both the student and the teacher in this case: specifically we used a ResNet using the modifications described in (Bulat & Tzimiropoulos, 2019).
Table 8 presents our results. Again, we observe that our method outperforms all methods considerably, showing that it can effectively transfer knowledge between different domains. Note that it was not clear to us where to place the distillation position for OFD, so although we included our result for this method, we emphasize that this result might be suboptimal.

heading: 5.4 FACIAL LANDMARK DETECTION
text: Given a face image, the task is to localise a set of facial landmarks in terms of their (x,y) coordinates. This is often solved by using a CNN to directly regress the (x,y) coordinates of the facial landmarks. In order to show the suitability of our method for this problem, we use the WFLW Wu et al. (2018) dataset, which is one of the hardest benchmarks for this task. Performance is measured in terms of Normalised Mean Error (lower is better), which is the standard metric for the problem. In our experiment, we use a ResNet50 as the teacher and a ResNet8 as the student. The results, shown in Table 9, confirm the superior performance of our method when compared to other state-of-the-art methods.

heading: 6 CONCLUSION
text: We presented a method for knowledge distillation that optimizes the output feature of the penultimate layer of the student network and hence is directly related to representation learning. A key to our method is the newly proposed Softmax Regression Loss which was found necessary for effective representation learning. We showed that our method consistently outperforms other stateof-the-art distillation methods for a wide range of experimental settings including multiple network architectures (ResNet, Wide ResNet, MobileNet) with different teacher-student capacities, datasets (CIFAR10/100, ImageNet), and domains (real-valued and binary networks).

heading: A APPENDIX
text: A.1 STUDY OF THE HYPER-PARAMETERS α AND β
We only performed a very basic search to find the best hyper-parameters. First, we fix α and search for the best β. Then, we used the best β, and search for the best α. This is sub-optimal compared to a full grid search over α and β. Furthermore, after some preliminary experimentation, we considered only for 2 values: 1 and 5. Notably, we found that for all teacher-student pairs but (T:WRN40 4, S:MV2) alpha=1 is the optimal value. Furthermore, on ImageNet, the optimal values were α = 1, β = 1 for all teacher-student pairs considered.
A.2 DATASETS AND TRAINING DETAILS
CIFAR-10 CIFAR-10 is a popular image classification dataset consisting of 50,000 training and 10,000 testing images equally distributed across 10 classes. All images are of resolution 32× 32px. Following (Zagoruyko & Komodakis, 2017), during training, we randomly cropped and horizontally flipped the images. The ResNet models were trained for 350 epochs using SGD. The initial learning rate was set to 0.1, and then it was reduced by a factor of 10 at epochs 150, 250 and 320. Similarly, the WRN models were trained for 200 epochs with a learning rate of 0.1 that was subsequently reduced by 5 at epochs 60, 120 and 160. In all experiments, we set the dropout rate to 0.
For traditional KD (Hinton et al., 2015), we set α = 0.9 and T = 4. For AT (Zagoruyko & Komodakis, 2017), as in (Zagoruyko & Komodakis, 2017; Tung & Mori, 2019), we set the weight of distillation loss to 1000. We note that, in our experiments, the AT loss is added after each layer group for WRN and the last two groups for ResNet as in (Zagoruyko & Komodakis, 2017). Following OFD (Heo et al., 2019a), we set the weight of distillation loss to 10−3. For RKD (Park et al., 2019), we set β1 = 25 for distance, and β2 = 50 for angle, as described in (Park et al., 2019; Tian et al., 2020). We did not compare with CRD (Tian et al., 2020) on CIFAR-10 because, in our experiments, we found that their parameter setting (used in their paper for CIFAR-100 and ImageNet-1K) does not obtain good performance on CIFAR-10.
CIFAR-100 For CIFAR-100 (Krizhevsky & Hinton, 2009), we used a standard data augmentation scheme (Zagoruyko & Komodakis, 2017) including padding 4 pixels prior to random cropping and horizontal flipping. We used SGD with weight decay 5e-4 and momentum 0.9. Batch size was set to 128. Learning rate was set to 0.1; then decayed by 0.1 at epochs 100, 150, until training reached 200 epochs (Heo et al., 2019a).
ImageNet-1K Images are cropped to 224 × 224 pixels for both training and evaluation. We used SGD with Nesterov momentum 0.9, weight decay 1e − 4, initial learning rate 0.2 which was then dropped by a factor of 10 every 30 epochs, training in total for 100 epochs (for CRD we trained with 10 more epochs as suggested by the authors). Batch size was set to 512. For simplicity and to enable a fair comparison, we used pretrained PyTorch models Paszke et al. (2017) as teacher networks Heo et al. (2019a); Tian et al. (2020). For binary experiments, we used Adam as the optimizer with initial learning 0.002 which was then reduced by a factor of 10 every 30 epochs, training in total for 100 epochs.
A.3 ADDITIONAL ABLATION STUDIES
Different losses for LSR: This part expands Section 4 of our paper by evaluating different losses for LSR. The following loss functions are compared:
1. L2 loss: LSR−L2(p, q) = ‖p− q‖2. This is the loss used in Section 4 of our paper 2. Cross Entropy loss (CE) with label y: LSR−CE(q, y) = H(q, y). 3. KL loss with temperature τ Hinton et al. (2015): LSR−KL(p, q) = KL(q/τ, p/τ).
The results, presented in Table 10, show that all loss functions offer significant improvement gains while LFM+LSR−L2 achieves the best accuracy. Therefore, in our paper, LSR−L2 is used in all cases.
Combining our method with KD and AT: Table 11 shows additional comparisons on CIFAR100 by combining our method with AT Zagoruyko & Komodakis (2017) and KD Hinton et al. (2015), respectively. The results show that a straightforward combination did not provide satisfactory results, however it could be possible that a more comprehensive investigation might prove to be beneficial.
A.4 ADDITIONAL COMPARISONS
This section provides additional comparisons using the evaluation framework of CRD Tian et al. (2020). Comparisons include distillation between models with the same architecture (e.g. ResNet56 to ResNet20) and between different architectures (e.g. ResNet50 to MobileNetV2). In order to
maximize the fairness of the comparison, we followed their experimental setting. Thus, we did not choose the training parameters, teacher-student architecture pairs or methods to compare against. The competing methods included are:
• Classic: Knowledge Distillation (KD) Hinton et al. (2015), FitNet Romero et al. (2015), Attention Transfer (AT) Zagoruyko & Komodakis (2017).
• Most recent: Similarity-Preserving KD (SP) (Tung & Mori, 2019), Correlation Congruence (CC) (Peng et al., 2019), Variational Information Distillation (VID) (Ahn et al., 2019), Relational Knowledge Distillation (RKD) (Park et al., 2019), Distillation of Activation Boundaries (AB) (Heo et al., 2019b), Factor Transfer (FT) (Kim et al., 2018), Flow of Solution (FSP) (Yim et al., 2017) and Contrastive Representation Distillation (CRD) (Tian et al., 2020).

Review summaries:
This paper proposes a new formulation of knowledge distillation (KD) for model compression. Different from the classic formulation that matches the logits between student and teacher models, this paper suggests to match the output features of the penultimate layers between student and teacher models, based on L2 distance. Two complementary variants are introduced, with one directly minimizing the distance of the original feature vectors and with the other minimizing the distance of the feature vectors projected on the teacher classifiers. The approach is evaluated in a variety of scenarios, such as different network architectures, teacher-student capacities, datasets, and domains, and compared with state-of-the-art results.

This paper proposes a new knowledge distillation method by enhancing the student network's representation learning. The proposed framework is very simple: use the teacher's final fully-connected layer (or, 'projection matrix $W$' in the paper) to obtain both the teacher's and the student's logit output and optimize the student network by minimizing the gap between two logits. Experiments cover various network architectures on CIFAR and ImageNet datasets and show improvements over the other competitive knowledge distillation methods. 

This paper proposed a novel method for knowledge distillation. The idea is to utilize the teacher’s pre-trained classifier to train the student’s penultimate layer feature by adopting two losses: (a) the Feature Matching loss LFM and (b) the Softmax Regression loss LSR. The latter is designed to take into account the classification task at hand, which imposes that for the same input image, the teacher’s and student’s feature produce the same output when passed through the teacher’s pre-trained and frozen classifier.