Title: CONTRASTIVE REPRESENTATION DISTILLATION

Abstract: Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.

1. INTRODUCTION:

Knowledge distillation (KD) transfers knowledge from one deep learning model (the teacher) to another (the student). The objective originally proposed by Hinton et al. (2015) minimizes the KL divergence between the teacher and student outputs. This formulation makes intuitive sense when the output is a distribution, e.g., a probability mass function over classes. However, often we instead wish to transfer knowledge about a representation. For example, in the problem of "cross-modal distillation", we may wish to transfer the representation of an image processing network to a sound (Aytar et al., 2016) or to depth (Gupta et al., 2016) processing network, such that deep features for an image and the associated sound or depth features are highly correlated. In such cases, the KL divergence is undefined.

Representational knowledge is structured -the dimensions exhibit complex interdependencies. The original KD objective introduced in (Hinton et al., 2015) treats all dimensions as independent, conditioned on the input. Let y T be the output of the teacher and y S be the output of the student. Then the original KD objective function, ψ, has the fully factored form: ψ(y S , y T ) = i φ i (y S i , y T i ) * . Such a factored objective is insufficient for transferring structural knowledge, i.e. dependencies between output dimensions i and j. This is similar to the situation in image generation where an L 2 objective produces blurry results, due to independence assumptions between output dimensions.

To overcome this problem, we would like an objective that captures correlations and higher order output dependencies. To achieve this, in this paper we leverage the family of contrastive objectives (Gutmann & Hyvärinen, 2010; Oord et al., 2018; Arora et al., 2019; Hjelm et al., 2018) . These objective functions have been used successfully in recent years for density estimation and representation learning, especially in self-supervised settings. Here we adapt them to the task of knowledge distillation from one deep network to another. We show that it is important to work in representation space, similar to recent works such as Zagoruyko & Komodakis (2016a); Romero et al. (2014) . However, note that the loss functions used in those works do not explicitly try to capture correlations or higher-order dependencies in representational space. resolution can be combined in this approach, which is a significant limitation since it requires student and teacher networks with very similar architectures. This technique achieves state of the art results for distillation (as measured by the generalization of the student network). FitNets (Romero et al., 2014) also deal with intermediate representations by using regressions to guide the feature activations of the student network. Since Zagoruyko & Komodakis (2016a) do a weighted form of this regression, they tend to perform better. Other papers (Yim et Koratana et al., 2019) have enforced various criteria based on representations. The contrastive objective we use in this paper is the same as that used in CMC (Tian et al., 2019) . But we derive from a different perspective and give a rigorous proof that our objective is a lower bound on mutual information. Our objective is also related to the InfoNCE and NCE objectives introduced in (Oord et al., 2018; Gutmann & Hyvärinen, 2010). Oord et al. (2018) use contrastive learning in the context of self-supervised representations learning. They show that their objective maximizes a lower bound on mutual information. A very related approach is used in (Hjelm et al., 2018) . InfoNCE and NCE are closely related but distinct from adversarial learning (Goodfellow et al., 2014) . In (Goodfellow, 2014) , it is shown that the NCE objective of Gutmann & Hyvärinen (2010) can lead to maximum likelihood learning, but not the adversarial objective.

3. METHOD:

The key idea of contrastive learning is very general: learn a representation that is close in some metric space for "positive" pairs and push apart the representation between "negative" pairs. Fig. 1 gives a visual explanation for how we structure contrastive learning for the three tasks we consider: model compression, cross-modal transfer and ensemble distillation.

3.1. CONTRASTIVE LOSS:

Given two deep neural networks, a teacher f T and a student f S . Let x be the network input; we denote representations at the penultimate layer (before logits) as f T (x) and f S (x) respectively. Let x i represent a training sample, and x j another randomly chosen sample. We would like to push closer the representations f S (x i ) and f T (x i ) while pushing apart f S (x i ) and f T (x j ). For ease of notation, we define random variables S and T for the student and teacher's representations of the data respectively:

EQUATION

EQUATION

EQUATION

Intuitively speaking, we will consider the joint distribution p(S, T ) and the product of marginal distributions p(S)p(T ), so that, by maximizing KL divergence between these distributions, we can maximize the mutual information between student and teacher representations. To setup an appropriate loss that can achieve this aim, let us define a distribution q with latent variable C which decides whether a tuple (f T (x i ), f S (x j )) was drawn from the joint (C = 1) or product of marginals (C = 0):

EQUATION

Now, suppose in our data, we are given 1 congruent pair (drawn from the joint distribution, i.e. the same input provided to T and S) for every N incongruent pairs (drawn from the product of marginals; independent randomly drawn inputs provided to T and S). Then the priors on the latent C are:

EQUATION

By simple manipulation and Bayes' rule, the posterior for class C = 1 is given by: q(C = 1|T, S) = q(T, S|C = 1)q(C = 1) q(T, S|C = 0)q(C = 0) + q(x, y|C = 1)q(C = 1)

= p(T, S) p(T, S)

EQUATION

Next, we observe a connection to mutual information as follows:

EQUATION

Then taking expectation on both sides w.r.t. p(T, S) (equivalently w.r.t. q(T, S|C = 1)) and rearranging, gives us:

EQUATION

where I(T ; S) is the mutual information between the distributions of the teacher and student embeddings. Thus maximizing E q(T,S|C=1) log q(C = 1|T, S) w.r.t. the parameters of the student network S increases a lower bound on mutual information. However, we do not know the true distribution q(C = 1|T, S); instead we estimate it by fitting a model h : {T , S} → [0, 1] to samples from the data distribution q(C = 1|T, S), where T and S represent the domains of the embeddings. We maximize the log likelihood of the data under this model (a binary classification problem):

EQUATION

We term h the critic since we will be learning representations that optimize the critic's score. Assuming sufficiently expressive h, h * (T, S) = q(C = 1|T, S) (via Gibbs' inequality; see Sec. 6.2.1 for proof), so we can rewrite Eq. 9 in terms of h * :

EQUATION

Therefore, we see that the optimal critic is an estimator whose expectation lower-bounds mutual information. We wish to learn a student that maximizes the mutual information between its representation and the teacher's, suggesting the following optimization problem:

EQUATION

An apparent difficulty here is that the optimal critic h * depends on the current student. We can circumvent this difficulty by weakening the bound in (12) to:

EQUATION

EQUATION

The first line comes about by simply adding N E q(T,S|C=0) [log(1h * (T, S))] to the bound in (12) . This term is strictly negative, so the inequality holds. The last line follows from the fact that L critic (h * ) upper-bounds L critic (h). Optimizing (15) w.r.t. the student we have:

EQUATION

which demonstrates that we may jointly optimize f S at the same time as we learn h. We note that due to (16), f S * = arg max f S L critic (h), for any h, also is a representation that optimizes a lower-bound (a weaker one) on mutual information, so our formulation does not rely on h being optimized perfectly.

We may choose to represent h with any family of functions that satisfy h : {T , S} → [0, 1]. In practice, we use the following:

EQUATION

)

where M is the cardinality of the dataset and τ is a temperature that adjusts the concentration level. In practice, since the dimensionality of S and T may be different, g S and g T linearly transform them into the same dimension and further normalize them by L-2 norm before the inner product. The form of Eq. ( 18) is inspired by NCE (Gutmann & Hyvärinen, 2010; Wu et al., 2018) . Our formulation is similar to the InfoNCE loss (Oord et al., 2018) in that we maximize a lower bound on the mutual information. However we use a different objective and bound, which in our experiments we found to be more effective than InfoNCE.

Implementation. Theoretically, larger N in Eq. 16 leads to tighter lower bound on MI. In practice, to avoid using very large batch size, we follow Wu et al. (2018) and implement a memory buffer that stores latent features of each data sample computed from previous batches. Therefore, during training we can efficiently retrieve a large number of negative samples from the memory buffer.

3.2. KNOWLEDGE DISTILLATION OBJECTIVE:

The knowledge distillation loss was proposed in Hinton et al. (2015) . In addition to the regular cross-entropy loss between the student output y S and one-hot label y, it asks the student network output to be as similar as possible to the teacher output by minimizing the cross-entropy between their output probabilities. The complete objective is:

EQUATION

)

where ρ is the temperature, α is a balancing weight, and σ is softmax function.

H(σ(z T /ρ), σ(z S /ρ))

is further decomposed into KL(σ(z T /ρ)|σ(z S /ρ)) and a constant entropy H(σ(z T /ρ)).

3.3. CROSS-MODAL TRANSFER LOSS:

In the cross-modal transfer task shown in Fig. 1 (b), a teacher network is trained on a source modality X with large-scale labeled dataset. We then wish to transfer the knowledge to a student network, but adapt it to another dataset or modality Y. But the features of the teacher network are still valuable to help with learning of the student on another domain. In this transfer task, we use the contrastive loss Eq. 10 to match the features of the student and teacher. Additionally, we also consider other distillation objectives, such as KD discussed in previous section, Attention Transfer 

3.4. ENSEMBLE DISTILLATION LOSS:

In the case of ensemble distillation shown in 1(c), we have M > 1 teacher networks, f Ti and one student network f S . We adopt the contrastive framework by defining multiple pair-wise contrastive losses between features of each teacher network f Ti and the student network f S . These losses are summed together to give the final loss (to be minimized): loss, which we call CRD (Contrastive Representation Distillation), consistently outperforms all other distillation objectives, including the original KD (an average relative improvement of 57%). Surprisingly, we found that KD works pretty well and none of the other methods consistently outperforms KD on their own. Another observation is that, while switching the teacher student combinations from same to different architectural styles, methods that distill intermediate representations tend to perform worse than methods that distill from the last several layers. For example, the Attention Transfer (AT) and FitNet methods even underperform the vanilla student. In contrast, PKT, SP and CRD that operate on last several layers performs well. This might because, architectures of different styles have their own solution paths mapping from the input to the output, and enforcing the mimic of intermediate representations thus might conflict with such inductive bias.

EQUATION

Capturing inter-class correlations. In Fig. 2 , we compute the difference between the correlation matrices of the teacher's and student's logits; for three different students: vanilla student without distillation, trained by AT, KD or CRD (our method). It is clear that the CRD objective captures the most correlation structure in the logit as shown by the smaller differences between teacher and student. This is reflected in reduced error rates.

Results on ImageNet For a fair comparison with Zagoruyko & Komodakis (2016a) and Lan et al.

(2018), we adopt the models from these papers, ResNet-34 as the teacher and ResNet-18 as the student.

As shown in Table 3 , the gap of top-1 accuracy between the teacher and student is 3.56%. The AT method reduces this gap by 0.95%, while ours narrow it by 1.42%, a 50% relative improvement.

Results on ImageNet validates the scalability of our CRD.

Transferability of representations:

We are interested in representations, and a primary goal of representation learning is to acquire general knowledge, that is, knowledge that transfers to tasks or datasets that were unseen during training. Therefore, we test if the representations we distill transfer well. A WRN-16-2 student either distills from a WRN-40-2 teacher, or is trained from scratch on CIFAR100. The student serves as a frozen representation extractor (the layer prior to the logit) for images from STL-10 or TinyImageNet (all images downsampled to 32x32). We then train a linear classifier to perform 10-way (for STL- AT in Table 4 . In this setting, all distillation methods except FitNet improve transferability of the learned representations on both STL-10 and TinyImageNet. While the teacher performs the best on the original CIFAR100 dataset, its representations transfer the worst to the other two datasets. This is perhaps the teacher's representations are biased towards the original task. Surprisingly, the student with CRD+KD distillation not only matches its teacher on CIFAR100 (see Table 4 ), but also transfers much better than the teacher, e.g., 3.6% improvement (STL-10) and 4.1% on TinyImageNet.

4.2. CROSS-MODAL TRANSFER:

We consider a practical setting where modality X has large amount of labeled data while modality Y does not. Transfering knowledge from X to Y is a common challenge. For example, while large-scale RGB datasets are easily accessible, other modalities such as depth images are much harder to label at scale but have wide applications. We demonstrate the potential of CRD for cross-modal transfer in two scenarios: (a) transfer from luminance to chrominance; (b) transfer from RGB to depth images.

Transferring from Luminance to Chrominace. We work on Lab color space, where L represents Luminance and ab Chrominance. We first train an L network on TinyImageNet with supervision.

Then we transfer knowledge from this L network to ab network on the unlabeled set of STL-10 with different objectives, including FitNet, KD, KD+AT and CRD. For convenience, we use the same architecture for student and teacher (they can also be different). Finally, we evaluate the knowlege of ab network by two means: (1) linear probing: we freeze the ab network and train a linear classifier on top of features from different layers to perform 10-way classification on STL-10 ab images. This is a common practice (Alain & Bengio, 2016; Zhang et al., 2017) to evaluate the quality of network representations; (b) fully finetuning: we fully finetune the ab network to obtain the best accuracy. We also use as a baseline the ab network that is randomly initialized rather than distilled. Architectures investigated include AlexNet, ResNet-18 and ResNet-50. The results shown in Figure 3 show that CRD is more efficient for transferring inter-modal knowledge than other methods. Besides, we also note KD+AT does not improve upon KD, possibly because attention of luminance and chrominance are different and harder to transfer. maps from depth images. We note that both knowledge transfer and downstream training are conducted on the same set of images, i.e., the training set. Table 5 reports the average pixel prediction accuracy and mean Intersection-over-Union across all classes. All distillation methods can transfer knowledge from the RGB ResNet to the depth CNN. FitNet surpasses KD and KD+AT. CRD significantly outperforms all other methods.

4.3. DISTILLATION FROM AN ENSEMBLE:

Better classification performance is often achieved by ensembles of deep networks, but these are usually too expensive for inference time, and distillation into a single network is a desirable task. We investigate the KL-divergence based KD and our CRD for this task, using the loss of Sec. 21. The network structures of each teacher and student are identical here, but an ensemble of multiple teachers can still provide rich knowledge for the student. To compare between KD and CRD on CIFAR100 dataset, we use WRN-16-2 and ResNet-20, whose single model error rates are 26.7% and 30.9% respectively. The results of distillation are presented in Figure 4 , where we vary the number of ensembled teachers. CRD with 8 teachers decreases the error rate of WRN-16-2 to 23.7% and ResNet20 to 28.3%. In addition, CRD works consistently better than KD in all settings we test. These observations suggest that CRD is capable of distilling an ensemble of models into a single one which performs significantly better than a model of the same size that is trained from scratch. ) is an alternative contrastive objective which select a single positive out from a set of distractors via a softmax function. We compare InfoNCE with our contrastive objective Eq 18, when using the same number of negatives. The last two rows in Table 6 show that our objective outperforms InfoNCE in 4 out of 5 teacher-student combinations.

Negative Sampling. In this paper, we consider two negative sampling plocies when giving an anchor x i : (1) x j j =i for the unsupervised case when we have no labels, or (2) x j yj =yi for supervised case, where y i represents the label associated with sample x i . One might imagine that the first sampling strategy will increase the intra-class variance as we may push apart positives and negatives from the same underlying class, while the second would not. We quantitatively measure and report the gap between these two strategies in Table 6 , which shows that the classification accuracy by the second sampling strategy is 0.81% higher for our objective and 0.62% higher for InfoNCE than the first one. 

4.5. HYPER-PARAMETERS AND COMPUTATION OVERHEAD:

There are two main hyper-parameters in our contrastive objectives: (1) the number of negative samples N in Eq. 18, and (2) temperature τ which modulates the softmax probability. We adopt WRN-40-2 as teacher and WRN-16-2 as student for parameter analysis. Experiments are conducted on CIFAR100 and the results are shown in Figure 5 . Temperature τ We varied τ between 0.02 and 0.3. As Figure 5 (b) illustrates, both extremely high or low temperature lead to a sub-optimal solution. In general, temperatures between 0.05 and 0. 

Number of negatives:



5. CONCLUSION:

We have developed a novel technique for neural network distillation, using the concept of contrastive objectives, which are usually used for representation learning. We experimented with our objective on a number of applications such as model compression, cross-modal transfer and ensemble distillation, outperforming other distillation objectives by significant margins in all these tasks. Our contrastive objective is the only distillation objective that consistently outperforms knowledge distillation across a wide variety of knowledge transfer tasks. Prior objectives only surpass KD when combined with KD. Contrastive learning is a simple and effective objective with practical benefits.

defining for convenience h (C = 1, S = s, T = t) = h(S = s, T = t) and h (C = 0, S = s, T = t) = 1h(S = s, T = t). The log likelihood function is:

EQUATION

By Gibbs' inequality, the max likelihood fit is h (C = c, S = s, T = t) = q(C = c|S = s, T = t), which also implies that h(S = s, T = t) = q(C = 1|S = s, T = t).

We now demonstrate that our objective in Eq. ( 10) is proportional to a summation over terms Eq. ( 22) for all s ∈ S and t ∈ T .

EQUATION

EQUATION

Notice that ( 26) is proportional to Eq. ( 10) from the main paper. For sufficiently expressive h, then, each term inside the expectation in Eq. ( 23) can be maximized, resulting in h * (T = t, S = s) = q(C = 1|T = t, S = t) for all s and t. For CIFAR-100, we initialize the learning rate as 0.05, and decay it by 0.1 every 30 epochs after the first 150 epochs until the last 240 epoch. For MobileNetV2, ShuffleNetV1 and ShuffleNetV2, we use a learning rate of 0.01 as this learning rate is optimal for these models in a grid search, while 0.05 is optimal for other models.

For ImageNet, we follow the standard PyTorch practice but train for 10 more epochs. Batch size is 64 for CIFAR-100 or 256 for ImageNet.

The student is trained by a combination of cross-entropy classification objective and a knowledge distillation objective, shown as follows:

EQUATION

For the weight balance factor β, we directly use the optimal value from the original paper if it is specified, or do a grid search with teacher WRN-40-2 and student WRN-16-2. This results in the following list of β used for different objectives:

Table 7 shows the effect of combining various objectives with KD. Most methods is able to slightly outperform KD after this combination. We also the check the compatibility of our distillation objective with KD and PKT. The combination of CRD and KD/PKT further improves over single CRD objective.

6.7. ADDITIONAL RESULTS ON TRANSFERABILITY OF REPRESENTATIONS:

In addtion to Table 4 shown in the main text, we also evaluate the transferabilit of the representations of various other distillation methods and summarize the results in Table 8 . In general, CRD achieves the best transferring accuracy in 7 out of 10 settings.

6.8. DEEP MUTUAL LEARNING SETTING:

In (Zhang et al., 2018b) , a Deep Mutual Learning setting was proposed where the teacher and student networks are trained simultaneously rather than sequentially. The benefit is that not only the student networks but also the teacher models will be improved. Here we investigate the possibility of incorporating different distillation objectives into this training framework. The results are summarizied in Table 9 . We observe that in general logit based distillation methods is better than non-logit based distillation methods in this setting, i.e., KD performs the best. We conjecture that during the mutual training phase, KD performs like an advanced label smoothing regularization and does not require the logit to be very accurate. But the feature-based distillation methods are hard to learn knowledge from the teacher model when it is handicapped. On the other hand, we notice that the combination of KD and CRD leads to better performance on the student side, as shown in the last row of Table 9 . We measure the transferability of the student network, by evaluating a linear classifier on top of its frozen representations on STL10 (abbreviated as "STL") and TinyImageNet (abbreviated as "TI"). The best accuracy is bolded and the second best is underlined.

:



Figures and Tables:

Zagoruyko &  Komodakis (2016a)  and FitNetRomero et al. (2014). Such transfer is conducted on a paired but unlabeled dataset D = {(x i , y i )|i = 1, ..., L, x i ∈ X , y i ∈ Y}. In this scenario, there is no true label y of such data for the original training task on the source modality, and therefore we ignore the H(y, y S ) term in all objectives that we test. Prior cross-modal work Aytar et al. (2016); Hoffman et al. (2016b;a) uses either L 2 regression or KL-divergence.

We evaluate our contrastive representation distillation (CRD) framework in three knowledge distillation tasks: (a) model compression of a large network to a smaller one; (b) cross-modal knowledge transfer; (c) ensemble distillation from a group of teachers to a single student network. Datasets (1) CIFAR-100 (Krizhevsky & Hinton, 2009) contains 50K training images with 0.5K images per class and 10K test images. (2) ImageNet (Deng et al., 2009) provides 1.2 million images from 1K classes for training and 50K for validation. (3) STL-10 (Coates et al., 2011) consists of a training set of 5K labeled images from 10 classes and 100K unlabeled images, and a test set of 8K images. (4) TinyImageNet (Deng et al., 2009) has 200 classes, each with 500 training images and 50 validaton images. (5) NYU-Depth V2 (Silberman et al., 2012) consists of 1449 indoor images, each labeled with dense depth image and semantic map.

Figure 2: The correlations between class logits of a teacher network are ignored by regular cross-entropy. Distillation frameworks use "soft targets" (Hinton et al., 2015) which effectively capture such correlations and transfer them to the student network, leading to the success of distillation. We visualize here the difference of correlation matrices of student and teacher logits, for different student networks on a CIFAR-100 knowledge distillation task: (a) Student trained without distillation, showing that the teacher and student cross-correlations are very different; (b) Student distilled by attention transfer (Zagoruyko & Komodakis, 2016a); showing reduced difference (see axis); (c) Student distilled by KL divergence (Hinton et al., 2015), also showing reduced difference; (d) Student distilled by our contrastive objective, showing significant matching between student's and teacher's correlations. In this visualization, we use WRN-40-2 as teacher and WRN-40-1 as student.

Figure 3: Top-1 classification accuracy on STL-10 using chrominance image (ab channel in Lab color space). We initialize the chrominance network randomly or by distilling from a luminance network, trained with large-scale labeled images. We evaluate distillation performance by (a) linear probing and (b) fully finetuning.

Figure 4: Distillation from an ensemble of teachers. We vary the number of ensembled teachers and compare KD with our CRD by using (a) WRN-16-2 and (b) ResNet20. Our CRD consistently achieves lower error rate.

ABLATIVE STUDY InfoNCE v.s. Ours. InfoNCE (Oord et al., 2018

Figure 5: Effects of varying the number of negatives, shown in (a), or the temperature, shown in (b).

We have validated different N : 16, 64, 256, 1024, 4096, 16384. As shown in Figure5(a), increasing N leads to improved performance. However, the difference of error rate between N = 4096 and N = 16384 is less than 0.1%. Therefore, we use N = 16384 for reporting the accuracy while in practice N = 4096 should suffice.

al., 2017; Huang & Wang, 2017; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019;

Table content: <table/>

Test

Table content: <table><tr><td>Teacher</td><td>WRN-40-2</td><td>WRN-40-2</td><td colspan="2">resnet56</td><td colspan="2">resnet110</td><td colspan="2">resnet110</td><td>resnet32x4</td><td>vgg13</td></tr><tr><td>Student</td><td>WRN-16-2</td><td>WRN-40-1</td><td colspan="2">resnet20</td><td colspan="2">resnet20</td><td colspan="2">resnet32</td><td>resnet8x4</td><td>vgg8</td></tr><tr><td>Teacher</td><td>75.61</td><td>75.61</td><td/><td>72.34</td><td/><td>74.31</td><td colspan="2">74.31</td><td>79.42</td><td>74.64</td></tr><tr><td>Student</td><td>73.26</td><td>71.98</td><td/><td>69.06</td><td/><td>69.06</td><td colspan="2">71.14</td><td>72.50</td><td>70.36</td></tr><tr><td>KD  *</td><td>74.92</td><td>73.54</td><td/><td>70.66</td><td/><td>70.67</td><td colspan="2">73.08</td><td>73.33</td><td>72.98</td></tr><tr><td>FitNet  *</td><td colspan="8">73.58 (↓) 72.24 (↓) 69.21 (↓) 68.99 (↓) 71.06 (↓) 73.50 (↑) 71.02 (↓)</td></tr><tr><td>AT</td><td colspan="8">74.08 (↓) 72.77 (↓) 70.55 (↓) 70.22 (↓) 72.31 (↓) 73.44 (↑) 71.43 (↓)</td></tr><tr><td>SP</td><td colspan="8">73.83 (↓) 72.43 (↓) 69.67 (↓) 70.04 (↓) 72.69 (↓) 72.94 (↓) 72.68 (↓)</td></tr><tr><td>CC</td><td colspan="8">73.56 (↓) 72.21 (↓) 69.63 (↓) 69.48 (↓) 71.48 (↓) 72.97 (↓) 70.71 (↓)</td></tr><tr><td>VID</td><td colspan="8">74.11 (↓) 73.30 (↓) 70.38 (↓) 70.16 (↓) 72.61 (↓) 73.09 (↓) 71.23 (↓)</td></tr><tr><td>RKD</td><td colspan="8">73.35 (↓) 72.22 (↓) 69.61 (↓) 69.25 (↓) 71.82 (↓) 71.90 (↓) 71.48 (↓)</td></tr><tr><td>PKT</td><td colspan="8">74.54 (↓) 73.45 (↓) 70.34 (↓) 70.25 (↓) 72.61 (↓) 73.64 (↑) 72.88 (↓)</td></tr><tr><td>AB</td><td colspan="8">72.50 (↓) 72.38 (↓) 69.47 (↓) 69.53 (↓) 70.98 (↓) 73.17 (↓) 70.94 (↓)</td></tr><tr><td>FT  *</td><td colspan="8">73.25 (↓) 71.59 (↓) 69.84 (↓) 70.22 (↓) 72.37 (↓) 72.86 (↓) 70.58 (↓)</td></tr><tr><td>FSP  *</td><td>72.91 (↓)</td><td>n/a</td><td colspan="6">69.95 (↓) 70.11 (↓) 71.89 (↓) 72.62 (↓) 70.23 (↓)</td></tr><tr><td>NST  *</td><td colspan="8">73.68 (↓) 72.24 (↓) 69.60 (↓) 69.53 (↓) 71.96 (↓) 73.30 (↓) 71.53 (↓)</td></tr><tr><td>CRD</td><td colspan="8">75.48 (↑) 74.14 (↑) 71.16 (↑) 71.46 (↑) 73.48 (↑) 75.51 (↑) 73.94 (↑)</td></tr><tr><td colspan="9">CRD+KD 75.64 (↑) 74.38 (↑) 71.63 (↑) 71.56 (↑) 73.75 (↑) 75.46 (↑) 74.29 (↑)</td></tr><tr><td>Teacher</td><td>vgg13</td><td>ResNet50</td><td/><td colspan="2">ResNet50</td><td colspan="2">resnet32x4</td><td>resnet32x4</td><td>WRN-40-2</td></tr><tr><td>Student</td><td>MobileNetV2</td><td colspan="2">MobileNetV2</td><td>vgg8</td><td/><td colspan="2">ShuffleNetV1</td><td>ShuffleNetV2</td><td>ShuffleNetV1</td></tr><tr><td>Teacher</td><td>74.64</td><td>79.34</td><td/><td>79.34</td><td/><td colspan="2">79.42</td><td>79.42</td><td>75.61</td></tr><tr><td>Student</td><td>64.6</td><td>64.6</td><td/><td>70.36</td><td/><td colspan="2">70.5</td><td>71.82</td><td>70.5</td></tr><tr><td>KD  *</td><td>67.37</td><td>67.35</td><td/><td>73.81</td><td/><td colspan="2">74.07</td><td>74.45</td><td>74.83</td></tr><tr><td>FitNet  *</td><td>64.14 (↓)</td><td>63.16 (↓)</td><td/><td colspan="2">70.69 (↓)</td><td colspan="2">73.59 (↓)</td><td>73.54 (↓)</td><td>73.73 (↓)</td></tr><tr><td>AT</td><td>59.40 (↓)</td><td>58.58 (↓)</td><td/><td colspan="2">71.84 (↓)</td><td colspan="2">71.73 (↓)</td><td>72.73 (↓)</td><td>73.32 (↓)</td></tr><tr><td>SP</td><td>66.30 (↓)</td><td>68.08 (↑)</td><td/><td colspan="2">73.34 (↓)</td><td colspan="2">73.48 (↓)</td><td>74.56 (↑)</td><td>74.52 (↓)</td></tr><tr><td>CC</td><td>64.86 (↓)</td><td>65.43 (↓)</td><td/><td colspan="2">70.25 (↓)</td><td colspan="2">71.14 (↓)</td><td>71.29 (↓)</td><td>71.38 (↓)</td></tr><tr><td>VID</td><td>65.56 (↓)</td><td>67.57 (↑)</td><td/><td colspan="2">70.30 (↓)</td><td colspan="2">73.38 (↓)</td><td>73.40 (↓)</td><td>73.61 (↓)</td></tr><tr><td>RKD</td><td>64.52 (↓)</td><td>64.43 (↓)</td><td/><td colspan="2">71.50 (↓)</td><td colspan="2">72.28 (↓)</td><td>73.21 (↓)</td><td>72.21 (↓)</td></tr><tr><td>PKT</td><td>67.13 (↓)</td><td>66.52 (↓)</td><td/><td colspan="2">73.01 (↓)</td><td colspan="2">74.10 (↑)</td><td>74.69 (↑)</td><td>73.89 (↓)</td></tr><tr><td>AB</td><td>66.06 (↓)</td><td>67.20 (↓)</td><td/><td colspan="2">70.65 (↓)</td><td colspan="2">73.55 (↓)</td><td>74.31 (↓)</td><td>73.34 (↓)</td></tr><tr><td>FT  *</td><td>61.78 (↓)</td><td>60.99 (↓)</td><td/><td colspan="2">70.29 (↓)</td><td colspan="2">71.75 (↓)</td><td>72.50 (↓)</td><td>72.03 (↓)</td></tr><tr><td>NST  *</td><td>58.16 (↓)</td><td>64.96 (↓)</td><td/><td colspan="2">71.28 (↓)</td><td colspan="2">74.12 (↑)</td><td>74.68 (↑)</td><td>74.89 (↑)</td></tr><tr><td>CRD</td><td>69.73 (↑)</td><td>69.11 (↑)</td><td/><td colspan="2">74.30 (↑)</td><td colspan="2">75.11 (↑)</td><td>75.65 (↑)</td><td>76.05 (↑)</td></tr><tr><td>CRD+KD</td><td>69.94 (↑)</td><td>69.54 (↑)</td><td/><td colspan="2">74.58 (↑)</td><td colspan="2">75.12 (↑)</td><td>76.05 (↑)</td><td>76.27 (↑)</td></tr></table>

Top-1 test accuracy (%) of student networks on CIFAR100 of a number of distillation methods (ours is CRD) for transfer across very different teacher and student architectures. CRD outperforms KD and all other methods. Importantly, some methods that require very similar student and teacher architectures perform quite poorly. E.g. FSP (Yim et al., 2017) cannot even be applied; AT(Ba & Caruana, 2014) and FitNet (Zagoruyko & Komodakis, 2016a) perform very poorly etc. We denote by * methods where we used our reimplementation based on the paper; for all other methods we used author-provided or author-verified code. Average over 3 runs.

Table content: <table/>

Top

Table content: <table><tr><td/><td colspan="2">Teacher Student AT</td><td>KD</td><td>SP</td><td colspan="2">CC Online KD * CRD CRD+KD</td></tr><tr><td colspan="2">Top-1 26.69</td><td colspan="4">30.25 29.30 29.34 29.38 30.04</td><td>29.45</td><td>28.83</td><td>28.62</td></tr><tr><td>Top-5</td><td>8.58</td><td colspan="4">10.93 10.00 10.12 10.20 10.83</td><td>10.41</td><td>9.87</td><td>9.51</td></tr></table>

We transfer the representation learned from CIFAR100 to STL-10 and TinyImageNet datasets by

Table content: <table><tr><td/><td colspan="4">Student KD AT FitNet CRD CRD+KD Teacher</td></tr><tr><td>CIFAR100→STL-10</td><td>69.7</td><td>70.9 70.7 70.3 71.6</td><td>72.2</td><td>68.6</td></tr><tr><td>CIFAR100→TinyImageNet</td><td>33.7</td><td>33.9 34.2 33.5 35.6</td><td>35.5</td><td>31.5</td></tr><tr><td colspan="5">freezing the network and training a linear classifier on top of the last feature layer to perform 10-way (STL-10)</td></tr><tr><td colspan="5">or 200-way (TinyImageNet) classification. For this experiment, we use the combination of teacher network</td></tr><tr><td colspan="4">WRN-40-2 and student network WRN-16-2. Classification accuracies (%) are reported.</td><td/></tr></table>

Performance on the task of using depth to predict semantic segmentation labels. We initialize the depth network either randomly or by distilling from a ImageNet pre-pretrained ResNet-18 teacher.

Table content: <table><tr><td colspan="6">Metric (%) Random Init. KD KD+AT FitNet CRD</td></tr><tr><td>Pix. Acc.</td><td>56.4</td><td>58.9</td><td>60.1</td><td>60.8</td><td>61.6</td></tr><tr><td>mIoU</td><td>35.8</td><td>38.0</td><td>39.5</td><td>40.7</td><td>41.8</td></tr></table>

Ablative study of different contrastive objectives and negative sampling policies on CIFAR100. For contrastive objectives, we compare our objective with InfoNCE (Oord et al., 2018); For negative sampling policy, when given an anchor image xi from the dataset, we consider either randomly sample negative xj such that (a) i = j, or (b) yi = yj where y represents the class label. Average over 5 runs.

Table content: <table><tr><td colspan="2">sampling objective</td><td>WRN-40-2 WRN-16-2</td><td>resnet110 resnet20</td><td>resnet110 resnet32</td><td>resnet32x4 resnet8x4</td><td>vgg13 vgg8</td></tr><tr><td>i = j</td><td>InfoNCE Ours</td><td>74.78 74.48</td><td>70.56 70.64</td><td>72.67 72.64</td><td>74.69 74.67</td><td>73.24 73.39</td></tr><tr><td>y i = y j</td><td>InfoNCE Ours</td><td>75.15 75.48</td><td>71.39 71.46</td><td>73.53 73.48</td><td>75.22 75.51</td><td>73.74 73.94</td></tr></table>

2 work well on CIFAR100. All experiments but those on ImageNet use a temperature of 0.1. For ImageNet, we use τ = 0.07. The optimal temperature may vary across different datasets and require further tuning.Computational Cost We use ResNet-18 on ImageNet for illustration. CRD uses extra 260 MFLOPs, which is about 12% of the original 2 GFLOPs. In practice, we did not notice significant difference of training time on ImageNet (e.g., 1.75 epochs/hour v.s. 1.67 epochs/hour on two Titan-V GPUs). The memory bank for storing all 128-d features of ImageNet only costs around 600MB memory, and therefore we store it on GPU memory.

Table content: <table/>

Test accuracy (%) of student networks on CIFAR100 of combining distillation methods with KD; we check the compatibility of our objective with KD as well as PKT. ↑ denotes outperformance over KD and ↓ denotes underperformance. 35.6 70.2 34.3 74.8 40.2 71.6 35.7 73.5 40.1 CRD+KD 72.2 35.5 70.5 34.1 74.6 39.3 70.7 34.6 73.1 39.9

Table content: <table><tr><td>Teacher</td><td>WRN-40-2</td><td colspan="2">resnet110</td><td colspan="2">resnet32x4</td><td colspan="2">vgg13</td><td>vgg13</td><td/><td>ResNet50</td><td>resnet32x4</td></tr><tr><td>Student</td><td>WRN-16-2</td><td colspan="2">resnet20</td><td/><td>resnet8x4</td><td>vgg8</td><td/><td colspan="2">MobileNetV2</td><td>vgg8</td><td>ShuffleNetV2</td></tr><tr><td>Teacher</td><td>75.61</td><td>74.31</td><td/><td/><td>79.42</td><td colspan="2">74.64</td><td>74.64</td><td/><td>79.34</td><td>79.42</td></tr><tr><td>Student</td><td>73.26</td><td>69.06</td><td/><td/><td>72.50</td><td colspan="2">70.36</td><td>64.60</td><td/><td>70.36</td><td>71.82</td></tr><tr><td>KD</td><td>74.92</td><td>70.67</td><td/><td/><td>73.33</td><td colspan="2">72.98</td><td>67.37</td><td/><td>73.81</td><td>74.45</td></tr><tr><td colspan="8">FitNet+KD 75.12 (↑) 70.67 (↑) 74.66 (↑) 73.22 (↑)</td><td colspan="2">66.90 (↓)</td><td>73.24 (↓)</td><td>75.15 (↑)</td></tr><tr><td>AT+KD</td><td colspan="7">75.32 (↑) 70.97 (↑) 74.53 (↑) 73.48 (↑)</td><td colspan="2">65.13 (↓)</td><td>74.01 (↑)</td><td>75.39 (↑)</td></tr><tr><td>SP+KD</td><td colspan="7">74.98 (↑) 71.02 (↑) 74.02 (↑) 73.49 (↑)</td><td colspan="2">68.41 (↑)</td><td>73.52 (↓)</td><td>74.88 (↑)</td></tr><tr><td>CC+KD</td><td colspan="7">75.09 (↑) 70.88 (↑) 74.21 (↑) 73.04 (↑)</td><td colspan="2">68.02 (↑)</td><td>73.48 (↓)</td><td>74.71 (↑)</td></tr><tr><td>VID+KD</td><td colspan="7">75.14 (↑) 71.10 (↑) 74.56 (↑) 73.19 (↑)</td><td colspan="2">68.27 (↑)</td><td>73.46 (↓)</td><td>74.85 (↑)</td></tr><tr><td>RKD+KD</td><td colspan="7">74.89 (↓) 70.77 (↑) 73.79 (↑) 72.97 (↓)</td><td colspan="2">67.87 (↑)</td><td>73.51 (↓)</td><td>74.55 (↑)</td></tr><tr><td>PKT+KD</td><td colspan="7">75.33 (↑) 70.72 (↑) 74.23 (↑) 73.25 (↑)</td><td colspan="2">68.13 (↑)</td><td>73.61 (↓)</td><td>74.66 (↑)</td></tr><tr><td>AB+KD</td><td colspan="7">70.27 (↓) 70.97 (↑) 74.40 (↑) 73.35 (↑)</td><td colspan="2">68.23 (↑)</td><td>73.65 (↓)</td><td>74.99 (↑)</td></tr><tr><td>FT+KD</td><td colspan="7">75.15 (↑) 70.88 (↑) 74.62 (↑) 73.44 (↑)</td><td colspan="2">66.99 (↓)</td><td>72.98 (↓)</td><td>75.06 (↑)</td></tr><tr><td>NST+KD</td><td colspan="7">74.67 (↓) 71.01 (↑) 74.28 (↑) 73.33 (↑)</td><td colspan="2">63.77 (↓)</td><td>71.74 (↓)</td><td>75.24 (↑)</td></tr><tr><td>CRD</td><td colspan="7">75.48 (↑) 71.46 (↑) 75.51 (↑) 73.94 (↑)</td><td colspan="2">69.73 (↑)</td><td>74.30 (↑)</td><td>75.65 (↑)</td></tr><tr><td>CRD+KD</td><td colspan="7">75.64 (↑) 71.56 (↑) 75.46 (↑) 74.29 (↑)</td><td colspan="2">69.94 (↑)</td><td>74.58 (↑)</td><td>76.05 (↑)</td></tr><tr><td colspan="8">CRD+PKT 75.91 (↑) 71.65 (↑) 75.90 (↑) 74.57 (↑)</td><td colspan="2">69.59 (↑)</td><td>74.68 (↑)</td><td>75.94 (↑)</td></tr><tr><td>Teacher</td><td colspan="2">WRN-40-2</td><td/><td colspan="2">resnet56</td><td colspan="2">resnet32x4</td><td colspan="2">vgg13</td><td colspan="2">resnet32x4</td></tr><tr><td>Student</td><td colspan="2">WRN-16-2</td><td/><td colspan="2">resnet20</td><td colspan="2">resnet8x4</td><td colspan="2">MobileNetV2</td><td colspan="2">ShuffleNetV2</td></tr><tr><td>dataset</td><td>STL</td><td>TI</td><td colspan="2">STL</td><td>TI</td><td>STL</td><td>TI</td><td>STL</td><td>TI</td><td>STL</td><td>TI</td></tr><tr><td>Vanilla</td><td colspan="11">69.7 33.7 69.1 30.4 71.8 36.7 64.2 29.1 65.1 28.9</td></tr><tr><td>KD</td><td colspan="11">70.9 33.9 69.1 32.0 71.8 36.2 66.3 29.9 69.5 33.4</td></tr><tr><td>FitNet</td><td colspan="11">70.3 33.5 68.1 29.5 73.5 40.2 66.9 31.4 71.7 36.2</td></tr><tr><td>AT</td><td colspan="11">70.7 34.2 70.1 31.0 74.1 39.2 66.3 28.4 72.5 37.3</td></tr><tr><td>SP</td><td colspan="11">71.3 34.1 68.0 30.2 72.9 37.6 67.8 31.6 68.8 33.4</td></tr><tr><td>CC</td><td colspan="11">70.5 33.7 68.6 30.4 72.1 37.1 66.3 30.6 69.9 34.4</td></tr><tr><td>VID</td><td colspan="11">71.0 34.6 69.9 32.4 73.9 39.6 66.1 31.1 71.4 36.5</td></tr><tr><td>RKD</td><td colspan="11">71.6 35.1 70.6 32.6 73.9 38.8 68.4 32.9 71.5 37.8</td></tr><tr><td>PKT</td><td colspan="11">71.6 34.8 69.1 31.6 73.5 38.0 69.5 33.5 70.7 36.4</td></tr><tr><td>AB</td><td colspan="11">70.8 32.2 68.5 29.7 73.4 37.6 67.6 31.0 71.3 36.2</td></tr><tr><td>FT</td><td colspan="11">71.7 34.7 70.9 33.5 75.2 40.4 68.9 32.5 72.9 38.7</td></tr><tr><td>FSP</td><td colspan="8">69.6 33.6 68.1 30.4 72.4 37.1 n/a</td><td>n/a</td><td>n/a</td><td>n/a</td></tr><tr><td>NST</td><td colspan="11">70.2 32.6 68.5 30.1 73.9 38.7 64.9 27.8 72.3 37.8</td></tr><tr><td>CRD</td><td>71.6</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table/>

Peer review (iclr2020):
Summary & Pros - This paper proposes a well-principled distillation method based on contrastive loss maximizing the mutual information between teacher and student models. - This paper provides extensive experiments that demonstrate the effectiveness of the proposed method. The performance gap compared to the existing distillation approaches seem to be significant. Major Concerns: - The authors claim that *none of the other methods consistently outperform KD on their own*. I feel that this claim is somewhat aggressive because some of them outperform KD without combining with KD, e.g., Table 1 in FT (Kim et al., 2018) and Table 2 in SP (Tung & Mori, 2019). Since the distillation (or transfer) methods are typically sensitive to hyperparameters (e.g., architecture types, transfer connections between layers), I also wonder how to set the hyperparameters for baselines, especially in Table 2, because choosing the transfer connections between different architectures is very important when using feature-based methods such as FitNet and AT. - Moreover, the baselines are developed for improving distillation performance, not replacing KD. Especially, feature-based methods (e.g., FitNet, NST) are easily combined with logit-based ones (e.g., KD). So I think the compatibility between the proposed and exisiting methods should be checked. However, in this paper, only Table 4 shows the compatibility with KD (CRD+KD). - The authors compare the proposed method with only KD, AT, FitNet except Table 1-3. For example, when evaluating the transferability (Table 4), why other baselines are not compared? - VID also maximizes MI between penultimate layers. What is the key difference and why CRD perform better? I think detailed verfication should be provided in the paper. Minor Comments - Comparison with Online-KD is unfair because it does not use pre-trained ResNet32. - Why only use penultimate layers? CRD between intermediate layers is also available like VID. - A result in Table 1 is missing (FSP WRN40-2 -> WRN40-1). - In Section 4, both CKD (contrastive knowledge distillation) and CRD (contrastive representation distillation) are used, so one of them should be removed. - In Section 4.1 Transferability paragraph, *Table 3* should be changed to *Table 4*. - On page 4, *space* after *before the inner product.* should be removed. I think the proposed method is well-principled and provides meaningful improvements on various distillation settings, thus this paper seems to be above the borderline. It would be better if additional supports for the above concerns is provided in a rebuttal.