Contrastive Representation Distillation
Authors: Yonglong Tian
Dilip Krishnan
Phillip Isola
Venue:  (2020)
Keywords: Knowledge Distillation
Representation Learning
Contrastive Learning
Mutual Information

Abstract:   Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. When combined with knowledge distillation, our method sets a state of the art in many transfer tasks, sometimes even outperforming the teacher network.

Published as a conference paper at ICLR 2020
CONTRASTIVE REPRESENTATION DISTILLATION
Yonglong Tian
Dilip Krishnan Google Research di
Phillip Isola
MIT CSAIL
MIT CSAIL
Nonglong@mit.ees
com
phillip phillipi@mit edu
ABSTRACT
Often we wish to transfer representational knowledge from one neural network another. Examples include distilling a large network into smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which train student to capture significantly more information in the teacher's 's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that resulting objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation.
INTRODUCTION
Knowledge distillation (KD) transfers knowledge from one deep learning model (the teacher) to another (the student). The objective originally proposed by Hinton et al. (2015) minimizes the KL divergence between the teacher and student outputs. This formulation makes intuitive sense when the output is distribution, e.g., probability mass function over classes. However, often we instead wish to transfer knowledge about a representation. For example, in the problem of "cross-modal distillation", we may wish to transfer the representation of an image processing network to sound (Aytar et al., 2016) or to depth (Gupta et al., 2016) processing network, such that deep features for an image and the associated sound depth features are highly correlated. In such cases, the KL divergence is undefined.
Representational knowledge is structured the dimensions exhibit complex interdependencies. The original KD objective introduced in (Hinton et al., 2015) treats all dimensions as independent, conditioned the input. Let be the output of the teacher and y s be the output of the student. Then the original KD objective function, t. has the fully factored form: v y² = E. dx(y y Such factored objective is insufficient for transferring structural knowledge, i.e. dependencies between output dimensions i and j. This is similar to the situation in image generation where an Le objective produces blurry results, due independence assumptions between output dimensions.
To overcome this problem, we would like an objective that captures correlations and higher order output dependencies. To achieve this, in this paper we leverage the family of contrastive objectives (Gutmann & Hyvärinen, 2010; Oord et al., 2018; Arora et al., 2019; Hjelm et al., 2018). These objective functions have been used successfully in recent years for density estimation and representa- tion learning, especially in self-supervised settings. Here we adapt them to the task of knowledge distillation from one deep network to another. We show that important to work in representation space, similar to recent works such as Zagoruyko & Komodakis (2016a); Romero et al. (2014). However, note that the loss functions used in those works do not explicitly try to capture correlations or order dependencies in representational space.
www.c//////o.eeeee hwamm/oobbittin wcw://obbbttoog///eeeeeee wtw.ab/ttinonn//sssstiiii. In particular, in (Hinton et al., 2015), (i(a, b) loogbb Vi
Code: http:/

---

Published as conference paper at ICLR 2020
(x))
(x,)
f (zz)
f
f
F
I E X
E y Student
Teacher
Student
Teacher
Teachers
Student
(a) Model compression
(b) Cross-modal transfer
(c) Ensemble distillation
The three distillation settings we consider: (a) compressing a model, (b) transferring knowledge from one modality (c.g., RGB) to another (c.g., depth), (c) distilling an ensemble of nets into single network. The constrastive objective encourages the teacher and student to map the same input close representations (in metric space), and different inputs distant representations,
Figure I:
indicated in the shaded circle.
Our objective maximizes lower-bound to the mutual information between the teacher and student representations. We find that this results in better performance several knowledge transfer tasks. We conjecture that this because the contrastive objective better transfers all the information the teacher's representation, rather than only transferring knowledge about conditionally independent output class probabilities. Somewhat surprisingly, the contrastive objective even improves results the originally proposed task of distilling knowledge about class probabilities, for example, compressing a large CIFAR 100 network into a smaller one that performs almost as well. We believe this is because the correlations between different class probabilities contains useful information that regularizes the learning problem. Our paper forges connection between two literatures that have evolved mostly independently: knowledge distillation and representation learning. This connection allows us to leverage strong methods from representation learning to significantly improve the SOTA on knowledge distillation. Our contributions are:
1. A contrastive-bars based objective for transferring knowledge between deep networks.
2. Applications model compression, cross-modal transfer, and ensemble distillation.
3. Benchmarking 12 recent distillation methods; CRD outperforms all other methods, c.g.. 57% average relative improvement over the original KD (Hinton et al., 2015) 1, which, surprisingly, performs the second best.
RELATED WORK
The seminal work of Buciluă et al. (2006) and Hinton et al. (2015) introduced the idea of knowledge distillation between large, cumbersome models into smaller, faster models without losing too much generalization power. The general motivation was that training time, the availability of computation allows "slop" in model size, and potentially faster learning. But computation and memory constraints inference time necessitate the of smaller models. Buciluă et al. (2006) achieve this by matching output logits; Hinton et al. (2015) introduced the idea of temperature in the softmax outputs to better represent smaller probabilities in the output of single sample. These smaller probabilities provide useful information about the learned representation of the teacher model; some tradeoff between large temperatures (which increase entropy) or small temperatures tend to provide the highest transfer of knowledge between student and teacher. The method in (Li et al., 2014) was also closely related to (Hinton al., 2015).
Attention transfer (Zagoruyko & Komodakis, 201 focuses on the features maps of the network as opposed the output logits. Here the idea is elicit similar response patterns in the teacher and student feature maps (called "attention"), However, only feature maps with the same spatial
Acciun-AAciua Acska -Acct where Acc Accia and Accian represent the accuracies of CRD, KD, and vanilla training of the i-th student model, respectively.
Average relative improvement EN
2

---

Published as a conference paper at ICLR 2020
resolution can be combined in this approach, which is a significant limitation since it requires student and teacher networks with very similar architectures. This technique achieves state of the art results for distillation (as measured by the generalization of the student network). FitNets (Romero et al., 2014) also deal with intermediate representations by using regressions to guide the feature activations of the student network. Since Zagoruyko & Komodakis (2016a) do a weighted form of this regression, they tend to perform better. Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018 88; Yim al., 2017; Huang Wang, 2017; Ahn al., 2019; Koratana et al., 2019) have enforced various criteria based on representations. The contrastive objective we use in this paper is the same as that used in CMC (Tian al.. 2019). But we derive from a different perspective and give a rigorous proof that our objective is a lower bound on mutual information. Our objective is also related to the InfoNCE and NCE objectives introduced in (Oord et al., 2018; Gutmann Hyvärinen, 2010). Oord et al. (2018) use contrastive learning in the context of self-supervise representations learning. They show that their objective maximizes a lower bound on mutual information. A very related approach is used in (Hjelm et al., 2018). InfoNCE and NCE are closely related but distinct from adversarial learning (Goodfellow et al., 2014). In (Goodfellow, 2014), it is shown that the NCE objective of Gutmann & Hyvärinen (2010) can lead to maximum likelihood learning, but not the adversarial objective.
3 METHOD
The key idea of contrastive learning is very general: learn a representation that is close in some metric space for "positive" pairs and push apart the representation between "negative" pairs. Fig. 1 gives visual explanation for how we structure contrastive learning for the three tasks we consider: model compression, cross-modal transfer and ensemble distillation.
3.1 CONTRASTIVE Loss
Given two deep neural networks, a teacher fT and student fs Let be the network input; denote representations at the penultimate layer (before logits) as ff (x) and (s(x) respectively. Let I; represent a training sample, and Iÿ another randomly chosen sample. We would like to push closer the representations (x:) and f (ax)) while pushing apart (x;:) and f (x;;) For ease of notation, we define random variables S and T for the student and teacher's representations of the data respectively:
~~ Pdata (I)
data
5 = 50(())
student's representation
(2)
T= f-x)
4 teacher's representation
Intuitively speaking, we will consider the joint distribution p(S, T) and the product of marginal distributions (TS)) so that, by maximizing KL divergence between these distributions, we can maximize the mutual information between student and teacher representations. To setup an appropriate loss that can achieve this aim, let define a distribution q with latent variable which decides whether tuple (f" (xx)), f (xx,) was drawn from the joint (C = 1) or product of marginals (C = 0):
q STC 1) - P(T, QT,S p(T)pp pS))
Now, suppose in our data, we are given 1 congruent pair (drawn from the joint distribution, i.e. the same input provided to T and S) for every N incongruent pairs (drawn from the product of marginals; independent randomly drawn inputs provided to T and S). Then the priors on the latent C are:
N
qCC N+1
(C=
(5)
N +
By simple manipulation and Bayes' rule, the posterior for class C = 1 is given by:
STC C) 1)
q(c = 1[T,S) =
(6)
+ 0)
=
P(T,S)
(7)
PTT, + (p(T))(S)
3

---

Published as a conference paper at ICLR 2020
Next, we observe connection to mutual information as follows:
D(T,S) (C 1) log p(T,5) NN((Tp(S)
log(1 N P( (T)(p))
logg(N) + log P(T,S)
(8)
P(T.S)
p(T)p(S)
Then taking expectation on both sides w.r.t. p(T,S) (equivalently w.r.t. g(T, 1)) and rearranging, gives us:
() 2 log(N) + (u(r,,)c=1) log gC = 1) 4 MI bound
where () is the mutual information between the distributions of the teacher and student embed- 1[T, 5) w.n.t the parameters of the student network increases a lower bound mutual information. However, we do know the true distribution (CC' = 1|7, S); instead we estimate it by fitting model h {T, S } - [0. 1] to samples from the data distribution g(C = 1 S), where T and represent the domains of the embeddings. We
dings. Thus maximizing ()) log 9(C
maximize the log likelihood of the data under this model (a binary classification problem):
Leriti (h) E 1i h(T, (T, log(h(T,
(10)
h" arg max Ceritic
optimal critic
(11)
We term h the critic since we will be learning representations that optimize the critic's score. Assuming sufficiently expressive h, h² (T, = q(C = 1(T, (via Gibbs' inequality; see Sec. 6.2.1
for proof), so we can rewrite Eq. 9 in terms of ht:
T(T;S) log(N) E h (T, (T)
(12)
Therefore, we see that the optimal critic is an estimator whose expectation lower-bounds mutual infor- mation. We wish to learn student that maximizes the mutual information between its representation and the teacher's s, suggesting the following optimization problem:
S arg max E.
h² (T,
(13)
fz
apparent difficulty here that the optimal critic h* depends on the current student. circumvent this difficulty by weakening the bound in (12) to:
We can
T(T;5) 2 log(N) FxCTT,(C==) h (T, S))
(oo - h" (T, (T,
(14)
-log(N> Leritic (h²) log( (N) ))= max (h)
(15)
2 log(N) + (h)
(16)
The first line comes about by simply adding (E (T, (T) - This term is strictly negative, so the incquality holds. The last line follows from the fact that Cenite (h*)
to the bound in (12).
upper-bounds Levite (h). Optimizing (15) w.r.t. the student we have:
S = arg max max Lentie (h)
4 our final learning problem (17)
88
arg max max E Eo +N NE, h(T,S))
(18)
1s
which demonstrates that we may jointly optimize f at the same time we learn h. We note that due (16), fsa arg max Lertte (h), for any h, also is a representation that optimizes lower-bound (a weaker one) mutual information, so our formulation does not rely on h being optimized perfectly.
We may choose to represent h with any family of functions that satisfy h (T.S) - [0, 1). In practice, we use the following:
JTTY h(T,)) 6 (TY
(19)
N M
where M is the cardinality of the dataset and 7 is temperature that adjusts the concentration level. In practice, since the dimensionality of S and T may be different, g and g" linearly transform them
4

---

Published as a conference paper at ICLR 2020
into the same dimension and further normalize them by C-2 norm before the inner product. The form of Eq. (18) is inspired by NCE (Gutmann & Hyvärinen, 2010; Wu et al., 2018). Our formulation is similar to the InfoNCE loss (Oord al., 2018) in that we maximize lower bound on the mutual information. However we use a different objective and bound, which in our experiments we found to be more effective than InfoNCE.
Implementation. Theoretically, larger N in Eq. 16 leads to tighter lower bound on MI. In practice, avoid using very large batch size, we follow Wu et al. (2018) and implement memory buffer that stores latent features of each data sample computed from previous batches. Therefore, during training we can efficiently retrieve a large number of negative samples from the memory buffer.
3.2 KNOWLEDGE DISTILLATION OBJECTIVE
The knowledge distillation loss was proposed in Hinton et al. (2015). In addition to the regular cross-entropy loss between the student output y and one-hot label y, it asks the student network output to be as similar as possible to the teacher output by minimizing the cross- cross-entropy between their output probabilities. The complete objective is:
LKD = (1 - a) (v. + ap HD)(2"/))
(20)
where p the temperature, a a balancing weight, and o softmax function. (( / is further decomposed into K [<a(=" /p)lo(=>> (pp))
(p).o(211p))
/p)) and a constant entropy H(o /p)).
3.3 CROSS-MODAL TRANSFER LOSS
In the cross-modal transfer task shown in Fig. 1(b), a teacher network is trained on a source modality X' with large-scale labeled dataset. We then wish to transfer the knowledge to student network, but adapt it to another dataset or modality y. But the features of the teacher network are still valuable to help with learning of the student another domain. In this transfer task, loss Eq. 10 to match the features of the student and teacher. Additionally, we also consider other distillation objectives. such as KD discussed in previous section, Attention Transfer Zagoruyko & Komodakis (201 6a) and FitNet Romero et al. (2014). Such transfer is conducted on paired but unlabeled dataset D {(z,, y; 1, .... L, x2 E X, yi v). this scenario, there is no true label of such data for the original training task on the source modality, and therefore we ignore the H (y, y term all objectives that we test. Prior cross-modal work Aytar et al. (2016); Hoffman et al. (201 6b;a) uses either L2 regression or KL-divergence.
the contrastive
3.4 ENSEMBLE DISTILLATION LOss
In the case of ensemble distillation shown in I(c), we have M > 1 teacher networks, f and one student network fs. We adopt the contrastive framework by defining multiple pair-wise contrastive losses between features of each teacher network ff, and the student network f.. These losses are summed together to give the final loss (to be minimized):
LCRD = H(y. Σ CeriticTii,.
(21)
EXPERIMENTS
We evaluate our contrastive representation distillation (CRD) framework in three knowledge distilla- tion tasks: (a) model compression of a large network to smaller one; (b) cross-modal knowledge transfer; (c) ensemble distillation from group of teachers single student network.
Datasets (1) CIFAR-100 (Krizhevsky & Hinton, 2009) contains 50K training images with 0.5K images per class and 10K test images. (2) ImageNet (Deng et al., 2009) provides 1.2 million images from 1K classes for training and 50K for validation. (3) STL-10 (Coates et al., 2011) consists of training set of 5K labeled images from 10 classes and 100K unlabeled images, and test set of 8K images. (4) TinylmageNet (Deng al., 2009) has 200 classes, each with 500 training images and 50 validaton images. (5) NYU-Depth V2 (Silberman et al., 2012) consists of 1449 indoor images, each labeled with dense depth image and semantic map.
5

---

Published as a conference paper at ICLR 2020
Teacher Student
WRN-40-2 WRN-16-2
WRN-40-2 WRN-40-1
resnet56 resner20
resnet resnet20
resnet resnet32
resnet32x4
vgg
resnet8x4
vgg8
Teacher
75.61
75.61
72.34
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
69.06
71.14
72.50
70.36
KD
74.92
73.54
70.66
70.67
73.08
73.33
72.98
FitNet
73.58 (1)
72.24 (L)
69.21 (L)
68.99 (1)
71.06 (L)
73.50 (1)
71.02 (L)
AT
74.08 (1)
72.77 ())
70.55 (L)
70.22 (U)
72.31 (L)
73.44 (1)
71.43 (L)
SP
73.83 (1)
72.43 (L)
69.67
70.04 (J)
72.69
72.94 (1)
72.68 ())
CC
73.56 (L)
72.21
(9.63
69.48 (L)
71.48 (L)
72.97 (L)
70.71 ())
VID
74.11 (L)
73.30
70.38
70.16
72.61
73.09
71.23 ())
RKD
73.35 (L)
72.22 (L) (9.61 69.25
71.82 (L)
71.90
71.48 ())
PKT
(4)
73.45 (J) (L) 70.25 (1)
72.61
(1)64
72.88 (L)
AB
72.50 (L)
72.38 (L) (9.47
69.53 (1)
70.98 (L)
73.17
70.94 ())
FT
73.25 (1)
71.59 (U) (L)
70.22 (U) 72.37 (1)
72.86
70.58
FSP"
72.91 (1)
n/a
(9.95
70.11(()
71.89 (L)
72.62 (L)
70.23 (1)
NST
73.68 (1)
72.24 ())
69.60 (L)
69.53 (1)
71.96 (L)
73.30 (L)
71.53
CRD
75.48 (1)
74.14 (1)
71.16 (1)
71.46 (1)
73.48 (1)
75.51 (1)
73.94 (
CRD+KD
75.64 (1)
74.38 (1)
71.63 (1)
71.56 (1)
73.75 (1)
75.46 (1)
74.29 (1)
Table 1: Test accuracy (%) of student networks CIFAR100 of number of distillation methods (ours CRD); see Appendix citations other methods. denotes outperformance over KD and denotes underperformance. note that CRD is only method always outperform KD (and also outperforms all other methods). We denote by methods where we used our reimplementation based on the paper; for all other methods used nuthor-provided author-verified code. Average over 5 runs.
Teacher Student
1g MobileNerV2
ResNet50 MobileNetV2
ResNet50 vgg8
resnet32x4 ShufflcNetVi
resnet32x4 ShuffleNetV2
WRN-40-2 ShuffleNetVi
Teacher Student
74.64
79.34
79.34
79.42
79,42
75.61
64.6
64.6
70.36
70.5
71.82
70.5
KD*
67.37
67.35
73.81
74.07
74.45
74.83
FitNet"
64.14 (L)
63.16 (L)
70.69 (1)
73.59 (L)
73.54 (1)
73.73 ())
AT
59.40 (L)
58.58 (1)
71.84 (1)
71.73
72.73 (L)
73.32 ())
SP
66.30 (L)
68.08
(1)
73.48 (1)
74.56 (1)
74.52 (J)
CC
64.86
65.43
(L)
71.14
71.29 (1)
71.38 ())
VID
65.56
67.57
(L)
73.38 (1)
73.40 (L)
73.61 ())
RKD
64.52 ())
(4.43
7L)50
72.28 (L)
73.21 (L)
72.21 ())
PKT
67.13 ())
(6.52
73.01 (L)
71)10
74.69 (T)
73.89 ())
AB
66.06 (
(7)20
70.65 (L)
73.55
74.31 (L)
73.34 (
FT'
61.78 (
60.99
70.29 (L)
71.75
72.50 (L)
72.03 ())
NST
58.16 (L)
(L)
71.28 (1)
74)12
74.68 (1)
74.89 (1)
CRD
69.73 (1)
69.11
74.30 (1)
75.11
75.65 (1)
76.05 (1)
CRD+KD
69.94 (1)
69.54 (1)
74.58 (1)
75.12 (1)
76.05 (1)
76.27 (1)
Table 2:
Top-1 test accuracy (%) of student networks on CIFAR 100 of numher of distillation methods (ours is CRD) for transfer across very different teacher and student architectures. CRD outperforms KD and all other methods. Importantly, some methods that require very similar student and teacher architectures perform quite poorly. E.g. FSP (Yim et al., 2017) cannot even be applied: (Ba & Caruana. 2014) FitNet (Zagoruyko Komodakis, 201 perform very poorly etc. We denote by methods where we used our reimplementation based on the paper, for all other methods we used author--rovide
author-verifie code. Average over 3 runs.
4.1 MODEL COMPRESSION
Setup We experiment CIFAR-100 and ImageNet with student-teacher combinations of various capacity, such as ResNet (He et al., 2016) or Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016b).
Results CIFAR100 Table and Table compare top-1 accuracies of different distillation objec- tives (for details, see Section 6. Table investigates students and teachers of the same architectural style, while Table 2 focuses students and teachers from different architectures. We observe that

---

Published as a conference paper at ICLR 2020
(a) Student vanilla
(b) Student: AT
(c) Student: KD
(d) Student: ours (CRD)
The correlations between class logits of a teacher network are ignored by regular cross-entropy. Distillation frameworks use "soft targets" (Hiinton et al., 2015) which effectively capture such correlations and transfer them to the student network, leading the success of distillation. We visualize here the difference of correlation matrices of student and teacher logits, for different student networks on CIFAR- 100 knowledge distillation task: (a) Student trained without distillation, showing that the teacher and student cross consscorrclations are very different; (b) Student distilled by attention transfer (Zagoruyko & Komodakis, 2016a); showing reduced difference (see axis); (c) Student distilled by KI. divergence (Hinton et al., 2015), also showing reduced difference; (d) Student distilled by our contrastive objective, showing significant matching between student's and teacher's 's correlations. In this visualization, we use WRN-40-2 as teacher and WRN-40-1 as student.
Figure 2:
Teacher
Student
AT
KD
SP
CC
Online KD
CRD
CRD+KD
Top-1 1 T
26.69
30.25
29.30
29.34
29.38
30.04
29.45
28.83
28.62
8.58
10.93
10.00
10.12
10.20
10.83
10.41
9.87
9.51
Table 3: Top-1 and Top-S error rates (%) of student network ResNet- 18 ImageNet validation set. We use ResNet-34 released by PyTorch team as our teacher network, and follow the standard training practice of ImageNet on PyTorch except that we train for 10 more epochs. We compare our CRD with KD (Hinton et al., 2015), AT (Zagoruyko& & Komodakis, 2016a) and Online- KD (Lan et al., 2018). "*" reported by the original paper Lan et al. (2018) using an ensemble of online ResNets teacher, pretrained ResNet-34 was used.
loss, which we call CRD (Contrastive Representation Distillation), consistently outperforms all other distillation objectives, including the original KD (an average relative improvement of 57%). Surpris- ingly, we found that KD works pretty well and none of the other methods consistently outperforms KD on their own. Another observation is that, while switching the teacher student combinations from same to different architectural styles, methods that distill intermediate representations tend to perform worse than methods that distill from the last several layers. For example, the Attention Transfer (AT) and FitNet methods even underperform the vanilla student. In contrast, PKT, SP and CRD that operate on last several layers performs well. This might because, architectures different styles have their own solution paths mapping from the input to the output, and enforcing the mimic of intermediate representations thus might conflict with such inductive bias.
Capturing inter-class correlations. In Fig. 2, compute the difference between the correlation matrices of the teacher's and student's logits; for three different students: vanilla student without distillation, trained by AT, KD or CRD (our method). It is clear that the CRD objective captures the most correlation structure in the logit as shown by the smaller differences between teacher and student. This is reflected in reduced error rates.
Results on ImageNet For a fair comparison with Zagoruyko & Komodakis (201 and Lan et al. (2018), we adopt the models from these papers, ResNet-34 the teacher and 18 the student. As shown in Table 3, the gap of top-1 accuracy between the teacher and student is 3.56%. The AT method reduces this gap by 0.95%, while ours narrow it by 1.42%, 50% relative improvement. Results on ImageNet validates the scalability of our CRD.
Transferability of representations We are interested in representations, and primary goal of representation learning is to acquire general knowledge, that is, knowledge that transfers to tasks or datasets that were unseen during training. Therefore, we test the representations we distill transfer well. A WRN-16-2 student either distills from a WRN-40-2 teacher, or is trained from scratch on CIF The student serves as a frozen representation extractor (the layer prior to the logit) for images from STL-10 or TinyImageNet (all images downsampled to 32x32). We then train a linear classifier to perform 10-way (for STL-10) or 200-way (for Tiny ImngeeegNee) classification to quantify the transferability of the representations. We compare CRD with multiple baselines such as KD and
7

---

Published as a conference paper at ICLR 2020
Student
KD
FitNet
CRD
CRD+KD
Teacher
(IFARIO--STL-1 -innRooo-TiyyllmaeeNee
69.7
70.9
70.7
70.3
71.6
72.2
68.6
33.7
33.9
34.2
33.5
35.6
35.5
31.5
Table 4: We transfer the representation learned from CIFARIOO 100 to STL-10 and ImygeeeeNet datasets by freezing the network and training linear classifier top of the last feature layer perform 10-way (STL-10) or 200-way (Tinylmage (inylmageNet) classification. For this experiment, we use the combination of teacher network WRN-40-2 and student network WRN-16-2. Classification accuracies (%) are reported.
Ransom in!.
Uict
S2:
C10
STL:10 Accuracy 199)
STL 10 Accurscy
FiNeet, ResNet50
KD, ResNer50
30
KD-AT. ResNet50
Reshet50
Residual Block ID
(a) Accuracy on STL-10 with linear probing
(b) Accuracy on STL-10 with fully finetuning
Top-1 classification accuracy on STI-10 using chrominance image (ab channel in Lab color space). We initialize the chrominance network randomly or by distilling from a luminance network, trained with large-scale labeled images. We evaluate distillation performance by (a) linear probing and (b) fully finetuning.
Figure 3:
AT in Table 4. In this setting, all distillation methods except FitNet improve transferability of the learned representations both STL-10 and TinylmageNet. While the teacher performs the best on the original CIFAR 100 dataset, its representations transfer the worst to the other two datasets. This is perhaps the teacher's representations are biased towards the original task. Surprisingly, the student with CRD+KD distillation not only matches its teacher on CIFAR (0 (see Table 4), but also transfers much better than the teacher, e.g., 3.6% improvement (STL-10) and 1% on TinyImageNet.
4.2 CROSS-MODAL TRANSFER
We consider practical setting where modality X' has large amount of labeled data while modality ) does not. Transfering knowledge from X' to is a common challenge. For example, while large-scale RGB datasets are easily accessible, other modalities such as depth images are much harder to label at scale but have wide applications. We demonstrate the potential of CRD for cross-modal transfer in two scenarios: (a) transfer from luminance to chrominance; (b) transfer from RGB to depth images.
Transferring from Luminance to Chrominace. We work on Lab color space, where represents Luminance and ab Chrominance. We first train an L network on TinylmageNet with supervision. Then we transfer knowledge from this L network to ab network on the unlabeled set of STL-10 with different objectives, including FitNet, KD, KD+AT and CRD). For convenience, architecture for student and teacher (they can also be different). Finally, we evaluate the knowlege of ab network by two means: (1) linear probing: we freeze the ab network and train a linear classifier on top of features from different layers to perform 10-way classification on STL-10 ab images. This a common practice (Alain & Bengio, 2016; Zhang al., 2017) evaluate the quality of network representations; (b) fully finetuning: we fully finetune the ab network to obtain the best accuracy. We also use as a baseline the ab network that randomly initialized rather than distilled. Architectures investigated include AlexNet, ResNet- 8 and ResNet-50. The results shown in Figure 3 show that CRD is more efficient for transferring inter-modal knowledge than other methods. Besides, we also note KD+AT does not improve upon KD, possibly because attention of luminance and chrominance are different and harder to transfer.
the same
Transferring from RGB to Depth. We transfer the knowledge of ResNet-18 teacher pretrained on ImageNet to a 5-layer student CNN operating on depth images. We follow a similar transferring procedure on NYU-Depth training set, except that global features, proposed by Hjelm et al. (2018), to overcome the problem of insufficient data samples in the depth domain. Then the student network is further trained predict semantic segmentation
trick of contrasting between local and
8

---

Published as a conference paper at ICLR 2020
Metric (%)
Random Init.
KD
KD+AT
FitNet
CRD
Pix. Acc.
56.4
58.9
60.1
60.8
61.6
mloU
35.8
38.0
39.5
40.7
41.8
Table 5:
Performance on the task of using depth to predict semantic segmentation labels. We initialize the depth network either randomly or distilling from ImageNet pre-pretrained ResNet-18 18 teacher.
25.25
KD, WRN-16-2
KD, resnet20
25.00
CKD, WRN-16-2
CRD, resnet20
24.75
Error Rate
24.50
24.25
23.0
24.00-
285-
23.75
23.50-
Number Ensembled Teachers
Number of Ensembled Teachers
(8) WideRResNet16622
(b) ResNel20
Figure 4: Distillation from an ensemble of teachers. We vary the number of ensembled teachers and compare KD with our CRD by using
WRN- TRN-16-22 and (b) ResNet20. Our CRD consistently achieves lower error rate.
maps from depth images. We note that both knowledge transfer and downstream training are conducted on the same set images, i.e., the training set. Table reports average pixel prediction across all classes. All distillation methods can transfer knowledge from the RGB ResNet to the depth CNN. FitNet surpasses KD and KD+AT. CRD significantly outperforms all other methods.
accuracy and mean Intersection-over-nniie
4.3 DISTILLATION FROM ENSEMBLE
Better classification performance is often achieved by ensembles of deep networks, but these usually too expensive for inference time, and distillation into a single network is desirable task. We investigate the KL-divergence based KD and CRD for this task, using the loss of Sec. 21. The network structures of each teacher and student are identical here, but an ensemble of multiple teachers can still provide rich knowledge for the student. To compare between KD and CRD on CIFAR100 dataset, we use WRN-16-2 and ResNet-20. 20. whose single model етrоr rates are 26.7% and 30.9% respectively. The results of distillation are presented in Figure 4, where we vary the number of ensembled teachers. CRD with 8 teachers decreases the error rate of WRN- 16-2 to 23.7% and ResNet20 to 28.3%. In addition, CRD works consistently better than KD in all settings we test. These observations suggest that CRD is capable of distilling ensemble of models into single which performs significantly better than a model the same size that is trained from scratch.
4.4 ABLATIVE STUDY
InfoNCE Ours. InfoNCE (Oord et al., 2018) is alternative contrastive objective which select a single positive out from set of distractors via a softmax function. We compare InfoNCE with our contrastive objective Eq 18, when using the same number of negatives. The last rows in Table show that our objective outperforms InfoNCE in 4 out of 5 teacher-student combinations.
Negative Sampling. In this paper, we consider (1) f for the unsupervised case when we have no labels. or (2) Iy for supervised case, where I represents the label associated with sample æ;. One might imagine that the first sampling strategy will increase the intra-class variance as we may push apart positives and negatives from the same underlying class, while the second would not. We quantitatively measure and report the gap between these two strategies in Table 6, which shows that the classification accuracy by the second sampling strategy is 0.81 % higher for objective and 0.62% higher for InfoNCE than the first one.
negative sampling plocies when giving anchor

---

Published as a conference paper at ICLR 2020
WRN-40-2 WRN-16-22
resnet resnet20
resnet resnet32
resnet32x4 resnet8x4
vgg
sampling
objective
vgg8
InfoNCE
74.78
70.56
72.67
74.69
73.24
+
Ours
74.48
70.64
72.64
74.67
73.39
InfoNCE
75.15
71.39
73.53
75.22
73.74
Vi # Vj
Ours
75.48
71.46
73.48
75.51
73.94
Table Ablative study of different contrastive objectives and negative sampling policies on CIFARIO0. For contrastive objectives, we compare our objective with InfoNCE (Oord al., 2018) ; For negative sampling policy, when given anchor image from the dataset, consider either randomly sample negative zy such that (a) J. (b) yu # y where y represents the class label. Average over runs.
T: WRN WAN-40-2, 2, S: WAN-WAN- 16-2
26.25
T: WRN WRN-40 S: WRN 16-2
Eroo Rate (N6
Eron Rate (86)
26.00
25.75
5.35
Number of Negatives in NCE
Temperature NCE
(a) Effects varying
(b) Effects varying
Figure 5: Effects of varying the number of negatives, shown (a), or the temperature, shown in (b),
4.5
AND COMPUTATION OVERHEAD
There are two main hyper-parameters in our contrastive objectives: (1) the number of negative samples N in Eq. 18, and (2) temperature which modulates the softmax probability. We adopt WRN-40-2 as teacher and WRN-16-2 as student for parameter analysis. Experiments are conducted on CIFAR100 and the results shown in Figure 5.
Number of negatives N We have validated different N: 16, 64, 256, 1024, 4096, 16384. As shown in Figure 5(a), increasing N leads to improved performance. However, the difference of error rate between = 4096 and = 16384 is less than 0.1 1.1. Therefore, we use 16384 for reporting the accuracy while in practice N 4096 should suffice.
Temperature 7 We varied r between 0.02 and 0.3. As Figure 5(b) illustrates, both extremely high or low temperature lead sub-optimal solution. In general, temperatures between 0.05 and 0.2 work well on CIFAR 100. All experiments but those on ImageNet use a temperature of 0.1. For ImageNet, we use 7 0.07. The optimal temperature may vary across different datasets and require further tuning.
Computational Cost We use ResNet- 18 ImageNet for illustration. CRD uses extra 260 MFLOPs, which is about 12% of the original 2 GFLOPs. In practice, we did not notice significant difference of training time on ImageNet (e.g.. 1.75 epochs/hour v.s. 1.67 epochs/hour on two Titan-V GPUs). The memory bank for storing all 128-d features of ImageNet only costs around 600MB memory, and therefore we store on GPU memory.
CONCLUSION
We have developed a novel technique for neural network distillation, using the concept contrastive objectives, which are usually used for representation learning. We experimented with our objective on a number of applications such as model compression, cross-modal transfer and ensemble distillation, outperforming other distillation objectives by significant margins in all these tasks. Our contrastive objective is the only distillation objective that consistently outperforms knowledge distillation across wide variety of knowledge transfer tasks. Prior objectives only surpass KD when combined with KD. Contrastive learning a simple and effective objective with practical benefits.
Acknowledgments. We thank Baoyun Peng for providing the code of CC (Peng al., 2019) and Frederick Tung for verifying our reimplementation of SP (Tung Mori, 2019). This research was supported in part by Google Cloud and ilytte..
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9163-9171, 2019. 3, 13, 15
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 2016. 8
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: /90229922... 2019. 1
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 2016. 1.5
Jimmy Ba and Rich Caruana. Do deep nets really need to deep? Advances neural information processing systems, 2014. 6
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541. ACM, 2006.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-22 223, 2011. 5
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014. 3
lan J Goodfellow. On distinguishability criteria for estimating generative models. arXiv preprint arXiv:1412.6511, 2014.
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2827-2836. 2016. 1
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010. 3,5
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. 6, 14
Bycongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3777-3787, 2019. 13, 15
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255/ 025331, 2015. 2, 7, 13, 15, 16
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Maree Marchildon Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 066700 2018 3, 8
Judy Hoffman, Saurabh Gupta, and Trevor Darrell. Learning with side information through modality hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2016a. 5
11

---

Published as a conference paper at ICLR 2020
Judy Hoffman, Saurabh Gupta. Jian Leong. Sergio Guadarrama, and Trevor Darrell. Cross-modal adaptation for rgb-d detection. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp- 5032-5039, IEEE, 2016b. 5
Zehao Huang and Naiyan Wang- Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: (007:0/777.121/ 2017. 3, 13, 15
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Advances in Neural Information Processing Systems, pp. 2760-2769, 2018.
3, 13, 15
Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: Learned intermediate representation training for model compression. In International Conference on Machine Learning, 2019. 3
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 5
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In Advances in neural information processing systems, 2018.
Jinyu Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size dnn with output- distribution-based criteria. In Fifteenth annual conference of the international speech communica- tion association, 2014. 2
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807 2018. I, 3, 9, 10
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3967-3976, 2019. 13, 15
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-284, 2018. 13,
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaon- ing Zhang. Correlation congruence for knowledge distillation. arXiv preprint arXiv: 01802, 2019, 10, 13, 15
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 141.6554,,,6550 2014. 1, 3,3, 5,
13, 15
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 14
Chieh Chen. Mo-
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. European Conference on Computer Vision, 2012. 5
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014. 14
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 190.:00999,, 2019. 3
Frederick Tung and Greg Mori. arXiv:1907.0900 2019. 10, 13, 15
knowledge distillation. arXiv preprint
12

---

Published as a conference paper at ICLR 2020
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 5
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017. 3, 6, 13, 15
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.03928, 2016a. 1, 2, 3, 5, 6, 7, 13, 15, 16
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016b. 6, 14
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 8
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 6848-6856, 2018a. 14
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018b. 15, 18
APPENDIX
6.1 OTHER METHODS
We compare to the following other state-of-the-ar methods from the literature:
1. Knowledge Distillation (KD) (Hinton et al., 2015)
2. Fitnets: Hints for thin deep nets (Romero et al., 2014)
3. Attention Transfer (AT) (Zagoruyko & Komodakis, 2016a)
4. Similarity-Preservinng Knowledge Distillation (SP) (Tung & Mori, 2019);
5. Correlation Congruence (CC) (Peng al., 2019)
6. Variational information distillation for knowledge transfer (VID) (Ahn et al., 2019)
7. Relational Knowledge Distillation (RKD) (Park al., 2019)
8. Learning deep representations with probabilistic knowledge transfer (PKT) (Passalis & Tefas, 2018)
9. Knowledge transfer via distillation of activation boundaries formed by hidden neurons (AB) (Heo al., 2019)
10. Paraphrasing complex network: Network compression via factor transfer (FT) (Kim et al., 2018)
11. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning (FSP) (Yim al., 2017)
12. Like what you like: Knowledge distill via neuron selectivity transfer (NST) (Huang & Wang, 2017)
6.2 CONTRASTIVE LOSS - DETAILS
6.2.1 PROOF THAT h (T, S) = cC
We wish to model some true distribution q(C|T = t, S s). C' a binary variable, so we can model
(C S s) Bernoulli distribution with single parameter h(S 8, T t) [0, 1].
13

---

Published as a conference paper at ICLR 2020
defining for convenience h'(C s, h(s s, = =t) and (C =,
t) = 1 - h(S = s,T t). The log likelihood function is:
Be-<(((18==7=0000g (C = c, = =
(22)
By Gibbs' inequality, the max likelihood fit (C c, = = q(C = = t). which also implies that h(S = 8, = -) 1|5 = =s,T=()
We now demonstrate that our objective in Eq. (10) is proportional summation terms Eq. (22) for all & E S and ( ET.
E, (.q(((S Bergooooesess..eaao//o -, O)
(23)
= (C c, -, 1)]]
(24)
= Bu..gg(s1/(c)))))===11p h( T = 1)]+
E
- h(S = s,T = t))l
(25)
N ( hS
[)]+
N N +
E B, P. h( s, = f))
(26)
Notice that (26) is proportional to Eq. (10) from the main paper. For sufficiently expressive h, then, each term inside the expectation in Eq. (23) can be maximized, resulting in li* (T t, S = 8) = (C = 1[T = t, t) for all and t.
6.3 NETWORK ARCHITECTURES
Wide Residual Network (WRN) (Zagoruyko & Komodakis, 2016b). WRN-d-w represnets wide resnet with depth d and width factor w.
resnet (He et al., 2016). We use resnet-d to represent cifar-style resnet with 3 groups of basic blocks, each with 16, 32, and 64 channels respectively. In our experiments, resnet8 x4 and resnet32 x4 indicate a 4 times wider network (namely, with 64, 128, and 256 channels for each of the block)
ResNet (He al., 2016). ResNet-d represents ImageNet-style ResNet with Bottleneck blocks and more channels.
MobileNetV2 Sandler et al. (2018). our experiments, we use width multiplier of 0.5.
vgg (Simonyan & Zisserman, 2014). the vgg net used in our experiments are adapted from its original ImageNet counterpart.
ShuffleNetV1 (Zhang et al., 2018a) 8a), ShuffleNetV2 (Tan et al., 2019). ShuffleNets are proposed for efficient training and we adapt them input of size 32x32.
6.4 IMPLEMENTATION DETAILS
All methods evaluated in our experiments use SGD.
For CIFAR-100, we initialize the learning rate as 0.05, and decay it by 0.1 every 30 epochs after the first 150 epochs until the last 240 epoch. For MobileNetV2, ShuffleNetVi and ShuffleNetV2, we use a learning rate of 0.01 as this learning rate is optimal for these models in grid search, while 0.05 is optimal for other models.
For ImageNet, we follow the standard PyTorch practice but train for more epochs. Batch size 64 for CIFAR-100 or 256 for ImageNet.
The student trained by combination of cross-entropy classification objective and knowledge distillation objective, shown as follows:
L Leross-entrapy &Listill
(27)
For the weight balance factor 3, we directly use the optimal value from the original paper if it is specified, or do a grid search with teacher WRN-40-2 and student WRN-16-2. This results in the following list of 3 used for different objectives:
14

---

Published as a conference paper at ICLR 2020
1. Fitnets (Romero et al., 2014): 8 100
2. AT (Zagoruyko & Komodakis, 201 8 = 1000
3. SP (Tung & Mori, 2019): 3 3000
4. CC (Peng al., 2019): 3 0.02
5. VID (Ahn et al., 2019): / =
6. RKD (Park et al., 2019): 31 25 for distance and $2 50 for angle. For this loss, we combine both term following the original paper.
7. PKT (Passalis & Tefas, 2018): 3 = 30000
8. AB (Heo et al., 2019): 0, distillation happens in separate pre-training stage where only distillation objective applies.
9. FT (Kim al., 2018): 500
10. FSP (Yim et al., 2017): 3 0, distillation happens in separate pre-training stage where only distillation objective applies.
11. NST (Huang & Wang, 2017): B 50
12. CRD: 3 0.8, in general / E [0.5, 1.5] works reasonably well.
For KD(Hinton et al., 2015), we follow Eq. 20 and set 0.9 and T = 4.
6.5 VISUALIZATION OF THE CORRELATION DISCREPANCY
We visualize the correlation discrepancy for different distillation objectives across various combi- nations of student and teacher networks. As shown in Fig. 6, our contrastive distillation objective siginificantly outperforms other objectives, in terms minimizing the correlation discrepancy be- tween student and teacher networks. The normalized correlation coefficients are computed at the logit layer.
6.6 COMBINING DIFFERENT DISTILLATION OBJECTIVES
Table 7 shows the effect of combining various objectives with KD. Most methods able to slightly outperform KD after this combination. We also the check the compatibility of our distillation objective with KD and PKT. The combination of CRD and KD/PKT further improves over single CRD objective.
6.7 ADDITIONAL RESULTS ON TRANSFERABIL.T REPRESENTATIONS
In addtion Table 4 shown in the main text, also evaluate the transferabilit of the representations of various other distillation methods and summarize the results in Table 8. In general, CRD achieves the best transferring accuracy out of 10 settings.
6.8 DEEP MUTUAL LEARNING SETTING
In (Zhang et al., 2018b), Deep Mutual Learning setting was proposed where the teacher and student networks are trained simultancously rather than sequentially. The benefit is that not only the student networks but also the teacher models will be improved. Here we investigate the possibility of incor- porating different distillation objectives into this training framework. The results are summarizied in Table 9, We observe that in general logit based distillation methods is better than non-logit based distillation methods in this setting, i.e., KD performs the best. We conjecture that during the mutual training phase, KD performs like an advanced label smoothing regularization and does not require the logit to be very accurate. But the feature-batt based distillation methods hard to learn knowledge from the teacher model when it is handicapped. On the other hand, we notice that the combination of KD and CRD leads better performance the student side, shown in the last row of Table
15

---

Published as a conference paper at ICLR 2020
(a) Student: vanilla.
(b) Student: AT
(c) Student: KD
(d) Student: ours
(CRD)
60
(1) Teacher: WRN-40-2, Student: WRN-40-1
wrggeaneessoooiiiniiii (2) Teacher: WRN-40-2, Student: WRN-16-2
(3) Teacher: ResNet-56, Student: ResNet-20
Figure 6: The correlations between class logits oulput the teacher network show the "dark knowledge" Hinton al. (2015) must transferred to a student networks. A student network captures these correlations tends to perform better at the task. We visualize here the difference between correlation matrices of the student and teacher at the logits, for different student networks on Cifar 100 knowledge distillation task: (a) A student trained without distillation; (b) A student distilled by attention transfer Zagoruyko & Komodakis (2016a) (e) A student distilled by KI. divergence Hinton al. (2015); (d) student distilled by contrastive objective. Our ohjective greatly improves the structured knowledge (correlations) the output units.
6.9 STANDARD DEVIATION FOR RESULTS ON CIFAR-100 BENCHMARK
The standard deviation over multiple runs on CIFAR-100 benchmark is provided in Table 10 for student and teacher models of the same architectural type, and in Table for student and teacher models of different architectural types.
16

---

Published as a conference paper at ICLR 2020
Teacher Student
WRN-40-2 WRN-16-2
resnet resnet20
resnet32x4 resnet8x4
vggl3
vgel3 MobileNetV2
ResNet50
resmet32x4
vgg8
vgg8
ShuffleNetV2
Teacher
75.61
74.31
79.42
74.64
74.64
79.34
79.42
Student
73.26
69.06
72.50
70.36
64.60
70.36
71.82
KD
74.92
70.67
73.33
72.98
67.37
73.81
74.45
FD
75.12 (1)
70.67 (1)
74.66 (1)
73.22 (1)
66.90 (L)
73.24 (J)
75.15 (1)
AT+KD
75.32 (1)
70.97 (1)
74.53 (1)
73.48 (1)
65.13 (L)
74.01 (1)
75.39 (1)
SP+KD
74.98 (1)
71.02 (1)
74.02 (1)
73.49
68.41 (T)
73.52 (L)
74.88 (1)
CC+KD
75.09 (1)
70.88 (1)
74.21
73.04
68.02
73.48 (J)
74.71 (1)
VID+KD
74.56 71.10 71)
71)
68.27 (1)
73.46 (L)
74.85 (1)
RKD+KD
74.89 (L) 70.77 (1)
73.79
72.97 (L)
67.87 (1)
73.51 (4)
74.55 (1)
PKT+KD
75.33 (1) 70.72 (1)
74.23
73)25
68.13
73.61 ())
74.66 (
AB+KD
70.27 (1)
70.97
74.40
(1)
68)23
73.65 (1)
74.99 (1)
FT+KD
75.15 (1)
70.88 (1)
74.62
73.44 (1)
66.99 ())
72.98 (L)
75.06 (1)
NST+KD
74.67 (L)
71.01 (1)
74.28 (1)
73.33 (1)
63.77 (L)
71.74 (L)
75.24 (1)
CRD
75.48 (1)
71.46 (1) 75.51
(1)
69.73 (1)
74.30 (1)
75.65 (1)
CRD+KD
75.64 (1)
71.56 (1)
75.46
74.29 (t)
69.94 (1)
74.58 (1)
76.05 (1)
CRD+PKT
75.91 (1)
71.65 (1)
75.90 (1)
74.57 (1)
69.59 (1)
74.68 (1)
75.94 (1)
Table Test accuracy (%) student networks on CIPAR100 combining distillation methods with KD;
we check the compatibility of our objective with KD as well as PKT. 1 denotes outperformance over KD and
denotes underperformance.
Teacher Student
WRN-40-2 WRN-16-2
resnet56 resnet20
resnct32x4 resnet8x4
vgg13
resnet32x4
MobileNetV2
ShuflleNeV2
dataset
STL
TI
STL
TI
STL
TI
STL
TI
STL
TI
Vanilla
69.7
33.7
69.1
30.4
71.8
36.7
64.2
29.1
65.1
28.9
KD
70.9
33.9
69.1
32.0
71.8
36.2
66.3
29.9
69.5
33.4
FitNet
70.3
33.5
68.1
29.5
73.5
40.2
66.9
31.4
71.7
36.2
AT
70.7
34.2
70.1
31.0
74.1
39.2
66.3
28.4
72.5
37.3
SP
71.3
34.1
68.0
30.2
72.9
37.6
67.8
31.6
68.8
33.4
CC
70.5
33.7
68.6
30.4
72.1
37.1
66.3
30.6
69.9
34.4
VID
71.0
34.6
69.9
32.4
73.9
39.6
66.1
31.1
71.4
36.5
RKD
71.6
35.1
70.6
32.6
73.9
38.8
68.4
32.9
71.5
37.8
PKT
71.6
34.8
69.1
31.6
73.5
38.0
69.5
33.5
70.7
36.4
AB
70.8
32.2
68.5
29.7
73.4
37.6
67.6
31.0
71.3
36.2
FT
71.7
34.7
70.9
33.5
75.2
40.4
68.9
32.5
72.9
38.7
FSP
69.6
33.6
68.1
30.4
72.4
37.1
n/a
n/a
n/a
n/a
NST
70.2
32.6
68.5
30.1
73.9
38.7
64.9
27.8
72.3
37.8
CRD
71.6
35.6
70.2
34.3
74.8
40.2
71.6
35.7
73.5
40.1
CRD+KD
72.2
35.5
70.5
34.1
74.6
39.3
70.7
34.6
73.1
39.9
Table We measure the transferability of the student network, by evaluating linear classifier on top of its frozen representations on STL10 (abbreviated as "STI.") and TinylmageNet (abhreviated as "TT"). The best. accuracy bolded and the second best is underlined.
17

---

Published as a conference paper at ICLR 2020
Teacher Student
WRN-40-2
WRN-40-2 WRN-40-1
resnet56 resnet20
resnetl resner20
resnetl10 resnet32
resnet32x4 resmet8x4
vgg13
ResNet50
WRN-16-2
vgg8
vgg8
Vanilla
75.61
75.61
72.34
74.31
74.31
79.42
74.64
79.34
KD
77.89
77.42
74.51
75.95
76.70
79.90
76.53
79.92
AT
76.02
76.42
72.44
73.72
73.69
79.08
73.21
79.33
SP
76.36
76.08
73.11
73.58
73.81
79.50
75.75
73.34
T
CC
76.42
76.30
71.95
72.78
74.19
79.44
74.80
77.76
CRD
77.30
77.23
73.25
75.32
74.37
79.81
75.25
78.65
CRD+KD
78.01
77.39
73.86
75.31
75.53
79.36
77.23
79.04
Vanilla
73.26
71.98
69.06
69.06
71.14
72.50
70.36
70.36
KD
74.98
73.66
70.85
70.71
73.24
74.77
73.39
74.00
AT
73.04
71.57
69.20
69.04
70.87
72.32
69.55
69.56
SP
72.82
71.47
69.45
69.31
70.75
72.63
70.57
70.66
S
CC
73.19
71.46
68.82
69.51
71.24
72.57
71.12
70.36
CRD
75.22
73.53
70.92
70.80
72.65
75.24
73.19
73.21
CRD+KD
75.89
74.12
70.90
71.60
73.07
75.34
74.08
74.22
Table accuracy (%) of student teacher networks on CIFAR100 with Deep Mutual Train- ing Zhang et al. (2018b) setting. where the teacher and student networks are trained sinultaneously rather than sequentially. We use "T" and "S" to denote the teacher and student models, respectively.
Teacher Student
WRN-40-2 WRN-16-2
WRN-40-2 WRN-40-1
resnet56 resner20
resnet resnet20
resnet1 resnet32
resnet32x4 resnet8x4
vgg13
vgg8
Teacher
75,61
75.61
72.34 69.06
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
71.14
72.50
70.36
74.92 - 0.28 73.54
+ 0.20
70.66 + 0.24
70.67 + 0.27
73.08
1 0.18
73.33
0.25
72.98
0.19
FitNet
73.58 -: 0.32 72.24
+ 0,24 69.21 + 0.36
68.99 1 0.27 71.06
1 0.13
73.50 0.28
71.02
0.31
AT
74.08 + 0.25 72.77 + 0.10 70.55 0.27 70.22 0.22+00.11
72.31
0.08
73.44
71.43
0.09
SP
73,83 0.12 72.43
0.27 69.67
0.20 0.20
0..41
72.94
72.68
0.19
CC
73.56 72.21
0.25 69.63 0.32 69.48
71.48
0.221
72.97
70.71 0.24
VID
74.11 73.30
0.13 70.38 0.14 70.16
72.61
73.09
71.23 0.06
RKD
73.35 72,22 1 0,20 69.61
0.06 69.25
1
71.90-10.11
71.48 0.05
PKT
73,45
0.19 70.34
0.04
70.25
73.44
72.88 1 0.09
AB
1.250 72.38
0.31
09.47
70,98 0.15
73.17
70,94 0
FT
73.25 0.20 71.59
0.15
69.84
0.12 0.12
72.37
7
70.58 0.08
ESP
72,91 = 0.24
n/a
69,95 + 0.21 70.11
0.11
72.62
70,23 0.23
NST
73.68 + 0.11
72.24 + 0,22
69.60 + 0.13
71.96
0.07 73.30
0.28
71.53 + 0.13
CRD
75.48 + 0.09
74.14 + 0.22 71.16 + 0.17 71.46 0.09
73.48 + 0.13 75.51 + 0.18
73.94 + 0.22
CRD+KD
75.64 + 0.21 74.38 + 0.11 71.63 + 0.15 71.56 1 0.16 73.75 1 0.24 75.46 + 0.25 74.29 + 0.12
Table 10:
Test accuracy (%) of student networks CIFAR100 of number of distillation methods (ours is CRD). Standard deviation is provided.
18

---

Published as a conference paper at ICLR 2020
Teacher Student
vgg13 MobileNetV2
ResNe150 MobileNetV2
ResNet50 vgg8
resnet32x4 ShuffleNetVi
resnet32x4 ShuffleNetV2
WRN-40-2
ShuffleNetVI
Teacher
74.64
79.34
79.34
79.42
79.42
75.61
Student
64.6
64.6
70.36
70.5
71.82
70.5
KD
67.37 + 0.32
67.35
0.32
73.81
+ 0.13
74.07 + 0.19
74.45 + 0.27
74.83 0.17
FitNet
64.14 0.50
63.16
0.47
70.69 + 0.22
73.59
0.15
73.54
0.22
73.73 0.32
AT
59.40 0.20
58.58 + 0.54
71.84 0.28
71.73
0.09
73.32 0.35
SP
66.30 0.38
68.08 0.38
73.34
0.42
0.22
74.52 = 0.24
CC
64.86 = 0.25
70.25
7.329 0.06
71.38 0.25
VID
65.56 0.42
67.57
0.28
10.30
73.38
0.11
73,61 0.12
RKD
64.52 0.45
0.42
71.50 0.07
0.39
0.221
72.21 0.16
PKT
67.13 0.30
0.33
73.01
0.14
0.25
74.69
73.89 0.16
AB
66.06 0.48
-
70.65
+ 0.09
73.55 0.31
7.131
73.34 0.09
FT
61.78 0.33
60,99
1.997..7
70.29
0.19
71.75 0.20
0.15
72.03 0.16
NST
58.16 0.26
64.96
0.44
71.28
0.13
74.12
0.19
74.68
14.6810.22 0.26
74.89
0.25
CRD
69.73 0.42
69.11
0.28
74.30
1 0.14
75.11
0.32
75.65
0.10
76.05 0.14
CRD+KD
69.94 0.05
69.54
0.39
74.58
+ 0.27
75.12 0.35
76.05
0.09
76.27 0.29
Table 11: Top-1 test accuracy (%)
student networks on CIFARI00 of number of distillation methods
(ours is CRD) for transfer across very different teacher student architectures. Standard deviation provided.
19


Reviews:
[{"review_id": "SkeiTRlTYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I do not necessarily see something wrong with the paper, but I'm not convinced of the significance (or sufficient novelty) of the approach. \n\nThe way I understand it, an independent assumption on internal representation is relaxed by capturing correlations between them and higher order dependencies between them using a different objective function(distance measure), the problem then becomes nothing but trying to minimize another distance metric between teacher and student networks on an intermediate layer. \n\nComparing with original distillation method (KD) I'm not sure how significant the improvement is. And technically this is just a distance metric between the internal representation or output of the student and teacher. Sure it is a more involved distance metric, however it is in the spirit of what the distillation work is all about and I do not see this as being fundamentally different, or at least not different enough for an ICLR paper.\n\nThe experimental results also suggest only a marginal improvement compared to other methods. It would be helpful to also include the variance of each experiment i.e., it was mentioned that the results were averaged by repeating 5 experiments to make sure the proposed method consistently better than others.  Rightnow, It is hard to compare with other approaches. The paper gives off a feeling that this method is not novel. Why this particular distance metric between the representations? Why not just L2?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyl91UHRFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines a contrastive objective measuring the mutual information between the representations learned by a teacher and a student networks for model distillation. The objective enforces correlations between the learned representations. When combined with the popular KL divergence between the predictions of the two networks, the proposed model shows consistently improvement over existing alternatives on three distillation tasks. \n\nThis is a solid work – it is based on sound principles and provides both rigorous theoretical analysis and extensive empirical evidence. I only have two minor suggestions.\n\n1, From Section 3.2 to Section 3.4, it is not clear to me that on the model compression task, are both the proposed contrastive loss and the loss in Eq. (10) used?\n\n2, The “Deep mutual learning”, Zhang et al, CVPR’18 paper needs to be discussed. I’d also like to see some experiments on the effects of training the teacher and student networks jointly from scratch using the proposed loss. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkevRLo4cB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary & Pros\n- This paper proposes a well-principled distillation method based on contrastive loss maximizing the mutual information between teacher and student models.\n- This paper provides extensive experiments that demonstrate the effectiveness of the proposed method. The performance gap compared to the existing distillation approaches seem to be significant.\n\nMajor Concerns:\n- The authors claim that \"none of the other methods consistently outperform KD on their own\". I feel that this claim is somewhat aggressive because some of them outperform KD without combining with KD, e.g., Table 1 in FT (Kim et al., 2018) and Table 2 in SP (Tung & Mori, 2019). Since the distillation (or transfer) methods are typically sensitive to hyperparameters (e.g., architecture types, transfer connections between layers), I also wonder how to set the hyperparameters for baselines, especially in Table 2, because choosing the transfer connections between different architectures is very important when using feature-based methods such as FitNet and AT.\n- Moreover, the baselines are developed for improving distillation performance, not replacing KD. Especially, feature-based methods (e.g., FitNet, NST) are easily combined with logit-based ones (e.g., KD). So I think the compatibility between the proposed and exisiting methods should be checked. However, in this paper, only Table 4 shows the compatibility with KD (CRD+KD).\n- The authors compare the proposed method with only KD, AT, FitNet except Table 1-3. For example, when evaluating the transferability (Table 4), why other baselines are not compared?\n- VID also maximizes MI between penultimate layers. What is the key difference and why CRD perform better? I think detailed verfication should be provided in the paper.\n\nMinor Comments\n- Comparison with Online-KD is unfair because it does not use pre-trained ResNet32.\n- Why only use penultimate layers? CRD between intermediate layers is also available like VID.\n- A result in Table 1 is missing (FSP WRN40-2 -> WRN40-1).\n- In Section 4, both CKD (contrastive knowledge distillation) and CRD (contrastive representation distillation) are used, so one of them should be removed.\n- In Section 4.1 Transferability paragraph, \"Table 3\" should be changed to \"Table 4\".\n- On page 4, \"space\" after \"before the inner product.\" should be removed.\n\nI think the proposed method is well-principled and provides meaningful improvements on various distillation settings, thus this paper seems to be above the borderline. It would be better if additional supports for the above concerns is provided in a rebuttal.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new distillation method with theoretical and empirical supports.

Given reviewers' comments and AC's reading, the novelty/significance and application-scope shown in the paper can be arguably limited. However, the authors extensively verified and compared the proposed methods and existing ones by showing significant improvements under comprehensive experiments. As the distillation method can enjoy a broader usage, I think the propose method in this paper can be influential in the future works.

Hence, I think this is a borderlines paper toward acceptance.
Decision: Accept (Poster)