Prototypical Contrastive Predictive Coding
Authors: Kyungmin Lee
Venue: ICLR 2022 Poster (2022)
Keywords: Knowledge distillation
contrastive learning
self-supervised learning

Abstract: Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model to a smaller student model or self-supervised learning via self-distillation. Knowledge distillation is an original method to solve these problems, which minimizes a cross-entropy loss between the prototypical probabilistic outputs of teacher and student networks. On the other hand, contrastive learning has shown its competency in transferring representations as they allow students to capture the information of teacher representations. In this paper, we amalgamate the advantages of knowledge distillation and contrastive learning by modeling the critic of a contrastive objective by the prototypical probabilistic discrepancy between two features. We refer to it as prototypical contrastive predictive coding and present efficient implementation using the proposed objective for three distillation tasks: supervised model compression, self-supervised model compression, and self-supervised learning via self-distillation. Through extensive experiments, we validate the effectiveness of our method and show that our method achieves state-of-the-art performance in supervised / self-supervised model compression. 

Published conference paper at ICLR 2022
REPRESENTATION DISTILLATION BY PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING
Kyungmin Lee
Agency for Defense Development kyungmn 1 com
ABSTRACT
Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model a smaller student model or self- supervised learning via self-distillation. Knowledge distillation is an original method to solve these problems, which minimizes a cross-entropy loss between the prototypica probabilistic outputs of teacher and student networks. On the other hand, contrastive learning has shown its competency in transferring representations they allow students to capture the information of teacher representations. In this paper, we amalgamate the advantages of knowledge distillation and contrastive learning by modeling the critic of a contrastive objective by the prototypical proba- bilistic discrepancy hetween two features. We refer to it as prototypical contrastive predictive coding and present efficient implementation using the proposed objective for three distillation tasks: supervised model compression, self-supervised model compression, and self-supervised learning via self-distillation. Through extensive experiments, we validate the effectiveness of our method compared various supervised and self-supervised knowledge distillation baselines.
INTRODUCTION
In machine learning, knowledge distillation (KD) is problem that aims to transfer the knowledge from one network (a teacher) another one (a student). The original method introduced by Hinton et al. (2015); Buciluă et al. (2006) minimizes the cross-entropy between the probabilistic outputs of teacher and student networks. Even though the simplicity of its implementation, many other distillation methods cannot easily outperform KD. Therefore the concept of knowledge distillation has been expanded to various machine learning tasks other than supervised model compressions, such as self-supervised learning (Caron et al., 2020; 2021) or self-supervised model compression (Fang al., 2021). Those methods rely on minimizing the discrepancy of probabilistic outputs of teacher and student networks, thus require a function that maps data or feature into a probability space. It is straightforward to use linear mapping for it, which we refer to a prototypes.
On the other hand, contrastive learning has its merits in capturing correlations and high-order dependencies over teacher representations as they are trained by pulling the positives and pushing over the negatives. Especially, contrastive objectives such as NCE (Gutmann & Hyvärinen, 2010) or CPC (as known as InfoNCE) (Oord al., 2018; Bachman al., 2019; Hjelm et al., 2018) are proven be a lower bound the mutual information allowing student network to capture the information from the teacher representations. While many methods (Chen et al., 2020b;c) used those objectives for representation lcarning, Tian al. (2019) proposed contrastive representation distillation (CRD) and demonstrated the effectiveness of contrastive learning in transferring knowledge from one network another.
However, the current contrastive learning requires large negative samples with careful sampling. To ameliorate, we combine the prototypical method and contrastive objective inherit their advantages. To that end, we propose prototypical contrastive predictive coding (ProtoCPC), which utilizes the prototypes to generate probability distribution and model the critic by the discrepancy between two probabilistic outputs. Furthermore, ProtoCPC objective is lower bound to the mutual

---

Published conference paper at ICLR 2022
information between teacher and student representations alike CPC. But unlike CPC, does not require commodious negative samples or careful sampling policies.
Given the ProtoCPC objective, we present efficient applications on three distillation tasks: super- vised model compression, self-supe model compression, and self-supervised learning via self-distillation. While ProtoCPC is simple in its implementation, we observe that it significantly boosts the performance of representation distillation. Our method outperforms KD, and partially outperform CRD on CIFAR-100 and ImageNet. Moreover, our method achieves state-of-the of-thhe-rrt performance on transferring various self-supervised teacher representations to small self-supervised model. Lastly, by construing self-supervised learning as self-distillation, we validate the effective- ness of ProtoCPC in representation learning. Our contributions are following:
We propose a novel prototypical contrastive objective for transferring representational knowledge between models.
We apply the proposed objective three distillation tasks: supervised model compression, self-supervised model compression, and self-supervised learning via self-distillation.
Experiments show the effectiveness of our method in various representation distillation benchmarks, especially achieving state-of-the-art performance in supervised/sel--suprrvied model compression.
2 METHOD
2.1 PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING
Given data a with a random variable x ~ X, let the teacher network ft and the student network fs, where they map z into R and Rs respectively. Also let T and S be random variables for representation f=(X) and (x) respectively. One can transfer the representational knowledge 7' to 5 by maximizing the mutual information 5)) where it is defined by the KI-ddieegeenc between the joint distribution p(T. 5) and the product of marginal distribution p(T (SS). However, as estimation and optimization of mutual information is challenging. many approaches count on maximizing variational lower bound to the mutual information (Poole et al., 2019). The contrastive predictive coding (Oord et al., 2018), mutual information (Oord et al., 2018; Tian al., 2020; Bachman et al., 2019; Song & Ermon, 2020), and has shown its competency in both representation learning (He et al., 2020; Chen et al., 2020b;c) and representation distillation (Tian et al., 2019). Formally, given
known as InfoNCE is guaranteed lower bound to the
S with a N-1 sampled from T, i.c. (2t, =a) ~ p(T, S) and
positive 2 = zςa and N- negatives { f==1
[(y. =x) N pS)) the following inequality holds for any critic h PR x RDs + R4:
h(())2))
T(T;S) 2
log
(I)
EE Mazyy 23)
Previous works set the critic by the exponential of cosine similarity between two unit feature vectors, i.e. h(2,, 24) exp( 2s) where žι and 2, fz-normalieed. Theoretically, the lower bound becomes tighter as N + oo. In practice, the CPC objective requires using extremely large batch size or memory buffer that stores the negatives as it requires pairwise computation between and sty8.
On the other hand, we project feature vectors into probability space. To do that, append linear prototypes W P and W e RPa* K at the top of frr and f so that they have same output dimension of K. For brevity, let Σ = S 2, and E = W 2, and T, 5 be random variables for 2, and 2. respectively. Then we set probability of student p. by K K-categorical distribution defined by the softmax operator on E = W5z, 2g with temperature T >
(Ta) exp
Σ exp /T.)
Similarly we define probability of teacher Ptù with temperature T. > 0. Then we define the critic between E, and E, by the negative exponential of cross-entropy between ps and pt, i.e.,
h(,, =.) -H(p.p.) e>iten (1) logp!")
(3)

---

Published conference paper at ICLR 2022
Then h a positive bounded function and is maximized when pr matches with p.- Then by plugging Eq. 3 into Eq. I, it follows that
H (p
(T;5) 2 ()
2
E
log
(4)
f===0 H (ppssppo)
explpe 2z/
log
(5)
N 2=== exp (pts Zo/ Eo/To)
explpt 3z/TT
> E
log
(6)
N L Ei=1 Ptj exp (Za (Te)
exp(pt 23/TB)
EE
log
2 IProOCPC:
Ek=1 g(A) exp
(k) /Te)
where ((4) = ΣØ= Pty is mean of teachers' probability which we call prior. The first in- equality is from data processing inequality, the second equality is from crossing out the constant term, and third inequality is from Jensen's inequality. We define Prototypical Contrastive Predictive Coding (ProtoCPC) objective IPoRCCPC in Eq. 7. In addition, we define ProtoCPC loss CroNcCPC by the negative of ProtoCPC objective, thus minimizing ProtoCPC loss is equivalent to a variational maximization of mutual information between student and teacher representations.
N-1
Relationship with CPC While many lower bounds the mutual information were proposed, Tschannen et al. (2019) observe that the tightness of bound does not necessarily imply a better representation learning performance. From then, many works focused on analyzing the components of contrastive objective itself which are responsible for the empirical success. Wang & Isola (2020) argued that the contrastive loss is composed of alignment and uniformity loss, where alignment loss accounts for the similarity of two positive features, and uniformity loss measures how the features are scattered the unit hypersphere and show that both losses are important contrastive learning.
We draw an analogy on ProtoCPC by dissecting into alignment and uniformity losses. Since the alignment loss is straightforward, we focus on the uniformity loss. We show that one can interpret the uniformity loss CronCCC-Uul by the re-substitution entropy estimator of zg via von-Mises Fisher kernel density estimation (vMF-KDE) (Ahmad & Lin, 1976):
K
CPronCPC-Unit Be, 8
log
( exp( (M) /T)) E,,,iS log q
eepp(ux 24/TT)
(8)
R=
1=
-Ez,~ s DoM (z=)] + log Z-MF - -H(az) + log ZuMP,
where each Wa is a k-th column of Ws and acts mean direction of k-th vMF distribution and (R) acts as a prior for each k-th vMF distribution. The ÂvMP-KDE is thus the mixture of K vMF distribution with prior (h)) and then the uniformity loss is re re-substitution entropy (z.). The Zom a normalizing constant for vMF distribution. Remark that uniformity loss CPC objective is also re-substitution entropy with vMF-KDE, but the mean directions are given by negative samples Rt and the prior is uniform. It shows that the ProtoCPC objective allows modeling of complex mixture vMF distribution by exploiting prior term and using prototypes remove the dependency on negative samples.
Prior momentum Since ProtoCPC is contrastive, it requires sufficient negatives to perform learning. However, unlike CPC, ProtoCPC only requires prior that accounts for negatives. Therefore, we use exponential moving average (EMA) on prior for better estimation. At each iteration, we update g by following update rule:
(k) mpg (8) + (1mpp)
1
(k) Pc
(10)
N
J=1
where mp > O is momentum rate. The prior momentum allows better estimation of prior term regardless of the size of negative samples.

---

Published conference paper at ICLR 2022
Cow Slenscco
Ps
P
P
Ps
P
h
hog
hay
ff
f
f
fos
for
a
y
a
(a) Supervised Model Compression
(b) Self-supervised Model Compression
(c) Sclf-distillation
Demonstration of three distillation tasks using proposed ProtoCPC objective. (a) Compressing a supervised model, (b) compressing a self-supervised model, (c) self-supervised learning via self-distillation. Our method is contrative objective with the prohability distributions where it is mediated by the prototypes. The figures of white color is training parameters while gray colors are frozen throughout the training. The dashed lines represent the copying of parameters such as exponential moving average (EMA).
Figure I:
Assignment of teacher probability While KD used softmax operator for both probabilities of teacher and student networks, many self-supervised methods (Asano al., 2019; Caron al., 2020) reported that the softmax operator can lead to collapse, i.e. every representation fall into the same one. To compromise, many prototypical self-supervised methods resort on Sinkhorn-K Knopp iterative algorithm by formulating assignment of teacher probability as an optimal transport problem:
N K
argmax (Pt, Ze) + (P)),, S.I. P PRRR NxK
P.1x = 1N. PN PN
1K
(11)
(k) is a entropy and where Ze is matrix whose rows are Zjj. H(P)) - Σk= Pt log Pr
O is temperature that controls the smoothness distribution. Then the Eq. 11 can be solved by
only few steps of Sinkhorn- iteration (Cuturi, 2013; Altschuler et al., 2017) which iteratively projects P. into following form:
(k) B /Tc Pty IxmIBwe*t /5"
(12)
where 35 is normalizing constant. We refer this to Sinkhorn Knopp (SK) operator. The SK operator allows each prototype to be non-zero, i.e. g( non-zero for each k. In practice, we only conduct a few steps of SK iterations.
2.2 APPLICATIONS OF PROTOCPC TO DISTILLATION TASKS
Fig. depicts explanation on how we structure prototypical contrastive learning for three distillation tasks we consider: supervised model compression, self-supervised model compression and self- distillation.
Supervised model compression Given (supervised) pre-trained teacher network f". the goal of supervised model compression is to train smaller student network f by utilizing the representational knowledge from the teacher. To do that, Hinton et al. (2015) proposed to minimize the knowledge distillation loss, where is weighted sum of the supervised loss and the distillation loss. Here, the supervised loss is conventional cross-entropy loss between the one-hot label and the output of S The distillation loss is cross-entropy between teacher's, probability output and student's probability output p.. Formally, is given by following:
min miinl (x(x)))((x)) [Esup ( v) + 37 fd
(13)
where y is label, pa( and (xx) are computed by the softmax operator each (T(x)) and (x) with temperature > O, and is balancing weight. Our approach to replace the distillation loss by the ProtoCPC loss and use SK operator for Pt- Then our loss becomes following:
min ((xx)) [Suu (xx) (x:) v) + 37 (0 P.
(14)

---

Published conference paper at ICLR 2022
Remark that since the supervised networks already contain prototypes the last layer. we use prototypes for both supervised learning and distillation at the same time. Note that CRD (Tian et al., 2019) uses binary contrastive loss for distillation loss by maintaining large memory buffer of negative samples. Also, CRD requires an additional linear layer to implement inner- product critic. However, our method conducts contrastive learning without additional training of embeddings or storing memory buffer of negative samples.
Self-supervised model compression Given large self-supervised pre-trained teacher network g², the goal of self-super model compression is to transfer the representational knowledge of g to a smaller self-supervised student network. Fang et al. (2021) first proposed SEED, where the student network is trained by minimizing the cross-entropy loss between the similarity score of teacher and student. The similarity score is computed by the pairwise computation over a memory buffer filled with features of the teacher network.
Our approach uses prototypes to generate the probabilistic output of teacher and student represen- tations, and use ProtoCPC loss for effective distillation. Let the pre-trained teacher network o be a composition of base encoder f" and the projection head h". . Then we train student network g² where it is composed of smaller encoder S and projection head & of the same architecture as hT output
Then we append prototypes W and W for each g" and n to ensure that they have the of dimension K. Given data æ, we train the student network by minimizing the ProtoCPC loss between the probability of teacher pa(c)) and probability of student (2)). For ps we use softmax operator with temperature т, >> and for Pt we use SK operator with temperature Tr over the batch of samples. Then the objective is given by following:
min E (Cmmarcllnnee) 8 W.
(15)
We can further use multi-crops data augmentation (Caron al., 2020), where augments multiple small crops of a data during the training to expedite the training. Let ăj _Mr)) be small local crops with transformation M, then the self-supervised model compression with multi-crops is given by following:
min E E. (Comccrc(px(())..220 - E (omo) g P.WS
(16)
For prototypes of teacher network W we copy the parameters of W to W at each iteration. This allows our method apply to any self-supervised teacher networks. Note that if the teacher network trained by prototypical methods such as DINO (Caron et al., 2021) or SWAV (Caron et al.. 2020), we can re-use use the pre-trained prototypes for W We present ablation studies on setting prototypes for W..
Self-distillation Additionally, we show that ProtoCPC loss enhance the representation learn- ing of a self-supervised model, where we interpret as a self-distillation method. We built on DINO (Caron et al., 2021), which is the state-of oh-the-art method in self-supervised learning that first regarded the self supf-ssu learning as knowledge distillation problem.
Let the student network be the composition of base encoder fög. we append projection head hés consists of MLP layers and a prototypical layer Wag- Then we set the teacher by the momentum encoder (He et al., 2020: Chen et al., 2020b; Grill et al., 2020; Caron et al., 2021) on the student network, where is composition of encoder feg. projection head hey, and prototype Wez. We use EMA for the weights of teacher network, where it is update by 8. + A0e- + (1 - x)e., where A a momentum rate. Within the training, generate a pair of views (I1, 222 from data X and they are passed each student and teacher networks. Then we compute ProtoCPC loss between the probability of teacher network pe (#1) ) and probability of student network (22), and symmetrically between pr (12) and (2()). The objectives are given by:
(pmnucre((((1))pp(2 (ponecee(((( p.
minEE X. 0,
(17)
where A(-) is data augmentation operator. Note that multi-crope crops data augmentation can be further applied.

---

Published conference paper at ICLR 2022
3
RELATED WORK
Contrastive learning There is a large body of works on contrastive objectives for representation learning. The contrastive objectives such as CPC (Oord et al., 2018; Song & Ermon, 2020; Bachman al., 2019; Hjelm al., 2018) and multi-view data augmentation (Tian et al., 2020) are keys for the success of contrastive representation learning. With sophisticated implementation such as momentum encoder (He et al., 2020; Chen et al., 2020b; Grill et al., 2020), or caching negative samples (Wu et al., 2018; Chen al., 2020b;c) had dramatically reduced the gap between self- supervised and supervised models. While the contrastive objectives are proven to be a lower bound to the mutual information (Poole et al., 2019; Song & Ermon, 2020), some studies attribute that the success of representation learning the contrastive objective itself rather than the tightness of mutual information estimation (Bachman ct al., 2019). Then many works focus on analyzing the contrastive objective itself (Wang & Isola, 2020) count on the metric-learning perspective of contrastive learning that deviates into different objectives (Grill al., 2020; Chen & He, 2021; Zhontar al., 2021; Bardes et al., 2021). The key difference between many contrastive objectives and our method is that their discrepancy measure is given by the f--norm of two features, while our method projects feature into a probability space and characterize the discrepancy with probabilistic measure. Our work is also related to methods that use clustering for representation learning (Tian et al., 2017; Zhuang et al., 2019; Li et al., 2020), which have been developed into prototypical methods (Asano al., 2019; Li et al., 2020; Caron al., 2020; 2021).
Knowledge distillation The idea that transferring the large pre-trained model's representation into a smaller one has been embodied by knowledge distillation (Hinton et al., 2015; Buciluă et al., 2006). Then various distillation criteria or utilization of intermediate feature maps were studied to enhance the performance (Romero al., 2014; Zagoruyko & Komodakis, 201 6a; Tung Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2019; Kim et al., 2018; Yim et al., 2017; Huang & Wang. 2017). However, many of them were not able to outperform KD by far. Recently, Tian al. (2019) introduced contrastive representation distillation (CRD) which utilizes binary contrastive objective for distillation loss and shows the empirical superiority on various knowledge distillation benchmarks.
Moreover, recent studies focus on distilling the knowledge of large self self-supervised models to the smaller ones using the concept of knowledge distillation (Fang et al., 2021: Shen et al., 2021; Noroozi et al., 2018; Chen el al., 2020a). Fang et al. (2021) proposed self-supervised representation distillation (SEED) which used features of teacher networks to compute probability distribution for distillation. One can interpret SEED as the prototypical method where prototypes are given by the queue of teacher features.
4 EXPERIMENT
4.1 SUPERVISED MODEL COMPRESSION
Setup We experiment on CIFAR- 100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) with various teacher-student combinations such as ResNet (He et al., 2016) and Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016b). We compare with various distillation base- lines such as KD (Hinton et al., 2015) and CRD (Tian et al., 2019).
Results on CIFAR-100 Table 1 and Table compare top-1 accuracies our method on super- vised model compression CIFAR-100. Table I investigates teachers and students of the same architectural style, while Table is focused students and teachers from different architectures. We observe that our ProtoCPC consistently outperforms KD and other distillation methods, yet ProtoCPC is on par with CRD where it also uses a contrastive learning method for distillation. Remark that CRD requires a large memory buffer of negative samples with careful sampling according the class information and additional training of embeddings. However, our ProtoCPC does not require massive negative samples and additional training of linear embeddings since we use the last linear layer as prototypes for our ProtoCPC.

---

Published conference paper at ICLR 2022
Teacher Student
WRN-40-2
WRN-40-2 WRN-40-1
resnet56 resnet20
resmetl10 resnet20
resnel110
resmel32x4
vg
WRN-16-2
resmet32
resmet8x4
'gg
Teacher Student
75.61
75.61
72.34
74.31
74.31
79.42
74.64
73.26
71.98
69.06
69.06
71.14
72.50
70.36
KD
74.92
73.54
70.66
70.67
73.08
73.33
72.98
FitNet
73.58 (U)
72.24 (1)
69.21 (1)
68.99 (U)
71.06 (1)
73.50 (1)
71.02 (U)
AT
74.08 (
72.77 (L)
70.55 (L)
70.22 (1)
72.31 (1)
73.44 (1)
71.43 ())
SP
73.83 ())
72.43 (1)
69.67 (U) 70.04 (1)
72.69 (1)
72.94 (U)
72.68 (L)
CC
73.56 ())
(2)
69.63 (1)
71.48 69)48
72.97 (U)
70.71 (L)
VID
74.11 (3)
73)30
70.38 (1)
72.61 (L)
73.09
71.23
RKD
73.35 (
72)
69.61
(9)25
71.82
71.48 71.90
PKT
74.54 ())
73.45 (L)
70.34
(0)25
72.61
73.64
72.88 ())
72.50 ())
72.38
69.47 (U) (L)
70.98
73.17())
70.94 (L)
FT
73.25 (
71.59
69.84 (U) 70.22
72.37 (L)
72.86
70.58 (U)
ESP
72.91 (1)
n/a
69.95 (U) 70.11
71.89
72.62
70.23 (L)
NST
73.68 (
(1)
69.60 (1) (9.53
71.96
73.30
71.53
CRD
75.48 (7)
71)
71.16(1)
71.46 (1) 73.48
75.51 (1)
73.94
ProtoCPC
75.79 (7)
74.23
71.1 (1)
71.04
73.55
75.02 (1)
73.79 (1)
KD+CRD 75.64 (1)
74.38 (1)
71.63 (1)
71.56 (1) 73.75
75,46 (1)
74.29 (1)
PrODCPC+CRD
75.92 (
74.75 (1)
70.99 (1)
71.47 (1)
71)52
75.55 (1)
74.40 (1)
Table 1: Top-1 test accuracy (%) of student networks on CIFAR-100 with various distillation methods (our method denoted by ProtoCPC). (1) denotes outperformance over KD and bold font denotes the best accuracy within the bascline. Note that ProtoCPC always outperform KD well as other baselines except CRD. Our ProtoCPC outperforms CRD in out of benchmarks. We also provide results combining ProtoCPC and CRD, showing that our method is compatible with CRD they learn different structure. The results are averaged over runs.
Teacher Student
1 MobileNetV2
ResNet50 MobileNetV2
ResNet50 vgg8
resnet32x4 ShuftleNetVi
resnet32x4 ShuflleNetV2
WRN-40-2
ShuffleNetVi
Teacher
74,64
79.34
79.34
79,42
79.42
75.61
Student
64.6
64.6
70.36
70.5
71.82
70.5
KD
67.37
67.35
73.81
74.07
74.45
74.83
FitNet
64.14 (1)
63.16 (L)
70.69 (L)
73.59 (L)
73.54 (L)
73.73 (L)
AT
59.40 (1)
58.58 (1)
71.84 (1)
71.73 (1)
72.73 (1)
73.32 (1)
SP
66.30 (1)
68.08 (1)
73.34 (1)
73.48 (1)
74.56 (1)
74.52 (L)
CC
64,86 (L)
65.43 (L)
70.25 (L)
71.14
71.29
71.38 (1)
VID
65.56 (L)
67.57 (1)
70.30 (L)
73.38
73.40 (1)
73.61 (L)
RKD
64.52 (L)
64.43
71.50 (L)
72.28
73.21
72.21 (1)
PKT
67.13 (L)
(L)
73.01 (L)
74.10
74.69 (1)
73.89 (L)
AB
66.06 (1)
67.20
70.65 (L)
73.55
74.31 (1)
73.34 (
FT
61.78 (L)
(0.99
70.29 (L)
71.75 (L)
72.50
72.03 (L)
NST
58.16 ())
64.96
71.28 (L)
74.12 (1)
74.68 (1)
74.89 (1)
CRD
69.73 (1)
69.11
74.30 (1)
75.11
75.65 (1)
76.05 (1)
ProtoCPC
69.09 (1)
69.50 (1)
74.32 (1)
75.24 (1)
76.50
76,28 (1)
KD+CRD
69.94 (1)
69.54 (1)
74.58 (1)
75.12
76.05 (1)
76.27 (1)
ProtoCPC+CRD
69.77 (1)
70.79 (1)
74.95 (1)
76.01 (1)
76.27 (1)
76.82 (1)
Table 2: Top-1 test accuracy(%) of student networks CIFAR- 100 of various distillation methods (ours is ProtoCPC) for transfer across very different teacher and student architectures. ProteCPC outprrforms KD and all other methods except CRD, Our method outperforms CRD 4 out of 6 benchmarks. We also observe that combining ProtoCPC and CRD boosts the performance significantly. Average over 3 runs.
Results on ImageNet In Table 3, compare ProtoCPC to different distillation methods such as AT (Zagoruyko & Komodakis, 2016a) and Online-KD (Lan et al., 2018) on ImageNet, where pre-trained ResNet-34 is used for teacher and ResNet-18 is student. We observe that ProtoCPC outperforms various knowledge distillation baselines such as KD, and slightly better than CRD. Remark that our method simpler in its implementation and shows better performance than CRD.

---

Published conference paper at ICLR 2022
Teacher
Student
AT
KD
SP
CC
Online KD
CRD
ProtoCPC
Top-1 Top-5
26.69
30.25
29.30
29.34
29.38
30.04
29.45
28.83
28.54
8.58
10.93
10.00
10.12
10.20
10.83
10.41
9.87
Table 3: Top-1 and Top-5 error rates (%) of student network ResNet-18 18 on ImageNet validation set. For fair comparison, we use same setting as in Tian et al. (2019). We compare our ProtoCPC with KD (Hinton et al., 2015), AT (7agoruyko & Komodakis, 2016a), SP (Tung & Mori, 2019), CC (Peng et al., 2019), Online KD (Lan et al., 2018), and CRD (Tian et al., 2019).
SELF-SUPERVISED MODEL COMPRESSION
Setup We experiment distillation of various self-supervised networks to ResNet-18 (He et al., 2016) ImageNet (Deng al., 2009) without class labels. We consider following self-supervised teacher networks: MoCo-v2 (Chen al., 2020b) pre-trained ResNet-50, SwAV (Caron al., 2020) pre-trained ResNet-50 and DINO (Caron et al., 2021) pre-trained ResNet-50 and vision trans- former (Dosovitskiy al., 2020). We train for 100 epochs and we additionally conduct experiments on using multi-crops data augmentation for SwAV evaluation, we follow linear evaluation protocol which conducts supervised learning on the linear layer appended the top of the frozen feature and k-nearest neighbor classification (k-NN).
DINO pre-trained ResNet-50 networks. For
Main results Table show the main results of our self-supervised model compression compared to self-supervised learning (SSL) of itself. We observe that ProtoCPC outperforms SSL with a large margin, especially showing superior performance k-NN classification. Note that our method works well for various self-supervised teacher networks and even works well when the teacher and student networks are of different architectures (vision transformer teacher to ResNet student).
MoCo ResNet-50
SwAV ResNet-50
DINO ResNet-50
DINO DciT-S/16
Linear
k-NN
Linear
k-NN
Linear
k-NN
Linear
k-NN
Teacher
71.1
61.9
75.3
65.7
75.3
67.5
77.0
74.3
Supervised 69.5
69.5
69.5
69.5
69.5
69.5
69.5
69.5
SSL. 52.5
36.7
57.5
48.2
58.2
50.3
58.2
50.3
ProteCPC 61.1
55.6
63.1
57.7
63.5
60.3
65.5
63.2
Table 4: Main result of our ProtoCPC on distillation of various self-supervised teacher models to ResNet-18. 18. The teacher models are MoCo (Chen et al., 2020h) ResNet-50, SwAV (Caron et al., 2020) ResNet-50 and DINO (Caron et al., 2021) ResNet-50 and DeiT small with paich size 16. The self-superviser denotes the result of self-supervised learning on ResNet- 18 with the same method of teacher network.
Comparison with SEED Table 5 compares our method with SEED (Fang et al., 2021). an original method for self-supervised model com- pression. We observe that method consis- tently outperforms SEED in both k-NV and lin- ear evaluation with the same teacher network applied. Although training for shorter epochs, our method achieves outperforms SEED. When the teacher is pre-trained by SwAV, our method outperforms SEED without using multi-crops data augmentation, and the gap becomes larger when we use multi-cne multi-crops as well. When using DINO self-supervised teacher and multi-crops data augmentation, achieve the best results in representation learning of ResNet-18.
Teacher
Method
Epochs
Linear
k-NN
SEED
200
60.5
49.1
MoCo
ProtoCPC
100
61.1
55.6
SEED
100
61.1
SEED
200"
62.6
SwAV
ProtoCPC
100
63.1
57.7
ProtoCPC
100'
63.9
57.0
ProtoCPC 100
63.5
60.3
DINO
ProtoCPC
100"
65.3
60.7
Table 5: Comparison of our method with SEED. de- notes training with multi-crops. Every teacher networks are ResNet-50 and student networks are ResNet-18.
4.3 SELLE-DISTILLATU
Setup
We evaluate our ProtoCPC objective in self celf-supervised learning, where we cast it as self- distillation. We follow DINO (Caron et al., 1021) l) for experimental procedures, where the teacher

---

Published conference paper at ICLR 2022
network is by the momentum encoder, and linear prototypes are attached to the features to distill
the knowledge.
We change the loss function from cross-entropy loss to ProtoCPC loss without any adjustment other settings.
Architecture
Method
Linear
A-NN
use prior momentum rate of 0.9 for ProtoCPC loss. We evaluate on both ResNet-50 and DeiT-S/16, where DINO performs well on both architectures. For evalu- ation, we report the results of linear evaluation protocol and k-nn classification.
DINO
70.9
62.33
ResNet-50
ProtoCPC
71.8(+0.9)
63.7(+1.44)
DINO
74.0
69.3
DeiT-S/16
ProtoCPC
75.0(+1.0)
70.6(+1.3)
Table 6: Results of representation learning trained by DINO (Caron et al., 2021) and ProtoCPC (ours). All experiments are run by ours with 100 epochs.
Results Table 6 compare the performance of representation learning of our method and DINO. We observe that ProtoCPC enhances the performance of representation learning without the adjustment of sypee-paraaeters. Also, in Table 7, we demonstrate the effect of prior mo- mentum rate p and observe that non-zero mp results in better representation learning.
my
0.0
0.9
0.99
Linear
73.4
75.0
74.5
N-NN 69.6
70.6
70.5
Table 7: Ablations on ProtoCPC loss with prior momen- tum rate my on DeiT-S/16 trained for 100 epochs.
4.4 ABLATION STUDY
Cross-entropy ProtoCPC and Softmax Sinkhorn-Knopp The original KD is imple- mented by cross-entropy loss between the softmax output of the teacher and student network. Our method uses ProtoCPC loss and SK operator for teacher network. We provide ablation studics on the choice of the loss function and probability assignment operator the subset of supervised model compression tasks. In Table 8, we observe that using the SK operator boosts the performance of KD, and using ProtoCPC loss further enhances the performance.
Teacher's prototypes for self-supervised model compression For prototypical self-supervised teachers, one can use the pre-trained prototypes for distillation. We present an ablation between using pre-trained prototypes and copying student's prototypes for teachers. Table 9 show the results distillation from SwAV ResNet-50 and DINO ResNet-50 teachers. We observe from SwAV teacher, using pre pre-trained prototypes performs better, while distilling from DINO teacher, copying from student's prototypes performs better. We suspect that SWAV used single prototypes throughout pre-training, the pre-trained prototypes contain representational knowledge. On the other hand, the pre-trained prototypes of DINO lack such representational knowledge.
when distilling
Loss
WRN-40022 WRN-16-2
resnet110 remnc220
reset111 resnc132
romne13244 resnct8y4
vgg13
Teacher
Method
Linear
k-NN
New P Old P
60.8
54.5
CE
SM
74.92
70.67
73.08
73.33
72.98
SWAV
63.1
57.7
75.20
71.06
72.97
73.31
New P
63.5
60.2
ProtoCPC
SK
SM
75.36
75.79
71.04
71.08
73.55
73.38
75.02
74.84
73.79
73.66
DINO
60.3
56.6
Table 8: Ablation study of losses and assignments of on super- vised model compression CIFAR- We compare ProtoCPC and cross-entropy (CE), and Baankhom--Koopp (SK) operator and softmax (SM) operator. Average over runs.
Table 9: Ablation of setting teacher's pro- tolypes for supe-supe model compres- sion on ResNet-18, 18. New P denotes copy- ing student's prototype and P denotes using pre-trained prototypes.
5 CONCLUSION
In this paper, we propose prototypical contrastive predictive coding, simple yet effective method for the distillation of network by combining the prototypical method and contrastive learning. Our experiments show the effectiveness of our objective on various applications such as supervised/ self-supervised model compression and self-supervised learning by sell-distillation.

---

Published conference paper at ICLR 2022
REFERENCES
Ibrahim Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.). IEEE Transactions on Information Theory, 22(3):372-375, 1976.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9163-9171, 2019.
Jason Altschuler, Jonathan Weed, and Philippe Rigollet. Near-lineat time approximation algorithms for optimal transport via sinkhorn iteration. arXiv preprint (rXXv:1705.09634 2017.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv: 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910. 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variame-iivvrianc--vvaai regularization for self-supervised learning, arXiv preprini urXiv:2 2021.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu- Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541, 2006.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. J0029, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020c.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26:2292-2800 23000, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Iece, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recogmition at scale, arXiv preprint arXiv:2010.1 11929, 2020.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arxiv:2101.044 1111111..41,..222 2021.
Tean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self supervised learning, arXiv preprint arXiv:2006.0775 07733, 2020.
10

---

Published conference paper at ICLR 2022
Michael Gutmann and Aapo Hyvärinen. conte-con estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirleenth international conference on artificial intelligence and stafistics, pp. 297-304, JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan. Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17299-938- 2020.
Byeongho Heo. Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3779-3787, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv 2015.
preprint
Devon Hjelm. Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1808 1808006670, 2018.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint 1XXix:777.0011999 2017.
Jangho Kim, SeongUk kk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. arXiv preprint arXiv: arXiv:1802.0444 180.049777 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xu Lam, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. arXiv preprint arXiv:1806. 04606, 2018.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning unsupervised representations. arXiv preprint arXiv:2005. 04966, 2020.
Ningning Ma, Xiangyu Zhang, Hai-Tac Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. Proceedings of the IEEE Conference on Computer Vision and 2018.
Pattern Recognition, pp.
Aaron van den Oord, Yazhe and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini 1800:v::00.03311 2018
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268 284, 2018.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Ben Poolc, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLRR, 2019.

---

Published conference paper at ICLR 2022
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprini arXiv: arXiv:1412.655: 1etr1.1522,, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 2510-4520, 2018.
Zhiqiang Shen, Zechun Liu, Jie Qin, Lei Huang, Kwang- Ting Cheng, and Marios Savvides, S2-bmm: Bridging the gap between self supervvid real and 1-bit neural networks via guided distribution cali- bration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2165-2174, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. arXiv preprint arXiv:2007.09852 2020.
Kai Tian, Shuigeng Zhou, and Jihong Guan. Deepcluster: A general clustering framework based on deep learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 809-825, Springer, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint 1910: 10699, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp- 776-794. Springer, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. arXiv preprint arXiv:1907. 13625, 2019.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1365-1374, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9939-9933. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7732-3742, 2018.
Junho Yim. Donggyu Joo. Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Sergey Zogoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03322. 2016a.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1rXiv:1605.07146. 2016b.
Jure Zbontar, Li Jing. Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002-6012, 2019.

---

Published conference paper at ICLR 2022
A
DETAILS IN EXPERIMENTS
PSEUDOCODE FOR PROTOCPC L.0SS
The PyTorch style pseudo-code for our ProtoCPC is demonstrated in Algorithm 1.
Algorithm 1 ProtoCPC loss PyTorch-style pseudocode.
tpe, tpt: student and teacher temperatures
# m: prior momentum rate X) initialize prior with uniform (t.detach()
prior = torch. def (xt,
pt (x(st tpt)
xs = zs tps
prior msprior (1-m) K torch dim=0) i sum prior is always
align -torch. sum zs, dim=1)
loss unif torch. logsumexp (25 torch. torch. log(prorr, loss loss _asinn + ioss_unif
return m)
A.2 SUPERVISED MODEL COMPRESSION
Many of followings are borrowed from CRD (Tian et al., 2019).
A.2.1 OTHER METHONS
We compare the following other state-of-the-art methods from the literature:
Knowledge Distillation (KD) (Hinton et al., 2015)
Fitnets: Hints for thin deep nets (Romero et al., 2014)
3. Attention Transfer (AT) (Zagoruyko & Komodakis, 2016a)
Similartty-Preserving Knowledge Distillation (SP) (Tung Mori, 2019);
5. Correlation Congruence (CC) (Peng et al., 2019)
Variational information distillation for knowledge transfer (VID) (Ahn al., 2019)
7. Relational Knowledge Distillation (RKD) (Park et al., 2019)
Learning deep representations with probabilistic knowledge transfer (PKT) (Passalis Tefas, 2018)
Knowledge transfer distillation activation boundaries formed by hidden neurons (AB) (Heo et al., 2019)
10. Paraphrasing complex network: Network compression via factor transfer (FT) (Kim al., 2018)
11. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning (FSP) (Yim et al., 2017)
12. Like what you like: Knowledge distill via neuron selectivity transfer (NST) (Huang & Wang. 2017)
13. Contrastive representation distillation (CRD) (Tian et al., 2019)
A.2.2 NETWORK ARCHITECTURES
Wide Residual Network (WRN) (Zagoruyko & Komodakis, 2016b). WRN-d-w represnets wide resnet with depth dl and width factor w.
resnet (He al., 2016). We use resnet-d represent cifar-style resnet with 3 groups of basic blocks, each with 16, 32, and 64 channels respectively. our experiments, resnet8 x4 and resnet32 x4 indicate times wider network (namely, with 64, 128, and 256 channels for each of the block)

---

Published conference paper at ICLR 2022
ResNet (He et al., 2016). ResNet-d represents ImageNet-style ResNet with Bottleneck blocks and more channels.
MobileNetV2 Sandler et al. (2018). In experiments, use width multiplier of 0.5. veg (Simonyan Zisserman, 2014), the vgg net used in our experiments are adapted from its original ImageNet counterpart.
ShuffleNetV (Zhang al., 2018), ShuffleNetV2 (Ma al., 2018). ShufileNets proposed for efficient training we adapt them to input of size 32x32.
A.2.3 IMPLEMENTATION DETAILS
All methods evaluated our experiments use SGD.
CIFAR-100, we initialize the learning rate as 0.05, and decay by every 30 epochs after the first 150 epochs until the last 240 epoch. For MobileNetV2, ShuffleNetVI and ShuffleNetV2, we learning rate of 0.01 as this learning rate is optimal for these models a grid search, while 0.05 is optimal
other models.
For ImageNet, we follow the standard PyTorch practice but train for 10 more epochs. Batch size is 64 for CIFAR- 1-00 or 256 for ImageNet.
The student trained by combination of cross-entropy classification objective and knowledge distillation objective, shown as follows:
C a- &
(18)
For the weight balance factor 3, we directly use the optimal value from the original paper if it is specified, or do a grid search with teacher WRN-40-2 and student WRN-16-2. This results in the following list of used for different objectives:
KD (Hinton et al., 2015):
0.1,8 0.9 and
Fitnets (Romero et al., 2014): a = 1 100
3. AT (Zagonyyko & Komodakis, 2016a):
1, 1000
SP (Tung Mori, 2019): a = 1, 3000
5. CC (Peng et al., 2019): a = 1,3 = 0.02
VID (Ahn al., 2019):
1,3
7. RKD (Park et al., 2019): a 1, = 25 for distance and 92 = 50 for angle. For this loss,
we combine both term following the original paper.
PKT (Passalis & Tefas, 2018): a 1. 30000
9. AB (Heo et al., 2019): 3 = 0, distillation happens in a separate pre-training stage where only distillation objective applies.
10. FT (Kim et al., 2018): a = 1,3 = 500
11. FSP (Yim al., 2017): 3 = 0, distillation happens in separate pre-training stage where only distillation objective applies.
12. NST (Huang& & Wang, 2017): a = 1, 50
13. CRD (Tian al., 2019): 1, 3 0.8
Similar KD, ProtoCPC used 1, = 1.75 and temperature T = 4. For ablation study in Table 8, used 0.1,3 0.9 for CE losses and a 1, 1.75 for ProtoCPC losses.
A.3
SELF-SUPERVISED MODEL COMPRESSION
A.3.1 PRE-TRAINED TEACHERS
MoCo-v2 (Chen et al., 2020b) uses InfoNCE loss and momentum encoder. Since InfoNCE requires large negatives, they pertain large queue which updates by with features of momentum encoder. We used the MoCo-v2 ResNet-50 (He et al., 2016)
rule
trained for 800 epochs.

---

Published conference paper at ICLR 2022
SwAV (Caron et al., 2020) a prototypical method that generates probability by adopting prototypes. They generate probability of a given feature by computing similarity with respect to prototypes and minimize the cross-entropy between probability outputs of two different views of image. They used Sinkhorn Knopp iteration for probability output. Also, they first proposed crops strategy, which additionally use small crops an image to expedite the training. We used the SWAV ResNet-50 trained for 800 epochs with multi-crops applied.
DINO (Caron et al.. 2021) is a prototypical method which uses momentum encoder. The training progress is similar to SwAV except that they momentum encoder the prototypes and use online centering hefore the softmax operator. They showed that their method is effective in training vision transformer as well as convnet such as ResNet-50. We used the DINO ResNet-50 and DeiT-S/16 (Dosovitskiy al., 2020) trained for 800 epochs with multi-crops applied.
We archive the checkpoints of teacher models from the author's original implementation.
For each method, also conduct self- supervuee learning on ResNet- 18 for fair comparison. We used same paper-parameters that were used to train ResNet-50 except that we trained for 100 epochs. In Table 4, we report the results of self-supervised learning on ResNet-18.
A.3.2 NETWORK ARCHITECTURES
We set the projection heads of student network to be same as the teacher network. When teacher is DINO DeiT, the teacher network do not contain batch normalization, but add batch normalization projection heads when training ResNet-18 student. Remark that projection heads of MoCo and AW have output dimension of 128, and the projection head of DINO has output of 256. Then we set the number of prototypes to be K 65536 throughout the experiments. For SwAV, since we use pre-trained prototypes, number prototypes is 3000. Every features are normalized before computation with prototypes, and prototypes are normalized during the training.
A.3.3 TRAINING HYPERPARAMETERS
For probability of teacher, we use SK operator with 3 steps of iteration and T: = 0.04. För probability of student, set 0.1. The prior momentum for ProtoCPC loss is 0.9. We use SGD optimizer with batch size 512 and weight decay le-4. The learning rate and is decayed by cosine learning rate schedule to le-6.
A.3.4 EVALUATION
For evaluation, we use both linear evaluation protocol and k--narest neighbor classification. For linear evaluation protocol, we freeze the trained weight and train a linear classifier at the top of the frozen feature. We train with SGD optimizer with batch size 256 and use learning rate of 0.3 with 100 epochs. We use cosine learning rate decay schedule and we don't 't use weight decay. For A-NN classification, we follow weighted k-NN with 7 0.07 done in (Wu et al., 2018).
A.4 SELFF-DISTILLLTINN
Most of our implementation is borrowed from DINO (Caron et al., 2021). For self-distillation experiments, only changed cross-entropy loss to ProtoCPC loss. Detailed syper-parameten are following:
ResNet-50: we used 2 layers MLP projection head with output dimension 65536. We trained for 100 epochs with 2x224 + 6x96 multi-crops, batch size 512, learning rate 0.6 with cosine anncaling schedule, weight decay lc-4, teacher temperature t. 0.04, student temperature = 0.1, prior momentum m09.
DeiT-S/16: we used layers MLP projection head with output dimension 65536. We trained for 100 epochs with 2x224 8x96 multi-crops, batch size 512, learning rate 0.001 with cosine annealing schedule, weight decay le-4, teacher temperature T = 0.04, student temperature T, 0.1, prior momentum =09.


Reviews:
[{"review_id": "cDmFLXK7Sbk", "reviewer": "Reviewer_hGXe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper is well written and organized.\nThe proposed method can be integrated into other methods and Experiments are done on different tasks to validate effectiveness.\nWeakness: \nThe main concern is the novelty of this method. It combines the prototypical method and contrastive learning while these two techniques are well researched in the literature. Also, compared with CRD+KD  the improvement is marginal in the knowledge distillation task.\n\nAfter rebuttal:\nI keep my original rate.  I think this paper is well presented with decent results. ", "summary_of_the_paper": "This paper proposes a prototypical contrastive predictive coding by combining the prototypical method and contrastive learning. Experiments are done on applications such as supervised/ self-supervised model compression and self-supervised learning by self-distillation to validate its effectiveness.", "summary_of_the_review": "This paper is well written and experiments are done thoroughly. The main concern is the novelty as it combines existing techniques without substantial findings.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OCS76cqBFiL", "reviewer": "Reviewer_EXkK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this to be an interesting paper. The results look decent, the method makes sense, and the derivations appear to be sound. The paper is clearly written and well presented.\n\nThere are however, several weaknesses to the paper. First and foremost, almost all the methods appear in prior work, albeit used in slightly different ways. The use of representations of negatives, updated with momentum, is reminiscent of MoCo; in both cases the basic method -- momentum -- and outcome -- smaller batches of negatives -- are the same, even though how the negatives are represented is different. Prior work has used contrastive losses for knowledge distillation (e.g., CRD), and has used prototypes for contrastive learning (e.g., DINO, SwAV). The Sinkhorn-Knopp method was introduced in SeLa. The present paper is novel in how it puts together these pieces and applies them, but the key technical ideas are all already in the literature, as far as I can tell.\n\nNonetheless, as an engineered system, the paper achieves solid performance. The ablations and comparisons are thorough, and I'm convincined that the ProtoCPC objective does have benefits over the baselines, although the numerical gains are not large. \n\nOne point the paper makes is that ProtoCPC avoids the large batches required by some of the competing methods, and that this could have advantages in terms of computational efficiency. I would have liked to see a quantification of this point, in terms of total number of network calls, memory consumption, or wall clock time.\n\nTwo minor comments:\n1. I'm not sure the name ProtoCPC is well-chosen. Contrastive Predictive Coding involved a \"predictive\" component in the form of the linear transformation W that predicts z_{t+k} from c_t. I don't really see a \"prediction\" component in present paper, at least not in the same sense as in CPC.\n2. The title also could be improved: there is no mention of distillation in the title yet the rest of the paper is all about applications to distillation.\n3. “Our method infers” —> “Our method inherits”?", "summary_of_the_paper": "This paper studies a new contrastive loss function for knowledge distillation and self-supervised representation learning. The paper demostrates results on transferring knowledge from a teacher to student using both supervised and self-supervised objectives, and on representation learning based on self-distillation. As prior work has already applied contrastive learning to each of these problems the main novelty of the paper is the exact form of the contrastive loss. The loss is computed over projections of the embeddings into a k-dimensional simplex (probability space) where cross-entropy can be used as the similarity function. This allows for encoding \"negatives\" into a distribution over k prototypes that are updated with EMA over iterations of training rather than requiring a large batch of negatives during each iteration of training. The results appear to slightly outperform SOTA methods.", "summary_of_the_review": "I think this is a fine paper, well executed, and with decent results. However, I think the novelty is minimal compared to prior work that introduced the key methods that are used, and the empirical gains are also not that large. I therefore think this paper is slightly below the threshold of typical ICLR papers in terms of value to the community, although it is above the threshold in terms of sound science.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GYJJlkgI4Zu", "reviewer": "Reviewer_dY6T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As for citation, the original method of KD was introduced in \"Model compression, KDD 2006\". The authors cite this paper in the Related Work section, but it might be more appropriate to also cite it in the Introduction section. \n\nPlease reply to the below questions.\n\n1. Although the paper title called \"PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING\", the experiments are only conducted on knowledge distillation. Could you provide other evidence of ProtoCPC's advantages (e.g. Mutual Information Estimation and Representation Learning without distillation experiments)? Otherwise, the paper could be called \"**Distillation** with PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING\".\n2. Is it possible to compare CRD with ProtoCPC beyond the Supervised Knowledge Distillation? The authors claim that \"Our method outperforms KD as well as CRD\nin supervised model compression on CIFAR-100 and ImageNet\". From the experiment results, it doesn't necessarily outperform CRD. \n3. In **Prior momentum** of 2.1, the authors state that \"The prior momentum allows better estimation of prior term regardless of the size of negative sample\". However, from the equations (6) (7) & (10), the sampled N is still required in the prior. What is the actual reason that we can ignore a large number of negative samples?\n4. Furthermore, as the authors claim their ProtoCPC \"does not require massive negative samples\" when performing contrastive learning, will this method boost the training efficiency?\n\nMinor comment:\n\n- In Table 3 description, the references of SP and CC are missing.", "summary_of_the_paper": "This paper proposes to combine knowledge distillation (KD) and contrastive learning for distillation tasks. Concretely, it models the critic of a contrastive objective by the prototypical probabilistic discrepancy between two features. The authors then carry out extensive experiments on supervised model compression, self-supervised model compression, and self-supervised learning through self-distillation. The empirical results show that the proposed method outperforms other strong baselines. ", "summary_of_the_review": "This paper proposes to combine knowledge distillation and contrastive learning for distillation tasks, which is also well-motivated. Extensive experiments on benchmarks validate its effectiveness. It's a nice paper with solid, but somewhat incremental, technical contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DQwsuyACiiZ", "reviewer": "Reviewer_tvAr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n(i) The proposed method is well described and seems to work as shown by sufficient experiments in various distillation settings. \n(ii) Fairly well-written paper and easy to follow various distillation settings.\n\nWeakness:\n(i) A detailed justification on why momentum-based estimation helps the ProtoCPC loss is missing? \n(ii) Empirically the method seems to work, a critique on why prototypes (clustering) help is missing? Motivation on prototypes would be appreciated.\n\nQuestions:\n(i) The pseudo-code of the ProtoCPC loss only takes z_t and z_s as input. How are the negative examples provided for momentum estimation? \n(ii) Currently, the prototypes are computed using a linear layer. Are more complex networks helpful or deterimental?\n(iii) Is the runtime of the method a criterion to evaluate efficient learning?\n\nTypo:\ncomputation between z_s and z_tjS -> computation between z_s and z_tj\n", "summary_of_the_paper": "This paper provides a method for combining contrastive learning and clustering (prototypical probabilities) for three knowledge distillation tasks - supervised model compression, self-supervised model compression, and self-supervised learning with self-distillation. \nTraditional contrastive learning methods rely on the similarity of feature vectors from the teacher and student networks using cosine similarity, log bilinear model (van den Oord et al), etc. This paper proposes to project the feature vectors into a probability space via linear prototypes (layer) of the same dimension (i.e, cluster the feature vectors). The proposed method overcomes the disadvantage of having large negative examples for contrastive learning by maintaining an EMA of prior that accounts for the negative examples.\n\nThe main contribution of this paper is the ProtoCPC loss that combines the advantages of contrastive predictive coding loss using prototypes over the teacher and student representation. \n\nThe authors empirically show that the proposed method is useful for various distillation tasks on image classification tasks on CIFAR-100 and Imagenet datasets.", "summary_of_the_review": "I enjoyed reading the paper and feel it has value to the community. The experimental study justifies the usefulness of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a prototypical contrastive predictive coding by combining the prototypical method and contrastive learning, and presents its efficient implementation for three distillation tasks: supervised model compression, self-supervised model compression, and self-supervised learning via self-distillation. The paper is well-written, and the effectiveness of the proposed method is validated through extensive experiments.  Reviewers generally agree the paper has clear merits despite some weaknesses for improvement. Overall, I would like to recommend it for acceptance and encourage authors to incorporate all the review comments and suggestions in the final version.
Decision: Accept (Poster)