SEED: Self-supervised Distillation For Visual Representation
Authors: Zhiyuan Fang
Jianfeng Wang
Lijuan Wang
Lei Zhang
Yezhou Yang
Zicheng Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Self Supervised Learning
Knowledge Distillation
Representation Learning

Abstract: This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named $\textbf{SE}$lf-Sup$\textbf{E}$rvised $\textbf{D}$istillation (${\large S}$EED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that ${\large S}$EED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, ${\large S}$EED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset. 

Published as a conference paper at ICLR 2021
SEED: SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION
Zhiyuan Fang!, Jianfeng Wang", Lijuan Wang', Lei Zhang Yezhou Yang', Zicheng Liu Microsooft Corporation
1 Arizona State University,
(zy- f fang, yz. yang) @asu. edu
(jianfw, lijuanw, leizhang, z zliu) Smicrosoft.com
ABSTRACT
This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem. propose a new learning paradigm, named SEli-SupErvised Distillation (SEED), where we leverage larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, train a student encoder to mimic the similarity score distribution inferred by teacher over set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervisee baselines, SEED improves the top- accuracy from 42.2% to 67.6 67.6% on EfficcentNet-B00 and from 36.3% to 68.2% on MobileNet- V3-Large on the ImageNet-11 1k dataset.
INTRODUCTION
The burgeoning studies and success self-supervised learning (SSL) for visual representation are mainly marked by its extraordinary potency of learning from unlabeled data at scale. Accompanying with the SSL its phenomenal benefit of obtaining task-agnostic representations while allowing the training to dispense with prohibitively expensive data labeling. Major ram- ifications of visual SSL include pretext tasks (Noroozi Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018; Zhang et al., 2019; Feng et al., 2019), con- trastive representation learning (Wu et al., 2018; He al., 2020; Chen et al., 2020a), online/offline cluster- ing (Yang et al., 2016; Caron et al., 2018; Li et al., 2020; Caron et al., 2020; Grill al., 2020), etc. Among them, several recent works (He et al., 2020; Chen al., 2020a; Caron al., 2020) have achieved comparable or even better accuracy than the super- vised pre- pre-training when transferring downstream tasks, e.g. semi-supervised classification, object detec- tion.
ImageNi I T (%)
Res-30 30
70
Res-34 34
Res-18 18
60
A A
A-31.9
50
Eff-B1
SEED MoCo-V2
Eff-BO
40
Mob-V3
25
Numbers of Parameters (Millions)
Figure SEED vs. MoCo-V2 (Chen et al.. 2020c)) on ImageNet-IK linear probe accuracy. The vertical axis is the top-1 accuracy and the horizontal axis is the number of Iearnablc pa- rameters for different network architectures. Di- rectly applying self supervised contrastive Iearn- ing (MoCo-V2) does not work well for smaller architectures, while our method (SEED) leads to dramatic performance boost. Details the setting can be found in Section 4.
The aforementioned tor-performing SSL algorithms all involve large networks (e.g., ResNet-50 (He et al., 2016) or larger), with, however, little attention small networks. Empirically, we find that existing techniques like contrastive learning do not work well on small networks. For instance, the linear probe top-1 accuracy on ImageNet using MoCo-V2 (Chen et al., 2020c) is only 36.3% with MobileNet- V3-Large (see Figure 1), which much lower compared with its supervised training accuracy

---

Published as a conference paper at ICLR 2021
75.2% (Howard et al., 2019). For EfficientNet-BO, the accuracy is 42.2% compared with its supervised training accuracy 77.1% (Tan & Le, 2019). We conjecture that this is because smaller models with fewer parameters cannot effectively learn instance level discriminative representation with large amount of data.
To address this challenge, we inject knowledge distillation (KD) (Buciluă et al., 2006; Hinton et al., 2015) into self-supervised learning and propose self-supervised distillation (dubbed as SEED) as new learning paradigm. That is, train the larger, and distill to the smaller both in self-supervised manner. Instead of directly conducting self-supervised training on smaller model, SEED first trains a large model (as the teacher) in a seff-supervised way, and then distills the knowledge to the smaller model (as the student). Note that the conventional distillation is for supervised learning, while the distillation here is in the self-supervised setting without any labeled data. Supervised distillation can be formulated as training student to mimic the probability mass function over classes predicted by teacher model. unsupervised knowledge distillation setting, however, the distribution over classes is not directly attainable. Therefore, we propose simple yet effective self-supervised distillation method. Similar to (He et al., 2020; Wu et al., 2018), we maintain a queue of data samples. Given an instance, first use the teacher network to obtain its similarity scores with all the data samples in the queue as well as the instance itself. Then the student encoder is trained to mimic the similarity score distribution inferred by the teacher over these data samples.
The simplicity and flexibility that SEED brings are self-evident. ing/prototypical computing procedure retrieve the pseudo-labels latent classes. 2) The teacher model can be pre-trained with any advanced SSL approach, e.g., MoCo-V2 (Chen et al., 2020c), SimCLR (Chen al., 2020a), SWAV (Caron al., 2020). 3) The knowledge can be distilled to any target small networks (either shallower, thinner, or totally different architectures).
1) It does not require any cluster-
To demonstrate the effectiveness, we comprehensively evaluate the learned representations on series of downstream tasks, e.g., fully/semi-superry supervvsed classification, object detection, and also assess the transferability to other domains. For example, on ImageNet- Ik dataset, SEED improves the linear probe accuracy of EfficientNet-BB from 42.2% to 67.6% (a gain over 25%), and MobileNet-V V3 from 36.3% to 68.2% (a gain over 31 1) compared to MoCo-V V2 baselines, as shown in Figure and Section 4.
Our contributions can be summarized as follows:
We are the first address the problem of sulf-supervised visual representation learning for small models.
We propose a self-supervised distillation (SEED) technique to transfer knowledge from a large model a small model without any labeled data.
With the proposed distillation technique (SEED), we significantly improve the state-of-the the art SSL performance on small models.
We exhaustively compare a variety of distillation strategies to show the validity of SEED under multiple settings.
2 RELATED WORK
Among the recent literature in self-supervised learning, contrastive based approaches show prominent results on downstream tasks. Majority of the techniques along this direction are stemming from noise-contrastive estimation (Gutmann& & Hyvärinen, 2010) where the latent distribution is estimated by contrasting with randomly or artificially generated noises. Oord et al. (2018) first proposed Info-NCE to learn image representations by predicting the future using an auto-regressive model for unsupervised learning. Follow-up works include improving the efficiency (Hénaff et al., 2019), and using multi view as positive samples (Tian et al., 2019b). As these approaches can only have the access to limited negative instances, Wu et al. (2018) designed a memory-bank to store the previously seen random representations negative samples, and treat each of them independent categories (instance discrimination). However, this approach also comes with a deficiency that the previously stored vectors inconsistent with the recently computed representations during the earlier stage of pre-training. Chen et al. (2020a) mitigate this issue by sampling negative samples from a large batch. Concurrently, He et al. (2020) improve the memory-bani based method and propose to
2

---

Published as a conference paper at ICLR 2021
the momentum updated encoder for the remission of representation inconsistency. Other techniques include Misra & Maaten (2020) that combines the pretext-invariant objective loss with contrastive learning, and Wang & Isola (2020) that decomposes contrastive loss into alignment and uniformity objectiveness.
Knowledge distillation (Hinton et al., 2015) aims to transfer knowledge from cumbersome model to a smaller one without losing too much generalization power, which is also well investigated in model compression (Buciluă al., 2006). Instead of mimicking the teacher's output logit, attention transfer (Zagoruyko & Komodakis, 2016) formulates knowledge distillation on attention maps. Similarly, works (Ahn et al., 2019; Yim et al., 2017; Koratana et al., 2019; Huang & Wang, 2017) have utilized different learning objectives including consistency on feature maps, consistency on probability mass function, and maximizing the mutual information. CRD (Tian al., 2019a), which is derived from CMC (Tian et al., 2019b), optimizes the student network by similar objective to Oord et al. (2018) using derived lower bound on mutual information. However, the aforementioned efforts all focus on task-specific distillation (e.g. image classification) during the fine-tuning phase rather than a task-agnostic distillation in the pre-training phase for the representation learning. Several works natural language pre- pre-training proposed to leverage knowledge distillation for smaller yet stronger small models. For instances, DistillBert (Sanh et al., 2019), TinyBert (Jiao et al., 2019). 9). and MobileBert (Sun al., 2020), have used knowledge distillation for model compression and shown their validity on multiple downstream tasks. Similar works also emphasize the value of smaller and faster models for language representation learning by leveraging knowledge distillation (Ture et al., 2019; Sun et al., 2019). These works all demonstrate the effectiveness of knowledge distillation for language representation learning small models, while are not extended to the pre-training for visual representations. Notably, recent concurrent work CompRess (Abbasi Koohpayeganí et al., 2020) also point out the importance to develop better SSL method for smaller models. SEED closely relates to the above techniques but aims to facilitate visual representation learning during pre-training phase using distillation technique for small models, which as far as we know has not yet been investigated.
3 METHOD
3.1 PRELIMINARY ON KNOWLEDGE DISTIILATTOO
Knowledge distillation (Hinton et al., 2015; Buciluă et al., 2006) is an effective technique to transfer knowledge from a strong teacher network to target student network. The training task can be generalized as the following formulation:
0g arg min
Σ Crup (x,, As, y/) Lusssin (x,, As, 0r),
where x; is an image, Y: is the corresponding annotation, O5 is the parameter set for the student network, and Or is the set for the teacher network. The loss Lup is the alignment error between the network prediction and the annotation. For example in image classification task (Mishra & Marr, 2017; Shen & Savvides, 2020; Polino et al., 2018; Cho & Hariharan, 2019), is normally a cross entropy loss. For object detection (Liu et al., 2019; Chen et al., 2017), it includes bounding box regression as well. The loss of Ldisill is the mimic error of the student network towards a pre-trained teacher network. For example in (Hinton al., 2015), the teacher signal comes from the softmax prediction of multiple large-scale networks and the loss is measured by the Kullback-Leibler divergence. In Romero al. (2014), the task to align the intermediate feature map values and to minimize the squared 12 distance. The effectiveness has been well demonstrated in the supervised setting with labeled data, but remains unknown for the unsupervised setting, which is our focus.
3.2 SELF-SUPERVISED DISTILLATION FOR VISUAL REPRESENTATION
Different from supervised distillation, SEED aims to transfer knowledge from large model to small model without requiring labeled data, so that the learned representations in small model can be used for downstream tasks. Inspired by contrastive SSL, formulate simple approach for the distillation on the basis of instance similarity distribution over a contrastive instance queue. Similar to He et al. (2020), we maintain instance queue for storing data samples' encoding output from the
3

---

Published as a conference paper at ICLR 2021
L2 Norm
SoftMax
Student
Student Probability
Aug
Enqueue
+/
D
Cros-Entropy
L2- Lo-Morm
SoftM ax
Teacher
Inner product
Instance Queue: D
Pre- trained and Frozen
Teacher Probability
Figure Illustration of our self-supervised distillation pipeline. The teacher encoder pre-trained by SSI. and kept frozen during the distillation. The student encoder is trained by minimizing the cross entropy of probabilities from teacher & student for augmented view of an image, computed over dynamically maintained queue.
teacher. Given new sample, we compute its similarity scores with all the samples in the queue using both the teacher and the student models. We require that the similarity score distribution computed by the student matches with that computed by the teacher, which is formulated as minimizing the cross entropy between the student and the teacher's similarity score distributions (as illustrated in Figure 2).
Specifically, for a randomly augmented view X; of an image, it is first mapped and normalized into feature vector representations z f (x,)/Ilf² (x,)l2. and z f (xx)/Ilff (x,),22, where 2 2z² E IRP and f and f denote the teacher and student encoders, respectively. Let D [d]...k]] denote the instance queue where K is the queue length and d, is the feature vector obtained from the teacher encoder. Similar to the contrastive learning framework, D is progressively updated under the "first-in fersstiiffisste first-out" strategy as distillation proceeds. That is, we en-queue the visual features of the current batch inferred by the teacher and de-queue the earliest of iteration. Note that the maintained samples in queue are mostly random and irrelevant to the entropy between the similarity score distribution computed
samples the end
target instance x;- Minimizing the by the student and teacher based on D softly contrasts x, with randomly selected samples, without directly aligning with the teacher encoder. To address this problem, we add the teacher's embedding (z!)) into the queue and form D = [k. dx+1] with dx+1 = =
Let ppppp="":> (xx; Or; D denote the similarity score between the extracted teacher feature zf and dy's (j = 1, K + 1) computed by the teacher model. p" (x;; fr; D+) is defined as
exp(z d,/T²)
p= D+) (p
P3 EaLD exp(z. d//T)"
(2)
and TT is temperature parameter for the teacher. Note, we use ()T to represent the feature from the teacher network and use (.) to represent the inner product between two features.
Similarly let p (x:;; D+) denote the similarity score computed by the student model, which is defined as
exp(z, dd//5) Σa-D+ exp(z, d/79)²
Dd 8 (x:; D+) [v] ... ...P.PP.... Px+1l
where D
and 8 is temperature parameter for the student.
Our self-supervised distillation can be formulated as minimizing the cross entropy between similarity scores of the teacher, p" (xx;; D+), and the student, d (x; 0s, D+), over all the instances X,, that is.
N
0g arg min As
TT (x1; Bas0rr.Dt D+) log log (x;; 1x;00, D*)
(4)
N
exp(z] d Ed-D+ exp(z,' d/7T) Ed-D+ exp(2ş d/73)*
log exp(z d
arg min 0g
Since the teacher network is pre-trained and frozen, the queued features are consistent during training the student network. The higher the value of p is, the larger weight will be laid on r/. Due
4

---

Published as a conference paper at ICLR 2021
to the 12 normalization, similarity score between 2 and dk+1 remains constant 1 before softmax normalization, which is the largest among 23. Thus, the weight for p is the largest and can be adjusted solely by tuning the value of -" By minimizing the loss, the feature of of can be aligned with z and meanwhile contrasts with other unrelated image features in D. We further discuss the relation of these two goals with our learning objective in Appendix A.5.
Relations with Info-NCE loss. When -J 0, the softmax function for p" smoothly approaches to a one-hot vector, where pk+ equals and all others 0. In this extreme case, the loss becomes
exp(z? /T/TT)
ACCE
log
(5)
Σd-D+ exp(z d/r))
which is similar to the widely-used Info-NCE loss (Oord al., 2018) in contrastive-baset SSL (see discussion in Appendix A.6.
4 EXPERIMENT
4.1 PRE-TRAINING
Self-Supervised Pre-training of Teacher Network. By default, we use MoCo-V2 (Chen et al., 2020c) to pre-train the teacher network. Following (Chen et al., 2020a), we ResNet the network backbone with different depths/widths and append a multi-layer-perceptron (MLP) layer (two linear layers and one ReLU (Nair & Hinton, 2010) activation layer in between) the end of the encoder after average pooling. The dimension of the last feature dimension is 128. All teacher networks are pre- pre-trained for 200 epochs due to the computational limitation unless explicitly specified. As our distillation is independent with the teacher pre- pre-training algorithm, we also show results with other self-supervised pre-trained models for teacher network, egg. SWAV (Caron et al., 2020), SimCLR (Chen et al., 2020a).
Sule-Supervised Distillation on Student Network. We choose multiple smaller networks with fewer learnable parameters as the student network: MobileNettw3-Large (Howard et al., 2017), EfficientNet-BO (Tan & Le, 2019), and smaller ResNet with fewer layers (ResNet-18, 34). Similar to the pre-training for teacher network, we add one additional MLP layer the basis of the student network. Our distillation is trained with a standard SGD optimizer with momentum 0.9 and a weight decay parameter of le-4 for 200 epochs. The initial learning rate is set 0.03 and updated by cosine decay scheduler (Nair& & Hinton, 2010) with 5 warm-up epochs and batch size 256. In Eg. 4, the teacher temperature is set - 0.01 and the student temperature is S 0.2. The queue size of K is 65,536. In the following subsections and appendix, we also show results with different hyper-parameter values, e.g., for -" and K.
4.2 FINE-TUNING AND EVALUATION
In order to validate the effectiveness of self-supervised distillation, choose assess the perfor- mance of representations of the student encoder on several downstream tasks. We first report its performances of linear evaluation and semi-supervised linear evaluation on the ImageNet ILSVRC- 2012 (Deng et al., 2009) dataset. To measure the feature transferability brought by distillation, we also conduct evaluations on other tasks, which include object detection and segmentation on the VOCO7 (Everingham et al.) and MS-COCO (Lin et al., 2014) datasets. At the end, we compare the transferability of the features learned by distillation with ordinary self-supervised contrastive learning on the tasks of linear classification on datasets from different domains.
Linear and KNN Evaluation on ImageNet. We conduct the supervised linear classification on ImageNet-1K, which contains ~ 1.33 images for training, and 50,000 images for validation, spanning 1,000 categories. Following previous works in (He et al., 2020; Chen et al., 2020a), we train a single linear layer classifier on top of the frozen network encoder after self-supervised pre- training/distillttion. SGD optimizer is used to train the linear classifier for 100 epochs with weight decay to be 0. The initial learning rate is set as 30 and is then reduced by factor of 10 at 60 and 80 epochs (similar as in Tian et al. (2019a)). Notably, when training the linear classifier for MobileNet and EfficientNet, reduce the initial learning rate to 3. The results reported in terms of Top-1
5

---

Published as a conference paper at ICLR 2021
Table 1: ImageNet-Ik Ik test accuracy (%) using KNN and linear classification for multiple students and MoCo- v2 pre-trained deeper teacher architectures. denotes MoCo-V2 self-supervised learning baselines before distillation. indicates using a deeper teacher encoder pre- pre-trained by SWAV, where additional small small-patches are also utilized during distillation and trained for 800 epochs. K denotes Top-1 accuracy using KNN. T-1 and T-5 denote Top-1 and T accuracy using linear evaluation. First column shows Top-1 Acc. of Teacher network. First row shows the supervised performances of student networks.
S
Eff-bo T-1
Eff-bi T-1
Mob-v3 T-1
R-18
R-34
T-1
T
K
T-5
K
T-5
K
T-5
K
T-1
T-5
K
T-1
T-5
Supervised Acc.
77.3
79.2
75.2
72.1
75.0
30.0
42.2
68.5
34.4
50.7
74.6
27.5
36.3
62.2
36.7
52.5
77.0
41.5
57.4
81.6
R-50
46.0
61.3
82.7
46,1
61.4
83.1
44.8
55.2
80.3
43.4
57.9
82.0
45.2
58.5
82.6
67.4
+10.7
18.8
+17.3
+18.9
+181
+6.7
+5.1
R-101
50.1
63.0
83.8
50.3
63.4
84.6
48.8
59.9
83.5
48.6
58.9
82.5
50.5
61.6
84,9
70.3
+20L1
-208
+1533
+159
+12.7
+0.00
+21 13
+236
+21.3
+11.9
+6.4
+5.5
-9.0
+4.2
+3.3
R-152
50.7
65.3
86.0
52.4
67.3
86,9
49.5
61.4
84.6
49.1
59.5
83.3
51.4
62.7
85.8
74.2
+20.7
+23.1
117.5
+10.0
166.6
+12.3
+22.0
+25.1
+224
+124
+7.0
16.3
+9.9
+5.3
142
R50
57.4
67.6
87.4
60.3
68.0
87.6
55.9
68.2
88,2
55.3
63.0
84.9
58.2
65.7
86.8
77.3
+1733
+26.0
+5.2
Net--bo
MobileNet -arge
18
Top-1 Accuracy (96)
00
1%
1%
1%
10%
10%
30
10%
100%
100%
100%
Number of Parameters of the Teacher (Millions)
Figure ImageNet- Ik Top-1 accuracy for semi- supervuper evaluations using 1% (red line). 10% (blue line) of the annotations for linear fine-tuning, in comparison with the fully supervised (green line) linear evaluation baseline for SEED. For the points whose Teacher's 's number of parameters is at O. we show the supe-super linear evaluation results of MoCo-V V2 without any distillation. The Student models tend to perform better the semi-supervised tasks after distillation from larger Teachers.
and Top-5 accuracy. We also perform classification using K-Nearest Neighbors (KNN) based on the learned 128d vector from the last MLP layer. The sample is classified by taking the most frequent label of its K (K = 10) nearest neighbors.
Table 1 shows the results with various teacher networks and student networks. We list the baseline of contrastive self-supervised pre-training using MoCo-V2 (Chen et al., 2020c) in the first row for each student architecture. We can clearly that smaller networks perform rather worse. For example, MobileNet- V3 can only reach 36.3%. This aligns well with previous conclusions from (Chen et al., 2020a;b) that bigger models are desired perform better in contrastive-based self-supervised pre- training. We conjecture that this is mainly caused by the inability of smaller network to discriminate instances in large-scale dataset. The results also clearly demonstrate that the distillation from a larger network helps boosting the performances of small networks, and show obvious improvement. For instance, with MoCo-V2 V2 pre-trained ResNet- (for 400 epochs) as the teacher network, the Top-1 accuracy of MobileNet- La-ag can be significantly improved from 36.3% to 61.4%. Furthermore, we use ResNet-50x2 (provided in Caron al. (2020)) as the teacher network and adopt the multi-crop trick (see A.2 for details). The accuracy can be further improved to 68.2% (last row of Table 1) for MobileNet- V3-LLarge with 800 epochs of distillation. We note that the gain benefited from distillation becomes more distinct smaller architectures and we further study the effect of various teacher models in ablations.
Semi-Supervised Evaluation on ImageNet. Following (Oord et al., 2018: Korblith et al., 2019; Kolesnikov et al., 2019), we evaluate the representation on the semi-supervised task, where a fixed provided with the annotations.
1% 10% subsets of ImageNet training data (Chen et al., 2020a) After the self-supervised learning with and without distillation, we also train a classifier on top of the representation. The results shown in Figure 3, where the baseline without distillation is depicted

---

Published as a conference paper at ICLR 2021
Table 2: Object detection and instance segmentation results using contrastive self-supervised learning and SEED distillation using ResNet-18 18 hackhone: bounding-her beerddxxggxxxxoe-o AP (AP*)) and mask AP (Apmk) evaluated on VOC07-val and COCO testing split. More results on different backbones he found in the Appendix. Subscript in green represents improvement is larger than 0.3.
VOC Obj. Det.
COCO Obj. Det.
COCO Inst. Segm.
S
T
Ap
APE8
AP
Ap
AP
AP75
Apm
AP
AP75
X
46.1
74.5
48.6
35.0
53.9
37.7
31.0
51.1
33.1
R-50
46.1 0.00
(4.8(+0.3)
(+..)
35.3(+0.3)
54.2(+0.3)
37.8(-0.1)
31.1(+0.1)
51.1(0.0)
33.2(+0.D)
R-18 R-101
(6.8(+0.7)
(+13)+
(+.7()
(4.3
(+.3
(7.9(+0.2)
(+.3
51.3(+0.2)
33.4(40.3)
R-152
46.8(+0.7)
(+14)+ (0.2(+
35.4(+0.4) 54.4(+0.5)
38.0(+0.3)
(1.3()0.3)
(1.4(+0.3) 33.4(+0.3)
CIFAR-10 10
CIFAR-100
SUN-397
Top- Accuracy (%)
65.0
49
68
62.5
48
60.0
17
64
57.5
ResNet-18
46
Efficient
55.0
45
20
40
60
20
40
60
20
40
60
Number of Parameters of the Teacher (Millions)
Figure ImageNet- Ik Accuracy (%) student network (EfficientNet-BB BO ResNet- transferred to other domains (CIFAR-10, CIFAR-100, SUN-397 datasets) with and without distillation from lager architectures (ResNet-50/101/152).
when teacher parameters are O. As we can see, the accuracy is also improved remarkably with SEED distillation, and a stronger teacher network with more parameters leads to better performed student network.
Transferring to Classification. To further study whether the improvement of the learned represen- tations by distillation is confined to ImageNet, we evaluate on additional classification datasets to study the generalization and transferability of the feature representation. We strictly follow the linear evaluation and fine-tuning settings from (Kornblith et al., 2019; Chen et al., 2020a; Grill et al., 2020), that a linear layer trained on the basis of frozen features. We report Top-1 Accuracy of models before and after distillation from various architectures on CIFAR-10. CIFAR- (Krizhevsky et al., 2009), SUN-397 (Xiao et al., 2010) datasets (see Figure 4). More details regarding pre-prot preppoccsssigg and training can be found in A.1. Notably, observe that our distillation surpasses contrastive self-supervised pre-training consistently on all benchmarks, verifying the effectiveness of SEED. This also proves the generalization ability of the learned representations from distillation to wide range of data domain and different classes
Transferring to Detection and Segmentation. We conduct two downstream tasks here. The first is Faster R-CNN (Ren et al., 2015) model for object detection trained on VOC-07+12 train+val set and evaluated on VOC-07 test split. The second is Mask R-CNN (He et al., 2017) model for the object detection and instance segmentation on COCO 2017 dataset (Lin et al., 2014). The pre-trained model serves as the initial weight and following He et al. (2020), we fine-tune all the layers of the model. More experiment settings can be found in A.2. The results are illustrated in Table 2. As we can see, on VOC, the distilled pre-trained model achieves large improvement. With ResNet- 152 as the teacher network, the Resnet 8- b-based Faster R-CNN model shows +0.7 point improvement on AP, +1.4 improvement on AP50 and +1.6 on AP75. On COCO, the improvement relatively minor and the reason could be that COCO training set has ~ 18k training images while VOC has only ~ 16.55k training images. A larger training set with more fine-tuning iterations reduces the importance of the initial weights.
4.3 ABLATION STUDY
We now explore the effects of distillation using different Teacher architectures, Teacher Pre-training algorithms, various distillation strategies and syper-parameters.
7

---

Published as a conference paper at ICLR 2021
Table 3: ImageNet-Ik Accuracy (%) of student net- work (ResNet-18) distilled from variants self- supervised ResNet-50, P-E/D-F represent the pre- training and distillation epochs. T./ represent testing accuracy of Teacher and Student. represents distillation using additional small patches. First row the ResNet-18 SSL baseline using MoCo-v2 trained
64
EfficientNet-bl
ImageNet Top- Aocuracy (%)
Width x
Width x
62
ResNet- 18
60
for 200 epochs.
Teacher
P-F
D-E
T. Top-1
S. Top-1 I S. Top-5
58
x
X
52.5
77.0
MoCo
200
200
60.6
52.1
77.0
SimCLR
200
200
65.6
57.5
81.7
56
MoCo-v2
200
200
67.4
57.9
82.0
Teacher: R50 R101
R152
R50
R101
R152
800
200
71.1
60.5
83.5
Figure Accuracy (%) student networks
SWAV
800
100
75.3
61.1
83.8
800 200
75.3
61.7
84.2
(EfficientNettb00 and ResNet-18) on ImageNet dis-
800
400
75.3
62.0
84.4
tilled from wider MoCo-v2 pre-trained ResNet
SWAV
800 200
75.3
62.6
84.8
(Re5Neet50110//15222,,
Different Teacher Networks. Figure summarizes the accuracy of ResNet- 18 and EfficientNet-Bi distilled from wider and deeper ResNet architectures. We clear performance improvement as depth and width of teacher network increase: compared to ResNet-50, deeper (ResNet- 101) and wider (ResNet-50x2) substantially improve the accuracy. However, further architectural enlargement has relatively limited effects, and we suspect the accuracy might be limited by the student network capacity in this case.
Different Teacher Pre-training Algorithms. In Table 3, we show the Top-1 accuracy of ResNet-18 distilled from ResNet-50 with different pre-training algorithms, i.e., MoCo-V1 (He et al., 2020), MoCo-V2 (Chen et al., 2020c), SimCLR (Chen et al., 2020a), and SWAV (Caron et al., 2020)). Notably, the aforementioned methods all unanimously adopt contrastive-based pre-training except SWAV, which is based upon online clustering. We find that our SEED is agnostic to pre-training approaches, making it easy to use any self-supervised models (including clustering-bet based approach like SWAV) in self-supervised distillation. In addition, we observe that more training epochs for both teacher SSL and distillation epochs can bring beneficial gain.
Other Distillation Strategies. We explore several alternative distillation strategies. 12-Distance: where the 12-distance of teacher student's embeddings are minimized, motivated by Romero et al. (2014). K-Means: exploit K-Means clustering to assign a pseudo-label based the teacher network's representation. Online Clustering: we continuously update the clustering centers during distillation for pseudo-label generation. Binary Contrastive Loss: we adopt Info-NCE alike loss for contrastive distillation (Tian et al., 2019a). We provide details for other strategies A.4. Table shows the results for each method ResNet-18 (student) distilled from ResNet-50. From the results, the simple /2-distance minimizing approach can achieve decent accuracy, which demonstrates the effectiveness of applying the distillation idea to the self-supervised learning. Beyond that, we study the effect of the original SSL (MoCo-Va V2) supervision as supplementary loss to SEED and find it does not bring additional benefits distillation. We find close results from these two strategies (Top-1 linear Acc.), SEED achieves 57.9%, while SEED MoCo-V V2 achieves 57.6%. This implies that the loss of SEED can to a large extent cover the original SSL loss, and it is not necessary to conduct SSL any further during distillation. Meanwhile, our proposed SEED outperforms these alternatives with highest accuracy, which shows the superiority of aligning the student towards the teacher and contrasting with the irrelevant samples.
Other Hyper-Panne Parameters. Table summarizes the distillation performances on multiple datasets using different temperature TT. We observe better performance when decreasing 7" to 0.01 for ImageNet-Ik 1k and CIFAR-10 10 dataset, and to 1c-3 for CIFAR-100 datasets. When T is large. the softmax-normalized similarity score of p. between z!" and instance d in the queue D + also becomes large, which means the student's feature should be less discriminative with the features of other images some extent. When -" is O, the teacher model will generate one-hot vector, which only treats 2i as a positive instance and all others in the queue as negative. Thus, the best 7 is a trade-off' depending on the data distribution, We further compare effect of different parrmmerrseesers A.8.
8

---

Published as a conference paper at ICLR 2021
Table 4: Top- accuracy of linear classifica- results on ImageNet using different distillation strategies on Re (student) and ResNet-50 (teacher) architectures.
Table 5: Effect 7" for the distillation of ResNet- 18 (student), ResNet-50 (teacher) on multiple datascts.
ImageNet
CIFAR-10
CIFAR-100
T
Method
Top-1 1 Acc. Top-5 Acc.
Top-1
Top-5
Top-1
Top-1
12-Distance
55.3
80.3
0.3
54.8
80.0
78.7
46.6
K-Means
51.0
75.8
0.1
54.9
80.1
83.0
50.1
Online Clustering
56.4
81.2
0.05
56.5
81.3
84.4
56.2
Binary Contr. Loss
57.4
81.5
0.01
57.9
82.0
87.5
60.6
SEED MoCo-V2
57.6
81.8
1e-3
57.6
81.8
86.9
60.8
SEED
57.9
82.0
5 CONCLUSIONS
Self-Supervised Learning is acknowledged for its remarkable ability in learning from unlabeled, and large scale data. However, a critical impedance for the SSI. pre-training on smaller architecture comes from its low capacity of discriminating enormous number of instances. Instead of directly learning from unlabeled data, we proposed SEED as a novel sulf-supervised learning paradigm, which learns representation by self-supervised distillation from bigger SSL pre-trained model. We show in extensive experiments that SEED effectively addresses the weakness of supervueeee learning for small models and achieves state-of-the of-the-art results various benchmarks of small architectures.
REFERENCES
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self-supervised learning by compressing representations. Advances in Neural Information Processing Systems, 33, 2020.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9163-9171, 2019.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09880 09882, 2020.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances Neural Information Processing Systems, pp. 742-751, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint wrXiv:202.005009 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXhv:2006.10029 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0420 2020c.

---

Published as a conference paper at ICLR 2021
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4794 4802, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://wwwpasccal
The
Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation feature decoupling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10364-10374, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0777 2018
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint 0tt::20006007333 2020.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. In European conference on computer vision, pp. 87-102. Springer, 2016.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, arXiv preprint arXiv:1905.09272 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1XX3::5533.05311 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019,
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. 2017.
arXiv preprint
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. bery Distilling bert for natural language understanding. arXiv preprint 2019.
10

---

Published as a conference paper at ICLR 2021
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self self-supervised visual representa- tion learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1920-1929, 2019.
Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: representation training for model compression. In International Conference on Machine Learning, pp. 3509-3518, 2019.
Learned intermediate
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.044 04966, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation, In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arXiv: /711v 1,08552, 2017.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariee representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6707-67 17, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1eXX.v1807.03745 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, pp. 8026-8037, 2019.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint arXiv: 080200568, 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards veal-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99,9999999 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: /910.0/ 108, 2019.
Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint Xiiv:2009.08453 2020.

---

Published as a conference paper at ICLR 2021
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint
2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02988 2020.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations, 2019a.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 19X6v:9906.080 05849, 2019b.
Iulia Ture, Ming-We Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint
2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 2020.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-34444 3492. IEEE, 2010.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5147-5156, 2016.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.0398, 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE international conference on computer vision, pp. 1476-1485, 2019.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2547-2555, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666, Springer, 2016.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
We discuss more details and different hyperparameters for SEED during distillation.
A.
PSEUDO-IMPLEEN ISEUDO-IAPLEMENTTAIONS
We provide pseudo-code of the SEED distillation in PyTorch Paszke et al. (2019) style:
'Q: maintaining queue of previcus representations:
I: Cumbersome encoder as Teacher.
3 S: Target encoder as Student.
temp T, temp_S: temperatures of the Teacher & Student.
activate evaluation mode for Teacher to freeze BN and updation. T.eval ()
for images in enumerate (loader) Enumerate single crop-view
11
12
$ augment image to got one identical view
13
images aug (images)
15
batch-size
16
maages.saap. [0]
17
18
# extract embedding from 8: 1 X D
S S (images)
20
x_$ terch. norm (K_S, p-2, dim=1)
21
22
use the gradient-free mode
with torch torch.no_grad())
X_T (imagee) embedding from 1 D
25
X torch. norm (X_T, p=2, dim=1)
26
27
insert the current batch embedding from T enqueue (Q, X_I)
30
probabil lity scores distríbution for T, S: (N + 1)
31
S_Dist torch, einsum ('bd, dn -> bn', [X_S], Q.t .c.clnn (omee())deeeeeee) deeaach ())
32
T Dist torch. einsum (' bd, dn -> bn', [X_T], Q.t O.clone detach ()
34
Apply temperatures for soft-labele S_Dist /- temp_s
35
36
Dist SoftMax (T (T_Dist/temp_T, dim=1)
loss computation, use log softmax for stable computation
39
loss -torch.mul (T Distist Log_ (o Disti dim=1)) ) sum ()
40
41
update the random sample queue dequeue (Q, B) pop-out earliest B instances
43
44
SGD updation
45
lass.backwardd)
update (8. params)
13

---

Published as a conference paper at ICLR 2021
A.1.1 DATA AUGMENTATIONS
Both our teacher pre-training and distillation adopt the data augmentations follows:
Random Resized Crop: The image is randomly resized with a scale of {0.2, 1.0), then cropped the size of 224x224.
Random Color Jittering: with brightness to be {0.4, 0.4, 0.4, 0.1] with probability at 0.8.
Random Gray Scale transformation: with probability at 0.2.
Random Gaussian Blur transformation: with (0.1, 0.2) and probability at 0.5.
Horizontal Flip: Horizontal flip is applied with probability at 0.5.
A.1.2 PRE-TRAINING AND DISTILLATION ON MOBILENET AND EFFICIENTNET
MobileNet (Howard et al., 2017) and EfficientNett (Tan Le, 2019) have been considered as the smaller counterparts with larger models, i.e., ResNet-50 (with supervised training, EfficientNet- BO hits 77.2% Top-1 Acc., and MobileNer- V3-lare reaches 72.2% on ImageNet testing split). Nevertheless, un-matched performances training: i.e., Self-Super Learning (MoCo-V2 on MobileNet-V V3 only yields 36.3% Top-1 Acc. on ImageNet. We conjecture that several reasons might lead to this dilemma:
observed in the task of self-supervised contrastive pre-
1. The inability of models with less parameters for handling large volume of categories and data, which exists also in other domains, i.e., face recognition (Guo al., 2016; Zhang et al., 2017).
2. Less possibility for optimum parameters to be chosen when transferring to downstream tasks: models with more parameters after pre-training might produce plenty comucoopa of optimum parameters for fine-tuning.
To narrow the dramatic performance gap between smaller architectures using contrastive SSL with the larger, we explore with architectural manipulations and training syper-parammters. In specific, we find that by adding deeper projection head largely improves the representation quality, a.k.a., better performances on linear evaluation. We experiment with adding one additional linear projection head on the top of convolutional backbones.
Similarly, we also expand the MLP projection head on EfficientNet-bO. Though recent work shows that fine-tuning from middle layer of the projection head can produce largely different result (Chen et al., 2020b), we consistently just use the representations from convolutional trunk without adding extra layers during the phase of linear evaluation. As shown in Table 6, pre-training with deeper projection head dramatically helps the improvement on linear evaluations, adding 17% Top-1. 1. Acc. for Mobile-v3. -arger and we report the improved baselines in the main paper (see the first row in Table 1 of the main paper). We keep most of the hyper-paun parameters as the distillation on ResNet except reducing the weight-decay of them to le-5, following (Tan & Le, 2019; Sandler et al., 2018).
Table 6: Linear evaluations on ImageNet of EfficientNet and MobileNet pre- preitrained using MoCo-v2. A deeper projection head largely boosts the linear evaluation performances smaller architectures.
Model
Deeper MLPs
Top-1 1 Ace. Top-5 Acc.
EfficientNettbo0
x 39.1
64.6
EfficientNet-b0
42.2
68.5
Mobile-v3-
X
19.0
41.3
Mobile-v3-large
36.3
62.2
A.2 ADDITIONAL DETAILS OF EVALUATIONS
We list additional details regarding evaluation experiments in this section.
ImageNet-Ik Semi-Supervised Linear Evaluation. Following Zhai et al. (2019); Chen et al. (2020a), we train the FC layers the basis of our student encoder after distillation using fraction
14

---

Published as a conference paper at ICLR 2021
Table 7: Before and after distillation Top- 1/5 test accuracy (%) ImageNet of EfficienttNet-00 and MobileNet- large without deeper MLPs.
Student
Teacher
Top-1
Top-5
X
39.1
64.6
ResNet-50
59.2
81.2
EfficientNet-b0
ResNet-100 101
62.8
84.7
ResNet-152
63.3
85.6
19.0
41.3
ResNet-50
50.9
77.7
MobilcNet-v3
ResNet- 101
57.6
82.6
ResNet-152
58.3
82.9
Table 8: ImageNet-lk test accuracy (%) under KNN and linear classification on ResNet-550 encoder with deeper, MoCo-V2/SWAV pre-trained teacher architectures. X denotes MoCo-V2 self-supervised learning baselines before distillation. indicates using stronger teacher encoder pre-traines by SWAV with additional amall-patches during distillation.
Stud.
ResNett-0
Teac.
Epoch
KNN
Top-1
Top-5
X
200
46.1
67.4
87.8
ResNet-50
46.1
67.5
87.8
200
.0.0
10.1
.0.0
ResNet-101
52.3
69.1
88.7
200
46.2
+1.7
-0.9
ResNet-152
53.2
70.4
90.5
200
ResNet-50
59.0
74.3
92.2
800
+12.9
+6.9
+44
of labeled ImageNet- Ik dataset (1% and 10%), and evaluate it on the whole test split. The fraction of labeled dataset is constructed in a class-balanced way, with roughly and 128 images per class". We use SGD optimizer and set initial learning rate to be 30 with multiplier BatchSize/256 without weight decaying for 100 epochs. We use the step-wise scheduler for the learning rate updating with 5 warm-up epochs, and the learning rate is reduced by 10 at 60 and 80 epochs. On smaller architectures like EfficientNet and MobileNet, we reduce the initial learning rate to 3. During training, the image is center-cropped to the size of 224x x224 with just Random Horizontal Flip the data augmentation. For testing, we first resize the image to 256 and use the center cropped 224 for pre-processing. In Table 8, we show the distillation results larger encoder (ResNet-550) when using different teacher networks.
Transfer Learning. We test the transferability of the representations learned from self-supervised distillation by conducting the linear evaluations using offline features on several other datasets. Specifically, single layer logistic classifier is trained following (Chen et al., 2020a; Grill al., 2020) using SGD optimizer without weight decay and momentum parameter at 0.9. We use CIFAR- 10, CIFAR- 100 (Krizhevsky et al., 2009) and SUN-397 (Xiao et al., 2010) as our testing beds.
As the size for CIFAR dataset 32x32, we resize all images to x24 pixels along the shorter side using bicubic resampling method, followed by center crop operation. We set the learning rate le-3 constantly and train it for 120 epochs. The hyper-parameters are searched using 10 fold cross-validation on the train split and report its final top-1 accuracy on the test split.
CIFAR:
*The full image ids for semi-supervised evaluation on ImageNet-1k can be found at https: httpp:::////tit///// com/ googl research rch rttre//meeeeeeeeeeeeeeeeeeee magenet sesgeent__uubets...
15

---

Published as a conference paper at ICLR 2021
Table 9: Object detection and instance segmentation fine-tuned on VOCO7: bounding-box AP (AP")) and mask AP (APT)A) evaluated on VOCO7-val. The first row shows the baseline from MoCo-v2 backbones without distillation.
VOC Object Detection
Student
Teacher
Aphh
Aph
APh
53.6
79.1
58.7
ResNet-50
53.7 (+0.1)
79.4 (+0.3) 59.2 (+0.5)
ResNet-34
ResNet-10
101
54.1 (+0.5)
79.8 (+0.7)
59.1 (+0.4)
ResNet-152
54.4 (+08)
80.1 (+1.) 59.9 (+1.2)
x
57.0
82.4
63.6
ResNet-50
57.0 (40.0)
82.4 (-0.0) 63.6 (+0.0)
ResNet-50
ResNet- 101 57.1 (40.1
82.8 (+0.4)
63.8 (+0.2)
152 57.3 (+0.3)
82.8 (+0.4) 63.9 (+0.3)
Table 10: Object detection and instance segmentation fine-tuned on COCO: bounding-box AP (AP4)) and mask AP (AP"*)) evaluated on COCO-va12017. The first several rows show the baselines from unsupervised backbones without distillation.
Object Detection
Instance Segmentation
Student
Teacher
Apbb
Apbb
APbb 75
Apmk
Apmk
A Apmk
x
38.1
56.8
40.7
33.0
53.2
35.3
ResNer50
38.4 (+0.3)
57.0 (+0.2)
41.0 (+0.3)
33.3 (+0.3)
53.6 (+0.4)
35.4 (+0.1)
ResNet34
ResNet101
38.5 (40.4)
57.3 (+0.5)
41.4 (+0.7)
33.6 (+0.6)
54.1 (+0.9)
35.6 (+0.3)
ResNet152
38.4 (40.3)
57.0 (+0.2)
41.0 (+0.3) 33.3 (+0.3)
53.7 (40.5)
35.3 (+0.0)
We further extend our transferring evaluation to the scene dataset SUN-397 for more diverse testing. The official dataset specifies 10 different train/test splits, with each contains 50 images per category covering 397 different scenes. We follow (Chen et al., 2020a; Grill al., 2020) and use the first train/test split. For the validation set, we randomly pick 10 images (yielding 20% of the dataset), with identical optimizer parameters as CIFAR.
SUN-397:
Object Detection and Instance Segmentation. As indicated by (He et al., 2020), features produced by self-supervised pre-training have divergent distributions in downstream tasks, thus resulting the supervised pre-training picked hyper-parameters not applicable. To relieve this, He et al. (2020) uses feature normalization during the fine-tuning phase and train the BN layers. Different from previous transferring and linear evaluations where we exploit only offline features, model for detection and segmentation trained with all parameters tuned. For this reason, annotations on COCO for segmentation gives much higher influence for the backbone model than the VOC dataset (see Table 9), and gives an offset to the pre-traine pre-training difference (see Table 10). Thus, this makes the performance boosting by pre-training less obvious, and leads to trivial AP differences before and after distillation.
Object Detection on PASCAL VOC-07: We train a C4 (He et al., 2017) based Faster R-CNN (Ren et al., 2015) as the detector with different ResNet architectures (ResNet-18, ResNet-34 and ResNet-50) for evaluating the transferability of features for object detection tasks. We use Detectron2 (Wu et al., 2019) for the implementations. We train detector for 48k iterations with a batch size of 32 (8 images per GPU). The base learning rate is set to 0.01 with 200 warm-up iterations. We set the scale of images for training as [400, 800] and 800 at inference.
Object Detection and Segmentation COCO: We use Mask R-CNN (He et al., 2017) with the C4 backbone for the object detection and instance segmentation task on COCO dataset, with 2x schedule. Similar to the VOC detection, we tune the BN layers and all parameters. The model is trained for 180k iterations with initial learning rate set to 0.02. We set the scale of images for training [600, 800] and 800 inference.
16

---

Published as a conference paper at ICLR 2021
Table 11:
Linear evaluations on ImageNet ResNet-18 after distillation from the SWAV pre-trained ResNet-50 using either single view, cross-views, or small patch views.
Method
Multi- Views))
Top-1 Acc.
Top-5 Acc.
Identical- Viee
x2224
61.7
84.2
Cross-Views
2x224
58.2
81.7
Multi-Crops + Cross- Views
1x224 + 6x96x96
61.9
84.4
Multi-Crops Identical-Vin
62.6
84.8
S
S
S
S
T
T
T
(a)
(b)
(c)
(d)
Figure 6: We experiment with different strategies of using views during distillation, which include: (a). Identical view for distillation. (b). Cross view distillation. (c). Large-small cross view distillation. (d). Large-small identical view distillation.
A.3 SINGLE CROP V.S. MULTI-CROPS VIEW(S) FOR DISTILLATION
In contrary with most contrastive SSL methods where two different augmented views of an image utilized as the positive samples (see Figure 6-a). SEED uses an identical view for each image (see Figure 6-b) during distillation and yields better performances, we have also experimented with two strategies of using small patches. To be specific, we follow the set-up in SWAV (Caron et al., 2020), that small patches of the size 96x96 sampled the scale of (0.05, 0. 04). Then, we apply the same augmentations as introduced previously as data pre-processing. Figure. 6-c shows the way that is similar SWAV for small patth learning, where both large and small patches are fed into the student encoder, with the learning target (z" to be the embedding of large view from the teacher encoder. Figure. 6-d is the strategy we use during distillation, that both views are fed into student and teacher to produce the embeddings for small-views (z.) z and large views (z,,, zi') ). Based on that, the distillation is formulated separately on the small and large views. Notably, we maintain two independent queues for storing historical data samples for the large and small views.
is shown in Table. 11. In addition,
A.4 STRATEGIES FOR OTHER DISTILLATION METHODS
We compare the effect of distillation using different strategies with SEED.
12-Distance: We train the student encoder by minimizing the squared 12-distance of representations from student (z;)) and teacher (z,) for an identical view X;-
K-Means: We experiment with the K-Means clustering method to retrieve pseudo class labels for distillation. Specifically, we first extract offline image features using the SSL pre-trained Teacher network without any image augmentations. Based on this, we conduct our K-Means clustering with 4k and 16k unique centroids. Then the final centroids are used to produce pseudo labels for unlabelled instances. With that, we carry out the distillation by training the model on classification task using the produced labels the ground-truth To avoid trivial solutions that the majority of images are assigned to few clusters, we sample images based on a uniform distribution over pseudo-label as clustering proceeds. We observe very close results when adjusting numbers of centroids.
Duline-Clustering: With K-Means for pseudo-label generation training. it does not lead to satisfying results (5100% on ResNet-11 18 with 5e Teacher) as instances might have not been accurately categorized by limited frozen centroids. Similar to (Caron et al., 2018; Li et al., 2020), we resort to the "in-batch" and dynamical clustering substitute the frozen K K-Means method. We conduct
17

---

Published as a conference paper at ICLR 2021
K-Means clustering within batch and continuously update the centroid based the teacher feature representation as distillation goes on. This alleviates the above problems and yields a substantial performance improvement on ResNet- to 56.4%.
We resort to CRD (Tian et al., 2019a) and adopt an info-NCE loss-alike training objective in unsupervised distillation tasks. Specifically, treat representation features from Teacher and Student for instance X; as positive pairs, and random instances from D as negative samples:
Binary Contrastive Loss:
0s
arg min Σ log [ K [log h(1 h(2z. d)))],
(6)
0s
where d E D, h(-) is any family of functions that satisfy h: {z, d) - [0, 11, e.g., cosine similarity.
A.5 DISCUSSIONS ON SEED
Our proposed learning objective for SEED composed of two goals, that to align the encoding z by the student model with z² produced by the teacher model; meanwhile, z also softly contrasts with random samples maintained in the D. This can be formulated more directly minimizing the 12 distance of z² 2 together with the cross-entropy computed using D:
=
Σ
N
Aa
2.
1
(x Ao p" +x(x;00 (er,D) D) log log (x; 0g 0s
D)}
N
explz, dg/T²) EddD exp(x d/7") EddD exp(z;
exp(z; 9
d /
Aa 2
x 2
da- Σ
log
d/ d/r5)
Directly optimizing Eq. 7 can lead to apparent difficulty in searching optimal hyper-paramett parametrrs (Ner λb, -" and 7)), Our proposed objective on D indeed is approximated upper-bound of the above objectiveness however much simplified:
CSED
Σ
p (x;; O,,0 D+) log p. (x,00,,")) D*)
N
N
K+1
exp(z? -d//77)
cxp(z d
(8)
llog EddD exp(z d/75)'
EdnDt exp(z) d//T)
where we let w/ denote the weighting term regulated under -T. Since the (K + 1)th element in D our supplemented vector zi'', the above objective can be expanded into:
LSEED Σ W (-25. S + log Σ exp(z .d/5))
d
(9)
K
Σ
w (-2. .d//75
log
exp(z5 d/r3))}
j=1
d>D
Note that the LSE term in the first line is strictly non-negative the range of inner product for S
and d lies between |1, +1]:
LSE(D*+ 22,5) 2 log (M (M-xx((11/75)) = log (M exp(-5)) > 0,
(10)
where M denotes the cardinality of the maintained queue D and to 65,536 in our experiment with S = 0.2 constantly. Meanwhile, the ISE term in the second line satisfies the following incquality:
LSE 25) LSE(D, 2)
(11)
18

---

Published as a conference paper at ICLR 2021
Thus, this demonstrates that the objective for SEED Eq. is equivalent to minimizing weakened upper-bound of e.q.
N Σ
SEED
p² (xx;0r,D)) (x; D+)
N
N
K
(-2 /2/ - + Σ wj :d// + log E
xxp(zz/d/79)
WK+1
N
1=1
dND
N
z 2 p (x,: OI D) log pd (x (xx; D) w K+1
Σ
N
(12)
This proves that our SEED directly relates to a more intuitive distillation formulation as Eg. 7 (12 + cross entropy loss), and it implicitly contains the objective of aligning and contrasting. However, our training objective is much simplified. During practice, we find by regulating -", both training losses produce equal results.
A.6 DISCUSSION ON THE RELATIONSHIP OF SEED WITH INFO-NCE
The objective of distillation can be considered as soft version of Info-NCE (Oord et al., 2018), with the only difference to be that SEED learns from the negative samples with probabilities instead of treating them all strictly negative samples. To be specific, following Info-NCE, the "hard" style contrastive distillation can be expressed as aligning with representations from the Teacher encoder and contrasting with all random instances:
Ōs arg min LNCE arg min log exp(z z //(TT)
(13)
Ea-D exp(z d/r)
0g
0g
which can be further deduced with two sub-terms consisting of positive sample alignment and contrasting with negative instances:
Σ{ /T+ log Σ explz d/t)}
LNCE
(14)
alignment
contrasting
Similarly, the objective of SEED can be dissembled into the weighted form of alignment and contrasting terms:
SSEED
1
Σ
Σ
expl2? d,/TJ)
llog EanD exp(2 dd/75
exp(z d
N
M
EaDD explz? d//T)
N K+1
exp(z? /
Σ
z / + log Σ
exp(z d/7
N M
Σ Ed exp(z? d/7T)
d
alignment
contrasting
(15)
where the normalization term be considered soft labels, W [w] ... w wk+]] which can weight the above loss as:
N
K+1
SEED N + M
Σ
Σ
w z 2 + log Σ exp(z; d
(16)
d~D
When tuning hyper-pann parpe--prrmmter T towards 0, W can be altered into the format of one-hot vector with w 1, which is then degraded to the case of contrastive distillation as in equation 14. In practice, the choice of optimal - can be dataset-specific. We show that the higher TT (with labels be more 'soft') can actually yield better results on other datasets, e.g., CIFAR-10 10 (Krizhevsky et al., 2009).
19

---

Published as a conference paper at ICLR 2021
A.7 COMPATIBILITY WITH SUPERVISED DISTILLATION
SEED conducts sele--sue distillation at the pre-training phase for the representation learning. However, we verify that SEED is compatible with traditional supervised distillation that happened during fine-tuning phrase at downstream, and can even produce better results. We begin with the SSL pre-training on a larger architecture (ResNet- 152) using MoCo-V2 and train for 200 epochs as the teacher network. As images in CIFAR- 100 are in the size of x2,32, we modify the first conv layer in ResNet with kernel size = and stride
We then compare the Top-1 accuracy of a smaller 18 CIFAR-100 when using different distillation strategies when all parameters are trainable. First, we use SEED to pre-train ResNet- 18 with Res-152 as the teacher model, and then evaluate in the test split of CIFAR-100 using linear fine-tuning task. As we keep all parameters trainable during the fine-tuning phase, distillation on the pre-training only yields trivial boost: 75.4% v.S. 75.2%. Then, adopt the traditional distillation method, e.g., (Hinton et al., 2015), to first fine-tune the ResNet-152 model, and then use its output class probability facilitate the linear classification task on ResNet-18 in the fine-tuning phrase. This improves the linear classification accuracy on ResNet-18 to 76.0%. At the end, we initialize the R8 with our SEED pre- traitrained ResNet- 18, and equip with the supervised classification distillation during fine-tuning. With that, we find that the performance of ResNet- 18 is further boosted to 78. %. We can conclude that our SEED compatible with traditional supervised distillation that mostly happened at downstream for specific tasks, e.g., classification, object detection.
Table 12: CIFAR- 100 Top-1 Accuracy(%) of ResNet- 18 with (or without) distillation different phase: self- supervised pre-training stage, and supervised classification fine-tuning. All backbone parameters of ResNet- are trainable experiments.
Pre-training Distill. Fine-tuning Distill.
Top-1 Acc
75.2
X
75.4
76.0
78.1
ImageNet Top-1 Accuracy (%)
63.5
63.0
62.5
62.0
61.5
61.0
6
Log(Size of Data Sample Queue)
Figure 7: Linear evaluation accuracy (%) of distillation between ResNet-18 (as the Student) and ResNet-50 (as the Teacher) using different size of queue when LR=0.03 and weight decay=le-6. Note the axis is the log(.) value of queue lengths.
A.8 ADDITIONAL ABLATION STUDIES
We study effects of different hyper-parameters to distillation using ResNet-18 (as Student) and SWAV pre-trained Re (as Teacher) with small patch views. In specific, we list the Top-1 Acc. validation split of ImageNet- 1k using different lengths of queue (K=128, 512, 1,024, 4,096, 8.192, 16,384, 65,536) in Figure. 7. With the increasing of random data samples, the distillation boosts the accuracy of learned representations, however within limited range: +1.5 when the queue size is
20

---

Published as a conference paper at ICLR 2021
Table 13: Linear evaluation accuracy (%) of dis- tillation between ResNet-18 (as Student) and ResNet-50 (as the Teacher) using different learn- ing rates when the queue size is 65,536 and weight decay=le-6.
TTbble 14: Linear evaluation accuracy (%) of distillation between ResNet- 18 (as the Student) and ResNet-50 (as Teacher) using different weight decays when the queue size is 65,536 and LR=0.03.
WD
Top-1
Ace.
Top-5 Acc.
LR
Top-1 I Acc.
Top-5 Acc.
le-2
11.8
27.7
58.9
83.1
1e-3
62.3
84.7
0.1
62.9
85.3
le-4
61.9
84.4
0.03
63.3
85.4
le-5
61.6
84.2
0.01
62.6
85.0
le-6
63.3
85.4
65,536 compared with 256. Furthermore, Table. 13 and 14 summarize the linear evaluation accuracy under different learning rates and weight decays.
21


Reviews:
[{"review_id": "tN2Oe-tty0_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper address the problem of knowledge distillation in self-supervised learning, where the representational knowledge from the larger model (i.e., teacher) is used to guide the learning of a smaller model (i.e, student). To achieve this, an instance queue is used to compute the similarity score between teacher model features and the feature of a given image, and the learning objective is to minimise the cross-entropy loss of the similarity between teacher and student models. The paper provides comprehensive empirical results to justify the efficacy of the proposed approach.\n\nJustification of rating:\nThe paper provides a comprehensive evaluation of the proposed approach on various network architecture and downstream tasks. Overall, I feel this work does not have sufficient theoretical or algorithmic contributions. The key contribution is the idea of apply knowledge distillation for self-supervised learning.\n\nStrengths:\n+ This is the first work that addresses self-supervised learning (SSL) with knowledge distillation. It empirically shows SSL with a small model is challenging (consistent with finding from Chen et al., 2020a;b), and proposed a technique (SEED) to transfer knowledge.\n+ This paper provides comprehensive experiment on image classification task (self-supervised, semi-supervised and supervised), object detection/segmentation, domain transfer, as well as provide ablation studies on various model architecture and parameters. The results show knowledge distillation is effective for self-supervised learning. \n\nWeakness: \n- The core novelty of this work is the idea of conduct knowledge distillation in self-supervised learning. The key weakness is that the knowledge distillation approach and the instance queue approach are previously proposed and known to the research community. This work empirically shows how it can be combined for the task on hand.\n\nMinor comments: \n- In section 3.2, before eqn (2), it is best to change “the similarity score between $x_i$ and $\\vec{d}_j$’s” to \"the similarity between the extracted (teacher/student) feature $z_i$ and $\\vec{d}_j$’s”. This is to avoid confusion as one might wonder how can the similarity between the input image and the $\\vec{d}_j$ be computed.\n- Please move the table’s caption to the top of the table.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7b7-3ILkl5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a  knowledge distillation (KD) approach for self-supervised learning (SSL) with small neural network models. The authors first observe that the state-of-the-art contrastive learning-based SSL does not obtain good performance on small models, due  to the larger  model capacity required for instance discrimination. To tackle this problem, they propose a SEED, a  KD method where the smaller student model learns to mimic its larger teacher model’s similarity distribution between an instance and its augmented views, using a cross-entropy based objective. The authors perform various experiments to show that -- 1)  SEED obtains substantial improvement in SSL-based imagenet classification performance for small models as compared to SSL training without SEED, 2) the performance gains are also substantial for transfer learning on other classification tasks, 3) the performance gains are smaller for downstream tasks of object detection and instance segmentation, with performance gains reducing for the larger COCO dataset, as compared to VOC, 4) SEED is robust to choice of SSL method, and performs better than other KD approaches.\n\nStrengths: The paper is clearly written and well-organized. The SEED approach  is well-motivated and sensible. Experimental validation and the ablation studies are quite thorough. Performance gains on classification tasks are substantial. The method is simple to implement. \n\nWeaknesses: 1. Performance gains on downstream tasks of detection and instance segmentation are much lower -- how would the authors propose to improve these? 2. If the primary goal is to improve SSL performance on small models, I would have liked to see more analysis on how different design choices of setting up contrastive learning affect model performance and if these could aid performance improvement, in addition to knowledge distillation. \n\nQuestions and suggestions: 1. Adding fully-supervised baselines for small models in table 1 will be useful in understanding the gap between full supervision and SSL for these models. 2. In figure 3, does 100% (green line) represent the student network trained with 100% of labeled imagenet supervised data? It is hard to interpret what these numbers represent. 3. Minor point: Some citations, which should not be in parentheses, are in parentheses (e.g., Romero et al. page 8). Please fix this in the revision. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1XoaxQiosYw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to distill knowledge from large teacher networks to small student networks in self-supervised learning. Experimental results show significant improvements on small networks.\n\nConcerns:\n\n- During the self-supervised distillation phase, it is not clear to me only distillation loss is applied or self-supervised learning loss is combined with distillation loss for learning student network. If only distillation loss is applied, does it make sense to train a student network using both self-supervised learning loss and distillation loss, such as MoCo-v2 used in most experiments? \n- The caption of Figure 3 is confusing, it would be good to explain it more clearly.\n- It seems that improvements on object detection and instance segmentation (Table 2) are relatively small compared to other experiments, are there any explanations? Could it be possible to use smaller student networks in this experiment as well? \n- In the experiment of different sizes of sample queue, does it mean that the larger the better? What is the intuition behind it?\n- Strong data augmentation is needed for most of self-supervised methods, but normally for distillation, it is not common to use very strong data augmentation, why does the paper decide to use the same data augmentation for both self-supervised learning and distillation learning?\n- It is quite similar to a very recent paper as shown in the following, it would be good to discuss the differences in the paper.\n\n@inproceedings{koohpayegani2020compress,\n  title={CompRess: Self-Supervised Learning by Compressing Representations},\n  author={Koohpayegani, Soroush Abbasi and Tejankar, Ajinkya and Pirsiavash, Hamed},\n  booktitle={NeurIPS},\n  year={2020}\n}", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is definite consensus on this paper, with all reviewers expressing very favorable opinions. The author responses are very well articulated and address the main concerns expressed by the reviewers. The paper is very well-written and the ablation study well-executed. Some recent related work was missed in the original submission, but this was adequately addressed in rebuttal. The proposed approach is novel technique for feature representation learning. The clarifications to the manuscript and the new analyses are especially appreciated. 
Decision: Accept (Poster)