Unsupervised Representation Transfer for Small Networks: I Believe I Can Distill On-the-Fly
Authors: Hee Min Choi
Hyoa Kang
Dokwan Oh
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation learning
knowledge distillation
unsupervised learning

Abstract: A current remarkable improvement of unsupervised visual representation learning is based on heavy networks with large-batch training. While recent methods have greatly reduced the gap between supervised and unsupervised performance of deep models such as ResNet-50, this development has been relatively limited for small models. In this work, we propose a novel unsupervised learning framework for small networks that combines deep self-supervised representation learning and knowledge distillation within one-phase training. In particular, a teacher model is trained to produce consistent cluster assignments between different views of the same image. Simultaneously, a student model is encouraged to mimic the prediction of on-the-fly self-supervised teacher. For effective knowledge transfer, we adopt the idea of domain classifier so that student training is guided by discriminative features invariant to the representational space shift between teacher and student. We also introduce a network driven multi-view generation paradigm to capture rich feature information contained in the network itself. Extensive experiments show that our student models surpass state-of-the-art offline distilled networks even from stronger self-supervised teachers as well as top-performing self-supervised models. Notably, our ResNet-18, trained with ResNet-50 teacher, achieves 68.3% ImageNet Top-1 accuracy on frozen feature linear evaluation, which is only 1.5% below the supervised baseline.

Unsupervised Representation Transfer for Small Networks: Believe I Can Distill On-the-Fly
Hee Min Choi, Hyoa Kang, and Dokwan Oh
Samsung Advanced Institute of Technology Suwon, South Korea
{chm. choi, hyoa. kang, dokwan. oh) ah)assumungg. com
Abstract
A current remarkable improvement of unsupervised visual representation learning is based on heavy networks with large-batch training. While recent methods have greatly reduced the gap between supervised and unsupervised performance of deep models such as ResNet-50, this development has been relatively limited for small models. In this work, propose novel unsupervised learning framework for small networks that combines deep self-supervised representation learning and knowledge distillation within one-phase training. In particular, a teacher model is trained to produce consistent cluster assignments between different views of the same image. Simultancously, student model is encouraged to mimic the prediction on-the-fi self-supervised teacher. For effective knowledge transfer, we adopt the idea of domain classifier that student training is guided by discriminative features invariant to the representational space shift between teacher and student. We also introduce network driven multi-view generation paradigm capture rich feature information contained in the network itself. Extensive experiments show that our student models surpass ofat--of-the-ar offline distilled networks even from stronger self-supervised teachers as well as top-performing self-supervised models. Notably, ResNet-18, trained with ResNet-50 teacher, achieves 68.3% ImageNet Top-1 1 accuracy on frozen feature linear evaluation, which is only 1.5% below the supervised baseline.
Introduction
Recently, there has been growing attention in unsupervised and self-supervised learning where the goal is to effectively learn useful features from a large amount of unlabeled data. Current self- supervised visual representation learning methods appear to approach and possibly even outperform the fully-supervised counterpart g 25 29]. All top- cop-performing self-supervised learning (SSL)
algorithms for visual representation involve training deep models on powerful computers. In particular, their smallest architecture is ResNet-50 [26), and the networks are trained with large batches (e.g..
4096 images) on multiple specialized hardware devices such 128 TPU cores 6 g 25 Yet. this heavy implementation is not a viable option in a resource-limited environment, and there is evidence that the SSL methods even do not work well on light models (Figure and 22 Also, we need strong small networks that can operate
system
chip (SoC) for real-world applications.
Existing methods SSL have demonstrated that deeper models learn general visual representation more effectively with unlabeled data 8 40 Moreover, it has been empirically shown that predictive performance of bigger networks better transferred into smaller ones Inspired by these, we are
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

70
69.8
Pretrained Epochs
ImageNet Top-1 accuracy (%)
Ours
Method
100
200
400
800
Supervised
69.8
Supervised
SimCLR
SimCLR [
47.1
49.9
51.8
53.4
55
MoCo-v2 -
MoCo-v2
48.6
49.9
51.9
56.1
BYOL -00
BYOL 25
44.2
47.5
46.8
47.1
BYOL-GA -4-
BYOL-GA 25
54.2
56.9
61.4
61.9
SwAV
SwAV 6
57.7
61.2
63.7
64.9
OSS (ours)
OSS (ours)
60.0
64.1
65.8
68.3
200
400 number of epochs
800
Figure 1: ImageNet Top-1 accuracy of linear probe ResNet-18 18 representations obtained from various supervsue methods with 256-batch training. BYOL-GA denotes the implementation of BYOL with gradient accumulation (to 4096 batches). Green dotted line indicates supervised ResNet- 18. Our method (red star) surpasses state-of-the-art self-supervised methods.
interested in pretraining a Light-weight model by distilling representational knowledge from a deep self-supervised network instead of directly performing self-supervised training small one.
Most of previous distillation methods in unsupervised representation learning literature are offline in the sense that they leverage an already trained self-supervisee teacher to transfer feature information
students [2 31 40 57]. Often, their sequential training pipelines require extra pre/post-processing
steps in order to boost performance. Unlike these approaches, propose novel unsupervised representation learning framework for small networks that combines deep self-supervised training and knowledge distillation [30] within one-phase training. To be specific, our teacher learns clusters and representation, and at the same time student trained to align its prediction to the clustering of On- the-ff fly Self Supervisee teacher. From this, we refer to method as OSS. The main advantage over offline distillation is that our teacher better distills proper signals into the student at each training stage by solving the same task online. In distillation process, it is difficult for low-capacity student to perfectly mimic the large teacher's behavior [36]. To deal with this, we incorporate the idea of domain classifier 23 That is, add a feature classifier trained not be able distinguish between teacher and student embeddings, and this leads to the emergence of representational wwwell-ppppeaiaaaeeeee invariant features in the course of joint training.
There is evidence that increasing the number of different views improves the quality of resulting features in SSL 66 8 55
Existing approaches to multi-view generation rely random image transformation techniques. Different from these methods, we introduce a new network driven paradigm in order to utilize rich feature information contained in the network itself. In particular, apply random dropout and empirically show that this apparently improves label efficiency (Section
Our method is conceptually simple but surprisingly works well with typical 256-batch training on single 8-GPU machine. Extensive experiments show that our student networks outperform tatt--ofthh-aand self-supervised models (Figure. [] as well as students from top- pop-perforiing distillation approaches even with stronger self-supervised teachers (Tables] [ 5) on linear evaluation. We also evaluate learned representations on series of vision tasks with multiple network architectures and various benchmark datasets and demonstrate significant performance gain (Tables and 4 and Figure 3) Overall, we make the following contributions.
We propose a novel unsupervised visual representation learning framework for small net- works that simultaneously conducts self-supervised teacher training and knowledge distilla- tion.
We introduce network driven view generation paradigm and adversarial distillation scheme to better capture representational information from networks. This the first work show their effectiveness in the context of unsupervised visual representation learning.
The proposed method significantly improves unsupervised pretraining performance of small models, and its effectiveness shown under multiple settings of network architectures, datasets and vision tasks.

---

Teacher
gradient
h
harop
fy
Linear
Linear
by
drop
Dropout
Tearher
Clusteriig
Loss
BN
Classifier
Feature
GR
LD
Claa
Rel.U
ReL.U
Student
x
View
Generator
hs
Drupout
gq
W
Normalize
Normalize
Student
Head
Embeddings
Prototypes
Loss
Figure 2: Overview of proposed architecture. OSS framework takes as input two randomly aug- mented views 21 and Z2, and they are processed by teacher's ' and student's 's backbones fe and In- The processed outputs are then projected lower dimensional embeddings and w via teacher's and student's projection heads lur and hş. We add dropout to each linear layer in both projection heads (hdrop's) for generating additional views of embeddings / and w. We note that h and dro share weights. Network training is guided by cluster prediction mechanism, and the target cluster assignments are obtained by online clustering using the mapping of teacher's feature embeddings to prototype vectors C. To distill useful discriminative feature information invariant to the representa- tional space shift between teacher and student, we use feature classifier connected to the last layer of each projection head with gradient reversal layer (GR).
2 Related Work
Unsupervised visual representation learning. In unsupervised or self-supervised visual repre- sentation learning, we learn rich feature information by solving pretext tasks whose labels can be obtained from image data itself. Learning methods can be divided into several categories by types of pretext tasks used therein: construction-basee methods 43 62). prediction- methods 10 20 24 25
38 39], cluster-based methods [ 5 6), and contrastive methods g
29] Our
work is mostly related to 6 a recent clustering- -baeed method.
Multi-view generation. Contrastive and clustering-basee self-supervised visual representation learning methods exploit random image transformation techniques to generate different views of each image. This is because comparing more views during training improves the performance of the resulting model 66] g 37 55]. Our method uses dropout [50] for obtaining more views to better utilize useful feature information as well as applying existing data augmentation based strategies.
Knowledge distillation from self-supervised models. Most of knowledge distillation [30] tech- niques are proposed in the context of supervised learning, where a shallow network trained via supervision signals from both labels and deep network [1 41 44 47 52
53 58 In unsupervised
learning settings, recent studies 40) and [57] exploit clustering of teacher's 's embeddings as distillation signals. More recently, CompRess 31 maintains queues for storing data sample's 's embeddings and calculates instance- extracted from teacher (and student). Its training objective is to minimize the distance between teacher and student score distributions. SEED (22 is a similar concurrent work of (31]. Unlike afore- mentioned methods that rely on already trained static teachers, our framework injects distillation step into clustering-band based self-supervised teacher learning. The only existing online technique is 4 which combines contrastive learning and co-teaching.
similarity scores between all the samples in the queue and the input features
Domain adversarial learning. DANN 23 learns features that are discriminative for the main classification task and indiscriminate with respect to the domain shift. This is achieved by adding a domain classifier with a gradient reversal layer. In particular, network parameters are updated to minimize the main classifier layer that reverses the sign of gradient. We adopt this idea in our framework to transfer discriminative features invariant to the representational space change between teacher and student. Similar idea with generative adversarial network [35] is applied in supervised knowledge distillation in [2]]
and maximize the domain classifier loss via a gradient reversal

---

3 Proposed Framework
Our goal is to learn visual features of small networks effectively without access to any labeled data. In order to do this, we embed distillation step into the process of on-the-fly self- teacher learning. Inspired by [57], which empirically shows clustering improves generalization power of representation, our online framework is developed upon [6], the state-of-the-art mustering-based self-supervised method. We also incorporate the idea of domain classifier to facilitate learning of useful features invariant to the representational space change between teacher and student. In nutshell, the training objective function of our network is given by
LT + Lg + Lp,
where Lr, Ls, and LD are designed for teacher network training (Sections 3.1 and 3.2 knowledge
distillation for student network (Sections 3.1 3.n (.2), and domain classifier (Section 3.3) respectively.
Overall architecture is presented in Figure and a pseudo code is available A.1
3.1 Teacher training and knowledge distillation
We now outline teacher network (e.g. ResNet-50) training and knowledge distillation framework. Our architecture takes as input two randomly augmented views F11 and T22 of an image Tn- The two views are processed by teacher's backbone network fo and projection head by that consists in a multi-layer perceptron (MLP). The processed features are then projected to the unit sphere. We denote teacher's 's backbone features fn1 = fe (2m1) and fra fa(22)), and the projected hr((n2)
outputs as 2n1 = [nrUni) and 2n2 = Un2)a=) Also, the two images are feed-forwarded to student's s backbone network 9,, (e.g., ResNet- The outputs are projected via an MLP head hg and then normalized well. Similarly, denote student's backbone features 9n1 9, (2 and
(fa))
hs((n))) and w,z = ( In2 = 9r (2n)), and their projected output vectors as Wh1 (0>s(0x1)]22
hs((m2)
We randomly initialize prototype vectors C = {Ci,.... } where K is the number of clusters. Code vectors, 4r 's for cluster assignments are computed from mapping teacher's 's embedding representation on 's to the trainable prototype vectors C and then solving an optimization problem for online clustering. We then solve swapped prediction problem 6 for teacher and simultaneously distill teacher's knowledge into student by enforcing consensus in prediction.
Teacher training. A training objective for the teacher network is solve swapped cluster prediction problem [6] of predicting the code Qnl from the feature 2nz and gn2 from zn1:
Le(21, 22)
Σ [Lo(Eni, (n2) + Lo(Zn2, qm1)].
Here, Tee is the cross entropy loss between the codes and softmax probabilities (with a temperature parameter T) of dot products of zni and all prototypes in C:
(b) exp ( c+) P.1 ΣE ky exp 2 [w))
Le(Zn1, ((2))
Σ (k) Inn log P
where
(3)
The loss function 2 is jointly minimized with respect to the prototypes and the parameters of teacher backbone fe and projection head by implicitly contained in 2n 's. The codes qn are computed using Sinkhorn Sinkhorn-Knopp algorithm [18 under an
constraint for each augmented
mini-batch with stored embeddings as done in [
Knowledge distillation. The student model is trained to follow teacher's code assignment. The training loss has exactly the same form of (2) with features from student and targets from teacher's codes:
I. (wn, w2)
Σ [Ta(Wss, (m2) + I.c(wn2, 9n1)]
Specifically, the cross-entropy loss L, is similarly calculated using student's embedding vectors is and the prototype vectors C. Again, minimization of this loss is performed with respect to the prototypes and the parameters of student backbone By and projection head hg used to produce Wn's. ss.

---

3.2 Network driven multi-view generation
In addition applying multi-crop strategy [6] we propose a network driven multi-view generation method to utilize rich feature information contained in the network. In particular, we add dropout layer after each linear layer in the projection heads to generate more feature embeddings. Additional loss terms from small-crop features {&nw 1V and dropout features { V+ are given by
Lawv (2,
Σ (E(1.2)
Σ
Σ
[Te(Enus qni) + Lé Ins)];
where we note that codes 9n 's are computed from non-dropout full-crop features. To summarize, the teacher loss LT the training objective
[ is the sum of ( and 5
Lr(21, 22, 2,, Le(21. 22) L 2).
The corresponding loss Ls for student in [ is similarly computed by the sum of (4) and Lmy (wi, u)) obtained from student's small-crop and dropout generated features (w, ú)) place of teacher's (3, using
3.3 Feature classifier
In order transfer good features that discriminative for the main clustering task and indiscriminate with respect to the representational space shift between teacher and student, we adopt the idea of domain classifier [23]. and call feature classifier. The feature classifier is adversarially trained not to be able to identify from which the embedding is originated via gradient reversal layer. To be specific, the feature classifier D) consists in an MLP that projects the features 2-D space. We compute the cross entropy loss between the feature label (0 for teacher and 1 for student) and softmax probability of the feature classifier outputs given by
exp (D (2ne))
= ()
log d - log d where df Exe(o.1) exp (Zik))
and (8) Smi is similarly obtained as d with was in place of 2nd For both teacher and student, the feature classifier is connected to the last layer of the MLP projection head (hT and hg) via gradient reversal layer that scales the gradient by a certain negative constant a during backpropagation training. Thus, minimizing Lp reduces discriminative power of the feature classifier.
Experiments
We perform extensive experiments across different combinations cracher-student architecture pairs and various tasks and datasets to test the performance of our student network. In this section, focus on evaluating our ResNet- 18 features trained with ResNet-50 teacher and a batch size 256 for 800 epochs on ImageNet-1K [19 training set (see A.T for details). For self-supervised ResNet-18
Table 1: Top-1 accuracy (%) of linear probe on ImageNet for ResNet- 18 distilled from SwAV's ResNet-50. * indicates the final accuracy of on- line teacher.
Table 2: Top-1 accuracy (%) of ResNet- fine- tuned small subsets of ImageNet. Our method outperforms both supervised and all considered unsupervised methods.
Distillation Method
Teacher Top-1
Student Top-1
Top-1 Accuracy
Method
1%
10%
Supervised
69.8
Scratch
14.9
50.4
CC 40
75.6
60.8
SimCLR 8
28.8
54.2
CRD [22
75.6
58.2
MoCo-v2 11
25.2
54.1
SEED 22
75.3
62.6
BYOL 251
25.6
52.3
CompRess-2g
75.6
62.4
SwAV
39.7
60.4
CompRess-Ig
31
75.6
65.6
CompRess-1g
31
37.8
59.0
OSS (ours)
73.0*
68.3
OSS (ours)
47.8
63.6
5

---

Table 3: Transfer performance (%) of ResNet-18 finetuned on small-scale datasets for varying tasks. The evaluation measure is 50 (average precision with intersection over union threshold of 50%) for detection and instance segmentation and mloU (mean intersection over union metric) for semantic segmentation. Overall, our model works best across all transfer tasks and datasets.
AP
A So
mloU VOC12
Method
VOCO7
COCO-1K
COCO-10K
COCO-IK
COCO-10K
Supervised
66.9
17.0
33.6
15.6
31.1
42.2
SimCLR 8
66.1 (-0.8)
16.0
33.0 (-0.6)
14,8 (-08)
30.6 (405)
40.6 (-1.6)
MoCo-v2
65.4 (11.5)
11.0 (60))
28.8 (448)
9.9 (s57)
26.9 (422)
41.3 (499
BYOL 25
64.9 (2.00
14.3 (2.7)
30.4 (3.2)
12.9 (-27)
28.0 (3.))
39,9 (-2.)
SwAV
68.1 (41.2)
18.6 (-1.6)
36,8 (v3.2)
17.2 (+1.6)
34.4 (43.3)
42.1 (4.1)
CompRess-1q
31
66.4 (0.5)
14.8 (222)
31.7 (1.9)
13.5 (2.0)
29.4 (1.7)
42.0(02)
OSS (ours)
68.3 (+14)
20.0 (-3.0)
37.9 (+4.3)
18.4 (+2.8
35.4 (-4.)
43.2 (+1.0)
baselines, use top-performing SSL models of SimCLR 8. MoCo-v2 111, BYOL 225 and SWAV
[6] that are trained with 256 batches for 800 epochs in pytorch 42] unless explicitly specified (see A.2) By default, the supervised performance comes from official torchvisi on models.
4.1
Linear evaluation on ImageNet
We validate method by linear evaluation of frozen features, following the protocol [6] described in A.3 We report 1-crop. top-1 classification accuracy of ResNet-18 on ImageNet validation set. Figure 1 shows that OSS significantly surpasses all the state-of-the-ar SSL methods. Note that OSS requires shorter training epochs than the SSL algorithms to promise better performance. Indeed, our 200-epoch model (64.1%) %) is almost on par with 800-epoch SwAV's (64.9%). Moreover, our 800-epoch model achieves 68.3%, which is only 1.5% below the accuracy of the supervised model. clustering-based SWAV (6) works better than contrastive methods of MoCo-v2 (T and SimCLR [8] We conjecture this is because clustering is relatively easier than instance discrimination for light networks in large-scale dataset. BYOL-GA indicates the implementation of BYOL (25) with gradient accumulation (GA) to 4096 batches (see Appendix G2 of (25)). For a fair comparison, we will use its non-GA version (B YOL) throughout the following experiments as all other methods are trained using 256 batches without GA.
In Table [ also compare our approach with various offline distillation methods using SwAV's ResNet-50 teachers obtained from (31 and (22). We note that (31] uses two extra normalization layers in the classifier for linear evaluation. Nevertheless, all the offline compression models from stronger Sw SwAY's ResNet-50 teacher do not still match our student's performance. A plausible reason for this that large models easily overfit the training data, so the static teachers provide less extra knowledge beyond hard annotations (2). A summary of the offline techniques along with an extensive list of their performance is found in B We also test general representational power of the pretrained frozen encoders on other classification datasets in B.2
4.2 Data efficiency evaluation ImageNet
We evaluate the performance on ImageNet classification task when finetuning our 800-epoch pre- trained ResNet-18 model with small subsets of trainset using labels. We follow semi-supervised protocol [6] detailed in A.4 and use 1% and 10% splits of [8). We report top-1 accuracy on ImageNet validation set Table Our method outperforms self-supervised approaches for all label fractions. We note that the gap in % data regime is larger than the gap in 10% counterpart. We also carry out the same data efficiency evaluation of R8sNet-18 features obtained from the sfate-of-the-an art offline knowledge distillation method of CompRess lq [3I with SWAV [6]'s ResNet-50 teacher. We use the officially released model and find that our student significantly outperforms by +100% in 1% and +4.6% in 10% settings. In fact, our network driven multi-view generation significantly improves label efficiency, and we will investigate this in ablations. Also, another strong low-shot learning performance on Places365 633 datasct is available in B.3

---

Table Top-1 accuracy (%) of linear evaluation on Im- ageNet for various shallow models. The second column (Params) the number of parameters in each backbone network, and the unit M denotes 10%. Params and super- vised performance are obtained from the corresponding papers.
Table 5: Top-1 aceuracy (%) of linear probe on ImageNet for MobileNet-v2 distilled from MoCo-v2's ResNet-50 teacher. in- dicates the final accuracy of online teacher. Our method outperforms all considered of- fline approaches.
Model
Params
Method
Top-1
Teacher Top-1
Student Top-1
Distillation Method
Supervised
69.8
ResNet- 18 26
11.5M
SwAV [
61.2
Supervised
72.0
OSS (ours)
64.1
CC 40
70.8
59.2
Supervised
75.5
CRD 522
70.8
54.1
RegNetY-600MP 45
6.1M
SwAV [
67.0
Reg (60)
70.8
48.0
OSS (ours)
67.4
Reg-BN
81
70.8
62.3
Supervised
77.1
CompRess-1e
31
70.8
63.0
EffeieentNetBii 51
5.3M
SwAV 6
59.3
CompRess-2g
70.8
65.8
OSS (ours)
64.1
OSS (ours)
70.4
66.1
Supervised
72.0
MobileNet-v2 48
3,4M
SwAV 6
63.2
OSS (ours)
66.1
4.3 Transfer to other vision tasks
now test transferability of ImageNet pretrained representation to different vision tasks. order to better pin-point the gain contributed by each pretraining method, we chose small-scale datasets such as PASCAL VOC (1 and small subsets of COCO [33] is because a large amount of data with long finetuning schedule reduces the importance of the initial model. For COCO data, randomly sample five non-overlapping 1K and 10K train splits as [28). and the results are averaged the splits (see for sensitivity analysis). We then transfer 800-epoch trained ResNet-18 backbones to object detection on VOC07 with Faster-RCNN [46), semantic segmentation on VOC12 with FCN 34 and object detection and instance segmentation COCO splits with Mask R-CNN (7] (see A.5 A.6 f.. details). Table 3) shows that our student produces better results than top-performing self-supervised and offline distilled models across all network architectures, datasets, and tasks. Notably, OSS is the only method that outperforms the supervised baselines on COCO-IK and VOC12. More downstream performance Cityscapes [17) is illustrated in B.4
Ablation Studies
5.1 Various teacher-student combinations
Smaller students. We validate framework with various teacher-student combinations. Specifi-
cally, we consider a ResNet-50 teacher and three small students: MobileNet-v2 [48) RegNetY-600MF
and EfficientNet-BOO [51). We pretrain the three pairs for 200 epochs on ImageNet [19] trainset
and conduct linear probe on the validation set (see A.I and A.3 for details). Table 4 shows that our
Aocurac Stxdatt
67
Pretrain Epochs
Teacher Params
Top-1 Student
Teacher
Teacher
64
ResNet-34
21.5M
62.1
61.0
ResNet-50
200
23.9M
70.8
62.9
62
200 epochs
ResNet-50w2
94.0M
75.1
400 epochs
ResNet-34
21.5M
64.6
63.1
60
ResNet-50
400
23.9M
65.5
100
ResNet-50w2
94.0M
75.5
66.4
Numberof of the Teacher
Figure 3: ImageNet Top-1 accuracy (%) of linear evaluation of ResNet- distilled from various teachers. The third column (Params) is the number of parameters in the teacher network, and the unit M is 1 Our student benefits more from larger teacher and longer training schedule.
7

---

Table 6: Effect of proposed dropout and distillation schemes the classification accuracy (%) of ResNet-18 features on ImageNet. Dropout strategy combined with feature classifier improves label efficiency, and online distillation works better than offline version.
Method Name
Scheme
Dropout Applied
Feature Classifier
Top-I
Accuracy
Top-5 Accuracy
Name
Frozen
1%
10%
Frozen
1%
10%
off.KD-Drop
66.0
33.9
61.5
86.9
59,4
84.2
KD
68.2
42.5
63.2
88.2
69.0
85.4
Ours
KD-Drop
68.4
47.2
63.3
88.5
73.3
85.6
KD-Drop-Ady
68.3
47.8
63.6
88.4
73.8
85.8
Original
64.9
39.7
60.4
86.3
66.3
83.6
SwAV
Drop
V
64.2
43.7
60.3
85.9
70.1
83.7
method outperforms SwAV [6] for all student architectures while the gain contributed by OSS not equal different networks. One possible reason is that
SwAV's hyperparameters, this may place SwAV at am advantage. We also compare our approach with other offline distillation methods using MobileNet-v2 provided in [31). In Table 5 our MobileNet-v2 features perform better than those compressed from MoCo-v2 11 's ResNet-50 teacher even our teacher's ' accuracy is lower during training. More results for EfficientNet-Bi encoder are found in B.I and semi-supervised learning performance is summarized in B.6
Larger teachers. We perform an analysis with ResNet- 18 students distilled from three different resnet teachers: ResNet-34, ResNet-50, and ResNet-50w2. ResNet-50w2 is 50-layer resnet with 2x wider channels. Here, we only consider frsource-friendly pairs that are trainable on a single 8-GPU machine, so we do not explore beyond ResNer-50w2. We pretrain them using our framework on ImageNet trainset for 800 epochs and carry out inear probe on the validation set at 200 and 400 epochs. Figure shows that method benefits more from bigger teachers and longer train iterations. As in supervised knowledge distillation, student's s performance is not necessarily proportional to the gap (in size) between teacher and student [36].
5.2 Framework design
Questions and experiment settings. natural question that arises is how effective is "offline version of the proposed method"' where student learning SwAV's framework with dropout while the target code comes from SwAV's pretrained (frozen) teacher? Also, one might be curious about the performance gain from each proposed technique: network driven multi-view generation and feature classifier. To answer these, we train ResNet-50 (teacher) and ResNet-18 (student) pair linear probe and semi-supervised evaluation (see A.3 and A.4).. The distillation epoch is 130 for the offline scheme (off. Off.KKDDroop) from SwAV's 800-epoch teacher following [3T] and 800 for the other online versions. Implementation details of off-.DD are available in A.8 Throughout this experiment, the dropout rates are 0.1 and 0.05 at the two liner layers in the projection head (see Figure
under the 4 settings in Table and carry
Table 7: Classification accuracy (%) of ResNet-50 finetuned on small subsets of ImageNet. Results of OSS come from training with ResNet-18 stu- dent. The proposed framework produces data ficient teacher well.
Table 8: Effects of dropout rates on low-shot learning classification accuracy on ImageNet for ResNet-18 student. Strong dropout im- proves data efficiency on 1% setup while
performance is less sensitive at 10% setup.
Top-1
Accuracy
Top-5 Accuracy 10%
Dropout rate
Top-1
Accuracy
Method
1%
10%
1%
Linearl
Linear2
1%
10%
Supervised
25.4
56.4
48.4
80.4
0.00
0.05
37.5
60.0
0.10
0.05
43.5
59.6
SimCLR
48.3
65.6
75.5
87.8
0.10
0.10
43.7
59.5
BYOL
53.2
68.8
78.4
89.0
0.20
0.05
44.0
59.1
SwAV
53,9
70.2
78.5
89.9
0.20
0.10
44.0
59.4
OSS (ours)
57.5
71.1
81.4
90.5

---

Original
Supervised Student
offline DD-Drop
KD
KD-Drop
KD-Drop-Adv
Image
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
visualization of feature maps of different dropout and distillation schemes using some images in ImageNet validation set. Proposed method (KD-Drop-Adv) shows strongest activation on the target class region (Meatloaf, Marmoset and Bobsled) among all schemes.
Figure 4: Grad-CAr 49
Student's performance analysis. From Table 6 we observe that online schemes achieve higher accuracy than the offline version (off. KD-Drop). Our intuition is that teachers better figure out and distill appropriate information in the course of joint training by solving the same task online. Yet, our offline method even outperforms all offline techniques in Table in linear evaluation, and this shows superiority of our overall framework. On the other hand, strong performance of our method does not come free from (online) knowledge distillation. We argue that the proposed network driven strategy brings beneficial gain in mining of rich knowledge contained in the network. Indeed, applying random dropout significantly increases data efficiency as demonstrated in 1%-label evaluation. Also, there is evidence that our feature classifier generally yields additional small improvement in low-shot learning performance. From this empirical analysis, we find that the dropout strategy, combined with the feature classifier, is more effective in lower data regime.
Teacher's performance analysis. It is natural to ask whether the proposed network driven view generation with feature classifier produces more data-efficiens teacher as the same setup is applied to the teacher network as well. We answer this question by performing the label efficiency evaluation of teacher on ImageNet. We compare the results with those of ResNet-50 models obtained from SSL methods Table Our teacher outperforms the state-of-the -ottte--rr-t- self-supervised networks on both 1 5% and 10% regimes. We note that the training objective of our ResNet-50 teacher is exactly the same as SwAV's s ResNet-50 model except for dropout and feature classifier. This verifies that the performance gain over SwAV (+3.6% on 1% set and +0.9% on 10% set in top-1 accuracy) is sorely contributed by our proposed strategy.
Dropout rates. In order to provide more intuition on our network driven generation paradigm, we experiments over multiple choices of dropout rates in the proposed OSS framework (KD-Drop- Adv). We do not explore aggressive dropout rates because self-supervised training signal is already noisy. Table X summarizes low-shot classification results on ImageNet. We find that strong dropout improves label efficiency at 1 1% regime. Yet, accuracy is less sensitive to the choice of dropout probabilities when relatively larger amount of data is available.
Qualitative analysis of feature distributions. We visualize feature maps of some images in Im- ageNet validation set using Grad-CAM [49] in Figure 4 Grad-CAM identifies regions where the network focuses on. We show the results of teacher and student networks trained by our framework with different schemes in Table 6 As in Figure 4 the proposed method (KD-Drop-Adis shows stronger activation on the target class region than other schemes. While the feature classifier seems to have a little effect on the quantitative performance, the feature map illustrates clear improvement (over KD-Drop) in the coverage of useful object region. Different from other setups, the student from KD-Drop-Adv reveals high activation on both chunks of meat and wider part of the bobsled and monkey. From this, we see training with the feature classifier helps learning the true data distribution better. More Grad-CAM examples are available in B.8

---

5.3 Dropout applied to SwAV
also report the effect of applying our dropout strategy on the performance of SwAV 6 features. We train a dropout added version of SwAV for 800 epochs using ResNet-18 model. We apply the same dropout scheme to the projection head and compare with the original model. As shown in T dropout improves its label efficiency well.
Conclusion
We introduce a unified unsupervised visual representation learning framework for small models that combines deep self-supervised learning with knowledge distillation. Our student outperforms the state-of-the-an self-supervised shallow models as well as the networks obtained from top- performing offline compression techniques. We hope our work will attract community' attention to development of unsupervised pretraining methods for light-weight networks.
Disclosure of Funding
We received no third party funding for this work.
References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9163-9171, 2019.
[2] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geof- frey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference Learning Representations (ICLR), 2018.
[3] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. International Conference on Learning Representations (ICLR), 2020.
[4] Prashant Bhat, Elahe Arani, and Bahram Zonooz. Distill on the Go: Online knowledge distillation in self-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2678-2687, 2021.
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), 2018
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2020.
[7] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai. Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark, 2019, arXiv:1906.07155.
[8] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, pages 1597-1607, 2020.
[9] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
10

---

[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750- 15758, 2021.
[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020. arXiv: 2003.04297.
[12] Inseop Chung, Seonguk Park, Jangho Kim, and Nojun Kwak. Feature-map- map-level online adver- sarial knowledge distillation. In Proceedings of the 37th International Conference on Machine Leurning, pages 2006-2015, 2020.
[13] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3606-3613, 2014.
[14] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Leurning Research, pages 215-223, 2011. URL htt.htttt/pprcccedingg./ mlr wwwppppesssvv.ee v15/ coates 1atmtellla..mm
[15] MMSegmentation Contributors. MMSegmentation,, an open source semantic segmentation toolbox. http://tthbb..ccoooooomm//aaaa/a
com
2020.
[16] OpenSelfSup Contributors. OpenSelfSup, open source representation learning toolbox. attpss//ggt t/github...........m.... wtt.ccoow...o.....oooooooooeeoee open- http://nnmmmmaaappeeeeeelff----e 2020.
[17] Marius Cordts, Mohamed Omran, Sehastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke. Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213-3223, 2016.
[18] Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transportation distances. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2013.
[19] Deng. W. Dong, Socher. L. Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[20] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning the by context prediction. In Proceedings the IEEE International Conference on Computer Vision (ICCV), pages 1422-1430, 2015.
[21] Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes VOC challenge. International Journal of Computer Vision, 88 (2):303-338, 2010.
[22] Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised distillation for visual representation. In International Conference on Learning Representations (ICIR), 2021. URL http:////pppppeeeeeeeeee///orora
[23] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarias training of neural networks. Journal of Machine Learning Research, 17(59): 2016.
[24] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
[25]
Stan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.

---

[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[27] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[28] Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking ImageNet pre-training. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4917-4926, 2019.
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015.
[31] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. CompRess: Self- supervised learning by compressing representations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[32] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[33] Tsung-Ya Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755, Cham, 2014.
[34] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tie segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2015.
[35] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2794-2802, 2017.
[36] Seyed-Imas Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. In AAAI Conference on Artificial Intelligence, pages 5191-5198, 2020.
[37] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant represen- tations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6707-6717, 2020.
[38] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 527-544, 2016.
[39] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV), pages 69-84, 2016.
[40] Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self- supervised learning via knowledge transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9359-9367, 2018.
[41] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
12

---

[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8024-8035, 2019. URL http::/ neur wtt.::/.ennnrii.cccccccoc cc/ppperr 9015-pytorch-an--igg wwwpaannnneaeeeeeeeeeeeeeeeee https///eeeeea.pppoeoooooreoeeee
library
pdf
[43] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536-2544, 2016.
[44] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation, Proceedings of the International Conference Computer Vision (ICCV), 2019.
[45] Iliaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[46] Shaoqing Ren, Kaiming He. Ross Girshick, and Jian Sun. Faster R-CNN:
Towards real- time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 91-99, 2015.
[47] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In International Conference on Learning Representations (ICLR), 2015.
[48] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510-4520, 2018.
[49] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618-626, 2017.
[50] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 2014.
[51] Mingxing Tan and Quoc Le. EfficientNet:
Rethinking model scaling for convolutional neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114, PMLR, 09-15 Jun 2019.
[52] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations (ICLR), 2020.
[53] Frederick Tung and Greg Mori. momilarity-preservvvee knowledge distillation. In Proceedings of the International Conference on Computer Vision (ICCV), pages 1365-13744 2019.
[54] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Lo, and Ross Girshick. Detectron2. https:/ :/gi thub.
2019.
[55] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3742-3733, 2018
[56] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from abbey to 7oo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492 2010.
13

---

[57] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. ClusterFit: Improving generalization of visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6509-6511 2020.
[58] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[59] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017. MXXiv:108.00888 03888.
[60] L. Yu, O. Yazici, X. Liu, J. van de Weijer, Y. Cheng, and A. Ramisa. Learning metrics from teachers: Compact networks for image embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1902229111 2019.
[61] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4L: Self-supervised semi-supervised learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1476-1485, 2019.
[62] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 649-666, 2016.
[63] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions Putlern Analysis and Machine Intelligence, 4056):1452-1464, 2018.
14


Reviews:
[{"review_id": "WAkpgjtPWh6", "reviewer": "Reviewer_Ymks", "summary": "This paper presents a self-supervised distillation method that aims to learn better-performing representations for small models.  In particular, the teacher model is trained to produce consistent cluster assignments between different small views of the image while the student model is encouraged to mimic the prediction of the teacher. Comprehensive experiments on ImageNet and other downstream tasks validate the effectiveness of the self-supervised distillation method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a distillation-based self-supervised representation method, which is largely based on the previous efforts of SEED [1] and CompRess [2]. In essence, the author proposes to train both the Teacher and Student architecture, with the Teacher conducting the clustering and providing the soft labels for the Student. The author also proposes to leverage the multi-view strategies similar to [1] and [3]. The whole paper is well-organized, with comprehensive experiments and ablations, and clear paper writing. The exhibited performances are competitive and show great improvement over previous self-supervised distillation baselines and self-supervised methods. \n\nThe concerns from the reviewer mainly come from the following aspects of incremental technical novelty comparing to [1]; and the doubts that whether the over-performing numbers come at the great training cost of additional optimizing the bulky Teacher architecture; In addition, it is much preferred to show comparisons with SEED and CompRess instead of SWAV/SimCLR. \n\nSpecific comments are listed as follows:\n\n1. About the novelty of contribution: It seems the proposed methods are mostly inspired and originated from [1] and [2]. In particular, SEED also adopted the `\"instance dictionary\", which I think has the same effect as the prototype clustering. Also, it seems [1]'s clustering is also based on the representations from the Teacher and also the multi-view strategy. Comparing to [1], what are the main differences that distinguish OSS? In other words, what are the reasons that further improve the performances? Is it the Teacher's continuing optimization jointly with the Student?\n\n2. Concerns about Figure 1. The figure demonstrates the advantages of using the self-supervised distillation methods comparing with previous state-of-the-art SSL methods. However, as a consensus, distillation usually leads to efficient convergencies and better performances under the same training epochs. Is it a fair comparison with SSL methods instead of SEED and CompRess? \n\nI strongly suggest the author instead show the numbers w.r.t. to the number of epochs of SEED and CompRESS instead of SWAV/SimCLR.\n\n3. Additional training cost. OSS shows great performance improvement over SSL and [1], [2], however the requirement to keep the Teacher model trainable during the distillation&training. I understand that the longer training epochs exist in most previous SSL baselines, while this also unavoidably narrows the OSS' scope of applications. I.e., given a generic SSL learned Teacher arch, isn't OSS sacrificing the cost of training a cumbersome Teacher for better performances? \n\nSuggestion: report also the offline distillation numbers when the Teacher arch is frozen and not updated. This quantifies the improvements gained from the online/offline setting.\n\n\n4. It is strongly suggested to append specific discussion about the differences of OSS with CompRess instead of SimCLR when no distillation technique is leveraged. The current draft majorly compares with MoCo/v2 and other SSL methods and over-emphasizes its position in SSL, which I think is a bit one-sided. \n\nSuggestion: focus more on the other self-supervised distillation and reduce the statement of comparison over SSL methods.\n\nOverall, I appreciate the authors with their efforts for comprehensive experiments and the showed performance-boosting is inspiring. I'm very prone to further update my ratings with further responses from the authors and I still hold quite positive opinions for this submission after fixing my concerns.\n\n\nReferences\n\n[1]. Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised distillation for visual representation. In International Conference on Learning Representations (ICLR), 2021.\n\n[2]. Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. CompRess: Self380 supervised learning by compressing representations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.\n\n[3]. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi  Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " the authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "K0l84taDvwX", "reviewer": "Reviewer_RWpQ", "summary": "The authors target an interesting and valuable problem: how to train an unsupervised/self-supervised small model by distillation large models. They proposed the OSS framework, which is equipped with a distillation strategy, a multi-view generation strategy, and an adversarial feature generation strategy. The proposed framework is shown to work better than current SOTA self-supervised frameworks and distillation methods. In some specific regimes, the proposed framework works even better than the supervised models. This work makes a valuable exploration for unsupervised/self-supervised learning with light-weight models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work tackles an interesting problem of unsupervised learning with light-weight models. The proposed OSS framework is effective and the experimental results could address all the claims of this paper. However, I still have some concerns regarding the model design and experimental settings.\n\nModel Design:\n> - For Eq. (5), since $\\hat{z}$ are the dropout features and they are derived from $\\tilde{z}$ through dropout, and during a long time of training, different versions of $\\hat{z}$ could be sampled. And the goal of the multi-view generation is to make the model can equally make use of the multi-view versions of the features. In other words, we want $\\hat{z}$ and $\\tilde{z}$ can work equally for the model. So, why is it necessary to include a cross-entropy loss term for $\\tilde{z}$, or I may want to ask why the sum of $L_{c}(\\hat{z})$ and $L_{c}(\\tilde{z})$ can work as such a role?  \n> - For Eq. (7), the adversarial training. How did this loss backward to the main feature models? Shall we use $L_{F} = -\\sum \\sum (log d_{T}^{(1)} + log d_S^{(0)})$? \n\nExperiments:\n> - In Fig. 1 and Table 1, how about the performance of the \"vanilla SwAV Teacher\" model and the \"Supervised Teacher Model\" (Baseline)? More results can help to better understanding the results and the improvements. \n> - In line 186, the authors mentioned \"A plausible reason for this is that large models easily overfit to the training data, so the static teachers provide less extra knowledge beyond hard annotations\", so why OSS model is not influenced by this issue?\n> - In Table 6, how about the performance of the variant KD-Adv?\n\nTypos:\n> - line 2 \"greatly have reduced\" --> have greatly reduced", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the related work, the authors summarized the unsupervised methods into several categories: construction-based methods, prediction-based methods, cluster-based methods and contrastive methods. In this paper, the authors mainly explore the proposed OSS framework on one of the cluster-based methods. More valuable explorations may be applied to other cluster-based methods and also methods on the other categories, at least the contrastive methods may also work.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DWh_HsQtRkD", "reviewer": "Reviewer_jWLC", "summary": "The paper proposes an unsupervised representation learning method. It follows a teacher-student paradigm. The teacher (ResNet-50) is learned by fitting samples to clusters defined by randomly initialized prototypes and fitting different views of the same sample to the same cluster. The views are created with feature dropouts instead of random geometric transformations. The student model (ResNet-18) is distilled from the teacher online, i.e., trained together with the teacher.  Adversarial learning is also applied to align the features between the student and teachers.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\n1.\tOnline distillation is relatively underexplored in the particular setting of unsupervised representation learning.\n2.\tThe proposed method is a reasonable combination of various techniques. Though the separate components are not new, a working combination of them is always valuable. \n3.\tExperimental results in distilling ResNet50 to ResNet18 demonstrate that the proposed method outperforms previous distillation-based methods significantly and outperforms other unsupervised learning methods trained directly on ResNet18.  \n\nCons:\n\n1.\tOnline teacher-student update is not new in the teacher-student paradigm.  \n2.\tWhile the benefits of using a small network are generally justifiable, constraining a method strictly to small networks makes its potential application narrow. The absence of results with larger networks also raises the question of whether the method can work on larger networks at all. The paper could be stronger if it explores settings beyond ResNet50->ResNet18. \n3.\tThe online distillation and adversarial learning ideas are orthogonal to how the other aspect of unsupervised learning. That means such a distillation strategy can be combined with other unsupervised learning methods, such as SimCLR, MOCO, BYOL, SwAV. While outperforming existing offline distillation methods, the proposed method might be a better mate to a different unsupervised learning method. \n\n---\n\nThanks for the authors' response.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is about general unsupervised learning. I did not see particular limitations to address.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers were supportive of the ideas in the paper and recommended acceptance - I agree with them. I also believe that studying small networks is an important area in itself due to the need for lightweight models on "edge devices". The reviewers have provided a number of valuable comments which the authors should integrate into their final version.
Decision: Accept (Poster)