Bag of Instances Aggregation Boosts Self-supervised Distillation
Authors: Haohang Xu
Jiemin Fang
XIAOPENG ZHANG
Lingxi Xie
Xinggang Wang
Wenrui Dai
Hongkai Xiong
Qi Tian
Venue: ICLR 2022 Poster (2022)
Keywords: Self-supervised learning
knowledge distillation
instance bagging

Abstract: Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well as its augmentations as an individual class and try to distinguish them from all other images. However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization. This is especially true for small-scale models, in which we find the performance drops dramatically comparing with its supervised counterpart. In this paper, we propose a simple but effective distillation strategy for unsupervised learning. The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance. Our method, termed as BINGO, which is short for Bag of InstaNces aGgregatiOn, targets at transferring the relationship learned by the teacher to the student. Here bag of instances indicates a set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation is to aggregate compact representations over the student with respect to instances in a bag. Notably, BINGO achieves new state-of-the-art performance on small-scale models, i.e., 65.5% and 68.9% top-1 accuracies with linear evaluation on ImageNet, using ResNet-18 and ResNet-34 as the backbones respectively, surpassing baselines (52.5% and 57.4% top-1 accuracies) by a significant margin. The code is available at https://github.com/haohang96/bingo.

Published as a conference paper at ICLR 2022
BAG OF INSTANCES AGGREGATION BoOSTS SELF-SUPERVISER DISTILLATION
Haohang Xu Jiemin Fang14~ Xiaopeng Zhang² Lingxi Xie
Xinggang Wang' Wenrui Dai' Hongkai Xiong! Qi Tian? 'Shanghai Jiao Tong University Huawei Inc.
'Institute of Artificial Intelligence, Huazhong University Science Technology
"School of EIC, Huazhong University of Science & Technology (xuhachang, dai weiwrrr xi onghongka i /Bsjtu.edu.com
[jaminteng, sgwant thust edu. fexphistery, 198800xc/egmalle comail.comm com tian wat..ggl@hawwei.com
ABSTRACT
Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well its augmentations an individual class and try to distinguish them from all other images. However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization. This is especially true for small scale models, which we find the performance drops dramat- ically comparing with its supervised counterpart. this paper, we propose a simple but effective distillation strategy for unsupervised learning. The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance. Our method, termed BINGO, which is short for Bag of InstaNces aGgregatiOn, targets at transferring the relationship learned by the teacher the student. Here bag of instances indicates set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation aggregate compact representations over the student with respect instances in bag. Notably, BINGO achieves new state-of-the-ar performance on small scale models, i.e., 65.5% and 68.9% top-I accuracies with linear evaluation on ImageNet, using ResNet- 18 and ResNet-34 as backbone, respectively, surpass- ing baselines (52.5% and 57. 57.4% top-1 accuracies) by a sigmificant margin. The code is available at https: / htt.:ttcc..cccccoatoaoeo333 com./haoaaggg/////ngg
INTRODUCTION
Convolutional Neural Networks (CNNs) have achieved great success in the field of computer vision, including image classification HHe et al. 2016), object detection (Ren et al., 2015) and semantic segmentation (Chen et al. 2017). However, most of the time, CNNs cannot succeed without enormous Luman-annotateo data. Recently, self-supervised learning, typified by contrastive learning (He et al.) 2020 Chen et al.) 2020a), has been fighting with the annotation-eage -eager challenge and achieves great success. Most current self-supervised methods yet focus on networks with large size, e.g., ResNet- 50 (He et al.] (0116) with more than 20M parameters, but real-life implementation usually involves computation-limat limited scenarios, e.g., mobile/edge devices.
Due to annotation lacking in unsupervised tasks, learning from unlabeled data becomes challenging. Recent contrastive learning methods (He et al., 2020 Chen et al. (220aa) tackle this problem by narrowing gaps between embeddings of different augmentations from the same image. Techniques like momentum encoder for stable updating, memory bank for storing negative pairs, complicated data augmentation strategies etc., proposed to avoid collapse and promote the performance. With the above techniques, contrastive learning methods show promising performance. However,
"qqual contributions.
The work was done during the internship of Haohang Xu and Jiemin Fang Huawei Inc.

---

Published as a conference paper at ICLR 2022
MoCa v2 SE3D
Dod's BINGO
51.2
70-
60
SEED
45.5
65
54.7
BINGO
Top-1 Acc. (%)
60-
Top-1 Acc. (%)
50-
50.5
47.1
45.8
44.3
$2.5
50
40
76.1
30
Eff-BO MobileNet-v3 ResNet-18
ResNet-34
1% Labels
10% Labels
(a) Lincar classification accuracy on ImageNet over dif- ferent student architectures distilled by ResNet-50x2 teacher model
(b) Semi-supervised learning by fine-tuning 1% and 10% labeled images on ImageNet using ResNet-18 student and ResNet-152 152 teacher model
Figure I: Overall performance comparisons between BINGO and other unsupervised distillation methods.
contrastive learning requires discriminating all instances, due the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization. This issue becomes severe for small scale models, which earry too few parameters to the enormous data. Inspired by supervised learning that knowledge from large models can effectively promote the learning ability of small models with distillation, exploring knowledge distillation on unsupervised small models becomes an important topic.
Compress (Fang et al. 2020) and SEED (Fang et al. 2020) are two typical methods for unsupervised distillation, which propose transfer knowledge from the teacher in terms of similarity distributions among different instances. However, as the similarity distribution is computed by randomly sampling instances from dynamically maintained queue, this kind of knowledge is mostly constructed based on instances with low relation, which fails to effectively model similarity of those highly related samples. To solve this issue, we propose new self-supervised distillation method, which transfers knowledge by aggregating bags of related instances, named BINGO. In our empirical studies, transferring knowledge based highly related samples helps boost performance more effectively compared with previous relation-agnostic methods. Specifically, we select an unsupervised pretrained large model as the teacher. First, we map the conventional instance- wise dataset into a bag-wiss one. Each original instance is set all the other instances' embeddings produced by the teacher model, we feed instances which show high similarity with the anchor instance into the bag. Then apply the bagged dataset to the small model distillation process. To this end, we propose bag aggreggaggnn distillation loss, which consists of two components: inter-sample distillation and sarra- distillation. For intra-sample distillation, embeddings of the student and teacher from two augmentations of the same instance are pushed together; for inter-sample distillation, embeddings of all instances in bag are pushed be more similar with the anchor one, Equipped with the two proposed distillation loss, the bag- bag-based knowledge from the teacher can be well transferred the student, which shows significant advantages
anchor instance of the bag. By matching similarities of
over previous relation- agnostic ones (Fang et al. 2020 Abbasi Koohpayegani et al. 2020
Our contributions can be summarized as follows.
We propose new self-supervised distillation method, which bags related instances by matching similarities of instance embeddings produced by the teacher. The bagged dataset can effectively boost small model distillation by aggregating instance embeddings in bags. The proposed relation- relation-guided guided method shows stronger performance than previous relation- agnostic ones.
BINGO promotes the performance of both ResNet- 18 and -34 to new ofate-of-the-arr (SOTA) ones in unsupervised scenarios. It is worth noting that the distilled models also present far better performance compared with previous SOTA methods on other tasks, i.e., KNN classification and semi-supervised learning.
2

---

Published as a conference paper at ICLR 2022
BINGO provides new paradigm for unsupervised distillation where knowledge between instances with high relation could be more effective than relation relation-agnostic ones. This may be inspiring for further explorations knowledge transfer in unsupervised scenarios.
RELATED WORK
Self-supervieed Learning As generic framework to learn representations with unlabeled data, self-supervised learning has experienced remarkable progress over the past few years. By constructing a series of pretext tasks, supf-super learning aims at extracting discriminative representations from input data. Previous methods obtain self-supervised representations mainly via a corrupting and recovering manner, from perspectives of spatial ordering (Noroozi & Favaro 2016), rotation changes (Komodakis & Gidaris 2018), in-painting (Pathak et 2016)) or colorization (Zhang et al. 2016), et al. Recently, contrastive learning based methods (He et al. 2020 Chen et al.) 2020a) emerge and significantly promote the performance of self-supervised learning, which aim maximizing the mutual information between two augmented views of a image. A series of subsequent works (Grill
et al. 2020 Xu al. 2020b Dwibedi et al. 2021 further improve the performance to very high level. Khosla et al. (2020) applies contrastive learning on supervised learning, which selects the
positive samples from the same category. Caron et al. (2020) proposes align the distribution of one instance's 's different views on other categories. However, few of them pay attention to self-supervised learning on small-scale models, which are of critical importance to implement self-supervised models on lightweight devices. We propose an effective method to boost the self-supervieed learning of small models, which takes advantage of relation-base based knowledge between data and shows superior performance than previous ones.
Knowledge Distillation Knowledge distillation aims to transfer knowledge from a model (teacher) to another one (student), usually from a large to small one, which is commonly used for improving the performance of the lightweight model. Hinton et al. (2015) first proposes knowledge distillation via minimizing the KL--ivergencee between the student and teacher's logits, which uses the predicted class probabilities from the teacher as soft labels to guide the student model. Instead of mimicking Logits,,Romero teacher's logits, [Romero et al. 2014) transfers the knowledge by minimizing the Eg distance between
intermediate outputs of the teacher and student model. To solve the dimension mismatch, Romero et al. (004) uses randomly initialized projection layer to enlarge the dimension of a narrower student model. Based on Romero et al.) (2014). Zagoruyko & Komodakis (2016) utilizes knowledge stored in the attention map generated by the teacher model, and pushes the student model pay attention to the area where the teacher focuses on. Zhou et all (2021) 1) improves weighted soft labels to adaptively improve the vhasivariance tradeoff of each sample. Besides perspectives of soft labels and intermediate features, relation between samples is also an important knowledge. Park et al. (2019) and Liu et al. (019)) train student model by aligning the pair-wise similarity graph with the teacher. Recently, some works extend the above distillation method into
learning scenarios.
Tian et al. (0199 uses the contrastive loss to learn cross-modality consistency. Xuu et al. (2020a) (Fang et al. (2020) and Abbasi Koohpayegani et al. (2020) share similar methodology with Caron et al. (2020) of aligning feature distribution between views of the same instances. The distribution is computed as the pair-wise similarities between student's outputs and features stored in memory bank. However, the above relation based self-supervised distillation methods only compute the similarity between anchor sample and randomly sampled instances from maintained queue, which ignores the relation between sampled and anchor instances. Choi et al. (202T) uses the teacher model to produce cluster assignments, and encourages the student model to mimic the output of the trainable teacher model on- on-the-fly, which achieves promising results. Gao et al. (2021) strengthens the student model by adding regularization loss on the original contrastive loss, which aims at minimizing the l distance between the student's 's and teacher's 's embedding. Navaneet et al. (2021) also achieves competitive results with feature regression in self-supervised distillation. We propose to transfer the relation knowledge between models via a new type of dataset, which bags related instances. By aggregating the bagged instances, the relation knowledge can be effectively transferred.
3 APPROACH
In this section, we introduce the proposed BINGO in details. First, we discuss how to bag samples in the instance-wise dataset. After the samples are bagged, the bag-aggregation based knowledge
3

---

Published as a conference paper at ICLR 2022
Instance-wii wise Dataset
Features
Bag-wise Dataset
Bag Aggregation
Positive
Positive
Teacher
B Bagging via
Anchor
Feature Similarity
Fosttiie
Positivg
Details about how to aggregate a bag
Bag-wise Dataset
Anchor Instance X
Instance sampling
t-T
Teacher
a
Bag sampling
Lintra
Positive Instance Xp
Linter
Sa a'
Student
Sp
t-T
An overview of the proposed method. The samples are first bagged via feature similarity. Then the related instances in a bag is aggregated via intra-sample and inter-sample distillation loss. The figure on top-right is an intuitive explanation of how bag aggregation works.
Figure 2:
distillation is introduced. We also discuss how to compute bag- aag-agggegttion loss and how they improve the performance of the lightweight model. The overall framework is illustrated in Fig. 2
3.1 BAGGING INSTANCES WITH SIMILARITY MATCHING
Given the unlabeled training set X = {x1, X2...., XN). we define the corresponding bag-wise training set as {S21 : 52 {N ) where each bag Q2 consists of set of instances. To transfer the instance-wise dataset to bag- bag-wise one, we first feed X into pretrained teacher model fr and get the corresponding features V [V11, V2,..., VN) where v; = fr (x;). For each anchor sample Xa in the dataset, we find positive samples which share high similarity with the anchor sample. Then the anchor sample as well as the similar samples are combined to form a bag. The samples one bag have compact representation in the embedding space. Several mapping function can be used to find similar samples:
K-nearest Neighbors For each anchor sample x, in the instance- wise dataset, first compute the pairwise similarity with all samples in the dataset S, = (Va Vi 1,2, NV. The bag Q corresponding to x, is defined as:
Ωa top rank K),
where top-ranks K) returns the indices of top K items in a set.
K-means Clustering Given the training feature set {V1, V2.... VN). we first assign pseudo-label qi to each sample i, where qi E (qi..... 9k} The clustering process is performed by minimizing the following term,
vfcg Cq.
Vi
where Cq. denotes the centering feature all features belonging to the label qi iee. cq. =
Eq Vj, Vj I...,
The bag S2 of anchor sample x, is defined as:
Ω (i qi qa, Vi = 1,2, N).
4

---

Published as a conference paper at ICLR 2022
Ground Truth Label If the ground truth label is available, also bag samples with the numan-annotated semantic labels. Given the label set Y = Y2.... we can bag related instances of the anchor sample x, via:
La (i yi Ya, vi 1,,
In this paper, we use K-nearest neighbors as the bagging strategy. More details about performance of using the K-means clustering based bagging strategy can found Appendix. Note that bagging instances via the ground truth label is just used to measure the upper bound of the proposed method.
3.2 KNOWLEDGE DISTILLATION VIA BAG AGGREGATION
Once we get the bag-wise dataset 52 utilizing pretrained teacher model, it can be used for distillation process. In each feed-forward process, the anchor sample x, and the positive sample x, which belong to the same bag ft are sampled together in one batch. We propose the bag- aag-aggregation distillation loss including the intra-sample distillation loss Lintra and inter-sample distillation loss Linter.
To aggregate the representations within bag into more compact embeddings, we minimize the following target function:
min L = I (I(fs(xs),
(5)
89
where metric function measure the distance between two embeddings there many metrics can be selected, such as cosine similarity, euclidean distance, etc. Here we use the normalized cosine similarity, i.e., the contrastive loss commonly used in self-supervised learning to measure the distance between x; and the anchor sample X.. The target function in Eq. 5 can be divided into two components:
C=L(fs(t((())), (T
(Ufs(ts(x)))
x.
Three separate data augmentation operators 6,, b2 t3 are randomly sampled from the same family of MoCo-v2 augmentations T, which is also adopted in SEED(Fang al. 2020) and DisCo (Gao et al.) 202T). where the first item focuses on pulling different views (augmentations) of the same sample together, and the second item aims at pulling different samples that are within same bag into more related ones. We term the first item as Lintra and the second item as Linter-
Intra-Sample Distillation The intra-sample distillation loss is variant of conventional contrastive loss. Contrastive learning aims to learn representations by discriminating the positive key among negative samples. Given two augmented views x and x' of one input image, MoCo (Chen al. 2020c) uses online encoder fg and a momentum encoder fx to generate embeddings of the positive pairs: = (x). fxxx''. The contrastive loss defined as
Ceontrast =- log exp(q kt/r)
expla ki/T)
During distillation, we simply replace fq and fk by the student model fs and teacher model fr. while weights of the teacher model fr are pretrained and are not updated during distillation. The intra-sample distillation loss can be formulated as
Lintra = log exp(fs(t((x.)) fr((z(xa))/7
(8)
Ei (xp((s(ttxx.,)) k, /T)
where 7 is the temperature parameter. We select negative samples k- in memory bank, which widely used in MoCo (He al. 2020) and many subsequent contrastive learning methods. The memory bank is a queue of data embeddings and the queue size is much larger than typical mini- batch size. After each forward iteration, items in the queue are progressively replaced by the current output of the teacher network.
Inter-Sample Distillation Given the anchor sample x, and a positive sample x,, in the bag D,,. natural to map highly related samples to more similar representations. In other words, we want the
5

---

Published as a conference paper at ICLR 2022
bag filled with related samples to be more compact. Inspired by Eq. distillation loss as
define the inter-
Linter = - log cxp(fs(ts(xp)) (tll)).fftttt frr(t((x))//T)
Σ (ts(()(t kj /r)
The intra- and inter-sample distillation loss serve as different roles. The intra-sample distillation works like conventional distillation (Hinton al., 2015 Romero al. (014), which aims minimizing distances between outputs of the teacher and student model given the same input. However, the inter- sample distillation mainly focuses on transferring the data relation knowledge taking the bag-wise dataset as the carrier, which is obtained from the pretrained teacher model.
4 EXPERIMENTS
In this section, evaluate the feature representations of the distilled student networks several widely used benchmarks. We first report the performance on ImageNet under the linear evaluation and semi-supervised protocols. Then we conduct evaluation on several downstream tasks including object detection and instance segmentation, as well as some ablation studies to diagnose how each component and parameter affect the performance.
4.1 PRE-TRAINING DETAILS
Pre-training of Teacher Model Two models are used teachers: ResNett-0 trained with MoCo- v2 (Chen et al. 2020c) for 800 epochs and ResNet-50x2 trained with SwAV for 400 epochs. The officially released weights are used to initialize teacher models for fair comparisons.
Self-supervised Distillation of Student Model Two models used students: ResNet-18 and ResNet-34. Following the settings of MoCo in Chen et al. (2020c), we add a 2-layer MI.P on top of the last averaged pooling layer form 128-d embedding vector. During distillation, the model is trained with the SGD optimizer with momentum 0.9 and weight decay 0.0001 for 200 epochs on ImageNet (Deng et al. 2009). The batch size and learning rate are set as 256 and 0.03 for GPUs, which simply follow the hyper-parameter settings as in Chen et al. (2020c). The learning rate is decayed to by a cosine scheduler. The CutMix used in Gidaris al. (2021) and Xu et al. (20206) is also applied to boost the performance. The temperature and the size of memory bank are set as 0.2 and 65,536 respectively. For the bagging strategy, we use K-nearest neighbors unless specified.
4.2 EXPERIMENTS ON IMAGENET
KNN Classification We evaluate representation of student model using nearest neighbor classifier. KNN classifier can evaluate the learned feature more di- rectly without any parameter tuning. Following Caron
Table I: KNN classification accuracy on Ima- geNet. We report the results on the validation set with 10 nearest neighbors. ResNet-50x2 used as the teacher.
let al. (2020): Abbasi Koohpayegani et al.) (200)) Fang 2020), extract features from center-cropped
al.
images after the last averaged pooling layers. For con- venient comparisons with other methods, we report the validation accuracy with 10 NN (we use the stu- dent model distilled from ResNet-50x2).. As shown in Table BINGO achieves 61.0% and 64.9% accu- racies on ResNet- 18/34 models, respectively, which outperforms previous methods significantly.
Method
ResNet-18 ResNet-34
SEED (Fang et al. 2020) BINGO
55.3
58.2
61.0
64.9
Linear Evaluation In order to evaluate the performance of BINGO, we train linear classifier upon the frozen representation, following the common evaluation protocol in Chen et al. (2020c). For fair comparisons, we use the same hyper- parameeessetters Fang et al. (200); Gao al. (2021) during linear evaluation stage. The classifier is trained for 100 epochs, using the SGD optimizer with 30 as initial learning rate. As shown in Table BINGO outperform previous DisCo and SEED with the
'Checkpoints of teacher models be downloaded from https://gt thub.com/ racebook

---

Published as a conference paper at ICLR 2022
Table Linear classification accuracy on ImageNet over different student architectures. Note that when using R50x2 the teacher, SEED distills for 800 epochs while DisCo and BINGO distill for 200 epochs. The numbers in brackets indicate the accuracies of teacher models. "T denotes the teacher and "S" denotes the student.
S
R-18
R-34
Method
T-1
T-5
T-1
T-5
Supervised (Fang et al.
2020
69.5
72.8
MoCo-Va V2 (Baseline) (Fag et al.
20200)
52.3
77.0
57.4
81.6
SEED
Fang al.
2020
(-50
57.6
81.8
58.5
82.6
DisCo
Gao et al.
2021
R-50 (67.4)
60.6
83.7
62.5
85.4
BINGO
(67.4)
61.4
84.3
63.5
85.7
BINGO
R-50 (71.1)
64.0
85.7
66.1
87.2
SEED
Fang al.
2020
R-152 (74.1)
59.5
83.3
62.7
85.8
DisCo
Gao et al
2021
R-152 (74.1)
65.5
86.7
68.1
88.6
BINGO
R-152 (74.1)
65.9
87.1
69.1
88.9
SEED
Fang et al.
2020
R50x2 (77.3)
63.0
84.9
65.7
86.8
DisCo
Gao et al
2021
R50x2 (77.3)
65.2
86.8
67.6
88.6
BINGO
R50x2 (77.3)
65.5
87.0
68.9
89.0
Table 3: Transfer learning accuracy (%) on COCO detection.
Mask R-CNN, ResNet- 18, Detection
Method
1x schedule
2x schedule
AP
AP
AP
APs
APM
APr
APd
AP
AP
APs
APM
APr
MoCo v2
31.3
50.0
33.5
16.5
33.1
41.1
34.4
53.9
37.0
18.9
36.8
45.5
BINGO
32.0
51.0
34.7
42.0
34.9
54.2
37.7
20.0
46.0
Table 4: Transfer learning accuracy (%)
COCO instance segmentation.
Mask R-CNN, ResNet- 18, Instance Segmentation
Method
1 schedule
2x schedule
Apms
AP
AP
APs
APM
AP2
Apms
APm
AP%
APs
APM
APL
MoCo v2
28.8
47.2
30.6
12.2
29.7
42.7
31.5
51.1
33.6
14.1
32.9
46.9
BINGO
29.6
48.2
31.5
12.8
30.8
43.0
31.9
51.7
33,9
14.9
33.1
47.2
same teacher and student models. Note that SEED distills for 800 epochs while BINGO runs for 200 epochs with ResNet-50x2 teacher, which demonstrates the effectiveness of BINGO.
Transfer Object Detection and Instance Segmentation We evaluate the generalization ability of the student model on detection and instance segmentation tasks. The COCO dataset is used for evaluation. Following He al. (2020), we use Mask R-CNN (He et al. (017]) for object detection and instance segmentation and fine-tune all the parameters of student model ResNet- 18 end-to-end. As shown Table and Table BINGO consistently outperforms models pretrained without distillation. The detection and segmentation results of ResNet-34 can be found in the appendix.
Semi-supervised Classification Following previous works Chen et al. 2020a b), we evaluate the proposed method by fine-tuning the student model ResNet- 18 with 1% and 10% labeled data, We follow the training split settings as in CCen et al. (2020a) for fair comparisons. The network is fine- tuned for epochs with SGD optimizer. The learning rate the last randomly initialized fe layer is set as 10. As shown in Table using ResNet-18 student model, BINGO obtains best accuracies with same ResNet-50 and ResNet- teacher when using and 10% labels, respectively.
Transfer to CIFAR-10/CIFAR-I1 classification Following the evaluation protocol in Fang et al. (2020); Gao al. (202)) assess the generalization of BINGO the CIFAR-10/CIFE - dataset. As shown in Table 6, compared with the previous state-of-the-ar art method DisCo (Gao et al. 2021 BINGO outperforms it by 1.5% and 3.2% respectively.
7

---

Published as a conference paper at ICLR 2022
Table 5: Semi-supervised learning by fine-tuning 1% and 10% images on ImageNet using ResNet-18
Method
T
1% labels
10% labels
MoCo v2 baseline
30.9
45.8
Compress (Abbasi Koohpayegani et al. 2020
R-50 (67.4)
41.2
47.6
SEED
Fang el al
2020
R-50 (67.4)
39.1
50.2
DisCo Gao et al
2021
R-50 (67.4)
39.2
50.1
BINGO
R-50 (67.4)
42.8
57.5
Compress (Abbasi
Koohpayegani et al. 2020
R-152 (74.1)
SEED Fang et al
2020
R-152 (74.1)
44.3
54.8
DisCo Gao et al.
2021
R-152 (74.1)
47.1
54.7
BINGO
R-152 (74.1)
50.3
61.2
BINGO
R-50x2 (77.3)
48.2
60.2
Table 6: Linear classification accuracy on CIFAR-10/100 with ResNet-18.
Table Lower and Upper bound performance exploration via the bagging criterion.
Method
T
CIFAR- 11FAR-00/100
Bagging Criterion
Accuracy (%)
MoCo v2 baseline
77.9/48. 1
Random Initialized model
46.6
SEED
R-50 (67.4)
82.3/56.8
pnpervised-pretraine model
64.8
DisCo
R-50 (67.4)
85.3/63.3
Ground-truth labels
65.8
BINGO
R-50 (67.4)
86.8/66.5
Self-supervised pretrained model
64.0
4.3 ABLATION STUDY
In this section, we conduct detailed ablation studies to diagnose how each component affect the performance of the distilled model. Unless specified, all results in this section are based ResNet-18, and distilled for 200 epochs.
Impact in K-nearest Neighbors We inspect the influence in K-nearest neighbors bagging strategy. As shown in Fig. 3 the results are relatively robust for range of k (==,,,,0,,00) In addition, we find that the classification accuracy decrease with & = 10, 20 compared with A: = because the noise introduced when i becomes large. However, the performance with relative small & = no better than = 5, we think the diversity is sacrificed when we only select the top-1 nearest neighborhood all the time.
Lower and Upper Bound of The Proposed Method As shown in Table 7 using data relation extracted from a ran- dom initialized model gives a poor performance of 46.6%, which can be lower bound of our method. Then we try to explore the upper bound performance by bagging instances via a seervvseed-peetrainee improvement of 0.8% over using data relation extracted from the unsupervised pretrained teacher model. When we directly the ground truth labels to bag instances, get highest upper bound performance, i.e., 65.8% Top-1 accuracy.
Ok y-dag
model, the performance gets an
$3.
L-]
b-10
Topsk
Figure 3: Top-1 accuracy with dif- ferent l in K-nearest neighbors.
Impacts of Data-Relation and Teacher Parameters. In our experiments, both the data relation and model parameters of teacher model are used to distill student model. We diagnose how each component affects the distillation performance. As shown Table model parameters represent whether to load the teacher model parameters into the teacher part in Fig. data relation refers to the Cinter loss. Besides, Lintra is intrinsically a basic contrastive-leaniin paradigm on two views of one instance. matter the teacher's 's parameters are loaded not, using the the data relation from pretrained teacher model always gets better results than using data relation from online student model, which verifies the efficiency of transferring teacher's data relation to student model. Interestingly, we find that BINGO even gets good result only utilizing teacher's data relation (Row 3 of Table 8). which is about 10% higher than model training without distillation.
8

---

Published as a conference paper at ICLR 2022
Table 8: Effects of utilizing teacher's data-relation and teacher's pretrained weights. The column of Student Relation means that we bag data with features extracted from student model online and the column of Teacher Relation means that we bag data with features extracted from pretrained teacher model. When teacher parameters are not used, we replace the pretrained teacher model as momentum update of student model like He et al. 2020).
Teacher Parameters
Student Relation
Teacher Relation
Accuracy
x
x
x
52.2 (w/o distillation)
x
x
57.2
/
X
X
62.2
X
62.0
/
62.5
X
64.0
Table 10: Top-1 accuracy of linear classification results on ImageNet with ResNet-34 under different
epochs. ResNet-152 is used as teacher model. "T" denotes the teacher and "S" denotes the student.
Method
S
T
Distillation Epochs
Top-1 Accuracy
Top-5 Accuracy
SEED
R-34
R-152(74.1)
200
62.7
85.8
BINGO
R-34
R-152(74.1)
100
67.8
88.4
DisCo
R-34
R-152(74.1)
200
68.1 1
88.6
BINGO
R-34
R-152(74.1)
200
69.1
88.9
Compare with Other Distillation Methods. We now compare with several other distillation strategies to verify the effectiveness of our method. We compare with two distillation schemes: feature-based distilla- tion method and relation-bas -ased distillation, which is termed KD and RKD, respectively. Feature- based distillation method aims at minimizing 12- 12-distance of teacher& & student's embeddings. Relation-based distillation method aims at minimizing the difference between sater-sample-eee similarrty graph obtained from teacher and student model. As shown in Table g BINGO outperforms all theses alternative methods.
Table 9: Top-1 accuracy of linear classifi- cation results on ImageNet using different distillation methods on ResNet-18 student model (ResNet-50 is used as teacher model)
Method
Top-1
MoCo- V2 baseline (Gao 2021
52.2
MoCo- V2 + KD (Fang et al.
2020)
55.3
MoCo- V2 + RKD
61.6
DisCo KD (Gao al. 2021
60.6
DisCo RKD (Gao 2021
60.6
BINGO
64.0
Computational Complexity. As shown in Fig. 2 the batch size is doubled during each forward propagation. The computation cost is increased compared with SEED (Fang al., 2020) and MoCo- v2 (Chen et al. 2020c). As for DisCo (Gao et af.) 2021), the computational complexity is also increased due to the multiple forward propagation for one sample: once for the mean student, twice for the online student and twice for the teacher model. The total number of forward propagation is 5, 2.5x bigger than SEED and MoCo-v2. For the above analysis, BINGO has similar computation cost with DisCo. To compare with SEED and DisCo under the same cost, we distill ResNet-34 from ResNet-152 for 100 and 200 epochs respectively. We compare results of 100 epochs with SEED and 200 epochs with DisCo. The results are shown in Table and and BINGO still shows significantly better performance than the two compared methods with the same training cost.
CONCLUSIONS
This paper proposes a new self-supervised distillation method, named BINGO, which bags related instances by matching embeddings of the teacher. With the instance-wise dataset mapped into bag-wise one, the new dataset can be applied to the distillation process for small models. The knowledge which represents the relation of bagged instances can be transferred by aggregating the bag. including inter-sample and intra-sample aggregation. Our BINGO follows a relation Dolatioo-guided guided principle, which shows stronger effectiveness than previous relation relation-agnostic methods. The proposed relation-basee distillation is a general strategy for improving unsupervised representation, and we hope would shed light on new directions for unsupervised learning.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 61932022, Grant 217201060011 Grant 61971285, and in part by the Program of Shanghai Science and Technology Innovation Project under Grant 2051 100100.
REPRODUCIBILITY STATEMENT
We will release the source code at mttppps://gttauu..ccm/aaaaaaaapp the reproducibility of this paper.
guarantee
ETHICS STATEMENT
The proposed approach seeks to improve the performance of unsupervised learning methods for small scale models. This will improve the application of unsupervised learning methods in more realistic computing resource-constrained scenarios. As far as we know, our method does not violate any ethical requirements.
REFERENCES
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self-supervised learning by compressing representations, Advances in Neural Information Processing Systems, 33: 12980-12992, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
Liang-Chiel Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33:22243-22255, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297, 2020c.
Hee Min Choi, Hyoa Kang, and Dokwan Oh. Unsupervised representation transfer small networks: I believe i can distill on-the-fly fly. Advances in neural information processing systems, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from friends: Nearest-neighbor contrastive learning of visual representations. arXiv preprint arXiv:2104 14548, 2021.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2022
Yuting Gao, Jia- Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning. arXiv preprint arXiv:2104.09124 2021.
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online hag-of-visual-worde generation for self-supervised learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6830-6840, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Neural Information Processing Systems, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1r3353150,.0253,, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33, 2020.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7096-7104, 2019.
K L Navancet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Simreg: Regression simple yet effective tool for self-supervised knowledge distillation. In British Machine Vision Conference (BMVC), 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976. 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 14/2,650, 2014.
11

---

Published as a conference paper at ICLR 2022
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations, 2019.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference on Computer Vision, pp. 988-604. Springer, 2020a.
Haohang Xu, Xiaopeng Zhang, Hao Li, Lingxi Xie, Hongkai Xiong, and Qi Tian. Seed the views: Hierarchical semantic alignment for contrastive representation learning, arXiv preprint arXiv:2012..0733, 2020b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros, Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang. Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In Interna- tional Conference on Learning Representations, 2021.
A APPENDIX
A.1 RESULTS OF K-MEANS BAGGING STRATEGY
Table 11: Top-1 accuracy with different cluster numbers C in K-means clustering.
Number of Clusters (C)
Accuracy
.000
62.7
10000
63.5
20000
63.8
50000
63.6
We additionally evaluate the performance of using K-means clustering as the bagging strategy according to Eq. in the main text. Given the pseudo-label q {q), 92,..., qN) and the anchor instance X,, the bag associated with x, is defined
Ω2 -i
1,2, .).
(10)
For implementation. ResNet-18 and ResNet-50 are used as the student and teacher model respectively, We evaluate the linear classification accuracy of the student model ImageNet-I We study various cluster numbers C as shown in Tab. [1] We find that a bigger cluster number can bring better results than smaller one. Noting that the linear classification accuracy bagging with K-nearest neighbors (where k=5) is slightly better than bagging via K-means clustering (with C=20000), i.e. 64.0% vs. 63.8%. Moreover, bagging with KNN is more convenient implement, so we choose the KNN-based bagging strategy in implementation.
A.2 OBJECT DETECTION AND INSTANCE SEGMENTATION OF RESNET-34
We further evaluate generalization ability of one more student model, i.e. ResNet-34, on object detection and instance segmentation tasks. The COCO (Lin et al. 2014) dataset is used for evaluation. Following He et al. (2020), we use Mask R-CNN (He et al. 2017) ) for object detection and instance segmentation, and fine-tune parameters of the student model. i.e. ResNet-34 distilled from the ResNet- 152 teacher model. As shown Table 12, BINGO consistently outperforms models pretrained with
distillation, SEED Fang et al. 2020 and DisCo Gao et al. (2011)
12

---

Published as a conference paper at ICLR 2022
Table 12: Transfer learning performance on COCO Lin et al. (2014) object detection and instance discrimination with ResNet-34 distilled from ResNet-152. "bb" denotes bounding box and "mk" denotes mask.
Mask R-CNN, ResNeet-3
Method
Object Delection
Instance Discrimination
ApP
AP
AP75
APs
APM
AP2
Apm
AP
AP
APs APM APz
MoCo v2
38.1
56.8
40.7
33.0
53.2
35.3
SEED
Fang et al.
2020)
38.4
57.0
41.0
33.3
53.7
35.3
DisCo
Gao al
2021
39.4
58.7
42.7
34.4
55.4
36.7
BINGO
39.9
59.4
43.5
22.8
43.3
52.1
35.7
56.5
38.2
16.8
37.9
51.6
0.6
0.-
(a) MoCo-V2 v2 baseline
(b) Distill w/o Bag togg-Agggattion
(c) Distill Bag Fag-Aggregttion
Figure 4: t-sne visualization of student's 's representations pretrained with the MoCo-v2 baseline (a), and distilled with (b) and without bag aggregation (c).
A.3 ANALYSIS AND DISCUSSIONS
We now inspect what the student learns during the distillation. Firstly we compute the average distance between anchor sample X, and its positive samples Xp in a bag Qr over whole dataset:
BagDis
E
E x2,
((fs((x)) -s(xx))
(11)
According to Eq. we compute the averaged distance in the bag using distilled student model. As shown in Table 3 the averaged distance in a bag smallest when the student model is distilled with bag-aggregation loss. We also compute the intra-class distance among all intra-class pairwise samples. As shown in Table [4] the proposed method also aggregate the bag of labels with the same ground truth labels on the unseen validation set.
Table 13:
Averaged distance between
Table 14: Averaged intra-class distance on Ima- geNet validation set
anchor and positive samples in the same bag
Method
Distance
Method
Distance
MoCo-V V2 baseline
0.38
MoCo-V2 baselinc
0.88
Distill w/o Bag- Aggrregatn
0.36
Distill w/o Bag-Aggregttion
0.72
Distill w/ Bag-Aggregation
0.32
Distill w/ Bag- aag-AAgregation
0.65
We visualize the last embedding feature to understanding the aggregating properties of the proposed method. 10 classes randomly selected from validation set. We provide the visualization of the student features. As shown in Fig. 4 the same color denotes features with the same label. It can be seen that BINGO gets more compact representations compared with models without distillation or distilling without pulling related samples in a bag.
13


Reviews:
[{"review_id": "p2F49GHYvmt", "reviewer": "Reviewer_dQqV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an interesting problem that is under-studied. It does tackle the problem following the same approach (a variant of KD applied to this problem). It presents clear novelties respect to prior art, but largely follows prior art in terms of the approach. The explanations are very clear and the paper is well written. Some related references could be added (see bellow), but overall it is complete. The experimental results seem a bit less complete for a number of reasons, although the results look to some extent promising.\n\nI have two main comments: methodology/story and experiments.\n\nIn terms of the methodology, the approach seems to be pulling the k most similar examples together. Obviously, the model capacity is a problem and forcing the model to distinguish between similar instances is challenging. This is a softer approach in which semantically similar instances are grouped together and thus the whole learning is less ambitious (I feel one related work not included is \"Khosla et al., Supervised Contrastive Learning, NeurIPS 2020\", which targets a similar goal but uses the supervised learning labels). The question is how this \"lack of modelling ambition\" affects downstream tasks. When I was reading the methodology I thought this setting would be good for ImageNet (in the end you'll \"cluster\" different classes together), but maybe not so good for other downstream applications. \n\nThis takes me to the limitations of the experimental validation. Right now there is a very weak comparison on COCO, where competing methods are not included. There's no comparison on PASCAL or instance segmentation, etc.\n\nOther comparison that is, in my opinion, not very conclusive is the semi supervised learning on imagenet. Other works (e.g. DisCo) show a graph of performance vs teacher. The largest teacher yields better performance than the one shown here, but I didn't see a clarification of which teacher is used.\n\nFinally, for being a method for SSL on low-compute models, it focuses on ResNet18. Is the MobileNet family not considered for example? both DisCo and SEED do use MobileNetV3 and EfficientNet for example. \n\n\nSmall issues: \nThe related work focuses more on pretext tasks than on contrastive method. This could do with some rebalancing.\nI'd suggest citing the conference paper rather than the arxiv version\nWhy on Table 1, the first two lines of the penultimate block have a different teacher performance than the 3rd line?\nThere's a typo in page 6: DINGO -> BINGO\nTable 7, 3rd line, looks like a very weird result. Are you sure of that one? Do you have any explanation? or maybe I misunderstood it?\nSome references\nKhosla et al., Supervised Contrastive Learning, NeurIPS 2020 (referred to above)\nXu et al., Knowledge Distillation Meets Self-Supervision, ECCV'20 (basically SEED is this paper minus the supervised learning bit)\nDwibedi et al., With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations, ICCV'21\n", "summary_of_the_paper": "The paper tackles the problem of training a self-supervised model using low(er)-compute models. Following the literature, the authors present a variant of distillation that achieves better performance than competing distillation methods. The base algorithm is based on contrastive learning (MoCoV2). The variant uses the teacher to find, for each instance in the training set, the k closest examples within the training set. Then, it uses two augmentations of the training instance as a positive pair (as customary for contrastive), but then also uses an example sampled from the k nearest to construct another positive pair. One element of each pair is passed through the teacher and another through the student, and each pair generates a loss. ", "summary_of_the_review": "While the method is clear and has an interesting premise, I am left wondering if the rationale behind the methodology is the right one. Experimental evidence is not very strong, and there are some clear aspects lacking in the evaluation. The most important issues are the lack of proper validation on downstream tasks, and the lack of variety of student architectures. (the score of 2 on \"Empirical Novelty And Significance\" is due to lack of completeness of experiments but can be raised with stronger evidence).\n\n(updated to 6 after rebuttal)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fOS2YmGq-T-", "reviewer": "Reviewer_Xmuw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some suggestion regards the writing: \n\n- Wrong reference for Compress in the beginning of third paragraph: \n\"Compress (Fang et al., 2021) and SEED (Fang et al., 2021) are two typical methods for unsupervised\ndistillation\"  \n-  large quantity of exemplars -> large number of exemplars\n- drops dramatically comparing with its supervised counterpart -> compared with its\n- termed as BINGO, which is short -> termed BINGO\n- targets at transferring the relationship -> targets transferring ...\n- while DisCo and BINGO distills for 200 epochs -> distill for \n- all theses alternative methods -> these\n- the proposed method also aggregate the bag -> aggregates\n- using the the data relation from pretrained teacher -> duplicate ``the''\n\n\n======================================================================================\n\nMain Review: \n\nThis paper is well-established upon the previous self-supervised distillation efforts that aims to train stronger small models via the self-supervised fashion. Previous works like Compress and SEED mainly focus on the instance-level contrastive learning, where all samples are treated equivalently as negative samples without relation-distillation. The key contribution of the BINGO is the propose of inter-sample loss (eq. 9) that additionally push sample embeddings with identical pseudo-labels closer.  Intuitively, the cluster -> label assignment measure is similar to SWAV, where the Sinkhorn algorithm is instead adopted and the representation of target model itself is leveraged for clustering. \n\nThe major concern from my side is the technical novelty of the BINGO and the proposed relation-distillation: it feels to me that the main reason that inter-sample distillation works is its mimicking the representations from other categories. How is the relation-distillation reflected in the inter-sample distillation loss? As it's not mimicking any categorical distributions and I'm not quite sure that can be viewed as relation across samples. \n\n======================================================================================\n\nSome Questions Regards the Implementation & Evaluations details:\n\n- The fact that K-Means clustering method works well is mostly because the features from Teacher model is kept frozen. A recent work in [1] proposes to use the trainable Teacher and Sinkhorn instead as clustering algorithm. This deserves further discussion to differentiate BINGO with it. \n\n- Table 1: results of BINGO seems to have used a stronger ResNet-50 as the Teacher (71.1% v.s. 67.4% from SEED and DisCo), what is the reason for this inconsistent comparisons?\n\n- Transfer learning results (detection/segmentation) only include the MoCo-v2 results, how about the numbers comparing with DisCo and SEED?\n\n- It seems that the number of clustering centers may lead to pretty diverse results, what is the intuition in selecting the optimal K? \n\n- According to SEED & SWAV, they uses multi-view strategy, is this adopted in BINGO? As I never see any description about it. \n\n\n[1] Unsupervised Representation Transfer for Small Networks: I Believe I Can Distill On-the-Fly, Hee Min Choi, Hyoa Kang, and Dokwan Oh, Samsung Advanced Institute of Technology, Suwon, South Korea, NeurIPS 2021\n\n", "summary_of_the_paper": "This paper presents a new self-supervised distillation learning schema which aims to address the ``low-relation'' distillation objective across instances in priori arts. In particular, the developed bagging instance technique cluster the offline Teacher representations into $K$ bags, assigning sample categorical labels. The overall training objective of BINGO is composed of the intra-sample distillation term, which retains the contrastive learning principle; and the inter-sample distillation term that select pseudo-positive samples from the clustering labels for further contrastive learning. Quantitative experiments on ImageNet classification task, transferring learning and semi-supervised learning task validate the effectiveness of proposed method.", "summary_of_the_review": "- The paper is well-constructed with clear motivation and comprehensive quantitative experiments. \n- Experiments show that the proposed BINGO learning schema is valid and bring incremental improvement over previous SEED and Compress. \n- My main concern is the technical contribution of BINGO somehow overlaps with [1] as previously mentioned. And some experimental settings are not consistent or clearly explained. \nI'd like to adjust my initial ratings after reading the response afterwards.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PpNZEEyw9U", "reviewer": "Reviewer_HSTU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall idea to novel to me, the idea of using pretrained teacher to aggregate similar samples is the main contribution and difference compared with previous works. The ablation study has well demonstrated the effectiveness of each component, but I'm a bit concerned about the main result since the comparison seems not fair with the previous method. \n\nFrom my point of view, the experiment is not quite fair to compare with the previous method.\n\n1. From Figure 2, it's clear to see that both X_{a} and X_{p} have to be passed into the student network, which means the batch size is doubled compared with SEED and DisCo. From my experience, the training will be at least 50% slower. On the other hand, increasing the number of positive samples can constantly improve the accuracy in contrastive learning. I'm looking forward to more experiments under the same training costs.\n\n2. In Table 1 (first half result), the teacher of SEED and DisCo has an accuracy of 67.4%, but BINGO is trained with a better teacher (71.1%), which is not a fair comparison. I'm looking for the result with the same teacher. (under the same training cost).\n\n3. Table 3 and Table 4 only report the result of ResNet18 and compare it with the MoCo baseline. The distilled network has a better performance than the baseline method is not a surprising phenomenon. I'm looking forward to more comparisons between BINGO, SEED, and DisCO. (under the same training cost). \n\n", "summary_of_the_paper": "This paper proposes a new knowledge distillation framework for unsupervised representation learning. Unlike the previous distillation method, where the knowledge transfer only happens between the same image, BINGO utilized a pretrained teacher model to aggregate similar samples into bags. The goal is to aggregate compact representations over the student with respect to instances in a bag. \n", "summary_of_the_review": "The paper is well written and easy to follow. As I have mentioned, I'm a bit concerned about the main result. Interestingly, I have reviewed this paper in another venue, but my questions are still not addressed well. However, I would like to increase my score if they can be handled well here. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NtGKDaGMfd4", "reviewer": "Reviewer_5UEk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The BINGO method seems to be simple conceptually, yet yields good results on unsupervised pretraining on small network.\n\nPositive points\n+ well exposed, although some aspects would benefit from more detail, see below \n+ interesting experiments, visualizations and ablation studies\n\nA few points would strengthen the paper:\n1) eq. (6) does not make use of data augmentation, yet its description mentions data augmentations, please clarify where it is used; are each instance different augmentations of an instance, for the intra as well as for the inter-sample loss?\n\n2) eq. (8) make negative samples appear, but it is not clear how these samples are selected and they are not referred to in the text, please make clear for self-containment.\n\n3) It would be interesting to highlight the accuracy of the Teacher model for reference in some of the tables.\n\n4) p.7 \"In our experiments, both the data relation and model parameters of teacher model are used to distill student model\". I am not sure what this means. Does the \"model parameters\" refer to the Lintra part of the loss, and the \"data relation\" refers to the Linter part of the loss, and is this an ablation of the impact of the two components of the loss? Please clarify this experiment.\n\nMinor:\na) Language is understandable, but could benefit from further proofreading, e.g. in the abstract: \"targets at transferring\" -> \"aims at transferring\"; or p.2 \"This may be heuristic for\" -> This might be a useful insight for... (for example); p.3: \"rare of them pay attention\" -> \"few of them pay attention\"; etc.\n\nb) p.2: the explanation for inter-sample and intra-sample distillation components are inverted in the introduction.\n\nc) Reference [Caron et al. 2020] was published in NeurIPS 2020 and there is no reason to use the arXiv version in the citation. Please double-check the other citations as well.", "summary_of_the_paper": "The paper introduces a method called BINGO for unsupervised feature learning of small models that rely on the distillation from larger teacher networks, i.e. unsupervised distillation.\nBINGO distills knowledge from a large model pretrained in an unsupervised manner to a small student network.\nThe main novelty is that the distillation from the teacher to the student leverages the relations learned by the teacher on the dataset, i.e. aims to transfer similarity clusters formed by K nearest neighbours.\nThe unsupervised pretraining of the student models is evaluated with linear or K-NN classification on ImageNet, in a limited labeled data setting, or by assessing generalizability to detection and instance segmentation. The method obtains competitive results w.r.t the state of the art with small models.\nAblation studies are done to investigate the impact of the different ingredients of the method.", "summary_of_the_review": "The paper presents an interesting method and good supporting experiments. I would like to see the few points of clarification I raised above addressed in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four experts reviewed the paper. All but Reviewer HSTU recommended acceptance. The authors clearly did a great job with the rebuttal, which convinced two reviewers to raise their scores above the acceptance threshold. Notably, the reviewers found the newly added experiments impressively strong. The rebuttal also addressed some clarification questions. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance. As mentioned by the reviewers, some experiments and discussions in the rebuttal should be included in the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)