Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a method called BINGO for unsupervised feature learning of small models that rely on the distillation from larger teacher networks, i.e. unsupervised distillation.
BINGO distills knowledge from a large model pretrained in an unsupervised manner to a small student network.
The main novelty is that the distillation from the teacher to the student leverages the relations learned by the teacher on the dataset, i.e. aims to transfer similarity clusters formed by K nearest neighbours.
The unsupervised pretraining of the student models is evaluated with linear or K-NN classification on ImageNet, in a limited labeled data setting, or by assessing generalizability to detection and instance segmentation. The method obtains competitive results w.r.t the state of the art with small models.
Ablation studies are done to investigate the impact of the different ingredients of the method.
main_review: The BINGO method seems to be simple conceptually, yet yields good results on unsupervised pretraining on small network.

Positive points
+ well exposed, although some aspects would benefit from more detail, see below 
+ interesting experiments, visualizations and ablation studies

A few points would strengthen the paper:
1) eq. (6) does not make use of data augmentation, yet its description mentions data augmentations, please clarify where it is used; are each instance different augmentations of an instance, for the intra as well as for the inter-sample loss?

2) eq. (8) make negative samples appear, but it is not clear how these samples are selected and they are not referred to in the text, please make clear for self-containment.

3) It would be interesting to highlight the accuracy of the Teacher model for reference in some of the tables.

4) p.7 "In our experiments, both the data relation and model parameters of teacher model are used to distill student model". I am not sure what this means. Does the "model parameters" refer to the Lintra part of the loss, and the "data relation" refers to the Linter part of the loss, and is this an ablation of the impact of the two components of the loss? Please clarify this experiment.

Minor:
a) Language is understandable, but could benefit from further proofreading, e.g. in the abstract: "targets at transferring" -> "aims at transferring"; or p.2 "This may be heuristic for" -> This might be a useful insight for... (for example); p.3: "rare of them pay attention" -> "few of them pay attention"; etc.

b) p.2: the explanation for inter-sample and intra-sample distillation components are inverted in the introduction.

c) Reference [Caron et al. 2020] was published in NeurIPS 2020 and there is no reason to use the arXiv version in the citation. Please double-check the other citations as well.
summary_of_the_review: The paper presents an interesting method and good supporting experiments. I would like to see the few points of clarification I raised above addressed in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of training a self-supervised model using low(er)-compute models. Following the literature, the authors present a variant of distillation that achieves better performance than competing distillation methods. The base algorithm is based on contrastive learning (MoCoV2). The variant uses the teacher to find, for each instance in the training set, the k closest examples within the training set. Then, it uses two augmentations of the training instance as a positive pair (as customary for contrastive), but then also uses an example sampled from the k nearest to construct another positive pair. One element of each pair is passed through the teacher and another through the student, and each pair generates a loss. 
main_review: The paper tackles an interesting problem that is under-studied. It does tackle the problem following the same approach (a variant of KD applied to this problem). It presents clear novelties respect to prior art, but largely follows prior art in terms of the approach. The explanations are very clear and the paper is well written. Some related references could be added (see bellow), but overall it is complete. The experimental results seem a bit less complete for a number of reasons, although the results look to some extent promising.

I have two main comments: methodology/story and experiments.

In terms of the methodology, the approach seems to be pulling the k most similar examples together. Obviously, the model capacity is a problem and forcing the model to distinguish between similar instances is challenging. This is a softer approach in which semantically similar instances are grouped together and thus the whole learning is less ambitious (I feel one related work not included is "Khosla et al., Supervised Contrastive Learning, NeurIPS 2020", which targets a similar goal but uses the supervised learning labels). The question is how this "lack of modelling ambition" affects downstream tasks. When I was reading the methodology I thought this setting would be good for ImageNet (in the end you'll "cluster" different classes together), but maybe not so good for other downstream applications. 

This takes me to the limitations of the experimental validation. Right now there is a very weak comparison on COCO, where competing methods are not included. There's no comparison on PASCAL or instance segmentation, etc.

Other comparison that is, in my opinion, not very conclusive is the semi supervised learning on imagenet. Other works (e.g. DisCo) show a graph of performance vs teacher. The largest teacher yields better performance than the one shown here, but I didn't see a clarification of which teacher is used.

Finally, for being a method for SSL on low-compute models, it focuses on ResNet18. Is the MobileNet family not considered for example? both DisCo and SEED do use MobileNetV3 and EfficientNet for example. 


Small issues: 
The related work focuses more on pretext tasks than on contrastive method. This could do with some rebalancing.
I'd suggest citing the conference paper rather than the arxiv version
Why on Table 1, the first two lines of the penultimate block have a different teacher performance than the 3rd line?
There's a typo in page 6: DINGO -> BINGO
Table 7, 3rd line, looks like a very weird result. Are you sure of that one? Do you have any explanation? or maybe I misunderstood it?
Some references
Khosla et al., Supervised Contrastive Learning, NeurIPS 2020 (referred to above)
Xu et al., Knowledge Distillation Meets Self-Supervision, ECCV'20 (basically SEED is this paper minus the supervised learning bit)
Dwibedi et al., With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations, ICCV'21

summary_of_the_review: While the method is clear and has an interesting premise, I am left wondering if the rationale behind the methodology is the right one. Experimental evidence is not very strong, and there are some clear aspects lacking in the evaluation. The most important issues are the lack of proper validation on downstream tasks, and the lack of variety of student architectures. (the score of 2 on "Empirical Novelty And Significance" is due to lack of completeness of experiments but can be raised with stronger evidence).

(updated to 6 after rebuttal)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new self-supervised distillation learning schema which aims to address the ``low-relation'' distillation objective across instances in priori arts. In particular, the developed bagging instance technique cluster the offline Teacher representations into $K$ bags, assigning sample categorical labels. The overall training objective of BINGO is composed of the intra-sample distillation term, which retains the contrastive learning principle; and the inter-sample distillation term that select pseudo-positive samples from the clustering labels for further contrastive learning. Quantitative experiments on ImageNet classification task, transferring learning and semi-supervised learning task validate the effectiveness of proposed method.
main_review: Some suggestion regards the writing: 

- Wrong reference for Compress in the beginning of third paragraph: 
"Compress (Fang et al., 2021) and SEED (Fang et al., 2021) are two typical methods for unsupervised
distillation"  
-  large quantity of exemplars -> large number of exemplars
- drops dramatically comparing with its supervised counterpart -> compared with its
- termed as BINGO, which is short -> termed BINGO
- targets at transferring the relationship -> targets transferring ...
- while DisCo and BINGO distills for 200 epochs -> distill for 
- all theses alternative methods -> these
- the proposed method also aggregate the bag -> aggregates
- using the the data relation from pretrained teacher -> duplicate ``the''


======================================================================================

Main Review: 

This paper is well-established upon the previous self-supervised distillation efforts that aims to train stronger small models via the self-supervised fashion. Previous works like Compress and SEED mainly focus on the instance-level contrastive learning, where all samples are treated equivalently as negative samples without relation-distillation. The key contribution of the BINGO is the propose of inter-sample loss (eq. 9) that additionally push sample embeddings with identical pseudo-labels closer.  Intuitively, the cluster -> label assignment measure is similar to SWAV, where the Sinkhorn algorithm is instead adopted and the representation of target model itself is leveraged for clustering. 

The major concern from my side is the technical novelty of the BINGO and the proposed relation-distillation: it feels to me that the main reason that inter-sample distillation works is its mimicking the representations from other categories. How is the relation-distillation reflected in the inter-sample distillation loss? As it's not mimicking any categorical distributions and I'm not quite sure that can be viewed as relation across samples. 

======================================================================================

Some Questions Regards the Implementation & Evaluations details:

- The fact that K-Means clustering method works well is mostly because the features from Teacher model is kept frozen. A recent work in [1] proposes to use the trainable Teacher and Sinkhorn instead as clustering algorithm. This deserves further discussion to differentiate BINGO with it. 

- Table 1: results of BINGO seems to have used a stronger ResNet-50 as the Teacher (71.1% v.s. 67.4% from SEED and DisCo), what is the reason for this inconsistent comparisons?

- Transfer learning results (detection/segmentation) only include the MoCo-v2 results, how about the numbers comparing with DisCo and SEED?

- It seems that the number of clustering centers may lead to pretty diverse results, what is the intuition in selecting the optimal K? 

- According to SEED & SWAV, they uses multi-view strategy, is this adopted in BINGO? As I never see any description about it. 


[1] Unsupervised Representation Transfer for Small Networks: I Believe I Can Distill On-the-Fly, Hee Min Choi, Hyoa Kang, and Dokwan Oh, Samsung Advanced Institute of Technology, Suwon, South Korea, NeurIPS 2021


summary_of_the_review: - The paper is well-constructed with clear motivation and comprehensive quantitative experiments. 
- Experiments show that the proposed BINGO learning schema is valid and bring incremental improvement over previous SEED and Compress. 
- My main concern is the technical contribution of BINGO somehow overlaps with [1] as previously mentioned. And some experimental settings are not consistent or clearly explained. 
I'd like to adjust my initial ratings after reading the response afterwards.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new knowledge distillation framework for unsupervised representation learning. Unlike the previous distillation method, where the knowledge transfer only happens between the same image, BINGO utilized a pretrained teacher model to aggregate similar samples into bags. The goal is to aggregate compact representations over the student with respect to instances in a bag. 

main_review: The overall idea to novel to me, the idea of using pretrained teacher to aggregate similar samples is the main contribution and difference compared with previous works. The ablation study has well demonstrated the effectiveness of each component, but I'm a bit concerned about the main result since the comparison seems not fair with the previous method. 

From my point of view, the experiment is not quite fair to compare with the previous method.

1. From Figure 2, it's clear to see that both X_{a} and X_{p} have to be passed into the student network, which means the batch size is doubled compared with SEED and DisCo. From my experience, the training will be at least 50% slower. On the other hand, increasing the number of positive samples can constantly improve the accuracy in contrastive learning. I'm looking forward to more experiments under the same training costs.

2. In Table 1 (first half result), the teacher of SEED and DisCo has an accuracy of 67.4%, but BINGO is trained with a better teacher (71.1%), which is not a fair comparison. I'm looking for the result with the same teacher. (under the same training cost).

3. Table 3 and Table 4 only report the result of ResNet18 and compare it with the MoCo baseline. The distilled network has a better performance than the baseline method is not a surprising phenomenon. I'm looking forward to more comparisons between BINGO, SEED, and DisCO. (under the same training cost). 


summary_of_the_review: The paper is well written and easy to follow. As I have mentioned, I'm a bit concerned about the main result. Interestingly, I have reviewed this paper in another venue, but my questions are still not addressed well. However, I would like to increase my score if they can be handled well here.
review_title: null

Meta-review: Four experts reviewed the paper. All but Reviewer HSTU recommended acceptance. The authors clearly did a great job with the rebuttal, which convinced two reviewers to raise their scores above the acceptance threshold. Notably, the reviewers found the newly added experiments impressively strong. The rebuttal also addressed some clarification questions. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance. As mentioned by the reviewers, some experiments and discussions in the rebuttal should be included in the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)