Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new self-supervised learning framework that leverages scene images to enhance object-level representation learning that only relies on object images. The basic idea is to apply unsupervised region proposal and k-NN based object similarity to diversify intra- and inter-image pairs with more variances to aid object-level representation learning. The improvements on various downstream tasks, the ablations and visualizations all demonstrate the advantages of the presented framework. The main contribution is to utilize scene images for contrastive self-supervised learning.
main_review: The main novelty of the paper is to enhance the contrastive self-supervised learning by including not only object images from ImageNet, but also scene images from Coco. The paper is written well in general and the experiments are extensive.

I did not give a higher rating due to the following:
- The technical contribution of the paper is not very significant as the multi-stage framework largely depends on previous contrastive learning work as BYOL. The improvement is basically due to the data augmentation from Coco which is not technically difficult.
- The generality of the framework could have been better demonstrated by not just on BYOL. Please discuss other possibilities in more detail.
- The trade-off between the performance gain and the additional cost due to multiple stages needs to be discussed more. 

Some minor things of writing:
- Please cite the published paper other than its preprints on arXiv (e.g., BYOL)
- L169 the symmetric loss is not clear here.
- L104 ‘detect’ -> ‘detecting’
- L129 ‘is comprised of’ -> ‘comprises’ 
- L203 ‘models Following’ -> ‘models. Following’

limitations_and_societal_impact: The limitations and broader impact of the work are discussed in the supplementary material and look good to me.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes an object-level self-supervised pre-training approach which extends and tries to improve existing work on image-level self-supervised pre-training. The authors motivate the problem by mentioning that image-level self-supervised pre-training methods are infeasible for images with many objects due to the hard constraints imposed by such methods. The paper proposes to apply self-supervised pre-training to similar objects in and across images. The authors use pre-trained image-level model to select the best candidate object pairs across images and then pre-train self-supervised models using the selected object regions. The authors report some performance improvements on the COCO dataset. 
main_review: The paper proposes a slightly interesting approach for self-supervised pre-training. The proposed approach achieves promising initial results - improving over existing image-level pre-training approaches and even fully-supervised baselines. The motivation behind this work is sound. and the authors have explained the limitations of existing works well.

However, a large number of aspects have not been considered in the paper. Such aspects need to be dealt with to ensure completeness of the paper. I list a few major limitations of the paper here. 

1. The most important limitation of the paper is a missing study on how the proposed approach scales with the availability of more data. The authors should try to train their model on larger datasets like OpenImages. This will show whether the proposed approach has any potential for larger-scale training.

2. The use of a image-level model to first find the nearest images/objects and then training the same model on the retrieved objects seems circular. It's not clear why the model should change/learn anything new if the similar objects are already retrieved using the same model. What additional information does this framework gain?

3. The authors should also provide a baseline by training the ORL on the ImageNet dataset to highlight the problem with using iconic images for pre-training. 

4.  In table 5d, the authors compare the performance when using selective search vs the performance when using ground-truth boxes. The authors show that using the ground-truth boxes leads to a lower performance than when using selective search. The authors hypothesize that this might be due to more diverse boxes returned by selective search. Did the authors try using jittered ground-truth boxes to induce more diversity in the ground-truth boxes? That might lead to better performance than selective search.

5. Finally, I think the paper can be made much stronger if the authors conduct downstream experiments on datasets other than COCO. Does the proposed approach lead to similar performance improvements when the pre-training dataset is different from the downstream dataset? 
limitations_and_societal_impact: The authors have adequately discussed the limitations and societal impact, but only in the supplementary material and not as part of the text of the main paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a technique to adapt self-supervised contrastive learning to scene images where multiple objects are present in each image. Specifically, the proposed technique has three stages: 1. image-level pre-training with standard contrastive learning; 2. Finding object pairs images (crops) using selective search proposals and pre-trained features from stage-1; and 3. Doing contrastive learning with object image pairs. Several experimental comparisons demonstrate consistent improvements w.r.t. baselines that does only image-level contrastive learning.
main_review: Strengths:
- Using self-supervised image features and selective search for correspondence object discovery is simple yet interesting technique.
- Experimental results on wide range of datasets and on several tasks.
- Consistent performance improvements w.r.t. image-level self-supervised learning techniques.

Weaknesses:
- The proposed stage-wise technique is more like an engineering solution to mine more/cleaner positive and negative pairs for contrastive learning. As such, the technical innovation is incremental.
- There is no discussion or comparisons with several works that extend image-level self-supervised learning to more local pixel features. Couple of works that I can find: [a,b].
- L65 says that this is the first study to extent self-supervised learning beyond image-level contrastive learning. This is an over-claim and not correct in my opinions as there are several existing works that propose pixel-level contrastive learning techniques.

Clarifications:
- Line 51 says that the proposed technique uses 'off-the-shelf contrastive learning pre-trained model'. Does this mean the stage-1 is already pre-trained on ImageNet data? Or, is it trained from scratch on COCO dataset? This point is important to clarify in the rebuttal.

[a] Pinheiro et al. Unsupervised learning of dense visual representations. NeurIPS 2020.
[b] Chaitanya et al. Contrastive learning of global and local features for medical image segmentation with limited annotations. NeurIPS 2020.

**Post Rebuttal**:

It seems reviewers generally agree that the proposed approach is more like an engineered system rather than providing any sound technical innovations. Like other positive reviewers, I do not find any clear issues in this work and experiments are reasonably convincing. It would be better if authors had provided experimental comparisons to more dense pixel-level contrastive learning works. After carefully considering author response and reviewer discussion, I tend to keep my original rating (6).
limitations_and_societal_impact: Limitations and broader impact are added in the supplementary. It is better to make the discussion of limitations more specific to this work (like, when the proposed algorithm fails and possible remedies) instead of mentioning general limitations of learning on large-scale datasets.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims to learn object-centric self-supervised representation from non-iconic scene images. In addition to the standard self-supervised objective (e.g., BYOL), the paper suggests attracting the object patch with the (i) nearby patch of the chosen object and (ii) similar objects in different images. For (ii), the paper uses the similarity score of the standard image-level self-supervised model. This additional training signal from similar object patches improves the transfer performance of the learned representation on classification and detection/segmentation tasks.
main_review: Learning representations from non-iconic images is an important direction of self-supervised learning. There have been a large number of concurrent works on this topic (including DenseCL and Self-EMD discussed in the paper), but most of them reported the trade-off between image-level (e.g., classification) and pixel/object-level (e.g., detection/segmentation) tasks. Unlike the prior work that leverages the spatial features before the final pooling layer to improve detection performance, this paper simply matches the pooled features of related object patches, improving both classification and detection performance. Furthermore, the paper demonstrates that the model can find the visual correspondence of objects and produces a better attention map.

While the paper is well-written and easy-to-follow, I have two questions in detail:

I. How does the three-stage pipeline is exactly implemented?

- How long does the image-level model in Stage 1 pretrained? Is it a strict two-step approach of training Stage 1 and Stage 3 for 800 epochs, respectively? Does Stage 3 reinitialize the model or fine-tune the one from Stage 1? If latter, the other baseline models also should be fine-tuned again using the same computational budget.
- How often does Stage 2 perform? Are the correspondence maps only computed once? Do the correspondence maps become better when using the model trained from Stage 3? If it gives a better correspondence, would additional repetitions of Stage 2 and 3 give further improvement?

II. How does the attention maps are produced?

- The results seem promising, but I could not find the implementation detail. Also, it would be informative to provide quantitative results.

p.s. Typo: Missing punctuation after textbf in line 203.
limitations_and_societal_impact: Both limitations and societal impact are well discussed in Appendix, though they could be moved to the main text following the intention of extending the page limits to nine pages.
review_title: null

Meta-review: - The proposed method is tackling an important problem. The reviewers found the approach is reasonable but technically not very novel. 
- The many concerns from the reviewers are addressed well enough by the rebuttal. 
- The strength is mainly the experiment results which are wide, consistent and reasonably convincing.
- The clarity is good enough but can be improved. 
Decision: Accept (Poster)