Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper provides a self supervised method tailored for object detection (in the RCNN
paradigm), inspired from BYOL.  
The general idea is to use a non-parametric method (selective search) to generate some
boxes, then apply a BYOL-style loss between ROI-Aligned features of each box coming from
different augmentations of the image. 

It differs from other SSL methods (BYOL, SWAV,...) in that it instead of pretraining solely
the backbone, this methods ensures that all the components of the detector (FPN, RCNN head)
are pre-trained.

It shows convincing results on COCO and pascal as well as some subsets, in a few shot setting.
The code is made available as part of the supplementary, thus helping with reproducibility.

main_review: Overall, the paper is very well written, clearly motivated and ablated. The contribution is
solid and seems widely applicable. The paper is lacking a few details and references that
I'm listing in the following section, but overall I would recommend acceptance.

## Missing references
Here are a couple of references which would make the related work more complete:

- [3] is a CVPR 2021 paper investigating SSL for detection in the DETR framework
- [4] explores SSL for object detection without using Imagenet images
- [5] has some similar contrastive ideas, targets the RetinaNet architecture
- [6,7] more recent SSL methods (for classification)

## Technical details on the SSL recipe

This paper is inspired by BYOL, but seem to diverge in a number of subtle way that are not
always clearly explained or ablated. I list them in this section.

### Stop gradients

Regular BYOL prevents gradients from flowing back to the target network, by using a
stop-gradient mechanism.
This is an important aspect of BYOL, it has been shown to be a mandatory for other methods eg [2].

In this paper, I didn't see any mention of a stop-gradient mechanism. Is it applied? If not
is there a reason for it? This needs to be clarified in my opinion.

### Symmetric loss
    
In section 3.3, the paper indicates that the contrastive loss is symmetrized by swapping V1
and V2/V3 and re-computing the loss.

This is not a cheap choice, since it incurs doubling the number of forwards. However, the
paper doesn't clearly motivate it, and as far as I can tell there is no ablation on the
impact this symmetrization has on the performance. Could this be clarified?

### Multi view

This paper uses 3 views in the loss, compared to BYOL which uses only 2.
This change is ablated in table 4, and improves performance by roughly 0.6 Box AP, while
having very little effect on the mask AP.

According to the paper, "multiple views (V3) further directs SoCo towards learning
scale-invariant and translation-invariant representations". Is there any evidence of that
fact? For example, if it really improves scale invariance, I would expect the performance on
small objects to improve more significantly than large, since the third view is a
down-sampled version of the image. Is this the case in practice? Similarly, table 5(a) would
possibly benefit from reporting the AP break-down by size.

What is the computational cost incurred by this third view?

In table 1, the paper further reports results with 4 views, with even better results.
It seems that adding views help, is there a reason to believe that it will saturate if one
kept adding more scales or crops?


Finally, the V3 is a deterministic transform of V2. Would it help to add some
stochasticity in the process? Eg resize to a random size (possibly bigger than the original)
instead of always downsampling by 2?  


### Batchsize

The study on the batch size (table 5b) is interesting and a bit counter-intuitive. Usually
SSL methods tend to always benefit from bigger batch-size. Is there any intuition why this
is not the case for this particular case?

## Longer training?

In [1], the authors have shown that after sufficient iterations, the effect of imagenet
pre-training tends to be erased, and random init tends to perform as well supervised
pre-training.

This paper reports only short schedules (1x and 2x), which arguably are important for
practitioners. However, I would be interested to find out if this SoCo method keeps its edge
over the supervised baseline on a long schedule (say 6x)?

If it does, I would consider it an even more impressive result.

## Pretraining dataset

The method uses pre-training on ImageNet. While this aids comparison with various SSL
methods that are not detection aware, I would argue that this paper could be made stronger
by also using more complex images during pre-training. One potential option would be to
pre-train on the whole train set of COCO as well as the unlabeled set of COCO. If size is an
issue, other detection datasets could be appended (Object365, OpenImages, ...)

Beyond the pure performance implications, it would be interesting to see if some of the
conclusions of the ablations change for such a pretraining. In particular Table 5.c shows
that adding more randomly selected boxes doesn't help, and the paper blames it on the fact
that "most of the images in the ImageNet dataset only contain a few objects". Pretraining on
a real detection dataset would alleviate this issue, and potentially unlock the full
potential of the method.

## Localization pretraining?

In detection, both classification and localization play a big role in the final performance
of the model. Because of the contrastive criterion that is being used, it seems that more
emphasis is put on the classification part than on the regression part.

Would it be possible to include some regression-aware objectives in the pre-training as
well?
One idea that comes to mind is similar to the box-jitter idea that is used in this paper:
it could be possible to train the localization part of Mask-RCNN to cancel the jitter, ie
regress the original position of the box. Has anything along those lines been tried?


[1] Rethinking ImageNet Pre-training, He at al.

[2] Exploring Simple Siamese Representation Learning, Chen et al

[3] UP-DETR: Unsupervised Pre-training for Object Detection with Transformers, Dai et al

[4] Self-EMD: Self-Supervised Object Detection without ImageNet, Liu et al

[5] Unsupervised Pretraining for Object Detection by Patch Reidentification, Ding et al

[6] Barlow Twins: Self-Supervised Learning via Redundancy Reduction, J.Zbontar et al.

[7] Emerging Properties in Self-Supervised Vision Transformers, M.Caron et al.

# Post rebuttal update:
In light of the excellent rebuttal, I increase my score to 9
limitations_and_societal_impact: No particular societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes SoCo, a method for self-supervised representation learning for downstream object detection and instance segmentation tasks.

They argue for stronger alignment between downstream tasks and self-supervised pre-training. They demonstrate this in the case of object detection by (a) developing a representation learning scheme that encourages object level scale and translation invariance and (b) pre-training several components of the downstream detector such as FPN and R-CNN head unlike prior work which mostly just trains the ResNet backbone.

The former property will help their model detect objects at several locations and scales as is needed for object detection.  The latter property helps them get stronger alignment between the upstream and downstream setups. To this end they propose "scale aware assignment". The ideas is that, representations for small objects are taken from finer FPN pyramid levels and for bigger objects are taken from coarser FPN pyramid levels. This matches what happens in the actual FPN where anchors at finer pyramid levels are smaller and at coarser pyramid levels are bigger.

The overall recipe is as follows: Selective search, a hand-crafted region proposal technique, is used to suggest boxes on all images. K of these boxes are randomly selected in each training step per image and three views of that image are constructed using random resized crop and resize operations. FPN features are extracted using RoiAlign for these box locations using "scale aware assignment". A BYOL loss is applied between representations of the same box across the three views. As in BYOL an EMA teacher network and projection head are used.

They pretrain R50-C4 and R50-FPN  models upstream and fine-tune corresponding Mask-RCNN models downstream for MS COCO and Pascal VOC object detection and MS COCO instance segmentation. Low data regime is explored using 5% and 10% "Mini" COCO setups. ImageNet linear eval results are included in the appendix. Ablation experiments evaluate whether stronger alignment in terms of architecture between pre-training and fine-tuning were helpful and sensitivity w.r.t. hyper-parameters such as view size, momentum for EMA, batch-size etc.
main_review: The paper is well written and easy to understand. The experiments section is detailed using various datasets and different amounts of data and meaningful ablation experiments. Self-supervised learning for detection is of significance to the community and this paper furthers an emerging theme of trying to pre-train the entire model and do more than just learn a global image representation.

Recommendations to further improve the paper are below.

InsLoc [12], DetCon [13], and ReSim [41] are all closely related to the submission. Of these only InsLoc is explained in some detail in the related works section. DetCon suggests regions and aligns representations across multiple views. ReSim uses a sliding window approach instead of heuristic region proposal methods, but in the end also identifies positive pairs based on spatial alignment and uses a contrastive loss to align representations. $\mathrm{ReSim-FPN}^T$ infact transfers the FPN downstream similar to how the proposed method does. But ReSim does not have "scale aware assignment". ReSim and DetCon are contemporary works to this submission. They came out on arxiv just 2 months before the deadline, but a more detailed discussion of these three papers will be a great addition to the related works section.

Please also add "Self-Supervised Representation Learning from Flow Equivariance. Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun Tech report, arXiv, Jan. 2021" (This is a dense representation learning method that uses flow as an additional data augmentation).

Also "UP-DETR: Unsupervised Pre-training for Object Detection with Transformers. Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen. CVPR 2021" is similar to the current work as it pre-trains all of DETR, and not just the backbone, and transfers it downstream.

Lastly Romijnders, Rob, et al. "Representation learning from videos in-the-wild: An object-centric approach." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021, has similar ideas but use a supervised object detector as their region proposal module and use predicted class labels to define positives.

In Table 5, is row 1 basically equal to BYOL? Is row-2 basically InsLoc? If not what are the differences?

In Table 5, how do you pre-train a FPN model without scale aware assignment in rows 3 and 4?

The performance deltas due to parameter changes in the ablation experiments are very small. This,  together with the lack of error bars, makes it difficult to know when a parameter is critical and when it has no effect. Computational costs might prohibit the addition of error bars everywhere but even showing just 1 number with error bars could convey a sense of the variances involved.

BYOL is not really contrastive. So calling it contrastive learning when there are no negatives involved is confusing. But I do not know what else to suggest.


## Post rebuttal update
The rebuttal response addresses my concerns. The rebuttal also included several experiments that further strengthen the paper. I continue to support the acceptance of this paper.
limitations_and_societal_impact: The model does poorly on ImageNet classification downstream. This limitation has been identified and reported in the appendix. Societal impact has also been discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims to design a self-supervised pre-training method for the downstream object detection task.  The authors propose to perform self-supervised object-level representation learning. Specifically, this method uses selective search to obtain object proposals in unlabeled images and extract the object-level features as in Mask R-CNN. Clear improvements have been achieved on COCO object detection and instance segmentation with Mask R-CNN.
main_review: ## Strength
- The proposed object-level pre-training method is simple, straightforward, and motivated.
- The paper is well written and organized, making it easy to read and follow.
- The method achieves strong performance in COCO semi-supervised setting (5%/10% training data).

## Weakness
- The used representation method as in BYOL is not exactly contrastive learning.  The method name in the title and main paper is not appropriate.  In the BYOL paper, the authors distinguish their method from contrastive methods. 
- The fine-tuning experiments are mainly conducted on COCO with a single architecture Mask R-CNN. But the pre-training should be general, even in the single task of object detection. To prevent overfitting a single dataset and a single arch. The authors are suggested to experiment with different datasets, e.g., Cityscapes, and different architectures, e.g., RetinaNet and FCOS. For different archs, you can just extract the pre-trained backbone and compare it to supervised or other self-supervised methods.
- The proposed method performs worse than the previous method PixPro on VOC.
- In Table 4, why does FPN make the performance worse?

## Others
- Did you try pre-training on non-object-centric datasets like COCO? The paper only reports results on ImageNet which is object-centric. I guess that's also why you can only set a small K for each image. It would be more interesting if the method can benefit more from data like COCO.

## After Rebuttal
The rebuttal has addressed all my concerns. Thus I raise the rating from 6 to 7.

limitations_and_societal_impact: Yes, they are in the appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission addresses the problem of constructing a good initialized pretrained network for the object detection task. Compared to previous work, the authors argue that pretraining should be adapted to object detection by focusing on object-level features learning. Instead of pretraining on ImageNet, the authors pretrain the model by designing an object level contrastive learning framework. Experiments on COCO and PASCAL show that the proposed method achieves higher performance. 
main_review: ### Strengths
+ The motivation and novelty of the submission is clearly explained. The Figure 1 is almost self-explainable for the framework.

+ The details of implementation are provided in the submission. Besides, the authors also provide reproducible code in the supplementary. 

+ The authors provide ablation study for different aspects, including hyper-parameters, different datasets to verify the proposed method, which is much appreciated.

### Weakness
- Limited application scope: As claimed in the title, this proposed method is used for "Pretraining for Detection". This means this method should be generalized for all kinds of detectors for object detection. However, the major novelty is proposing a pretraining framework which focuses on object-level feature learning by using some unsupervised proposal generators. This method may perform well on two-stage detectors (also verified in the experiments). However, for other types of detectors, such as one-stage detectors and anchor-free detectors, there are no experiments to verify. 

- Ablation studies of the proposal generator: There are two aspects which may be needed to make the method more convincing.
  * In this submission, the authors only adopt Selective Search which is an old unsupervised proposal generator. The influence of different selections of proposal generators is not verified. If the proposed method is not sensitive to such selection, the value of the submission can further increase.
  * According to Figure 2 in Section A.3 (Supplementary), the distribution of proposal size highly concentrates on the size of 20-60. However, in line 168-169, the authors assign different proposals to different layers of FPN according to their area. This may influence the learning effects of lateral layers ($P_3, P_4, P_5$), because #proposals is small. 
limitations_and_societal_impact: * The title may be too broad. It would be better to limit to two-stage detector, or the authors need to verify such method also works well on one-stage or anchor-free detectors. 
* To show the proposed method is insensitive to the proposal generator's selection, it would be better to choose different proposal generator other than Selective Search and do parallel experiments
* The authors may need to address the imbalance of the proposals for different layers' learning problem as mentioned in the weakness part above.
review_title: null

Meta-review: All reviewers agree this is a solid paper with a good novel contribution, great experimental results, and detailed analysis/ablation. The authors did a good job addressing the reviewers' concerns in their responses too. Clear accept.
Decision: Accept (Spotlight)