Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on unsupervised object detection pre-training using object priors. Specifically, two kinds of object priors are used in this paper. The first one is region prior which is generated from the encoder attention maps. The second one is instance prior which is obtained by selecting high-quality outputs from the detector decoder. Experiments on COCO and PASCAL VOC show that the proposed approach obtains better results than the previous unsupervised object detection pre-training approaches when only limited number of annotated data is available.
strengths_and_weaknesses: ### Strengths
- The proposed approach is interesting.
- Promising results are obtained when limited number of annotated data is available.

### Weaknesses
- Novelty of region prior. There is already published work which uses attention maps to generate object bounding boxes to pre-train object detector [a]. The authors should discuss and compare with this paper.
- Marginal improvement when more annotated data is available. From Table 3, when using the full COCO training set, only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg.

[a] DAP: Detection-Aware Pre-training with Weak Supervision
questions: From Table 3, when using the full COCO training set, only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg. Is there any explanation about why this happens and any thought on how to improve it?
limitations: Only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents an unsupervised pre-training strategy for DETR-Like object detectors. The core idea is to generate object priors from the encoder to guide the self-supervised pre-training.



strengths_and_weaknesses: Strengths:
* Using the encoder itself to generate object priors is a good idea.
* The proposed method outperforms other methods designed for pre-training DETR-like object detectors.

Weakness:
* Missing important comparisons: For a rough categorization, the previous contrastive pre-training methods focused on (1)  **image classification** (e.g., Mocov2 and SimCLR) (2) **general dense prediction tasks**[1, 2, 3, 4] (such as object detection and segmentation) (3) **detr-like object detectors**. This paper belongs to (3), and only compared previous works of (1) and (3). But the detailed comparisons to the (2) are missing. It would be much better if the authors can provide deep discussions.
* The advantages of pre-training is the generalization to various downstream tasks. I wonder if the representation learned in this method can be also used for other tasks such as segmentation.

[1] Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning. CVPR 2021.
[2] Dense Contrastive Learning for Self-Supervised Visual Pre-Training. CVPR 2021.
[3] Region Similarity Representation Learning. ICCV 2021.
[4] Deeply Unsupervised Patch Re-Identification for Pre-training Object Detectors. TPAMI 2022.

questions: See weakness.
limitations: Yes. The authors mentioned "there still exists a large gap from supervised training on class agnostic object proposal evaluation, which calls for further studies."
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission proposes a pre-training approach for object detection transformer, it generates pseudo bounding box regression targets by leveraging the attention cues from the pre-trained backbone network. Specifically, both object proposals generated merely in encoder attention map and object instance prediction from decoder head are used for pseudo boxes. The method is evaluated on low-data and full data fine-tuning, as well as object proposal generation. 
strengths_and_weaknesses: 
**Strengths**
- The submission is well-written and easy to follow. The math definitions and theories are clear and concise.
- It is surprising that bounding box prediction merely supervised by the attention cues and self-supervising signal could achieve such superior object proposal results, which has not been done before.

**Weaknesses and parts to be improved**
- I listed some of my concerns in the questions, which I'm not very sure, the authors are encouraged to address my concerns and I would consider raising my score.
- The approach is simple but it seems the technical contributions are sparse, the major components of attention map generation and region proposal generation are lent from prior work.
- In section 2.4, the authors mention it _uses a pre-trained SWAV model to extract features_,  however, the details are not given. Specifically, is the regions cropped from the image and fed to the pre-trained SWAV backbone, or some feature cropping method is used, e.g., ROIalign.
- In the introduction, the authors mention they are inspired by DINO and referred to the attention maps in the figures, however in the method section, it actually uses the Normalized Cuts algorithm to obtain the attention map. I believe it is better to rewrite the part in the introduction to be more clear and indicate in the figure captions how the attention map is obtained.

**minor comments**
- In line 84, can then **be** generated
- In equation 3, FB is not defined
- In equation 8, B and R should be defined properly, I guess R should be replaced with other characters as it is already used in equation 1.
- In line 144, SWAV is not cited
- In line 212, Detreg should be DETReg
- In Tab.2 and Tab.3, citations are missing for the prior works.
- In Tab.4, I suggest adding the IOU metric for calculating Recall, which is more clear.
questions: 1. I actually have a question regarding the effectiveness of detection pre-training. For the **Full-data finetuning** experiment, the supervised training is compared as a baseline, however, would it be fair to add the pre-training time to the baseline? I guess when you extend the baseline training time, it would easily catch up with the detection pre-training methods, which would pose a concern on the effectiveness of detection pre-training for full-data finetuning.
2. Although the work does not employ low-level unsupervised proposal methods like Selective Search, would it be more advantageous to incorporate these proposals which are generated from low-level cues? The intuition is that they may provide better or complementary coverage to some objects such as small ones, or others that have clear boundaries to the background.
3. I'm interested in how the pre-trained object feature embedding loss affects the method, have you tried removing the feature embedding loss? Do you have any perspectives, e.g., it is a regularization that avoids the feature representation deviating too much from the pre-trained representation, or maybe it is the major driven force in the proposed object detection pre-training, as it enforces the region features to be close to an image-level pre-trained model like SwAW. Could you provide more ablation study on this part?
4. Have you tried other unsupervised pretraining models as initialization and embedding loss, e.g., BYOL, do you have any ideas or perspectives on this aspect?
limitations: The submission does not discuss the limitations, it is encouraged to include more discussion on the limitation and possible future works and improvements that could be done.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an unsupervised object detection pre-training framework that can generate object priors and learn detector jointly. This work is inspired by DINO, the self-attention maps can generate region prior bounding boxes. Authors utilize Deformable DETR as their main framework. The proposed method achieves good results on low data-regime object detection on COCO and full-data finetuing on PASCAL VOC.
strengths_and_weaknesses: Strengths:
1) The motivation is good. Generating object priors by attention maps is an alternative of the popular selective search algorithm. However, a pre-trained backbone is important and necessary.

Weaknesses:
1) In Section 3, authors should claim which detection framework they used for pre-training in the begging of section 3, which will make the paper clearer. 
2) The proposed method highly rely on the pre-trained backbone, authors initialize the ResNet50 backbone of JoinDet with SwAV, did authors explore other unsupervised pre-training methods, e.g. BYOL, SimCLR?
3) Authors should highlight the setting of the pre-training methods, supervised/SwAV only pre-trains backbone, while JoinDet pre-trains whole network. 
4) There are experiments conducted in the setting where pre-training is performed on COCO/ImageNet and finetuning is performed on PASCAL VOC (Table2). Why authors did not do experiments in the setting where pre-training is conducted on ImageNet and fine-tuning is performed on COCO. I hope to see this experiment in the rebuttal.
5) Why authors only do experiments under low-data regime on COCO, is it consistent on PASCAL VOC?
6) The improvement over DETReg is somewhat limited.
questions: My biggest concern comes from the experiments and the limited improvements under fine-tuning setting, see weaknesses above. 
limitations: Authors have discussed broader impacts in their supplementary material.
review_title: null

Meta-review: metareview: The paper received mixed reviews. Three reviewers rated borderline accept and one reviewer rated borderline reject. The authors provided detailed responses to the raised concerns/questions and supported their responses with additional ablation study, experimental result on new dataset (e.g., VOC). 

For reviewer fpzy (who gave borderline reject), the requested additional analysis have been provided by the authors. The major remaining issue is "The improvement over DETReg is somewhat limited". The results presented in the paper did show consistent improvement over DETReg on three settings with at least 1 mAP improvement. 

After reading the reviews and the responses, while there are no enthusiastic supports from the reviewers, the AC does not find sufficient ground to reject the paper. This paper introduces new ideas for unsupervised object detection pretraining and show consistent improvement over the baselines over three evaluation settings. The AC believes that this work would benefit the community and thus recommends to accept.
confidence: Certain
Decision: Accept