Unsupervised Object Detection Pretraining with Joint Object Priors Generation and Detector Learning
Authors: Yizhou Wang
Meilin Chen
SHIXIANG TANG
Feng Zhu
Haiyang Yang
LEI BAI
Rui Zhao
Yunfeng Yan
Donglian Qi
Wanli Ouyang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Unsupervised pretraining methods for object detection aim to learn object discrimination and localization ability from large amounts of images. Typically, recent works design pretext tasks that supervise the detector to predict the defined object priors. They normally leverage heuristic methods to produce object priors, \emph{e.g.,} selective search, which separates the prior generation and detector learning and leads to sub-optimal solutions. In this work, we propose a novel object detection pretraining framework that could generate object priors and learn detectors jointly by generating accurate object priors from the model itself. Specifically, region priors are extracted by attention maps from the encoder, which highlights foregrounds. Instance priors are the selected high-quality output bounding boxes of the detection decoder. By assuming objects as instances in the foreground, we can generate object priors with both region and instance priors. Moreover, our object priors are jointly refined along with the detector optimization. With better object priors as supervision, the model could achieve better detection capability, which in turn promotes the object priors generation. Our method improves the competitive approaches by \textbf{+1.3 AP}, \textbf{+1.7 AP} in 1\% and 10\% COCO low-data regimes object detection. 


Unsupervised Object Detection Pretraining with Joint Object Priors Generation and Detector Learning
Yizhou Wang
Meilin Chen Shixiang Tang² Feng Zhu
Haiyang Yang*
Lei Bai Rui Zhao Yunfeng Yan'
Donglian Qi'
Wanli Ouyang
'Zejjiang University "The University of Sydney "SenseTime Research
*Shanghai Al Laboratory SNanjing University
'Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China
Eyizhouwang, merlinis, yvonnech, qidl)0zju.ede edu.cn stan3906duni. sydney. wetineyyeuuu... edu,
(zhufeng, (haorui>@eensetten com baisanshi @gmannhii@mmmm.m.mmmmmm samaail.cooo com
hyyang@smail .jj..du... edu. wanli. wanli. ouyangdsydney. edu.au
Abstract
Unsupervised pretraining methods for object detection aim to learn object discrim- ination and localization ability from large amounts of images. Typically, recent works design pretext tasks that supervise the detector to predict the defined object priors. They normally leverage heuristic methods to produce object priors, egg., selective search, which separates the prior generation and detector learning and leads to sub-optimal solutions. In this work, we propose a novel object detection pretraining framework that could generate object priors and learn detectors jointly by generating accurate object priors from the model itself. Specifically, region priors are extracted by attention maps from the encoder, which highlights fore- grounds. Instance priors are the selected high-quality output bounding boxes of the detection decoder. By assuming objects instances in the foreground, we generate object priors with both region and instance priors. Moreover, our object priors are jointly refined along with the detector optimization. With better object priors supervision, the model could achieve better detection capability, which in turn promotes the object priors generation. Our method improves the competitive approaches by +1.3 AP, +1.7 AP in 1 % and 10% COCO low-data regimes object detection.
Introduction
Object detection is fundamental task in computer vision, whose goal is to predict of bounding boxes and category labels for all objects of interest in the images. The success of modern detectors [4]
35
12
1
27
29 31
32 41 relies the large-scale datasets with precious annotations which
very costly and even infeasible for enormous images available in the Internet. Therefore, unsupervised
pretraining methods 122 3 for object detection are proposed to ease the burden of these human
annotations, which enable the detectors to be fast deployed in the real world by finetuning the pretrained model with only a few annotated images.
While various unsupervised pretraining methods, egg. SwAW 5. been shown to learn good backbones for object detection 28
BYOL
18
DetCo [1], have
only few methods are able to pretrain the detection head, which is also a key component of complete detection architecture [24 20 25). To enable the localization of the pretrained detection head, few recent
52
46
37
'Equal contribution. The work was done during an internship Sense Time. CCoresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

predictions
backbone
encoder
decoder
backgrounds
input image
region priors (a)
instance priors
training epochs
(b)
Figure 1: (a) JoinDet extracts supervision from the knowledge learned by the model itself. The self-attention highlights foregrounds and the detector outputs instance predictions. (b) During training, object prior generation and detector optimization can mutually guide each other, leading to better region priors and better instance priors.
methods use hand-craft methods [58] 45 11
to generate object priors on unlabeled images to construct the pretext tasks. Specifically, UP-DETR [2 and DETReg B generate object priors by randomly cropping patches and selective search 45
respectively. Then, both of them require the detection model to learn detect those object priors for pretraining. Although these methods bring possibilities of pretraining detection heads, we argue that they suffer from two main drawbacks: (1) Both random cropping and selective search can only provide limited object-related supervision. For random cropping. it does not explicitly focus objects. Selective search, as a hand-crafted and heuristic box generation method, relies on low-level features of input images to generate object priors, which is an extra time-consuming non-deep-learning process. (2) The supervision can not be progressively refined during training. For example, the object priors of selective search are fixed through the pretraining process, which limits the upper bound of the methods.
In this paper, we propose a novel unsupervised object detection pretraining method that can generate object priors and learn the detector jointly. In our method, we mine the supervision (object priors) based on the model itself, and show that the generated object priors can be jointly refined with detection optimization. Inspired by DINO [6. the self-attention maps can generate region prior bounding boxes, as they highlight the foreground regions (see Fig. a)). Meanwhile, the output bounding boxes of the detector can generate instance prior bounding boxes, although may be located the backgrounds (see Fig. [a)). By combining knowledge region priors and instance priors together, can generate reliable object priors only from the detector model. Furthermore, the object prior generation and detector optimization can mutually evolved (see Fig. [b)). Supervised by reliable object priors, the model can gradually learn better object localization information, producing better region and instance priors for better object priors.
Given the object priors generated along with detector optimization, the supervision signal for the detector learning shifts quickly the beginning, which leads to undesirable learning instabilities. In order stabilize the pretraining process, both preceding and current object priors should be considered, and the update should be implemented in a slowly progressing way. To this end, we accomplish this regularization by designing a momentum box update strategy. Specifically, we match the current object priors to their neighboring and proceeding priors, and implement the update as a momentum-basee average of their object prior locations.
Our contributions are three-fold. (1) We propose to mine supervision from the knowledge of the model to be learned itself for unsupervised object detection pretraining, which can generate supervision signals. (2) We design an unsupervised object detection pretraining method with Joint region priors generation and Detector learning (JoinDet) which jointly generates reliable object priors and learns object detection
progressive way. (3) We propose Box Smooth Module to stabilize the pretraining process on JoinDet. Our method outperforms state-of-the--rts on low data-regime object detection on MS COCO, and full-data finetuning on PASCAL VOC. Detailed related works provided in supplementary materials.

---

Region Generation Module
Instance Generation Module
Box Smooth Module
M
pr
backgrounds
pll
R
R -
772
+ 1 m
Backbone
Encoder
Decoder
DETR loss
G
Figure 2: JoinDet generates region priors from the transformer encoder and instance priors from the transformer decoder. Reliable object priors can be achieved by concatenating region and instance priors. The Box Smooth Module then smooths object priors refinement in a momentum manner to preserve previous more accurate supervision and stabilize the pretraining process.
2
Methodology
Our goal is to mine effective supervision from the knowledge learned by the model itself for pre- training without annotations. To this effect, propose unsupervised object detection pretraining method with joint object priors generation and detector learning (JoinDet) which dynamically gener- ates reliable object priors and learns object detection. Specifically, we first extract the eigen attention map from the transformer to highlight foregrounds. We then generate region priors from the eigen attention map using the Region Generation Module and choose reliable predictions as instance priors using the Instance Generation Module. By combining region and instance priors, reliable object priors are generated and set as the pseudo-labels for the object detection pretraining process. To stabilize the optimization process, we further propose Box Smooth Module to smooth the refinement of object priors during pretraining.
During pretraining, the supervision is alternated between two steps: (1) generating reliable object priors by combining region priors and instance priors, including Step 1-3 below. (2) optimizing the detection model with reliable object priors (Step 4). Specifically.
Stepl: Generate region priors (Sec. 2.7. At n-th epoch, given image the output patch features F of the transformer encoder E can be computed by F = E(G())), where 9 is the backbone network. To better highlight foregrounds, we generate the eigen attention map M with M KCFF), which is detailed later. The region priors P can then be generated by the Region Generation Module.
Step2:
Generate instance priors (Sec. 2.2 With the DETR-based detection frameworks, object queries and patch features are fed into a transformer decoder D together to generate foreground prediction boxes, i.e., Pa, = D(F, q). Guided by the eigen attention map M, instance priors can be generated by the Instance Generation Module denoted as IG, i..., P = (G((n.,M)
Step3: Smooth object priors refinement by the Box Smooth Module (Sec. 2.3 Reliable object priors RR can be generated by combining region priors P and instance priors pi with R, = Pr + P Given R, and the last object priors R.-1, the momentum averaged object priors are computed by RR BS 11) where BS represents the Box Smooth Module. The Ro is initialized with the region priors Pr before pretraining the detector.
Step4: Train the model with updated supervision using prediction loss (Sec. 2.4j. Given model predictions and updated supervision Rn, we train the transformer parameters using set prediction
loss.
2.1 Region priors generation
There are two steps of region priors generation. First, compute the eigen attention map on patch features from the transformer encoder. Second, extract the foreground bounding boxes as region priors.
3

---

Generate the eigen attention map. Our main motivation follows [47 39] is that the eigen attention maps in the vision transformer can well highlight salient foreground objects. Specifically, given an input image x and DETR with backhone G, transformer encoder &, the eigen attention map M is extracted from the transformer encoder by
(I) where K. represents the computation method of the eigen attention map mentioned in [47]. and FF E Rixw represents output patch features from transformer encoder. The details of the eigen attention map computation method
M 1==((F== -(=(()))(((((()))
elaborated in supplementary materials.
Extract region priors. As foregrounds are highlighted in the eigen attention map, the bounding boxes of foreground regions can be extracted as region priors using Region Generation Module RG. Concretely, .M firstly partitioned into background parts (denoted by 0) and foreground parts (denoted by 1) by bi-partition mask m:
if Miy >a 0, otherwise,
1,
mij-
where i, j represent the position index, a priors pP" are then generated by computing the minimum bounding rectangles of all connected foreground regions.
Tww Σ ش My is the average score of M. Region
In addition, at the beginning of pretraining, to generate the initial supervision Rg, we compute Pj using the unsupervised pretrained backbone G by
R2 P = RG(K()((x))
(3)
Although region priors represent highlighted foregrounds in the eigen attention map, they do not divide foregrounds into foreground instances, and therefore instance priors should be mined.
2.2 Instance priors generation
The foreground predictions from the box prediction branch of DETR provide instance information. With object queries and output patch features FF passing transformer decoder D, DETR generates a set box predictions P by
p = DDF.qq) (P, = (b
where N denotes the number of output predictions and by. P. denotes the predicted bounding box, and logits in foreground- foreground-backkrouudd predictions, respectively. The prediction score š, can be computed by 8, Softmax((), (β,). As the model only predicts background (class label 0) and foreground (class label 1), we only reserve foreground predictions p...
(5) Although foreground predictions can define instances, some of them might correspond to the back- ground, which should be removed by the Instance Generation Module to improve reliability.
pf [bbj. 85)1 arg max DJ 1
Instance Generation Module IG, in which foreground predictions pf and the eigen attention map M interact together to eliminate background predictions and generate instance priors P Specifically, boxes with low prediction scores in p/ are removed with
pi = IG(PI .M) (P/IP/ e SP.j E [1,N),N,/N, 2 t):
(6)
where P is the element in ps, N is the number of elements in the Pl, Ny is the number of foreground pixels in the eigen attention map M bounded by Pj. N, the number of total pixels in M bounded by Pf- Here, t is threshold set to 0.5. Guided by highlighted foregrounds in M, background predictions in pl can be removed. The remaining predictions are treated as reliable instance priors Pt.

---

2.3
Box Smooth Module
Given generated region priors P and instance priors P', reliable object priors R can therefore be generated by
R pr Pt
(7)
where @ denotes the concatenation operation. We use reliable object priors R. to supervise the pretraining process.
As region priors P and instance priors P are dynamically updated along with network optimization, the fast evolution of reliable object priors R may lead to the instability of the training process. Motivated by Weight Box Fusion [40], we propose a Box Smooth Module to overcome these problems, which preserves accurate object priors from previous supervision and reduces the perturbation using a momentum update strategy. Specifically, given current object priors Ry, together with previous supervision Rn- we cluster all boxes in R., and Rn-1 using IoU metric to generate K" box clusters. We recompute the box coordinates and scores using all boxes in the k-th cluster to generate new box by. The supervision R, {bbb, sk)] is then updated in momentum strategy by
DE md s,b E Lj=1.bEER (1 (1-m")))
8, + +E T+U
by
Lim1
(8)
Emm s + E/= (1-11mmm)) m*)sj
where k-th cluster contains boxes in Rn-1 and U boxes in Rn, and m² E [0, 1] represents the momentum coefficient to adjust the influence of previous supervision Rn-1 and current object priors Rn m is empirically set as 0.45. When m² is set to I, the supervision will not be updated. When m" is set to 0, the supervision will be directly changed to current object priors R2. When initializing the pretraining (n = 0), we set Rg = Pō as the initial supervision. With such design, we smooth the refinement of object priors stabilize the training process.
Discussion, As no labels are used supervise the model, generated region priors and instance priors have relatively large variations between training epochs. The Box Smooth Module is a critical design the framework, which suppresses inaccurate previous supervision and slows down the shifting speed of supervision to guarantee stable optimization of the model. Note that Weighted Box Smooth just a simple implementation refine the supervision in a momentum update strategy, which different from its original use ensemble boxes from different object detection models.
2.4 Objective function
Here, we describe the objective function of JoinDet during pretraining. Following DETReg 3 JoinDet has three prediction heads: fes which predicts the class (foreground or background), fhoz which predicts bounding boxes, and femb which reconstructs features of the predicted regions. Assume K boxes {bx)]
j left after the Box Smooth Module, use pretrained SWAV (5) model of box regions, assign class label {uh- (cx, b,, ] Let { denote N query embedding outputted by transformer decoder, the JoinDet outputs can be denoted as: Dj = (v)) b, foor (v,), 2, = femb(v;). The model predictions can be defined as ý = {ūj = (Ø,, bj, 2,)
to extract features Z = {
1, and let targets =
k=1
Following DETR (4) we use Hungarian bipartite matching algorithm 26 to match yj and رJ by finding the permutation that minimizes the optimal matching cost between y and ū,:
arg min 5
Ematch (Vix (o()))
where Lmatch is the pairwise matching cost matrix. After finding the optimal a, the set prediction loss is defined as:
N
Huungarion (Ui, 05)
Σ
AcLose(CjsPaai) + Lo (b,, ba(s) + de Aemob 2agi)
(10)
where Lete is the classification loss which can be implemented via the cross-entropy loss or Focal Loss, Looz is based on L1 loss and GloU Loss [36]. Following DETReg, we set Lemb as Lemi La())
||z; 2.())11.
5

---

Table 1: Low-Data regimes object detection results. Finetuning using 1% and 10% of the labels data on COCO. Following DETReg [3] all models are pretrained for 50 epochs on COCO without labels and finetuned using 1% and 10% labeled data. Results on val2017 are reported.
Pretraining
COCO 1%
COCO 10%
Supervised
11.31+0.3
26.34++..1
SwAV [5]
11.79+0.3
27.79+0.2
ReSim [50]
11.07+00.4
26.56++0.3
DETReg 3
14.58.0..3
29.12:0.22
JoinDet
15.89-0.2
30.87
Experiment
3.1 Implementation details
Architecture. We choose Deformable-DETR |57 as the baseline model for its high accuracy and fast convergence. Following DETReg [3) in prediction heads, fhor and femb are multi-layer perceptrons (MLPs) with 2 hidden layers of size 256 followed by ReLU (34) layer. The output dimensions of foor and femb are and 256, respectively. We use a single fully-connect layer with output dimension 2 as fols.
Datasets. For a fair comparison, we pretrain our JoinDet on ImageNet- IK 13 ImageNet- 100 and MS COCO [30] without labels, and evaluate our method on MS COCO and PASCAL VOC. ImageNet- IK contains 1.28M object-centric images with 1000 class categories. Similar other works 38 21 561. we use subset of ImageNet-IK that contains 125K images and 100 classes as ImageNet- MS COCO is a popular object detection dataset that contains 121K scene images with 80 object categories labeled with bounding boxes. PASCAL VOC contains 20K images with 21 object categories.
backbone of JoinDet with SWAV (5), which is fixed throughout the training stage. We pretrain Join for 5 and 50 epochs on ImageNet-IK and MS COCO, respectively, where the pretraining schedules are set the same as DETReg. We update object priors every and epochs for ImagNet- 1K and MS COCO, respectively. As our method relies on the self-attention map in the transformer encoder. most generated object priors focus on big objects, so we use large-scale jittering mentioned in [17 to alleviate the imbalance problem. More implementation details and experimental results are provided in supplementary materials. All experiments are implemented on 16 NVIDIA V100 GPUs.
Training. Following DETReg, we initialize the ResNet50 223
3.2 Comparison with State-of-the-art methods
Low-Data regimes object detection. To evaluate the effectiveness of JoinDet, we conduct exper- iments in a low-data regimes object detection setting on COCO, where only 1% or 10% amounts of COCO labeled data are used during finetuning. The experimental results are presented in Tab. T Firstly, our method outperforms existing backbone pretraining methods, with significant perfor- mance gain of 4.68 AP, and 4.53 AP over the supervised pretraining baseline on 1% and 10% COCO data, respectively. Second, when compared with DETReg, which also fully pretrains the model, our method remains 1.31 AP, and 1.75 AP performance gain on 1% and 10% COCO data, respectively, showing the effectiveness of progressively refined object priors.
Full-data finetuning. To further evaluate JoinDet, we finetune on two datasets: MS COCO and PASCAL VOC. On MS COCO, following previous works (3, we finetune JoinDet using similar training schedule on MS COCO train2017 and evaluate on ca/2017. On PASCAL VOC, we finetune on trainval07 + 12 and evaluate on test07. Comparison with state-of-the the art methods is presented in Tab. and Tab. 3 Tab. 2 shows that JoinDet achieves higher performance on VOC. Concretely, JoinDet outperforms DETReg with +0.2 AP and +1.0 AP when pretraining on ImageNet- KK and COCO, respectively, When using COCO as the pretraining dataset, JoinDet shows considerable performance gain. We suggest that, as JoinDet is based on mining object priors on the pretraining dataset, more diverse objects contained COCO provide more diverse supervision on JoinDet, which further boosts its performance when transferred to downstream datasets. Second, JoinDet achieves

---

Table 2: Object detection results when finetuned on PASCAL VOC. All models are based on Deformable DETR 57
Table 3: Object detection results when fine- tuned on MS COCO. All models are based on Deformable DETR [57.
Pretraining
Pretruin dataset
Epochs
AP
AP50
AP75
Supervised
ImageNet-IK IK
100
59.5
82.6
65.6
Pretraining
Epochs
AP50
AP75
SwAV
ImageNet-11
100
61.0
83.0
68.1
Supervised
44.5
63.6
48.7
DETReg
ImageNet-IK
100
63.5
83.3
70.3
SwAV B
50
45.2
64.0
49.5
JoinDet
ImageNet-IK
100
63.7
83.8
70.7
UP-DETR 12
50
44.7
63.7
48.6
DETReg
3
COCO
100
63.4
83.7
70.8
DETReg
3
50
45.5
64.1
49.9
JoinDet
COCO
100
64.4
84.0
71.3
JoinDet
45.6
64.3
49.8
AP50 on PASCAL VOC test07
DETReg
DETReg
--
JoinDet
AP5O on COCO va/2017
joinDet
80
60
72
52
10
20
30
40
10
20
30
40
Epochs
Epochs
Figure 3: AP50 (COCO style) learning curves with DETReg and JeinDet on VOC when finetun- ing with small epochs.
Figure 4: AP50 (COCO style) learning curves with DETReg and JoinDet on COCO when fine- tuning with small epochs.
+0.4 AP performance gain on SwAV and comparable performance with DETReg full-data COCO finetuning, Second, as shown in Fig- 3 and Fig. JoinDet is finetunigg-efficient, which shows considerable improvement when finetuning with small epochs.
Class agnostic object proposal evaluation. To demonstrate that JoinDet learns better to detect objects without labels, following DETReg, examine the class agnostic performance of JoinDet on Tab. First, JoinDet outperforms other pretraining approaches, indicating that using progres- sive region priors improves the localization ability. Second, pretraining COCO shows larger performance gain of +1.7 AP, which shows our method benefits more on scene images with various foreground objects.
4 Ablation
Default Settings. We implement ablations on single-scale deformable DETR. JoinDet is unsupervised pretrained on COCO for 50 epochs with a learning rate drop at epoch 40. The supervision mined from the model outputs is updated every 10 epochs. The results this section are evaluated by full-data finetuning on PASCAL VOC with 25 epochs. We train 3 different models with different random seeds and report the mean result of AP.
Select effective object priors. To assess the effectiveness of joint usage of region priors and instance priors, we compare the performance of JoinDet variants using different object priors: (a) box predictions P, (b) initial region priors P (c) region priors pr, (d) instance priors P (c) region priors pr instance priors P As shown Tab. using both region and instance priors (Tab, 5 outperforms DETReg by +3.0 AP, +1.5 AP with 5 epochs and 25 epochs finetuning, respectively, which verifies the effectiveness of joint usage of two priors generated by the model. Second, when region priors or instance priors are removed, we observe -0.6. -1.9 AP performance drop with 25 epochs finetuning, respectively, which suggests that region and instance priors can mutually benefit each other. Third, when no region priors are used to initialize the pretraining, directly using the box predictions of the model can only generate unreliable object priors for supervision, leading to significant performance drop of -13.4 AP with 25 epochs finetuning.

---

Table 4: Class agnostic object proposal evaluation on MS COCO val2017. Models are pretrained on ImagNet- 100 and COCO for 50 epochs. The top 100 proposals are evaluated. Compared with other methods, the higher Average Recall of JoinDet shows that it detects more objects without supervision from ground truth.
Method
Pretrain
AP
AP50
AP75
AR@1 1
AR@ 10
AR@ @ 100
Selective Search
0.2
0.5
0.1
0.2
1.5
10.9
UP-DETR [2]
ImageNet- 100
0.0
0.0
0.0
0.0
0.0
0.4
DETReg S
ImageNet-100
1.0
3.1
0.6
0.6
3.6
12.7
JoinDet
ImageNet-100
2.5
5.3
1.8
2.3
6.7
13.9
DETReg 3
COCO
1.3
3.0
1.0
0.6
3.5
11.7
JoinDet
COCO
3.0
7.4
2.3
2.3
8.8
17.4
Table 5: Pretrain JoinDet with different object priors. All experiments are pretrained using single- scale Deformable DETR COCO for 50 epochs and finetuned on VOC.
Method
Object priors R from
Pr from
Update
10 epochs
25 epochs
DETReg
Selective search
46.0
53.9
(a)
P
38.4
42.1
(b)
pr
Backbone
46.5
53,6
(C)
pr
Encoder
46.7
53.5
(d)
pi
Encoder
48.1
54.8
(e) pp 4 P
Encoder
49.0
55.4
Smooth the refinement of object priors. To demonstrate the effectiveness of the proposed Box Smooth Module, conduct other updating methods, including directly changing and updating by NMS (see Fig.5p. Directly changing, which updates the supervision without considering previous generated object priors, leads to fast change on the supervision signal. When updating by NMS, object priors are only updated when they have large IoU with current object priors, making relatively slow updating speed. As summarized in Tab. 6 noticeable performance drops observed without the Box Smooth Module, especially when the updating method is altered by directly changing, which verifies the effectiveness of the Box Smooth Module. The large performance gaps are due to the facts the: (1) Directly changing neglects useful object priors in previous supervision. which makes object-related object priors appear and disappear during pretraining, leading an unstable training process; (2) Although updating by NMS avoids the disappearance of reliable object priors previous supervision, fails to refine the box coordinates in slow and smooth way. The quick shifting of the supervision signal leads to undesirable learning instabilities.
Augmentations during pretraining, As our method mines object priors the self-attention map in the transformer encoder, most generated priors focus on big objects. To assess the importance of small object priors during pretraining, we ablate on different augmentations during pretraining. We use the initial supervision Ro, which bounds the highlighted regions in the eigen attention map generated by the backbone fixed supervision. All experiments are pretrained over 50 epochs on COCO with Deformable DETR and evaluated on PASCAL VOC with 100 epochs finetuning. It can be observed in Tab. that both scale jittering and copy & paste improve pretraining with large margin, indicating the importance of adding small object priors when mining supervision from the self-attentic map.
Visualization
To qualitatively evaluate the generated object priors our method, we visualize the evolution of the eigen attention map and generated object priors during pretraining, and compare them with that in supervised trained Deformable DETR in Fig.F First, generated object priors are progressively refined to objects in images during pretraining. Second, supervised by updating object priors, foregrounds in the eigen attention map can further highlighted (see Fig. a()). Third, with the help of progressive instance priors, the generated object priors can still become object-related and similar to ground truth

---

Generated boxes
Directly change
Update by NMS
Box Smooth Module
Previous reliable object priors R
Current reliable object priors R,,
Figure 5: Qualitative samples for different object priors updating methods. Directly changing updates the supervision without considering previous object priors. Updating by NMS reserves previous object priors which have small IoUs with current object priors. Box Smooth Modulc suppresses inaccurate previous object priors and slows down the shifting speed of object priors.
Table 6: VOC finetuning results of different object priors updating methods during pretraining. All meth- ods are pretrained on COCO without labels.
Table 7: VOC finetuning results of differ- ent augmentations during pretraining. All methods are pretrained COCO without labels.
Method
Change
10 epochs
25 epochs
Method
AP
Directly change
Fast
48.0
53.0
Using the initial supervision Ro
62.3
Update by NMS
Mid
48.3
54.5
+scale jittering [17
63.4
Box Smooth Module
Slow
49.0
55.4
+copy & paste 17
with small boxes
63.5
when it is hard for the eigen attention map be further refined, e.g. when foreground objects fill images (Fig. ((b)) or objects appear in a simple background ([c). In the former condition, our method highlights all objects while the supervised method focuses on labeled people. In the latter condition, foregrounds and backgrounds have been well divided at the beginning epochs.
6
Related Works
Unsupervised pretraining for backbone. Recent unsupervised pretraining methods, which rely on pretext tasks to learn visual representations 99. 21 18
have shown considerable performance on transfer learning tasks, outperforming their supervised counterparts. However, compared with considerable performance gains classification-relan ment on dense-prediction tasks (00 [6] is limited. To this end, a growing number of works explore pretext tasks for object detection and instance segmentation. DenseCL [46] and PixPro (53] contrast pixel feature on the same physical location under different views to learn pixel-level representations. DetCo (51) exploits supervision on features from different stages of the backbone and from global and local patches to learn consistent representations on image-level and patch-level. [2] proposes point-level region contrast, which enables the model to learn the point-level to help localization, and at the region-level level to help holistic object recognition. Despite the good performance, all these works only focus on pretraining the backbone of object detectors, neglecting the detector heads. When these methods are transferred to object detection, the detector heads are initialized from scratch and not benefit from pretraining, which limits their performance on object detection. contrast, our JoinDet, which utilizes object priors generated by the model itself as supervision, pretrains the entire model to promote detector learning.
49 33
55 43).
tasks, the improve-
Unsupervised pretraining for object detector. Pretraining the backbone with a pretext task for dense-prediction tasks leaves untrained detection heads which are also a core component when transferring to object detection 24]. Few works attempt to remedy this problem by pretraining the entire detector with various unsupervised pretext tasks. SoCo (88) utilizes selective search to generate object priors and perform contrastive learning object-level features from the detector head. UP-DETR [2] and DETReg 3 pretrain the detection heads of DETR by forcing them to predict object priors generated by randomly cropping and selective search, respectively. However, randomly cropping hardly provides any effective object prior, and selective search heuristic method that is time-consuming, independent from the pretraining process. In contrast to these methods,

---

Progressive refined & and M
Progressive refined
Progressive refined
Pretraining epochs
Our method
Locu
GT
Object priors R
The eigen atten
Object priors
The eigen atten
Object priors &
eigen atten J
(8)
(b)
(6)
Figure 6: Visualization results of generated object priors and the eigen attention map of JoinDet on COCO. In (a), we show that generated object priors and the eigen attention maps can be progressively and mutually refined. In (b) and (c), generated object priors are still refined when is hard for the eigen attention maps to be further refined during the pretraining process.
proposed JoinDet jointly generates object priors and learns detection, which can gradually update the object priors with learned and improved ones for better supervision during pretraining.
Attention unsupervised pretraining as supervision. NNCLR [15] and DINO (6] show that the attention maps of the visual transformer generate semantic segmentation masks even though the model is pretraimed without labels. This suggests that self-learned attention can provide effective supervision for dense-prees cense-prediction tasks. STEGO [9 utilizes off-the fff-the-shelf pretrained DINO to extract cross-image feature correspondence (cross-image attention) as supervision to distill segmentation features and train unsupervised segmentation model. Different from STEGO, our JoinDet exploits the self-attention maps in the transformer encoder to generate multiple object priors as supervision during training. We also show that the self-attention maps can be jointly refined during training generate progressive object priors for better supervision.
Conclusion
In this paper, we mine the information from the knowledge learned by the object detection model itself to pretrain the model without labels. We propose unsupervised object detection pretraining method which can jointly generate reliable object priors and learn to detect objects. The core is at generating region priors from the transformer encoder and instance priors from the transformer decoder to achieve reliable object priors for supervision. With careful design, the model can gradually learn better object localization capability producing better object priors to optimize itself. Our method shows considerable improvements in low-data regimes object detection and fast convergence full-data finetuning. However, there still exists large gap from supervised training class agnostic object proposal evaluation, which calls for further studies.
Acknowledgment
This work is supported by the Key R&D Project Zhejiang Province (No.2022C01056) and the National Natural Science Foundation of China (N..621278
10

---

References
[1] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In CVPR, 2014.
[2]
Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level region contrast for object detection pre-training. CVPR, 2022.
[3] Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region
priors for object detection, In CVPR, 2021.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020. [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In JCCV, 2021.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML,, 2020.
[8] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In NeurlPS, 2020.
[9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003 00:0299907. 04297, 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[11] Ming-Ming Cheng, Ziming Zhang, Wen- Yan Lin, and Philip Torr. Bing: Binarized normed
gradients for objectness estimation at 300fps. In CVPR, 2014.
[12] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers.
CVPR, 2021.
[13] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[14] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet:
Keypoint triplets for object detection. In ICCV, 2019.
[15] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisser- man. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
[16] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.
The pascal visual object classes (voc) challenge. IJCV, 2010.
[17] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
[18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020. [19] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint Ttti:220330841444 2022.
[20] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. TPAMI, 2022.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[24] Gabriel Huang, Issam Laradji, David Vazquez, Simon Lacoste-Julien, and Pau Rodriguez. A survey of self-supervised and few-shot object detection. arXiv preprint 10.11111, 2021.
[25] Leng Jiaxu, Chen Taiyue, Gao Xinbo, Yu Yongtao, Wang Ye, Gao Feng, and Wang Yue. A com- parative review of recent few-shot object detection algorithms. arXiv preprint arXiv:2111.00000) 2021.
11

---

[26]
Harold W Kuhn. The hungarian method for the assignment problem. NRL, 1955.
Detecting objects as paired keypoints. In ECCV, 2018. [28] Zhaowen Li, Yousong Zhu, Fan Yang, Wei Li, Chaoyang Zhao, Yingying Chen, Zhiyang Chen, Jiahao Xie, Liwei Wu, Rui Zhao, et al. Univip: A unified framework for self-supervised visual pre-training. arXiv preprini arXiv:2203. 06965, 2022.
[27] Hei Law and Jia Deng. Comnrnet:
[29] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense
object detection. In ICCV, 2017.
[30] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[31] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
[32] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV, 2021.
[33] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarinn repre- sentations. In CVPR, 2020.
[34] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines.
In ICML, 2010.
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[36] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and a loss for bounding box regression. In CVPR,2013
[37] Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. In CVPR, 2021.
[38] Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning. arXiv preprint arXiv:21 1450, 2021.
[39] Jianbo Shi and Jitendra Malik. Normalized euts and image segmentation. TPAMI, 2000.
[40] Roman Solovyev, Weimin Wang, and Tatiana Gabruseva. Weighted boxes fusion: Ensembling boxes from different object detection models. IMAGE VISION COMPUT, 2021.
[41] Peize Sun, Rufeng Zhang, Yi Jiang. Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR, 2021.
[42] Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020.
[43] Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, and Wanli Ouyang. Relative contrastive loss for unsupervised representation learning. In ECCV, 2022.
[44] Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Chenyu Wang, and Wanli Ouyang. Unifying vision unsupervised contrastive learning from a graph perspective. In ECCV, 2022.
[45] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 2013.
[46] Xinlong Wang, Rufeng Zhang. Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
[47] Yangtao Wang, Xi Shen, Shell Hu, Yuan Yuan, James Crowley, and Dominique Vaufreydaz. Self sulf-superiieed transformers for unsupervised object discovery using normalized cut. arXiv preprint arXiv:2202.115333 2022.
[48] Fangyun Wei, Yue Gao, Zhirong WU, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. In NewrIPS, 2021.
[49] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In CVPR, 2018
[50] Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In ICCV, 2021.
[51] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and
Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021.
[52] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. In NewrIPS, 2021.
12

---

[53] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao. Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. CVPR. 2021.
[54] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ICML, 2021.
[55] Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2021.
[56] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In ICCV, 2021.
[57] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for and object detection. In JCLR, 2020.
[58]
Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questionssanswers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 3.1 See supplementary materials for more details.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
A
Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in the appendix. This section will often be part of the supplemental material.
14


Reviews:
[{"review_id": "j5e9zzbwX2t", "reviewer": "Reviewer_fpzy", "summary": "This paper proposes an unsupervised object detection pre-training framework that can generate object priors and learn detector jointly. This work is inspired by DINO, the self-attention maps can generate region prior bounding boxes. Authors utilize Deformable DETR as their main framework. The proposed method achieves good results on low data-regime object detection on COCO and full-data finetuing on PASCAL VOC.", "questions": "My biggest concern comes from the experiments and the limited improvements under fine-tuning setting, see weaknesses above. ", "limitations": "Authors have discussed broader impacts in their supplementary material.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The motivation is good. Generating object priors by attention maps is an alternative of the popular selective search algorithm. However, a pre-trained backbone is important and necessary.\n\nWeaknesses:\n1) In Section 3, authors should claim which detection framework they used for pre-training in the begging of section 3, which will make the paper clearer. \n2) The proposed method highly rely on the pre-trained backbone, authors initialize the ResNet50 backbone of JoinDet with SwAV, did authors explore other unsupervised pre-training methods, e.g. BYOL, SimCLR?\n3) Authors should highlight the setting of the pre-training methods, supervised/SwAV only pre-trains backbone, while JoinDet pre-trains whole network. \n4) There are experiments conducted in the setting where pre-training is performed on COCO/ImageNet and finetuning is performed on PASCAL VOC (Table2). Why authors did not do experiments in the setting where pre-training is conducted on ImageNet and fine-tuning is performed on COCO. I hope to see this experiment in the rebuttal.\n5) Why authors only do experiments under low-data regime on COCO, is it consistent on PASCAL VOC?\n6) The improvement over DETReg is somewhat limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fSYYf8bI2pK", "reviewer": "Reviewer_EWLn", "summary": "This paper presents an unsupervised pre-training strategy for DETR-Like object detectors. The core idea is to generate object priors from the encoder to guide the self-supervised pre-training.\n\n\n", "questions": "See weakness.", "limitations": "Yes. The authors mentioned \"there still exists a large gap from supervised training on class agnostic object proposal evaluation, which calls for further studies.\"", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Using the encoder itself to generate object priors is a good idea.\n* The proposed method outperforms other methods designed for pre-training DETR-like object detectors.\n\nWeakness:\n* Missing important comparisons: For a rough categorization, the previous contrastive pre-training methods focused on (1)  **image classification** (e.g., Mocov2 and SimCLR) (2) **general dense prediction tasks**[1, 2, 3, 4] (such as object detection and segmentation) (3) **detr-like object detectors**. This paper belongs to (3), and only compared previous works of (1) and (3). But the detailed comparisons to the (2) are missing. It would be much better if the authors can provide deep discussions.\n* The advantages of pre-training is the generalization to various downstream tasks. I wonder if the representation learned in this method can be also used for other tasks such as segmentation.\n\n[1] Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning. CVPR 2021.\n[2] Dense Contrastive Learning for Self-Supervised Visual Pre-Training. CVPR 2021.\n[3] Region Similarity Representation Learning. ICCV 2021.\n[4] Deeply Unsupervised Patch Re-Identification for Pre-training Object Detectors. TPAMI 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GAvIsoJW7oA", "reviewer": "Reviewer_34gJ", "summary": "This submission proposes a pre-training approach for object detection transformer, it generates pseudo bounding box regression targets by leveraging the attention cues from the pre-trained backbone network. Specifically, both object proposals generated merely in encoder attention map and object instance prediction from decoder head are used for pseudo boxes. The method is evaluated on low-data and full data fine-tuning, as well as object proposal generation. ", "questions": "1. I actually have a question regarding the effectiveness of detection pre-training. For the **Full-data finetuning** experiment, the supervised training is compared as a baseline, however, would it be fair to add the pre-training time to the baseline? I guess when you extend the baseline training time, it would easily catch up with the detection pre-training methods, which would pose a concern on the effectiveness of detection pre-training for full-data finetuning.\n2. Although the work does not employ low-level unsupervised proposal methods like Selective Search, would it be more advantageous to incorporate these proposals which are generated from low-level cues? The intuition is that they may provide better or complementary coverage to some objects such as small ones, or others that have clear boundaries to the background.\n3. I'm interested in how the pre-trained object feature embedding loss affects the method, have you tried removing the feature embedding loss? Do you have any perspectives, e.g., it is a regularization that avoids the feature representation deviating too much from the pre-trained representation, or maybe it is the major driven force in the proposed object detection pre-training, as it enforces the region features to be close to an image-level pre-trained model like SwAW. Could you provide more ablation study on this part?\n4. Have you tried other unsupervised pretraining models as initialization and embedding loss, e.g., BYOL, do you have any ideas or perspectives on this aspect?", "limitations": "The submission does not discuss the limitations, it is encouraged to include more discussion on the limitation and possible future works and improvements that could be done. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths**\n- The submission is well-written and easy to follow. The math definitions and theories are clear and concise.\n- It is surprising that bounding box prediction merely supervised by the attention cues and self-supervising signal could achieve such superior object proposal results, which has not been done before.\n\n**Weaknesses and parts to be improved**\n- I listed some of my concerns in the questions, which I'm not very sure, the authors are encouraged to address my concerns and I would consider raising my score.\n- The approach is simple but it seems the technical contributions are sparse, the major components of attention map generation and region proposal generation are lent from prior work.\n- In section 2.4, the authors mention it _uses a pre-trained SWAV model to extract features_,  however, the details are not given. Specifically, is the regions cropped from the image and fed to the pre-trained SWAV backbone, or some feature cropping method is used, e.g., ROIalign.\n- In the introduction, the authors mention they are inspired by DINO and referred to the attention maps in the figures, however in the method section, it actually uses the Normalized Cuts algorithm to obtain the attention map. I believe it is better to rewrite the part in the introduction to be more clear and indicate in the figure captions how the attention map is obtained.\n\n**minor comments**\n- In line 84, can then **be** generated\n- In equation 3, FB is not defined\n- In equation 8, B and R should be defined properly, I guess R should be replaced with other characters as it is already used in equation 1.\n- In line 144, SWAV is not cited\n- In line 212, Detreg should be DETReg\n- In Tab.2 and Tab.3, citations are missing for the prior works.\n- In Tab.4, I suggest adding the IOU metric for calculating Recall, which is more clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "91chIjSkys8", "reviewer": "Reviewer_oHTr", "summary": "This paper focuses on unsupervised object detection pre-training using object priors. Specifically, two kinds of object priors are used in this paper. The first one is region prior which is generated from the encoder attention maps. The second one is instance prior which is obtained by selecting high-quality outputs from the detector decoder. Experiments on COCO and PASCAL VOC show that the proposed approach obtains better results than the previous unsupervised object detection pre-training approaches when only limited number of annotated data is available.", "questions": "From Table 3, when using the full COCO training set, only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg. Is there any explanation about why this happens and any thought on how to improve it?", "limitations": "Only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The proposed approach is interesting.\n- Promising results are obtained when limited number of annotated data is available.\n\n### Weaknesses\n- Novelty of region prior. There is already published work which uses attention maps to generate object bounding boxes to pre-train object detector [a]. The authors should discuss and compare with this paper.\n- Marginal improvement when more annotated data is available. From Table 3, when using the full COCO training set, only marginal improvement (0.1%) is obtained by the proposed approach comparing to the previous state-of-the-art object prior based pre-training approach DETReg.\n\n[a] DAP: Detection-Aware Pre-training with Weak Supervision", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept