Aligning Pretraining for Detection via Object-Level Contrastive Learning
Authors: Fangyun Wei
Yue Gao
Zhirong Wu
Han Hu
Stephen Lin
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: self-supervsied learning
object detection
pretraining

Abstract: Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning.  Such generality for transfer learning, however, sacrifices specificity if we are interested in a certain downstream task. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. In this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-level representations are introduced via selective search bounding boxes as object proposals; 2) the pretraining network architecture incorporates the same dedicated modules used in the detection pipeline (e.g. FPN); 3) the pretraining is equipped with object detection properties such as object-level translation invariance and scale invariance. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-art results for transfer performance on COCO detection using a Mask R-CNN framework. Code is available at https://github.com/hologerry/SoCo.

Aligning Pretraining for Detection via Object-Level Contrastive Learning
Fangyun Wei"
Yue Gao
Zhirong Wu
Han Hu
Stephen Lin
Microsoft Research Asia {fawe, yuegao, wuzhiron, hanhu,
Abstract
Image-level contrastive representation learning has proven be highly effecctvee as generic model for transfer learning. Such generality for transfer learning, however, sacrifices specificity if we are interested in certain downstream task. We argue that this could be sub-optimal and thus advocate design principle which encourages alignment between the self supe-supe pretext task and the downstream task. this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-level representations are introduced via selective search bounding boxes object proposals; 2) the pretraining network architecture incorporates the same dedicated modules used in the detection pipeline (e.g. FPN); the pretraining is equipped with object detection properties such as object-level translation invariance and scale invariance. Our method, called Selective Object COntrastive learning (SoCo), achieves state-of-the-ard results for transfer performance on COCO detection using a Mask R-CNN framework. Code is available https://tthubb.oommlllogrryy///
1 Introduction
Pretraining and finetuning has been the dominant paradigm training deep neural networks in computer vision. Downstream tasks usually leverage pretrained weights learned on large labeled datasets such as ImageNet [I] for initialization. As result, supervised ImageNet pretraining has been prevalent throughout the field. Recently, self-supervised pretraining 2 4 5 7 9 has achieved considerable progress and alleviated the dependency on labeled data. These methods aim to learn generic visual representations for various downstream tasks by means of image-level pretext tasks, such as instance discrimination. Some recent works [10 I1 12 [3] observe that the image-level representations sub-optimal for dense prediction tasks such object detection and semantic segmentation. A potential reason is that image-level pretraining may overfit to holistic representations and fail learn properties that are important outside of image classification.
The goal of this work is to develop self-supervised pretraining that is aligned to object detection. In object detection, bounding boxes widely adopted the representation for objects. Translation and scale invariance for object detection are reflected by the location and size of the bounding boxes. An obvious representation gap exists between image-level pretraining and the object-level bounding boxes of object detection.
Motivated by this, we present object-level self-supervised pretraining framework, called Selective Object COntrastive learning (SoCo), specifically for the downstream task of object detection. To introduce object-level representations into pretraining, SoCo utilizes off-the-shelf selective search [14) to generate object proposals. Different from prior image-level contrastive learning methods which
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

treat the whole image as an instance, SoCo treats each object proposal in the image as an independent instance. This enables us to design a new pretext task for learning object-level visual representations with properties that are compatible with object detection. Specifically, SoCo constructs object- views where the scales and locations of the same object instance are augmented. Contrastive learning follows to maximize the similarity of the object across augmented views.
The introduction of the object-level representation also allows us further bridge the gap in network architecture between pretraining and finetuning. Object detection often involves dedicated modules, e.g., feature pyramid network (FPN) [5], and special-purpose sub-networks, e.g., R-CNN head [I6] 17 In contrast to image-level contrastive learning methods where only feature backbones are pretrained and transferred, SoCo performs pretraining over all the network modules used in detectors. As result, all layers of the detectors can be well-initialized.
Experimentally, the proposed SoCo achieves state-of-the-ae transfer performance from ImageNet to COCO. Concretely, by using Mask R-CNN with an R50-FPN backbone, obtains 43.2 Aph / 38.4 Apm on COCO with a 1x schedule, which are +4.3 Aph /33.0 Ap better than the supervised pretraining baseline, and 44.3 AP 39.6 Apmk on COCO with a 2x schedule, which are +3.0 Apb +2.3 Ap better than the supervised pretraining baseline. When transferred to Mask R-CNN with an R50-C4 backbone, it achieves 40.9 Ap 35.3 Apmk and 42.0 Ap / 36.3 Apmk on the x and 2x schedule, respectively.
2 Related Work
Unsupervised feature learning has long history in training deep neural networks. Auto-encoders [8 and Deep Boltzmann Machines [99 learn layers of representations by reconstructing image pixels. Unsupervised pretraining is often used as a method for initialization, which is followed by supervised training on the same dataset such as handwritten digits for classification.
Recent advances in unsupervised learning, especially self-supervised learning, tend to formulate the problem in transfer learning scenario, where pretraining and finetuning are trained on different datasets with different purposes. Pretext tasks such as colorization (20). context prediction [21],
inpainting [22] and rotation prediction 22 force the network to learn semantic information order to solve the pretext tasks. Contrastive methods based on the instance discrimination pretext task 2 learn to map augmented views of the same instance to similar embeddings. Such approaches [2
5, 9 22, 26 have shown strong transfer ability for a number of downstream tasks, sometimes even outperforming the supervised counterpart by large margins
With new technical innovations such as learning clustering assignments large-scale contrastive learning has been successfully applied on non-curated data sets (27 28) While the linear evaluation result on ImageNet classification has improved significantly B3 the progress on transfer performance for dense prediction tasks has been limited. Due this, a growing number of works investigate pretraining specifically for object detection and semantic segmentation. The idea is to shift image- level representations to pixel-level or region-level representations. VADer [29], PixPro [10] and
DenseCI. [11 propose to learn pixel-level representations by matching point features of the same physical location under different views. InsLoc [12 learns to match region-level features from composited imagery. DetCon [ additionally uses the bottom-up segmentation of MCG [30) for supervising intra-segment pixel variations. UP-DETR [31] proposes a pretext task named random query patch detection to pretrain DETR detector. Self-EMD [32] explores self-supervised representation learning for object detection without using Imagenet images.
Our work is also motivated and inspired by object detection methods which advocate architectures and training schemes invariant to translation and scale transformations [15] B3). We find that incorporating them in the pretraining stage also benefits object detection, which has largely been overlooked in previous self-supervised learning works.
3 Method
We introduce a new contrastive learning method which maximizes the similarity of object-level fea- tures representing different augmentations of the same object. Meanwhile, we introduce architectrall alignment and important properties of object detection into pretraining when designing the object- level pretext task. We use an example where transfer learning is performed on Mas with a R50-FPN backbone to illustrate the core design principles. To further demonstrate the extensibility
2

---

Online Network
RolA
Head
wy RPN
Projection
Prediction
Crop Resio
Contrastive Loss
EMA Update
View
Aligh
Search
Solection
Projection
P5
1300234
Badebone w/ FPN
Ps
P2
Downsampling
Rlii
Target
Projection
Object-Level
Preprocessing
Feature Extractor
Scale-Aware Assignment
Contrastive Learning
Overview of SoCo. SoCo utilizes selective search to generate set of object proposals for each raw image. K proposals are randomly selected in each training step. We construct three views {Vi, V22 VaJ where the scales and locations of the same object are different. We adopt a backbone with FPN to encode image-level features and RoLAlign to extract object-level features. Object proposals are assigned to different pyramid levels according to their image areas. Contrastive learning performed at the object level to learn translation -nvolatioonnvvriannt invariant and scale- invariant representations. The target network is updated by an exponential moving average of the online network.
Figure I:
and flexibility of our method, we also apply SoCo on a R50-C4 structure. In this section, we first present an overview of the proposed SoCo Section 3.1 Then we describe the process of object proposal generation and view construction in Section Finally, the object-level contrastive learning and our design principles are introduced in Section
3.1 Overview
Figure displays an overview of SoCo. SoCo aims to align pretraining to object detection in two aspects: I) network architecture alignment between pretraining and object detection; 2) introducing central properties of detection. Concretely, besides pretraining a backbone as done in existing self- supervised contrastive learning methods, SoCo also pretrains all the network modules used in object detector, such as FPN and the head in the Mask R-CNN framework. As a result, all layers the detector be well- cell-initialized. Furthermore, SoCo strives to leam object-level representations which are not only meaningful, but also invariant to translation and scale. To achieve this, it encourages diversity of scales and locations of objects by constructing multiple augmented views and applying a scale-aware assignment strategy for different levels of a feature pyramid. Finally, object-level contrastive learning is applied to maximize the feature similarity of the same object across augmented views.
3.2 Data Preprocessing
we use selective search [4), an unsupervised object proposal generation algorithm which takes into account color similarity, texture similarity, size of region and fit between regions, to generate set of object proposals each of the raw images. We represent each object proposal as bounding box where (Ir, y) denotes the coordinates of the bounding box center, and and h are the corresponding width and height, respectively. We keep only the proposals that satisfy the following requirements: 1) 1/3 < w/h h < 3: 2) 0.3 < vwh/ / /WW < 0.8, where W and H denote width and height of the input image. The object proposal generation step is performed offline. In each training iteration, randomly select K proposals for each input image.
30 and Fast R-CNN 35 Object Proposal Generation. Inspired by R-CNN
{x, y, w, h).
View Construction. Three views, namely V1, V/2 and Vg, are used in SoCo. The input image is resized to 224 224 to obtain V,. Then we apply a random crop with a scale range of [0.5, 1.0] on Vi in generating V22 1/2 is then resized to the same size as Vi and object proposals outside of V22 are of these
dropped. Next, we downsample V22 to a fixed size (e.g. 112 x 112) to produce V₃- cases, the bounding boxes transformed according to the cropping and resizing of the RGB images

---

(see [ Data Preprocessing). Finally, each view is randomly and independently augmented. We adopt the augmentation pipeline of BYOL [3] but discard the random crop augmentation since spatial transformation is already applied on all three views. Notice that the scale and location of the same object proposal are different across the augmented views, which enables the model to learn Translaion-invariian and scale-invariant object-level representations.
Box Jitter. To further encourage variance of scales and locations of object proposals across views, we adopt a box jitter strategy on the generated proposals as an object-level data augmentation. Specifically, given an object proposal b = {z. y, w. h), we randomly generate jittered box b = w; 2) i y h; 3) i where r E 121.1, 0.1]. The box jitter is randomly applied on each proposal with a probability of 0.5.
{ ú, æi, h as follows: 1) å =
r w; 4) h h + r h,
3.3
Object-Level Contrastive Learning
The goal of SoCo is to align pretraining to object detection. Here, we use the representative framework Mask R-CNN 17 with feature pyramid network (FPN) [5] to demonstrate our key design principles. The alignment mainly involves aligning the pretraining architecture with that of object detection and integrating important object detection properties such as object-level translation invariance and scale invariance into the pretraining.
Aligning Pretraining Architecture to Object Detection. Following Mask R-CNN, we use backbone with FPN as the image-level feature extractor S4. We denote the output of FPN as {P2, Ps. P3} with stride of (4. 8, 16, 32). Here. we do not use Ps due to its low resolution. With the bounding box representation b, RoIAlign [17) is applied to extract the foreground feature from the corresponding scale level. For further architectural alignment, we additionally introduce an R-CNN head f into pretraining. The object-level feature representation h of bounding box b is extracted from an image view V as:
h (=olAligg (VV))
(1)
SoCo uses two neural networks to learn, namely an online network and a target network. The online network and the target network share the same architecture but with different of weights. Concretely, the target network weights J/d 1e are the exponential moving average (EMA) with the momentum coefficient 7 of the online parameters fg and f/" Denote the set of object proposals {b,} considered in the image. Let h be the object-level representation of proposal b, in view V/1 and h4, h" be the representation of by in view V2, Vg. They are extracted using the online network and the target network respectively,
h = 10t (RoIAlign(Fd
(2)
h/. f (RolAlign (V2), 6)),, =-= (RolAlign( f (Va), 6.)).
(3)
We follow BYOL 3 for learning contrastive representations. The online network is appended with projector go, and a predictor go for obtaining latent embeddings. Both go and qo are two-layer MLPs. The target network is only appended with the projector gę for avoiding trivial solutions. We use u; of and of" to denote the latent embeddings of object-level representations (hs, h4s hi' J, respectively:
V = qo (go (h:),,
of = (h())), of = ge (h()).
(4)
The contrastive loss for the i-th object proposal is defined as: (us v()
(Us, v!")
C = 2 ||ull22 |1//12 ||ulll 11
(5)
Then we can formulate the overall loss function for each image as:
Σ c..
(6)
where K is the number of object proposals. We symmetrize the loss L in boby separately feeding Vi to the target network and (V2, V/} to the online network to compute E. At each training iteration, we perform stochastic optimization step to minimize SSoCo L C.

---

Scale-Aware Assignment. Mask R-CNN with FPN uses the IoU between anchors and ground-truth boxes determine positive samples. It defines anchors to have areas of {322, 64", 128?, 256²) pixels on {Pa, Pa. Pa, Ps), respectively, which means ground-truth boxes of scale within range are assigned to a specific pyramid level. Inspired by this, we propose scale-aware as- signment strategy, which greatly encourages the pretraining model to learn object-level scale- invariant representations. Concretely, we assign object proposals of area within a range of {0 - 48" 49 - 962, 972 1922, 1932 224"] pixels to {12, Ps, Pa, Po). respectively. Notice that the maximum proposal size is 224 x 224 since all of the views are resized to a fixed resolution smaller than 224. The advantage is that the same object proposals at different scales are encouraged to learn consistent representations through contrastive learning. As a result, SoCo learns object-level scale-invariant visual representations, which is important for object detection.
Introducing Properties of Detection to Pretraining. Here, we discuss how SoCo promotes impor- tant properties of object detection in the pretraining. Object detection uses tight bounding boxes to represent objects. To introduce object-level representations, SoCo generates object proposals by selective search. Translation invariance and scale invariance at the object level are regarded as the most important properties for object detection, i.e., feature representations of objects belonging to same category should be insensitive to scale and location. Recall that V/2 is randomly cropped patch of V1. Random cropping introduces box shift and thus contrastive learning between V12 and V/2 encourages the pretraining model to learn location invariant representations. V3 is generated by downsampling V2, which results in a scale augmentation of object proposals. With our scale- assignment strategy, the contrastive loss between V1 and V33 guides the pretraining towards learning scale-invariant visual representations.
Extending to Other Object Detectors. SoCo can be easily extended to align to other detectors besides Mask R-CNN with FPN. Here, we apply SoCo to Mask R-CNN with a C4 structure, which is a popular non-FPN detection framework. The modification is three-fold: 1) for all object proposals, RolAlign is performed on C4; 2) the R-CNN head replaced by the entire 5-th residual block; 3) view V/3 is discarded and the remaining V12 and V22 are kept for object-level contrastive learning. Experiments and comparisons with state-of-the of-the-art art methods in Section 4.3 demonstrate the extensibility and flexibility of SoCo.
4 Experiments
4.1 Pretraining Settings
Architecture.
Through the introduction of object proposals, the architectural discrepancy is reduced between pretraining and downstream detection finetuning. Mask R-CNN [17) is commonly adopted framework to evaluate transfer performance. To demonstrate the extensibility and flexibility of SoCo, we provide details of SoCo alignment for the detection architectures R50-FPN and R50- C4. SoCo-R50-FM
ResNet-50 (36) with FPN (15) is used as the image-level feature encoder. RoLAlign [17 is then used to extract Rol features on feature maps {12,, P3, Ps, P3) with stride of {4, 8, 16, 32). According to the image areas of object proposals, each Rol feature is then transformed to an object-level representation by the head network as in Mask R-CNN. SoCo-R50- C4: on the standard ResNet-50 architecture, we insert the Rol operation on the output of the 4-th residual block. The entire 5-th residual block is treated as the head network to encode object-level features. Both the projection network and prediction network are 2-layer MLPs which consist of a linear layer with output size 4096 followed by batch normalization 37 rectified linear units (ReLU) 38,, and a final linear layer with output dimension 256.
Dataset. We adopt the widely used ImageNet which consists of 1.28 million images for self-supervised pretraining.
Data Augmentation. Once all views are constructed, we employ the data augmentation pipeline of BYOL B. Specifically, we apply random horizontal flip, color distortion, Gaussian blur, grayscaling. and the solarization operation. We remove the random crop augmentation since spatial transformations have already been applied on all views.
Optimization. We use a 100-epoch training schedule in all the ablation studies and report the results of 100-epochs and 400-epochs in the comparisons with state-of-the-art methods. We use the LARS optimizer [39] with cosine decay learning rate schedule [40] and warm-up period of 10 epochs.
5

---

Table 1: Comparison with satee-of-the-ar art methods COCO by using Mask R-CNN with R50-FPN.
Schedule APm
Schedule APm
Methods
Epoch
Ap
AP
7P
AP
AP
APth
AP
AP
AP
AP
Seratch
31.0
49.5
33.2
28.5
46.8
30.4
38.4
57.5
42.0
34.7
54.8
37.2
Supervised
90
38.9
59.6
42.7
35.4
56.5
38.1
41.3
61.3
45.0
37.3
58.3
40.3
MoCo +
200
38.5
58.9
42.0
35.1
55.9
37.7
40.8
61.6
44.7
36.9
58.4
39.7
MoCo v2
200
36.4
57.2
41.7
40.5
InfoMin 6
200
40.6
60.6
44.6
36.7
57.7
39.4
42.5
62.7
46.8
38.4
59.7
41.4
BYOL
300
40.4
61.6
44.1
37.2
58.8
39.8
42.3
62.6
46.2
38.3
59.6
41.1
SwAV 7
400
46.3
60.0
41.0
ReSim-FPN
40.3
PixPro 10
400
41.4
61.6
45,4
InsLoc 12
400
42.0
62.3
45.8
37.6
59.0
40.5
43.3
63.6
47.3
38.8
60.9
41.7
DenseCL.
40.3
58.9
40.1
DetCons
1000
41.8
37.4
42.9
38.1
DetCon,
1000
42.7
38.2
43.4
38.7
SoCo
100
42.3
62.5
46.5
37.6
59.1
40.5
43.2
63.3
47.3
38.8
60.6
41.9
SoCo
38.2
61.3
SoCo*
400
43.2
63.5
47.4
38.4
60.2
41.4
44.3
64.6
48.9
39.6
61.8
42.5
The base learning rate |raas is set to 1.0 and scaled linearly [1]] with the batch size (Ir Iraase x BatchSize BatchSize/2566. The weight decay is 1.0 e-5 The total batch size is set to 2048 over 16 Nvidia V100 GPUs. For the update the target network, following [3], the momentum coefficient starts from 0.99 and is increased 1 during training. Synchronized batch normalization is enabled.
4.2 Transfer Learning Settings
COCO 42 and Pascal VOC 43 datasets are used for transfer learning. Detectron2 44 used as the code base.
COCO Object Detection and Instance Segmentation. We the COCO train2017 which contains 1118 images with bounding box and instance segmentation annotations 80 object categories. Transfer performance evaluated on the COCO va12017 set. We adopt the Mask R-CNN detector [17 with the R50-FPN and the R50-C4 backbones. We report Apho AP 50 and AP for object detection, and A AP and AP for instance segmentation. We report the results under the COCO x and 2x schedules in comparisons with state-of-the-art methods. All ablation studies are conducted on the COCO x schedule.
Pascal VOC Object Detection. We use the Pascal VOC trainval07+11 set which contains ~ 16..5 images with bounding box annotations 20 object categories as the training set. Transfer performance is evaluated the Pascal VOC test2007 set. We report the results of Apbb, APt and AP for object detection.
Optimization. All the pretrained weights except for projection and prediction are loaded into the object detection network for transfer. Following (5) synchronized batch normalization is used across all layers including the newly initialized batch normalization layers in finetuning. For both the COCO and Pascal VOC datasets, we finetune with stochastic gradient descent and batch size of 16 split across 8 GPUs. For COCO transfer, we use weight decay of 2.5 x e- and a base learning rate of 0.02 that increases linearly for the first 1000 iterations and drops twice by a factor 10, after & and of the total training time. For Pascal VOC transfer, the weight decay is 1.0 e and the base learning rate is set to 0.02 and divided by 10 at and l1 of the total training time.
12
4.3 Comparison with State-of-the-Art Methods
Mask R-CNN with R50-FPN on COCO. Table I shows the transfer results for Mask R-CNN with R50-FPN backbone. We compare SoCo with the state-of of-theo-----rtt unsupervised pretraining methods on the COCO x and 2x schedules. We report our results under 100 epochs and 400 epochs of pretraining. On the 100-epoch training schedule, SoCo achieves 42.3 Apob 37.6 Apmk and 43.2 Apbb / 38.8 Apmk on the 1x and 2x schedules, respectively. Pretrained for 400 epochs, SoCo outperforms all previous pretraining methods designed for either image classification object detection, achieving 43.0 APbb 38.2 AP for the 1x schedule and 44.0 AP / 39.0 Apmk for the 2x

---

Table 2: Comparison with state-of-the methods COCO using Mask R-CNN with R50-C4.
Methods
Epoch
Aplo
AP
AP
1x
Schedule Apmk
AP
Ap
AP
AP
2
Schedule Ap
AP
AP7S
Scratch
26.4
44.0
27.8
29.3
46.9
30.8
35.6
54.6
38.2
31.4
51.5
33.5
Supervised
38.2
58.2
41.2
33.3
54.7
35.2
40.0
59.9
43.1
34.7
56.5
36.9
MoCo
200
38.5
58.3
41.6
33.6
54.8
35.6
40.7
60.5
44.1
35.4
57.3
37.6
SimCI.R 9
200
39.6
59.1
42.9
34.6
55.9
37.1
MoCo v2
800
39.3
58.9
42.5
34.3
55.7
36.5
41.2
60.9
44.6
35.8
57.7
38.2
InfoMin [6]
200
39.0
58.5
42.0
34.1
55.2
36.3
41.3
61.2
45.0
36.0
57.9
38.3
BYOL 3
300
40.3
60.5
43.9
35.1
56.8
37.3
SwAV 7
39.6
60.1
42.9
34.7
56.6
36.6
SimSiam
200
39.2
59.3
42.1
34.4
56.0
36.7
PixPro 10
400
40.5
59.8
44.0
InsLoc
12
400
39.8
59.6
42.9
34.7
56.3
36.9
41.8
61.6
45.4
36.3
58.2
38.8
SoCo
100
40.4
60.4
43.7
34.9
56.8
37.0
41.1
61.0
44.4
35.6
57.5
38.0
SoCo
400
40.9
60.9
44.3
35.3
57.5
37.3
42.0
61.8
45.6
36.3
58,5
38.8
Table Comparison with state-of-the-arr methods on Pascal VOC. Faster R-CNN with R50-C4 is adopted. Table is split to two sub-tables.
(a) Sub-table 1.
(b) Sub-table 2.
Methods
Epoch
Aphb
APh
APh
Methods
Epoch
Aphh
APh
APh
Scratch
33.8
60.2
33.1
MoCo 4
200
55.9
81.5
62.6
Supervised
90
53.5
81.3
58.8
SimCLR 9
1000
56,3
81.9
62.5
MoCo v2
800
57.6
82.7
64.4
ReSim-C4
45
200
58.7
83.1
66.3
InfoMin 6
200
57.6
82.7
64.6
PixPro 10
400
60.2
83.8
67.7
BYOL 3
300
$1.9
81.0
56.5
InsLoc 12
400
58.4
83.0
65.3
SwAV 7
400
45.1
77.4
46.5
DenseCI.
58.7
82.8
65.2
SimSiam X
200
57.0
82.4
63.7
SoCo
100
59.1
83.4
65.6
ReSim -PN
200
59.2
82.9
65.9
SoCo
59.7
83.8
66.8
schedule. Moreover, we propose an enhanced version named SoCo*, which constructs an additional view V₂ for pretraining. Similar the construction step of V2, Va randomly cropped patch of
Vi. We resize Va to 192 x 192 and perform the same forward process as for 1/2 and Vg. Compared with SoCo, SoCo* further boosts the performance and obtains an improvement of +0.2 Aph +0.2 Apmk on the 1 x schedule, achieving 43.2 Apbs / 38.4 Apmk, and +0.3 Apab, +0.6 A on the 2x schedule, achieving 44.3 Apto 39.6 A ppnk respectively.
Mask R-CNN with R50-C4 on COCO. To demonstrate the extensibility and flexibility of SoCo, also evaluate transfer performance using Mask R-CNN with R50-C4 backbone the COCO benchmark. We report the results SoCo under 100 epochs and 400 epochs. Table 2 compares the proposed method previous kate-of-theean methods. Without bells and whistles, SoCo obtains state-of-the-art performance, achieving 40.9 Apbb 35.3 Ap and 42.0 Apbb / 36.3 Apmk on the COCO 1 x and 2 schedules, respectively.
Faster R-CNN with R50-C4 on Pascal VOC. We also evaluate the transfer ability of SoCo on Pascal VOC benchmark. Following (5). we adopt Faster R-CNN with R50-C4 backbone for transfer learning. Table 3 shows the comparison. SoCo obtains an improvement +6.2 Apt against supervised pretraining baseline, achieving 59.7 Apbo for VOC object detection.
4.4 Ablation Study
To further understand the advantages of SoCo, we conduct series of ablation studies that examine the effectiveness object-level contrastive learning, the effects of alignment between pretraining and detection, and different hyper-panne parameters. For all ablation studies, we use Mask R-CNN with R50-FPN backbone for transfer learning and adopt 100-epoch SoCo pretraining schedule. Transfer performance is evaluated under the COCO 1 x schedule. For the ablation of each hyper-parameter or component, we fix all other typer-parameters to the following default settings: view V/ with 112 x 112 resolution, batch size of 2048, momentum coefficient 0.99, proposal number K 4.
7

---

of Table Ablation study on the effectiveness of aligning pretraining to object detection.
Whole
Selective
Scale-aware Assignment
Box
Multi View
FPN
Head
Ap
Apmk
Image
Search
Jitter
38.1
34.4
40.6 (+2.5)
36.8 (+2.4)
40.2 (42.1)
36.2 (+1.8)
41.2 (+3.1)
37.0 (+2.6)
41.6 (43.5)
37.3 (+2.9)
41.7 (+3.6)
37.5 (+3.1)
42.3 (+4.2)
37.6 (+3.2)
Table 5: Ablation studies on hyper-parameters for the proposed SoCo method.
(a) Study on image size of view Vi3.
(c) Study on proposal generation and proposal number K.
Image Size
Apbb
A ppmk
Selective Search
Random
Apbb
Apmk
96
42.1
37.7
41.6
37.3
112
42.3
37.6
42.3
37.6
128
42.1
37.7
8
41.6
37.4
160
42.0
37.6
16
41.2
37.0
192
42.2
37.8
41.4
36.9
NaN
NaN
8
NaN
NaN
(b) Study on batch size.
(d) Study on momentum coefficient T.
Batch Size
Aphh
Apmk
Apbb
Apmk
512
41.7
37.6
1024
41.9
37.6
0.98
35.0
31.7
2048
42.3
37.6
0.99
42.3
37.6
4096
41.4
37.3
0.993
41.8
37.6
box jitter and scale-aware assignment used, FPN and R-CNN head pretrained and transferred, and selective search is used as the proposal generator.
Effectiveness of Aligning Pretraining to Object Detection. We ablate each component of SoCo step by step to demonstrate the the effectiveness of aligning pretraining object detection. Table 4 reports the studies. The baseline treats the whole image detection properties or architecture alignment. obtains 38.1 Aph and 34.4 APmk, Selective search introduces object-level representations. By leveraging generated object proposals and conducting contrastive learning at the object level, our method obtains an improvement of +2.5 Apbb and +2.4 Apmk Next, further minimize the architectural discrepancy between pretraining and object detection pipeline by introducing FPN and an R-CNN head into the pretraining architecture. We observe that only introducing FPN into pretraining slightly hurt the performance. In contrast, including both FPN and the R-CNN head improves the transfer performance to 41.2 Apbb 37.0 Apmk which verifies the effectiveness of architectural alignment between self-supervised pretraining and downstream tasks. On top of architectural alignment, we further leverage scale- aware assignment which encourages scale-invariant object-level visual representations, and an improvement of +3.5 Apbb / +2.9 Apink against the baseline is found. The box jitter strategy slightly elevates performance. Finally, multiple views (Vg) further directs SoCo towards learning cale-invariann and translation- invariant representations, and our method achieves 42.3 Aph and 37.6 Apmk
instance, without considering any
Ablation Study Hyper- Parametess, Table exammnnes sensitivity the SoCo.
Table 5)a) ablates the resolution of view V/. SoCo is found to be insensitive to the image size of V/. We use 112 112 the default resolution due its slightly better transfer performance and low training computation cost.

---

Table 6: Transfer Learning LVIS dataset using Mask R-CNN with R50-FPN.
x Schedule
2x
Schedule
Method
Epoch
Apl
APh
AP
Apmi
AP
AP7
Ap
Aphi
30
AP
APm
AP
AP7S
Supervised
90
20.4
32.9
21.7
19.4
30.6
20.5
23.4
36.9
24.9
22.3
34.7
23.5
SoCo*
400
26.3
41.2
27.8
25.0
38.5
26.8
28.3
43.5
30.7
26.9
41.1
28.7
Table Transfer learning on RetinaNet and FCOS.
(a) Transfer RetinaNet.
(b) Transfer FCOS.
Method
Epoch
Ap
AP 50
AP
Method
Epoch
Apbo
AP
AP
75
50
Supervised
90
36.3
55.3
38.6
Supervised
90
36.6
56.0
38.8
SoCo* 400
38.3
57.2
41.2
SoCo*
400
37.4
56.3
39.9
Table 5 b) ablates the training batch size of SoCo. It can be seen that larger smaller batch sizes hurt the performance. We use a batch size of 2048 by default.
Table (ic) ablates the effectiveness of object proposal generation strategies. To demonstrate the superiority of meaningful proposals generated by selective search, we randomly generate K bounding boxes in each of the training images to replace the selective search proposals (namely Random in Table Ste).. SoCo can still yield satisfactory results using a single random bounding box learn object-level representations, which is not surprising, since the RolAlign operation performed on the randomly generated bounding boxes also introduces object-level representation learning. This fact indirectly demonstrates that downstream object detectors can benefit from self-supervised pretraining that involves object-level representations. However, the result is still slightly worse than the case where only one object proposal generated by selective search is used in the pretraining. For the cases where we randomly generate 4 and 8 bounding boxes, the noisy and meaningless proposals cause the pretraining diverge. From the table, we also find that applying more object proposals (K 8 and K = 16) generated by selective search can harmful. One potential reason is that most of the images in the ImageNet dataset only contain a few objects, larger K may create duplicated and redundant proposals.
Table d) ablates the momentum coefficient r of the exponential moving average, and T 0.99 yields the best performance.
4.5 More Experiments
Transfer Learning on LVIS Dataset. In addition COCO and PASCAL VOC object detection, we also consider the challenging LVIS v1 dataset [46] to demonstrate the effectiveness and generality of our approach. The LVIS v1 detection benchmark contains 1203 categories a long-tailed distribution with few training samples. It is considered to be more challenging than the COCO benchmark. We use Mask R-CNN with R50-FPN backbone and follow the standard LVIS x and 2x training schedules. We do not any training or post-procedd post-prcesssing tricks. Table 6 shows the results. We can see that our method achieves +5.9 A556 Apmk and +4.9 APbb/+4.6 A improvements under the LVIS 1 and 2x training schedules, respectively.
Transfer Learning on Different Object Detectors. addition to two-stage detectors, we further conduct experiments on RetinaNet [47] and FCOS 48), which are representative works for single- stage detectors and anchor-free detectors. We transfer the weights of backbone and FPN layers learned by SoCo* to RetinaNet and FCOS architectures and follow the standard COCO 1 x schedule. We use MMDetection [49] as code base and default optimization configs are adopted. Table 7 shows the comparison.
Evaluation on Mini COCO. Transfer to the full COCO dataset may be of limited significance due to the extensive supervision available from its large-scale annotated training data. To further demonstrate the generalization ability of SoCo, we conduct an experiment on mini version of the COCO dataset, named Mini COCO. Concretely, we randomly select 5% or 10% of the training data from COCO train2017 to form Mini COCO benchmarks. Mask-RCNN with R50-FPN backbone adopted to evaluate the transfer performance the COCO 1x schedule. COCO va12017 is still used

---

Table 8: Results Mini COCO 1 schedule. Mask R-CNN with R50-FPN backbone is adopted.
Mini COCO (5%)
Mini COCO (10%)
Methods
Epoch
Ap
AP
AP
Apmn
AP
AP
Ap
AP
AP
Apm
AP
AP
Supervised
90
19.4
36.6
18.6
18.3
33.5
17.9
24.7
43.1
25.3
22.9
40.0
23.4
SoCo
100
24.6
42.2
25.6
22.1
38.8
22.4
29.2
47.7
31.1
26.1
44.4
27.0
SoCo
400
26.0
43.2
27.5
22.9
40.0
23.4
30.4
48.6
32.5
26.8
45.2
27.9
SoCo*
400
26.8
45.0
28.3
23.8
41.4
24.2
31.1
49.9
33.4
27.6
46.4
28.9
Table 9: Pretraining on non-object-centrie dataset.
Pretraining dataset
Epoch
Apbb
Aph 5u
Aphb 75
Apmk
Apmk
AP
75
ImageNet
100
42.3
62.5
46.5
37.6
59.1
40.5
ImageNet-subsee
100
36.9
56.2
40.1
33.2
53.0
35.6
COCO train set unlabeled
100
37.3
56.5
40.6
33.5
53.4
36.0
COCO train set unlabeled set
530
40.6
61.1
44.4
36.4
58.1
38.7
as the evaluation set. The other settings remain unchanged. Table summarizes the results. Compared with the supervised pretraining baseline which achieves 19.4 AP 18.3 APmk and 24.7 Aph 22.9 Apmkk on the 5% and 10% Mini COCO benchmarks, SoCo obtains large improvements of +6.6 Apbb / +4.6 Apmk and +5.7 Apb / +3.9 Apmk, respectively. SoCo* further boosts performance with improvements of +7.4 Aph +5.5 APmk and +6.4 AP3b +4.7 Apmk over the supervised pretraining baseline.
Pretraining on Non-object-centric Dataset. We use COCO training set and unlabeled set as pretraining data. Since ImageNet is ~5.3 times larger than COCO, we pretrain our method on COCO 100 epochs and prolonged 530 epochs respectively. Furthermore, we also generate dataset. named ImageNet-Subbss which contains the same number of images of COCO to verify the impact of different pretraining datasets. Table shows the results. Compared with the ImageNet pretrained model, the COCO pretrained model under 530 epochs is lower by 1.7 AP, possibly because the scale of COCO is smaller than ImageNet. Compared with the ImageNet-Subse pretrained model, the COCO pretrained model under 100 epochs is +0.4 AP higher, which demonstrates the importance of domain alignment between the pretraining dataset and detection dataset.
Longer Finetuning. We transfer SoCo* to Mask R-CNN with R50-FPN under the COCO 4x schedule. Table TO shows the results. Our model continues to improve the performance with the 4x finetuning schedule, surpassing the supervised baseline by +2.6 Apbb/ +1.9 Apmk
Table 10: Finetuning COCO 4x schedule using Mask R-CNN with R50-FPN.
Method
Epoch
Aphb
Aphh
APhh
Apmk
Apmk
APhh
Supervised
90
41.9
61.5
45.4
37.7
58.8
40.5
SoCo* 400
44.5
64.2
48.8
39.6
61.5
42.4
Conclusion
In this paper, we propose a novel object-level self-supervised pretraining method named Selective Object COntrastive learning (SoCo), which aims to align pretraining to object detection. Different from prior image-level contrastive learning methods which treat the whole image as an instance, SoCo treats each object proposal generated by the selective search algorithm an independent instance, enabling SoCo to learn object-level visual representations. Further alignment obtained in two ways. One is through network alignment between pretraining and downstream object detection, such that all layers of the detectors can be well-initialized. The other by accounting for important properties of object detection such as object-level translation invariance and scale invariance. SoCo achieves of-te-of-the-ar transfer performance on COCO detection using a Mask R-CNN detector. Experiments on two-stage and single-stage detectors demonstrate the generality and extensibility of SoCo.
10

---

References
[I] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[2] Zhirong Wu, Yuanjun Xiong, Yu Stella, and Dahua Lin. Unsupervised feature learning via non con-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[3] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
[4] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297 2020.
[6] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint WiXi::200600882,, 2020.
[8] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 2020.
[9] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[10] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning, arXiv preprint arXiv:2011.10043: 2020.
[11] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. arXiv preprint arX1v:2011.09157, 2020.
[12] Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self- supervised detection pretraining. arXiv preprint 1rXiv:2102.08318, 2021.
[13] Olivier J Hénaff, Skanda Koppula, Jean- Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. arXiv preprint arXiv:2103.10957 2021.
[14] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104(2):154-171, 2013.
[15] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[16] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 550.001997 2015.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.

---

[18] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
[19] Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statisticx, pages 448-455. PMLR, 2009.
[20] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6874-6883, 2017.
[21] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pages 1422-1430, 2015.
[22] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544 2016.
[23] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0772. 2018.
[24] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Interna- PMLR, 2020.
tional Conference on Machine Learning, pages
[25] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03225 2021.
[26] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
[27] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3959-29668 2019.
[28] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Pai, Man- nat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features the wild. arXiv preprint arXiv:2103.01900 2021.
[29] Pedro O Pinheiro, Amjad Almahairi, Ryan Y Benmaleck, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. arXiv preprint arXiv:201 J. 1.05499, 2020.
[30] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 328-335, 2014.
[31] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1601-1610, 2021.
[32] Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprint 2xXi:2001..16677 2020.
[33] Bharat Singh and Larry S. Davis. An analysis of scale invariance in object detection snip, 2018.
[34] Ross Girshick. Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
[35] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
12

---

[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[37] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR. 2015.
[38] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
[39] Yang You, Igor Gitman, and Boris Ginsburg, Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv: 708.03883. 6:12, 2017.
[40] llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:6608.080
2016.
[41] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint (rXiv:176.002677 2017.
[42] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755 Springer, 2014.
[43] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[44] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019.
https://gi http://tttt..ooooeoooooorereeeee
[45] Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. arXiv preprint 103.v:2/03./29000 2021.
[46] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5356-5364, 2019.
[47] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[48] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9636, 2019.
[49] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprini arXiv:1906.097 077190060,7,,,, 2019.
13


Reviews:
[{"review_id": "vYqP10qaFB", "reviewer": "Reviewer_SV22", "summary": "This submission addresses the problem of constructing a good initialized pretrained network for the object detection task. Compared to previous work, the authors argue that pretraining should be adapted to object detection by focusing on object-level features learning. Instead of pretraining on ImageNet, the authors pretrain the model by designing an object level contrastive learning framework. Experiments on COCO and PASCAL show that the proposed method achieves higher performance. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n+ The motivation and novelty of the submission is clearly explained. The Figure 1 is almost self-explainable for the framework.\n\n+ The details of implementation are provided in the submission. Besides, the authors also provide reproducible code in the supplementary. \n\n+ The authors provide ablation study for different aspects, including hyper-parameters, different datasets to verify the proposed method, which is much appreciated.\n\n### Weakness\n- Limited application scope: As claimed in the title, this proposed method is used for \"Pretraining for Detection\". This means this method should be generalized for all kinds of detectors for object detection. However, the major novelty is proposing a pretraining framework which focuses on object-level feature learning by using some unsupervised proposal generators. This method may perform well on two-stage detectors (also verified in the experiments). However, for other types of detectors, such as one-stage detectors and anchor-free detectors, there are no experiments to verify. \n\n- Ablation studies of the proposal generator: There are two aspects which may be needed to make the method more convincing.\n  * In this submission, the authors only adopt Selective Search which is an old unsupervised proposal generator. The influence of different selections of proposal generators is not verified. If the proposed method is not sensitive to such selection, the value of the submission can further increase.\n  * According to Figure 2 in Section A.3 (Supplementary), the distribution of proposal size highly concentrates on the size of 20-60. However, in line 168-169, the authors assign different proposals to different layers of FPN according to their area. This may influence the learning effects of lateral layers ($P_3, P_4, P_5$), because #proposals is small. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The title may be too broad. It would be better to limit to two-stage detector, or the authors need to verify such method also works well on one-stage or anchor-free detectors. \n* To show the proposed method is insensitive to the proposal generator's selection, it would be better to choose different proposal generator other than Selective Search and do parallel experiments\n* The authors may need to address the imbalance of the proposals for different layers' learning problem as mentioned in the weakness part above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "jCDyLhOQqa", "reviewer": "Reviewer_SXPo", "summary": "This paper proposes SoCo, a method for self-supervised representation learning for downstream object detection and instance segmentation tasks.\n\nThey argue for stronger alignment between downstream tasks and self-supervised pre-training. They demonstrate this in the case of object detection by (a) developing a representation learning scheme that encourages object level scale and translation invariance and (b) pre-training several components of the downstream detector such as FPN and R-CNN head unlike prior work which mostly just trains the ResNet backbone.\n\nThe former property will help their model detect objects at several locations and scales as is needed for object detection.  The latter property helps them get stronger alignment between the upstream and downstream setups. To this end they propose \"scale aware assignment\". The ideas is that, representations for small objects are taken from finer FPN pyramid levels and for bigger objects are taken from coarser FPN pyramid levels. This matches what happens in the actual FPN where anchors at finer pyramid levels are smaller and at coarser pyramid levels are bigger.\n\nThe overall recipe is as follows: Selective search, a hand-crafted region proposal technique, is used to suggest boxes on all images. K of these boxes are randomly selected in each training step per image and three views of that image are constructed using random resized crop and resize operations. FPN features are extracted using RoiAlign for these box locations using \"scale aware assignment\". A BYOL loss is applied between representations of the same box across the three views. As in BYOL an EMA teacher network and projection head are used.\n\nThey pretrain R50-C4 and R50-FPN  models upstream and fine-tune corresponding Mask-RCNN models downstream for MS COCO and Pascal VOC object detection and MS COCO instance segmentation. Low data regime is explored using 5% and 10% \"Mini\" COCO setups. ImageNet linear eval results are included in the appendix. Ablation experiments evaluate whether stronger alignment in terms of architecture between pre-training and fine-tuning were helpful and sensitivity w.r.t. hyper-parameters such as view size, momentum for EMA, batch-size etc.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to understand. The experiments section is detailed using various datasets and different amounts of data and meaningful ablation experiments. Self-supervised learning for detection is of significance to the community and this paper furthers an emerging theme of trying to pre-train the entire model and do more than just learn a global image representation.\n\nRecommendations to further improve the paper are below.\n\nInsLoc [12], DetCon [13], and ReSim [41] are all closely related to the submission. Of these only InsLoc is explained in some detail in the related works section. DetCon suggests regions and aligns representations across multiple views. ReSim uses a sliding window approach instead of heuristic region proposal methods, but in the end also identifies positive pairs based on spatial alignment and uses a contrastive loss to align representations. $\\mathrm{ReSim-FPN}^T$ infact transfers the FPN downstream similar to how the proposed method does. But ReSim does not have \"scale aware assignment\". ReSim and DetCon are contemporary works to this submission. They came out on arxiv just 2 months before the deadline, but a more detailed discussion of these three papers will be a great addition to the related works section.\n\nPlease also add \"Self-Supervised Representation Learning from Flow Equivariance. Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun Tech report, arXiv, Jan. 2021\" (This is a dense representation learning method that uses flow as an additional data augmentation).\n\nAlso \"UP-DETR: Unsupervised Pre-training for Object Detection with Transformers. Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen. CVPR 2021\" is similar to the current work as it pre-trains all of DETR, and not just the backbone, and transfers it downstream.\n\nLastly Romijnders, Rob, et al. \"Representation learning from videos in-the-wild: An object-centric approach.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021, has similar ideas but use a supervised object detector as their region proposal module and use predicted class labels to define positives.\n\nIn Table 5, is row 1 basically equal to BYOL? Is row-2 basically InsLoc? If not what are the differences?\n\nIn Table 5, how do you pre-train a FPN model without scale aware assignment in rows 3 and 4?\n\nThe performance deltas due to parameter changes in the ablation experiments are very small. This,  together with the lack of error bars, makes it difficult to know when a parameter is critical and when it has no effect. Computational costs might prohibit the addition of error bars everywhere but even showing just 1 number with error bars could convey a sense of the variances involved.\n\nBYOL is not really contrastive. So calling it contrastive learning when there are no negatives involved is confusing. But I do not know what else to suggest.\n\n\n## Post rebuttal update\nThe rebuttal response addresses my concerns. The rebuttal also included several experiments that further strengthen the paper. I continue to support the acceptance of this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The model does poorly on ImageNet classification downstream. This limitation has been identified and reported in the appendix. Societal impact has also been discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "SG4j7Nlo6iP", "reviewer": "Reviewer_ABcQ", "summary": "This paper provides a self supervised method tailored for object detection (in the RCNN\nparadigm), inspired from BYOL.  \nThe general idea is to use a non-parametric method (selective search) to generate some\nboxes, then apply a BYOL-style loss between ROI-Aligned features of each box coming from\ndifferent augmentations of the image. \n\nIt differs from other SSL methods (BYOL, SWAV,...) in that it instead of pretraining solely\nthe backbone, this methods ensures that all the components of the detector (FPN, RCNN head)\nare pre-trained.\n\nIt shows convincing results on COCO and pascal as well as some subsets, in a few shot setting.\nThe code is made available as part of the supplementary, thus helping with reproducibility.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is very well written, clearly motivated and ablated. The contribution is\nsolid and seems widely applicable. The paper is lacking a few details and references that\nI'm listing in the following section, but overall I would recommend acceptance.\n\n## Missing references\nHere are a couple of references which would make the related work more complete:\n\n- [3] is a CVPR 2021 paper investigating SSL for detection in the DETR framework\n- [4] explores SSL for object detection without using Imagenet images\n- [5] has some similar contrastive ideas, targets the RetinaNet architecture\n- [6,7] more recent SSL methods (for classification)\n\n## Technical details on the SSL recipe\n\nThis paper is inspired by BYOL, but seem to diverge in a number of subtle way that are not\nalways clearly explained or ablated. I list them in this section.\n\n### Stop gradients\n\nRegular BYOL prevents gradients from flowing back to the target network, by using a\nstop-gradient mechanism.\nThis is an important aspect of BYOL, it has been shown to be a mandatory for other methods eg [2].\n\nIn this paper, I didn't see any mention of a stop-gradient mechanism. Is it applied? If not\nis there a reason for it? This needs to be clarified in my opinion.\n\n### Symmetric loss\n    \nIn section 3.3, the paper indicates that the contrastive loss is symmetrized by swapping V1\nand V2/V3 and re-computing the loss.\n\nThis is not a cheap choice, since it incurs doubling the number of forwards. However, the\npaper doesn't clearly motivate it, and as far as I can tell there is no ablation on the\nimpact this symmetrization has on the performance. Could this be clarified?\n\n### Multi view\n\nThis paper uses 3 views in the loss, compared to BYOL which uses only 2.\nThis change is ablated in table 4, and improves performance by roughly 0.6 Box AP, while\nhaving very little effect on the mask AP.\n\nAccording to the paper, \"multiple views (V3) further directs SoCo towards learning\nscale-invariant and translation-invariant representations\". Is there any evidence of that\nfact? For example, if it really improves scale invariance, I would expect the performance on\nsmall objects to improve more significantly than large, since the third view is a\ndown-sampled version of the image. Is this the case in practice? Similarly, table 5(a) would\npossibly benefit from reporting the AP break-down by size.\n\nWhat is the computational cost incurred by this third view?\n\nIn table 1, the paper further reports results with 4 views, with even better results.\nIt seems that adding views help, is there a reason to believe that it will saturate if one\nkept adding more scales or crops?\n\n\nFinally, the V3 is a deterministic transform of V2. Would it help to add some\nstochasticity in the process? Eg resize to a random size (possibly bigger than the original)\ninstead of always downsampling by 2?  \n\n\n### Batchsize\n\nThe study on the batch size (table 5b) is interesting and a bit counter-intuitive. Usually\nSSL methods tend to always benefit from bigger batch-size. Is there any intuition why this\nis not the case for this particular case?\n\n## Longer training?\n\nIn [1], the authors have shown that after sufficient iterations, the effect of imagenet\npre-training tends to be erased, and random init tends to perform as well supervised\npre-training.\n\nThis paper reports only short schedules (1x and 2x), which arguably are important for\npractitioners. However, I would be interested to find out if this SoCo method keeps its edge\nover the supervised baseline on a long schedule (say 6x)?\n\nIf it does, I would consider it an even more impressive result.\n\n## Pretraining dataset\n\nThe method uses pre-training on ImageNet. While this aids comparison with various SSL\nmethods that are not detection aware, I would argue that this paper could be made stronger\nby also using more complex images during pre-training. One potential option would be to\npre-train on the whole train set of COCO as well as the unlabeled set of COCO. If size is an\nissue, other detection datasets could be appended (Object365, OpenImages, ...)\n\nBeyond the pure performance implications, it would be interesting to see if some of the\nconclusions of the ablations change for such a pretraining. In particular Table 5.c shows\nthat adding more randomly selected boxes doesn't help, and the paper blames it on the fact\nthat \"most of the images in the ImageNet dataset only contain a few objects\". Pretraining on\na real detection dataset would alleviate this issue, and potentially unlock the full\npotential of the method.\n\n## Localization pretraining?\n\nIn detection, both classification and localization play a big role in the final performance\nof the model. Because of the contrastive criterion that is being used, it seems that more\nemphasis is put on the classification part than on the regression part.\n\nWould it be possible to include some regression-aware objectives in the pre-training as\nwell?\nOne idea that comes to mind is similar to the box-jitter idea that is used in this paper:\nit could be possible to train the localization part of Mask-RCNN to cancel the jitter, ie\nregress the original position of the box. Has anything along those lines been tried?\n\n\n[1] Rethinking ImageNet Pre-training, He at al.\n\n[2] Exploring Simple Siamese Representation Learning, Chen et al\n\n[3] UP-DETR: Unsupervised Pre-training for Object Detection with Transformers, Dai et al\n\n[4] Self-EMD: Self-Supervised Object Detection without ImageNet, Liu et al\n\n[5] Unsupervised Pretraining for Object Detection by Patch Reidentification, Ding et al\n\n[6] Barlow Twins: Self-Supervised Learning via Redundancy Reduction, J.Zbontar et al.\n\n[7] Emerging Properties in Self-Supervised Vision Transformers, M.Caron et al.\n\n# Post rebuttal update:\nIn light of the excellent rebuttal, I increase my score to 9", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No particular societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "BdiJQ3QampS", "reviewer": "Reviewer_uFr7", "summary": "The paper aims to design a self-supervised pre-training method for the downstream object detection task.  The authors propose to perform self-supervised object-level representation learning. Specifically, this method uses selective search to obtain object proposals in unlabeled images and extract the object-level features as in Mask R-CNN. Clear improvements have been achieved on COCO object detection and instance segmentation with Mask R-CNN.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n- The proposed object-level pre-training method is simple, straightforward, and motivated.\n- The paper is well written and organized, making it easy to read and follow.\n- The method achieves strong performance in COCO semi-supervised setting (5%/10% training data).\n\n## Weakness\n- The used representation method as in BYOL is not exactly contrastive learning.  The method name in the title and main paper is not appropriate.  In the BYOL paper, the authors distinguish their method from contrastive methods. \n- The fine-tuning experiments are mainly conducted on COCO with a single architecture Mask R-CNN. But the pre-training should be general, even in the single task of object detection. To prevent overfitting a single dataset and a single arch. The authors are suggested to experiment with different datasets, e.g., Cityscapes, and different architectures, e.g., RetinaNet and FCOS. For different archs, you can just extract the pre-trained backbone and compare it to supervised or other self-supervised methods.\n- The proposed method performs worse than the previous method PixPro on VOC.\n- In Table 4, why does FPN make the performance worse?\n\n## Others\n- Did you try pre-training on non-object-centric datasets like COCO? The paper only reports results on ImageNet which is object-centric. I guess that's also why you can only set a small K for each image. It would be more interesting if the method can benefit more from data like COCO.\n\n## After Rebuttal\nThe rebuttal has addressed all my concerns. Thus I raise the rating from 6 to 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they are in the appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree this is a solid paper with a good novel contribution, great experimental results, and detailed analysis/ablation. The authors did a good job addressing the reviewers' concerns in their responses too. Clear accept.
Decision: Accept (Spotlight)