Aligning Pretraining for Detection via Object-Level Contrastive Learning

1 Introduction . Pretraining and finetuning has been the dominant paradigm of training deep neural networks in computer vision . Downstream tasks usually leverage pretrained weights learned on large labeled datasets such as ImageNet [ 1 ] for initialization . As a result , supervised ImageNet pretraining has been prevalent throughout the field . Recently , self-supervised pretraining [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] has achieved considerable progress and alleviated the dependency on labeled data . These methods aim to learn generic visual representations for various downstream tasks by means of image-level pretext tasks , such as instance discrimination . Some recent works [ 10 , 11 , 12 , 13 ] observe that the image-level representations are sub-optimal for dense prediction tasks such as object detection and semantic segmentation . A potential reason is that image-level pretraining may overfit to holistic representations and fail to learn properties that are important outside of image classification . The goal of this work is to develop self-supervised pretraining that is aligned to object detection . In object detection , bounding boxes are widely adopted as the representation for objects . Translation and scale invariance for object detection are reflected by the location and size of the bounding boxes . An obvious representation gap exists between image-level pretraining and the object-level bounding boxes of object detection . Motivated by this , we present an object-level self-supervised pretraining framework , called Selective Object COntrastive learning ( SoCo ) , specifically for the downstream task of object detection . To introduce object-level representations into pretraining , SoCo utilizes off-the-shelf selective search [ 14 ] to generate object proposals . Different from prior image-level contrastive learning methods which ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . treat the whole image as an instance , SoCo treats each object proposal in the image as an independent instance . This enables us to design a new pretext task for learning object-level visual representations with properties that are compatible with object detection . Specifically , SoCo constructs object-level views where the scales and locations of the same object instance are augmented . Contrastive learning follows to maximize the similarity of the object across augmented views . The introduction of the object-level representation also allows us to further bridge the gap in network architecture between pretraining and finetuning . Object detection often involves dedicated modules , e.g. , feature pyramid network ( FPN ) [ 15 ] , and special-purpose sub-networks , e.g. , R-CNN head [ 16 , 17 ] . In contrast to image-level contrastive learning methods where only feature backbones are pretrained and transferred , SoCo performs pretraining over all the network modules used in detectors . As a result , all layers of the detectors can be well-initialized . Experimentally , the proposed SoCo achieves state-of-the-art transfer performance from ImageNet to COCO . Concretely , by using Mask R-CNN with an R50-FPN backbone , it obtains 43.2 APbb / 38.4 APmk on COCO with a 1× schedule , which are +4.3 APbb / +3.0 APmk better than the supervised pretraining baseline , and 44.3 APbb / 39.6 APmk on COCO with a 2× schedule , which are +3.0 APbb / +2.3 APmk better than the supervised pretraining baseline . When transferred to Mask R-CNN with an R50-C4 backbone , it achieves 40.9 APbb / 35.3 APmk and 42.0 APbb / 36.3 APmk on the 1× and 2× schedule , respectively . 2 Related Work . Unsupervised feature learning has a long history in training deep neural networks . Auto-encoders [ 18 ] and Deep Boltzmann Machines [ 19 ] learn layers of representations by reconstructing image pixels . Unsupervised pretraining is often used as a method for initialization , which is followed by supervised training on the same dataset such as handwritten digits for classification . Recent advances in unsupervised learning , especially self-supervised learning , tend to formulate the problem in a transfer learning scenario , where pretraining and finetuning are trained on different datasets with different purposes . Pretext tasks such as colorization [ 20 ] , context prediction [ 21 ] , inpainting [ 22 ] and rotation prediction [ 23 ] force the network to learn semantic information in order to solve the pretext tasks . Contrastive methods based on the instance discrimination pretext task [ 2 ] learn to map augmented views of the same instance to similar embeddings . Such approaches [ 24 , 4 , 5 , 9 , 25 , 26 ] have shown strong transfer ability for a number of downstream tasks , sometimes even outperforming the supervised counterpart by large margins [ 4 ] . With new technical innovations such as learning clustering assignments [ 7 ] , large-scale contrastive learning has been successfully applied on non-curated data sets [ 27 , 28 ] . While the linear evaluation result on ImageNet classification has improved significantly [ 3 ] , the progress on transfer performance for dense prediction tasks has been limited . Due to this , a growing number of works investigate pretraining specifically for object detection and semantic segmentation . The idea is to shift imagelevel representations to pixel-level or region-level representations . VADer [ 29 ] , PixPro [ 10 ] and DenseCL [ 11 ] propose to learn pixel-level representations by matching point features of the same physical location under different views . InsLoc [ 12 ] learns to match region-level features from composited imagery . DetCon [ 13 ] additionally uses the bottom-up segmentation of MCG [ 30 ] for supervising intra-segment pixel variations . UP-DETR [ 31 ] proposes a pretext task named random query patch detection to pretrain DETR detector . Self-EMD [ 32 ] explores self-supervised representation learning for object detection without using Imagenet images . Our work is also motivated and inspired by object detection methods which advocate architectures and training schemes invariant to translation and scale transformations [ 15 , 33 ] . We find that incorporating them in the pretraining stage also benefits object detection , which has largely been overlooked in previous self-supervised learning works . 3 Method . We introduce a new contrastive learning method which maximizes the similarity of object-level features representing different augmentations of the same object . Meanwhile , we introduce architectural alignment and important properties of object detection into pretraining when designing the objectlevel pretext task . We use an example where transfer learning is performed on Mask-RCNN with a R50-FPN backbone to illustrate the core design principles . To further demonstrate the extensibility and flexibility of our method , we also apply SoCo on a R50-C4 structure . In this section , we first present an overview of the proposed SoCo in Section 3.1 . Then we describe the process of object proposal generation and view construction in Section 3.2 . Finally , the object-level contrastive learning and our design principles are introduced in Section 3.3 . 3.1 Overview . Figure 1 displays an overview of SoCo . SoCo aims to align pretraining to object detection in two aspects : 1 ) network architecture alignment between pretraining and object detection ; 2 ) introducing central properties of detection . Concretely , besides pretraining a backbone as done in existing selfsupervised contrastive learning methods , SoCo also pretrains all the network modules used in an object detector , such as FPN and the head in the Mask R-CNN framework . As a result , all layers of the detector can be well-initialized . Furthermore , SoCo strives to learn object-level representations which are not only meaningful , but also invariant to translation and scale . To achieve this , it encourages diversity of scales and locations of objects by constructing multiple augmented views and applying a scale-aware assignment strategy for different levels of a feature pyramid . Finally , object-level contrastive learning is applied to maximize the feature similarity of the same object across augmented views . 3.2 Data Preprocessing . Object Proposal Generation . Inspired by R-CNN [ 34 ] and Fast R-CNN [ 35 ] , we use selective search [ 14 ] , an unsupervised object proposal generation algorithm which takes into account color similarity , texture similarity , size of region and fit between regions , to generate a set of object proposals for each of the raw images . We represent each object proposal as a bounding box b = { x , y , w , h } , where ( x , y ) denotes the coordinates of the bounding box center , and w and h are the corresponding width and height , respectively . We keep only the proposals that satisfy the following requirements : 1 ) 1/3 ≤ w/h ≤ 3 ; 2 ) 0.3 ≤ √ wh/ √ WH ≤ 0.8 , where W and H denote width and height of the input image . The object proposal generation step is performed offline . In each training iteration , we randomly select K proposals for each input image . View Construction . Three views , namely V1 , V2 and V3 , are used in SoCo . The input image is resized to 224× 224 to obtain V1 . Then we apply a random crop with a scale range of [ 0.5 , 1.0 ] on V1 in generating V2 . V2 is then resized to the same size as V1 and object proposals outside of V2 are dropped . Next , we downsample V2 to a fixed size ( e.g . 112 × 112 ) to produce V3 . In all of these cases , the bounding boxes transformed according to the cropping and resizing of the RGB images ( see Figure 1 , Data Preprocessing ) . Finally , each view is randomly and independently augmented . We adopt the augmentation pipeline of BYOL [ 3 ] but discard the random crop augmentation since spatial transformation is already applied on all three views . Notice that the scale and location of the same object proposal are different across the augmented views , which enables the model to learn translation-invariant and scale-invariant object-level representations . Box Jitter . To further encourage variance of scales and locations of object proposals across views , we adopt a box jitter strategy on the generated proposals as an object-level data augmentation . Specifically , given an object proposal b = { x , y , w , h } , we randomly generate a jittered box b̂ = { x̂ , ŷ , ŵ , ĥ } as follows : 1 ) x̂ = x + r · w ; 2 ) ŷ = y + r · h ; 3 ) ŵ = w + r · w ; 4 ) ĥ = h + r · h , where r ∈ [ −0.1 , 0.1 ] . The box jitter is randomly applied on each proposal with a probability of 0.5 . 3.3 Object-Level Contrastive Learning . The goal of SoCo is to align pretraining to object detection . Here , we use the representative framework Mask R-CNN [ 17 ] with feature pyramid network ( FPN ) [ 15 ] to demonstrate our key design principles . The alignment mainly involves aligning the pretraining architecture with that of object detection and integrating important object detection properties such as object-level translation invariance and scale invariance into the pretraining . Aligning Pretraining Architecture to Object Detection . Following Mask R-CNN , we use a backbone with FPN as the image-level feature extractor f I . We denote the output of FPN as { P2 , P3 , P4 , P5 } with a stride of { 4 , 8 , 16 , 32 } . Here , we do not use P6 due to its low resolution . With the bounding box representation b , RoIAlign [ 17 ] is applied to extract the foreground feature from the corresponding scale level . For further architectural alignment , we additionally introduce an R-CNN head fH into pretraining . The object-level feature representation h of bounding box b is extracted from an image view V as : h = fH ( RoIAlign ( f I ( V ) , b ) ) . ( 1 ) SoCo uses two neural networks to learn , namely an online network and a target network . The online network and the target network share the same architecture but with different sets of weights . Concretely , the target network weights f Iξ , f H ξ are the exponential moving average ( EMA ) with the momentum coefficient τ of the online parameters f Iθ and f H θ . Denote the set of object proposals { bi } considered in the image . Let hi be the object-level representation of proposal bi in view V1 , and h′i , h ′′ i be the representation of bi in view V2 , V3 . They are extracted using the online network and the target network respectively , hi = f H θ ( RoIAlign ( f I θ ( V1 ) , bi ) ) , ( 2 ) h′i = f H ξ ( RoIAlign ( f I ξ ( V2 ) , bi ) ) , h ′′ i = f H ξ ( RoIAlign ( f I ξ ( V3 ) , bi ) ) . ( 3 ) We follow BYOL [ 3 ] for learning contrastive representations . The online network is appended with a projector gθ , and a predictor qθ for obtaining latent embeddings . Both gθ and qθ are two-layer MLPs . The target network is only appended with the projector gξ for avoiding trivial solutions . We use vi , v′i and v′′i to denote the latent embeddings of object-level representations { hi , h ′ i , h ′′ i } , respectively : vi = qθ ( gθ ( hi ) ) , v ′ i = gξ ( h ′ i ) , v ′′ i = gξ ( h ′′ i ) . ( 4 ) The contrastive loss for the i-th object proposal is defined as : Li = −2 · 〈vi , v′i〉 ‖vi‖2 · ‖v′i‖2 − 2 · 〈vi , v ′′ i 〉 ‖vi‖2 · ‖v′′i ‖2 . ( 5 ) Then we can formulate the overall loss function for each image as : L = 1 K K∑ i=1 Li , ( 6 ) where K is the number of object proposals . We symmetrize the loss L in Eq . 6 by separately feeding V1 to the target network and { V2 , V3 } to the online network to compute L̃ . At each training iteration , we perform a stochastic optimization step to minimize LSoCo = L+ L̃ . Scale-Aware Assignment . Mask R-CNN with FPN uses the IoU between anchors and ground-truth boxes to determine positive samples . It defines anchors to have areas of { 322 , 642 , 1282 , 2562 } pixels on { P2 , P3 , P4 , P5 } , respectively , which means ground-truth boxes of scale within a range are assigned to a specific pyramid level . Inspired by this , we propose a scale-aware assignment strategy , which greatly encourages the pretraining model to learn object-level scaleinvariant representations . Concretely , we assign object proposals of area within a range of { 0 − 482 , 492 − 962 , 972 − 1922 , 1932 − 2242 } pixels to { P2 , P3 , P4 , P5 } , respectively . Notice that the maximum proposal size is 224× 224 since all of the views are resized to a fixed resolution smaller than 224 . The advantage is that the same object proposals at different scales are encouraged to learn consistent representations through contrastive learning . As a result , SoCo learns object-level scale-invariant visual representations , which is important for object detection . Introducing Properties of Detection to Pretraining . Here , we discuss how SoCo promotes important properties of object detection in the pretraining . Object detection uses tight bounding boxes to represent objects . To introduce object-level representations , SoCo generates object proposals by selective search . Translation invariance and scale invariance at the object level are regarded as the most important properties for object detection , i.e. , feature representations of objects belonging to same category should be insensitive to scale and location . Recall that V2 is a randomly cropped patch of V1 . Random cropping introduces box shift and thus contrastive learning between V1 and V2 encourages the pretraining model to learn location-invariant representations . V3 is generated by downsampling V2 , which results in a scale augmentation of object proposals . With our scale-aware assignment strategy , the contrastive loss between V1 and V3 guides the pretraining towards learning scale-invariant visual representations . Extending to Other Object Detectors . SoCo can be easily extended to align to other detectors besides Mask R-CNN with FPN . Here , we apply SoCo to Mask R-CNN with a C4 structure , which is a popular non-FPN detection framework . The modification is three-fold : 1 ) for all object proposals , RoIAlign is performed on C4 ; 2 ) the R-CNN head is replaced by the entire 5-th residual block ; 3 ) view V3 is discarded and the remaining V1 and V2 are kept for object-level contrastive learning . Experiments and comparisons with state-of-the-art methods in Section 4.3 demonstrate the extensibility and flexibility of SoCo .

Summary: This paper proposes SoCo, a method for self-supervised representation learning for downstream object detection and instance segmentation tasks.  They argue for stronger alignment between downstream tasks and self-supervised pre-training. They demonstrate this in the case of object detection by (a) developing a representation learning scheme that encourages object level scale and translation invariance and (b) pre-training several components of the downstream detector such as FPN and R-CNN head unlike prior work which mostly just trains the ResNet backbone.  The former property will help their model detect objects at several locations and scales as is needed for object detection.  The latter property helps them get stronger alignment between the upstream and downstream setups. To this end they propose "scale aware assignment". The ideas is that, representations for small objects are taken from finer FPN pyramid levels and for bigger objects are taken from coarser FPN pyramid levels. This matches what happens in the actual FPN where anchors at finer pyramid levels are smaller and at coarser pyramid levels are bigger.  The overall recipe is as follows: Selective search, a hand-crafted region proposal technique, is used to suggest boxes on all images. K of these boxes are randomly selected in each training step per image and three views of that image are constructed using random resized crop and resize operations. FPN features are extracted using RoiAlign for these box locations using "scale aware assignment". A BYOL loss is applied between representations of the same box across the three views. As in BYOL an EMA teacher network and projection head are used.  They pretrain R50-C4 and R50-FPN  models upstream and fine-tune corresponding Mask-RCNN models downstream for MS COCO and Pascal VOC object detection and MS COCO instance segmentation. Low data regime is explored using 5% and 10% "Mini" COCO setups. ImageNet linear eval results are included in the appendix. Ablation experiments evaluate whether stronger alignment in terms of architecture between pre-training and fine-tuning were helpful and sensitivity w.r.t. hyper-parameters such as view size, momentum for EMA, batch-size etc.