Aligning Pretraining for Detection via Object-Level Contrastive Learning

1 Introduction . Pretraining and finetuning has been the dominant paradigm of training deep neural networks in computer vision . Downstream tasks usually leverage pretrained weights learned on large labeled datasets such as ImageNet [ 1 ] for initialization . As a result , supervised ImageNet pretraining has been prevalent throughout the field . Recently , self-supervised pretraining [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] has achieved considerable progress and alleviated the dependency on labeled data . These methods aim to learn generic visual representations for various downstream tasks by means of image-level pretext tasks , such as instance discrimination . Some recent works [ 10 , 11 , 12 , 13 ] observe that the image-level representations are sub-optimal for dense prediction tasks such as object detection and semantic segmentation . A potential reason is that image-level pretraining may overfit to holistic representations and fail to learn properties that are important outside of image classification . The goal of this work is to develop self-supervised pretraining that is aligned to object detection . In object detection , bounding boxes are widely adopted as the representation for objects . Translation and scale invariance for object detection are reflected by the location and size of the bounding boxes . An obvious representation gap exists between image-level pretraining and the object-level bounding boxes of object detection . Motivated by this , we present an object-level self-supervised pretraining framework , called Selective Object COntrastive learning ( SoCo ) , specifically for the downstream task of object detection . To introduce object-level representations into pretraining , SoCo utilizes off-the-shelf selective search [ 14 ] to generate object proposals . Different from prior image-level contrastive learning methods which ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . treat the whole image as an instance , SoCo treats each object proposal in the image as an independent instance . This enables us to design a new pretext task for learning object-level visual representations with properties that are compatible with object detection . Specifically , SoCo constructs object-level views where the scales and locations of the same object instance are augmented . Contrastive learning follows to maximize the similarity of the object across augmented views . The introduction of the object-level representation also allows us to further bridge the gap in network architecture between pretraining and finetuning . Object detection often involves dedicated modules , e.g. , feature pyramid network ( FPN ) [ 15 ] , and special-purpose sub-networks , e.g. , R-CNN head [ 16 , 17 ] . In contrast to image-level contrastive learning methods where only feature backbones are pretrained and transferred , SoCo performs pretraining over all the network modules used in detectors . As a result , all layers of the detectors can be well-initialized . Experimentally , the proposed SoCo achieves state-of-the-art transfer performance from ImageNet to COCO . Concretely , by using Mask R-CNN with an R50-FPN backbone , it obtains 43.2 APbb / 38.4 APmk on COCO with a 1× schedule , which are +4.3 APbb / +3.0 APmk better than the supervised pretraining baseline , and 44.3 APbb / 39.6 APmk on COCO with a 2× schedule , which are +3.0 APbb / +2.3 APmk better than the supervised pretraining baseline . When transferred to Mask R-CNN with an R50-C4 backbone , it achieves 40.9 APbb / 35.3 APmk and 42.0 APbb / 36.3 APmk on the 1× and 2× schedule , respectively . 2 Related Work . Unsupervised feature learning has a long history in training deep neural networks . Auto-encoders [ 18 ] and Deep Boltzmann Machines [ 19 ] learn layers of representations by reconstructing image pixels . Unsupervised pretraining is often used as a method for initialization , which is followed by supervised training on the same dataset such as handwritten digits for classification . Recent advances in unsupervised learning , especially self-supervised learning , tend to formulate the problem in a transfer learning scenario , where pretraining and finetuning are trained on different datasets with different purposes . Pretext tasks such as colorization [ 20 ] , context prediction [ 21 ] , inpainting [ 22 ] and rotation prediction [ 23 ] force the network to learn semantic information in order to solve the pretext tasks . Contrastive methods based on the instance discrimination pretext task [ 2 ] learn to map augmented views of the same instance to similar embeddings . Such approaches [ 24 , 4 , 5 , 9 , 25 , 26 ] have shown strong transfer ability for a number of downstream tasks , sometimes even outperforming the supervised counterpart by large margins [ 4 ] . With new technical innovations such as learning clustering assignments [ 7 ] , large-scale contrastive learning has been successfully applied on non-curated data sets [ 27 , 28 ] . While the linear evaluation result on ImageNet classification has improved significantly [ 3 ] , the progress on transfer performance for dense prediction tasks has been limited . Due to this , a growing number of works investigate pretraining specifically for object detection and semantic segmentation . The idea is to shift imagelevel representations to pixel-level or region-level representations . VADer [ 29 ] , PixPro [ 10 ] and DenseCL [ 11 ] propose to learn pixel-level representations by matching point features of the same physical location under different views . InsLoc [ 12 ] learns to match region-level features from composited imagery . DetCon [ 13 ] additionally uses the bottom-up segmentation of MCG [ 30 ] for supervising intra-segment pixel variations . UP-DETR [ 31 ] proposes a pretext task named random query patch detection to pretrain DETR detector . Self-EMD [ 32 ] explores self-supervised representation learning for object detection without using Imagenet images . Our work is also motivated and inspired by object detection methods which advocate architectures and training schemes invariant to translation and scale transformations [ 15 , 33 ] . We find that incorporating them in the pretraining stage also benefits object detection , which has largely been overlooked in previous self-supervised learning works . 3 Method . We introduce a new contrastive learning method which maximizes the similarity of object-level features representing different augmentations of the same object . Meanwhile , we introduce architectural alignment and important properties of object detection into pretraining when designing the objectlevel pretext task . We use an example where transfer learning is performed on Mask-RCNN with a R50-FPN backbone to illustrate the core design principles . To further demonstrate the extensibility and flexibility of our method , we also apply SoCo on a R50-C4 structure . In this section , we first present an overview of the proposed SoCo in Section 3.1 . Then we describe the process of object proposal generation and view construction in Section 3.2 . Finally , the object-level contrastive learning and our design principles are introduced in Section 3.3 . 3.1 Overview . Figure 1 displays an overview of SoCo . SoCo aims to align pretraining to object detection in two aspects : 1 ) network architecture alignment between pretraining and object detection ; 2 ) introducing central properties of detection . Concretely , besides pretraining a backbone as done in existing selfsupervised contrastive learning methods , SoCo also pretrains all the network modules used in an object detector , such as FPN and the head in the Mask R-CNN framework . As a result , all layers of the detector can be well-initialized . Furthermore , SoCo strives to learn object-level representations which are not only meaningful , but also invariant to translation and scale . To achieve this , it encourages diversity of scales and locations of objects by constructing multiple augmented views and applying a scale-aware assignment strategy for different levels of a feature pyramid . Finally , object-level contrastive learning is applied to maximize the feature similarity of the same object across augmented views . 3.2 Data Preprocessing . Object Proposal Generation . Inspired by R-CNN [ 34 ] and Fast R-CNN [ 35 ] , we use selective search [ 14 ] , an unsupervised object proposal generation algorithm which takes into account color similarity , texture similarity , size of region and fit between regions , to generate a set of object proposals for each of the raw images . We represent each object proposal as a bounding box b = { x , y , w , h } , where ( x , y ) denotes the coordinates of the bounding box center , and w and h are the corresponding width and height , respectively . We keep only the proposals that satisfy the following requirements : 1 ) 1/3 ≤ w/h ≤ 3 ; 2 ) 0.3 ≤ √ wh/ √ WH ≤ 0.8 , where W and H denote width and height of the input image . The object proposal generation step is performed offline . In each training iteration , we randomly select K proposals for each input image . View Construction . Three views , namely V1 , V2 and V3 , are used in SoCo . The input image is resized to 224× 224 to obtain V1 . Then we apply a random crop with a scale range of [ 0.5 , 1.0 ] on V1 in generating V2 . V2 is then resized to the same size as V1 and object proposals outside of V2 are dropped . Next , we downsample V2 to a fixed size ( e.g . 112 × 112 ) to produce V3 . In all of these cases , the bounding boxes transformed according to the cropping and resizing of the RGB images ( see Figure 1 , Data Preprocessing ) . Finally , each view is randomly and independently augmented . We adopt the augmentation pipeline of BYOL [ 3 ] but discard the random crop augmentation since spatial transformation is already applied on all three views . Notice that the scale and location of the same object proposal are different across the augmented views , which enables the model to learn translation-invariant and scale-invariant object-level representations . Box Jitter . To further encourage variance of scales and locations of object proposals across views , we adopt a box jitter strategy on the generated proposals as an object-level data augmentation . Specifically , given an object proposal b = { x , y , w , h } , we randomly generate a jittered box b̂ = { x̂ , ŷ , ŵ , ĥ } as follows : 1 ) x̂ = x + r · w ; 2 ) ŷ = y + r · h ; 3 ) ŵ = w + r · w ; 4 ) ĥ = h + r · h , where r ∈ [ −0.1 , 0.1 ] . The box jitter is randomly applied on each proposal with a probability of 0.5 . 3.3 Object-Level Contrastive Learning . The goal of SoCo is to align pretraining to object detection . Here , we use the representative framework Mask R-CNN [ 17 ] with feature pyramid network ( FPN ) [ 15 ] to demonstrate our key design principles . The alignment mainly involves aligning the pretraining architecture with that of object detection and integrating important object detection properties such as object-level translation invariance and scale invariance into the pretraining . Aligning Pretraining Architecture to Object Detection . Following Mask R-CNN , we use a backbone with FPN as the image-level feature extractor f I . We denote the output of FPN as { P2 , P3 , P4 , P5 } with a stride of { 4 , 8 , 16 , 32 } . Here , we do not use P6 due to its low resolution . With the bounding box representation b , RoIAlign [ 17 ] is applied to extract the foreground feature from the corresponding scale level . For further architectural alignment , we additionally introduce an R-CNN head fH into pretraining . The object-level feature representation h of bounding box b is extracted from an image view V as : h = fH ( RoIAlign ( f I ( V ) , b ) ) . ( 1 ) SoCo uses two neural networks to learn , namely an online network and a target network . The online network and the target network share the same architecture but with different sets of weights . Concretely , the target network weights f Iξ , f H ξ are the exponential moving average ( EMA ) with the momentum coefficient τ of the online parameters f Iθ and f H θ . Denote the set of object proposals { bi } considered in the image . Let hi be the object-level representation of proposal bi in view V1 , and h′i , h ′′ i be the representation of bi in view V2 , V3 . They are extracted using the online network and the target network respectively , hi = f H θ ( RoIAlign ( f I θ ( V1 ) , bi ) ) , ( 2 ) h′i = f H ξ ( RoIAlign ( f I ξ ( V2 ) , bi ) ) , h ′′ i = f H ξ ( RoIAlign ( f I ξ ( V3 ) , bi ) ) . ( 3 ) We follow BYOL [ 3 ] for learning contrastive representations . The online network is appended with a projector gθ , and a predictor qθ for obtaining latent embeddings . Both gθ and qθ are two-layer MLPs . The target network is only appended with the projector gξ for avoiding trivial solutions . We use vi , v′i and v′′i to denote the latent embeddings of object-level representations { hi , h ′ i , h ′′ i } , respectively : vi = qθ ( gθ ( hi ) ) , v ′ i = gξ ( h ′ i ) , v ′′ i = gξ ( h ′′ i ) . ( 4 ) The contrastive loss for the i-th object proposal is defined as : Li = −2 · 〈vi , v′i〉 ‖vi‖2 · ‖v′i‖2 − 2 · 〈vi , v ′′ i 〉 ‖vi‖2 · ‖v′′i ‖2 . ( 5 ) Then we can formulate the overall loss function for each image as : L = 1 K K∑ i=1 Li , ( 6 ) where K is the number of object proposals . We symmetrize the loss L in Eq . 6 by separately feeding V1 to the target network and { V2 , V3 } to the online network to compute L̃ . At each training iteration , we perform a stochastic optimization step to minimize LSoCo = L+ L̃ . Scale-Aware Assignment . Mask R-CNN with FPN uses the IoU between anchors and ground-truth boxes to determine positive samples . It defines anchors to have areas of { 322 , 642 , 1282 , 2562 } pixels on { P2 , P3 , P4 , P5 } , respectively , which means ground-truth boxes of scale within a range are assigned to a specific pyramid level . Inspired by this , we propose a scale-aware assignment strategy , which greatly encourages the pretraining model to learn object-level scaleinvariant representations . Concretely , we assign object proposals of area within a range of { 0 − 482 , 492 − 962 , 972 − 1922 , 1932 − 2242 } pixels to { P2 , P3 , P4 , P5 } , respectively . Notice that the maximum proposal size is 224× 224 since all of the views are resized to a fixed resolution smaller than 224 . The advantage is that the same object proposals at different scales are encouraged to learn consistent representations through contrastive learning . As a result , SoCo learns object-level scale-invariant visual representations , which is important for object detection . Introducing Properties of Detection to Pretraining . Here , we discuss how SoCo promotes important properties of object detection in the pretraining . Object detection uses tight bounding boxes to represent objects . To introduce object-level representations , SoCo generates object proposals by selective search . Translation invariance and scale invariance at the object level are regarded as the most important properties for object detection , i.e. , feature representations of objects belonging to same category should be insensitive to scale and location . Recall that V2 is a randomly cropped patch of V1 . Random cropping introduces box shift and thus contrastive learning between V1 and V2 encourages the pretraining model to learn location-invariant representations . V3 is generated by downsampling V2 , which results in a scale augmentation of object proposals . With our scale-aware assignment strategy , the contrastive loss between V1 and V3 guides the pretraining towards learning scale-invariant visual representations . Extending to Other Object Detectors . SoCo can be easily extended to align to other detectors besides Mask R-CNN with FPN . Here , we apply SoCo to Mask R-CNN with a C4 structure , which is a popular non-FPN detection framework . The modification is three-fold : 1 ) for all object proposals , RoIAlign is performed on C4 ; 2 ) the R-CNN head is replaced by the entire 5-th residual block ; 3 ) view V3 is discarded and the remaining V1 and V2 are kept for object-level contrastive learning . Experiments and comparisons with state-of-the-art methods in Section 4.3 demonstrate the extensibility and flexibility of SoCo .

Summary: The paper aims to design a self-supervised pre-training method for the downstream object detection task.  The authors propose to perform self-supervised object-level representation learning. Specifically, this method uses selective search to obtain object proposals in unlabeled images and extract the object-level features as in Mask R-CNN. Clear improvements have been achieved on COCO object detection and instance segmentation with Mask R-CNN.