Unsupervised Object-Level Representation Learning from Scene Images

1 Introduction . Unsupervised visual representation learning aims at obtaining transferable features with abundant unlabeled data . Recent self-supervised learning ( SSL ) methods based on contrastive learning [ 60 , 22 , 37 , 5 , 19 , 4 , 7 ] have largely narrowed the gap and even surpassed the supervised counterpart on a number of downstream tasks [ 30 , 49 , 15 , 47 , 35 , 23 ] . These methods build upon the instance discrimination task that maximizes the agreement between different data-augmented views of the same image . Despite their success , current SSL methods are primarily pre-trained on the unlabeled ImageNet [ 8 ] dataset that contains iconic images with single object as shown in Figure 1 ( a ) . The underlying object-centric constraint of ImageNet makes it hard to be applied in real world scenarios where more complex scene images with multiple objects are available . Meanwhile , naïvely adopting the off-the-shelf contrastive learning methods on scene images introduces inconsistent learning signals since random crops of the same image may correspond to different objects as shown in Figure 1 ( b ) . Indeed , it has been shown that current contrastive learning methods tend to struggle on more complex scene datasets [ 19 , 50 , 34 , 58 ] like COCO [ 33 ] or Places365 [ 72 ] . Therefore , it is imperative to design an effective object-level representation learning paradigm as illustrated in Figure 1 ( c ) to harness massive unlabeled scene images in the wild . 1Project page : https : //www.mmlab-ntu.com/project/orl/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we are interested in going beyond ImageNet to obtain better representations on noniconic images . Apparently , it is challenging to learn representations from scene-level images since they are entangled with many concepts including structures , objects , backgrounds and relationships . It remains an open question how to take advantages of spatial information of multiple objects naturally residing in the scene images when no object annotations are available , let alone further deriving object-level correspondence to construct positive object-instance pairs . To tackle these challenges , we introduce a novel object-level unsupervised representation learning framework tailored for scene images . Our framework is based on a key insight of the current contrastive learning methods : they can implicitly group different images with similar visual concepts together even though they are explicitly optimized to group different views of the same image . This phenomenon reveals that image-level contrastive learning has already induced a latent space with rich visual concepts . Though the latent space usually entangles other scene concepts like structures , backgrounds and relationships , it will be useful for object discovery if appropriately deployed . Through computing the similarity of sampled regions between k-nearest-neighbor ( KNN ) images , we conclude two observations : 1 ) image-level contrastive learning encodes priors for region correspondence discovery across images ; 2 ) high-response regions are usually objects or object parts . Based on the observation above , we propose a multi-stage framework for unsupervised object-level representation learning . Specifically , we first extract potential object-based regions in scene images using the unsupervised region proposal algorithms ( e.g. , selective search [ 56 ] ) . We then propose a region correspondence generation scheme to leverage the off-the-shelf image-level contrastive learning pre-trained model to discover corresponding object-instance pairs for the proposed regions in the embedding space . Finally , we use the obtained object-instance pairs to construct positive sample pairs for object-level representation learning . Figure 1 ( d ) shows several cross-image object-instance pairs discovered by our framework on COCO dataset using the latent prior of BYOL [ 19 ] , the state-of-the-art image-level contrastive learning method . The discovered inter-corresponding pairs substantially provide diverse intra-class variances at the object-instance level to aid object-level representation learning . Overall , our main contributions are summarized as follows : 1 ) We observe that existing image-level contrastive learning methods have priors to discover objectlevel correspondence across images . We leverage this prior for the first time for unsupervised cross-image object-level correspondence discovery . 2 ) With the obtained correspondence , we introduce a novel multi-stage self-supervised learning pipeline , termed as ORL , for object-level representation learning from scene images , going beyond object-centric ImageNet . 3 ) We contribute the first study for object-level SSL . ORL substantially outperforms image-level contrastive learning approaches pre-trained on COCO dataset ( ∼118k images with labels discarded ) , setting a new state of the art on this challenging dataset that contains diverse scenes in the wild . The COCO pre-trained ORL even surpasses supervised ImageNet pre-training on several considered downstream tasks . When SSL is conducted on a larger “ COCO+ ” dataset ( COCO train2017 set plus COCO unlabeled2017 set , ∼241k images in total ) , ORL further improves the performance , demonstrating its potential to benefit from more unlabeled scene data . 2 Related work . Self-supervised learning . Self-supervised learning builds unsupervised representations by exploiting the internal priors or structures of data in the form of a pretext task . A wide range of pretext tasks have been proposed in the past few years . Examples include patch context prediction [ 10 ] , jigsaw puzzles [ 39 ] , inpainting [ 43 ] , colorization [ 31 , 70 ] , cross-channel prediction [ 71 ] , visual primitive counting [ 40 ] , and rotation prediction [ 14 ] . Although good representations emerge with these pretext tasks , they are prone to lose generality due to their hand-crafted nature . Recently , contrastive learning [ 20 ] that performs instance discrimination [ 60 , 22 , 37 , 5 , 19 , 4 , 7 ] has shown great potential in this field , largely narrowing the gap to fully supervised learning . The core idea of contrastive learning is to gather positive pairs and separate negative pairs in the embedding space . A positive pair is usually formed with two transformed views of the same image while the negative pairs are formed with different images . Typically , contrastive learning methods require a large number of negative samples to avoid mode collapse . These samples can be maintained within a mini-batch [ 42 , 27 , 66 , 26 , 2 , 5 ] , a memory bank [ 60 , 53 , 74 , 37 ] or a queue [ 22 , 6 ] . BYOL [ 19 ] and SwAV [ 4 ] further remove the necessity of involving negative pairs . BYOL directly predicts the features of one view from another view , while SwAV predicts the cluster assignments between multiple views of the same image . Despite their improved performance , the existing image-level contrastive learning methods are largely confined to the underlying object-centric bias of ImageNet . More recently , a group of works that perform pixel-level [ 45 , 58 , 64 , 50 , 34 , 25 ] or region-level [ 48 , 65 , 61 , 62 , 9 ] representation learning have emerged . Our work is more related to region-level representation learning but substantially different from this line of research in the following aspects : 1 ) they still largely pre-train on object-centric ImageNet while we pre-train on non-iconic scene images , 2 ) they align pre-training specifically for dense prediction downstream tasks while we target at more general-purpose representation learning that improves performance in both dense prediction and classification tasks , 3 ) their randomly cropped local regions do not contain the explicit object notion as ours , and 4 ) they only rely on intra-image transformations ( e.g. , random cropping ) to construct corresponding positive pairs from the same image while we leverage the discovered high-level semantic correspondence to construct positive pairs across images . There are also a few prior attempts [ 18 , 3 , 16 ] for self-supervised learning on non-curated scene images . As opposed to our work , most of them consider larger models and datasets to explore the limit of current self-supervised learning methods without further considering the object-level information residing in scene images . Visual correspondence . Visual correspondence aims at finding pairwise pixels or regions across images that result from the same scene [ 67 ] , which can be regarded as similarity learning of visual descriptors among matched points or patches . While early efforts learn dense correspondence with labeled data [ 21 , 68 , 29 , 55 , 41 ] , some recent works learn the similarity between the parts or landmarks of the data in an unsupervised manner [ 52 , 51 ] . Our work substantially differs from this line of research from original intention . Previous works aim at accurately detecting all correspondence given two images , whereas our work focuses on retrieving high-quality correspondence to improve representation learning . < latexit sha1_base64= '' 5uEXhRHmlsZUMlLuTEklpEFlXBE= '' > AAAB6XicbVDLSgNBEOyNrxhfUY9eBoPoKexKQL0F9OAxinlAsoTZSW8yZHZ2mZkNhJA/8OJBEa/+kTf/xkmyB00saCiquunuChLBtXHdbye3tr6xuZXfLuzs7u0fFA+PGjpOFcM6i0WsWgHVKLjEuuFGYCtRSKNAYDMY3s785giV5rF8MuME/Yj2JQ85o8ZKj6PzbrHklt05yCrxMlKCDLVu8avTi1kaoTRMUK3bnpsYf0KV4UzgtNBJNSaUDWkf25ZKGqH2J/NLp+TMKj0SxsqWNGSu/p6Y0EjrcRTYzoiagV72ZuJ/Xjs14bU/4TJJDUq2WBSmgpiYzN4mPa6QGTG2hDLF7a2EDaiizNhwCjYEb/nlVdK4LHuV8s1DpVS9y+LIwwmcwgV4cAVVuIca1IFBCM/wCm/O0Hlx3p2PRWvOyWaO4Q+czx9IMI05 < /latexit > < latexit sha1_base64= '' Ii91hgWGVTQhpT9DsAPPeeou2Cw= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQL0F9OAxAfOAZAmzk95kzOzsMjMbCCFf4MWDIl79JG/+jZNkD5pY0FBUddPdFSSCa+O6305uY3Nreye/W9jbPzg8Kh6fNHWcKoYNFotYtQOqUXCJDcONwHaikEaBwFYwupv7rTEqzWP5aCYJ+hEdSB5yRo2V6uNeseSW3QXIOvEyUoIMtV7xq9uPWRqhNExQrTuemxh/SpXhTOCs0E01JpSN6AA7lkoaofani0Nn5MIqfRLGypY0ZKH+npjSSOtJFNjOiJqhXvXm4n9eJzXhjT/lMkkNSrZcFKaCmJjMvyZ9rpAZMbGEMsXtrYQNqaLM2GwKNgRv9eV10rwqe5Xybb1Sqt5nceThDM7hEjy4hio8QA0awADhGV7hzXlyXpx352PZmnOymVP4A+fzB+erjQg= < /latexit > < latexit sha1_base64= '' 4TCA4BNonmNCsppdstdpWECVqBg= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQL0F9OAxAfOAZAmzk95kzOzsMjMrhpAv8OJBEa9+kjf/xkmyB00saCiquunuChLBtXHdbye3tr6xuZXfLuzs7u0fFA+PmjpOFcMGi0Ws2gHVKLjEhuFGYDtRSKNAYCsY3cz81iMqzWN5b8YJ+hEdSB5yRo2V6k+9Ysktu3OQVeJlpAQZar3iV7cfszRCaZigWnc8NzH+hCrDmcBpoZtqTCgb0QF2LJU0Qu1P5odOyZlV+iSMlS1pyFz9PTGhkdbjKLCdETVDvezNxP+8TmrCK3/CZZIalGyxKEwFMTGZfU36XCEzYmwJZYrbWwkbUkWZsdkUbAje8surpHlR9irl63qlVL3N4sjDCZzCOXhwCVW4gxo0gAHCM7zCm/PgvDjvzseiNedkM8fwB87nD+qzjQo= < /latexit > 3 Methodology . We propose a new multi-stage self-supervised learning framework , i.e. , ORL , for object-level representation learning from scene images . ORL extends the existing image-level contrastive learning framework to object level by leveraging priors from image-level instance discrimination . The overall pipeline of ORL is illustrated in Figure 2 . It contains three stages : image-level pre-training , correspondence discovery , and object-level pre-training . We detail each stage as follows . 3.1 ORL pipeline . Preliminary : Contrastive learning . Our pipeline contains several contrastive learning modules in Stage 1 and 3 . Without loss of generality , we consider BYOL [ 19 ] as our basic contrastive learning module . BYOL uses two neural networks : the online network fθ ( x ) and the target network gξ ( x ) . The target network provides the regression target to train the online network while its weights ξ are updated by an exponential moving average of the online parameters θ with a decay rate τ ∈ [ 0 , 1 ] following BYOL . Given two input images x1 and x2 , the loss function is defined as : L ( x1 , x2 ) , ‖fθ ( x1 ) − gξ ( x2 ) ‖22 , ( 1 ) We name it an “ intra- ” version of BYOL if x1 and x2 are two augmented views from the same image , otherwise an “ inter- ” one . Stage 1 : Image-level pre-training . The foremost stage is to obtain an unsupervised pre-trained model from image-level tasks . As shown in Figure 2 Stage 1 , given two augmented views v and v′ from the same input image x , we pre-train the network following the loss function Limage = L ( v , v′ ) , constituting a standard image-level BYOL pre-training . This stage can be freely replaced with other image-level contrastive learning methods . We adopt BYOL here for its simplicity and effectiveness . Stage 2 : Correspondence discovery . We employ the pre-trained image-level contrastive learning model in Stage 1 to mine object-level correspondence for the whole dataset . As shown in Figure 2 Stage 2 , the overall discovery process comprises three steps . ( i ) Image-level nearest-neighbor retrieval . Specifically , for each query image x in the training set D , we first retrieve its top K nearest neighbors Nk , k = 1 , ... , K , by cosine distance in the embedding space using the features learned from the first stage to form image-level pairs that contain similar visual context . ( ii ) Region-of-interest ( RoI ) generation . To generate object-based RoIs , we apply unsupervised region proposal algorithms , e.g. , selective search [ 56 ] , for each image in the pair . Considering the redundancy of generated proposals ( each image can have thousands of proposals ) , we filter certain number of them with some pre-defined thresholds2 including the minimal scale , the range of aspect ratio , and the maximal intersection-over-union ( IoU ) among the filtered boxes . After the filtering operation , we select the top 100 proposals ranked with objectiveness as the candidate RoI set for subsequent RoI pair retrieval . To extract features with the equally-sized input that is compatible with the backbone , we crop and resize each RoI to 224× 224 . Note that even top RoIs ranked with objectiveness are still very noisy , containing a large proportion of non-object regions . ( iii ) Top-ranked RoI pair retrieval . For each query RoI from x , we compute its cosine similarity in the embedding space with all RoIs from its nearest-neighbor image Nk using the features learned from Stage 1 again . Within the calculated cosine similarity matrix Mk ∈ R100×100 , we retrieve top-ranked N RoI pairs to construct the set of object-level corresponding pairs { Bnk } , where n = { 1 , ... , N } . These high-response corresponding regions are almost objects or object parts . Finally , we save the nearest-neighbor image id and bounding box coordinate information of each corresponding pair . Stage 3 : Object-level pre-training . With the corresponding inter-image RoI ( inter-RoI ) pairs obtained in Stage 2 , we perform object-level representation learning following the BYOL framework as shown in Figure 2 Stage 3 . Specifically , given an input image x , we first randomly select one nearest-neighbor imageNk to obtain the corresponding set of inter-RoI pairs { Bnk } . We then randomly select one inter-RoI pair Bnk as a positive pair . With the bounding box coordinate stored in Bnk , we crop the corresponding inter-RoIs from x and Nk , respectively , and resize each patch to 96×96 , constituting two patches p1 and p2 . We feed the two patches to the online network and target network separately to compute the loss Linter-RoI = L ( p1 , p2 ) . To make full use of discovered objects , we introduce the intra-RoI contrastive learning via augmenting object patches . Specifically , we randomly select one filtered bounding box from x obtained in Stage 2 , and spatially jitter the box around its original location with the following operations3 : ( i ) a random box center shifting within 50 % of its width and height , ( ii ) a random area scaling between 50 % and 200 % of the original box , and ( iii ) a random aspect ratio between 1/2 and 2/1 . Similarly , we crop the two intra-RoIs p and p′ , and resize each patch to 96×96 for forward propagation to compute the loss Lintra-RoI = L ( p , p′ ) . The diverse spatial jittering of the bounding box encourages the network to preserve common object information and disregard the background , thus further improving the localization ability . We keep the two original global views in BYOL as well since they preserve the global image-level information compared with the local patches . The final loss for our ORL can thus be formulated as : LORL = λ1Limage + λ2Lintra-RoI + λ3Linter-RoI , ( 2 ) where λ1 , λ2 , λ3 are the loss weights to balance each term . We set all loss weights to 1 by default . Following BYOL , we also compute the symmetric loss L̃ORL by separately feeding v′ , p′ , p2 to the online network and v , p , p1 to the target network .

Summary: This paper aims to learn object-centric self-supervised representation from non-iconic scene images. In addition to the standard self-supervised objective (e.g., BYOL), the paper suggests attracting the object patch with the (i) nearby patch of the chosen object and (ii) similar objects in different images. For (ii), the paper uses the similarity score of the standard image-level self-supervised model. This additional training signal from similar object patches improves the transfer performance of the learned representation on classification and detection/segmentation tasks.