Unsupervised Object-Level Representation Learning from Scene Images

1 Introduction . Unsupervised visual representation learning aims at obtaining transferable features with abundant unlabeled data . Recent self-supervised learning ( SSL ) methods based on contrastive learning [ 60 , 22 , 37 , 5 , 19 , 4 , 7 ] have largely narrowed the gap and even surpassed the supervised counterpart on a number of downstream tasks [ 30 , 49 , 15 , 47 , 35 , 23 ] . These methods build upon the instance discrimination task that maximizes the agreement between different data-augmented views of the same image . Despite their success , current SSL methods are primarily pre-trained on the unlabeled ImageNet [ 8 ] dataset that contains iconic images with single object as shown in Figure 1 ( a ) . The underlying object-centric constraint of ImageNet makes it hard to be applied in real world scenarios where more complex scene images with multiple objects are available . Meanwhile , naïvely adopting the off-the-shelf contrastive learning methods on scene images introduces inconsistent learning signals since random crops of the same image may correspond to different objects as shown in Figure 1 ( b ) . Indeed , it has been shown that current contrastive learning methods tend to struggle on more complex scene datasets [ 19 , 50 , 34 , 58 ] like COCO [ 33 ] or Places365 [ 72 ] . Therefore , it is imperative to design an effective object-level representation learning paradigm as illustrated in Figure 1 ( c ) to harness massive unlabeled scene images in the wild . 1Project page : https : //www.mmlab-ntu.com/project/orl/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we are interested in going beyond ImageNet to obtain better representations on noniconic images . Apparently , it is challenging to learn representations from scene-level images since they are entangled with many concepts including structures , objects , backgrounds and relationships . It remains an open question how to take advantages of spatial information of multiple objects naturally residing in the scene images when no object annotations are available , let alone further deriving object-level correspondence to construct positive object-instance pairs . To tackle these challenges , we introduce a novel object-level unsupervised representation learning framework tailored for scene images . Our framework is based on a key insight of the current contrastive learning methods : they can implicitly group different images with similar visual concepts together even though they are explicitly optimized to group different views of the same image . This phenomenon reveals that image-level contrastive learning has already induced a latent space with rich visual concepts . Though the latent space usually entangles other scene concepts like structures , backgrounds and relationships , it will be useful for object discovery if appropriately deployed . Through computing the similarity of sampled regions between k-nearest-neighbor ( KNN ) images , we conclude two observations : 1 ) image-level contrastive learning encodes priors for region correspondence discovery across images ; 2 ) high-response regions are usually objects or object parts . Based on the observation above , we propose a multi-stage framework for unsupervised object-level representation learning . Specifically , we first extract potential object-based regions in scene images using the unsupervised region proposal algorithms ( e.g. , selective search [ 56 ] ) . We then propose a region correspondence generation scheme to leverage the off-the-shelf image-level contrastive learning pre-trained model to discover corresponding object-instance pairs for the proposed regions in the embedding space . Finally , we use the obtained object-instance pairs to construct positive sample pairs for object-level representation learning . Figure 1 ( d ) shows several cross-image object-instance pairs discovered by our framework on COCO dataset using the latent prior of BYOL [ 19 ] , the state-of-the-art image-level contrastive learning method . The discovered inter-corresponding pairs substantially provide diverse intra-class variances at the object-instance level to aid object-level representation learning . Overall , our main contributions are summarized as follows : 1 ) We observe that existing image-level contrastive learning methods have priors to discover objectlevel correspondence across images . We leverage this prior for the first time for unsupervised cross-image object-level correspondence discovery . 2 ) With the obtained correspondence , we introduce a novel multi-stage self-supervised learning pipeline , termed as ORL , for object-level representation learning from scene images , going beyond object-centric ImageNet . 3 ) We contribute the first study for object-level SSL . ORL substantially outperforms image-level contrastive learning approaches pre-trained on COCO dataset ( ∼118k images with labels discarded ) , setting a new state of the art on this challenging dataset that contains diverse scenes in the wild . The COCO pre-trained ORL even surpasses supervised ImageNet pre-training on several considered downstream tasks . When SSL is conducted on a larger “ COCO+ ” dataset ( COCO train2017 set plus COCO unlabeled2017 set , ∼241k images in total ) , ORL further improves the performance , demonstrating its potential to benefit from more unlabeled scene data . 2 Related work . Self-supervised learning . Self-supervised learning builds unsupervised representations by exploiting the internal priors or structures of data in the form of a pretext task . A wide range of pretext tasks have been proposed in the past few years . Examples include patch context prediction [ 10 ] , jigsaw puzzles [ 39 ] , inpainting [ 43 ] , colorization [ 31 , 70 ] , cross-channel prediction [ 71 ] , visual primitive counting [ 40 ] , and rotation prediction [ 14 ] . Although good representations emerge with these pretext tasks , they are prone to lose generality due to their hand-crafted nature . Recently , contrastive learning [ 20 ] that performs instance discrimination [ 60 , 22 , 37 , 5 , 19 , 4 , 7 ] has shown great potential in this field , largely narrowing the gap to fully supervised learning . The core idea of contrastive learning is to gather positive pairs and separate negative pairs in the embedding space . A positive pair is usually formed with two transformed views of the same image while the negative pairs are formed with different images . Typically , contrastive learning methods require a large number of negative samples to avoid mode collapse . These samples can be maintained within a mini-batch [ 42 , 27 , 66 , 26 , 2 , 5 ] , a memory bank [ 60 , 53 , 74 , 37 ] or a queue [ 22 , 6 ] . BYOL [ 19 ] and SwAV [ 4 ] further remove the necessity of involving negative pairs . BYOL directly predicts the features of one view from another view , while SwAV predicts the cluster assignments between multiple views of the same image . Despite their improved performance , the existing image-level contrastive learning methods are largely confined to the underlying object-centric bias of ImageNet . More recently , a group of works that perform pixel-level [ 45 , 58 , 64 , 50 , 34 , 25 ] or region-level [ 48 , 65 , 61 , 62 , 9 ] representation learning have emerged . Our work is more related to region-level representation learning but substantially different from this line of research in the following aspects : 1 ) they still largely pre-train on object-centric ImageNet while we pre-train on non-iconic scene images , 2 ) they align pre-training specifically for dense prediction downstream tasks while we target at more general-purpose representation learning that improves performance in both dense prediction and classification tasks , 3 ) their randomly cropped local regions do not contain the explicit object notion as ours , and 4 ) they only rely on intra-image transformations ( e.g. , random cropping ) to construct corresponding positive pairs from the same image while we leverage the discovered high-level semantic correspondence to construct positive pairs across images . There are also a few prior attempts [ 18 , 3 , 16 ] for self-supervised learning on non-curated scene images . As opposed to our work , most of them consider larger models and datasets to explore the limit of current self-supervised learning methods without further considering the object-level information residing in scene images . Visual correspondence . Visual correspondence aims at finding pairwise pixels or regions across images that result from the same scene [ 67 ] , which can be regarded as similarity learning of visual descriptors among matched points or patches . While early efforts learn dense correspondence with labeled data [ 21 , 68 , 29 , 55 , 41 ] , some recent works learn the similarity between the parts or landmarks of the data in an unsupervised manner [ 52 , 51 ] . Our work substantially differs from this line of research from original intention . Previous works aim at accurately detecting all correspondence given two images , whereas our work focuses on retrieving high-quality correspondence to improve representation learning . < latexit sha1_base64= '' 5uEXhRHmlsZUMlLuTEklpEFlXBE= '' > AAAB6XicbVDLSgNBEOyNrxhfUY9eBoPoKexKQL0F9OAxinlAsoTZSW8yZHZ2mZkNhJA/8OJBEa/+kTf/xkmyB00saCiquunuChLBtXHdbye3tr6xuZXfLuzs7u0fFA+PGjpOFcM6i0WsWgHVKLjEuuFGYCtRSKNAYDMY3s785giV5rF8MuME/Yj2JQ85o8ZKj6PzbrHklt05yCrxMlKCDLVu8avTi1kaoTRMUK3bnpsYf0KV4UzgtNBJNSaUDWkf25ZKGqH2J/NLp+TMKj0SxsqWNGSu/p6Y0EjrcRTYzoiagV72ZuJ/Xjs14bU/4TJJDUq2WBSmgpiYzN4mPa6QGTG2hDLF7a2EDaiizNhwCjYEb/nlVdK4LHuV8s1DpVS9y+LIwwmcwgV4cAVVuIca1IFBCM/wCm/O0Hlx3p2PRWvOyWaO4Q+czx9IMI05 < /latexit > < latexit sha1_base64= '' Ii91hgWGVTQhpT9DsAPPeeou2Cw= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQL0F9OAxAfOAZAmzk95kzOzsMjMbCCFf4MWDIl79JG/+jZNkD5pY0FBUddPdFSSCa+O6305uY3Nreye/W9jbPzg8Kh6fNHWcKoYNFotYtQOqUXCJDcONwHaikEaBwFYwupv7rTEqzWP5aCYJ+hEdSB5yRo2V6uNeseSW3QXIOvEyUoIMtV7xq9uPWRqhNExQrTuemxh/SpXhTOCs0E01JpSN6AA7lkoaofani0Nn5MIqfRLGypY0ZKH+npjSSOtJFNjOiJqhXvXm4n9eJzXhjT/lMkkNSrZcFKaCmJjMvyZ9rpAZMbGEMsXtrYQNqaLM2GwKNgRv9eV10rwqe5Xybb1Sqt5nceThDM7hEjy4hio8QA0awADhGV7hzXlyXpx352PZmnOymVP4A+fzB+erjQg= < /latexit > < latexit sha1_base64= '' 4TCA4BNonmNCsppdstdpWECVqBg= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQL0F9OAxAfOAZAmzk95kzOzsMjMrhpAv8OJBEa9+kjf/xkmyB00saCiquunuChLBtXHdbye3tr6xuZXfLuzs7u0fFA+PmjpOFcMGi0Ws2gHVKLjEhuFGYDtRSKNAYCsY3cz81iMqzWN5b8YJ+hEdSB5yRo2V6k+9Ysktu3OQVeJlpAQZar3iV7cfszRCaZigWnc8NzH+hCrDmcBpoZtqTCgb0QF2LJU0Qu1P5odOyZlV+iSMlS1pyFz9PTGhkdbjKLCdETVDvezNxP+8TmrCK3/CZZIalGyxKEwFMTGZfU36XCEzYmwJZYrbWwkbUkWZsdkUbAje8surpHlR9irl63qlVL3N4sjDCZzCOXhwCVW4gxo0gAHCM7zCm/PgvDjvzseiNedkM8fwB87nD+qzjQo= < /latexit > 3 Methodology . We propose a new multi-stage self-supervised learning framework , i.e. , ORL , for object-level representation learning from scene images . ORL extends the existing image-level contrastive learning framework to object level by leveraging priors from image-level instance discrimination . The overall pipeline of ORL is illustrated in Figure 2 . It contains three stages : image-level pre-training , correspondence discovery , and object-level pre-training . We detail each stage as follows . 3.1 ORL pipeline . Preliminary : Contrastive learning . Our pipeline contains several contrastive learning modules in Stage 1 and 3 . Without loss of generality , we consider BYOL [ 19 ] as our basic contrastive learning module . BYOL uses two neural networks : the online network fθ ( x ) and the target network gξ ( x ) . The target network provides the regression target to train the online network while its weights ξ are updated by an exponential moving average of the online parameters θ with a decay rate τ ∈ [ 0 , 1 ] following BYOL . Given two input images x1 and x2 , the loss function is defined as : L ( x1 , x2 ) , ‖fθ ( x1 ) − gξ ( x2 ) ‖22 , ( 1 ) We name it an “ intra- ” version of BYOL if x1 and x2 are two augmented views from the same image , otherwise an “ inter- ” one . Stage 1 : Image-level pre-training . The foremost stage is to obtain an unsupervised pre-trained model from image-level tasks . As shown in Figure 2 Stage 1 , given two augmented views v and v′ from the same input image x , we pre-train the network following the loss function Limage = L ( v , v′ ) , constituting a standard image-level BYOL pre-training . This stage can be freely replaced with other image-level contrastive learning methods . We adopt BYOL here for its simplicity and effectiveness . Stage 2 : Correspondence discovery . We employ the pre-trained image-level contrastive learning model in Stage 1 to mine object-level correspondence for the whole dataset . As shown in Figure 2 Stage 2 , the overall discovery process comprises three steps . ( i ) Image-level nearest-neighbor retrieval . Specifically , for each query image x in the training set D , we first retrieve its top K nearest neighbors Nk , k = 1 , ... , K , by cosine distance in the embedding space using the features learned from the first stage to form image-level pairs that contain similar visual context . ( ii ) Region-of-interest ( RoI ) generation . To generate object-based RoIs , we apply unsupervised region proposal algorithms , e.g. , selective search [ 56 ] , for each image in the pair . Considering the redundancy of generated proposals ( each image can have thousands of proposals ) , we filter certain number of them with some pre-defined thresholds2 including the minimal scale , the range of aspect ratio , and the maximal intersection-over-union ( IoU ) among the filtered boxes . After the filtering operation , we select the top 100 proposals ranked with objectiveness as the candidate RoI set for subsequent RoI pair retrieval . To extract features with the equally-sized input that is compatible with the backbone , we crop and resize each RoI to 224× 224 . Note that even top RoIs ranked with objectiveness are still very noisy , containing a large proportion of non-object regions . ( iii ) Top-ranked RoI pair retrieval . For each query RoI from x , we compute its cosine similarity in the embedding space with all RoIs from its nearest-neighbor image Nk using the features learned from Stage 1 again . Within the calculated cosine similarity matrix Mk ∈ R100×100 , we retrieve top-ranked N RoI pairs to construct the set of object-level corresponding pairs { Bnk } , where n = { 1 , ... , N } . These high-response corresponding regions are almost objects or object parts . Finally , we save the nearest-neighbor image id and bounding box coordinate information of each corresponding pair . Stage 3 : Object-level pre-training . With the corresponding inter-image RoI ( inter-RoI ) pairs obtained in Stage 2 , we perform object-level representation learning following the BYOL framework as shown in Figure 2 Stage 3 . Specifically , given an input image x , we first randomly select one nearest-neighbor imageNk to obtain the corresponding set of inter-RoI pairs { Bnk } . We then randomly select one inter-RoI pair Bnk as a positive pair . With the bounding box coordinate stored in Bnk , we crop the corresponding inter-RoIs from x and Nk , respectively , and resize each patch to 96×96 , constituting two patches p1 and p2 . We feed the two patches to the online network and target network separately to compute the loss Linter-RoI = L ( p1 , p2 ) . To make full use of discovered objects , we introduce the intra-RoI contrastive learning via augmenting object patches . Specifically , we randomly select one filtered bounding box from x obtained in Stage 2 , and spatially jitter the box around its original location with the following operations3 : ( i ) a random box center shifting within 50 % of its width and height , ( ii ) a random area scaling between 50 % and 200 % of the original box , and ( iii ) a random aspect ratio between 1/2 and 2/1 . Similarly , we crop the two intra-RoIs p and p′ , and resize each patch to 96×96 for forward propagation to compute the loss Lintra-RoI = L ( p , p′ ) . The diverse spatial jittering of the bounding box encourages the network to preserve common object information and disregard the background , thus further improving the localization ability . We keep the two original global views in BYOL as well since they preserve the global image-level information compared with the local patches . The final loss for our ORL can thus be formulated as : LORL = λ1Limage + λ2Lintra-RoI + λ3Linter-RoI , ( 2 ) where λ1 , λ2 , λ3 are the loss weights to balance each term . We set all loss weights to 1 by default . Following BYOL , we also compute the symmetric loss L̃ORL by separately feeding v′ , p′ , p2 to the online network and v , p , p1 to the target network .

Summary: This work proposes a technique to adapt self-supervised contrastive learning to scene images where multiple objects are present in each image. Specifically, the proposed technique has three stages: 1. image-level pre-training with standard contrastive learning; 2. Finding object pairs images (crops) using selective search proposals and pre-trained features from stage-1; and 3. Doing contrastive learning with object image pairs. Several experimental comparisons demonstrate consistent improvements w.r.t. baselines that does only image-level contrastive learning.