Object-aware Contrastive Learning for Debiased Scene Representation

1 Introduction . Self-supervised learning of visual representations from unlabeled images is a fundamental task of machine learning , which establishes various applications including object recognition [ 1 , 2 ] , reinforcement learning [ 3 , 4 ] , out-of-distribution detection [ 5 , 6 ] , and multimodal learning [ 7 , 8 ] . Recently , contrastive learning [ 1 , 2 , 9–15 ] has shown remarkable advances along this line . The idea is to learn invariant representations by attracting the different views ( e.g. , augmentations ) of the same instance ( i.e. , positives ) while contrasting different instances ( i.e. , negatives ) .2 Despite the success of contrastive learning on various downstream tasks [ 16 ] , they still suffer from the generalization issue due to the unique features of the training datasets [ 17–19 ] or the choice of data augmentations [ 19–21 ] . In particular , the co-occurrence of different objects and background in randomly cropped patches ( i.e. , positives ) leads the model to suffer from the scene bias . For example , Figure 1a presents two types of the scene bias : the positive pairs contain different objects ( e.g. , giraffe and zebra ) , and the patches contain adjacent object and background ( e.g. , zebra and safari ) . Specifically , the co-occurrence of different objects is called contextual bias [ 22 ] , and that of object and background is called background bias [ 23 ] . Attracting the patches in contrastive learning ∗Equal contribution 1Code is available at https : //github.com/alinlab/object-aware-contrastive . 2Some recent works ( e.g. , [ 14 , 15 ] ) attract the positives without contrasting the negatives . While we mainly focus on contrastive learning with negatives , our method is also applicable to the positive-only methods . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . makes the features of correlated objects and background indistinguishable , which may harm their generalization ( Figure 1b ) because of being prone to biases ( Figure 1c ) . Contribution . We develop a novel object-aware contrastive learning framework that mitigates the scene bias and improves the generalization of learned representation . The key to success is the proposed contrastive class activation map ( ContraCAM ) , a simple yet effective self-supervised object localization method by contrasting other images to find the most discriminate regions in the image . We leverage the ContraCAM to create new types of positives and negatives . First , we introduce two data augmentations for constructing the positive sample-pairs of contrastive learning : object-aware random crop and background mixup that reduce contextual and background biases , respectively . Second , by equipping ContraCAM with an iterative refinement procedure , we extend it to detect multiple objects and entire shapes , which allows us to generate masked images as effective negatives . We demonstrate that the proposed method can improve two representative contrastive ( or positiveonly ) representation learning schemes , MoCov2 [ 28 ] and BYOL [ 14 ] , by reducing contextual and background biases as well as learning object-centric representation . In particular , we improve : • The representation learning under multi-object images , evaluated on the COCO [ 25 ] dataset , boosting the performance on the downstream tasks , e.g. , classification and detection . • The generalizability of the learned representation on the background shifts , i.e. , objects appear in the unusual background ( e.g. , fish on the ground ) , evaluated on the Background Challenge [ 23 ] . • The generalizability of the learned representation on the distribution shifts , particularly for the shape-biased , e.g. , ImageNet-Sketch [ 29 ] , and corrupted , e.g. , ImageNet-C [ 30 ] datasets . Furthermore , ContraCAM shows comparable results with the state-of-the-art unsupervised localization method ( and also with the supervised classifier CAM ) while being simple . 2 Object-aware Contrastive Learning . We first briefly review contrastive learning in Section 2.1 . We then introduce our object localization and debiased contrastive learning methods in Section 2.2 and Section 2.3 , respectively . 2.1 Contrastive learning . Contrastive self-supervised learning aims to learn an encoder f ( · ) that extracts a useful representation from an unlabeled image x by attracting similar sample x+ ( i.e. , positives ) and dispelling dissimilar samples { x−i } ( i.e. , negatives ) . In particular , instance discrimination [ 10 ] defines the same samples of different data augmentations ( e.g. , random crop ) as the positives and different samples as negatives . Formally , contrastive learning maximizes the contrastive score : scon ( x ; x + , { x−n } ) : = log exp ( sim ( z ( x ) , z̄ ( x+ ) ) /τ ) exp ( sim ( z ( x ) , z̄ ( x+ ) ) /τ ) + ∑ x−n exp ( sim ( z ( x ) , z̄ ( x−n ) ) /τ ) , ( 1 ) where z ( · ) and z̄ ( · ) are the output and target functions wrapping the representation f ( x ) for use , sim ( · , · ) denotes the cosine similarity , and τ is a temperature hyperparameter . The specific form of z ( · ) , z̄ ( · ) depends on the method . For example , MoCov2 [ 28 ] sets z ( · ) = g ( f ( · ) ) , z̄ ( · ) = gm ( fm ( · ) ) where g ( · ) is a projector network to indirectly match the feature f ( x ) and fm ( · ) , gm ( · ) are the momentum version of the encoder and projectors . On the other hand , BYOL [ 14 ] sets z ( · ) = h ( g ( f ( · ) ) ) , z̄ ( · ) = gm ( fm ( · ) ) , where h ( · ) is an additional predictor network to avoid collapse of the features because it only maximizes the similarity score ssim ( x ; x+ ) : = sim ( z ( x ) , z̄ ( x+ ) ) [ 14 , 15 ] . Scene bias in contrastive learning . Despite the success of contrastive learning , they often suffer from the scene bias : entangling representations of co-occurring ( but different ) objects , i.e. , contextual bias [ 22 ] , or adjacent object and background , i.e. , background bias [ 23 ] , by attracting the randomly cropped patches reflecting the correlations ( Figure 1a ) . The scene bias harms the performance ( Figure 1b ) and generalization of the learned representations on distribution shifts ( Figure 1c ) . To tackle the issue , we propose object-aware data augmentations for debiased contrastive learning ( Section 2.3 ) utilizing the object locations inferred from the contrastively trained models ( Section 2.2 ) . 2.2 ContraCAM : Unsupervised object localization via contrastive learning . We aim to find the most discriminative region in an image , such as objects for scene images , compared to the other images . To this end , we extend the ( gradient-based ) class activation map ( CAM ) [ 31 , 32 ] , originally used to find the salient regions for the prediction of classifiers . Our proposed method , contrastive class activation map ( ContraCAM ) , has two differences from the classifier CAM . First , we use the contrastive score instead of the softmax probability . Second , we discard the negative signals from the similar objects in the negative batch since they cancel out the positive signals and hinder the localization , which is crucial as shown in Table 1 and Appendix C.1 ) . Following the classifier CAM , we define the saliency map as the weighted sum of spatial activations ( e.g. , penultimate feature before pooling ) , where the weight of each activation is given by the importance , the sum of gradients , of the activation for the score function . Formally , let A : = [ Akij ] be a spatial activation of an image x where 1 ≤ i ≤ H , 1 ≤ j ≤ W , 1 ≤ k ≤ K denote the index of row , column , and channel , and H , W , K denote the height , width , and channel size of the activation . Given a batch of samples B , we define the score function of the sample x as the contrastive score scon in Eq . ( 1 ) using the sample x itself as a positive3 and the remaining samples B \ x as negatives . Then , the weight of the k-th activation αk and the CAM mask CAM : = [ CAMij ] ∈ [ 0 , 1 ] H×W are : CAMij = Normalize ( ReLU ( ∑ k αkA k ij ) ) , αk = ReLU 1 HW ∑ i , j ∂scon ( x ; x , B \ x ) ∂Aki , j , ( 2 ) where Normalize ( x ) : = x−min xmax x−min x is a normalization function that maps the elements to [ 0 , 1 ] . We highlight the differences from the classifier CAM with the red color . Note that the ReLU used to compute αk in Eq . ( 2 ) discards the negative signals . The negative signal removal trick also slightly improves the classifier CAM [ 33 ] but much effective for the ContraCAM . We further improve the ContraCAM to detect multiple objects and entire shapes with an iterative refinement procedure [ 34 ] : cover the salient regions of the image with the ( reverse of ) current CAM , predict new CAM from the masked image , and aggregate them ( see Figure 2 ) . It expands the CAM regions since the new CAM from the masked image detects the unmasked regions . Here , we additionally provide the masked images in the batch ( parellely computed ) as the negatives : they are better negatives by removing the possibly existing similar objects . Also , we use the original image x as the positive to highlight the undetected objects . Formally , let CAMt be the CAM of iteration t and CAMt : = [ CAMtij ] = [ maxl≤t CAM l ij ] be the aggregated CAM mask . Also , let x t be the image softly masked by the ( reverse of ) current aggregated mask , i.e. , xt : = ( 1− CAMt−1 ) x for t ≥ 2 3It does not affect the score but is defined for the notation consistency with the iterative extension . and x1 = x where denotes an element-wise product , and Bt : = { xtn } be the batch of the masked images . Then , we define the score function for iteration t as : stcon ( x ) : = scon ( x t ; x , ∪l≤t ( Bl \ xl ) ) . ( 3 ) We substitute the contrastive score scon in Eq . ( 2 ) with the stcon in Eq . ( 3 ) to compute the CAM of iteration t , and use the final aggregated mask after T iterations . We remark that the CAM results are not sensitive to the number of iterations T if it is large enough ; CAM converges to the stationary value since soft masking xt regularizes the CAM not to be keep expanded ( see Appendix C.2 ) . We provide the pseudo-code of the entire Iterative ContraCAM procedure in Appendix A . Note that contrastive learning was known to be ineffective at localizing objects [ 35 ] with standard saliency methods ( using a classifier on top of the learned representation ) since attracting the randomly cropped patches makes the model look at the entire scene . To our best knowledge , we are the first to extend the CAM for the self-supervised setting , relaxing the assumption of class labels . Selvaraju et al . [ 36 ] considered CAM for contrastive learning , but their purpose was to regularize CAM to be similar to the ground-truth masks ( or predicted by pre-trained models ) and used the similarity of the image and the masked image ( by ground-truth masks ) as the score function of CAM .

Summary: This paper proposes a new learning scheme for representation learning. Using class activation maps, augmentations are obtained for contrastive learning that are aware of the objects in the image. Results include results on multiple classification, segmentation, and detection datasets.   Note on 10 sept 2021: rebuttal has been read and discussed with colleague reviewers and AC