Robust Contrastive Learning Using Negative Samples with Diminished Semantics

1 Introduction . Recent studies on self-supervised learning have shown great success in learning visual representations without human annotations . The gap between unsupervised and supervised learning has been progressively closed by contrastive learning [ 53 , 48 , 7 , 18 , 49 , 6 , 16 , 56 ] . In the meantime , CNNs trained in the supervised setting are known to learn correlations between labels and superfluous features such as local patches [ 4 , 3 ] , texture [ 14 ] , high-frequency components [ 51 ] , and even artificially added features [ 26 ] , which has raised concerns about deploying these models in a real scenario [ 30 , 15 ] . CNNs trained by contrastive learning methods are no exception [ 23 ] . In this paper , we propose to construct negative samples that only preserve non-semantic features . We show that using contrastive learning methods trained with these negative samples can mitigate these concerns . Contrastive learning methods exploit carefully designed augmentations to construct positive pairs and pull their representations together . These augmentations are crucial to contrastive learning [ 7 , 6 ] . A common assumption behind these augmentations is to preserve the semantics of the input images while perturbing other superficial signals . This inspires us to generate negative samples and inject additional implicit biases on the visual features learned by the models . Specifically , we utilize 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . augmentations that diminish the semantic features while keeping the undesired features such as texture . By pushing apart the representations of such negative samples and input images , the models are expected to rely more on the semantics of the images and less on superficial features . Inspired by the non-semantic features , we propose two methods to craft negative samples . The first method relies on texture synthesis tools from classic approaches [ 12 , 52 ] . It generates realistic texture images based on two patches extracted from input images , as shown in Figure 1 ( d ) . For each image in ImageNet , we generate its texture version and form a dataset which we call ImageNet-Texture . The second method constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image , as shown in Figure 1 ( e ) . Comparing the non-semantic negative samples with two semantic positive samples in Figure 1 ( b ) and Figure 1 ( c ) , the dog from the input image is still recognizable in the positive samples but hard to understand from negative samples . Instead , local statistics such as the fur and color of the dog are preserved in the negative samples . The generated non-semantic samples can be readily used with existing contrastive learning methods that distinguish positive pairs from negative pairs such as MoCo [ 18 ] and SimCLR [ 7 ] . Despite their simplicity , we show that these non-semantic negative samples are actually harder than the standard negative samples used by these methods , which are inefficient at leveraging hard negatives [ 13 , 31 ] . Further , our negative pairs can also be used by contrastive learning methods that do not explicitly use negative samples , such as BYOL [ 16 ] . We evaluate our methods with two contrastive learning methods , MoCo [ 18 , 8 ] and BYOL [ 16 ] , on three datasets , ImageNet-100 , ImageNet-1K and STL10 . When using our proposed augmentations to generate negative samples and minimize their representation similarity to the input images , we notice a consistent improvement on the generalization performance over backbone methods [ 8 , 16 ] and previous negative example generation strategies [ 31 , 43 ] , especially under out-of-distribution ( OOD ) settings . We conduct a systematic analysis of how the shape-texture trade-off influences model performance based on the proposed ImageNet-Texture dataset . We control the penalty on similarities between the non-semantic negative examples and the query samples . This impacts the trade-off between using shape and texture features . We find that the relative importance of texture and shape features varies across different datasets . For example , shape bias benefits ImageNet-Sketch [ 50 ] more than the original ImageNet validation set . On the other hand , texture bias benefits finer-grained classification more , such as dog breed classification included in ImageNet . These results complement previous evidence showing the effectiveness of shape features in classifying 16 coarse classes [ 14 ] . Such preference for one feature over the other is also observed intra-dataset : the texture is more important for some classes such as dishrag and plaque . These observations make us question the relationship between shape and texture features as the implicitly necessary bias of CNNs and advocate for an adaptive combination of both when deploying the model in real scenarios . In summary : • We propose texture-based and patch-based augmentations to generate negative samples from input images , and show that these negative samples improve the generalization of contrastive learning . • We introduce the ImageNet-Texture dataset , which contains texture versions of ImageNet images generated by texture synthesis tools . • We provide fine-grained analysis on the shape-texture trade-off of CNNs , and show different scenarios when one is preferred over the other . 2 Negative Samples with Diminished Semantics . CNNs are apt to learn low level features such as texture under supervised settings [ 3 , 14 , 51 ] ; this has been recently witnessed under the contrastive learning setting as well [ 23 ] . To mitigate this problem , we propose two methods , texture-based and patch-based augmentations , to generate negative samples for contrastive learning . Texture-based augmentation generates realistic images based on texture synthesis and patch-based augmentation exploits more comprehensive local features by sampling patches from input images . By penalizing learned similarities between the representations of images and their non-semantic counterparts , the model is encouraged to rely less on the undesired features and focus more on the semantics . In practice , we find the two negative samples play similar roles and the patch-based method works slightly better . In this section , we start with an overview of contrastive learning and show how non-semantic negatives are used in these frameworks . Then we elaborate on the two approaches to generate negative samples with diminished semantics . 2.1 Contrastive learning with non-semantic negatives . Given an encoder network f and an image x , we denote the output of the network as z = f ( x ) . We use zi and zp to denote the representations of the query sample xi and a positive sample xp generated from the same input image with augmentations that preserve semantics . For contrastive learning methods like MoCo [ 18 ] and SimCLR [ 7 ] , zn denotes the representation of the standard negative sample xn extracted from the memory bank ( MoCo ) or other images in the current batch ( SimCLR ) . zns is the representation of the proposed negative sample xns which contains particular non-semantic features of the input image with the semantic part weakened . We extend the noise-contrastive estimation ( NCE ) loss as below : LNCE = − ∑ i∈I log exp ( zTi zp/τ ) exp ( zTi zp/τ ) + exp ( αzTi zns/τ ) + ∑ n∈N exp ( zTi zn/τ ) , ( 1 ) where τ is a temperature parameter and α is an additional scaling parameter for non-semantic negatives . A larger α implies a stronger penalty on the similarity between the representations of the query image and its non-semantic version . In Appendix B.1 we discuss other possible ways to apply α . Methods like BYOL [ 16 ] do not explicitly rely on negative samples . Nevertheless , BYOL adapts the loss to maximize the agreement of positive pairs . Therefore , we explicitly use the non-semantic negative sample with their loss to minimize its similarity to the query sample : LBYOL = ‖zi − zp‖ − α‖zi − zns‖ = 2− 2α− 2zTi zp + 2αzTi zns . ( 2 ) We overload α to be the parameter that controls the penalty on the similarity between the representations of input image and its non-semantic version under BYOL , with similar intention as MoCo and SimCLR above . To minimize either LNCE or LBYOL , the encoder must learn features from xi that are not contained in xns but shared with xp . 2.2 Texture-based negative sample generation . We use texture synthesis tools to generate negative samples . Texture synthesis aims to generate realistic images that preserve as much local structure as possible from an example image [ 19 , 11 , 42 ] . For instance , as shown in Figure 1 ( d ) , the texture of the input dog image preserves the fur and colors of the dog . Notably , in previous discussion of robustness [ 3 , 50 ] , such local structure has often been recognized as highly correlated with the labels yet superfluous to generalization . For example , under large domain shift due to lighting , motion , and even modality , the texture is more apt to change than the semantic features , such as the shape . Furthermore , CNNs trained on ImageNet are more likely to classify images based on the texture features rather than the shape features which are instead preferred by humans due to their transferability [ 14 ] . To encourage the model to rely more on the shape features , we propose a two-step method to generate the texture image of input images as the negative samples for contrastive learning . To be specific , we first sample two patches from given images as the input to the texture synthesis algorithms . One patch is extracted from the center of the image . This patch is expected to reflect the texture of the object according to the implicit bias contained in the ImageNet dataset that most of the objects are center-oriented in the images [ 2 ] . The other patch is extracted from a random location to reflect other possible textures of the image ( e.g . background , peripheral region of the object ) . In this work , we extract patches with size 96× 96 when image size allows , otherwise 48× 48 patches are extracted . Second , we adopt off-the-shelf texture synthesis algorithms [ 12 , 52 , 1 ] to generate texture images based on the two patches . These non-parametric algorithms iteratively sample pixels from given patches that share a similar neighborhood with the current pixel . Specifically , we use the open-source software built on these methods [ 12 , 52 , 1 ] with multi-threaded CPU support implemented in Rust 1 . For each sample in the ImageNet dataset , we generate one 224× 224 texture image to construct a dataset that has the same training and validation size as the ImageNet dataset . We call this dataset ImageNet-Texture . More examples can be found in Appendix A.1 . 2.3 Patch-based negative sample generation . To simulate the local information contained in the images [ 4 , 3 ] , we propose an efficient patch-based method to generate non-semantic images . Given an image and a patch size d , we sample patches of size d from ( d 224d e ) 2 non-overlapping random locations that lie entirely in the image . The patches are then tiled and cropped into 224 × 224 as negative samples . Compared with the texture-based method , this generation process takes negligible time , therefore it can be implemented as part of the data loading process in parallel with training . By doing so , each training sample can be paired with different negative samples generated from different patches every time it is used , compared with the two fixed patches selected when generating texture images . Different from the texture-based method that generates realistic images , the patch-based method generates images with artificial lines as shown in Figure 1e . One might be concerned with possible degenerate solution where the model outputs a low similarity whenever it detects the repeated sharp changes in the horizontal or vertical directions , which could be done with a single layer of convolution . However , interestingly , we find that the model does not find such a simple solution in practice . This is also noticed in a previous study where the image and its copy with a patch cut out are non-trivially distinguished by the model [ 34 ] . To mitigate this potential issue , we randomly sample patch size d from a prior distribution instead of using a fixed d in practice , which allows the model to look at texture at different scales .

Summary: This paper introduces non-semantic negative samples into self-supervised learning. These non-semantic samples force the network to reduce the reliance on texture-based features and focus on semantic ones, which are more general in different scenarios.  Experiments on several image classification datasets verify the robustness of the proposed method. In addition, texture-shape bias trade-off is also thoroughly analyzed.