Robust Contrastive Learning Using Negative Samples with Diminished Semantics
Year: 2022

Abstract:
Unsupervised learning has recently made exceptional progress because of the development of more effective contrastive learning methods. However, CNNs are prone to depend on low-level features that humans deem non-semantic. This dependency has been conjectured to induce a lack of robustness to image perturbations or domain shift. In this paper, we show that by generating carefully designed negative samples, contrastive learning can learn more robust representations with less dependence on such features. Contrastive learning utilizes positive pairs that preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples in a reversed way, where only the superfluous instead of the semantic features are preserved. We develop two methods, texture-based and patch-based augmentations, to generate negative samples. These samples achieve better generalization, especially under out-of-domain settings. We also analyze our method and the generated texture-based samples, showing that texture features are indispensable in classifying particular ImageNet classes and especially finer classes. We also show that model bias favors texture and shape features differently under different test settings. Our code, trained models, and ImageNet-Texture dataset can be found at https://github.com/ SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives.

heading: 1 Introduction
text: Recent studies on self-supervised learning have shown great success in learning visual representations without human annotations. The gap between unsupervised and supervised learning has been progressively closed by contrastive learning [53, 48, 7, 18, 49, 6, 16, 56]. In the meantime, CNNs trained in the supervised setting are known to learn correlations between labels and superfluous features such as local patches [4, 3], texture [14], high-frequency components [51], and even artificially added features [26], which has raised concerns about deploying these models in a real scenario [30, 15]. CNNs trained by contrastive learning methods are no exception [23]. In this paper, we propose to construct negative samples that only preserve non-semantic features. We show that using contrastive learning methods trained with these negative samples can mitigate these concerns.
Contrastive learning methods exploit carefully designed augmentations to construct positive pairs and pull their representations together. These augmentations are crucial to contrastive learning [7, 6]. A common assumption behind these augmentations is to preserve the semantics of the input images while perturbing other superficial signals. This inspires us to generate negative samples and inject additional implicit biases on the visual features learned by the models. Specifically, we utilize
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
augmentations that diminish the semantic features while keeping the undesired features such as texture. By pushing apart the representations of such negative samples and input images, the models are expected to rely more on the semantics of the images and less on superficial features.
Inspired by the non-semantic features, we propose two methods to craft negative samples. The first method relies on texture synthesis tools from classic approaches [12, 52]. It generates realistic texture images based on two patches extracted from input images, as shown in Figure 1(d). For each image in ImageNet, we generate its texture version and form a dataset which we call ImageNet-Texture. The second method constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image, as shown in Figure 1(e). Comparing the non-semantic negative samples with two semantic positive samples in Figure 1(b) and Figure 1(c), the dog from the input image is still recognizable in the positive samples but hard to understand from negative samples. Instead, local statistics such as the fur and color of the dog are preserved in the negative samples.
The generated non-semantic samples can be readily used with existing contrastive learning methods that distinguish positive pairs from negative pairs such as MoCo [18] and SimCLR [7]. Despite their simplicity, we show that these non-semantic negative samples are actually harder than the standard negative samples used by these methods, which are inefficient at leveraging hard negatives [13, 31]. Further, our negative pairs can also be used by contrastive learning methods that do not explicitly use negative samples, such as BYOL [16]. We evaluate our methods with two contrastive learning methods, MoCo [18, 8] and BYOL [16], on three datasets, ImageNet-100, ImageNet-1K and STL10. When using our proposed augmentations to generate negative samples and minimize their representation similarity to the input images, we notice a consistent improvement on the generalization performance over backbone methods [8, 16] and previous negative example generation strategies [31, 43], especially under out-of-distribution (OOD) settings.
We conduct a systematic analysis of how the shape-texture trade-off influences model performance based on the proposed ImageNet-Texture dataset. We control the penalty on similarities between the non-semantic negative examples and the query samples. This impacts the trade-off between using shape and texture features. We find that the relative importance of texture and shape features varies across different datasets. For example, shape bias benefits ImageNet-Sketch [50] more than the original ImageNet validation set. On the other hand, texture bias benefits finer-grained classification more, such as dog breed classification included in ImageNet. These results complement previous evidence showing the effectiveness of shape features in classifying 16 coarse classes [14]. Such preference for one feature over the other is also observed intra-dataset: the texture is more important for some classes such as dishrag and plaque. These observations make us question the relationship between shape and texture features as the implicitly necessary bias of CNNs and advocate for an adaptive combination of both when deploying the model in real scenarios. In summary:
• We propose texture-based and patch-based augmentations to generate negative samples from input images, and show that these negative samples improve the generalization of contrastive learning.
• We introduce the ImageNet-Texture dataset, which contains texture versions of ImageNet images generated by texture synthesis tools.
• We provide fine-grained analysis on the shape-texture trade-off of CNNs, and show different scenarios when one is preferred over the other.

heading: 2 Negative Samples with Diminished Semantics
text: CNNs are apt to learn low level features such as texture under supervised settings [3, 14, 51]; this has been recently witnessed under the contrastive learning setting as well [23]. To mitigate this problem, we propose two methods, texture-based and patch-based augmentations, to generate negative samples for contrastive learning. Texture-based augmentation generates realistic images based on texture synthesis and patch-based augmentation exploits more comprehensive local features by sampling patches from input images. By penalizing learned similarities between the representations of images and their non-semantic counterparts, the model is encouraged to rely less on the undesired features and focus more on the semantics. In practice, we find the two negative samples play similar roles and the patch-based method works slightly better. In this section, we start with an overview of contrastive learning and show how non-semantic negatives are used in these frameworks. Then we elaborate on the two approaches to generate negative samples with diminished semantics.

heading: 2.1 Contrastive learning with non-semantic negatives
text: Given an encoder network f and an image x, we denote the output of the network as z = f(x). We use zi and zp to denote the representations of the query sample xi and a positive sample xp generated from the same input image with augmentations that preserve semantics. For contrastive learning methods like MoCo [18] and SimCLR [7], zn denotes the representation of the standard negative sample xn extracted from the memory bank (MoCo) or other images in the current batch (SimCLR). zns is the representation of the proposed negative sample xns which contains particular non-semantic features of the input image with the semantic part weakened. We extend the noise-contrastive estimation (NCE) loss as below:
LNCE = − ∑ i∈I log exp
( zTi zp/τ ) exp ( zTi zp/τ ) + exp ( αzTi zns/τ ) + ∑ n∈N exp ( zTi zn/τ
) , (1) where τ is a temperature parameter and α is an additional scaling parameter for non-semantic negatives. A larger α implies a stronger penalty on the similarity between the representations of the query image and its non-semantic version. In Appendix B.1 we discuss other possible ways to apply α.
Methods like BYOL [16] do not explicitly rely on negative samples. Nevertheless, BYOL adapts the loss to maximize the agreement of positive pairs. Therefore, we explicitly use the non-semantic negative sample with their loss to minimize its similarity to the query sample:
LBYOL = ‖zi − zp‖ − α‖zi − zns‖ = 2− 2α− 2zTi zp + 2αzTi zns. (2)
We overload α to be the parameter that controls the penalty on the similarity between the representations of input image and its non-semantic version under BYOL, with similar intention as MoCo and SimCLR above. To minimize either LNCE or LBYOL, the encoder must learn features from xi that are not contained in xns but shared with xp.

heading: 2.2 Texture-based negative sample generation
text: We use texture synthesis tools to generate negative samples. Texture synthesis aims to generate realistic images that preserve as much local structure as possible from an example image [19, 11, 42]. For instance, as shown in Figure 1(d), the texture of the input dog image preserves the fur and colors of the dog. Notably, in previous discussion of robustness [3, 50], such local structure has often been recognized as highly correlated with the labels yet superfluous to generalization. For example, under large domain shift due to lighting, motion, and even modality, the texture is more apt to change than the semantic features, such as the shape. Furthermore, CNNs trained on ImageNet are more likely to classify images based on the texture features rather than the shape features which are instead preferred by humans due to their transferability [14]. To encourage the model to rely more on the shape features, we propose a two-step method to generate the texture image of input images as the negative samples for contrastive learning.
To be specific, we first sample two patches from given images as the input to the texture synthesis algorithms. One patch is extracted from the center of the image. This patch is expected to reflect the texture of the object according to the implicit bias contained in the ImageNet dataset that most
of the objects are center-oriented in the images [2]. The other patch is extracted from a random location to reflect other possible textures of the image (e.g. background, peripheral region of the object). In this work, we extract patches with size 96× 96 when image size allows, otherwise 48× 48 patches are extracted. Second, we adopt off-the-shelf texture synthesis algorithms [12, 52, 1] to generate texture images based on the two patches. These non-parametric algorithms iteratively sample pixels from given patches that share a similar neighborhood with the current pixel. Specifically, we use the open-source software built on these methods [12, 52, 1] with multi-threaded CPU support implemented in Rust 1. For each sample in the ImageNet dataset, we generate one 224× 224 texture image to construct a dataset that has the same training and validation size as the ImageNet dataset. We call this dataset ImageNet-Texture. More examples can be found in Appendix A.1.

heading: 2.3 Patch-based negative sample generation
text: To simulate the local information contained in the images [4, 3], we propose an efficient patch-based method to generate non-semantic images. Given an image and a patch size d, we sample patches of size d from (d 224d e)
2 non-overlapping random locations that lie entirely in the image. The patches are then tiled and cropped into 224 × 224 as negative samples. Compared with the texture-based method, this generation process takes negligible time, therefore it can be implemented as part of the data loading process in parallel with training. By doing so, each training sample can be paired with different negative samples generated from different patches every time it is used, compared with the two fixed patches selected when generating texture images.
Different from the texture-based method that generates realistic images, the patch-based method generates images with artificial lines as shown in Figure 1e. One might be concerned with possible degenerate solution where the model outputs a low similarity whenever it detects the repeated sharp changes in the horizontal or vertical directions, which could be done with a single layer of convolution. However, interestingly, we find that the model does not find such a simple solution in practice. This is also noticed in a previous study where the image and its copy with a patch cut out are non-trivially distinguished by the model [34]. To mitigate this potential issue, we randomly sample patch size d from a prior distribution instead of using a fixed d in practice, which allows the model to look at texture at different scales.

heading: 2.4 How hard are the texture-based and patch-based negative samples?
text: Constrastive learning methods are known to struggle with finding hard negative samples [13, 31] and researchers have proposed several ways to better leverage hard negatives [31, 43]. An intermediate question is how hard are our proposed negative samples compared with those standard negative samples used in previous constrastive learning methods [7, 18], i.e. random training samples.
We use the official MoCo-v2 model pretrained on the ImageNet-1K dataset for 200 epochs to calculate the cosine similarities between different kinds of pairs across the ImageNet training set. We plot the histogram of these similarities in Figure 2. As shown in the Figures 2a and 2b, most positive pairs and negative pairs have similarity close to 1 and 0 respectively. Specifically, the average similarities across the training samples are 0.94257 and 0.00018 for positive and negative pairs. As shown in
1https://github.com/EmbarkStudios/texture-synthesis
the Figures 2d and 2c, the distributions of patch-based and texture-based negative samples are very different from those of standard negative samples; their similarity distributions have heavy tails in the positive region. Specifically, the distribution of patch-based and texture-based negative samples have average similarity 0.29503 and 0.35248 across the dataset, which shows that they remain difficult after training with standard negative examples.

heading: 3 Experiments
text: In this section, we evaluate the two kinds of non-semantic negative samples with two contrastive learning methods, MoCo and BYOL, on the ImageNet dataset. We also experiment using its subset, the ImageNet-100 dataset [48, 31], which allows us to perform more comprehensive experiments. We report accuracy on out-of-domain (OOD) datasets including the ImageNet-C(orruption) [22], ImageNet-S(ketch) [50], Stylized-ImageNet [14], and ImageNet-R(endition)[21] datasets as an evaluation of the model’s robustness to domain shifts. ImageNet-C and Stylized-ImageNet contain images transformed from the images in the ImageNet validation set with common corruption and transferred style. ImageNet-S and ImageNet-R are collected independently from the ImageNet dataset and share all or a subset of the classes in the ImageNet dataset with a focus on sketch and other rendering modalities. We show that with our proposed non-semantic negatives, contrastive learning generalizes better under domain shifts. For patch-based negatives, it also improves the performance on the in-domain dataset.

heading: 3.1 ImageNet-100
text: We follow the hyperparameters used in [31] to train MoCo-v2 on the ImageNet-100 dataset with a memory bank size k = 16384 or a halved memory bank size. We also conduct experiments following the hyperparameters in a concurrent study [44] except that we keep k = 16384 for our method. For patch-based augmentation parameters, we use patch size sampled from a uniform distribution d ∼ U(16, 72). The parameter α is indicated behind each model name. We discuss the impact of α in detail in the next section. More ablations on the patch-based augmentations can be found in Appendix C.3. For ImageNet-C, we report the average accuracy across 5 levels of corruption severity.
We repeat the experiments, including both the pretraining and linear evaluation, for 3 runs and report the mean and standard deviation in Table 1. As shown in the table, when following the
previous memory bank size, using both patch-based and texture-based negatives improve the OOD generalizations. Specifically, patch-based augmentation increases the accuracy on ImageNet-S by 5.79% and ImageNet-R by 5.97% when α = 3. When α = 2, it also increases the in-domain accuracy by 1.47% and accuracy on ImageNet-C by 2.05%. The similar trend shared by standard ImageNet and ImageNet-C with different α can be attributed to the resemblance of the images in the two dataset, especially those corrupted images with a lower level of severity. We show the performance of the model with α = 3 is actually better on the highest corruption level as shown in Appendix C.2. The improvement achieved using texture-based negatives is less, probably because the information contained in the texture image is restricted due to the limited access to the two fixed patches. When the memory bank is halved to be 8, 192, the baseline MoCo model has slightly worse performance, decreasing from 77.88 to 77.73. But with patch-based hard negative samples, the MoCo-v2 model instead achieved the best accuracy 79.54 on the ImageNet-100 validation set, IamgeNet-C and IamgeNet-R. We discuss this more in a later section.
Similar to Figure 2, in Figure 3 we visualize the distribution of the cosine similarity between the query sample and both semantic and non-semantic samples calculated based on the models trained with and without patch-based negative samples. The shift of the distribution towards the origin in Figure 3d meets the expectation that our method reduces the similarity of input images and patch-based negative samples. Specifically, the average similarity decreases from 0.4040 to 0.3252 to 0.1593 when α increases from 0, namely no patched-based negatives, to 2 to 3. Interestingly, we notice in Figure 3c that the similarity to texture-based negative samples also decreases, and the average similarity decreases from 0.4114 to 0.3541 to 0.1896, although we did not explicitly penalize it. This demonstrates the resemblance of patch-based negative samples to texture-based negative samples. Given the better performance achieved with patch-based negative samples, for the rest of the experiments, we mainly focus on the patch-based methods. But we still conduct our analysis on the texture-based samples. We also find a marginal decrease in the positive similarity (−0.0068) and negative similarity (−0.0008) when α = 2 and a substantial decrease in the positive similarity (−0.0737) and increase in the negative similarity (0.0036) when α = 3. A similar figure for texture-based negative samples can be found in the Appendix in Figure 12.

heading: 3.2 ImageNet-1K
text: We follow the official hyperparameters [8] to train MoCo-v2 with our patch-based negative samples on the ImageNet-1K dataset. For the parameter α and patch size d, we follow the same configuration used on the ImageNet-100 dataset. We compare our results against the MoCo-v2 baseline [8] and the hard negative mixing algorithm, MoCHi [31]. Due to limited computational resources, we report the metrics evaluated with the official model without repeated runs. The results are shown in Table 2.
More results can be found in Appendix C.6. Note that for ImageNet-C, we show the mCE metric [22], for which smaller is better. For the other datasets, we show the top-1 accuracy.

heading: 3.3 Extension to other non-semantic features
text: Model Top-1 Accuracy
MoCo-v2 [8] 70.44 + Patch-based NS 76.42
note that the expected color distribution of our patch-based negatives is identical to that of the query images, and the actual distribution of samples is close. We conduct experiments with MoCo-v2 on the ImageNet-100 dataset while removing the color jittering from the augmentations. The accuracy of models with and without patch-based negatives are reported in Table 3. We found that patch-based negatives contribute significant effectiveness in preventing the models from learning such a color distribution shortcut.

heading: 3.4 Memory bank size
text: Contrastive learning methods based on negative samples suffer from ineffective excavation of hard negatives [13, 31] and resort to large batch sizes [7] or memory bank [18]. In this section, we study whether our proposed negative samples can mitigate this problem on the STL-10 and ImageNet-100 datasets. We keep the hyperparameters intact and vary the memory bank size. We report the accuracy of the MoCo-v2 baseline with and without patch-based negative samples on STL-10 dataset in Table 4. We also compare with [43] which exploits hard negatives through reweighting. We found that with proper hyperparameters the MoCo-v2 baseline already beats the reweighting results with SimCLR.
As shown in Figures 4 and 5, using patch-based non-semantic negatives consistently improves the MoCo baseline when the number of standard negatives varies. When slightly decreasing the memory bank size on the STL-10 dataset as shown in Figure 4 and ImageNet-100 in Table 1, the performance with patch-based negatives increases. This is probably because, according to Eq. 1 and analysis in Appendix B.1, a smaller memory bank size causes a larger contribution of non-semantic negatives to the loss, and consequently a larger regularization. To further demystify this observation, we conduct experiments with evenly sampled memory bank sizes between 4096 and 16384 and report the average and standard deviation across 3 runs in Figure 5. We confirm a consistent recession of baseline accuracy when decreasing memory back sizes [18, 43]. However, the steady improvement led by the non-semantic negatives effectively mitigates the problem - the decrease caused by a smaller memory bank is less substantial and using patch-based negatives always beats the baseline.

heading: 4 Discussion

heading: 4.1 Controlling the shape-texture trade-off with α
text: There has been a growing interest in understanding the cause and impact of the trade-off between shape and texture bias of CNNs [14, 23, 35, 27]. CNNs trained on ImageNet are known to be over-reliant on the texture features [14]. Contrastive learning with our non-semantic negatives serve as not only an effective method to reduce such reliance, but a natural tool to study such a trade-off.
We train MoCo-v2 models with different α from 1 to 5 on the ImageNet-100 dataset. As shown in Figure 6a, we find that α effectively controls the trade-off on the model bias to the shape and texture features. α = 0 is used to denote the baseline method. Specifically, a larger α in the loss function 1 leads to a larger penalty on the similarity between the representations of query samples and non-semantic samples, consequently a larger shape bias. We follow [14] to calculate shape bias on the stimuli images with conflicted shape and texture clues generated by style transfer. We show the corresponding accuracy on the shape and texture labels in Figure 6b.
As shown in Figure 6b, when α increases the texture accuracy on the stimuli dataset monotonically decreases while the shape accuracy monotonically increases. To further study how the trade-off between shape and texture bias impacts the model performance, we first compare the accuracy on the ImageNet validation dataset and ImageNet-Sketch dataset [50] when α varies in Figure 6c. We find that on both datasets, slightly increased shape bias over baseline (α = 2) improves performances. Interestingly, the accuracy peak on the ImageNet-Sketch appears at α = 3 while the peak appears at α = 2 on the ImageNet validation dataset. In addition, for even larger α the accuracy on the ImageNet-Sketch dataset still outperforms the baseline while the standard accuracy gets hurt. This shows that different downstream tasks may benefit differently from differently shape-biased models.
We plot the histograms of similarities calculated by the models trained with different α in Appendix
Figure 13. We find that large α makes the original pretext task challenging - the model cannot effectively pull together the representations of positive pairs. Specifically, when α increases from 1 to 5, the average similarity of the positive pairs decrease from 0.9267 to 0.7541. This demonstrates that it is hard for the model to learn representations that are completely independent of the texture features contained in the non-semantic images.

heading: 4.2 Rethinking the shape-texture trade-off through class-based analysis
text: The initial discussion on the shape-texture trade-off shows that humans rely more on the shape features while CNNs rely more on texture features and increasing shape bias can improve accuracy and robustness [14]. However, similar to [40, 23], we notice that increasing shape bias does not always improve the generalization and robustness of the models. To better understand this phenomenon, we provide two observations based on the analysis of the ImageNet-Texture dataset and our method to explain why a texture-biased model is helpful with classification on the ImageNet dataset.
First, we find that an increasing shape bias often leads to more errors among the fine-grained classes. The initial discussion of the shape-bias [14] only pays attention to the selected 16 coarse classes. We thus compare the finer and coarse class accuracy on the dog images of ImageNet dataset as in Figure 6d. For coarse class accuracy, the predictions are counted to be correct whenever the image is
classified as a dog class, no matter which dog class is predicted, while for the finer class accuracy, only those predictions of target dog classes are counted to be correct. We notice that the finer class accuracy drops more significantly when shape bias increases as opposed to the coarse class accuracy. For example, when α = 3, the finer class accuracy drops from 72.9 to 67.6 while the coarse class accuracy slightly decreases from 97.8 to 97.7. Therefore, for datasets with numerous fine-grained classes like ImageNet, a texture-biased model is more helpful for a higher accuracy, which confirms the previous conjecture [54] . Second, in Appendix Figure 9, we show a scatter plot of texture accuracy vs. standard accuracy of different model architectures and a histogram of accuracy on individual ImageNet-Texture classes. We identify several classes where using only texture features is sufficient to achieve a high classification accuracy. These classes are all missing in the previous study [14]. As shown in Figure 7 and Appendix Figure 10, texture serves as a more important clue than the shape for these classes.

heading: 5 Related Work
text: Contrastive learning based self-supervised learning Recent contrastive learning based selfsupervised learning methods including MoCo [18], SimCLR [7], InfoMin[49], SimSiam[9], BYOL [16], SwAV [6], Barlow Twins [56] have proven helpful in learning visual representations. These methods rely on different pretext tasks to increase the agreement among the different views of the same image. The augmentations used to generate these views are essential to the success of these contrastive learning methods [7, 6] by preventing shortcuts such as the use of simple color histogram [7]. There is an ongoing trend of developing novel augmentations [6, 49] or adaptively applying augmentations [54] and consistent improvement has been achieved with these studies. However, it is intractable to eliminate every shortcut and sometimes tricky to craft the correct positive pairs. Different from these methods, we show that augmentations that perturb the semantic features and craft negative samples can be more effective to impose additional regularization. For example, to prevent models from relying on local features, it is much easier to destroy global features and create negatives than to remove all the local features and create positives. Furthermore, by maximizing the difference between natural images and their non-semantic versions in the representation space, the models are coerced to avoid any potential shortcuts shared by them.
Methods like MoCo [18] and SimCLR [7] distinguish positive pairs from negative pairs that are picked from the rest of the dataset. However, most of the negatives prove to be unnecessary and insufficient [13, 31]. To excavate effective negative samples, these methods heavily depend on the large batch sizes [7] or memory bank [18]. Utilizing hard negative samples has long been recognized as an effective approach to boost model performance [17, 29, 55, 46]. In the contrastive learning studies, [10, 43] modify the contrastive learning loss to make it assign greater weights to the hard negative samples. [31] proposes to synthesize hard negative samples by taking linear combinations of the hardest negative samples. Our work is orthogonal to these ideas in the way that we propose to generate negative samples from given images themselves to reduce the reliance on the undesired features. In addition, two recent works [25, 32] study the application of adversarial examples as hard positive and negative samples in contrastive learning. [45] augments the images by manipulating their foregrounds and backgrounds to generate negative and positive samples. Compared with these studies, we mainly focus on the OOD evaluation of the models. In addition, our patch-based augmentation is also related to the self-Supervised learning methods that adopt the pretext task based on jigsaw [41, 39, 20], which we discuss in the Appendix B.2.
Robustness and out-of-domain generalization of CNNs High test accuracy provides no guarantee that a network learns high-level semantic features instead of low-level superfluous features that exist in both training and test dataset [30]. An increasing number of studies have corroborated such concerns and found that CNNs can rely on local patches [4, 3], texture [14], high-frequency components [51] and even artificially added features [26] to achieve high test accuracy. These superficial correlations become brittle under large domain shifts [22, 21]. This still remains an unsolved problem [47] and is rarely discussed in the contrastive learning setting.
Among all these undesired features, the shape-texture bias has been widely discussed in recent studies [14, 36]. Previous work has shown that CNNs trained on the ImageNet dataset are biased to texture features and such over-reliance can hurt the generalization performance of CNNs [14, 24]. Several studies have aimed at mitigating this problem [38, 35] or providing a better understanding [23, 27]. In this paper, we introduce a dataset called ImageNet-Texture, which can help future studies on these problems. Our method also effectively controls the trade-off between shape and texture bias. We provide new insights about this problem based on the analysis of our method and dataset.

heading: 6 Closing Remarks
text: Conclusion CNNs are prone to learn discriminative features that are vulnerable under domain shifts. In this paper, we first demonstrate the regularization power of contrastive learning to discard any undesired features by generating appropriate negative samples. We explore two approaches, the patch-based and texture-based augmentations, to craft negative samples with only local features preserved. We show that the representations learned by contrastive learning with such negative samples depend less on the local features, and consequently generalize better under OOD settings. We hope this paper can encourage people to rethink the role that negative samples play in contrastive learning, which hopefully leads to more efficient methods to generate negative samples.
Limitations The problem of dependence on superficial features exists in various domains beyond vision, such as language [37, 28]. Therefore, it is intriguing to consider generalizing such an idea to other modalities. In addition, as the mechanism to ensure that contrastive learning models trained on large datasets to discard the bias of the datasets is yet to be invented, severe social issues in fairness or privacy may be raised as a result [5, 33]. In this paper, we show how to calibrate the bias towards texture features using proposed negative samples. It is also worth considering whether contrastive learning can be used to address any of these negative effects triggered by bias in datasets.
Acknowledgments and Disclosure of Funding
The authors thank the National Science Foundation, grant no. IIS-1910132 and the Guaranteeing AI Robustness Against Deception (GARD) program from DARPA for their support of this project. The authors thank Yannis Kalantidis for his help with reproducing MoCo-v2 on the ImageNet-100 dataset and Joshua Robinson for providing the checkpoints of IFM models for comparison.

Review summaries:
The paper proposes to augment current contrastive self-supervised learning (SSL) methods such as BYOL and MoCo with additional negative samples that are augmented so that they remove semantics from the original image, thus forcing the learning of more robust representations that focus less on superficial image features. The authors propose to do so by generating negative samples based on textures and patches from an original image. The experiments suggest that models learned with the augmented negative samples generalize better.

The paper proposes texture-based and patch-based augmentations to generate negative samples from input images to improve the robustness and generalization ability of contrastive learning. The texture-based augmentation generates negative texture images based on two patches extracted from input images using texture synthesis tools. The and patch-based augmentations constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image. Experiments on ImageNet and its variances demonstrate the effectiveness of the proposed method on improving generalization ability of existing contrastive learning methods. 

This paper introduces non-semantic negative samples into self-supervised learning. These non-semantic samples force the network to reduce the reliance on texture-based features and focus on semantic ones, which are more general in different scenarios.  Experiments on several image classification datasets verify the robustness of the proposed method. In addition, texture-shape bias trade-off is also thoroughly analyzed.

This paper proposes new augmentation strategies for contrastive learning that produce negative samples designed to mimic the texture of points, but not have their shape data. The goal is to make models not rely on texture data. The authors report extensive experiments on the effectiveness of their method, and discuss how different image classes rely on different features in the shape-texture trade-off.