Robust Contrastive Learning Using Negative Samples with Diminished Semantics
Authors: Songwei Ge
Shlok Kumar Mishra
Chun-Liang Li
Haohan Wang
David Jacobs
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
out-of-the-domain
self-supervised learning
hard negative mining

Abstract: Unsupervised learning has recently made exceptional progress because of the development of more effective contrastive learning methods. However, CNNs are prone to depend on low-level features that humans deem non-semantic. This dependency has been conjectured to induce a lack of robustness to image perturbations or domain shift. In this paper, we show that by generating carefully designed negative samples, contrastive learning can learn more robust representations with less dependence on such features. Contrastive learning utilizes positive pairs which preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples in a reversed way, where only the superfluous instead of the semantic features are preserved. We develop two methods, texture-based and patch-based augmentations, to generate negative samples.  These samples achieve better generalization, especially under out-of-domain settings. We also analyze our method and the generated texture-based samples, showing that texture features are indispensable in classifying particular ImageNet classes and especially finer classes. We also show that the model bias between texture and shape features favors them differently under different test settings. 

Robust Contrastive Learning Using Negative Samples with Diminished Semantics
Songwei Ge Univeristy of Maryland songweigêcs. umd. edu
Shiok Mishra Univeristy of Maryland shlokm@cs. umd. edu
Haohan Wang
Carnegie Mellon University haohanw@cs. cmu. edu
Chun-Liang Li Google Cloud AI chunl maumliinnggoogle.coo
David Jacobs Univeristy of Maryland dwj@cs. umd. edu
Abstract
Unsupervised learning has recently made exceptional progress because of the devel- opment of more effective contrastive learning methods. However, CNNs are prone to depend on low-level features that humans deem non-semantic. This dependency has been conjectured to induce lack of robustness to image perturbations domain shift. In this paper, we show that by generating carefully designed negative samples, contrastive learning can learn more robust representations with less de- pendence on such features. Contrastive learning utilizes positive pairs that preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples in a reversed way, where only the superfluous instead of the semantic features are preserved. We develop methods, texture-basee and patch-based augmentations, to generate negative sam- ples. These samples achieve better generalization, especially under out-of-domain settings. We also analyze our method and the generated texture-based samples, showing that texture features are indispensable in classifying particular ImageNet classes and especially finer classes. We also show that model bias favors texture and shape features differently under different test settings. Our code, trained models, and ImageNet- Texture dataset can be found at https://githnn com/ SangweiGe/Contraeeee Learning with- Non semantic-Negativee
Introduction
Recent studies on self-supervised learning have shown great success in learning visual representations without human annotations. The gap between unsupervised and supervised learning has been progres- sively closed by contrastive learning [53, 48, 7, 18, 49, 6, 16, 56]. In the meantime, CNNs trained in the supervised setting are known learn correlations between labels and superfluous features such as local patches [4, 3], texture [14], high-frequency components [51], and even artificially added features [26], which has raised concerns about deploying these models
real scenario [30, 15]. CNNs trained by contrastive learning methods are no exception [23]. In this paper, we propose to construct negative samples that only preserve non-sen semantic features. We show that using contrastive learning methods trained with these negative samples can mitigate these concerns.
Contrastive learning methods exploit carefully designed augmentations to construct positive pairs and pull their representations together. These augmentations are crucial to contrastive learning [7, 6]. common assumption behind these augmentations while perturbing other superficial signals. This inspires us to generate negative samples and inject additional implicit biases on the visual features learned by the models. Specifically, we utilize
preserve the semantics of the input images
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Semantic positive samples
Non- semantic negative samples
SONK (a) Input image
(b) Query sample
(c) Positive sample
(d) Texture- NS (e) Patch-based NS
Figure I: We propose to construct negative samples (NS) from input images for contrastive learning
with augmentations that only preserve non-semantic information such texture and local features.
augmentations that diminish the semantic features while keeping the undesired features such as texture. By pushing apart the representations of such negative samples and input images, the models are expected to rely more on the semantics of the images and less on superficial features.
Inspired by the non-semantic features, we propose two methods to craft negative samples. The first method relies on texture synthesis tools from classic approaches [12, 52]. generates realistic texture images based on two patches extracted from input images, as shown in Figure 1(d). For each image in ImageNet, we generate its texture version and form dataset which we call ImageNet mmateNet-Texture The second method constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image, as shown in Figure ((e). Comparing the non-semantic negative samples with two semantic positive samples in Figure (b) and Figure (c)), the dog from the input image is still recognizable in the positive samples but hard to understand from negative samples. Instead, local statistics such the fur and color of the dog preserved in the negative samples.
The generated non-semantic samples can be readily used with existing contrastive learning methods that distinguish positive pairs from negative pairs such as MoCo [18] and SimCLR [7]. Despite their simplicity, we show that these semantic negative samples are actually harder than the standard negative samples used by these methods, which are inefficient at leveraging hard negatives [13, 31]. Further, our negative pairs can also be used by contrastive learning methods that do not explicitly use negative samples, such as BYL [16]. We evaluate our methods with two contrastive learning methods, MoCo [18, 8] and YOO [16], on three datasets, ImageNet- 100, ImageNet- 1K and STI.- 10. When using our proposed augmentations to generate negative samples and minimize their representation similarity to the input images, we notice a consistent improvement on the generalization performance over backbone methods [8, 16] and previous negative example generation strategies [31, 43), especially under duitrf-distribution (OOD) settings.
We conduct a systematic analysis of how the thape-texture trade-off influences model performance based on the proposed TeageNet-TTxture dataset. We control the penalty on similarities between the non-semantic negative examples and the query samples. This impacts the trade-off between using shape and texture features. We find that the relative importance of texture and shape features varies across different datasets. For example, shape bias benefits ImageNet-Skett [50] more than the original ImageNet validation set. On the other hand, texture bias benefits finer-grat anerrgraiiedd classification more, such as dog breed classification included in ImageNet. These results complement previous evidence showing the effectiveness of shape features in classifying 16 coarse classes [14]. Such preference for one feature over the other is also observed intra-dataset: the texture is more important for some classes such as dishrag and plaque. These observations make us question the relationship between shape and texture features as the implicitly necessary bias of CNNs and advocate for an adaptive combination of both when deploying the model in real scenarios. In summary:
We propose baaed and patch- patch-based augmentations to generate negative samples from input images, and show that these negative samples improve the generalization of contrastive learning.
We introduce the ImageNet- Texture dataset, which contains texture versions of ImageNet images generated by texture synthesis tools.
We provide fine-grained analysis on the shape-texture trade-off of CNNs, and show different scenarios when one is preferred over the other.

---

2 Negative Samples with Diminished Semantics
CNNs are apt learn low level features such texture under supervised settings [3. 14, 51]; this has been recently witnessed under the contrastive learning setting as well [23]. To mitigate this problem, we propose two methods, eexture-based and patch-based augmentations, to generate negative samples for contrastive learning. Texture-basse augmentation generates realistic images based on texture synthesis and patch-based augmentation exploits more comprehensive local features by sampling patches from input images. By penalizing learned similarities between the representations of images and their non non-semantic counterparts, the model encouraged to rely less on the undesired features and focus more on the semantics. In practice, we find the two negative samples play similar roles and the patch-basee based method works slightly better. In this section, we start with an overview of contrastive learning and show how non-semantic negatives are used in these frameworks. Then we elaborate on the two approaches to generate negative samples with diminished semantics.
2.1 Contrastive learning with non-semantic negatives
Given encoder network ʃ and an image x, we denote the output of the network as z = f(x). We use 2; and 2p to denote the representations of the query sample æ; and a positive sample xp generated from the same input image with augmentations that preserve semantics. For contrastive learning methods like MoCo [18] and SimCLR [7], zn. denotes the representation the standard negative sample n, extracted from the memory bank (MoCo) or other images in the current batch (SimCLR). 2ns is the representation of the proposed negative sample Tne which contains particular non-semantic features of the input image with the semantic part weakened. We extend the noise-contrastive estimation (NCE) loss as below:
exp (2.T2p17)
LNCE = Σ log iE7
(I)
exp zp/T) exp (az, zns/T) EmEN exp (z) (2//2//T)
where is temperature parameter and is an additional scaling parameter for non- semantic negatives. A larger a implies stronger penalty on the similarity between the representations of the query image and its non-semantic version. In Appendix B.1 we discuss other possible ways to apply a.
Methods like BYOL [16] do not explicitly rely on negative samples. Nevertheless, BYOL adapts the loss to maximize the agreement of positive pairs. Therefore, we explicitly use the non-semantic negative sample with their loss to minimize its similarity to the query sample:
LBOOL = |z, zpl - al| - = = 2 - 2a - 22 xp + 2az zns-
(2)
We overload a to be the parameter that controls the penalty on the similarity between the representa- tions of input image and its non-semantic version under BYOL, with similar intention as MoCo and SimCLR above. To minimize either LNCE or LBYOL, the encoder must learn features from x, that are not contained in Zns but shared with Tp-
2.2 Texture-based negative sample generation
We use texture synthesis tools to generate negative samples. Texture synthesis aims to generate realistic images that preserve as much local structure as possible from an example image [19, 11, 42]. For instance, as shown in Figure I(d), the texture of the input dog image preserves the fur and colors of the dog. Notably, in previous discussion of robustness [3, 50). such local structure has often been recognized as highly correlated with the labels yet superfluous to generalization. For example, under large domain shift due to lighting, motion, and even modality, the texture is more apt to change than the semantic features, such as the shape. Furthermore, CNNs trained on ImageNet are more likely to classify images based on the texture features rather than the shape features which are instead preferred by humans due to their transferability [14]. To encourage the model to rely more on the shape features, propose two-step method to generate the texture image of input images the negative samples for contrastive learning.
To be specific, we first sample two patches from given images as the input to the texture synthesis algorithms. One patch is extracted from the center of the image. This patch is expected to reflect the texture of the object according to the implicit bias contained in the ImageNet datasct that most
3

---

of the objects are center- criter--riente in the images [2]. The other patch is extracted from random location reflect other possible textures of the imgee (e.g. background, peripheral region of the object). In this work, we extract patches with size 96 96 when image size allows, otherwise 48 48 patches are extracted. Second, we adopt off-the-shelf texture synthesis algorithms [12, 52, to generate texture images based on the two patches. These aon-parametrc algorithms iteratively sample pixels from given patches that share a similar neighborhood with the current pixel. Specifically, we use the open-source software built on these methods [12, 52, I] with multi-threaded CPU support implemented in Rust 1. For each sample the ImageNet dataset, we generate one 224 x 224 texture image construct dataset that has the same training and validation size the ImageNet dataset. We call this dataset ImageNet- ImageNet-Texture. Texture. More examples can be found in Appendix A.I.
2.3 Patch-based negative sample generation
To simulate the local information contained in the images [4. 3], we propose an efficient patch-based method to generate non-semantic images. Given an image and patch size d, we sample patches of size d from ( 22 2) non-overlapping random locations that lie entirely in the image. The patches then tiled and cropped into 224 224 negative samples. Compared with the texture-base based method, this generation process takes negligible time, therefore it can be implemented as part of the data loading process in parallel with training. By doing so, each training sample can be paired with different negative samples generated from different patches every time is used, compared with the two fixed patches selected when generating texture images.
Different from the texture-based method that generates realistic images, the patch-based method generates images with artificial lines as shown in Figure le. One might be concerned with possible degenerate solution where the model outputs low similarity whenever detects the repeated sharp changes in the horizontal or vertical directions, which could be done with single layer of convolution. However, interestingly, we find that the model does not find such a simple solution in practice. This is also noticed in previous study where the image and its copy with patch cut out are non-trivion non-trivially distinguished by the model [34]. To mitigate this potential issue, we randomly sample patch size from prior distribution instead of using fixed d in practice, which allows the model to look at texture at different scales.
2.4 How hard are the cexture-based and patch-based negative samples?
Constrastive learning methods are known to struggle with finding hard negative samples [13, 31] and researchers have proposed several ways to better leverage hard negatives [31, 43]. An intermediate question is how hard are our proposed negative samples compared with those standard negative samples used in previous constrastive learning methods [7, 18], i.e. random training samples.
X000
:000
BX000
40000
-0000
2000
XX3M
2000
2.3
Coalre dirilriity
Coarealtilatty
Czalrs striirryy
(a) Positive Sample
(b) Standard NS
(c) Texture-based NS
(d) Patch-based NS
The histogram of cosine similarity between the representations of query sample and its paired samples, namely positive sample and standard, texture-based, and patch-based negative samples (NS), using MoCo-v2 model trained on the ImageNet-1 K dataset for 200 epochs.
Figure 2:
We use the official MoCo-v2 model pretrained on the ImageNet-11 IK dataset for 200 epochs to calculate the cosine similarities between different kinds pairs across the ImageNet training set. We plot the histogram of these similarities in Figure 2. As shown in the Figures 2a and 2b, most positive pairs and negative pairs have similarity close to 1 and respectively. Specifically, the average similarities across the training samples are 0.94257 and 0. 0.00018 for positive and negative pairs. As shown in

---

the Figures 2d and 2c, the distributions of patch-based and texture-bast negative samples are very different from those of standard negative samples; their similarity distributions have heavy tails in the positive region. Specifically, the distribution of patch- and texture-based negative samples have average similarity 0. 0.29503 and 0.35248 across the dataset, which shows that they remain difficult after training with standard negative examples.
3 Experiments
In this section, evaluate the two kinds of non-semantic negative samples with two contrastive learning methods, MoCo and B BYL,, on the ImageNet dataset. We also experiment using its subset, the ImageNet- 100 dataset [48, 31], which allows
perform more comprehensive experiments. We report accuracy on out-of-domain (OOD) datasets including the ImageNet-C(orruppiee [22], ImageNet-S(ketth) [50], Stylized-Imagell
[14], and (mageNet-R(endition]] datasets evaluation of the model's robustness to domain shifts. ImageNet-C and Boylized-ImageNe contain images transformed from the images in the ImageNet validation with common corruption and transferred style. ImageNet-S and ImageNet-R are collected independently from the ImageNet dataset and share all subset of the classes in the ImageNet dataset with focus on sketch and other rendering modalities. We show that with our proposed non-semantic negatives, contrastive learning generalizes better under domain shifts. For patch- patch-based negatives, it also improves the performance on the in-domain dataset.
3.1 ImageNet-100
ImageNet
ImageNet-C
ImageNet-S
Stylized-ImageNet
ImageNet-R
MoCo-v2 - = 16384
07.88
43.08 0..27
28. 0.55
16.20
32.9210.12
Texture- based a = 2
77.76 1..11
43.5811033
29. 19.11:10.30
16.59
33.361:0.15
Patch-based 2
79.35=0.12
45.13
31.76220.6
17.37
34.78+0015
Patch-based
75.58+0.52
44.4520.16
34.0320.58
18.60
36.89::0.1
MoCo-v2 = 8192
77.73+0..8
43.22
28. 0..5
16.83
33. 33.194-0.44
+ Patch-based a = 2
79.54
45,48
33.3620045
17.81
36.31
MoCo-v2*
80.00+a.14
45.15+0.42
30.38+0.30
16.68
30.38
+ IFM [44]
0.05
80.86
47.36
31.35
18.18
36.79
+ IFM [44] c 0.1
81.22
47.46
31.87
18.42
37.23
+ IFM [44] c 0.2
81.02
47.19
31.55
18.68
37.14
Patch-basee
81.49
47.48
34,20
17.95
38.45:00199
BYOL
78. 76.7.8...
44.43++0..5
35.84+0335 38
15.01
39.53+t061
+ Patch-based a 0.05
10.33
44.60
36.76220.51
15.52:0022
41.16:10.99
InsDis [53]
68.52
28.93
16.67
9.86
19.60
CMC [48]
79.34
39.28
24.04
13.88
32.68
InfoMin [49]
82.74
48.87
38.43
18.14
40.68
Supervised
86.26
49.17
34.95
21.20
39.76
Table 1: Top-1 accuracy the ImageNet-100 100 dataset and its OOD variants. We consider the supervised baseline as well as several self-supervised baselines including MoCo-v2, BYOL, InsDis, CMC, and InfoMIN. For our main comparison using MoCo-v2 and BYOL, also report the standard deviation of 3 runs. For MoCo models, represents the size the memory bank. We use to denote the experiments that the training setting in concurrent work IFM [44].
We follow the hyperparameters used in [31] train MoCo-v2 the ImageNet- 100 datasct with memory bank size & = 16384 or a halved memory bank size. We also conduct experiments following the hyperparamcters in concurrent study [44] except that we keep & 16384 for our method. For patch- patch-based augmentation parameters, we use patch size sampled from a uniform distribution d ~ (116, 72). The parameter is indicated behind each model name. We discuss the impact of a in detail in the next section. More ablations on the patch- based augmentations can be found in Appendix C.3. For ImageNet-C, we report the average accuracy across levels of corruption severity.
We repeat the experiments, including both the pretraining and linear evaluation, for 3 runs and report the mean and standard deviation in Table 1. As shown in the table, when following the
5

---

previous memory bank size, using both patch- and texture- based negatives improve the OOD generalizations. Specifically, patch-based augmentation increases the accuracy on ImageNet-S by 5.79% and ImageNet-R by 5.97% when = 3. When a = 2, it also increases the in-domain accuracy by 1.47% and accuracy on ImageNet-C by 2.05%. The similar trend shared by standard ImageNet and ImageNet-C: with different a can be attributed to the resemblance of the images in the two dataset, especially those corrupted images with a lower level of severity. We show the performance of the model with 3 is actually better on the highest corruption level shown in Appendix C.2. The improvement achieved using texture-based negatives is less, probably because the information contained in the texture image is restricted due the limited access to the two fixed patches. When the memory bank is halved to be 8, 192, the baseline MoCo model has slightly worse performance, decreasing from 77.88 to 77.73. But with patch-based hard negative samples, the MoCo-v2 model instead achieved the best accuracy 79.54 on the ImageNet- 100 validation set, lamgeNet-C and lamgeNet-R. We discuss this more in later section.
Praqueter
15006
2000
9.0
33
4.0
CD
Aa
33
0,3
(a) Positive Sample
(b) Standard NS
(c) Texture- Texture-based NS
(d) Patch-based NS
Figure 3: Histogram of cosine similarity between the representations of query sample and its paired positive sample, standard, eexture-based and patch- negative samples (NS), using models trained without (blue) and with patch-based negative samples (red: a 2, green: a = 3).
we visualize the distribution of the cosine similarity between the query sample and both semantic and non-semantic samples calculated based on the models trained with and without patch-based negative samples. The shift of the distribution towards the origin in Figure 3d meets the expectation that our method reduces the similarity of input images and vatch-based negative samples. Specifically, the average similarity decreases from 0.4040 0.3252 to 0.1593 when a increases from O, namely no patched-basee negatives, to 2 to 3. Interestingly, notice in Figure 3c that the similarity to texture-batt based negative samples also decreases, and the average similarity decreases from 0.4114 to 0.3541 to 0. 0.1896, although we did not explicitly penalize it. This demonstrates the resemblance of patch-based negative samples texture-based negative samples. Given the better performance achieved with patch-based negative samples, for the rest of the experiments, we mainly focus on the patch- methods. But we still conduct our analysis on the texture-based samples. We also find a marginal decrease in the positive similarity (-0.0068) and negative similarity (-0.0008) when a and substantial decrease in the positive similarity (-0.0737) and increase in the negative similarity (0.0036) when a = 3. A similar figure for bastede-based negative samples can be found in the Appendix in Figure 12.
Similar to Figure 2, in Figure 3
3.2 ImageNet-1K
ImageNet
ImageNet-C
ImageNet-S
Stylized-ImageNett
ImageNet-R
MoCo-v2 [8]
67.60
87.7
17.47
5.55
27.81
MoCHi [31]
67.56
88.7
16.32
5.94
25.71
Patch-base NS a
67.92
87.6
18.58
6.34
28.95
Table 2: Top-1 accuracy on the ImageNet- 1K dataset and its sketch, stylized, rendition variants, and mCE the ImageNet-C dataset.
We follow the official hyperparameters [8] to train MoCo-v2 with our patch- based negative samples on the ImageNet- KK dataset. For the parameter a and patch size d, we follow the same configuration used on the ImageNet- 100 dataset. We compare our results against the MoCo-v2 baseline [8] and the hard negative mixing algorithm, MoCHi [31]. Due to limited computational resources, we report the metrics evaluated with the official model without repeated runs. The results are shown in Table 2.

---

More results can be found in Appendix C.6. Note that for ImageNet-C, we show the mCE metric [22]. for which smaller is better. For the other datasets, we show the top-1 accuracy.
3.3 Extension other non-semantic features
Non-semantic features are sometimes referred to as "shortcuts" in the contrastive learning literature [6, 7]. Models that leverage such features often exhibit unfavorable generalization to downstream tasks. For example, without color jittering, SimCLR [9] tends to utilize color histograms to reduce the training loss. In this section, show that models trained with non- semantic negatives are coerced to avoid the shortcuts shared between query images and their non-semantic counterparts. In the example of the color shortcut, we note that the expected color distribution of our patch-base based negatives identical that of the query images, and the actual distribution of samples is close. We conduct experiments with MoCo-v2 on the ImageNet- 100 dataset while removing the color jittering from the augmentations. The accuracy of models with and without patch-basce negatives are reported in Table 3. We found that patch-base negatives contribute significant effectiveness in preventing the models from learning such a color distribution shortcut.
Model
Top-1 Accuracy
MoCo-v2 [8]
70.44
+ Patch-based NS
76.42
Table 3: Test accuracy of MoCo-v2 on the ImageNet- 100 dataset after removing color jittering and adding patch-based negatives.
3.4 Memory bank size
Contrastive learning methods based on negative samples suffer from ineffective excavation hard negatives [13, 31] and resort to large batch sizes [7] or memory bank [18]. In this section, we study whether our proposed negative samples can mitigate this problem on the STL-110 and ImageNet- 100 datasets. We keep the hyperparameters intact and vary the memory bank size. We report the accuracy of the MoCo-v2 baseline with and without patch-based negative samples on STL-10 dataset in Table 4. We also compare with [43] which exploits hard negatives through reweighting. We found that with proper hyperparameters the MoCo-v2 baseline already beats the reweighting results with SimCLR.
SimCLR [7]
80.16
80.0
Debiased [10]
84.90
79.5
+ Hard [43]
87.42
accuracy
eccurac
70.0
McGo-/2
MoCo-V2
MoCo-v2 [8] + Patch-bbsed NS
88.00
Potch-based NS
78.5
Patch-based NS
89.36
78.0
77.5
Table 4: Top-1 accuracy on
128 256 512 1024
4096 16384
4096
6144
8192 10240 12285 14336 16384 memory bank size
the STL-10 dataset.
memory bank size
8 denotes results visually extracted from Figure 2 the STL-10 dataset with different memory bank sizes.
Figure 4: Top-1 accuracy on
Figure 5: Top-1 accuracy on the ImageNet- 100 dataset with differ- ent memory bank sizes.
[43].
As shown in Figures 4 and 5, using patch-based non-semantic negatives consistently improves the MoCo baseline when the number of standard negatives varies. When slightly decreasing the memory bank size on the STL-I0 dataset as shown in Figure 4 and ImageNet- in Table 1, the performance with patch-based negatives increases. This is probably because, according to Eq. and analysis in Appendix B. smaller memory bank size causes a larger contribution of non-semantic negatives to the loss, and consequently larger regularization. To further demystify this observation, we conduct experiments with evenly sampled memory bank sizes between 4096 and 16384 and report the average and standard deviation across runs Figure We confirm a consistent recession of baseline accuracy when decreasing memory back sizes [18, 43]. However, the steady improvement led by the non-semantic negatives effectively mitigates the problem the decrease caused by a smaller memory bank is less substantial and using natch-based negatives always beats the bascline.
7

---

Discussion
4.1 Controlling the shape-texture trade-off with a
There has been growing interest in understanding the cause and impact of the trade-off between shape and texture bias of CNNs [14, 23, 35, 27]. CNNs trained on ImageNet are known to be over-reliant the texture features [14]. Contrastive learning with our non-semantic negatives serve as not only an effective method to reduce such reliance, but a natural tool to study such a trade-off.
coanse eccuracy
finer class accuracy
shoge sccurary arr
n5
accurss
0:30
023
alpha
alpha
aipha
(a) Shape-bias
(b) Shape-texture accuracy (c) Sketch-clean accuracy (d) Coarse-finee accuracy
Figure 6: Larger a monotonically increases the model bias to shape features over texture features. Model performance is impacted by such trade-off differently under different settings. In all scenarios, slightly calibrated shape bias improves model performance. The test settings represented in the red lines gain more from the increased shape bias than the settings represented in the green lines.
We train MoCo-v2 models with different a from 1 to 5 the ImageNet- 100 dataset. As shown in Figure 6a, we find that a effectively controls the trade-off on the model bias to the shape and texture features. a is used denote the baseline method. Specifically, larger a in the loss function 1 leads to a larger penalty on the similarity between the representations of query samples and non-semantic samples, consequently larger shape bias. We follow [14] to calculate shape bias on the stimuli images with conflicted shape and texture clues generated by style transfer. We show the corresponding accuracy on the shape and texture labels in Figure 6b.
As shown in Figure 6b, when a increases the texture accuracy on the stimuli dataset monotonically decreases while the shape accuracy monotonically increases. To further study how the or between shape and texture bias impacts the model performance, we first compare the accuracy on the ImageNet validation dataset and ImageNet-Sketch dataset [50] when varies in Figure 6c. We find that on both datasets, slightly increased shape bias over baseline (a = 2) improves performances. Interestingly, the accuracy peak the ImageNet-Sketch appears
3 while the peak appears at a 2 on the ImageNet validation dataset. In addition, for even larger the accuracy on the ImageNNet-Skttch dataset still outperforms the baseline while the standard accuracy gets hurt. This shows that different downstream tasks may benefit differently from differently shape-biased models.
We plot the histograms of similarities calculated by the models trained with different in Appendix Figure 13. We find that large a makes the original pretext task challenging the model cannot effectively pull together the representations of positive pairs. Specifically, when increases from 1 to 5, the average similarity of the positive pairs decrease from 0.9267 to 0.7541. This demonstrates that it is hard for the model to learn representations that are completely independent of the texture features contained in the non-semantic images.
4.2 Rethinking the shape-texture trade-off through class-based analysis
The initial discussion on the shape-texture trade-off shows that humans rely more on the shape features while CNNs rely more texture features and increasing shape bias improve accuracy and robustness [14]. However, similar to [40, 23], we notice that increasing shape bias does not always improve the generalization and robustness of the models. To better understand this phenomenon, we provide two observations based on the analysis of the ImageNet-Textur dataset and our method to explain why texture- biased model is helpful with classification the ImageNet dataset.
First, we find that an increasing shape bias often leads to more errors among the fine-grained classes. The initial discussion of the shape-bias [14] only pays attention to the selected 16 coarse classes. We thus compare the finer and coarse class accuracy on the dog images of ImageNet dataset as in Figure 6d. For coarse class accuracy, the predictions counted to be correct whenever the image is

---

n04589990, window screen, 0.58
T033007733, dishrag, 0.82
n02892201, plaque 0.64
Figure 7: A ResNet-50 model trained on the ImageNet dataset achieves decent accuracy when only texture features are available some classes. A normal image and its texture version are displayed for some of these classes. The caption indicates the class ID, name, and accuracy on texture images.
classified as a dog class, no matter which dog class is predicted, while for the finer class accuracy, only those predictions of target dog classes are counted to be correct. We notice that the finer class accuracy drops more significantly when shape bias increases as opposed to the coarse class accuracy. For example, when a = 3, the finer class accuracy drops from 72.9 to 67.6 while the coarse class accuracy slightly decreases from 97.8 to 97.7. Therefore, for datasets with numerous fine-grained classes like ImageNet, a texture-biased model is more helpful for a higher accuracy, which confirms the previous conjecture [54]. Second, in Appendix Figure 9, we show a scatter plot of texture accuracy vs. standard accuracy of different model architectures and a histogram of accuracy on individual ImageNet-Texttee
classes. We identify several classes where using only texture features is sufficient to achieve a high classification accuracy. These classes are all missing in the previous study [14]. As shown in Figure 7 and Appendix Figure 10, texture serves more important clue than the shape for these classes.
Related Work
Contrastive learning based self-supervised learning Recent contrastive learning based self- supervised learning methods including MoCo [18]. SimCLR [7]. InfoMin[49]] SimSiam[9], BYOL [16], SwAV [6]. Barlow Twins [56] have proven helpful in learning visual representations. These methods rely on different pretext tasks to increase the agreement among the different views of the same image. The augmentations used to generate these views are essential the success of these contrastive learning methods [7, 6] by preventing shortcuts such as the use of simple color histogram [7]. There is ongoing trend of developing novel augmentations [6, 49] or adaptively applying augmentations [54] and consistent improvement has been achieved with these studies. However, it is intractable to eliminate every shorteut and sometimes tricky to craft the correct positive pairs. Different from these methods, we show that augmentations that perturb the semantic features and craft negative samples can be more effective to impose additional regularization. For example, to prevent models from relying on local features, it is much easier to destroy global features and create negatives than to remove all the local features and create positives. Furthermore, by maximizing the difference between natural images and their non-semantic versions in the representation space, the models are coerced to avoid any potential shortcuts shared by them.
Methods like MoCo [18] and SimCLR [7] distinguish positive pairs from negative pairs that are picked from the rest of the dataset. However, most of the negatives prove be unnecessary and insufficient [13, 31J. To excavate effective negative samples, these methods heavily depend on the large batch sizes [7] or memory bank [18]. Utilizing hard negative samples has long been recognized as an effective approach to boost model performance [17, 29, 55, 46]. In the contrastive learning studies, [10, 43] modify the contrastive learning loss make assign greater weights the hard negative samples. [31] proposes to synthesize hard negative samples by taking linear combinations of the hardest negative samples. Our work is orthogonal to these ideas in the way that we propose to generate negative samples from given images themselves to reduce the reliance on the undesired features. In addition, two recent works [25, 32] study the application of adversarial examples as hard positive and negative samples in contrastive learning. [45] augments the images by manipulating their foregrounds and backgrounds to generate negative and positive samples. Compared with these studies, we mainly focus on the OOD evaluation of the models. In addition, our batch-based augmentation also related to the cal--Supeeeiee learning methods that adopt the pretext task based on jigsaw [41, 39, 20], which we discuss in the Appendix B.2.

---

Robustness and out-of-domain generalization of CNNs High test accuracy provides no guar- antee that a network learns high-level semantic features instead of low-level superfluous features that exist in both training and test dataset [30]. An increasing number of studies have corroborated such concerns and found that CNNS can rely on local patches [4, 3], texture [14], eighfrreueeccy components [51] and even artificially added features [26] to achieve high test accuracy. These super- ficial correlations become brittle under large domain shifts [22, 21J. This still remains an unsolved problem [47] and is rarely discussed in the contrastive learning setting.
Among all these undesired features, the thape-texture bias has been widely discussed in recent studies [14, 36.. Previous work has shown that CNNs trained on the ImageNet dataset are biased to texture features and such over-reliance can hurt the generalization performance of CNNs [14, 24]. Several studies have aimed at mitigating this problem [38, 35] or providing a better understanding [23, 27J. In this paper, we introduce a dataset called SoggeNet-Texture which can help future studies on these problems. Our method also effectively controls the trade-off between shape and texture bias. We provide new insights about this problem based on the analysis of our method and dataset.
Closing Remarks
Conclusion CNNs are prone to learn discriminative features that are vulnerable under domain shifts. In this paper, we first demonstrate the regularization power of contrastive learning to discard any undesired features by generating appropriate negative samples. We explore two approaches, the patch-based and cetture-based augmentations, to craft negative samples with only local features preserved. We show that the representations learned by contrastive learning with such negative samples depend less on the local features, and consequently generalize better under OOD settings. We hope this paper can encourage people to rethink the role that negative samples play contrastive learning, which hopefully leads to more efficient methods to generate negative samples.
The problem of dependence on superficial features exists in various domains beyond vision, such as language [37, 28]. Therefore, it is intriguing to consider generalizing such an idea to other modalities. In addition, as the mechanism to ensure that contrastive learning models trained on large datasets to discard the bias of the datasets is yet to be invented, severe social issues in fairness or privacy may be raised as a result [5, 33]. In this paper, we show how to calibrate the bias towards texture features using proposed negative samples. It is also worth considering whether contrastive learning can be used to address any of these negative effects triggered by bias in datasets.
Limitations
Acknowledgments and Disclosure of F Funding
The authors thank the National Science Foundation, grant no. 19101121 and the Guaranteeing AI Robustness Against Deception (GARD) program from DARPA for their support of this project. The authors thank Yannis Kalantidis for his help with reproducing MoCo-v2 on the ImageNet-100 dataset and Joshua Robinson for providing the checkpoints of IFM models for comparison.
References
M. Ashikhmin. Synthesizing natural textures. In Proceedings of the 2001 symposium on Interactive 3D) graphics. pages 217-226, 2001.
[2] L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord. Are done with imagenet?, 2020.
[3] W. Brendel and M. Bethge. Approximating CNNs with bag-of- lor--loll--tttree models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019.
[4] T. B. Brown, Mané, A. Roy, M. Abadi, and Gilmer. Adversarial patch. arXiv preprint 17X.:1112..00655.. 2017.
[5] J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
[6] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988) 2020.
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
10

---

[8] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning, arXiv preprini SxXi::2003.04297,, 2020.
[9] X. Chen and K. He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.
2020.
[10] C.-Y. Chuang. J. Robinson, Y.-C. Lin, A. Torralba, and S. Jegelka. Debiased contrastive learning. In
H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8765-8775. Curran Associates, Inc., 2020.
[11] J. S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 361-368, 1997.
[12] A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033 1038, IEEE, 1999.
[13] J. Frankle, D. J. Schwah, A. S. Morcos, et Are negatives created equal in contrastive instance discrimination? arXiv preprint arXiv:2010.0660 2020.
[14] R. Geirhos, Rubisch, C. Michaclis, M. Bethge, A. Wichmann, and W. Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustmess. In International Conference on Learning Representations, 2019.
[15] R. Geirhos, R. M. Temme, J. Rauber, H. H. Schütt, M. Bethge, and F. A. Wichmann. Generalisation in humans and deep neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7549-7561, 2018.
[16] J.-B. Grill, Strub, F. Altché. C. Tallec. P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284 Curran Associates, Inc., 2020.
[17] B. Harwood, V. Kumar B G, G. Carneiro, I. Reid, and T. Drummond. Smart mining for deep metric learning. In Proceedings the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[18] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9733 2020.
[19] D. J. Heeger and J. R. Bergen. Pyramid- based texture amalysisssynthesis. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 229-238, 1995.
[20] O. Henaff. Data-cfficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pages 41 182-4192. PMIR, 2020.
[21] D. Hendrycks, S. Basart. N. Mu, Kadavath, F. Wang, E. Dorundo. R. Desai, Zhu, Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv: 262..1166,,, 2020.
[22] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[23] K. Hermann, T. Chen, and S. Korbllith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[24] K. L.. Hermann and A. K. Lampinen. What shapes feature representations? exploring datasets, architectures, and training, 2020.
[25] C.-H. Ho and N. Nvasconcelos. Contrastive learning with adversarial examples. In H. Larochelle.
M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages
Curran Associates, Inc., 2020.
[26] H. Huang, X. Ma, S. M. Erfani, J. Bailey, and Y. Wang. Unlearnable examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021.
[27] M. A. Islam, M. Kowal, P. Esser, S. Jia, B. Ommer, K. G. Derpanis, and N. Bruce. Shape or texture: Understanding discriminative features in [enn]s. In International Conference on Learning Representations, 2021.
[28] R. Jia and P. Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, 2017.
[29] S. Jin, A. RoyChowdhury, H. Jiang, A. Singh, A. Prasad, D. Chakraborty, and E. Learned- Learned-Miller. Unsuper- vised hard example mining from videos for improved object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 307-324, 2018

---

[30] J. Jo and Y. Bengio. Measuring the tendency of cans to learn surface statistical regularities. arXiv preprint arXiv:1711.1156. 2017.
[31] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel. and D. Larlus. Hard negative mixing for contrastive learning. In Neural Information Processing Systems (NeurIPS), 2020.
[32] M. Kim, J. Tack, and S. J. Hwang. Adversarial self-supervised contrastive learning. In H. Larochelle,
M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2983-2994. Curran Associates, Inc., 2020.
[33] S. Kiritchenko and S. M. Mohammad. Examining gender and race bias in two hundred sentiment analysis systems, arXiv preprint arXiv: 10001100500000 2018.
[34] C.-L., K. Sohn, J. Yoon, and T. Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. arXiv preprint arxiv:2104.044 104004015 2021.
[35] Y. Li, Q. Yu, M. Tan, J. Mei, P. Tang, W. Shen, A. Yuille, and cihang xie. Shape-texture debiased neural network training. In International Conference on Learning Representations, 2021.
[36] G. Malhotra and J. Bowers. The contrasting roles of shape in human vision and convolutional neural networks. In CogSci, 2019.
[37] T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong reasons: Diagnosing syntactic hcuristics in natural language inference. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages
2019.
[38] S. Mishra, A. Shah, A. Bansal, J. Choi, A. Shrivastava, A. Sharma, and D. Jacobs. Learning visual representations for transfer learning by suppressing texture. arXiv preprint arXiv:2011.01900 2020.
[39] I. Misra and L. V. d. Maaten. Self-supervised learning of pretext- invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717 2020.
[40] C. K. Mummadi, R. Subramaniam, R. Hutmacher, J. Vitay, V. Fischer, and J. H. Metzen. Does enhanced shape bias improve neural network robustness to common corruption? In ICLR, 2021.
[41] M. Noroozi and P. Favaro. Unsupervised learning visual representations by solving puzzles. In European conference on computer vision, pages 69-84. Springer, 2016.
[42] Portilla and E. P. Simoncelli. parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1):49-71 70, 2000.
[43] J. Robinson, C.-Y. Chuang, S. Sra, and Jegelka. Contrastive learning with hard negative samples. International Conference on Learning Representations, 2021.
[44] J. Robinson, I.. Sun, K. Yu, K. Batmanghelich, S. Jegelka, and S. Sra. Can contrastive learning avoid shortcut solutions? NeurIPS, 2021.
[45] C. K. Ryali, D. J. Schwab, and A. S. Morcos. Leveraging background augmentations encourage semantic focus in self-supervised contrastive learning. arXiv preprint arXiv:2103. 12719, 2021.
[46] E. Simo-Serra E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno- Noguur. Diseriminative learning of deep convolutional feature point descriptors. In Proceedings of the IEEE International Conference on Computer Vision, pages 1 118-126, 2015.
[47] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Mcasuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33, 2020.
[48] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In ECCVV2020)
[49] Y. Tian, C. Sun, B. Poolc, D. Krishman, C. Schmid, and P. Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[50] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[51] H. Wang, X. Wu, Z. Huang, and E. P. Xing. figh-freeuency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684 8694, 2020.
[52] L.-Y. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th annual conference Computer graphics and interactive techniques, pages 479-488, 2000.
[53] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation, pages 3733 3742, 2018
[54] T. Xiao, X. Wang, A. A. Efros, and T. Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2020.
12

---

[55] H. Xuan, A. Stylianou, X. Liu, and R. Pless. Hard negative examples are hard, but useful. In European Conference on Computer Vision, pages 126-142. Springer, 2020.
[56] J. Zbontar, L.. Jing, Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03220 2021.
13


Reviews:
[{"review_id": "teS0nWF2zGL", "reviewer": "Reviewer_ffpV", "summary": "This paper proposes new augmentation strategies for contrastive learning that produce negative samples designed to mimic the texture of points, but not have their shape data. The goal is to make models not rely on texture data. The authors report extensive experiments on the effectiveness of their method, and discuss how different image classes rely on different features in the shape-texture trade-off.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a nice paper with strong central ideas and extensive experiments. The shape-texture trade-off is interesting and important, and I'm glad to see work using contrastive learning to try to control the trade-off.\n\nI think the work could be improved if the authors spend a bit more time explaining the results and discussing where the lift comes from. The experiments are extensive, but there are multiple effects that are not explained. The two major questions that stick out to me are:\n\n1. In table 1, there are some tasks where patch-based with alpha = 2 works better, and some tasks where patch-based with alpha = 3 works better. I could not find any explanation about this discrepancy. *Why* does one method outperform in some cases, but not others? Why does patch-based with alpha = 3 underperform baselines by two points on ImageNet, but alpha = 2 outperforms by two points?\n\nSection 3.1 spends considerable space on exposition of the results, but does not spend enough space discussing how the method creates lift, or under what circumstances to expect lift.\n\n2. The results from Figures 4 and 5 about memory bank size are striking, and I don't think that the text explanation or the analysis in appendix B.1 adequately explain it. Performance consistently increases with memory bank size -- and even appears to grow faster than log-linear in memory bank size -- but then significantly drops at the last tested memory bank size. I would expect a smoother degradation for the explanation that the benefit from larger memory bank size is being outweighed by less impact from the non-semantic negatives (note that increasing memory bank size is still helpful for MoCo). More experiments would be helpful to understand this phenomenon and validate the hypothesis.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "gEK-5jtLJoI", "reviewer": "Reviewer_PE6h", "summary": "This paper introduces non-semantic negative samples into self-supervised learning. These non-semantic samples force the network to reduce the reliance on texture-based features and focus on semantic ones, which are more general in different scenarios.  Experiments on several image classification datasets verify the robustness of the proposed method. In addition, texture-shape bias trade-off is also thoroughly analyzed.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to eliminate CNNs’ reliance on low-level features and improve the generalization ability of models. To achieve this, two negative sample generation methods are proposed. These methods construct negative samples which only preserve non-semantic features. This idea is straightforward and sound. \n\nExperiments on multiple datasets show the robustness of models trained with non-semantic samples, verifying the efficacy of the proposed methods. Additionally, detailed analysis on the texture-shape bias trade-off is provided.\n\nIn general, this paper is well written and it sheds light on the texture-shape bias of existing self-supervised methods. The improved generalization performance would help self-supervised pre-trained models to be applied in real scenarios.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. As shown in Table 1 and discussed in Section 4, models are sensitive to the penalty hyperparameter $\\alpha$ and different datasets may require different settings. However, tuning a pre-trained model for each scenario is not affordable.  And using the same setting for all scenarios may lead to sub-optimal results or even hurt the performance. Discussions about how to apply the proposed methods in real scenarios should be added.\n\n\n2. Using the non-semantic samples with MoCo or BYOL require extra computation cost for these samples. Time and GPU memory costs should be discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "QwglmrutfHn", "reviewer": "Reviewer_CTYi", "summary": "The paper proposes texture-based and patch-based augmentations to generate negative samples from input images to improve the robustness and generalization ability of contrastive learning. The texture-based augmentation generates negative texture images based on two patches extracted from input images using texture synthesis tools. The and patch-based augmentations constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image. Experiments on ImageNet and its variances demonstrate the effectiveness of the proposed method on improving generalization ability of existing contrastive learning methods.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The idea of adding negative samples that only share the same texture as the anchor image to avoid learning the texture semantic is novel. \n\nQuality: The paper has the following weaknesses:\n1.\tThe paper lacks comparison with other baselines. For example, in [1], the authors proposed to use texture randomization in the augmentations which can also learn texture-invariant features. Also, in [1] the authors reported higher accuracy of MoCo-v2 on ImageNet-100 (81.0%), while the MoCo-v2 accuracy in this paper is less than 78%. \n2.\tThe paper proposed to learn less about the texture semantics by adding negative samples that only share the texture sematic with the anchor images. However, the texture semantics can still be helpful in some cases. Similar to [1], it would be good to try to ensemble models trained with the proposed negative augmentations and without it and see whether this could improve the performance.\n3.\tThe proposed method is restricted to removing texture semantics and lack of extension to other semantics. One interesting experiment to do is to remove color distortion in the augmentations of positive samples and add negative samples that share the same color distribution of the anchor image to see whether the proposed method could be helpful to avoid the color distribution shortcut.\n\nClarity: The paper is well written and easy to follow.\n\nSignificance: the paper proposes a simple method to construct negative samples to prevent contrastive learning from learning texture information which could affect its generalization ability. However, it is not always desirable to remove the texture semantics, since it is also an important semantics in image classification task. Moreover, the proposed method is restricted to the texture semantics and the authors do not show that it could be extended to other semantics. Therefore, the significance of the paper is limited.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "7ei2zzeDgl5", "reviewer": "Reviewer_zVco", "summary": "The paper proposes to augment current contrastive self-supervised learning (SSL) methods such as BYOL and MoCo with additional negative samples that are augmented so that they remove semantics from the original image, thus forcing the learning of more robust representations that focus less on superficial image features. The authors propose to do so by generating negative samples based on textures and patches from an original image. The experiments suggest that models learned with the augmented negative samples generalize better.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe method proposes to augment the negative samples of constrative SSL methods. This has been done before in the literature ([31], [43] for example), but not necessarily for textures or image patches. The role of texture and patch-based features for supervised learning models has been studied before (e.g. [3]). Therefore, it is a combination of ideas that have been investigated before, but in its current form it is relatively novel.\n\n**Clarity**\n\nThe paper is well written. The claims and hypothesis of the authors are clearly stated, and the experiments and analysis is easy to follow.\n\n**Quality**\n\nThe premise of the paper is clear, and the experimental section is quite extensive. That being said, there are some issues with the approach. \n\nFirst, the modification of the BYOL objective is ad-hoc and not really justified. An ablation that also shows the difference in using a regular negative sample or a texture-based negative sample for this method would be useful to understand whether the reported improved performance is due to adding a negative sample or because of the particular mining strategy for the negative. Additionally, the improved performance for the augmented model on a few datasets (ImageNet-100 e.g.) is quite below the standard deviation of the runs and therefore not really significant.\n\nSecond, for most approaches the introduction of another negative sample effectively increases the batch size of the training model or equivalently, the model is trained on more samples. An ablation showing that this does not have an effect on the regular approach would be useful.\n\nThirdly, and following the previous point, some of the reported metrics of the baseline methods do not seem to match the published results and seem to be slightly lower. For example, MoCoV2 reports improved performance well above 67.60 on Imagenet 1K when trained for longer, and MoCHi reports better performance over the MoCoV2 baseline and in this paper its reported worse performance. It would be useful to have experiments with the same setup/performance as in the original papers, and with configurations that maximize the performance of a model to make sure that the reported improvements are significant and not necessarily due to differing training setups such as feeding more examples to the model.\n\n**Significance**\nFor the above reasons, it is not fully well supported that the proposed method extensions and negative mining strategies show a significant advantage to the baseline methods (considering also the more complex setup involved), however, even with these doubts the paper might be of interest to the community and therefore my favorable rating.\n\n**Review updates**\nAfter reading the rest of the reviews and the authors' response, I am still not convinced about the results using a modified BYOL and the performance discrepancy with MoCHI. However, as stated in the original review, I believe the paper might be of interest to the community despite these issues and therefore stand by my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly address the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see direct ethical concerns with the paper and the authors briefly mention concerns in the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper copes with generating negative samples for contrastive learning, separating the signal from texture and shapes. The idea has been judged sufficiently novel and sound. The experiments are convincing, and the additional results presented in the rebuttal improved the empirical validation further. I recommend this paper for acceptance, and recommend the authors take into account all reviewer’s comments for preparing the final version of the manuscript.
Decision: Accept (Poster)