Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces non-semantic negative samples into self-supervised learning. These non-semantic samples force the network to reduce the reliance on texture-based features and focus on semantic ones, which are more general in different scenarios.  Experiments on several image classification datasets verify the robustness of the proposed method. In addition, texture-shape bias trade-off is also thoroughly analyzed.
main_review: This paper aims to eliminate CNNs’ reliance on low-level features and improve the generalization ability of models. To achieve this, two negative sample generation methods are proposed. These methods construct negative samples which only preserve non-semantic features. This idea is straightforward and sound. 

Experiments on multiple datasets show the robustness of models trained with non-semantic samples, verifying the efficacy of the proposed methods. Additionally, detailed analysis on the texture-shape bias trade-off is provided.

In general, this paper is well written and it sheds light on the texture-shape bias of existing self-supervised methods. The improved generalization performance would help self-supervised pre-trained models to be applied in real scenarios.

limitations_and_societal_impact: 1. As shown in Table 1 and discussed in Section 4, models are sensitive to the penalty hyperparameter $\alpha$ and different datasets may require different settings. However, tuning a pre-trained model for each scenario is not affordable.  And using the same setting for all scenarios may lead to sub-optimal results or even hurt the performance. Discussions about how to apply the proposed methods in real scenarios should be added.


2. Using the non-semantic samples with MoCo or BYOL require extra computation cost for these samples. Time and GPU memory costs should be discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes texture-based and patch-based augmentations to generate negative samples from input images to improve the robustness and generalization ability of contrastive learning. The texture-based augmentation generates negative texture images based on two patches extracted from input images using texture synthesis tools. The and patch-based augmentations constructs non-semantic images by tiling randomly sampled patches of different sizes from the input image. Experiments on ImageNet and its variances demonstrate the effectiveness of the proposed method on improving generalization ability of existing contrastive learning methods.

main_review: Originality: The idea of adding negative samples that only share the same texture as the anchor image to avoid learning the texture semantic is novel. 

Quality: The paper has the following weaknesses:
1.	The paper lacks comparison with other baselines. For example, in [1], the authors proposed to use texture randomization in the augmentations which can also learn texture-invariant features. Also, in [1] the authors reported higher accuracy of MoCo-v2 on ImageNet-100 (81.0%), while the MoCo-v2 accuracy in this paper is less than 78%. 
2.	The paper proposed to learn less about the texture semantics by adding negative samples that only share the texture sematic with the anchor images. However, the texture semantics can still be helpful in some cases. Similar to [1], it would be good to try to ensemble models trained with the proposed negative augmentations and without it and see whether this could improve the performance.
3.	The proposed method is restricted to removing texture semantics and lack of extension to other semantics. One interesting experiment to do is to remove color distortion in the augmentations of positive samples and add negative samples that share the same color distribution of the anchor image to see whether the proposed method could be helpful to avoid the color distribution shortcut.

Clarity: The paper is well written and easy to follow.

Significance: the paper proposes a simple method to construct negative samples to prevent contrastive learning from learning texture information which could affect its generalization ability. However, it is not always desirable to remove the texture semantics, since it is also an important semantics in image classification task. Moreover, the proposed method is restricted to the texture semantics and the authors do not show that it could be extended to other semantics. Therefore, the significance of the paper is limited.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes new augmentation strategies for contrastive learning that produce negative samples designed to mimic the texture of points, but not have their shape data. The goal is to make models not rely on texture data. The authors report extensive experiments on the effectiveness of their method, and discuss how different image classes rely on different features in the shape-texture trade-off.
main_review: This is a nice paper with strong central ideas and extensive experiments. The shape-texture trade-off is interesting and important, and I'm glad to see work using contrastive learning to try to control the trade-off.

I think the work could be improved if the authors spend a bit more time explaining the results and discussing where the lift comes from. The experiments are extensive, but there are multiple effects that are not explained. The two major questions that stick out to me are:

1. In table 1, there are some tasks where patch-based with alpha = 2 works better, and some tasks where patch-based with alpha = 3 works better. I could not find any explanation about this discrepancy. *Why* does one method outperform in some cases, but not others? Why does patch-based with alpha = 3 underperform baselines by two points on ImageNet, but alpha = 2 outperforms by two points?

Section 3.1 spends considerable space on exposition of the results, but does not spend enough space discussing how the method creates lift, or under what circumstances to expect lift.

2. The results from Figures 4 and 5 about memory bank size are striking, and I don't think that the text explanation or the analysis in appendix B.1 adequately explain it. Performance consistently increases with memory bank size -- and even appears to grow faster than log-linear in memory bank size -- but then significantly drops at the last tested memory bank size. I would expect a smoother degradation for the explanation that the benefit from larger memory bank size is being outweighed by less impact from the non-semantic negatives (note that increasing memory bank size is still helpful for MoCo). More experiments would be helpful to understand this phenomenon and validate the hypothesis.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to augment current contrastive self-supervised learning (SSL) methods such as BYOL and MoCo with additional negative samples that are augmented so that they remove semantics from the original image, thus forcing the learning of more robust representations that focus less on superficial image features. The authors propose to do so by generating negative samples based on textures and patches from an original image. The experiments suggest that models learned with the augmented negative samples generalize better.
main_review: **Originality**

The method proposes to augment the negative samples of constrative SSL methods. This has been done before in the literature ([31], [43] for example), but not necessarily for textures or image patches. The role of texture and patch-based features for supervised learning models has been studied before (e.g. [3]). Therefore, it is a combination of ideas that have been investigated before, but in its current form it is relatively novel.

**Clarity**

The paper is well written. The claims and hypothesis of the authors are clearly stated, and the experiments and analysis is easy to follow.

**Quality**

The premise of the paper is clear, and the experimental section is quite extensive. That being said, there are some issues with the approach. 

First, the modification of the BYOL objective is ad-hoc and not really justified. An ablation that also shows the difference in using a regular negative sample or a texture-based negative sample for this method would be useful to understand whether the reported improved performance is due to adding a negative sample or because of the particular mining strategy for the negative. Additionally, the improved performance for the augmented model on a few datasets (ImageNet-100 e.g.) is quite below the standard deviation of the runs and therefore not really significant.

Second, for most approaches the introduction of another negative sample effectively increases the batch size of the training model or equivalently, the model is trained on more samples. An ablation showing that this does not have an effect on the regular approach would be useful.

Thirdly, and following the previous point, some of the reported metrics of the baseline methods do not seem to match the published results and seem to be slightly lower. For example, MoCoV2 reports improved performance well above 67.60 on Imagenet 1K when trained for longer, and MoCHi reports better performance over the MoCoV2 baseline and in this paper its reported worse performance. It would be useful to have experiments with the same setup/performance as in the original papers, and with configurations that maximize the performance of a model to make sure that the reported improvements are significant and not necessarily due to differing training setups such as feeding more examples to the model.

**Significance**
For the above reasons, it is not fully well supported that the proposed method extensions and negative mining strategies show a significant advantage to the baseline methods (considering also the more complex setup involved), however, even with these doubts the paper might be of interest to the community and therefore my favorable rating.

**Review updates**
After reading the rest of the reviews and the authors' response, I am still not convinced about the results using a modified BYOL and the performance discrepancy with MoCHI. However, as stated in the original review, I believe the paper might be of interest to the community despite these issues and therefore stand by my original rating.
limitations_and_societal_impact: The authors briefly address the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper copes with generating negative samples for contrastive learning, separating the signal from texture and shapes. The idea has been judged sufficiently novel and sound. The experiments are convincing, and the additional results presented in the rebuttal improved the empirical validation further. I recommend this paper for acceptance, and recommend the authors take into account all reviewer’s comments for preparing the final version of the manuscript.
Decision: Accept (Poster)