Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Hypothesis: Convolutional Neural Networks are biased towards either texture or shape according to the dataset used for training. If so, can we develop an approach that can do shape-texture debiased learning?

This hypothesis is in line with Geihros et al., 2019 who observed that ImageNet trained models are texture-biased. 

Verification of the hypothesis: The paper qualitatively and quantitatively shows the influence of a shape-biased and a texture-biased model.

How to solve the problem?: The authors make two observations - (1). the model with shape-biased representations and the models with texture-biased representations are highly complementary to each other; and (2) being biased towards either shape cues or texture cues may hurt the performance but there exists a sweet-spot along the interpolation path of shape-texture debiased model that allows acquiring both shape and texture representations and achieve superior performance than vanilla models. 

The rest of the paper consists of experiments to demonstrate this property and show analysis for different aspects of the model. 

Question: One thing that I did not understand is how shape-texture debiasing can help against adversarial attacks? The authors present some quantitative analysis demonstrating it but I still don't get the rationale behind it.


review_title: interesting observation, simple experiments!

final_score: rating: 7: Good paper, accept
review_content: The paper tackles the problem of an existing bias in the classification networks, which makes them focus on a particular set of features (either local, "textures", or global, "shapes"). While it is clear how to train shape- and texture-biased networks (by preparing datasets with items easier to recognize using global or local features respectively), the authors claim that a combined training on both of these datasets would improve the performance of the network.

Pros.

1. The proposed method tackles an important problem and improves over some baselines (when compared to plain training or other similar regularizations, like Mixup and CutMix) on ImageNet-based datasets. Also, it can be applied to different problems like segmentation and improve the results, which shows the generality of the approach.

2. The authors try to explain the behavior of the method by providing extensive motivation and some experimental confirmations (unfortunately, they are quite limited).

3. The paper is extremely well-written and concise.

Cons.

1. The authors claim that whether or not the trained model will have bias depends on the statistics of items in the dataset: the more items easier to recognize using shape features, the more shape-biased the model will be. A simple test for that hypothesis would be to mix shape- and texture-biased datasets and train a model or their concatenation. If the authors' hypothesis is correct, this would aid in the final performance of the model, but I did not find such an experiment or any discussion of it in the experiments section. It seems to me that a major part of the performance of this method could simply come from data augmentation (basically, the same principle as described in the Mixup paper).

2. Following on the previous point, it would be really helpful to see attention visualizations for the proposed model, similar to Figure 4. Right now it is not possible to discern whether or not it has any noticeable differences from the baseline shape model in terms of attention maps, and it could reinforce the proposed explanation of the method.

3. Another important baseline that is missing is evaluating an ensemble of two models: one texture- and one shape-biased. It is a useful baseline, since it is hyperparameter-free, while the proposed method requires a grid-search to find an optimal parameter for blending the labels. Importantly, while the authors state that there is a "sweet spot" in this continuous hyperparameter, they do not propose a way to efficiently search for it other than grid-search. While they show that for the ImageNet dataset the network is quite robust to this hyperparameter, the same result may not be true for other datasets. Searching for this hyperparameter may prove to be significantly more costly than training an ensemble of two models.

In general, the proposed method itself is quite simple and has a lot of familiarities to the previous approaches, which the authors note. The main distinguishing feature of the proposed approach is the mechanism by which it works, suggested by the authors. And, in my opinion, there is not enough numerical and qualitative justification that this approach works the way it is explained.

Yet, assuming that some additional clarifications for the mechanisms behind the proposed approach will be provided in the rebuttal, I place my initial rating as "accept".
review_title: Good paper with noticeable limitations

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The authors propose a method to mitigate the bias towards either texture or shape, in convolutional network training. The method follows the idea from Geirhos et al (2019), but use images randomly sampled from the same dataset, instead of style transfer from paintings. Then, depending on a manually selected hyperparameter, the weights of conflicting labels are blended by weighted average of the one-hot encoding.

###################

Reasons for score

The method proposed is simple, yet effective; however, parts of the method can be more principled, for example, the sampling of textures used for style transfer and debiasing - instead of using random pairs, random sampling from certain classes or external texture or material dataset would help understand better how debiasing works.
Since the model claims that the method is debiasing the model (or can bias it towards shape or texture), it would be interesting to validate this claim on a texture database (e.g. Materials in Context, Flickr Material Database (FMD) or Describable Textures Dataset (DTD) ). The expected outcome would be - shape biased model underperforms; texture biased model, improves texture recognition. The last two datasets are small (1K images; 6K images), so it would not increase the computational overhead. 

###################

Strong points of the paper:

- visualization of the cues of the two models (texture and shape), using CAM (Class Activation Mapping);
- showing the orthogonality of shape and texture-biased models (Fig. 4), with examples of tasks where each model is more accurate or less acurate.
- significant improvement on Stylized ImageNet 
- usecase of the method to mitigate an adversarial attacker (FGSM).

###################

Weaker aspects:

- method is a  bit too simple (however, it is effective).
 The whole paper can be summarized as: pick a random (texture) image, style transfer and blend the labels via weighted average.
The contribution in addition to Geirhos et al seems to consist in changing the domain of the texture source, with same dataset images. 

- What is the difference between the "Vanilla" training and shape biased training? My understanding from Fig. 2 and equation (1) - the two seem the same -- however they have separate columns in Table1 -- with similar results.
Please clarify the distinction.

- the method would benefit from a more principled way of choosing textures.
The proposed method consists of  (Sec 2.1., Data Generation: "... first select a pair of images from the training set uniformly at random, and then apply style transfer to blend their shape and texture".  How the algorithm decides which of the images in the pair is a texture and which one contains shape information? Can the authors make public a list of such fixed pairs, for other researchers to evaluate the methods on?
Just using style transfer (please also cite the method used for style transfer -- or if it is novel, please describe it) is very similar to the approach in Geirhos et al (2019), and it is not clear what the contribution is apart from changing the domain of the conflicting texture information.
It would be interesting to use a texture or material dataset (e.g. Materials in Context, Flickr Material Database, Describable Textures Dataset) as a source of texture for style transfer, or use class-specific textures (e.g. fur-like texture, to bias non-animal classes towards animal classes, e.g. cat, dog). 

- lack of evaluation of impact on other tasks, such as transfer learning:
How does debiasing affect the features on transfer learning tasks? For example, using the features from the debiased model (pretrained) on some external dataset, e.g. with a linear classifier on top of the features.

- style transfer seems to be performed on entire image -- e.g. Figure 2 - could the authors please clarify this?
The example in Fig. 1 shows using the shape of the object, while Fig. 2  There are more recent datasets, such as MS-COCO or LVIS which provide image segmentation which could be used for training and evaluation - and also for guiding the style transfer, e.g. restricting the texture style transfer or retexturing only to the object mask.
There is an evaluation on Semantic segmentation in Sec. 4.4; however, it is not clear from Fig 5 why the background is replaced with the texture source.


- not too much information on how / why the proposed method improves against FGSM.

- Please add the benchmarks (state-of-the-art and methods compared with, e.g. Geirhos et al in Table 2). It's difficult to chase the numbers in the paper to compare the method with existing work.

review_title: Promising results, but could benefit from more in depth analysis.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Note that I don't follow this research area closely, this is thus more an educated impression of the paper.

The paper is extremely simple. It builds on the idea of Geirhos et al. 2019 to use style transfer to augment IN training data by mixing shape and texture information but when using a training image resulting from mixing the shape of an image with label A and the texture of an image with label B, they use 0.8*A + 0.2*B as label (which is similar to mixup/cutmix, except the merging is done using style transfer). This alone seem to provide a clear boost on IN and different variants.

My main worry/question, since Geirhos paper has had a lot of attention (as well as mixup/cutmix), is why nobody has done this yet? The paper is very clear and there don't seem to be any hidden difficulty (except maybe using an auxiliary BN?) If indeed there are no similar work, I think the clear performance boost on IN with a simple and clearly explained method justifies publication.

I would however complain the analysis is not very strong. In particular, a natural alternative to the proposed approach would be to jointly train the convolutional layers but use three different final fc layers to classify synthetic images based on shape, synthetic images based on texture, or natural images. This experiment might help to clarify why the method works.

Other comments:
- I am not sure the "debiaised" term is really appropriate
- I would prefer to see the results of 4.3 in tables (some could actually be in table 1), they would be more readable, more exhaustive and allow more effective analysis (with the current presentation I have a hard time believing the current approach peforms better than cutmix) + I would like to actually see the curve corresponding to the sensitivity to gamma
- for the segmentation results, the results are really not sufficient: what are the results with gamma=1?
- end of p.7 "which can BE cropped"
- personally, I think it's a shame one has to go to p.5 to understand what the method is actually doing when it is so simple. In general, a lot could be shortened in the first 4 pages, and dedicated to better analysis and more exhaustive evaluation
review_title: very simple paper, good IN results, could use more detailed analysis

Meta-review: After the rebuttal stage, three of four reviewers recommend acceptance, and one gives a borderline score but argues they lean positive. Concerns seem well addressed; the method is simple yet effective.
Decision: Accept (Poster)