Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of the paper:
This paper tries to study whether increasing shape-bias of a neural network trained with imagined will make it more robust to common corruptions such as gaussian noises.
The paper falsified this point by producing a data augmentation method which leads to more shape biased network yet less susceptive to common corruptios.
The paper further hypothesize that it’s the stylization augmentation that leads to increase robustness of the network by ablation studies.

Strength:
The paper does provide a counter example to the common hypothesis that increase shape bias can lead to more robust network. This provides insight to future researches on understanding how shape-bias and texture-bias affects on neural network robustness. If the results of the counter example is reproducible and significance, then the claim is convincing and the paper did a good job verifying such hypothesis.

Weakness:
1. The experiment results seemed to be limited in this dataset. In order to make such general claim, I would expect results for different datasets (e.g. CIFAR), large scale datasets (e.g. the whole imagined), and different network and training procedure (e.g. not just resent).
2. This might be some minor things, but it would be nice if there are statistic significance test for the results (or at least show the variance of couple runs). When I looked at the difference of the number, it would be nice that one can make sure such results is statistically significant with respect with difference runs.
3. Most of the paper is very empirical, and there is little insight or theory or principal ways that organize the results. 


Justifications:
While I do like the paper that provide insight and show negative results falsifying some prevalent claim, but the paper provides rather limited evaluation without theoretical insights. Since I’m not an expert in this field, I will recommend borderline scores to hear about the authors’ response.

-------------------
UPdate: the authors' reply address my concerns well, so I raise my rating to the acceptance side.
review_title: interesting results and experiments, might need more comprehensive experiments

final_score: rating: 7: Good paper, accept
review_content: The paper disproves the hypothesis that addressing shape bias improves robustness to corruptions of neural networks, which has been stated by the previous studies [1, 2]. The paper demonstrates that the degree of shape bias of a model is not correlated with classification accuracy on corrupted images via experiments. For the experiments, this paper presents two novel methods to encourage CNNs to be shape-biased: 1) edge dataset and 2) style randomization (SR). In the experiments, the authors train CNNs to be shape-biased to various degrees based on the proposed methods. Additionally, they compare test accuracies of the models to evaluate shape bias and robustness to corruption. In addition, this paper shows that through fine-tuning the affine parameters of the normalization layers, a CNN trained on original images can achieve comparable, if not better, performance than a CNN trained with data augmentation.

Pros:

- The paper clearly demonstrates that shape bias is not correlated with robustness to corrupted images. The test accuracy on texture-shape cue conflict images effectively indicates the degree of a model’s shape bias. Also, the authors visualize the results that explicitly compare the models’ accuracy on corrupted images.

- The authors present an interesting analysis towards Stylized ImageNet (SIN) dataset by separating the dataset into different factors that are used to generate SIN. According to this paper, the newly made datasets based on each factor leads a CNN to be shape-biased to various degrees. This provides insightful perspectives on shape bias and corruption robustness. Specifically, it is intriguing that a model trained on a stylized dataset using styles from in-distribution images achieves comparable performance compared to what uses styles from out-of-distribution images, such as paintings.

Cons:

- The novelty of the two proposed methods, edge dataset and SR, is limited. First, the edge dataset is created based on an existing model. The authors simply convert the non-binary edge map into a binary one. Furthermore, there are not enough experiments or evidence that validate the effectiveness of this method compared to the original method. Additionally, SR is also similar to existing methods, except that they change the target distribution from training samples’ distribution to a uniform distribution.

- According to this paper, a CNN trained on edge dataset is more shape-biased but achieves lower accuracy on corrupted images compared to a CNN trained on SIN. The authors state that the result indicates that shape bias is not correlated to corruption robustness.
However, it seems to be unfair to compare edge dataset and SIN to disprove the correlation between shape bias and corruption robustness.
Edge dataset, unlike SIN, does not contain any information except edges. Therefore, a CNN trained on edge dataset ‘learns only shape information, while a SIN-trained CNN learns other information as well, but ‘focuses more on the shape’. Since the edge dataset provides much less information of the original images, it is trivial that a CNN trained on edge dataset achieves lower performance compared to a SIN-trained CNN. 
The previous approaches [1, 2] mentioned in this paper also imply that shape bias is basically related to how much the model focuses on shape information, not to how much shape information the model is given.
Therefore, it would be more reasonable to compare a SIN-trained CNN with other CNNs that are trained on datasets containing a similar amount of information, but have different levels of concentration on shape information.

[1] Geirhos et al., “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.” ICLR’19

[2] Michaelis et al., “Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming.” ICLR’20


--------------------------
After rebuttal:

Thank you for the additional explanation.

The comments by the authors effectively address my concerns. Although the edge dataset and SR are quite similar to the existing methods, the authors clearly present the usefulness of them as well as their additional contributions. Also, additional training details support the adequacy of the comparison in the experiments. Therefore, I would like to increase my final score to 7: Good paper, accept.

review_title: I vote for ‘marginally above acceptance threshold’. While the novelty of the paper seems to be insufficient and the experiments seems to have several weaknesses, the paper presents an interesting analysis towards the relationship between shape bias and robustness to corruptions of neural networks

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ##Updated Review##

I'd like to thank the reviewers for their responses, and for updating to the much clearer naming scheme for the different methodologies!  Much easier to follow with those names.

I maintain that this is high quality novel work that contradicts a widely held belief within the field and as such is a clear accept.

# Main Idea
The main idea is that training to reduce texture bias in convnets in favor of shape bias is often thought to cause the concomitant increase in corruption robustness, but this has not been tested directly.  The authors propose a systemic study of this relationship and conclude that both shape bias and increase corruption robustness are both byproducts of style-variation during training, that is they share a common causal mechanism, but that shape bias does not itself cause corruption robustness.

They accomplish this by creating a new augmented dataset which encourages learning shape features (created from the edge maps of the training images) but does not also induce robustness against common corruptions.

#### Additional findings from the study:
Shape bias gets maximized when edge information and stylization are combined without including any texture information.

Corruption robustness is maximized by superimposing the image (and its textures) on the above stylized edges.

They propose that corruption robustness seems to benefit most from style variation in the vicinity of the image manifold.

# General Strengths
I think the paper makes a compelling case.

The Geirhos result, while elegant and interesting, has always struck me as a roundabout way to get to the shape bias question.  These authors more directly attack it, and show that the relationship doesn’t hold, while showing the real. Value of the Geirhos result was in showing that manifold-local variation increased robustness (and also consequently shape bias).

In addition to the elegance of the result, the authors use of “texture randomization via texture feature randomization” is a an elegant (and much faster) implementation than regenerating an entire dataset with many different textures.   The insight is good, the image itself doesn’t need to be rerendered, but the networks interpretation of its texture needs to be scrambled.  This doesn’t restrict you to the statistics of any database on which you might extract textures

# Weaknesses
Doesn’t report I-SIN results across ImageNetC corruption dataset.
Would have preferred to see the specific corruption type results broken out for more than just the two shown and the average robustness (the variation and performance on different types of corruption could be useful information).

Also make sure the names are consistent across graphs and paragraphs.  Not the best naming scheme (too many very similar acronyms which aren’t immediately evocative of the underlying point).
review_title: Clear, thorough and convincing demonstration that shape-bias does not inherently confer corruption robustness

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper delves deeper into understanding shape-based representation of CNNs in an empirical way. Based on the stylized images, it proposes to use edge maps to more explicitly feed shape information to learning models. Besides, the common way to let models learn the shape-based representation is to train on the dataset contained the shape information while the texture information is severely distorted. The paper takes the point of changing the statistics of feature maps would result in style changes and proposed style-randomization to help CNNs better focus on shape information. Also, it connects the biasing degree on shape information of models with the defensive performance against common corruptions, like Gaussian additive noise, blur, and etc. An intuitive conclusion was drawn that there is no clear correlation between shape bias and robustness against common corruptions, and justified by extensive experiments. 

The studied direction is important for understanding the learned representation of CNNs. The proposed points, using edge maps and style randomization, are simple and effective for making CNNs focus on shape information. Also, the discussion with the common corruptions is intriguing and helps us better understand the relationships between the shape-based representation and the robustness. 

The experiments are technically sound and results are sufficient to support their arguments, though more analysis would add support to the claims. The experimental setups are described in detail that makes it easy to replicate the experiments. Overall this paper is easy to follow. The reviewer would encourage the author to share their codes and datasets. 

-------------------------------------------------------------------
Some concerns and comments are listed below:

Since most of the experiments are conducted on ImageNet-20 and ResNet-18 (appendix lists few results on ImageNet-200 and ResNet-50), the reviewer doubts if the conclusions are still valid on different datasets and architectures. Especially for the aspects of the architecture, different architectures have different inductive biases and may result in different phenomena. 

According to Table 2, 3 and discussion on the bottom of the 6th page, the Stylized-Edge data would increase shape bias larger than Edge. This is a little bit counter-intuitive due to Stylized-edge only contain more uncorrelated texture information. The reviewer wonders if the quality of the Edge data is not very high and result in the phenomenon. To verify this, the author better to do experiments on the high-quality edge data with human annotations or performing edge algs on simple data.

Table 3 and A5 list the results of E-SIN and SE+IN. The reviewer wonders if there are any "intermediate" results such as E-IN, E+IN to better compare. Also, the reviewer is curious about the useful of superposition. Will the concat or mix data bring similar benefits, compared to superposition?

Currently, to test the degree of focusing on shape information, the author runs the random-shuffled images and tests on stylized-imagenet. This is good but the reviewer wonders if we directly test on the edge maps of validation sets.

To sum up, the reviewer thinks this paper would bring some new understandings to the community. 

--------------after rebuttal-------------------

I've read all reviews and the rebuttal, and thank the authors for their efforts and extra experiments. I think it is ok to be accepted due to provide further understandings about the learning representation with solid experimental results.
review_title: Take a step further on understanding shape-based representation

Meta-review: This work investigates the recently proposed hypothesis that enhanced shape bias improves neural network robustness to common corruptions. Several interesting experiments are performed to better understand the contributing factors that lead to improved robustness of models trained with texture randomization. Of particular note, the authors design a data augmentation strategy that verifiably increases the shape bias of model, but for which corruption robustness is not improved. Reviewers agreed that this is an interesting counter-example to the shape-bias hypothesis and improves our understanding of why stylization improves robustness. Given the carefully designed experiments investigating an important topic I recommend accept.
Decision: Accept (Poster)