Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##Updated Review## 

I'd like to thank the authors for their response and am looking forward to seeing the result of their edge detection comparison.  I maintain my review of this paper.

# Main Idea
The main idea is that many networks learn shortcuts based on texture.  The authors propose to use anisotropic diffusion to augment training using images with suppressed texture.  This retains edge information without texture.  The authors claims to show that they achieve SOTA results on detection and classification in different datasets, and that their method is particularly effective for transfer learning tasks.

This work interpolates between work that tries to replace and resample texture and work that attempts to remove texture by focusing exclusively on edges with no other image content.  What is left after texture suppression is not just edges, but natural looking images with no discernible high frequency texture information (but high frequency edge information intact).  Color and low freqncy texture remains intact in these images.  In addition, shape information about the underlying natural objects is well preserved.  I think this is important because it has the effect suppressing specific high frequency texture while preserving low-frequency natural image statistics from within the training distribution.

Additionally, it should be noted that the authors trained their networks with a combination of standard image net and their augmented version. Which means that the original texture information is still present in at least a part of the dataset.  The training set now contains the same edge and shape information with frequency-localized texture variation.

# Interesting observations:
Interestingly, when comparing to the original Geirhos Stylized ImageNet performance on MoCo V2, the method presented here significantly outperforms.
### The authors note that: 
One reason for this failure using the SIN dataset could be that the model is able to memorize the textures in the stylized images since it only has 79,434 styles. This is not a problem in the original fully-supervised setting where the authors used SIN for supervised image classification. In that case, the network can learn to ignore texture to discriminate between classes. 

This points to the benefit of retaining some set of natural image statistics (from a larger distribution) within the training set.

Very intriguingly, the presented method also outperforms a network trained on SIN when evaluated on sketch-image net, a dataset comprised almost entirely of edge-like sketch images lacking almost all texture.  I find this result to be the most surprising of all those presented as these image lack all natural image texture statistics.  The performance is not out of this world, but it outperforms training on native image net and on SIN.  It would be interesting to see the comparison of performance here to a network trained exclusively on image-net images with only the edges extracted (perhaps using canny edge detectors).

# Weaknesses
Avoid sentences like the following as they are not completely supported by the work. "This forces the network to be less reliant on lower-level information to solve the pretext task and, hence, learn representations that focus on higher-level concepts.”  

Sometimes the magnitude of differences in results between different methods (while being presented in tables as relatively modest improvements) are overstated in the text.  The results speak for themselves and there is no need to overstate small differences. 

I find the saliency maps unconvincing (both because they are only a cherry picked subset) and because attending to the entire object does not mean the network is not also attending to the object’s texture.  Additionally, these methods are highly subjective in their evaluation (and likely to change depending on the method for saliency map production). Finally, the supplement shows examples that in fact don’t follow the established pattern presented in the original paper.  Finally, the analysis and results of the paper do not depend on this result and as such I think you should just remove it. 
review_title: Interesting exploration of how texture suppression assists in domain generalization

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper tackles the issue that CNNs may over-emphasize texture. The authors propose to suppress texture with anisotropic diffusion, as a way of data augmentation in training. The strategy is shown to be helpful for various downstream tasks, especially in a transfer learning setting.

I believe the paper has several advantages:
+ The proposed method is simple and well-motivated, and seems to provide consistent boosts across several settings.
+ The paper is clearly presented, figure 1 provides a good summary of the paper.
+ The visual analysis in sec 4.3 is quite interesting. The proposed technique is helpful in reducing the attention on texture in these examples.

Meanwhile, I have several concerns:
- technical novelty: the main contribution of the paper is proposing anisotropic diffusion as a way of data augmentation, which is somewhat trivial. Though it's quite simple, I feel it's quite heuristic and it's not supported by sufficient theoretical analysis or exploring of different design choices.
- experimental analysis: I am happy to see that the proposed technique achieved improvements across different settings. However, I think additional analysis is needed to better demonstrate that the representation focuses less on texture. Additional experimental analysis can also help to better understand the proposed technique and gain further insights.
- transfer learning: I am also not fully convinced by the effectiveness of the transfer learning setting. The experiments are conducted on Sketch-ImageNet, where images have little texture. Therefore, augmentation by suppressing texture naturally has an advantage in such a setting. I am a bit skeptical about the proposed technique in a more general setup, such as learning from synthetic data. (synthetic data usually has a different texture than real imagery). Besides, I think more baselines need to be added for transfer learning setting, such as techniques used in domain adaptation.

review_title: Interesting idea, however not enough technical contribution or experimental analysis.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
In this paper, the authors propose suppressing texture (as low level cue) as it seems to provide shortcuts that prevent the network from learning higher level representations. The method achieves state-of-the-art on Sketch Imagenet (> +10%)

Pros:
- identifies the low level texture as a source of shortcuts which prevent learning, especially in unsupervised or self-supervised setting.
- simple idea, building on earlier research from the 90s; Gaussian blur was used before as source of augmentation; this work builds on top of the idea, and ensures that edges are preserved, mitigating the effect of texture influence.
- great improvement on Sketch ImageNet;

Cons / areas of improvement:
- the authors incorrectly claim testing on DTD dataset -- Please revisit Sec 4.2.1; Based on the citation, it looks like the authors are evaluating on a synthetic dataset, based on ShapeNet, with textures from DTD; Please clarify this in the paper.
- Table2: needs clarification, at least on DTD - is it using transfer learning on the right dataset?  
- would be great to have one extra line with state-of-the-art for each of the datasets, e.g. in the form: 70.21 [citation]; when reporting numbers on DTD, it is fine to use only the first split (see SimCLR), as long as the evaluation protocol is specified in the dataset description paragraph.

Style, typos, glitches:
- wrong citation for DTD dataset; (Newel & Deng use objects from ShapeNet, and apply textures from DTD: https://www.robots.ox.ac.uk/~vgg/data/dtd/#citation).
- please explain all the notation in equations (1) and (2), e.g. \delta I, what is “c”, in equation (1) and (2); is it the same c? 
Table1: citation or Stylized ImageNet; -- what is referred to as “Supervised”? 

The idea seems simple, it is well validated through experiments, but needs some improvements in clarifying the evaluation on transfer to DTD dataset.
review_title: Simple yet effective idea, clearly presented.

Meta-review: This paper adopts an idea from 1990 for reducing reliance on texture, and shows that this idea improves the quality of visual representations in a variety of tasks. Initially reviewer scores were 7/5/4 but those improved slightly to 7/6/4 (changed in comment, not final review) after the rebuttal stage-- thus, one accept, one borderline, and one reject score. Reviewers have concerns about the great simplicity of the approach, where the only contribution is from prior work. Some reviewers request comparisons in a proper domain adaptation setting. While the large number of experimental settings somewhat balance out the concerns, overall, support for acceptance is not strong enough at this stage.
Decision: Reject