Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**

This paper proposes two novel norms: Selfnorm and Crossnorm for model robustness. Selfnorm recalibrates style in features to reduce texture sensitivity. Crossnorm performs style augmentation (swap channel-wise means and variances) to reduce texture bias. The authors also give some discussions on the distinction and connection between SelfNorm and CrossNorm. The proposed norms can be applied to different settings and tasks. Their method shows state-of-the-art robustness performance on both fully and semi-supervised settings, and classification and segmentation tasks.

**Clarity**

- The paper is well organized and easy to read.

**Strengths**

- Contributions clearly stated and validated.
- Comprehensive experiments to show the effectiveness of their method. 
- The proposed method is domain agnostic and can be applied to different settings and tasks.

**Weaknesses**

- SelfNorm is a learned normalization, thus I think the statement "they take opposite actions ... at separate stages testing v.s. training ..." (page 1) is confused. Should SelfNorm be used in both testing and training?

- In the experiments, both SelfNorm and CrossNorm units are placed in a ResNet block. What is the order of their placement? It's not clear in the paper.

- It is better to keep the original formula (as shown in Figure 1(a)) in Eq.2 as well so that it is easy for readers to understand.

- The related residual channel attention (CA) [a] should be discussed in this paper. Because the residual channel attention can be also considered as a normalization that recalibrates/rescales features with channel-wise attention. The authors are suggested to give some discussion on the difference between the CA mechanism and proposed Norms. CA is also a unit that is inserted in the ResNet block so that it's interesting to have an experimental comparison as well.

[a] Zhang at al., Image Super-Resolution Using Very Deep Residual Channel Attention Networks, in ECCV'18

review_title: Interesting norms for model robustness

final_score: rating: 7: Good paper, accept
review_content: This paper investigates the CNN model robustness against problems, e.g., texture sensitivity and bias. In particular, this paper proposes to recalibrate style using SelfNorm motivated by the fact that attention help emphasize essential styles and suppress trivial ones and reduce texture bias using CrossNorm by swapping feature maps within one instance. The opposite two processes forms a unity of using style to advance model robustness. The intuition of the proposed method is clearly presented and the paper is well structured. Detailed comments are summarized as follows:

Pros:
	- The paper contributes a solution by forming a unity of opposites in using style for model robustness. Both the motivation and intuition are clearly presented. The idea is somewhat novel for me.
	- The method can be easily implemented and integrated into classical frameworks.
	- Extensive experiments under different settings and tasks show the effectiveness of the proposed method.

Cons:
	- In Sec. 3 Unity of Opposites, the authors explains that SelfNorm works during testing while CrossNorm functions only in training. The statement cannot clearly illustrates the process in training and testing. I suppose both SelfNorm and CrossNorm would be used during training to get a trade-off point and only SelfNorm is used during testing. It is encourage to add an algorithm flowchart to indicate the process.
	- The authors explains SelfNorm recalibrate feature style while  CrossNorm performs style augmentation. It is suggested to visualize the attention/feature maps before and after using the proposed techniques to illustrate their benefits.
The experiment results of SNCN are not significant compared with AugMix though their combination achieves new state-of-the-art.
review_title: he paper contributes a solution by forming a unity of opposites in using style for model robustness. Both the motivation and intuition are clearly presented. The idea is somewhat novel for me.

final_score: rating: 7: Good paper, accept
review_content: 1. Summary

The paper presents two new methods to improve corruption robustness and domain generalization: SelfNorm, a way to adapt style information during inference, and CrossNorm, a simple data augmentation technique diversifying image style in feature space. Both methods are tested on Cifar10/100-C, ImageNet-C, Semi-Supervised Cifar and GTA V -> Cityscapes.


2. Strengths
+ The approach is straight forward and apparently very effective
+ The evaluation is performed reasonably
+ The method is tested not only on corruptions but also on domain adaptation. This is great as both tasks can be seen as two sides of the same metal and it is great to see more methods testing on both.

3. Weaknesses
- The main problem is the write up. In the introduction "texture sensitivity" is introduced as a concept but never picked up later on. In general the motivation is very high level drawing a lot on the concepts of style, texture and content but the method itself is rather down to earth modifying instance normalization parameters. In my opinion the high level arguments and colorful terms (style, shape sensitivity, unity of opposites etc.) distract rather than add to the story. They may serve as inspirations but without extensive experiments it is hard to related concepts like style to the manipulation of network parameters. 
- The ablation study was hard to follow and tbh I think it could have been shortened presenting only the most important results in the paper and the rest in the appendix.
- Conversely I felt the paper would profit from more figures and visualizations (e.g. Figure 7). The method is very simple and I think that is a major strength. It should thus also be possible to present it in a more concise form.


4. Recommendation 

As it is right now I think the paper has to be rejected because the write up is just too chaotic and vague. I do however like the method a lot and I would vote for accept if it was rewritten substantially to focus more on an understandable presentation of the method than abstract concepts and colorful terms. 


4. Questions/Recommendations
- It is probably beyond the scope of the review period but a demonstration of the technique on some non image data would be amazing making the theoretical argument in Section 4.1 that the method can work on other domains much more powerful.
- It would be nice to include a discussion of "Improving robustness against common corruptions by covariate shift adaptation".

6. Additional feedback 
- Figures 1 and 2 are a bit hard to understand.
- IMHO it would be more interesting to include Figure 7 in the paper and move Figure 3 into the appendix.
- The use of IBN should be mentioned in Table 2
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
The paper introduces two simple modules, SelfNorm (SN) and CrossNorm (CN), that are highly modular and can theoretically be attached to different parts of the CNNs to control the balance between style and content cues for their recognition. SN is a lightly parametrised module that gives freedom to the CNN to dynamically re-adjust the means and variances of the feature maps in the original instance normalization framework. The authors argue that SN then learns to emphasize _important styles_ and suppresses less important ones, with the underlying assumption that the first and second-order statistics are often sufficient and necessary representations for style. CN is a twin invention that randomly swaps the mean and variance statistics of features of two channels. This leads to diversified virtual styles in the training data, effectively factoring out the model's dependence on style cues for recognition, according to the authors.

**Pros**
It is great news to the field that such simple recipes introduce gains in the performances. From the industrial point of view, the expansion of choices for model design means the likelihood of introducing uniform gains across the board in industrial applications (think about how much $ worth it will be if it cuts the error rate by 0.1 pp across all image classifier applications). From the research point of view, it is exciting to confirm yet again in 2020 that there is much room for improvement in those modular components, after the introduction of SE blocks and the likes.

**Cons**
The above excitement should be checked with a pinch of salt. I find it difficult to follow the rationale behind key assumptions (e.g. that shape = content and texture = style) and the experimental section is, I have to admit, a bit chaotic. And I suspect if the chaos is intended, for downplaying the discouraging side of the results.

1. **Shape=content and texture=style? No.**
First of all, we need a decent, if not mathematical, definition of the four terms above. To me, "content" is the causal cue that constitutes the GT label for the task at hand; I refer to all the rest cues as "style". "Texture" to me is a local pattern that can be captured by sliding windows of, say, size 10x10 pixels and "shape" is any pattern that is more global than texture. Under this terminology, I find it hard to agree that texture is not content. Texture does contribute, as the causal cue, to the final task at hand in many real-world computer vision applications - texture classification, fine-grained cat categorization, medical diagnosis with CT scans, and semantic parsing for detecting snowy and watery road conditions, to name a few. 

2. **The oversimplistic view of the world that underlies this paper makes the justifications for the proposed methods all the more fragile.**
It is difficult to agree that SN and CN, which are argued to control style and content for the benefit of the recognition task at hand, are really working as speculated. It is also disputable whether the first and second-order statistics really encode the style and/or texture component. I find it a bit dangerous to let such a convenient and simplifying view of the matters be published and guide researchers in the field to adopt the same kind of viewpoint. In the revision, try to introduce more depths in your arguments and include more empirical analysis to make a point SN and CN work in the promised way.

3. **Introduce order in the experimental reports.**
So I made a table below to summarize the experimental setups in this paper (sorry for the dots everywhere - formatting tricks ;)). I'm having a hard time convincing myself that SN and CN really work empirically, given those highly specific choices of settings per task and analysis. For example, why is ImageNet performance for SN and CN not compared against augmentation baselines considered in the CIFAR experiments in Tab1? What are the individual performances of CN and SN for CIFAR on those 4 architectures in Tab1? What is the effect of location for SN in a CNN (equivalent analysis for CN is presented in Tab4)? There are many, many questions unanswered. I do observe a few improvements introduced by the two modules here and there, but I can't forgo the impression that these are only selected highlights that comply with the authors' arguments. Please introduce the much-needed order in the experimental section, and only then will I be able to assess if the new technology is truly innovative.
| .  Section  .  | .  Data  . | .  Arch  . | .  Evaluation  . | .  Baselines  . | .  Authors' methods  . | 
| -- | -- | -- | -- | -- | -- |
| .  Tab1  . | .  CIFAR  . | .  4 archs  . | .  mCE,CleanAcc  . | .  Cutout,Mixup,Cutmix,AA,Advtr,AugMix  . | .  SNCN, SNCN+AugMix |
| .  Fig2  . | .  CIFAR  . | .  28-2WideResNet  . | .  mCE,CleanAcc  . | .  WA,RA  . | .  CN |
| .  Fig4  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | .  SN,CN |
| .  Tab4  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | . CN |
| .  Tab5  . | .  CIFAR  . | .  40-2WideResNet  . | .  mCE  . | .  VanillaModel  . | . SN,CN,SNCN,SNCN+Crop,SNCN+Crop+CR |
| .  Tab2  . | .  ImageNet  . | .  ResNet50 . | .  mCE,CleanAcc  . | .  PU,AA,MaxBlur,SIN,AugMix  . | . CN,SN,SNCN+AugMix |

**Key reasons for the rating**
The simplicity and effectiveness of the technologies argued by the authors are eclipsed by the oversimplifying assumptions and inefficient experimental exposition. The rating reflects this disappointment. Please aim to improve the paper in the rebuttals and paper revisions.
review_title: Not sure yet if the paper can be published.

Meta-review: 
 This paper proposes two mechanisms, SelfNorm (used during training and inference) leveraging an attention-based recalibration of mean and standard deviation for instance normalization, and CrossNorm which performs cross-channel swapping of mean/stdev. Is is shown that the combination (often combined with AugMix) performs well across several datasets in terms of model robustness. Overall the paper has strength in the fact that the method is interesting, simple to implement, and modular. However, reviewers brought up a number of issues including the overstated motivation/writing, lack of clarity, and most importantly need for clear experimental results (comparing to uniform/standard baselines) and identification of the separate mechanisms. It is especially uncertain why it is necessary that they are used *together* (often with AuxMix as well) to obtain the strong performance. As a result, the score for this paper is borderline, tending towards a weak acceptance. 

It is appreciated that the authors provided a lengthy rebuttal, including new results in a different domain (NLP); however, the reviewers agreed that not all of the concerns were addressed.  After a lengthy discussion, all of the reviewers agree that while the method is simple, modular, and effective when combined (hence the positive scores from some reviewers), the authors fail to describe the underlying reason for the method's gains, especially with respect to the individual parts (SelfNorm vs. CrossNorm) and why the results only come when these rather independently derived modules are used together. The exposition of the experimental results, with differing baselines/conditions that make it very hard to understand where the effect is coming from, exacerbates this issue. 

As a result of these concerns, I recommend rejection of this paper. However, the method is interesting and results promising, so I hope that the authors can clarify the writing and improve the presentation of the results (specifically separating out the effects of SelfNorm and CrossNorm, as well as analyzing how they interact together to improve results) and submit to a future venue. 
Decision: Reject