Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to fuse multiple data augmentations in an adversarial manner. It achieves good robustness compared with AugMix on various natural distortions.
main_review: Pos:
1. The idea is novel and effective.
2. The paper is well written and easy to understand.

Neg:
1. I would like to see whether the AugMax could be combined with other data augmentations like CutMix.
2. The performance on SA seems to be comparable with AugMix. Could the authors provide results on more data augmentations?
limitations_and_societal_impact: None.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents an interesting new data augmentation called AugMax. It is motivated by diversity-hardness tradeoff of data augmentations. The authors suggest they are two complementary dimensions to achieve out-of-distribution robustness, and a disentangled normalization module termed DuBIN (Dual-Batch-and-Instance Normalization) could alleviate the resultant training difficulty. 
main_review: Pros:
-	Strong and clear motivation. The authors took a unified view towards two existing categories of data augmentation: random and adversarial. Either of the two was shown to improve domain shift robustness, but their complementariness was not previously revealed. The authors are the first to show a unification between the two is necessary to achieve robustness
-	Solid technical approach. The authors first proposed AugMax to reconcile diversity with hardness by learning an adversarially weighted combination of multiple random augmentations. Since this stronger augmentation leads to more diverse input distribution that is harder to fit, they further designed a new normalization strategy called DuBIN to disentangle the instance-wise feature heterogeneity of AugMax samples.
-	State of the art experimental results. Their simple approach yields significantly improved OoD robustness, outperforming strong previous methods by 2.01%, 2.52%, 1.84% and 15 2.13% on CIFAR10-C, CIFAR100-C, Tiny ImageNet-C and ImageNet-C.
-	Convincing ablation experiments. The authors compared different Diversity and Hardness Strategies The results confirm that diversity and hardness are two complementary dimensions, and their proper unification strategy is superior to alternative. 
-	Writing is very clear and easy to follow. 

Cons: 
-	I am not fully convinced why AdvMix is not as good as AugMax in Table 10. Both seem to be sensible options to me to combine diversity and hardness. The author also did not provide an insight why.
-	In Table 8, DuBIN seems to boost both AugMix and AugMax, at similar margins over DuBN. That sets doubts on whether the authors’ motivation of adopting DuBIN is really the case, e.g., because AugMax introduces a particularly heterogenous input distribution. 
-	AugMax looks computationally heavy, with default n = 10. There was also no mention of training overhead. Please address this
-	(minor) The overall innovation is reasonable, but not super strong. It is mainly composedly of existing techniques: adversarial training + AugMix; and BN + IN. To be fair, packing them together is still well-motivated and smart.

limitations_and_societal_impact: No concern was found. This paper should have positive social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel data augmentation method AugMax which adversarially learns the parameters for integrating randomly selected augmentation operators to achieve a unification of diversity and hardness. To solve the issue of feature heterogeneity incurred by AugMax, this paper proposes DuBIN which simultaneously uses DuBN and IN instead of only DuBN. AugMax-DuBIN can improve OOD robustness while almost maintaining generalization.
main_review: Pros:
1.	This paper is well organized. The motivation of AugMax-DuBIN is clear. AugMax utilizes the strengths of both AugMix and PGD attack to gain a unification of diversity and hardness. DuBIN combines IN with DuBN to reduce the feature diversity which is validated by smaller batch normalization variance brought by BuBIN.
2.	Observing from the experimental results, AugMax-DuBIN can significantly improve robust accuracy and decrease mean corruption error.

Cons:
1.	Please illustrate the differences between robustness accuracy (RA) and mCE Since RA is also evaluated as the average accuracy on corruptions. In addition, please provide more details about the corruption methods.
2.	It is better to provide the ablation study on the effect of early-stopping step $k$ in Algorithm 1. I suppose it is a critical hyper-parameter for the performance of AugMax-DuBIN. In addition, please illustrate the differences between AugMax with $k=10$ and PGD with 10 iteration steps. Also, it is better to conduct a fair comparison between FAT[1] which utilizes early-stepped PGD to improve robustness.
3.	This paper lacks some theoretical analyses to further validate the efficacy of AugMax.
4.	It is a bit wired to claim achieve a unification of diversity and hardness. Figure 1 illustrates AugMax gains a trade-off between diversity and hardness. It seems difficult to obtain enough diversity and enough hardness simultaneously. Please provide some discussions about this concern.

[1] Attacks which do not kill training make adversarial learning stronger. ICML 2020

limitations_and_societal_impact: Limitations: Please carefully refer to "Cons" in Main Review.
No potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed AugMax, which increases both the diversity and the hardness. To make the training easier, the paper design a normalization strategy DuBIN.  Experimental results show that the combination of AugMax and DuBIN exhibits good performance.
main_review: The paper can inspire future AT methods to deal with OOD samples. The idea is intuitive and easy to follow.

My concerns:
1. I am confused by the DuBIN layer and not sure how it works, can you show me the formulations of the 2 normalizations?
2. 
2. In Table 10, you reported the performance of AT on CIFAR100. Can your method be extended to the AT settings, i.e., can your method defend the adversarial examples.
3. You state that "AugMax can also lead to a higher feature heterogeneity, which may require larger model capacity to encode." Have you tried Different Normalization Layers on different models (e.g., resnet, wrn, etc.) to show how model capacity affects the performance.


limitations_and_societal_impact: Please refer to Main Review.
review_title: null

Meta-review: This paper focuses on an interesting new data augmentation called AugMax. The proposal is to reconcile diversity with hardness by learning an adversarially weighted combination of multiple random augmentations. Since this stronger augmentation leads to more diverse input distribution that is harder to fit, they further designed a new normalization strategy called DuBIN to disentangle the instance-wise feature heterogeneity of AugMax samples. The philosophy behind sounds quite interesting to me, namely, diversity-hardness tradeoff of data augmentations. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)