Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper makes the interesting observation that the disentangled affine transformation is what really matters in the Mixture BN, instead of the disentangled BN statistics as previously believed by the community.
strength_and_weaknesses: Overall, I think the message conveyed by this paper is simple yet important: The disentangled affine transformation is what really matters in the Mixture BN, instead of the disentangled BN statistics as previously believed by the community. The experimental results effectively support the conclusions. The paper is also well-written and is easy to follow for readers without expert knowledge in Mixture BN. 

I have one curious question for the authors:

In order to provide more evidence that clean and adversarial samples have similar BN statistics when we don't use disentangled affine transformation, could you please show similar visualization results as those in Figure 4, but remove all affine transformations in all BN layers? This can be achieved by disabling affine transformation when defining PyTorch BN layers. Usually, in normal training, disabling the affine transformations in BN layers won't harm performance much. 

clarity,_quality,_novelty_and_reproducibility: please see above
summary_of_the_review: please see above
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper firstly attempts at training a model with cross-domain BN statistics. Through untwining NS and AP in dual BN, it demonstrates that what makes it effective lies in two sets of AP instead of disentangled NS (as claimed in prior work).
It points out a hidden flaw in prior work for visualizing the NS to highlight a (large) domain gap between adversarial and clean samples. After fixing it, it reveals that the adversarial-clean domain gap is not as large as prior work suggests.
strength_and_weaknesses: \+ Through untwining NS and AP in dual BN, it demonstrates that what makes it effective lies in two sets of AP instead of disentangled NS (as claimed in prior work).

\+ It points out a hidden flaw in prior work for visualizing the NS to highlight a (large) domain gap between adversarial and clean samples. After fixing it, it reveals that the adversarial-clean domain gap is not as large as prior work suggests.

\- The technical contribution may be limited. This paper perform a lot of experiments with the dual BN for Hybrid-AT. It tests many different AT and BN settings. But it mainly uses Hybrid-AT techniques and variants of BN. The technical contribution may be limited. 

\- The application of the proposed method may be limited. The model must have BN layers so that it can be applied. Besides, it basically have two paths in the model for adversarial and clean data. During training, it knows how to activate each path since the model knows whether the data is clean or adversarial. But during inference without knowing the data is clean or adversarial, how can the model know which path to activate?  
clarity,_quality,_novelty_and_reproducibility: see the strength and weakness. 
summary_of_the_review: It mainly compares with the work (Xie & Yuille, 2020) and points out some key insights different from Xie & Yuille, 2020. It performs comprehensive experiments to check certain assumptions. But it mainly use existing techniques. The technical contribution may be limited. And the application may be limited. It may still have problems to apply this method in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the motivation of applying dual BN in a hybrid adversarial training scene, where the model is trained on both adversarial samples and clean samples. In contrast to the popular belief that normalization statistics should be estimated separately for clean samples and adversarial samples to achieve stronger robustness, this paper reveals that what makes dual BN effective lies in two sets of affine parameters. Moreover, the paper demonstrates that the adversarial-clean domain gap is not as large as many might expect and it is similar to its noisy-counterpart under the same perturbation/noise magnitude. Overall, this paper raises an interesting topic but the algorithm part is a bit weak.
strength_and_weaknesses: Strength:
This paper reconsiders the motivation of applying dual BN in adversarial training with hybrid samples and explores the effectiveness of normalization statistics and affine parameters separately.

This paper provides a closer looker at dual BN: disentangling the mixture distribution for normalization statistics and introducing two sets of affine parameters (AP). Then they find that the performance improvement mainly comes from more affine parameters.

The authors point out a visualization flaw in previous work: when AP is fixed, the robustness gap between the NS calculated on clean or adversarial samples is quite subtle.


Weakness:

This paper lacks a description of the existing development and problems in the field of applying batch normalization in adversarial training. It is not clear why they want to explore the reasonableness of the motivation for applying dual batch normalization proposed in the previous work. For example, is there some bad phenomenon with the application of dual batch normalization? or some limitations not considered in previous work? In a word, the authors do not give a reason why they carry out this work. 

The cross BN model looks like a mixture of ordinary BN and Dual BN. There is not much to take in this model. Although the authors provided two variants: setup1 and setup2, the contribution still looks incremental.

In Figure 2, Cross-AT method has a similar performance to Hybrid AT. However, from the statement that “the cross-AT still keeps its clean branch in the Hybrid-AT.., but the model weights are updated only by the adversarial branch’’, Cross-AT does not use clean samples to update the model weights, how does its performance on clean samples achieve the similar with Hybrid AT?

In the last paragraph of section 5, the paper claims that “future works utilizing dual BN in Hybrid AT are suggested to discard the two-domain hypothesis and embrace the two-task hypothesis”, the authors emphasize they have two intriguing findings. However, the authors have not clarified what contributions this work will bring to the development of the related field. 

In Page 2 Line 13, the authors claim that “This conjecture is motivated by an interesting observation This conjecture is motivated by an interesting observation that swapping NS in the Hybrid-AT has little influence on the model performance”. Although the experiment results shown in Figure 2 may prove the little influence of Normalization statistics, the experiment setting itself is hard to interpret. That is, the authors may not illustrate what motivates them to this experiment or if is there any interpretable meaning about the operation of swapping NS.  

In the last paragraph of section 3, the authors claim that “Such a success inspires us to suspect that the merit of dual BN over single BN in Hybrid-AT …”. However, as the results shown in Figure 2, although there is no significant drop in performance after “Cross-AT”, there is also no significant gain to prove the effectiveness of “Cross-AT”. Therefore, the claim of “success” may not be accurate.

In table 2, although the test accuracy of Setup1(AP_{adv}) has improved compared to Single BN and Dual BN in PGD-10 and AA, the test accuracy has obviously dropped in the clean dataset. So the claim on Page 2 Line 2 “two sets of APs alone significantly improve the robustness.” may be weakly supported by the experiment.

The paper conducts a lot of experiments but lacks some necessary theoretical analysis in dual BN and Hybrid-AT. Moreover, a novel effective method is expected according to the theoretical finding.


Minor issues:

In the second contribution listed in section 1: “We point out that it also introduces two sets of AP”, what“it”refers to is not clear.

In section 2.1, the part named “Experimental setups” is expected to be close to the experiment result in section 4 or section 5 of the article.

In Table 2, Table 3, there is no description of symbolic subscripts of AP_adv  AP_clean,.


clarity,_quality,_novelty_and_reproducibility: Please see my comments above.
summary_of_the_review: The contribution of the paper is limited and the reason why exploring the reasonableness of the motivation for applying dual batch normalization proposed in the previous work is not clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work reviewed the existing solution of Dual BN for adversarial robustness by focusing on the affine parameters in BN. It is argued that the stronger adversarial robustness in Dual BN is mainly attributed to additional affine parameters. Extensive ablation studies are presented to validate the argument. Moreover, it is argued that domain gap between adversarial and clean samples are not significantly larger than between noisy and clean samples, suggesting new explanations are required. 

strength_and_weaknesses: strength:

1. There are many empirical evaluations to support the hypotheses

2. The logic and experiments are easy to follow.

Weakness:

1. This is more like a commentary paper pertaining to an existing paradigm, i.e. Dual BN. Despite some insights into the effectiveness of affine parameters, there is no novel algorithm proposed.

2. From practical point of view, is there any recommended practice that one could adopt based on the analysis? Answering this quesiton could substantially improve the quality of this paper. 

3. I am wondering whether this analysis could generalize to modern architectures, e.g. ViT, where BN no longer exists.

Minor:

4. It is recommended to evaluate mixing up NS_adv and NS_clean in Table 1 such that the impact of using more data to estimate BN is eliminated.


clarity,_quality,_novelty_and_reproducibility: This paper presents the idea clearly. The hypothesis on affine parameters' impact is novel. The proposed ablation studies should be easy to reproduce.
summary_of_the_review: Overall, this work presents interesting empirical insight into the effectiveness of Dual BN. However, the major weakness lies in the lack of novel solutions. It is hard to grasp useful practical guidelines for improving adversarial robustness from this work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: We have set up a video call with reviewers to discuss this work and all reviewers agree that the observation and analysis is interesting and inspiring. However, it is hard to directly use this understanding to improve robustness in practice, which greatly limits the power and contribution of this work to a broader research community.
justification_for_why_not_higher_score: Limited impact.
Decision: Reject