Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose Sandwich Affine strategy to separate the affine layer in BN into one shared sandwich affine layer, cascaded by several parallel independent affine layers. Such method should well address the inherent feature distribution heterogeneity in many tasks. Following this idea, the SaAuxBN and SaIN have also been introduced in this paper. The extensive experiments demonstrate the effectiveness of such methods in neural architecture search (NAS), image generation, adversarial training, and style transfer.


Pros: 
+ Neat motivation; 
+ Simple methods with clear contribution;
+ Extensive experiments;

Cons:
- The implementation details in Sec3.2 in not clear. (1) “Specifically, the number of independent affine layers in the SaBN equals to the total number of candidate operation paths of the connected previous layer.” Are you sure it doesn’t equals to the total number of candidate operations in next layer? (2) I think it will be better to add the SaBN in Fig3, and point out the correspondence between the candidate operations and the conditional affine layer. (3)  “The categorical index i of SaBN during searching is obtained by applying a multinomial sampling”. Could you please provide a more detailed explanation ?

- According to Fig.4, SaBN makes the searching process more stable, could you please provide some explanation about this phenomenon? What is the difference of the architecture parameters (i.e. alpha) learned by BN and SaBN? 
- Does the author use SaBN to search the architecture, and use BN to fine-tune the searched model? Some implementation details are missing.

- Do the results in Table2 denote the Top-1 accuracy? If yes, please specify in the caption.

- A lot of reference about the normalization methods published in recent years are still missing.


Other Comment:
If we change the order of the Sandwich Affine layer and Conditioned Affine layer, how about the performance in such case? Could you give some analysis about the different about your methods and the above case?
review_title: Simple approach that achieves acceptable results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers improving the performance of various normalizers by factorizing the affinity operations in normalization layer into on shared affinity operation, as well as several several independent affinity operation, each of which is corresponding to a specific data distribution. The experiments on various tasks (e.g. NAS, GAN, adversarial defense) demonstrate the effectiveness of proposed methods.

Overall, the idea is simple yet effective. It is a good practical way to improve the performance of BN and IN.

My questions are as follows:

(1)The section 3.2 is still unclear to me. The SaBN output multiple tensors, each of which is associated with the candidate operation in the previous layer. How to merge these tensors as the input for the next layer? Just do the summation? In A.2.3, why the number of independent affine parameter becomes n^2 in the case of multiple connected.

(2)What is the difference between the searched architectures by using BN, CCBN and SaBN? Could the authors give some analysis of the impact on the searched architecture when using SaBN?

(3)In Fig.11, the authors show some generated results on ImageNet. I think more results generated from the original methods should be given for the comparison.

(4)I want to know the limitations or some failure results of the proposed method, so that we can have a more comprehensive understanding of the proposed normalizer.
review_title: Simple idea and solution that work well in practice

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a new variant of batch normalization. Here are my comments.

1. paper organization. 

There are only four sections, and it seems that Section 3 is the main section, which contains both contributions and experiments. I would like the authors to clarify that the contribution is only in the formulation paragraph. It seems to me that the paper organization is poor. There are little descriptions, explanations, and discussions about the proposed method (Eqn 3)

2. the contribution.

If I understand correctly, the only difference between previous work (Eqn 2) and the proposed method (Eqn 3) is the additional parameter gamma-sa/beta-sa. At first glance, the two parameters can be absorbed into gamma_i and beta_i in Eqn 2. (I am saying eqn 2 and eqn 3 are equivalent by some reparameterization.

3. experiments

It seems that the experimental results in table 2 are rather poor. Could you give more explanation on this?

Overall, I don't think the current format, evaluations, and paper significance fit the conference.

review_title: Review summary

final_score: rating: 3: Clear rejection
review_content: The paper presents a modification to conditional batch normalization, wherein an extra affine layer is introduced between the standardization and the conditional affine layer. The empirical results show the benefits of introducing this extra "sandwich" affine layer.

The intuition behind the approach makes sense, but the formulation makes it difficult to see why SaBN is in fact beneficial compared to CCBN. It does not appear that the approach imposes any restrictions or regularization on the CCBN affine parameters; therefore the only difference between CCBN and SaBN seems to be a different parameterization. I would then expect both CCBN and SaBN to reach the same optimal training loss. It may be that the reparameterization provided by SaBN yields the optimization trajectories that lead to better-generalizing solutions, but it is not explained in the paper whether, or why, this happens. One way to probe this might be to reparameterize each gamma in BN or CCBN as $\gamma=\gamma_1 \gamma_2$ (or even $\gamma = \gamma_1^2$) and study the behavior of the resulting model.

The paper proposes to measure the heterogeneity in the found representations (between different branches of CCBN) via the CAPV measure. In its definition, I could not find what the overbars signify but I assume it means the average over the channels. Also, the indexing of gammas from 0 to N should probably be from 1 to C, for consistency with Eq. (2). The definition of CAPV as the variance of gammas seems problematic, however, in ReLU models with batch normalization: models whose gammas differ by a constant factor represent the same function, so the variance of gammas can be arbitrarily changed without affecting the model. A more useful measure of heterogeneity would need to take this scale invariance into account.

The paper shows several empirical studies, including one for architecture search -- with the main paper using DARTS, and the appendix using GDAS. This choice seems suboptimal, given that in the NAS-Bench-201 paper (https://openreview.net/pdf?id=HJxyZkBKDr), Table 5 seems to indicate that DARTS performs much worse than GDAS. In this work, the appendix devoted to GDAS seems to indicate that (1) there is no consistency on using or not using affine, between CIFAR-100 and Imagenet, and (2) GDAS-SaBN is not statistically significantly better than the better of GDAS and GDAS-affine.

The results in this paper are encouraging, but I believe the paper needs to explain more clearly why SaBN is expected to work (given that it preserves the hypothesis class as well as the minimum of the training objective). Additionally, since it appears to amount to a per-dimension reparameterization, the reader might expect that some of the other reparameterizations could have a similar effect (including such simple interventions as changing the learning rates for some of the gamma or beta parameters), and compellingly demonstrating that the specific reparameterization given by SaBN outperforms such alternatives would make the paper stronger.
review_title: Review

Meta-review: This work proposes a novel reparameterization of batch normalization that is hypothesized to give a better inductive bias for learning several tasks, including neural architecture search, conditional image generation, adversarial robustness and neural style transfer. The reviewers indicate that this is useful and is of interest to the ICLR audience, but they are not satisfied with the analysis offered in the paper. Specifically, the reviewers request that the authors provide a more detailed analysis of why the proposed reparameterization improves results, given that it does not change the expressive power of the model class. Additionally, the reviewers have some concerns about the structure of the paper. I therefore recommend rejecting the paper at this time.
Decision: Reject