Sandwich Batch Normalization
Authors: Xinyu Gong
Wuyang Chen
Tianlong Chen
Zhangyang Wang
Venue:  (2021)
Keywords: normalization

Abstract: We present Sandwich Batch Normalization ($\textbf{SaBN}$), a frustratingly easy improvement of Batch Normalization (BN) with only a few lines of code changes. SaBN is motivated by addressing the inherent $\textit{feature distribution heterogeneity}$ that one can be identified in many tasks, which can arise from model heterogeneity (dynamic architectures, model conditioning, etc.), or data heterogeneity (multiple input domains). A SaBN factorizes the BN affine layer into one shared $\textit{sandwich affine}$ layer, cascaded by several parallel $\textit{independent affine}$ layers. Its variants include further decomposing the normalization layer into multiple parallel ones, and extending similar ideas to instance normalization. We demonstrate the prevailing effectiveness of SaBN (as well as its variants) as a $\textbf{drop-in replacement in four tasks}$: neural architecture search (NAS), image generation, adversarial training, and style transfer. Leveraging SaBN immediately boosts two state-of-the-art weight-sharing NAS algorithms significantly on NAS-Bench-201; achieves better Inception Score and FID on CIFAR-10 and ImageNet conditional image generation with three state-of-the art GANs; substantially improves the robust and standard accuracy for adversarial defense; and produces superior arbitrary stylized results. We also provide visualizations and analysis to help understand why SaBN works. All our codes and pre-trained models will be released upon acceptance. 

Under review as a conference paper at ICLR 2021
SANDWICH BATCH NORMALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present Sandwich Batch Normalization (SaBN), frustratingly easy improve- ment of Batch Normalization (BN) with only a few lines of code changes. SaBN is motivated by addressing the inherent featren distribution heterogeneity that one can be identified in many tasks, which can arise from model heterogeneity (dy- namic architectures, model conditioning, etc.), ), or data heterogeneity (multiple in- put domains). A SaBN factorizes the BN affine layer into one shared sandwich affine layer, cascaded by several parallel independent affine layers. Its variants include further decomposing the normalization layer into multiple parallel ones, and extending similar ideas to instance normalization. We demonstrate the pre- vailing effectiveness of SaBN (as well its variants) as a drop-in replacement in four tasks: neural architecture search (NAS), image generation, adversarial train- ing, and style transfer. Leveraging SaBN immediately boosts two state-of-the-art weight-sharing NAS algorithms significantly on Inception Score and FID on CIFAR-IO and ImageNet conditional image genera- tion with three state-of-the art GANs; substantially improves the robust and stan- dard accuracy for adversarial defense; and produces superior arbitrary stylized results. We also provide visualizations and analysis to help understand why SaBN works. All our codes and pre-trainee models will be released upon acceptance.
achieves better
INTRODUCTION
This paper presents simple, liggh, and easy-to-implement modification of Batch Normal- ization (BN) (loffe & Szegedy 2015), yet strongly motivated by various observations (Zajac et al.) 2019 Deecke et al. 20188] Xie et al. 2019; Xie & 2019) drawn from a number of application fields, that BN has troubley standardizing hidden features with very heterogeneous structures, e,g. from a multi-modee modal distribution. We call the phenomenon feature distribution heterogeneity. Such heterogeneity of hidden features could arise from multiple causes, often upplication-depeneent
2019)
One straightforward cause is due to input data heterogeneity. For example, when training a deep network on diverse set of visual domains, that possess significantly different statistics, BN is found to be ineffective normalizing the activations with only a single mean and variance (Deecke)
ct al. 2018), and often needs to be re-set or adapted (Li al. 2016).
Another intrinsic cause could arise from model heterogeneityy i.e., when the training is, or could be equivalently viewed as, on set of different models. For instance, in neural architecture search (NAS) using weight sharing (Liu et al., 2018: Dong & Yang 2019), training the super-network during the search phase could be considered training a large set of sub-models (with many over- lapped weights) simultaneously. As another example, for conditional image generation (Miyato) al. 2018). the generative model could be treated set of category-specinne sub-models packed together, one of which would be "activated"" by the conditional input each time.
The vanilla BN (Figure [(a) (a)) fails to perform well when there is data or model heterogeneity. Recent trends split the affine layer into multiple ones and leverage input signals to modulate or select be- tween them (De Vries al. 2017 Deecke al. 2018) (Figure (b)); or even further, utilize several independent BNs to address such disparity (Zajac et al. 2019 Xie et al.) 2019 Xie & Yuille 2019) Yu et al. 2018) (Figure (c)). While those relaxations alleviate the data or model heterogeneity, suggest that they might be "too loose" in terms of the normalization or regularization effect.
Let us take adversarial training (AT) (Madry et al. 2017) as concrete motivating example to illus- trate our rationale. AT is by far the most effective approach to improve a deep model's adversarial robustness. The model is trained by mixture of the original training set ("clean examples") and

---

Under review as a conference paper at ICLR 2021
Affine
Aftines
Affine
Sandwich Affine
Normalization
Normllization
Normaliaation
Feature map
Feature map
Conditional information
map
Conditional information
(a) Batch Norm
(b) Categorical Conditional Batch Norm
(c) Sandwich Batch Norm
Figure 1: Illustration of (a) the original batch normalization (BN), composed of normalization layer and one affine layer. (b) Categorical Conditional BN, composed one normalization layer following of inde- pendent affine layers to intake conditional information; (c) our proposed Sandwich BN, sequentially composed of one normalization layer, one shared sandwich affine layer, and a a set of independent affine layers.
its attacked counterpart with some small perturbations applied ("adversarial examples"), Yet, latest works (Xie al., 20T9 Xie & Yuille 20T9) pointed out that clean and adversarial examples behave like two different domains with distinct statistics on the feature level (Li & Li. 2017 Pang et al. 2018). Such data heterogeneity puts vanilla BN jeopardy for adversarial training, where the two domains are treated as one. (Xie et al. 2019 Xie & Yuille 2019) demonstrated a helpful remedy to improve AT performance by using two separate BNs for clean and adversarial examples respectively, which allows either BN to learn more stable and noiseless statistics over its own focused domain.
But what may be missing? Unfortunately, using two separate BNs ignores one important fact that the two domains, while being different, are not totally independent. Considering that all adversarial images are generated by perturbing clean counterparts only minimally, esize the two domains to be largely overlapped least (i.e., they still share some hidden features despite the different statistics). To put simple: while it is oversimplified to normalize the two domains "same one", it is also unfair and unnecessary treat them "disparate two".
convincing to hypoth-
More application examples can be found that all share this important structural feature prior, that (informally) call as "harmony in diversity". For instance, weight-sharing NAS algorithms (Liu iet al. 2018: Dong & Yang. 2019; Yu et al., 2018) train a large variety of child models, constituting model heterogeneity; but most child architectures inevitably have many weights in common since they are sampled from the same super net. Similarly, while conditional GAN (Miyato et al., 2018) has to produce diverse images classes, those classes often share the same resolution and many other dataset-specific characteristics (e.g., the object-centric bias for CIFAR images); that is even more true when the GAN trained produce classes of one super-category, e.g.. dogs and cats.
Recognizing the need to address feature normalization with "harmony in di- versity", we propose a new SaBN illustrated in (c). SaBN modifies BN in "frustratingly simple" way: it is equipped with two cascaded affine layers: shared unconditional sandwich affine layer, followed by a set of independent affine layers that can be conditioned. Compared to Cate- gorical Conditional BN, the new sandwich affine layer is designed to inject inductive bias, that all re-scaling transformations will have a shared factor, indicating the commodity. Experiments on the applications of NAS and conditional generation demonstrate that SaBN addresses the model heterogeneity issue elegantly, and improves their performance in a plug-and-play fashion.
Our Contributions:
To better address the data heterogeneity altogether, SaBN could further integrate the idea of split/auxiliarry BNs (Zajac et al., 2019 Xie et al., 2019, Xie Yuille 2019) Yu et al.] 2018). to decompose the normalization layer into multiple parallel ones. That yields the new variant called SaAuxBN. We demonstrate it using the application example of adversarial training. Lastly, we ex- tend the idea of SaBN to Adaptive Instance Normalization (AdaIN) (Huang Belongie 2017), and show the resulting SaAdaIN to improve arbitrary style transfer.
RELATED WORK
2.1 NORMALIZATION IN DEEP LEARNING
Batch Normalization (BN) Loffe Szegedy 2015 made critical contributions to training deep convolutional networks and nowadays becomes cornerstone of the latter for numerous tasks. BN normalizes the imput mini-bater batch of samples by the mean and variance, and then re-scale them with learnalle affine parameters. The success of BNs was initially attributed to overcoming internal co-
2

---

Under review as a conference paper at ICLR 2021
variate shift (loffe & Szegedy 2015), but later on raises many open discussions on its effect of im- proving landscape smoothness (Santurkar et al. 2018); enabling larger learning rates (Bjorck et al. 2018) and reducing gradient sensitivity (Arora et al. 2018); preserving the rank of pre-activation
weight matrices (Daneshmand et al. 2020); decoupling feature length and direction (Kohler et al.,
2018|; capturing domain-specific artifacts (Li et al. 2016); reducing BN's dependency hatch SizeIoffe (2017): (inghh& & Krishnan| (2020); Preventing model from elimination singularities QQao al. 2019) and even characterizing an important portion of network expressivity (Frankle al.
2020 Inspired by BN, a number of task-specific modifications are proposed by exploiting different
normalization axes, such as Instance Normalization (IN) (Ulyanov et al.) (016) for style transfer;
Layer Normalization (LN) (Ba et al.] (016)) for recurrent networks; Group Normalization (GN)
(Wu & He, 2018) for tackling small batch sizes; StochNorm Kou et al. (2020) for fine-tuning task; Passport-awas Normalizatior/Zhang et al. (020) for model TP protection; and Li et al. (019a);
Wang ct al. (2020); Zheng et al. (2020) for image generation.
Several normalization variants have been proposed by modulating BN parameters, mostly the affine layer (mean and variance), to improve the controlling flexibility for more sophisticated usage. For example, Harm al. Vries et al., (017) presents Conditional BN, whose affine parameters generated as a function of the input. Similarly, Conditional IN (Dumoulin et al., (016) assigns each style with independent IN affine parameters. In (Miyato al. (0188), the authors developed Categorical Conditional BN for conditional GAN image generation, where each generated class has its independent affine parameters. Huang & Belongic (Huang & Belongie (017)) presented Adaptive IN (AdaIN), which used the mean and variance of style image to replace the original affine parameter, achieving arbitrary style transfer. Spatial adaptivity (Park et al. 2019) and channel attention (Li et al. 2019b) managed to modulate BN with higher complexities.
A few latest works investigate to use multiple normalization layer instead of one in BN. (Deecke) eet al. 2018) developed mode normalization by employing mixture-of -o--eeerteeeeeeeeeeeeeeeeeeeeeee to separate incoming data into several modes and separately normalizing each mode. (Zając et al.) 2019) used two separate BNs to address the domain shift between labeled and unlabeled data in semi-supervised learning. Very recently, (Xie & Yuille, 2019; Xie et al., 2019) reveal the two-domain issue in adversarial training and find improvements by using two separate BNs (AuxBN).
2.2
BRIEF BACKGROUNDS FOR RELATED APPLICATIONS
We leverage four important applications test beds. All of them appear to be oversimplified by using the vanilla BN, where the feature homogeneity and heterogeneity are not properly handled. We briefly introduce them below, and will concretely illustrate where the heterogeneity comes from and how our methods resolve the bottlenecks in Sec.
Generative A Adversarial Network Generative adversarial networks (GANs) have been prevailing since its origin (Goodfellow al.) 2014a) for image generation. Many efforts have been made to improve GANS, such as modifying loss function (Arjovsky et al., 2017; Gulrajani et al. 2017
2018), improving network architecture (Zhang al. [0188] Karras al. 2019 Gong et al. 2019) and adjusting training procedure (Karras et al. 2017). Recent works also tried to improve the generated image quality by proposing new normalization modules, such as Categorical Conditional BN and spectral normalization (Miyato et al. 2018).
Neural Architecture Search (NAS) The goal of NAS is to automatically search for an optimal model architecture for the given task and dataset. It was first proposed in (Zoph & Le 2016) where reinforcement learning algorithm iteratively samples, trains and evaluates candidate models from the search space. Due to its prohibitive time cost, weight-sharing mechanism was introduced (Pham
let al.
2018) and becomes popular strategy to accelerate the training of sampled models (Liu
et al.
2018). However, weight-sharing causes performance deterioration due to unfair training (Chu
ct al.
2019 In addition, few NAS benchmarks (Ying et al., 2019 Dong & Yang) 2020
Zela
et al.
2020) were recently released, with ground-truth accuracy for candidate models pre-recorded, enabling researchers to evaluate the performance of search method more easily.
Adversarial Robustness Deep networks are notorious for the vulnerability to adversarial attacks (Goodfellow et al.) 2014b). In order to enhance adversarial robustness, countless training approaches have been proposed. (Dhillon et al. 2018 Papernot& & McDaniel, 2017, Xu et al.) 2017 Meng & Chen 2017 Liao et al. 2018: Madry et al. (017)). Among them, adversarial training (AT) (Madry
ct al. 2017) are arguably the strongest, which train the model over a mixture of clean and perturbed
data. Overall, the normalization in AT has, to our best knowledge, not been studied in depth. A
3

---

Under review as a conference paper at ICLR 2021
pioneer work (Xie et al., 2019) introduce auxiliary batch norm (AuxBN) to improve the clean image recognition accuracy.
Neural Style Transfer Style transfer is a technique generating stylized image, by combining the content of one image with the style of another. Various improvements made on the normalization methods in this area. [Ulyanov et al. (2016) proposed Instance Normalization, improving the styl- ized quality of generated images. Conditional Instance Normalization (Dumoulin et al., (0166) and Adaptive Instance Normalization (Huang & Belongie 2017) are proposed, enabling the network to perform arbitrary style transfer.
3 SANDWICH BATCH NORMALIZATION
Formulation: Given the input feature x E BNXCXH*W (N denotes the batch size, C the number of channels, H the height, and W the width), the vanilla Batch Normalization (BN) works as:
h x (x)
o (x) +
(1)
where u(x) and (x) are the running estimates (or batch statistics) of input z's mean and variance along the (N, H. W) dimension. 7 and B are the learnable parameters of the affine layer, and both are of shape C. However, as the vanilla BN only has a single re-scaling transform, it will simply treat any latent heterogeneous features as a single distribution. As an improved variant, Categorical Conditional BN (CCBN) (Miyato et al., 2018) is proposed to remedy the heterogeneity issue in the task of conditional image generation, boosting the quality of generated images. Categorical Conditional BN has set of independent affine layers, whose activation is conditioned by the input domain index. It can be expressed as:
h = 7i x - (x(x) -((xx) xxx)
Br, I,..,
where 7/ and 33 parameters of the i-th affine layer. Concretely, is the expected output class in the image generation task (Miyato & Koyama| 2018). However, we argue that this "separate/split" modification might be "too loose", ignoring the fact that the distributions of the expected generated images overlap largely (due to similar texture, appearance, illumination, object location, scene lay- out, etc.). Hence to better handle both the latent homogeneity and heterogeneity in x, we present Sandwich Batch Normalization (SaBN), that is equipped with both shared sandwich affine layer and set of independent affine layers. SaBN can be concisely formulated as:
x =(((x) + Bra) + Be, 1,.... C. o(x)
h = (i(Ysal
As depicted in Fig. (d), Yea and Br.. denote the new sandwich affine layer, while 7/ and Br are the i-th affine parameters, conditioned on categorical inputs. Implementation-wisse. SaBN only takes few lines of code changes compared to vanilla BN: please see appendix Fig. 6 pseudo codes.
3.1 UNIFYING НOMOGENEITY & HITTRRGEENEITY FOR CONDITIONAL IMAGE GENERATION
As one of the state-of-ther the-th-tttrttttt GANs, SNGAN (Miyato et al. 2018) suc- cessfully generate high quality im- ages in conditional image generation tasks with Categorical Conditional BN (CCBN). Intuitively, it uses in- dependent affine layers to disentan- gle the image generation of different classes Thus the generative model could be treated as a set of category- specific sub-models in one model. one of which would be "activated" by the conditional class input each time, incurring the model heterogeneity. aging separate affine layers ignores the fact that the expected generated domains are not totally inde- pendent with each other. These classes share the same image resolution, similar illumination, object location, and some intrinsic dataset-specific characteristics, which suggest strong homogeneity.
Table 1: The best Inception Scores ("IS", t) and FIDs (L) achieved by conditional SNGAN, BigGAN, and AutoGAN-topl, using CCBN and SaBN on CIFAR-10 and ImageNet (dogs & cats).
CIFAR-10
ImageNNt (dags cars)
Model
IS
FID
IS
FID
AutoGAN-upp BigGAN SNGIAN
8.91
8.57
8.76
10.18
16.75
78.14
AutoGAN MAIGGAN--pI-SAA SaBN
SaBN
8.72(+0.29)
9.11(-1.40)
BigGAN-SaBi S.GAN-S.BN
9.01(+0.10)
8.03|
-0.5g)
8.80(+0.13)
-1.211
18.31(11.56
10.38(-18.70)
However, only lever-
4

---

Under review as a conference paper at ICLR 2021
This motivates to unify both ho- mogeneity and heterogeneity for condi- tional image generation with SaBN. We choose three representative GAN mod- els, SNGAN, BigGAN (Brock et al. 2018) and AutoGAN-topl (Gong et al.
10
2.25
25
Sdtt aifice
CCBM
200
1.75
1.99
1.25
2019). as our backbones. SNGAN and BigGAN are equipped with Categorical Conditional BN. AutoGAN-topl orig- inally has no normalization layer and was designed for unconditional image generation, we manually insert Categorical Conditional BN into its generator to adapt it to the con- ditional image generation task. We then construct SNGAN- SaBN, BigGAN-SaBN and AutoGAN- SaBN, by replacing all Categorical Conditional BN in the above basclines with our SaBN.
0.50
Nctwokk dusti
Figure 2: The CAPV value of 7 (left) and the shared sandwich parameter 7sa 's value (right) along the network depth.
We test all the above models on CIFAR-IU 10 dataset (Krizhevsky et al. 2009) (10 categories, resolu- tion 32 x 32). Furthermore, task on ImageNet (Deng et al. 2009p. using the subset of all 143 classes belonging to the dog and cat super-classes, cropped reso resolution 128 x 128) following (Miyato al., 2018)'s 's setting. Inception Score (Salimans et al., 2016) (the higher the better) and FID (Heusel et al., (2017) (the lower the bet- ter) are adopted as evaluation metrics. We summarize the best performance the models has achieved during training into Table | We find that SaBN can consistently improve the generative quality of all three baseline GAN models. We also visualize the image generation results of all compared and 24 at appendix.
test SNGAN and SNGAN-SNSS SSBN high resolution image generation
GANs in Fig. 21
Understanding SaBN by Visualization. One might be curious about the effectiveness of SaBN, since at the inference time, the shared sandwich affine layer can be multiplied/merged into the independent affine layers, making the inference form of SaBN completely the same Categorical Conditional BN. Hence to better understand how SaBN benefits the conditional image generation task, dive into the inductive role played by the shared sandwich affine parameter.
Nemainttr Modal:
Opestion
Sunxcndiz
Faol edge
layers-
scpple
Direpeccal balci
We choose SNGAN and SNGAN-SaBN on ImageNet as our testhed. Specifically, we propose a new mea- surement called Class-wise Affine Parameters Vari- ance (CAPV), which aims to indicate how much class- specitic heterogeneity introduced with the of in- dependent affine parameters. For Categorical Con- ditional BN (CCBN), we define its CAPV for y as VyC +ar((F1,52 52. 5c]), where 5/ rep- resents the channel-wise average value of Y. Simi- larly, the CAPV of SaBN is defined as VSaBN(y) (7) = Var(ITSa Vi, 7sa 72, Tsa . Tcl). A larger CAPV value implies more heterogeneity.
Figure 3: We depict two consecutive layers in the super-network. By default, a BN is inte- grated into each operation in vanilla DARTS, except Zero and Skip-connection operation. The output of each layer is the sum of all opera- tion paths' output, weighted by their associated architecture parameter a. Model heterogene- ity is introduced during the summation process, due to the difference among each parallel oper- ations. In the meanwhile, different operations still maintain intrinsic homogeneity and are not completely independent, as they share the same original input, and their gradient is also estimated from the same loss function.
We plot the CAPV values for both SNGAN and SNGAN-SaBN at each layer in Fig. 2 (left). The solid blue line (SaBN) is lower than orange line (CCBN) in the shallow layers, but soon surpasses the latter at the deeper layers. We additionally plot dashed blue line, obtained by removing the shared sandwich affine layer from the SaBN parameters (i.e., only independent affine). Their comparison indicates that the shared sandwich affine layer helps inject an inductive bias: compared to using Categorical Conditional BN, training with SaBN will enforce to shallow layers to preserve more feature homogeneity, while encouraging the deeper layer to introduce more class-specific feature heterogeneity. In other plain words, SaBN seems to have shallower and deeper layers more "fo- cused" on each dedicated role (common versus class-specific feature extractors). We also plot the value of shared sandwich affine parameters Ysa (averaged across channel) along the network depth, the right figure in Fig 2 also aligned with our above observation.
5

---

Under review as a conference paper at ICLR 2021
3.2 ARCHITECTURE HETEROGENEITY NEURAL ARCHITECTURE SEARCH (NAS)
Recent NAS works formulate the search space super-network which contains all candidate operations and architectures, and the goal is to find a sub-network of a single path that of the optimal performance. To support the search over the architectures in the super-network, DARTS (Liu al. 2018) assigns each candidate operation a trainable parameter and the search problem is solved by alternatively optimizing the architecture parameters a and the model weights via stochastic gradient descent. The architecture parameters a can be treated as the magnitude of each operation, which will help rank the best candidates after the search phase.
In Fig. 3 we concretely illustrate the origin of the model heterogeneity in the supernet. To disentangle the mixed model heterogeneity during search process and still maintain the intrinsic homogeneity, we replace the BN in each operation path with SaBN in the second layer (same for the first layer if is also downstream of layers ahead). Ideally, the number of independent affine layers shall be set to the total number of unique ar- chitectures in the search space, en- abling the architecture-wis disen- tanglement. However, would be impractical the search space size (number of unique architecture) is usually larger than 10a. Thus adopt greedy way that only consider to disentangle pre- vious layer's s architecture (operations). Specifically, the number of independent affine layers in the SaBN equals to the total number of candidate operation paths of the connected previous layer. The categorical index i of SaBN during searching is obtained by applying a multinomial sampling on the softmax of previous layers' architecture parameters: softmax([ao, an, a 02,
CSPA 12c
AATTS
Sapti
Figure 4: Results of architecture search on CIFAR-100 and ImageNet16-120 based on DARTS. At the end of each scarching epoch, the architecture derived from current values. The x-axis the searching epoch. The y-axis is the ground truth test accuracy of current epoch's 's architecture, obtained via querying NAS- -AS-Bench- 201. Each experiment is run for three times with different random seeds. Each eurve in the figure is averaged across them.
To validate the efficacy of our SaBN, we conduct ablation study of four settings: 1) no affine (i.e.. 7 = 1, = as in vanilla DARTS), where by default the learning of affine parame- ters of BN in each operation path is disabled (Liu et al., 2018): 2) homogeneity only ("DARTS- affine"). where the learning of affine parameters of BN are enabled in each operation path in "DARTS"; 3) heterogeneity only EDARRS-Categorical path of "DARTS" are replaced by the Categorical Conditional BN (Miyato et al., 2018); : 4) ho- mogeneity and heterogeneity ("DARTS-SaBN"), where all operation paths' BNs in "DARTS" are replaced by SaBN. Following the suggestion by Dong & Yang (2020), experiments use batch statistics instead of keeping rumning estimates of the mean and variance in the normalization layer. We conduct our experiments on CIFAR-
where the BN in each operation
Table 2: The searched results top-1 accuracy of the four meth- ods on NAS-Bench-201 Our proposed approach achieves the highest accuracy, with the lowest standard deviation.
100 (Krizhevsky et al.) 2009) and ImageNet16-120 (Chrabaszez et al. 2017 using NAS-Bench-201 (Dong & Yang 2020) (Fig. 4. Early stopping is applied for the searching phase as sug- gested in (Liagg et al. 2019). We can observe that SaBN dominates on both CIFAR-100 and ImageNet cases. Sur- prisingly, we also notice that by simply turning the affine in the original DARTS, DARTS-affine gains fairly strong improvements. The performance gap between DARTS-affine and DARTS- demonstrate the effectiveness of the independent affine layers in SaBN.
Method
CIFAR-100 100
ImageNet
DARTS
44.05 1 7.47
36.47 +: 7.06
DARTS-afline DARTS-CCEN DARTS- (ours)
63.46 1 2.41
37.26 +: 7.65
62.16 2.62
31.25 + 6.20
71.56 + 1.39
45.85 + 0.72
Experiments also shows that CCBN does help improve search performance. However, it falls largely behind SaBN, indicating the shared sandwich affine layer to also be vital. In Fig. [15 appendix, can observe the shared sandwich affine layer helps to preserve more homogeneity. The ground-truth accuracy of the final searched architecture is summarized in Tab. Besides, also find SaBN

---

Under review as a conference paper at ICLR 2021
works well on another weight-sharing search method, GDAS (Dong & Yang 2020). The results shown in Sec. [A.2.3 at appendix.
3.3 SANDWICH AUXILIARY BATCH NORM IN ADVERSARIAL ROBUSTNESS
AdvProp (iie et al. 2019) successfully utilized adversarial examples to boost network Standard Testing Accuracy (SA) by introducing Auxiliary Batch Norm (AuxBN). The design is quite simple: an additional BN is added in parallel of the original BN, where the original BN (clean branch) takes the clean image as input, while the additional BN (adversarial branch) is fed with only adversarial examples during training. That intuitively disentangles the mixed clean and adversarial distribution (data heterogeneity) into two splits, guaranteeing the normalization statistics and re-scaling are exclusively performed in either domain.
However, one thing missed is that the domains of clean and adversarial images overlap largely, as adversarial images are generated by perturbing clean counterparts minimally. This inspires to present a novel SaAuxBN, by leveraging Norminn-specitn normalizations and affine layers, and also shared sandwich affine layer for homogeneity preserving, SaAuxBN can be defined as:
X 14:(x) o3 (x)
h = K(7sal
Bra) + B 0,
(4)
(x(x)) and (x) denote the i-th (moving) mean and variance of input, where i = 0 for adversarial images and i 1 for clean images. We use independent normalization layer decouple the data from two different distributions, i... the clean and adversarial.
For fair comparison, we follow the settings in Madry et al. (0117). In the adversarial training, we adopt foo based 10 steps Projected Gradient Descent (PGD) (Madry et al.) (017)) with step size
255 and maximum perturbation magnitude same configuration is adopted.
8 As for assessing RA, PGD-20 with the 255
We replace AuxBN with SaAuxBN in AdvProp (Xie et al. 2019) and find it can further improve SA of the network with its clean branch. The experiments are conducted on CIFAR-10 (Krizhevsky
al. 2009) with ResNet-18 (He al. 2016) backbone, and the results are presented in Tab.
Table 3: Performance (SA) of different BN settings on clean branch.
Evaluation
BN
AuxBN (clean branch)
ModeNorm
SaAuxBN (clean branch)
Clean (SA)
87.25
94.38
86.10
94,47
We further conduct experiment to test the SA and Robust Testing Accuracy (RA) of the network using the adversarial branch of AuxBN and SaAuxBN. The comparison results are presented in Tab. 4
Table 4: Performance (SA&RA) of different BN settings. During evaluation, only the adversarial path is activated in AuxBN and SaAuxBN.
Evaluation
BN
AuxBN (ady branch)
ModeNorm
SaAuxBN (ady branch)
Clean (SA)
87.25
85.08
86. 10
87.07
PGD-10 (RA)
42.82
43.30
44.60
45.72
PGD-20 (RA)
40.84
41.98
42.81
44,23
Tab. 4 shows that BN still achieves the highest performance on SA, but falls lot on RA compared with other methods. Our proposed SaAuxBN is on par with the vanilla BN in terms of SA, while has significantly better results on RA than any other approaches. Compared with SaAuxBN, AuxBN suffers from worse SA and RA, indicating that both the shared sandwich affine layer are key to the disentanglement of adversarial domain from the clean domain, with their shared statistics properly preserved. The behaviors of AuxBN and SaAuxBN suggests that the sandwich affine layer here mainly encourages to enhance feature homogeneity.
visualized in Fig. 20 at appendix, which
We additionally include ModeNorm (Deecke et al. 2018) an ablation in our experiments, which was proposed to deal with multi-modal distributions inputs, i.e., data heterogeneity. It shares some similarity with AuxBN both consider multiple independent norms. ModeNorm achieves fair

---

Under review as a conference paper at ICLR 2021
performance on both SA and RA, while still lower than SaAuxBN. The reason might be the output of ModeNorm is a summation of two features weighted by a set of learned gating functions, which still mixes the statistics from two domains, leading to inferior performance in the attack scenario.
3.4 STYLE TRANSFER WITH SANDWICH ADAPTIVE INSTANCE NORMALIZATION
Huang& & Belongie (Huang& & Belongie 2017) achieves arbitrary style transfer by introducing Adap- tive Instance Norm (AdalN). The AdaIN framework is composed of three parts: Encoder, AdaIN and Decoder. Firstly, the Encoder will extract content feature and style feature from content and style images. Then the AdaIN is leveraged to perform style transfer on feature space, producing stylized content feature. The Decoder is learnt to decode the stylized content feature to stylized images. This framework is trained end-to-end with two loss terms, a content loss and a style loss.
Concretely, AdaIN firstly performs normalization on the content feature, then re-scale the normal- ized content feature with style feature's statistic. It can be formulated as:
h aly) /xxx (x)
+ ply),
(5)
where y is the style input, x is the content input. Note that and here are quite different from BN, which are performed along the spatial axes (H, W) for each sample and each channel. Obviously, style-dependent re- scale may be too loose and might further amplify the intrinsic data heterogeneity brought by the variety of the input content images, undermining the network's ' ability of maintaining the content information in the outpur. In order to reduce the data heterogeneity, we propose to insert a shared sandwich affine layer after the normalization, which introduce homogeneity for the style-dependent re-scaling transformation. Hereby, we present SaAdaIN:
h (ly)(Taa(( x mxx)
+ Baaa) + p(y),
(6)
(x)
Besides AdaIN, we also in- clude Instance-Level Meta Nor-
Certent less
Syll Incs
malization with Instance Norm (ILM+IN) proposed by Jia al.) (211)) as a task-specific com- parison baseline. Its style- independent affine is not only conditioned on style informa- tion, but also controlled by the input feature. Our training set- tings for all models are kept iden- tical with (Huang & Belongie| 2017]. We depict the loss curves of the training process in Fig. 5 We can notice that both the content loss and style loss of the proposed SaAdaIN is lower than that of AdaIN and ILM+IN. This observation demonstrates that the shared sandwich affine layer in our SaAdaIN is beneficial for the network to preserve the semantic information of the original input content, while also better migrating and merging style information.
1.0
Figure S: The content loss and the style loss of using AdaIN, ILM+IN and SaAdaIN. The noisy shallow-color curves are the original data. The foreground smoothed curves obtained via applying exponential moving average on the original data.
Furthermore, the qualitative visual results are demonstrated in Fig. 26 appendix (Better zoomed in and viewed in color). The leftmost column displays content images and referenced style images. The next three columns are the stylized outputs using AdaIN, ILM+IN and SaAdaIN, respectively.
CONCLUSION
We present SaBN and its variants as plug-and-play normalization modules, which are motivated by addressing model & data heterogeneity issues. We demonstrate their effectiveness on several tasks, including neural architecture search, adversarial robustness, conditional image generation and arbitrary style transfer. Our future work plans to investigate the performance of SaBN on more applications, such as semi-super learning (Zajac et al. 2019), slimmable models, and once-for- all training (Yu et al. 2018).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.0720 07875, 2017.
Wasserstein gan.
arXiv preprint
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. arXiv preprint arXiv:/812.03981, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 11x7x:1707.08819 2017.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Faimas: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint 1oXiv:1977.01880 2019.
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. The- oretical understanding of batch-normalizatioo:: A markov chain perspective. arXiv preprint arXiv:2003.01652, 2020.
Harn De Vries, Florian Strub, Jérémic Mary, Hugo Larochelle, Olivier Pictquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Lucas Deecke, lain Murray, and Hakan Bilen. 11X.v:1810.05566, 2018.
Mode normalization.
arXiv preprint
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial de- fense. arXiv preprint arXiv: 2018.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture four gpu hours. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1761-1770, 2019.
Xuanyi Dong and Yi Yang. Nas-bench- 102: Extending the scope reproducible neural architecture search. arXiv preprint 1.Xiv:2001.00320 2020.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.0762,, 2016.
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in enns. arXiv preprint TrXXi:2003.00/55 2020.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 3224-3234, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 141iv: 2014b.

---

Under review as a conference paper at ICLR 2021
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp. 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in narch-normalizze models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03160 2015.
Songhao Jia, Ding-Jie Chen, and Hwann-Tzong Chen. Instance-level meta normalization. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4865-4873, 2019.
Alexia Jolicceur-Martineau. gan. arXiv preprint 1807:0807700734 2018.
The relativistic discriminator: key element missing from standard
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Ming Zhou, Klaus Neymeyr, and Thomas Hof- mann. Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization. arXiv preprint 17X.v.1805.10644, 2018.
Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. Advances in Neural Information Processing Systems, 33, 2020.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Boyi Li, Felix Wu, Kilian Q Weinberger, and Serge Belongic. Positional normalization. In Advances in Neural Information Processing Systems, pp. 1622-1634, 2019a.
Xilai Li, Wei Sun, and Tianfu Wu. Attentive normalization. arXiv preprint wrxhv:1908.01259 2019b.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In Proceedings of the IEEE International Conference Computer Vision, pp. 5764- 2017.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. arXiv preprint 160hv:1603.0477, 2016.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with carly stopping. arXiv
preprint arxiv: / 000300 2019.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using level representation guided denoiser. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
10

---

Under review as a conference paper at ICLR 2021
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09000 09055, 2018.
Aleksander Madry,
Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706: (70.o0030000.833 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 145, 2017.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv:1802.056637, 2018.
arXiv preprint
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv:1802.05955 2018.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection adversarial exam- ples. In Advances in Neural Information Processing Systems, pp- 4579-4589, 2018.
Nicolas Papernot and Patrick McDaniel. Extending defensive distillation. arXiv preprint
arXiv:1705.05264, 2017.
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2337-2346 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint Stti::8802.33200 2018.
Siyuan Qiao, Huiyu Wang. Chenxi Liu, Wei Shen, and Alan Yuille. Rethinking normalization and elimination singularity in neural networks. arXiv preprint arXiv:191..09733 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018.
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch de- pendence in the training of deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11237-11246, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 2016.
Yi Wang. Ying-Cong Chen, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Attentive normalization for conditional image generation. In Proceedings of the IEEE/VV Conference Computer Vision and Pattern Recognition, pp. 5094-5103, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3-19, 2018.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training. arXiv preprint arXiv:1906.0377 037877 2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. arXiv preprint urXiv:191.09666 2019.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint mxXiv/174..00 155, 2017.

---

Under review as a conference paper at ICLR 2021
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101; 101: Towards reproducible neural architecture search. arXiv preprint arXiv:1902.09335 2019.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. 2018.
arXiv preprint
Michal Zajac, Konrad Zoina, and Stanistaw Jastrzebski. Split batch normalization: Improving semi- supervised learning under domain shift. arXiv preprint arXiv:1904.0333 2019.
Arber Zela, Julien Siems, and Frank Hutter. Nas-bench-Ishott. Benchmarking and dissecting one- shot neural architecture search. arXiv preprint arX1v22001.10422, 2020.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.083112 2018.
Jie Zhang, Dongdong Chen, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Passport-awaid normalization for deep model protection. Advances Neural Information Processing Systems, 33, 2020.
Heliang Zheng, Jianlong Fu, Yanhong Zeng, Jiebo Luo, and Zun Zha. Learning semantic- aware normalization for generative adversarial networks. Advances in Neural Information Pro- cessing Systems, 33, 2020.
Pan Zhou, Caiming Xiong, Richard Socher, and Steven Chu Hong Hoi. Theory-inspired path- regularized differential network architecture search. Advances in Neural Information Processing Systems, 33, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:161..0157,, 2016.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
def BatchNorm (x, gamma, beta, running_mean, mean, sunning_var, eps=le-5, momentum=0.11
# x: input features with shape H,
gamma, beta: scale factors with shape [1,C,1,1]
mean, var tf s.f.nn.moment (x, [0, 2, 31, keep dims True)
running_meet mean (1 momentum) * running_men mean + momentun mean conning_var
momentum) running var + momentum var x = (x running_mean) tf.sqrt (running_var + eps)
return x ganma + beta
def SaBatchNorm sa_gamma, sa beta, gammas, betas, index, Lomning_mean, running_var, eps=ie-5, momentum=0.1).
# x: input features with shape H,
# sa_gamma, beta: shared scale factor with shape [1,C,1,1] # ganmas, betas: a list of scale factors with shape [1,c,1,1] mean, var tf. nn moments (x, [0, 2, 31, keep_dims running mean = (1 um)entum) * running_mes meag + momentun mean running_va var
True)
monentum) zunning_van var + momentum var
x = (x cunning_meant tf.sqrt (running_var + eps) x = x sa_ganma + sa beta
return x ganmas [index]
betas [index]
def chNor (x, sa_ _a_gamm,, sa sa_beta,
gammas, betas, index, running ganning_means, running_vave vars, eps=le-5, momentum=0.1); # x: input features with shape [N, C, H]
sa_ganma, sa_beta: shared scale factors with shape [1, I gammas, betas: a list of scale factors with shape [1,0,1, mean, var tf.nn.fmoentee moments (x, [0, 2, 3], keep dims True) running_meaat means [index] (1 um)
junning_meand [index]
momentum mean running_vary vars [index] = (1 momentum) running_vars varr [index]
momentum * var
x (x runninggmest means [index]) / tf. sqrt (running_van _arr [index] eps)
x = x sa sa_ganma + sa_ sa_beta
return gammas [index] betas [index]
Figure 6: Pseudo Python code of BN, SaBN and SaAuxBN with TensorFlow. We highlight the main difference between approaches with vanilla BN.
13

---

Under review as a conference paper at ICLR 2021
A.2 ADDITIONAL RESULTS IN NAS
A.2.1 SEARCH SPACE OF NAS-BENCII-21
Summation
Layer
Concatenation
Fixed edge
feature,
cellm
cell
m-1
Edge
Fiigr
cellm
17
featurez
E
rdge 1
L1
cell m+1
features
output
Figure The search space of NAS Bench-201.
Our experiments of NAS are conducted on NAS-Bench-200 which adopts cell-based search space, shown in Fig. 7 The whole search space is composed by stacking several cells. Each cell consists of several layers Lÿ, which is composed of several parallel operation paths as illustrated in Fig.
in our paper. L denotes the k-th layer which take feature, input. In Sec. 3.1 of our paper, have concretely illustrate the detailed implementation of the SaBN under the circumstance layer only has one connected previous layer. However, connected previous layers. Take L3 as an example, it is connected with L3 and L1. In this case, the number of independent affine parameter in SaBN at L3 would be n², where is the number of parallel operation paths in each layer.
can observe there is layer having multiple
A.2.2 THE ADDITIONAL RESULTS SABN IN DARTS
Visualization of architecture parameters in DARTS
We visualize the searching curves the architecture parameters on each edge of the cell. in Fig
10 for DARTS, DARTS BARTS-CCBN and DARTS SaBN respectively. In the case of DARTS (Fig. 8
can see that "skip_ connect" dominates on most of edges during the search, yielding bad architectures when search ends. DARTS-CCBN is in favor of both Skii_conneer"" and "none" operations at last, architecture parameters climbs at first, but drops immediately after a few epochs, whereas the "nor_conv_3a" _33"" takes the lead in most of edges.
in Fig. / For our DARTS-SaBN in Fig. TO the weight of "skip_connec""
Visualization of discovered architectures in DARTS
We visualize the discovered architectures of DARTS, DARTS- CCBN and DARTS- in Fig. It shows the architecture that discovered by DARTS is fully dominated by further analyze the operator composition of searched architectures of DARTS and DARTS-
11
SaBN in Fig. [2 based on their final searched architecture parameters. We can clearly see that "skip_connee" is highly preferred by DARTS, where DARTS-SaBN favors "nor_conv_3x3"
14

---

Under review as a conference paper at ICLR 2021
Edge
Edge
0.24
0.25
022
018
2do conmact
200
0.4
178 baarching laoch
Start kaodh
0.2
0225
0248
0.75
0131
0.55
suachingg
Figure 8: Architecture parameters (after softmax) on each edge in DARTS.
Edipe B
Edge 1
Edge
0.00
0.2
0.13
022
6.27
2ww. 160
3cl 300
FOC.BRL Jr1
032
2.2
0:5
Starching
Figure 9: Architecture parameters (after softmax) each edge in -CBTS-
Edge-
Edge
6:3
0200
care
0100
15
15
20
18
20
Edpe
Edge
Edpe
2.8
0.30
0.25
300
3.20
320
8.0
Socet Farch
Figure Architecture parameters (after softmax) on each edge in our MARTS-SaBN
15

---

Under review as a conference paper at ICLR 2021
Bok: (npui
(mpaa
to52_0 (tgpa))
Aki__contect
node
BoC cony
node
skip copnect
3nd2
skip_snniied
stAl__
107,0000
Aip_ommcee
BOC_COB
skip comEEct
axppconeeet
DOE BOK_EWAY_AA
comnec
nor
TOY cone 363
sude (eutpuc)
mook
Bolk (CArq]a))
(оutрu))
scarchod archiltcture
(c) DARTS-CCBBE scarched architecture
DARTS-SaHN's architocture
Figure 11: The architectures searched by DARTS are dominated by "skki_cconec"" and the architecture of DARTS-CCBN is full of both "skip_connect" connect" and "none". contrast, DARTS-SaBN highly prefers "nor conv ox3".
Ratios
0.0995
DARTS DARTS-SSBN
avg_pool_ _ppo_oxx3 3x3
0.1369
0.2843
nor_conv_3a
0.1979
0.2081
nor_conv _x1
0.1609
0.2457
skip_connect
0.328B
0.1625
0.1754
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Figure 12: The operation statistics of the searched architecture from DARTS and DARTS-SaBN,
Analysis of Optimization and Generalization Ability
We found SaBN benefits NAS w.r.t. both the generalization ability and the optimization. In DARTS, the goal is to learn the architecture parameter a and the model weights jointly. Specifically, a is optimized to minimize the architecture loss (val a) using validation dataset, where the weights w is optimized minimize the weight loss Lirain (w, a:) with training dataset.
our experiment, we first visualize the architecture loss as well as weight loss in Fig. 13 Compared with DARTS, the loss value of MARTS-CCBI is larger in the first several epochs, but becomes similar in the later. This indicates that the independent affine layers slow the optimization at the beginning, due the additional parameters. Compared DARTS-SaBN with DARTS-CCBN, can observe the losses of DARTS-SaBN lower than BARTS-CCBN at early stage, indicating the additional sandwich affine layer is beneficial the optimization of the model. This is achieved by the injected inductive bias of commodity for features from different operations, leading to easier optimization. However, the losses of both DARTS and DARTS-CCBN become lower than DARTS- SaBN in the later stage. This is caused by the architecture collapse in the later searching stage, which can be observed in Fig. S where the supernet starts to be dominated by skip connections in DARTS and DARTS-CCBN, leading to easier optimization Zhou et al. (2020).
We further visualize the gap between the (ws, a)) and Turan (w, a) in Fig. 14 with the setting of DARTS, DARTS-CCHS and DARTS-SaBN. The gap in DARTS-SaBN is lower than that of DARTS and DARTS-CCBN indicating SaBN can also help to improve the generalization ability The
of the supernet. The inserted shared sandwich affine layer also serves as regularization term here, since it is updated by data from all previous operations, thus improving the model generalization ability.
CAPV of SaBN in DARTS
16

---

Under review as a conference paper at ICLR 2021
Weight losss
Architecture
DARTS-SaON DARTS DARTS-CCON
4.0
DARTS-SSaON
DARTS
DARTS-CCON
Loss value
2.5
2.0
2.0
1.5
1.0
1.5
10
Epoch
Epoch
Figure 13: The architecture loss Lval (w, a.) and the weight loss Tunin (w, o.) of DARTS, DARTS-CCBN and DARTS-SaBN.
Weight architecture loss gap
0.5
DARTS-SaBN
DARTS
DARTS-CCEN
0.2
10
15
20
30
35
Figure 14: The architecture--wiihh loss gap of DARTS, DARTS-CCES and DARTS-SaBN.
We use CAPV as the measurement to analyze the behavior of SaBN in DARTS, which is introduced in Sec. 3.1. We compare our proposed DARTS-SaBN with DATRS-CCBN, w.r.t their CAPV at each layer, in Fig 15
1e-5
CAPV of
25
ScBN
ScBN w/o Sa affine
(CUNN
2.0
1.0
12
Netwoek depth
Figure 15:
The CAPV for DARTS-CCBL and DARTS-SaBN, The shared sandwich affine layer is learned to reduce the feature heterogeneity across the whole network.
A.2.3 THE RESULTS OF GDAS
GDAS (Dong Yang 2019) an advantaged extension of DARTS (Liu al.] 2018) where the forward activation of all paths during search is replaced by the activation of sampled single path. This achieved by introducing Gumbel-softmax to construct a differentiable one-hot vec- tor [a af oz,...0,_1l 1 {0, 1), from weight vector (On, 01 02,...0n-1}
17

---

Under review as a conference paper at ICLR 2021
Table 5: Comparison of four variants of GDAS NAS-
We conducted four experiments, the original GDAS, GDAS-affine GDAS- CategoricalCBN and GDAS-SaBN, which are quite similar to DARTS experi- ments settings. The only difference is that the category index in CCBN and SaBN obtained by using Gumbel-softmax instead of multinomial sampling. For
Bench-201. Our GDAS-SaBN achieves the highest top-1 ac- curacy.
Method
CIFAR-IOP 100
ImageNet
GDAS
67.52 0.06
39.16 0.32
GDAS-afiine UDAS-CCBN
64.79 5.22
42.57 + 1.05
57.62 = 12.22
32.52 + 11.48
GDAS-SaBl (ours)
68.67 1.02
43.47 + 2.01
all experiments we use batch statisties instead of running mean and variance(Dong & Yang 2020
The experiments results are shown in Fig. T6 On CIFAR-100, only GDAS-SaBN outperforms the original GDAS. On ImageNer16-1000 Both GDAS-SaBN and GDAS-affine improve the search takes the lead finally. The comparative results of GDAS-SaBi and GDAS-CCBN indicate the importance of the shared sandwich affine layer. The ground-truth accu- racy of the final searched architecture summarized in Tab.
results, while GDAS
CIFAR-100
ImageNet16-120
60
Greund Trath Test Aecurccy
Sroune Truit Test accurscy
-55
26
30
GDAS
GDAS affine
GDAS affinc
CategricaiCUN
GDAS SaRN
10
GDAS-SaRN
100
150
200
250
10
20
30
40
50
60
70
90
The Searching Epoch
The Searching Epoch
Figure 16: Results of architecture search on CIFAR-100 100 and ImageNet16-122 1200 based on GDAS. Each curve is the average result of three runs using different random seeds.
A.3 ADDITIONAL. RESULTS OF ADVERSARIAL. ROBUSTNESS
A.3.1 ANALYSIS OF OPTIMIZATION AND GENERALIZATION ABILITY
Our experiments also indicate that SaAuxBN benefits the model on both optimization and general- ization. We mainly focus on two losses in our experiments, clean branch loss L [[fetem (Zetean) ). y) and adversarial branch loss ((faav(Fadv) ). y), where denotes the model and 2. y denotes the input data, label respectively. We visualize the clean branch loss L(felean (Ietea)) y) and adversarial branch loss ([aa((aaa)) y) of SaAuxBN and AuxBN two figures show that SaAuxBN leads to lower training and testing loss, demonstrating SaAuxBN is beneficial to the model optimization. This is achieved by the injected inductive bias of commodity for features from different domain (clean and adversarial), leading to easier optimization.
training set and test set in Fig. 17
18 The above
Besides, we also show the train-test loss gap of both losses in Fig. 19 For adversarial branch loss L(favv (Zadv), ), y) in the left side, we can observe SaAuxBN has significantly smaller loss gap com- pared with AuxBN. The train-test loss gaps of clean branch loss [[fetean(Tettan) y) for SaAuxBN and AuxBN in the right are quite similar. These evidence prove that SaAuxBN also benefits the model in generalization ability. The inserted shared sandwich affine layer also serves as regular- ization term, since it is updated by data from both adversarial domain as well as clean domain, thus improving the generalization ability.
A.3.2 CAPV SABN SABN IN ADVERSARIAL RORUSTNESS
We visualize the CAPV of model with AuxBN and SaAuxBN which trained under AdvProp framework in Fig. 20
18

---

Under review as a conference paper at ICLR 2021
Adversarial branch loss training set
Clean branch loss training set
SaAudEN
AuxBN
2.0
Lost volu 1og scale)
100
1.9
1.0
1.7
1.6
1.5
1.4
SaAuxBN
AuxeRN
1.3
25
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epoch
Epoch
Figure 17: The adversarial branch loss ((ffax(Taak) ), y) and clean branch loss T(fetem (Jeetam))))))))))))))) y) on training
J, y denote model, input and label respectively. The model with SaAuxBN has lower training loss.
Adversarial branch on test set
Clean branch loss on test set
Whrry
2-2
AuxBN
AuxBN
LosE value (og scale)
100
Lo85 value
Epoch
Epoch
Figure 18: The adversarial branch loss L(favv y) and clean branch loss [(fetem (Nekass, on testing The model with SaAuxBN has lower test loss.
Train-test gap
Train-test gap for branch
SaAuNEN
SaAuxBB
AuxEN
AuxeN
0.15
0.30
volue
Cap volue
0.10
0.05
M
0.00
-0.05
25
50
75
100
125
150
175
200
25
100
125
150
175
200
Figure 19: The train-test loss gap for adversarial branch loss ((fat((asa)) ). w) and clean branch loss [[fetem (Zetean) ). V).
19

---

Under review as a conference paper at ICLR 2021
10-3
CAPV of Y
StAuxBN
1.4
SaAuxBN w/ Sa affine
AuxBN
1.0
CAPV
0.8
0.6
0.4
0.2
0.0
Network depth
Figure 20: The CAPV value for models with AuxBN and SaAuxBN. We can observe the shared sandwich affine layer is learned to reduce CAPV, i.e., the feature heterogeneity.
20

---

Under review as a conference paper at ICLR 2021
A.4 ADDITIONAL RESULTS OF IMAGE GENERATION
A.4.1 VISUAL RESULTS
SNGAN Vs SNGAN-SaBN
(a) SNGAN's results
(b) SNGAN-SaBN results
Figure 21: The image generation results of SNGAN and SNGAN-SaBB -aBNN on ImageNet. Each column is corre- sponding to specific image class.
(a) SNGAN's results
(b) SNGAN-SaBN's results
Figure image generation results SNGAN and SNGAN-SaBN on CIFAR-10.
21

---

Under review as a conference paper at ICLR 2021
AutoGAN Vs AutoGAN-SaBN
(u) AutoGAN's results
(b) AutoGAN-SSa -aBBN's results
Figure The image generation results AutoGAN AutoGAN-SaBN on CIFAR-10.
BigGAN Vs BigGAN-SaBN
BigGAN's results
(b) BigGAN-SaBN's results
Figure 24: The image generation results of BigGAN and BigGAN-SaBN CIFAR-10.
A.4.2 ANALYSIS OF OPTIMIZATION
We found that SaBN makes optimization casier. We've ve visualized the generator training loss of both SNGAN (originally with CCBN) and SNGAN-SaBN on ImageNet dataset in Fig. [5 It can be clearly observed that SNGAN-SaBN has lower generator training loss, yielding better optimization.
A.S VISUAL RESULTS OF NEURAL STYLE TRANSFER
22

---

Under review as a conference paper at ICLR 2021
SIGGAN
0
50
100
150
200
Epoch
Figure The generator loss SNGAN SNGAN-SaBN ImageNet. SNGAN-SaBN achieves lower
loss value.
Content
AdaIN
ILM+IN
SaAdaIN (ours)
Figure 26: The visual results of style transfer. An ideally stylized output should be semantically similar to the content image, while naturally incorporate the style information from the referenced style image.
23


Reviews:
[{"review_id": "k6jUJz4rT3Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers improving the performance of various normalizers by factorizing the affinity operations in normalization layer into on shared affinity operation, as well as several several independent affinity operation, each of which is corresponding to a specific data distribution. The experiments on various tasks (e.g. NAS, GAN, adversarial defense) demonstrate the effectiveness of proposed methods.\n\nOverall, the idea is simple yet effective. It is a good practical way to improve the performance of BN and IN.\n\nMy questions are as follows:\n\n(1)The section 3.2 is still unclear to me. The SaBN output multiple tensors, each of which is associated with the candidate operation in the previous layer. How to merge these tensors as the input for the next layer? Just do the summation? In A.2.3, why the number of independent affine parameter becomes n^2 in the case of multiple connected.\n\n(2)What is the difference between the searched architectures by using BN, CCBN and SaBN? Could the authors give some analysis of the impact on the searched architecture when using SaBN?\n\n(3)In Fig.11, the authors show some generated results on ImageNet. I think more results generated from the original methods should be given for the comparison.\n\n(4)I want to know the limitations or some failure results of the proposed method, so that we can have a more comprehensive understanding of the proposed normalizer.", "extra_scores": {}, "extra_text": {}}, {"review_id": "C_iybo7L4h", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose Sandwich Affine strategy to separate the affine layer in BN into one shared sandwich affine layer, cascaded by several parallel independent affine layers. Such method should well address the inherent feature distribution heterogeneity in many tasks. Following this idea, the SaAuxBN and SaIN have also been introduced in this paper. The extensive experiments demonstrate the effectiveness of such methods in neural architecture search (NAS), image generation, adversarial training, and style transfer.\n\n\nPros: \n+ Neat motivation; \n+ Simple methods with clear contribution;\n+ Extensive experiments;\n\nCons:\n- The implementation details in Sec3.2 in not clear. (1) “Specifically, the number of independent affine layers in the SaBN equals to the total number of candidate operation paths of the connected previous layer.” Are you sure it doesn’t equals to the total number of candidate operations in next layer? (2) I think it will be better to add the SaBN in Fig3, and point out the correspondence between the candidate operations and the conditional affine layer. (3)  “The categorical index i of SaBN during searching is obtained by applying a multinomial sampling”. Could you please provide a more detailed explanation ?\n\n- According to Fig.4, SaBN makes the searching process more stable, could you please provide some explanation about this phenomenon? What is the difference of the architecture parameters (i.e. alpha) learned by BN and SaBN? \n- Does the author use SaBN to search the architecture, and use BN to fine-tune the searched model? Some implementation details are missing.\n\n- Do the results in Table2 denote the Top-1 accuracy? If yes, please specify in the caption.\n\n- A lot of reference about the normalization methods published in recent years are still missing.\n\n\nOther Comment:\nIf we change the order of the Sandwich Affine layer and Conditioned Affine layer, how about the performance in such case? Could you give some analysis about the different about your methods and the above case?", "extra_scores": {}, "extra_text": {}}, {"review_id": "4pBc6e8_UNd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a modification to conditional batch normalization, wherein an extra affine layer is introduced between the standardization and the conditional affine layer. The empirical results show the benefits of introducing this extra \"sandwich\" affine layer.\n\nThe intuition behind the approach makes sense, but the formulation makes it difficult to see why SaBN is in fact beneficial compared to CCBN. It does not appear that the approach imposes any restrictions or regularization on the CCBN affine parameters; therefore the only difference between CCBN and SaBN seems to be a different parameterization. I would then expect both CCBN and SaBN to reach the same optimal training loss. It may be that the reparameterization provided by SaBN yields the optimization trajectories that lead to better-generalizing solutions, but it is not explained in the paper whether, or why, this happens. One way to probe this might be to reparameterize each gamma in BN or CCBN as $\\gamma=\\gamma_1 \\gamma_2$ (or even $\\gamma = \\gamma_1^2$) and study the behavior of the resulting model.\n\nThe paper proposes to measure the heterogeneity in the found representations (between different branches of CCBN) via the CAPV measure. In its definition, I could not find what the overbars signify but I assume it means the average over the channels. Also, the indexing of gammas from 0 to N should probably be from 1 to C, for consistency with Eq. (2). The definition of CAPV as the variance of gammas seems problematic, however, in ReLU models with batch normalization: models whose gammas differ by a constant factor represent the same function, so the variance of gammas can be arbitrarily changed without affecting the model. A more useful measure of heterogeneity would need to take this scale invariance into account.\n\nThe paper shows several empirical studies, including one for architecture search -- with the main paper using DARTS, and the appendix using GDAS. This choice seems suboptimal, given that in the NAS-Bench-201 paper (https://openreview.net/pdf?id=HJxyZkBKDr), Table 5 seems to indicate that DARTS performs much worse than GDAS. In this work, the appendix devoted to GDAS seems to indicate that (1) there is no consistency on using or not using affine, between CIFAR-100 and Imagenet, and (2) GDAS-SaBN is not statistically significantly better than the better of GDAS and GDAS-affine.\n\nThe results in this paper are encouraging, but I believe the paper needs to explain more clearly why SaBN is expected to work (given that it preserves the hypothesis class as well as the minimum of the training objective). Additionally, since it appears to amount to a per-dimension reparameterization, the reader might expect that some of the other reparameterizations could have a similar effect (including such simple interventions as changing the learning rates for some of the gamma or beta parameters), and compellingly demonstrating that the specific reparameterization given by SaBN outperforms such alternatives would make the paper stronger.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2e6Mleg6t7r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new variant of batch normalization. Here are my comments.\n\n1. paper organization. \n\nThere are only four sections, and it seems that Section 3 is the main section, which contains both contributions and experiments. I would like the authors to clarify that the contribution is only in the formulation paragraph. It seems to me that the paper organization is poor. There are little descriptions, explanations, and discussions about the proposed method (Eqn 3)\n\n2. the contribution.\n\nIf I understand correctly, the only difference between previous work (Eqn 2) and the proposed method (Eqn 3) is the additional parameter gamma-sa/beta-sa. At first glance, the two parameters can be absorbed into gamma_i and beta_i in Eqn 2. (I am saying eqn 2 and eqn 3 are equivalent by some reparameterization.\n\n3. experiments\n\nIt seems that the experimental results in table 2 are rather poor. Could you give more explanation on this?\n\nOverall, I don't think the current format, evaluations, and paper significance fit the conference.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a novel reparameterization of batch normalization that is hypothesized to give a better inductive bias for learning several tasks, including neural architecture search, conditional image generation, adversarial robustness and neural style transfer. The reviewers indicate that this is useful and is of interest to the ICLR audience, but they are not satisfied with the analysis offered in the paper. Specifically, the reviewers request that the authors provide a more detailed analysis of why the proposed reparameterization improves results, given that it does not change the expressive power of the model class. Additionally, the reviewers have some concerns about the structure of the paper. I therefore recommend rejecting the paper at this time.
Decision: Reject