Title: DIFFERENTIABLE LEARNING-TO-NORMALIZE VIA SWITCHABLE NORMALIZATION

Abstract: We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. S-N switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig. 1 ). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been released in https://github.com/switchablenorms/. * The first two authors contribute equally.

1. INTRODUCTION:

Normalization techniques are effective components in deep learning, advancing many research fields such as natural language processing, computer vision, and machine learning. In recent years, many normalization methods such as Batch Normalization (BN) (Ioffe & Szegedy, 2015) , Instance Normalization (IN) (Ulyanov et al., 2016) , and Layer Normalization (LN) (Ba et al., 2016) have been developed. Despite their great successes, existing practices often employed the same normalizer in all normalization layers of an entire network, rendering suboptimal performance. Also, different normalizers are used to solve different tasks, making model design cumbersome.

To address the above issues, we propose Switchable Normalization (SN), which combines three types of statistics estimated channel-wise, layer-wise, and minibatch-wise by using IN, LN, and BN respectively. SN switches among them by learning their importance weights. By design, SN is adaptable to various deep networks and tasks. For example, the ratios of IN, LN, and BN in SN are compared in multiple tasks as shown in Fig. 1 (a) . We see that using one normalization method uniformly is not optimal for these tasks. For instance, image classification and object detection prefer the combination of three normalizers. In particular, SN chooses BN more than IN and LN in image classification and the backbone network of object detection, while LN has larger weights in the box and mask heads. For artistic image style transfer (Johnson et al., 2016) , SN selects IN. For neural architecture search, SN is applied to LSTM where LN is preferable than group normalization (GN) (Wu & He, 2018) , which is a variant of IN by dividing channels into groups.

The selectivity of normalizers makes SN robust to minibatch size. As shown in Fig. 1 (b) , when training ResNet50 (He et al., 2016) on ImageNet (Deng et al., 2009) with different batch sizes, SN is close to the "ideal case" more than BN and GN. For (8, 32) as an example 1 , ResNet50 trained with SN is able to achieve 76.9% top-1 accuracy, surpassing BN and GN by 0.5% and 1.0% respectively. In general, SN obtains better or comparable results than both BN and GN in all batch settings. Figure 1 : (a) shows that SN adapts to various networks and tasks by learning importance ratios to select normalizers. In (a), a ratio is between 0 and 1 and all ratios of each task sum to 1. (b) shows the top-1 accuracies of ResNet50 trained with SN on ImageNet and compared with BN and GN in different batch settings. The gradients in training are averaged over all GPUs and the statistics of normalizers are estimated in each GPU.

For instance, all methods are compared to an ideal case, 'ideal BN', whose accuracies are 76.4% for all settings. This ideal case cannot be obtained in practice. In fact, when the minibatch size decreases, BN's accuracies drop significantly, while SN and GN both maintain reasonably good performance. SN surpasses or is comparable to both BN and GN in all settings.

Overall, this work has three key contributions.

(1) We introduce Switchable Normalization (S-N), which is applicable in both CNNs and RNNs/LSTMs, and improves the other normalization techniques on many challenging benchmarks and tasks including image recognition in ImageNet (Russakovsky et al., 2015) , object detection in COCO (Lin et al., 2014) , scene parsing in Cityscapes (Cordts et al., 2016) and ADE20K (Zhou et al., 2017) , artistic image stylization (Johnson et al., 2016) , neural architecture search (Pham et al., 2018) , and video recognition in Kinetics (Kay et al., 2017) . ( 2) The analyses of SN are presented where multiple normalizers can be compared and understood with geometric interpretation. (3) By enabling each normalization layer in a deep network to have its own operation, SN helps ease the usage of normalizers, pushes the frontier of normalization in deep learning, as well as opens up new research direction. We believe that all existing models could be reexamined with this new perspective. We'll make the code of SN available and recommend it as an alternative of existing handcrafted approaches.

In the following sections, we first present SN in Sec.2 and then discuss its relationships with previous work in Sec.3. SN is evaluated extensively in Sec.4.

2. SWITCHABLE NORMALIZATION (SN):

We describe a general formulation of a normalization layer and then present SN. A General Form. We take CNN as an illustrative example. Let h be the input data of an arbitrary normalization layer represented by a 4D tensor (N, C, H, W ), indicating number of samples, number of channels, height and width of a channel respectively as shown in Fig. 2 . Let h ncij and ĥncij be a pixel before and after normalization,

H:

where n ∈ [1, N ], c ∈ [1, C], i ∈ [1, H], and j ∈ [1, W ].

Let µ and σ be a mean and a standard deviation. We have

EQUATION

where γ and β are a scale and a shift parameter respectively. is a small constant to preserve numerical stability. Eqn.(1) shows that each pixel is normalized by using µ and σ, and then re-scale and re-shift by γ and β.

IN, LN, and BN share the formulation of Eqn.( 1), but they use different sets of pixels to estimate µ and σ. In other words, the num-bers of their estimated statistics are different. In general, we have

EQUATION

where k ∈ {in, ln, bn} is used to distinguish different methods. I k is a set pixels and |I k | denotes the number of pixels. Specifically, I in , I ln , and I bn are the sets of pixels used to compute statistics in different approaches.

IN was established in the task of artistic image style transfer (Johnson et al., 2016; Huang & Belongie, 2017) . In IN, we have

µ in , σ 2 in ∈ R N ×C and I in = {(i, j)|i ∈ [1, H], j ∈ [1, W ]}, meaning that IN has 2N C elements of

statistics, where each mean and variance value is computed along (H, W ) for each channel of each sample. LN (Ba et al., 2016) was proposed to ease optimization of recurrent neural networks (RNNs). In LN, we have µ ln , σ 2 ln ∈ R N ×1 and

I ln = {(c, i, j)|c ∈ [1, C], i ∈ [1, H], j ∈ [1, W ]},

implying that LN has 2N statistical values, where a mean value and a variance value are computed in (C, H, W ) for each one of the N samples.

BN (Ioffe & Szegedy, 2015) was first demonstrated in the task of image classification (He et al., 2016; Krizhevsky et al., 2012) by normalizing the hidden feature maps of CNNs. In BN, we have µ bn , σ 2 bn ∈ R C×1 and

I bn = {(n, i, j)|n ∈ [1, N ], i ∈ [1, H], j ∈ [1, W ]}

, in the sense that BN treats each channel independently like IN, but not only normalizes across (H, W ), but also the N samples in a minibatch, leading to 2C elements of statistics.

2.1. FORMULATION OF SN SN has an intuitive expression:

EQUATION

where Ω is a set of statistics estimated in different ways. In this work, we define Ω = {in, ln, bn} the same as above where µ k and σ 2 k can be calculated by following Eqn.(2). However, this strategy leads to large redundant computations. In fact, the three kinds of statistics of SN depend on each other. Therefore we could reduce redundancy by reusing computations,

EQUATION

showing that the means and variances of LN and BN can be computed based on IN. Using Eqn.(4), the computational complexity of SN is O(N CHW ), which is comparable to previous work.

Furthermore, w k and w k in Eqn.

(3) are importance ratios used to weighted average the means and variances respectively. Each w k or w k is a scalar variable, which is shared across all channels. There are 3×2 = 6 importance weights in SN. We have

Σ k∈Ω w k = 1, Σ k∈Ω w k = 1, and ∀w k , w k ∈ [0, 1],

and define w k = e λ k Σ z∈{in,ln,bn} e λz and k ∈ {in, ln, bn}.

(5)

Here each w k is computed by using a softmax function with λ in , λ ln , and λ bn as the control parameters, which can be learned by back-propagation (BP). w k are defined similarly by using another three control parameters λ in , λ ln , and λ bn .

Training. Let Θ be a set of network parameters (e.g. filters) and Φ be a set of control parameters that control the network architecture. In SN, we have Φ = {λ in , λ ln , λ bn , λ in , λ ln , λ bn }. Training a deep network with SN is to minimize a loss function L(Θ, Φ), where Θ and Φ can be optimized jointly by back-propagation (BP). This training procedure is different from previous meta-learning algorithms such as network architecture search (Colson et al., 2007; Liu et al., 2018; Pham et al., 2018) 

BN (Ioffe & Szegedy, 2015) γ, β 2C p, µ, σ, µ , σ O(N CHW ) 2C IN (Ulyanov et al., 2016) γ, β 2C µ, σ O(N CHW ) 2CN LN (Ba et al., 2016) γ, β 2C µ, σ O(N CHW ) 2N GN (Wu & He, 2018) γ, β 2C g, µ, σ O(N CHW ) 2gN BRN (Ioffe, 2017) γ, β 2C p, , r, d µ, σ, µ , σ O(N CHW ) 2C BKN (Wang et al., 2018) A C 2 p, µ, Σ, µ , Σ O(N C 2 HW ) C + C 2 WN (Salimans & Kingma, 2016) γ C - - - - SN γ, β, {w k } k∈Ω 2C + 6 {µ k , σ k } k∈Ω O(N CHW ) 2C + 2N +2CN

Table 1 : Comparisons of normalization methods. First, we compare their types of parameters, numbers of parameters (#params), and hyper-parameters. Second, we compare types of statistics, computational complexity to estimate statistics, and numbers of statistics (#statistics). Specifically, γ, β denote the scale and shift parameters. µ, σ, Σ are a vector of means, a vector of standard deviations, and a covariance matrix. µ represents the moving average. Moreover, p is the momentum of moving average, g in GN is the number of groups, is a small value for numerical stability, and r, d are used in BRN. In SN, k ∈ Ω indicates a set of different kinds of statistics, Ω = {in, ln, bn}, and w k is an importance weight of each kind.

of the network and all the SN layers, and feed the network with a certain number of mini-batches randomly chosen from the training set. Second, we average the means and variances produced by all these mini-batches in each SN layer. The averaged statistics are used by BN in SN.

We find that batch average makes training converged faster than moving average. It can be computed by using a small amount of samples. For example, top-1 accuracies of ResNet50 on ImageNet by using batch average with 50k and all training samples are 76.90% and 76.92% respectively. They are trained much faster and slightly better than 76.89% of moving average. Appendix A shows more results.

Implementation. SN can be easily implemented in existing softwares such as PyTorch and Tensor-Flow. The backward computation of SN can be obtained by automatic differentiation (AD) in these softwares. Without AD, we need to implement back-propagation (BP) of SN, where the errors are propagated through µ k and σ 2 k . We provide the derivations of BP in Appendix H.

3. RELATIONSHIPS TO PREVIOUS WORK:

In Table 1 , we compare SN to BN, IN, LN, and GN, as well as three variants of BN including Batch Renormalization (BRN), Batch Kalman Normalization (BKN), and WN. In general, we see that SN possesses comparable numbers of parameters and computations, as well as rich statistics. Details are presented below.

• First, SN has similar number of parameters compared to previous methods, as shown in the first portion of Table 1 . Most of the approaches learn a scale parameter γ and a bias β for each one of the C channels, resulting in 2C parameters. SN learns 6 importance weights as the additional parameters. We see that BKN has the maximum number of C 2 parameters, as it learns a transformation matrix A for the means and variances. WN has C scale parameters without the biases.

Furthermore, many methods have p and as hyper-parameters, whose values are not sensitive because they are often fixed in different tasks. In contrast, GN and BRN have to search the number of groups g or the renormalization parameters r, d, which may have different values in different networks. Moreover, WN does not have hyper-parameters and statistics, since it performs normalization in the space of network parameters rather than feature space. Salimans & Kingma (2016) ; Luo et al. (2019) showed that WN is a special case of BN.

• Second, although SN has richer statistics, the computational complexity to estimate them is comparable to previous methods, as shown in the second portion of Also, approximating the covariance matrix in a minibatch is nontrivial as discussed in (Desjardins et al., 2015; Luo, 2017b; a) . BN, BRN, and BKN also compute moving averages.

• Third, SN is demonstrated in various networks, tasks, and datasets. Its applications are much wider than existing normalizers and it also has rich theoretical value that is worth exploring.

We would also like to acknowledge the contributions of previous work that explored spatial region (Ren et al., 2016) and conditional normalization (Perez et al., 2017) .

4. EXPERIMENTS:

This section presents the main results of SN in multiple challenging problems and benchmarks, such as ImageNet (Russakovsky et al., 2015) , COCO (Lin et al., 2014) , Cityscapes (Cordts et al., 2016) , ADE20K (Zhou et al., 2017) , and Kinetics (Kay et al., 2017) , where the effectiveness of SN is demonstrated by comparing with existing normalization techniques.

4.1. IMAGE CLASSIFICATION IN IMAGENET:

SN is first compared with existing normalizers on the ImageNet classification dataset of 1k categories. All the methods adopt ResNet50 as backbone network. The experimental setting and more results are given in Appendix C.

Comparisons. The top-1 accuracy on the 224×224 center crop is reported for all models. SN is compared to BN and GN as shown in Table 2 . In the first five columns, we see that the accuracy of BN reduces by 1.1% from (8, 16) to (8, 8) and declines to 65.3% of (8, 2), implying that BN is unsuitable in small minibatch, where the random noise from the statistics is too heavy. GN obtains around 75.9% in all cases, while SN outperforms BN and GN in almost all cases, rendering its robustness to different batch sizes. In Appendix, Fig. 6 plots the training and validation curves, where SN enables faster convergence while maintains higher or comparable accuracies than those of BN and GN.

The middle two columns of Table 2 average the gradients in a single GPU by using only 16 and 32 samples, such that their batch sizes are the same as (8, 2) and (8, 4). SN again performs best in these single-GPU settings, while BN outperforms GN. For example, unlike (8, 4) that uses 8 GPUs, BN achieves 76.5% in (1, 32), which is the best-performing result of BN, although the batch size to compute the gradients is as small as 32. From the above results, we see that BN's performance are sensitive to the statistics more than the gradients, while SN are robust to both of them. The last two columns of Table 2 BKN (Wang et al., 2018) estimated the statistics in the current layer by combining those computed in the preceding layers. It estimates the covariance matrix rather than the variance vector. In particular, how to connect the layers requires careful design for every specific network. For ResNet50 with (8, 32), BKN achieved 76.8%, which is comparable to 76.9% of SN. However, for small minibatch, BKN reported 76.1% that was evaluated in a micro-batch setting where 256 samples are used to compute gradients and 4 samples to estimate the statistics. This setting is easier than (8, 4) that uses 32 samples to compute gradients. Furthermore, it is unclear how to apply BRN and BKN in the other tasks such as object detection and segmentation.

4.2. OBJECT DETECTION AND INSTANCE SEGMENTATION IN COCO:

Next we evaluate SN in object detection and instance segmentation in COCO (Lin et al., 2014) . Unlike image classification, these two tasks benefit from large size of input images, making large memory footprint and therefore leading to small minibatch size, such as 2 samples per GPU (Ren et al., 2015; Lin et al., 2016) . In this case, as BN is not applicable in small minibatch, previous work (Ren et al., 2015; Lin et al., 2016; He et al., 2017) often freeze BN and turns it into a constant linear transformation layer, which actually performs no normalization. Overall, SN selects different operations in different components of a detection system (see Fig. 1 ), showing much more superiority than both BN and GN. The experimental settings and more results are given in Appendix D.

Table 3 reports results of Faster R-CNN by using ResNet50 and the Feature Pyramid Network (FPN) (Lin et al., 2016) . A baseline BN † achieves an AP of 36.7 without using normalization in the detection head. When using SN and GN in the head and BN † in the backbone, BN † +SN improves the AP of BN † +GN by 0.8 (from 37.2 to 38.0). We investigate using SN and GN in both the backbone and head. In this case, we find that GN improves BN 

4.3. SEMANTIC IMAGE PARSING IN CITYSCAPES AND ADE20K:

We investigate SN in semantic image segmentation in ADE20K (Zhou et al., 2017) and Cityscapes (Cordts et al., 2016) . The empirical setting can be found in Appendix E.

Table 5 reports mIoU on the ADE20K validation set and Cityscapes test set, by using both singlescale and multi-scale testing. In SN, BN is not synchronized across GPUs. In ADE20K, SN outperforms SyncBN with a large margin in both testing schemes (38.7 v.s. 36.4 and 39.2 v.s. 37.7), and improve GN by 3.0 and 2.9. In Cityscapes, SN also performs best compared to SyncBN and GN. For example, SN surpasses SyncBN by 1.5 and 2.1 in both testing scales. We see that GN performs worse than SyncBN in these two benchmarks. Fig. 9 in Appendix compares the importance weights of SN in ResNet50 trained on both ADE20K and Cityscapes, showing that different datasets would choose different normalizers when the models and tasks are the same.

4.4. VIDEO RECOGNITION IN KINETICS:

We evaluate video recognition in Kinetics dataset (Kay et al., 2017) , which has 400 action categories. We experiment with Inflated 3D (I3D) convolutional networks (Carreira & Zisserman, 2017) and employ the ResNet50 I3D baseline as described in (Wu & He, 2018) . The models are pretrained from ImageNet. For all normalizers, we extend the normalization from over (H, W ) to over (T, H, W ), where T is the temporal axis. We train in the training set and evaluate in the validation set. The top1 and top5 classification accuracy are reported by using standard 10-clip testing that averages softmax scores from 10 clips sampled regularly.

Table 6 shows that SN works better than BN and GN in both batch sizes. For example, when batch size is 4, top1 accuracy of SN is better than BN and GN by 1.2% and 0.5%. It is seen that SN already surpasses BN and GN with batch size of 8. SN with batch size 8 further improves the results.

4.5. ON THE OTHER TASKS:

We also evaluate SN in the tasks of artistic image stylization (Johnson et al., 2016) and efficient neural architecture search (Pham et al., 2018) . The results are presented in Appendix F and G, where SN achieves competitive results.

5. DISCUSSIONS AND FUTURE WORK:

This work presented Switchable Normalization (SN) to learn different operations in different normalization layers of a deep network. This novel perspective opens up new direction in many research fields that employ deep learning, such as CV, ML, NLP, Robotics, and Medical Imaging. This work has demonstrated SN in multiple tasks of CV such as recognition, detection, segmentation, image stylization, and neural architecture search, where SN outperforms previous normalizers without bells and whistles. The implementations of these experiments will be released. Our analyses (Luo et al., 2018) suggest that SN has an appealing characteristic to balance learning and generalization when training deep networks. Investigating SN facilitates the understanding of normalization approaches (Shao et al., 2019; Pan et al., 2019; Luo, 2017a; b) , such as sparse SN (Shao et al., 2019) and switchable whitening (Pan et al., 2019) . 8, 32) . We see that SN with batch average produces faster and more stable convergence than the other methods.

In SN, BN employs batch average rather than moving average. We provide comparisons between them as shown in Fig. 5 , where SN is evaluated with both moving average and batch average to estimate the statistics used in test. They are used to train ResNet50 on ImageNet. The two settings of SN produce similar results of 76.9% when converged, which is better than 76.4% of BN. We see that SN with batch average converges faster and more stably than BN and SN that use moving average. In this work, we find that for all batch settings, SN with batch average provides results better than moving average. We also found that the conventional BN can be improved by replacing moving average by using batch average.

B PROOF OF REMARK 1:

Remark 1. Suppose x to be an image patch with zero mean and unit variance, w i to be a filter of the i-th channel, i ∈ {1, 2, ..., C}, and v to be the filter norm of WN. Eqn.( 1) can be rewritten as ĥin = γ wi T x wi 2 + β; ĥbn = γ wi T x wi 2 + β, s.t. γ ≤ v; and ĥln = γ For BN, we follow the derivations in (Luo et al., 2019) where the batch statistics µ bn and σ bn are treated as random variables. BN can be reformulated as population normalization (PN) and adaptive gamma decay. Let L = 1 P P j=1 E µ bn ,σ bn ( ĥj bn ) be the expected loss function of BN by integrating over random variables µ bn and σ bn . We have L

1 P P j=1 ( hj pn ) + ζ(h)γ 2

, where hj pn = γ h j -µ P σ P + β represents population normalization (PN) with h j = w T x j . µ P and σ P are the population mean and population standard deviation. ζ(h) is a data-dependent coefficient. Therefore, ζ(h)γ 2 represents adaptive gamma regularization whose strength is depended on training data. With normalized input, we have µ P = 0 and σ P = 1. Thus PN can be rewritten as WN, that is, hpn = γ wi T x wi 2 + β. Let WN be defined as v wi T x wi 2 . Then v in WN and γ in PN have the same learning dynamic. However, the adaptive gamma regularization imposes the γ ≤ v constraint to BN, since WN does not have regularization on v. Compared to WN, we express BN as ĥbn = γ wi T x wi 2 + β, s.t. γ ≤ v.

C IMAGENET C.1 EXPERIMENTAL SETTING:

All models in ImageNet are trained on 1.2M images and evaluated on 50K validation images. They are trained by using SGD with different settings of batch sizes, which are denoted as a 2-tuple, (number of GPUs, number of samples per GPU). For each setting, the gradients are aggregated over all GPUs, and the means and variances of the normalization methods are computed in each GPU. The network parameters are initialized by following (He et al., 2016) . For all normalization methods, all γ's are initialized as 1 and all β's as 0. The parameters of SN (λ k and λ k ) are initialized as 1. We use a weight decay of 10 -4 for all parameters including γ and β. All models are trained for 100 epoches with a initial learning rate of 0.1, which is deceased by 10× after 30, 60, and 90 epoches. For different batch sizes, the initial learning rate is linearly scaled according to (Goyal et al., 2017) . During training, we employ data augmentation the same as (He et al., 2016) . The top-1 classification accuracy on the 224×224 center crop is reported.

C.2 MORE RESULTS:

Fig. 6 (a) plots the validation curves of SN. Fig. 6 (b) and (c) compare the training and validation curves of SN, BN and GN in (8, 32) and (8, 2) respectively. From all these curves, we see that SN enables faster convergence while maintains higher or comparable accuracies than those of BN and GN.

Ablation Study of Importance Weights. In particular, the selected operations of each SN layer are shown in Fig. 7 . We have several observations. First, for the same batch size, the importance weights of µ and σ could have notable differences, especially when comparing 'res1,4,5' of (a,b) and 'res2,4,5' of (c,d) . For example, σ of BN (green) in 'res5' in (b,d) are mostly reduced compared to µ of BN in (a,c). As discussed in (Ioffe & Szegedy, 2015; Salimans & Kingma, 2016) , this is because the variance estimated in a minibatch produces larger noise than the mean, making training instable. SN is able to restrain the noisy statistics and stabilize training.

Second, the SN layers in different places of a network may select distinct operations. In other words, when comparing the adjacent SN layers after the 3 × 3 conv layer, shortcut, and the 1 × 1 conv layer, we see that they may choose different importance weights, e.g. 'res2,3'. The selectivity of operations in different places (normalization layers) of a deep network has not been observed in previous work.

Third, deeper layers prefer LN and IN more than BN, as illustrated in 'res5', which tells us that putting BN in an appropriate place is crucial in the design of network architecture. Although the stochastic uncertainty in BN (i.e. the minibatch statistics) acts as a regularizer that might benefit generalization, using BN uniformly in all normalization layers may impede performance.

D COCO DATASET:

SN is easily plugged into different detection frameworks implemented by using different softwares. We implement it on existing detection softwares of PyTorch and Caffe2-Detectron (Girshick et al., 2018) respectively. We conduct 3 settings, including setting-1: Faster R-CNN (Ren et al., 2015) on PyTorch; setting-2: Faster R-CNN+FPN (Lin et al., 2016) on Caffe2; and setting-3: Mask R-CNN (He et al., 2017) +FPN on Caffe2. For all these settings, we choose ResNet50 as the backbone network. In each setting, the experimental configurations of all the models are the same, while only the normalization layers are replaced. All models of SN are finetuned from (8, 2) in ImageNet. portance weights for µ and σ of (8, 32), while (c,d) show those of (8, 2). The y-axis represents the importance weights that sum to 1, while the x-axis shows different residual blocks of ResNet50. The SN layers in different places are highlighted differently. For example, the SN layers follow the 3 × 3 conv layers are outlined by shaded color, those in the shortcuts are marked with ' ', while those follow the 1 × 1 conv layers are in flat color. The first SN layer follows a 7 × 7 conv layer. We see that SN learns distinct importance weights for different normalization methods as well as µ and σ, adapting to different batch sizes, places, and depths of a deep network. Experimental Settings. For setting-1, we employ a fast implementation (Yang et al., 2017) of Faster R-CNN in PyTorch and follow its protocol. Specifically, we train all models on 4 GPUs and 3 images per GPU. Each image is re-scaled such that its shorter side is 600 pixels. All models are trained for 80k iterations with a learning rate of 0.01 and then for another 40k iterations with 0.001. For setting-2 and setting-3, we employ the configurations of the Caffe2-Detectron (Girshick et al., 2018) . We train all models on 8 GPUs and 2 images per GPU. Each image is re-scaled to its shorter side of 800 pixels. In particular, for setting-2, the learning rate (LR) is initialized as 0.02 and is decreased by a factor of 0.1 after 60k and 80k iterations and finally terminates at 90k iterations. This is referred as the 1x schedule in Detectron. In setting-3, the LR schedule is twice as long as the 1x schedule with the LR decay points scaled twofold proportionally, referred as 2x schedule. For all settings, we set weight decay to 0 for both γ and β following (Wu & He, 2018) . All the above models are trained in the 2017 train set of COCO by using SGD with a momentum of 0.9 and a weight decay of 10 -4 on the network parameters, and tested in the 2017 val set. We report the standard metrics of COCO, including average precisions at IoU=0.5:0.05:0.75 (AP), IoU=0.5 (AP .5 ), and IoU=0.75 (AP .75 ) for both bounding box (AP b ) and segmentation mask (AP m ). Also, we report average precisions for small (AP s ), medium (AP m ), and large (AP l ) objects. In the upper part of Table 7 , the baseline is denoted as BN † , where the BN layers are frozen. We see that freezing BN performs significantly better than finetuning BN (29.6 v.s. 19.3) . SN and GN enable finetuning the normalization layers, where SN obtains the best-performing AP of 33.0 in this setting. Fig. 8 (a) compares their AP curves.

Results of Setting-1. As shown in:

As reported in the lower part of Table 7 , SN and GN allow us to train COCO from scratch without pretraining on ImageNet, and they still achieve competitive results. For instance, 29.5 of SN ‡ outperforms BN ‡ by a large margin of 9.5 AP and GN ‡ by 1.2 AP. Their learning curves are compared in Fig. 8 (b) .

Results of Setting-2 and -3. The results of setting-2 and setting-3 are presented in the paper.

E SEMANTIC IMAGE PARSING:

Setting. Similar to object detection, semantic image segmentation also benefits from large input size, making the minibatch size is small during training. We use 2 samples per GPU for ADE20K and 1 sample per GPU for Cityscapes. We employ the open-source software in PyTorch2 and only replace the normalization layers in CNNs with the other settings fixed. For both datasets, we use DeepLab (Chen et al., 2018) with ResNet50 as the backbone network, where output stride = 8 and the last two blocks in the original ResNet contains atrous convolution with rate = 2 and rate = 4 respectively. Following (Zhao et al., 2017) , we employ "poly" learning rate policy with power = 0.9 and use the auxiliary loss with the weight 0.4 during training. The bilinear operation is adopted to upsmaple the score maps in the validation phase.

ADE20K. SyncBN and GN adopt the pretrained models on ImageNet. SyncBN collects the statistics from 8 GPUs. Thus the actual "batchsize" is 16 during training. To evaluate the performance of SN, we use SN (8, 2) in ImageNet as the pretrained model. For all models, we resize each image to 450 × 450 and train for 100, 000 iterations. We performance multi-scale testing with input size = {300, 400, 500, 600}. Cityscapes. For all models, we finetune from their pretrained ResNet50 models. SN finetunes from (8, 2). For all models, the batchsize is 8 in finetuning. We use random crop with the size 713 × 713 and train for 700 epoches. For multi-scale testing, the inference scales are {1.0, 1.25, 1.5, 1.75}.

Ablation Study. Fig. 9 compares the importance weights of SN in ResNet50 trained on both ADE20K and Cityscapes. We see that even when the models and tasks are the same, different training data encourage SN to choose different normalizers.

F ARTISTIC IMAGE STYLIZATION:

We evaluate SN in the task of artistic image stylization. We adopt a recent advanced approach (Johnson et al., 2016) , which jointly minimizes two loss functions. Specifically, one is a feature reconstruction loss that penalizes an output image when its content is deviated from a target image, and the other is a style reconstruction loss that penalizes differences in style (e.g. color, texture, exact boundary). Johnson et al. (2016) ; Huang & Belongie (2017) show that IN works better than BN in this task.

We compare SN with IN and BN using VGG16 (Simonyan & Zisserman, 2014) as backbone network. All models are trained on the COCO dataset (Lin et al., 2014) . For each model in training, we resize each image to 256×256 and train for 40, 000 iterations with a batch size setting of (1, 4). We do not employ weight decay or dropout. The other training protocols are the same as (Johnson et al., 2016) . In test, we evaluate the trained models on 512×512 images selected following (Johnson et al., 2016) . 

G NEURAL ARCHITECTURE SEARCH:

We investigate SN in LSTM for efficient neural architecture search (ENAS) (Pham et al., 2018) , which is designed to search the structures of convolutional cells. In ENAS, a convolutional neural network (CNN) is constructed by stacking multiple convolutional cells. It consists of two steps, training controllers and training child models. A controller is a LSTM whose parameters are trained by using the REINFORCE (Williams, 1992) algorithm to sample a cell architecture, while a child model is a CNN that stacks many sampled cell architectures and its parameters are trained by back-propagation with SGD. In (Pham et al., 2018) , the LSTM controller is learned to produce an architecture with high reward, which is the classification accuracy on the validation set of CIFAR-10 (Krizhevsky, 2009) . Higher accuracy indicates the controller produces better architecture.

We compare SN with LN and GN by using them in the LSTM controller to improve architecture search. As BN is not applicable in LSTM and IN is equivalent to LN in fully-connected layer (i.e. both compute the statistics across neurons), SN combines LN and GN in this experiment. Fig.10 (b) shows the validation accuracy of CIFAR10. We see that SN obtains better accuracy than both LN and GN.

H BACK-PROPAGATION OF SN:

For the software without auto differentiation, we provide the backward computations of SN below. Let ĥ be the output of the SN layer represented by a 4D tensor (N, C, H, W ) with index n, c, i, j. Let ĥ = γ h+β and h = h-µ √ σ 2 + , where µ = w bn µ bn +w in µ in +w ln µ ln , σ 2 = w bn σ 2 bn +w in σ 2 in +w ln σ 2 ln , and w bn + w in + w ln = 1. Note that the importance weights are shared among the means and variances for clarity of notations. Suppose that each one of {µ, µ bn , µ in , µ ln , σ 2 , σ 2 bn , σ 2 in , σ 2 ln } is reshaped into a vector of N × C entries, which are the same as the dimension of IN's statistics. Let L be the loss function and ( ∂L ∂µ ) n be the gradient with respect to the n-th entry of µ. We have 

EQUATION

EQUATION

:

https://github.com/CSAILVision/semantic-segmentation-pytorch

Figures and Tables:

Figure 2: The size of feature maps is N ×C ×H ×W (N = 4 in this example). Different normalizers estimate statistics along different axes.

Figure 5: Comparisons of 'BN', 'SN with moving average', and 'SN with batch average', when training ResNet50 on ImageNet in (8, 32). We see that SN with batch average produces faster and more stable convergence than the other methods.

By combining them, Eqn.(3) is rewritten as ĥsn = γ wi T x wi 2+wln C j =i wj 2 + β, s.t. w bn γ ≤ v, where w ln , w bn are the weights and γ ≤ v is a constraint. Proof. Eqn.(1) shows that IN, LN, and BN can be generally computed as ĥk = γ wi T x-µ k σ k + β, k ∈ {in, ln, bn}. When x is normalized to zero mean and unit variance, we have ĥin = γ wi T x wi 2 + β and ĥln = γ wi T x wi 2+ C j =i wj 2 + β according to their definitions.

validation curves of SN in different batch sizes.

train and validation curves of (8,32), SN v.s. BN and GN.

train and validation curves of (8,2), SN v.s. GN.

Figure 6: Comparisons of learning curves. (a) visualizes the validation curves of SN with different settings of batch size. The bracket (•, •) denotes (#GPUs, #samples per GPU). (b) compares the top-1 train and validation curves on ImageNet of SN, BN, and GN in the batch size of (8,32). (c) compares the train and validation curves of SN and GN in the batch size of (8,2).

Figure 7: Selected operations of each SN layer in ResNet50. There are 53 SN layers. (a,b) show the im-

Figure 8: Average precision (AP) curves of Faster R-CNN on the 2017 val set of COCO. (a) plots the results of finetuning pretrained networks. (b) shows training the models from scratch.

importance weights of σ of (8,2) in ADE20K.

importance weights of µ of (8,2) in Cityscapes. importance weights of σ of (8,2) in Cityscapes.

Figure 9: Selected normalizers of each SN layer in ResNet50 for semantic image parsing in ADE20Kand Cityscapes. There are 53SN layers. (a,b)  show the importance weights for µ and σ of (8, 2) in ADE20K, while (c,d) show those of (8, 2) in Cityscapes. The y-axis represents the importance weights that sum to 1, while the x-axis shows different residual blocks of ResNet50. The SN layers in different places are highlighted differently. For example, the SN layers follow the 3 × 3 conv layers are outlined by shaded color, those in the shortcuts are marked with ' ', while those follow the 1 × 1 conv layers are in flat color.

Figure 10: (a) shows the losses of BN, IN, and SN in the task of image stylization. SN converges faster than IN and BN. As shown in Fig.1 and the supplementary material, SN adapts its importance weight to IN while producing comparable stylization results. (b) plots the accuracy on the validation set of CIFAR-10 when searching network architectures.

Fig.10 (a) compares the style and feature reconstruction losses. We see that SN enables faster convergence than both IN and BN. As shown in Fig.1 (a), SN automatically selects IN in image stylization. Some stylization results are visualized in Fig.11.

. In previous work, Φ represents as a set of network modules with different learning capacities, where Θ and Φ were optimized in two BP stages iteratively by using two training sets that are non-Let x be an image patch with zero mean and unit variance, w i be a filter of the ith channel, i ∈ {1, 2, ..., C}, and v be the filter norm of WN. Eqn.(1) can be rewritten by ĥin = γ wi T x wi 2 +β; ĥbn = γ wi T x wi 2 +β, s.t. γ ≤ v; and ĥln = γ SN in Eqn.(3) can be reformulated by ĥsn = w in ĥin +w bn ĥbn +w ln ĥln = γ .t. w bn γ ≤ v, where w in , w ln , w bn are the weights and γ ≤ v is a constraint.

Table content: <table><tr><td/><td>Parameter</td><td/><td colspan="4">Statistical Estimation</td></tr><tr><td>params</td><td>#params</td><td>hyper-params</td><td>statistics</td><td colspan="3">computation complexity</td><td>#statistics</td></tr><tr><td colspan="7">overlapped. For example, previous work divided an entire training set into a training and a validation</td></tr><tr><td colspan="7">set. However, if Θ and Φ in previous work are optimized in the same set of training data, Φ would</td></tr><tr><td colspan="7">choose the module with large complexity to overfit these data. In contrast, SN essentially prevents</td></tr><tr><td colspan="7">overfitting by choosing normalizers to improve both learning and generalization ability as discussed</td></tr><tr><td>below.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="7">Analyses of SN. To understand SN, we theoretically compare SN with BN, IN, and LN by repre-</td></tr><tr><td colspan="7">senting them using weight normalization (WN) (Salimans &amp; Kingma, 2016) that is independent of</td></tr><tr><td colspan="7">mean and variance. WN is computed as v w T x w 2 , where w and x represent a filter and an image patch.</td></tr><tr><td colspan="4">WN normalizes the norm of each filter to 1 and rescales to v.</td><td/><td/></tr><tr><td colspan="5">Remark 1. wi T x wi 2+ C j =i wj 2</td><td colspan="2">+β. By combining them,</td></tr><tr><td/><td/><td/><td/><td/><td colspan="2">wi T x wi 2+wln C j =i wj 2</td><td>+</td></tr><tr><td colspan="6">w 1 w 1 IN v γ 1 γ 2 β, sw 1 w 2 w 2 Remark 1 simplifies SN in Eqn.(3), enabling us to com-pare different normalizers geometrically by formulating them with respect to WN. In Fig.3, ĥin of IN can be computed similarly to WN with an additional bias β, where the norms of all filters are normalized to 1 and then rescaled to γ. As γ and v have the same learning dynam-w 2 v ic, the length of γ would be identically to v (see w 1 , w 2 γ 1 of IN). Moreover, ĥbn in BN can be rewritten as WN with γ 2 regularization over γ, making it shorter than v. Compared</td><td>v v</td><td>w 1 BN γ 1 γ 2 γ 1 w 1</td><td>w 2</td></tr><tr><td colspan="3">to IN and LN, Luo et al. (2019) shows that the regulariza-tion of BN improves generalization and increases angle</td><td/><td/><td/><td>γ 2</td><td>w 2</td></tr><tr><td colspan="3">between filters, preventing them from coadaptation (see</td><td>LN</td><td/><td/><td>SN</td></tr><tr><td colspan="3">w 1 , w 2 of BN). Furthermore, ĥln in LN normalizes each filter among channels where the filter norm is less con-strained than IN and BN. That is, LN allows γ &gt; v to increase learning ability. Finally, ĥsn in SN inherits the</td><td colspan="4">Figure 3: Geometric view of directions and lengths of the filters in IN, BN, LN, and SN by comparing them to WN.</td></tr><tr><td colspan="3">benefits from all of them and enables balance between</td><td/><td/><td/></tr><tr><td colspan="7">learning and generalization ability. For example, when the batch size is small, the random noise</td></tr><tr><td colspan="7">from the batch statistics of BN would be too strong. SN is able to maintain performance by decreas-</td></tr><tr><td colspan="7">ing w bn and increasing w ln , such that the regularization from BN is reduced and the learning ability</td></tr><tr><td colspan="7">is enhanced by LN. This phenomenon is supported by our experiment. More results are provided in</td></tr><tr><td>Appendix B.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="7">Variants of SN. SN has many extensions. For instance, a pretrained network with SN can be fine-</td></tr><tr><td colspan="7">tuned by applying the argmax function on its control parameters where each normalization layer</td></tr><tr><td colspan="7">selects only one normalizer, leading to sparse SN. For (8, 32) as an example, SN with sparsity</td></tr><tr><td colspan="7">achieves top-1 accuracy of 77.0% in ImageNet with ResNet50, which is comparable to 76.9% of SN</td></tr><tr><td colspan="7">without sparsity. Moreover, when the channels are divided into groups, each group could select its</td></tr><tr><td colspan="7">own normalizer to increase representation power of SN. Our preliminary results suggest that group</td></tr><tr><td colspan="7">SN performs better than SN in some senses. For instance, group SN with only two groups boosts</td></tr></table>



Table content: <table/>

Comparisons of top-1 accuracies on the validation set of ImageNet, by using ResNet50 trained with SN, BN, and GN in different batch size settings. The bracket (•, •) denotes (#GPUs, #samples per GPU). In the bottom part, 'GN-BN' indicates the difference between the accuracies of GN and BN. The '-' in (8, 1) indicates BN does not converge. The best-performing result of each setting is shown in bold.

Table content: <table><tr><td>Figure 4: Importance weights v.s. batch sizes. The</td></tr><tr><td>bracket (•, •) indicates (#GPUs, #samples per GPU).</td></tr><tr><td>SN doesn't have BN in (8, 1).</td></tr></table>

Faster R-CNN+FPN using ResNet50 and FPN with 1x LR schedule. BN † represents BN is frozen. The best results are bold.

Table content: <table><tr><td colspan="3">backbone head AP b AP b .5 AP b .75</td><td>AP m AP m .5 AP m .75</td></tr><tr><td>BN  †</td><td>-</td><td>38.6 59.5 41.9</td><td>34.2 56.2 36.1</td></tr><tr><td>BN  †</td><td colspan="2">GN 39.5 60.0 43.2</td><td>34.4 56.4 36.3</td></tr><tr><td>BN  †</td><td>SN</td><td>40.0 61.0 43.3</td><td>34.8 57.3 36.3</td></tr><tr><td>GN</td><td colspan="2">GN 40.2 60.9 43.8</td><td>35.7 57.8 38.0</td></tr><tr><td>GN</td><td>SN</td><td>40.4 61.4 44.2</td><td>36.0 58.4 38.1</td></tr><tr><td>SN</td><td>SN</td><td>41.0 62.3 45.1</td><td>36.5 58.9 38.7</td></tr></table>

Mask R-CNN using ResNet50 and FPN with 2x LR schedule. BN † represents BN is frozen without finetuning. The best results are bold.

Table content: <table/>

Results of Kinetics dataset. In training,

Table content: <table><tr><td colspan="5">Published as a conference paper at ICLR 2019</td><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">ADE20K</td><td colspan="2">Cityscapes</td><td/><td colspan="2">batch=8, length=32</td><td colspan="2">batch=4, length=32</td></tr><tr><td/><td>mIoUss</td><td>mIoUms</td><td>mIoUss</td><td>mIoUms</td><td/><td>top1</td><td>top5</td><td>top1</td><td>top5</td></tr><tr><td>SyncBN</td><td>36.4</td><td>37.7</td><td>69.7</td><td>73.0</td><td>BN</td><td>73.2</td><td>90.9</td><td>72.1</td><td>90.0</td></tr><tr><td>GN</td><td>35.7</td><td>36.3</td><td>68.4</td><td>73.1</td><td>GN</td><td>73.0</td><td>90.6</td><td>72.8</td><td>90.6</td></tr><tr><td>SN</td><td>38.7</td><td>39.2</td><td>71.2</td><td>75.1</td><td>SN</td><td>73.5</td><td>91.3</td><td>73.3</td><td>91.2</td></tr><tr><td colspan="5">Table 5: Results in ADE20K validation set and C-</td><td/><td/><td/><td/><td/></tr><tr><td colspan="5">ityscapes test set by using ResNet50 with dilated con-</td><td colspan="5">the clip length of 32 frames is regularly sampled</td></tr><tr><td colspan="5">volutions. 'ss' and 'ms' indicate single-scale and multi-</td><td colspan="5">with a frame interval of 2. We study a batch size</td></tr><tr><td colspan="5">scale inference. SyncBN represents mutli-GPU synchro-</td><td colspan="5">of 8 or 4 clips per GPU. BN is not synchronized</td></tr><tr><td colspan="5">nization of BN. SN finetunes from (8, 2) pretrained mod-</td><td colspan="5">across GPUs. SN finetunes from (8, 2) pretrained</td></tr><tr><td>el.</td><td/><td/><td/><td/><td>model.</td><td/><td/><td/><td/></tr></table>

reports results of Mask R-CNN(He et al., 2017) with FPN. In the upper part, SN is compared to a head with no normalization and a head with GN, while the backbone is pretrained with BN, which is then frozen in finetuning (i.e. the ImageNet pretrained features are the same). We see that the baseline BN † achieves a box AP of 38.6 and a mask AP of 34.2. SN improves GN by 0.5 box AP and 0.4 mask AP, when finetuning the same BN † backbone.More direct comparisons with GN are shown in the lower part of Table4. We apply SN in the head and finetune the same backbone network pretrained with GN. In this case, SN outperforms GN by 0.2 and 0.3 box and mask APs respectively. Moreover, when finetuning the SN backbone, SN surpasses GN by a large margin of both box and maskAP (41.0 v.s. 40.2 and 36.5 v.s. 35.7). Note that the performance of SN even outperforms 40.9 and 36.4 of the 101-layered ResNet(Girshick et al., 2018).

Table content: <table/>

Faster R-CNN for detection in COCO using ResNet50 and RPN. BN † represents BN is frozen without finetuning. The superscript ' ‡' indicates the backbones are trained from scratch without pretraining on ImageNet.

Table content: <table><tr><td>backbone</td><td>head</td><td>AP</td><td>AP.5</td><td>AP.75</td><td>AP l</td><td>APm</td><td>APs</td></tr><tr><td>BN  †</td><td>BN  †</td><td>29.6</td><td>47.8</td><td>31.9</td><td>45.5</td><td>33.0</td><td>11.5</td></tr><tr><td>BN</td><td>BN</td><td>19.3</td><td>33.0</td><td>20.0</td><td>32.3</td><td>21.3</td><td>7.4</td></tr><tr><td>GN</td><td>GN</td><td>32.7</td><td>52.4</td><td>35.1</td><td>49.1</td><td>36.1</td><td>14.9</td></tr><tr><td>SN</td><td>SN</td><td>33.0</td><td>52.9</td><td>35.7</td><td>48.7</td><td>37.2</td><td>15.6</td></tr><tr><td>BN  ‡</td><td>BN</td><td>20.0</td><td>33.5</td><td>21.1</td><td>32.1</td><td>21.9</td><td>7.3</td></tr><tr><td>GN  ‡</td><td>GN</td><td>28.3</td><td>46.3</td><td>30.1</td><td>41.2</td><td>30.0</td><td>12.7</td></tr><tr><td>SN  ‡</td><td>SN</td><td>29.5</td><td>47.8</td><td>31.6</td><td>44.2</td><td>32.6</td><td>13.0</td></tr></table>

SN is compared with both BN and GN in the Faster R-CNN. In this setting, the layers up to conv4 of ResNet50 are used as backbone to extract features, and the layers of conv5 are used as the Region-of-Interest head for classification and regression. As the layers are inherited from the pretrained model, both the backbone and head involve normalization layers. Different results of Table7use different normalization methods in the backbone and head. Its upper part shows results of finetuning the ResNet50 models pretrained on ImageNet. The lower part compares training COCO from scratch without pretraining on ImageNet.

Table content: <table/>

and the gradients for λ in , λ ln , and λ bn are∂L ∂λ in = w in (1 -w in )

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="2">∂L ∂µ</td><td colspan="6">= -</td><td colspan="2">√</td><td>1 σ 2 +</td><td>H,W i,j</td><td>∂L ∂ hncij</td><td>,</td><td>(8)</td></tr><tr><td colspan="2">∂L ∂h ncij</td><td>=</td><td colspan="3">∂L ∂ hncij</td><td>•</td><td colspan="2">√</td><td colspan="6">1 σ 2 +</td><td>+</td><td>2w in (h ncij -µ in ) HW</td><td>∂L ∂σ 2</td></tr><tr><td>+</td><td colspan="7">2w ln (h ncij -µ ln ) CHW</td><td colspan="4">C c=1</td><td colspan="2">(</td><td>∂L ∂σ 2 ) c +</td><td>2w bn (h ncij -µ bn ) N HW</td><td>N n=1 (</td><td>∂L ∂σ 2 ) n</td></tr><tr><td>+</td><td>w in HW</td><td colspan="2">∂L ∂µ</td><td>+</td><td colspan="5">w ln CHW</td><td colspan="5">C c=1</td><td>(</td><td>∂L ∂µ</td><td>) c +</td><td>w bn N HW</td><td>N n=1 (</td><td>∂L ∂µ</td><td>) n ,</td><td>(9)</td></tr><tr><td colspan="4">The gradients for γ and β are</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="2">∂L ∂γ</td><td colspan="3">=</td><td colspan="4">N,H,W n,i,j</td><td>∂L ∂ ĥncij</td><td>• hncij ,</td><td>(10)</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">∂L ∂β</td><td/><td/><td colspan="2">=</td><td>N,H,W n,i,j</td><td>∂L ∂ ĥncij</td><td>,</td><td>(11)</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ in + (</td><td>∂L ∂σ 2 ) nc σ 2 in</td></tr><tr><td/><td/><td/><td/><td colspan="7">-w in w ln</td><td colspan="4">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ ln + (</td><td>∂L ∂σ 2 ) nc σ 2 ln</td></tr><tr><td/><td/><td/><td/><td colspan="8">-w in w bn</td><td/><td colspan="2">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ bn + (</td><td>∂L ∂σ 2 ) nc σ 2 bn ,</td><td>(12)</td></tr><tr><td/><td>∂L ∂λ ln</td><td/><td colspan="12">= w ln (1 -w ln )</td><td>N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ ln + (</td><td>∂L ∂σ 2 ) nc σ 2 ln</td></tr><tr><td/><td/><td/><td/><td colspan="7">-w in w ln</td><td colspan="4">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ in + (</td><td>∂L ∂σ 2 ) nc σ 2 in</td></tr><tr><td/><td/><td/><td/><td colspan="8">-w ln w bn</td><td/><td colspan="2">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ bn + (</td><td>∂L ∂σ 2 ) nc σ 2 bn ,</td><td>(13)</td></tr><tr><td colspan="2">∂L ∂λ bn</td><td colspan="13">= w bn (1 -w bn )</td><td>N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ bn + (</td><td>∂L ∂σ 2 ) nc σ 2 bn</td></tr><tr><td/><td/><td/><td/><td colspan="7">-w in w bn</td><td colspan="4">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ in + (</td><td>∂L ∂σ 2 ) nc σ 2 in</td></tr><tr><td/><td/><td/><td/><td colspan="7">-w ln w bn</td><td colspan="4">N,C n,c</td><td>(</td><td>∂L ∂µ</td><td>) nc µ ln + (</td><td>∂L ∂σ 2 ) nc σ 2 ln .</td><td>(14)</td></tr></table>

Peer review (iclr2019):
 In this work, the authors propose Switchable Normalization (SN), which *learns* to switch / select different normalization algorithms (including batch normalization (BN), Instance Normalization (IN), Layer Normalization (LN)) in layers of the networks and in different applications. The idea is motivated by observations (shown in Fig 1) that, 1) different tasks tend to have applied different normalization methods; 2) some normalization methods (e.g. BN) are fragile to very small batch size. The authors propose a general form for different normalization methods, which is a Gaussian normalization and then scale and shift by scalars. Different normalization methods utilize different statistics as the mean and the variance of the Gaussian normalization. The authors further propose to learn the combination weights on mean and variance, which is w_k and w*_k in Eqn (3). To avoid duplicate computation, the authors also do some careful simplification on computing mean and variance with all of the three normalization methods. In the experiment part, the authors demonstrate the effectiveness of the proposed SN method on various kinds of tasks, including ImageNet classification, object detection and instance segmentation in COCO, semantic image parsing and video recognition. In all of the tasks tested, which also cover the common application in computer vision, SN shows superior and robust performance. Pros: + Neat motivation; + Extensive experiments; + Clear illustration; Cons - There are still some experiment results missing, as the authors themselves mentioned in the Kinetics section (but the reviewer thinks it would be ready); - In Page 3 the training section and Page 4, the first paragraph, it mentioned Θ and Φ (which are the weights for different normalization methods) are jointly trained and different from the previous iterative meta-learning style method. The authors attribute "In contrast, SN essentially prevents overfitting by choosing normalizers to improve both learning and generalization ability as discussed below". The reviewer does not see it is well justified and the reviewer thinks optimizing them jointly could lead to instability in the training (but it did not happen in the experiments). The authors should justify the jointly training part better. - Page 5 the final paragraph, the reviewer does not see the point there. "We would also like to acknowledge the contributions of previous work that explored spatial region (Ren et al., 2016) and conditional normalization (Perez et al., 2017). " Please make it a bit more clear how these works are related.