Title: NORMALIZING THE NORMALIZERS: COMPARING AND EXTENDING NETWORK NORMALIZATION SCHEMES

Abstract: Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work well in the recurrent setting. In this paper we propose a unified view of normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework in the context of convolutional neural nets and recurrent neural networks, showing improvements over baselines in image classification, language modeling as well as super-resolution.

1. INTRODUCTION:

Standard deep neural networks are difficult to train. Even with non-saturating activation functions such as ReLUs (Krizhevsky et al., 2012) , gradient vanishing or explosion can still occur, since the Jacobian gets multiplied by the input activation of every layer. In AlexNet (Krizhevsky et al., 2012) , for instance, the intermediate activations can differ by several orders of magnitude. Tuning hyperparameters governing weight initialization, learning rates, and various forms of regularization thus become crucial in optimizing performance.

In current neural networks, normalization abounds. One technique that has rapidly become a standard is batch normalization (BN) in which the activations are normalized by the mean and standard deviation of the training mini-batch (Ioffe & Szegedy, 2015) . At inference time, the activations are normalized by the mean and standard deviation of the full dataset. A more recent variant, layer normalization (LN), utilizes the combined activities of all units within a layer as the normalizer (Ba et al., 2016) . Both of these methods have been shown to ameliorate training difficulties caused by poor initialization, and help gradient flow in deeper models.

A less-explored form of normalization is divisive normalization (DN) (Heeger, 1992) , in which a neuron's activity is normalized by its neighbors within a layer. This type of normalization is a well established canonical computation of the brain (Carandini & Heeger, 2012) and has been extensively studied in computational neuroscience and natural image modelling (see Section 2). However, with few exceptions (Jarrett et al., 2009; Krizhevsky et al., 2012) it has received little attention in conventional supervised deep learning.

Here, we provide a unifying view of the different normalization approaches by characterizing them as the same transformation but along different dimensions of a tensor, including normalization across examples, layers in the network, filters in a layer, or instances of a filter response. We explore the effect of these varieties of normalizations in conjunction with regularization, on the prediction performance compared to baseline models. The paper thus provides the first study of divisive normalization in a range of neural network architectures, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and tasks such as image classification, language modeling and image super-resolution. We find that DN can achieve results on par with BN in CNN networks and out-performs it in RNNs and super-resolution, without having to store batch statistics. We show that casting LN as a form of DN by incorporating a smoothing parameter leads to significant gains, in both CNNs and RNNs. We also find advantages in performance and stability by being able to drive learning with higher learning rate in RNNs using DN. Finally, we demonstrate that adding an L1 regularizer on the activations before normalization is beneficial for all forms of normalization.

2. RELATED WORK:

In this section we first review related work on normalization, followed by a brief description of regularization in neural networks.

2.1. NORMALIZATION:

Normalization of data prior to training has a long history in machine learning. For instance, local contrast normalization used to be a standard effective tool in vision problems (Pinto et al., 2008; Jarrett et al., 2009; Sermanet et al., 2012; Le, 2013) . However, until recently, normalization was usually not part of the machine learning algorithm itself. Two notable exceptions are the original AlexNet by Krizhevsky et al. (2012) which includes a divisive normalization step over a subset of features after ReLU at each pixel location, and the work by Jarrett et al. (2009) who demonstrated that a combination of nonlinearities, normalization and pooling improves object recognition in two-stage networks.

Recently Ioffe & Szegedy (2015) demonstrated that standardizing the activations of the summed inputs of neurons over training batches can substantially decrease training time in deep neural networks. To avoid covariate shift, where the weight gradients in one layer are highly dependent on previous layer outputs, Batch Normalization (BN) rescales the summed inputs according to their variances under the distribution of the mini-batch data. Specifically, if z j,n denotes the activation of a neuron j on example n, and B(n) denotes the mini-batch of examples that contains n, then BN computes an affine function of the activations standardized over each mini-batch:

zn,j = γ z n,j -E[z j ] 1 |B(n)| (z n,j -E[z j ]) 2 + β E[z j ] = 1 |B(n)| m∈B(n) z m,j

However, training performance in Batch Normalization strongly depends on the quality of the aquired statistics and, therefore, the size of the mini-batch. Hence, Batch Normalization is harder to apply in cases for which the batch sizes are small, such as online learning or data parallelism. While classification networks can usually employ relatively larger mini-batches, other applications such as image segmentation with convolutional nets use smaller batches and suffer from degraded performance. Moreover, application to recurrent neural networks (RNNs) is not straightforward and leads to poor performance (Laurent et al., 2015) .

Several approaches have been proposed to make Batch Normalization applicable to RNNs. Cooijmans et al. (2016) and Liao & Poggio (2016) collect separate batch statistics for each time step. However, neither of this techniques address the problem of small batch sizes and it is unclear how to generalize them to unseen time steps.

More recently, Ba et al. (2016) proposed Layer Normalization (LN), where the activations are normalized across all summed inputs within a layer instead of within a batch:

zn,j = γ z n,j -E[z n ] 1 |L(j)| (z n,j -E[z n ]) 2 + β E[z n ] = 1 |L(j)| k∈L(j) z n,k

where L(j) contains all of the units in the same layer as j. While promising results have been shown on RNN benchmarks, direct application of layer normalization to convolutional layers often leads to a degradation of performance. The authors hypothesize that since the statistics in convolutional layers can vary quite a bit spatially, normalization with statistics from an entire layer might be suboptimal. Ulyanov et al. (2016) proposed to normalize each example on spatial dimensions but not on channel dimension, and was shown to be effective on image style transfer applications (Gatys et al., 2016) . Liao et al. (2016a) proposed to accumulate the normalization statistics over the entire training phase, and showed that this can speed up training in recurrent and online learning without a deteriorating effect on the performance. Since gradients cannot be backpropagated through this normalization operation, the authors use running statistics of the gradients instead.

Exploring the normalization of weights instead of activations, Salimans & Kingma (2016) proposed a reparametrization of the weights into a scale independent representation and demonstrated that this can speed up training time.

Divisive Normalization (DN) on the other hand modulates the neural activity by the activity of a pool of neighboring neurons (Heeger, 1992; Bonds, 1989) . DN is one of the most well studied and widely found transformations in real neural systems, and thus has been called a canonical computation of the brain (Carandini & Heeger, 2012) . While the exact form of the transformation can differ, all formulations model the response of a neuron zj as a ratio between the acitivity in a summation field A j , and a norm-like function of the suppression field

EQUATION

where {u i } are the summation weights and {w k } the suppression weights.

Previous theoretical studies have outlined several potential computational roles for divisive normalization such as sensitivity maximization (Carandini & Heeger, 2012) , invariant coding (Olsen et al., 2010) , density modelling (Ballé et al., 2016) , image compression (Malo et al., 2006) , distributed neural representations (Simoncelli & Heeger, 1998) , stimulus decoding (Ringach, 2009; Froudarakis et al., 2014) , winner-take-all mechanisms (Busse et al., 2009) , attention (Reynolds & Heeger, 2009) , redundancy reduction (Schwartz & Simoncelli, 2001; Sinz & Bethge, 2008; Lyu & Simoncelli, 2008; Sinz & Bethge, 2013) , marginalization in neural probabilistic population codes (Beck et al., 2011) , and contextual modulations in neural populations and perception (Coen-Cagli et al., 2015; Schwartz et al., 2009) .

2.2. REGULARIZATION:

Various regularization techniques have been applied to neural networks for the purpose of improving generalization and reduce overfitting. They can be roughly divided into two categories, depending on whether they regularize the weights or the activations.

Regularization on Weights::

The most common regularizer on weights is weight decay which just amounts to using the L2 norm squared of the weight vector. An L1 regularizer (Goodfellow et al., 2016) on the weights can also be adopted to push the learned weights to become sparse. Scardapane et al. (2016) investigated mixed norms in order to promote group sparsity.

Regularization on Activations: Sparsity or group sparsity regularizers on the activations have shown to be effective in the past (Roz, 2008; Kavukcuoglu et al., 2009) and several regularizers have been proposed that act directly on the neural activations. Glorot et al. (2011) add a sparse regularizer on the activations after ReLU to encourage sparse representations. Dropout developed by Srivastava et al. (2014) applies random masks to the activations in order to discourage them to co-adapt. DeCov proposed by Cogswell et al. (2015) tries to minimize the off-diagonal terms of the sample covariance matrix of activations, thus encouraging the activations to be as decorrelated as possible. Liao et al. (2016b) utilize a clustering-based regularizer to encourage the representations to be compact. 

3. A UNIFIED FRAMEWORK FOR NORMALIZING NEURAL NETS:

We first compare the three existing forms of normalization, and show that we can modify batch normalization (BN) and layer normalization (LN) in small ways to make them have a form that matches divisive normalization (DN). We present a general formulation of normalization, where existing normalizations involve alternative schemes of accumulating information. Finally, we propose a regularization term that can be optimized jointly with these normalization schemes to encourage decorrelation and/or improve generalization performance.

3.1. GENERAL FORM OF NORMALIZATION:

Without loss of generality, we denote the hidden input activation of one arbitrary layer in a deep neural network as z ∈ R N ×L . Here N is the mini-batch size. In the case of a CNN, L = H × W × C, where H, W are the height and width of the convolutional feature map and C is the number of filters.

For an RNN or fully-connected layers of a neural net, L is the number of hidden units.

Different normalization methods gather statistics from different ranges of the tensor and then perform normalization. Consider the following general form:

z n,j = i w i,j x n,i + b j (2) v n,j = z n,j -E An,j [z] (3) zn,j = v n,j σ 2 + E Bn,j [v 2 ] (4)

where A j and B j are subsets of z and v respectively. A and B in standard divisive normalization are referred to as summation and suppression fields (Carandini & Heeger, 2012) . One can cast each normalization scheme into this general formulation, where the schemes vary based on how they define these two fields. These definitions are specified in Table 1 . Optional parameters γ and β can be added in the form of γ j zn,j + β j to increase the degree of freedom. 

3.2. NEW MODEL COMPONENTS:

Smoothing the Normalizers: One obvious way in which the normalization schemes differ is in terms of the information that they combine for normalizing the activations. A second more subtle but important difference between standard BN and LN as opposed to DN is the smoothing term σ, in the denominator of Eq. ( 1). This term allows some control of the bias of the variance estimation, effectively smoothing the estimate. This is beneficial because divisive normalization does not utilize information from the mini-batch as in BN, and combines information from a smaller field than LN. A

Model Range Normalizer Bias BN A n,j = {z m,j : m ∈ [1, N ], j ∈ [1, H] × [1, W ]} B n,j = {v m,j : m ∈ [1, N ], j ∈ [1, H] × [1, W ]} σ = 0 LN A n,j = {z n,i : i ∈ [1, L]} B n,j = {v n,i : i ∈ [1, L]} σ = 0 DN A n,j = {z n,i : d(i, j) ≤ R A } B n,j = {v n,i : d(i, j) ≤ R B } σ ≥ 0

Table 1 : Different choices of the summation and suppression fields A and B, as well as the constant σ in the normalizer lead to known normalization schemes in neural networks. d(i, j) denotes an arbitrary distance between two hidden units i and j, and R denotes the neighbourhood radius. similar but different denominator bias term max(σ, c) appears in (Jarrett et al., 2009) , which is active when the activation variance is small. However, the clipping function makes the transformation not invertible, losing scale information.

Moreover, if we take the nonlinear activation function after normalization into consideration, we find that σ will change the overall properties of the non-linearity. To illustrate this effect, we use a simple 1-layer network which consists of: two input units, one divisive normalization operator, followed by a ReLU activation function. If we fix one input unit to be 0.5, varying the other one with different values of σ produces different output curves (Fig. 2 , left). These curves exhibit different non-linear properties compared to the standard ReLU. Allowing the other input unit to vary as well results in different activation functions of the first unit depending on the activity of the second (Fig. 2 , right). This illustrates potential benefits of including this smoothing term σ, as it effectively modulates the rectified response to vary from a linear to a highly saturated response.

In this paper we propose modifications of the standard BN and LN which borrow this additive term σ in the denominator from DN. We study the effect of incorporating this smoother in the respective normalization schemes below.

L1 regularizer: Filter responses on lower layers in deep neural networks can be quite correlated which might impair the estimate of the variance in the normalizer. More independent representations help disentangle latent factors and boost the networks performance (Higgins et al., 2016) . Empirically, we found that putting a sparse (L1) regularizer

EQUATION

on the centered activations v n,j helps decorrelate the filter responses (Fig. 5 ). Here, N is the batch size and L is the number of hidden units, and L L1 is the regularization loss which is added to the training loss.

A possible explanation for this effect is that the L1 regularizer might have a similar effect as maximum likelihood estimation of an independent Laplace distribution. To see that, let p v (v) ∝ exp (-v 1 ) and x = W -1 v, with W a full rank invertible matrix. Under this model p

x (x) = p v (W x) |det W |.

Then, minimization of the L1 norm of the activations under the volume-conserving constraint det A = const. corresponds to maximum likelihood on that model, which would encourage decorrelated responses. We do not enforce such a constraint, and the filter matrix might even not be invertible. However, the supervised loss function of the network benefits from having diverse non-zero filters. This encourages the network to not collapse filters along the same direction or put them to zero, and might act as a relaxation of the volume-conserving constraint.

3.3. SUMMARY OF NEW MODELS:

DN and DN*: We propose DN as a new local normalization scheme in neural networks. In convolutional layers, it operates on a local spatial window across filter channels, and in fully connected layers it operates on a slice of a hidden state vector. Additionally, DN* has a L1 regularizer on the pre-normalization centered activation (v n,j ).

BN-s and BN*::

To compare with DN and DN*, we also propose modifications to original BN: we denote BN-s with σ 2 in the denominator's square root, and BN* with the L1 regularizer on top of BN-s.

LN-s and LN*::

We apply the same changes as from BN to BN-s and BN*. In order to narrow the differences in the normalization schemes down to a few parameter choices, we additionally remove the affine transformation parameters γ and β from LN such that the difference between LN* and DN* is only the size of the normalization field. γ and β can really be seen as a separate layer and in practice we find that they do not improve the performance in the presence of σ 2 .

4. EXPERIMENTS:

We evaluate the normalization schemes on three different tasks:

• CNN image classification: We apply different normalizations on CNNs trained on the CIFAR-10/100 datasets for image recognition, each of which contains 50,000 training images and 10,000 test images. Each image is of size 32 × 32 × 3 and has been labeled an object class out of 10 or 100 total number of classes.

• RNN language modeling: We apply different normalizations on RNNs trained on the Penn Treebank dataset for language modeling, containing 42,068 training sentences, 3,370 validation sentences, and 3,761 test sentences.

• CNN image super-resolution: We train a CNN on low resolution images and learn cascades of non-linear filters to smooth the upsampled images. We report performance of trained CNN on the standard Set 14 and Berkeley 200 dataset.

For each model, we perform a grid search of three or four choices of each hyperparameter including the smoothing constant σ, and L1 regularization constant α, and learning rate on the validation set.

4.1. CIFAR EXPERIMENTS:

We used the standard CNN model provided in the Caffe library. The architecture is summarized in Table 2 . We apply normalization before each ReLU function. We implement DN as a convolutional operator, fixing the local window size to 5 × 5, 3 × 3, 3 × 3 for the three convolutional layers in all the CIFAR experiments.

We set the learning rate to 1e-3 and momentum 0.9 for all experiments. The learning rate schedule is set to {5K, 30K, 50K} for the baseline model and to {30K, 50K, 80K} for all other models. At every stage we multiply the learning rate by 0.1. Weights are randomly initialized from a zero-mean normal distribution with standard deviation {1e-4, 1e-2, 1e-2} for the convolutional layers, and {1e-1, 1e-1} for fully connected layers. Input images are centered on the dataset image mean.

Table 3 summarizes the test performances of BN*, LN* and DN*, compared to the performance of a few baseline models and the standard batch and layer normalizations. We also add standard regularizers to the baseline model: L2 weight decay (WD) and dropout. Adding the smoothing constant and L1 regularization consistently improves the classification performance, especially for ResNet Experiments. Residual networks (ResNet) (He et al., 2016) , a type of CNN with residual connections between layers, achieve impressive performance on many image classification benchmarks. The original architecture uses BN by default. If we remove BN, the architecture is very difficult to train or converges to a poor solution. We first reproduced the original BN ResNet-32, obtaining 92.6% accuracy on CIFAR-10, and 69.8% on CIFAR-100. Our best DN model achieves 91.3% and 66.6%, respectively. While this performance is lower than the original BN-ResNet, there is certainly room to improve as we have not performed any hyperparameter optimization. Importantly, the beneficial effects of sigma (2.5% gain on CIFAR-100) and the L1 regularizer (0.5%) are still found, even in the presence of other regularization techniques such as data augmentation and weight decay in the training.

Since the number of sigma hyperparameters scales with the number of layers, we found that setting sigma as a learnable parameter for each layer helps the performance (1.3% gain on CIFAR-100). Note that training this parameter is not possible in the formulation by Jarrett et al. (2009) . The learned sigma shows a clear trend: it tends to decrease with depth, and in the last convolution layer it approaches 0 (see Fig. 3 ).

4.2. RNN EXPERIMENTS:

To apply divisive normalization in fully connected layers of RNNs, we consider a local neighborhood in the hidden state vector h j-R:j+R , where R is the radius 

v j = z j - 1 2R + 1 R r=-R z j+r (6) zj = v j σ 2 + 1 2R+1 R r=-R v 2 j+r (7)

We follow Cooijmans et al. (2016) 's batch normalization implementation for RNNs: normalizers are separate for input transformation and hidden transformation. Let BN (•), LN (•), DN (•) be BatchNorm, LayerNorm and DivNorm, and g be either tanh or ReLU.

EQUATION

EQUATION

EQUATION

Note that in recurrent BN, the additional parameters γ and β are shared across timesteps whereas the moving averages of batch statistics are not shared. For the LSTM version, we followed the released implementation from the authors of layer normalization1 , and apply LN at the same places as BN and BN*, which is after the linear transformation of W x x and W h h individually. For LN* and DN, we modified the places of normalization to be at each non-linearity, instead of jointly with a concatenated vector for different non-linearity. We found that this modification improves the performance and makes the formulation clearer since normalization is always a combined operation with the activation function. We include details of the LSTM implementation in the Appendix.

The RNN model is provided by the Tensorflow library (Abadi et al., 2016) and the LSTM version was originally proposed in Zaremba et al. (2014) . We used a two-layer stack-RNN of size 400 (vanilla RNN) or 200 (LSTM). R is set to 60 (vanilla RNN) and 30 (LSTM). We tried both tanh and ReLU as the activation function for the vanilla RNN. For unnormalized baselines and BN+ReLU, the initial learning rate is set to 0.1 and decays by half every epoch, starting at the 5th epoch for a maximum of 13 epochs. For the other normalized models, the initial learning rate is set to 1.0 while the schedule is kept the same. Standard stochastic gradient descent is used in all RNN experiments, with gradient clipping at 5.0.

Table 4 shows the test set perplexity for LSTM models and vanilla models. Perplexity is defined as ppl = exp(x log p(x)). We find that BN and LN alone do not improve the final performance relative to the baseline, but similar to what we see in the CNN experiments, our modified versions BN* and LN* show significant improvements. BN* on RNN is outperformed by both LN* and DN. By applying our normalization, we can improve the vanilla RNN perplexity by 20%, comparable to an LSTM baseline with the same hidden dimension. We also evaluate DN on the low-level computer vision problem of single image super-resolution.

We adopt the SRCNN model of Dong et al. (2016) as the baseline which consists of 3 convolutional layers and 2 ReLUs. From bottom to top layers, the sizes of the filters are 9, 5, and 52 . The number of filters are 64, 32, and 1, respectively. All the filters are initialized with zero-mean Gaussian and standard deviation 1e-3. Then we respectively apply batch normalization (BN) and our divisive normalization with L1 regularization (DN*) to the convolutional feature maps before ReLUs. We construct the training set in a similar manner as Dong et al. (2016) by randomly cropping 5 million patches (size 33 × 33) from a subset of the ImageNet dataset of Deng et al. (2009) . We only train our model for 4 million iterations which is less than the one adopted by SRCNN, i.e., 15 million, as the gain of PSNR and SSIM by spending that long time is marginal.

We report the average test results, utilizing the standard metrics PSNR and SSIM (Wang et al., 2004) , on two standard test datasets Set14 (Zeyde et al., 2010) and BSD200 (Martin et al., 2001) . We compare with two state-of-the-art single image super-resolution methods, A+ (Timofte et al., 2013) and SRCNN (Dong et al., 2016) . All measures are computed on the Y channel of YCbCr color space. We also provide a visual comparison in Fig. 4 .

As show in Tables 5 and 6 DN* outperforms the strong competitor SRCNN, while BN does not perform well on this task. The reason may be that BN applies the same statistics to all patches of one image which causes some overall intensity shift (see Figs. 4 ). From the visual comparisons, we can see that our method not only enhances the resolution but also removes artifacts, e.g., the ringing effect in Fig. 4 .

4.4. ABLATION STUDIES AND DISCUSSION:

Finally, we investigated the differential effects of the σ 2 term and the L1 regularizer on the performance. We ran ablation studies on CIFAR-10/100 as well as PTB experiments. The results are listed in Table 7 .

We find that adding the smoothing term σ 2 and the L1 regularization consistently increases the performance of the models. In the convolutional networks, we find that L1 and σ both have similar effects on the performance. L1 seems to be slightly more important. In recurrent networks, σ 2 has a much more dramatic effect on the performance than the L1 regularizer. There are several factors that could explain the improvement in performance. As mentioned above, adding the L1 regularizer on the activations encourages the filter responses to be less correlated. This can increase the robustness of the variance estimate in the normalizer and lead to an improved scaling of the responses to a good regime. Furthermore, adding the smoother to the denominator in the normalizer can be seen as implicitly injecting zero mean noise on the activations. While noise injection would not change the mean, it does add a term to the variance of the data, which is represented by σ 2 . This term also makes the normalization equation invertible. While dividing by the standard deviation decreases the degrees of freedom in the data, the smoothed normalization equation is fully information preserving. Finally, DN type operations have been shown to decrease the redundancy of filter responses to natural images and sound (Schwartz & Simoncelli, 2001; Sinz & Bethge, 2008; Lyu & Simoncelli, 2008) . In combination with the L1 regularizer this could lead to a more independent representation of the data and thereby increase the performance of the network.

5. CONCLUSIONS:

We have proposed a unified view of normalization techniques which contains batch and layer normalization as special cases. We have shown that when combined with a sparse regularizer on the activations, our framework has significant benefits over standard normalization techniques. We have demonstrated this in the context of both convolutional neural nets as well as recurrent neural networks. In the future we plan to explore other regularization techniques such as group sparsity. We also plan to conduct a more in-depth analysis of the effects of normalization on the correlations of the learned representations. We plot the effect of σ and L1 regularization on the validation performance in Figure 6 . While sigma makes the most contributions to the improvement, L1 also provides much gain for the original version of LN and BN. In LSTM experiments, we found that have an individual normalizer for each non-linearity (sigmoid and tanh) helps the performance for both LN and DN. Eq. 12-14 are the standard LSTM equations, and let N be the normalizer function, our new normalizer is replacing the nonlinearity with Eq. 15-16. This modification can also be thought as combining normalization and activation as a single activation function. 

:

https://github.com/ryankiros/layer-norm

We use the setting of the best model out of all three SRCNN candidates.

Figures and Tables:

Figure 1: Illustration of different normalization schemes, in a CNN. Each H × W -sized feature map is depicted as a rectangle; overlays depict instances in the set of C filters; and two examples from a mini-batch of size N are shown, one above the other. The colors show the summation/suppression fields of each scheme.

Fig. 1 shows a visualization of the normalization field in a 4-D ConvNet tensor setting. Divisive normalization happens within a local spatial window of neurons across filter channels. Here we set d(•, •) to be the spatial L ∞ distance.

Figure 2: Divisive normalization followed by ReLU can be viewed as a new activation function. Left: Effect of varying σ in this activation function. Right: Two units affect each other's activation in the DN+ReLU formulation.

Figure 3: Input scale (|x|) vs. learned σ at each layer, color coded by the layer number in ResNet-32, trained on CIFAR-10 (left), and CIFAR-100 (right).

Fig.5plots randomly sampled pairwise pre-normalization responses (after the linear transform) in the first layer at the same spatial location of the feature map, along with the average pair-wise

Figure 5: First layer CNN pre-normalized activation joint histogram

Figure 6: Validation accuracy on CIFAR-10/100 showing effect of sigma constant (a, b) and L1 regularization (c, d) on BN, LN, and DN

CIFAR CNN specification

Table content: <table><tr><td>Type</td><td>Size</td><td>Kernel</td><td>Stride</td></tr><tr><td>input</td><td>32 × 32 × 3</td><td>-</td><td>-</td></tr><tr><td>conv +relu</td><td colspan="2">32 × 32 × 32 5 × 5 × 3 × 32</td><td>1</td></tr><tr><td>max pool</td><td colspan="2">16 × 16 × 32 3 × 3</td><td>2</td></tr><tr><td>conv +relu</td><td colspan="3">16 × 16 × 32 5 × 5 × 32 × 32 1</td></tr><tr><td>avg pool</td><td>8 × 8 × 32</td><td>3 × 3</td><td>2</td></tr><tr><td>conv +relu</td><td>8 × 8 × 64</td><td colspan="2">5 × 5 × 32 × 64 1</td></tr><tr><td>avg pool</td><td>4 × 4 × 64</td><td>3 × 3</td><td>2</td></tr><tr><td colspan="2">fully conn. linear 64</td><td>-</td><td>-</td></tr><tr><td colspan="2">fully conn. linear 10 or 100</td><td>-</td><td>-</td></tr></table>

CIFAR-10/100 experiments

Table content: <table><tr><td>Model</td><td colspan="2">CIFAR-10 Acc. CIFAR-100 Acc.</td></tr><tr><td>Baseline</td><td>0.7565</td><td>0.4409</td></tr><tr><td colspan="2">Baseline +WD +Dropout 0.7795</td><td>0.4179</td></tr><tr><td>BN</td><td>0.7807</td><td>0.4814</td></tr><tr><td>LN</td><td>0.7211</td><td>0.4249</td></tr><tr><td>BN*</td><td>0.8179</td><td>0.5156</td></tr><tr><td>LN*</td><td>0.8091</td><td>0.4957</td></tr><tr><td>DN*</td><td>0.8122</td><td>0.5066</td></tr></table>

PTB Word-level language modeling experiments

Table content: <table><tr><td>Model</td><td>LSTM</td><td colspan="2">TanH RNN ReLU RNN</td></tr><tr><td colspan="3">Baseline 115.720 149.357</td><td>147.630</td></tr><tr><td>BN</td><td colspan="2">123.245 148.052</td><td>164.977</td></tr><tr><td>LN</td><td colspan="2">119.247 154.324</td><td>149.128</td></tr><tr><td>BN*</td><td colspan="2">116.920 129.155</td><td>138.947</td></tr><tr><td>LN*</td><td colspan="2">101.725 129.823</td><td>116.609</td></tr><tr><td>DN*</td><td colspan="2">102.238 123.652</td><td>117.868</td></tr><tr><td colspan="4">of the neighborhood. Although the hidden states are randomly initialized, this structure will impose</td></tr><tr><td colspan="2">local competition among the neighbors.</td><td/></tr></table>

Average test results of PSNR and SSIM on Set14 Dataset.

Table content: <table><tr><td>Model</td><td colspan="4">PSNR (x3) SSIM (x3) PSNR (x4) SSIM (x4)</td></tr><tr><td colspan="2">Bicubic 27.54</td><td>0.7733</td><td>26.01</td><td>0.7018</td></tr><tr><td>A+</td><td>29.13</td><td>0.8188</td><td>27.32</td><td>0.7491</td></tr><tr><td colspan="2">SRCNN 29.35</td><td>0.8212</td><td>27.53</td><td>0.7512</td></tr><tr><td>BN</td><td>22.31</td><td>0.7530</td><td>21.40</td><td>0.6851</td></tr><tr><td>DN*</td><td>29.38</td><td>0.8229</td><td>27.64</td><td>0.7562</td></tr></table>

Average test results of PSNR and SSIM on BSD200 Dataset.

Table content: <table><tr><td>Model</td><td colspan="4">PSNR (x3) SSIM (x3) PSNR (x4) SSIM (x4)</td></tr><tr><td colspan="2">Bicubic 27.19</td><td>0.7636</td><td>25.92</td><td>0.6952</td></tr><tr><td>A+</td><td>27.05</td><td>0.7945</td><td>25.51</td><td>0.7171</td></tr><tr><td colspan="2">SRCNN 28.42</td><td>0.8100</td><td>26.87</td><td>0.7378</td></tr><tr><td>BN</td><td>21.89</td><td>0.7553</td><td>21.53</td><td>0.6741</td></tr><tr><td>DN*</td><td>28.44</td><td>0.8110</td><td>26.96</td><td>0.7428</td></tr><tr><td colspan="3">4.3 SUPER RESOLUTION EXPERIMENTS</td><td/><td/></tr></table>

Comparison of standard batch and layer normalation (BN and LN) models, to those with only L1 regularizer (+L1), only the σ smoothing term (-s), and with both (*). We also compare divisive normalization with both (DN*), versus with only the smoothing term (DN).

Table content: <table><tr><td/><td>Model</td><td colspan="5">CIFAR-10 CIFAR-100 LSTM</td><td colspan="3">Tanh RNN ReLU RNN</td><td/></tr><tr><td/><td>Baseline</td><td colspan="2">0.7565</td><td>0.4409</td><td/><td colspan="2">115.720 149.357</td><td colspan="2">147.630</td><td/></tr><tr><td/><td colspan="3">Baseline +L1 0.7839</td><td>0.4517</td><td/><td colspan="2">111.885 143.965</td><td colspan="2">148.572</td><td/></tr><tr><td/><td>BN</td><td colspan="2">0.7807</td><td>0.4814</td><td/><td colspan="2">123.245 148.052</td><td colspan="2">164.977</td><td/></tr><tr><td/><td>BN +L1</td><td colspan="2">0.8067</td><td>0.5100</td><td/><td colspan="2">123.736 152.777</td><td colspan="2">166.658</td><td/></tr><tr><td/><td>BN-s</td><td colspan="2">0.8017</td><td>0.5005</td><td/><td colspan="2">123.243 131.719</td><td colspan="2">139.159</td><td/></tr><tr><td/><td>BN*</td><td colspan="2">0.8179</td><td>0.5156</td><td/><td colspan="2">116.920 129.155</td><td colspan="2">138.947</td><td/></tr><tr><td/><td>LN</td><td colspan="2">0.7211</td><td>0.4249</td><td/><td colspan="2">119.247 154.324</td><td colspan="2">149.128</td><td/></tr><tr><td/><td>LN +L1</td><td colspan="2">0.7994</td><td>0.4990</td><td/><td colspan="2">116.964 152.100</td><td colspan="2">147.937</td><td/></tr><tr><td/><td>LN-s</td><td colspan="2">0.8083</td><td>0.4863</td><td/><td colspan="2">102.492 133.812</td><td colspan="2">118.786</td><td/></tr><tr><td/><td>LN*</td><td colspan="2">0.8091</td><td>0.4957</td><td/><td colspan="2">101.725 129.823</td><td colspan="2">116.609</td><td/></tr><tr><td/><td>DN</td><td colspan="2">0.8058</td><td>0.4892</td><td/><td colspan="2">103.714 132.143</td><td colspan="2">118.789</td><td/></tr><tr><td/><td>DN*</td><td colspan="2">0.8122</td><td>0.5066</td><td/><td colspan="2">102.238 123.652</td><td colspan="2">117.868</td><td/></tr><tr><td>Baseline</td><td>BN</td><td>BN +L1</td><td>BN-S</td><td>BN*</td><td>LN</td><td>LN +L1</td><td>LN-S</td><td>LN*</td><td>DN</td><td>DN*</td></tr><tr><td>Corr. 0.19</td><td>Corr. 0.43</td><td>Corr. 0.17</td><td>Corr. 0.23</td><td>Corr. 0.17</td><td>Corr. 0.55</td><td>Corr. 0.17</td><td>Corr. 0.20</td><td>Corr. 0.16</td><td>Corr. 0.21</td><td>Corr. 0.20</td></tr><tr><td>MI 0.37</td><td>MI 1.20</td><td>MI 0.66</td><td>MI 0.80</td><td>MI 0.66</td><td>MI 1.41</td><td>MI 0.67</td><td>MI 0.74</td><td>MI 0.64</td><td>MI 0.81</td><td>MI 0.73</td></tr></table>

This is different from the released implementation of LN and BN in LSTM, which separately normalized the concatenated vector W h h t-1 and W x x t . For all LN* and DN experiments we choose this new formulation, whereas LN experiments are consistent with the released version.W h h t-1 + W x x t + b (12) c t = σ(f t ) c t-1 + σ(i t ) tanh(g t )(13) h t = σ(o t ) tanh(c t ) include results on another standard dataset Set5Bevilacqua et al. (2012) in Table8and show more visual results in Fig.7.

Table content: <table><tr><td></td><td>f t</td><td></td></tr><tr><td> </td><td>i t o t g t</td><td colspan="2">  = (14)</td></tr><tr><td/><td/><td>σ(x) = σ(N (x))</td><td>(15)</td></tr><tr><td/><td/><td>tanh(x) = tanh(N (x))</td><td>(16)</td></tr><tr><td colspan="3">C MORE RESULTS ON IMAGE SUPER-RESOLUTION</td></tr></table>

Peer review (iclr2017):
This paper empirically studies multiple combinations of various tricks to improve the performance of deep neural networks on various tasks. Authors investigate various combinations of normalization techniques together with additional regularizations. The paper makes few interesting empirical observations, such that the L1 regularizer on top of the activations is relatively useful for most of the tasks. In general, it seems that this work can be significantly improved by providing more precise study of existing normalization techniques. Also, studying more closely the overall volumes of the summation and suppression fields (e.g. how many samples one needs to collect for a robust enough normalization) would be useful. In more detail, the work seems to have the following issues: * Divisive normalization, is used extensively in Krizhevsky12 (LRN). It is almost exactly the same definition as in equation 1, however with slightly different constants. Therefore claiming that it is less explored is questionable. * It is not clear whether the Divisive normalization does subtract the mean from the activation as there is a contradiction in its definition in equation 1 and 3. This questions whether the *General Formulation of Normalization* is correct. * In seems that Divisive normalization is used also in Jarrett09, called Contrast Normalization, with a definition more similar to equation 3 (subtracting the mean). * In case of the RNN experiments, it would be more clear to provide the absolute size of the summation and suppression field as BN may be inferior to DN due to a small batch size. * It is unclear what and how are measured the results shown in Table 10. Also it is unclear what are the sizes of the suppression/summation fields for the CIFAR and Super Resolution experiments. Minor, relatively irrelevant issues: * It is usually better to pick a stronger baseline for the tasks. The selected CIFAR model from Caffe seems to be quite far from the state of the art on the CIFAR dataset. A stronger baseline (e.g. the widely available ResNet) would allow to see whether the proposed techniques are useful for the more recent models as well. * Double caption for Table 7/8.