Title: Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
Authors: Mengye Ren
Renjie Liao
Raquel Urtasun
Fabian H. Sinz
Richard S. Zemel
Venue: ICLR 2017

Abstract: Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work  well in the recurrent setting. In this paper we propose a unified view of  normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework  in the context of convolutional neural nets and recurrent neural networks, showing  improvements over baselines in image classification, language modeling as well as super-resolution.  


Review discussion:
- There does not seem to be consistently the best normalization method and the L1 regularization seems to be rather important. Do you have any intuition why BN* performs the best on CIFAR while DN* on the super-resolution task?
- Do you think that it would be possible to see similar effects on larger datasets?
Such as ILSVRC etc.
- What are the radii of the summation R_A and suppression fields R_B?
- Would it be possible to elaborate more about what is visualized in table 10? Is there some intuition why the BN and LN have almost diagonal structure, while additional L1 regularization removes that?
Q: There does not seem to be consistently the best normalization method
and the L1 regularization seems to be rather important. Do you have
any intuition why BN* performs the best on CIFAR while DN* on the
super-resolution task?
A: We think that the reason why DN* outperforms BN* on super-resolution
is that BN applies the same statistics to all patches of one
image which causes some overall intensity shift. DN, on the other
hand, use a per image statistic.
While this is useful for
super-resolution having a larger pool of data to average over might be
more advantageous in the case of CIFAR. DN* only averages over a
batchsize of one, although it uses all channels. In the case of CIFAR
the bigger batch size for BN* might just be more helpful.
Also, CIFAR
images are pretty small.
Therefore, the difference in the spatial
averaging range between BN* and DN* might not be relevant here.
Q: Do you think that it would be possible to see similar effects on
larger datasets?
Such as ILSVRC etc.
A: Yes, we expect that. Larger datasets with more classes and variety
require bigger networks, so regularization and normalization is just
as important there.
Q: What are the radii of the summation R_A and suppression fields R_B?
The radius of DN for CIFAR and RNN experiments are mentioned in the
paper. Cifar: Local window size 5x5, 3x3, 3x3 (for each conv layer),
and for RNN, 30 for LSTM and 60 for RNN.
Q: Would it be possible to elaborate more about what is visualized in
table 10? Is there some intuition why the BN and LN have almost
diagonal structure, while additional L1 regularization removes that?
A: We visualize the joint histogram of filter responses (the v's) for
4 pairs of filters to visualize correlations between those responses
on lower levels. We do not yet understand why the unregularized
normalization schemes can lead to such strong correlations. We do,
however, have an intuition why the L1 regularizer helps decrease them.
A possible explanation for this effect is that the L1 regularizer
might have a similar effect as maximum likelihood estimation of an
independent Laplace distribution. To see that, let $
p_v\left(\mathbf{v}\right) \propto\exp\left(-\left\Vert
\mathbf{v}\right\Vert _{1}\right) $ and $\mathbf{x}=W^{-1}\mathbf{v}$
with full rank invertible matrix $W$. Under this model
$p_x\left(\mathbf{x}\right) =p_v\left(W\mathbf x\right)\left|\det
W\right|$.
Then, minimization of the L1 norm of the activations under
the volume conserving constraint $\det A=\text{const.}$ corresponds to
maximum likelihood on that model, which would encourage decorrelated
responses. We do not enforce such a constraint, and the filter matrix
might even not be invertible.
However, the supervised loss function of
the network benefits from having diverse non-zero filters. This
encourages the network to not collapse filters on the same direction
or put them to zero, and might act as a relaxation of the volume
conserving constraint.
*** Paper Summary ***
This paper proposes a unified view on normalization. The framework encompases layer normalization, batch normalization and local contrast normalization. It also suggests decorrelating the inputs through L1 regularization of the activations. Results are reported on three tasks: CIFAR classification, PTB Language models and super resolution on Berkeley dataset.
*** Review Summary ***
Overall, I feel it is good to refresh the community about local normalization schemes and other mechanism to favor unit competitions. The paper reads well and reports results on various setups, with sufficient discussion.
*** Detailed Review ***
The paper is clear and reads well. It lacks a few reference to prior research.
Also I am surprised that "Local Contrast Normalization" is not said anywhere, as it is a common terminology in the neural network and vision literature.
It is unclear to me why you chose to pair L1 regularization of the activation and normalization. They seem complementary. Would it make sense to apply L1 regularization to the baseline to highlight it is helpful on its own.
Overall, it seems the only thing that brings a consistent improvement across all setups.
On related work, maybe it would be worthwhile to insist that Local Contrast Normalization (LCN) used to be very popular [Pinto et al, 2008, Jarret et al 2009, Sermanet et al 2012; Quoc Le 2013] and effective. It is great to connect this litterature to current work on layer normalization and batch normalization.
Similarly, sparsity or group sparsity of the activation has shown effective in the past [Rozell et al 08, Kavukcuoglu et al 09] and need more exposure today.
Finally, since dropout is so popular but interact poorly with normalizer estimates, I feel it would be worthwhile to report results with dropout beyond the baseline and discuss how the different normalization scheme interact with it.
Overall, I feel it is good to refresh the community about local normalization schemes and other mechanism to favor unit competitions. The paper reads well and reports results on various setups, with sufficent discussion.
*** References ***
Jarrett, Kevin, Koray Kavukcuoglu, and Yann Lecun. "What is the best multi-stage architecture for object recognition?." 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009.
Pinto, N., Cox, D., DiCarlo, J.: Why is real-world visual object recognition hard?
PLoS Comput Biol 4 (2008)
Le, Quoc V. "Building high-level features using large scale unsupervised learning." 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013.
P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house
numbers digit classification. In ICPR, 2012.
C. Rozell, D. Johnson, and B. Olshausen. Sparse coding via thresholding and local competition in neural circuits.Neural Computation, 2008.
K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y. LeCun. Learning invariant features through topographic filter maps. In CVPR, 2009.
Q: "Local Contrast Normalization" is not said anywhere, as it is a common
terminology in the neural network and vision literature.
A: Please see our response to AnonReviewer2 for a comparison to Local Contrast
Normalization. We will reference local contrast normalization papers in an
updated version of the paper.
Q: It is unclear to me why you chose to pair L1 regularization of the
activation and normalization. They seem complementary. Would it make sense to
apply L1 regularization to the baseline to highlight it is helpful on its own.
Overall, it seems the only thing that brings a consistent improvement across
all setups.
A: We did add L1 to the baseline in our paper submission, see CIFAR-10/100
table results. Adding L1 without mean centering does not seem to be a sensible
choice. We also tried to add L1 to mean centered models (without having the
denominator). They consistently underperform DN* models (1% decrease on
CIFAR-10 and 3% decrease on CIFAR-100).
We also added a potential explanation why L1 regularization helps and what our
motivation was to consider it in an updated version of the paper (also see our
response below).
Q: On related work, maybe it would be worthwhile to insist that Local Contrast
Normalization (LCN) used to be very popular [Pinto et al, 2008, Jarrett et al
2009, Sermanet et al 2012; Quoc Le 2013] and effective. It is great to connect
this literature to current work on layer normalization and batch
normalization.
Similarly, sparsity or group sparsity of the activation has
shown effective in the past [Rozell et al 08, Kavukcuoglu et al 09] and need
more exposure today.
A: Thanks for the suggestions. We added them to an updated version of the
paper.
AnonReviewer4: Finally, since dropout is so popular but interact poorly with
normalizer estimates, I feel it would be worthwhile to report results with
dropout beyond the baseline and discuss how the different normalization scheme
interact with it.
We tried dropout in combination with normalization schemes. It did not seem to
yield any improvement in performance and therefore we abandoned it later. The
normalization was only on convolutional layers, whereas dropout is applied on
fully connected layers. It may be that dropout and normalization do not have
too much interaction since they are applied at difference places.
This paper empirically studies multiple combinations of various tricks to improve the performance of deep neural networks on various tasks. Authors investigate various combinations of normalization techniques together with additional regularizations.
The paper makes few interesting empirical observations, such that the L1 regularizer on top of the activations is relatively useful for most of the tasks.
In general, it seems that this work can be significantly improved by providing more precise study of existing normalization techniques.
Also, studying more closely the overall volumes of the summation and suppression fields (e.g. how many samples one needs to collect for a robust enough normalization) would be useful.
In more detail, the work seems to have the following issues:
* Divisive normalization, is used extensively in Krizhevsky12 (LRN). It is almost exactly the same definition as in equation 1, however with slightly different constants.
Therefore claiming that it is less explored is questionable.
* It is not clear whether the Divisive normalization does subtract the mean from the activation as there is a contradiction in its definition in equation 1 and 3. This questions whether the "General Formulation of Normalization" is correct.
* In seems that Divisive normalization is used also in Jarrett09, called Contrast Normalization, with a definition more similar to equation 3 (subtracting the mean).
* In case of the RNN experiments, it would be more clear to provide the absolute size of the summation and suppression field as BN may be inferior to DN due to a small batch size.
* It is unclear what and how are measured the results shown in Table 10.
Also it is unclear what are the sizes of the suppression/summation fields for the CIFAR and Super Resolution experiments.
Minor, relatively irrelevant issues:
* It is usually better to pick a stronger baseline for the tasks. The selected CIFAR model from Caffe seems to be quite far from the state of the art on the CIFAR dataset. A stronger baseline (e.g. the widely available ResNet) would allow to see whether the proposed techniques are useful for the more recent models as well.
* Double caption for Table 7/8.
Q: Divisive normalization, is used extensively in Krizhevsky12 (LRN). The
claim that DN is less explored is questionable.
A: Please see our general response above.
In brief, the difference to
Krizhevsky is not just constants but the spatial dimension we sum over.
Furthermore, Krizhevsky et al. use normalization for one particular task while
we systematically explore it on several benchmarks.
Q: It is not clear whether the Divisive normalization does subtract the mean
from the activation as there is a contradiction in its definition in equation
1 and 3.
A: Equation (1) is the divisive normalization model as used on the
neuroscience literature. Equation (3) is our divisive normalization model. We
do subtract the mean in all cases. The reason is that divisive normalization
can be seen as a non-linear radial transformation which has strong links to
redundancy reduction and density estimation on natural images and sound (see
references Schwartz/Simoncelli, Lyu/Simoncelli, Sinz/Bethge, Balle/Simoncelli).
However, these links rely on zero mean data (or elementw-ise transformations
thereof).
Therefore, even the neuroscience version often implicitly assumes
mean centered data.
In practice, we found that mean subtraction is important
as it helps accelerate the training. We also tried a variant with only mean-
subtraction (on BN, LN and DN) and found that the performance is consistently
lower than divisive normalization, (1% decrease on CIFAR-10, and 3% decrease
on CIFAR-100).
Q: This questions whether the "General Formulation of Normalization" is
correct.
A: Our model technically includes the case where the mean is not subtracted by
choosing A to be empty. As we show in Table 1, different choices of sigma, A,
and B recover BN, LN, or DN. One exception is the form used in Jarrett et al:
x |-> x/max(||x||, c), which is not a special case of our model.
However, DN
can be seen as a smooth, invertible approximation of it, since the function
values for ||x|| -> infty and ||x||->0 are the same and DN is bounded and
monotonically increasing (i.e. sigmoidal) in between.
Q: It seems that Divisive normalization is used also in Jarrett09, called
Contrast Normalization, with a definition more similar to equation 3
(subtracting the mean).
A: As we explain in our general points above, our DN differs in important
aspects from Jarrett et al. and Krizhevsky et al.
In brief, Krizhevsky et al.
is invertible, but does not use spatial summation and does not use mean
centering (like we do). Jarrett et al. use mean centering but is not generally
invertible. As we argue above, we think that these choices matter for
classification performance.
Q: In case of the RNN experiments, BN may be inferior to DN due to a small
batch size.
A: We experimented BN with larger batch size, which helped  BN achieve a
slightly better performance which was  still worse than DN models. Varying
batch size was not considered in our experiments as the original learning rate
schedule was tied to the number of epochs, and change batch size will end up
searching for the best learning rate schedule. In the new set of experiments,
we searched for the best learning rate schedule for BN with larger batch size.
We include a comparison here, but note that it may not be fair again since DN
results were based on smaller mini-batches.
For ReLU RNN models, we are able to increase the learning by 10 by having a
batch size of 120, which leads to the most improvement.
ReLU RNN (large batch=120)
BN large batch:       128
BN* large batch:      124
DN small batch:       118 (In the paper)
DN* small batch:      117 (In the paper)
Tanh RNN (large batch=120)
BN large batch:       131 (Adding L1 did not help)
DN small batch:       132 (In the paper)
DN* small batch:      123 (In the paper)
LSTM RNN (large batch=60)
BN* large batch:      116
DN* small batch:      102
Q: It is unclear what and how are measured the results shown in Table 10.
Also
it is unclear what are the sizes of the suppression/summation fields for the
CIFAR and Super Resolution experiments.
A: The plots in table 10 show representative 2d histograms of activations in
the first layer before normalization along with the average pairwise mutual
information (MI) and correlation. This is to show that BN increases the
dependencies between the activations on early layers and that the smoother and
the L1 regularizer can decrease those. This could be one of the reasons for
their better performance.
On CIFAR, the suppression/summation fields are 5x5, 3x3, 3x3 for each
convolutional layer.
On super resolution, the suppression field is 5x5 and and the suppression
field is 7x7 for the first two convolutional layers.
Q: It is usually better to pick a stronger baseline for the tasks (e.g. the
widely available ResNet).
A: Since we submitted the paper, we have been experimenting with ResNet-32 as
a much larger architecture, on CIFAR-10 and CIFAR-100. The original
architecture uses BN by default. If we remove BN, the architecture is very
difficult to train or converges to a poor solution. We first reproduced
original BN ResNet-32, with 92.6% on CIFAR-10, and 69.8% on CIFAR-100.
We then train DN using the same hyperparameters.
Importantly, the effect of
sigma (2.5% gain on CIFAR-100) and L1 regularizer (0.5% gain on CIFAR-100) is
still found, even in the presence of other regularization techniques such as
various data augmentation and weight decay in the training. This suggests that
both the smoother and L1 regularizer should be included when applying divisive
normalized models.
So far our best DN model  achieves 91.3% on CIFAR-10 and 66.6% on CIFAR-100.
While this performance is lower than the original ResNet, there is certainly
room to improve as we have not performed any hyperparameter optimization, and
instead have used the same hyperparameters of the original BN ResNet. These
were found via extensive tuning with BN, but other hyperparameter settings are
likely better for DN.
During training the DN network we made an interesting observation:  Since the
number of sigma hyperparameters scales with the number of layers, we found
that setting sigma as a learnable parameter for each layer helps the
performance (1.3% gain on CIFAR-100). Note, that training this parameter is
not possible in the formulation by Jarrett et al. The learned sigma shows a
clear trend: it tends to decrease with depth (see Fig. 1), and the sigma in the
last convolution layer is approaching to 0.
Reasons for that effect could be that the overall scale of the data changes
over layers. Since there seems to be an optimal range for sigma, that range
would shift as well.
Another reason could be that preserving the scale of the
data becomes less and less relevant towards deeper layers classification layer
and the non-invertible (sigma=0) solution is more helpful in the final
classification layer.
Fig 1. Link to view the image: http://imgur.com/a/8l7lz
Learned sigma in ResNet-32 on CIFAR-100 dataset.
We run more than 30 experiments with different random seeds on DN ResNet-32 on
both CIFAR-10 and CIFAR-100 to investigate the pattern of learned sigmas are
every layer.
In particular, we plot the relationship between sigma and input
scale (local standard deviations of activations). The points are color coded
by the layer number (from purple to yellow). We make several observations:
1) At lower layers (dark purple), sigma is usually between medium to large to
scale down the input values.
2) At higher layers (light yellow), sigma goes to zero (as activation scale
might not matter for classification problem).
3) In the middle, there is usually a reliable relationship between sigma and
the input scale, which maybe useful to make the parameter search more
efficient.
Fig 2. Link to view the images
http://imgur.com/a/muBFj
Learned Sigma vs. Input Scale on ResNet-32
The authors present a unified framework for various divisive normalization schemes, and then show that a somewhat novel version of normalization does somewhat better on several tasks than some mid-strength baselines.
Pros:
* It has seemed for a while that there are a bunch of different normalization methods out there, of varying importance in varying applications, so having a standardized framework for them all, and evaluating them carefully and systematically, is a very useful contribution.
* The paper is clearly written.
* From an architectural standpoint, the actual comparisons seem well motivated.   (For instance, I'm glad they tried DN* and BN* -- if they hadn't tried those, I would have wanted them too.)
Cons:
* I'm not really sure what the difference is between their new DN method and standard cross-channel local contrast normalization.  (Oh, actually -- looking at the other reviews, everyone else seems to have noticed this too.   I'll not beat a dead horse about this any further.)
* I'm nervous that the conclusions that they state might not hold on larger, stronger tasks, like ImageNet, and with larger deeper models.  I myself have found that while with smaller models on simpler tasks (e.g. Caltech 101), contrast normalization was really useful, that it became much less useful for larger architectures on larger tasks.
In fact, if I recall correctly, the original AlexNet model had a type of cross-unit normalization in it, but this was dispensed with in more recent models (I think after Zeiler and Fergus 2013) largely because it didn't contribute that much to performance but was somewhat expensive computationally.
Of course, batch normalization methods have definitely been shown to contribute performance on large problems with large models, but I think it would be really important to show the same with the DN methods here, before any definite conclusion could be reached.
Q: What the difference is between the new DN method and standard cross-channel
local contrast normalization.
A: The main difference between standard cross-channel contrast normalization
and DN is the smoother (sigma) in the denominator. Contrast normalization can
be recovered by setting sigma=0 and choosing the summation and suppression
fields appropriately.
However, as we show in Figure 5 in Appendix A, non-zero
values of sigma consistently yield better performances than sigma=0 (contrast
normalization).
Thus, the difference between local contrast normalization and
DN matters for the performance in practice.
Q: Conclusions might not hold on larger, stronger tasks, like ImageNet, and
with larger deeper models...
A: We are currently training VGG networks on ImageNet, and we will update our
response once we get results.
We tried VGG-16 on ImageNet using top-1 accuracy with center crop. We apply
normalization (BN and DN) on the first convolutional layer of each stage of
the same feature map size. The baseline without any normalization scheme
reaches 68.63%. The BN version reaches 69.19%, and the DN version also gets
69.24%. Due to time limit, we have not yet finished training the version with
an L1 sparse regularizer. DN performs better than our trained baseline network
and on par with BN. This shows that DN (which, in contrast to BN, normalizes
on just a single image) is still useful to improve the accuracy over baseline
even on larger problems.
However, we acknowledge that these numbers are still
below the public released VGG weights which has 71% top-1 accuracy. The
original VGG paper did not release the exact learning rate schedule so we
couldn’t fully reproduce the reported number within our time limit.
We thank the reviewers for their constructive comments. Before we address each
item individually, we would like to emphasize a few general points.
Our normalization scheme is the first to work well for both DNNs and RNNs.
Previously batch normalization (BN) was problematic for RNNs while layer
normalization (LN) did not perform well for DNNs. We demonstrate that our
normalization scheme performs on par or outperforms BN and LN on several
benchmark problems.
While our scheme is similar in spirit to the ones used by
Jarrett et al. or Krizhesvky et al., it differs in important aspects that we
believe are important in practice.
One important feature not present in current normalization schemes is the
smoother in the denominator which makes the normalization equation
invertible/information preserving (unlike Jarrett, like Krizhevsky). One
contribution of our paper is to show that the smoother boosts performance of
the normalization scheme (e.g. Figure 5 in Appendix A).
While we use an
invertible normalization equation of the form x/sqrt(||x||^2 + c), Jarrett et
al. use a clipping equation of the form x/max(||x||, c) which is only
invertible for very small signals (||x||<c).
We subtract the mean before normalizing (like Jarrett, unlike Krizhevsky). For
mean centered responses, divisive normalization is a non-linear radial
transformation that has strong links to redundancy reduction on natural
signals (see references by Schwartz/Wainwright/Simoncelli, Lyu/Simoncelli,
Sinz/Bethge). We show that the L1 regularizer in combination with
normalization improves performance for all normalization schemes. We give
potential explanations for why L1 regularization helps (e.g. Table 10).
Thus, while our normalization scheme uses existing elements, we empirically
demonstrate which elements are important in practice by means of a systematic
comparison of normalization schemes on a diverse set of problems. Our proposed
normalization scheme DN* yields better performance on several benchmark tasks,
has a wider application range than BN (i.e., RNN, super resolution), and is
easier to use in practice since it does not require to keep track of
statistics during training.
Individual responses will be posted as direct replies under the comments.

Meta-review:
role: PC
data: comment: On the one hand, the topic is considered important and the paper is technically correct. On the ohter hand, novelty and theoretical depth are a bit lacking. Overall, this is a borderline paper. 

Still, the Program Chairs recommend it for a poster presentation given the importance of the topic.

Decision: Accept (Poster)