Venue: ICLR 2017 conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: *** Paper Summary ***

This paper proposes a unified view on normalization. The framework encompases layer normalization, batch normalization and local contrast normalization. It also suggests decorrelating the inputs through L1 regularization of the activations. Results are reported on three tasks: CIFAR classification, PTB Language models and super resolution on Berkeley dataset.

*** Review Summary ***

Overall, I feel it is good to refresh the community about local normalization schemes and other mechanism to favor unit competitions. The paper reads well and reports results on various setups, with sufficient discussion. 

*** Detailed Review ***

The paper is clear and reads well. It lacks a few reference to prior research. Also I am surprised that "Local Contrast Normalization" is not said anywhere, as it is a common terminology in the neural network and vision literature. 

It is unclear to me why you chose to pair L1 regularization of the activation and normalization. They seem complementary. Would it make sense to apply L1 regularization to the baseline to highlight it is helpful on its own. Overall, it seems the only thing that brings a consistent improvement across all setups.

On related work, maybe it would be worthwhile to insist that Local Contrast Normalization (LCN) used to be very popular [Pinto et al, 2008, Jarret et al 2009, Sermanet et al 2012; Quoc Le 2013] and effective. It is great to connect this litterature to current work on layer normalization and batch normalization. Similarly, sparsity or group sparsity of the activation has shown effective in the past [Rozell et al 08, Kavukcuoglu et al 09] and need more exposure today.

Finally, since dropout is so popular but interact poorly with normalizer estimates, I feel it would be worthwhile to report results with dropout beyond the baseline and discuss how the different normalization scheme interact with it.

Overall, I feel it is good to refresh the community about local normalization schemes and other mechanism to favor unit competitions. The paper reads well and reports results on various setups, with sufficent discussion. 

*** References ***

Jarrett, Kevin, Koray Kavukcuoglu, and Yann Lecun. "What is the best multi-stage architecture for object recognition?." 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009.

Pinto, N., Cox, D., DiCarlo, J.: Why is real-world visual object recognition hard?
PLoS Comput Biol 4 (2008)

Le, Quoc V. "Building high-level features using large scale unsupervised learning." 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013.

P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house
numbers digit classification. In ICPR, 2012.

C. Rozell, D. Johnson, and B. Olshausen. Sparse coding via thresholding and local competition in neural circuits.Neural Computation, 2008.

K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y. LeCun. Learning invariant features through topographic filter maps. In CVPR, 2009.

review_title: Overall, I feel it is good to refresh the community about local normalization schemes and other mechanism to favor unit competitions. The paper reads well and reports results on various setups, with sufficient discussion. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper empirically studies multiple combinations of various tricks to improve the performance of deep neural networks on various tasks. Authors investigate various combinations of normalization techniques together with additional regularizations. 

The paper makes few interesting empirical observations, such that the L1 regularizer on top of the activations is relatively useful for most of the tasks. 

In general, it seems that this work can be significantly improved by providing more precise study of existing normalization techniques. Also, studying more closely the overall volumes of the summation and suppression fields (e.g. how many samples one needs to collect for a robust enough normalization) would be useful.

In more detail, the work seems to have the following issues:
* Divisive normalization, is used extensively in Krizhevsky12 (LRN). It is almost exactly the same definition as in equation 1, however with slightly different constants. Therefore claiming that it is less explored is questionable.
* It is not clear whether the Divisive normalization does subtract the mean from the activation as there is a contradiction in its definition in equation 1 and 3. This questions whether the "General Formulation of Normalization" is correct.
* In seems that Divisive normalization is used also in Jarrett09, called Contrast Normalization, with a definition more similar to equation 3 (subtracting the mean).
* In case of the RNN experiments, it would be more clear to provide the absolute size of the summation and suppression field as BN may be inferior to DN due to a small batch size.
* It is unclear what and how are measured the results shown in Table 10. Also it is unclear what are the sizes of the suppression/summation fields for the CIFAR and Super Resolution experiments.

Minor, relatively irrelevant issues:
* It is usually better to pick a stronger baseline for the tasks. The selected CIFAR model from Caffe seems to be quite far from the state of the art on the CIFAR dataset. A stronger baseline (e.g. the widely available ResNet) would allow to see whether the proposed techniques are useful for the more recent models as well.
* Double caption for Table 7/8.
review_title: Well written but with little novelty

final_score: rating: null
review_content: - There does not seem to be consistently the best normalization method and the L1 regularization seems to be rather important. Do you have any intuition why BN* performs the best on CIFAR while DN* on the super-resolution task?
- Do you think that it would be possible to see similar effects on larger datasets? Such as ILSVRC etc.
- What are the radii of the summation R_A and suppression fields R_B?
- Would it be possible to elaborate more about what is visualized in table 10? Is there some intuition why the BN and LN have almost diagonal structure, while additional L1 regularization removes that?
review_title: Pre-review Questions

final_score: rating: 7: Good paper, accept
review_content: The authors present a unified framework for various divisive normalization schemes, and then show that a somewhat novel version of normalization does somewhat better on several tasks than some mid-strength baselines. 

Pros:

* It has seemed for a while that there are a bunch of different normalization methods out there, of varying importance in varying applications, so having a standardized framework for them all, and evaluating them carefully and systematically, is a very useful contribution.
* The paper is clearly written.   
* From an architectural standpoint, the actual comparisons seem well motivated.   (For instance, I'm glad they tried DN* and BN* -- if they hadn't tried those, I would have wanted them too.) 

Cons:

* I'm not really sure what the difference is between their new DN method and standard cross-channel local contrast normalization.  (Oh, actually -- looking at the other reviews, everyone else seems to have noticed this too.   I'll not beat a dead horse about this any further.)

* I'm nervous that the conclusions that they state might not hold on larger, stronger tasks, like ImageNet, and with larger deeper models.  I myself have found that while with smaller models on simpler tasks (e.g. Caltech 101), contrast normalization was really useful, that it became much less useful for larger architectures on larger tasks.   In fact, if I recall correctly, the original AlexNet model had a type of cross-unit normalization in it, but this was dispensed with in more recent models (I think after Zeiler and Fergus 2013) largely because it didn't contribute that much to performance but was somewhat expensive computationally.  Of course, batch normalization methods have definitely been shown to contribute performance on large problems with large models, but I think it would be really important to show the same with the DN methods here, before any definite conclusion could be reached. 

review_title: Review of "NORMALIZING THE NORMALIZERS: COMPARING AND EXTENDING NETWORK NORMALIZATION SCHEMES"

Meta-review: On the one hand, the topic is considered important and the paper is technically correct. On the ohter hand, novelty and theoretical depth are a bit lacking. Overall, this is a borderline paper. 

Still, the Program Chairs recommend it for a poster presentation given the importance of the topic.
Decision: Accept (Poster)