Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies normalization strategies for deep residual networks, both theoretically and empirically, and finds the strategies that don't rely on batch-statistics to have particular failure modes, the two most notable ones being channel-wise collapse, where instances collapse to a constant in each channel, and reduction in instance-variability, where the first and second order statistics per channel are identical for each instance, thereby reducing expressivity and the ability to recognize higher order concepts. The paper then proceeds to propose proxy-normalization, a normalization method suitable for small-batch learning (i.e., not relying on batch-statistics) that, when combined with other normalization strategies, such as Layer Norm or Group Norm, alleviates these failure modes. On ImageNet and CIFAR experiments with deep residual networks (RN50 and larger), the work finds that Proxy-Normalization (with sufficient additional regularization such as label-smoothing, dropout, and stochastic depth), can be used to match the performance of BN without relying on batch statistics. The proposed method standardizes the activations of the network according to the mean and variance of a channel-wise gaussian random variable, with learnable mean and variance parameters.
main_review: ### Originality & Significance
- Improving residual normalization strategies to have no dependence on batch statistics is an important problem with great practical applicability. A broad body of work has been dedicated to this problem.
- The proposed Proxy-Normalization strategy looks novel to the best of my knowledge, but is somewhat related to existing normalization strategies based on proxy-statistics of the activations [1,2,3].
- The power decomposition is quite intriguing, and provides a nice grounding for subsequent arguments about normalization strategies and failure modes. Since such power quantities can be measured, it also makes it easy to directly verify these arguments with empirical observations, as the authors do in Figure 3. Their logic is as follows: Power concentration in P1 (at the dataset scale) means all instances have collapsed in that channel (equal to a constant). Collapsing channels result in a pseudo-linearity failure, whereby the network does not effectively utilize its depth. Lack of power concentration in P2 and P4 imply that all instance activations have the same exact mean and variance (no inter-instance variability). Lack of inter-instance variability (low power in P2 and P4) reduces expressivity of the network, making it more difficult to recognize higher order concepts.

### Clarity
The paper is well written and relatively clear, however some details are unclear.
- For example, from my understanding C_k in theorem 1 is the number of channels at k. Can you please clarify this point?
- The arguments relating inter-instance variability to expressivity is also not entirely clear and is a little hand-wavy.

### Quality
There are sufficient technical contributions in the paper, both empirically and theoretically. However, there are limitations
- The proposed Proxy Normalization method while conceptually simple, introduces some non-trivial complexity into training, in having to sample from a Gaussian random variable with learnable parameters, and then feed that into the activation before standardizing.
- The proposed proxy-normalization is demonstrated to address the channel collapse issue (large P1), but is not demonstrated to address the inter-instance variability issue. The concentration of power in the dataset scale (channel collapse) is more much more stark with LN and GN, but the improvement in inter-instance variability (expressivity), to be determined by P2+P4 moving away from 0, is not as significant in Figure 3, but I suppose that according to the provided arguments LN and GN should not really have this issue.
- The proposed method requires additional regularization factors to match BN performance, and from my parsing of the appendix, these factors seem to vary from one architecture to another. Furthermore, it is unclear which regularization strategies are necessary or why. Such an ablation and detailed understanding would significantly improve the paper.


[1] Arpit et al., Normalization propagation: A parametric technique for removing internal covariate shift in deep networks, 2016.

[2] Laurent et al., Recurrent normalization propagation, 2017.

[3] Shekhovtsov et al., Normalization of neural networks using analytic variance propagation, 2018.
limitations_and_societal_impact: No clear negative societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes some theoretical analyses on different normalization schemes in deep neural networks. It found that LN suffers from the collapsing effect over the number of layers, which makes the computation more and more linear, whereas IN suffers from a lack of expressivity. Towards fixing these issues, the paper proposed ProxyNorm that adds another normalization layer after the activation function. Empirical results suggest that the combination of ProxyNorm and LayerNorm or GroupNorm can lead to matching or sometimes superior performance compared to BatchNorm.
main_review: ------------------------------------------
### Strengths
- Insightful analysis on different failure modes of LN and IN.
- Proposed a working method that shows empirical improvement over standard LN and GN, with a matching performance to BN. This is a very exciting result.
- Overall, the experiment numbers seem solid.
- The experimental details and implementation details are included in the appendix.

------------------------------------------
### Weaknesses
- **More empirical studies on the collapsing phenomenon in LN and GN.** Currently the plot in Figure 3 is not very clear about the collapsing effect. $\mathcal{P}^{(1)}$ does grow, but it does not converge to 1, and the rest of the terms do not converge to 0, unlike suggested by the theorem. So I am wondering if the paper could improve the empirical studies to verify the collapsing effect. First, you don’t have to use a ResNet-50, but it could be a regular convolutional network with many more layers. Second, I would also like to see other metrics to measure the linearity of the layers empirically over the number of layers. E.g. the proportion of negative entries in the pre-activation.
- **Analysis depends on the use of ReLU activations:** Line 111 states that the activation needs to be positive homogeneous and non-zero, and based on this it is clear that we only have scaled or flipped ReLU type activations. There have also been other developments of activation functions such as SeLU that have self-normalizing effects. I wonder if this can be baked into the analysis and lead to other insights? Instead of adding another normalization layer after the activation function.
- **Reliance to be preceded by another normalization function:** First of all, I don’t quite understand the requirement stated in Line 257-258: “The only required constraint for PN to work optimally is that each activation step should be immediately preceded by a normalization step.” I expect to see more concrete justification for this requirement and the transition of going into LN+PN and GN+PN is too fast. Secondly, doing a double normalization makes the computation less elegant. The work would have more impact by just introducing one single normalization step.
- **Guarantees on LN+PN/GN+PN:** I might have entirely missed the point but I would appreciate it if there is a theorem or remark showing that LN+PN / GN+PN is able to avoid the collapsing effect.

------------------------------------------
### Clarification
- **Computing statistics analytically:** If we assume that we have a ReLU activation, can we analytically compute the mean and variance of the proxy distribution? It can simplify the modification to standard networks a lot, by simply introducing pre-computed affine transformations instead of collecting statistics from samples.
- **The reason for a proxy distribution:** It also was not clear to me what is the purpose for adding a proxy distribution. Why not just normalize the post-activations using the statistics from themselves?

------------------------------------------
### Conclusion
In conclusion, I found this is a nice paper for advancing our understanding of neural network normalizations. However, I found the proposed solution somehow less elegant, and there is some disconnect between the theoretical and the experimental sections. Some practical design choices are not well explained (e.g. the need for applying together with an existing normalization layer), and a lack of theoretical guarantees on the proposed LN+PN and GN+PN. Overall, I think the paper is above the bar for acceptance.

Update: The review addresses most of my concerns and I have raised the score from 6 to 7.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper analyzes the properties of normalization methods. It shows how layer normalization induces a collapse towards channel-wise constant functions with assumptions, and how the constraint imposed by normalization affects the performance of instance normalization. It also proposes Proxy Normalization that emulates batch normalization’s behavior and performance, when combined with batch-independent normalization (e.g, layer normalization or group normalization) .
main_review: **Pros:**

+This paper summarizes many properties of normalization methods, even though these properties of normalization have been separately shown in previous works.   
+The analysis shown in section 3 is novel, even though I have concerns on the claims.  
+The experiments of this paper are comprehensive.  


**Cons:**

-It is not rigorous to state that BN preserve the expressivity. Particularly, there are work [1*] showing that the batch size affects the layer representation/expressivity of BN, and further affects the model’s representation/expressivity. For example, using a batch size of 2 for a MLP, it is clear that activation is constraint to be 1 and -1 during training. Therefore, BN could change the expressivity, noting that the optimization of BN is over the mini-batch statistics, not the population statistics during training. 

-I do not recognize the practicality of Proxy Normalization:  
 (1) This paper should conduct experiments on small batch size scenario (e.g., batch size of 2 on ImageNet or object detection tasks on CoCo) to support its claims in Line-306 and 309. Based on the current experimental results, I do not find why one person should use PN+GN/LN, rather than BN, considering that GN/LN introduces extra computational cost during inference.   
(2) There are two extra hyper-parameters $\tilde{\beta}_c^l$ and  $\tilde{\gamma}_c^l$, How to determine them? Do the results sensitive to these hyper-parameters? This paper should conduct experiments to discuss the sensitivity of these two hyper-parameters over different architecture/dataset. 


-Some description and notation is not rigorous:   
(1) Line 140, why  $\tilde{y}^l$ is unlikely to have its channel-wise distributions collapsed?, if the $\beta$ is biased to a very large negative value, all the normalized output will less than 0, leading to all 0 output.   
(2) Line 117~118, what is $E_\mathbf{x}$, a, does this mean the expectation over the mini-batch dimension? If yes, why denote $\mathbf{x} \in R^{H \times W \times C_l}$? 


**Ref:**

[1*] Group Whitening: Balancing Learning Efficiency and Representational Capacity, arxiv: 2009.13333



**Update**: I have read the authors responses and other reviewer’s comments, The authors mostly addressed my concerns relating to the clarity. I still has the concerns on the practicality, considering that LN/GN is required (For their scale invariant to stabilize training) in the solution, while LN/GN introduces extra computational cost during inference. 
I do recognize the analysis shown in section 3, which is novel to me. I thus keep positive to this paper and keep my score as 6.

limitations_and_societal_impact: yes
review_title: null

Meta-review: The paper proposes a theoretical analysis on different normalization schemes. It demonstrates that normalizing schemes which don't rely on batch-statistics suffer from channel collapse or lack or expressivity.  It then proposes a novel normalization scheme to address those issues.

Although the practicality of the algorithm could be improved, proxy-norm is able to match BN while not using the batch statistics. Additionally, the analysis presented in the paper is novel and reviewers agreed that it would be of value to the community.

Decision: Accept (Poster)