Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper conducts an empirical analysis of the interaction between batch normalization and bounded activation functions. Specifically, the paper compares the architecture using batch normalization after a bounded activation(Swap model) and the architecture using a bounded activation after batch normalization(Convention model). Motivated by the observation that the swap model outperforms the convention model significantly when a bounded activation is used, the authors designed experiments to identify the reasons for these performance differences. The paper shows that in terms of asymmetric saturation, the Swap model and the Convention model behave differently and argues that high sparsity induced from the asymmetric saturation has a strong association with the generalization performance.
strength_and_weaknesses: ### Strengths
- The authors discover that in the Swap model with a bounded activation, each feature map is saturated on one side of the asymptotic value of the bounded activation.


### Weaknesses
- It is a bit confusing whether the asymmetric saturation has a strong association with generalization performance.
    - Although there is another noticeable observation that the saturation is very low in higher block depths, this is not discussed at all.
    - Can you explain more how to exclude the possibility that low saturation at higher blocks or the combination of both could be a reason for better generalization?
- It is also confusing whether the sparsity has a strong association with generalization performance
    - Since the sparsity metric is $s^l = 1 - t^l$ where $t_l$ is the saturation metric, layerwise sparsity can be obtained from Figure 3. The relation between the sparsity of the Swap model and the Convention model is different depending on which layer is considered. In such case, it seems a bit of a stretch to draw a conclusion that the higher the sparsity is the better the generalization is. 
    - In a sense, this contradicts with the authors' argument 'Our saturation metric can dismiss the channel properties due to the summarization of channels in the layer.'
    - Can we say that different sparsity distributions over layers with the same average sparsity will have similar generalization performance?
- The coverage of the analysis is a bit limited. 
    - The analysis is claimed to be valid with bounded nonlinearity and without residual connection, excluding many widely used architectures. Also, it seems difficult to generalize or apply the claim of the paper to commonly used cases.
    - Even though it is subjective, it does not seem that the Swap model with Tanh performs comparably to the Convention model with ReLU. 

### Questions
- Can you elaborate on ' Because Tanh has non-linearity in everyplace except the origin, it can not follow the design of residual connection proposed'?
    - Does that mean that Tanh has gradient 1 at the origin? What does it mean by nonlinear in everyplace?
    - What does it mean by 'following the design of residual connection'?
- In Table 1, with ReLu, the Convention model is better than the Swap model. Have you considered or performed a similar analysis to understand this reversed behavior?
- What is 'the center of the function'? This term is not defined precisely. The center of the domain of the function or the center of the image of the function?
- NWDBN is not explained until Figure 8 and is frequently used before Figure 8. Even though I guess that NWDBN may stand for No Weight Decay Batch Normalization, acronyms should be explained when it is first used.
- What is the formula of LeCun Tanh? There are many typos for LeCun Tanh.
clarity,_quality,_novelty_and_reproducibility: Even though the observation that the Swap model with a bounded activation is interesting, the arguments that connect the observation and other experiments to the conclusions are not convincing. It seems that the training details are well-provided enough to enable reproducibility.
summary_of_the_review: It is interesting to know that with a bounded activation, the order between BN and the activation causes drastically different qualitative behavior. However, the presentation of the idea can be improved further by replacing vaguely defined terms and expressions. The arguments supporting the conclusions seem weak.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper reports that when a bounded activation function is used, the Swap order of network block "conv/fc+act+bn" achieves performance better than the Convention order of block "conv/fc+bn+act". The authors find that the bounded activation function accompanies high saturation in each network block and asymmetry saturation in each channel within the block. The asymmetry saturation with batch normalization can induce high sparsity, which assists the generalization performance. Three metrics were proposed to measure saturation, asymmetry saturation (skewness), and sparsity respectively. Experiments show that when several bounded activation functions are used, Swap order performs better than Conventional order in the classification tasks, but still cannot outperform ReLU-based networks.
strength_and_weaknesses: Strength:
- This paper investigates an interesting topic: the order of batch normalization(bn) and activation function(act), and gives a partial answer: placing bn after act gives better test performance when bounded activation functions were used.
- The logic chain is quite clear and easy to understand: this paper first reports the improvement of the Swap order, then analyzes the order from the point of saturation and asymmetric saturation, and found that the bn after asymmetric saturation can induce sparsity, which may be the reason for improvement. 
- The finding of this paper is interesting. For a bounded act, the saturation is asymmetric, and it may assist the performance.

Weaknesses:
- The saturation metric seems not complete enough. Consider that if the maximum absolute value of G^l is small, which means that the values of G^l are centered around 0, the metric would be close to 1 also. 
- Fig.3 and fig.4 only show high saturation and high skewness in shallow blocks, in deep blocks things seem changed. What does that mean? Is it means that in the deep block, we should use Convention order?
- Although the finding is interesting, the conclusion seems less practical. The performance of the Swap order with bounded acts cannot outperform the Convention order with the ReLU act. We still can use ReLU directly.
- Besides, (this may be a overclaim) since the authors report a counterintuitive example that the Swap order, which achieves better performance, accompanies saturation,  the author had better discuss it because intuitively the bounded act can induce gradient vanishment problems.
clarity,_quality,_novelty_and_reproducibility: Quality: borderline
Clarity: adequate
summary_of_the_review: This paper discusses an interesting topic, but the analysis seems less comprehensive. I would like to rate it as borderline reject. If the author can clarify my problems, I am willing to consider changing my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: It is shown that putting batch normalization after activation function enhance optimization with SGD. Then authors experimentally compares the distribution of hidden representation for two different locations of batchnormalization. In particular, paper investigated the sparsity, and asymmetric saturation. A stark contrast is reported for asymmetric saturation when BN is placed before and after activation. 
strength_and_weaknesses: **Strength** 
It is surprising that a simple swap of batch normalization and activation can enhance the performance of neural nets for bounded activations. 

**Weakness** 

- I am not convinced about the argument that asymmetric saturation enhances generalization. Are authors sure that the enhanced performance is not due to accelerated training? Is it possible to compare the convergence for two different neural architectures?
- For asymmetric saturation, is it possible to create saturation for conventional architectures; thereby enhancing their performance? 
- How do stepsize, batch size, network width, depth, channel size affect the observation? 
- Would be nice to analytically derive the reason for asymmetric saturation at initialization to make sure it is independent of the dataset.
- Is the swap only useful for image classification and convolution nets. Is the swap beneficial for MLPs as well? 
- Does the swap for tanh activation outperform ReLU with conventional BN location?  
clarity,_quality,_novelty_and_reproducibility: Paper is well written. But, I think the approach is not very novel and the paper is thin on technical levels.
summary_of_the_review: The paper reports an interesting observation in learning neural networks. However, this observation requires further justifications and investigations. 

Thanks for the rebuttal response!
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates how the order of batch normalization (BN) placed in the network affects the performance, when using the unbounded activation functions (e.g., Tanh). It shows that the Swap model (BN after the nonlinearity) using unbounded activation functions has significantly better performance than the conventual model (BN before the nonlinearity) using the same unbounded activation functions. It further shows activation values of each channel of Swap model is asymmetrically saturated, by looking into the output of channel-activations. It claims this asymmetrically saturated activations of Swap model increases the sparsity, which improves the performance of the model. 
strength_and_weaknesses: 
**Strengths:**

1. normalization and nonlinear (activation) layers are basic layer/module in DNNs, and it is a good plus to investigate their interaction. This paper investigates How BN interact with bounded activation and shows several interesting observations，e,g., the activation values
of each channel of Swap model is asymmetrically saturated. 

2. The view in investigating the correlation among saturation (especially the channel-wise saturation), sparsity and performance is new to me, and this paper also provides quantitative metric to evaluate the saturation and sparsity. 



**Weaknesses:** 

**1. One big concern for me is that some claims are not clearly clarified or rigorous.**

(1) In Introduction section, I donot understand this claim “In contrast, the one-sided property of asymmetric saturation causes at least half of the sample values after normalization to be almost zero, allowing the Swap model to have even higher sparsity than the Convention model”.  I find the only support is from Section 5.1. Indeed I donot understand why “The asymptotic values of combined Tanh with normalization operation are $\frac{+1-\hat{\mu}}{\hat{\sigma}}$ and $\frac{+1+\hat{\mu}}{\hat{\sigma}}$.”?  Can this paper provide further clarification? Besides, I understand “$\hat{\mu}$ becomes around -1 or 1 value”, but why $\hat{\sigma}$ is calculated as an appropriate size to produce a high skewness”? and what is a appropriate size?

(2) In Section 3.2, “This is counterintuitive as excessive saturation is considered an undesirable situation in the previous works” is somewhat misleading. I believe the experiments are based on the model at the end of training. I believe the “excessive saturation is considered an undesirable situation in the previous works” is for the initial training stages (the model has not learned information from the datasets), but it is not undesirable in the end of the training. Indeed, a good model with high confidence to the prediction may have excessive saturation neurons. 

(3) The statement in Section 3 “When training a neural network with bounded activation functions with a center of the function at the origin, the output increases due to the weight gradually increasing.” is not rigorous. Does sigmoid (bounded activation) has a center of the function at the origin? It is also has likelihood that the bounded activation is saturated in the initial training, which depends how the model is initialized. Furthermore, why the weight gradually increasing? I think it also depends on the how much the weight decay is used and (for a model with weight decay)? 

(4) It seems to be contradictory base on the statement “In short, higher saturation decreases sparsity” in Section 5.2 and the observation that Swap model has high saturation, but why the experimental results show that Swap model has higher sparsity? Is any wrong understanding of me? 

(5) The observations seem to be not uniformly hold over all layers of VGG, from Figure 3 and 4. Why the activations of Swap model after the 10 th layer has lower saturation and almost the same Skewness, compare to the Conventional model, in Figure 3 and 4? That is not consistent to the observations of Swap model before the 10th layer. 

**2. I also have concern on the experimental setup and analysis.**

(1) Considering this statement “Because Tanh has non-linearity in everyplace except the origin, it can not follow the design of residual connection proposed in He et al. (2016). Thus, we choose architectures where a skip connection does not exist.”, does this mean that the MobileNet used in this paper has no residual connection in its ResBlock? If yes, this paper only performs experiments on the VGG-like (feed forward neural network), but not for the ResNet-like network.  By the way, I donnot understand why residual connection is not allowed for Tanh? Does this design cannot well trained or the model cannot be conducted by design?

(2) In Figure 7, it is true that the increase in the models sparsity and accuracy are highly correlated. But I am wondering whether this results still hold by further increasing the weight decay on the affine parameters (the current maximum weight decay is $5e^{-4}$)? Intuitively, further increasing the weight decay on the affine parameters will further increase the sparsity, based on the statements shown in this paper. Does it further improve the accuracy of the network? What is the sparsity and accuracy, if we use a weight decay on the affine parameters of $1e^{-3}$, $1e^{-2}$, $1e^{-1}$?

(3) Even though Swap model using BN after bounded activation functions performances better than conventional model, one problem is that all the swap models using bounded activation functions shown in Table 1 has significantly lower accuracy compared to the conventional model using ReLU (the widely used unbounded activations). Based on this, I cannot well recognize the contributions of this paper, from the perspective of practice. It is better to provide the results that a swap model using BN+bounded activation has better performance than the conventional model using BN+ ReLU. 


(4) It is good that this paper submit the code, but when I check how this paper calculate the saturation, skewness and sparsity in the code, I cannot find it. Do the authors tell me in which line and which file, the saturation, skewness and sparsity are calculated?

clarity,_quality,_novelty_and_reproducibility: The clarity is somewhat unclear and I have concerns on the main claims, the quality and novelty is somewhat good. I believe the experiments can be somewhat reproduced based on the descriptions of this paper.
summary_of_the_review: This paper provides a somewhat new view in investigating BN’s position with different nonlinearity. The observations are interesting but with unclear clarification. I personally have main concerns and currently tend to negative for this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper points out an intersting fact that Swap order, i.e., putting batch normalization (BN) after bounded activation function (BAF) can achieve better performance than putting BN before BAF  in image classification. However, I am not fully convinced by the claim that there is a relation between the saturation and generalization. Reviewers also mentioned that the bounded activation function like tanh with Swap order can not outperform the unbounded activation function with convention order. The rebuttal did not address these concerns. For these reasons, I recommend reject.

justification_for_why_not_higher_score: I am not fully convinced by the claim that there is a relation between the saturation and generalization. Reviewers also mentioned that the bounded activation function like tanh with Swap order can not outperform the unbounded activation function with convention order.
Decision: Reject