Batch Normalization and Bounded Activation Functions
Authors: Dongjin Kim
Woojeong Kim
TaeJoo Park
Suhyun Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Batch Normalization
Activation Functions
Saturation
Sparsity

Abstract: Since Batch Normalization was proposed, it has been commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e., using Batch Normalization after activation functions, has also been attempted, but it is generally not much different from the conventional order when ReLU is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance on various benchmarks and architectures than the conventional order. We report this remarkable phenomenon and closely examine what contributes to this performance improvement in this paper. One noteworthy thing about swapped models is the extreme saturation of activation values, which is usually considered harmful. Looking at the output distribution of individual activation functions, we found that many of them are highly asymmetrically saturated. The experiments inducing a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance. In addition, we found that Batch Normalization after bounded activation functions has another important effect: it relocates the asymmetrically saturated output of activation functions near zero. This enables the swapped model to have higher sparsity, further improving performance. Extensive experiments with Tanh, LeLecun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation. 

Under review as a conference paper at ICLR 2023
BATCH NORMALIZATION AND BOUNDED ACTIVATION FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Since Batch Normalization was proposed. it has been commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e. using Batch Normalization after activation functions, has also been attempted, but it generally not much different from the conventional order when ReL.U is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance on various bench- marks and architectures than the conventional order. We report this remarkable phenomenon and closely examine what contributes to this performance improve- ment in this paper. One noteworthy thing about swapped models is the extreme saturation of activation values, which is usually considered harmful. Looking at the output distribution of individual activation functions, we found that many of them are highly asymmetrically saturated. The experiments inducing a different degree of asymmetric saturation support the hypothesis that asymmetrrc satura- tion helps improve performance. In addition, we found that Batch Normalization after bounded activation functions has another important effect: relocates the asymmetrically saturated output of activation functions near zero. This enables the swapped model to have higher sparsity, further improving performance. Ex- tensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric satura- tion.
INTRODUCTION
Batch Normalization (BN) has become a widely used technique in deep learning. It was proposed to address the internal covariate shift problem by maintaining a stable output distribution among layers. The characteristics of the output distribution of weighted summation operation, which is symmetric, non-sparse, and "more Gaussian" " (Hyvärinen & Oja 2000), Loffe & Szegedy| (2015) placed the BN between the weight and activation function. Thus, the "weight-Bi Ba-achtt--attitttionn order, which we call "Convention" in this paper, has been widely used to construct one block in many architectures (Simonyan & Zisserman 2014 Howard al., 2017). "Swap" models, swapping the order of BN and the activation function in block. have been also attempted but no significant and consistent difference between the two orders has been observed in the case of ReLU. For instance, Hasani & Khotanlou (2019) evaluated the effect of position of BN in terms of training speed and concluded that there is no clear winner and the result depends on the datasets and architecture types.
However, in the case of bounded activation functions, we empirically found that Swap order exhibits substantial improvements test accuracy than the Convention order with diverse architectures and datasets. We investigate the reason for this accuracy difference between the Convention and the Swap model with bounded activation function based on empirical analysis. For simplicity, our anal- yses are mainly conducted on Tanh model, but applicable to similar antisymmetric and bounded activation functions. We presents the results with LeCun Tanh and Softsign at the end the experi- mental section.
One key difference between Swap and Convention models is the distribution of activation values, shown in Figure [ In the Swap model. most activation values near the asymptotic values of the bounded activation function, that is, highly saturated. This is unanticipated since it is a common belief that high saturation should be avoided. To investigate this paradox, took one step further

---

Under review as a conference paper at ICLR 2023
Convention
Convention
3.5
Swap
3.0
2.5
Density
2.0
Activation
1.5
Swap
1.0
0.5
0.0
0.0
-0.5
0.0 0.5
1.0
Activation value
Activation
Figure I: The activation distributions of layer are almost symmetric (left) in both Convention and Swap models with Tanh. However, the activation distributions of channels in the layer quite different. Symmetric distributions similar to that of the layer appeared similar to layer distribution in channels the Convention model (right top). On the other hand, the Swap model have a one- sided distribution of boundary (bottom right). We chose ten consecutive channels from the 8th layer of the VGGI6 model trained on CIFAR-100.
and looked the output distribution of individual activation functions, not just a whole layer. To our very surprise, even though the distribution is fairly symmetric at the layer level, the activation values of each channel are biased toward either one of the asymptotic values, or asymmetrically saturated. We assume that this asymmetric saturation is key factor for the performance improvement of the Swap model since enables Tanh to behave like a one-sided activation function. In the experiments designed examine whether asymmetric saturation is related to the performance of models with bounded activation functions, we can observe that the accuracy and the degree of asymmetric saturation are highly correlated.
BN after Tanh does not just incur asymmetric saturation but also shifts the biased distribution near zero, which has the important effect of increasing sparsity. Sparsity is generally considered to be a desirable property. For instance, Glorot et al. (201T) studied the benefits of ReLU compared to Tanh in terms of sparsity. One thing to note that each channel is symmetrically saturated, BN will not increase sparsity much since the mean is already close to O. In contrast, the one-sided property of asymmetric saturation causes least half of the sample values after normalization to be almost zero, allowing the Swap model to have even higher sparsity than the Convention model. Ramachandran et al. (2017) explored novel activation functions by an automatic search for different activation functions. The top activation functions found by search are one-sided, and the boundary value is near zero, similar ReLU, The penalized Tanh activation (Xu et al. 2016). inserting leaky ReLU before Tanh, also introduces skewed distribution, and the penalized Tanh achieved the same level of generalization as ReLU-activated CNN. Analogous to the activation functions found in the previous studies, asymmetric saturation combined with normalization makes bounded activation function behave much like ReI.U, achieving comparable performance.
Our findings follows:
The Swap model using Batch Normalization after bounded activation functions performs better than the Convention model in many architectures and datasets.
We discover the asymmetric saturation at the channel level and investigate its importance through carefully-designed experiments.
We identify the high sparsity induced by Batch Normalization after bounded activation functions and perform experiment examine the impact of sparsity performance.
2

---

Under review as a conference paper at ICLR 2023
PROPERTIES MEASURE
Sparsity
Tanh
BN
Asymmetric saturation
Saturation
BN
Tanh
Weight
Weight
Figure 2:
Illustration of Block designs of the Convention order (left) and Swap order (right), and locations for property measurement.
2 SETTINGS FOR INVESTIGATION AND NOTATION
Models. The main purpose of the investigation is to analyze the benefits of using BN after bounded activation functions, more specifically, bounded activation function that is an odd function and has two boundaries. We examine the VGG-like model trained on CIFAR-100 with replacing the activation function from ReLU to Tanh. However, because the VGG architecture was proposed for the ImageNet dataset, the model is overparameterized for the CIFAR dataset. It incurs poor performance and difficulty investigate the Swap model. Thus, we cut out the last convolution layers and select the best model based on the validation accuracy. The model with five cut-out layers shows the best accuracy as in Appendix A.6 We call this model "VGGI6_11 and use this architecture to investigate Conv and Swap orders. Although VGG11 model has already been proposed Simonyan & Zisserman (2014). the validation accuracy of VGG16 is significantly higher than VGGII (VGGI1: (YGGI: Conv 64.55%, Swap 69.94, VGG16 11: Conv 69.5% Swap 74. 1% ). At inference time, The BN normalizes the input distribution to have zero-mean and unit-variance by using the running statistics (e.g.. P for running mean and ô for related to running variance), and then applies the affine transformation, which has a scaling parameter and shifting parameter 3. The Convention model normalizes the outputs of the weighted summation operation conducted in the weight layer, and then Tanh activates the block outputs. On the other hand, in the Swap model, Tanh directly activates the weight layer outputs, and then BN is applied generate block outputs.
Metrics. We consider properties to investigate each order: saturation, asymmetric saturation, and sparsity. We measure the degree of saturation the outputs of Tanh in the layer units. To measure the asymmetric saturation, we collect the outputs of Tanh in channel units. For the sparsity measure, we collect the outputs of each block in the channel units. Layer structure and measurement locations are illustrated in Figure?? 2
Setups for experiment. For the experiment Section 4.2 the weight decay on the convolution layer is fixed, and we vary the weight decay intensity on BN. This experiment's learning rate and the convolution layer's weight decay followed the NWDBN model's hyperparameters. NWDBN is the Convention based model, but the affine parameters of BN are zero. Based on these hyperparam- eters, we increase the intensity of weight decay on 3 in BN from 0.0 to 0.001 by 0.0001. For the experiment in Section 5.3 the learning rate and convolution layer's 's weight decay followed the Swap model's hyperparameters. Then, we change the weight decay intensity on the affine transformation parameters in BN. The intensity list of weight decay are 0, 1e-6, 5e-6. le-5, 5e-5, le-4, and 5e-4. For the experiment in Section 7.T we train models on 4 benchmarks (CIFAR- 10, CIFAR-100, Tiny ImageNet, and ImageNet), base-architecture (VGG16_11, MobileNet), and activation functions (ReLU, Tanh). Because Tanh has non linearity in everyplace except the origin. it can not follow the design of residual connection proposed in He et al.) (2016). Thus, we choose architectures where a skip connection does not exist. For the experment in Section 7.2 we trained VGG16_11 with 3 activation functions (Tanh, Lucun Tanh, Softsign) on CIFAR- 100 dataset. All results except the ImageNet dataset are conducted on 3 random seeds and averaged over seeds for all the measure values and accuracy. We use the SGD optimizer, weight decay regularization, and 2-step learning rate decaying strategy that decays by 0.1. We conduct grid search to obtain the best model for investigation. We explore learning rate and weight decay. The hyperparameters that we use are demonstrated in Appendix A.T
3

---

Under review as a conference paper at ICLR 2023
3 OVERLY SATURATED TANH BUT WELL-GENERALIZED MODEL
Saturation refers to a situation where most of the outputs bounded activation functions are close to the asymptotic value of the function. When training a neural network with bounded activation function whose center is the origin, the output increases due to the weight gradually increasing. The increased output values map close the asymptote in bounded activation functions, shown in the experiment in Glorot & Bengio (2010). Thus, saturation is bound to occur. However, exces- sive saturation results gradient vanishing problem. The gradient of points near the asymptotic values is almost O. Therefore, the gradients of saturated activations vanished. Various methods were proposed to prevent excessive saturation. Glorot & Bengio 2010) proposed an initialization scheme, Rakitianskaia & Engelbrechi] (2015a b) proposed metric to measure the degree of saturation for monitoring the training. Bhat et al. (1990) pre-scaled the inputs of the activation function, and Chen & Chang 1996) proposed adaptable bounded activation.
3.1 SATURATION METRIC
We introduce a saturation metric based on how closely outputs the values to the maximum ab- solute value of the output range of the function. The target outputs for measuring the saturation G = [ (u--92---'')l ] E TR-N is the flattened outputs of Ith layer in fully-connected block or convolu- tion block. is SD' for fully-connected blocks and SC H''' for convolution blocks, where denotes the total number of test samples, D' denotes the dimension size of layer outputs in 1 fully- connected block, and C², H', W respectively denotes the number of channels, height, and width in wth convolution block. We take the absolute value of the input and divide it by the maximum absolute value to normalize it to [0, 1]. The formulation for normalization of 7th element 7ch layer feature map, g.. as follows:
| g
where g' E R is the maximum absolute value of G'. Since the possible output range of the bounded function is fixed. We use the absolute asymptotic value of the bounded function as a all element of g' for measuring saturation. For instance, we g for the Tanh model. We averaged all the normalized values in a layer for our saturation metric. The formulation of our saturation metric on 7th layer, t', as follows:
SV 1E
e has the range of [o, 1]: it approaches if G is highly saturated as illustrated in Appendix A.3 Also, as an implementation issue, the calculation performed in units of mini-batch, and the details are described in appendix A.IT
3.2 HIGH SATURATION IN THE SWAP MODEL.
Even if only the layer order changed from the Con- vention order to the Swap order, there was a 4.61% test accuracy improvement. The results of this model and other models can be found in Table T However, when
Convention
Swap
n
we measure the layer saturation in both models, the Swap model has highly saturated layers. The maximum satura- tion of the Swap model (0.86) is significantly higher than the Convention model (0.45). The saturation of the Swap model shows over 0.7 in almost half of the layers. Even more, some layers overly saturated at almost 0.86. On the other hand, the saturation of the Convention model is lower than 0.5 over all layers. (Figure 3) This is counter- intuitive as excessive saturation is considered an undesir- able situation in the previous works.
0.5
0.1
6.0
Block depth
Figure 3: Layer Saturation of Conven- tion and Swap models
4

---

Under review as a conference paper at ICLR 2023
4 ASYMMETRIC SATURATION
Our saturation metric can dismiss the channel properties due to the summarization of channels the layer. Thus, we conduct channel inspection. Interestingly, when we examine channel distribution, the saturation in that layer has biased to one asymptotic value. Asymmetric saturation appears in most channels on the excessively saturated layer in the Swap model. contrast, the channel distribution of the Convention is almost zero centralized.
4.1 ASYMMETRIC SATURATION METRIC
The target outputs for measuring the asymmetry Q = [o -22 ...qMl 1M E RM is the flattened activation outputs of (th layer and cd dimension for fully-connected block or ch channel for convo- lution block. M is S for fully-connected blocks and SH' for convolution blocks. To measure the channel asymmetry more precisely, we introduce skewness, the metric for measuring the asymmetry. The formulation of the sample skewness for 1 layer and tt channel, kh. as follows:
V MM(M 1)
1 S
(g 1
3
M
(3)
M 2
(1,
2
where na8 E is the mean of 1th layer and th channel's activation outputs. The skewness value has directional distribution information, negative for left-skewed and positive for right-skewed. How- ever, we want to measure asymmetry regardless of direction. Thus we take the absolute value to remove the directional information. The metric for the layer skewness, R., as below:
-Cc R.' = CCC -- (k4.)
The layer distributions in both Convention and Swap models are symmetry, the channel distribu- tions quite different. Thus, measure the asymmetry channel-wise, not layer-wise, like the saturation metric. As an implementation issue, the calculation was performed in units mini-batch, and the details are described in appendix A.IT
As shown in Figure All of the layer skewness in the Convention model measured close to O. Therefore there has little asymmetric distribution. However, in the Swap model, the skewness of layers is relatively higher than in the Convention model. Furthermore, the skewness values are high along the high saturation blocks. It, therefore, implies that saturation occurs with asymmetry. The re- lationship between our skewness metric and the different distribution shapes is illustrated in Appendix A.3
5.0
Convention Swap
4.0
Skewness
3.0
2.0
1.0
depth
4.2 EFFECT OF ASYMMETRIC SATURATION ON GENERALIZATION PERFORMANCE
Figure 4: Layer Skewness in Conven- tion and Swap models
In order demonstrate the effectiveness of asymmetric saturation, we introduce method to control the level of asymmetry the Convention model. First, let us organize the reason why the Convention model cannot make use of asymmetric saturation. We assume that the Convention model can not generate asymmetric saturation well due the weight decay effect on affine transform parameters in BN. In the experiment to verify the mean and variance effects on skewness, we can confirm that both statistical values, the mean and variance of Tanh input, affect asymmetry on Tanh output. The skewness value of Tanh's output on the different input mean and standard deviation can be found Appendix A.4 From this perspective, the affine parameters with weight decay generate the input of Tanh to utilize the center of Tanh by decreasing the mean and variance of its input. Thus, it could decrease the asymmetry of the Tanh output. Therefore, we train a model with no weight decay BN to encourage asymmetric saturation in the Convention model. As a result, the NWDBN model shows improved accuracy of 72.27% % compared to the Convention model 69.5%. To closely examine the effects of asymmetric saturation test accuracy,
5

---

Under review as a conference paper at ICLR 2023
Nora (TTah(())))
Nor (TTan((a))x)))
5.0
A= +0, 0=1
2.5
i = +0.92, 0.1888
A= - 0.85, o=
-3.0
Nor [l
-7.5
A= 0.92, 0
f 0.85, 0.27
-10
-5
Figure 6: Shapes of combined Tanh with normalization functions, and samples related to BN statis- tics. The functions are plotted as lines, and the samples are plotted as dots. We choose some normal distributions whose samples generate ja and ē after the Tanh and randomly generate input samples for Tanh. Note that the and are the statistics of Tanh output in the Swap order.
we increase the intensity weight decay on the Beta parameter, which can eliminate the biasing of the asymmetric saturation in the NWDBN model. As shown in Figure [5] increasing weight decay intensity decreases the skewness in the NWDBN model. Additionally, the test accuracy decreased along with the skewness.
SPARSITY
5.1 ASYMMETRIC SATURATION WITH BATCH NORMALIZATION CAN INDUCE HIGH SPARSITY
Sparsity is desirable property in deep learning. One of the successes of the method that introduces a sparsity is the Relu. ReLU achieves high generalization perfor- mance by utilizing the strengths of sparsity (Glorot et al. 2011]. The sparsity of ReLU is due the bound- ary placed at O. Thus ReLU activates all negative inputs to O. The other work that shows the advantage of hav- ing one asymptote at 0 is Ramachanddan et al. (2017). They conducted an automatic search strategy look up various activation functions used. The top prominent ac- tivation functions identified through search are one-sided with boundary value close zero, like ReLU. Also, Xu iet al. (016)) introduced penalized Tanh activation, which places leaky ReLU before Tanh enhance the perfor- mance of Tanh, which perform as well as ReLU and in- troduce asymmetry in Tanh.
1.6
Avg. Skewness
Accuracy
Avg. Skewness
0.6
0.4
70
69.5
66 tean www.eeeeeeeeeeeeeeeee de hu aest5e Intensity of Weight deray on Beta
Figure 5: Relation between accuracy and averaged skewness over layers. The AwweSnsseesss averaged all the layer- wise skewness in each model with dif- ferent weight decay intensity. The NWDBN model is denoted as 0.0 inten- sity in the graph.
We found that the Swap model also can increase the spar- sity by shifting the majority of the values O when asym- metric saturation occurs. The normalization BN makes the distribution be zero mean. When the asymmetric saturation occured on precede Tanh, the majority of activations are saturated on one side of Tanh output. Thus, the normalization applied on this distribution the majority of values are shifted to near zero which incurs a increasement of sparsity.
5.2 SPARSITY COMPARISON
The NWDBN model shows better performance than the Convention model by inspiring the asym- metry. but it underperforms the Swap model. We found that the rise of asymmetric saturation the NWDBN model gives benefit in terms of asymmetry but decreases the sparsity. In other words,

---

Under review as a conference paper at ICLR 2023
increased asymmetry of activations in the Convention model generates more activation values close to -1 or 1, which incurs less sparse block output. Based on this intuition, we hypothesize that the Swap model has strength sparsity. To compare the models, introduce our sparsity metric to verify the sparsity on each model.
We leverage our saturation metric and modify it for the sparsity metric. Our saturation metric mea- sures the degree to which many values are saturated with the maximum value. On the other hand, sparsity is measured by how a small number of coefficients contain a large proportion of the energy. The more saturated the distribution, the more coefficients divide the total energy. In short, higher saturation decreases sparsity. Therefore, the sparsity metric can be regarded as the reverse of the saturation metric. However, there is differences to the saturation metric. Whereas the saturation is measured on the output of Tanhh, sparsity is measured on the output of the blocks, i.e., the sparsity of the Conv model is measured on the Tanh output, and the sparsity of the Swap model is measured on the BN output. Thus, for the measuring the sparsity, we modify g' in Equation 1 to the vector of maximum absolute output in unit-wise, g' E RD' for fully connected block and IRC for convolution block. Then, we normalize !o by the corresponding unit value in g'. The formulation of modified normalized element, D.. is 1 where d is the corresponding dimension or channel index of tthh output. Consequently, the modified saturation metric, f. is EN / and our sparsity metric for 4th layer, s', is 1 - - Also, we investigated how our sparsity metric satisfies the conditions of the sparsity metric. We demonstrate our sparsity metric based on the desired heuristic criteria of sparsity measures described in Hurley & Rickard (2009). Our sparsity metric satisfies criteria among 6 criteria. The proof can be found in Appendix A0
We first measured saturation each model's block output measure the sparsity and subtracted the saturation value from 1. Then, averaged the sparsity over layers. The sparsity of each model is as follow: Convention (0.717951), NWDBN (0.287974), Swap (0.848927). The Swap model shows the largest sparsity. The result also shows that the Convention model can generate sparse distribution. Because of the weight decay on BN, zero-centered distribution insert the Tanh in Convention model. Lastly, as we expect, the NWDBN model shows the lowest sparsity. However, Since the NWDBN model has a higher asymmetry than the Convention model, the NWDBN model can outperform the Convention model.
5.3 EFFECT OF SPARSITY ON GENERALIZATION PERFORMANCE
In this section, we encourage the sparsity in the Swap model and investigate its effects test accuracy. As mentioned in Section [5 the Swap order can enhance the sparsity when asymmetric saturation occurs. This sparsity can be promoted in training by affine parameters in BN. Decaying affine parameters gathers the most values to 0 during the training phase. Note that the normalization operation shifts the majority near zero, and affine transformation imposes the majority of distribution more cen- tered to 0. To enhance the sparsity of the Swap model, increase the weight decay of affine transformation parameters. The larger weight decay may further increase the sparsity of BN output. As shown in Figure 7 the increase in the model's sparsity and accuracy are highly correlated.
0.85
Aug.Sparsity Accuracy
74.5
0.84
74.0
Avg. Sparsity
0.81
0.82
73.5
0.81
73.0
0.80
0.79
Se.c6 5e:05 Suid4" Intensity of Weight decay on Gamma and Beta
Figure 7: Influence of sparsity on accuracy.
measure the averaged saturation over layers in the Swap model trained with each random seed and calculate the sparsity by our sparsity metric.
SUMMARY OF THE MAIN ANALYSIS
We trained 3 types (Convention, NWDBN, Swap) of models in the above analysis experiments. Each model creates different output distribution of layers due to differences in structure and reg- ularization effects. Output distributions of these models are described in Figure g The Convention model, which is illustrated in Figure 3 (top), normalizes extracted features from the convolution
7

---

Under review as a conference paper at ICLR 2023
Convolution
Normalizication
Affine
Convention
Laper outout
Layer sugpet
Layer oetoud
Layer carput
SA wright
Convolution Layer
Normaliatzation
MWGBN
Middle
Law
Spariiyy
1344
Leger
Convolution Layer
Tash
Normalisizatio
Affine
Swap
High High
Sparthy
Asymmetry
Lager cutput
Lager output
Lays Adp
The distribution VGGI6's 5th block's s output on randomly chosen channels. We chose block where all 3 models were considerably saturated. All test samples in the CIFAR-100 dataset are used to construct the distribution.
Figure 8:
Table 1: Test accuracy with different activation functions and layer orders for VGG16 and Mo- bileNet.
VGGI6 Tanh
MobileNet Tanh
VGGI6 Relu
MobileNet Relu
Dataset
Convention
Swap
Convention
Swap
Convention
Swap
Convention
Swap
CIFAR-I0
91.75
92.90
91.54
92.48
93.69
93.04
92.2
91.93
CIFAR-100
64.84
72.17
64.47
70.63
73.68
71.79
70.06
69.49
Tiny ImageNet
49.29
57.05
50.85
51.79
61.54
59.045
59.79
59.1
ImageNet
60.85
67.04
64.26
72.07
73.83
72.95
70.48
71.1
layer. After that, affine parameters applied to the normalized features. These affine parameters generate zero centralized activation caused by the effect of weight decay. The NWDBN order also normalizes the extracted feature from convolution layer. Still, Unlike the Convention model, there are no downscaling effects on affine transform parameters. For this reason, the input distribution to Tanh can generate distribution away from zero and produce relatively high asymmetry distribu- tion than the Convention model. We can observe that asymmetric saturation is generated through Tanh in sparsity, which negates the benefits of sparsity. Far from the above models, the Swap model applied Tanh to extracted features from convolution layer, and BatchNorm followed. Therefore, Tanh generates asymmetric saturation, then it could be significant number of activations will be moved near zero, helping to increase sparsity. The layer output distribution can be found in Figure (bottom).
However, the asymmetric saturation in the NWDBN model leads low
EXTENDED EXPERIMENTS
7.1 RESULTS VARIOUS DATASETS AND ARCHITECTURES
We mainly investigated VGG16.11 11 with Tanh model trained CIFAR- 100 dataset. In this section, we adopt Swap order on varied settings, which are various datasets (CIFAR-10, CIFAR-100, 100, Tiny ImageNet, ImageNet), architectures (VGG, MobileNet), and activation functions (ReLU, Tanh).
The Swap order and the Convention order of the ReLU model do not show large difference in generalization performance than the difference of Tanh model, and this could be ReL.U has the
8

---

Under review as a conference paper at ICLR 2023
Table VGGI6 11 with bounded activation functions on CIFAR-100, we used averaged skewness over layers for calculating the difference of skewness.
Order
Swap Convention AAvg. Www..keenness
Activation
Convention
Swap
Tanh
69.5
74.11
2.38
LeCun Tanh
67.82
74.46
1.90
Softsign 70.01
73.65
1.28
structural ability produce asymmetric and sparse activations. However, in the case of Tanh, every model with Swap order outperforms the Convention ordered models with significant generalization improvement. The Convention order slightly performs better than the Swap order except for the ImageNet dataset on ReLU model. The Swap MobileNet with Tanh especially performs better than the Convention Mobilenet with ReLU on CIFAR and ImageNet datasets. The results can found in [ Also, all Swap models generate asymmetry on Tanh.
The asymmetric saturation tends to occur from the front layers. Also, can find that the range of the asymmetric saturation existence block is related to the amount of dataset information and dataset resolution. For example, when compar- ing the CIFAR-10 10 and CIFAR-100, the asym- metrically saturated layers happen further back. When comparing the Tiny ImageNet, and Ima- geNet, the model trained on the ImageNet gen- erates asymmetric saturation until the last con- volution layer. These results are shown in Fig- ure g
12.0
CIFAR-10
10.0
CIFAB-190
TinyimappNeet
ImageNet
8.0
6.0
4.0-
2.0-
0.0
2
6
10 11 12
14
Block depth
Asymmetric saturation of the Swap model on various dataset. There are no BN on fully connected layer in VGGI6 for Tiny Ima- gaNet and ImageNet dataset, we only measure the skewness a convolution layer.
Figure 9:
7.2 RESULTS OF OTHER BOUNDED ACTIVATION FUNCTIONS
Our main investigations are based on the Tanh activation function. this section, we test whether similar behavior is observed with other activation functions, such as LeCun Tanh (LeCun et al. 2012) and Softsign (Turian et al. 2009). In
detail, we use the formula LeCun tanh as follows 1.7159 x tanh (2xnnput) antisymmetric, just like Tanh. Softsign was proposed to prevent vanishing gradients by alleviating the saturation of neurons. It grows polynomially rather than exponentially, approaching its asymp- totes much slower (Glorot & Bengio 2010). LeCun Tanh has gentle slope and wider output range than Tanh. The asymmetric saturation caused by the Swap order occurs not only in Tanh but also in other activation functions. The shapes of these functions and layer skewness were shown in Appendix A.5] The Swap with Softsign and LeCun Tanh have improved performance compared to the Convention. can be found Table When swapping, asymmetric saturation happens the least in Softsign, which makes it challenging to create a saturation state. Furthermore, the Softsign model shows lower performance than the Tanh model, which could generate more saturation with the most significant slope in the Swap, even though the Convention model had the highest performance.
3 They are bounded and
CONCLUSION
In this work, we report that the Swap models perform better than the Convention models in many cases and analyze what brings about performance improvement. Asymmetric saturation at the chan- nel level and sparsity induced by BN are two key factors explaining the better performance of the Swap models. With asymmetric saturation and normalization by BN, the final distributions gener- ated by BN layers of the Swap models much resemble those by ReLU. This explains why the Swap models outperform the Convention models and often show results comparable to the ReLU models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Naveen V Bhat, Peter A Minderman, Thomas McAvoy, and Nam Sun Wang. Modeling chemical process systems via neural computation. IEEE Control Systems Magazine, 10(3):24-30, 1990.
Chyi-Tsong Chen and Wei-Der Chang. A feedforward neural network with function shape autotun- ing. Neural networks, 94):627-641, 1996.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Moein Hasani and Hassan Khotanlou. An empirical study on position of the batch normalization layer in convolutional neural networks. In 2019 5th Iranian Conference Signal Processing and Intelligent Systems (ICSPIS), pp. 1-4. IEEE, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1704.04861, 2017.
Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):422344741 2009.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neurul networks, 13(4-5):4114433 2000.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Rober Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Anna Rakitianskaia and Andries Engelbrecht. Measuring saturation in neural networks. 2015 IEEE Symposium Series on Computational Intelligence, pp. 1423-1430. IEEE, 2015a.
Anna Rakitianskaia and Andries Engelbrecht. Saturation in pso neural network training: Good or evil? 2015 IEEE Congress on Evolutionary Computation (CEC), pp. 125-132. IEFE, 2015b.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for activation functions. arXiv preprint arXiv: 1=Xiv:1710.059411 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Joseph Turian, James Bergstra, and Yoshua Bengio. Quadratic features and deep architectures for chunking. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pp. 245 248, 2009.
Bing Xu, Ruitong Huang, and Mu Li. Revise saturated activation functions. arXiv preprint arXiv:1602.05980 05980, 2016.
10

---

Under review as a conference paper at ICLR 2023
Table 3: Training hyperparameters of the VGGI6 Tanh models
Convention
Swap
CIFAR-TO
CIFAR-I00
Tiny ImageNet
ImageNer
CIFAR-TO 0
CIFAR-I00
Thay ImageNer
ImageNer
Training Epochs
200
200
100
Learingg Rate
0.1
0.01
0.01
0.01
0.01
0.1
0.01
0.01
Leuming Rate Drop
100.
150
Weight Decay
0,0001 128
0.0005
0.001
0.0001
0.001
0.0005
0.001 128
0.001
Batch Size
128
256
128
128
256
Table 4: Training hyperparameters of the VGG16 ReLU models
Convention
Swap
CIFAR-10
CIFAR-100
Tiny ImageNet
ImageNet
CIFAR-10
CIFAR-100
Tiny ImageNer
ImageNet
Training Epochs
200
200
200
100
200
200
200
100
Leaming Rate
0.01
0.1
0.1
0.01
0.01
0.01
earning Rate Drop
100, 150
100, 150
100, 150
30
100, 150
100, 150
100, 150
60, 90
Weight Decay
0.0001
0.001
0.0005
Batch Size
128
128
128
256
128
128
128
256
Table 5: Training hyperparameters of the MobileNet Tanh models
Convention
Swap
CIFAR-10
CIFAR-IOU
Tiny ImageNet
ImageNet
CIFAR-I0
CIFAR-100
ImageNet
ImageNNet
Training Epochs
200
200
200
100
200
200
100
Leamiing Rate
0.1
0.1
0.01
0.1
0.1
0.1
0.1
0.1
Lerming Drop
150
100.
Weight Decay
0.00001
0.0005
0,0001
0,0001
0.0001
0.0005
0,0001
0.0001
Batch Size
128
128
128
256
Table 6: Training hyperparameters of the MobileNet ReLU models
Convention
Swap
CIFAR-10
CIFAR-IOU
Tay ImageNet
ImageNet 100
CIFAR-10
CIFAR-I00
Hly ImageNet
ImageNer
Training Epochs
200
200
200
2001
200
200
100
Learning Rate
0.01
0.01
0,01
0.01
0.01
0.01
0.1
Learning Rate Drop
100, 150
100, 150
100, 150
30,
100, 150
100, 150
100, 150
60, 90
Weight Decay
0.001
0.0001
Batch Size
128
128
128
256
128
128
128
256
A APPENDIX
A.1 TRAINING HYPERPARAMETER
We sweep the learning rate and weight decay hyperparameter. The learning rate was 0.1 and 0.OI. For CIFAR and Tiny-ImageNet datasets, we trained models with a batch size of 128, and the learning rate was reduced by one-tenth 100 and 150 of the total 200 epochs, and we swept 4 weight decay of 0.005, 0.001, 0.0005, and 0,0001. For ImageNet datasets, we trained models with batch size of 256, and the learning rate was reduced by one-tenth 30 and 60 of the total 100 epochs, and we swept weight decay of 0.001, 0.0005, and 0.0001. We chose the best averaged-accuracy model for the 3 random seeds and averaged the values of these three models for all measurements for analysis. Because of the computation issue, we only use seed for ImageNet dataset with early stopping.
The hyperparameters used training are shown in Table 3 5|
A.2
No BN
We also compare the saturation and skewness between the Convention model and the model without BN, we call this "NoBN" model. As shown in Figure 10 asymmetric saturation also occurs in the model without BN, we call this "NoBN" model. However, the NoBN model can not utilize the advantages of batch normalization (e.g., high learning rate), it shows low test accuracy than the Convention model even though asymmetric saturation exists compared to the Convention model. The accuracy of the Convention model is 64.84% and the accuracy of the NoBN model is 61.06%.
11

---

Under review as a conference paper at ICLR 2023
1.0
0.9
Convention
NoBN
6-
NoBN
0.8
S Gaturabi oo
0.7
5-
Skewness
0.6
0.5
0.4
3
2
0.2
0.1
0.0
2
10
11 12
13
6
12 13
Block depth
Block depth
Figure 10: Layer saturation (left) and skewness (right) of the Convention VGG and the NoBN VGG model trained on CIFAR-100.
A.3 SATURATION SKEWNESS MEASUREMENT VALUES
Our saturation metric becomes when the distribution is gathered O, and it increases the elements in the distribution close to the maximum expression range. For the uniform distribution, the degree of saturation was measured at 0.5. The measurement on different distributions can be found in Figure [I (left). Skewness is the metric for measuring the asymmetry of the distribution. Skewness is calculated increase. We calculate the absolute on skewness in our asymmetry metric, thus the increases are regardless of the direction. The measurement on different distributions can be found Figure [T] (right).
when the distribution is symmetric, and it increases as the asymmetry
2.0
Saturation
Skewness
1.5
0.5
2.0
Density
0.3
0.0
1.0
4-
0.5
0.0
-1.0
-0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
Activation value
Activation value
The degree of saturation on different distributions (left) and the degree of skewness on different distributions (right)
Figure 11:
A.4 THE EFFECTS OF THE MEAN AND STANDARD DEVIATION OF INPUT DISTRIBUTION ON TANH
The mean and variance of input distribution on Tanh affect the asymmetry of Tanh output. The skewness of Tanh output depends on the mean, and standard deviation can found Figure [2 The maximum skewness of varied mean distribution is increased on the increase of mean. However, the maximum skewness does not align with the input standard deviation increases. The skewness decreases not only the small input standard deviation but also the large input standard deviation. Ad- ditionally, in the same mean condition, a decrease in standard deviation from the maximum skewness point more rapidly decreases the skewness than increase in standard devation.
12

---

Under review as a conference paper at ICLR 2023
2.5
Input Mean
0.3095196
2.0
Skewness
1.47221949
1.5
1.0
0.5
0.0
0.1
0.5
1.0
1.5
2.0
Input Standard deviation
Figure 12: The skewness of Tanh output depend on the mean and standard deviation of Tanh input
A.5 SKEWNESS TENDENCIES ON VARIOUS ACTIVATION FUNCTIONS
The key to the success of the Tanh model with the Swap order is asymmetric saturation. We show that asymmetrie saturation also appears in the other bounded activations, such as LeCun Tanh and SoftSign. The Conv model with the 3 types of activation functions shows low layer-wise skewness. The skewness is less than over the overall layer. However, a significant skewness increment arises when the Swap order is applied. The SoftSign shows minor improvement in skewness due to its property preventing saturation.
5.0
Tanh
1.5
SoftSign
Activation Value
1.0
4.0
LecunTanh
Skewness
Convention
0.5
3.0
Swap
0.0
2.0
-05
-1.0
1.0
-1.5
0.0
7
8
9
Input
Block depth
Figure 13: Shapes of activation functions (left) and skewness tendency of different activation func- tions (right), dashed line represents the Convention model and the solid line represents the Swap model.
A.6 BEST DEPTH MODEL SEARCHING ON VGG16
To find appropriate model for CIFAR, measured the accuracy of models without the last convolution layers of VGG16. We train them from scratch using VGG16's training hyperparameters. The accuracy gradually increases until the VGG16_11 model, and decreases after that. The results are shown in Table 7 One thing to note is that the omitted layers have low skewness in the VGG16 model. The layer-wise skewness considerably decrease after the 8th block, which is the same number of convolution layers in the best performance model. The layer-wise skewness is shown in Figure
Table 7: Performance of shortened Swap VGG16 models. The number of removed convolution layers in the VGG16.n model is the difference between 16 and n.
VGG16_15
VGGT6_14
VGG16_13
VGGI6_12
VGGT6,
VGGT6.10
VGG16.9
VGGT6.N
Accuracy 72.17
73.02
73.48
73.85
73.76
73.92
72.57
70.91
70.69
13

---

Under review as a conference paper at ICLR 2023
Swap
5.0
4.0
Skewness
2.0
1.0
4
5
6
10
11
12
13
14
Block depth
Figure 14: Skewness of the layers in original VGG16 models
A.7 THE INPUTS WEIGHIT LAYER THE GRADIENTS OF TANH AND WEIGHT
The vanishing gradients problem is inevitable when excessive saturation occurs. However, the Swap model can alleviate the gradient vanishing problem. The forward propagation among the convolution and Tanh layers in the Swap model is as follows: = Wx, a Tanh(y). Here, X is hwe-by-1 vector, and W a d-by-n matrix, where is the height, w the width, the number of channels, d is the number of filters, and is the size of column i.e., the gradient of is obtained by the of the corresponding dimension element. As result, the larger solve the vanishing gradients problem. The Conv block's output is Tanh's output in the range of [-1, 1], while the Swap block's output can have larger value since BN has no limit. A vanishing gradient occurs Tanh of the Swap model in the experiment. However, it is alleviated the gradient of convolution weight due to the large x, and shows a similar scale to the gradients of convolution weight in the Cony model. In the gradient gradients on Tanh of the Swap model are smaller than those of the Conv model. On the other hand, the Swap model has larger scale of than the Conv model. Thus, Conv and Swap models have similar scale when looking at the gradient of the convolution weight.
hwvc. In backpropagation,
the shallow layers, the backpropagation
Figure 15: Plots for mean of absolute value of Convolution input(left) and mean of absolute gradient of Convolution weight value(center) and mean of absolute gradient of tanh input value(right) in the Convention(top) model and the Swap(bottom) model
14

---

Under review as a conference paper at ICLR 2023
A.8 LEARNING CURVE OF CONV AND SWAP MODELS
Both models were trained with the same hyperparameters. At the beginning of training, the training loss of the Swap model decreases faster than that of the Conv model, but when training is complete, the training losses of the two models become almost the same. However, through the validation loss, we can see that the Swap model has better generalization ability. The training loss is shown in the Figure 16 and the test loss is shown in the Figure 17
1600
VGG11 Swap
VGG11 VGG11_Conventen Convention
1200
Train Loss
1000
800
400
200
100
150
200
250
300
EPOCHS
Pionea 16. Teinning lose of Conn and Curon modale
VG611 Swap
VG611_Swap
140
VG611 Convention
120
Test Loss
100
80
40
300
EPOCHS
Figure 17: Test loss of Conv and Swap models.
A.9 RELATION BETWEEN PERFORMANCE AND SPARSITY FOR LARGE AFFINE PARAMETERS
We followed the As the size of the weight decay applied to the affine parameters increased, the sparsity decreased. Accordingly, it was confirmed that the performance also decreased.
0.86
75
Sg.rsttsity
0.84
70
Accuracy
0.83
0.81
Avg.Sparsity
Accuracy
0.8
60
12:02
11
Intensity of Weight decay Gamma and Beta
Figure 18: Accuracy drops as the sparsity decreases for large affine parameters.
A. PROOF OF PROPERTIES FOR SPARSITY METRICS OF INVERSE SATURATION
Gd is vector [gl, g2, g3, ... gN] Ô' is vector [gl, g2, g3, ... gN]
15

---

Under review as a conference paper at ICLR 2023
Theorem 1.1: S satifies S(aG") = S(''), s((")) va R,a 0. Proof: scaling the G' also scale the D'
a|G²| aG
G¹ g'
:.
= C
Theorem 1.2: satifies S(a + G*) < S(C'), a E R a > (We also exclude the case mentioned
in Hurley & Rickard 2009) that all elements of G are the same.)
Proof:
EN g + Na
S(G' + a) = Ng + No
if Ng 2 then x +o No +Na
x Ng
S(a + G*) < S(G))
Theorem 1.3: S satifies S(G') = S(G')G"')))
(II is concatenation)
Proof: We define concat t) which means concatenate vector X as t times. S(concal(G', ())
Then
12 8i = S(G))
NN
Theorem 1.4: S satifies vi33 = Bi > O, such that Va > 0:
S(Lgi-..g + 3 + a...)) > 5((gi + B...))
We choose sufficiently large 3 that [g!| + B g. Let assume that S(lgi...g + 3 + a...) S ([9g + 3....). Then
IN k= + +
=k of + N(g) +
IN
31
No +
NN Dh +8+a
ΣN - g +
N(g + +
N + +
En En + + + +
Exyith + 9 + 3
of + +
g
ER
Entigh
9 + B + a
g +
1
x
Ø + 8 + a
g +
:. s(o( + a....) ((gg + 3...))
Theorem 1.5: S satifies S(G' > S(G))
Proof:
ΣN EN
g
N
>
N +1
N
A.11
ALGORITHMS
For more details, we take channel-wir wise summation with respect to batchs, but we divide the summa- tion value by D and accumulate as batch statistics. Because the whole step is same taking average with respect to total sample, we can divide by the total size first and sum all values as batchs later. We follow this step due to the memory usage.
16

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Calculating skewness the layers
Input: (s 1,2...., S) mini batch B2xxHx samplee samples
N = -ooaalmumbrerff
Output: skeuness R².
Function Statistics (X,mean, (X.meon.urrianeeskkuneeess tarianeee,.kkknsssooottioo)
= x }.
for = 1, 1,,,,,, do
b,, h', w² is the shape of X
B= x W x w', D= N x 1 x w
# B IRO Ob, xno block' < (la)(XX)) by channels
if condition is meon then
# I2C RR
batchhstatistic sum of block! along the channels
10
2. botch. statistic stachsstaisstic
mean' batch_statistid end
12
14
condition is variance then
# RCC RC
batch. statistiie sum of (block! - mean!)? along the channels
2. batch statistic + statistic
3. variance += b-bbhh end
19
20
if
condition skewness then
# C² + RBCxB
21
batchsstatistid + sum of (block' -mean113 Vvariance!"
22
along the channels
Alatisstatisti statistid DDx(D-1)
23
D-2 3. skewness' += batch statistic end
25
26
end
27 end
28 mean (mean', mean", meean",..,,menn ".
29 variance = {variance" variance variance
skewness (skewness skewness skewness
31 for 8 = 1, 2.., do
X <- gelactivalion((.. get activation output values the layers.
update mean with Statistics (X (X.meanccomdition +- mean)
33
34 end
for
1,2, S do X + gelactivation(zz) get activation output values over the layers.
36
37
update variance with Statistics (X menan..crriancce.mmmiiiooo variance)
end
39
for
& = ...,...,
S do X <- gettactivatio(rr.) get activation output values the layers.
41
update skeuness with
Statistics ww..stiisss.m..mmms.aaa.a (Xmeenn.orrdnnesskkeenssssooooo skewness)
42 end
R C
skewness' Average of absolute of each channel values in skewness' along the layers.
45 return skewness
17

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Calculating (Empirical) saturation over the layers
Input: F, 1, S) mini batch 12
N total number of samples
saturation_type empirical or not
activationtypeislLCCu or not.
Output: saturation R²_
if saturation-rype is empirical then
upper t channel -wiee maximum absolute value else if activation type is LeCun tanh then
upper <- 1.7159
else end
upper t- 1
saturation (saturation's , Sanfrurtiion.. }.
for s = 1, 1, ..., do
if' saturation_type is empirical then
11
X < getblock(x..) get block output values over the layers.
end
13
else
14
X getactivation(z.)) get activation output values over the layers.
15
end
= [xl x2 .'. for 1,
X do
17
C¹ b,, h', w is the shape of X
18
B B=b, b, x 1 x w D= C x
11
w
19
# #RC' x - RC² xh, x xmo
20
1. block! + (Xa) by channels.
21
2. Take absolute to block!
22
block! + Wook!
23
upper
# TR. x
4. sum of all values of block!
25
26
saturation + sum
27
saluration' += saturation
28
end
end
30
return saturation
18


Reviews:
[{"review_id": "lnEDe2K9w_W", "reviewer": "Reviewer_AJ9Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It is shown that putting batch normalization after activation function enhance optimization with SGD. Then authors experimentally compares the distribution of hidden representation for two different locations of batchnormalization. In particular, paper investigated the sparsity, and asymmetric saturation. A stark contrast is reported for asymmetric saturation when BN is placed before and after activation. ", "summary_of_the_review": "The paper reports an interesting observation in learning neural networks. However, this observation requires further justifications and investigations. \n\nThanks for the rebuttal response! ", "strength_and_weaknesses": "**Strength** \nIt is surprising that a simple swap of batch normalization and activation can enhance the performance of neural nets for bounded activations. \n\n**Weakness** \n\n- I am not convinced about the argument that asymmetric saturation enhances generalization. Are authors sure that the enhanced performance is not due to accelerated training? Is it possible to compare the convergence for two different neural architectures?\n- For asymmetric saturation, is it possible to create saturation for conventional architectures; thereby enhancing their performance? \n- How do stepsize, batch size, network width, depth, channel size affect the observation? \n- Would be nice to analytically derive the reason for asymmetric saturation at initialization to make sure it is independent of the dataset.\n- Is the swap only useful for image classification and convolution nets. Is the swap beneficial for MLPs as well? \n- Does the swap for tanh activation outperform ReLU with conventional BN location?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NgaYNPtRD0o", "reviewer": "Reviewer_DHjF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts an empirical analysis of the interaction between batch normalization and bounded activation functions. Specifically, the paper compares the architecture using batch normalization after a bounded activation(Swap model) and the architecture using a bounded activation after batch normalization(Convention model). Motivated by the observation that the swap model outperforms the convention model significantly when a bounded activation is used, the authors designed experiments to identify the reasons for these performance differences. The paper shows that in terms of asymmetric saturation, the Swap model and the Convention model behave differently and argues that high sparsity induced from the asymmetric saturation has a strong association with the generalization performance.", "summary_of_the_review": "It is interesting to know that with a bounded activation, the order between BN and the activation causes drastically different qualitative behavior. However, the presentation of the idea can be improved further by replacing vaguely defined terms and expressions. The arguments supporting the conclusions seem weak.", "strength_and_weaknesses": "### Strengths\n- The authors discover that in the Swap model with a bounded activation, each feature map is saturated on one side of the asymptotic value of the bounded activation.\n\n\n### Weaknesses\n- It is a bit confusing whether the asymmetric saturation has a strong association with generalization performance.\n    - Although there is another noticeable observation that the saturation is very low in higher block depths, this is not discussed at all.\n    - Can you explain more how to exclude the possibility that low saturation at higher blocks or the combination of both could be a reason for better generalization?\n- It is also confusing whether the sparsity has a strong association with generalization performance\n    - Since the sparsity metric is $s^l = 1 - t^l$ where $t_l$ is the saturation metric, layerwise sparsity can be obtained from Figure 3. The relation between the sparsity of the Swap model and the Convention model is different depending on which layer is considered. In such case, it seems a bit of a stretch to draw a conclusion that the higher the sparsity is the better the generalization is. \n    - In a sense, this contradicts with the authors' argument 'Our saturation metric can dismiss the channel properties due to the summarization of channels in the layer.'\n    - Can we say that different sparsity distributions over layers with the same average sparsity will have similar generalization performance?\n- The coverage of the analysis is a bit limited. \n    - The analysis is claimed to be valid with bounded nonlinearity and without residual connection, excluding many widely used architectures. Also, it seems difficult to generalize or apply the claim of the paper to commonly used cases.\n    - Even though it is subjective, it does not seem that the Swap model with Tanh performs comparably to the Convention model with ReLU. \n\n### Questions\n- Can you elaborate on ' Because Tanh has non-linearity in everyplace except the origin, it can not follow the design of residual connection proposed'?\n    - Does that mean that Tanh has gradient 1 at the origin? What does it mean by nonlinear in everyplace?\n    - What does it mean by 'following the design of residual connection'?\n- In Table 1, with ReLu, the Convention model is better than the Swap model. Have you considered or performed a similar analysis to understand this reversed behavior?\n- What is 'the center of the function'? This term is not defined precisely. The center of the domain of the function or the center of the image of the function?\n- NWDBN is not explained until Figure 8 and is frequently used before Figure 8. Even though I guess that NWDBN may stand for No Weight Decay Batch Normalization, acronyms should be explained when it is first used.\n- What is the formula of LeCun Tanh? There are many typos for LeCun Tanh.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M8AbgAa7y3", "reviewer": "Reviewer_5V5C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper reports that when a bounded activation function is used, the Swap order of network block \"conv/fc+act+bn\" achieves performance better than the Convention order of block \"conv/fc+bn+act\". The authors find that the bounded activation function accompanies high saturation in each network block and asymmetry saturation in each channel within the block. The asymmetry saturation with batch normalization can induce high sparsity, which assists the generalization performance. Three metrics were proposed to measure saturation, asymmetry saturation (skewness), and sparsity respectively. Experiments show that when several bounded activation functions are used, Swap order performs better than Conventional order in the classification tasks, but still cannot outperform ReLU-based networks.", "summary_of_the_review": "This paper discusses an interesting topic, but the analysis seems less comprehensive. I would like to rate it as borderline reject. If the author can clarify my problems, I am willing to consider changing my score.", "strength_and_weaknesses": "Strength:\n- This paper investigates an interesting topic: the order of batch normalization(bn) and activation function(act), and gives a partial answer: placing bn after act gives better test performance when bounded activation functions were used.\n- The logic chain is quite clear and easy to understand: this paper first reports the improvement of the Swap order, then analyzes the order from the point of saturation and asymmetric saturation, and found that the bn after asymmetric saturation can induce sparsity, which may be the reason for improvement. \n- The finding of this paper is interesting. For a bounded act, the saturation is asymmetric, and it may assist the performance.\n\nWeaknesses:\n- The saturation metric seems not complete enough. Consider that if the maximum absolute value of G^l is small, which means that the values of G^l are centered around 0, the metric would be close to 1 also. \n- Fig.3 and fig.4 only show high saturation and high skewness in shallow blocks, in deep blocks things seem changed. What does that mean? Is it means that in the deep block, we should use Convention order?\n- Although the finding is interesting, the conclusion seems less practical. The performance of the Swap order with bounded acts cannot outperform the Convention order with the ReLU act. We still can use ReLU directly.\n- Besides, (this may be a overclaim) since the authors report a counterintuitive example that the Swap order, which achieves better performance, accompanies saturation,  the author had better discuss it because intuitively the bounded act can induce gradient vanishment problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8ARzkqx71Tx", "reviewer": "Reviewer_Qoao", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how the order of batch normalization (BN) placed in the network affects the performance, when using the unbounded activation functions (e.g., Tanh). It shows that the Swap model (BN after the nonlinearity) using unbounded activation functions has significantly better performance than the conventual model (BN before the nonlinearity) using the same unbounded activation functions. It further shows activation values of each channel of Swap model is asymmetrically saturated, by looking into the output of channel-activations. It claims this asymmetrically saturated activations of Swap model increases the sparsity, which improves the performance of the model. ", "summary_of_the_review": "This paper provides a somewhat new view in investigating BN’s position with different nonlinearity. The observations are interesting but with unclear clarification. I personally have main concerns and currently tend to negative for this paper. ", "strength_and_weaknesses": "\n**Strengths:**\n\n1. normalization and nonlinear (activation) layers are basic layer/module in DNNs, and it is a good plus to investigate their interaction. This paper investigates How BN interact with bounded activation and shows several interesting observations，e,g., the activation values\nof each channel of Swap model is asymmetrically saturated. \n\n2. The view in investigating the correlation among saturation (especially the channel-wise saturation), sparsity and performance is new to me, and this paper also provides quantitative metric to evaluate the saturation and sparsity. \n\n\n\n**Weaknesses:** \n\n**1. One big concern for me is that some claims are not clearly clarified or rigorous.**\n\n(1) In Introduction section, I donot understand this claim “In contrast, the one-sided property of asymmetric saturation causes at least half of the sample values after normalization to be almost zero, allowing the Swap model to have even higher sparsity than the Convention model”.  I find the only support is from Section 5.1. Indeed I donot understand why “The asymptotic values of combined Tanh with normalization operation are $\\frac{+1-\\hat{\\mu}}{\\hat{\\sigma}}$ and $\\frac{+1+\\hat{\\mu}}{\\hat{\\sigma}}$.”?  Can this paper provide further clarification? Besides, I understand “$\\hat{\\mu}$ becomes around -1 or 1 value”, but why $\\hat{\\sigma}$ is calculated as an appropriate size to produce a high skewness”? and what is a appropriate size?\n\n(2) In Section 3.2, “This is counterintuitive as excessive saturation is considered an undesirable situation in the previous works” is somewhat misleading. I believe the experiments are based on the model at the end of training. I believe the “excessive saturation is considered an undesirable situation in the previous works” is for the initial training stages (the model has not learned information from the datasets), but it is not undesirable in the end of the training. Indeed, a good model with high confidence to the prediction may have excessive saturation neurons. \n\n(3) The statement in Section 3 “When training a neural network with bounded activation functions with a center of the function at the origin, the output increases due to the weight gradually increasing.” is not rigorous. Does sigmoid (bounded activation) has a center of the function at the origin? It is also has likelihood that the bounded activation is saturated in the initial training, which depends how the model is initialized. Furthermore, why the weight gradually increasing? I think it also depends on the how much the weight decay is used and (for a model with weight decay)? \n\n(4) It seems to be contradictory base on the statement “In short, higher saturation decreases sparsity” in Section 5.2 and the observation that Swap model has high saturation, but why the experimental results show that Swap model has higher sparsity? Is any wrong understanding of me? \n\n(5) The observations seem to be not uniformly hold over all layers of VGG, from Figure 3 and 4. Why the activations of Swap model after the 10 th layer has lower saturation and almost the same Skewness, compare to the Conventional model, in Figure 3 and 4? That is not consistent to the observations of Swap model before the 10th layer. \n\n**2. I also have concern on the experimental setup and analysis.**\n\n(1) Considering this statement “Because Tanh has non-linearity in everyplace except the origin, it can not follow the design of residual connection proposed in He et al. (2016). Thus, we choose architectures where a skip connection does not exist.”, does this mean that the MobileNet used in this paper has no residual connection in its ResBlock? If yes, this paper only performs experiments on the VGG-like (feed forward neural network), but not for the ResNet-like network.  By the way, I donnot understand why residual connection is not allowed for Tanh? Does this design cannot well trained or the model cannot be conducted by design?\n\n(2) In Figure 7, it is true that the increase in the models sparsity and accuracy are highly correlated. But I am wondering whether this results still hold by further increasing the weight decay on the affine parameters (the current maximum weight decay is $5e^{-4}$)? Intuitively, further increasing the weight decay on the affine parameters will further increase the sparsity, based on the statements shown in this paper. Does it further improve the accuracy of the network? What is the sparsity and accuracy, if we use a weight decay on the affine parameters of $1e^{-3}$, $1e^{-2}$, $1e^{-1}$?\n\n(3) Even though Swap model using BN after bounded activation functions performances better than conventional model, one problem is that all the swap models using bounded activation functions shown in Table 1 has significantly lower accuracy compared to the conventional model using ReLU (the widely used unbounded activations). Based on this, I cannot well recognize the contributions of this paper, from the perspective of practice. It is better to provide the results that a swap model using BN+bounded activation has better performance than the conventional model using BN+ ReLU. \n\n\n(4) It is good that this paper submit the code, but when I check how this paper calculate the saturation, skewness and sparsity in the code, I cannot find it. Do the authors tell me in which line and which file, the saturation, skewness and sparsity are calculated?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject