Tanh Works Better with Asymmetry
Authors: Dongjin Kim
Woojeong Kim
Suhyun Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Batch Normalization
Activation Functions
Saturation
Sparsity
TL;DR: With bounded activation functions, using batch normalization after activation functions is better because of asymmetric saturation and sparsity.

Abstract: Batch Normalization is commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e., using Batch Normalization after activation functions, has also been attempted, but its performance is generally not much different from the conventional order when ReLU or a similar activation function is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance than the conventional order on various benchmarks and architectures. This paper reports this remarkable phenomenon and closely examines what contributes to this performance improvement. By looking at the output distributions of individual activation functions, not the whole layers, we found that many of them are asymmetrically saturated. The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance. In addition, Batch Normalization after bounded activation functions relocates the asymmetrically saturated output of activation functions near zero, enabling the swapped model to have high sparsity, further improving performance. Extensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation.  Finally, based on this investigation, we test a Tanh function shifted to be asymmetric. This shifted Tanh function that is manipulated to have consistent asymmetry shows even higher accuracy than the original Tanh used in the swapped order, confirming the asymmetry's importance. The code is available at https://github.com/hipros/tanh_works_better_with_asymmetry.

Tanh Works Better With Asymmetry
Dongjin Kim 1,3 Woojeong Kim? Suhyun Kim
Department of Computer Science and Engineering, Korea University
2 *Department of Computer Science. Cornell University 3 KKorea Institute of Science and Technology {npclinic3, kwj962004, dr suhyun. kiim-mgmill..mmmmmmmmmm com
Abstract
Batch Normalization is commonly located in front of activation functions, as pro- posed by the original paper. Swapping the order, i.c., using Batch Normalization after activation functions, has also been attempted, but its performance is generally not much different from the conventional order when ReLU or similar activation function is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance than the conventional order on various benchmarks and architectures. This paper reports this remarkable phenomenon and closely examines what contributes to this performance improvement. By looking at the output distributions of individual activation functions, not the whole layers, found that many of them asym- metrically saturated. The experiments designed to induce different degree asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance. In addition, Batch Normalization after bounded activation functions relocates the asymmetrically saturated output of activation functions near zero, enabling the swapped model to have high sparsity, further improving performance. Extensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation. Finally, based on this investigation, shifted to be asymmetric. This shifted Tanh function that manipulated to have consistent asymmetry shows even higher accuracy than the original Tanh used in the swapped order, confirming the asymmetry' 's importance. The code is available at httts:///github: htt.gggthhh.....cicmeeeeeeeeeeee com/hipros/tanh, works www.wwortrtreeee_eeeeeeeee
test Tanh function
pnnteerrwtthh with_itttheeee asymmetry
Introduction
Batch Normalization (BN) is a widely used technique in deep learning. It was proposed to address the internal covariate shift problem by maintaining stable output distribution among layers. As the characteristics of the output distribution of the weighted summation operation tend to be symmetric, non-sparse, and Gaussian [10), Ioffe & Szegedy (11) placed the BN between the weight and activation order, which we call "Convention" in this paper, has
function. Thus, the Breright-N-aattiatiiin been widely used to construct block in many architectures [8] "Swap" models, swapping the order of BN and the activation function in a block, have also been attempted, but no significant and consistent difference between the two orders has been observed in the case of ReLU similarly shaped activation functions. For instance, Hasani & Khotanlou 6 evaluated the effect of the position of BN terms of training speed and concluded that there no clear winner and the result depends on the datasets and architecture types.
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X2 Asymmetry Law Sparaity
Convention
Conediation
Tarh
Butth Nerrodiaaion
C2
eurgort
Swap
Low
We examine the layer and channel-wise distributions of Tanh activation (left) and the feature map distribution single convolution block (right). Tanh produces nearly symmetrical layer activation in the Convention and Swap models (left-left). However, the channel distributions vary, with the Convention model showing symmetry (left top) and the Swap model presenting one-sided distributions (left bottom). The Convention model, with zero-centered Tanh input, exhibits low asymmetry and sparsity (right top). The Swap model, without the zero-centered constraint of BN, enhances asymmetry and, due to BN's placement after Tanh, sparsity (right bottom). The ranges of output are marked as lines above the plot.
Figure 1:
However, in the case of bounded activation functions, we empirically found that the Swap order exhibits substantial improvements in test accuracy than the Convention order with diverse architectures and datasets. This paper empirically investigates the reason for this accuracy difference between the Convention and the Swap models with bounded activation functions. For simplicity, our analyses are mainly conducted with Tanh but are applicable to similar antisymmetric and bounded activation functions. We present the results with LeCun Tanh and Softsign at the end of the experimental section.
One key difference between Swap and Convention models is the distribution of activation values, as shown in Figure[llett), In the Swap model, the mean of convolution layer output shifts substantially away from O. Thus, most activation values are near the asymptotic values of the bounded activation function, that is, highly saturated. This is unanticipated since it is a common belief that high saturation should avoided. To investigate this paradox, we took one step further and looked at the output distribution of each activation function, not just just whole layer. To our very surprise, even though the distribution fairly symmetric at the layer level, the activation values of each channel asymmetrically saturated. Through
biased toward either one of the asymptotic values in-depth investigation, we come to hypothesis that this asymmetric saturation a key factor for the performance improvement of the Swap model since it enables Tanh to behave like one-sided activation function. In the experiments to examine whether asymmetric saturation related to the performance of models, observe that the accuracy and the degree of asymmetric saturation highly correlated.
BN after Tanh also shifts the distribution biased toward either -1 or 1 by asymmetric saturation near zero, which has the important effect of increasing sparsity, as depicted in Figure rright). Sparsity is widely considered to be a desirable property. For instance, Glorot et al. [5) studied the benefits of ReLU compared to Tanh in terms of sparsity. One thing to note is that if each channel is symmetrically saturated, BN will not increase sparsity much since the mean is already close zero. In contrast, the one-sided property of asymmetric saturation causes at least half of the sample values after normalization to be almost zero. It allows the Swap model to have even higher sparsity than the Convention model. Asymmetric saturation combined with normalization makes a bounded activation function behave much like ReLU.
Based on the above lessons, we test enforcing the asymmetric saturation and sparsity of Tanh outputs. This involves shifting the inputs of Tanh by introducing asymmetry, and then shifting the outputs in reverse by -lanh(T) to boost sparsity. This equivalent to using a shifted Tanh new activation function. Our experimentation across different architectures and datasets reveals that this adjusted

---

Tanh function with fixed asymmetry consistently outperforms the original Tanh used in the Swap order. Its effectiveness even comparable that ReLU. This validates the hypothesis that Tanh works more efficiently with asymmetric saturation. Our findings follows:
The Swap model using Batch Normalization after bounded activation functions performs better than the Convention model in many architectures and datasets.
We discover the asymmetric saturation at the channel level and investigate its importance through carefully designed experiments.
We identify the high sparsity induced by Batch Normalization after bounded activation functions and perform an experiment to examine the impact of sparsity on performance.
Our shifted Tanh with enforced asymmetric saturation shows higher accuracy than the Swap Tanh, which reconfirms the importance of asymmetry.
2 Settings and Notation
Model For our investigation, we choose the VGG-like model due to its simple and straightforward design. This model is trained on the CIFAR-100 dataset [12]. As a modification, we replace the ReLU activation function with Tanh. However, because the VGG architecture was proposed for the ImageNet dataset, the model is overparameetrrzzed for the CIFAR dataset. It incurs poor performance and difficulty in investigating the Swap model. Thus, we cut out the last convolution layers and select the best model based on the validation accuracy. The model with five cut-out layers shows the best accuracy, as the Appendix. We call this model "VGG16_11 11" and use this architecture to investigate Convention and Swap orders.
Metric We consider three properties to investigate each order: saturation, asymmetric saturation, and sparsity. Saturation measured on Tanh and quantifies how closely outputs the values to the maximum absolute output range limit, also known as the asymptotic value. We involve the skewness metric in statistics to measure asymmetry on Tanh. One thing that differs from the skewness is the absolute. We absolute the skewness to quantify the asymmetry regardless of skewed direction. Sparsity is measured on the output of the block, i.e., Tanh for Convention and BN for Swap. To measure the sparsity, we inversely utilize our saturation metric.
Experiment setup In Section 4.2 we verify the effect of asymmetry by slightly cutting off the asymmetry in the asymmetry-boostee on-BN (NWDBN) model. The NWDBN model is ordered as Convention, and the weight decay does not apply to BN. We fix the intensity of the weight decay on the weight layer and vary the decay intensity on BN to regulate asymmetry. The initial hyperparameters followed the best hyperparameter of the NWDBN model. Then, we increase the intensity of weight decay on in BN from 0.0 0.001 by 0.0001. In Section 5.3, we investigate the effect of sparsity by slightly increasing it. The initial hyperparameters followed the Swap model's best hyperparameters. Then, we change the weight decay intensity on the and 3 in BN. The searched intensities of weight decay are 0, 1e-6, 5e-6, le-5, 5e-5, le-4, and 5e-4. We verify the effects of the layer order in various datasets, networks, and activation functions in Section 7.1 We train models on four benchmarks 100 C0F, Tiny ImageNet [13), and ImageNet 3) three aase-architectures (VGG, MobileNet, PreAct-ResNet), and three activation functions (Rel.U, Tanh, ShiftedTanh). For the ResNet, the Swap order incurs extra non- lincariy functions, which hinder the fair comparison between the Convention and Swap model. Except for the first stage, the first shortcut connection in each stage has convolution and BN layers in the block. In the case of the Swap Tanh model, Tanh replaces BN and vice-versa. Thus, the number of non-lincarity functions increases as the number of the linear projection layer in the shortcut. We, therefore, report the results of TreAcc-ReNNet, which only has the convolution layer in the shortcut connection. (Furthermore, upon training ResNet20 with Tanh the CIFAR- 100 dataset, we observed that the Swap model surpassed the Convention model, achieving an accuracy of 69.06% compared to the Convention model's 's 68.97%.)
Convention model, which refer the No-Weight-Decay
All results, except the ImageNet dataset, are conducted on three random seeds. The measured values and accuracy are averaged over seeds. We use the SGD optimizer with momentum of 0.9, weight
3

---

decay. We adopt 2-step learning rate decaying strategy that decays by 0.1. We conduct a grid search obtain the best model for investigation. We search different combinations initial learning rates (0.1 and 0,01) and weight decay values (1e-4, Se-4, le-3, and 5e-3). The specific hyperparameters for each model can be found in the Appendix.
The input of an arbitrary layer in the convolution neural network is denoted as X' e Raxexhxu where n, c, ht and w represent the number of samples, channels, height, and width, respectively. We reshape X' to X E Rexm where T E Rc where T = tanh tanh(X.,)). J.. BN normalizes the channel distribution to have zero mean and unit variance using estimated statistics. After normalization, affine transformation with scaling parameter y E IR and shifting parameter [ E I is applied channel-wise. Thus, the output of
Notation
n/w. The output of the Tanh function is
BN is given by: Baj - 7/ X1.- Rs.
Bi. where is a constant added for numerical stability, j
and o are estimated mean and standard deviation of each channel, respectively, computed over the mini-batch of training data samples.
Overly Saturated but Well-Generalized Model
The phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function [5] When training a neural network with bounded activation functions, the outputs of neurons increase due to the gradually increasing weights [4]. The increased output values become close to the near asymptote in bounded activation functions, as shown in the experiment Glorot & Bengio (4. However, excessive saturation damages the neural network, and it is regarded as situation to evade [5] 16
1I MI Various methods were proposed to prevent excessive saturation. Glorot & Bengio 4 proposed an initialization scheme, Rakitianskaia & Engelbrecht [15 [6 proposed a metric to measure the degree of saturation for monitoring the training, Bhat al. [ pre-scaled the inputs of the activation function, and Chen Chang 2 proposed adaptable bounded activation.
3.1 Saturation Metric
Our saturation metric quantifies the extent to which the output values approach the function's 's output range limit, also known as the asymptotic value. To normalize the input, we calculate the absolute value and divide it by the maximum absolute value, resulting in a range of [0, 1]. The saturation of layer is defined as:
T
7= -
where represents the maximum absolute value of the output range of the activation function. Specifically, for the Tanh function, is equal to 1. lies in [0, 1). The large value of this metric indicates T are highly saturated, as illustrated in the Appendix.
3.2 High Saturation in the Swap Model
Even if only the layer order is changed from the Convention order to the Swap order, there a 4.61% 11 improvement in test accuracy. However, when measure the layer saturation in both models, the Swap model has highly saturated layers. The layer-wise Saturation of Swap and Convention models can be seen in Figure a). This is counterintuitive excessive saturation is considered undesirable in previous works 4 15 16 I 2]. The saturation almost half of the layers in the Swap model is over 0.7. Some layers
have saturation over 0.8. On the other hand, the saturation of the Convention model is lower than 0.5 over all layers.
4 Asymmetric Saturation
In order to scrutinize saturation, we perform an in-depth analysis at the channel level, one step further than the layer level. Fascinatingly, we notice that most channels display asymmetric saturation in the Swap model, especially layers with excessive saturation. On the other hand, the channel distribution in the Convention model tends to be centered around zero, even at the channel level.

---

Convention
Convention Swap
Swap
Saturation
0.6
Skewness
0.5
0.4
0.3
0.2
0.1
0.0
0.0
Block depth
Block depth
Figure (a) Layer-wise Saturation of Convention and Swap models. (b) Layer Skewness in Convention and Swap models. The two figures display 95% confidence interval.
4.1 Asymmetric Saturation Metric
Both the Convention and Swap models demonstrate symmetric distributions at the layer level. However, notable differences arise at the channel level. To quantify and compare the degree of asymmetry present at the channel level, we measure the asymmetry of each channel individually and calculate the average to determine the overall skewness of the layer. We adopt sample skewness metric to gauge the asymmetry of the Tanh output. The skewness of given zth channel, denoted pi. calculated follows:
m
m
(m - (m 2)
where nn represents the sample third central moment of T..:, while of represents the sample standard deviation cubed T... The averaged absolute skewness layer, k. herein referred to simply as Skewness, is given by:
k = pp:l-
Figure b) illustrates the Skewness for all layers in the Convention and Swap models. In the Convention model, the Skewness of most layers is close zero, indicating a minimal presence of asymmetric distribution. Conversely, the Swap model exhibits higher Skewness compared to the Convention model. Moreover, the regions with high saturation blocks within the Swap model show particularly elevated Skewness values. This suggests that saturation in these areas occurs asymmetrically in channel distributions.
4.2 Effect of Asymmetric Saturation on Performance
In order examine the effect of asymmetric sat- uration, we devise a method to control the level of asymmetry in the Convention model. One key factor that limits the emergence of asym- metric saturation in the Convention model the influence of weight decay on 7 and 33 within BN. Especially, small B values by weight decay con- strain the distribution to be zero-centered, which incurs decrease the asymmetry of the Tanh output. In our experiment, the NWDBN model successfully encourages asymmetric saturation and has higher Skewness than the Convention model. Moreover, the NWDBN model shows improved accuracy of 72.22% compared to the Convention model 69.5%. To better examine the effects of asymmetric saturation performance, gradually increase the intensity of weight
Avg. Skewnksssss
Accuracy
Avg. Skeunese
1.0
0.8
0.6
sode Intensity Weight decay
Figure 3: Relation between accuracy and averaged Skewness over layers. The "Avg. Skewness" aver- aged all the layer-wise wise Skewness in each model.
5

---

7.5
(TTanh(Tan)())
5.0
A = + 0=
Nor (..an(TTnxx)))
A= = 0.92, 6= 0.18
0.0
A = 0.85, 6
-2.5
No xx=
-7.5-
A + 0.92, 6 0.18
-10.0
ü 0.85, 0
10
X
Figure 4: Red lines represent Tanh activation functions combined with normalization. The dots represent the output values of the combined function for different input samples. Samples of same color indicate they generate the same BN statistics the Tanh output. The notation Nora.a represents the normalization function BN.
decay the 3 parameter from zero, which indicates that slightly eliminates the asymmetry in the NWDBN model. As the weight decay on gets bigger, Skewness becomes smaller, and the test accuracy degrades, shown in Figure 3 This strong correlation clearly supports our hypothesis: asymmetric saturation helps improve performance.
Sparsity
5.1
Asymmetric Saturation with Batch Normalization Can Induce High Sparsity
ReLU achieves high generalization performance by utilizing the strengths of sparsity 5 The sparsity introduced by ReLU is a consequence of its single boundary at zero, which transforms all negative inputs to zero. In addition, work by Ramachandran et al. [17] shows the benefits of having single asymptote zero. They carried out an automated strategy investigate variety of activation functions. The most notable activation functions discovered through this search were one-sided and had boundary value close zero, similar ReLU.
We note that the Swap model effectively generates sparsity, pushing a majority of values toward when asymmetric saturation occurs. This shifting is credited to the normalization process in Batch Normalization (BN), which recenters the distribution around zero. Thus, when the asymmetric saturation occurs, this saturation subsequently shifted towards zero during the normalization process, enhancing sparsity. The combined effect of Tanh with normalization, along with samples of the function's is output showing different BN statistics in the Swap order, depicted in Figure The majority of values and the asymptotic lines that the samples saturated are shifted to near zero.
5.2 Sparsity Metric
Intuitively, a sparse representation is one in which small number of coefficients contain a large proportion of the energy (9]. We utilize saturation metric to measure sparsity. Saturation quantifies the extent to which many values reach the maximum. We measure the sparsity by the extent to which many values far from the maximum. However, sparsity is measured consider the output of Tanh for the Convention model and the output of BN for the Swap model. BN has boundaries, empirical saturation, i, as follows:
the block output,
empirically calculate for normalization in Equation We denote the
1
m
B
f
(4)
mB
max

---

where B is the maximum absolute element in B i... B = max, Baj. For the Convention model, we apply the above metric to T' instead of B. Finally, the sparsity s is defined as:
1
(5)
Our sparsity metric satisfies five criteria among six desired criteria of sparsity measures in Hurley & Rickard 99.
The proof of each criterion can be found in the Appendix.
5.3 Effect of Sparsity on Performance
In this section, we investigate the significance of sparsity in the Swap model and its influence on test accuracy. There's 's a marked relationship between increased sparsity and test accuracy,
0.85
Avg. Sparsity Accuracy
74.5
0.84-
Aug Sparsity
0.83
73.5 8
The result can be seen in Figure 5
0.82
0.81
As elaborated in Section 5.1 the Swap order has the capability to intensify sparsity when there's asymmetric saturation. However, the introduced sparsity in the Swap model might be reduced due to the shifting operation present in the affine transformation of BN. This reduction occurs be- cause effectively decaying the affine parameters can help maintain values close to zero through- out the training. To preserve the sparsity in the Swap model, we increase the weight decay on the parameters involved in the affine transformation. Such an amplified weight decay can further boost the sparsity of the BN output.
0.80-
0.79-
0.78
0.0 16:06 5e-95 1e45 sels 1e:4 56/04 72.0
Intensity of Weight decay on Gamma and Beta
Figure 5: Relation between accuracy and averaged Sparsity over layers. The "Avg. Sparsity" averaged all the layer-wise Sparsity in each model.
6 Shifted Tanh for Asymmetric and Sparse Output
Based on the investigation, we modify the Tanh function, referred to as Shifted Tanh, to encourage asymmetry and sparsity. Different from the Swap Tanh model, Shifted Tanh can generate asymmetry and sparse activation on zero-centered input. Shifted Tanh shares properties similar to ReLU. This property accelerates the model to use asymmetry and sparsity for better generalization.
The Shifted Tanh involves shifting the inputs by a factor of E R before the Tanh operation, introducing asymmetry, and then shifting the outputs in reverse by the degree of the shift while considering the Tanh operation. The formulation of the Shifted Tanh as follows:
T + tanh(a + 7) tanh(()).
(6)
We choose r = -1 based on the best accuracy from the ablation study on weight t, as shown in the Appendix.
7
Extended Experiments
7.1 Investigation on Various Datasets and Architectures
Our study delves into the Tanh models trained on the CIFAR-100 dataset using the VGG16_1 architecture. In this section, we evaluate the Swap order various datasets and architectures and compare its accuracy against the Convention order.
From the results presented in Table significant difference is observed in performance between the Swap and Convention orders when using Tanh; the Swap order consistently delivers better results. In contrast, for both ReLU and shifted Tanh models, the sequence of layers has negligible effect on accuracy. Moreover, the Convention model usually exhibits slight edge over the Swap model. We assume that the zero-centered distribution BN augments the asymmetry and sparsity the Shifted Tanh in the Convention arrangement. Hence, the Convention model's 's accuracy slightly surpasses the Swap model, mirroring patterns seen with ReLU. It's noteworthy that the enhanced accuracy of shifted Tanh rivals that of the ReLU model.
7

---

Table Experimental comparison between orders on various datasets and models.
Tanh
Shifted Tanh
ReLU
Datasets
Models
Convention
Swap
Convention
Swap
Convention
Swap
VGG16
91.75
92.90=0.0
93.9220.17
93.6020.17
93.6920.0s
93.04
CIFAR-I0
MobileNct
91.661a.u
92.53_o.12
92.85 10.10
92.64
92.48.1o.2s
91.93
PreAct-ResNet18
92.4610.17
_4,41_o.1
94,4810.15
94.57
94.94_0.07
94.86
VGG16
64.95
70.93
23.87
73.21
73.68
71.79
CIFAR-100
MobileNet
64.501a.31
70.39_
72.751o.zz
71.88
70.27
69.49
PreAct-ResNet188
73,26+0.19
75.76=0.17
76.70+o.a5
76.51
78.06+0.29
77.39
VGG16 49.29
57.05
60. 00.6822..0
60.37=0.07
59.37
59.05
Tiny ImageNet
MobileNet
45.27+0.19
51,78=0.26
52.91 +0.20
52. 16
51.92+0.11
50.48
PreAct-ResNet34
59.06+0.26
64.94=0.0s
65.82
65.66+n.20
67.28
66.21 +0.18
VGG16
60.85
67.04
73.4
72.11
73.83
72.95
ImageNet
MobileNet
64,26
72.07
70.4
72.56
68.27
71.1
5.0
1.5
SoftSign
Activation value
1.0
4.0
LecunTanh
Convention
0.5
Skewness
3.0
Swap
0.0
2.0
0.5
1.0
-1.5
0.0
0 Input
1
2
3
4
6
7
8
9
Block depth
Figure 6: Shapes of activation functions (left) and Skewness tendency of different activation functions (right). The dashed line represents the Convention model and the solid line represents the Swap model in the right figure.
7.2 Effect of the Swap Order on Other Bounded Activation Functions
Asymmetric saturation is a crucial factor in the success of the Tanh model. The primary focus of our investigations lies with the Tanh activation function. However, in this section, we extend our exami- nation to include other bounded activation functions, namely LeCun Tanh [1] and Softsign [19
Softsign, defined as f has the same asymptotic values as Tanh, yet its growth rate is polynomial, not exponential, resulting in slower approach its asymptotes [ Therefore, it can mitigate the issue of vanishing gradients by reducing neuron saturation. LeCun Tanh, characterized by a gentler slope and broader output range than Tanh, has the formula 1.7159 Tanh 3 x:).
The asymmetric saturation caused by the Swap order occurs not only in Tanh but also in other bounded activation functions. The Convention model with the three types of bounded activation functions shows low layer-wise Skewness. The Skewness is less than one over the overall layer. On the other hand, significant Skewness increment arises when the Swap order is applied. The shapes of these functions and layer Skewness are shown in Figure
In all of the activation functions, the Swap models improve performance compared to the Convention. can found Table 2 When swapping, asymmetric saturation happens the least in Softsign, which makes it challenging to create saturation state. Furthermore, the Softsign model with the Swap order shows lower performance than the Tanh model with the Swap order, which could generate more saturation, even though the Convention model had the highest performance.

---

Table 2: Experimental comparison of accuracy with asymmetry between orders. We compare the accuracy with Skewness on VGGI6 I1 models trained by CIFAR-100. We used averaged Skewness over layers to calculate the difference of Skewness.
Order
Avy. Skewness
Activation functions
Convention
Swap
(Swap Convention)
Tanh
69.5
74.11
2.38
Softsign
70.01
73.65
1.28
LeCun Tanh
67.82
74.46
1.90
0.7
Convention
Swap
0.6
0.5
Accuracy
0.4
0.3
0.2
0.1
0.0
gaussian
shot
impulse
deforus
glass
mation
zoom
snow
frost
fog
brightness
contrast
elastic
pixelate
ipeg
Figure 7: Experimental comparison between orders on the different types corruption. We compare robust accuracy on CIFAR-100-C between the Convention and Swap ordered VGG16_11 11 trained by CIFAR-100.
7.3 Robustness the Asymmetrically Saturated Model
We evaluate the Swap model using corrupted version of the 110AR-100 dataset, termed CIFAR- 100-C 77. The results of robustness accuracy on CIFAR-100-C can seen Figure 7 The findings reveal an enhanced accuracy with the Swap order throughout all corruption categories. The Convention model has an average accuracy across all corruptions at 41.4%. However, the Swap model enhances this average to 45.3%.
Asymmetric saturation offers increased robustness to variations in input data. This robustness stems from the behavior of the Tanh function, especially when utilizing the output on its bounds to its midpoint. Near the center, particularly around zero, the Tanh function highly responsive to slight input changes, leading to considerable alterations in the output. As a result, the Convention model is more vulnerable to such input shifts, especially in the zero vicinity of the Tanh function. contrast, the Swap model activates Tanh's asymptotic region. Therefore, even pronounced input deviations lead to only slight output changes. Such trait makes the Swap model inherently robust, maintaining more consistent output even with input disparities.

---

Table 3: Experimental result of comparison between Convention and NWDBN models on VGG16 __1 trained by CIFAR-100.
Configurations Convention NWDBN 72.22
Accuracy 69.5
Avg. Skewness
Avg. Sparsity
0.254 0.718
0.718 0.288
7.4 Further Discussion the Effect of Asymmetry and Sparsity
The Swap model enhances its performance through both asymmetry and sparsity, making chal- lenging distinguish the contributions of each. We choose the Convention order to determine the primary factor influencing performance. When asymmetry increases in the Convention model, the majority of values have large values and decreases in sparsity. Thus, the NWDBN model, which encourages greater asymmetry in the Convention model, has a reduced sparsity while the level of asymmetry is enhanced. The result of accuracy and averaged Skewness and Sparsity on the Convention and NWDBN models can be seen in Table Notably, even with its diminished sparsity, the asymmetrically saturated NWDBN model surpasses the Convention model's performance. From these observations, we deduce that asymmetry plays a more pivotal role than sparsity.
8 Conclusion
In this work, we report that the Swap models with bounded activation functions perform better than the Convention models and analyze what brings about performance improvement. Our analysis reveals that the asymmetric saturation and the increased sparsity by the batch normalization after bounded activation functions contribute the increased accuracy of the Swap models. Interestingly, this makes bounded activation functions behave much like ReLU and enables them to achieve performance comparable that of ReLU, Most importantly, expands our understanding of the activation function and will further assist in developing superior activation functions in the future.
Acknowledgement
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) 10o.2021-0-0000 Development of Ultra-high Speech Quality Technology for Remote Multi-speakerr Conference System), and by the National Research Council of Science & Technology (NST) grant by the Korea govermment (MSIT) (CRC-20-02-KIST].
References
[1] Bhat, N. V., Minderman, P. A., McAvoy, T., and Wang, N. S. Modeling chemical process systems via neural computation. IEEE Control Systems Magazine, 10(3):24-30, 1990.
[2] Chen, C.-T. and Chang. W.-D. A feedforward neural network with function shape autotuning. Neural networks, 94))627 641, 1996.
[3] Deng. J., Dong, W., Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on computer vision and puttern recognition, pp. 248-255. Icce. 2009.
[4] Glorot, X. and Bengio, Y. Understanding the diffieulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 449-256.12 JMLR Workshop and Conference Proceedings, 2010.
[5] Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
10

---

[6] Hasani, M. and Khotanlou, H. An empirical study on position of the batch normalization layer in convolutional neural networks. In 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), pp. 1-4. IEEE, 2019.
[7] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representa- tions, 2019.
[8] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1 704.04861, 2017.
[9] Hurley, N. and Rickard, Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723 4741, 2009.
[10] Hyvärinen, A. and Oja, E. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 (0)4114443000 1-430, 2000.
[11] loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456 PMLR, 2015.
[12] Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
[13] Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7,7):3, 2015.
[14] LeCun, Y. A., Bottou, I.., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
[15] Rakitianskaia, A. and Engelbrecht, A. Measuring saturation in neural networks. In 2015 IEEE Symposium Series on Computational Intelligence, pp. 4423-1430. IEEE, 2015.
[16] Rakitianskaia, A. and Engelbrecht, A. Saturation in pso neural network training: Good evil? In 2015 IEEE Congress on Evolutionary Computation (CEC), pp. 125-132. IEEE, 2015.
[17] Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
[18] Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[19] Turian, J., Bergstra, J., and Bengio, Y. Quadratic features and deep architectures for chunking. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pp. 245-248, 2009.

---

A Limitation
While this paper contains an in-depth analysis of Tanh, it does not propose function that is superior to existing activation functions. Generally, it is known that Tanh does not perform better than ReLU, but it's hard to find a study that deeply analyzes the reasons for this. To our knowledge, this paper is the most comprehensive analysis of the performance degradation when Tanh used in conventional manner. Although it suggests strategies to utilize Tanh more effectively, achieving performance nearly par with ReL.U, it fundamentally does not assert that Tanh surpasses ReLUU in terms of effectiveness.
B Accuracy of the Shifted Tanh According to
In this section, we focus on the hyperparameter in the shifted Tanh function. We carry out experiments using VGG16 11, MobileNet, and PreAct-ResNet models trained on CIFAR- 10 and CIFAR-100 100 datasets, testing different values of the parameter (-1.5, -1.2, -1.0, -0.8, and -0.5). The performance results corresponding to each of these T values are presented Table B. these experiments, all hyperparameters, except for are maintained at their best-averaged accuracy settings based on each model trained on the CIFAR dataset. Our aim to identify the effective value without pushing all inputs into the saturation zone due to excessive asymmetry. The best-performance t is identified by averaging the accuracy across different models for each T and selecting the one with the highest performance. In our case, -1 is the best.
Table B.1: The results of shifted Tanh models on various values.
1.5
-1.2
-1.0
-0.8
-0.5
Accuracy 80.56
83.86
84.15
84.01
83.58
C
Metric
C.1
Saturation and Skewness on Different Distribution Patterns
2.0
Saturation
Skewness
0.7
1.5-
0.5
2.0
Density
0.3
0.0
1.0
0.5
0.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Activation value
Activation value
Figure C.l: The Saturation (left) and Skewness (right) values on different distributions.
Our saturation metric is designed to quantify the concentration of elements to the maximum absolute output values. The metric has bounds from 0 to 1, and it equates to O when all the elements are 0 in the Tanh case. Conversely, it grows larger as elements tend towards the maximum limit the output range. For example, the saturation metric is evaluated at 0.5 in uniform distribution. For saturation values of the Tanh output on the different Gaussian distributions, refer to Figure C.I (left).
Skewness is measure that quantifies the asymmetry within a distribution. Symmetric distribution will yield the Skewness O, and the Skewness increases with the rise in asymmetry. Notably, we calculate the absolute skewness value in our asymmetry metric, ensuring that the metric remains
12

---

unaffected by the direction of skewness. For better understanding of how this measure applies to various distributions, Figure C.1 (right) provides Skewness on each distribution.
C.2 Impacts of Mean and Standard Deviation Variations on Tanh Asymmetry
2.5
Input Mean
0.0
2.0
0.69314718
Skewness
1.47221949
1.5
1.0
0.5
0.0
0.1
1.0
1.5
2.0
Input Standard deviation
Figure C.2: Skewness value on the different means and standard deviations of the Gaussian distribu- tion for Tanh input.
The mean and variance of input distribution on Tanh affect the asymmetry of Tanh output. The Skewness of Tanh output on various mean and standard deviation can be found in Figure C.2 In the Skewness of varied mean distribution, the Skewness increased on the mean increase. However, the maximum Skewness does not align with the standard deviation increases. The Skewness decreases not only the small input standard deviation but also the large input standard deviation. Additionally, in the same mean condition. decrease in standard deviation from the maximum skewness point more rapidly decreases the Skewness than an increase in standard deviation.
C.3 The Proof Demonstrating the Sparsity Metric Criteria
This section examines the adequacy of our sparsity metric against the heuristic criteria for sparsity measures defined by Hurley Rickard [9]. Our sparsity metric meets five of these criteria: Scaling, Rising Tide. Cloning, Bill Gates, and Babies.
We consider the absolute operation as pre-applied the metric input, according Hurley & Rickard 9). Thus, our sparsity metric is represented as f x ~ 8, where x E TR is a vector of absolute values in channel in BN output, and -=aax signifies the maximum value in x, i.c.. x m max; Iz. We prove our sparsity metric in the setting of one-channel, which can be expanded the multi-channel setting without loss of generality. Below are the five criteria our sparsity metric meets and proof for each.
Proof of Scaling, Sparsity is scale invariant. That is, for any scalar R, where > 0, the function satisfies:
f(ax) = f(x).
The invariance of scale means that sparsity considers relative differences rather than absolute magni- tudes.
Proof. If scale the vector by a, also scale the maximum positive value m. + by a. Therefore, we have:
Im ar; 21
f(ax)
max +
mm
ma.
fxx).
13

---

This demonstrates that the sparsity of the vector x remains the same when the vector is scaled by any positive scalar a.
Proof of Rising Tide. Adding constant each element decreases sparsity. Formally, for any scalar E R, where a > O, the function f satisfies:
f(a < f(x).
We exclude the case, as mentioned in 9 where all elements of x are the same. This property also indicates that sparsity increases more values approach zero.
Proof. We begin by noting the sparsity of the vector + oc:
I; + ma
aa) =1
mar + ma
Assume for contradiction that f(x + a) 2 f(x). Given that mx 2 2= Ond ma the following inequalities can be derived:
ym
i= + ma
D marmax+
mar ma
mm 04 + ma
22
mar + ma
mar
which contradicts the initial assumption, thus implying that
f(a + x) < f(x),
as desired.
Proof of Cloning. Sparsity is invariant under cloning,
That is, for vector x, the function f satisfies:
f(x) /(x[xx) = [[x||x||XX -...
[==..._[x[[xx]]]] |(x||x||........./ 1|x),
where || denotes concatenation, such that x||x = P2, Pen 21, Fm]
The principle of cloning implies that sparsity remains the same even if set of values is replicated.
Proof. Define function g x, ~ y, where 8 Z and y RRmd is the result of concatenating
times. We then have:
1mm T3 8mamax + 2m [
[g[(x 8))
=
maxax
= f(x).
This shows that the sparsity of x remains the same even when it is concatenated with itself times.
Proof of Bill Gates. As one individual element becomes infinitely large, the sparsity increases. Formally, for every i, there exists p O, such that for all O:
11.,..., 32 MWm,,..,
a, .... (m,)) [( -.... Jt Ps Ps...,, Fmm)
The implication is that if single value grows without bounds, the sparsity also increases indefinitely.
14

---

Proof. We first choose a sufficiently large p such that vi, I; + p > x xm + Assume for contradiction that ([11,,,,, p + 0t; -..-.: Fm]) S (((11 -. Ps. Nm. This implies:
Ym
2h 2 +
k= 2K + p 3
m(x; + + a)
m[;; + p)
E -k +
Σ k= + P
m(x; +pt:
m(x; + p)
D + 24 + ++
E + + P
I4 + p p
I2 +
24 + p +
Iitp
1
x
24 + p+ xit
leading to contradiction. Therefore,
f( 24 + p a..., Zmll > f[E11.... Is + Ps..., Fm.).
Proof of Babies. Adding a new element of zero increases sparsity. Formally, for a vector x, the function f satisfies:
f(x|10) fxx)
Concatenating zero elements to the existing values increases the relative difference to the other values, which increases sparsity.
Proof.
k= IK
m +
m
We start by subtracting one from both sides of the inequality:
2
The negative sign can be removed by reversing the inequality:
Multiplying both sides of the inequality by m(m + 1) (since is positive integer, is also positive, and the inequality sign will not change), we get:
m < (m + 1)2
Subtracting Em m. Ti from both sides of the inequality, obtain:
M
Since Fg is not negative value, the sum is also non non-negative, which verifies the inequality. Therefore, the original inequality is proven.
15

---

D Deciding Model Configurations for Investigation
D.1 Investigation of Depth Impact on Accuracy in VGG16 Models
Table E.2: Experimental results of shortened VGG16 models with the Swap order for CIFAR- 100.
The number of removed convolution layers the VGGI6_n model is the difference between 16 and n.
VGG16
VGGI6_15
VGG16
VGGI6_15 13
VGGI6
VGG16_I1 II
VGGI6 10
VGG16_9
VGG16.F
Accuracy
70.93
72.97
73.63
73.77
74.00
74.11
72.31
70.95
70.76
Table E.3: Experimental comparison between the official VGGI and our VGG16_11 models trained by CIFAR-100.
Order
Models
Convention
Swap
VGG11
64.55
69.94
VGG16
69.5
74.11
To identify model for focused analysis using the CIFAR dataset, we examined various VGG16 variants. This examination progressively removes convolution layers from the end towards the front of VGG16. The result can be seen in Table E.2 We observed an increase in accuracy until peaking at the VGG16_I II model, followed by a decline. Although VGGI model has already been proposed in Simonyan & Zisserman [18), the validation accuracy of VGG16 1 is significantly higher than VGG11 on CIFAR-100 100. The results can be seen in Table E.
E
Training Hyperparameters
We sweep the learning rate and weight decay hyperparameter. The learning rate 0.1 and 0.01. For CIFAR and Timy-ImageNet datasets, we trained models with batch size of 128, and the learning rate was reduced by one-tenth 100 and 150 of the total 200 epochs, and we swept four weight decay of 0,005, 0.001, 0.0005, and 0.0001. 1. For ImageNet datasets, we trained models with a batch size of 256, and the learning rate was reduced by one-tenth 30 and 60 the total 100 epochs, and we swept three weight decays of 0.001, 0.0005, and 0.0001. We chose the best averaged accuracy model for the three random seeds and averaged the values of these three models for all measurements for analysis. Because of the computation issue, we only use one seed for the ImageNet dataset with early stopping. The hyperparameters VGG, MobileNet, and PreAct-ResNet are shown E. E.3 respectively.
16

---

E.1
VGG
Table F.1: Hyperparameters VGG11 with Tanh
Convention
Swap
NWDBN
Training Epochs
200
200
200
Learning Rate
0.1
0.1
0.1
Learning Rate Drop
100, 150
100, 150
100, 150
Weight Decay
0.0005
0.0005
0.0005
Batch Size
128
128
128
Table F.2: Hyperparameters of VGG16 with Tanh
Swap
CIFAR-10
CIFAR-100
Tiny ImageNNtt
ImageNet
CIFAR-10
CIFAR-100
Tiny ImageNet
ImageNNt
Training Epochs
200
200
Learning Rate
0.1
0.01
0.01
0.01
0.01
0.01
Leariing Kate Drop
100, 150
100, 150
100, 150
30, 60
100, 150
100, 150
100, 150
30,
Weight
0.001
0.0005
0.001
0.001
Batch Size
128
128
128
256
128
128
128
256
Table F.3: Hyperparameters of VGG16 with the shifted Tanh
Convention
Swap
CIFAR-10
CIFAR-100
Tiny ImageNet
ImugeNNtt
CIFAR-10
CIFAR-100
Tiny ImageNNtt
ImageNet
Training Epochs
200
200
200
200
200
200
90
Learning Rate
0.1
0.1
0.1
0.1
0.1
0.1
0,1
0.1
Drop
100,
Weight Decay
0.0001
0,0005
0.0001
0.0001
0,0005
0.0005
0,0005
0,0001
Batch Size
128
128
128
256
128
128
128
256
Table F.4: Hyperparameters of VGG16 with the ReLU
Convention
CIFAR-10
CIFAR-100
Tiny ImageNet
ImageNet
CIFAR-10
CIFAR-100
Thy ImageNet
ImageNet
Tiiniin Epochs
200
200
200
90
200
200
200
90
Learning Rate
0.11
0.01
Learinng Rate Drop
100. 150
100,
100. 150
100,
100, 150
100,
30,
Weight Decay
0.001
0.005
0.0001
0.0001
0.001
0.008
0.001
0,0005
Batch Size
128
128
128
17

---

E.2
MobileNet
Table F.3: Hyperparameters for MobileNet with Tanh
Convention
Swap
CIFAR-10
CIFAR-100
Thny ImageNet
ImageNNt
CIFAR-10
CIFAR-100
Thy ImageNet
ImageNet
Training Epochs
200
2001
200
200
200
2001
90
Learning Rate
Learning Ratc Drop
100, 150
100, 150
100, 150
30, 60
100, 150
100. 150
100, 150
30, 60
Weight Decay
0.0001
0,90015
0.005
0.0001
0.0001
0,0005
0.005
0.0001
Table F.6: Hyperparameters for MobileNet with the Shifted Tanh
Convention
Swap
CIFAR-10
CIFAR-100
Thyy ImageNNet
ImageNet
CIFAR-10
CIFAR-100
Thy ImageNet
ImageNet
Training Epochs
200
200
200
90
Learning Rate
0.1
0.1
0.01
0.1
0.1
0.1
0.01
0.01
Learnig Rate
100, 150
100, 150
100, 150
30, 60
100, 150
100, 150
100, 150
30,
Weight Decay
0.0001
0,0005
0.005
0.0001
0,0005
0.0005
0.005
0.0005
Ratch Size
128
128
128
256
128
128
128
256
Table F.7: Hyperparameters for MobileNet with ReLU
Convention
Swap
Tiny ImageNet
ImageNet
Tiny ImageNet
ImugeNet
Training Epochs
200
200
200
200
200
200
90
Learning
Leaming Rate Drop
100, 150
100, 150
100, 150
30, 60
100. 150
100, 150
100, 150
30,
Weight Decay
0.001
0,005
0,0001
0.0001
0.001
0.005
0.001
0.0001
18

---

E.3 PreAct-ResNet
Table F.8: Hyperparameters for & and with Tanh. PreAct-ResNet18 for the CIFAR dataset, and PreAct-ResNec14 is for the Tiny ImageNet dataset.
Convention
Swap
CIFAR-10
CIFAR-100
Tiny ImageNet
CIFAR-10
CIFAR-100
Tiny Image.Net
Training Epochs
200
200
Learning Rate
Learning Rate Drop
100, 150
100, 150
100, 150
100. 150
100, 150
100. 150
Weight Decay
0.0001
0.0001
0.0001
0.0005
0.0005
0.0005
Batch Size
128
128
128
128
128
128
Table F:9: Hyperparameters for PreAct-ResNer18 and PreAct-ResNer34 with the Shifted Tanh. PreAct-ResNet18 is for the CIFAR dataset, and PreAct-ResNNtke is for the Tiny ImageNet dataset.
Convention
Swap
CIFAR-10
CIFAR-100
Tiny ImageNet
CIFAR-10
CIFAR-100
Tiny ImageNet
Training Epochs
200
200
200
200
200
200
Learning Rate
0.1
0.1
0.01
0.1
0.1
0.1
Learning Rate Drop
100, 150
100, 150
100, 150
100, 150
100, 150
100, 150
Weight Decay
0.0001
0.0005
0.001
0.0005
0,0005
0,0005
Batch Size
128
128
128
128
128
128
Table F.10: Hyperparameters for PreAct-ResNer18 and PreAct-ResNet34 with ReLU. PreAct- 8 is for the CIFAR dataset, and PreAcc-ResNer34 is for the Tiny ImageNet dataset.
Convention
Swap
CIFAR-10
CIFAR-100
Tiny ImageNet
CIFAR-10
CIFAR-100
Tiny ImageNet
Training Epochs
200
200
Learning Rate
0.01
Learning Rate Drop
100, 150
100. 150
100, 150
100, 150
100. 150
100. 150
Weight Decay
0.005
0.005
0.0005
0.005
0.005
0,0005
Batch Size
128
128
128
128
128
128
19


Reviews:
[{"review_id": "FQRxonhPzg", "reviewer": "Reviewer_yn6T", "summary": "The paper investigate the order of Batch Normalization and activation functions, and founded bounded activation functions like Tanh works better in the swapped order unlike bounded one like ReLU. To explain this, the authors analyze the asymmetric saturation levels at both the layer and channel levels and find that the Swap model has higher saturation levels, especially in layers with excessive saturation. They also introduce a new model, NWDBN, that encourages asymmetric saturation and show that it improves accuracy compared to the convention model. The paper concludes that asymmetric saturation can help improve performance in neural networks.", "questions": "- The paper did not explain well why higher sparsity induced by asymmetric saturation would booster the performance.\n- Is there a way to control Asymmetric Saturation to archieve trade-off between sparsity and performance?", "limitations": "As stated in Questions.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper revolve around the hypothesis that \"asymmetric saturation helps improve performance\", and analyize that ReLU and Tanh bring different asymmetric saturation levels, this sound interesting to me.\n- The paper identify the high sparsity induced by Batch Normalization after bounded activation is functions and validate that the higher sparsity induced would booster the performance.", "weaknesses": "- The logic chain of the story in this paper need some improvement, see more in Questions part.\n- The input of convolution layer in Figure 1 upper and lower part seem different, can the authoer explain why?\n- Why is x-axis in Figure 1 lower-right BN figure of range (-5, 2.5), while I didn't see any value in the range of (-5, -1), this plot could be misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and valuable suggestions.\n\n**Weakness 1.**\n\n**Question 1.**\nThe robustness of the noise obtained through the sparsity results in improved performance. Sparse representation indicates that relatively few units represent the data sample. Thus, even if the perturbation is added to the input, this noise less affects sparse features. \nThe Swap model has strength on sparsity compared to the Convention model. In Section 3.3, we verified the robustness of the Swap model on the corrupted dataset. The increased sparsity of the Swap model outperforms the Convention model. \n\n**Question 2.**\nThank you for the insightful question. However, it is not easy to completely isolate only asymmetry to achieve trade-offs between sparsity and performance.\n\n**Weakness 2.**\nFor the Swap model, the mean of convolution layer output needs to shift substantially away from 0 to achieve asymmetry on Tanh. Consequently, this results in a distribution that is biased in one direction.\n\nConversely, in the Convention model, there is no reason to shift the mean of convolution output. The normalization of BN adjusts the mean of the convolution layer output to zero. This normalization process can lead to the output of the convolution layer non-biased.\n\n\n**Weakness 3.** \nThe subplot shows most samples near 0, with a few outliers like -5, remarking the Swap model's high sparsity. Due to the line's thinness, such values may be hard to discern. \nWe will enhance Figure 1 for a more clear representation.\n"}, {"review_id": "HPUNikERhe", "reviewer": "Reviewer_AMuf", "summary": "This paper investigates neural network classifiers with bounded activation functions. The authors first observe that swapping the batch norm and activation order improves performance with bounded activation functions. They then observe that asymmetric saturation and sparsity occurs in the swap model compared to convention, and show how it correlates with accuracy. They then propose a modified activation function that promotes asymmetric saturation and shows that even in convention order, it benefits performance. ", "questions": "See weaknesses.", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper notes an interesting observation regarding the swap model with bounded activation\n- It supports the hypothesis that asymmetric saturation and sparsity benefit benefit. This is achieved by caring the weight decay on the batch norm layers.\n- The comparable performance with the modified tanh activation with relu is quite interesting. I wonder if ReLU-like behavior is what is the best for performance or if there is room for improvement past this. ", "weaknesses": "- The evaluation is somewhat limited, since it is only done on VGG on image classification tasks. It’s not clear if these results would also apply to different architectures, e.g. transformer-based ones, or on different tasks such as segmentation or text classification.\n- Comparison with different ReLUs, such as LeakyRELU or ELU. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and valuable suggestions.\n\n**Weakness 2**\nThe table below presents the accuracy of VGG16 models trained on CIFAR-100. The accuracy of both leaky ReLU and ELU is the best hyperparameter, and the accuracies are averaged over three seeds.\n\n|              | Convention |   Swap   |\n|:------------:|:----------:|:--------:|\n|     ReLU     |    73.68   |   71.79  |\n|  leaky ReLU  |    74.75   |   73.61  |\n|      ELU     |    69.85   |   72.27  |\n|     Tanh     |    64.84   |   72.17  |\n| Shifted Tanh |    73.87   |   73.21  | \n\nThe shifted Tanh model with Convention outperforms the others except for the leaky ReLU model with Convention.\n\nInterestingly, the ELU model, like Tanh, exhibits enhanced performance in the Swap order, reinforcing our assertion that \"asymmetry enhances performance.\"\n\nGiven ELU's asymptote away from the x-axis, its capability to establish asymmetry is constrained in the Convention order due to BN's zero-mean output. However, the Swap order can amplify its asymmetry, leading to performance boosts in multiple models, not just VGG.\n\n|              | Convention |   Swap   |\n|:------------:|:----------:|:--------:|\n|  MobileNet   |    68.72   |   70.26  |\n|PreAct-ResNet18|    74.64   |   75.6  |\n\n"}, {"review_id": "akW2kzjDJQ", "reviewer": "Reviewer_s3KQ", "summary": "This paper investigates the performance of different activation function orders in deep learning models with batch normalization. The authors focus on the conventional order, where batch normalization is placed before the activation function, and the swapped order, where batch normalization is placed after the activation function. Surprisingly, they find that the swapped order achieves significantly better performance than the conventional order when using bounded activation functions like Tanh. The paper provides a thorough analysis of the underlying mechanisms and presents empirical evidence to support their findings.\n\n", "questions": "no questions ", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "\n- Novelty: The paper explores an interesting, and up to my knowledge previously overlooked aspect of activation function order in the context of batch normalization. The findings challenge the conventional wisdom and offer a new perspective on designing deep learning models.\n\n- Comprehensive experiments: The authors conduct extensive experiments and carefully examine the output distributions of individual activation functions. Their investigation into the asymmetric saturation phenomenon provides lots of intuitions into the behavior of bounded activation functions that make intuitive sense. \n\n- While focusing on Tanh as the primary activation function, the authors demonstrate that their findings are applicable to similar antisymmetric and bounded activation functions. \n\n- Performance Improvement: The swapped order, combined with bounded activation functions and batch normalization, consistently outperforms the conventional order across various benchmarks and architectures. The results highlight the potential for achieving superior performance by exploiting the benefits of asymmetry and sparsity.\n\n- Quality of presentation of ideas: the paper is very well written and gives the ideas in a clear and easy to follow manner. The contributions are clearly stated and there are no over claims in the text. \n", "weaknesses": "*Limited scope*\nMy main concern with this paper is that the limited scope of the network configurations considered for the experiments. For example, the results presented in Table 1 shows that conventional models with ReLU are mostly equal or better than all swap models, as well as the ReLU activation with swap order. Can authors make any further comments on possible reasons for this? For example, one might discount the strength of the main empirical evidence that is presented (differential performance between swapped & conventional models for Tanh & Tanh shifted), is only observed for the sub-optimal models and not the original model. While this does not directly contradict the main message of the paper, it significantly weakens its potential impact and scope. If the authors can address these questions by further experiments this could potentially strengthen the main conclusions made in the paper. \n\nOn a similar note, given the central hypothesis that asserts \"The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance, \" can this hypothesis explain the differential performance between ReLU & Tanh too? Can this hypothesis explain the difference & benefit of having BN layers at all? (if you remove them, you don't have the asymmetric saturation ? There seems to be plenty of adjacent configurations that can be added to expand on the generalizability of this central idea of the paper.  These types of additional experiments could strengthen the study's conclusions. \n\n*Lack of Theoretical Analysis* Although the paper presents compelling empirical evidence, a deeper theoretical analysis would enhance the understanding of why and how the swapped order, asymmetry, and sparsity contribute to performance improvement. Incorporating theoretical insights could strengthen the paper's contribution to the field. \n\n*Figures & tables:* One minor but important issue: The tables and figures currently lack confidence intervals or standard deviations \n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and valuable suggestions.\n\n**Weakness: Limited scope**\n\n- Possible reasons for the Convention ReLU performance\n\nAs the original author of BN[1] suggested, Convention seems to be fundamentally better than Swap. The good performance of Convention ReLU derives from these advantages. However, in the case of Tanh, the asymmetry caused by the Swap overcomes the disappeared advantages of the BN in the Convention order. We can confirm that Convention is better in the shifted Tanh, which can produce asymmetry even without a Swap. \n\n- Differential performance between swapped & conventional models\n\nWould you be kind enough to clarify what you mean by \"sub-optimal model\"? For reference, the VGG used in Table 1 is not VGG11 but VGG16. Only VGG16 has only BN added to the original model in that table, while all other models remain in their original versions.\n\n- Explanation of the differential performance between ReLU & Tanh\n\nWe believe it's possible. We assume that the primary reason Tanh underperforms compared to ReLU in general cases is due to asymmetry. Therefore, when we tested with Swap or shifted Tanh to induce asymmetry, we observed a performance similar to ReLU's. From this, we deduce that the reason for the lower performance of Tanh compared to ReLU was its asymmetry.\n\n - The Difference and benefits of having BN\n\nThere are various roles of BN, and there have been many studies on it. This paper focuses on analyzing the correlation between Tanh and BN rather than showing the general advantages of BN. Nevertheless, we assume that BN is needed to complement the low sparsity caused by the asymmetry of Tanh. \n\nIn the case of Tanh, BN increases the sparsity of the block output in the Swap structure, which helps improve the performance of the model, especially when it is combined with asymmetry.\n\nThe layer-wise Skewness and Sparsity of the VGG16_11 without BN, which is called the NoBN model, and the Swap model using Tanh can be seen in Figure 2 in the PDF file. \n\nIn the absence of BN, increasing the weight size can indeed produce asymmetry. However, the degree of skewness observed isn't as prominent as that in the Swap model. We interpret this subdued asymmetry in the NoBN model as a result of reduced sparsity. This limited generation of strong asymmetry could potentially contribute to a decline in performance.\n\nOn the other hand, in the Swap structure, even if a high asymmetry is generated due to the BN located behind, it can create a high sparsity. \n\n**Weakness: Figures & tables**\n\nWe have added confidence intervals for Figure 1 to the PDF. \u001cWe will update the standard deviation for Table 1 in the main paper.\n\n[1] Ioffe, Sergey, and Christian Szegedy. \"Batch normalization: Accelerating deep network training by reducing internal covariate shift.\" International conference on machine learning. pmlr, 2015."}, {"review_id": "lpwRbO06vw", "reviewer": "Reviewer_hcmY", "summary": "This paper proves the hypothesis that asymmetric saturation benefits network performance by swapping the position of Batch Normalization and Tanh activation functions. The Swap model generates high sparsity and asymmetric saturation which enables Tanh to behave like a one-sided activation function. Experimental results show the asymmetric distributions consistently outperforms the symmetric ones. However, because the BN and Tanh combination hardly appears in modern networks and there is no convincing theory facilitating network architecture design, the contribution of this paper is of less significance.", "questions": "1）The paper discusses sparsity and asymmetry, which have distinct mathematical definitions. However, it remains unclear which factor is the key determinant of network performance.\n2） Whether the proposed swap model effectively addresses the issue of vanishing gradients?\n", "limitations": "The authors did not discuss Limitations in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1) This paper is easy to follow.\n2) This paper provides a comprehensive experimental demonstration to validate asymmetric activation functions are superior to symmetric ones. This implies that ReLU-like activation functions are better than tanh activation functions.\n", "weaknesses": "1) One major concern is the lack of prevalence of the BN and Tanh combination in modern networks. As a result, the analysis and experiments conducted in this study hold limited empirical significance.\n2) The benefits of asymmetric activation functions have previously been demonstrated from the perspectives of gradient [1] and expressivity [2]. However, this paper fails to contribute new explanations or adequately discuss the limitations of existing works.\n3) The experiments in this paper are restricted to a limited range of network architectures. For example, even the widely used ResNet18 and ResNet50 models have not been evaluated on the ImageNet dataset. \n4) A noticeable error can be identified in Figure 1, where the right bottom subplot indicates that 'BN' exhibits lower sparsity compared to 'Tanh,' contradicting the accompanying label stating 'High Sparsity.'\n\n[1] Maas, A. L., Hannun, A. Y., & Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. 2013.\n[2] Hanin, Boris, and David Rolnick. Complexity of linear regions in deep networks. 2019.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and valuable suggestions.\n\n**Weakness 1.**\nOur main contribution lies beyond the combination of BN and Tanh. We have delved into understanding the significance of asymmetry in the activation functions, using Tanh as a base. This analysis can offer insights for future research on activation function design and interpretation.\n\nThe derived insights can be applied to other functions, like ELU, which shares structural similarities with Tanh. ELU has the asymptote away from the x-axis. Thus, when ELU is used in the Convention order, its ability to generate asymmetry is hindered due to BN's zero-centered output. This imposed limitation on asymmetry drops the model accuracy. The accuracy of VGG16_11 trained by CIFAR-100 with ELU and other activation functions (Tanh, ReLU, leaky ReLU) can be seen in Table 1 in the PDF. \n\nNotably, Tanh and ELU outperform in the Swap order compared to the Convention order. This remarks our observation that restricted asymmetry degrades the accuracy. This enhanced performance can be seen in different models, not just VGG. Table 2 in the PDF file shows the accuracy of the ELU model in various settings. \n\nMoreover, shifted Tanh remains effective even in non-BN settings. Examining Convention and Swap orders, we discerned the importance of asymmetry and sparsity. Such understandings hold potential for performance enhancements, regardless of BN's presence. Notably, in the BN-free VGG16_11 model, shifted Tanh outperforms the Tanh, with an accuracy of 92.27% against Tanh's 89.68%.\n\n**Weakness 2.**\nAs far as we know, no previous works discuss the advantages of asymmetry of the activation function.\nOur work differs in a critical aspect: asymmetric saturation. Specifically:\n - [1] investigates gradient flow in the linear region of the ReLU in deep networks.\n - [2] delves into the expressivity stemming from the linear regions formed by piecewise linear functions.\n\nThese studies primarily focus on the linear aspects and not on asymmetric saturation.\nTo illustrate further, [1] values the \"sparse-dispersed code\", where a few coding units are active at a given moment for an image, but all units contribute equally to coding over their lifetime. While this shared the sparsity perspective with ours, it is distinct from our main finding: asymmetric saturation helps improve performance.\n\nThus, our paper sheds light on the benefits of asymmetric saturation, offering a novel perspective on the activation function.\n\n**Weakness 3.**\nWe recognize the widespread use of ResNet architectures. However, they aren't suitable for our study, which focuses on analyzing the performance difference in the layer order.\n\nIn the specific residual blocks of ResNet, BN exists in the skip connection. Thus, the number of BN and activation functions are different. This results in the Swap model having as many activation functions as BNs in the skip-connection, complicating a direct comparison with the Convention model.\n\nNevertheless, the results of the ResNet model with Tanh on ImageNet and CIfar-100 are below. The ResNet-18 was conducted at one hyper-parameter (lr 0.1, wd 0.0001) due to limited time, and ResNet-20 for the CIFAR-100 is the best hyper-parameter accuracy.\n\n|              | Convention |   Swap   |\n|:------------:|:----------:|:--------:|\n|  ResNet-18 (ImageNet)   |    63.08   |   69.96  |\n|ResNet-20 (CIFAR-100)|    68.97   |   69.06  |\n\nSwap shows improved performance compared to Convention.\n\nAdditionally, we extended our evaluation to PreAct-ResNet50 with Tanh on ImageNet. PreAct-ResNet, different from ResNet, lacks BN in its skip connection. This means that when the Swap order is employed, it does not introduce the challenges seen in ResNet. Our results with PreAct-ResNet50, at the best hyper-parameter. (Convention: 62.95, Swap: 72.82) The Swap model also outperforms the Convention model.\n\n**Weakness 4.**\nThe sparse distribution refers to a state in which a small number of samples have large values.\nThe right bottom subplot shows a state where most have a value of 0 and a small number of samples have a large value (e.g., -5).\nThe averaged Sparsity over layers on Convention and Swap for VGG11 is as follows. (Convention: 0.718, Swap: 0.849)\n\nThe lines in the subplot are thin and do not seem visible.\nWe will update a figure that can add the existence range of values later.\n\n**Question 1.** \nIt is not easy to control asymmetry and sparsity completely independently. However, we assume that asymmetry is more important in general. This can be confirmed by comparing the asymmetry and sparsity between the NWDBN and Convention models.\n\nWhen using NWDBN, it was not shown because the sparsity metric was not introduced. While Convention shows an average layer Skewness of 0.254 and Sparsity of 0.718, NWDBN presents a pronounced Skewness of 0.718 and a lower Sparsity of 0.288. Even with reduced Sparsity, NWDBN surpasses Convention as asymmetry increases.\n\n**Question 2.**\nThe vanishing gradients problem is inevitable when using Tanh with excessive saturation. However, BN in the Swap order somewhat alleviates the vanishing gradients caused by asymmetric saturation in two aspects. \n\n1. the scaling effect on the gradient by normalization, specifically $\\sigma$ in BN. When asymmetric saturation occurs in Tanh, the $\\sigma$ becomes smaller. The normalization in the forward pass is $(x-\\mu)/\\sigma=z$. And it becomes $dL/dx=1/\\sigma*dL/dz$, which has the effect of scaling due to the $1/\\sigma$.\n\n2. the unboundedness of the convolution input. It can calculate a large weight gradient. In the Swap model, the convolution layer gets its input from BN, unlike the Convention model, which uses Tanh's output. When calculating the weight gradient of convolution weight, unbounded large input can create a larger gradient than Convention. \n\nIn this respect, the Swap model can exhibits performs well across various settings even with asymmetric saturation.\n"}]

Author rebuttal:
We thank the reviewers for their positive feedback and valuable suggestions.

Meta-review: This paper studies different activation function orders the context of batch normalization. It investigates the setting where batch normalization is placed before the activation function, and the swapped order, and find empirically that the swapping leads to improved results for bounded activation functions, eg. tanh. The underlying mechanisms are analyzed, showing that asymmetric saturation benefits network performance by generating sparser activations. While this is theoretically interesting, one point of criticism from the reviewers is the limited practical significance of the work, because, the combination of BatchNorm and tanh is barely used in SotA models. Yet, reviewers and AC agree that the observation made is interesting. The authors are encouraged to include the additionally provided results on models beyond VGG into the camera ready version.
Decision: Accept (poster)