Title: Tanh Works Better with Asymmetry
Venue: NeurIPS 2023
Accepted: yes
Keywords: Batch Normalization

Activation Functions

Saturation

Sparsity
Abstract: Batch Normalization is commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e., using Batch Normalization after activation functions, has also been attempted, but its performance is generally not much different from the conventional order when ReLU or a similar activation function is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance than the conventional order on various benchmarks and architectures. This paper reports this remarkable phenomenon and closely examines what contributes to this performance improvement. By looking at the output distributions of individual activation functions, not the whole layers, we found that many of them are asymmetrically saturated. The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance. In addition, Batch Normalization after bounded activation functions relocates the asymmetrically saturated output of activation functions near zero, enabling the swapped model to have high sparsity, further improving performance. Extensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation.  Finally, based on this investigation, we test a Tanh function shifted to be asymmetric. This shifted Tanh function that is manipulated to have consistent asymmetry shows even higher accuracy than the original Tanh used in the swapped order, confirming the asymmetry's importance. The code is available at https://github.com/hipros/tanh_works_better_with_asymmetry.

Tanh Works Better With Asymmetry

Abstract

Batch Normalization is commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e., using Batch Normalization after activation functions, has also been attempted, but its performance is generally not much different from the conventional order when ReLU or a similar activation function is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance than the conventional order on various benchmarks and architectures. This paper reports this remarkable phenomenon and closely examines what contributes to this performance improvement. By looking at the output distributions of individual activation functions, not the whole layers, we found that many of them are asymmetrically saturated. The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance . In addition, Batch Normalization after bounded activation functions relocates the asymmetrically saturated output of activation functions near zero, enabling the swapped model to have high sparsity, further improving performance. Extensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation. Finally, based on this investigation, we test a Tanh function shifted to be asymmetric. This shifted Tanh function that is manipulated to have consistent asymmetry shows even higher accuracy than the original Tanh used in the swapped order, confirming the asymmetry's importance. The code is available at https://github.com/hipros/tanh_works_better_with_asymmetry .

1 Introduction

Batch Normalization (BN) is a widely used technique in deep learning. It was proposed to address the internal covariate shift problem by maintaining a stable output distribution among layers. As the characteristics of the output distribution of the weighted summation operation tend to be symmetric, non-sparse, and Gaussian [10], Ioffe & Szegedy [11] placed the BN between the weight and activation function. Thus, the 'weight-BN-activation' order, which we call 'Convention' in this paper, has been widely used to construct a block in many architectures [18, 8]. 'Swap' models, swapping the order of BN and the activation function in a block, have also been attempted, but no significant and consistent difference between the two orders has been observed in the case of ReLU or similarly shaped activation functions. For instance, Hasani & Khotanlou [6] evaluated the effect of the position of BN in terms of training speed and concluded that there is no clear winner and the result depends on the datasets and architecture types.

BN

BN

Convention

Swap

-1.5   -1.5    -0.5

0.0

+0.5   +1.0

-1.0        -0.5

0.0

+0.5       +1.0

-2.0       -1.0

0.0

+1.0

-5.0

0.0

+5.0

-1.0        -0.

0.0

+0.5        +1.0

Layer output

Layer output

Layer output

Layer output

Layer output

Layer output

Output range

-5.0 -4.0 -3.0  -2.0 -1.0

0.0 +

1.0 +2.0

Convolution

Convolution

However, in the case of bounded activation functions, we empirically found that the Swap order exhibits substantial improvements in test accuracy than the Convention order with diverse architectures and datasets. This paper empirically investigates the reason for this accuracy difference between the Convention and the Swap models with bounded activation functions. For simplicity, our analyses are mainly conducted with Tanh but are applicable to similar antisymmetric and bounded activation functions. We present the results with LeCun Tanh and Softsign at the end of the experimental section.

One key difference between Swap and Convention models is the distribution of activation values, as shown in Figure 1(left). In the Swap model, the mean of convolution layer output shifts substantially away from 0. Thus, most activation values are near the asymptotic values of the bounded activation function, that is, highly saturated. This is unanticipated since it is a common belief that high saturation should be avoided. To investigate this paradox, we took one step further and looked at the output distribution of each activation function, not just a whole layer. To our very surprise, even though the distribution is fairly symmetric at the layer level, the activation values of each channel are biased toward either one of the asymptotic values or asymmetrically saturated . Through an in-depth investigation, we come to a hypothesis that this asymmetric saturation is a key factor for the performance improvement of the Swap model since it enables Tanh to behave like a one-sided activation function. In the experiments to examine whether asymmetric saturation is related to the performance of models, we observe that the accuracy and the degree of asymmetric saturation are highly correlated.

BN after Tanh also shifts the distribution biased toward either -1 or 1 by asymmetric saturation near zero, which has the important effect of increasing sparsity, as depicted in Figure 1(right). Sparsity is widely considered to be a desirable property. For instance, Glorot et al. [5] studied the benefits of ReLU compared to Tanh in terms of sparsity. One thing to note is that if each channel is symmetrically saturated, BN will not increase sparsity much since the mean is already close to zero. In contrast, the one-sided property of asymmetric saturation causes at least half of the sample values after normalization to be almost zero. It allows the Swap model to have even higher sparsity than the Convention model. Asymmetric saturation combined with normalization makes a bounded activation function behave much like ReLU.

Based on the above lessons, we test enforcing the asymmetric saturation and sparsity of Tanh outputs. This involves shifting the inputs of Tanh by τ , introducing asymmetry, and then shifting the outputs in reverse by -tanh ( τ ) to boost sparsity. This is equivalent to using a shifted Tanh as a new activation function. Our experimentation across different architectures and datasets reveals that this adjusted

Tanh function with fixed asymmetry consistently outperforms the original Tanh used in the Swap order. Its effectiveness is even comparable to that of ReLU. This validates the hypothesis that Tanh works more efficiently with asymmetric saturation.

Our findings are as follows:

The Swap model using Batch Normalization after bounded activation functions performs better than the Convention model in many architectures and datasets.

We discover the asymmetric saturation at the channel level and investigate its importance through carefully designed experiments.

We identify the high sparsity induced by Batch Normalization after bounded activation functions and perform an experiment to examine the impact of sparsity on performance.

Our shifted Tanh with enforced asymmetric saturation shows higher accuracy than the Swap Tanh, which reconfirms the importance of asymmetry.

2 Settings and Notation

Model For our investigation, we choose the VGG-like model due to its simple and straightforward design. This model is trained on the CIFAR-100 dataset [12]. As a modification, we replace the ReLU activation function with Tanh. However, because the VGG architecture was proposed for the ImageNet dataset, the model is overparameterized for the CIFAR dataset. It incurs poor performance and difficulty in investigating the Swap model. Thus, we cut out the last convolution layers and select the best model based on the validation accuracy. The model with five cut-out layers shows the best accuracy, as in the Appendix. We call this model 'VGG16_11' and use this architecture to investigate Convention and Swap orders.

Metric We consider three properties to investigate each order: saturation, asymmetric saturation, and sparsity. Saturation is measured on Tanh and quantifies how closely outputs the values to the maximum absolute output range limit, also known as the asymptotic value. We involve the skewness metric in statistics to measure asymmetry on Tanh. One thing that differs from the skewness is the absolute. We absolute the skewness to quantify the asymmetry regardless of skewed direction. Sparsity is measured on the output of the block, i.e., Tanh for Convention and BN for Swap. To measure the sparsity, we inversely utilize our saturation metric.

Experiment setup In Section 4.2, we verify the effect of asymmetry by slightly cutting off the asymmetry in the asymmetry-boosted Convention model, which we refer to the No-Weight-Decayon-BN (NWDBN) model. The NWDBN model is ordered as Convention, and the weight decay does not apply to BN. We fix the intensity of the weight decay on the weight layer and vary the decay intensity on BN to regulate asymmetry. The initial hyperparameters followed the best hyperparameter of the NWDBN model. Then, we increase the intensity of weight decay on β in BN from 0.0 to 0.001 by 0.0001. In Section 5.3, we investigate the effect of sparsity by slightly increasing it. The initial hyperparameters followed the Swap model's best hyperparameters. Then, we change the weight decay intensity on the γ and β in BN. The searched intensities of weight decay are 0, 1e-6, 5e-6, 1e-5, 5e-5, 1e-4, and 5e-4. We verify the effects of the layer order in various datasets, networks, and activation functions in Section 7.1. We train models on four benchmarks (CIFAR-10, CIFAR-100, Tiny ImageNet [13], and ImageNet [3]), three base-architectures (VGG, MobileNet, PreAct-ResNet), and three activation functions (ReLU, Tanh, ShiftedTanh). For the ResNet, the Swap order incurs extra non-linearity functions, which hinder the fair comparison between the Convention and Swap model. Except for the first stage, the first shortcut connection in each stage has convolution and BN layers in the block. In the case of the Swap Tanh model, Tanh replaces BN and vice-versa. Thus, the number of non-linearity functions increases as the number of the linear projection layer in the shortcut. We, therefore, report the results of PreAct-ResNet, which only has the convolution layer in the shortcut connection. (Furthermore, upon training ResNet20 with Tanh on the CIFAR-100 dataset, we observed that the Swap model surpassed the Convention model, achieving an accuracy of 69.06% compared to the Convention model's 68.97%.)

All results, except the ImageNet dataset, are conducted on three random seeds. The measured values and accuracy are averaged over seeds. We use the SGD optimizer with a momentum of 0.9, weight

decay. We adopt a 2-step learning rate decaying strategy that decays by 0.1. We conduct a grid search to obtain the best model for investigation. We search different combinations of initial learning rates (0.1 and 0.01) and weight decay values (1e-4, 5e-4, 1e-3, and 5e-3). The specific hyperparameters for each model can be found in the Appendix.

Notation The input of an arbitrary layer in the convolution neural network is denoted as X ∈ R n × c × h × w , where n , c , h , and w represent the number of samples, channels, height, and width, respectively. We reshape X to X ∈ R c × m , where m = nhw . The output of the Tanh function is T ∈ R c × m where T i,j = tanh ( X i,j ) . BN normalizes the channel distribution to have zero mean and unit variance using estimated statistics. After normalization, an affine transformation with scaling parameter γ ∈ R c and shifting parameter β ∈ R c is applied channel-wise. Thus, the output of

BN is given by: B i,j = γ i ( X i,j -ˆ µ i √ ˆ σ 2 i + ϵ ) + β i , where ϵ is a constant added for numerical stability, ˆ µ i and ˆ σ i are estimated mean and standard deviation of each channel, respectively, computed over the mini-batch of training data samples.

3 Overly Saturated but Well-Generalized Model

The phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function [15]. When training a neural network with bounded activation functions, the outputs of neurons increase due to the gradually increasing weights [4]. The increased output values become close to the near asymptote in bounded activation functions, as shown in the experiment in Glorot & Bengio [4]. However, excessive saturation damages the neural network, and it is regarded as a situation to evade [4, 15, 16, 1, 2, 11]. Various methods were proposed to prevent excessive saturation. Glorot & Bengio [4] proposed an initialization scheme, Rakitianskaia & Engelbrecht [15, 16] proposed a metric to measure the degree of saturation for monitoring the training, Bhat et al. [1] pre-scaled the inputs of the activation function, and Chen & Chang [2] proposed adaptable bounded activation.

3.1 Saturation Metric

Our saturation metric quantifies the extent to which the output values approach the function's output range limit, also known as the asymptotic value. To normalize the input, we calculate the absolute value and divide it by the maximum absolute value, resulting in a range of [0, 1]. The saturation t of a layer is defined as:

where a represents the maximum absolute value of the output range of the activation function. Specifically, for the Tanh function, a is equal to 1. t lies in [0 , 1) . The large value of this metric indicates T are highly saturated, as illustrated in the Appendix.

3.2 High Saturation in the Swap Model

Even if only the layer order is changed from the Convention order to the Swap order, there is a 4.61% improvement in test accuracy. However, when we measure the layer saturation in both models, the Swap model has highly saturated layers. The layer-wise Saturation of Swap and Convention models can be seen in Figure 2(a). This is counterintuitive as excessive saturation is considered undesirable in previous works [4, 15, 16, 1, 2]. The saturation of almost half of the layers in the Swap model is over 0.7. Some layers even have a saturation over 0.8. On the other hand, the saturation of the Convention model is lower than 0.5 over all layers.

4 Asymmetric Saturation

In order to scrutinize saturation, we perform an in-depth analysis at the channel level, one step further than the layer level. Fascinatingly, we notice that most channels display asymmetric saturation in the Swap model, especially in layers with excessive saturation. On the other hand, the channel distribution in the Convention model tends to be centered around zero, even at the channel level.

1

2

3

4

5

6

7

8

9

Block depth

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Saturation

Convention

Swap

1

2

3

4

5

6

7

8

9

Block depth

0.0

1.0

2.0

3.0

4.0

5.0

Skewness

Convention

Swap

4.1 Asymmetric Saturation Metric

Both the Convention and Swap models demonstrate symmetric distributions at the layer level. However, notable differences arise at the channel level. To quantify and compare the degree of asymmetry present at the channel level, we measure the asymmetry of each channel individually and calculate the average to determine the overall skewness of the layer. We adopt sample skewness as a metric to gauge the asymmetry of the Tanh output. The skewness of a given i th channel, denoted p i , is calculated as follows:

where µ 3 i represents the sample third central moment of T i, : , while σ 3 i represents the sample standard deviation cubed of T i, : . The averaged absolute skewness of a layer, k , herein referred to simply as Skewness, is given by:

Figure 2(b) illustrates the Skewness for all layers in the Convention and Swap models. In the Convention model, the Skewness of most layers is close to zero, indicating a minimal presence of asymmetric distribution. Conversely, the Swap model exhibits higher Skewness compared to the Convention model. Moreover, the regions with high saturation blocks within the Swap model show particularly elevated Skewness values. This suggests that saturation in these areas occurs asymmetrically in the channel distributions.

4.2 Effect of Asymmetric Saturation on Performance

In order to examine the effect of asymmetric saturation, we devise a method to control the level of asymmetry in the Convention model. One key factor that limits the emergence of asymmetric saturation in the Convention model is the influence of weight decay on γ and β within BN. Especially, small β values by weight decay constrain the distribution to be zero-centered, which incurs a decrease in the asymmetry of the Tanh output. In our experiment, the NWDBN model successfully encourages asymmetric saturation and has higher Skewness than the Convention model. Moreover, the NWDBN model shows improved accuracy of 72.22% compared to the Convention model 69.5% . To better examine

0.0

1e-04 2e-04 3e-04 4e-04 5e-04 6e-04 7e-04 8e-04 9e-04 1e-03

Intensity of Weight decay on Beta

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Avg.Skewness

69.5

70

70.5

71

71.5

72

72.5

Avg.Skewness

the effects of asymmetric saturation on performance, we gradually increase the intensity of weight

10

5

0

5

10

x

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

Nor

,

(

(

x

))

Nor

,

(

(

x

)),

x

( ,

)

+0.92,

0.18

0.85,

0.27

Nor

,

(

(

x

))

= +0,  =1

+0.92,

0.18

0.85,

0.27

decay on the β parameter from zero, which indicates that slightly eliminates the asymmetry in the NWDBN model. As the weight decay on β gets bigger, Skewness becomes smaller, and the test accuracy degrades, as shown in Figure 3. This strong correlation clearly supports our hypothesis: asymmetric saturation helps improve performance .

5 Sparsity

5.1 Asymmetric Saturation with Batch Normalization Can Induce High Sparsity

ReLU achieves a high generalization performance by utilizing the strengths of sparsity [5]. The sparsity introduced by ReLU is a consequence of its single boundary at zero, which transforms all negative inputs to zero. In addition, work by Ramachandran et al. [17] shows the benefits of having a single asymptote at zero. They carried out an automated strategy to investigate a variety of activation functions. The most notable activation functions discovered through this search were one-sided and had a boundary value close to zero, similar to ReLU.

We note that the Swap model effectively generates sparsity, pushing a majority of values toward zero when asymmetric saturation occurs. This shifting is credited to the normalization process in Batch Normalization (BN), which recenters the distribution around zero. Thus, when the asymmetric saturation occurs, this saturation is subsequently shifted towards zero during the normalization process, enhancing sparsity. The combined effect of Tanh with normalization, along with samples of the function's output showing different BN statistics in the Swap order, are depicted in Figure 4. The majority of values and the asymptotic lines that the samples saturated are shifted to near zero.

5.2 Sparsity Metric

Intuitively, a sparse representation is one in which a small number of coefficients contain a large proportion of the energy [9]. We utilize our saturation metric to measure sparsity. Saturation quantifies the extent to which many values reach the maximum. We measure the sparsity by the extent to which many values are far from the maximum. However, as sparsity is measured on the block output, we consider the output of Tanh for the Convention model and the output of BN for the Swap model. BN has no boundaries, so we empirically calculate a for normalization in Equation 1. We denote the empirical saturation, ˆ t , as follows:

where B max i, : is the maximum absolute element in B i, : , i.e., B max+ i, : = max j B i,j . For the Convention model, we apply the above metric to T instead of B . Finally, the sparsity s is defined as:

Our sparsity metric satisfies five criteria among six desired criteria of sparsity measures in Hurley & Rickard [9]. The proof of each criterion can be found in the Appendix.

5.3 Effect of Sparsity on Performance

In this section, we investigate the significance of sparsity in the Swap model and its influence on test accuracy. There's a marked relationship between increased sparsity and test accuracy. The result can be seen in Figure 5.

As elaborated in Section 5.1, the Swap order has the capability to intensify sparsity when there's asymmetric saturation. However, the introduced sparsity in the Swap model might be reduced due to the shifting operation present in the affine transformation of BN. This reduction occurs because effectively decaying the affine parameters can help maintain values close to zero throughout the training. To preserve the sparsity in the

0.0

1e-06

5e-06

1e-05

5e-05

1e-04

5e-04

Intensity of Weight decay on Gamma and Beta

0.78

0.79

0.80

0.81

0.82

0.83

0.84

0.85

Avg.Sparsity

72.0

72.5

73.0

73.5

74.0

74.5

Avg.Sparsity

Swap model, we increase the weight decay on the parameters involved in the affine transformation. Such an amplified weight decay can further boost the sparsity of the BN output.

6 Shifted Tanh for Asymmetric and Sparse Output

Based on the investigation, we modify the Tanh function, referred to as Shifted Tanh, to encourage asymmetry and sparsity. Different from the Swap Tanh model, Shifted Tanh can generate asymmetry and sparse activation on zero-centered input. Shifted Tanh shares properties similar to ReLU. This property accelerates the model to use asymmetry and sparsity for better generalization.

The Shifted Tanh involves shifting the inputs by a factor of τ ∈ R before the Tanh operation, introducing asymmetry, and then shifting the outputs in reverse by the degree of the shift while considering the Tanh operation. The formulation of the Shifted Tanh is as follows:

We choose τ = -1 based on the best accuracy from the ablation study on weight τ , as shown in the Appendix.

7 Extended Experiments

7.1 Investigation on Various Datasets and Architectures

Our study delves into the Tanh models trained on the CIFAR-100 dataset using the VGG16_11 architecture. In this section, we evaluate the Swap order in various datasets and architectures and compare its accuracy against the Convention order.

From the results presented in Table 1, a significant difference is observed in performance between the Swap and Convention orders when using Tanh; the Swap order consistently delivers better results. In contrast, for both ReLU and shifted Tanh models, the sequence of layers has a negligible effect on accuracy. Moreover, the Convention model usually exhibits a slight edge over the Swap model. We assume that the zero-centered distribution of BN augments the asymmetry and sparsity of the Shifted Tanh in the Convention arrangement. Hence, the Convention model's accuracy slightly surpasses the Swap model, mirroring patterns seen with ReLU. It's noteworthy that the enhanced accuracy of shifted Tanh rivals that of the ReLU model.

5

0

5

Input

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Activation value

1

2

3

4

5

6

7

8

9

Block depth

0.0

1.0

2.0

3.0

4.0

5.0

Skewness

SoftSign

LecunTanh

Convention

Swap

7.2 Effect of the Swap Order on Other Bounded Activation Functions

Asymmetric saturation is a crucial factor in the success of the Tanh model. The primary focus of our investigations lies with the Tanh activation function. However, in this section, we extend our examination to include other bounded activation functions, namely LeCun Tanh [14] and Softsign [19].

Softsign, defined as x | x | +1 , has the same asymptotic values as Tanh, yet its growth rate is polynomial, not exponential, resulting in a slower approach to its asymptotes [4]. Therefore, it can mitigate the issue of vanishing gradients by reducing neuron saturation. LeCun Tanh, characterized by a gentler slope and a broader output range than Tanh, has the formula 1 . 7159 × Tanh ( 2 3 × x ) .

The asymmetric saturation caused by the Swap order occurs not only in Tanh but also in other bounded activation functions. The Convention model with the three types of bounded activation functions shows low layer-wise Skewness. The Skewness is less than one over the overall layer. On the other hand, a significant Skewness increment arises when the Swap order is applied. The shapes of these functions and layer Skewness are shown in Figure 6.

In all of the activation functions, the Swap models improve performance compared to the Convention. It can be found in Table 2. When swapping, asymmetric saturation happens the least in Softsign, which makes it challenging to create a saturation state. Furthermore, the Softsign model with the Swap order shows lower performance than the Tanh model with the Swap order, which could generate more saturation, even though the Convention model had the highest performance.

gaussian

shot

impulse

defocus

glass

motion

zoom

snow

frost

fog

brightness

contrast

elastic

pixelate

jpeg

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Convention

Swap

7.3 Robustness of the Asymmetrically Saturated Model

We evaluate the Swap model using a corrupted version of the CIFAR-100 dataset, termed as CIFAR100-C [7]. The results of robustness accuracy on CIFAR-100-C can be seen in Figure 7. The findings reveal an enhanced accuracy with the Swap order throughout all corruption categories. The Convention model has an average accuracy across all corruptions at 41.4% . However, the Swap model enhances this average to 45.3% .

Asymmetric saturation offers increased robustness to variations in input data. This robustness stems from the behavior of the Tanh function, especially when utilizing the output on its bounds to its midpoint. Near the center, particularly around zero, the Tanh function is highly responsive to slight input changes, leading to considerable alterations in the output. As a result, the Convention model is more vulnerable to such input shifts, especially in the zero vicinity of the Tanh function. In contrast, the Swap model activates Tanh's asymptotic region. Therefore, even pronounced input deviations lead to only slight output changes. Such a trait makes the Swap model inherently robust, maintaining a more consistent output even with input disparities.

7.4 Further Discussion on the Effect of Asymmetry and Sparsity

The Swap model enhances its performance through both asymmetry and sparsity, making it challenging to distinguish the contributions of each. We choose the Convention order to determine the primary factor influencing performance. When asymmetry increases in the Convention model, the majority of values have large values and it decreases in sparsity. Thus, the NWDBN model, which encourages greater asymmetry in the Convention model, has a reduced sparsity while the level of asymmetry is enhanced. The result of accuracy and averaged Skewness and Sparsity on the Convention and NWDBN models can be seen in Table 3. Notably, even with its diminished sparsity, the asymmetrically saturated NWDBN model surpasses the Convention model's performance. From these observations, we deduce that asymmetry plays a more pivotal role than sparsity.

8 Conclusion

In this work, we report that the Swap models with bounded activation functions perform better than the Convention models and analyze what brings about performance improvement. Our analysis reveals that the asymmetric saturation and the increased sparsity by the batch normalization after bounded activation functions contribute to the increased accuracy of the Swap models. Interestingly, this makes bounded activation functions behave much like ReLU and enables them to achieve performance comparable to that of ReLU. Most importantly, it expands our understanding of the activation function and will further assist in developing superior activation functions in the future.

References

Bhat, N. V., Minderman, P. A., McAvoy, T., and Wang, N. S. Modeling chemical process systems via neural computation. IEEE Control Systems Magazine , 10(3):24-30, 1990.

Chen, C.-T. and Chang, W.-D. A feedforward neural network with function shape autotuning. Neural networks , 9(4):627-641, 1996.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255. Ieee, 2009.

Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.

Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.

Hasani, M. and Khotanlou, H. An empirical study on position of the batch normalization layer in convolutional neural networks. In 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS) , pp. 1-4. IEEE, 2019.

Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations , 2019.

Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv , abs/1704.04861, 2017.

Hurley, N. and Rickard, S. Comparing measures of sparsity. IEEE Transactions on Information Theory , 55(10):4723-4741, 2009.

Hyvärinen, A. and Oja, E. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pp. 448-456. PMLR, 2015.

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.

Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade , pp. 9-48. Springer, 2012.

Rakitianskaia, A. and Engelbrecht, A. Measuring saturation in neural networks. In 2015 IEEE Symposium Series on Computational Intelligence , pp. 1423-1430. IEEE, 2015.

Rakitianskaia, A. and Engelbrecht, A. Saturation in pso neural network training: Good or evil? In 2015 IEEE Congress on Evolutionary Computation (CEC) , pp. 125-132. IEEE, 2015.

Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941 , 2017.

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Turian, J., Bergstra, J., and Bengio, Y . Quadratic features and deep architectures for chunking. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers , pp. 245-248, 2009.

A Limitation

While this paper contains an in-depth analysis of Tanh, it does not propose a function that is superior to existing activation functions. Generally, it is known that Tanh does not perform better than ReLU, but it's hard to find a study that deeply analyzes the reasons for this. To our knowledge, this paper is the most comprehensive analysis of the performance degradation when Tanh is used in a conventional manner. Although it suggests strategies to utilize Tanh more effectively, achieving performance nearly on par with ReLU, it fundamentally does not assert that Tanh surpasses ReLU in terms of effectiveness.

B Accuracy of the Shifted Tanh According to τ

In this section, we focus on the hyperparameter τ in the shifted Tanh function. We carry out experiments using VGG16_11, MobileNet, and PreAct-ResNet models trained on CIFAR-10 and CIFAR-100 datasets, testing different values of the parameter τ (-1.5, -1.2, -1.0, -0.8, and -0.5). The performance results corresponding to each of these τ values are presented in Table B.1. In these experiments, all hyperparameters, except for τ , are maintained at their best-averaged accuracy settings based on each model trained on the CIFAR dataset. Our aim is to identify the effective τ value without pushing all inputs into the saturation zone due to excessive asymmetry. The best-performance τ is identified by averaging the accuracy across different models for each τ and selecting the one with the highest performance. In our case, -1 is the best.

C Metric

C.1 Saturation and Skewness on Different Distribution Patterns

1.0

0.5

0.0

0.5

1.0

Activation value

0.0

0.5

1.0

1.5

2.0

Density

Saturation

0.7

0.5

0.3

1.0

0.5

0.0

0.5

1.0

Activation value

0

2

4

6

8

Skewness

5.0

2.0

0.0

Our saturation metric is designed to quantify the concentration of elements to the maximum absolute output values. The metric has bounds from 0 to 1, and it equates to 0 when all the elements are 0 in the Tanh case. Conversely, it grows larger as elements tend towards the maximum limit of the output range. For example, the saturation metric is evaluated at 0.5 in a uniform distribution. For saturation values of the Tanh output on the different Gaussian distributions, refer to Figure C.1 (left).

Skewness is a measure that quantifies the asymmetry within a distribution. Symmetric distribution will yield the Skewness of 0, and the Skewness increases with the rise in asymmetry. Notably, we calculate the absolute skewness value in our asymmetry metric, ensuring that the metric remains

unaffected by the direction of skewness. For a better understanding of how this measure applies to various distributions, Figure C.1 (right) provides Skewness on each distribution.

C.2 Impacts of Mean and Standard Deviation Variations on Tanh Asymmetry

0.1

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

2.5

Skewness

Input Mean

0.0

0.3095196

0.69314718

1.47221949

The mean and variance of input distribution on Tanh affect the asymmetry of Tanh output. The Skewness of Tanh output on various mean and standard deviation can be found in Figure C.2. In the Skewness of varied mean distribution, the Skewness is increased on the mean increase. However, the maximum Skewness does not align with the standard deviation increases. The Skewness decreases not only the small input standard deviation but also the large input standard deviation. Additionally, in the same mean condition, a decrease in standard deviation from the maximum skewness point more rapidly decreases the Skewness than an increase in standard deviation.

C.3 The Proof of Demonstrating the Sparsity Metric Criteria

This section examines the adequacy of our sparsity metric against the heuristic criteria for sparsity measures defined by Hurley & Rickard [9]. Our sparsity metric meets five of these criteria: Scaling, Rising Tide, Cloning, Bill Gates, and Babies.

We consider the absolute operation as pre-applied to the metric input, according to Hurley & Rickard [9]. Thus, our sparsity metric is represented as f : x ↦→ s , where x ∈ R m + is a vector of absolute values in a channel in BN output, and x max+ signifies the maximum value in x , i.e., x max+ = max i x i . We prove our sparsity metric in the setting of one-channel, which can be expanded to the multi-channel setting without loss of generality. Below are the five criteria our sparsity metric meets and proof for each.

Proof of Scaling. Sparsity is scale invariant. That is, for any scalar α ∈ R , where α > 0 , the function f satisfies:

The invariance of scale means that sparsity considers relative differences rather than absolute magnitudes.

Proof. If we scale the vector x by α , we also scale the maximum positive value x max+ by α . Therefore, we have:

This demonstrates that the sparsity of the vector x remains the same when the vector is scaled by any positive scalar α .

Proof of Rising Tide. Adding a constant to each element decreases sparsity. Formally, for any scalar α ∈ R , where α > 0 , the function f satisfies:

We exclude the case, as mentioned in [9], where all elements of x are the same.

This property also indicates that sparsity increases as more values approach zero.

Proof. We begin by noting the sparsity of the vector x + α :

Assume for contradiction that f ( x + α ) ≥ f ( x ) . Given that mx max+ > Σ i m =1 x i > 0 and mα > 0 the following inequalities can be derived:

which contradicts the initial assumption, thus implying that

as desired.

Proof of Cloning. Sparsity is invariant under cloning.

That is, for a vector x , the function f satisfies:

where || denotes concatenation, such that x || x = [ x 1 , x 2 , . . . , x m , x 1 , x 2 , . . . , x m ] .

The principle of cloning implies that sparsity remains the same even if a set of values is replicated.

Proof. Define a function g : x , δ ↦→ y , where δ ∈ Z + and y ∈ R mδ is the result of concatenating x δ times. We then have:

This shows that the sparsity of x remains the same even when it is concatenated with itself δ times.

Proof of Bill Gates. As one individual element becomes infinitely large, the sparsity increases. Formally, for every i , there exists ρ > 0 , such that for all α > 0 :

The implication is that if a single value grows without bounds, the sparsity also increases indefinitely.

Proof. We first choose a sufficiently large ρ such that ∀ i, x i + ρ > x max+ . Assume for contradiction that S ([ x 1 , ..., x i + ρ + α, ..., x m ]) ≤ S ([ x 1 , ..., x i + ρ, ..., x m ]) . This implies:

̸

leading to a contradiction. Therefore,

Proof of Babies. Adding a new element of zero increases sparsity. Formally, for a vector x , the function f satisfies:

Concatenating zero elements to the existing values increases the relative difference to the other values, which increases sparsity.

Proof.

We start by subtracting one from both sides of the inequality:

The negative sign can be removed by reversing the inequality:

Multiplying both sides of the inequality by m ( m +1) (since m is a positive integer, m +1 is also positive, and the inequality sign will not change), we get:

Subtracting m Σ m k =1 x k from both sides of the inequality, we obtain:

Since x k is not a negative value, the sum is also non-negative, which verifies the inequality. Therefore, the original inequality is proven.

̸

̸

̸

D Deciding Model Configurations for Investigation

D.1 Investigation of Depth Impact on Accuracy in VGG16 Models

To identify a model for focused analysis using the CIFAR dataset, we examined various VGG16 variants. This examination progressively removes convolution layers from the end towards the front of VGG16. The result can be seen in Table E.2. We observed an increase in accuracy until peaking at the VGG16_11 model, followed by a decline. Although a VGG11 model has already been proposed in Simonyan & Zisserman [18], the validation accuracy of VGG16_11 is significantly higher than VGG11 on CIFAR-100. The results can be seen in Table E.3.

E Training Hyperparameters

We sweep the learning rate and weight decay hyperparameter. The learning rate was 0.1 and 0.01. For CIFAR and Tiny-ImageNet datasets, we trained models with a batch size of 128, and the learning rate was reduced by one-tenth at 100 and 150 of the total 200 epochs, and we swept four weight decay of 0.005, 0.001, 0.0005, and 0.0001. For ImageNet datasets, we trained models with a batch size of 256, and the learning rate was reduced by one-tenth at 30 and 60 of the total 100 epochs, and we swept three weight decays of 0.001, 0.0005, and 0.0001. We chose the best averaged accuracy model for the three random seeds and averaged the values of these three models for all measurements for analysis. Because of the computation issue, we only use one seed for the ImageNet dataset with early stopping. The hyperparameters of VGG, MobileNet, and PreAct-ResNet are shown in E.1, E.2, E.3, respectively.

E.1 VGG

E.2 MobileNet

E.3 PreAct-ResNet

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves the hypothesis that asymmetric saturation benefits network performance by swapping the position of Batch Normalization and Tanh activation functions. The Swap model generates high sparsity and asymmetric saturation which enables Tanh to behave like a one-sided activation function. Experimental results show the asymmetric distributions consistently outperforms the symmetric ones. However, because the BN and Tanh combination hardly appears in modern networks and there is no convincing theory facilitating network architecture design, the contribution of this paper is of less significance.\n\nSTRENGTHS:\n1) This paper is easy to follow.\n2) This paper provides a comprehensive experimental demonstration to validate asymmetric activation functions are superior to symmetric ones. This implies that ReLU-like activation functions are better than tanh activation functions.\n\nWEAKNESSES:\n1) One major concern is the lack of prevalence of the BN and Tanh combination in modern networks. As a result, the analysis and experiments conducted in this study hold limited empirical significance.\n2) The benefits of asymmetric activation functions have previously been demonstrated from the perspectives of gradient [1] and expressivity [2]. However, this paper fails to contribute new explanations or adequately discuss the limitations of existing works.\n3) The experiments in this paper are restricted to a limited range of network architectures. For example, even the widely used ResNet18 and ResNet50 models have not been evaluated on the ImageNet dataset. \n4) A noticeable error can be identified in Figure 1, where the right bottom subplot indicates that 'BN' exhibits lower sparsity compared to 'Tanh,' contradicting the accompanying label stating 'High Sparsity.'\n\n[1] Maas, A. L., Hannun, A. Y., & Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. 2013.\n[2] Hanin, Boris, and David Rolnick. Complexity of linear regions in deep networks. 2019.\n\nQUESTIONS:\n1）The paper discusses sparsity and asymmetry, which have distinct mathematical definitions. However, it remains unclear which factor is the key determinant of network performance.\n2） Whether the proposed swap model effectively addresses the issue of vanishing gradients?\n\nLIMITATIONS:\nThe authors did not discuss Limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performance of different activation function orders in deep learning models with batch normalization. The authors focus on the conventional order, where batch normalization is placed before the activation function, and the swapped order, where batch normalization is placed after the activation function. Surprisingly, they find that the swapped order achieves significantly better performance than the conventional order when using bounded activation functions like Tanh. The paper provides a thorough analysis of the underlying mechanisms and presents empirical evidence to support their findings.\n\nSTRENGTHS:\n- Novelty: The paper explores an interesting, and up to my knowledge previously overlooked aspect of activation function order in the context of batch normalization. The findings challenge the conventional wisdom and offer a new perspective on designing deep learning models.\n\n- Comprehensive experiments: The authors conduct extensive experiments and carefully examine the output distributions of individual activation functions. Their investigation into the asymmetric saturation phenomenon provides lots of intuitions into the behavior of bounded activation functions that make intuitive sense. \n\n- While focusing on Tanh as the primary activation function, the authors demonstrate that their findings are applicable to similar antisymmetric and bounded activation functions. \n\n- Performance Improvement: The swapped order, combined with bounded activation functions and batch normalization, consistently outperforms the conventional order across various benchmarks and architectures. The results highlight the potential for achieving superior performance by exploiting the benefits of asymmetry and sparsity.\n\n- Quality of presentation of ideas: the paper is very well written and gives the ideas in a clear and easy to follow manner. The contributions are clearly stated and there are no over claims in the text.\n\nWEAKNESSES:\n*Limited scope*\nMy main concern with this paper is that the limited scope of the network configurations considered for the experiments. For example, the results presented in Table 1 shows that conventional models with ReLU are mostly equal or better than all swap models, as well as the ReLU activation with swap order. Can authors make any further comments on possible reasons for this? For example, one might discount the strength of the main empirical evidence that is presented (differential performance between swapped & conventional models for Tanh & Tanh shifted), is only observed for the sub-optimal models and not the original model. While this does not directly contradict the main message of the paper, it significantly weakens its potential impact and scope. If the authors can address these questions by further experiments this could potentially strengthen the main conclusions made in the paper. \n\nOn a similar note, given the central hypothesis that asserts \"The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance, \" can this hypothesis explain the differential performance between ReLU & Tanh too? Can this hypothesis explain the difference & benefit of having BN layers at all? (if you remove them, you don't have the asymmetric saturation ? There seems to be plenty of adjacent configurations that can be added to expand on the generalizability of this central idea of the paper.  These types of additional experiments could strengthen the study's conclusions. \n\n*Lack of Theoretical Analysis* Although the paper presents compelling empirical evidence, a deeper theoretical analysis would enhance the understanding of why and how the swapped order, asymmetry, and sparsity contribute to performance improvement. Incorporating theoretical insights could strengthen the paper's contribution to the field. \n\n*Figures & tables:* One minor but important issue: The tables and figures currently lack confidence intervals or standard deviations\n\nQUESTIONS:\nno questions\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper investigates neural network classifiers with bounded activation functions. The authors first observe that swapping the batch norm and activation order improves performance with bounded activation functions. They then observe that asymmetric saturation and sparsity occurs in the swap model compared to convention, and show how it correlates with accuracy. They then propose a modified activation function that promotes asymmetric saturation and shows that even in convention order, it benefits performance.

STRENGTHS:
- The paper notes an interesting observation regarding the swap model with bounded activation
- It supports the hypothesis that asymmetric saturation and sparsity benefit benefit. This is achieved by caring the weight decay on the batch norm layers.
- The comparable performance with the modified tanh activation with relu is quite interesting. I wonder if ReLU-like behavior is what is the best for performance or if there is room for improvement past this.

WEAKNESSES:
- The evaluation is somewhat limited, since it is only done on VGG on image classification tasks. It’s not clear if these results would also apply to different architectures, e.g. transformer-based ones, or on different tasks such as segmentation or text classification.
- Comparison with different ReLUs, such as LeakyRELU or ELU.

QUESTIONS:
See weaknesses.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The paper investigate the order of Batch Normalization and activation functions, and founded bounded activation functions like Tanh works better in the swapped order unlike bounded one like ReLU. To explain this, the authors analyze the asymmetric saturation levels at both the layer and channel levels and find that the Swap model has higher saturation levels, especially in layers with excessive saturation. They also introduce a new model, NWDBN, that encourages asymmetric saturation and show that it improves accuracy compared to the convention model. The paper concludes that asymmetric saturation can help improve performance in neural networks.

STRENGTHS:
- The paper revolve around the hypothesis that "asymmetric saturation helps improve performance", and analyize that ReLU and Tanh bring different asymmetric saturation levels, this sound interesting to me.
- The paper identify the high sparsity induced by Batch Normalization after bounded activation is functions and validate that the higher sparsity induced would booster the performance.

WEAKNESSES:
- The logic chain of the story in this paper need some improvement, see more in Questions part.
- The input of convolution layer in Figure 1 upper and lower part seem different, can the authoer explain why?
- Why is x-axis in Figure 1 lower-right BN figure of range (-5, 2.5), while I didn't see any value in the range of (-5, -1), this plot could be misleading.

QUESTIONS:
- The paper did not explain well why higher sparsity induced by asymmetric saturation would booster the performance.
- Is there a way to control Asymmetric Saturation to archieve trade-off between sparsity and performance?

LIMITATIONS:
As stated in Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10