SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tanh Works Better With Asymmetry
Dongjin Kim $^{1,3}$  Woojeong Kim $^{2}$  Suhyun Kim $^{3*}$
$^{1}$ Department of Computer Science and Engineering, Korea University
$^{2}$ Department of Computer Science, Cornell University
$^{3}$ Korea Institute of Science and Technology
{npclinic3, kwj962004, dr.suhyun.kim}@gmail.com
Batch Normalization is commonly located in front of activation functions, as proposed by the original paper. Swapping the order, i.e., using Batch Normalization after activation functions, has also been attempted, but its performance is generally not much different from the conventional order when ReLU or a similar activation function is used. However, in the case of bounded activation functions like Tanh, we discovered that the swapped order achieves considerably better performance than the conventional order on various benchmarks and architectures. This paper reports this remarkable phenomenon and closely examines what contributes to this performance improvement. By looking at the output distributions of individual activation functions, not the whole layers, we found that many of them are asymmetrically saturated. The experiments designed to induce a different degree of asymmetric saturation support the hypothesis that asymmetric saturation helps improve performance. In addition, Batch Normalization after bounded activation functions relocates the asymmetrically saturated output of activation functions near zero, enabling the swapped model to have high sparsity, further improving performance. Extensive experiments with Tanh, LeCun Tanh, and Softsign show that the swapped models achieve improved performance with a high degree of asymmetric saturation. Finally, based on this investigation, we test a Tanh function shifted to be asymmetric. This shifted Tanh function that is manipulated to have consistent asymmetry shows even higher accuracy than the original Tanh used in the swapped order, confirming the asymmetry's importance. The code is available at https://github.com/hipros/tanh_works_better_with_asymmetry.
Batch Normalization (BN) is a widely used technique in deep learning. It was proposed to address the internal covariate shift problem by maintaining a stable output distribution among layers. As the characteristics of the output distribution of the weighted summation operation tend to be symmetric, non-sparse, and Gaussian [10], Ioffe & Szegedy [11] placed the BN between the weight and activation function. Thus, the "weight-BN-activation" order, which we call "Convention" in this paper, has been widely used to construct a block in many architectures [18, 8]. "Swap" models, swapping the order of BN and the activation function in a block, have also been attempted, but no significant and consistent difference between the two orders has been observed in the case of ReLU or similarly shaped activation functions. For instance, Hasani & Khotanlou [6] evaluated the effect of the position of BN in terms of training speed and concluded that there is no clear winner and the result depends on the datasets and architecture types.
Figure 1: We examine the layer and channel-wise distributions of Tanh activation (left) and the feature map distribution in a single convolution block (right). Tanh produces nearly symmetrical layer activation in the Convention and Swap models (left-left). However, the channel distributions vary, with the Convention model showing symmetry (left top) and the Swap model presenting one-sided distributions (left bottom). The Convention model, with zero-centered Tanh input, exhibits low asymmetry and sparsity (right top). The Swap model, without the zero-centered constraint of BN, enhances asymmetry and, due to BN's placement after Tanh, sparsity (right bottom). The ranges of output are marked as lines above the plot.
However, in the case of bounded activation functions, we empirically found that the Swap order exhibits substantial improvements in test accuracy than the Convention order with diverse architectures and datasets. This paper empirically investigates the reason for this accuracy difference between the Convention and the Swap models with bounded activation functions. For simplicity, our analyses are mainly conducted with Tanh but are applicable to similar antisymmetric and bounded activation functions. We present the results with LeCun Tanh and Softsign at the end of the experimental section.
One key difference between Swap and Convention models is the distribution of activation values, as shown in Figure 1(left). In the Swap model, the mean of convolution layer output shifts substantially away from 0. Thus, most activation values are near the asymptotic values of the bounded activation function, that is, highly saturated. This is unanticipated since it is a common belief that high saturation should be avoided. To investigate this paradox, we took one step further and looked at the output distribution of each activation function, not just a whole layer. To our very surprise, even though the distribution is fairly symmetric at the layer level, the activation values of each channel are biased toward either one of the asymptotic values or asymmetrically saturated. Through an in-depth investigation, we come to a hypothesis that this asymmetric saturation is a key factor for the performance improvement of the Swap model since it enables Tanh to behave like a one-sided activation function. In the experiments to examine whether asymmetric saturation is related to the performance of models, we observe that the accuracy and the degree of asymmetric saturation are highly correlated.
BN after Tanh also shifts the distribution biased toward either -1 or 1 by asymmetric saturation near zero, which has the important effect of increasing sparsity, as depicted in Figure 1(right). Sparsity is widely considered to be a desirable property. For instance, Glorot et al. [5] studied the benefits of ReLU compared to Tanh in terms of sparsity. One thing to note is that if each channel is symmetrically saturated, BN will not increase sparsity much since the mean is already close to zero. In contrast, the one-sided property of asymmetric saturation causes at least half of the sample values after normalization to be almost zero. It allows the Swap model to have even higher sparsity than the Convention model. Asymmetric saturation combined with normalization makes a bounded activation function behave much like ReLU.
Based on the above lessons, we test enforcing the asymmetric saturation and sparsity of Tanh outputs. This involves shifting the inputs of Tanh by  $\tau$ , introducing asymmetry, and then shifting the outputs in reverse by  $-tanh(\tau)$  to boost sparsity. This is equivalent to using a shifted Tanh as a new activation function. Our experimentation across different architectures and datasets reveals that this adjusted
Tanh function with fixed asymmetry consistently outperforms the original Tanh used in the Swap order. Its effectiveness is even comparable to that of ReLU. This validates the hypothesis that Tanh works more efficiently with asymmetric saturation.
Our findings are as follows:
- The Swap model using Batch Normalization after bounded activation functions performs better than the Convention model in many architectures and datasets.
- We discover the asymmetric saturation at the channel level and investigate its importance through carefully designed experiments.
- We identify the high sparsity induced by Batch Normalization after bounded activation functions and perform an experiment to examine the impact of sparsity on performance.
- Our shifted Tanh with enforced asymmetric saturation shows higher accuracy than the Swap Tanh, which reconfirms the importance of asymmetry.
2 Settings and Notation
Model For our investigation, we choose the VGG-like model due to its simple and straightforward design. This model is trained on the CIFAR-100 dataset [12]. As a modification, we replace the ReLU activation function with Tanh. However, because the VGG architecture was proposed for the ImageNet dataset, the model is overparameterized for the CIFAR dataset. It incurs poor performance and difficulty in investigating the Swap model. Thus, we cut out the last convolution layers and select the best model based on the validation accuracy. The model with five cut-out layers shows the best accuracy, as in the Appendix. We call this model "VGG16_11" and use this architecture to investigate Convention and Swap orders.
Metric We consider three properties to investigate each order: saturation, asymmetric saturation, and sparsity. Saturation is measured on Tanh and quantifies how closely outputs the values to the maximum absolute output range limit, also known as the asymptotic value. We involve the skewness metric in statistics to measure asymmetry on Tanh. One thing that differs from the skewness is the absolute. We absolute the skewness to quantify the asymmetry regardless of skewed direction. Sparsity is measured on the output of the block, i.e., Tanh for Convention and BN for Swap. To measure the sparsity, we inversely utilize our saturation metric.
Experiment setup In Section 4.2, we verify the effect of asymmetry by slightly cutting off the asymmetry in the asymmetry-boosted Convention model, which we refer to the No-Weight-Decay-on-BN (NWDBN) model. The NWDBN model is ordered as Convention, and the weight decay does not apply to BN. We fix the intensity of the weight decay on the weight layer and vary the decay intensity on BN to regulate asymmetry. The initial hyperparameters followed the best hyperparameter of the NWDBN model. Then, we increase the intensity of weight decay on  $\beta$  in BN from 0.0 to 0.001 by 0.0001. In Section 5.3, we investigate the effect of sparsity by slightly increasing it. The initial hyperparameters followed the Swap model's best hyperparameters. Then, we change the weight decay intensity on the  $\gamma$  and  $\beta$  in BN. The searched intensities of weight decay are 0, 1e-6, 5e-6, 1e-5, 5e-5, 1e-4, and 5e-4. We verify the effects of the layer order in various datasets, networks, and activation functions in Section 7.1. We train models on four benchmarks (CIFAR-10, CIFAR-100, Tiny ImageNet [13], and ImageNet [3]), three base-architectures (VGG, MobileNet, PreAct-ResNet), and three activation functions (ReLU, Tanh, ShiftedTanh). For the ResNet, the Swap order incurs extra non-linearity functions, which hinder the fair comparison between the Convention and Swap model. Except for the first stage, the first shortcut connection in each stage has convolution and BN layers in the block. In the case of the Swap Tanh model, Tanh replaces BN and vice-versa. Thus, the number of non-linearity functions increases as the number of the linear projection layer in the shortcut. We, therefore, report the results of PreAct-ResNet, which only has the convolution layer in the shortcut connection. (Furthermore, upon training ResNet20 with Tanh on the CIFAR-100 dataset, we observed that the Swap model surpassed the Convention model, achieving an accuracy of  $69.06\%$  compared to the Convention model's  $68.97\%$ .)
All results, except the ImageNet dataset, are conducted on three random seeds. The measured values and accuracy are averaged over seeds. We use the SGD optimizer with a momentum of 0.9, weight
decay. We adopt a 2-step learning rate decaying strategy that decays by 0.1. We conduct a grid search to obtain the best model for investigation. We search different combinations of initial learning rates (0.1 and 0.01) and weight decay values (1e-4, 5e-4, 1e-3, and 5e-3). The specific hyperparameters for each model can be found in the Appendix.
Notation The input of an arbitrary layer in the convolution neural network is denoted as  $\mathcal{X} \in \mathbb{R}^{n \times c \times h \times w}$ , where  $n, c, h$ , and  $w$  represent the number of samples, channels, height, and width, respectively. We reshape  $\mathcal{X}$  to  $\pmb{X} \in \mathbb{R}^{c \times m}$ , where  $m = nhw$ . The output of the Tanh function is  $\pmb{T} \in \mathbb{R}^{c \times m}$  where  $T_{i,j} = \tanh(X_{i,j})$ . BN normalizes the channel distribution to have zero mean and unit variance using estimated statistics. After normalization, an affine transformation with scaling parameter  $\gamma \in \mathbb{R}^c$  and shifting parameter  $\beta \in \mathbb{R}^c$  is applied channel-wise. Thus, the output of BN is given by:  $B_{i,j} = \gamma_i\left(\frac{X_{i,j} - \hat{\mu}_i}{\sqrt{\hat{\sigma}_i^2 + \epsilon}}\right) + \beta_i$ , where  $\epsilon$  is a constant added for numerical stability,  $\hat{\mu}_i$  and  $\hat{\sigma}_i$  are estimated mean and standard deviation of each channel, respectively, computed over the mini-batch of training data samples.
3 Overly Saturated but Well-Generalized Model
The phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function [15]. When training a neural network with bounded activation functions, the outputs of neurons increase due to the gradually increasing weights [4]. The increased output values become close to the near asymptote in bounded activation functions, as shown in the experiment in Glorot & Bengio [4]. However, excessive saturation damages the neural network, and it is regarded as a situation to evade [4, 15, 16, 1, 2, 11]. Various methods were proposed to prevent excessive saturation. Glorot & Bengio [4] proposed an initialization scheme, Rakitianskaia & Engelbrecht [15, 16] proposed a metric to measure the degree of saturation for monitoring the training, Bhat et al. [1] pre-scaled the inputs of the activation function, and Chen & Chang [2] proposed adaptable bounded activation.
3.1 Saturation Metric
Our saturation metric quantifies the extent to which the output values approach the function's output range limit, also known as the asymptotic value. To normalize the input, we calculate the absolute value and divide it by the maximum absolute value, resulting in a range of [0, 1]. The saturation  $t$  of a layer is defined as:
$$
t = \frac {1}{c} \Sigma_ {i = 1} ^ {c} \frac {\sum_ {j = 1} ^ {m} \left| T _ {i , j} \right|}{m a}, \tag {1}
$$
where  $a$  represents the maximum absolute value of the output range of the activation function. Specifically, for the Tanh function,  $a$  is equal to 1.  $t$  lies in [0, 1). The large value of this metric indicates  $T$  are highly saturated, as illustrated in the Appendix.
3.2 High Saturation in the Swap Model
Even if only the layer order is changed from the Convention order to the Swap order, there is a  $4.61\%$  improvement in test accuracy. However, when we measure the layer saturation in both models, the Swap model has highly saturated layers. The layer-wise Saturation of Swap and Convention models can be seen in Figure 2(a). This is counterintuitive as excessive saturation is considered undesirable in previous works [4, 15, 16, 1, 2]. The saturation of almost half of the layers in the Swap model is over 0.7. Some layers even have a saturation over 0.8. On the other hand, the saturation of the Convention model is lower than 0.5 over all layers.
4 Asymmetric Saturation
In order to scrutinize saturation, we perform an in-depth analysis at the channel level, one step further than the layer level. Fascinatingly, we notice that most channels display asymmetric saturation in the Swap model, especially in layers with excessive saturation. On the other hand, the channel distribution in the Convention model tends to be centered around zero, even at the channel level.
Figure 2: (a) Layer-wise Saturation of Convention and Swap models. (b) Layer Skewness in Convention and Swap models. The two figures display a  $95\%$  confidence interval.
4.1 Asymmetric Saturation Metric
Both the Convention and Swap models demonstrate symmetric distributions at the layer level. However, notable differences arise at the channel level. To quantify and compare the degree of asymmetry present at the channel level, we measure the asymmetry of each channel individually and calculate the average to determine the overall skewness of the layer. We adopt sample skewness as a metric to gauge the asymmetry of the Tanh output. The skewness of a given  $i^{\mathrm{th}}$  channel, denoted  $p_i$ , is calculated as follows:
$$
p _ {i} = \frac {m ^ {2}}{(m - 1) (m - 2)} \frac {\mu_ {i} ^ {3}}{\sigma_ {i} ^ {3}} \tag {2}
$$
where  $\mu_i^3$  represents the sample third central moment of  $T_{i,:}$ , while  $\sigma_i^3$  represents the sample standard deviation cubed of  $T_{i,:}$ . The averaged absolute skewness of a layer,  $k$ , herein referred to simply as Skewness, is given by:
$$
k = \frac {1}{c} \Sigma_ {i = 1} ^ {c} | p _ {i} |. \tag {3}
$$
Figure 2(b) illustrates the Skewness for all layers in the Convention and Swap models. In the Convention model, the Skewness of most layers is close to zero, indicating a minimal presence of asymmetric distribution. Conversely, the Swap model exhibits higher Skewness compared to the Convention model. Moreover, the regions with high saturation blocks within the Swap model show particularly elevated Skewness values. This suggests that saturation in these areas occurs asymmetrically in the channel distributions.
4.2 Effect of Asymmetric Saturation on Performance
In order to examine the effect of asymmetric saturation, we devise a method to control the level of asymmetry in the Convention model. One key factor that limits the emergence of asymmetric saturation in the Convention model is the influence of weight decay on  $\gamma$  and  $\beta$  within BN. Especially, small  $\beta$  values by weight decay constrain the distribution to be zero-centered, which incurs a decrease in the asymmetry of the Tanh output. In our experiment, the NWDBN model successfully encourages asymmetric saturation and has higher Skewness than the Convention model. Moreover, the NWDBN model shows improved accuracy of  $72.22\%$  compared to the Convention model  $69.5\%$ . To better examine
Figure 3: Relation between accuracy and averaged Skewness over layers. The "Avg. Skewness" averaged all the layer-wise Skewness in each model.
the effects of asymmetric saturation on performance, we gradually increase the intensity of weight
Figure 4: Red lines represent Tanh activation functions combined with normalization. The dots represent the output values of the combined function for different input samples. Samples of the same color indicate they generate the same BN statistics on the Tanh output. The notation  $\mathsf{Nor}_{\hat{\mu},\hat{\sigma}}$  represents the normalization function of BN.
decay on the  $\beta$  parameter from zero, which indicates that slightly eliminates the asymmetry in the NWDBN model. As the weight decay on  $\beta$  gets bigger, Skewness becomes smaller, and the test accuracy degrades, as shown in Figure 3. This strong correlation clearly supports our hypothesis: asymmetric saturation helps improve performance.
5 Sparsity
5.1 Asymmetric Saturation with Batch Normalization Can Induce High Sparsity
ReLU achieves a high generalization performance by utilizing the strengths of sparsity [5]. The sparsity introduced by ReLU is a consequence of its single boundary at zero, which transforms all negative inputs to zero. In addition, work by Ramachandran et al. [17] shows the benefits of having a single asymptote at zero. They carried out an automated strategy to investigate a variety of activation functions. The most notable activation functions discovered through this search were one-sided and had a boundary value close to zero, similar to ReLU.
We note that the Swap model effectively generates sparsity, pushing a majority of values toward zero when asymmetric saturation occurs. This shifting is credited to the normalization process in Batch Normalization (BN), which recenters the distribution around zero. Thus, when the asymmetric saturation occurs, this saturation is subsequently shifted towards zero during the normalization process, enhancing sparsity. The combined effect of Tanh with normalization, along with samples of the function's output showing different BN statistics in the Swap order, are depicted in Figure 4. The majority of values and the asymptotic lines that the samples saturated are shifted to near zero.
5.2 Sparsity Metric
Intuitively, a sparse representation is one in which a small number of coefficients contain a large proportion of the energy [9]. We utilize our saturation metric to measure sparsity. Saturation quantifies the extent to which many values reach the maximum. We measure the sparsity by the extent to which many values are far from the maximum. However, as sparsity is measured on the block output, we consider the output of Tanh for the Convention model and the output of BN for the Swap model. BN has no boundaries, so we empirically calculate  $a$  for normalization in Equation 1. We denote the empirical saturation,  $\hat{t}$ , as follows:
$$
\hat {t} = \frac {1}{c} \Sigma_ {i = 1} ^ {c} \frac {\Sigma_ {j = 1} ^ {m} | B _ {i , j} |}{m B _ {i , :} ^ {\max } +}, \tag {4}
$$
where  $B_{i,:}^{max}$  is the maximum absolute element in  $B_{i,:}$ , i.e.,  $B_{i,:}^{\max +} = \max_j B_{i,j}$ . For the Convention model, we apply the above metric to  $T$  instead of  $B$ . Finally, the sparsity  $s$  is defined as:
$$
s = 1 - \hat {t} \tag {5}
$$
Our sparsity metric satisfies five criteria among six desired criteria of sparsity measures in Hurley & Rickard [9]. The proof of each criterion can be found in the Appendix.
5.3 Effect of Sparsity on Performance
In this section, we investigate the significance of sparsity in the Swap model and its influence on test accuracy. There's a marked relationship between increased sparsity and test accuracy. The result can be seen in Figure 5.
As elaborated in Section 5.1, the Swap order has the capability to intensify sparsity when there's asymmetric saturation. However, the introduced sparsity in the Swap model might be reduced due to the shifting operation present in the affine transformation of BN. This reduction occurs because effectively decaying the affine parameters can help maintain values close to zero throughout the training. To preserve the sparsity in the
Figure 5: Relation between accuracy and averaged Sparsity over layers. The "Avg. Sparsity" averaged all the layer-wise Sparsity in each model.
Swap model, we increase the weight decay on the parameters involved in the affine transformation. Such an amplified weight decay can further boost the sparsity of the BN output.
6 Shifted Tanh for Asymmetric and Sparse Output
Based on the investigation, we modify the Tanh function, referred to as Shifted Tanh, to encourage asymmetry and sparsity. Different from the Swap Tanh model, Shifted Tanh can generate asymmetry and sparse activation on zero-centered input. Shifted Tanh shares properties similar to ReLU. This property accelerates the model to use asymmetry and sparsity for better generalization.
The Shifted Tanh involves shifting the inputs by a factor of  $\tau \in \mathbb{R}$  before the Tanh operation, introducing asymmetry, and then shifting the outputs in reverse by the degree of the shift while considering the Tanh operation. The formulation of the Shifted Tanh is as follows:
$$
x \mapsto \tanh  (x + \tau) - \tanh  (\tau). \tag {6}
$$
We choose  $\tau = -1$  based on the best accuracy from the ablation study on weight  $\tau$ , as shown in the Appendix.
7 Extended Experiments
7.1 Investigation on Various Datasets and Architectures
Our study delves into the Tanh models trained on the CIFAR-100 dataset using the VGG16_11 architecture. In this section, we evaluate the Swap order in various datasets and architectures and compare its accuracy against the Convention order.