SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From the results presented in Table 1, a significant difference is observed in performance between the Swap and Convention orders when using Tanh; the Swap order consistently delivers better results. In contrast, for both ReLU and shifted Tanh models, the sequence of layers has a negligible effect on accuracy. Moreover, the Convention model usually exhibits a slight edge over the Swap model. We assume that the zero-centered distribution of BN augments the asymmetry and sparsity of the Shifted Tanh in the Convention arrangement. Hence, the Convention model's accuracy slightly surpasses the Swap model, mirroring patterns seen with ReLU. It's noteworthy that the enhanced accuracy of shifted Tanh rivals that of the ReLU model.
Table 1: Experimental comparison between orders on various datasets and models.
Datasets	Models	Tanh	Shifted Tanh	ReLU
Convention	Swap	Convention	Swap	Convention	Swap
CIFAR-10	VGG16	91.75±0.08	92.90±0.05	93.92±0.17	93.60±0.17	93.69±0.08	93.04±0.11
MobileNet	91.66±0.03	92.53±0.12	92.85±0.10	92.64±0.21	92.48±0.23	91.93±0.30
PreAct-ResNet18	92.46±0.17	94.41±0.15	94.48±0.15	94.57±0.18	94.94±0.07	94.86±0.18
CIFAR-100	VGG16	64.95±0.31	70.93±0.12	73.87±0.05	73.21±0.31	73.68±0.08	71.79±0.09
MobileNet	64.50±0.34	70.39±0.13	72.75±0.22	71.88±0.54	70.27±0.22	69.49±0.28
PreAct-ResNet18	73.26±0.19	75.76±0.17	76.70±0.35	76.51±0.21	78.06±0.29	77.39±0.23
Tiny ImageNet	VGG16	49.29±0.18	57.05±0.28	60.68±0.09	60.37±0.07	59.37±0.09	59.05±0.33
MobileNet	45.27±0.19	51.78±0.26	52.91±0.26	52.86±0.15	51.92±0.11	50.48±0.26
PreAct-ResNet34	59.06±0.26	64.94±0.05	65.82±0.18	65.66±0.20	67.28±0.23	66.21±0.18
ImageNet	VGG16	60.85	67.04	73.4	72.11	73.83	72.95
MobileNet	64.26	72.07	70.4	72.56	68.27	71.1
Figure 6: Shapes of activation functions (left) and Skewness tendency of different activation functions (right). The dashed line represents the Convention model and the solid line represents the Swap model in the right figure.
7.2 Effect of the Swap Order on Other Bounded Activation Functions
Asymmetric saturation is a crucial factor in the success of the Tanh model. The primary focus of our investigations lies with the Tanh activation function. However, in this section, we extend our examination to include other bounded activation functions, namely LeCun Tanh [14] and Softsign [19].
Softsign, defined as  $\frac{x}{|x| + 1}$ , has the same asymptotic values as  $\mathrm{Tanh}$ , yet its growth rate is polynomial, not exponential, resulting in a slower approach to its asymptotes [4]. Therefore, it can mitigate the issue of vanishing gradients by reducing neuron saturation. LeCun  $\mathrm{Tanh}$ , characterized by a gentler slope and a broader output range than  $\mathrm{Tanh}$ , has the formula  $1.7159 \times \mathrm{Tanh}(\frac{2}{3} \times x)$ .
The asymmetric saturation caused by the Swap order occurs not only in  $\mathrm{Tanh}$  but also in other bounded activation functions. The Convention model with the three types of bounded activation functions shows low layer-wise Skewness. The Skewness is less than one over the overall layer. On the other hand, a significant Skewness increment arises when the Swap order is applied. The shapes of these functions and layer Skewness are shown in Figure 6.
In all of the activation functions, the Swap models improve performance compared to the Convention. It can be found in Table 2. When swapping, asymmetric saturation happens the least in Softsign, which makes it challenging to create a saturation state. Furthermore, the Softsign model with the Swap order shows lower performance than the Tanh model with the Swap order, which could generate more saturation, even though the Convention model had the highest performance.
Table 2: Experimental comparison of accuracy with asymmetry between orders. We compare the accuracy with Skewness on VGG16_11 models trained by CIFAR-100. We used averaged Skewness over layers to calculate the difference of Skewness.
Activation functions	Order	Δavg. Skewness(Swap - Convention)
Convention	Swap
Tanh	69.5	74.11	2.38
Softsign	70.01	73.65	1.28
LeCun Tanh	67.82	74.46	1.90
Figure 7: Experimental comparison between orders on the different types of corruption. We compare robust accuracy on CIFAR-100-C between the Convention and Swap ordered VGG16_11 trained by CIFAR-100.
7.3 Robustness of the Asymmetrically Saturated Model
We evaluate the Swap model using a corrupted version of the CIFAR-100 dataset, termed as CIFAR-100-C [7]. The results of robustness accuracy on CIFAR-100-C can be seen in Figure 7. The findings reveal an enhanced accuracy with the Swap order throughout all corruption categories. The Convention model has an average accuracy across all corruptions at  $41.4\%$ . However, the Swap model enhances this average to  $45.3\%$ .
Asymmetric saturation offers increased robustness to variations in input data. This robustness stems from the behavior of the Tanh function, especially when utilizing the output on its bounds to its midpoint. Near the center, particularly around zero, the Tanh function is highly responsive to slight input changes, leading to considerable alterations in the output. As a result, the Convention model is more vulnerable to such input shifts, especially in the zero vicinity of the Tanh function. In contrast, the Swap model activates Tanh's asymptotic region. Therefore, even pronounced input deviations lead to only slight output changes. Such a trait makes the Swap model inherently robust, maintaining a more consistent output even with input disparities.
Table 3: Experimental result of comparison between Convention and NWDBN models on VGG16_11 trained by CIFAR-100.
Configurations	Accuracy	Avg. Skewness	Avg. Sparsity
Convention	69.5	0.254	0.718
NWDBN	72.22	0.718	0.288
7.4 Further Discussion on the Effect of Asymmetry and Sparsity
The Swap model enhances its performance through both asymmetry and sparsity, making it challenging to distinguish the contributions of each. We choose the Convention order to determine the primary factor influencing performance. When asymmetry increases in the Convention model, the majority of values have large values and it decreases in sparsity. Thus, the NWDBN model, which encourages greater asymmetry in the Convention model, has a reduced sparsity while the level of asymmetry is enhanced. The result of accuracy and averaged Skewness and Sparsity on the Convention and NWDBN models can be seen in Table 3. Notably, even with its diminished sparsity, the asymmetrically saturated NWDBN model surpasses the Convention model's performance. From these observations, we deduce that asymmetry plays a more pivotal role than sparsity.
In this work, we report that the Swap models with bounded activation functions perform better than the Convention models and analyze what brings about performance improvement. Our analysis reveals that the asymmetric saturation and the increased sparsity by the batch normalization after bounded activation functions contribute to the increased accuracy of the Swap models. Interestingly, this makes bounded activation functions behave much like ReLU and enables them to achieve performance comparable to that of ReLU. Most importantly, it expands our understanding of the activation function and will further assist in developing superior activation functions in the future.
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-00456, Development of Ultra-high Speech Quality Technology for Remote Multi-speaker Conference System), and by the National Research Council of Science & Technology (NST) grant by the Korea government (MSIT) [CRC-20-02-KIST].
[1] Bhat, N. V., Minderman, P. A., McAvoy, T., and Wang, N. S. Modeling chemical process systems via neural computation. IEEE Control Systems Magazine, 10(3):24-30, 1990.
[2] Chen, C.-T. and Chang, W.-D. A feedforward neural network with function shape autotuning. Neural networks, 9(4):627-641, 1996.
[3] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
[4] Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
[5] Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
[6] Hasani, M. and Khotanlou, H. An empirical study on position of the batch normalization layer in convolutional neural networks. In 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), pp. 1-4. IEEE, 2019.
[7] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[8] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1704.04861, 2017.
[9] Hurley, N. and Rickard, S. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723-4741, 2009.
[10] Hyvarinen, A. and Oja, E. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411-430, 2000.
[11] Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
[12] Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
[13] Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[14] LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
[15] Rakitianskaia, A. and Engelbrecht, A. Measuring saturation in neural networks. In 2015 IEEE Symposium Series on Computational Intelligence, pp. 1423–1430. IEEE, 2015.
[16] Rakitianskaia, A. and Engelbrecht, A. Saturation in pso neural network training: Good or evil? In 2015 IEEE Congress on Evolutionary Computation (CEC), pp. 125-132. IEEE, 2015.
[17] Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
[18] Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[19] Turian, J., Bergstra, J., and Bengio, Y. Quadratic features and deep architectures for chunking. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pp. 245-248, 2009.
While this paper contains an in-depth analysis of Tanh, it does not propose a function that is superior to existing activation functions. Generally, it is known that Tanh does not perform better than ReLU, but it's hard to find a study that deeply analyzes the reasons for this. To our knowledge, this paper is the most comprehensive analysis of the performance degradation when Tanh is used in a conventional manner. Although it suggests strategies to utilize Tanh more effectively, achieving performance nearly on par with ReLU, it fundamentally does not assert that Tanh surpasses ReLU in terms of effectiveness.
B Accuracy of the Shifted Tanh According to  $\tau$
In this section, we focus on the hyperparameter  $\tau$  in the shifted Tanh function. We carry out experiments using VGG16_11, MobileNet, and PreAct-ResNet models trained on CIFAR-10 and CIFAR-100 datasets, testing different values of the parameter  $\tau$  (-1.5, -1.2, -1.0, -0.8, and -0.5). The performance results corresponding to each of these  $\tau$  values are presented in Table B.1. In these experiments, all hyperparameters, except for  $\tau$ , are maintained at their best-averaged accuracy settings based on each model trained on the CIFAR dataset. Our aim is to identify the effective  $\tau$  value without pushing all inputs into the saturation zone due to excessive asymmetry. The best-performance  $\tau$  is identified by averaging the accuracy across different models for each  $\tau$  and selecting the one with the highest performance. In our case, -1 is the best.
Table B.1: The results of shifted Tanh models on various  $\tau$  values.
τ	-1.5	-1.2	-1.0	-0.8	-0.5
Accuracy	80.56	83.86	84.15	84.01	83.58
C Metric
C.1 Saturation and Skewness on Different Distribution Patterns
Figure C.1: The Saturation (left) and Skewness (right) values on different distributions.
Our saturation metric is designed to quantify the concentration of elements to the maximum absolute output values. The metric has bounds from 0 to 1, and it equates to 0 when all the elements are 0 in the Tanh case. Conversely, it grows larger as elements tend towards the maximum limit of the output range. For example, the saturation metric is evaluated at 0.5 in a uniform distribution. For saturation values of the Tanh output on the different Gaussian distributions, refer to Figure C.1 (left).
Skewness is a measure that quantifies the asymmetry within a distribution. Symmetric distribution will yield the Skewness of 0, and the Skewness increases with the rise in asymmetry. Notably, we calculate the absolute skewness value in our asymmetry metric, ensuring that the metric remains
unaffected by the direction of skewness. For a better understanding of how this measure applies to various distributions, Figure C.1 (right) provides Skewness on each distribution.
C.2 Impacts of Mean and Standard Deviation Variations on Tanh Asymmetry
Figure C.2: Skewness value on the different means and standard deviations of the Gaussian distribution for Tanh input.
The mean and variance of input distribution on Tanh affect the asymmetry of Tanh output. The Skewness of Tanh output on various mean and standard deviation can be found in Figure C.2. In the Skewness of varied mean distribution, the Skewness is increased on the mean increase. However, the maximum Skewness does not align with the standard deviation increases. The Skewness decreases not only the small input standard deviation but also the large input standard deviation. Additionally, in the same mean condition, a decrease in standard deviation from the maximum skewness point more rapidly decreases the Skewness than an increase in standard deviation.
C.3 The Proof of Demonstrating the Sparsity Metric Criteria
This section examines the adequacy of our sparsity metric against the heuristic criteria for sparsity measures defined by Hurley & Rickard [9]. Our sparsity metric meets five of these criteria: Scaling, Rising Tide, Cloning, Bill Gates, and Babies.
We consider the absolute operation as pre-applied to the metric input, according to Hurley & Rickard [9]. Thus, our sparsity metric is represented as  $f: \mathbf{x} \mapsto s$ , where  $\mathbf{x} \in \mathbb{R}^{m+}$  is a vector of absolute values in a channel in BN output, and  $x^{\max+}$  signifies the maximum value in  $\mathbf{x}$ , i.e.,  $x^{\max+} = \max_{i} x_{i}$ . We prove our sparsity metric in the setting of one-channel, which can be expanded to the multi-channel setting without loss of generality. Below are the five criteria our sparsity metric meets and proof for each.
Proof of Scaling. Sparsity is scale invariant. That is, for any scalar  $\alpha \in \mathbb{R}$ , where  $\alpha > 0$ , the function  $f$  satisfies:
$$
f (\alpha \mathbf {x}) = f (\mathbf {x}).
$$
The invariance of scale means that sparsity considers relative differences rather than absolute magnitudes.
Proof. If we scale the vector  $\mathbf{x}$  by  $\alpha$ , we also scale the maximum positive value  $x^{\max +}$  by  $\alpha$ . Therefore, we have:
$$
\begin{array}{l} f (\alpha \mathbf {x}) = 1 - \frac {\sum_ {i = 1} ^ {m} \alpha x _ {i}}{m \alpha x ^ {\max +}} \\ = 1 - \frac {\sum_ {i = 1} ^ {m} x _ {i}}{m x ^ {\max +}} \\ = f (\mathbf {x}). \\ \end{array}
$$
This demonstrates that the sparsity of the vector  $\mathbf{x}$  remains the same when the vector is scaled by any positive scalar  $\alpha$ .
Proof of Rising Tide. Adding a constant to each element decreases sparsity. Formally, for any scalar  $\alpha \in \mathbb{R}$ , where  $\alpha > 0$ , the function  $f$  satisfies:
$$
f (\alpha + \mathbf {x}) <   f (\mathbf {x}).
$$
We exclude the case, as mentioned in [9], where all elements of  $\mathbf{x}$  are the same.
This property also indicates that sparsity increases as more values approach zero.
Proof. We begin by noting the sparsity of the vector  $\mathbf{x} + \alpha$ :
$$
f (\mathbf {x} + \alpha) = 1 - \frac {\sum_ {i = 1} ^ {m} x _ {i} + m \alpha}{m x ^ {\max} + m \alpha}.
$$
Assume for contradiction that  $f(\mathbf{x} + \alpha) \geq f(\mathbf{x})$ . Given that  $mx^{\max +} > \Sigma_{i = 1}^{m}x_{i} > 0$  and  $m\alpha > 0$  the following inequalities can be derived:
$$
\frac {\sum_ {i = 1} ^ {m} x _ {i} + m \alpha}{m x ^ {\max } + m \alpha} > \frac {\sum_ {i = 1} ^ {m} x _ {i}}{m x ^ {\max } +}
$$
$$
1 - \frac {\Sigma_ {i = 1} ^ {m} x _ {i} + m \alpha}{m x ^ {\max} + m \alpha} <   1 - \frac {\Sigma_ {i = 1} ^ {m} x _ {i}}{m x ^ {\max} +}
$$
which contradicts the initial assumption, thus implying that
$$
f (\alpha + \mathbf {x}) <   f (\mathbf {x}),
$$
Proof of Cloning. Sparsity is invariant under cloning.
That is, for a vector  $\mathbf{x}$ , the function  $f$  satisfies:
$$
f (\mathbf {x}) = f (\mathbf {x} | | \mathbf {x}) = f (\mathbf {x} | | \mathbf {x} | | \mathbf {x}) = \dots = f (\mathbf {x} | | \mathbf {x} | | \dots | | \mathbf {x}),
$$
where  $||$  denotes concatenation, such that  $\mathbf{x}||\mathbf{x} = [x_1,x_2,\dots ,x_m,x_1,x_2,\dots ,x_m]$
The principle of cloning implies that sparsity remains the same even if a set of values is replicated.
Proof. Define a function  $g: \mathbf{x}, \delta \mapsto \mathbf{y}$ , where  $\delta \in \mathbb{Z}^+$  and  $\mathbf{y} \in \mathbb{R}^{m\delta}$  is the result of concatenating  $\mathbf{x}$  and  $\delta$  times. We then have:
$$
\begin{array}{l} f (g (\mathbf {x}, \delta)) = 1 - \frac {\delta \Sigma_ {i = 1} ^ {m} x _ {i}}{\delta m x ^ {\mathrm {m a x}} +} \\ = 1 - \frac {\sum_ {i = 1} ^ {m} x _ {i}}{m x ^ {\max } +} \\ = f (\mathbf {x}). \\ \end{array}
$$
This shows that the sparsity of  $\mathbf{x}$  remains the same even when it is concatenated with itself  $\delta$  times.
Proof of Bill Gates. As one individual element becomes infinitely large, the sparsity increases. Formally, for every  $i$ , there exists  $\rho > 0$ , such that for all  $\alpha > 0$ :
$$
f ([ x _ {1}, \dots , x _ {i} + \rho + \alpha , \dots , x _ {m} ]) > f ([ x _ {1}, \dots , x _ {i} + \rho , \dots , x _ {m} ])
$$
The implication is that if a single value grows without bounds, the sparsity also increases indefinitely.
Proof. We first choose a sufficiently large  $\rho$  such that  $\forall i, x_i + \rho > x^{\max +}$ .
Assume for contradiction that  $S([x_1, \dots, x_i + \rho + \alpha, \dots, x_m]) \leq S([x_1, \dots, x_i + \rho, \dots, x_m])$ .
$$
\begin{array}{l} 1 - \frac {\sum_ {k = 1} ^ {m} x _ {k} + \rho + \alpha}{m (x _ {i} + \rho + \alpha)} \leq 1 - \frac {\sum_ {k = 1} ^ {m} x _ {k} + \rho}{m (x _ {i} + \rho)} \\ \frac {\sum_ {k = 1} ^ {m} x _ {k} + \rho + \alpha}{m (x _ {i} + \rho + \alpha)} \geq \frac {\sum_ {k = 1} ^ {m} x _ {k} + \rho}{m (x _ {i} + \rho)} \\ \frac {\sum_ {k \neq i} x _ {k} + x _ {i} + \rho + \alpha}{x _ {i} + \rho + \alpha} \geq \frac {\sum_ {k \neq i} x _ {k} + x _ {i} + \rho}{x _ {i} + \rho} \\ \frac {\sum_ {k \neq i} x _ {k}}{x _ {i} + \rho + \alpha} \geq \frac {\sum_ {k \neq i} x _ {k}}{x _ {i} + \rho} \\ \frac {1}{x _ {i} + \rho + \alpha} \not \geq \frac {1}{x _ {i} + \rho}, \\ \end{array}
$$
leading to a contradiction. Therefore,
$$
f ([ x _ {1}, \dots , x _ {i} + \rho + \alpha , \dots , x _ {m} ]) > f ([ x _ {1}, \dots , x _ {i} + \rho , \dots , x _ {m} ]).
$$
Proof of Babies. Adding a new element of zero increases sparsity. Formally, for a vector  $\mathbf{x}$ , the function  $f$  satisfies:
$$
f (\mathbf {x} | | 0) > f (\mathbf {x}).
$$
Concatenating zero elements to the existing values increases the relative difference to the other values, which increases sparsity.
$$
1 - \frac {\sum_ {k = 1} ^ {m} x _ {k}}{m + 1} > 1 - \frac {\sum_ {k = 1} ^ {m} x _ {k}}{m}
$$
We start by subtracting one from both sides of the inequality:
$$
- \frac {\Sigma_ {k = 1} ^ {m} x _ {k}}{m + 1} > - \frac {\Sigma_ {k = 1} ^ {m} x _ {k}}{m}.
$$
The negative sign can be removed by reversing the inequality:
$$
\frac {\sum_ {k = 1} ^ {m} x _ {k}}{m + 1} <   \frac {\sum_ {k = 1} ^ {m} x _ {k}}{m}.
$$
Multiplying both sides of the inequality by  $m(m + 1)$  (since  $m$  is a positive integer,  $m + 1$  is also positive, and the inequality sign will not change), we get:
$$
m \Sigma_ {k = 1} ^ {m} x _ {k} <   (m + 1) \Sigma_ {k = 1} ^ {m} x _ {k}.
$$
Subtracting  $m\Sigma_{k = 1}^{m}x_{k}$  from both sides of the inequality, we obtain:
$$
0 <   \Sigma_ {k = 1} ^ {m} x _ {k}.
$$
Since  $x_{k}$  is not a negative value, the sum is also non-negative, which verifies the inequality. Therefore, the original inequality is proven.
D Deciding Model Configurations for Investigation
D.1 Investigation of Depth Impact on Accuracy in VGG16 Models
Table E.2: Experimental results of shortened VGG16 models with the Swap order for CIFAR-100. The number of removed convolution layers in the VGG16_n model is the difference between 16 and n.
VGG16	VGG16_15	VGG16_14	VGG16_13	VGG16_12	VGG16_11	VGG16_10	VGG16_9	VGG16_8
Accuracy	70.93	72.97	73.63	73.77	74.00	74.11	72.31	70.95	70.76
Table E.3: Experimental comparison between the official VGG11 and our VGG16_11 models trained by CIFAR-100.
Models	Order
Convention	Swap
VGG11	64.55	69.94
VGG16_11	69.5	74.11
To identify a model for focused analysis using the CIFAR dataset, we examined various VGG16 variants. This examination progressively removes convolution layers from the end towards the front of VGG16. The result can be seen in Table E.2. We observed an increase in accuracy until peaking at the VGG16_11 model, followed by a decline. Although a VGG11 model has already been proposed in Simonyan & Zisserman [18], the validation accuracy of VGG16_11 is significantly higher than VGG11 on CIFAR-100. The results can be seen in Table E.3.
E Training Hyperparameters
We sweep the learning rate and weight decay hyperparameter. The learning rate was 0.1 and 0.01. For CIFAR and Tiny-ImageNet datasets, we trained models with a batch size of 128, and the learning rate was reduced by one-tenth at 100 and 150 of the total 200 epochs, and we swept four weight decay of 0.005, 0.001, 0.0005, and 0.0001. For ImageNet datasets, we trained models with a batch size of 256, and the learning rate was reduced by one-tenth at 30 and 60 of the total 100 epochs, and we swept three weight decays of 0.001, 0.0005, and 0.0001. We chose the best averaged accuracy model for the three random seeds and averaged the values of these three models for all measurements for analysis. Because of the computation issue, we only use one seed for the ImageNet dataset with early stopping. The hyperparameters of VGG, MobileNet, and PreAct-ResNet are shown in E.1, E.2, E.3, respectively.
E.1 VGG
Table F.1: Hyperparameters of VGG11 with Tanh
Convention	Swap	NWDBN
Training Epochs	200	200	200
Learning Rate	0.1	0.1	0.1
Learning Rate Drop	100, 150	100, 150	100, 150
Weight Decay	0.0005	0.0005	0.0005
Batch Size	128	128	128
Table F.2: Hyperparameters of VGG16 with Tanh
Convention	Swap
CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet	CIFAR-10	CIFAR-100	Tiny ImageNet	ImageNet
Training Epochs	200	200	200	90	200	200	200	90
Learning Rate	0.1	0.01	0.01	0.01	0.01	0.1	0.01	0.01
Learning Rate Drop	100, 150	100, 150	100, 150	30, 60	100, 150	100, 150	100, 150	30, 60
Weight Decay	0.0001	0.0005	0.001	0.0001	0.001	0.0005	0.001	0.001
Batch Size	128	128	128	256	128	128	128	256