Title: ENHANCING BATCH NORMALIZED CONVOLUTIONAL NETWORKS USING DISPLACED RECTIFIER LINEAR UNITS: A SYSTEMATIC COMPARATIVE STUDY

Abstract: In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory technique to train deep networks currently. We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p < 0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

1. INTRODUCTION:

The recent advances in deep learning research have produced more accurate image, speech, and language recognition systems and generated new state-of-the-art machine learning applications in a broad range of areas such as mathematics, physics, healthcare, genomics, financing, business, agriculture, etc. Although advances have been made, accuracy performance enhancements have usually demanded considerably deeper or more complex models, which tend to increase the required computational resources (processing time and memory usage).

Instead of increasing deep models depth or complexity, a less computational expensive alternative approach to enhance deep learning performance across-the-board is to design more efficient activation functions. Even if computational resources are no issue, to employ enhanced activation functions nevertheless contributes to speeding up learning and achieving higher accuracy. Indeed, by allowing the training of deep neural networks, the discovery of Rectified Linear Units (ReLU) (Nair & Hinton, 2010; Glorot et al., 2011; Krizhevsky et al., 2012) was one of the main factors that contributed to deep learning advent. ReLU allowed achieving higher accuracy in less time by avoiding the vanishing gradient problem (Hochreiter, 1991) . Before ReLU, activation functions such as Sigmoid and Hyperbolic Tangent were unable to train deep neural networks because of the absence of the identity function for positive input.

However, ReLU presents drawbacks. For example, some researchers argument that zero slope avoids learning for negative values (Maas et al., 2013; He et al., 2016b) . Therefore, other activation functions like Leaky Rectifier Linear Unit (LReLU) (Maas et al., 2013) , Parametric Rectifier Linear Unit (PReLU) (He et al., 2016b) and Exponential Linear Unit (ELU) (Clevert et al., 2015) were proposed (Appendix A). Unfortunately, there is no consensus about how these proposed nonlinearities compare to ReLU, which therefore remains the most used activation function in deep learning.

Similar to activation functions, batch normalization (Ioffe & Szegedy, 2015) currently plays a fundamental role in training deep architectures (Appendix B) . This technique normalizes the inputs of each layer, which is equivalent to normalizing the outputs of the deep model previous layer. However, before being used as inputs for the subsequent layer, the normalized data are typically fed into activation functions (nonlinearities), which necessarily skew the otherwise normalized distributions. In fact, ReLU only produces non-negative activations, which is harmful to the previously normalized data. The outputs mean values after ReLU are no longer zero, but rather necessarily positives. Therefore, the ReLU skews the normalized distribution (Section 2).

Aiming to mitigate the mentioned problem, we concentrate our attention on the interaction between activation functions and batch normalization. We conjecture that nonlinearities that are more compatible with batch normalization present higher performance. After that, considering that an identity transformation preserves any statistical distribution, we assume that to extend the identity function from the first quadrant to the third implies less damage to the normalization procedure.

Hence, we investigate and propose the activation function Displaced Rectifier Linear Unit (DReLU), which partially prolongs the identity function beyond origin. Hence, DReLU is essentially a ReLU diagonally displaced into the third quadrant. Different from all other previous mentioned activation functions, the inflection of DReLU does not happen at the origin, but in the third quadrant.

Considering the widespread adoption and practical importance, we used Convolutional Neural Networks (CNN) (LeCun et al., 1998; Krizhevsky et al., 2012) in our experiments. Moreover, as particular examples of CNN architectures, we used the previous ImageNet Large Scale Visual Recognition Competition (ILSVRC) winners Visual Geometry Group (VGG) (Simonyan & Zisserman, 2014) and Residual Networks (ResNets) (He et al., 2016a; c) . These architectures have distinctive designs and depth to promote generality to the conclusions of this work. In this regard, we evaluated how replacing the activation function impacts the performance of well established and widely used standard state-of-the-art models. Finally, we decided to employ the two most broadly used computer vision datasets by deep learning research community: CIFAR-100 (Krizhevsky, 2009) and CIFAR-10 (Krizhevsky, 2009) .

In this systematic comparative study, performance assessments were carried out using statistical tests with a significance level of 5% (Appendix C.5). At least ten executions of each of experiment were executed. However, when the mentioned significance level was not achieved, ten additional runs were performed.

2. DISPLACED RECTIFIER LINEAR UNITS:

Consider x the input of a layer composed of a generic transformation Wx+b followed by a nonlinearity, for instance, ReLU. After the addition of the batch normalization layer, the overall joint transformation performed by the block (composed layer) is given by:

EQUATION

For a moment, consider the intermediate activation y produced inside the block:

EQUATION

Without loss of generality, assume γ = 1 and β = 0 (Ioffe & Szegedy, 2015) . Therefore, since y is the output of a batch normalization layer, we can rewrite unbiased estimators for the expected value and variance of any given dimension k as follows (Appendix B):

Ê[ŷ (k) ] = 0, Var[ŷ (k) ] = 1 (3)

Consequently, we investigate the expected value and variance of the activations distribution produced by the combined layer. Therefore, if z is the output of the block using a ReLU nonlinearity, it immediately follows that:

EQUATION

The application of ReLU removes all negative values from a distribution. Hence, it necessarily produces positive mean as output. Therefore, it can be written:

Ê[z (k) ] = Ê[ReLU(ŷ (k) )] > Ê[ŷ (k) ] = 0 (5)

Considering that a distribution with Ê[ŷ (k) ] = 0 has to present negative values, replacing all negative activations with zeros makes the variance of z (k) necessarily lower than the variance of the original distribution ŷ(k) . Consequently, we can rewrite:

Var[z (k) ] = Var[ReLU(ŷ (k) )] < Var[ŷ (k) ] = 1 (6)

Therefore, despite batch normalization, the activations after the whole block are not perfectly normalized since these outputs present neither zero mean nor unit variance. Consequently, regardless of the presence of a batch normalization layer, after the ReLU, the inputs passed to the next composed layer have neither mean of zero nor variance of one that was the objective in the first place.

In this sense, ReLU skews an otherwise previous normalized output. In other words, ReLU reduces the correction of the internal covariance shift promoted by the batch normalization layer. Consequently, we conclude the ReLU bias shift effect (Clevert et al., 2015) is directly related to the drawback ReLU generates to the batch normalization.

Consequently, we propose DReLU, which is essentially a diagonally displaced ReLU. It generalizes both ReLU and SReLU (Clevert et al., 2015) by allowing its inflection to move diagonally from the origin to any point of the form (-δ, -δ). If δ = 0, DReLU becomes ReLU. If δ = 1, DReLU becomes SReLU. Therefore, the slope zero component of the activation function provides negative activations, instead of null ones. Unlike ReLU, in DReLU learning can happen for negative inputs since gradient is not necessarily zero. The following equation defines DReLU:

EQUATION

DReLU can be regarded as a generalization of the Shifted Rectifier Linear Unit (SReLU) (Clevert et al., 2015) . In fact, instead of always prolong the identity to the point (-1, -1), in DReLU we established a hyperparameter δ that defines the most appropriate point (-δ, -δ) where the inflection should happen. In this sense, SReLU is a particular case of DReLU where δ = 1. Considering that the experiments we performed to determine the DReLU hyperparameter contemplates δ = 1 as a possible value for δ, it was not necessary to include SReLU in the present comparative study. Indeed, our experiments showed that the addition of the parameter δ allowed DReLU to significantly outperform SReLU in all of our hyperparameter definition experiments (Appendix C.6). Considering that DReLU replaces ReLU in Eq. 1, the activations of the composed layer become:

EQUATION

Since DReLU extends the identity function into the third quadrant, it is no longer possible to conclude Eq. 5 is valid. Therefore, the consequence presented in the mentioned equation is probably at least minimized. In this case, we can conclude that Ê[z (k) ] DReLU is much probably near to zero than Ê[z (k) ] ReLU . Hence, we can rewrite:

EQUATION

Furthermore, DReLU exhibits a noise-robust deactivation state for very negative inputs, a feature not granted by LReLU and PReLU. A noise-robust deactivation state is achieved by setting the slope zero for highly negative values of input (Clevert et al., 2015) . Some authors argument that activation functions with this propriety improve learning (Clevert et al., 2015) . Finally, DReLU is less computationally complex than LReLU, PReLU, and ELU. In fact, since DReLU has the same shape of ReLU, it essentially has the same computational complexity.

3. EXPERIMENTS:

In this comparative study, we define an experiment as the training of a deep model using a distinct activation function on a given dataset. If not otherwise mentioned, we conducted ten executions of each experiment. We define a scenario as the set of experiments regarding all activation functions on a specific dataset using a particular model. In this regard, this paper presents the consolidated results of six scenarios (two datasets versus three models) that correspond to 30 experiments, which in turn represents a total of 320 executions (training of deep neural networks). In two cases, we executed 20 instead of 10 runs of a given experiment to achieve the desired statistical significance.

We trained the models during 100 epochs since it was enough to the test accuracy to saturate. At epochs 40 and 70, we evaluated the test accuracy of the partially trained models. Therefore, we were able to assess how fast each model was learning to generalize based on the activation function used by the model. This is important for compare the expected performance of the activation functions in applications where the models need to provide high test accuracy training only a few tens of epochs.

Since the training time of an epoch shows no significant difference among the activation functions, we said that the nonlinearity that provided the best test accuracy in these terms to be learning faster.

The Appendix C provides a detailed explanation of the performed experiments.

All experiments were conducted without using dropout (Srivastava et al., 2014) since recent studies have shown that, despite improving the training time, dropout provides unclear contributions to the overall deep model performance (Ioffe & Szegedy, 2015) . Moreover, dropout has recently become a technique restricted to fully-connected layers, which in turn are being less used and replaced by an average pooling layer in more recent architectures (He et al., 2016a; Huang et al., 2016; He et al., 2016c; Zagoruyko & Komodakis, 2016; 2017) . Therefore, since currently fully connected layers are rarely used in modern CNN, the usage of dropout is accordingly becoming unusual. This can be demonstrated by observing that the most recent CNN models are not using dropout, but only batch normalization (He et al., 2016a; Huang et al., 2016; He et al., 2016c; Zagoruyko & Komodakis, 2016; 2017) . Particularly in the case of DenseNets (Huang et al., 2016) , the results just using batch normalization are significantly better than using both techniques.

This recent tendency of design modern deep networks using only batch normalization but avoiding dropout can also be observed in the discriminative and generative convolutional models recently used in Generative Adversarial Networks (GANs) (Radford et al., 2015) . Hence, we emphasize that we designed the experiments of this comparative study to reflect the scenario we believe is currently the most likely and relevant from the perspective of training modern CNNs, which contemplates the use exclusively of batch normalization and no dropout.

The comparative study provided in the paper was designed to be self-contained to avoid misleading comparisons of experiments performed in entirely different situations. In this sense, it should be noticed that the results presented by the papers that proposed the activation functions which are being used in this study (LReLU (Maas et al., 2013) , PReLU (He et al., 2016b) and ELU (Clevert et al., 2015) ), must not be compared to the ones presented here because of the following reasons.

First, the studies previously mentioned were performed with use of dropout and without batch normalization, which is a technique that was not available when the mentioned studies were conducted. The only exception is ELU, where a few tested scenarios used batch normalization. However, even in those cases, dropout was always and intensively employed. Second, the cited studies did not use standardized models such as VGG or ResNet where the only factor of change was the compared activation functions. For example, in ELU paper, hand designed models were used to compare the performance of the activation function or in some cases completely different models were compared. Third, the results presented by the mentioned works did not use statistical tests. In this sense, considering the variation of the performance of the experiments based on different initialization or data shuffling, the conclusions may not be much trustable from a statistical point of view.

Hence, the conclusions regarding the performance of the cited activation functions based on their original paper may not be valid in the context where only batch normalization, but no dropout, is used to regularize the deep models. In fact, one of the significant contributions of this paper is providing a systematic statistical supported comparative study using standardized models in the currently predominant scenario of using batch normalization without dropout.

4. RESULTS AND DISCUSSION:

In the following subsections, we analyze the tested scenarios. In each case, we first discuss the activation functions learning speed based on test accuracy obtained for the partially trained models. Subsequently, we comment about the test accuracy performances of the activation functions, which corresponds to the respective model test accuracy evaluated after 100 epochs. Naturally, we consider that an activation function presents better test accuracy if it showed the higher test accuracy for the final trained models on a particular dataset.

In all scenarios, the null hypotheses were the test accuracy samples taken from different activation functions originated from the same distribution. In other works, all the compared activation functions have the same test accuracy performance in the particular scenario. The null hypotheses were rejected for all scenarios (Table 1 ), which means that with statistical significance (p < 0.05) at least one of the activation functions presents a test accuracy performance that is different from the others activation functions. Therefore, we used the Conover-Iman post-hoc tests for pairwise multiple comparisons for all combination of datasets and models (Tables 3, 4 , 5, 7, 8, 9). In these tables, the best results and p-values of the comparison of DReLU to other activation functions are in bold. 

4.1. CIFAR-100 DATASET:

The Table 2 presents the mean of the nonlinearities layers mean activations performed in the CIFAR-100 training dataset. It shows that DReLU is more capable of reducing the bias shift effect during training than ReLU. Therefore, as expected and in agreement with 9, all of our experiments showed that the identity mapping extension produced less damage to the normalization performed by the previous layer and in fact mitigated the bias shift effect when compared to ReLU. In the case of ResNet-56, DReLU overcame the test accuracy results of the other activation functions on either 40 and 70 epochs once again. Therefore, we concluded DReLU generated the fastest learning (Fig. 2 ). Regarding test accuracy, DReLU outperformed ReLU (p < 0.00228) and all other options, with exception to LReLU. Although, no statistical significance was achieved in this pairwise comparison (Table 4 ). In ResNet-100, DReLU also provided the fastest learning in all situations (Fig. 3 ). Finally, DReLU test accuracy outperformed ReLU (p < 2.3 × 10 -5 ) and all others activation functions again (Table 5 ). Therefore, in CIFAR-100 as a whole, DReLU presented the fastest learning for all three models considered. The results showed DReLU always outperformed ReLU test accuracy in all studied models. Besides, DReLU was the most accurate in two evaluated models and the second in the other scenario. In VGG-19 model, DReLU provided faster learning than any other activation function for either 40 and 70 epochs (Fig. 4 ). Moreover, DReLU presented the best test accuracy performance (Table 7 ).

We performed 20 executions of either DReLU and ReLU experiment. In the case of ResNet-56, DReLU provided faster learning than any other nonlinearity on either 40 and 70 epochs in this scenario (Fig. 5 ). Furthermore, DReLU was again the most accurate followed by ReLU (Table 8 ). In relation to ResNet-110, DReLU provided the fastest leaning on either 40 and 70 epochs once more (Fig. 6 ). DReLU was the most accurate solution also for this scenario (Table 9 ). In this case, we also performed 20 runs of DReLU and ReLU. Hence, in the CIFAR-10 as a whole, DReLU also presented the best learning speed for all considered models. Moreover, the results showed DReLU again surpassed ReLU test accuracy in all analyzed scenarios. Furthermore, DReLU was the most accurate activation function in all evaluated models. 4.3 DISCUSSION Primarily, we reemphasize the studies that proposed the activation functions compared in this paper used significantly different (specifically designed) models from the (standardized) ones used in this study. In this sense, it is not possible to make a direct comparison between their results and the ones presented in this work. In fact, in the mentioned papers, the usage regular of dropout may have produced a no ideal performance from ReLU since its fast training capacity was probably reduced.

In the mentioned papers, no experiments were executed using batch normalization without dropout.

As this study presents a significantly different scenario, we can expect different conclusions from this work. Moreover, this paper performed statistical tests to prove that, in the conditions which experiments were executed, the proposed solution is consistently better than the other options. To do that, we performed a significant number of executions of each experiment. It is an important point to consider since the performance of same models presents a slight variation each time they are trained on a given dataset.

Taking into consideration the previous comments, in our experiments, regarding the CIFAR-100 dataset, DReLU presented the fastest learning for all three models considered. Moreover, the results showed DReLU always outperformed ReLU test accuracy in all studied models. Besides, DReLU was the most accurate in two evaluated models and the second in the other one. In the CIFAR-10 dataset, DReLU also presented the best learning speed for all considered models. Moreover, the results showed DReLU surpassed ReLU test accuracy in all analyzed scenarios. Actually, DReLU was the most accurate activation function in all evaluated models.

It is important to mention that we commonly observed that ReLU usually produced the second best training speed and test accuracy performance. This apparent surprise result may be explained by the use of batch normalization. Indeed, the correction of the internal covariate shift problem enabled by the batch normalization technique acted relatively in benefit of ReLU and detriment of the other previously proposed units. Hence, batch normalization significantly helped to avoid the so-called "dying ReLU" problem (Karpathy, 2017; Maas et al., 2013; Cunningham et al., 2017) .

In fact, even if a substantial gradient pushes the weights to the zero gradient region of ReLU, the normalization process tends to bring them back to inflection region of ReLU, which avoids the ReLU to die. This fact can explain why ReLU typically outperforms LReLU, PReLU, and ELU in these situations but apparently did not when these activation functions were proposed a few years ago before the batch normalization advent.

The fact that batch normalization relatively helped ReLU in detriment of LReLU, PReLU and ELU make particularly impressive the ability of DReLU to overcome the performance of ReLU in exclusively batch normalized networks. Particularly remarkable is the ability of DReLU to enhance the training speed during the first decades significantly.

5. CONCLUSION:

In this paper, we have proposed a novel activation function for deep learning architectures, referred to as DReLU. The results showed that DReLU presented better learning speed than the all alternative activation functions, including ReLU, in all models and datasets. Moreover, the experiments showed DReLU was more accurate than ReLU in all situations. Besides, DReLU also outperformed test accuracy results of all others investigated activation functions (LReLU, PReLU, and ELU) in all scenarios with one exception. The experiments used batch normalization but avoided dropout. Furthermore, they were designed to cover standard and commonly used datasets (CIFAR-100 and CIFAR-10) and models (VGG and Residual Networks) of several depths and architectures.

In addition to enhancing deep learning performance (learning speed and test accuracy), DReLU is less computationally expensive than LReLU, PReLU, and ELU. Moreover, the mentioned gains were obtained by just replacing the activation function of the model, without any increment in depth or architecture complexity, which usually increases the computational resource requirements as processing time and memory usage.

This paper showed that the batch normalization procedure acted in the benefice of ReLU while other previews proposed activation functions appear not to perform as expected. We believe this happened because batch normalization avoids the so-called "dying ReLU" problem, something that others activation functions were already not affected by in first place.

Furthermore, considering some evaluated models included skip connections, which are a tendency in the design of deep architectures like ResNets, we conjecture the results may generalize to other deep architectures such DenseNets (Huang et al., 2016 ) that also use this structure.

A ACTIVATION FUNCTIONS:

Currently, all major activation functions adopt the identity transformation to positive inputs, some particular function for negative inputs, and an inflection on the origin. In the following subsections, we describe the activation functions compared in this work.

- A.1 RELU ReLU has become the standard activation function used in deep networks (Fig. 7a ). Its simplicity and high performance are the main factors behind this fact. The follow equation defines ReLU:

2 -1 0 1 2 0 1 2 0 0 (a) -2 -1 0 1 2 0 1 2 0 0 (b) -2 -1 0 1 2 0 1 2 0 0 (c) -2 -1 0 1 2 0 1 2 0 0 (d)

EQUATION

The Eq. ( 10) implies ReLU has slope zero for negative inputs and slope one for positive values. It was first used to improve the performance of Restricted Boltzmann Machines (RBM) (Nair & Hinton, 2010) . After that, ReLU was used in other neural networks architectures (Glorot et al., 2011) . Finally, ReLU has provided superior performance in the supervised training of convolutional neural network models (Krizhevsky et al., 2012) .

B BATCH NORMALIZATION:

In machine learning, normalizing the distribution of the input data decreases the training time and improves test accuracy (Tax & Duin, 2002) . Consequently, normalization also improves neural networks performance (LeCun et al., 2012) . A standard approach to normalizing input data distributions is the mean-standard technique. In this method, the input data is transformed to present zero mean and standard deviation of one.

However, if instead of working with shallow machine learning models, we are dealing with deep architectures; the problem becomes more sophisticated. Indeed, in a deep structure, the output of a layer works as input data to the next. Therefore, in this sense, each layer of a deep model has his own "input data" that is composed of the previous layer output or activations. The only exception is the first layer, for which the input is the original data.

In fact, consider that the stochastic gradient decent (SGD) is being used to optimize the parameters θ of a deep model. Assume S a sample of m training examples in a mini-batch; then the SGD minimizes the loss given by the equation:

EQUATION

Furthermore, consider x the output of the layer i-1. These activations are fed into layer i as inputs.

In turn, the outputs of layer i are fed into layer i+1 producing the overall loss given bellow:

EQUATION

In the above equation, F i+1 and F i are the transformation produced by the layers i + 1 and i, respectively. The G function represents the mapping perpetrated by the above layers combined with the loss function adopted as the criterion. Considering that the output of layer i is given by y = F i (x, Θ i ), we can rewrite the above equation as follows:

EQUATION

Applying equation (13) to equation ( 15) and considering a learning rate λ, we can write the equation to update the parameters of the layer i+1 as the follows:

EQUATION

In fact, the above equation is mathematically equivalent to training the layer i+1 on the input data given by y, which in turn is the output of the previous layer. Therefore, indeed, we can understand the output of the previous layer as "input data" for the effect of training the current layer using SGD.

Considering each layer has its own "input data" (the output of the previous layer), normalizing only the actual input data of a deep neural network produces a limited effect in enhancing learning speed and test accuracy. Moreover, during the training process, the distribution of the input of each layer changes, which makes training even harder. Indeed, the parameters of a layer are updated while its input (the activations of the previous layer) is modified.

This phenomenon is called internal covariant shift, which is a major factor that hardens the training of deep architectures (Ioffe & Szegedy, 2015) . In fact, while the data of shallow models is normalized and static during training, the input of a deep model layer, which is the output of the previous one, is neither a priori normalized nor static throughout training.

Batch normalization is an effective method to mitigate the internal covariant shift (Ioffe & Szegedy, 2015) . This approach, which significantly improves training speed and test accuracy, proposes normalizing the inputs of the layers when training deep architectures.

The layers inputs normalization is performed after the each mini-batch training to synchronizing with the deep network parameters update. Therefore, when using batch normalization, for a input x = (x (1) , x (2) , . . . , x (d) ), each individual dimension is transformed as follows:

EQUATION

C.6 HYPERPARAMETER EXPERIMENTS Clearly, δ must not go to infinity because we would lose the nonlinearity in such a case. Therefore, a compromise that causes less damage to the normalization while keeps the activation function nonlinearity has to be achieved. To define the hyperparameter value of DReLU, we performed the experiments showed in Table 10 . The results are the mean and standard deviation values of each experiment five executions. Based on these experimental results, we decided to set δ = 0.05. The blue line presents the performance of SReLU, which can be observed is considerably lower than the DReLU using the chosen hyperparameter δ = 0.05. 

:



Figures and Tables:

Figure 1: VGG-19 model test accuracy means and standard deviations on the CIFAR-100 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 2: ResNet-56 model test accuracy means and standard deviations on the CIFAR-100 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 3: ResNet-110 model test accuracy means and standard deviations on the CIFAR-100 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 4: VGG-19 model test accuracy means and standard deviations on the CIFAR-10 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 5: ResNet-56 model test accuracy means and standard deviations on the CIFAR-10 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 6: ResNet-110 model test accuracy means and standard deviations on the CIFAR-10 dataset. (a) 40 trained epochs. (b) 70 trained epochs.

Figure 7: Previous proposed activation function. (a) ReLU. (b) LReLU. (c) PReLU. (d) ELU.

Kruskal-Wallis test results 4.135 × 10 -9 2.383 × 10 -8 2.48 × 10 -8 1.922 × 10 -11 4.097 × 10 -8 4.7 × 10 -10

Table content: <table><tr><td/><td/><td>CIFAR-100</td><td/><td/><td>CIFAR-10</td><td/></tr><tr><td>Score</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>χ2 (4)</td><td>44.918</td><td>41.253</td><td>41.169</td><td>56.087</td><td>40.115</td><td>49.451</td></tr><tr><td>p-value</td><td/><td/><td/><td/><td/><td/></tr></table>

CIFAR-100 averaged nonlinearities layers mean activationsIn relation to VGG-19, the experiments showed DReLU outperformed the test accuracy results of ReLU and all other assessed activation functions on either 40 and 70 epochs. In this sense, the results demonstrated that DReLU produced the fastest training (Fig.1). Moreover, DReLU presented the best final test performance (Table3) in this case.

Table content: <table><tr><td/><td/><td>40 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0950 ± 0.0016</td><td>0.1288 ± 0.0016</td><td>0.0852 ± 0.0020</td></tr><tr><td>DReLU</td><td>0.0578 ± 0.0010</td><td>0.0972 ± 0.0030</td><td>0.0551 ± 0.0012</td></tr><tr><td/><td/><td>70 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0744 ± 0.0008</td><td>0.1058 ± 0.0015</td><td>0.0680 ± 0.0015</td></tr><tr><td>DReLU</td><td>0.0362 ± 0.0009</td><td>0.0743 ± 0.0024</td><td>0.0390 ± 0.0008</td></tr><tr><td/><td/><td>100 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0640 ± 0.0006</td><td>0.0943 ± 0.0013</td><td>0.0594 ± 0.0013</td></tr><tr><td>DReLU</td><td>0.0261 ± 0.0007</td><td>0.0618 ± 0.0023</td><td>0.0300 ± 0.0007</td></tr></table>

CIFAR-100 VGG-19 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>73.40 ± 0.22</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LReLU</td><td>73.01 ± 0.22</td><td>p &lt; 2.2 × 10 -5</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>71.55 ± 0.24</td><td>p &lt; 1.7 × 10 -13</td><td>p &lt; 1.1 × 10 -6</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>67.57 ± 0.17</td><td>p &lt; 2 × 10 -16</td><td>p &lt; 5.6 × 10 -14</td><td>p &lt; 6.7 × 10 -6</td><td>-</td></tr><tr><td colspan="2">DReLU 73.69 ± 0.21</td><td>p &lt; 0.00012</td><td/><td/><td/></tr></table>

CIFAR-100 ResNet-56 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>73.70 ± 0.22</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="3">LReLU 74.20 ± 0.31 p &lt; 6.2 × 10 -5</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>72.53 ± 0.29</td><td colspan="2">p &lt; 1.5 × 10 -5 p &lt; 5.4 × 10 -12</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>70.13 ± 0.41</td><td>p &lt; 5.7 × 10 -11</td><td>p &lt; 2 × 10 -16</td><td>p &lt; 0.000 59</td><td>-</td></tr><tr><td colspan="2">DReLU 74.12 ± 0.38</td><td>p &lt; 0.002 28</td><td>p &lt; 0.243 00</td><td colspan="2">p &lt; 2.7 × 10 -10 p &lt; 2.4 × 10 -15</td></tr></table>

CIFAR-100 ResNet-110 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>75.70 ± 0.28</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LReLU</td><td>75.94 ± 0.29</td><td>p &lt; 0.0286</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>74.86 ± 0.37</td><td>p &lt; 1.2 × 10 -5</td><td>p &lt; 5.3 × 10 -9</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>70.90 ± 0.42</td><td>p &lt; 3.6 × 10 -11</td><td>p &lt; 2.9 × 10 -14</td><td>p &lt; 0.0005</td><td>-</td></tr><tr><td colspan="3">DReLU 76.20 ± 0.31 p &lt; 2.3 × 10 -5</td><td>p &lt; 0.0286</td><td colspan="2">p &lt; 1.2 × 10 -5 p &lt; 3.6 × 10 -11</td></tr><tr><td colspan="3">4.2 CIFAR-10 DATASET</td><td/><td/><td/></tr></table>

CIFAR-10 averaged nonlinearities layers mean activations

Table content: <table><tr><td/><td/><td>40 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0630 ± 0.0009</td><td>0.1006 ± 0.0027</td><td>0.0669 ± 0.0009</td></tr><tr><td>DReLU</td><td>0.0293 ± 0.0010</td><td>0.0671 ± 0.0016</td><td>0.0376 ± 0.0011</td></tr><tr><td/><td/><td>70 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0449 ± 0.0006</td><td>0.0783 ± 0.0014</td><td>0.0515 ± 0.0006</td></tr><tr><td>DReLU</td><td>0.0139 ± 0.0003</td><td>0.0471 ± 0.0011</td><td>0.0244 ± 0.0009</td></tr><tr><td/><td/><td>100 Epochs Performance Evaluation</td><td/></tr><tr><td>Nonlinearity</td><td>VGG-19</td><td>ResNet-56</td><td>ResNet-110</td></tr><tr><td>ReLU</td><td>0.0364 ± 0.0004</td><td>0.0673 ± 0.0011</td><td>0.0435 ± 0.0005</td></tr><tr><td>DReLU</td><td>0.0066 ± 0.0002</td><td>0.0351 ± 0.0008</td><td>0.0161 ± 0.0007</td></tr></table>

CIFAR-10 VGG-19 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>93.82 ± 0.13</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LReLU</td><td>93.35 ± 0.18</td><td>p &lt; 1.4 × 10 -9</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>93.27 ± 0.22</td><td>p &lt; 2.2 × 10 -10</td><td>p &lt; 0.694 49</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>91.40 ± 0.15</td><td>p &lt; 2 × 10 -16</td><td>p &lt; 0.000 22</td><td>p &lt; 0.000 81</td><td>-</td></tr><tr><td colspan="2">DReLU 93.92 ± 0.11</td><td>p &lt; 0.00330</td><td/><td/><td/></tr></table>

CIFAR-10 ResNet-56 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>93.29 ± 0.21</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LReLU</td><td>93.01 ± 0.21</td><td>p &lt; 1.2 × 10 -6</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>93.01 ± 0.10</td><td>p &lt; 4.7 × 10 -5</td><td>p &lt; 0.9726</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>90.21 ± 0.11</td><td>p &lt; 4.0 × 10 -13</td><td>p &lt; 1.2 × 10 -6</td><td>p &lt; 1.3 × 10 -6</td><td>-</td></tr><tr><td colspan="2">DReLU 93.52 ± 0.14</td><td>p &lt; 0.0027</td><td/><td/><td/></tr></table>

CIFAR-10 ResNet-110 100 epochs performance evaluation Test accuracy means, standard deviations and post hoc tests p-values

Table content: <table><tr><td>Unit</td><td>Accuracy (%)</td><td>ReLU</td><td>LReLU</td><td>PReLU</td><td>ELU</td></tr><tr><td>ReLU</td><td>94.00 ± 0.18</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LReLU</td><td>93.56 ± 0.26</td><td>p &lt; 1.6 × 10 -7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PReLU</td><td>93.74 ± 0.09</td><td>p &lt; 2.3 × 10 -5</td><td>p &lt; 0.257 54</td><td>-</td><td>-</td></tr><tr><td>ELU</td><td>90.76 ± 0.29</td><td>p &lt; 1.8 × 10 -13</td><td>p &lt; 0.004 73</td><td>p &lt; 0.000 13</td><td>-</td></tr><tr><td colspan="2">DReLU 94.11 ± 0.18</td><td>p &lt; 0.03614</td><td colspan="3">p &lt; 1.6 × 10 -7 p &lt; 2.3 × 10 -5 p &lt; 1.8 × 10 -13</td></tr></table>



Table content: <table><tr><td/><td/><td colspan="2">: Hyperparameter experiments</td></tr><tr><td/><td colspan="3">Displaced Rectifier Linear Unit</td></tr><tr><td/><td/><td colspan="2">CIFAR-10 test accuracy</td></tr><tr><td>δ</td><td colspan="4">VGG-19 (%) ResNet-56 (%) ResNet-110 (%) Mean (%)</td></tr><tr><td colspan="2">0.01 93.89±0.20</td><td>93.45±0.13</td><td>93.98±0.19</td><td>93.77</td></tr><tr><td colspan="2">0.05 93.91±0.18</td><td>93.50±0.07</td><td>94.10±0.10</td><td>93.83</td></tr><tr><td colspan="2">0.10 94.00±0.13</td><td>93.43±0.25</td><td>93.93±0.14</td><td>93.78</td></tr><tr><td colspan="2">0.25 93.67±0.19</td><td>92.93±0.19</td><td>93.31±0.12</td><td>93.30</td></tr><tr><td colspan="2">0.50 92.70±0.08</td><td>91.62±0.37</td><td>92.04±0.17</td><td>92.12</td></tr><tr><td colspan="2">1.00 90.96±0.16</td><td>88.64±0.33</td><td>89.26±0.42</td><td>89.62</td></tr></table>

Peer review (iclr2018):
 The key argument authors present against ReLU+BN is the fact that using ReLU after BN skews the values resulting in non-normalized activations. Although the BN paper suggests using BN before non-linearity many articles have been using BN after non-linearity which then gives normalized activations (https://github.com/ducha-aiki/caffenet-benchmark/blob/master/batchnorm.md) and also better overall performance. The approach of using BN after non-linearity is termed "standardization layer" (https://arxiv.org/pdf/1301.4083.pdf). I encourage the authors to validate their claims against simple approach of using BN after non-linearity.