SAU: SMOOTH ACTIVATION FUNCTION USING CONVO- LUTION WITH APPROXIMATE IDENTITIES
Year: 2021

Abstract:
Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on the CIFAR100 dataset.

text: Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on the CIFAR100 dataset.

heading: 1 INTRODUCTION
text: Deep networks form a crucial component of modern deep learning. Non-linearity is introduced in such networks by the use of activation functions, and the choice has a substantial impact on network performance and training dynamics. Designing a new novel activation function is a difficult task. Handcrafted activations like Rectified Linear Unit (ReLU) (Nair & Hinton (2010)), Leaky ReLU (Maas et al. (2013)) or its variants are very common choices for activation functions and exhibits promising performance on different deep learning tasks. There are many activations that have been proposed so far and some of them are ELU (Clevert et al. (2016)), Parametric ReLU (PReLU) (He et al. (2015a)), Swish (Ramachandran et al. (2017)), Tanhsoft (Biswas et al. (2021d)), EIS (Biswas et al. (2021b)), Padé Activation Unit (PAU) (Molina et al. (2020)), Orthogonal Padé Activation Unit (OPAU) (Biswas et al. (2021a)), ACON (Ma et al. (2021)), ErfAct (Biswas et al. (2021c)), Mish (Misra (2020)), GELU (Hendrycks & Gimpel (2020)), ReLU6 (Krizhevsky (2010)), Softplus (Zheng et al. (2015)) etc. Nevertheless, ReLU remains the favourite choice among the deep learning community due to its simplicity and better performance when compared to Tanh or Sigmoid, though it has a drawback known as dying ReLU in which the network starts to lose the gradient direction due to the negative inputs and produces zero outcome. In 2017, Swish (Ramachandran et al. (2017)) was proposed by the Google brain team. Swish was found by automatic search technique, and it has shown some promising performance across different deep learning tasks.
Activation functions are usually handcrafted. PReLU (He et al. (2015a)) tries to overcome this problem by introducing a learnable negative component to ReLU (Nair & Hinton (2010)). Maxout (Goodfellow et al. (2013)) and Mixout (Hui-zhen Zhao (2017)) are constructed with piecewise linear components, and theoretically, they are universal function approximators, though they increase the number of parameters in the network. Recently, meta-ACON (Ma et al. (2021)), a smooth activation, have been proposed, which is the generalization of the ReLU and Maxout activations and can smoothly approximate Swish. Meta-ACON has shown some good improvement on both small models and highly optimized large models. PAU (Molina et al. (2020)) and OPAU (Biswas et al. (2021a)) are two promising candidates for trainable activations, which have been introduced recently based on rational function approximation.
In this paper, we introduce a smooth approximation of known non-smooth activation functions like ReLU or Leaky ReLU based on the approximation of identity. Our experiments show that the proposed activations improve the performance of different network architectures as compared to ReLU on different deep learning problems.

heading: 2 MATHEMATICAL FORMALISM

heading: 2.1 CONVOLUTION
text: Convolution is a binary operation, which takes two functions f and g as input, and outputs a new function denoted by f ∗ g. Mathematically, we define this operation as follows
(f ∗ g)(x) = ∫ ∞ −∞ f(y)g(x− y) dy. (1)
The convolution operation has several properties. Below, we will list two of them which will be used larter in this article.
P1. (f ∗ g)(x) = (g ∗ f)(x), P2. If f is n-times differentiable with compact support over R and g is locally integrable over
R then f ∗ g is at least n-times differentiable over R.
Property P1 is an easy consequence of definition equation 1. Property P2 can be easily obtained by moving the derivative operator inside the integral. Note that this exchange of derivative and integral requires f to be of compact support. An immediate consequence of property P2 is that if one of the functions f or g is smooth with compact support, then f ∗ g is also smooth. This observation will be used later in the article to obtain smooth approximations of non-differentiable activation functions.

heading: 2.2 MOLLIFIER AND APPROXIMATE IDENTITIES
text: A smooth function φ over R is called a mollifier if it satisfies the following three properties:
1. It is compactly supported. 2. ∫ R φ(x) dx = 1.
3. lim →0 φ (x) := lim →0
1
φ(x/ ) = δ(x), where δ(x) is the Dirac delta function.
We say that a mollifier φ is an approximate identity if for any locally integrable function f over R, we have
lim →0
(f ∗ φ )(x) = f(x) pointwise for all x.

heading: 2.3 SMOOTH APPROXIMATIONS OF NON-DIFFERENTIABLE FUNCTIONS
text: Let φ be an approximate identity. Choosing = 1/n for n ∈ N, one can define φn(x) := nφ(nx). (2)
Using the property of approximate identity, for any locally integrable function f over R, we have lim n→∞ (f ∗ φn)(x) = f(x) pointwise for all x.
That is, for large enough n, f ∗ φn is a good approximation of f . Moreover, since φ is smooth, φn is smooth for each n ∈ N and therefore, using property P2, f ∗ φn is a smooth approximation of f for large enough n.
Let σ : R → R be any activation function. Then, by definition, σ is a continuous and hence, a locally integrable function. For a given approximate identity φ and n ∈ N, we define a smooth approximation of σ as σ ∗ φn, where φn is defined in equation 2.

heading: 3 SMOOTH APPROXIMATION UNIT (SAU)
text: Consider the Gaussian function
φ(x) = 1√ 2π e− x2 2
which is a well known approximate identity. Consider the Leaky Rectified Linear Unit (Leaky ReLU) activation function
LeakyReLU[α](x) = { x x ≥ 0 αx x < 0
Note that LeakyReLU[α] activation function is hyperparametrized by α and it is non-differentiable at the origin for all values of α except α = 1. For α = 0, LeakyReLU[α] reduces to well known activation function ReLU (Nair & Hinton (2010)) while for constant and trainable α, LeakyReLU[α] reduces to Leaky ReLU (Maas et al. (2013)) and Parametric ReLU (He et al. (2015a)) respectively. For a given n ∈ N, and α 6= 1, a smooth approximation of LeakyReLU[α] is given by
G(x, α, n) = (LeakyReLU[α] ∗ φn)(x) = 1
2n
√ 2
π e
−n2x2 2 + (1 + α)
2 x+ (1− α) 2 x erf ( nx√ 2 ) (3)
where erf is the Gaussian error function
erf(x) = 2√ π ∫ x 0 e−t 2 dt.
It is noticeable that this function is not zero centered but passes by extremely close neighbourhood of zero. To make the function zero centered, we have multiplied the first term of (3) by a linear component x. To further investigate these two functions as a possible candidates for activation function, we have conducted several experiments on MNIST (LeCun et al. (2010)), CIFAR10 (Krizhevsky (2009)), and CIFAR100 (Krizhevsky (2009)) datasets with PreActResNet-18 (He et al. (2016)), VGG-16 (with batch-normalization) (Ioffe & Szegedy (2015)) (Simonyan & Zisserman (2015)), and DenseNet-121 (Huang et al. (2016)) models, and we notice that both of them performs almost similar in every cases. So, for rest of the paper, we will only consider the approximate identity of Leaky ReLU (α = 0.25) given in (3) as the activation function. We call this function Smooth Approximation Unit (SAU). Approximation of Leaky ReLU (α = 0.25) by SAU is given in figure 1. It is clear from the figure 1 that SAU can approximate Leaky ReLU (as well as ReLU or its variants) quite well.
We note that in GELU (Hendrycks & Gimpel (2020)) paper, a similar idea is utilized to obtain their activation functions. They use the product of xwith the cumulative distribution function of a suitable probability distribution (see (Hendrycks & Gimpel (2020)) for further details).

heading: 3.1 LEARNING ACTIVATION PARAMETERS VIA BACK-PROPAGATION
text: Back-propagation algorithm (LeCun et al. (1989)) and gradient descent is used in neural networks to update Weights and biases. Parameters in trainable activation functions are updated using the same
technique. The forward pass is implemented in both Pytorch (Paszke et al. (2019)) & TensorflowKeras (Chollet et al. (2015)) API, and automatic differentiation will update the parameters. Alternatively, CUDA (Nickolls et al. (2008)) based implementation (see (Maas et al. (2013))) can be used and the gradients of equation (3) for the input x and the parameter α can be computed as follows:
∂G ∂x = −nx 2
√ 2
π e
−n2x2 2 + (1 + α)
2 + (1− α) 2 erf ( nx√ 2 ) + n(1− α)√ 2π x e− n2x2 2 (4)
∂G ∂α = x 2
( 1− erf ( nx√ 2 )) . (5)
where
d
dx erf(x) = 2√ π e−x 2
α and n can be either hyperparameters or trainable parameters. Now, note that the class of neural networks with SAU activation function is dense in C(K), where K is a compact subset of Rn and C(K) is the space of all continuous functions over K. The proof follows from the following proposition (see (Molina et al. (2020))).
Proposition 1. (Theorem 1.1 in Kidger and Lyons, 2020 (Kidger & Lyons (2020))) :- Let ρ : R → R be any continuous function. Let Nρn represent the class of neural networks with activation function ρ, with n neurons in the input layer, one neuron in the output layer, and one hidden layer with an arbitrary number of neurons. Let K ⊆ Rn be compact. Then Nρn is dense in C(K) if and only if ρ is non-polynomial.

heading: 4 EXPERIMENTS
text: To explore and compare the performance of SAU, we consider eight popular standard activation functions on different standard datasets and popular network architectures on standard deep learning problems like image classification, object detection, semantic segmentation, and machine translation. We consider the following activations to compare with SAU: ReLU, Leaky ReLU, Parametric ReLU (PReLU), ELU, ReLU6, Softplus, PAU, Swish, and GELU. It is evident from the experimental results in next sections that SAU outperform in most cases compared to the standard activations. For the rest of our experiments, we have considered α as a trainable parameter and n as a hyperparameter. We have initialized α at 0.25 and updated it via backpropagation according to (5). The value of n is considered 20000. All the experiments are conducted on an NVIDIA V100 GPU with 32GB RAM.

heading: 4.1 IMAGE CLASSIFICATION

heading: 4.1.1 MNIST, FASHION MNIST AND THE STREET VIEW HOUSE NUMBERS (SVHN) DATABASE
text: In this section, we present results on MNIST (LeCun et al. (2010)), Fashion MNIST (Xiao et al. (2017)), and SVHN (Netzer et al. (2011)) datasets. The MNIST and Fashion MNIST databases have a total of 60k training and 10k testing 28 × 28 grey-scale images with ten different classes. SVHN consists of 32 × 32 RGB images with a total of 73257 training images and 26032 testing images with ten different classes. We have applied standard data augmentation methods like rotation, zoom, height shift, shearing on the three datasets. We report results with AlexNet (Krizhevsky et al. (2012)) architecture in Table 1. For all the experiments to train a model on these three datasets, we use a batch size of 128, stochastic gradient descent (Robbins & Monro (1951), Kiefer & Wolfowitz (1952)) optimizer with 0.9 momentum & 5e−4 weight decay, and trained all networks up-to 100 epochs. We begin with 0.01 learning rate and decay the learning rate with cosine annealing (Loshchilov & Hutter (2017)) learning rate scheduler. We report more experiments on these datasets with LeNet (Lecun et al. (1998)), VGG-16 (Simonyan & Zisserman (2015)) (with batchnormalization (Ioffe & Szegedy (2015))) and a custom-designed model in the appendix section.

heading: 4.1.2 CIFAR
text: The CIFAR (Krizhevsky (2009)) is one of the most popular databases for image classification consists of a total of 60k 32× 32 RGB images and is divided into 50k training and 10k test images. CIFAR has two different datasets- CIFAR10 and CIFAR100 with a total of 10 and 100 classes, respectively. We report the top-1 accuracy on Table 2 and Table 3 on CIFAR10 and CIFAR100 datasets respectively. We consider MobileNet V1 (Howard et al. (2017)), MobileNet V2 (Sandler et al. (2019)), Shufflenet V2 (Ma et al. (2018)), PreActResNet (He et al. (2016)), ResNet (He et al. (2015b)), Inception V3 (Szegedy et al. (2015)), DenseNet (Huang et al. (2016)), WideResNet (Zagoruyko & Komodakis (2016)), VGG (Simonyan & Zisserman (2015)) (with batch-normalization (Ioffe & Szegedy (2015))), and EfficientNet B0 (Tan & Le (2020)). For all the experiments to train a model on these two datasets, we use a batch size of 128, stochastic gradient descent (Robbins & Monro (1951), Kiefer & Wolfowitz (1952)) optimizer with 0.9 momentum & 5e−4 weight decay, and trained all networks up-to 200 epochs. We begin with 0.01 learning rate and decay the learning rate with cosine annealing (Loshchilov & Hutter (2017)) learning rate scheduler. Standard data augmentation methods like width shift, height shift, horizontal flip, and rotation is applied on both datasets. It is noticeable from these two tables that replacing ReLU by SAU, there is an increment in top-1 accuracy from 1% to more than 5% in most of the models. A more detailed result on these two datasets with the other baseline activations are reported in the appendix section. Training and test accuracy & loss curves for baseline activation functions and SAU are given in Figures 2 and 3 respectively on CIFAR100 dataset on ShuffleNet V2 (2.0x) network. From these learning curves, it is evident that after training few epochs, SAU has stable & smooth learning, faster convergence speed, and higher accuracy and lower loss on the test dataset compared to other baseline activation functions.

heading: 4.1.3 TINY IMAGENET
text: In this section, we present results on the Tiny ImageNet dataset, a similar kind of image classification database like the ImageNet Large Scale Visual Recognition Challenge(ILSVRC). Tiny Imagenet contains 64 × 64 RGB images with total 100,000 training images, 10,000 validation images, and 10,000 test images and have total 200 image classes. We report the mean of 6 different runs for Top-1 accuracy in table 4 on WideResNet 28-10 (WRN 28-10) (Zagoruyko & Komodakis (2016)) model. We consider a batch size of 64, 0.2 dropout rate (Srivastava et al. (2014)), SGD optimizer (Robbins & Monro (1951), Kiefer & Wolfowitz (1952)), He Normal initializer (He et al. (2015a)), initial learning rate(lr rate) 0.1, and lr rate is reduced by a factor of 10 after every 50 epochs up-to 300 epochs. Standard data augmentation techniques like rotation, width shift, height shift, shearing, zoom, horizontal flip, fill mode is applied to improve performance. It is evident from the table that the proposed function performs better than the baseline functions, and top-1 accuracy is stable (mean±std) and got a good improvement for SAU over ReLU.

heading: 4.2 OBJECT DETECTION
text: A standard problem in computer vision is object detection, in which the network model try to locate and identify each object present in the image. Object detection is widely used in face detection, autonomous vehicle etc. In this section, we present our results on challenging Pascal VOC dataset (Everingham et al. (2010)) on Single Shot MultiBox Detector(SSD) 300 (Liu et al. (2016)) with VGG-16(with batch-normalization) (Simonyan & Zisserman (2015)) as the backbone network. No pre-trained weight is considered for our experiments in the network. The network has been trained with a batch size of 8, SGD optimizer (Robbins & Monro (1951); Kiefer & Wolfowitz (1952)) with 0.9 momentum, 5e−4 weight decay, 0.001 learning rate, and trained up to 120000 iterations. We report the mean average precision (mAP) in Table 5 for the mean of 6 different runs.

heading: 4.3 SEMANTIC SEGMENTATION
text: Semantic segmentation is a computer vision problem that narrates the procedure of associating each pixel of an image with a class label. We present our experimental results in this section on the popular Cityscapes dataset (Cordts et al. (2016)). The U-net model (Ronneberger et al. (2015)) is considered as the segmentation framework and is trained up-to 250 epochs, with adam optimizer (Kingma & Ba (2015)), learning rate 5e−3, batch size 32 and Xavier Uniform initializer (Glorot & Bengio (2010)). We report the mean of 6 different runs for Pixel Accuracy and the mean IntersectionOver-Union (mIOU) on test data on table 6.

heading: 4.4 MACHINE TRANSLATION
text: Machine Translation is a deep learning technique in which a model translate text or speech from one language to another language. In this section, we report results on WMT 2014 English→German dataset. The database contains 4.5 million training sentences. Network performance is evaluated on the newstest2014 dataset using the BLEU score metric. An Attention-based 8-head transformer network (Vaswani et al. (2017)) in trained with Adam optimizer (Kingma & Ba (2015)), 0.1 dropout rate (Srivastava et al. (2014)), and trained up to 100000 steps. Other hyperparameters are kept similar as mentioned in the original paper (Vaswani et al. (2017)). We report the mean of 6 different runs on Table 7 on the test dataset(newstest2014).

heading: 5 BASELINE TABLE
text: In this section, we present a table for SAU and the other baseline functions, which shows that SAU beat or perform equally well compared to baseline activation functions in most cases. We report a detailed comparison with SAU and the baseline activation functions based on all the experiments in earlier sections in Table 8. We notice that SAU performs remarkably well in most of the cases when compared with the baseline activations.

heading: 6 CONCLUSION
text: In this paper, we propose a new novel smooth activation function using approximate identity, and we call it a smooth activation unit (SAU). The proposed function can approximate ReLU or its different variants (like Leaky ReLU etc.) quite well. For our experiments, we consider SAU as a trainable activation function, and we show that on a wide range of experiments on different deep learning problems, the proposed functions outperform the known activations like ReLU, Leaky ReLU or Swish in most cases which shows that replacing the hand-crafted activation functions by SAU can be beneficial in deep networks.

heading: A APPENDIX
text: A.1 BASELINE ACTIVATION FUNCTIONS
1. ReLU: ReLU (Nair & Hinton (2010)) is a widely used activation function and defined as f(x) = max(x, 0).
2. ELU: ELU (Clevert et al. (2016)) is defined as f(x) = max(x, 0) +min(α(ex − 1), 0).
3. Softplus: Softplus (Zheng et al. (2015)) is a smoothing approximation of ReLU and defined as f(x) = ln(1 + ex).
4. Leaky ReLU: Leaky ReLU (Maas et al. (2013)) is a variant of ReLU and is defined as f(x) = max(x, 0.01x)
5. ReLU6: ReLU6 (Krizhevsky (2010)) is a bounded variant of ReLU defined as f(x) = min(max(x, 0), 6).
6. Parametric ReLU: Parametric ReLU (PReLU) (He et al. (2015a)) is a trainable activation function and is defined as f(x) = max(x, αx) where α is the learnable parameter.
7. PAU: PAU (Molina et al. (2020)) is a smooth approximation of Leaky ReLU function by rational approximation of polynomials up-to a given order. PAU is a trainable activation function.
8. Swish: Swish (Ramachandran et al. (2017)) is a popular activation function and defined as f(x) = x
1 + e−βx , where β is either a hyperparameter or trainable parameter.
9. GELU: GELU (Hendrycks & Gimpel (2020)) can be viewed as a smooth approximation of ReLU and GELU is defined as f(x) = x2 (1 + erf( x√ 2 )).
A.2 MNIST, FASHION MNIST, AND SVHN
We report a more detailed experiment on MNIST, Fashion MNIST, and SVHN datasets on VGG-16, LeNet, and a custom-designed model in Table 9, 10, and 11 respectively. We design the custom network with CNN layers with 3 × 3 kernels and max-pooling layers with 2 × 2 kernels. We consider Channel depths of size 128 (twice), 64 (thrice), 32 (twice), with a dense layer of size 128, Max-pooling layer(thrice), and dropout (Srivastava et al. (2014)). We have applied batchnormalization (Ioffe & Szegedy (2015)) before the activation function layer, and more details about this architecture are given in figure 4. For the experimental setup on these three networks, we follow the same experimental setup we use for AlexNet architecture on these three datasets. The results reported in Table 9, 10, and 11 is also counted in the baseline table. Best performing activation(s) is(are) reported in bold text.
A.3 EXPERIMENTAL RESULTS WITH BASELINES ON CIFAR DATASETS
We report a more detailed experiments on CIFAR10 and CIFAR 100 datasets with Leaky ReLU, ReLU6, PReLU, ELU, Softplus, Swish, and GELU activations. Experimental results on CIFAR10 are reported in Table 12, 13 and experimental results on CIFAR100 are reported in Table 14 and
15. The results are reported on MobileNet V1 (MN V1) (Howard et al. (2017)), MobileNet V2 (MN V2) (Sandler et al. (2019)),) Shufflenet V2 (SF V2) (Ma et al. (2018)), PreActResNet (PARN) (He et al. (2016)), ResNet (RN) (He et al. (2015b)), Inception V3 (IN V3) (Szegedy et al. (2015)), DenseNet (DN) (Huang et al. (2016)), WideResNet (WRN) (Zagoruyko & Komodakis (2016)), VGG (Simonyan & Zisserman (2015)) (with batch-normalization (Ioffe & Szegedy (2015))), and EfficientNet B0 (EN B0) (Tan & Le (2020)) networks architectures. We follow the same hyperparameter setup as described in section 4.1.2. To count the numbers in the baseline table for all baseline activations, we also consider the results reported in Table 12, 13, 14, 15 along with results reported in Table 2 and 3. Comparing these six tables it is clear that SAU beats all the baseline activations in most of the cases on CIFAR10 and CIFAR100 datasets.

Review summaries:
In this paper, the authors introduce an activation function (SAU) based on smoothing of the leaky relu. More precisely, the authors use convolutions and approximate identities to achieve that goal.  Furthermore, they present the smooth version as a learnable activation function, whose parameters can be optimized via back-propagation and hyper-parameter searches.  The authors present the smoothing from the ground up and conclude their theoretical development with the gradients needed for back-propagation as well as a proof that guarantees the activation function allows the network to be an universal approximator. The empirical section describes the behavior of SAU on different datasets and architectures.

The authors propose a new activation function, named SAU, which is a smoothed approximation to Leaky/PReLU. The activation is derived by convolving a given activation function with a smooth approximation to the delta function. The paper only investigates using LeakyReLU as the target activation and the standard concentrated Gaussian approximation to the delta function. The degree of approximation is controlled via a hyperparameter denoted by $n$, controlling the variance of the Gaussian. The new activation is tested in various settings and with various architectures and is shown to outperform other alternatives to the common ReLU activation function.

This paper proposes a new activation function for deep neural networks. Through the lens of *approximate identities*, the authors come up with the new activation function which is a ``smoothed'' version of ReLU/Leaky ReLU, i.e., the Smooth Activation Unit (SAU). The authors conduct experiments on both vision (including classification, object detection, and semantic segmentation) and natural language processing tasks (machine translation) and compare the performance of the proposed SAU with previous widely used activation functions.