SAU: Smooth activation function using convolution with approximate identities

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin . Over the years , many smooth approximations of ReLU have been proposed using various smoothing techniques . We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities . In particular , we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models . We call this function Smooth Activation Unit ( SAU ) . Replacing ReLU by SAU , we get 5.12 % improvement with ShuffleNet V2 ( 2.0x ) model on the CIFAR100 dataset . 1 INTRODUCTION . Deep networks form a crucial component of modern deep learning . Non-linearity is introduced in such networks by the use of activation functions , and the choice has a substantial impact on network performance and training dynamics . Designing a new novel activation function is a difficult task . Handcrafted activations like Rectified Linear Unit ( ReLU ) ( Nair & Hinton ( 2010 ) ) , Leaky ReLU ( Maas et al . ( 2013 ) ) or its variants are very common choices for activation functions and exhibits promising performance on different deep learning tasks . There are many activations that have been proposed so far and some of them are ELU ( Clevert et al . ( 2016 ) ) , Parametric ReLU ( PReLU ) ( He et al . ( 2015a ) ) , Swish ( Ramachandran et al . ( 2017 ) ) , Tanhsoft ( Biswas et al . ( 2021d ) ) , EIS ( Biswas et al . ( 2021b ) ) , Padé Activation Unit ( PAU ) ( Molina et al . ( 2020 ) ) , Orthogonal Padé Activation Unit ( OPAU ) ( Biswas et al . ( 2021a ) ) , ACON ( Ma et al . ( 2021 ) ) , ErfAct ( Biswas et al . ( 2021c ) ) , Mish ( Misra ( 2020 ) ) , GELU ( Hendrycks & Gimpel ( 2020 ) ) , ReLU6 ( Krizhevsky ( 2010 ) ) , Softplus ( Zheng et al . ( 2015 ) ) etc . Nevertheless , ReLU remains the favourite choice among the deep learning community due to its simplicity and better performance when compared to Tanh or Sigmoid , though it has a drawback known as dying ReLU in which the network starts to lose the gradient direction due to the negative inputs and produces zero outcome . In 2017 , Swish ( Ramachandran et al . ( 2017 ) ) was proposed by the Google brain team . Swish was found by automatic search technique , and it has shown some promising performance across different deep learning tasks . Activation functions are usually handcrafted . PReLU ( He et al . ( 2015a ) ) tries to overcome this problem by introducing a learnable negative component to ReLU ( Nair & Hinton ( 2010 ) ) . Maxout ( Goodfellow et al . ( 2013 ) ) and Mixout ( Hui-zhen Zhao ( 2017 ) ) are constructed with piecewise linear components , and theoretically , they are universal function approximators , though they increase the number of parameters in the network . Recently , meta-ACON ( Ma et al . ( 2021 ) ) , a smooth activation , have been proposed , which is the generalization of the ReLU and Maxout activations and can smoothly approximate Swish . Meta-ACON has shown some good improvement on both small models and highly optimized large models . PAU ( Molina et al . ( 2020 ) ) and OPAU ( Biswas et al . ( 2021a ) ) are two promising candidates for trainable activations , which have been introduced recently based on rational function approximation . In this paper , we introduce a smooth approximation of known non-smooth activation functions like ReLU or Leaky ReLU based on the approximation of identity . Our experiments show that the proposed activations improve the performance of different network architectures as compared to ReLU on different deep learning problems . 2 MATHEMATICAL FORMALISM . 2.1 CONVOLUTION . Convolution is a binary operation , which takes two functions f and g as input , and outputs a new function denoted by f ∗ g. Mathematically , we define this operation as follows ( f ∗ g ) ( x ) = ∫ ∞ −∞ f ( y ) g ( x− y ) dy . ( 1 ) The convolution operation has several properties . Below , we will list two of them which will be used larter in this article . P1 . ( f ∗ g ) ( x ) = ( g ∗ f ) ( x ) , P2 . If f is n-times differentiable with compact support over R and g is locally integrable over R then f ∗ g is at least n-times differentiable over R. Property P1 is an easy consequence of definition equation 1 . Property P2 can be easily obtained by moving the derivative operator inside the integral . Note that this exchange of derivative and integral requires f to be of compact support . An immediate consequence of property P2 is that if one of the functions f or g is smooth with compact support , then f ∗ g is also smooth . This observation will be used later in the article to obtain smooth approximations of non-differentiable activation functions . 2.2 MOLLIFIER AND APPROXIMATE IDENTITIES . A smooth function φ over R is called a mollifier if it satisfies the following three properties : 1 . It is compactly supported . 2 . ∫ R φ ( x ) dx = 1 . 3. lim →0 φ ( x ) : = lim →0 1 φ ( x/ ) = δ ( x ) , where δ ( x ) is the Dirac delta function . We say that a mollifier φ is an approximate identity if for any locally integrable function f over R , we have lim →0 ( f ∗ φ ) ( x ) = f ( x ) pointwise for all x . 2.3 SMOOTH APPROXIMATIONS OF NON-DIFFERENTIABLE FUNCTIONS . Let φ be an approximate identity . Choosing = 1/n for n ∈ N , one can define φn ( x ) : = nφ ( nx ) . ( 2 ) Using the property of approximate identity , for any locally integrable function f over R , we have lim n→∞ ( f ∗ φn ) ( x ) = f ( x ) pointwise for all x . That is , for large enough n , f ∗ φn is a good approximation of f . Moreover , since φ is smooth , φn is smooth for each n ∈ N and therefore , using property P2 , f ∗ φn is a smooth approximation of f for large enough n. Let σ : R → R be any activation function . Then , by definition , σ is a continuous and hence , a locally integrable function . For a given approximate identity φ and n ∈ N , we define a smooth approximation of σ as σ ∗ φn , where φn is defined in equation 2 . 3 SMOOTH APPROXIMATION UNIT ( SAU ) . Consider the Gaussian function φ ( x ) = 1√ 2π e− x2 2 which is a well known approximate identity . Consider the Leaky Rectified Linear Unit ( Leaky ReLU ) activation function LeakyReLU [ α ] ( x ) = { x x ≥ 0 αx x < 0 Note that LeakyReLU [ α ] activation function is hyperparametrized by α and it is non-differentiable at the origin for all values of α except α = 1 . For α = 0 , LeakyReLU [ α ] reduces to well known activation function ReLU ( Nair & Hinton ( 2010 ) ) while for constant and trainable α , LeakyReLU [ α ] reduces to Leaky ReLU ( Maas et al . ( 2013 ) ) and Parametric ReLU ( He et al . ( 2015a ) ) respectively . For a given n ∈ N , and α 6= 1 , a smooth approximation of LeakyReLU [ α ] is given by G ( x , α , n ) = ( LeakyReLU [ α ] ∗ φn ) ( x ) = 1 2n √ 2 π e −n2x2 2 + ( 1 + α ) 2 x+ ( 1− α ) 2 x erf ( nx√ 2 ) ( 3 ) where erf is the Gaussian error function erf ( x ) = 2√ π ∫ x 0 e−t 2 dt . It is noticeable that this function is not zero centered but passes by extremely close neighbourhood of zero . To make the function zero centered , we have multiplied the first term of ( 3 ) by a linear component x . To further investigate these two functions as a possible candidates for activation function , we have conducted several experiments on MNIST ( LeCun et al . ( 2010 ) ) , CIFAR10 ( Krizhevsky ( 2009 ) ) , and CIFAR100 ( Krizhevsky ( 2009 ) ) datasets with PreActResNet-18 ( He et al . ( 2016 ) ) , VGG-16 ( with batch-normalization ) ( Ioffe & Szegedy ( 2015 ) ) ( Simonyan & Zisserman ( 2015 ) ) , and DenseNet-121 ( Huang et al . ( 2016 ) ) models , and we notice that both of them performs almost similar in every cases . So , for rest of the paper , we will only consider the approximate identity of Leaky ReLU ( α = 0.25 ) given in ( 3 ) as the activation function . We call this function Smooth Approximation Unit ( SAU ) . Approximation of Leaky ReLU ( α = 0.25 ) by SAU is given in figure 1 . It is clear from the figure 1 that SAU can approximate Leaky ReLU ( as well as ReLU or its variants ) quite well . We note that in GELU ( Hendrycks & Gimpel ( 2020 ) ) paper , a similar idea is utilized to obtain their activation functions . They use the product of xwith the cumulative distribution function of a suitable probability distribution ( see ( Hendrycks & Gimpel ( 2020 ) ) for further details ) . 3.1 LEARNING ACTIVATION PARAMETERS VIA BACK-PROPAGATION . Back-propagation algorithm ( LeCun et al . ( 1989 ) ) and gradient descent is used in neural networks to update Weights and biases . Parameters in trainable activation functions are updated using the same technique . The forward pass is implemented in both Pytorch ( Paszke et al . ( 2019 ) ) & TensorflowKeras ( Chollet et al . ( 2015 ) ) API , and automatic differentiation will update the parameters . Alternatively , CUDA ( Nickolls et al . ( 2008 ) ) based implementation ( see ( Maas et al . ( 2013 ) ) ) can be used and the gradients of equation ( 3 ) for the input x and the parameter α can be computed as follows : ∂G ∂x = −nx 2 √ 2 π e −n2x2 2 + ( 1 + α ) 2 + ( 1− α ) 2 erf ( nx√ 2 ) + n ( 1− α ) √ 2π x e− n2x2 2 ( 4 ) ∂G ∂α = x 2 ( 1− erf ( nx√ 2 ) ) . ( 5 ) where d dx erf ( x ) = 2√ π e−x 2 α and n can be either hyperparameters or trainable parameters . Now , note that the class of neural networks with SAU activation function is dense in C ( K ) , where K is a compact subset of Rn and C ( K ) is the space of all continuous functions over K. The proof follows from the following proposition ( see ( Molina et al . ( 2020 ) ) ) . Proposition 1 . ( Theorem 1.1 in Kidger and Lyons , 2020 ( Kidger & Lyons ( 2020 ) ) ) : - Let ρ : R → R be any continuous function . Let Nρn represent the class of neural networks with activation function ρ , with n neurons in the input layer , one neuron in the output layer , and one hidden layer with an arbitrary number of neurons . Let K ⊆ Rn be compact . Then Nρn is dense in C ( K ) if and only if ρ is non-polynomial .

Summary: The authors propose a new activation function, named SAU, which is a smoothed approximation to Leaky/PReLU. The activation is derived by convolving a given activation function with a smooth approximation to the delta function. The paper only investigates using LeakyReLU as the target activation and the standard concentrated Gaussian approximation to the delta function. The degree of approximation is controlled via a hyperparameter denoted by $n$, controlling the variance of the Gaussian. The new activation is tested in various settings and with various architectures and is shown to outperform other alternatives to the common ReLU activation function.