Symmetric-APL Activations: Training Insights and Robustness to Adversarial Attacks

1 INTRODUCTION . Deep neural networks ( DNNs ) are a series of linear transformations followed by point-wise nonlinear functions . These non-linear functions are also referred to as activation functions . Activation functions directly influence the training dynamics and , therefore , affect the final performance of the DNN . For instance , activations such as Sigmoid and Tanh suffer from what is known as the “ vanishing gradient ” problem , when the inputs are in the saturation regions of these functions . This can adversely affect network performance and cause slow convergence . Rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) have been shown to perform better than Sigmoid and Tanh in many scenarios . Instead of using a fixed activation function , one can learn a parameterized activation function to add expressive power to the model . Towards this goal , significant efforts have been made to introduce different learnable activation functions ( Agostinelli et al. , 2014 ; He et al. , 2015 ; Ramachandran et al. , 2017 ; Jin et al. , 2016 ; Li et al. , 2016 ) . Although most of these adaptive activation functions outperform fixed activation functions in multiple tasks , there is very little insight into why . In this paper , we propose the Symmetric Adaptive Piecewise Linear unit ( Symmetric-APL ; pronounced as Symmetric apple : `` æp . @ l ) as a new adaptive activation function which builds upon the adaptive piecewise linear unit ( APL ) ( Agostinelli et al. , 2014 ) . Similar to APL units , S-APL has multiple hinges which are connected by linear pieces . The S-APL activation is designed to give the same expressive power to both the positive and negative sides of the x-axis . We show that SAPL can approximate any continuous and zero-centered function , which encompasses many types of learned and fixed activations functions . Using this flexible learned activation function , we can then analyze how the shape changes during training and use ablation studies to investigate the role that these changes play in optimization . This helps us better understand learned activation functions and explain why they often provide increases in performance . Finally , we evaluate the robustness of S-APL activated networks to adversarial attacks . Experiments are done using both black-box and open-box adversarial attacks where we observe S-APL equipped networks are more robust to attacks than their ReLU counterparts . Experiments show that the robustness improves against differential evolution attacks ( Su et al. , 2019b ) by ( up to ) 51 % on Lenet , 47 % on ResNet-18 ( He et al. , 2016 ) , 43 % on pure CNN , and 38 % on Net in Net ( Lin et al. , 2013 ) . For open-box adversarial attacks , the robustness also improves against FGSM and CW-L2 ( Goodfellow et al. , 2014 ) attack by ( up to ) 36 % , 28 % , and 25 % for Lenet , pure CNN , and ResNet-18 architectures , respectively using CIFAR-10 dataset . 2 RELATED WORKS . ReLU and its variants such as Leaky-ReLU ( Maas et al. , 2013 ) , P-ReLU ( He et al. , 2015 ) , exponential linear units ( ELU ) ( Clevert et al. , 2015 ) , and scaled exponential linear units ( SELU ) ( Klambauer et al. , 2017 ) are believed to solve the vanishing gradient problem and some are shown to better normalize the network . By changing the negative side of x-axis , these variants push the mean activations of each unit closer to zero and accelerate learning . Some of the pioneering attempts to learn activations in a neural network can be found in Poli ( 1996 ) , Weingaertner et al . ( 2002 ) , and Khan et al . ( 2013 ) where the authors proposed novel approaches to learning the best activation function per neuron among a pool of candidate activations by employing genetic and evolutionary algorithms . Maxout ( Goodfellow et al. , 2013 ) has been introduced as an activation function aimed at enhancing the model averaging properties of dropout ( Srivastava et al. , 2014 ) . However , not only is it limited to approximating convex functions , but it also requires a significant increase in parameters . Neural architecture search ( Ramachandran et al. , 2017 ) uses a combination of exhaustive and reinforcement learning based search . Although the authors discovered multiple novel activation functions , of which f ( x ) = x ∗ sigmoid ( βx ) performed slightly better than ReLU , the search space of activation functions is limited . S-ReLU ( Jin et al. , 2016 ) was proposed as an adaptive activation function which can mimic both convex and non-convex functions . It is worth mentioning that , in Lin et al . ( 2013 ) , the authors proposed the network in network approach where they replace activation functions in convolutional layers with small multi-layer perceptrons . Theoretically , due to Universal Approximation Theorem , this is the most expressive activation ; however , it requires many more parameters . 3 ADAPTIVE PIECEWISE ACTIVATION FUNCTION . 3.1 ADAPTIVE PIECEWISE LINEAR UNITS . Agostinelli et al . ( 2014 ) proposed an adaptive activation function which consists of multiple linear pieces . The formulation of this adaptive piecewise linear ( APL ) activation function is shown in Equation 1. hi ( x ) = max ( 0 , x ) + S∑ s=1 asimax ( 0 , −x+ bsi ) ( 1 ) Where hi is the non-linear transformation for the hidden unit i and S is a hyperparameter that determines the number of hinges . ai is the slope of the linear piece and bi is the location of the hinge . However , APL units are not zero-centered as they can have a non-zero output for a input of zero . This behavior provides no apparent beneficial purpose . Furthermore , APL units can only represent piecewise linear functions whose output hi ( x ) = x for x > u , for some scalar u . This significantly restricts the class of piecewise linear functions that APLs can express . 3.2 SYMMETRIC ADAPTIVE PIECEWISE LINEAR UNITS ( APL : `` æp . @ l ) To overcome the shortcomings of APL activation function , we define a new adaptive piecewise linear activation function that is zero-centered and gives the same expressive power to the positive and negative half of the input space . Since S-APL shares the variables as+ , b s + , a s − , and b s − among all the neurons of a layer ( e.i. , hi ( x , S ) does not depend on i ) , it requires fewer parameters in compare to the plain APL . We formulate the activation hi of the hidden unit i as the summation of symmetric hinge-shaped pieces . hi ( x , S ) = S∑ s=1 as+max ( 0 , x− bs+ ) + S∑ s=1 as−max ( 0 , −x− bs− ) ( 2 ) Similar to APL units , S is a hyperparameter which determines the number of hinges ( on each side of the x-axis ) . as+ and a s − control the slope of linear pieces in both the positive and negative sides of the x-axis , while bs+ , b s − ∈ [ 0 , +∞ ) determine the location of hinge s. The total number of additional parameters is 4S for a single layer . Because our networks are trained with batch normalization ( Ioffe & Szegedy , 2015 ) before each S-APL layer , the distribution of input to the S-APL function will follow that of a normal distribution . Therefore , we fix the hinges to be a pre-determined number of standard deviations away from the mean , decreasing the number of parameters to 2S . Based on Equation 2 , S-APL activation is zero-centered which leads to more stable statistical behavior during training . In case of using fixed hinges ’ locations for APL , due to increasing the number of parameters , S-APL offers more flexibility than the normal APL . However , this increase is of O ( constant ) which is negligible . With the assumption of as+ = a s − and b s + = b s − , since hsi ( x ) = h s i ( −x ) , S-APL allows neurons to activate symmetrically . As it is mentioned in Zhao & Griffin ( 2016 ) , networks with symmetric activation functions are more robust to adversarial fooling . We will delve into this property in later sections . S-APL can better handle the limits . It could be inferred from the definition that , in contrast to normal APL , increasing the absolute value of the input xwill also increase the number of terms affecting the hi ( x ) . However , normal APL requires the rightmost linear section in all the component functions to have a unit slope and bias 0 , which is not an appropriate constraint and undermines its representation ability . The following theorem shows that S-APL can approximate any non-linear , zero-centered , and Lipschitz continuous function in a closed interval of real numbers . It is worth mentioning that APL was not flexible enough to do so . Theorem 3.1 For any function f : [ A , B ] → R and ∈ R , ∃S ∈ N , where |f ( x ) − S-APL ( x ) | ≤ , assuming : • A and B are finite real numbers . • f is M-Lipschitz continuous . Proof : We start by dividing the interval [ A , B ] into S equal sub-intervals [ Ai , Bi ] . In other words , we put S equally distanced hinges in the interval [ A , B ] . So the length of each sub-interval will be B−AS . We start approximating f from the left most sub-interval which is [ A , A + B−A S ] . The approximation of f in that sub-interval will be a line which connects f ( A ) to f ( A+ B−AS ) . Due to M-Lipschitz continuity of f , for any x ∈ [ A , A+ B−AS ] : |f ( x ) − S-APL ( x ) | ≤ { |f ( A ) − f ( A+ B−AS ) + M ( B−A ) S | f ( A ) ≤ f ( A+ B−A S ) , |f ( A+ B−AS ) − f ( A ) + M ( B−A ) S | f ( A ) ≥ f ( A+ B−A S ) . ( 3 ) Again , due to M-Lipschitz continuity of f , we know that : |f ( A ) − f ( A+ B −A S ) | ≤ M ( B −A ) S ( 4 ) Using Equation 4 , we can re-write the inequality 3 as follows : |f ( x ) − S-APL ( x ) | ≤ 2M ( B −A ) S ( 5 ) So , by choosing S ≥ 2M ( B−A ) , we satisfy that |f ( x ) − S-APL ( x ) | ≤ . in order to complete the proof , we need to show that this approximation can be applied to other sub-intervals of [ A , B ] as well . We do so by induction as follows : Suppose that the approximation is done for sub-intervals [ A1 , B1 ] , ... , [ Ai , Bi ] , now we prove that for sub-interval [ Ai+1 , Bi+1 ] we can do the approximation which is connecting f ( Ai+1 ) to f ( Bi+1 ) . The slope of S-APL in the subinterval [ Ai , Bi ] is ∑i 1 a i + or ∑i 1 a i − ( depending on the sign of the sub-interval ) . However , the slope of S-APL in the sub-interval [ Ai+1 , Bi+1 ] will be either ∑i+1 1 a i + or ∑i+1 1 a i − . In both cases , the extra term ai+1+ or a i+1 − is not fixed and can change the slop to any arbitrary value . This fact plus the assumption of continuity of S-APL guarantee that f ( Ai+1 ) can be connected to f ( Bi+1 ) which was our proposed approximation . As we showed the approximation for the left most sub-interval , the induction is complete .

Summary: In this paper, a new activation function, i.e. S-APL is proposed for deep neural networks. It is an extension of the APL activation, but is symmetric w.r.t. x-axis. It also has more linear pieces (actually S pieces, where S can be arbitrarily large) than the existing activation functions like ReLU. Experimental results show that S-APL can be on par with or slightly better than the existing activation functions on MNIST/CIFAR-10/CIFAR-100 datasets with various networks. The authors also show that neural networks with the proposed activation can be more robust to adversarial attacks.