Symmetric-APL Activations: Training Insights and Robustness to Adversarial Attacks

1 INTRODUCTION . Deep neural networks ( DNNs ) are a series of linear transformations followed by point-wise nonlinear functions . These non-linear functions are also referred to as activation functions . Activation functions directly influence the training dynamics and , therefore , affect the final performance of the DNN . For instance , activations such as Sigmoid and Tanh suffer from what is known as the “ vanishing gradient ” problem , when the inputs are in the saturation regions of these functions . This can adversely affect network performance and cause slow convergence . Rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) have been shown to perform better than Sigmoid and Tanh in many scenarios . Instead of using a fixed activation function , one can learn a parameterized activation function to add expressive power to the model . Towards this goal , significant efforts have been made to introduce different learnable activation functions ( Agostinelli et al. , 2014 ; He et al. , 2015 ; Ramachandran et al. , 2017 ; Jin et al. , 2016 ; Li et al. , 2016 ) . Although most of these adaptive activation functions outperform fixed activation functions in multiple tasks , there is very little insight into why . In this paper , we propose the Symmetric Adaptive Piecewise Linear unit ( Symmetric-APL ; pronounced as Symmetric apple : `` æp . @ l ) as a new adaptive activation function which builds upon the adaptive piecewise linear unit ( APL ) ( Agostinelli et al. , 2014 ) . Similar to APL units , S-APL has multiple hinges which are connected by linear pieces . The S-APL activation is designed to give the same expressive power to both the positive and negative sides of the x-axis . We show that SAPL can approximate any continuous and zero-centered function , which encompasses many types of learned and fixed activations functions . Using this flexible learned activation function , we can then analyze how the shape changes during training and use ablation studies to investigate the role that these changes play in optimization . This helps us better understand learned activation functions and explain why they often provide increases in performance . Finally , we evaluate the robustness of S-APL activated networks to adversarial attacks . Experiments are done using both black-box and open-box adversarial attacks where we observe S-APL equipped networks are more robust to attacks than their ReLU counterparts . Experiments show that the robustness improves against differential evolution attacks ( Su et al. , 2019b ) by ( up to ) 51 % on Lenet , 47 % on ResNet-18 ( He et al. , 2016 ) , 43 % on pure CNN , and 38 % on Net in Net ( Lin et al. , 2013 ) . For open-box adversarial attacks , the robustness also improves against FGSM and CW-L2 ( Goodfellow et al. , 2014 ) attack by ( up to ) 36 % , 28 % , and 25 % for Lenet , pure CNN , and ResNet-18 architectures , respectively using CIFAR-10 dataset . 2 RELATED WORKS . ReLU and its variants such as Leaky-ReLU ( Maas et al. , 2013 ) , P-ReLU ( He et al. , 2015 ) , exponential linear units ( ELU ) ( Clevert et al. , 2015 ) , and scaled exponential linear units ( SELU ) ( Klambauer et al. , 2017 ) are believed to solve the vanishing gradient problem and some are shown to better normalize the network . By changing the negative side of x-axis , these variants push the mean activations of each unit closer to zero and accelerate learning . Some of the pioneering attempts to learn activations in a neural network can be found in Poli ( 1996 ) , Weingaertner et al . ( 2002 ) , and Khan et al . ( 2013 ) where the authors proposed novel approaches to learning the best activation function per neuron among a pool of candidate activations by employing genetic and evolutionary algorithms . Maxout ( Goodfellow et al. , 2013 ) has been introduced as an activation function aimed at enhancing the model averaging properties of dropout ( Srivastava et al. , 2014 ) . However , not only is it limited to approximating convex functions , but it also requires a significant increase in parameters . Neural architecture search ( Ramachandran et al. , 2017 ) uses a combination of exhaustive and reinforcement learning based search . Although the authors discovered multiple novel activation functions , of which f ( x ) = x ∗ sigmoid ( βx ) performed slightly better than ReLU , the search space of activation functions is limited . S-ReLU ( Jin et al. , 2016 ) was proposed as an adaptive activation function which can mimic both convex and non-convex functions . It is worth mentioning that , in Lin et al . ( 2013 ) , the authors proposed the network in network approach where they replace activation functions in convolutional layers with small multi-layer perceptrons . Theoretically , due to Universal Approximation Theorem , this is the most expressive activation ; however , it requires many more parameters . 3 ADAPTIVE PIECEWISE ACTIVATION FUNCTION . 3.1 ADAPTIVE PIECEWISE LINEAR UNITS . Agostinelli et al . ( 2014 ) proposed an adaptive activation function which consists of multiple linear pieces . The formulation of this adaptive piecewise linear ( APL ) activation function is shown in Equation 1. hi ( x ) = max ( 0 , x ) + S∑ s=1 asimax ( 0 , −x+ bsi ) ( 1 ) Where hi is the non-linear transformation for the hidden unit i and S is a hyperparameter that determines the number of hinges . ai is the slope of the linear piece and bi is the location of the hinge . However , APL units are not zero-centered as they can have a non-zero output for a input of zero . This behavior provides no apparent beneficial purpose . Furthermore , APL units can only represent piecewise linear functions whose output hi ( x ) = x for x > u , for some scalar u . This significantly restricts the class of piecewise linear functions that APLs can express . 3.2 SYMMETRIC ADAPTIVE PIECEWISE LINEAR UNITS ( APL : `` æp . @ l ) To overcome the shortcomings of APL activation function , we define a new adaptive piecewise linear activation function that is zero-centered and gives the same expressive power to the positive and negative half of the input space . Since S-APL shares the variables as+ , b s + , a s − , and b s − among all the neurons of a layer ( e.i. , hi ( x , S ) does not depend on i ) , it requires fewer parameters in compare to the plain APL . We formulate the activation hi of the hidden unit i as the summation of symmetric hinge-shaped pieces . hi ( x , S ) = S∑ s=1 as+max ( 0 , x− bs+ ) + S∑ s=1 as−max ( 0 , −x− bs− ) ( 2 ) Similar to APL units , S is a hyperparameter which determines the number of hinges ( on each side of the x-axis ) . as+ and a s − control the slope of linear pieces in both the positive and negative sides of the x-axis , while bs+ , b s − ∈ [ 0 , +∞ ) determine the location of hinge s. The total number of additional parameters is 4S for a single layer . Because our networks are trained with batch normalization ( Ioffe & Szegedy , 2015 ) before each S-APL layer , the distribution of input to the S-APL function will follow that of a normal distribution . Therefore , we fix the hinges to be a pre-determined number of standard deviations away from the mean , decreasing the number of parameters to 2S . Based on Equation 2 , S-APL activation is zero-centered which leads to more stable statistical behavior during training . In case of using fixed hinges ’ locations for APL , due to increasing the number of parameters , S-APL offers more flexibility than the normal APL . However , this increase is of O ( constant ) which is negligible . With the assumption of as+ = a s − and b s + = b s − , since hsi ( x ) = h s i ( −x ) , S-APL allows neurons to activate symmetrically . As it is mentioned in Zhao & Griffin ( 2016 ) , networks with symmetric activation functions are more robust to adversarial fooling . We will delve into this property in later sections . S-APL can better handle the limits . It could be inferred from the definition that , in contrast to normal APL , increasing the absolute value of the input xwill also increase the number of terms affecting the hi ( x ) . However , normal APL requires the rightmost linear section in all the component functions to have a unit slope and bias 0 , which is not an appropriate constraint and undermines its representation ability . The following theorem shows that S-APL can approximate any non-linear , zero-centered , and Lipschitz continuous function in a closed interval of real numbers . It is worth mentioning that APL was not flexible enough to do so . Theorem 3.1 For any function f : [ A , B ] → R and ∈ R , ∃S ∈ N , where |f ( x ) − S-APL ( x ) | ≤ , assuming : • A and B are finite real numbers . • f is M-Lipschitz continuous . Proof : We start by dividing the interval [ A , B ] into S equal sub-intervals [ Ai , Bi ] . In other words , we put S equally distanced hinges in the interval [ A , B ] . So the length of each sub-interval will be B−AS . We start approximating f from the left most sub-interval which is [ A , A + B−A S ] . The approximation of f in that sub-interval will be a line which connects f ( A ) to f ( A+ B−AS ) . Due to M-Lipschitz continuity of f , for any x ∈ [ A , A+ B−AS ] : |f ( x ) − S-APL ( x ) | ≤ { |f ( A ) − f ( A+ B−AS ) + M ( B−A ) S | f ( A ) ≤ f ( A+ B−A S ) , |f ( A+ B−AS ) − f ( A ) + M ( B−A ) S | f ( A ) ≥ f ( A+ B−A S ) . ( 3 ) Again , due to M-Lipschitz continuity of f , we know that : |f ( A ) − f ( A+ B −A S ) | ≤ M ( B −A ) S ( 4 ) Using Equation 4 , we can re-write the inequality 3 as follows : |f ( x ) − S-APL ( x ) | ≤ 2M ( B −A ) S ( 5 ) So , by choosing S ≥ 2M ( B−A ) , we satisfy that |f ( x ) − S-APL ( x ) | ≤ . in order to complete the proof , we need to show that this approximation can be applied to other sub-intervals of [ A , B ] as well . We do so by induction as follows : Suppose that the approximation is done for sub-intervals [ A1 , B1 ] , ... , [ Ai , Bi ] , now we prove that for sub-interval [ Ai+1 , Bi+1 ] we can do the approximation which is connecting f ( Ai+1 ) to f ( Bi+1 ) . The slope of S-APL in the subinterval [ Ai , Bi ] is ∑i 1 a i + or ∑i 1 a i − ( depending on the sign of the sub-interval ) . However , the slope of S-APL in the sub-interval [ Ai+1 , Bi+1 ] will be either ∑i+1 1 a i + or ∑i+1 1 a i − . In both cases , the extra term ai+1+ or a i+1 − is not fixed and can change the slop to any arbitrary value . This fact plus the assumption of continuity of S-APL guarantee that f ( Ai+1 ) can be connected to f ( Bi+1 ) which was our proposed approximation . As we showed the approximation for the left most sub-interval , the induction is complete .

Summary: This paper proposes a learnable piece-wise linear activation unit whose hinges are placed symmetrically. It gives a proof on the universality of the proposed unit on a certain condition. The superiority of the method is empirically shown. The change of the activation during training is analyzed and insight on the behavior is provided. The robustness to adversarial attacks is also empirically examined.