Discovering Parametric Activation Functions

1 INTRODUCTION . The rectified linear unit ( ReLU ( x ) = max { x , 0 } ) is the most commonly used activation function in modern deep learning architectures ( Nair & Hinton , 2010 ) . When introduced , it offered substantial improvements over the previously popular tanh and sigmoid activation functions . Because ReLU is unbounded as x→∞ , it is less susceptible to vanishing gradients than tanh and sigmoid are . It is also simple to calculate , which leads to faster training times . Activation function design continues to be an active area of research , and a number of novel activation functions have been introduced since ReLU , each with different properties ( Nwankpa et al. , 2018 ) . In certain settings , these novel activation functions lead to substantial improvements in accuracy over ReLU , but the gains are often inconsistent across tasks . Because of this inconsistency , ReLU is still the most commonly used : it is reliable , even though it may be suboptimal . The improvements and inconsistencies are due to a gradually evolving understanding of what makes an activation function effective . For example , Leaky ReLU ( Maas et al. , 2013 ) allows a small amount of gradient information to flow when the input is negative . It was introduced to prevent ReLU from creating dead neurons , i.e . those that are stuck at always outputting zero . On the other hand , the ELU activation function ( Clevert et al. , 2015 ) contains a negative saturation regime to control the forward propagated variance . These two very different activation functions have seemingly contradicting properties , yet each has proven more effective than ReLU in various tasks . There are also often complex interactions between an activation function and other neural network design choices , adding to the difficulty of selecting an appropriate activation function for a given task . For example , Ramachandran et al . ( 2018 ) warned that the scale parameter in batch normalization ( Ioffe & Szegedy , 2015 ) should be set when training with the Swish activation function ; Hendrycks & Gimpel ( 2016 ) suggested using an optimizer with momentum when using GELU ; Klambauer et al . ( 2017 ) introduced a modification of dropout ( Hinton et al. , 2012 ) called alpha dropout to be used with SELU . These results suggest that significant gains are possible by designing the activation function properly for a network and task , but that it is difficult to do so manually . This paper presents an approach to automatic activation function design . The approach is inspired by genetic programming ( Koza , 1992 ) , which describes techniques for evolving computer programs to solve a particular task . In contrast with previous studies ( Bingham et al. , 2020 ; Ramachandran et al. , 2018 ; Liu et al. , 2020 ; Basirat & Roth , 2018 ) , this paper focuses on automatically discovering activation functions that are parametric . Evolution discovers the general form of the function , while gradient descent optimizes the parameters of the function during training . The approach , called PANGAEA ( Parametric ActivatioN functions Generated Automatically by an Evolutionary Algorithm ) , discovers general activation functions that improve performance overall over previously proposed functions . It also produces specialized functions for different architectures , such as Wide ResNet , ResNet , and Preactivation ResNet , that perform even better than the general functions , demonstrating its ability to customize activation functions to architectures . 2 RELATED WORK . Prior work in automatic activation function discovery includes that of Ramachandran et al . ( 2018 ) , who used reinforcement learning to design novel activation functions . They discovered multiple functions , but analyzed just one in depth : Swish ( x ) = x ·σ ( x ) . Of the top eight functions discovered , only Swish and max { x , σ ( x ) } consistently outperformed ReLU across multiple tasks , suggesting that improvements are possible but often task specific . Bingham et al . ( 2020 ) used evolution to discover novel activation functions . Whereas their functions had a fixed graph structure , PANGAEA utilizes a flexible search space that implements activation functions as arbitrary computation graphs . PANGAEA also includes more powerful mutation operations , and a function parameterization approach that makes it possible to further refine functions through gradient descent . Liu et al . ( 2020 ) evolved normalization-activation layers . They searched for a computation graph that replaced both batch normalization and ReLU in multiple neural networks . They argued that the inherent nonlinearity of the discovered layers precluded the need for any explicit activation function . However , experiments in this paper show that carefully designed parametric activation functions can in fact be a powerful augmentation to existing deep learning models . Finally , Basirat & Roth ( 2018 ) used a genetic algorithm to discover task-specific piecewise activation functions . They showed that different functions are optimal for different tasks . However , the discovered activation functions did not outperform ELiSH and HardELiSH , two hand-designed activation functions proposed in the same paper ( Basirat & Roth , 2018 ) . The larger search space in PANGAEA affords evolution extra flexibility in designing activation functions , while the trainable parameters give customizability to the network itself , leading to consistent , significant improvement . 3 THE PANGAEA METHOD . 3.1 REPRESENTING AND MODIFYING ACTIVATION FUNCTIONS . Activation functions are represented as computation graphs in which each node is a unary or a binary operator ( Table 1 ) . The activation functions are implemented in TensorFlow ( Abadi et al. , 2016 ) , and safe operator implementations are chosen when possible ( e.g . the binary operator x1/x2 is implemented as tf.math.divide_no_nan , which returns 0 if x2 = 0 ) . The operators in Table 1 were chosen to create a large and expressive search space that contains activation functions unlikely to be discovered by hand . Operators that are periodic ( e.g . sin ( x ) ) and operators that contain repeated asymptotes were not included ; in preliminary experiments they often caused training instability . All of the operators have domain R , making it possible to compose them arbitrarily . PANGAEA begins with an initial population of P random activation functions . Each function is either of the form f ( x ) = unary1 ( unary2 ( x ) ) or f ( x ) = binary ( unary1 ( x ) , unary2 ( x ) ) , as shown in Figure 1 . Both forms are equally likely , and the unary and binary operators are also selected uniformly at random . Previous work has suggested that it is difficult to discover highperforming activation functions that have complicated computation graphs ( Bingham et al. , 2020 ) . The computation graphs in Figure 1 thus represent the simplest non-trivial computation graphs with and without a binary operator . During the search , all ReLU activation functions in a given neural network are replaced with a candidate activation function . No other changes to the network or training setup are made . The network is trained on the dataset , and the activation function is assigned a fitness score equal to the network ’ s accuracy on the validation set . Given a parent activation function , a child activation function is created by applying one of four possible mutations ( Figure 2 ) . Other possible evolutionary operators like crossover are not used in this paper . All mutations are equally likely with two special cases . If a remove mutation is selected for an activation function with just one node , a change mutation is applied instead . Additionally , if an activation function with greater than seven nodes is selected for mutation , the mutation is a remove mutation , in order to reduce bloat . Insert In an insert mutation , one operator in the search space is selected uniformly at random . This operator is placed on a random edge of a parent activation function graph . In Figure 2b , the unary operator Swish ( x ) is inserted at the edge connecting the output of tanh ( x ) to the input of x1 + x2 . After mutating , the parent activation function ( tanh ( x ) + |erf ( x ) | ) 2 produces the child activation function ( Swish ( tanh ( x ) ) + |erf ( x ) | ) 2 . If a binary operator is randomly chosen for the insertion , the incoming input value is assigned to the variable x1 . If the operator is addition or subtraction , the input to x2 is set to 0 . If the operator is multiplication , division , or exponentiation , the input to x2 is set to 1 . Finally , if the operator is the maximum or minimum operator , the input to x2 is a copy of the input to x1 . When a binary operator is inserted into a computation graph , the activation function computed remains unchanged . However , the structure of the computation graph is modified and can be further altered by future mutations . Remove In a remove mutation , one node is selected uniformly at random and deleted . The node ’ s input is rewired to its output . If the removed node is binary , one of the two inputs is chosen at random and is deleted . The other input is kept . In Figure 2c , the addition operator is removed from the parent activation function . The two inputs to addition , tanh ( x ) and |erf ( x ) | , can not both be kept . By chance , tanh ( x ) is discarded , resulting in the child activation function |erf ( x ) |2 . Change To perform a change mutation , one node in the computation graph is selected at random and replaced with another operator from the search space , also uniformly at random . Unary operators are always replaced with unary operators , and binary operators with binary operators . Figure 2d shows how changing addition to multiplication produces the activation function ( tanh ( x ) · |erf ( x ) | ) 2 . Regenerate In a regenerate mutation , every operator in the computation graph is replaced with another operator from the search space . As with change mutations , unary operators are replaced with unary operators , and binary operators with binary operators . Although every node in the graph is changed , the overall structure of the computation graph remains the same . Regenerate mutations are useful for increasing exploration , and are similar in principle to burst mutation and delta coding ( Gomez & Miikkulainen , 2003 ; Whitley et al. , 1991 ) . Figure 2e shows the child activation function −max { 0 , tanh ( SELU ( x ) ) } , which is quite different from the parent function in Figure 2a . Parameterization of Activation Functions After mutation ( or random initialization ) , activation functions are parameterized ( Figure 3 ) . A value k ∈ { 0 , 1 , 2 , 3 } is chosen uniformly at random , and k edges of the activation function graph are randomly selected . Multiplicative per-channel parameters are inserted at these edges and initialized to one . Whereas evolution is well suited for discovering the general form of the activation function in a discrete , structured search space , parameterization makes it possible to fine-tune the function using gradient descent . The function parameters are updated at every epoch during backpropagation , resulting in different activation functions in different stages of training . As the parameters are per-channel , the process creates different activation functions at different locations in the neural network . Thus , parameterization gives neural networks additional flexibility to customize activation functions .

Summary: The authors propose to search for activation functions with regularized evolution, an evolutionary algorithm proposed by Real et al. Various mutations are proposed that allow to investigate a larger search space than prior work. In particular, a mutation is added which adds trainable parameters to the activation function. The discovered activation functions are compared on three different architectures to several state-of-the-art activation functions.