Discovering Parametric Activation Functions
Authors: Garrett Bingham
Risto Miikkulainen
Venue:  (2021)
Keywords: activation function
parametric
evolution

Abstract: Recent studies have shown that the choice of activation function can significantly affect the performance of deep learning networks. However, the benefits of novel activation functions have been inconsistent and task-dependent, and therefore the rectified linear unit (ReLU) is still the most commonly used. This paper proposes a technique for customizing activation functions automatically, resulting in reliable improvements in performance. Evolutionary search is used to discover the general form of the function, and gradient descent to optimize its parameters for different parts of the network and over the learning process. Experiments with three different neural network architectures on the CIFAR-100 image classification dataset show that this approach is effective. It discovers different activation functions for different architectures, and consistently improves accuracy over ReLU and other recently proposed activation functions by significant margins. The approach can therefore be used as an automated optimization step in applying deep learning to new tasks.

Under review as a conference paper at ICLR 2021
DISCOVERING PARAMETRIC ACTIVATION FUNCTIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent studies have shown that the choice of activation function can significantly affect the performance of deep learning networks. However, the benefits of novel activation functions have been inconsistent and task dependent, and therefore the rectified linear unit (ReLU) is still the most commonly used. This paper proposes a technique for customizing activation functions automatically, resulting in reli- able improvements in performance. Evolutionary search used to discover the general form of the function, and gradient descent to optimize its parameters for different parts of the network and over the learning process. Experiments with four different neural network architectures age classification datasets show that this approach is effective. It discovers both general activation functions and specialized functions for different architectures, consistently improving accuracy over ReLU and other recently proposed actíva- tion functions by significant margins. The approach therefore be used automated optimization step in applying deep learning to new tasks.
the CIFAR-IO and CIFAR-100 im-
INTRODUCTION
The rectified linear unit (ReLU(x>) max ar, 0}) is the most commonly used activation function in modern deep learning architectures (Nair & Hinton 2010). When introduced, it offered substantial improvements over the previously popular tanh and sigmoid activation functions. Because ReLU is unbounded as x - oc, it is less susceptible to vanishing gradients than tanh and sigmoid are. It is also simple to calculate, which leads faster training times.
Activation function design continues to be an active area of research, and number of novel activation functions have been introduced since ReLU, each with different properties (Nwankpa et al.) 2018). In certain settings, these novel activation functions lead to substantial improvements in accuracy over ReLU, but the gains are often inconsistent across tasks. Because this inconsistency, ReLU still the most commonly used: is reliable, even though it may be suboptimal.
The improvements and inconsistencies are due to a gradually evolving understanding of what makes an activation function effective. For example, Leaky ReLU (Maas et al. 2013) allows small amount of gradient information to flow when the input is negative. was introduced to prevent ReLU from creating dead neurons, i.e. those that stuck at always outputting zero. On the other hand, the ELU activation function (Clevert et al.) 2015) contains negative saturation regime to control the forward propagated variance. These very different activation functions have seemingly contradicting properties, yet each has proven more effective than ReLU in various tasks.
There are also often complex interactions between an activation function and other neural network design choices, adding to the difficulty of selecting an appropriate activation function for a given task. For example, IRamachandran
al. (2018) warned that the scale parameter in batch normalization (Ioffe & Szegedy, 2015) should be set when training with the Swish activation function; Hendryyck Gimpel[(216) suggested using an optimizer with momentum when using GELU; Klambauer et al.) (2017) introduced modification of dropout (Hinton et al. 2012) called alpha dropout to be used with SELU. These results suggest that significant gains are possible by designing the activation function properly for network and task, but that it is difficult to do manually.
This paper presents an approach to automatic activation function design. The approach is inspired by genetic programming (Koza 1992), which describes techniques for evolving computer programs to solve a particular task. In contrast with previous studies (Bingham et al. 2020 Ramachandran et al. 2018: Liu et al. 2020; Basirat& & Roth 2018), this paper focuses automatically discovering activation functions that are parametric. Evolution discovers the general form of the function, while gradient descent optimizes the parameters of the function during training. The approach,

---

Under review as a conference paper at ICLR 2021
Table 1: The operator search space consists of basic unary and binary functions as well as existing activation functions (Appendix Dj. azx) (1 + e --z)- The unary operators bessel i0e and bessel_ile are the exponentially scaled modified Bessel functions of order and respectively.
Unary
Binary
0
[x]
crf(z)
tanh(x) ex -1
arcsinh(r)
(eLU(())
Softplus(a)) Softsign(x))
31 + 32
21
I
erfe(x)
arctanh(2)) (x:)
ELU(()
max
x
x²
sinh(x)
(xx
bessel (z)
sELU(
HardSigmoid(z)
(21,
et
cosh(2)
lx)(
bessel (2)(2)
Swish(z)
01/12
called PANGAEA (Parametric ActivatioN functions Generated Automatically by an Evolutionary Algorithm), discovers general activation functions that improve performance overall over previously proposed functions. It also produces specialized functions for different architectures, such as Wide ResNet, ResNet, and Preactivation ResNet, that perform even better than the general functions, demonstrating its ability to customize activation functions to architectures.
RELATED WORK
Prior work automatic activation function discovery includes that of Ramachandran et al. (2018). who used reinforcement learning design novel activation functions. They discovered multiple functions, but analyzed just one in depth: Swish(x) = x (xx). Of the top eight functions discovered, only Swish and max (x)) consistently outperformed ReLU across multiple tasks, suggesting that improvements are possible but often task specific.
Bingham et al. (2020) used evolution to discover novel activation functions. Whereas their functions had a fixed graph structure, PANGAEA utilizes a flexible search space that implements activation functions arbitrary computation graphs. PANGAEA also includes more powerful mutation operations, and a function parameterization approach that makes it possible to further refine functions through gradient descent.
Liu et al. 2020) evolved normalization-activtiii layers. They searched for a computation graph that replaced both batch normalization and ReLU in multiple neural networks. They argued that the inherent nonlincarity of the discovered layers precluded the need for any explicit activation function. However, experiments in this paper show that carefully designed parametric activation functions can in fact be a powerful augmentation to existing deep learning models.
Finally, Basirat Roth (2018) used a genetic algorithm discover task-spectns piecewise activation functions. They showed that differ- ent functions are optimal for different tasks. However, the discovered activation functions did not outperform ELiSH and HardELiSH, two hand-designed activation functions proposed in the same paper (Basirat & Roth 2018). The larger search space in PANGAEA affords evolution extra flexibility in designing activation functions, while the trainable pa- rameters give customizability to the network itself, leading to consistent, significant improvement.
(
(v)
Unary
Binary
Unary
Unary
Unnry
Figure 1: Random acti- vation function initializa- tion. The initial popula- tion consists of random samples of two kinds of computation graphs, ran- domly initialized with the operators in Table [ In this manner, the search starts with simple graphs and gradually expands to more complex forms.
THE PANGAEA METHOD
3.1 REPRESENTING AND MODIFYING ACTIVATION FUNCTIONS
Activation functions are represented as computation graphs in which each node is unary binary operator [T. The activation functions are implemented TensorFlow (Abadi et al., 2016), and safe operator implementations are chosen when possible (e.g. the binary operator 2 1/12 implemented as tf math di divide se_________ _n___ann which returns 0 if J2 0). The operators in Table I were chosen to create large and expressive search space that contains activation functions unlikely to be discovered by hand. Operators that are periodic (e.g. (x)) and operators that contain repeated asymptotes were not included; in
2

---

used in Under review as a conference paper at ICLR 2021
max fars,
(xnh(
Swish((.
ar
lanh(ar)
Lanhi
lanh(ar)
SELU
(a) Parent
(b) Insert
(c) Remove
(d) Change
(e) Regenerate
Figure 2: Evolutionary operations activation functions. In 'Insert' mutation, new operator is inserted in one the edges the computation graph, like the Swish(x)) (b). In a 'Remove' mutation, a node in the computation graph is deleted, like the addition in (c). In 'Change' mutation, an operator at a node is replaced with another, like addition with multiplication three mutations are useful in refining the function locally. In contrast, in 'Regenerate" mutation (e), every operator in the graph is replaced by random operator, thus increasing exploration.
(d). These first
preliminary experiments they often caused training instability, All of the operators have domain R. making it possible to compose them arbitrarily.
PANGAEA begins with initial population of P random activation functions. Each function is
either of the form f(x cynary (una (unaar2(())) or [(x) b y narnyunnaryyl (xx), unary22(4 (x)). as shown in Figure ! Both forms equally likely, and the unary and binary operators are also selected uniformly at random. Previous work has suggested that difficult to discover high- performing activation functions that have complicated computation graphs (Bingham et al. (020). The computation graphs Figure thus represent the simplest non-trivial computation graphs with and without binary operator.
During the search, all ReL.U activation functions in given neural network are replaced with a candidate activation function. No other changes to the network training setup made. The network is trained on the dataset, and the activation function assigned a fitness score equal to the network's accuracy on the validation set.
Given a parent activation function, a child activation function is created by applying one of four possible mutations (Figure 2 Other possible evolutionary operators like crossover this paper. All mutations are equally likely with two special cases. If a remove mutation is selected for an activation function with just one node, change mutation is applied instead. Additionally, if an activation function with greater than seven nodes is selected for mutation, the mutation is remove mutation, in order reduce bloat.
Insert In an insert mutation, one operator in the search space is selected uniformly at random. This operator is placed on random edge of a parent activation function graph. In Figure Zb, the unary operator Swish(r)) is inserted the edge connecting the output of tanh(x) to the imput of æ + ø2- After mutating, the parent activation function (tanh(x) + |erf(x)))> produces the child activation function (Swish(tanh(2)
+ |erf(x))) If binary operator randomly chosen for the insertion, the incoming input value is assigned to the variable æ If the operator is addition or subtraction, the input to 22 is set 0. If the operator multiplication, division, or exponentiation, the input to 22 is set to 1. Finally, the operator is the maximum or minimum operator, the input to xy is a copy of the input to 21. When a binary operator is inserted into computation graph, the activation function computed remains unchanged. However, the structure of the computation graph is modified and can be further altered by future mutations.
Remove In remove mutation, one node is selected uniformly at random and deleted. The node's input is rewired to its output. If the removed node is binary, one of the two inputs is chosen at random and is deleted. The other imput is kept. In Figure the addition operator is removed from the parent activation function. The two inputs to addition, tanh(z) and |erf(z)), cannot both kept. By chance, tan)(x) is discarded, resulting in the child activation function |erf(=:))2

---

Under review as a conference paper at ICLR 2021
Change To perform change mutation, one node in the computa- tion graph is selected at random and replaced with another operator from the search space, also uniformly at random. Unary opera- tors are always replaced with unary operators, and binary operators with binary operators. Figure 2 shows how changing addition multiplication produces the activation function (tanh(r) |erf(=)))??
Regenerate In regenerate mutation, every operator in the com- putation graph is replaced with another operator from the search space. As with change mutations, unary operators are replaced with unary operators, and binary operators with binary operators. Al- though every node in the graph is changed, the overall structure of the computation graph remains the same. Regenerate mutations useful for increasing exploration, and are similar in principle to burst mutation and delta coding (Gomez & Mikkullinenn 2003] Whitley et al., 1991). Figure 2 shows the child activation fune- tion max (0, tanh (SELU())) which is quite different from the parent function in Figure 2
[x|
arctan( x])
arctan (2r
Figure 3: Parameterization of activation functions. In this example, parameters are random
added to k edges, yielding the parametric activation function our (B\xl] (y))an(?xx)).
Parameterization of Activation Functions After mutation (or random initialization), activation functions are parameterized (Figure 3) A value k E 1, is chosen uniformly at random, and k edges of the activation function graph are randomly selected. Multiplicative per-channel parameters are inserted at these edges and initialized to one. Whereas evolution is well suited for discovering the general form of the activation function discrete, structured search space, parameterization makes it possible to fine-tune the function using gradient descent. The function parameters are updated at every epoch during backpropagation. resulting in different activation functions in different stages of training. As the parameters per-channel, the process creates different activation functions at different locations in the neural network. Thus, parameterization gives neural networks additional flexibility to customize activation functions.
3.2 DISCOVERING ACTIVATION FUNCTIONS WITH EVOLUTION
Activation functions are discovered by regularized evolution (Real al. 2019). Initially, / random activation functions are created, parameterized, and assigned fitness scores. To generate a new activation function, S functions are sampled with replacement from the current population. The function with the highest validation accuracy serves as the parent, and is mutated to create child activation function. This function parameterized and assigned a fitness score. The new activation function is then added to the population, and the oldest function in the population is removed, ensuring the population is always of size P. This process continues until functions have been evaluated in total, and the top functions over the history of the search are returned result.
Any activation function that achieves fitness score less than a threshold V discarded. These functions are not added to the population, but they do count towards the total number of C' activation functions evaluated for each architecture. This quality control mechanism allows evolution to focus only the most promising candidates.
To save computational resources during evolution, each activation function evaluated by training a neural network for 100 epochs using compressed learning rate schedule (Appendix B). After evolu- tion is complete, the top 10 activation functions from the entire search are reranked. Each function receives an adjusted fitness score equal the average validation accuracy from two independent 200-epoch training runs using the original learning rate schedule. The top three activation functions after reranking proceed the final testing experiments.
During evolution, it is possible that some activation functions achieve unusually high validation accuracy by chance. The 100-epoch compressed learning rate schedule may also have minor effect on which activation functions are optimal compared to full 200-epoch schedule. Reranking thus serves two purposes. Full training reduces bias from the compressed schedule, and averaging two such runs lessens the impact of activation functions that achieved high accuracy by chance.
4

---

Under review as a conference paper at ICLR 2021
4 DATASETS AND ARCHITECTURES
The experiments in this paper focus primarily on the CIFAR-100 image classification dataset (Krizhevsky et al.) 2009). This dataset is a more difficult version of the popular CIFAR-10 dataset, with 100 object categories instead of 10. Fifty images from each class were randomly selected from the training set to create a balanced validation set, resulting in a training/validationnesst split of 45K/5K/10K images.
To demonstrate that PANGAEA can discover effective activation functions in various settings, it is evaluated with three different neural networks. The models were implemented in TensorFlow (Abadi al. 2016), mirroring the original authors training setup closely possible (Appendix B
Wide Residual Network (WRN-10-4; Zagoruyko & Komodakis 2016) has a depth of 10 and widening factor of four. Wide residual networks provide an interesting comparison because they are shallower and wider than many other popular architectures, while still achieving good results. WRN-10-4 was chosen because its CIFAR- 100 accuracy is competitive, yet it trains relatively quickly.
Residual Network (ResNet-v1-56) He et al. 2016a), with a depth of 56, provides an important contrast to WRN- It is significantly deeper and has slightly different training setup, which may have an effect on the performance of different activation functions.
Preactivation Residual Network (ResNet-v2-5or He et al. 2016b) has identical depth to ResNet- v1-56, but is a fundamentally different architecture. Activation functions are not part of the skip connections, is the case in LesNet-v1-56. Since information does not have pass through an activation function, this structure makes it casier to train very deep architectures. PANGAEA should exploit this structure and discover different activation functions for ResNet-vv-56 and ResNet-v1-56.
5 RESULTS
Overview Separate evolution exper- iments were run to discover novel ac- tivation functions for each the three architectures. Evolutionary parame- ters P = 64, S = 16, C = 1,000, and V 20% were used since they were found to work well in prelimi- nary experiments.
0.75
0.71
Accureey
0.73
WRX-10-4
0.72
Jerskalied
ResNee
ResNet-v2-56
Acc w/ BeLU
0.70
Anmotated func.
Figure visualizes progress in these experiments. For all three architec- tures, PANGAEA quickly discovered activation functions that outperform ReLU. It continued to make further progress, gradually discovering bet- ter activation functions, and did not plateau during the time allotted for the experiment. Each run took approxi- mately 2,000 GPU hours on GeForce GTX 1080 GPUs (Appendix C).
0.68
0.68
200
400
600
800
1000
Activaaion Fimctions Exaluaced
Figure 4: Progress of PANGAEA three different neural networks. Evolution quickly discovered activation functions that outperform ReLU (shown = 0), and continued to improve throughout the experiment. The plots show the high- est validation accuracy of all activation functions evaluated so far after 100 epochs of training. Notable discovered ac- tivation functions are identified with a star and annotated. The improvements over ReLUU are meaningful, but the val- ues themselves are not directly comparable to the results in Table 2 which lists test set accuracy after 200 epochs.
Table 2 shows the final test accuracy for the top specialized activation func- tions discovered by PANGAEA in each run. For comparison, the accu- racy of the top general functions dis- covered in this process are also shown, as well the accuracy of 28 baseline activation functions. In sum, PANGAEA discovered the best activation function for ResNettvv-56,, the top two activation functions for ResNet-v1-56, and the top three activation functions for WRN-10-4.
Specialized Activation Functions For all three architectures, there is at least one baseline acti- vation function that outperforms ReLU by a statistically significant margin. This result already demonstrates the importance of activation function design, and suggests that the common practice of
5

---

Under review as a conference paper at ICLR 2021
Table 2: CIFAR-100 test set accuracy shown as median of ten runs, with mean + sample standard deviation in parenthesis. The top accuracy for each architecture is in bold. Asterisks indicate a statistically significant improvement in mean accuracy over ReLU, with * p S 0.05, ** p S 0.01, and *** if S 0.001; p-values are from one-tailed Welch's t-tests. The ++ or +++ indicate a statistically significant improvement in mean accuracy over all 28 baseline activation functions (Appendix D). with p < 0.01 or p < 0.001 in every case, respectively.
WRN-10-4
ResNet-v1-56
Res. Res.Net-v2-56
Specialized for WRN-10-4
(og()((az)) arcsinh(ar) (ar)
73.23 (73.1610441) -..
11.15 (10.4 20.14)
72.05 (64.30 21.32)
log(o(axr) ) Baresinh(z))
73.22 (73.20 = 0.17) **+
05.78 (18.63 1.5.121000 21.04)
55.40 (45.88 1.30000
Swish (wish(Swish(ar))
72.38 (72.49 = 0.58)
59.61 (39.86 a: 2.88)
74.70 (74.71 0.200)
Specialized for ResNet-v1-56 ax
70.35 (70.28 a. 0.37)
70.82 (71.01 1 0.64)
74.41 (74.36 0.45)
(3x))
70.62 (70.47 4 0.53)
70.30 (70.30 +: 0.58)
74.73 (74.70 0.22)
max (w)
71.96 (72:10 d: 0.33)
69.46 (69.43 d: 0.00)
74.97 (74.97
Specialized for ResNet-vv256
Softplus(ELU(zz))
71.51 (71.36 0.34)
69.94 (60.96 0.39;
75.60 (75.61 0.42
min (olllo(())))))))))()((()))))(( (og((((22))
72.05 (72.04 a 0.34)
69.63 (69.56
75.20 (75.19 1: 0.39) ***
SELU(Swish(s))
01.00 (01.00 4: 0.00)
01.00 (01.00 0.00;
75.06 (75.02 1 0.33) "
General Activation Functions
max (2)), , a (RLL((())L
72.50 (72.54 0.26) ...
69.97 (69.91 à 0.377
75.21 (75.20 A. 0.41) te
min(Swish(z)) (ReLU
72.44 (72.39 4: 0.29)
69.90 (69.82 0.40)
75,20 (70.27 a: 0.38)
(x))
72.38 (72.33 0.32)
69.49 (69.58 0.35)
75.45 (75.58 = 0.37)
Baseline Activation Functions
ReLU
71.44 (71.46 0.50)
69.78 (69.64 0.65)
74.13 (74.39 0.44)
ELiSH
01.00 (01.00 0.000
01.00 (01.00 0.00;
75.16 (75.20 0:31)
ELU
72.41 (72.30 0.32) V.w
69.59 (69.67 0.46)
74.86 (74.96 0.300
GELU
72.00 (71.95 +: 0.35)
70.16 (70.19 0.40)
74.84 (74.86 0:0:::00000 0.30) **
HardSigmoid
55.55 (64.90 + 1.00)
33.31 (32.65 = 4.06)
65.03 (64.90 0.69)
Leaky ReLU
71.76 (71.73 0.33)
60.77 (69.78 0.33)
74.75 (74.73 0.35)*
Mish
72.0 (71.95 0.41)
70.03 (69.88 0.54;
75.33 (75.32 0.29)
SELU
70.55 (70.53
68.51 (68.52 0.29)
73.86 (73.79 0.36)
sigmoid
56.45 (66.10 0.98)
37.07 (36.47 3.32)
66.72 (66.46 0.92)
Softplus
72.25 (72.27 0.26)
69.71 (69.7) 0.36;
75.47 (75.46 06 0.52) **a
Softsign
56.72 (06.30 0: 2.16)
58.33 (59.38 0.96)
69.31 (69.38 + 0.39)
Swish
72.27 (72.26 + 0.28)
69.60 (69.68 0.88)
75.17 (75.08 0.36)
tanh
56.29 (56.52 + 1.58)
63.89 (63.68 0.38)
70.53 0.40)
Parametric Baseline Functions
aReLU(55z)
72.01 (71.96 0.31)
68.91 (68.98 0.22)
73.60 (23.52:00.5)
(FLiSH((82)
01.00 (01-00 +: 0.00)
01.00 (01.00 0.00;
73.95 (75.94 0.53)
dELU(5a))
71.96 (71.98 + 0.24)
68.91 (69.06 0.37)
74.03 (73.97 0.40)
GGELU(,
71.86 (71 +: 0.34)
69.35 (69.30 0.35)
73.77 (73.83
aHardSigmoid
66.74 (06.70 70 0.64)
33.47 (34.33 = 6.53;
65.09 (65.10=0.00)
aLeaky (z)
71.70 (71.74 0.399
69.18 (60.11 0.47;
73.53 (73.44 0.200
aMish(3r)
72.02 (72.11 0.31)
69.66 (69.51 0.677
73.72 (73.72 0.32)
aSELU())))
71.04 (71.07 + 0.53)
68.06 (68.05 0.39;
73.44 (75.37 0.58)
asigmoid( (x))
67.16 (00 98 0.66)
43.72 (65.40 = 2.69;
66.80 (66.98 1::6::00 0.85)
Sootplus (3x)
71.82 (71.73 0.31)
68.84 (69 84 0.30)
73.92 (73.98 1.006.1000 0.37)
62. (62
0.83)
01.00 (9.18 13.75)
68.91 (65.87 : 0.38)
nSwish (Br)
72.36 (72.
0.29)
69.25 (60.25
73.97 (73.93 0.22)
(8x)
63.72 (60.55 0.58)
01.00 (02.592 6.07)
69.61 (69.56 0.62)
PReLU
72.25 (72.23 1. 0.37) vew
69.67 (69.77 0.40;
74.99 (75.10 1: 0.50)
PSwish x (B2))
72. 22 (72.40 4: 0.31)
70.19 (70.16 0.46)
75.37 (76.39 e: 0.29)
using ReLU by default is suboptimal. The best baseline activation function is different for different architectures, reinforeing the importance of developing specialized activation functions.
Because PANGAEA uses validation accuracy from single neural network to assign fitness scores to activation functions, there is selective pressure discover functions that exploit the structure of the network. The functions thus become specialized to the architecture. They increase the performance of that architecture; however, they may not be as effective with other architectures. Specialized activation function accuracies are highlighted in gray in Table 2 To verify that the functions are customized to a specific architecture, the functions were cross-evaluated with other architectures.

---

Under review as a conference paper at ICLR 2021
PANGAEA discovered two special- ized activation functions for WRN- 10-4 and one for ResNet-v1-56 that achieved statistically significant im-
[g(rles)] (nght(ae)) porosable (eoth:Biithinne) (oginnnee) (adee(
Epoot 300
provements in mean accuracy all baseline activation functions. All three specialized activation functions evolved for ResNet-v2-55 signifi- cantly outperformed ReLU as well. These results strongly demonstrate the power customizing activation func- tions to architectures.
Apoch 0
Deep Lager
Stallow Lagee
General Activation Functions Al- though the best performance tends to come from specialization, is also useful to discover activation functions that achieve high accuracy across mul- tiple architectures. For instance, they could used initially on a new ar- chitecture before spending compute on specialization. A powerful albeit computationally demanding approach would be to evolve general functions directly. by evaluating candidates on multiple architectures during evolu- tion. However, it turns out that each specialized evolution run already generates variety of functions, many of which are general.
-2
Figure 5: Adaptation of parametric activation functions over time and space. Top: The parameters change during training, resulting different activation functions in the early and late stages. The plots were created by averaging the values of a, 3, and y across the entire network at different training epochs. Bottom:
The parameters updated separately in each channel, inducing different activation functions at dif- ferent locations of neural network. The plots were created by averaging a, B, and y at each layer of the network after the completion of training.
To evaluate whether the PANGAEA runs discovered general functions as well, the top 10 functions from each run were combined into pool of 30 candidate functions. Fach candidate was assigned three fitness scores equal to the average validation accuracy from two independent training runs on each of the three architectures. Candidate functions that were Pareto-dominated, were functionally equivalent to one of the bascline activation functions, or had already been selected as a specialized activation function were discarded, leaving three Pareto-optimal general activation functions.
These functions indeed turned out to be effective as general activation functions: they all performed well all architectures. One outperformed all baseline activation functions WRN- 10-4, while two functions on ResNet-v1-56 and three functions on ResNet-v2-56 outperformed 25 of the 28 baseline functions. However, specialized activation functions, i.e. those specifically evolved for each architecture, still tend to give the biggest improvements.
Shapes of Discovered Functions Many of the top dis- covered activation functions are compositions of multiple unary operators. These functions do not exist in the core unit search space of Ramachandran et al. (018), which requires binary operators. They also do not exist in the S or S2 search spaces proposed by Bingham et al. (2020). which are too shallow. The design of the search space is therefore as important as the search algorithm itself. Pre- vious search spaces that rely on repeated fixed building blocks only have limited representational power. In con- trast, PANGAFA utilizes flexible search space that can represent activation functions in an arbitrary computation graph.
Table 3: CIFAR-100 test set accuracy shown as a median ten runs, with mean + sample standard deviation in parenthesis. The parametric evolved functions tend to outperform their non- parametric counterparts, demonstrating the value of parameterization.
log(o log(o((ar)) arrssih((r) log(or(azi) (affo(aai)--arcsinlee)
73.23 (73.18 0-41)
73. (78:30 0.37)
log(o(z)) arcsinh(z)
72.42 (72.51 0.30)
-Sissh SwishhSwwish
71.99 (71.97
ResNet-v1-56
our 8 (log(a(12))
70.82 (21.01 0.6s)
Figure 5 shows examples of parametric activation func- tions discovered by PANGAEA. As training progresses, gradient descent makes small adjustments the function parameters a, 3, and 7, resulting in activation functions that change over time. This result suggests that it is ad-
log(o(
7U.30 (70.30 0.58)
log (x))
69.44 (69.2) 0.45)
ResNet-v2-56
min (log((((z))) (ng()((rr))
75.20 (75.18 o.0 (70.53 0.37)
log(e(z))

---

Under review as a conference paper at ICLR 2021
vantageous to have activation function in the early stages of training when the network learns rapidly, and a different activation function in the later stages of training when the network is focused on fine-tuning. The parameters a, 3, and y are also learned separately for the different channels, resulting in activation functions that vary with location in a neural network. Functions in deep layers (near the output) are more nonlinear than those in shallow layers (closer to the input), possibly contrasting the need to form regularized embeddings with the need to form categorizations. In this manner, PANGAEA customizes the activation functions to both time and space for each architecture.
6 ABLATIONS AND VARIATIONS
Effect of Parameterization To understand the effect that parameterizing activation functions has on perfor- mance, the specialized functions (Table were trained without them. As Table 3 shows, when parameters are removed, performance drops. The function log(o(z)) is the only exception to this rule. but its high performance is gen-
Table 4: WRN-10-4 accuracy with different activation functions on CIFAR- 100, shown median of ten runs, with mean + sample standard deviation in parenthesis. PANGAFA discovers better activation functions than random search and nonparametrie evolution.
not surprising, since it was previously discovered eral activation function (Table 2). These results confirm that the learnable parameters contributed to the success of PANGAEA.
PANGARA
logla (ex))) arcsinh(s)
73.23 (70.15 6-41)
lor()(ax))) Sarcsinh(2))
73.22 (73.20 a o.ar:
(wish((wishhczzt)
72,38 (72,49 a 0.80;
Random Search oSwish(z)
Search Strategy As additional baseline comparisons, two alternative search strategies were used to discover ac- tivation functions for WRN-10-4. First, a random search baseline was established by applying random mutations without regard to fitness values. This approach corre- sponds setting evolutionary parameters P = 1, S 1, and V = 0%. Second, to understand the effects of func- tion parameterization, a nonparametric evolution baseline was run. This setting is identical to PANGAEA, except functions are not parameterized (Figure 3.. Otherwise, both baselines follow the same setup PA PANGAEA, in- cluding evaluating C 1, 1,000 candidate functions and reranking the most promising ones (Section 3.2
(72.8
Solpluu(z) ardianfozr)
72.78 (22.5) uas))
Rel D ((omesinh(((((()) SELU())))
72.63 (22.0-2 0013
Nongarametric Evolution coch(1) Swish(z)
72.81 (29.78 go9
(e' 1) Swish(ar)
72.57 (20.50 uss.)
Rel (Swishh)))
72.06 (2) 0.54)
ReLU Swish
71. 44 0
72.27 0.9)
Table 5: Specialized activation functions discovered for WRN-10-4, ResNet-vl- 56, and ResNet-vv-56 are evaluated on larger versions of those architec- tures: WRN-16-8, ResNet-v1-110, and ResNet-v2-110, respectively. CIFAR- 100 test accuracy is reported as the median of three runs, with mean + sample standard deviation in parenthe- sis. Specialized activation functions successfully transfer to WRN-16-8 and Res 10, outperforming ReLU.
Table shows the results of this experiment. Random search able to discover good functions that outperform ReLU,, but the functions are not as powerful as those dis- covered by PANGAEA. This result demonstrates the im- portance of fitness selection in evolutionary search. The functions discovered by nonparametric evolution similarly outperform ReLU but underperform PANGAEA. Interest- ingly, without parameterization, evolution is not as cre- ative: two of the three functions discovered are merely Swish multiplied by a constant. Random search and non- parametric evolution both discovered good functions that improved accuracy, but PANGAEA achieves the best per- formance by combining the ad advantages of fitness selection and function parameterization.
WRN-16-8
(gge)(ar)))ea arrsinh((r)
78.42 oua))
log (n(mrirr)): Sarcsinh(ar)
78.35 (76:00 0.17)
-Swish(Swish(aar)
77.90
ReLU
78.14 (76.18 0008)
ResNet-v1-110
or (og(a(12))
70.85 (70.56 0.50)
((ar)
70.40 (2034 0.0n)
mix (Swish((r). 0)
70.30 (70.0 aaig
Rel.1
71.15 05)
ResNet-v2-110
Scaling Up PANGAEA discovered specialized activa- tion functions for WRN- 10-4, ResNet-vil-6, and ResNet- v2-56. Table 5 shows the performance of these activation functions when paired with the larger WRN-16-8 16-8, ResNet-v1-110, and ResNet-v2-110 architectures. Due to time constraints, ReLU is the only baseline activation function in these experiments.
Softplus(ELU((s))
77.34 (77.14 0.38)
76.99 (70.88 0.19)
SwiU,,iiss(zz
77.04 0:14)
ReL.1
76.35 (76344 a013
Two of the three functions discovered for WRN-10-4 outperform ReLU with WRN-16-8, and all three functions discovered for ResNet-v2-55 outperform ReLU with ResNet-v2-110 Interestingly. ReLU achieves the highest accuracy for ResNett-v--110, where activation functions are part of the skip connections, but not for ResNet-v2-110, where they are not. Thus, it is easier achieve
8

---

Under review as a conference paper at ICLR 2021
high performance with specialized activation functions very deep architectures when they not confounded by skip connections. Notably, ResNet-v2-110 with Softplus(ELU(a) performs comparably to much larger ResNet-v2-1001 with ReLU (77.34 vs. 77.29, reported by He et al. (2016)))
Evolving novel activation functions be computationally expensive. The results in Table 5 suggest that it is possible to reduce this cost by evolving activation functions for smaller architectures, and then using the discovered functions with larger architectures.
AlI-CNN-C Finally, to verify that PANGAEA is effec- tive with different datasets and types of architectures, ac- tivation functions were evolved for the AlI-CNN-C (Sprin- genberg et al.] (213) architecture on the CIFAR-10 10 dataset. AII-CNN-Cis is quite distinet from the architectures consid- ered above: it contains only convolutional layers, activa- tion functions, and global average pooling layer, but it does not have residual connections. As shown in Table PANGAEA improves significantly over ReLU in this set- ting well. The accuracy improvement from 88.47% to 92.80% corresponds to an impressive 37.55% reduction in the error rate. This experiment provides further evidence
Table AlI-CNN-C accuracy with dif- ferent activation functions on CIFAR-10, shown as a median of ten runs, with mean + sample standard deviation parenthesis. PANGAEA improves performance significantly also with this different architecture and task.
o Rel RRl aSwish(z) cosh(BB)
92.80 (92.77 0.155)
92.67 (92.66 # 0.06)
aSwish(Brr) ReLU
92.63 (76. 15 1 (4.866)
88.47 (88.47 1 0.14)
that PANGAEA can improve performance for different architectures and tasks.
7 FUTURE WORK
is difficult to select an appropriate activation function for a given architecture because the activation function, network topology, and training setup interact in complex ways. It is especially promising that PANGAEA discovered activation functions that significantly outperformed the baselines, since the architectures and training setups were standard and developed with ReLU. A compelling research direction is to jointly optimize the architecture, training setup, and activation function.
More specifically, there has been significant recent research in automatically discovering the archi- tecture of neural networks through gradient- based, reinforcement learning, or neuroevolutionary methods (Elsken et al.) 2019; Wistuba et al.) 2019: Real et al.] 2019). In related work, evolution was used discover novel loss functions automatically (Gonzalez & Miikkulainen, 2019 2020 Liang et al. 2020). outperforming the standard cross entropy loss. In the future, may be possible to optimize many of these aspects of neural network design jointly. Just as new activation functions improve the accuracy of existing network architectures, it is likely that different architectures will be discovered when the activation function is not ReL.U. One such example is EfficientNet (Tan & Le 2019), which achieved state-of-the-an accuracy for ImagelNet (Deng al. 2009) using the Swish activation function (Ramachandran et al.) 2018 Fifwing et al.) 2018). Coevolution of activation functions, topologies, loss functions, and possibly other aspects of neural network design could allow taking advantage of interactions between them, leading to further improvements in the future.
8 CONCLUSION
This paper introduced PANGAEA, a technique for automatically designing novel, giggh-erforming. parametric activation functions. PANGAEA builds a synergy of two different optimization processes: evolutionary population- based search for the general form, and gradient descent- based fine-tuning of the parameters of the activation function. Compared to previous studies, the search space is extended to include deeper and more complex functional forms, including ones unlikely to be discovered by humans. The parameters are adapted during training and are different in different locations of the architecture, thus customizing the functions over both time and space. PANGAEA is able to discover general activation functions that perform well across architectures, and specialized functions taking advantage of a particular architecture, significantly outperforming previously proposed activation functions in both cases. It is thus promising step towards automatic configuration of neural networks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Abadi, P. Barham, J. Chen, 7. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265-283, 2016.
M. Basirat and P. M. Roth. The quest for the golden activation function. arXiv: /808. 000830 2018.
G. Bingham, W. Macke, and R. Mikkulainen. Evolutionary optimization of deep learning activation functions. In Genetic and Evolutionary Computation Conference (GECCO '00) July 8-12, 2020, Cancún, Mexico, 2020.
D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). CoRR, abs/1511.07225 2015.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F, Li. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
F. Gomez and R. Miikkulainen. Active guidance for finless rocket using neuroevolution. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 2084-2095, 2003.
S. Gonzalez and R. Mikkkulanenn. Improved training speed, accuracy, and data utilization through loss function optimization. arXiv::905.. /528, 2019.
S. Gonzalez and R. Mükkulainen. Evolving loss functions with multivariate taylor polynomial parameterizations. arXiv:2002.00059 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 1034, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
K. He, X. Zhang, Ren, and Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv:1606.08411 2016.
G. E. Hinton, N. Srivastava, A. Krizhevsky, Sutskever, and R. R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors. arXiv: T:X.vv/207.058.. 2012.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
J. R. Koza. Genetic programming: the programming of computers by means of natural selection, volume 1. MIT press, 1992.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
J. Liang, S. Gonzalez, and R. Miikkulainen. Population- basee training for loss function optimization. a4212 2020.
H. Liu, A. Brock. K. Simonyan, and Q. Le. Evolving normalization-actiittioo layers. 029667, 2020.
10

---

Under review as a conference paper at ICLR 2021
A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th international conference on machine learning (ICML-13), pp- 3, 2013.
D. Misra. Mish: A self regularized non-monotonic neural activation function. 1rXiv:190800868) 2019.
V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Pro- ceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall. Activation functions: Comparison of trends in practice and research for deep learning. arXiv:/811.03378, 2018.
P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver BC, Canada, April 30 May 2018, Workshop Track Proceedings, 2018.
E. Real, A. Aggarwal, Y. Huang, and Q V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015.
M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14, 2019.
D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the condor experience. Concurrency and computation; practice and experience, 37(2-4):333-356 2005.
D. Whitley, K. Mathias, and P. Fitzhorn. Delta-Coding: An iterative search strategy for genetic algorithms. In Proceedings of the International Conference on Genetic Algorithms, pp. 77-84, 1991.
M. Wistuba, A. Rawat, and Pedapati. A survey on neural architecture search. arXiv: M.X.v:9055.0192. 2019.
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv: 1oXiv:1605.07146.0

---

Under review as a conference paper at ICLR 2021
CIFAR-1 1.00 Test. Accuracy
CIFAR-10 100 Test Accuracy
77
Relative Imp improvemenn (%)
CIFAR-1 00 Test Accuracy
79
Relative Improvemee (%)
75
Relalive Improvemen poproveemen...
76
2.0
76
75
1.5
63
74
74
73
60
72
72
log parcspbbi)
Sattplus(ELUoo)
12
16
10
16
22
20
34
20 38 56 74 92110
164
WRN-10-X (Changing Width)
WRN-X-4 (Changing Depth)
ResNet-v2-X
Figure 6: CIFAR- 100 test accuracy for different neural networks and activation functions. Accuracy with ReLU is shown in blue, and accuracy with the specialized activation functions in red. The relative improvement of the specialized functions over ReLU is shown as a dotted green line, according the axis values on the right of each plot. Left: The depth of Wide ResNet is fixed at 10, and the width varies from to 16. Center: The depth of Wide ResNet varies from 10 to 34, while the width is fixed at four. Right: The depth of Preactivation ResNet ranges from 20 to 164. The width and depth of network can affect how much a specialized activation function outperforms ReLU.
A ADJUSTING ARCHITECTURE WIDTH AND DEPTH
To further investigate the effect of network size on the performance novel activation functions, two specialized activation functions were paired with neural networks of different widths and depths. Due to time constraints, the results in this experiment are based on single training runs.
The specialized activation function log(o(ar)) Barcsinh(z) was dis- covered for Wide ResNet of depth 10 and width four (WRN-10-4), Figure 6 shows the performance of this function when paired with Wide ResNets of different depths and widths.
Wide Residual Networks
For all widths tested, log(o as the width becomes large. This result implies that (og((((a))) Baresinh(*) gives the network more representational power than ReLU. As the width of the architecture is inereased, the additional network parameters partially offset this advantage, explaining the decreasing relative improvement of log(a(( (oxz) (arcsiinh(zr) over ReLU.
(arcsinh(z)) outperforms ReLU, albeit with diminishing returns
For fixed architecture width of four, log(o Barcsinh(z) outperforms ReLU only when the depth is 10 and 16. Surprisingly, the depth is increased to 22 and beyond, the performance of log( ar (ax)) ) Barcsinh(x)) drops. This result suggests that log(o(azr) (Brrssih(r)) is specialized to shallow architectures.
Preactivation Residual Networks The specialized activation function Softplus(ELU(z)) was discovered for Preactivation ResNet of depth 56 (ResNet-v2-56), Figure6]show the performance of this function when paired with Preactivation ResNets of different depths. Unlike with the Wide ResNets, there is no clear increase or decrease in relative improvement over ReLU as depth increases. Impressively, ResNet-v2- 164 with Softplus( (zLU(x)) achieved test set accuracy 78,01, outperforming the accuracy of ResNet-v2-1001 with ReLU (77.29) as reported by He et al. 2016b).
B TRAINING DETAILS
Wide Residual Network (WRN-10-4) When measuring final performance after evolution. the standard WRN setup used; all ReLU activations in WRN-10-4 are replaced with the evolved activation function, but no other changes to the architecture are made. The network is optimized using stochastic gradient descent with Nesterov momentum 0.9. The network is trained for 200 epochs; the initial learning rate is 1. and it is decreased by factor of 0.2 after epochs 60, 120, and 160. Dropout probability is set to 0.3, and L2 regularization 0.0005 is applied to the weights. Data augmentation includes featurewise center, featurewise standard deviation normalization, horizontal flip, and random 32 x 32 crops of images padded with four pixels on all sides. This setup was chosen to mirror the original WRN setup (Zagoruyko & Komodakis 2016) closely as possible.
12

---

Under review as a conference paper at ICLR 2021
During evolution of activation functions, the training is compressed to save time. The network is trained for only 100 epochs; the learning rate begins at 0.1 and is decreased by a factor of 0.2 after epochs 30, 60, and 80. Empirically, the accuracy achieved by this shorter schedule is sufficient to guide evolution; the computational cost saved by halving the time required to evaluate an activation function can then be used search for additional activation functions.
Residual Network (ResNet-v1-566) As with WRN-10-4, when measuring final performance with ResNet-v1-56, the only change to the architecture is replacing the ReLU activations with an evolved activation function. The network is optimized with stochastic gradient descent and momentum 0.9. Dropout is not used, and L.2 regularization of 0.0001 is applied to the weights. In the original ResNet experiments (He et al., 2016a), an initial learning rate of 0.01 was used for 400 iterations before increasing it to 0, and further decreasing it by factor of 0.1 after 32K and 48K iterations. An iteration represents a single forward and backward pass over one training batch, while an epoch consists of training over the entire training dataset. In this paper, the learning rate schedule is implemented by beginning with a learning rate of 0,01 for one epoch, increasing it to 0.1, and then decreasing it by factor of 0.1 after epochs 91 and 137. (For example, (48K iterations / /5K training images) * batch size of 128 = 137.) The network is trained for 200 epochs in total. Data augmentation includes random horizontal flip and random 32 x 32 crops of images padded with four pixels on all sides, as in the original setup (He et al., 2016a).
When evolving activation functions for ResNet-v1-56. the learning rate schedule is again compressed. The network is trained for 100 epochs; the initial warmup learning rate of 0.01 still lasts one epoch, the learning rate increases to 0.1, and then decreases by factor of 0.1 after epochs 46 and 68. When evolving activation functions, their relative performance is more important than the absolute accuracies they achieve. The shorter training schedule is therefore cost-efficient way of discovering high-performing activation functions.
Preactivation Residual Network (ResNet-v2-56) The full training setup, data augmentation, and compressed learning rate schedule used during evolution for ResNet-v2-56 are all identical to those for ResNet-v1-56 with one exception: with Res-eet-v2-66, it is not necessary warm up training with an initial learning rate of 0.01 (He et al. 20166), so this step is skipped.
AII-CNN-C When measuring final performance with AlI-CNN-C, the ReLU activation function replaced with an evolved one, but the setup otherwise mirrors that SfSiiingegbb et al. (2015) as closely as possible. The network is optimized with stochastic gradient descent and momentum 0.9. Dropout probability 0.5, and L2 regularization of 0.001 is applied to the weights. The data augmentation involves featurewise centering and normalizing, random horizontal flips, and random 32 32 crops of images padded with five pixels on sides. The initial learning rate set to 0.01, and it is decreased by factor of 0.1 after epochs 200, 250, and 300. The network is trained for 350 epochs in total.
During evolution of activation functions, the same training setup was used. It is not necessary to compress the learning rate schedule as was done with the residual networks because All-CNN-C trains more quickly.
CIFAR-10 As with CIFAR- 100, balanced validation set was created for CIFAR-10 by randomly selecting 500 images from each class, resulting in a training/validatioonness split of 45K/5K/10K images.
C IMPLEMENTATION AND COMPUTE REQUIREEENTS
siifhrperformacc computing in two clusters is utilized for the experiments. One cluster uses HTCondor (Thain et al. 2005) for scheduling jobs, while the other uses the Slurm workload manager. Training is executed on GeForce GTX 1080 GPUs on both clusters. When job begins executing, parent activation function is selected by sampling S = 16 functions from the P = 64 most recently evaluated activation functions. This is minor difference from the original regularized evolution (Real et al. 2019), which is based on a strict sliding window of size P. This approach may give extra influence to some activation functions, depending how quickly slowly jobs are executed in each
13

---

Under review as a conference paper at ICLR 2021
of the clusters. In practice the method is highly effective; it allows evolution to progress quickly by taking advantage extra compute when demand on the clusters is low.
It is difficult to know ahead of time how computationally expensive the evolutionary search will be. Some activation functions immediately result in an undefined loss, causing training to end. In that case only few seconds have been spent and another activation function immediately be evaluated. Other activation functions train successfully, but their complicated expressions result in longer-than-usual training times. In these experiments, evolution for WRN-10-4 took 2,314 GPU hours, evolution for ResNet-v1-56 took 1,594 GPU hours, and evolution for ResNet-v2-56 took 2,175 GPU hours. These numbers do not include costs for reranking and repeated runs in the final experiments. Although substantial, the computational cost is negligible compared to the cost in human labor designing activation functions. Evolution of parametric activation functions requires minimal manual setup and delivers automatic improvements in accuracy.
D BASELINE ACTIVATION FUNCTION DETAILS
Table 7: Baseline activation functions from the operator search space (Table 1 and final results
(Table 2)
Name
Definition
Reference(s)
ReLU
max(x, 0}
Nair & Hinton 2010
ELiSH
Ite-a if x 2 0 else d'=1
Basirat & Roth 2018
Ite-1
ELU
x20 else a(e" -1), with =1
Clevert 2015
GELU
<d(()) with (xx) = P(X S xx), X NOO 1),
Hendrycks Gimpel 2016
approximated 0.52(1 + tanhl 2/ + 0.044715x)))
HardSigmoid
max (1.0 + 0.5}]
Leaky ReLU
x x 0 else 0.01æ x (alp(Sootplus(z)) A if x 2 0 else Aa(e- -1,
Maas et 2013
Mish
Misra 2019
SELU
K Klambauer ct al. 2017
with X 1.05070998, 1.67326324
sigmoid
(1+e
Softplus
log(e" 11)
Softsign
=/ </<| +
Swish
(.), with (()) = (1
Ramachandran et al. 2018
and Eltwing 2018
tanh
PReLU
if 0 else where per-neuron learnable parameter initialized 0.25 x (Bax), where is per-channel learnable parameter
Hc al. 2015
PSwish
Ramachandran 2018
14


Reviews:
[{"review_id": "V8l2umihLvJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** This paper proposes to use evolutionary computation to search for optimal activation functions tailored to a given architecture. The proposed PANGEA method practically re-implements a partial Genetic Programming setup applied to searching for neural network activation functions, with interesting (though minor) results. Further investigation is necessary.\n\n**Quality:** The paper is well written, with a good structure. The literature review covers most basis, though with some crucial misses (GP and Random Guessing). The investigation presented is correct and through, though insufficient in breadth. The results are significant and interesting. The core of the paper is well put together, but it requires some more work.\n\n**Clarity:** The concepts are explained clearly and well exposed. The results are motivated and discussed. On the actual work presented, I am happy with the clarity of exposition and presentation.\n\n**Originality:** The work explores a highly promising direction, building on previous results but with a new twist. The results are bare-bone proof-of-concept, the baseline misses some foundation work, and the actual improvements are marginal at best.\n\n**Pros:**\n- Very interesting and promising research direction\n- Correct experimentation methods\n- Very well written and easy to read\n- I personally find this approach highly promising and innovative, and fully support its investigation\n\n**Cons:**\n- While the direction taken is most intriguing, the actual quantity of study is relatively small\n- The claims are not supported by sufficient experimentation (insufficient experiments)\n- Some claims are not supported at all (tailoring activation to architecture)\n- Missing referencing and comparison with GP.\n\n**Comments:**\n- Working with 1 dataset and 3 variations of 1 architecture is too small a baseline to prove the point. While I personally believe in the presented results, further experimentation is necessary.\n- The paper exposure is very pleasant, but it effectively hides the relatively small size of the presented work. As a results it sounds redundant at time, with over-sized pictures and over-complete tables, while the Conclusion paragraph is shorter than the Future Work.\n- Figures 4 and 5 are unnecessarily over-sized and need to be shrunk.\n- The lack of mention of Genetic Programming is unacceptable, especially given the paper available space. Reinventing part of it and rebranding it as PANGEA  for the specific application is interesting but insufficient.\n- To establish the performance of a highly randomized algorithm such as PANGEA, a pure-random parameter guessing baseline needs to be established. Under the proposed rebranding, that would be a run where the operator Regenerate has a 1.0 chance while the others have 0.0.\n- The claim that the evolved activation function is tailored to the architecture is central to the work, but it is not proven. It should be shown with cross-evaluation of activation functions and architectures or removed from the paper.\n- Might be an oversight on my part, but it is not clear how many activation functions were evolved for each architecture (3 are shown in the results table) and how their performance is aggregated (in the results table, it is not). As mainly randomized algorithms, evolutionary methods can converge to very different results over subsequent runs, especially in the presence of highly multimodal, indirectly computed fitness landscapes such as this. The method needs then to be evaluated as a whole, not the single produced results, as the performance eventually depends on the number of reruns (tending to infinity).\n- Paragraph 3.2: while Genetic Algorithms and related Genetic Programming are indeed discrete search algorithms, state-of-the-art Evolution Strategies are excellent continuous optimization algorithms. Please fix the claim accordingly.\n- First paragraph in 3.3: while reading up on ES, please adopt the established $(\\mu,\\lambda)$ and $(\\mu+\\lambda)$ terminology in place of the introduced $P$ and $S$.\n- The code is not published as open source, limiting distribution and reproducibility.\n\n**Final remarks:**\nThe number of papers crossing and integrating SGD (or DL) and EC (or NE) is growing over the years. This is very good news for the field, but makes it important that papers exploring new foundation directions such as this one are incontestable and thoroughly proven. If the paper is accepted to ICLR21, the authors have my congratulations and best wishes; but if not, rather than just resubmitting the same content to another conference, I urge them to expand their work beyond reproach, as this direction (and the proven quality of your experimentation and writing) has the potential for an absolutely top-quality publication.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AdFxyMONYR8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to search for activation functions with regularized evolution, an evolutionary algorithm proposed by Real et al. Various mutations are proposed that allow to investigate a larger search space than prior work. In particular, a mutation is added which adds trainable parameters to the activation function. The discovered activation functions are compared on three different architectures to several state-of-the-art activation functions.\n\nThe authors clearly describe their method. The idea and method are well-motivated and leave no scope for criticism. Experimental results indicate that the discovered activation functions are significantly better than a wide range of alternatives. Many related works are discussed, the authors might consider adding [1] to their discussion.\nCurrently unclear are the contributions of this work. In my understanding the authors claim to use different mutations and a different search space. Given the title, the main contribution seems to be the addition of parameters to the activation function. However, this idea is not new. It is e.g. used for PReLU but also the related work by Ramachandran et al. considers the option to add parameters as a unary operator. The authors attempt to study the influence of adding these parameters to the search space by investigating how the effective learning rate changes during training and at various depths. It would be interesting to see how the method performs without the proposed addition in section 3.2. Do any preliminary results indicate that this is significantly improving the results?\nVery related to this: in the appendix you define the swish activation as x*\\sigma(x). It would be interesting to know whether the authors used this function or the parametric version proposed by the original authors defined as x*\\sigma(\\beta * x).\n\nThe authors investigate whether the activation function generalize across architectures and notice that many do not. It would be interesting to know whether the discovered activation functions generalize to different data sets. The authors claimed that CIFAR-100 is more difficult than CIFAR-10. How do the transferred activation functions compare in this case to the baselines?\n\nThe generalization is only considered on ResNets. How about entirely different architectures such as some without residual connections, DenseNets, NASNet, EfficientNet or MobileNet? Will it also transfer to recurrent neural networks and other sorts of non-image architectures?\n\nConcluding, the paper is well-written and seems to provide a solid empirical evaluation. I think the authors could be more explicit about the novelties presented in this work and why they matter.\n\n\n[1] Jin-Young Kim, Sung-Bae Cho: Evolutionary Optimization of Hyperparameters in Deep Learning Models. CEC 2019: 831-837\n\n**After Authors' Response:**\nIn my understanding the main contribution of this work is the identification of a new search space for activation functions. This might be very similar to earlier ones but the authors convinced me that it is a useful contribution. All my comments were addressed and as far as I saw also the comments of all other reviewers were addressed. For this reason, I increased my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7iSQaTmIjNJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors focus on the task of discovering activation functions via evolutionary methods. The paper is well written, easy to follow and technically sound. \nMy main concern with this paper is that the authors mention the importance of having activation functions learned for particular networks/tasks, yet failed to compare against any of the learnable activation functions that have the same objective.\nSome of those activation functions are, e.g., APL [1], PAU [2], Mixtures of activations [3], and SReLU [4].\n\nAs a positive aspect, I see the benefit of your paper in that the given set of evolutionary operations might achieve some of the previous activation functions. However, your approach has an associated computational cost that is very high, and therefore it is very important to distill what makes it work and raise the following questions:\n\n- Is it possible that the improvements come from the parametrization?.\n- What is the behavior when you do exploration but not allowing parameters?.\n\nOne very interesting aspect of the functions that you learned, is that the form $log(\\sigma(x))$ shows up repeatedly and in some cases including parameters. Mathematically, this is equivalent to $-SoftPlus(-x)$. The general form can be easily achieved by a parametric version $\\alpha SoftPlus(\\beta x)$, which appears often in table 2. Do you think this would be one big take-away message? The swish suggestion of your paper?\nSince the SoftPlus is a soft version of the ReLU, one could also find interesting $\\alpha ReLU (\\beta x))$.\n\nIndeed, your search algorithm is useful, but the results seem to point in the direction of parametric learnable activation functions, therefore a comparison to that family of activations is very important.\n\n\n[1] F. Agostinelli, M. D. Hoffman, P. J. Sadowski, and P. Baldi. Learning activation functions to improve deep neural networks. In Workshop Track Proceedings of the International Conference on Learning Representations, 2015.\n\n[2] A. Molina, P. Schramowski, K. Kersting. Pade Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks. In International Conference on Learning Representations, 2020.\n\n[3] F. Manessi and A. Rozza. Learning combinations of activation functions. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 61–66. IEEE, 2018.\n\n[4] X. Jin, C. Xu, J. Feng, Y. Wei, J. Xiong, and S. Yan. Deep learning with s-shaped rectified linear activation units. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.\n\n[5] Ramachandran, P., Zoph, B. and Le, Q.V., 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the idea of searching parameterized activation functions, in contrast to the previous handcraft or learnable ones. It may be a counterpart of neural architecture search.

Pros:
1. The idea is very interesting.
2. The paper is well written.
3. The experiments show improvements over baseline activation functions.

Cons:
1. The AC fully agreed with Reviewer #4 that the whole literature of learnable activation function is neglected (Reviewer #2 also alluded to this issue). Although the authors added experiments with learnable baseline activation functionss, the literature review on learnable activation function was not included accordingly.
2. Although the idea of searching activation functions is interesting, the AC doubted the necessity. Since the rich literature of learnable activation functions is already there (note that it is more than introducing parameters to handcrafted ones), can we simply learn piecewise linear activation functions with more pieces so that it can approximate complex enough functions? This can be much more easily implemented (can go along with weight training on the standard deep learning platform) and the computation cost will be much lower. Such a comparison is absolutely necessary.
3. The AC was actually worried about the activation functions founded as they may be too complex, so the generalization issue (even numerical stability issue) may be a concern. More thorough testing is necessary (currently only tested on CIFAR-100 and three CNNs; and Reviewers #3 and #2 also concerned about this issue).

Although Reviewer #2 raised his/her score, the final average score is still below threshold. So the AC decided to reject the paper.
Decision: Reject