Efficient Activation Function Optimization through Surrogate Modeling
Authors: Garrett Bingham
Risto Miikkulainen
Venue: NeurIPS 2023 poster (2023)
Keywords: automl
activation function
surrogate modeling
fisher information matrix
eigenvalues
optimization
umap
imagenet

Abstract: Carefully designed activation functions can improve the performance of neural networks in many machine learning tasks.  However, it is difficult for humans to construct optimal activation functions, and current activation function search algorithms are prohibitively expensive.  This paper aims to improve the state of the art through three steps: First, the benchmark datasets Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT were created by training convolutional, residual, and vision transformer architectures from scratch with 2,913 systematically generated activation functions. Second, a characterization of the benchmark space was developed, leading to a new surrogate-based method for optimization. More specifically, the spectrum of the Fisher information matrix associated with the model's predictive distribution at initialization and the activation function's output distribution were found to be highly predictive of performance. Third, the surrogate was used to discover improved activation functions in several real-world tasks, with a surprising finding: a sigmoidal design that outperformed all other activation functions was discovered, challenging the status quo of always using rectifier nonlinearities in deep learning.  Each of these steps is a contribution in its own right; together they serve as a practical and theoretical foundation for further research on activation function optimization.

Efficient Activation Function Optimization through Surrogate Modeling
Garrett Bingham"
Risto Miikkulainen
The University of Texas at Austin
The University of Texas at Austin
and Cognizant Al Labs San Francisco, CA 94105
and Cognizant Al Labs San Francisco, CA 94105 ristoêcs utwxas.e...... edu
garrett.gjb.aa
Abstract
Carefully designed activation functions can improve the performance of neural networks in many machine learning tasks. However, it is difficult for humans construct optimal activation functions, and current activation function search algorithms are prohibitively expensive. This paper aims to improve the state of the art through three steps: First, the benchmark datasets act-Bench-CnM Act-Bench- Bencch-Ressee ResNett and Pett-Bench-Vis were created by training convolutional, residual, and vision transformer architectures from scratch with 2,913 systemati- cally generated activation functions. Second, characterization of the benchmark space was developed, leading to new surrogate-bar based method for optimization. More specifically, the spectrum of the Fisher information matrix associated with the model's predictive distribution initialization and the activation function's s output distribution were found to be highly predictive of performance, Third, the surrogate was used to discover improved activation functions in several real-world tasks, with a surprising finding: sigmoidal design that outperformed all other activation functions was discovered, challenging the status quo always using rectifier nonlinearities in deep learning. Each of these steps is contribution in its own right; together they serve as practical and theoretical foundation for further research activation function optimization.
Introduction
Activation functions important choice in neural network design 2 46) In order realize the benefits good activation functions, researchers often design new functions based on characteristics like smoothness, groundedness, monotonicity, and limit behavior. While these properties have proven useful, humans are ultimately limited by design biases and by the relatively small number of functions they can consider. On the other hand, automated search methods can evaluate thousands of unique functions, and as a result, often discover better activation functions than those designed by humans. However, such approaches do not usually have theoretical justification, and instead focus only on performance. This limitation results in computationally inefficient ad hoc algorithms that may miss good solutions and may not scale large models and datasets.
This paper addresses these drawbacks provide foundation for theory and algorithm development, convolutional, residual, and vision transformer based architectures were trained from scratch with 2,913 different activation functions, resulting in three activation function benchmark datasets: Act-Bench- CNN, Act Set--Bennh-Reeeeeee
data-driven way through three steps. First, in order to
"GB is currently research scientist at Google DeepMind. AQuaSurF code is available at https:// github. ctllb/.coo//ai wtt.cooccccooo wwt.//onnnia.ccoooo ai- llabs/aquauurz and the benchmark datasets are at htthtts://ee thub. com/ cognizant- labs/act- bench
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and
These datasets make it possible to analyze activation function properties at a large scale in order to determine which are most predictive of performance.
The second step was to characterize the activation functions in these benchmark datasets analytically, leading to a surrogate performance measure. Exploratory data analysis revealed two activation func- tion properties that are highly indicative of performance: (1) the spectrum of the Fisher information matrix associated with the model's predictive distribution at initialization, and (2) the activation function's is output distribution. Both sets of features contribute unique information. Both are predictive of performance on their own, but they are most powerful when used in tandem. These features were combined to create a metric space where low-dimensional representation of the activation functions was learned. This space was then used as a surrogate in the search for good activation functions.
In the third step, this surrogate was evaluated experimentally, first by verifying that it can discover known good functions in the benchmark datasets efficiently and reliably, and second by demonstrating that it can discover improved activation functions in new tasks involving different datasets, search spaces, and architectures. The representation turned out to be so powerful that an out-of-the-box regression algorithm was able to search it effectively. This algorithm improved performance on various tasks, and also discovered sigmoidal activation function that outperformed all baselines, a surprising discovery that challenges the common practice of using ReLU and its variants. The approach, called AQuaSurF (Activation Quality with a Surrogate Function), orders of magnitude more efficient than past work. Indeed, whereas previous approaches evaluated hundreds or thousands of activation functions, AQuaSurF requires only tens of evaluations in order to discover functions that outperform wide range of baseline activation functions in each context. Code implementing the AQuaSurF algorithm available at https: //gi com/ wwtp///atnntt..c.oo ai- aabsabs/ waa.aaaaaaaauurraaaa
Prior research on activation function optimization and Fisher information matrices is reviewed in Section A This work extends it in three ways. First, the benchmark collections are made wtt.ccocccccoc http///onniiaaaiiioo wwwaaaaaaaaaatabaattaaaaeeaaaaee
available at hhttp:://iithi for further research on activation function optimization. Second, the low-dimensional repre- sentation of the Fisher information matrix makes it practical surrogate measure, making it possible to apply it to not only activation function design, but potentially also other ap- plications in the future. Third, the already-discovere functions can be used immediately to improve performance in image processing tasks, and potentially in other tasks in the future.
providing a foundation
Act-Bench-CNN
Act-Bench-ResNet
Act-Bench-VIT
Namber of functiions
2020
Kept
12000
2 Activation Function Benchmarks
100
As the first step. three activation function bench- mark datasets are introduced: Act-Bench- Act-Bench- Resee, and Act-Bench- Baschh-Vi1 VVT. Each dataset contains training results for 2,913 unique activation functions when paired with different architectures and tasks: All-CNN-C on CIFAR- 10, ResNet-56 on CIFAR-10, and MobileViTv2-005 on Imagenette 22 24 31 [1
0.0
022 04 5.6
0. 1.0
0. 02 04 066 Valdittion Accuracy
0/3 02 04
Figure 1: Distribution of validation accuracies with 2,913 unique actívation functions from the three benchmark datasets. Many activation functions result in failed training (indicated by the chance ac- curacy of 0.1), suggesting that searching for activa- tion functions is challenging problem. However, most of these functions have invalid FIM eigenval- ues, and can thus be filtered out effectively.
51. These functions were created using the main three-node computation graph from PAN-
GAEA 5 Details are in Appendix B
Figure T shows the distribution of validation ac- curacies in these datasets. In all three datasets, the distribution is highly skewed towards func- tions that result in failed training. The plots sug-
ArtSunchhleggu
ALT- Mi---ench-ATT
8.-
0.1-
gest that it is difficult to design good activation functions, and explain why existing methods are computationally expensive. Notwithstanding this difficulty, the histograms show that many unique functions do achieve good performance. Thus, searching for new activation functions is worthwhile task that requires smart approach.
0.3 Act-teench-CNR
05 07 0.0 0.1 0.5 5.7 09
03
0. &
Act-Bench-Chhh
Ast-Bench-ReswWe
Figure 2: Distribution of validation accuracies across the benchmark datasets. Each point rep- resents a unique activation function's 'isions' performance on two of the three datasets. Some functions per- form well on all tasks, while others are specialized.

---

Figure 2 shows the same data as Figure 1 but with scatter plots that show how performance varies across different tasks. All three plots contain linearly correlated clusters of points in the upper right comnrr, suggesting that there are modifications to activation functions that make them more powerful across tasks. However, the best results come from discovering functions specialized to individual tasks, indicated by the clusters of points in the upper left and lower right corners.
The three benchmark datasets form a foundation for developing and evaluating methods for automated activation function design. In the next two sections, they are used to develop surrogate performance metric, making it possible to scale up activation function optimization to large networks and datasets.
3 Features and Distance Metries
To make efficient search for activation functions possible, the surrogate space needs to be low- dimensional, represent informative features, and have an appropriate distance metric. In the second step, an approach is developed based on (1) the eigenvalues of the Fisher information matrix and (2) the outputs of the activation function. This section motivates each feature type and develops metric for computing distances between activation functions. They form a surrogate in the next section.
FIM Eigenvalues The Fisher information matrix (FIM) is an important concept in characterizing neural network models. Viewed from various perspectives, the FIM determines a neural network's capacity for learning, ability generalize, the robustness of the network to small perturbations of its parameters, and the geometry of the loss function near the global minimum [6 21 27 33
Consider neural network f with weights O. Given inputs drawn from a training distribution Qx, the network defines the conditional distribution Ryj f(x:0). The FIM associated with this model is
F
J0 X-QQ. (y))(
Voly,,
where Cly, z) is the loss function representing the negative log-likelihood associated with Ry (x)))
The FIM has |0] eigenvalues. The distribution of eigenvalues can be represented by binning the eigenvalues into an m-bucket histogram, and this mn-dimensional vector serves as a computational characterization of the network. To calculate the FIM and its eigenvalues, this paper uses the K-FAC approach 20 38 Full details are in Appendix
Different activation functions induce different FIM eigenvalues for given neural network. They can be calculated at initialization without training; they can thus serve as a low-dimensional feature vector representation of the activation function. The FIM eigenvalues are immediately useful for filtering out poor activation functions; if they are invalid, the activation function is likely to fail in training (Figure D] However, in order to use them as a surrogate, a distance metric needs to be defined.
Given a neural network architecture f. let fa and fu be two instantiations with different activation functions & and v/. Let ii and in represent the distributions of eigenvalues corresponding to the weights in layer 1 of neural networks fo and fa. respectively, and let wy be the number of weights in layer of the networks. The distance between fo and fo is then computed as a weighted layer-wise sum of 1- Wassersstei distances
fw) Wi (14,)) w///w/
With this distance metric, the FIM eigenvalue vector representations encode low-dimensional embedding space for activation functions, making efficient search possible. Because the FIM eigenvalues depend on several factors (Equation [ including the activation function é, network architecture f. data distribution Q, and loss function L, they are susceptible to more potential sources of noise. Fortunately, incorporating activation function outputs helps to compensate for this noise.
Activation Function Outputs The shape of an activation function 1p can be described by a vector of sample values (x)). If the network's 's weights are appropriately initialized, the input activations to its neurons are initially distributed as (O, 1) 4. Therefore, the sampling I ~ NN(0, 1) provides an n-dimensionaa feature vector that represents the expected use of the activation function initialization. distance metric in this feature vector space can be defined naturally as the Euclidean distance
d(fe, fv) =
(22n)) x ~~~00 (0, 1).
(3)
3

---

Functions with similar shapes will have a small distance between them, while those with different shapes will have a large distance. Because these output feature vectors depend only on the activation function, they are reliable and inexpensive to compute. Most importandly together with the FIM eigenvalues, they constitute powerful surrogate search space, demonstrated in the next section.
4 Using the Features as a Surrogate
relu(o)
identity
In this section, the UMAP dimensionality reduc- tion technique is used to visualize the FIM and output features across the benchmark datasets. This visualization leads to combined surrogate space that can be used to accelerate the search for good activation functions.
Visualization with UMAP The features de- veloped above can be visualized using the UMAP algorithm [40]. UMAP is a dimension reduction approach similar to t-SNE, but is bet- ter at scaling to large sample sizes and preserv- ing global structure [55]. As first demonstra- tion, Figure shows a 2D representation of the 2,913 activation functions in the benchmark datasets. Each function was represented 80-dimensional vector of output values. Interpo- lating between embedded points confirms that UMAP learns good underlying representation.
UMAP also used to project the activation functions to nine two-dimensional spaces ac- cording to the distance metrics in Equations 2 and 3 In Figure 4 each column represents different benchmark dataset (Act-Bench-CNI) Act-Bench- Nesseet, or Act-Benchh-iiy and each row different distance metric (FIM eigen- values with m = [|] /100]. activation function outputs with 1, 1,000, or both). The plots only show activation functions that were not filtered out. Each point represents a unique function, colored according to its validation accuracy on the benchmark task. Although the performance of each activation function is already known, this information was not given UMAP; the embeddings are entirely unsupervised.
Figure 3: UMAP embedding of the 2,913 activa- tion functions in the benchmark datasets. Each point stands for unique activation function, repre- sented by an 80-dimensional output feature vector. The embedding locations of four common activa- tion functions are labeled. The black x's mark coordinates interpolating between these four func- tions, and the grid of plots on the bottom shows reconstructed activation functions at each of these points. UMAP interpolates smoothly between dif- ferent kinds of functions, suggesting that is good approach for learning low-dimensional repre- sentations activation functions.
Thus, Figure illlsttrates how predictive each feature type is of activation function performance in each dataset. The next subsections evaluate each feature type in this role in detail, and show that utilizing both features provides better results than either feature alone. Details in Appendix D
FIM Eigenvalues The first row of Figure the 2D UMAP embeddings of the FIM eigenvalue vectors associated with each activation function. There are clusters in these plots where the points share similar colors, indicating distinct activation functions with similar FIM cigenvalues. Such functions induce similar training dynamics in the neural network and lead to similar performance. On the other hand, some clusters contain activation functions with wide range of performances, and some points do not belong to any cluster at all. Overall, the plots suggest that FIM eigenvalues are a useful predictor of performance, but that incorporating additional features could lead to better results.
The middle row of Figure shows the 2D UMAP embeddings of the output vectors associated with each activation function. Points are close to each other in this space if the corresponding activation functions have similar shapes. These plots are demonstrably
Activation Function Outputs

---

more informative than the plots based on the FIM eigenvalues in three ways. First, the purple points are better separated from the others. This separation means that activation functions that fail (those achieving 0.1 chance accuracy) are better separated from those that do well. Second, most points' immediate neighbors have similar colors. This similarity means that activation functions with similar shapes lead to similar accuracy, and analyzing activation function outputs on their own is more informative than analyzing the FIM eigenvalues. Third, the plots include multiple regions where there are me-dimensional manifolds that exhibit smooth transitions in accuracy, from purple to blue to green to yellow. Thus, not only does UMAP successfully embed similar activation functions near each other, but it also is able to organize the activation functions in meaningful way.
There one drawback to this approach: the per- formant activation functions (those represented by yellow dots) are often in distinet clusters. This dispersion means that search algorithm would have to explore multiple areas of the search space in order to find all of the best func- tions. As the next subsection suggests, this issue can be alleviated by utilizing both FIM eigenval- ues and activation function outputs.
F' Elperaaier
tetivstos funde on Ourpue
-06
0.
-04
Combining Eigenvalues & Outputs The UMAP algorithm uses an intermediate fuzzy topological representation to represent relation- ships between data points, similar to a neigh- borhood graph. This property makes it possible to combine multiple sources of data by taking intersections or unions of the representations in order to yield new representations [40] The bottom row of Figure Aigureaatilizes utilizes both FIM eigen- values and activation function outputs by taking the union of the two representations. Thus, ac- tivation functions are embedded close to each other in this space if they have similar shapes, if they induce similar FIM eigenvalues, or both.
Equnaalees a Curners
-03
EAUSAl
-ELUUR)
Lamm
abulk)
aboox
Figure 4: UMAP embeddings of activation func- tions for each dataset (column) and feature type (row). Each point represents a unique activation function; the points are colored by validation ac- curacy on the given dataset. The colored triangles identify the locations of six well-known activation functions. The areas of similar performance are more continuous in the bottom row; that is, us- ing both FIM eigenvalues and activation function outputs provides a better low- oow-dimessional repre- sentation than either feature alone.
The bottom row of Figure 4 shows the bene- fits of combining the two features. Unlike the activation function output plots, which contain multiple clusters of high-performing activation functions in different locations in the embedding space, the combined UMAP model embeds all of the best activation functions in similar regions. The combined UMAP model also places poor activation functions (purple points) in the edge of the embedding space, and brings good functions (yellow points) to the center. Thus, the embedding space is more convex, and therefore easier to optimize.
In general, activation funetions with similar shapes lead to similar performances, and those with different shapes often produce different results. This property is why the middle row of Figure appears locally smooth. However, in some cases the shape of the activation function does not tell the whole story, and additional information is needed to ascertain its performance.
For example, the colored triangles in Figure S identify the location of six activation functions in the low-dimensional space. In the activation function output space (middle row), all of these functions are mapped to different regions of the space. The points spread apart because an activation function and its negative have very different shapes, i.c. their output will be different for every nonzero input (Figure 5). In contrast, in the FIM eigenvalue space (top row), the points for these pairs of functions overlap because the FIM eigenvalues are comparable (Figure 5). Indeed, assuming the weights are initialized from distribution symmetric about zero, negating an activation function does not change the training dynamics of a neural network, and they are functionally equivalent.
This issue complicates the search process in two ways. First, good activation functions are mapped to different regions of the embedding space, and so a search algorithm must explore multiple areas in order to find the best function. Second, distinct regions of the space may contain redundant
5

---

information: if (z) is known to be good activation function, it is not helpful to spend compute resources evaluating ELU(z) only to discover that it achieves the same performance.
Negating an activation function is a clear example activation function, but does not affect the training of neural network. More broadly, it is likely that there exist activation functions that differ in other ways (besides just negation), but that still induce similar training dynamics in neural networks. Fortunately, utilizing FIM eigenvalues and activation function outputs together provides enough information to tease out these relationships. FIM eigenvalues take into account the activation function, the neural network architecture, the loss function, and the data distribution. The eigenvalues are more meaningful features than activation function outputs, which only depend on the shape of the function. However, as Figure 4 shows, the FIM eigenvalues are noisier features, while the activation function outputs are quite reliable. Thus, utilizing both features a natural way to combine their strengths and address their weaknesses.
modification that changes the shape of the
All CNN.C
Constructing Surrogate These observa- tions suggest an opportunity for an effective sur- rogate measure: The UMAP coordinates in the bottom row of Figure g have the information needed to predict how well an activation func- tion will perform. They capture the essence of the m and n dimensional feature vectors, and distill it into a 2D representation that can be com- puted efficiently and used to guide the search for good functions. As the third step in this research, the next two sections evaluate this process ex- perimentally, demonstrating that is efficient and reliable, and that it scales to new and chal- lenging datasets and search spaces.
0.3m
CLUX 5.09
anttol 0.12
0.2
-50
-28
-14
-22
-10
ResNet-56
Euak 0.91
C1ub 0.01
0 0.55
Sattist 2.86
stslal: 035
atdjc) 0:23
0.0
-12
-20
-1c
-12
MobileviTv--0.5
Sattet 0.84
Sattixt:
atolet: 0.75
0.021
404 0.24
0.002-
5 Searching on the Benchmarks
logiA)
banhixi
tanhix)
absle)
Searching for activation functions typically re- quires training neural network from scratch in order to evaluate each candidate function fully, which is often computationally expensive. With the benchmark datasets, the results are already precomputed. This information makes it pos- sible to experiment with different search algo- rithms and conduct repeated trials to understand the statistical significance of the results. These results serve inform both algorithm design and feature selection, as shown in this section.
Figure 5: FIM eigenvalue distributions for differ- architectures and activation functions. The legends show the activation function and the cor- responding validation accuracy in different tasks. Although negating an activation function changes its shape, it does not substantially change its behav- ior nor its performance. FIM eigenvalues capture this relationship between activation functions. The eigenvalues are thus useful for finding activation functions that appear different but in fact behave similarly, and these discoveries in turn improve the efficiency of activation function search.
Setup Three algorithms were evaluated: weighted k-nearest regression with k (KNR), random forest regression (RFR), and support vector regression (SVR). Gaussian Process Regression (GPR) was also evaluated but found to be inconsistent in preliminary experiments (Appendix FF). Random search (RS) was included as a baseline comparison; did not utilize the FIM eigenvalue filtering mechanism. The algorithms were used out of the box with default hyperparameters from the scikit-learn package [47]. They were provided different activation function features in order to understand their potential to predict perfor- mance. The features included FIM eigenvalues, activation function outputs, or both. The features were preprocessed and embedded in a two-dimensional space by UMAP. These representations are visualized in Figure 4 the coordinates of each point correspond exactly to the information given to the regression algorithms.
The ReLU activation function is ubiquitous in machine learning. For many neural network archi- tectures, the performance with ReLU is already known 2 45 46), which makes it good starting point for search. For this reason, the search algorithms began by evaluating ReLU and seven other

---

randomly chosen activation functions. In general, such evaluation requires training from scratch, but with the benchmark datasets, it requires only looking up the precomputed results. The algorithms then used the validation accuracy of these eight functions to predict the performance of all unevalu- ated functions in the dataset. The activation function with the highest predicted accuracy was then evaluated. The performance of this new function was then added to the list of known results, and this process continued until 100 activation functions had been evaluated. Each experiment comprising a different search algorithm, activation function feature set, and benchmark dataset was repeated 100 times, Full details are Appendix
Act-Bench-CNN Act-Bench-Restee
Act-Bench-VIT
Results Figure 6 shows the results of the searches. Importantly, the curves do not depict just one search trial. Instead, they represent the average performance aggregated from 100 in- dependent runs, which is made possible by the benchmark datasets. As indicated by the shaded confidence intervals, the results are reliable and are not simply due to chance.
Do0 0 bee cap boo
0.20
0.9000
so
0.922
0-3.0
number of conclusions can be drawn from Figure First, all search algorithms, even ran- dom search, reliably discover activation func- tions that outperform ReLU. This finding is sup- ported by previous work (reviewed in Section A): Although ReLU is a good activation func- tion that performs well in many different tasks, better performance can be achieved with novel activation functions. Therefore, continuing to use ReLU in the future is unlikely to lead to best results; The choice of the activation function should be an important part of the design, sim- ilar to the choice of the network architecture or the selection of its hyperparameters.
oage 0,890
0.918
50
300
30
300
Evaluated
.... KNR (FIM Eigenvauue)
KNR (Elgonvalues & Ourputs)
RFR IFIM Eigenvaluss
RFR (Elgenvalues Outputsi
(TIM Eigenvalues)
(Bierwaaues Outputs!
KOUR (Punction Butputst
Random Search
RPA (Fanciion Outputs)
Benchmark Optimam
SVX (function Outputs)
Accuracy with Rulli
Figure 6: Search results on the three benchmark datasets. Each curve represents different search algorithm (KNR, RFR, or SVR) utilizing dif- ferent UMAP feature (FIM eigenvalues, function outputs, or both; these features are visualized in Figure 4. The curves represent the validation ac- curacy of the best activation function discovered so far, averaged across 100 independent trials, and the shaded areas show the 95% confidence inter- val around the mean. In cases, regression with UMAP features outperforms random search, and searching with both eigenvalues and outputs out- performs searching with either feature alone. Of the three regression algorithms, KNR performs the best, rapidly surpassing ReLU and quickly dis- covering near-optimal activation functions in all benchmark tasks. Thus, the features make it possi- ble to find good activation functions efficiently and reliably even with off-the-shelf search methods; the benchmark datasets make it possible to demon- strate these conclusions with statistical reliability.
Second, all regression algorithms outperform random search. This finding holds across the three types of activation function features and across the three benchmark datasets. The FIM eigenvalues and activation function outputs are thus important in predicting performance of ac- tivation functions.
Third, regression algorithms trained on both FIM eigenvalues and activation function outputs outperform algorithms trained on just eigenval- ues or outputs alone. This result is consistent across the regression algorithms and benchmark datasets. It suggests that the FIM eigenvalues and activation function outputs contribute com- plimentary pieces of information. The finding quantitatively reinforces the qualitative visual- ization in Figure 4 FIM cigenvalues are useful for matching activation functions that induce similar training dynamics in neural networks, activation function outputs enable a low-dimensional repre- sentation where search is more practical, and combining the two features results in a problem that is more convex and easier to optimize.
Fourth, the searches are efficient. Previous approaches require hundreds or thousands of evaluations 49]. In contrast, this paper leverages FIM eigenvalues and activation function outputs to reduce the problem to simple two-dimensional regression; the features are powerful enough that out-of-the-box regression algorithms can discover good functions with only tens of evaluations. This efficiency makes possible to search for better functions directly on large datasets such ImageNet [[]] demonstrated next.
to discover good activation functions [3] 6
7

---

Searching with New Settings
AII-CNN-C
ResNet-56
MobileViTv2-0.5
CIFAR-10D
CIFAR-100
ImageNet
The experiments in Section usedd precomputed datasets and search spaces to demonstrate that UMAP embeddings are predictive of activation function performance, and that KNR can find good functions based them. To verify that these conclusions extend beyond the benchmark tasks, this section contains three experiments demonstrating that AQuaSurF scales up to more challenging datasets and search spaces, that the activation functions can be transferred to other tasks, and that AQuaSurF extends to new archi- tectures and baseline functions.
0.08 0:0
ido
100
s0
10o
Functions Evaluated
sigmoiste)
relutat
sofiphante)
Lanhixl
solatk)
softsignee)
Figure 7: Progress of activation function searches. Each point represents the validation accuracy with a unique activation function, and the solid line indicates the performance of the best activation function found so far. AQuaSurF discovers new activation functions that outperform all baseline functions in every case.
Scaling Up the Datasets and Search Space In the first experiment, the tasks involve larger and more challenging datasets: All-CNN-C on CIFAR-100 -10, ResNet-56 on CIFAR-100 100, and Mobile ViT52-0.5 on ImageNet. Additionally, a larger space with 425,896 unique activation functions was searched, based on four-node computation graphs (Appendix B}. This space is large, diverse, and not precomputed, putting the conclusions from the benchmark experiments test in production setting.
Based on the benchmark results, KNR with A 3 was used as the search algorithm. The searches all began by evaluating the same eight existing activation functions: ELU, ReLU, SELU, sigmoid, Softplus, Softsign, Swish, and tanh. From this starting point, eight workers operated in parallel evaluating the functions with the highest predicted performance. Details are in Appendix
Figure 7 shows that all three searches find improved activation functions over time, and Figure I0 in Appendix B shows how the searches navigate the search space. In every experiment, new activation functions were discovered that outperform all baseline functions. Although the search space is large, the searches are efficient, requiring only tens of evaluations to improve performance. Impressively, the search with ResNet-56 on CIFAR-100 produced an activation function that outperformed all baselines just the second evaluation.
Table D shows the final results from AQuaSurF. The results reinforce the fact that substantial gains can be obtained when using better activation functions than the default ReLU, and especially those optimized specifically for the task.
Transferring to a New Task In the second experiment, the best activation functions from Table ! were transferred to new task: ResNet-50 on ImageNet. As demonstrated in Table 2 good functions can be discovered efficiently in smaller tasks and then used to improve performance in larger ones.
New Architectures and Baseline Functions In the third experiment, the CoAtNet architecture was trained on Imagenette [TO]. As hybrid convolution and attention architecture, CoAtNet presents new challenge for A AQuaSurF. The activation functions ELiSH, GELU, HardSigmoid, Leaky ReLU, and Mish [3] 23] 37 [43 were added the original set of baseline functions (Table I as well as to the set of unary operators, forming a new search space for AQuaSurF to explore (Appendix B).
The results show that AQuaSurF extends to architectures and baseline functions not considered in the benchmark tasks (Table (3). AQuaSurF discovered multiple activation functions that substantially outperform all baseline functions. Although the extended list of baseline functions presents more challenging task, it also provides the surrogate function more information that it uses for performance prediction, resulting in the discovery of even better functions.
7 Understanding the Discoveries
Aside from the raw performance improvements afforded by AQuaSurF, the experiments on the new settings are particularly interesting because they illustrate both the process of refining existing activation functions and the process of discovering novel designs.
8

---

Table 1: Accuracy with different activation functions. The CIFAR-100 results show the median test accuracy from three runs, and the ImageNet results show the validation accuracy from a single run. AQuaSurF discovers novel activation functions that outperform all baselines in every case. This result demonstrates both that good functions matter, and the power of optimizing them to the task.
AlI-CNN-C CIFAR-100
Re CIFAR-100
2- on ImageNet
(aniiimooidHHnnSiigmiidd BLU(())
0.6990
Swish 2r)
0.7469
-Z)) HardSigmoi((s)
0.6396
(Softsign(()) ELU(z)
0.6950
(ELUUsinhfe pritaniss 1))
0,7458
BLU(Swishh(- 2])
0.6394
Swish(e;//FLUUp
0.6931
2. =cfc(ELU(=))
0.7419
Swish(a) erfetbessel (0efa))
0.6336
FLU
0,6312
ELU
0.7411
ELU
0.6233
ReI.U
0.6897 ReLU
0.7348
ReLU
0.6139
SELU
0,0100 SELU
0.6967
SELU
0.60%6
sigmoid
0,0100
sigmoid
0.5766
sigmoid
0.5032
Softplus
0.6563
Seftplus
0.7397 Softplus
0
Sofsiga
0,2570
Softiign
0.6624 Softsign
0.5710
Swish
0.6913
Swish
0.7401
Swish
0.6383
tunh
0.3757
tanh
0.6754
tanh
0.6098
Refinement and Novelty Figure g shows dif- ferent activation functions discovered during the searches. (Plots of all 100 functions evaluated in each search are included in Figures [3 in Appendix B Visually, many the best functions (shown in 8 are similar existing functions like ELU and Swish, with subtle changes in their saturation value, the slope of the positive segment, and the width and depth of the nega- tive bump. This result is not surprising since these functions formed the starting point for the search. Indeed, after a few good functions were found, much of the search process focused on refining their design (Figure in Appendix [B). Although these refinements appear small, they were not known ahead of time and they are sig- nificant, as evidenced by the final results (Tables
Table 2: ResNet-50 top-1 accuracy on ImageNet. Results are the median of three runs. The best acti- vation functions discovered in the searches (Table I successfully transfer to this new task, with eight of the nine functions outperforming ReLU
-I (2)) HardSigmoid(2) SwiSh())/SELL((1
0.7776
0.7771
Swish (r) (eeesbbeleiee(()
0.7755
o (Softsign(z)) ELU)) (LU(x)
0.7734
SELU(sinh(
1))
0.7719
HarUSinnoid(HaaSSymiidd ELU(()
0.7718
FLU((Sish((zz)
0.7679
Swish(-2x)
0.7664
I erfc(ELU(())
0.7635
ReI.((z)
0.7660
13.
However. some of the best discovered activation functions, in- cluding the top function for the CoAtNet experiment, employ properties uncommon among the usual deep learning activa- tion functions (Figure Bp): Some of them have discontinuous derivatives at x 0: some do not saturate, but diverge as + =oo; some of them contain positive bumps (in contrast to e.g. Swish, which features a negative bump). Many of these functions performed comparably to the existing best functions, and all of them outperformed ReL.U. In the future, these designs may provide comprehensive foundation for discovering better activation functions for specific new tasks.
Table 3: CoAtNet validation ac-
curacy on Imagenette. AQuaSurF finds novel functions that outper- form all baselines.
erfe (r)ftplllllus min (Sottllus -z} (asiinhEELUSSiiinees
0.8907
0.8861
0.8828
ELiSH ELU
0,1000
0.8629
GELU
0.8841
HardSigmoid
0,8487
Leaky ReLU
0,8815
Together, the plots show that AQuaSurF is capable of both exploitation (Figure (1) and exploration (Figure (8). In the future, it will be interesting to explore tradeoffs between these concepts. A more comprehensive discussion of this and other future research directions is included in Appendix F
Mish
0.8762
ReLU
0.8772
SELU
0.8194
sigmoid
0.8586
Softplus
0.8678
Softsign
0.8530
Swish
0.8736
Discovering a Hybrid Rectiffer-Sigmoidal Activation Func- tion In the past, sigmoidal nonlinearities like sigmoid and tanh were often used because they saturate and thus prevent exploding signals. However, currently these functions are usually discarded in favor of rectifier nonlinearities like ReLU and its variants as these functions give better performance on modern deep learning benchmarks [2]. Indeed, in Tables and 3 sigmoid, tanh, HardSigmoid, and Softsign all perform relatively poorly. It is therefore surprising to see that the very best function discovered in the CoAtNet experiment, erfe (Softplus(z))) (bottom left of Figure 8)), is sigmoidal in shape.
tanh
0,8415

---

Why does this function perform so well? As
Arrge
shown in Figure the function saturates to as -x and to O as + 0o, and has an ap- proximately linear region in between. The regions of the function that the neural network actually utilizes in its feedforward pass are superimposed as histograms this plot. Interestingly, at initializa- tion, the network does not use the saturation regimes. The inputs to the func- tion are tightly concen- trated around x for all instances of the activa- tion function throughout the network. As training progresses, the network makes use of larger do- main of the activation function, and by the time training has concluded the network uses the saturation regimes approximately < -4 and > 1.
Figure 8: Sample activation functions discovered with AQuaSurF the four searches in Section 6 "HS" stands for HardSigmoid. (a) The top three functions (columns) discovered these functions refined versions of existing activation functions like ELU and Swish. (b) Selected novel activation functions. All of these functions outperformed ReLU and are distinct from existing activation functions. Such designs may serve as a foundation for further improve- and specialization in settings.
each search (rows). Many of
Thus, Figure shows that erfc(Softplus(z)) serves dual purpose. At initialization. it per- forms like a rectifier nonlinearity, but by the end of training, it acts like sigmoidal nonlinearity. This discovery challenges conventional wisdom about activation function design. It shows that neural networks use activation functions in dif- ferent ways in the different stages of training, and suggests that sigmoidal designs may play an important role after all.
At Initialization
1.0
After Training effc[Softpluss0)
0.8
0.6
0.4
Conclusion
This paper introduced three benchmark datasets, Act-Bench-CNA.
BBench-Resleee and Act-Bench-ViT, to support research on acti- vation function optimization. Experiments with these datasets showed that FIM eigenvalues and activation function outputs, and their low- dimensional UMAP embeddings, predict ac- tivation function performance accurately, and can thus be used as a surrogate for finding bet- ter functions, even with out-of-the-box regres- sion algorithms. These conclusions extended from the benchmark datasets challenging real- world tasks, where better functions were discov- ered with a variety of datasets, search spaces, and architectures. AQuaSurF also discovered highly performant sigmoidal activation function, challenging the conventional wisdom of using ReI.Uliike functions exclusively in deep learning. The study reinforces the idea that activation function design an important part of deep learning, and shows AQuaSurF is an efficient and flexible mechanism for doing it.
Figure 9: The best discovered function in the CoAt- Net experiment, erfc (eff((Solllls tion by the network. The red curve shows the acti- vation function itself, and the two histograms show the distributions of inputs the activation func- tion at initialization and after training, aggregated across all instances the activation function the entire network. The network uses the function like rectifier initialization and like sigmoidal acti- vation function after training. This result suggests that sigmoidal designs may be powerful after all, thus challenging the conventional wisdom.
and its utiliza-
10

---

References
[1] F. Agostinelli, M. Hoffmann P. Sadowski, and P. Baldi. Learning activation functions improve deep neural networks. arXiv:1412.6830, 2014.
A. Apicella, F. Donnarumma, F. Isgrò, and R. Prevete. survey on modern trainable activation functions. Neural Networks, 2021.
M. Basirat and M. Roth. The quest for the golden activation function. 1808 2018.
[4] G. Bingham and R. Mikkuulaienn, Autoinit: Analytic preservng weight initialization for neural networks. arXiv preprint arXiv:2109.08955 2021.
G. Bingham and R. Mikkulainen. Discovering parametric activation functions. Neural Networks, 148: 48-65, 2022.
[6] G. Bingham, W. Macke, and Mikkullanen. Evolutionary optimization of deep learning activation functions, Genetic and Evolutionary Computation Conference (GECCO '20), July 8-12, 2020, Cancúá, Mexico, 2020.
[7] D.-A Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). CoRR, abs/151 1.07289, 2015.
[8] E. D. Cubuk, B. Zoph, D. Mane. V. Vasudevan, and Q. V. Le. AutoAugment: Learning augmentation policies from data, arXiv: 1805.0951, 2018
[9] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition workshops, pages 702-703, 2020.
[10] Z. Dai, H. Liu, Q. V. Lc, and M. Tan. CoAtNet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3999 3377, 2021.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.F.. Li. ImageNet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee. 2009.
[12] S. Elfwing. E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approxi- mation in reinforcement learning. Neural Networks, 107:3-11,2014
[13] T. Elsken, H. Metzen, and F. Hutter. Neural architecture search: A survey, Journal of Machine Learning Research, 10555)1-21,, 2019.
[14] A. F. Emery and A. V. Nenarokomov. Optimal experiment design. Measurement Science and Technology. (6)):664 1998.
[15] Y. Furusho and K. Ikeda. Effects of skip-connection in resnet and batch-normalizztion on Fisher information matrix. In INNS Big Data and Deep Learning Conference, pages 341-348. Springer, 2019.
[16] Y. Furusho and K. Ikeda. Theoretical analysis of skip connections and batch normalization from gener- alization and optimization perspectives. APSIPA Transactions on Signal and Information Processing, 9, 2020.
[17] S. Gonzalez and R. Mikkulainen, Evolving loss functions with multivariate Taylor polynomial parameteri- zations. arXiv:2002.000 00059, 2020.
[18] S. Gonzalez and R. Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. 2020 IEEE Congress on Evolutionary Computation (CEC), pages 1-8, 2020. doi: 10. SWW./CCECE86602220011155555
[19] Goyal, R. Goyal, and B. Lall. Learning activation functions: A new paradigm of understanding neural networks. arXiv: 1906.0959, 2019.
[20] R. Grosse and J. Martens. A Kronecker-factoree approximate Fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573-582, PMLR, 2016.
[21] T. Hayase and R. Karakida. The spectrum of Fisher information of deep networks achieving dynamical isometry. In International Conference on Artificial Intelligence and Statistics, pages 334-342, PMLR, 2021.

---

[22] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference compuler vision, pages 630-645. Springer, 2016.
[23] D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs),
2016.
[24] J. Howard. Imagenette. httpp:://githus http::////hstoooooooo//aaaaaaa/s
2019. Accessed: 2022-07-27.
[25] L.. Huang, J. Qin, L.. Liu, F. Zhu, and I., Shao. Layer-wise conditioning analysis in exploring the learning dynamics of DNNs. In European Conference Computer Vision, pages 384-401. Springer, 2020.
[26] S. loffc and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning, pages 448-456, 2015.
[27] S. Jastrzebski, D. Arpit. O. Astrand, G. B. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, and K. J. Geras. Catastrophic Fisher explosion: Early phase Fisher matrix impacts generalization. In International Conference on Machine Learning, pages 4772-4784. PMLR, 2021.
[28] R. Karakida, S. Akaho, and S.-i. Amari. Universal statistics of Fisher information in deep neural networks: Mean field approach. 1032- 1041. PMLR, 2019.
22nd International Conference on Artificial Intelligence and Statistics, pages
[29] R. Karakida, S. Akaho, and S.-i. Amari. Pathological spectra Fisher information metric and its variants in deep neural networks. Neural Computation, 33(8):2274-2377, 2021.
[30] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normaiiiing neural networks. In Advances in neural information processing systems, pages 971 971-980, 2017.
[31] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[32] J. Lehman and K. O. Stanley. Ahandoning objectives: Evolution through the search for novelty alone. Evolationary computation, 19(2):189-25 2221119992222,22 2011.
[33] T. Liang, T. Poggio, A. Rakhlin, and J. Stokes. Fisher-Rao metric, geometry, and complexity of neural networks. In The 22nd international conference artificial intelligence and statistics, pages 888-896. PMLR, 2019.
[34] Z. Liao, T. Drummond, I, Reid, and G. Carneiro. Approximate Fisher information matrix characterize the training of deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 42 (1):15-26, 26, 2018.
[35] H. Liu, A. Brock, K. Simonyan, and Q V. Le. Evolving normalization layers. arX29:2000.002677 2020.
[36] Loshchilov
Hutter, Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101
2017.
[37] A. Maas, A. Y. Hanmun, and Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th international conference on machine learning (ICMI,-13, page 3, 2013.
[38] Martens and R. Grosse. Optimizing neural networks with Kronecker-factonn approximate curvature, In International conference on machine learning, pages 2408 2417. PMLR, 2015.
[39] J. Martens, J. Ba. and M. Johnson. Cronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018.
[40] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint a8X2 0...0.0..426,... 2018.
[41] S. Mchta and M. Rastegari. Separable self-attention for mobile vision transformers. arXiv preprint arXiv:2206.0266. 2022.
[42] J. Mellor. J. Turner, A. Storkey, and E, J. Crowley. Neural architecture search without training. 07Xiv200060046477 2020.
[43] D. Misra. Mish: A self regularized non-monotonic neural activation function. arXiv:1908.08633 2019.
[44] A. Molina, P. Schramowski, and K. Kersting, Padé activation units: End-to-end learning of flexible activation functions in deep networks. 19Xiv:107.06777 2019.
12

---

[45] V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedinggs of the 27th international conference machine learning (ICML-10), pages 807-814, 2010.
[46] C. Nwankpa, W. Ijomah, A. Gachagan, and S. Marshall, Activation functions: Comparison of trends in practice and research for deep learning, arXiv:1811.0337. 2018.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer,
R. Weiss, V. Dubourg, al. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12:2825 2830, 2011.
[48] J. Pennington and P. Worah. The spectrum of the Fisher information matrix of single-hidden-laye neural network. In NeurlPS, pages 5415-5424, 2018.
[49] P. Ramachandran, B. Zoph, and Q V. Le. Searching for activation functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3. 2018. Workshop Track Proceedings, 2018.
[S0] Y. Shen, Y. Li, J. Zheng, W. Zhang, Yao, J. Li, Yang. J. Liu, and B. Cui. Proxybo: Accelerating neural architecture search via Bayesian optimization with zero-cost proxies. arXiv preprini arXiv:2110.10422 2021.
[51] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshup track), 2015.
[52] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, Journal of machine learning research, 15(1):1929- 1958, 2014.
[53] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[54] M. Tavakoli, F. Agostinelli, and P. Baldi. SPLASH: Learnable activation functions for improving accuracy and adversarial robustness. arXiv:2006.0844 2020.
[55] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9 (11), 2008.
[56] C. White, A. Zela, R. Ru, Y. Liu, and F. Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34:28454-28466 2021.
[57] M. Wistuba, A. Rawat, and T. Pedapati. A survey neural architecture search. arXiv: arXiv: 1905.0139, 2019.
[58] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choc, and Y. Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pages 6023-6032, 2019.
[59] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
[60] B. Zoph and Q V. Le. Neural architecture search with reinforcement learning. arXi::1611.01578, 2016.
13

---

Applying activation function design is A Related Work
The techniques in this paper were inspired by prior research multiple areas, including neural architecture and activation function search, as well as research on the FIM.
Neural Architecture Search In neural architecture search [NAS: 13 57 60) the goal is to design
a neural network architecture automatically. NAS approaches typically focus on optimizing the type and location of the layers and the connections between them, but often use standard activation functions like ReLU. This work in complimentary to NAS approaches, because uses standard architectures but optimizes the design of the activation function.
Zero-cost NAS Proxies Recently, zero-cost NAS proxies have received increased attention 42 50 [56). These approaches aim to accelerate neural architecture search by using cheap surrogate calculations in place of expensive full training of architectures. This paper adopts similar approach, using FIM eigenvalues and actívation function outputs to predict which activation functions are likely to be most promising before dedicating resources to evaluating them.
Methods for automatically discovering activation functions include reinforcement learning 49) evolutionary computation 5 35), and gradient- based methods [1 5 19
Activation Function Search
44 54). This paper builds upon existing work, focusing efficient search and on understanding the properties that make activation functions effective.
Other Uses of the FIM This paper used FIM eigenvalues to predict the performance of different activation functions. The FIM is an important quantity in machine learning with several uses. One important example is optimal experiment design [14), where experiments are designed to be optimal with respect some criterion. The criteria vary, but are often functions of the eigenvalues of the FIM, such as the maximum or minimum eigenvalue, or the trace of the FIM (sum of the eigenvalues) or determinant of the FIM (product of the eigenvalues). Instead of choosing one optimality criterion and only considering one summary statistic, this paper keeps all of the eigenvalues of the FIM and learns an optimal distribution experimentally.
Past work has also used the eigenvalues of the FIM determine suitable values of the batch size
29 34] The FIM provides insights to the learning dynamies of SGD (27 and the dynamics of signal propagation at different layers in networks with and without batch normalization layers [25]. The FIM has also been used to develop second-on order optimization algorithms for neural networks
or learning rate for neural networks [15] 16 21
20
39
thus compelling further opportunity.
B Activation Function Search Spaces
The activation functions in this paper implemented as computation graphs from the PANGAEA search space [5] The space includes unary and binary operators, in addition to existing activation functions 7 12 30
45 49]. This approach allows specifying families of functions in compact manner. It is thus possible to focus the search on a space where good functions are likely to be located, and also to search it comprehensively.
Benchmark Datasets The benchmark datasets introduced in Section contain every activation function of the three-node form binary (unary (x), unary (x:) using the operators in Table The result 5,103 activation functions, of which 2,913 unique. This space is visualized in Figure 4
For Act BenchBenn and Act-Bench-Reslee Act-Bench-Van the results are from single runs due to computational costs.
the accuracies are the median from three runs. For
New Settings The first experiment in Dectionnoutiizeee a larger search space. Specifically, it was based on the following four-node computation graphs: binary (unary (unary (+)) unary (r) ) binary (unary (a:), .unary (unary (x))),
n-ary (unary (x)) ,unary (a)) .unary (x)), ) unary (binary (unary (x)) unary(x:) (x))), and unary (unary (unary (unary (x)))). The
unary and binary nodes used the operators in Table 4. and the n-ary node used the sum, product,
14

---

Table 4: Activation function search spaces were defined through computation graphs consisting of basic unary and binary operators well as existing activation functions S
Unary
Binary
erf(x)
ReLU(2)
I1 + D2
erfe(x)
ELU(x)
22
sinh (x)
SELU(z)
11 22
cosh (z)
Swish (z)
222
tanh(2) 2))
Softplus(a)
x²²
arcsinh(a))
(oftsign(z))
max x2)
x²
(r)
HardSigmoid
(21
bessel 10e
(x)
log(o(x))
bessel_ile(zz
maximum, and minimum operators. Together, these computation graphs create a search space with 1,023,516 functions, of which 425,896 are unique. This space is visualized in Figures) T0 T0 11
The third experiment in Section 6 i.e. CoAtNet Imagenette, added ELiSH, GELU, HardSigmoid, Leaky ReLU, and Mish as unary operators to the original benchmark search space. In this experiment, the search space comprised functions of the form binary (unary (unary (x)) unary (x)) and unary (unary (unary (x))). This search space contains 238,34 13,31 activation functions, of which 146,779 are unique.
C Fisher Information Matrix Details
In order to calculate the FIM, this paper uses the K-FAC approach 20 38 39 This technique is summarized in this Appendix, with notation similar to that of Grosse and Martens 20]
Preliminaries A feedforward neural network maps an input an = x to an output as = f(x: 0) through a series of layers. Each layer I E (1..... L is comprised of a weight matrix W,, bias
vector b, and an element-wise activation function or. With W. (b: Wi) and å = (1 a T
each layer implements the transformation
Wia--,,
(4)
a, (si).
(5)
veo(Wr) represent the vector of all network parameters. Parameter- ized by 0 and given inputs drawn from training distribution Qx. the neural network defines the conditional distribution R(x0)) The Fisher information matrix associated with this model is
Let (vec(W) T)
F= E0 1x4k,,.
x y- )-
As usual deep learning, the loss function C(y, z) represents the negative log-likelihood associated with ((x:0) and quantifies the discrepancy between the model's prediction z fxx 0) and the true label y. The network is trained to minimize the loss by updating its parameters according to the gradient VoC(y, fx:;
Approximations For ease of notation, write Dv VvL(y, (xx 0)). Recalling that 0 =
(vec(W,)T vec(Wr. the FIM can be expressed as an L. block matrix:
[vec(DW (eecDW))))((W.)))
E (eeo(
E ([vee(DWw)ve(DWW)>
E ([ve(DD:)ve((DW)))
15

---

ANN- C on CIFAR-1O 100
ResNet-S6 CIFAR-100
MobileVIT ImageNet
0.7
0.5
Valication Accuracy
0.3
0.1
Figure 10: Low-dimensional UMAP representation of the 425,896 function search space. The activation functions are embedded according to their outputs; each point represents unique function. The larger points represent activation functions that were evaluated during the searches; they are colored according their validation accuracy. Although the space is vast, the searches require only tens of evaluations to discover good activation functions.
16

---

Figure 11: A photograph three- dimensiienii scatter plot laser-engren easer-engraved into physical crystal cube. Each point represents one of the unique 425,896 unique activation functions in the search space. Points are arranged according
3D UMAP projection according activation function outputs; the points are the same as those shown in Figure [0] The cube shows the size and complexity of the search space, and the and 2D manifolds reveal underlying structure.
Note that DW Dsjā,_, and recall that vec(uv written as
Each block of the FIM can be
Fig = E VDe(W,
(8)
E (eccDs a _-
(9)
E [ā (s Ds |
(10)
E [ā x Ds, (a) @ Ds
(11)
E [ - Ds.
(12)
Two approximations are necessary in order make representation of the FIM practical. First, assume that different layers have uncorrelated weight derivatives. The FIM can then be approximated as block diagonal matrix, with F Fig 0 if + j. Second, if one approximates the pre-activation derivatives Ds; and activations a independent, then the diagonal blocks of the FIM can be
17

---

further decomposed into the Kronecker product of two smaller matrices:
F L1 TR | @ Ds Ds
R |ar-a|/-1] & & [Ds;Ds]]
(13)
Let S2 = [aaa] and r Ds ]. The approximate empirical FIM is then written
2o @ Fi
0
F
(14)
0
Pr.11 @ FT.
Layer-Specific Implementation The above example illustrates FIM approximation for simple feedforward network. However, most modern architectures contain several different kinds of layers. Some layers like pooling, normalization, dropout layers do not have trainable weights, and therefore these layers are not included in the FIM [26] [5].
Each diagonal entry Ω @ I corresponds to one layer with weights. The calculation differs slightly depending on the layer type, but otherwise the example above can be straightforwardly extended to more complicated networks. Calculations for three common layer types are presented below.
Dense Layers For dense layers, the matrices Ω and F₂ can be readily computed with one forward and backward pass through the network using mini-batch of data. The eigenvalues are then computed using standard techniques.
Convolutional Layers Convolutional layers require special consideration to calculate Q2 and Tı. For given layer, let M represent the batch size. the set of spatial locations (typically two-dimensional grid), A the set of spatial offsets from the center of the filter, and I and J the number of output and input maps, respectively. The activations are represented by the M |T| x J array Ar-1- The weights are represented by the / x |A] x J array W which is interpreted as an I x |A] matrix. The expansion operator I-] extracts patches around each spatial location and flattens them into vectors that become the rows of matrix: [ I] is M X J| matrix.
Similar the feedforward networks, the bias (if used) can be prepended to the weights matrix as W, (b: Wi) and homogeneous column of ones to the expanded activations as [A-11] = (1 A AI)))). This constructions allows the forward pass to be written as
S | WT,
(15)
A: = Φ (Si),
(16)
from which the factors are computed as
Ω |[ A [
(17)
r 1 E [DS] DS:]
(18)
Depthwise Convolutional Layers Depthwise convolutional layers utilize separate kernels for each channel. In this case, [A a M[TJJ x | matrix. Otherwise, the factors 22 and T2 are calculated the same way as they are for standard convolutional layers.
Eigenvalue Calculation Because FP is block-diagonal matrix, its eigenvalues simply the combined eigenvalues of each block: A(F) (X(P)) The eigenvalue calculation for one block F, = 1 & T₁ is further simplified by first computing the eigenvalues 1(22-1) and A(T;) for each Kronecker factor separately and then returning all pairwise products from the two sets of eigenvalues. For numerical stability, the eigenvalues can first be log-scaled and then all pairwise sums from the two sets are returned. Calculating the eigenvalues requires one forward and backward pass through the network with a mini-batch of data. The computational cost is therefore relatively cheap, especially compared with the cost of fully training network from scratch.
is possible for the FIM eigenvalues to invalid. For example, if the forward propagated activations or backward propagated gradients explode or vanish, then the diagonal entries Ω4 @ T, may be undefined. Such invalid values result from activation functions that are unstable. Therefore, invalid FIM cigenvalues provide good way to filter out bad activation functions.
18

---

D Features and Surrogate Details
This section describes how the activation function features were implemented and how the surrogate
was constructed.
Calculating FIM Eigenvalues The FIM eigenvalues were calculated for each activation function as discussed in Section 3] The cigenvalues were log-scaled for numerical stability. By definition, the number of eigenvalues is the same the number of weights in the neural network. To save space. the cigenvalues were binned to histograms. For a layer / with |0t] weights, [[0:]/100] equally sized bins from 100 100 were used. One histogram was computed for each layer in network, and all of the histograms were concatenated together into single feature vector for a given activation function. In this manner, the total dimensionality was 13,692 for AlI-CNN-C, 16,500 for ResNet-56, and 11,013 for MobileViTv2-0.5.
Calculating Activation Function Outputs The activation function outputs y = f(z) were caleu- lated for each activation function f by sampling =1,000 values ~ ~N 1) and truncating to the range [-5, 5, 5]. The same random inputs were used for all activation functions.
Per-Layer FIM Eigenvalues In Figure 5] the eigenvalues for the entire network are shown for completeness. However, the UMAP representations shown in Figure were produced by keeping the eigenvalues at each layer separate and computing weighted distance between them (according to Equation 2) As pointed out in the main text, FIM eigenvalues are informative but noisy features. In preliminary experiments, keeping the eigenvalues separate at each layer reduced some of this noise, resulting in more informative Figure and consequently improving the performance of the search algorithms.
FIM Eigenvalue Features Preliminary experiments aimed to predict activation function perfor- mance using common features in the literature, including maximum eigenvalue, minimum eigenvalue, sum of the eigenvalues, and product of the eigenvalues [14. More recently proposed features, such (second moment) / (first moment)². were also considered [48]. Ultimately, learning the relevant features from the entire eigenvalue distribution was found to be the most flexible and powerful approach.
UMAP Settings UMAP exposes a number parameters that can be used to customize its be- havior [40]. The metric parameter determines how distances computed between points, the n o_neeigbors parameter adjusts the tradeoff between the local and global structure of the data, and the min_dist parameter controls the minimum distance between points in the embedding space.
The plots Figure were produced by computing the distances between FIM eigenvalues and acti- vation function outputs. For the FIM eigenvalues UMAP (metrice 'manhattan,, n_ne m_neighbors=3, min_dist=O.11 1) was used, and for the activation function outputs UMAP (metric= feucl puclidean', m_neighbor==15, min_dist=0.1) was used. The distance metrics were chosen to implement Equations 3 and
In preliminary experiments, decreasing n_nneihborrs from the default of 15 down to for the FIM cigenvalues qualitatively improved the embedding for the combined features. The com- bined features were visualized with union model, i.c. umap_combined = umap_ soppffimeeigee +
umap_fn ft__ounututtteeeee (40]
E Experiment Details
This section specifies the details for the experiments in the main text of the paper. Several variations to the approach presented in the main text were also evaluated in preliminary experiments. The approach turned out to be robust to most of them, but the results also justify the choices used for the main experiments.
Training Details For CIFAR-10 10 and CIFAR-100. balanced validation sets were created by sampling 5,000 images from the training set. Full training details and hyperparameters are listed in Tables 6
19

---

Autolnit Table 5: Training details and hyperparameter values used in the CIFAR-I0 and CIFAR-100 experi-
ments.
All-CNN-C on CIFAR-I0 and CIFAR-100
Batch Size
128
Dropout
0.5
Epochs
25 for Act-Bench-CMN and search (Figure 7. 50 full evaluation (Table
Image Size
32 x 32
Learning Rate
Linear warmup to 0.1 for five epochs, then linear decay Yes
Mean/Std. Normalization
Momentum
0.9
Optimizer
SGD
Random Crops
32 x 32 crops of images padded with four pixels on all sides
Random Flips
Yes
Weight Decay
le
Weight Initialization
ResNet-56 on CIFAR-10 10 and CIFAR-100
Batch Size
128
Dropout
0.0
Epochs
Act-Bennch-Reshee and search (Figure 7 50 for full evaluation (Table
Image Size
32 x 32
Learning Rate
Linear warmup to 0.1 for five epochs, then linear decay
Mean/Std. Normalization
No
Momentum
0.9
Optimizer
SGD
Random Crops
32 x 32 crops of images padded with five pixels on all sides
Random Plips
Yes
Weight Decay
Te Autolnit 4
Weight Initialization
CoAtNet A smaller variant of the CoAtNet architecture was used in order to fit the model and data on the available GPU memory. The architecture has three convolutional blocks with 64 channels, four convolutional blocks with 128 channels, six transformer blocks with 256 channels, and three transformer blocks with 512 channels. This architecture is slightly deeper but thinner than the original CoAtNet-O architecture, which has two convolutional blocks with 96 channels, three convolutional blocks with 192 channels, five transformer blocks with 384 channels, and two transformer blocks with 768 channels 10 The models otherwise identical.
Search Implementation In order to predict performance for an unevaluated activation function, the function outputs and FIM eigenvalues must first be computed. Thus, the searches in Section 6 were implemented in three steps. First, activation function outputs for all 425,896 activation functions in the search space were calculated. This computation is inexpensive and easily parallelizable. Second, eight workers operated in parallel to sample activation functions uniformly random from the search space and calculate their FIM eigenvalues. Third, once the number of activation functions with FIM eigenvalues calculated reached 5,000, seven of the workers began the search by evaluating the functions with the highest predicted performance. The eighth worker continued calculating FIM eigenvalues for functions that their performance could be predicted during the search. This setup allowed taking best advantage of the available compute for the regression-tye search methods.
The experiments on ImageNet required substantially more compute than the experiments on CIFAR- 100. For this reason, all eight workers evaluated activation functions once the number of functions with FIM eigenvalues reached 7,000.
Computing FIM eigenvalues took approximately 26 seconds, 84 seconds, and 37 seconds per activation function for All-CNN-C, ResNet-56, and ViTv-c- respectively. This cost is not trivial, but is well worth it, the experiments in the main paper show.
wttp///......k..e..e. 0/keras_cv.....
20

---

Table 6: Training details and hyperparameter values used in the Imagenette and ImageNet experi-
ments.
MobileViTv2-0.5 on Imagenette and ImageNet
Batch Size
256
CutMix Alpha
58
1.0
Epochs
105
Evaluation Center Crop
95%
Image Size
160 x 160
Learning Rate
Linear warmup from le- 4e for five epochs, then cosine decay le
Mixup Alpha 59
0.1
Optimizer
AdamW [36
RandAugment
Magnitude six, applicd twice
Random Resized Crop
53
Minimum 8% of the original image
Woight Decay
0.02x current learning rate
ResNet-50 ImageNet
Batch Size
256
CutMix Alpha
188
1.0
Epochs
105
Evaluation Center Crop
95%
Image Size
160 x 160
learning Rate
Linear warmup from 1e to 2e for five epochs, then cosine decay to 1e 6
Mixup Alpha (59
0.1
Optimizer
AdamW 36
RandAugment
9
Magnitude six. applied twice
Random Resized Crop
53
Minimum 8% of original image
Weight Decay
0.02x current learning rate
Weight Initialization
Autolnit [4]
CoAtNet on Imagenette
Batch Size
256
CutMix Alpha
58
1.0
Epochs
105
Evaluation Center Crop
95%
Image Size
160
160
Learning
Linear warmup from 1e 4e for epuchs, then cosine decay to -7
Mixup Alpha 559
0.1
Optimizer
AdamW
36
RandAugment
9
Magnitude six, applied twice
Random Resized Crop
53
Minimum 8% of the original image
Weight Decay
0.02x current learning rate
Act-Bench-CNN
Act-Bench-ResNet
Act-Bench-VV
0.9
Aanaaio Accuracy
0.6
0.5
02
01
Figure 12: UMAP projections of FIM eigenvalues using the default hyperparameter of n_ t_neighhors=15. The embedding is informative but also noisy. Using n_neighbors=3, as shown in the main text, improved performance.
21

---

Unique Activation Functions Different computation graphs can result in the same activation function (e.g, max(z, O} and max of Section 6, repeated activation functions were filtered out. 1,000 inputs were sampled N(0, 1) and truncated [-5, 5). Two activation functions were considered the same if their outputs were
(0, 2²). In the benchmark dataset and in the larger search space
identical.
Improving the Combined UMAP Projection Figure 12 displays projection of FIM eigenvalues using default UMAP hyperparameters. The plots show the eigenvalues organized in multiple distinct one-dimensional manifolds. Again, FIM eigenvalues are noisy features; there some clusters of activation functions achieving similar performance, but there are also regions where performance varies widely. As mentioned in the main text, this issue was addressed by reducing the UMAP parameter n n__nighbors to 3. This change reduced the connectivity of the low low-dimensional FIM eigenvalue representation, resulting in a space with many distinct clusters (as seen in Figure 4
On its own, this setting did not improve the search on the benchmark datasets. However, it did improve performance when the FIM eigenvalues were combined with activation function outputs (as was discussed in Section 4 The reason is that the UMAP model for the activation function outputs did not decrease n_ o__eighhors, and so the combined UMAP model relied more the activation function outputs than it did on the FIM eigenvalues. As Figure 4 shows, the activation function outputs are reliable but sometimes project good activation functions to distinct regions in the search space. Introducing extra connectivity into the fuzzy topological representation via the FIM eigenvalues was sufficient address this issue, bringing good activation functions to common regions of the space.
Increasing the Dimension of the UMAP Projections The UMAP plots show two two-dimensional projections of FIM eigenvalues and activation function outputs. Regression algorithms were also trained on five and 10-dimensional projections. These runs resulted in comparable worse perfor- mance. Therefore, the two-dimensional projections were selected in the paper for simplicity and for consistency between the algorithm implementation and figure visualizations.
Gaussian Process Regression As an alternative search method, Gaussian process regression (GPR) was evaluated in activation function search. Several different acquisition mechanisms were used, including expected improvement, probability of improvement, maximum predicted value, and upper confidence bound. The approach worked well, but the results were inconsistent across the different acquisition mechanisms. GPR was also more expensive to run compared to the algorithms in the main text (KNR, RFR, SVR), and those algorithms were used instead for simplicity and efficiency.
The initial experiments with the KNR algorithm used k = 3. Experimenting with h: = (1, 5, 8} did not reliably improve performance, so k = 3 was kept.
Adjusting in KNR
Uniformly Spaced Inputs for Activation Function Outputs In an alternative implementation, equally spaced inputs from -5 to 5 were given to the activation functions instead of normally distributed inputs. This variation did not noticeably change the quality of the embeddings performance of the search algorithms. Therefore, normal inputs were used for consistency with Equation Figure the only exception; it used 80 inputs equally spaced from -5 to 5 and increased the UMAP parameter dist to 0.5. These settings improved the quality of the reconstructed activation functions in the plot.
the
15 and show plots and the validation accuracy of every candidate activation function evaluated in the searches for All-CNN-C on CIFAR- ResNet-56 on CIFAR-100, MobileViTVv-0.5 on ImageNet, and CoAtNet on Imagenette, respectively.
Evaluated Functions Figures 13
F
Future Work
This paper demonstrated that FIM eigenvalues and activation function outputs are efficient and reliable features that can predict performance of activation functions accurately. This finding enabled discovering better activation functions for various tasks, improving the state of the art in machine learning. Because the technique efficient, it was possible to scale it up to large datasets such as ImageNet. These discoveries inspire several avenues for future research, discussed below.
22

---

eda Shertalaar
Figure 13: Activation functions evaluated in the search for AlI-CNN-C on CIFAR-100.
23

---

2000
www.
Figure 14: Activation functions evaluated in the search for ResNet-56 on C1FAR-100.
24

---

Figure 15: Activation functions evaluated in the search for on ImageNet.
25

---

100 SAEEEE
200
Figure 16: Activation functions evaluated in the search for CoAtNet on Imagenette.
26

---

New Search Spaces The PANGAEA search space was used in this paper because it is known to work well for deep architectures [ In the future will be interesting to explore search spaces with different unary, binary, and n-ary operators. Beyond computation graphs, it may also be possible to apply techniques in this paper to optimize continuous vector representations of activation functions 14
Exploration vs. Exploitation The KNR approach was utilized to search for new activation functions because it performed well on the benchmark datasets (Section 5 In the future, it will be interesting to consider other algorithms and analyze their tradeoffs between exploration and exploitation. For example, in a resource-consstrance environment where improvement is needed quickly, a more exploitative approach could be used to find an improved activation function in a short time. On the other hand, if substantial compute is available, an approach that focuses exploration could be used to discover activation functions that perform well but are maximally different from functions used in modern architectures (Figure 8p). Novelty search 32 could serve suitable approach, and such discoveries could further understanding of how neural networks utilize different kinds of activation functions to learn.
Optimizing Multiple Activation Functions In a typical neural network design, the same activation function is used throughout the network. However, recent work has shown that it may be beneficial to have different activation functions at different locations, and further, that it may be useful to have different activation functions in the early and late stages of training [5]. Indeed, many hybrid architectures use Swish in convolutional layers and ReLU in attention layers [4]]. Unfortunately, it is difficult design these strategies manually, and so practitioners often use a single activation function for simplicity.
The techniques proposed in this paper may provide avenue toward optimizing multiple activation functions in tandem. For example, the features for multiple candidate activation functions could be concatenated into single feature vector, and this vector could be projected with UMAP to low-dimensional space where performance prediction is more straightforward.
Optimizing Parametric Activation Functions Parametric activation functions have learnable parameters that allow them to refine their shape via gradient descent. In some tasks, this extra flexibility results in better performance over fixed activation functions (5). The techniques introduced in this paper can be readily extended to optimizing the design of parametric activation functions as well. Because the surrogate considers the state of the network and activation function at initialization, it is possible to predict the performance by treating the activation function parameters as fixed to their initial values.
However, it may be possible to extend this idea further. Because the activation function parameters are implemented neural network weights, each parameter will have corresponding FIM eigenvalue. These extra eigenvalues will provide the surrogate with additional information that may help predict the performance more accurately.
For simplicity, current parametric activation functions usually initialize their parameters either to be 1.0 or to approximate some existing activation function, and the initialization is usually the same throughout the network. This method is likely suboptimal; the surrogate introduced in this paper could provide smarter approach. By adjusting the initial parameter values and observing the change in predicted performance, the surrogate can be used to find better initializations, including different ones at different layers in the network. This contribution could make parametric activation functions even more powerful.
Optimizing Other Aspects of Neural Network Design By fixing the neural network architecture and varying the activation function, this paper showed that is possible to use FIM eigenvalues to infer future performance. As the FIM is fundamental quantity in machine learning, it may be possible to apply a similar strategy to optimize other aspects of neural network design, such as normalization layers, loss functions, or data augmentation strategies
[8] 17 I8 35) If meaningful distance metric between such objects can be defined, then UMAP could be used to map them to a low-dimensional space where performance prediction is much simpler.
Similarly, one could use the FIM eigenvalues to optimize alternate objectives beyond accuracy. Robustness is particularly interesting objective, because the FIM can be used to describe neural
27

---

network's robustness to small parameter perturbations. Other objectives, such as interpretability. fair- ness, or inference cost, could also be considered. For example, one could consider a multidimensiooall regression approach where instead of just predicting accuracy, the surrogate would predict each of these quantities separately. Such a method could present the user with a Pareto front of activation functions involving tradeoff's between these quantities.
Reverse Engineering Activation Functions UMAP was used to project activation functions to a low-dimensional space, and regression algorithms to predict the performance of activation functions this space, i.e. to serve as a fitness function for the search. However, is possible that there no activation function that maps to the optimum of this fitness landscape. Indeed, because such search spaces are finite, the activation functions do not completely fill them. For example, there are empty regions in Figure 4 corresponding to activation functions outside of the predefined search space.
What should be done if an empty region of the embedding space has higher predicted fitness than any of the candidate activation functions? In the paper, these regions were simply ignored, and the activation function with the highest predicted fitness was used. However, the future, it may be possible to create activation functions that map to these empty spaces, an in so doing improve performance. One approach could be based on inverse transforms: Given a coordinate the low-dimensional embedding space, UMAP can apply inverse transform and return object that would have mapped to those coordinates. This technique was already used for visualization in Figure 3 Using this approach, UMAP could generate hypothetical desired FIM eigenvalue distribution, a list of activation function outputs.
There are two challenges to this approach. First, because UMAP is dimensionality- algorithm, different activation functions can map the same location in the embedding space. Thus, the mapping from embedding space back to activation functions is not well defined. Second, even if UMAP prescribes FIM eigenvalue distribution that is predicted result in good performance, may be difficult to manually design an activation function to satisfy that distribution.
However, generated list of prescribed activation function outputs is already a good start. From this list, it is possible to construct an activation function that interpolates through these points, either in piecewise linear fashion, with splines, using some other standard technique. Even without the corresponding FIM cigenvalues, such an approach could potentially improve the efficiency of novel activation function discovery, and lead to better designs for activation functions the future.
G Compute Infrastructure
The experiments in this paper were implemented using AWS g5 48xlarge instance with eight NVIDIA A10G GPUs. The total compute cost for the search experiments Section 6 was 14.49 GPU-hours for AlI-CNN-C on CIFAR-100, 21.67 GPU-hours for ResNet-56 on CIFAR-100, and 196.25 GPU-days for MobileViTv2-0. on ImageNet. This cost includes the time to train the eight baseline activation functions and then to evaluate 100 additional functions. The instance ran in Oregon (us-we tus-west-22 2) and was powered by renewable energy, so the experiments for this paper contributed no carbon emissions.
28


Reviews:
[{"review_id": "3gCBXq9l2F", "reviewer": "Reviewer_N57C", "summary": "This paper introduces a set of benchmark datasets for activation function search, and an efficient search method based on the analysis of the benchmarks.\n", "questions": "None", "limitations": "None", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed benchmark datasets are beneficial for further research.\n\n2. The method that searches activation functions through the function outputs of a limited number of samples seems effective and can significantly outperform the random search baseline.\n", "weaknesses": "1. The paper is hard to follow. The main text refers to many details in Appendix, but it is still complex and hard to get to the method. I suggest the authors refine the structure and make the technical details of the proposed method more clearly.\n\n2. Analysis is limited to show the efficiency of the method. The paper includes \"Efficient\" in the title, but I can only find the evaluation of efficiency in Appendix, and it should be compared with previous search methods to show how efficient it is. Besides, this method still needs to train multiple activation functions independently, which is also computationally expensive. \n\n3. The improvements are marginal. The authors should compare their method with existing approaches in both benchmark search and new tasks search. Besides, in Table 1, comparing with the popular searched activation Swish, the improvements are marginal.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer N57C**\n\n---\n\n> I suggest the authors refine the structure and make the technical details of the proposed method more clearly.\n\nThanks for the suggestion.  Many of these details are currently in the appendix.  We will use the extra page in the camera-ready version to include more of them and to refine the structure.  \n\n---\n\n> Analysis is limited to show the efficiency of the method…it should be compared with previous search methods to show how efficient it is.\n\nTo clarify, the efficiency comes from the number of evaluations needed to find a good function.  Previous approaches like PANGAEA and the algorithm that discovered Swish evaluated thousands of activation functions before discovering the best ones.  In contrast,     AQuaSurF is orders of magnitude more efficient.\n\nFigures 6 and 7 demonstrate this efficiency.  In particular, Figure 7 shows how AQuaSurF outperforms all baseline functions on ResNet-56 in just the second function evaluation.  We will revise the surrounding text to make these points clear.\n\nBecause the training setups and hyperparameters are not the same, the results from previous search methods are not directly comparable.  However, a comparison can still be made by looking at the relative improvements gained by using a discovered activation function instead of ReLU (calculated as (new_acc - relu_acc) / relu_acc).  For example, with ResNet-56 on CIFAR-100 AQuaSurF results in a relative improvement of 1.65%, and in the same scenario PANGAEA gives a relative improvement of 1.64%.  Thus, the two methods discover similarly effective activation functions, but AQuaSurF does so much more efficiently: requiring 100 function evaluations instead of 1,000.\n\n---\n\n> Besides, this method still needs to train multiple activation functions independently, which is also computationally expensive.\n\nYes, but in some domains, it is well worth it: Spending additional compute to improve performance even a small amount may translate to significant money saved or lives improved.  The important contribution here is that while previous methods required access to distributed computing environments, AQuaSurF can be run on a single commodity cloud instance.  Previously, only for well-resourced labs were able to take advantage of activation function optimization; now it is possible for everyday practitioners, with many more applications benefiting from it.\n\nFurthermore, this paper released three activation function benchmark datasets: Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT.  These resources make it possible to run search algorithms without a GPU at all.  We expect these benchmarks to be a valuable resource for the community, enabling future work to improve efficiency even further.\n\nFinally, recall that the activation functions discovered can be transferred to new tasks, even challenging ones such as ImageNet (Table 2).  Even though the best performance comes from customizing activation functions to specific tasks, these activation functions can still improve performance in other domains.\n\n---\n\n\n> Comparing with the popular searched activation Swish, the improvements are marginal.\n\nSwish can be seen as a state-of-the-art activation function, resulting from a significant effort to optimize activation functions.  Thus, even a small improvement over Swish is significant.  In some domains like medical diagnosis or stock trading, such small improvements can make a meaningful difference.  \n\nMoreover, Swish was developed in the context of tasks and architectures popular at the time, and it may not work as well in other contexts (as was already demonstrated with ResNet-v1-56 in Table 2 of the PANGAEA paper https://arxiv.org/pdf/2006.03179.pdf).  It is thus not the only activation function we will ever need; instead, it is important to be able to reliably and automatically discover better activation functions for any task and architecture that may come up in the future.  AQuaSurF’s sample efficiency will make it possible to improve performance in such new contexts as they arise.\n"}, {"review_id": "3wqtZtZHUY", "reviewer": "Reviewer_UeiT", "summary": "This paper addresses the optimization of activation functions in neural networks for improved performance in machine learning tasks. The authors create benchmark datasets and propose a surrogate-based optimization method based on a characterization of the benchmark space. They apply this method to discover better activation functions in CIFAR-100 and ImageNet tasks, showcasing its practical effectiveness.", "questions": "Weaknesses 2 and 3 raise concerns regarding the significance and necessity of the proposed problem and solution. If the authors are unable to address the issues of generalizability, I would be inclined to view their \"improvement\" as overfitting to specific tasks and networks.", "limitations": "No potential negative societal impact.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The authors create benchmark datasets (Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT) by training various architectures with numerous activation functions.\n\n2. The paper presents a novel surrogate-based optimization method that characterizes activation functions analytically. By utilizing the Fisher information matrix's spectrum and activation function output distribution, a low-dimensional representation is created.\n\n3. The proposed method, AQuaSurF, efficiently discovers improved activation functions in CIFAR-100 and ImageNet tasks, surpassing previous approaches in terms of evaluation efficiency.", "weaknesses": "1. The motivation and definition of using \"Activation Function Outputs\" as feature in Section 3 is not clearly explained.\n\n2. In Table 1, some widely used human-designed activation functions, such as ELU, ReLU, and Swish, consistently achieve top performance on various tasks with different networks. However, the top activation functions discovered by the proposed method vary across tasks and networks. This suggests a limited generalizability of the searched activation functions. In other words, when faced with a new task or utilizing a new network, the activation function needs to be searched again. Furthermore, this also implies that the searching method may overfit the specific task and network, rather than finding activation functions that are generally effective and meaningful.\n\n3. Related to the previous point, the design of the search space appears overly complicated, which also raises concerns about overfitting. As observed, the top activation functions discovered through the search process often involve complex combinations of existing human-designed activation functions. This complexity reduces their interpretability. Human-designed activation functions, on the other hand, are typically well-reasoned and supported by theory or hypotheses, allowing them to generalize effectively across tasks and networks. However, the searched activation functions are difficult to explain in terms of why they exhibit certain characteristics, and they lack generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer UeiT**\n\n---\n\n> The motivation and definition of using \"Activation Function Outputs\" as feature in Section 3 is not clearly explained.\n\nThe intuition behind using activation function outputs as a feature is that we expect activation functions with similar shapes to have similar performance.  From one perspective, Equation 3 quantifies the difference between two activation functions’ output distributions at initialization.  But from another point of view, Equation 3 is computing the pointwise distance between two activation function shapes, giving extra weight to the middle regions near x=0 where the activation functions are more likely to be utilized.\n\nIn the revision, we will explain this motivation for activation function outputs, and will clarify how Equation 3 implements this idea.  Thanks for pointing it out.\n\n---\n\n> The top activation functions discovered by the proposed method vary across tasks and networks.\n\nGeneral activation functions like ELU, ReLU, and Swish are useful for achieving good performance in many tasks.  However, in some tasks it is worth spending extra effort in order to achieve the absolute best performance. Customization can provide such an improvement. AQuaSurF is a way to discover customized activation functions that improve performance over the general-purpose baseline solutions in such tasks.\n\n---\n\n> When faced with a new task or utilizing a new network, the activation function needs to be searched again.\n\nYes, and this process allows taking advantage of customization. With previous techniques it was infeasible to perform such a new search for every task, but with AQuaSurF it is possible.  We hope that future work will build on the contributions in this paper, including the benchmark datasets and the code, and improve the efficiency even further.\n\nNote also that the best activation functions discovered often successfully transfer to new tasks and improve performance.  This is especially useful for challenging tasks such as ImageNet (Table 2). \n\n---\n\n> This also implies that the searching method may overfit the specific task and network.\n\nCustomization means finding an activation function that works as well as possible in the given context, i.e. architecture and task. The result may not work as well in another context---and that is precisely where the power of customization lies. While it is certainly possible to discover solutions that are general and apply to many contexts, they are essentially leaving money on the table. AQuaSurF provides a method for doing such customization separately for each context, thus taking advantage of any possible performance improvement.\n\nTo avoid overfitting, we use standard techniques: the networks are trained on the training set, the activation functions are evaluated on a held-out validation set, and final performance is measured on the test set.\n\n---\n\n> The top activation functions discovered through the search process often involve complex combinations of existing human-designed activation functions.\n\nThis is actually an advantage of using an automated search process: It is possible to use AQuaSurF to build on any human ideas, i.e. refine and combine them, as well as augment them with entirely new designs. Such solutions can be much more complex than the original human designs; it is thus possible to discover powerful activation functions that humans are not likely to discover on their own. \n\n---\n\n> The searched activation functions are difficult to explain in terms of why they exhibit certain characteristics.\n\nThis was true of previous work like PANGAEA and Swish, but this paper actually makes key contributions in understanding what properties make an activation function effective.  The two features the surrogate model uses are informative: Activation function outputs describe how the function modifies the forward-propagated signals before training begins, and FIM eigenvalues describe the curvature of the loss surface at initialization.  The paper thus suggests that we should not limit ourselves to only using activation functions that have a simple written form---properties such as function outputs and FIM eigenvalues matter more. Based on these observations, in the future it may be possible to develop a general theory of what makes activation functions effective. \n\nTo support this effort in practice, the benchmark datasets Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT, as well as the AQuaSurF software, will be a powerful resource.  They already made it possible for us to identify function outputs and FIM eigenvalues as useful predictors of performance; we expect that in the future they will be useful for the community to further theoretical understanding as well as practical development of activation functions.\n\n"}, {"review_id": "P9Jh7AoZuc", "reviewer": "Reviewer_x7yQ", "summary": "This paper introduces three benchmark datasets created by training CNN, ResNet, and ViT architectures using a set of activation functions generated from a three-node computation graph that combines unary and binary operations.\n\nThe benchmarks serve to showcase the efficacy of utilizing the 2D UMAP of the Fisher information matrix (FIM) spectrum and/or activation outputs as a cost-effective surrogate for predicting activation performance. Leveraging the 2D feature representation, an efficient activation optimization method, AQuaSurF, is developed by employing regression techniques to model activation accuracy across the 2D feature space, requiring only 100 function evaluations. The benchmark results further demonstrate the effectiveness and statistical reliability of this approach.\n\nThe proposed method is successfully applied to various vision tasks, where the discovered activation functions consistently outperform existing baseline activations. Moreover, the top activations identified through this search exhibit successful transferability to a new vision task.", "questions": "1- In the first paragraph of page 3 the authors observe, based on the scatter plots in Fig 2, that \"best results come from discovering functions specialized to individual tasks\". However, upon comparing the upper-left and lower-right corners of the plots with the upper-right region it appears that the best functions on one task also transfer effectively to, and are potentially among the best on, the other task. Is this interpretation correct?\n\n2- In the middle row of Fig 4. The UMAP depends only on the activations and not the model. However, there appears to be differences in the distribution of points in the 3 plots (and also compared to Fig.3). Is this because of filtering out failed activations and possible rescalings / reflections of the space? A brief comment on this would enhance clarity for readers.\n\n3- On lines 228-229 of the manuscript \"Thus, activation functions are embedded close to each other in this space if they have similar shapes, if they induce similar FIM eigenvalues, or both\", considering that the metric on the union of the representations is the sum of the metrics on the individual representations, then shouldn't the activations be close to each other only if they have both similar shapes and similar FIM eigenvalues?\n\n4- How does AQuaSurF compare with PANGAEA in terms of performance? given the partial similarity of the search spaces, is it possible to make a direct comparison between the two methods (e.g. by limiting the space to non-parametric functions)?", "limitations": "Limitations are partly addressed in the Future Work section in the appendix. There are no concerns regarding negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and easy to follow. \n\nThe approach of utilizing the UMAP embedding of the FIM spectrum with activation outputs to assess activation performance is novel and interesting. \n\nIn contrast to previous methodologies that relied mostly on evolutionary algorithms and required thousands of function evaluations, the method proposed in this work demonstrates efficiency by outperforming baselines with just 100 function evaluations. \n\nFurthermore, the benchmark datasets introduced in this work, may potentially help accelerate research on activation optimization. \n\nOverall, this paper offers valuable insights for assessing activation performance and also introduces a more efficient methodology for activation optimization.\n", "weaknesses": "In Section 6, the authors apply their proposed method to more challenging datasets and a larger activation search space, compared to those used to create the benchmarks. To further evaluate the effectiveness of the approach it would be beneficial to apply the method (KNR on UMAP embeddings) to vision tasks involving new network architectures as well. \n\nWhile the chosen baseline activations in Table.1 already include ReLU and Swish, used in the original three architectures studied in the paper, in order to further strengthen the results it would still be advantageous, and perhaps straightforward, to extend the list of baselines at least to those used in PANGAEA, including GELU, LeakyReLU etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer x7yQ**\n\n---\n\n> It would be beneficial to apply the method (KNR on UMAP embeddings) to vision tasks involving new network architectures as well.\n\nThis is a great idea.  We included CNN, ResNet, and ViT models in the paper to cover a wide range of possible architectures and would be happy to add additional comparisons to the revision.  We also made the AQuaSurF code publicly available and spent additional effort to write documentation and ensure that the code is easy to use, so we hope that additional comparisons in the future can be run by the community as well.\n\n---\n\n> In order to further strengthen the results it would still be advantageous, and perhaps straightforward, to extend the list of baselines at least to those used in PANGAEA, including GELU, LeakyReLU etc.\n\nThis is an excellent idea, and not only because it provides for more comparisons, but because it is not straightforward.  In order to conduct the experiment properly, we need to provide the surrogate with the performance of GELU, LeakyReLU, and any other activation functions we choose to compare against.  This extra information will naturally influence the surrogate’s predictions, and so we need to restart the search from scratch in order to make the comparison fair.  Thus, such comparisons raise an interesting question: How much does the performance of the surrogate depend on the number of initial activation functions it is given?  We are excited to run this experiment and will add it to the final revision.\n\n---\n\n> It appears that the best functions on one task also transfer effectively to, and are potentially among the best on, the other task.\n\nGood observation.  There are indeed some activation functions that perform well across multiple architectures. However, note that the best functions are specialized to a specific task.  Note also that Figure 2 only shows the distribution of accuracies for activation functions in the benchmark datasets.  When searching in larger spaces (as was done in Section 6), we do not know what the distribution of accuracies looks like.  The most important qualitative insight from Figure 2 is that specialized activation functions do exist, and so we should exploit this fact when searching for functions in more open-ended search spaces.  We will clarify this point in the main text.\n\n---\n\n> In the middle row of Fig 4. The UMAP depends only on the activations and not the model. However, there appears to be differences in the distribution of points in the 3 plots (and also compared to Fig.3). Is this because of filtering out failed activations and possible rescalings / reflections of the space?\n\nYes!  This is precisely what is happening.  Thank you for reading the paper so carefully – this is an extremely subtle point.  Indeed, the plots in the middle row of Figure 4 in principle should be the same, because they do not depend on the model.  They are different because the activation functions filtered out due to invalid eigenvalues (Figure 1) are in fact different across architectures.  Furthermore, UMAP is a stochastic algorithm, so even though there is substantial overlap in the activation functions it is embedding, the final results have small variations between them.  \n\nIn fact, if you look closely, you can actually see the “rescalings / reflections” of the space that you hypothesized.  In the middle row, Act-Bench-CNN and Act-Bench-ResNet are nearly perfect mirror images of each other.  You can see this in the arrangement of the overall points, but also with the embedding locations of the labeled activation functions ELU, -ELU, tanh, -tanh, abs, and -abs.  The Act-Bench-ViT plot appears different and has a few small clusters of purple points in the edges of the embedding space.  These are activation functions that were not filtered out for Act-Bench-ViT but were filtered out in the other tasks.  Indeed, if you remove these points, the Act-Bench-ViT embedding space becomes almost identical to the Act-Bench-CNN one (and is a mirror image of the Act-Bench-ResNet one).\n\nWe will clarify these points in the revision. Again, thank you for reading the paper so carefully.  This is an extraordinarily good insight, and we appreciate that our hard work is being given such a careful review.\n\n---\n\n> Considering that the metric on the union of the representations is the sum of the metrics on the individual representations, then shouldn't the activations be close to each other only if they have both similar shapes and similar FIM eigenvalues?\n\nWhat you are describing would correspond to an intersection of the representations, but we took a union of the representations.  So, activation functions are embedded close to each other if they have similar shapes, similar FIM eigenvalues, or both.  We tried the intersection approach but found the union of the representations to be more effective.  We will clarify this point in the main text.  (See https://umap-learn.readthedocs.io/en/latest/composing_models.html for more details.)\n"}, {"review_id": "v9FgcjABL6", "reviewer": "Reviewer_sps3", "summary": "The paper presents a new method for improving the performance of neural networks through the design of optimal activation functions. The authors created benchmark datasets by training convolutional, residual, and vision transformer architectures with systematically generated activation functions. They then developed a new surrogate-based method for optimization, which uses the spectrum of the Fisher information matrix and the activation function's output distribution to predict performance. The method was tested on CIFAR-100 and ImageNet tasks, and the results showed significant improvements in accuracy.\n\n", "questions": "None", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces an innovative approach to enhancing activation functions, surpassing existing techniques in both efficiency and effectiveness. \n\n2. The paper is exceptionally well-written, and the experiments conducted are notably thorough.\n\n3. The benchmark datasets created by the authors provide a foundation for future research on activation function properties and their impact on performance.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer sps3**\n\nThank you for the review.  Please let us know if you have any questions that we can address in the upcoming author-reviewer discussion period.\n\n"}]

Author rebuttal:
**Additional Response to Reviewer x7yQ**

---

> How does AQuaSurF compare with PANGAEA in terms of performance? given the partial similarity of the search spaces, is it possible to make a direct comparison between the two methods (e.g. by limiting the space to non-parametric functions)?

In principle this experiment can be run, but there are a number of challenges, and we do not think the results would be informative enough to justify the compute cost.

PANGAEA is expensive to run.  We could limit it to the same number of function evaluations as AQuaSurF, but then it is unlikely that it would discover anything useful (Figure 4, PANGAEA paper https://arxiv.org/pdf/2006.03179.pdf).  We could also limit PANGAEA to non-parametric functions, but in this case as well it is unlikely that PANGAEA would discover good functions (Tables 3 and 4, PANGAEA paper).  As noted in the response to Reviewer N57C, AQuaSurF gave a relative improvement of 1.65% over ReLU with ResNet-56 on CIFAR-100, while PANGAEA gave a relative improvement of 1.64%.  Thus, it appears that the two algorithms are similarly capable, but AQuaSurF is orders of magnitude more efficient , achieving a comparable result in 100 function evaluations instead of 1,000.

Importantly, instead of being in competition with PANGAEA, AQuaSurF can be viewed as an enhancement of it.  AQuaSurF used a similar search space as PANGAEA because it was shown to be powerful, and the main contribution was in efficiency gains, i.e., making activation function optimization so efficient that it can be run with commodity hardware as needed for new architectures and tasks.  It is likely that the surrogate model in AQuaSurF could be synergistic with other search algorithms and search spaces, and improve their efficiency as well.

Meta-review: The paper has received mixed reviews. There is a consensus among reviewers that the proposed AQuaSurf for searching activation functions from a surrogate model using limited outputs (i.e. FIM & Act) is novel. Further, the paper presents a benchmark dataset for searching activation functions in different network architectures (CNN,ResNet,ViT). As weaknesses, two reviewers (UeiT & N57C) are inclined that the paper is limited in terms of generalization and effectiveness of the search results over SOTA activations (e.g. Swish,ReLU,ELU). Authors have responded in the rebuttal and attempted to alleviate the concerns raised by the reviewers on describing the methodology and experiments. While reviewers related varying outputs from AQuaSurf to lack of generalization, the authors strongly believe this in fact is the strength of the searching method which can yield improvement over SOTA activations such as Swish. However, both reviewers still hold their original ratings. On the contrary, two other reviewers (x7yQ & sps3) strongly praise the work where the consensus is the paper introducing a novel approach for searching activation functions that require much smaller number of trials (a magnitude order faster) compared to previous methods which rely on evolutionary algorithms, requiring thousands of function evaluations to achieve optimum performance. Despite several concerns raised above, the AC finds the support from reviewers on the novelty of the paper, introducing new benchmark dataset, and effectiveness of the searching mechanisms enough to merit for publication. Further, we trust the authors will make good effort to address all concerns raised in reviews/rebuttal discussions in revised manuscript, as the final decision was significantly impacted by this agreement. The decision was discussed with and approved by the SAC.
Decision: Accept (poster)