Stochastic Adaptive Activation Function
Authors: Kyungsu Lee
Jaeseung Yang
Haeyun Lee
Jae Youn Hwang
Venue: NeurIPS 2022 Accept (2022)
Keywords: activation function
neurotransmission
adaptive activation

Abstract: The simulation of human neurons and neurotransmission mechanisms has been realized in deep neural networks based on the theoretical implementations of activation functions. However, recent studies have reported that the threshold potential of neurons exhibits different values according to the locations and types of individual neurons, and that the activation functions have limitations in terms of representing this variability. Therefore, this study proposes a simple yet effective activation function that facilitates different thresholds and adaptive activations according to the positions of units and the contexts of inputs. Furthermore, the proposed activation function mathematically exhibits a more generalized form of Swish activation function, and thus we denoted it as Adaptive SwisH (ASH). ASH highlights informative features that exhibit large values in the top percentiles in an input, whereas it rectifies low values. Most importantly, ASH exhibits trainable, adaptive, and context-aware properties compared to other activation functions. Furthermore, ASH represents general formula of the previously studied activation function and provides a reasonable mathematical background for the superior performance. To validate the effectiveness and robustness of ASH, we implemented ASH into many deep learning models for various tasks, including classification, detection, segmentation, and image generation. Experimental analysis demonstrates that our activation function can provide the benefits of more accurate prediction and earlier convergence in many deep learning applications.

Stochastic Adaptive Activation Function
Kyungsu Lee DGIST 42988 Daegu, South Korea ks ww..eeeddist.cc. ac.
Jaeseung Yang DGIST
42988 Daegu, South Korea yj yjs6813@dgist.ac
Haeyun Lee DGIST, SAMSUNG SDI 17084 Yong-In, South Korea haeyun. 1 lee.@ammung..g...... com
Jae Youn Hwang DGIST
42988 Daegu, South Korea jyhwangddgist.om ac..iitt....k... kr
Abstract
The simulation of human neurons and neurotransmission mechanisms has been realized in deep neural networks based on the theoretical implementations of activa- tion functions. However, recent studies have reported that the threshold potential of neurons exhibits different values according to the locations and types of individual neurons, and that the activation functions have limitations terms of representing this variability. Therefore, this study proposes simple yet effective activation function that facilitates different thresholds and adaptive activations according to the positions of units and the contexts of inputs. Furthermore, the proposed activation function mathematically exhibits more generalized form of Swish acti- vation function, and thus we denoted it Adaptive SwisH (ASH). ASH highlights informative features that exhibit large values in the top percentiles in an input, whereas it rectifies low values. Most importantly, ASH exhibits trainable, adaptive, and context-aware properties compared to other activation functions. Furthermore, ASH represents general formula of the previously studied activation function and provides a reasonable mathematical background for the superior performance. To validate the effectiveness and robustness of ASH, implemented ASH into many deep learning models for various tasks, including classification, detection, segmentation, and image generation. Experimental analysis demonstrates that our activation function can provide the benefits of more accurate prediction and earlier convergence in many deep learning applications.
Introduction
Searching for the optimal activation functions has been challenge in the field of artificial intelli- gence (Maas et al.) 2013) Ramachandran et al. 2017: Clevert et al. 2013). Early activation functions have been studied compensate for the non linearity of the artificial neural networks ameliorate the gradient vanishing problem (Hertz et al.) 1997 Hochreiter 1998). Recently, novel activation func- tions have been suggested with the zero-centerrr centered parametric properties that improve the training efficiency of deep neural networks (DNNs) (Maas et al. 2013; Clevert et al.) 2015). Currently, the activation functions focusing on the stability of DNNs probabilistic distribution of inputs have been proposed (Hendrycks, Gimpel 2016 Misra 2019 Advances the activation functions have
DDaegu Gyeongbuk Institute of Science and Technology corresponning author 'Our code is available at http://gtthubboo//yuugeeeee Aes.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

allowed DNNs to perform various tasks such as detecting or segmenting target objects in sophisticated images or even generating new images beyond the simple classifiers (Simonyan, Zisserman 2014
ZZha et al.
2019 Badrinarayanan et al. 2017 Goodfellow et al. 2014).
The activation functions has evolved behave more like human neuron (Sharma et al. 2017 Lee et al. 2017). However, Izhikevich| (2003); Evans et al. (2018) reported that the neurotransmission mechanism, including the membrane, action, and threshold potentials of human neurons, is subject to the location or the connection type of the neurons. Additionally, humans perceive objects with surrounding contexts using the N : N mapping of visions human neurons rather than the 1:1 mapping of pixel to an input node in a neural network (Liu et al. 2018)) The connections between neurons can be realized through the linear combinations of layers in DNNs. However, DNNs have limitations in terms of realizing contextual perception. This implies that the further improvement in deep neural networks and convolutional neural networks (CNNs) can be realized despite the impressive performance on image analysis (Tinsakul et al., 2019; Misra 2019). Therefore, the development of DNNs leaned to mimic human perception by realizing the mechanism of human Lindsay 202I).
neurons (Aggarwal, others 2018
ICurrently, the primary issue is that many activation functions exhibit passivity, in this paper, indicating that they determine outputs only concerning the value of one element rather than entire contexts. For instance, the Rectified Linear Unit (ReLU), defined as f(x) = max 0), determines the output values related to x (Fukushima, Miyake] 1982). whereas the softmax function generates output values as the ratio of the input value the totals (Goodfellow et al. 2017). Particularly, ReLU exhibits passivity, whereas softmax does not. Suppose an image can be classified by considering 80% of the total portion. Current activation functions are limited in terms of classifying such an image since only elements (pixels) of the image are used to rectify the image rather than ratio. Another limitation is that the activation functions are invariant. Although the parametric activation functions update their parameters during training, the resulting parameters are invariant during the inference phase (Xu et al., 2013] Bingham, Mükkkulainen 2022). Therefore, the limited rectification can be realized by the invariant parameters or thresholds regardless of new inputs from different domains (e.g., test set).
Contribution To realize the mechanism of human neurons that rectify inputs considering their contexts, we propose novel ASH activation function. The main contributions of this study are to suggest a simple yet effective activation function, ASH, and to implement ASH in a mathematically effective form. Going beyond the passive activation functions, ASH activation function is designed as (I) an active activation function that provides outputs regarding the context of inputs and (2) a conditional activation function that employs an adaptive threshold. Unlike ReLU or Leaky ReLU, the threshold value of ASH is adaptively changed by considering the contextual information.
fx 0
i 2 0,
f(x)
otherwise
In particular, the threshold value (0) is adaptively changed according to the input distribution without heavy calculation, and thus ASH provides outputs considering the contexts of inputs adaptively. By applying ASH, obtained the following theoretical and experimental results:
We conducted mathematical modeling on ASH in an effective form to ensure trainable and parametric properties, and thus ASH exhibits parametric and adaptive properties. The baseline threshold of ASH is initially trained during the training phase, and the threshold value is adaptively fine-tuned according to the contexts of inputs without heavy calculations.
We theoretically verified that ASH adaptively changes its threshold alongside the stochastic distribution of inputs. This implies that ASH provides outputs regarding the entire contexts of inputs, thus leading to enhanced feature extraction.
We theoretically verified that ASH exhibits generalized formula of Swish activation function and provided the mathematical explanations for the superior performance of Swish, which was empirically searched in the previous work.
We experimentally showed that ASH improves the performance of deep learning models on various tasks and shortens the convergence epoch.
Related works Activation functions affect the performance of the training process to determine a functional subspace of a DNN (Hayou et al. 2019). In particular, the non-linearity using activation

---

functions have been introduced to prevent the issue of the linear transformation causing simple feature extractions in the DNNs (Misra) 2019: Jarrett et al., 2009). DNNs with non-linearity have been employed to perform complex tasks LLeshno et al. 1993). In the early era, Rectified Linear Unit (ReLUU replaced the classical activation functions such as sigmoid and tanh (Nair, Hinton 2010) due to its simple and computational efficiency compared to other activation functions. During decades, many activation functions have been proposed including Leaky ReLU (Maas et al. 2013), Exponential Linear Unit (ELU) (Clevert et al. 2015), Gaussian Error Linear Unit (GELU) (Hendrycks, Gimpel| (2166), Scaled Exponential Linear Unit (SELU) (Klambauer et al.) 2017), and
Swish (Ramachandran et al. 2017) improve the performance and stability of learning parameters in DNNs. Those activation functions have solved the dying ReLU problem, which exhibits a zero value in the negative region, and improved DNNs more smoothly for stable optimization. In particular, ELU and SELU have realized internal normalization the layer using the zero-mean property. GELU exploited Gaussian error and could implement adaptive dropout to apply higher probabilistic intuition.
Problem statement For adaptive thresholding, ASH exploits a stochastic selection methodology such as a quick selection (Hoare 196T). In particular, for enhanced feature extraction, the informative elements, which exhibit large values, should be identified as an attention mechanism. contrast, some elements, which exhibit low relevance, should be required to be rectified. To this end, ASH is designed to identify informative elements but rectify others as O.
Let X E x#* xC be tensor (i.e., feature-map) disregarding the batch, but with a height (!),, width (W), and channel (C), and let X X be set of feature-maps. Let A be activation function such that A X X. We can then extract the jth element from X, and denote it as (6) Here, the goal of this study is to design the activation function represented as follows:
(x(0) 0
(f) is ranked the top-k percentile of X, otherwise
A(x
Note that, the novel activation function A is represented similar to ReLU, whereas its threshold is not invariant in contrast ReL.U, but it is subjected to the distribution of the input X. To simplify, let ((x((), k; X) be condition whether 2:(9) ranked in the top i percentile of X, and the negation of C' is denoted as -C. particular, the elements that satisfy C(x((*), k; X)) are from the first largest element to the (0.01kN) --th largest element in X, where N indicates the number of elements in X. Here, the elements can be extracted using simple algorithm as quick selection. Suppose set X that includes all elements in X such that X = (xx(1,, (r(1)) ..... ....,!!! x..) }. We can then construct subsets of X as Xe (x((0 X)C(=(0) k; X)} and (Xc) = (xx(1) E (x) k; X))- Equation (
can be then simplified as follows:
A(z!0)) = (0(0) 0 E &x.
=(o') otherwise
In summary, the activation function is designed to sample the top-k percentile from the input, where criteria are the values of elements. Sampling examples are presented in Fig. compared to ReLU,
0.24
0.33
0.12
-0.4
0.24
0.33
0.12
0.24
0.33
0.12
0
0.24
0.33
0.12
0.33
40.49
0.01
8.17
0.37
0.81
0.17
0.37
U.81
0.17
0.37
0.17
0.37
0.37
-0.44
-0.46
-0.07
0.27
-0.07
0.27
0.30
0.13
-0,03
0.3
(a) Feature-map
(b) ReLU
(c) ASH (80%)
(d) ASH (50%)
(e) ASH (30%)
Figure 1: Input feature-map and outputs by activation functions. ASH (k%) indicates that ASH activation function samples top-k% elements from input feature-map. The sampled elements by the activation functions are colored as blue.
3

---

2 Method
This study aims design an activation function for stochastic sampling of the top-k percentile elements from inputs. However, sorting or sampling methods such as a quick selection requires high computational costs. In contrast, sampling the top-k percentile can be realized using a Z-- score-based method in simple calculation despite the prerequisites of normal distribution (also known as Gaussian distribution). As discussed below, the outputs of neurons are normally distributed. Therefore, we employed the stochastic sampling to design ASH activation function.
In this section, (1) demonstrate that the outputs of neurons normally distributed, (2) construct model for stochastic sampling using a 7-score, (3) formulate ASH activation function, (4) verify that ASH parametric and trainable, and (5) search for general applications of ASH activation function.
2.1 Background
Gaussian distribution Many deep learning models employ normalization methods to improve their stability in training (loffe, Szegedy 2015 Ulyanov al.) 20166; Wu, He 2018). In previous study, Loffe. Szegedy (2013) reported that the output of the convolutional layer, x = Wa + b, is more likely have symmetric, non-sparse distribution, that is "more Gaussian" Since most deep learning models are based on convolutional operations, the outputs neurons are supposed to be normally distributed (Gaussian distribution). Therefore, it concluded that the inputs of the activation functions are normally distributed when activation functions follow convolutional layers, such that N (ua, o²), where an input feature-map an activation function, Pa and aμ are mean and standard deviation of x, respectively. Therefore, we can obtain the following proposition.
Proposition 1. The outputs of neurons in convolutional neural networks are normally distributed.
Top-k%
Figure 2: Top-k% sampling from normal distribution
Sampling from normal distribution This study aimed to sample the top-k% elements that exhibit large values in an input feature-map. Since the area under the normal distribution indicates the percentile, sampling the top-k%% from normal distribution is theoretically identical to the statement calculating the area under the curve presented in Fig. 2 Let be a tensor, a multi multi--imensiooall array or matrix. should be then normally distributed, and thus we can sample the elements (()) the top-&% from F using the following equation:
P( 2 x k, k [0
1-00 co]
(4)
However, heavy computational costs are incurred to obtain a trivial solution from the probability density function of the normal distribution, defined as V (=- /220r Therefore, leaned to probability theory to simplify the computation rather than calculus. To this end, we employed the standard normal distribution (Z-score normalization) for Equation (4) and we obtained the following equation:
1 PF
P(Z()) 2 2x) k. Z
s.t. Z N0.
(5)
OF
where 21 -3 pr)/opp which is the Z-normalized value from z/ and thus 2 is subjected to indicating percentile to sample, in terms of Z-table (Larsen, Marx 2005). Then, go Z-table and

---

easily find the proper value for 2g, intuitively. Therefore, the condition. 2 = - / /F 2
2k + f(i) 2 Hr + p, mathematically identical to sample the top-k% elements from F. To summarize, we obtained the following proposition.
Proposition 2. Element σ6 E X, which is normally distributed, in the top-A% of X if x(()) 2
μx + X, where 2 is a valuee subjected to k in Z-table (Larsen, Marx 2005).
The mechanisms convolutional neural networks (CNNs) have been studied in many previous works (Rumelhart et al. 1986, Bottoo 20T0 Zhang 2016 Hu et al. 2018). Training CNN models is subjected to the backpropagation derived from the partial derivatives of loss functions by the individual convolutional parameters. Let L be loss function for the deep learning model M and let W be one the variables in M. Then, the derivative of L in terms of W is represented aL and W is updated as W + W 17 aw as aw.
Differentiation
OL with a learning rate 7). On the other hand, suppose variable 0 be the threshold, the function f(x) our if x: 2 0, otherwise O. Thus, the partial derivative of f is represented as follows:
as - if 0.
fI, 2 da (0, otherwise
0s
0s
8x 00 otherwise
= 0 00
(6)
Here, is arithmetically combined with fxx. whereas 0 does not. Therefore, is trainable, but 0 is not trainable this context. In basic calculus, trivial that if the loss function not dependent on the variable, the partial derivative is zero, and thus the variable cannot be trained or optimized; In other words, it is invariant. Therefore, we obtain the following lemma:
Lemma 1. The derivative of a variable in the conditional statement is zero, and thus that the variable cannot be optimized.
2.2 ASH Activation Funetion
Let X be an input of ASH activation function (ÆE) and be a tensor of which elements are normally distributed. Furthermore, let 2.(() E the i-th element X. Then, by Proposition 2, ASH activation function, which samples the top-k% elements from the input, represented follows:
E(x(")) = -(00 (1) if 2 μx + 240x:
otherwise
where μa and o are the mean and the standard deviation of all elements in X, respectively, and Ze is the Z-score concerning percentile (k) to sample (i.e., = 1.96 if k = 2.5%, see Z-table). Equation 7 exhibits that ASH activation function is represented simple yet effective form with low computational costs.
Intuitively, we assumed that the activation level (percentile) is supposed be different by each neuron and the tasks of the deep learning model, similar to human neurons. However, in Equation (7.. the condition (z) 2 μx + xx(x) ) is invariant by Lemma 1. Note that, μ4 + 2g is variable and changeable with respect to X, but the sampled portion (k-% related 2x) from X is invariant. Therefore, ASH in Equation (7) is not parametric and trainable. To make ASH be trainable and
parametric, let Equation 7 be substituted using proxy function as 1E(x = (+) (x) such that
the proxy function ()) is represented as follows:
f
af px ZNOx 2 0,
f(x(0)
to otherwise
(8)
For simplicity, suppose that Heaviside step function Weisstein 2002) is defined H(x) d max(0, 2r), and thus f(x(())= Hzx μx xx Then, we obtain the arithmetical form to The formulate ASH activation function as follows:
px -xx==xxx)
Even with the arithmetic formula, ASH activation function Equation 9 still independent of zk- and thus the zạ is still not trainable. However, it is well known that the Heaviside step function is
5

---

analytically approximated as (=) = 1+ tanh(ax) with a large value of a (lliev et al. 2017 and thus ASH activation function is approximated using the smooth Heaviside step function follows:
E(x!0) = ( H H(x px -x0x)
= 2 (0) + -r (0) Lanh(a(a - px -xxxx))
(10)
1 200020(0)020000000 -x x
Since 25 arithmetically placed, 25 representing a sampling percentile is trainable, and thus ASH activation function is also trainable and parametric. By optimizing Zk. ASH activation functions exhibit different thresholds. Therefore, based on the stochastic sampling of inputs and different thresholds, similar to human neurons, synapses, and their potentials. As human neurons, the mechanism of ASH can be summarized as follows:
concluded that ASH exhibits different activation levels
(I) In the training phase, each ASH activation function optimizes its z and fine-tunes the threshold for the sampling percentile of inputs. Thus, it implies that ASH activation function realizes the arbitrary threshold potentials as human neurons (Clevert et al. 2015 examples of 2 related Equation
Evans et al. 2018). Some
7 are:
Example 1. A small value of žg. even a small negative value, implies the dense activation, and the dying ReLU problem can be solved.
Example A large value of 2ę implies the sparse activation, and the sparsity can be leveraged.
(2) In the training inference phase, ASH activation function rectifies the inputs using the learned threshold value and contexts of inputs. In particular, to sample the top-k percentile, ASH employs the mean and the standard deviation of inputs, representing the contexts of the entire inputs. Therefore, it implies that ASH realizes the adaptive activation considering the contexts of inputs. Some examples of the adaptive activation related Equation (7) are:
Example 3. A small threshold value (0,) is employed to caleulate the input that exhibits large mean and standard deviation values (X > 0g).
Example 4. large threshold value (0) is employed to calculate the input X' that exhibits large mean and standard deviation values (XX > 0 >>>>> 0.).
2.3 Generalized Activation Function
We found the innovation while representing Equation as follows:
10 using the sigmoid function S(x) = I+e-
fEx(() s( (()(((((z))) px - =xax))
(11)
= S(ax + b))
In a previous work, Ramachandran et al. (2017) introduced the leverage of automatic search tech- niques to discover the best performance activation function. The experiments empirically discovered that the Swish activation function is the best performance activation function, defined as .x:(()) (Ra- machandran al., 2017). Intuitively, the definition of the Swish activation function the same with Equation [11) and Equation (I1) represents more generalized formula. Therefore, ASH (Adap- tive SwisH) activation function provides the theoretical explanations for why Swish was the best performance activation function in the empirical evaluations. Therefore, we obtain the following.
Lemma 3. ASH activation function exhibits general formula for the Swish activation function.
Interestingly, the activation function designed for stochastic adaptive sampling is converged to the generalized Swish activation function. The extreme impression is that the stochastic percentile sampling by the activation function that mimics real neurons expresses the general formula of the swish activation function formerly known as ofate-of-the-att Therefore, the stochastic percentile

---

sampling can partially be applied to the Swish activation function. Additionally, it can be supposed that the Swish activation function achieved superior performance in the previous studies based on the utilization of stochastic percentile sampling.
This paper initially considered activation function that enables stochastic percentile sampling in a mathematically effective manner. However, we found that the mathematical expression of ASH supports the theoretical background of the Swish activation function. Therefore, this paper provides the theoretical backgrounds and rationales for the Swish activation function, which was empirically investigated. It is a significant innovation provide the mathematical theorem that the Swish activation function is derived from a stochastically designed activation function.
Main Result
Similar to previous study (Ramachandran et al. 2017), we compared ASH to several baseline activation functions on various models for different tasks using public datasets. Because many activation functions have been developed, we employed some of the most commonly used activation functions, namely ReLU, leaky ReLU (LReLU) (Maas et al. 2013), parametric ReLU (PReLU) (He et al. 2015), Softplus (Nair, Hinton 2010), ELU (Clevert et al. 2015), SELU (Klambauer et al. 2017). and GELU (Hendryeks, Gimpel| 2016). In our experiments, every hyper-arameter in ASH and the other activation functions set to be the same to demonstrate the advantages of ASH compared to other activation functions. In the tables, the highest accuracy values are highlighted in bold.
3.1 Classification Task
We first compared ASH to all the baseline activation functions on the CIFAR- 10 (rizhevssyy et al. 2009), CIFAR- 100 (Krizhevsky et al., 2009) datasets, and ImageNet (Russakovsky et al.) 2015) datasets. We employed environments from a previous study (Ramachandran et al. 2017) and re- implemented the baseline models of 1es (He al. 2016), wide ResNer28- 10 (Zagoruyko)
Komodakis] 2016), and DenseNet-1000 (Huang et al. 2017). 1. Based on these different environments,
small differences were reported previously, but we believe that the accuracy trends are similar. We first evaluated ASH activation function against other activation functions using the ImageNet 2012 classification dataset because ImageNet is a widely utilized dataset in classification tasks. We then evaluated all activation funcciinn using the CIFAR-I0 and CIFAR-100 datasets, which have been widely utilized as benchmarks.
Model
Top-1 (%)
Top-5
Acc. (%)
ReLU
76.4
0.09
75.6
0.10
77.1 0.11
91.2 0.09
90.7 0.06
90.7 1.
LReLU
77.6 0.07
78.0
0.03
76.6 + 0.07
91.6 0..0 0,10
91.2 : 0.07
92.3 + 0.07
PLeLU
77.0 0.13
78.7
0.03
0.09
92.9 1 0.03
92.3 1 0.14
92.2 1: 0.12
Softplus
0.11
77.3
0.03
0.00
9.12
91.7 0.05
93.8 0.11
ELU
71.6 0.09
73.7 = 0.09
0.10
85.600000
89.3
90.2 0.08
SELU
75.4 0.13
78.1 0.14
0.06
9.09
0.10
GELU
76.8 0.13
77.9
0.05
0.12
90.2 1 0.11
92.6 1 0.04
91.9 1 0.09
0.07
76.6
0.06
76.5 0.05
92.2
0.12
90.9
1 0.07
92.2
ASH
78.5 0.06
78.6 0.07
78.7 0.10
94,0= 0.08
94.71 0.07
94.1 0,08
Table
ImageNet datases, Three models are avergged. The values are mean and 95% confidence Interval (C.L.
Model
ResNet
WRN
Net
Model
ResNet
WRN
DenseNet
ReLU
94.4 0.04
95.6 0.03
95.7 = 0.02
ReLU
74.3.E...
784 0:04
84.0 0.09
LReLU
94.5 = 0.05
95.61
94.7 =::00 0.09
LReLU
75.3 1: 0.06
77.9 1 0.07
82.2 : 0.07
PLeLU
94.7 0.08
1..4
95.1 1
PLeLU
74.7 0.06
77.6 0.06
82.1 0.07
Softplus
94.3
94.2
95.2 = 0.07
Softplus
76.1 + 0.05
78.6 2.06
84,1 0.02
ELU
0.10
93.8
94.5 = 0.11
ELU
1..0
7644 0.09
80.8 0.05
SELU
94.5 = 0.05
9
94.9 = 0.10
SELU
73.4
74.4 0.08
81.4 + 0.06
GELU
95.2 1 0.04
95.7 1 0.06
94.8 1 0.10
GELU
75.0 0.05
78.2 0.10
84.0 1 0.02
Swish
95.5 0.09
95.6 1
75.7 0.10
78.9
84.0
ASH
95.7 = 0.08
96.7
0.04
96.0 = 0.11
ASH
76,5 2 0.08
79.2 =
0.06
84.6 0.06
Table CIFAR- meun values and C.I.
Table
100 memm values and C.I.
The ImageNet dataset evaluations were averaged based on the accuracy values of the three deep learning models. The results in Tables to highlight the outstanding performance of ASH activation function in terms of improving predictive accuracy. Because deep learning models for classification tasks demand sparsity, it intuitive that ASH activation function improves accuracy compared to other activation functions.
7

---

3.2 Detection Task
We compared ASH to all the baseline activation functions on the COCO (Lin et al. 2014) and PASCAL VOC (Everingham et al.] 2010) datasets for the detection task. Both of these datasets are widely used as benchmarks for detection tasks. We employed the same environments as the classification task and implemented the baseline models of Mask-R-CNN (He et al., 2017). SSD (Liu et al. 2016), and YOLOv4 (Bochkovskiy et al., 2020). For detection tasks, deep learning models output bounding boxes representing the locations of target objects. We exploited mAP@50 as evaluation metric based on its popularity for detection tasks.
Model
MR-CNN
SSD
YOLOv5
Model
MR-CNN
SSD
YOLOv5
ReLU
68.5 = 0.02
70.1 0.07
72.1 I 0.04
ReLU
65.8 0.03
67.3 0.07
69.3 1: 0.09
LReLU
68.9 0.06
0.10
12.6
LRcLU
+.07
68.4 0.07
70.5 0.07
PLeLU
69.4 = 0.10
7.11
73.1 = 0.11
PLeLU
67.8 0.03
69.2 1 0.07
71.2 2: 0.09
Softplus
69 0.03
71.0.:00
73.0
Satplux
67.9
0.03
71.4 d:
ELU
694
71.1
73.0 0.06
ELU
6..8
09.3 0.06
71.4 0.06
SELU
69.7=
71.2
73.3
SELU
6.07
69.9 0.03
72.1 0.11
GELU
70.0 = 0.08
71.6
0.05
73.7 = 0.04
GELU
68.9 0.07
70.5 0.03
72.4 + 0.11
Swish
70.4 1 0.08
72.0 1 0.03
74.0 1 0.08
Swish
69.0 0.04
70.6
0.02
72.6 1. 0.08
ASH
0.06
70.5
0.11
Table COCO with mean values
S. PASCAL VOC mean values
The quantitative results in Tables 4 and 5 highlight the outstanding performance of ASH activation function compared to other activation functions. A higher mAP indicates that the predicted bounding boxes are closer to the annotations. ASH activation function provides superior performance for detecting target objects in various datasets for various deep learning models. Because the deep learning models used for detection tasks demand locality to generate bounding boxes, is expected that zą will be small, demonstrating that greater activation can be realized using ASH activation function compared to the models used for the classification task.
3.3 Segmentation Task
We compared ASH to all of the baseline activation functions on the ADE20K (Zhou et al. 2017 and PASCAL VOC (Everingham et al. (2100) datasets for the segmentation task. Both datasets include many target objects in one scene. Therefore, they are widely utilized as benchmarks for segmentation tasks. We employed the same environments as the classification and detection tasks and implemented (Chen et al. 20188.
the baseline models of U-Net (Ronneherger et al. 2015). Deepl.abV3+ and EfficientNet (Tan, Le) 2019). Similar to other general benchmarks, we adopted intersection over union (IoU) and mean loU (mloU) values as evaluation metrics based on their popularity for segmentation tasks.
Similar the previous tasks, the quantitative results in Tables 6 and highlight the outstanding performance of ASH activation function compared to the other activation functions. Because locality important for segmenting target objects from the background in segmentation tasks, it is intuitive that ASH activation function improves locality during feature extraction. The experimental results demonstrate that superior segmentation performance can be realized by using ASH activation function. which aids significantly in localizing target objects.
Model
U-Net
DLY3+
EfficientNet
Model
U-Net
DIV3+
EfficientNet
ReLU
0.04
ReLU
0.05
LReLU
49. = 0.07
51.0 0.09
52.3 0.05
LReLUU
76.2
78.0 - 0.05
80.3 + 0.10
PLeLU
49,9 0.02
51.0
52.5 0.06
PLeLU
77.0
78.9 - 0.06
81.0 0.08
Somplus
50.1 = 0.11
51.2 1122:0.0
5188 0.03
Solpllu
77.1 0.09
78.8 4 0.03
81.2 2 0.10
ELU
50.3 1 0.04
$1.4 0.09 0.09
52.8 0.08
ELU
77.2 1 0.10
79.0 1 0.05
81.3 1 0.02
SELU
50.9
$2.1:0003
53.5
SELU
78.2
+
GELU
50.9 1 0.07
52.2 0.08
53.5 0.05
GELU
78.9 1. 0.06
80.7 1. 0.05
83.2 1. 0.08
Swish
51.3 0.05
52.4
0.07
Swish
78.8 +
4:
ASH
534
0.05
54.7 1. 0.08
56.3
0.09
ASH
81.2 11 0.04
83.2 3: 0.04
85,5 : 0.05
Table ADE20K with mean values and 93%
Table PASCAL
mean values 95% C.I.
8

---

VALIDATION LOSS
SeLU
GeLU
ASH
Swish
TRAINING EPOCH
Figure 3: Validation loss values alongside the training epoch for the activation functions. The validation losses averaged from the results of all experiments.
3.4 Training Time
We empirically explored the effectiveness of ASH activation function in terms of training time by monitoring the validation loss values for all activation functions. Fig. reveals that the loss values of ASH activation function exhibit steeper slope than those of the other activation functions. Therefore, because ASH activation function reaches convergence significantly faster than the other activation functions, we can empirically conclude that ASH has superior effect in terms of reducing training time.
Through our experiments, explored the outstanding performance of ASH activation function compared to other activation functions, including improvements in accuracy, sparsity, training time, and localization. In this study, the experimental results demonstrate the outstanding performance of ASH activation function. Additionally, supporting experiments and the results of other tasks such as image generation presented in the Supplementary Material.
4 Conclusions
In this paper, we proposed novel activation function to rectify inputs using an adaptive threshold considering the entire contexts inputs more like human neurons. To this end, we designed an activation function to extract elements in the top-k percentile from the input feature-map. Since sorting algorithm- based selections or quick selection algorithm demands a heavy computational cost, we employed the stochastic technique utilizing normal distribution to realize stochastic percentile sampling. Based on the mathematical derivations, we implemented ASH activation function in simple yet effective formula (xx)
sigmoid(az:+ (ax++)r))))) with low computational cost for sampling the top-k percentile from the input. addition, we implemented ASH activation function, realizing (1) the adaptive threshold by employing the 7-score-basse trainable variables and (2) the perception of entire contexts in rectifying an input by utilizing the mean and standard deviation the input. Mcanwhile, ASH activation function represented the generalized form of the Swish activation function that was empirically searched in the previous study. Therefore, this study also exhibited novel contribution of the mathematical proofs for the late-offtthearr performance of the Swish activation function. Experiments using various deep learning models on different tasks (classification, detection, and segmentation) demonstrated superior performance for ASH activation function, in terms of accuracy, localization, and training time.
Acknowledgment
This work was supported in part by the National Research Foundation of Korea (NRF) under Grant NRF-2020R 1A20500AA2B5000022 and in part by the Bio & Medical Technology Development Program of the National Research Foundation (NRF) funded by the Korean govermment (MSIT) (No.2017MBBA9G8084466

---

References
Aggarwal Charu C, others. Neural networks and deep learning // Springer. 2018. 10. 978-3.
Badrinarayanan Vijay, Kendall Alex, Cipolla Roberto. Segnet: A deep convolutional incoder-decodet architec- ture for image segmentation // IEEE transactions pattern analysis and machine intelligence. 2017. 39, 12. 2481 2495.
Bingham Garren, Miikkulainen Risto. Discovering parametric activation functions // Neural Networks, 2022.
Bochkovskiy Alexey, Wang Chien- Yao, Liao Hong- Yuan Mark. Yolov4: Optimal speed and accuracy of object detection // arviv preprint arXiv:2004. 10934. 2020.
Bottou Lčon. Large-scale machine learning with stochastic gradient descent // Proceedings of COMPSTAT 2010. 177 186.
Chen Liang- Chng,, Zhu Yukum, Papandreou George, Schroff Florian, Adam Hartwig. Encoder-decoder with atrous separable convolution for semantic image segmentation // Proccedings of the European conference on computer vision (ECCV). 2018. 801-818.
Clevert Djork-Arné, Unterthiner Thomas, Hochreiter Sepp. Fast and accurate deep network learning by exponential linear units (elus) I| arXiv preprint arXiv:1511.0722 1.07289.. 2015.
Evans Dominic A, Stempel A Vanessa, Vale Ruben, Ruchle Sabine, Lefter Yaara, Branco Taggo A synaptic threshold mechanism for computing escape decisions // Nature. 2018. 558, 7711. 590-594.
Everingham Mark, Van Gool Luc, Williams Christopher KI, Winn John, Zisserman Andrew. The pascal visual object classes (voc) challenge // International journal of computer vision. 2010. 88, 2. 303-33..
Fukushima Kumihiko, Miyake Sei. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition // Competition and cooperation in neural nets. 1982. 267-285.
Goodfellow Ian, Bengio Yoshua, Courville Aaron. Deep learning (adaptive computation and machine learning series) // Cambridge Massachusetts. 2017.321-350
Goodfellow lan, Pouget-Abodie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, Bengio Yoshua. Generative adversarial nets // Advances in neural information processing systems. 2014. 27.
Hayou Soufane, Doucet Arnaud, Rousseau Judith. On the impact of the activation function on deep neural networks training // International conference on machine learning. 2019, 2672-2680.
He Kaiming, Gkioxari Georgia, Dollár Piotr, Girshick Ross. Mask r-cnn // Proceedings of the IEEE international conference on computer vision. 2017. 2961-2969.
He Kaiming, Zhang Xiangyu, Ren Shaoqing, Sun Jian. Delving deep into rectifiers: Surpassing human-leve performance on imagenet classification // Proceedings the IEEE international conference on computer vision. 2015. 10344
He Kaiming, Zhang Xiangyu, Ren Shaoqing, Sun Jian. Identity mappings in deep residual networks // European conference on computer vision. 2016. 630-645.
Hendrycks Dan, Gimpel Kevin, Gaussian error linear units (gelus) // arxiv preprint arXiv: 1606.0841.. 2016.
John, Krogh Anders, Lautrup Benny, Lehmann Torsten. Nonlinear backpropagation: doing backpropagation without derivatives of the activation function // JEEE Transactions on neural networks. 1997. 8, 1321 13277
Hoare Charles AR. Algorithm 65: find // Communications of the ACM. 1961. 4, 7. 321-322.
Hochreiter Sepp. Recurrent neural net learning and vanishing gradient // International Journal Of Uncertainity, Fuzziness and Knowledge-Bain Based Systems. 1998. 6,2. 2. 107-116.
Hu Zheng, Li Yongping, Yang Zhiyong. Improving convolutional neural network using pseudo derivative ReLU 11 2018 5th International Conference on Systems and Informatics (ICSAI). 2018. 283 -877.
Huang Gao, Liu Zhuang, Van Der Muaten Laurens. Weinberger Kilian Q. Densely connected convolutional networks // Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. 4700-4708.
Iliev A., Kyurkchiev N., Markov S. On the approximation of the step function by some sigmoid functions // Mathematics and Computers in Simulation. 2017. 133. 223-234, Biomath 2014 and Biomath 2015.
10

---

loffe Sergey, Szegedy Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift // International conference on machine learning. 2015, 448 201 456.
Izhikevich Eugene M. Simple model of spiking neurons // IEFE Transactions neural networks. 2003. 14, 6. 1569-1572.
Jarrett Kevin, Kavukcuoglu Koray, Ranzato Marc' Aurelio, LeCunn Yann. What is the best multi-stage architecture for object recognition? // 2009 IEEE 12th international conference on computer vision. 2009. 2146-2111 2153.
Jinsakul Natinai. Tsai Cheng-Fa Fa, Tsai Chia-En, Wa Pensee. Enhancement of deep learning in image classification performance using xception Mathematics. 2019. 7, 12. 1170.
swish activation function colorectal polyp preliminary screening
Klambauer Günnter, Unterthiner Thomas, Mayr Andreas, Hochreiter Sepp. Self-normalizing neural networks Advances in neural information processing systems. 2017. 30.
Learning multiple layers of features from tiny images. 2009.
Larsen Richard J, Marx Morris An introduction to mathematical statistics. 2005.
Lee June-Goo, Jun Sanghoon, Cho Young- Won, Lee Hyunna, Kim Guk Bae, Seo Joon Beom, Kim Namkug. Deep learning in medical imaging: general overview Korean journal radiology. 2017. 18, 570-584.
Leshno Moshe, Lin Vladimir Pinkus Allan, Schocken Shimon. Multilayer feedforward networks a nonpolynomial activation function can approximate any function Neural networks. 1993. 6, 6. 861-86777
Tsung-Yi Yi, Maire Michuel, Belongie Serge, Hays James, Peroa Pietro Ramanan Deva, Dollár Piotr, Ziticck Lawrence. Microsoft coco: Common objects in context // European conference on computer vision. 2014. 740 740-755.
Lindsay Grace W. Convolutional neural networks as a model of the visual system: Past, present, and future /| Journal of cognitive neuroscience. 2021. 33, 10. 2017-2031.
Liu Wei, Anguelov Dragomir, Erhan Dumitru, Szegedy Christian, Reed Scott, Fu Cheng Yang. Berg Alexander C.
Ssd: Single shot multibox detector // European conference computer vision. 2016. 21-37.
Liu Yu, Chen Xun, Wang Zengfu. Wang Z Jane, Ward Rabab K. Wang Xuesong. Deep learning for pixel-level image fusion: Recent advances and future prospects // Information Fusion. 2018. 42. 158-173.
Maas Andrew L, Hannun Awni Y. Ng Andrew Y, others. Rectifier nonlinearities improve neural network acoustic models // Proc. ieml. 30, 1. 2013. 3.
Misra Diganta. Mish: A self regularized non- monmoontooicc activation function // arXiv preprint 1....11 2019.
Nair Vinod, Hinton Geoffrey E. Rectitied linear units improve restricted boltzmann machines // Ieml. 2010.
Ramachundran Prajit. Zoph Barret, Le Quoe V. Searching for activation functions // arXiv preprint arXiv:1710.05941. 2017.
Ronneberger Olaf. Fischer Philipp, Brox Thomas. U-net: Convolutional networks for biomedical image
segmentation // International Conference on Medical image computing and computer-asisted intervention. 2015. 234-241.
Rumelhart David E, Hinton Geoffrey E, Williams Ronald Learning representations by back- peeck-ropgggting errors nature. 1986. 323, 6088. 533-536 536.
Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Sean, Huang Zhiheng, Karpathy Andrej, Khosla Adiyya, Bernstein Michael, others Imagenet large scale visual recognition challenge /| International journal of computer vision. 2015. 115, 3. 211-252.
Sharma Sagar, Sharma Simone, Athaiya Anidhya. Activation functions in neural networks // towards data science. 2017. 6, 12. 310-316.
Simonyan Karen, Zisserman Andrew. Very deep convolutional networks for large-scale image recognition // arXiv preprint 100.:11409.1556.6 2014.
Tan Mingxing, Le Quoe. Efficientnet: Rethinking model scaling for convolutional neural networks // International conference machine learning. 2019. 6114

---

Ulyanov Dmitry, Vedaldi Andrea, Lempitsky Victor. Instance normalization: The missing ingredient for fast stylization arXiv preprint arXiv:1607.08022. 2016.
Weisstein Eric W. Heaviside step function // https://mathworld. wolfram. com/. 2002.
Wu Yuxxin, He Kaiming. Group normalization Proceedings of the European conference on computer vision (ECCV), 2018. 3-19.
Xu Bing. Wang Naiyan, Chen Tianqi. Li Mu. Empirical evaluation of rectified activations in convolutional network I/ arXiv preprint arXiv:1505, 150.000833. 2015.
Zagoruyko Sergey, Komodakis Nikos. Wide residual networks // arXiv preprint 16Xiv: 146. 2016.
Zhang Zhifei. Derivation of backpropagation convolutional neural network (cnn) University Tennessce, Knoxville, TN. 2016.
Zhao Zhong- Qiu, Zheng Peng, Xu Shou-tao, Wu Xindong. Object detection with deep learning: A review IEFE transactions on neural networks and learning systems. 2019. 30, 11. 3212-322..
Zhou Bolei, Zhao Hang. Puig Xavier, Fidler Sanja, Barriuso Adela, Torralba Antonio. Scene parsing through ade20k dataset // Proceedings of the IEEE conference on computer vision and pattem recognition. 2017. 633 641.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [N/A]
Did you include the license the code and datasets? [N/A]
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit, In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Introduction
(b) Did you describe the limitations of your work? [Yes] See Section 2.1, Appendix
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2.1
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 2
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Abstract (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] statistical analysis were provided using confidence interval
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "z6JwFlxiy8B", "reviewer": "Reviewer_oa51", "summary": "The paper \"Stochastic Adaptive Activation Function\" successfully establishes a link between the statistics of feature selection and the recently proposed Swish-activation function.\nBuilding upon the theoretical connection, an ASH (Adaptive Swish) non-linearity appears in the theory part.\nThe experimental part tests the new activation function Image net 2012, CIFAR-10, CIFAR-100, MS-COCO, and PASCAL VOC. The paper reports improved performance on every dataset and every network.\n", "questions": "- Line 17: The source code repositrory seems to be offline?\n- Line 8: Does ASH mean something?\n        - I found the meaning in Line 218. ASH stands for Adaptive SwisH. Perhaps this definition happens a little late in the paper?\n- Line 72: What does Swish mean? Are SiLU and Swish synonymous? Could the exact definition be briefly given here? \n- Equation 1: Is the ASH-function a real-valued variant of the modRelu introduced in\n    [arjovski16]? The authors may be interested in adding a citation and discussing the differences.\n    [arjovski16] Unitary Evolution Recurrent Neural Networks https://arxiv.org/pdf/1511.06464.pdf\n\n- Line 172: Do we have a citation for a Z-table, for convenience?\n- Table 1: What is the variance here?\n- Tables 2 and 3: Is the mean accuracy shown here? What about the variance?\n   \n\n", "limitations": "Limitations are not discussed at all. All numbers are perfect. Is the experimental section too good to be true?\n\nA discussion of limitations could include:\n- Potential problems that were observed during training or hyperparameter tuning. Perhaps some negative properties have been observed at some point while working with the ASH activation?\n- Reasons for not reporting the variance on the mean experiment values for the Image-Net experiments?\n- Perhaps some theoretical drawbacks exist that readers should be aware of?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper couples the Swish activation function to probability theory.\n- The authors find improved performance for their ASH activation function across the board.\nThe numerical results are in line with those reported in https://arxiv.org/pdf/1710.05941;%20http://arxiv.org/abs/1710.05941.pdf for CIFAR-10.\n- To the best of my knowledge, the contribution is novel. But I haven't been working looking into activation functions for some time (see confidence score below).\n\nWeaknesses:\n- The paper is hard to read at times.\n- For a statistically motivated paper, it would have been neat to see variance values alongside the reported mean accuracy in the experimental section.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "arD_qaGqu5Q", "reviewer": "Reviewer_aJG2", "summary": "The authors propose a new trainable activation function that unlike most activation functions that work element-wise, this one activates over complete layers. \n\nMore precisely, the activation function behaves like a ReLU, either activating with the input value, or cancelling to zero when deactivating. However, the activation criteria is whether the element is in the top k% of the elements in the layer.\n\nThe threshold is then controlled by a learnable parameter, making it a trainable activation function.", "questions": "The empirical evaluation shows that this activation function is valuable and it works. \nI have however a series of questions that could be interesting to analyze to better understand the proposed method.\n\nI'm thinking of the similarities of this activation with dropout. In particular, I'm wondering if the cancelation of the elements, should lead to a scale adjustment? It would seem like that could be beneficial as in dropout.\n\nNow, considering the cancelation induced by the activation function, there are big parts of the network that won't receive gradient updates if they are not in the top k%. Would it make sense to introduce something like a negative tail or a scaling factor for the elements below the top k%? Think of Leaky ReLU.\n\nFurthermore, as one would normally use these activation functions through the network. I'm wondering if there is some effect of vanishing gradients happening, or signal reduction, due to the cancellation imposed after each layer. I could imagine that if the learned parameter would cancel 90% of the signal at each layer, this could introduce a strong loss of information.\n\nI'm also wondering whether this is a new approach for signal compression?. Consider a fixed top k% parameter. This directly implies that the number of outputs after the activation is a fraction of the size of the output. This is potentially equivalent to a reduction layer.\nHere you do not get a dimensionality reduction, as for every input still different outputs could activate. But there is a compression in the amount of data reaching the next layer. If you were to control the percentage of information dropped, you could construct something like an auto-encoder and get the benefit of signal compression, without the effort needed in compressing the dimensionality. ", "limitations": "no negative societal impact", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very interesting and I believe it could have applications and connections to other areas in deep learning.\n\nThe paper is well written and the empirical evaluation shows that the proposed activation works well.\n\nThe major weakness I see, is that the authors compare their activation mostly with other non-trainable activations. However, it is known that trainable activation functions such as [1], etc., usually perform better and such comparisons are missing in the empirical evaluation.\n\n[1] Molina, Alejandro, Patrick Schramowski, and Kristian Kersting. \"Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks.\" International Conference on Learning Representations. 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WxEnunbhMei", "reviewer": "Reviewer_ecwn", "summary": "This paper proposes an rectifier based activation function where the threshold is set to the upper k% of the input tensor values.", "questions": "- How is k determined?  Is it fixed, or learnable?  Is it the same across a network, or different for each layer?  What values for k are used in the experiments?\n\n- The approximation of heaviside step function relies on a large value of alpha.  How does this affect the analysis in Equation 10?", "limitations": "- Some claims are excessive and unsubstantiated.  For example: \"our activation function exhibits outstanding performance in any deep learning applications,\" or \"the experiments empirically discovered that the Swish activation function is the best performance activation function.\"\n\n- The experiments do not show any case in which ASH is outperformed by any other function.  The implication is that it is universally the best activation function.  This implied claim is problematic because there are no repeated runs (with the exception of 3 runs in Table 1) nor any statistical significance tests to back up these results.  See checklist item 3c.\n\n- Figure 3 has no numbers on the x or y axes.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The main originality in this paper comes from determining a threshold based on a percentile from the input entries.  To my knowledge, this has not been considered before.  \n\n- Quality: The main issue with the paper is that there are no statistical significance tests for the final results.  This prevents drawing any substantial conclusions from the results, which hurts the paper's quality.\n\n- Clarity: The writing is at times difficult to follow and understand, but the mathematics appear sound.\n\n- Significance: The paper would be much more significant with repeated runs and statistical significance tests, but as-is, it is unsignificant.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LSr0VSxfLff", "reviewer": "Reviewer_djrD", "summary": "In this paper, the authors present the Adaptive SwisH (ASH) activation function, which is a more generalized form of the SwisH activation function. They present the parametric and adaptive properties of the ASH function and prove the baseline threshold of the ASH function is trainable during the training phase and that the threshold value adaptively changes according to the contexts of inputs. Additionally, they prove that the ASH function does not require heavy computations compared to some other activation functions. In terms of experiments, they show that the ASH activation function outperforms other activation functions on classification, object detection, and segmentation tasks.", "questions": "Why were only 3 models considered for each experiment (classification, object detection, segmentation)?", "limitations": "The authors addressed the fact that they did not search for the best performing parameters. Instead, they justified this by stating that allowing the network to learn these parameters on its own is more similar to how human neurons work. For future work, it would be nice to see the comparison between setting the threshold for zk and letting the network learn these parameters naturally.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- proved parametric and adaptive properties of ASH function\n- proved baseline threshold of the ASH function is trainable during the training phase and that the threshold value adaptively changes according to the contexts of inputs without heavy calculation\n- proved ASH function is a generalized version of the SwisH activation function\n\nWeaknesses\n    Overall\n    - research in the paper should be presented as a novel extension of the SwisH activation function since it is the more generalized form\n    Abstract\n    - should have placed the full name for the ASH activation function (Adaptive SwisH) in the \"abstract\" section\n    Method\n    - equation 10 needs more steps\n    Main Result\n    - Mention mathematical proofs for accuracy, sparsity, training time, and localization property as part of potential future work\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept