Entropy-based Activation Function Optimization: A Method on Searching Better Activation Functions
Authors: Haoyuan Sun
Zihao Wu
Bo Xia
Pu Chang
Zibin Dong
Yifu Yuan
Yongzhe Chang
Xueqian Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
Activation Functions
Information Entropy
Primary area: optimization
TL;DR: We theoretically prove existence of the worst activation function, propose a theoretical methodology to guide the optimization design of activation functions, and use it to derive a novel high-performance activation function.

Abstract: The success of artificial neural networks (ANNs) hinges greatly on the judicious selection of an activation function, introducing non-linearity into network and enabling them to model sophisticated relationships in data. However, the search of activation functions has largely relied on empirical knowledge in the past, lacking theoretical guidance, which has hindered the identification of more effective activation functions. In this work, we offer a proper solution to such issue. Firstly, we theoretically demonstrate the existence of the worst activation function with boundary conditions (WAFBC) from the perspective of information entropy. Furthermore, inspired by the Taylor expansion form of information entropy functional, we propose the Entropy-based Activation Function Optimization (EAFO) methodology. EAFO methodology presents a novel perspective for designing static activation functions in deep neural networks and the potential of dynamically optimizing activation during iterative training. Utilizing EAFO methodology, we derive a novel activation function from ReLU, known as Correction Regularized ReLU (CRReLU). Experiments conducted with vision transformer and its variants on CIFAR-10, CIFAR-100 and ImageNet-1K datasets demonstrate the superiority of CRReLU over  existing corrections of ReLU. Extensive empirical studies on task of large language model (LLM) fine-tuning, CRReLU exhibits superior performance compared to GELU, suggesting its broader potential for practical applications.

Published as a conference paper at ICLR 2025
ENTROPY-BASED ACTIVATION FUNCTION OPTIMIZA- TION: A METHOD ON SEARCHING BETTER ACTIVA- TION FUNCTIONS
Haoyuan Sun Zihao Wu² Bo Xia', Pu Chang', Zibin Dong", Yifu Yuan",
Yongzhe Chang" Xueqian Wang Tsinghua University Tianjin University Anhui Polytechnic University
ABSTRACT
The success of artificial neural networks (ANNs) hinges greatly on the judicious selection of an activation function, introducing non-linearity into network and enabling them to model sophisticated relationships in data. However, the search of activation functions has largely relied on empirical knowledge in the past, lacking theoretical guidance, which has hindered the identification of more effec- tive activation functions. In this work, we offer a proper solution to such issue. Firstly, we theoretically demonstrate the existence of the Worst Activation Function with Boundary Conditions (WAFBC) from the perspective of information entropy. Furthermore, inspired by the Taylor cxpansion form of information entropy fune- tional, we propose the Entropy- Activation Function Optimization (EAFO) methodology. EAFO methodology presents novel perspective for designing static activation functions in deep neural networks and the potential of dynamically opti- mizing activation during iterative training. Utilizing EAFO methodology, derive a novel activation function from ReLU, known as Correction Regularized ReLU (CRReLU). Experiments conducted with vision transformer and its variants on CIFAR-10, CIFAR- 100 and ImageNet- 1K datasets demonstrate the superiority of CRReLU over existing corrections of ReLU. Extensive empirical studies on task of large language model (LLM) fine-tuning, CRReLU exhibits superior performance compared to GELU, suggesting its broader potential for practical applications.
INTRODUCTION
Flourishing development of artificial intelligence is predominantly attributable to rapid advancements in artificial neural networks (ANNs) observed in recent years. Activation functions (AFs) play a critical role in the performance of ANNs due to their fundamental role in enabling nonlinear representations. Despite continuous development of novel activation functions and their empirical success in improving network performance, theoretical analysis towards these activation functions remain scarce in the research literature. In other words, proposal of improved activation functions is often based on empirical evidence without theoretical validations. which greatly hinders the search for better activation functions. Hence, theoretically reliable methodology on searching better activation functions holds significant value for the machine learning community and future research.
Major Questions
Is there the best activation function?
2. If the answer is no, can we give a methodology for searching better activation functions?
3. Can we provide an example to illustrate how the given methodology is applied in practice?
Our research endeavors are centered on addressing the above three questions. In our work, we initiate our exploration from the correlation between information entropy and Bayesian error rate; subsequently, we establish the connection between activation function and information entropy. To answer the first question, we derive the specific form that the worst activation function exists under boundary conditions and draw the conclusion that there is no best activation functions but only better
"Equal contribution: sun-hy23 mails muiils.singhua.edu.com 3020210034@ @jt..du..cc Corresponding authors: changyongzhe @ www.sssnggaaaac....... edw.enn wang. wwng.xg@w..innguua... www.ssigguu.eeu..oo

---

Published as a conference paper at ICLR 2025
ones. To answer the second question, propose novel method for optimizing activation functions, namely the Entropy-based Activation Function Optimization (EAFO) methodology. To answer derive novel activation function known as Correction Regularized ReLU
the third question,
(CRReILU) with the beginning of conventional ReL.U utilizing the EAFO methodology. Experiments involving the vision transformer (Dosovitskiy et al.] 2020) and its variants (Touvron al. 2021 Han ct al.
2021), conducted on 100, CIFAR-I00 (Krizhevsky et al. 2009) and ImageNet-IK (Deng
al. 2009) datasets, have consistently demonstrated the superior performance of CRReLU compared to other activation function baselines. Extensive experimental studies on the task of large language model (LLM) fine-tuning with direct preference optimization (DPO) method (Rafailov al. 2023)
also demonstrate that CRReLU surpasses GELU in performance, suggesting the wider applicability of CRReLU in practical scenarios. Moreover, the EAFO methodology also shows potential to further optimize activation functions during the iterative training of ANNs, although the specific optimization techniques remain a topic of ongoing research.
In summary, main contributions are follows:
We answer the first question: there no best activation function, but better ones. We theoreti- cally prove the existence of the Worst Activation Function with Boundary Conditions (WAFBC) from the perspective of information entropy; and starting from the worst activation function, performance of activation functions always improves.
We answer the second question: the Entropy-based Activation Function Optimization (EAFO) methodology is proposed, which provides novel perspective for designing static ac- tivation functions in deep neural networks and the potential of dynamically optimizing activation during iterative training.
We answer the third question: a novel activation function is derived, known as Correction Regularized ReLU (CRReLU), which starts from the ReLU utilizing the EAFO methodology. Experiments across mainstream architectures, datasets and tasks demonstrate that the proposed CRReLU exceeds existing activation functions, exhibiting exceptional performance.
RELATED WORK
With the development of deep learning, deep neural networks (DNNs) have gained significant promi- nence and achieved notable success across various domains. Recent advancements in the field of natural language processing, exemplified by large language models such as GPT-4 (OpenAl, 2023). LLama-3 (Hugo Touvron, 2023), and Gemini (Team. 2024), have propelled machine understanding and generation of natural language to unprecedented levels of accuracy. Additionally, deep neural networks have also achieved important applications in computer vision (Dosovitskiy al. 2020 Tou-
vron et al. 2021 Han et al., 202I). deep reinforcement learning (Schulman et al. 2017). autonomous driving( Pan al. 2024). and many other areas.
The nonlincarity of activation functions in neural networks is crucial for both enabling the efficient learning of complex patterns, and facilitating the extraction of intricate and hierarchical representa- tions from input data, thus allowing them to capture more complex relationships between input and output variables. In contrast, however, the nonlinear activation functions of deep neural networks also presents challenges during training, encompassing challenges like gradient vanishing (Bengio al. 1994), gradient exploding (Larochelle et al. 2009), and so on.
To address these challenges, researchers have explored alternative approaches for improvement, including the enhancement of activation functions. In the nascent stages of activation function devel- opment, scholars predominantly focused on rudimentary thresholding functions, initially directing their attention towards squashing functions such as the Sigmoid function and the Tanh function (Hornik) 1991). In order to mitigate the issues of vanishing and exploding gradients, various non- squashing functions have been proposed. Notably, ReLU (Hahnloser et al. 2000 Jarrett et al., 2009) Nair& & Hinton| 2010 has played pivotal role in the remarkable success of deep learning. The derivative of ReLO for positive inputs is one, thereby preventing the gradient from vanishing: however, negative values are mapped to zero, leading to two main issues: (1) The absence of information flow for negative values, known as dying ReLU; (2) The shift in subsequent layers due to positive bias maintained by activation.
2

---

Published as a conference paper at ICLR 2025
Given the aforementioned challenges, researchers have dedicated significant efforts improving the effectiveness of activation functions. The Leaky ReLU (Maas) (013)) activation function permits a small negative slope, ensuring some gradient can still be propagated even when input is less than zero. The Parametric ReLU (PReI.U) (He et al. 2015) is an extension of the Leaky ReLU,, where a is considered a learnable parameter that is learned from data rather than being predetermined. The Exponential Linear Unit (ELU) (Clevert et al.) 2016) outputs negative value when x is less than O, leading to the advantageous property of the average oulput approaching O. The Continuously Differ- entiable Exponential Linear Unit (CELU) (Barron 2017) proposes an alternative parameterization that simplifies analysis of the rectifier function and facilitates the tuning process of parameters in ELU. The Swish (also known as SiLU) (Ramachandran et al. 2017) has been shown to enhance training stability and performance in deep learning models due to smooth nature and improved gradient propagation. In Mish (Misra (2200) activation function, unboundedness of positíve values avoids the saturation led by a plateau, slight allowance for negative values enables better gradient flow, and the smoother activation function allows better information to flow deep into neural networks, thus resulting in better accuracy and generalization in performance.
MOTIVATION
In Section 2 shown that researchers have dedicated substantial efforts the exploration of improved activation functions, which are widely acknowledged to hold considerable significance for the advancement of deep learning. However, has also come our attention that proposals for these activation functions lack theoretical framework, indicating such searches are, to some extent, inefficient and aimless.
GELU(Gaussie Error Linear Unit) (Hendrycks & Gimpel 2023) was first proposed in 2016 and has since gained significant success in a variety of fields, especially with the emergence of large language models in recent years. It has been successfully incorporated into several cutting-edge neural network architectures, such as BERT(Devling et al. 2019). ViT (Dosovitskiy et al. 2020), GPT-4 2023), and so on, demonstrating its versatility and effectiveness. In the work conducted by Lee (2023) insightful mathematical properties of the GELU are finally unveiled, including its differentiability. boundedness, stationarity, and smoothness. Hence, it is often the case that superior performance exhibited by novel activation functions frequently lacks mathematical explanations for their observed enhancements. Understanding may merely limited the fact that it exhibits improved performance, which hampers exploration for better activation functions and interpretability of neural networks.
In light of the aforementioned challenges, our work endeavors to propose methodology for searching better activation functions, not only enabling the discovery of improved activation functions but also elucidating the reasons behind their superior performance the same time.
4 METHODOL.OGY
4.1
PROBLEM SETUP
4.1.1 BAYESIAN ERROR RATE AND INFORMATION ENTROPY
A deep neural network can be simplified as comprising a feature extraction layer, which is subse- quently followed by a fully connected layer for final classification. From a probabilistic perspective, in binary classification, the feature extraction layer can be conceptualized as transforming the shape of mixture distribution, thereby enabling the final fully connected layer to separate two distributions with hyperplane. Hence, the more overlapping two distributions are, the higher Bayesian error rate and the worse classification performance. Furthermore, a lower information entropy corresponds to higher likelihood of forming two distinct peaks (i.e. the smaller classification uncertainty, the easier to classify); and an increase in the overlap between two distributions also leads to the increase of information entropy (i.e. the greater classification uncertainty, the harder to classify). In addition, the above statements can be extended to multi-class classification, and further elaboration is omitted here. To better understand the statement that lower entropy indicates better classification, we conduct further discussion in Appendix
3

---

Published as a conference paper at ICLR 2025
4.1.2 ACTIVATION FUNCTION AND INFORMATION ENTROPY
Assuming the inverse function of the activation function is y(x), and the activation function is monotonically increasing. Many previous activation functions, such as Sigmoid and Tanh (Horiik) 1991), satisfy the assumption that the function has an inverse function in entire definition domain. Furthermore, when an activation function fails to meet the assumption, we can transform the part of such function satisfying this assumption, as is the case with the positive part of ReLUU.
Then we set data distribution before passing through the activation function obeys the distribution p(x). Thus, data distribution after passing through activation function is q(x) = ()xx)) (x), where y' (x) represents the derivative of yxx). Hence, we can express the information entropy as:
H(((x)) = - ((x) (xg = (ako))/gttaooo/attoto((oooootto-
(x)) = G6)((x, (x))
Therefore, the information entropy can be deemed as functional, which takes a function y(x) as input and produces real number output.
4.2 WORST ACTIVATION FUNCTION WITH BOUNDARY CONDITION (WAFBC)
Answer to Question 1: There is no best activation function, but only better ones. Furthermore, there exists the worst activation function, namely the Worst Activation Function with Boundary Condition (WAFBC).
Firstly, we would like to determine the extremum (whether maximum or minimum) of the functional H))))))). For further deductions, taking the simplest functional into consideration, e.g. setting H(y(x)) G (y' (x)),)(x) dx.
In order to research the influence brought by variations of function y(x), we apply a small perturbation en (x) to function y(x), and then the functional H (v(x) takes the form as:
(((()))++((()())) H (y(x) + en())) = / + (x),y(x) + +x))
We apply Taylor expansion to functional IH ()(x)) - (x)): we can obtain the following equation:
aG
ac
H()(s)+
(()())))()))))))) + 5y (x(x) Jy (xx) + 0(E) dr
(1)
--(((())) - & / [ σy (xx + 2y (x)) dx ((e)
a
a
the data distribution after passing through activation function. We easily get that for the inverse function y(x) of activation function, when x approaches the lower bound (e.g. the initial activation function value approaches lower bound), (xx) should approaches negative infinity; and when approaches the upper bound (e.g. the initial activation function value approaches upper bound), y(x) should approaches positive infinity. And for the sake that en (x) a small perturbation applied (x). we can draw the conelusion that (x) must be 0 at the boundaries.
As illustrated in Section 4.1.2 q(x) = p(x))=()(
Utilizing the method of integration by parts and boundary condition towards Equation we can derive the following results:
/
aG Jy n (x)xxx /
GG Ty dn (x)) ((x) σγ' x
a
d ( (GG
dx = - (xx) -f((>)
d IG dr Jy
(x))
dx
dx Jy
Thus, H + (x)) has the following expression:
2G aG δy
H (x)x + (x)) = H()(x)) +
n ((x)dx+0(4
de σン
In analogy to the extremum of ordinary functions, is expected that the first-order term should be 0 the extremum point. Such requirement for arbitrary (xx leads to the Euler-Lagrange equation:
aG a
= 0
(2)
dx σy
dy
4

---

Published as a conference paper at ICLR 2025
Proposition 1. If G is independent of x. L.e. G G(y,y), based the Euler-Lagrange equation expressed in Equation 2 then we have:
aG - y oy
G
=C
Detailed proof of Proposition 1 can be seen in Appendix A
Substitute G p(o(=))) (x)) into Equation B and perform the calculation, the final result is:
dx p({xx)) =c
Integrating both sides the equation simultaneously, the final solution is:
x= x ply)dy+
Based on the solution we get in Equation 4 for the sake that y(x) is the inverse function of the activation function, first integral equation can finally solved to obtain form the activation function as:
f(x) (()) p(t)dr +
(5)
where Ci and C2 are two constants based on the upper bound and lower bound of activation function. Equation [ shows the analytical form of the worst activation function with boundary condition. We this form in Appendix B Through the above derivation, extremum of
provide further discussion the functional is determined. Furthermore, we would like to deduce whether a maximum value minimum one. Applying Legendre condition to the functional extremum, then we have:
Gyy p(y(x))
<
Therefore, the derived extremum is a maximum extremum, and is a global maximum extremum actually, meaning the deduced activation function has the worst performance. However, the minimum of the functional is needed if would like obtain the best activation function. Nonetheless, based on calculation, the actual situation is that this functional only has global maximum but global minimum exists. Hence, there is no best activation function, but only better activation onnivationfunctionn In this scenario, WAFBC represents global maximum of the functional, indicating that the performance of activation functions consistently improves from WAFBC to any alternative activation functions.
4.3 ENTROPY-BASED ACTIVATION FUNCTION OPTIMIZATION (EAFO)
Entropy-buut based Activation Function Optimization (EAFO) methodology is proposed, which provides novel perspective for designing static activation functions in deep neural networks and the potential of dynamically optimizing activation during iterative training.
Answer to Question 2:
Let's reconsider the Taylor expansion of the functional
ac d
aG dy dr σу
(Ty(x)++Tm(6) H(y+)) +
n (x))
To minimize the information entropy of novel activation function, is advisable reduce the first- order term of Taylor expansion. In order to ensure that the information entropy of novel activation function has been indeed reduced, we would like to set (x) as the opposite sign to dx dy
ag AG
dy
which means we set:
- aG d JG Ty dr σÿ
n (x)
(6)
Substitute the analytical form of functional G()((x) into Equation we can derive the following equation:
6 perform the calculation,
(xx = -- - --(o(>)))) ('xx) y(x)
+ +((6))))))))
y(x)
5

---

Published as a conference paper at ICLR 2025
where p(x) is the probability density function (PDF) of data distribution before passing through the activation function: (x(x is the first order derivative of PDF; y(x) is inverse function of the activation function; y' (x) is the first order derivative of y(x); y" (x) is the second order derivative of yxx).
As a result, we have derived a correction term that is capable of decreasing information entropy, obtain the inverse function of the optimized activation function, denoted as g(x) y(x) + (x).. Finally, the optimized activation function can be obtained by deriving the inverse function of gxx).
expressing its general form in Equation 7 Subsequently,
In summary, we express the theoretical EAFO methodology as follows:
EAFO methodology outline
1) Utilize Equation] 7 and derive correction term (xx) given data distribution p(y) and inverse function of activation function y(x).
2) Sum the correction term with the inverse function to obtain the inverse function of the
optimized function, i.e. gxx) = -(xx)+ (x).
3) Derive the rigorous or approximate inverse function of g(x), yielding the optimized activation function.
Furthermore, EAFO methodology has also shown the potential of dynamically optimizing activation during iterative training. We are acknowledged that activation neural networks with Multi-Layer Perceptrons (MLPs) architecture is typically fixed. Recent studies, such as work done by Liu et al. (002)), have suggested the optimization of activation innovative network architectures (such Kolmogorov-Amool Networks). Furthermore, across true data distributions p(y). utilizing EAFO methodology. we may continuously optimize activation y(x) practically under Multi-Layer Perceptrons (MLPs) architecture with numerical methods. Moreover, in theory, it is feasible to optimize activation functions using methods such as gradient descent optimization of the information entropy functional; however, we are also aware that this would result in an explosion of computational complexity in large neural networks, which calls for practically efficient algorithms. Hence, at the present stage, the EAFO methodology is still the theoretical stage, providing guidance for calculating the analytical form of better activation functions.
4.4 CORRECTION REGULARIZED RELU (CRRELU) FROM RELU TO BETTER
Answer to Question 3: A novel activation function is derived, known Correction Regularized ReLU (CRReLU), which starts from ReLU utilizing the EAFO methodology.
As illustrated in Section 4.2 it is theoretically true that the worst activation function exists, and we can determine its exact form. Actually, beginning with the worst activation function, the value of the functional G consistently decreases, indicating an improvement the performance of activation function. This reveals the feasibility of searching improved activation function, which constitutes the crux of "optimization". In Section 4.3 EAFO is proposed as the optimization methodology. Hence, we easily think of optimizing from WAFBC to get a better-pe activation function. While it is true that such an idea is feasible, we also observe that WAFBC itself takes the form of a variable upper bound integral, which yields a complex form of n (x) and renders the deduced result not practically significant. Moreover, commencing optimization from WAFBC also leads to sluggish advancement. Therefore, from the practical perspective, we are inclined to start from an activation function that already demonstrates relatively good performance.
Here, we would like take ReLU (Hahnloser et al., 2000 Jarrett et al., 2009 Nair Hinton 2010) as the beginming, and show the process of finding a better activation function. Before the deduction, we also notice that ReL.U is lack of an inverse function over the entire domain. In this section, we would like to utilize following strategies for mitigating the aforementioned dilemma: the initial activation function only necessitates an inverse function specific regions where is required; and when encountering parts without an inverse function, we may employ practical approximations. Therefore, we initially examine the region where x is positive the case of ReLU, As shown in Equation 7 the derivation of correction term n (x) only requires original distribution p(y) and inverse function of the activation function y(x). Knowledge of activation function is easily available, whereas original distribution remains unexplored. However, in real experiments, original distribution of experimental

---

Published as a conference paper at ICLR 2025
data would surely exhibit substantial degree of morphological variability, thus lacking perfect analytical form. Hence, we assume the situation is that networks are large enough, according to the Central Limit Theorem, the data processed by them can be approximated
Gaussian distribution in MLPs (Williams) 1996] Lee et al. 2018 Park et al. 20200 Gao et al. 2023) and CNNs (Huang et al. (021). Certainly, such assumption may not always hold in networks of real experiments and modern architectures (such as Tranformers). Nevertheless, approximation of the exact solution for inverse function and introduction of the learnable parameter could have significantly mitigated the impact of such assumption, which can also be demonstrated by the insensitivity of CRReLU to data distribution shown in [5c
Now, let's consider the derivation from ReLU to CRReLU. For the sake of concise representation, we rewrite the data distribution and the derivative of data distribution as:
p()) = c p'6)) ye
Furthermore, ReLU has mathematical function defined as y = when x positive, meaning we have y(x) =x y and yxxx) = 0. Therefore,
d'(y(x)) p'(xx) -C- == -C-ee
Ultimately, incorporating p'(y) xx $f ,x(x) and
yx)) aandy"")) into Equation 7 we can obtain:
(xx) - 2
Furthermore, we make constant C as a learnable parameter & with the purpose of enabling self- optimization in networks. According to EAFO methodology, we can get the inverse function of revised activation function as follows:
gxx) = Exe
0
(8)
Finally, the optimized activation function CRReLU can be obtained by deriving the inverse function of g(x). However, obtaining the inverse function of Equation 8 presents challenge using conventional methods; as a consequence, we use the following function as a form of practical approximation.
f(x) = Exe
x x
We show the rationalization and reliability of utilizing Equation of Equation in Proposition
9 aas the approximate inverse function
Proposition 2. Known g(x) = exe between 8 (x)) and x is bounded with le- +.0.e
? fx) = x exe
2, for x O, the absolute value of error
Detailed proof of Proposition 2 can be seen in Appendix C
As illustrated in Section 4.2 (x) is the small perturbation; hence, from a theoretical perspective, (x) can be set as an infinitssimal Furthermore, given the knowledge that (x)) is bounded function, we can deduce that & is also an infinitesimal. Therefore, absolute value of error between g (f)x)) and is an infinitesimal of higher order. In practice, we typically initialize a small value, such 0.01 (as described in Section 5), implying that the absolute value of error a small value.
Finally, let's consider the part where is negative, When is negative, the inverse function ReLU can be visualized
ray emanating from the origin and extending to infinity, possessing an infinite slope; and when x is positive, constitutes a ray with the slope of Hence, the correction term solution for both positive and negative values of x can be considered identical, differing only by constant C. In Equation D and Proposition 2, it is shown that incorporating the correction term into a linear activation function have beneficial effects by reducing the information entropy. Therefore, we can obtain the full form Correction Regularized ReLU as:
f(x) = m0. Exe
(10)
Lipschitz Continuity Analysis of CRReLU. Lipschitz continuity constitutes stronger form of the continuity, which imposes an upper bound on the rate of variation of a function. We would like to begin with the defination of Lipschit functions.
7

---

Published as a conference paper at ICLR 2025
Definition 1 (Lipschitz Continuous Function(Khromm & Singh 2024 ). For function f: R - B defined on some domain domíf) [ R2, is called C-Lisschitz continuous, C > O, w.E.. some a-norm if. V x.y E dfm(f): ( S -C|l->>lle
We are interested in the smallest C that makes the above condition hold. This is what called the (true) Lipschitz constant of function f. Lipschitz continuity of an activation function is crucial for ensuring well-behhaved optimization landscape, thereby promoting efficient convergence during training; furthermore, a Lipschitz constant closer to indicates stronger Lipschitz continuity. Through derivation, we can calculate Lipschitz constant of GELU <Le 2023), SiL.U, Mish and CRReLUU
Fact 1. Lipschitz constant of GELU is 1.08;; Lipschitz constant of SiLU is 1.100; Lipschitz constant of Mish is 1.089.
Fact Under mild assumptions, Lipschitz constant of CRReLU is max + E, 0.446e).
Based on this, we further obtain the recommended initialization range for e of CRReLU in Corollary ! Corollary 1. In order to make Lipschitz constant of CRReLU remains lower than that of GELU, the range of (-0.188, 1a2,1888,0000 0.084). We recommend setting the initial value of within this range.
Detailed proof of Fact [ Fact Corollary can be seen in Appendix E. further discussion on the initialization of and training stability in Appendix H
Moreover, we conduct
Finally, we provide further details of CRReLU in Appendix D including python-like pseudocode of CRReLU in Appendix D.I and further discussion properties of CRReLU in Appendix D.2
5 EXPERIMENTS
Datasets. In experiments of image classification task, we adopt three datasets, ordered as CIFAR-10. CIFAR-100 (Krizhevsky et al. 2009) and ImageNet-1K 1K (Deng al. (009) in terms of the number of classification categories. In experiments large language model (LLM) fine-tuning task, we employ
two human preference datasets: SHP (Ethayarajh et al. 2022) and HH (Bai et al. 2022).
Baselines. We conduct experiments comparing the performance of CRReLU with several typical
existing corrections of ReLU as illustrated in Section 2 and Section PReLU (He et al. 2015). ELU
(Clevert et al.) 2016). CELU (Barron 2017). GELU (Hendrycks & Gimpel 2023), Swish (SiLU)
(Ramachandran et al. 2017) and Mish (Misra 2020
Experimental hyperparameters. For all transformer- based architectures, we directly set E to 0.01 without further optimization. Detailed experimental hyperparameters are provided in Appendix N
5.1 TASK OF IMAGE CLASSIFICATION
We conduct all experiments within CIFARIO and CIFAR100 on 4xRTX3090 and those within ImageNetIK 4XNVIDIA L20 for 100 epochs using the AdamW optimizer with weight decay of 0.05, truncated normal initialization (we provide further discussion on the reason for abandoning pre-trained initialization method in Appendix J.1), gradient clipping norm of 1.0, cross entropy loss function, and cosine annealing learning rate scheduler with linear warm-up. All experiments are conducted three runs, we report the mean and standard derivation.
Experiments of VITs on CIFAR-10 and CIFAR-100. Vision Transformer and its variants possess sufficiently complex structure and representational capability, garnering widespread attention from the community. Moreover, the assumption of Gaussian distribution has been theoretically proved as reasonable for sufficiently large MLPs and CNNs; however, the distribution of data under attention mechanism of transformers remains unexplored. Hence, we select vision transformer and its variants as our test model order to further investigat the insensitivity of CRReLU to data distribution. Phase of experiments on CIFAR- and CIFAR-100 involves the selection of Vision Transformer (ViT) (Dosovitskiy et al. 2020). Data-Efficient Image Transformer (DeiT) (Touvron et al. 2221 and Transformer in Transformer (TNT) (Han al. 2021). We report the top-one accuracy CIFAR-10 in Table and CIFAR- 100 in Table demonstrating CRReLU outperforms other existing corrections of ReLU on the CIFAR datasets.
Experiments of VITs on ImageNet-1K: ImageNet-IKK dataset poses a significant challenge to the information processing capability of neural networks due to its large image size and extensive range
8

---

Published as a conference paper at ICLR 2025
Table 1: Test accuracy of experiments conducted CIFAR-10 for 100 epochs.
Top-one Accuracy
GELU
ELU
PReLU
CELU
SiLU
Mish
|RRRELU
CIFAR-10 10
ViT-Tiny
70.44.0. 66.4 78.0 0.6 66.5+ 0.6 0.3 68.7--0.3
0..7
CIFAR- 10
Deiy-Tiny
72.440.7 67.610.6 75.4=0.1 67.7.10.8 69.910.55 70.2
77.0=0.3 73.7::0.5 69.5 75.8=0.3 68.7:10.2 71.1 71.610.8
CIFAR- 10
TNT-Small
7..9
Table 2: Test accuracy of experiments conducted on CIFAR- 100 for 100 epochs.
Top-one Accuracy
GELU
ELU
PReLU
CELU
SiLU
Mish
|RRRLU
CIFAR- 100
ViT-Tiny 32.6=0.88 28.9:0.11 43.2 28.9 31.2=0.66 30.6+0.8
46.6:10.6
CIFAR-100
DeiT-TTiny 46.6+00.9 40.5+0.5 50.0+0.5 40.5 43.5=0.6 43.
50.7++0.1
CIFAR-100
TNT-Small
47.5+0.88 43.6++..3 49.0++0.7 43.0+H0.5 45.0=0.9 45.5++0.8 50.9++0.4
Table Test accuracy experiments conducted on ImageNet-IK for 100 epochs.
Top-one Accuracy
GELU
ELU
PReLU
CELU
SiLU
Mish CRReLU
ImageNet-1K, ViT-Tiny 53.9 37,210.6 56.8+0.3 37.610.5 46.1 46.9 57.5
ImageNet- K Diny--iny 61.7++0.4 49. +9.7 60.8+0.4 48.9+0.8 58.5+0.7 58.9+0.3 6..6+0.2
of classification categories. Hence, we further conduct experiments on ImageNet-IK with Vision Image Transformer (DeiT) (Touvron et al. (021). We report the top-one accuracy on ImageNet- 1K in Table 3 (and we provide further discussion on the results of DeiT in Appendix (.2).
Transformer (ViT) (Dosovitskiy et al. 2020) and
Additional Experiments on Architecture and Dataset. In Appendix F we conduct experiments on
ConvNeXt and EuroSAT to verify generalization of CRReL.U to network architecture and dataset.
Entropy Analysis across Network Layers. In order to provide more insights on CRReLU's behavior, we would like to further compare the post-training entropy of neural networks trained with CRReLU and GELU. We apply post-trained ViT-Tiny with CRReLU and GELU on ImageNet1 K. By randomly selecting same ten batches of images from ImageNetIK: K, we compute the information entropy after each of the 12 layers. We present the mean and standard deviation of the results in Table 4
Table 4: Entropy calculation after activation on 12 layers of post-trained VIT Tiny ImageNetIK.
Layer
1
2
3
4
5
6
CRReLU 7.594
7.598+10.00 7.595 7.59910.003 GELU 7.536:10.046 7.541
7.592 7.573
7.584=10.004
7.501
7.580
7.583+40.004
Layer
7
8
9
10
11
12
7.557+:00005 7. 7.540+:00005 CRReLU 7.572:10.005
7.523 0.0
7.498
7.461
GELU 7.5855+0.004 7.58540.004 7.583+0.004 7.580 7.577 7.560=0.004
From the results presented above, it is shown that for GELU, the entropy after 12 layers of activation exhibits an overall increasing trend; conversely, CRReLU demonstrates a general declining trend. Furthermore, note that the reduction in entropy for CRReLU between layers 1 and 6 is not signifi- cant, whereas marked decline observed from layers 7 to 12. Moreover, we conduct experiments under mixed activation function (GGELU+6CRRELLU) with results presented in Appendix G
5.2 TASK OF LARGE LANGUAGE MODEL (LLM) FINE-TUNING
In order to further validate the effectiveness of CRReLU on larger networks and generalization to richer range of applications, we further perform supplementary experiments on L.L. fine-tuning task. We employ the Direct Preference Optimization (DPO) (Rafailov et al. 2023) method to fine-tune GPT-2 (Radford et al. 2019) on Stanford Human Preferences (SHP) dataset (Ethayarajh et al. 2022)
and Anthropic HH dataset (Bai et al. 2022). The parameter number of GPT-2 is 137 M. a relatively modest magnitude, hence we conduct full tinettunigg instead of LoRA- --s-e one on 2 3090090. Firstly, carry out supervised fine-tuning (SFT) with the purpose of mitigating distribution shift between the true reference distribution which is unavailable, and the reference policy utilized by DPO. Subsequently, we separately set the penalty coefficient B as 0.1, 1, 2, and 5, in order compare the

---

Published as a conference paper at ICLR 2025
Table 5: Metrics comparison between CRReLU and GELU in the task of LLM fine-tuning.
Evaluation Metrics
Evaluation Margin Reward
Evaluation Accuracyt
Evaluation Loss
CRReLU
0.1428 0.0002
0.6209=0.0001
0.6476
B = 0.1
GELU
0.0000=0.0003
0.6197++0.0001
0.6480--000000
CRReLU GELU
0.4627=0.0007
0.5757=0.0001
0.9202=-0.0002
β =
0.4560=000006
0.5729+000003
0.9387++000008
CRReLU
0.7757=0.0021
0.5631++0.0033
1.4610-10.0008
[ ==2
GELU
0.0075=000015
0.5006=10.0001
1.4814:10.0055
CRReLU
1.8473
0.5635:10.0002
3.2677:10.0006
B=5
GELU
0.0069
0.5568+10.0003
3.3034.10.0021
performance of CRReLU and GELU under different penalty coefficients, and then execute DPO. We report mean and standard deviation of evaluation metrics across the fine-tuning process in Table which demonstrates that CRReLU generally outperforms GELU in LLM fine-tuning task.
DISCUSSION
Pursuit of better activation functions has been a longstanding and fundamental topic in the realm of machine learning. However, prior research has consistently concentrated on empirical search, without an emphasis understanding the underlying mathematical mechanisms. This work aims to offer proper solution to such issue. Our investigation into the relationship between activation functions and information theory concepts reveals that information entropy can be represented Existence of the worst activation function with boundary condition (WAFBC) furnishes a solid theoretical basis for exploring better activation functions. In the process of solving WAFBC, we draw inspiration from the Taylor expansion form, leading us to propose Entropy-based Activation Function Optimization (EAFO) methodology. EAFO methodology presents a novel perspective for designing static activation functions in deep neural networks and shows the potential of dynamically optimizing activation during iterative training. Utilizing EAFO methodology, we derive novel activation function from ReLU, called Correction Regularized ReLU (CRReLU), Experiments involving image classification task and large language model (LLM) fine-tuning task demonstrate that CRReLU is comparable to surpasses existing corrections of ReLU. Overall, the EAFO methodology provides numerous promising avenues for future research on activation functions, and the CRReLU introduces novel addition to the set of high-performing activation functions.
functional.
Limitations and Future Work, Our findings raise several important questions for future work. Firstly, how can EAFO framework be systematically generalized to non-invertible activation func- tions? In the initial setting of EAFO methodology, the choice of activation function is restricted to those with invertible counterparts. One potential approach addressing this problem entails employing the Lebesgue integral form instead of the original Riemann integral utilized in the entropy calculation. Secondly, how effectively implement activation function iteration optimization during neural network training? Notwithstanding the demonstrated feasibility of iterative activation function optimization during neural network training, it currently hindered by the high computational complexity, particularly in large-scale neural networks. Applicability of the EAFO methodology to optimize activation in alternative network structures, such as Kolmogorov- Arnold Networks (KANs), also deserves further in-depth research. Therefore, the development of practical and efficient algo- rithms is an exciting direction for future work. We provide further discussion on dynamic optimization in Appendix K Moreover, although previous work has theoretically established the reasonability of Gaussian distribution assumptions for MI.s and CNNs, and we have empirically demonstrated superiority of CRReLU in modern architectures like transformers. the theoretical robustness under broader range of distributions is still worthy of further research. Additionally, since there have already been more than 400 activation functions, it worth exploring greater depth how to theoretically rank them; and we provide further discussion on activation function ranking in Appendix L Finally, while we have empirically validated the exceptional performance of CRReLU on image classification task and large language model fine-tuning task, its performance on other tasks remains to be explored (we provide further discussion on large language model inference task Appendix M). thereby warranting further investigation in the future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partly supported by the National Natural Science Foundation of China No.62103225), Natural Science Foundation of Shenzhen [No.JCYJ202380071 1604008), Natural Science Founda- tion of Guangdong Province NN..2024A151500000 and National Key Research and Development Program (N0.2022YFB470440)),
REFERENCES
Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation, 2020.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauma Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Jonathan T. Barron. Continuously differentiable exponential linear units, 2017.
Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
Drork-Arné Clevert Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 248-255, 2009.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11999 2020.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information, 2022.
Tianxiang Gao, Xiaokai Huo, Hailiang Liu, and Hongyang Gao. Wide neural networks as gaussian processes: Lessons from deep equilibrium models, 2023.
Richard Hahnloser, Rahul Sarpeshkar, Misha Mahowald, Rodney Douglas, and H. Seung. Digital selection and analogue amplification coexist in cortex-inspired silicon eireuit. Nature, 405: 947-51, 07 2000.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2226, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks, Neural Networks, 4: 251-257, 1991.
11

---

Published as a conference paper at ICLR 2025
Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo. Convolution- distribution assumption: Rethinking the criteria of channel pruning, 2021.
Thibaut Lavril etal. Hugo Touvron. Llama: Open and efficient foundation language models, 2023.
Kevin Jarrett, Koray Kavukcuoglu, Marc' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? 2009 IEEE 12th International Conference on Computer Vision, pp. 2146-2153, 2009.
Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. The Tweifth International Conference on Learning Representations, 2024. URL https:://ppereevieeeee net.
net/ Eorummid http://penneeeeee..ttteooouaaaaa |WsWD8zUh
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference on Machine Learning, pp- 5562-5571. PMIR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Vladimír Kunc and Jiff Kléma. Three decades of activations: A comprehensive survey of 400 activation functions for neural networks. arXiv preprint arXiv:2402.09092 2024.
H. Larochelle, Yoshua Bengio, Jérôme Louradour, and Pascal Lamblin. Exploring strategies for training deep neural networks. J. Mach. Learn. Res., 10:1-40, 2009.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes, 2018.
Minhyeok Lee. Gelu activation function in deep learning: A comprehensive mathematical analysis and performance, 2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-11986, 2022.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks, 2024.
Andrew L. Maas. Rectifier nonlincarities improve neural network acoustic models. 2013.
Seyed Iman Mirzadeh, Keivan Alizadeh- Aaizddeh-Vahid, Sachin Mehta, Carlo Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Diganta Misra. Mish: A self regularized non-monotonic activation function. In British Machine Vision Conference, 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
OpenAl. Gpt-4 technical report, 2023.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vip: Vision language planning for autonomous driving, 2024.
Daniel S. Park, Jaehoon Lee, Daiyi Peng, Yuan Cao, and Jascha Sohl-Dickstein. Towards nngp-guided neural architecture search, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.
12

---

Published as a conference paper at ICLR 2025
Jorge M Santos, Luis A Alexandre, and Marques de Sá. The enror entropy minimization algorithm for neural network classification. int. conf. on recent advances soft computing, pp. 92-97, 2004.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Luis M Silva, Joaquim Marques de Sá, and Luis A Alexandre. Neural network classification using shannon's entropy. In Esann, pp. 217-222, 2005.
Gemma Team. Gemma: Open models based on gemini research and technology, 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & amp; distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Christopher K. I. Williams. Computing with infinite networks. In Neural Information Processing Systems, 1996.
Aolin Xu and Maxim Raginsky. enformation-theoreee analysis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017.
Jirong Yi, Qiaosheng Zhang, Zhen Chen, Qiao Liu, and Wei Shao. Mutual information learned classifiers: An information-theoretic viewpoint of training deep learning classification systems. arXiv preprint 2rXv:22299.0055 2022.
13

---

Considering the total differential of Published as a conference paper at ICLR 2025
A
PROOF OF PROPOSITION 1
Proof. From Equation
2 we know that:
aG dx oy dy
ac
=0
dG
aG dx aG dy
a
dy
dG
aG
aG
(vxx))
+
y +
δy y
dx
dx dx dy dx
oy dx
dx
dy
Thus, we have:
d
dG
aC
d
dc
dr
σy
dx og
dG
aG
dG
d
aG
(1
+y
dr
dy
dx
dr
d ()
00
d
dx
dx
dr
d
OG
G (V/)).))
dx
dx
Therefore, know that
aG
G
=0
dx dx
dy
For the sake that G is independent of x, then we have that
xc dx
0. Hence,
d aG
G y
dx
Oy
Finally, we can draw the conclusion that:
2G G G- σy
C,
which completes the proof.
B FURTHER DISCUSSION ON WAFBC
Let's take several typical boundary conditions into consideration. Firstly, setting f(x) approaches when x tends to positive infinity; and f'xx approaches O, when x tends to negative infinity. Therefore, the solution takes the form of cumulative distribution function (CDF), which can be expresses as:
f(x) f p(r)dr
Similarly, fixing the difference between the upper and lower bounds of the activation function to be and making the activation function symmetric about the origin, the form can be written as:
f(x) ((x)=e( p(1)ds
Furthermore, in the event that the input data distribution is assumed to be approximately uniformly distributed, the worst activation function can be approximated as a linear function. Were it to approximate the input data distribution as normal distribution, then the form of the worst activation function would be closer to Sigmoid and Tanh. Additionally, the WAFBC possesses some intriguing properties, for example, it inherently has upper and lower bounds, which explain why bounded activation functions like Sigmoid and Tanh do not perform as well as unbounded functions like ReLU. We show the comparison of function curves in Figure I and Figure 2
14

---

Published as a conference paper at ICLR 2025
Comparison between Sigmoid and Standard Normal CDF
1,0
Standard Normal
Sigmoid
0.8
Function value
0.6
0.4
-2
x value
Figure 1: Comparison between Sigmoid and standard normal CDF
Comparison between Tanh and Standard Normal CDF multiplied by e
Standard Normal CDF multiplied
Tanh
1.0
Function value
0.5
0.0
-0.5
-1.0
-2
2
value
Figure Comparison between Tanh and Standard Normal CDF multiplied by e (has been transformed to achieve symmetry about origin)
C PROOF OF PROPOSITION 2
Before the proof of Proposition 2 we would like to show four facts without proof.
Fact 3. f(x) = 2
bounded function, and range of the function is [-e 1 4,e 1
Fact 4. f(x) xxee= is bounded function, and range of the function is [0,e 1.
Fact 5. f(x) = -2e- 222 is bounded function, and range of the function is [-e 2
Fact 6. Vr R, - -
We now commence the proof of Proposition 2
15

---

Published as a conference paper at ICLR 2025
Proof. Substituting the analytic expression into the formula and performing algebraic simplifications, we can obtain:
-x(x=exe
8 8(f(x)) = (x+exe (x) = x exe 4 -8 xx extexe $)e -t
(e)
x-exe
- - e -
xtere
2 -
x Ex
xexe"
1
2px0 t22
)1
- (1(mte
2
x
Exe
2
1
E²xe 2
Thus,
20re 2
+022000
- xtere $)² x)
-
1
1 - le(f(x)) -x = Exe I
E xe =e
2Exe 2
-
Exe
1 -
3
2Exe
2
+ -
<
Ex3
7
2
22
x2x2-=0 xx
e 22
1 + -
Exe
Exe
e 1 g² 0.5e -à de
The first inequality is established owing to Fact and the fact that when x is positive, the second term of absolute value be positive. The second inequality is established owing Fact The third inequality is established owing to Fact 4 and Fact Hence, we can draw the conclusion that the
absolute value error between (x)) and x is bounded with e e- - the proof.
which completes
D FURTHER DETAILS OF CRRELU
D.1 CORRECTION REGULARIZED RELU (CRRELU) PSEUDOCODE
Algorithm 1: Correction Regularized ReLU (CRReLU) Pseudocode
import torch import torch. import torch. nn. functional
class CRReLU (nn |onuule)))):
def init (self, 1r=0.01)
super (CRReLU, self) init ()
tensor self nn. Parammete tenso. (1x))
def forward (self,
F. relu (x) +self. 1wwx*torch.ccommmmmmom. exp exxxxx*+2xxxxx2 (-x**2/2)
16

---

Published as a conference paper at ICLR 2025
D.2 FURTHER DISCUSSION ON PROPERTIES OF CRRELU
We show the function curves with different & values for CRReLU in Figure 3 As depicted in the figure, existence of the correction term in CRReLU brings several good properties. It allows propagation of gradient when input is less than zero, serving to alleviate the dying ReLUU phenomenon certain degree; simultaneously, as x approaches negative infinity, CRReLU also converges thereby guaranteeing sparsity of models in the negative part.
CRReLU with different value
8=0.1
2.0
0=3.
00555
1.5
Function value
1.0
0.5
0.0
X value
Figure 3: CRReLU with different value
E
PROOF OF LIPSCHITZ CONTINUITY ANALYSIS
Remark 1 LLee 2023
Lipschitz constant of GELU is 1.084.
Proof of sketch. Firstly, compute the derivative of the GELU function as:
dGELU(x)
+
dx
22
To find a constant C > such that for all x R. we have that (GELU(X
C.
dx
Using the second derivative establish tight bound on the derivative. The second derivative of the GELU function is:
d GELUU(
(2-x²)
dx
V2T
Setting the second derivative equal to zero and solving for x, we obtain critical points at x /2 and V2. Computing the first derivative at 22:
dGELU(x)
dx L= 1.084
Thus, is found that derivation of GELU bounded by 1.084, proving its Lipschitz continuity. Remark 2. Lipschitz constant of SiLU is I.100.
Proof of sketch. In the following calculations and derivations, we refer to the derivation in Remark and make appropriate omissions. Firstly, we write the form of SiLU as:
SiLU(x) x
17

---

Published as a conference paper at ICLR 2025
We compute the derivative of the SiLU function as:
dSiLU(x) (x+ 1)+1
dx (1+e-1)2
and the second derivative as follows:
( e-(-e'(xx22 (2) +x+
dx²
(1+er)s
We solve the transcendental equation s through python program, have x1 -2.3994 and X2 2.3994. Furthermore, have the derivative of the SiLU is bounded in the range of [-0.100, 1.100J. Hence, Lipschitz constant of SiLU is 1.100.
dx²
Remark 3. Lipschitz constant of Mish is 1.089.
Proof of sketch. Form of the Mish function, its derivation and its second derivation are performed as:
2 22e
Mish(x) x - +2 +2
dMish(xx e'14(x 1) 2 + 6xx + e (4x
dx
(e2t -2= +22
MMsish(x) (3'2 2) 2e (x 1) - + 2x
dx2
(e2x + 2e + 2)
Similarly, we solve the transcendental equation Mishh(( 0 through python program, we then dx²
have xı -.25664 and x2 1.4906. Morever, the derivative of the Mish is bounded in the range of [-0.113, 1.089]. Hence, Lipschitz constant of Mish is 1.089.
Remark Under mild assumptions, Lipschitz constant of CRReLU max(1 + e,, 0.446)..
Proof of sketch.
- (x> 0) x
CRReLU(X)
Exe 2 (x <<))
Furthermore, under mild assumptions, we consider the derivative of CRReLU piecewise (disregarding the potential for non-differentiabiiity temporarily).
dCRReLU(x))
1 1+E(1 - -x)e
(x) 0)
dx
E(1 x(1-x2)e ((x<0) {((=1))=+(<<0))
Setting the second derviation to be 0
dCRReLU(X)
2 (xx) 3x) =
dx²
Then have that: x1 = =0,x2 V3,x3 V3. For and x3, directly calculate the values at that point:
dCRReLU(x)
dCRReLU(x)
0.446e:
bxg -0.4468
dx
dx
For 0, we calculate the limit of (CRReLU(s) around O:
dx
dCRReLU(xx) = lim dCRReLU(x)) x-00
lim
x
dx
Hence, under mild assumptions, to obtain the upper bound for derivative of CRReL.U, we have that:
C= max 3, e, 0.4466, -0.446e)
Let's 's further consider the Lipschitz constant and simplify it. If > O, it is obvious that C= + e; and if e < O, we can get C 0.446e. Hence, we can express the Lipschitz constant of CRReLU as:
max(1 + E, -0.446e)
18

---

Published as a conference paper at ICLR 2025
Corollary 1. In order to make Lipschitz constant of CRReLU remains lower than that of GELU, the range of (.0.88,0.084), We recommend setting the initial value of e within this range.
Proof of sketch. If O, considering the following inequality:
1 < 1.084, then we have: 0 << & 0.084
& < 0, considering the following inequality:
- 0.4468 < 1.084, then we have: - -0.188 < <<<
Corollary 2. I is in (0.084,0.089)yX-001999 -0.188), we have the CRReLU's Lipschitz continuity worsen than GELU, but better than Mish.
Corollary 3. If'E is in (0.089,0 0. 1e.10000--022444 -0.198), we have the CRReLU's Lipschitz continuity worsen than Mish, but better than SiLU.
F ADDITIONAL EXPERIMENTS ON ARCHITECTURE AND DATASET
F.1 ADDITIONAL ARCHITECTURE
In this section, we aim to enhance the evaluation of CRReLU's generalization to network architecture. We choose to validate performance on CIFARIO, CIFAR100, and ImageNet K with the Con tiny (Liu al. 2022). For all the experiments, we conduct three runs, and we report the mean and standard deviation in Table 6 Experiments within CIFAR 10 and CIFAR 100 are conducted on 4 RTX3090 and those within ImageNetIK are conducted on 4XNVIDIA L20.
Table 6: Test accuracy of experiments conducted ConvNeetttiny for 100 epochs.
Top-one Accuracy
GELU
ELU
PReLU
CELU
SiLU
Mish
CRRELU
CIFARIO CIFAR 100
ConnNNet 64.910.4 59.810.55 6..611.4 59.840.5 60.610.2 61.410.4
70.61:1.1
ConvNeXt 36.6:1..3 30.340.4 35.2 30.5+0.2 35.010.9 35.3
42.110.7
ImageNet1
ConvNeXt
72.9:t0.3 71.7+0.5 72.910.5 71.8+0.99 72.3+0.7 72.8+0.6 73.2+00.2
F.2 ADDITIONAL DATASET
In this section, we aim to enhance the performance of CRReLU on diverse datasets. We conduct experiments on EuroSAT (Helber et al.) 2019) with ConyNeXt-tiny (Liu et al., 2022). All experi- ments are performed three times, we report the mean and standard deviation. We conduct this part experiments on a single RTX3090 for 25 epochs using the Adam optimizer, learning rate of 0.0001, cross entropy loss function, batch size of 256.
Table 7: Test accuracy of experiments conducted with ConvNeXt-tiny on the EuroSAT.
GELU
ELU
PReLU
CELU
SiLU
Mish
CRReLU
83.09:1..06 0..27 81.3
81.12+0.27 8..8 82.23+10.08
83.26:10.52
G ADDITIONAL EXPERIMENTS ON MIXED ACTIVATION FUNCTION
In T is shown that reduction in entropy CRReLUU between layers and 6 not significant. Hence. in this section, we focus equipping different activation functions in the initial six layers and subsequent six ones. We employ GELU for layers to 6 and CRReLU for layers 7 to 12, denoting We conduct three runs CIFAR10, CIFAR 100, and ImageNet1 K, presenting mean and standard deviation Table g Experiments on CIFAR10 and CIFAR 100 are conducted on 4x RTX3090, and those ImageNet K carried out 4xNVIDIA L20.
this
From the results, appears that having only the last few layers equipped with CRReLU not as effective utilizing CRReLU throughout the entire network. Especially the results on ImageNet1 K,
19

---

Published as a conference paper at ICLR 2025
Table 8: Test accuracy of experiments conducted with VIT (12GELU, 6GELU+6CRR-LU, 12CR- ReLU) for 100 epochs.
12GELU
6GELUU+6RRRL.U
12CRReI.U
CIFARIO CIFAR 1O0
0.704
0.755+00.008
0.807+0.003
0.326=00.008
0.399=1.000
0.46611.006
ImageNetIK
0.539=100003
0.512=10.011
0.575=10.004
6GEL GGHLU+GGRRRELUU is significantly and stably worsen to GELU and all CRReLLU, which is quite surprising to us. We consider that this may be due to the fact that, while the reduction in entropy not significantly apparent in the carlier layers, CRReLU's focus on achieving lower entropy still facilitates superior feature extraction. It seems that when using GELU in the earlier layers and CRReLU in the later layers, on small-scale datasets, is still possible to benefit from the CRReLU mechanism in the later layers (the features learned in the earlier layers not good enough yet); however, on large-scale datasets, the features learned in the earlier layers (those equipped with GELU) might even have negative effect.
H FURTHER DISCUSSION ON INITIALIZATION AND TRAINING STABILITY
H.1 EXPERIMENTS WITH ADDITIONAL INITIALIZATION
In this section, we focus on exploring the impact of different initial values of E, as well as potential instabilities failure cases under different initialization schemes. We set -0.5,-0.22 -0.1, -0.05, -0.02, -0.01, 0.01, 0.02. 0.05, 0.1, 0.2, 0.5, I, and 10, conducting experiments with ViT-tiny on CIFARIO and CIFAR100. We conduct three runs, reporting mean and standard derivation in Table
Table 9: Test accuracy of experiments conducted with ViT for 100 epochs under different initializa- tions.
0.01
0.02
0.05
0.1
0.2
0.5
10
CIFARIO
80.7::0.4
80.110.1
79.7:1..3
79.6:1.22
78.1
1..1=1.0
68.7+0.3
60.3 29.9+0.8
CIFARIO0
46.6::0.6
46.0
45.6:10.4
44.9+0.3
43.6+#..4
36.4=0.9
22.7
-0.01
-0.02
-0.05
-0.1
-0.2
-0.5
CIFARIO
80. 60.140.44
80.0+0.3
80.
00.22
80.6+0.1
00.
80.4=0.1
CIFARI00
45.9+10.3
46.0
46. 1
06.1
46.0+0.1
45.8=0.5
From the above experiments, we can see different initialization strategy does can have an impact on the final result. When the initial values differ significantly from the range derived in Corollary (such as 0.5, 1, 10), we have observed that the performance of training will degrade severely, especially at and 10, where the training process becomes extremely unstable.
H.2 CHANGE OF E DURING TRAINING
In Table 9 we demonstrate that different initialization strategies can have an impact on the final result. Improper initialization can significantly impair performance and lead instability, In this section, set & to 0,01 and show the changes of & in each layer during the 100 epochs' training of ViT-tiny on CIFAR10 and CIFAR100 in Figure 4 and Figure 5 As the figures shown, initializing at 0.01 allows its stable change during the training process and could avoid extreme situations.
H.3 DISCUSSION ON THE INTRODUCED LEARNABLE PARAMETER &
In Section 4.2 we have successfully demonstrated existence of the worst activation function, and from the worst starting point, it always moves towards improvement, regardless of the direction taken. However, commencing from specific activation function, like ReLU here, does not invariably result in improvement across all directions. i.e. certain optimization paths may lead to deteriorated outcomes. Therefore, from the practical perspective, we introduce learnable parameter e with the aim of enabling self-optimization of networks. From another perspective, in the derivation from ReLU
20

---

Published as a conference paper at ICLR 2025
14007
Laper
Laper
Laye:
Touar
Figure 4: Visualization of & change in each layer during the training of ViT-tiny on CIFARIO.
Layer
Loyer
Laye
Layer
Lamer
Layel 30
Lakar 11
Taarr 12
Figure 5: Visualization of e change in each layer during the training of ViT-tiny CIFARIOD
to CRReLU, we assume that data follows Gaussian distribution, which might not be true in real experiments. Existence of the learnable parameter also weakens this assumption to some extent.
Furthermore, we would like to provide some insights on selecting optimal values of e for practitioners implementing CRReLU their own networks. Firstly, we suggest looking within the aforementioned scope (-0.18 18.00.04) (detailed in Corollary D]. Furthermore, we suggest testing multiple values of it within this range and using k-fold cross-validation to test the performance of model under different initial values. Train and evaluate the model different folds to find the optimal value. In addition, if the prior knowledge of the dataset and network structure is sufficient, it can also be fully utilized, for example, if the network tends to produce negative outputs, then increasing the value of can be considered to enable the model to better capture the features of the samples.
I FURTHER DISCUSSION ON LOWER ENTROPY INDICATES BETTER CLASSIFICATION
Our work largely based on the statement: lower entropy indicates better classification. In this section, we will further elaborate the reasonability of such a statement. We would like to further claborate mainly from three perspectives: intuitively, empirically, and theoretically. From the intuitive perspective, lower entropy indicates less uncertainty for feature representation, which usually means more information is captured in fewer features. In other words, lower entropy can suggest that features are more discriminative, better able to distinguish different categories patterns. From the empirical perspective, early work (Silva et al. (005)) experimentally showed that minimization of Shannon's 's entropy of the gap between the output and the desired target could achieve better performance compared to MSE and CE. In early work (Santos et al.) 2004), the authors experimentally illustrated that minimizing entropy of the error between output and desired targets yields exceptionally satisfactory classification performance. From the theoretical perspective, work (Yi al. (2022) proved that for training DNN classifiers essentially learns the conditional entropy of the underlying data distribution of the dataset (the information or uncertainty remained in the labels after revealing the input) and derived the mutual information (between the corresponding feature and the label) bounds for a classification data model (Section 7). Hence, the conditional entropy [H[(uupput input) will decrease with the process of training. In the work (Xu & Raginsky 2017), the authors derived upper bounds the generalization error in terms of the mutual information between its input and output. According to (Xu & Raginsky] 2017), a smaller mutual information means a smaller generalization upper bound, which in turn suggests better classification performance. We have
21

---

Published as a conference paper at ICLR 2025
mutual information I(input,output) H((uupput) H((utput input). Within the process of training, ol input) decreases; hence, in order to make the mutual information [iinput, as small as possible, should minimize the H(output). Therefore, consider that a lower entropy signifies better classification performance.
J FURTHER DISCUSSION ON BIAS TOWARDS ACTIVATION FUNCTION IN PRE-TRAINED MODELS
J.1 BIAS EXISTED IN PRE-TRAIN INITIALIZATION
In the experiments of this work, we choose the truncated normal initialization instead of the initializa- tion commonly selected in state-of-the-an (SOTA) methods through pre-training larger datasets (for instance, ImageNet K initialized using weights pre-trained on ImageNer22K). This is mainly based on our following observation. We discover that the pre-trained models that have been released exhibit an intrinsic bias towards the activation functions they utilize (in other words, when the weights of a model pre-trained with GELU are used as initialization, the performance with GELU as the activation function is always optimal. Thus, to facilitate a fair comparison on the activation fuctions, we abandon this initialization method and utilize the truncated normal initialization, which does not introduce any bias on the activation functions.
Furthermore, in the LLM fine-tuning tasks (Table 5) the initial model we utilized is the publicly released GPT-2, which employs the GELU activation function for pre-training. Based on aforemen- tioned observations, the model actually exhibits a bias towards GELU; however, the ultimate results indicate that CRReLU still surpasses GELU, albeit to a lesser extent. Thus, this also demonstrates to some extent the superiority of CRReLU when confronted with larger parameters.
J.2 BIAS EXISTED IN KNOWLEDGE DISTILLATION
In Table experiments on VIT clearly demonstrate superiority of CRReLLU over other activation functions, and those on DieT, GELU shows 0.1% higher mean accuracy compared to CRReLU. Such result attributed to the teacher-student strategy structure of DieT model. We utilize the fine-tuned "deit-tiny-patch 1-22 6-224 model teacher model, which is trained with GELU, As explained in the work (Abnar et al. 2020), through distillation, transformers will inherit inductive bias. Hence, training student model with GELU ImageNet-1K 1K with the help of teacher model, which has already been pre-trained on ImageNet-1K with GELU, is certain to achieve better results than other activation functions. Furthermore, it is observed that such issue is not limited CRReLU actually: when comparing GELU and PReLU, GELU is 2.9% lower than PReLU in the ViT model, whereas in the DeiT model, GELU is 0.9% higher than PReLU. We believe that such issues are related to the bias present in the teacher model towards activation functions (the currently used open-source implementation employs GELU).
K FURTHER DISCUSSION ON DYNAMIC OPTIMIZATION
As mentioned, the EAFO methodology has shown the potential for dynamic optimization of activation during iterative training. Although we have not yet obtained an effective dynamic optimization method, we still want to provide some discussion here for future research work.
Dynamic optimization during iterative training might introduce considerable computational com- plexity. Such a problem of computational complexity might require the utilization of more efficient optimization algorithms (or optimizer) to address. Some insights are provided as follows. Firstly, we suggest conducting such activation optimization at a "batch-level" (gradient updates are typically done at the mini-batch level), which can stabilize the entire training process on one hand, and on the other hand, can reduce the computational complexity of dynamic optimization. That is to say, we can update the network parameters at a mini-batch level; while updating the activation at a batch level. Furthermore, we recommend using techniques similar to momentum methods for the design of optimizers, so that the model can retain information about the speed of gradient descent from the past, thereby accelerating convergence and reducing computing cost overall. Finally, also would like to
22

---

Published as a conference paper at ICLR 2025
consider methods for the adaptive activation learning, similar Adam, by adjusting the activation learning rates through calculating first and second moment estimates of the gradients.
L FURTHER DISCUSSION ON ACTIVATION FUNCTION RANKING
In the past three decades, more than 400 activation functions have been proposed (Kunc & Kléma 2024). We are thinking about whether we can provide some insights for activation function ranking based on our framework. Here, we would like to provide a little insight through the comparison of information entropy. The information entropy takes the form as:
H(((x)) - - ply()))) (o)o))))))))))(((o((((())()))))(
where yxx) is the inverse function of the activation function. Insight 1. Under mild assumptions, PReLU with tunable parameters should outperform the Leaky- ReLU with fixed- sxxddpparameter..
Proof of sketch. For PReLU and Leaky-ReLU, they exhibit similar form: f(x) = {{
x
x> The
axx x x 0"
difference is that a is a tunable parameter for PReLU, whereas a fixed parameter for Leaky-ReLLU
x> 0
And the inverse functions take the form as: y(x) (x/0x < and negative components of entropy function and simplify the expression:
We further segregate the positive
H(y(x)) = [ p(y(())) (x)e))/((()lgl(/bxxx)()())))))>)
= =- ((x/a)/0 log(p(x/a)/axdxx
Hence, let's further consider the difference in information entropy between PReLU and Leaky-ReLU:
H(PReLUU - H((Laay-RRL))
=
where 01 represents the tunable parameter of PReLU; represents the fixed parameter of Leaky- ReLU. Moreover, according to the formula, due the PReLU's ability to dynamically adjust its parameters based on the data distribution p(-), the resulting mutual information will be lower compared to Leaky-ReLU with fixed parameters, resulting better classification performance.
Nevertheless, is also worthy noting that alteration of parameter & in response to the data distribution will undoubtedly vary across different network architectures. Hence, ranking different activation functions in a generalized condtion is actually rather challenging task at present stage; for different network architectures, initialization and stochasticity, theoretical understanding on ranking activation functions still requires a considerable amount of discussion and comprehension.
M FURTHER DISCUSSION ON LLM INFERENCE TASK
In addition to the LLM fine-tuning tasks, we believe that CRReLU could further potentially achieve a better balance between inference speed and diverse generation in the LLM inference task. In the work (Mirzadeh et al.) 2024), it is shown that leveraging the activation sparsity of ReLU, there will be a significant enhancement inference FLOPS. However, it is also noteworthy that contemporary open-source LLMs increasingly favor the use of GELU and SiLU, likely driven by considerations surrounding the diversity of model generation. Excessive activation sparsity might potentially diminish the generative diversity of the model. thereby reducing further illustrate in Figure 2(c) that as the parameter beta increases, the performance activation sparsity improves. Such observation is closely related the Lipschitz Continuity of the activation
engagement. The authors
23

---

Published as a conference paper at ICLR 2025
function (Kim al. 2021) (last paragraph of Section 3.1 claims that bounded inputs make dot- product self-attention Lipschitz). In Fact we have obtained the GELU's Lipschitz constant of 1.084; Mish's Lipschitz constant of 1.089; SiLU's Lipschitz constant of 1.110. To enhance the performance of CRRELUU resulting in a superior Lipschitz continuity compared to GELU, we derive the recommended range is (-0. 188, 0.084). Hence, CRReLU approaches zero within this range, CRReLU converges more closely to ReLU. In this scenario, activation sparsity of CRReLU improves, while may also potentially diminish the diversity of the generated outputs. Conversely, as it far away from zero within this range, the utilization of CRReLU could deteriorate activation sparsity, yet simultaneously possess potential to enhance diversity of generated outputs. Therefore, CRReLU has higher flexibility in how much we care about generation diversity compared to activation sparsity.
N DETAILS OF EXPERIMENTAL SETTINGS
N.1 TASK OF IMAGE CLASSIFICATION
Table 10: Experimental settings of ViT, DeiT and TNT on CIFAR-10 and CIFAR- 100 datasets
Image Size
32 32
Patch Size
Embedding Dim
192 for ViT-Tiny and DeiT-Tiny 384 for TNT-small
Optimizer
A with weight decay = 0.05
Cosine Annealing Learning Rate Scheduler
Learning Rate
Initial Ir= 2.5x 10 Ir drop = min Ir = 1 x x10-
Warm up
warmup epochs 20 warmup learning rate 1 1
Gradient Clipping
1.0
Training Epochs
100
Batch Size
256
Loss Function
CrossEntropy Loss
Normalization
Layer Norm
Data Augmentation
True (provided by timm)
Drop Out and Drop Path
False
Table 11: Experimental settings of ViT and DeiT on ImageNet- dataset
Image Size
224 224
Patch Size
16
Embedding Dim
192
Optimizer
AdamW with weight decay = 0.05
Cosine Annealing Learning Rate Scheduler
Learning Rate
Initial Ir= 2.5x 10 Ir drop = min Ir = 1 x 10
Warm up
warmup epochs 20 warmup learning rate = 1 10
Gradient Clipping
1.0
Training Epochs
100
Batch Size
256
Loss Function
CrossEntropy Loss
Normalization
Layer Norm
Data Augmentation
True (provided by timm)
Drop Out and Drop Path
False
24

---

Published as a conference paper at ICLR 2025
Table 12:
We record changes in parameter number when employing various activation functions. GELU, ELU, CELU, SiLU (Swish), and Mish are considered activation functions without learnable parameter (AFs without LP), while PReLU and CRReLU are considered activation functions with learnable parameter (AFs with LP). The results demonstrate that increase in parameter number introduced by the learnable parameter is negligible.
Parameter Number
CIFAR-10
CIFAR-100
ImageNet-IK IK
AFs without LP AFs with LP
5399818
5417188
5754472
ViT-Tiny
5399830
5417200
5754484
AFs without LP
5365076
5399816
5910800
DeiT-Tiny
AFs with LP
5365088
5399828
5910812
AFs without LP
21525298
21559948
TNT-Small
AFs with LP
21525322
21559972
N.2 TASK OF LARGE LANGUAGE MODEL (LLM) FINE-TUNING
Table 13: Experimental settings of GPT2 fine-tuning task
Batch Size
Optimizer
RMSprop (More Memory-Fificient)
Learning Rate
5x 10
with linear warmup steps of 150
Trainer
FSDPTrainer GPUs)
Max Gradient Norm
10.0
Max Length for an Input (Prompt + Response)
512
Max Length for Prompt
256
25


Reviews:
[{"review_id": "DXfqcwc94E", "reviewer": "Reviewer_YJtL", "summary": "In this work, the authors propose a theoretical framework for defining optimality of an activation function (without the optimization considerations). Using Taylor's expansion, the authors extend their framework to search for better activation functions (EAFO - Entropy based Activation Function Optimization) and later also define the worst activation function with boundary conditions. Using the EAFO framework and starting from ReLU, the authors derive a better and novel activation function CRReLU (Correction Regularized ReLU). The authors later demonstrate on three datasets CIFAR10, CIFAR100 and ImageNet-1k where the new found activation function outperforms ReLU on classification performance. Lastly, authors also show improved performance on LLM fine tuning tasks when the CRReLU was swapped out with the ReLU activation function.", "questions": "1. My main concern regarding the manuscript is—entropy as an indicator of better classification seems like a very strong statement. One of the key reasons why Sigmoid is not preferred over ReLU is due its optimization properties (vanishing gradients). Since the EAFO framework is completely agnostic to that, the contribution of this framework becomes significantly weaker. If the authors could empirically show how EAFO could be used in practice or justify the choice of entropy as an indicator for activation function optimality, that could help address my concerns\n2. Another suggestion is to actually compare the entropy post training of neural networks trained with different activations, not just at the end, but also in the intermediate layers. Since the activation function is being used throughout the network, does lower entropy also help there? If not, should only be the last few layers be equipped with CRReLU?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written clearly and concisely, and is easy to read. \n2. An information theoretic framework for defining optimality of activation functions for classification tasks is a great approach to search for activation functions and could potentially generate insights. The authors indicate several properties of worst activation functions in i.e. being bounded however this might require more careful analysis but serves as a good stepping stone for future follow up works.", "weaknesses": "1. The premise for EAFO is that extremas in the entropy space after transformation with the activation function correspond to better separability of features in the resulting space but that doesn’t mean better classification performance. Moreover, unlike in discrete space the entropy in the continuous random variables also changes with the scale. However, that might not have any impact on the classification performance. Why do the authors believe this is the right measure to define how good an activation function is?\n2. Can the authors rank different activation functions based on the EAFO framework? For example, comparison of ReLU and PReLU should point to PReU being better. Since there is already experimental evidence that PReLU is better, if EAFO could confirm it, that would be a great contribution. Similarly please consider ranking 3-4 activation functions to justify the utility of this framework.\n3. For the experiments, what are the error bars? How many training runs per result? This is important to understand the statistical significance of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05yFuMFRO1", "reviewer": "Reviewer_7aze", "summary": "The paper introduces a theoretical framework to learn a high-performance activation function. It theoretically shows that a worst activation function exist and empirically show that their proposed framework learns significantly improved activation functions compared to SoTA activation function.", "questions": "Why is the baseline accuracy on ImageNet and CIFAR-10 so low? State-of-the-art networks typically achieve test scores over 0.9 on CIFAR-10 and above 0.8 on ImageNet-1K.\n\nIn LLM fine-tuning tasks, the paper reports marginal improvements over GELU. Could the authors provide further insight into the specific benefits of CRReLU in this context, beyond numerical accuracy improvements?\n\nHow would CRReLU perform if evaluated on more diverse NLP tasks or models with larger parameters, and would any tuning adjustments be needed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed framework to learn activation is shown to perform significantly better than the SoTA activation function theoretically as well as empirically.\n\nWhile different networks and tasks require different activation functions for the best performance, the proposed framework simplifies this design choice by transferring the activation choice to automated learning during the optimization stage.", "weaknesses": "Although the paper demonstrates substantial empirical improvements, the reported results fall considerably short of state-of-the-art (SoTA) baseline accuracies. For instance, CNNs using ReLU activation commonly achieve test scores above 0.9.\n\nAdditionally, there is no direct comparison between SoTA neural network architectures (such as ViT and CNN) using their standard activation functions and those with the proposed activation function. This makes it unclear how much the new activation function improves upon SoTA.\n\nIn the LLM fine-tuning task, the improvement over GeLU activation is minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m6DkSWs3Xw", "reviewer": "Reviewer_LUV6", "summary": "This paper targets the fundamental challenge of activation function design in deep neural networks, which has relied heavily on empirical knowledge rather than a systematic understanding and theoretical foundations. The authors thus propose a new theoretical framework connecting information entropy to activation function performance, which verifies the existence of a worst-case activation function (WAFBC) and thereby develops an entropy-based optimization method (EAFO). The key theoretical contribution of this work is establishing that moving away from WAFBC can consistently improve the model’s performance, leading to a systematic approach for activation function optimization. Built upon this, the authors present Correction Regularized ReLU (CRReLU), demonstrating its great performance across vision transformers and language models. The experiments are comprehensive, covering both image classification (CIFAR-10/100, ImageNet-1K) and language model fine-tuning tasks, with thorough ablation studies and theoretical guarantees.", "questions": "**(Q1) Dynamic Optimization Implementation:** While the authors suggest the potential for dynamic optimization of activation functions during training, the practical implementation remains relatively unclear. Could the authors elaborate on:\n\n- Concrete strategies for making dynamic optimization computationally tractable in large networks?\n- Specific approaches to balance the frequency of activation function updates with computational overhead?\n- Empirical evidence or theoretical bounds on the expected performance gains from dynamic optimization? Understanding these aspects would help assess the practical value of the dynamic optimization extension.\n\n**(Q2) Initialization and Training Stability:** The choice of ε=0.01 as initialization appears somewhat arbitrary. Could the authors provide:\n\n- Analysis of how different initialization values affect training dynamics and final performance?\n- Guidelines for selecting optimal ε values based on network architecture or task requirements?\n- Can we investigate potential instabilities or failure cases under different initialization schemes? This information would be crucial for practitioners implementing CRReLU in their own networks.\n\n---\n**Additional Comment:**\n\nI hope my review helps to further strengthen this paper and helps the authors, fellow reviewers, and Area Chairs understand the basis of my recommendation. I also look forward to the rebuttal feedback and further discussions, and would be glad to raise my rating if thoughtful responses and improvements are provided.\n\n\n---\n## **-------------------- Post-Rebuttal Summary --------------------**\n\nThe additional experiments, discussions, and revised manuscript provided by the authors have significantly strengthened the work and addressed most of my concerns. I suppose this work can provide knowledge advancement to the field, and I look forward to the final revised manuscript, incorporating the additional information presented in the rebuttal stage.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**(S1) Theoretical Foundation:** This paper establishes a solid mathematical framework connecting information entropy to activation function performance. The derivation begins with principles of information theory and extends through functional analysis to establish clear relationships between data distributions and activation behavior. Specifically, the proof of the Worst Activation Function with Boundary Conditions (WAFBC) existence is clear, utilizing variational calculus and the Euler-Lagrange equation to demonstrate global maximality. As such, it not only provides insights into why certain activation functions perform better than others but also explains long-observed empirical phenomena, such as the superior performance of unbounded activation functions (like ReLU) compared to bounded ones (such as sigmoid and tanh), which offers both theoretical guarantees and practical optimization guidance.\n\n**(S2) Technical Originality and Soundness:** The proposed EAFO method represents a significant advancement in activation function design. Unlike previous ones that largely relied on empirical knowledge, EAFO provides a principled and systematic framework. The derivation of correction terms through analysis of the information entropy functional's Taylor expansion is insightful, enabling both static design and potential dynamic optimization. The introduction of learnable parameters in CRReLU demonstrates a thoughtful balance between theoretical purity and practical adaptability. Moreover, its potential extension to dynamic optimization during training seems to open new research directions, while maintaining backward compatibility with existing architectures and optimization techniques. \n\n**(S3) Thorough Experiments:** Experiments in this work are comprehensive and well-designed, covering multiple network architectures and task domains. Extensive ablation studies and sensitivity analyses are also conducted to show the methods’ effectiveness. Concretely, the evaluation across vision transformers (ViT, DeiT, TNT) and LLMs (GPT-2) shows broad applicability, while the performance improvements on classical computer vision benchmarks (like CIFAR-10/100 and ImageNet-1K) provide strong practical validation. The large-scale experiments on language model fine-tuning using Direct Preference Optimization (DPO) provide valuable insights into the method's scalability and generalization capabilities. Moreover, the computational efficiency analysis is particularly useful, showing minimal overhead despite the addition of learnable parameters. \n\n**(S4) Presentation Clarity:** \nThis manuscript exhibits great clarity in presenting mathematical concepts and empirical results. The progression from theoretical foundations through practical implementation is logical and well-structured, making the work accessible to a broader audience while maintaining technical insights. The mathematical derivations are with appropriate detail and clear step-by-step explanations, facilitating reproducibility and future extensions. In addition, the thorough implementation details, including pseudo-code and network architecture considerations, ensure the practical applicability of this work.", "weaknesses": "**(W1) Theoretical Limitations:** The authors make the assumption of Gaussian distribution for the input data distributions in this paper. While it is mathematically convenient, it requires more rigorous justification. While the authors cite the Central Limit Theorem and previous works supporting this assumption in deep neural networks, modern architectures like transformers with complex operators like self-attention mechanisms may exhibit significantly different distribution patterns. This work would benefit from a more detailed analysis of how distribution deviations affect the theoretical guarantees. In addition, the convergence properties during the training process, particularly the interaction between the learnable parameter and standard network weights, lack thorough theoretical treatment. \n\n**(W2) Experimental Concerns:** The experimental results, while generally strong, reveal several areas requiring deeper investigation. The performance compared to GELU in DeiT experiments raises important questions about the interaction between CRReLU and knowledge distillation processes. It deserves a more thorough analysis, potentially exploring alternative distillation strategies that are more compatible with the CRReLU's properties. Besides, the initialization strategy for the learnable parameter appears somewhat arbitrary (set to 0.01). Moreover, the absence of experiments on classical CNN architectures leaves a significant gap in demonstrating the method's generality, particularly given the widespread use of CNN-based network architectures.\n\n**(W3) Dynamic Optimization Challenges:** This work employs dynamic optimization during training, which potentially faces several practical challenges. For example, the computational complexity analysis of dynamic optimization is insufficient, particularly for large-scale networks where activation function optimization could introduce substantial overhead. The interaction between dynamic activation optimization and common training techniques (batch normalization, residual connections, dropout) also requires more detailed analysis. I recommend the authors conduct more experimental validation and analysis to address these issues.\n\n**(W4) Implementation and Scalability Considerations:** The practical implementation of EAFO and CRReLU requires more detailed treatment, particularly regarding numerical stability and computational efficiency at scale. Discussion of potential gradient flow issues when the learnable parameter ε takes extreme values, and the mitigation strategies are all not provided. Additionally, the paper would benefit from analysis of how the method performs under resource-constrained conditions, such as mobile devices or edge computing scenarios. All these could provide more insights to the researchers and practitioners in the community, and thus propel further research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "chxrgMM2Z3", "reviewer": "Reviewer_m9pv", "summary": "The paper presents a systematic approach to address the problem of activation function optimization in artificial neural networks (ANNs). By leveraging information entropy theory, the authors theoretically demonstrate the existence of the worst activation function under boundary conditions (WAFBC). They then propose the Entropy-based Activation Function Optimization (EAFO) methodology, which provides a framework for designing better activation functions. Utilizing this methodology, the authors derive a novel activation function called Correction Regularized ReLU (CRReLU) from the conventional ReLU. Extensive experiments on vision transformer variants and large language model (LLM) fine-tuning tasks demonstrate the superior performance of CRReLU over existing ReLU variants.", "questions": "1. How does the EAFO methodology perform when applied to other activation functions, especially those without an inverse function, such as Swish or Mish?\n2. Can the authors provide empirical evidence or theoretical analysis of CRReLU's performance under non-Gaussian data distributions, such as heavy-tailed or multimodal distributions, to address concerns about the robustness of the method?\n3. What potential approaches or algorithms, such as gradient-based optimization or stochastic methods, can be explored to mitigate the computational complexity introduced by dynamic optimization during iterative training?\n4. Would the authors consider conducting additional experiments on diverse datasets, such as medical imaging (e.g., MICCAI) or remote sensing data (e.g., EuroSAT), and architectures like convolutional neural networks (e.g., ResNet) or graph neural networks, to further validate the generalizability of CRReLU?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical Rigor: \nThe paper provides a solid theoretical foundation for activation function optimization by introducing the concept of WAFBC and the EAFO methodology. This approach is novel and offers a fresh perspective on designing activation functions.\n2. Practical Application: \nThe derived CRReLU activation function shows significant improvements in performance across various tasks, including image classification and LLM fine-tuning, demonstrating the practical applicability of the proposed methodology.\n3. Comprehensive Experiments: \nThe authors conduct extensive experiments on multiple datasets and architectures, validating the effectiveness of CRReLU and providing a thorough evaluation of the proposed method.", "weaknesses": "1. Limited Generalizability: \nThe paper primarily focuses on ReLU and its variants. It would be valuable to explore the applicability of the theoretical framework to activation functions without an inverse function, such as Swish or Mish.\n2. Computational Complexity: \nThe dynamic optimization during iterative training introduces significant computational complexity, which the paper does not address. The authors should discuss potential approaches or algorithms, such as gradient-based optimization or stochastic methods, that might mitigate these computational complexity issues, or provide a more detailed analysis of the trade-offs between performance gains and computational costs.\n3. Assumption of Gaussian Distribution: \nThe assumption that data follows a Gaussian distribution simplifies the derivation of CRReLU but may not hold in all real-world scenarios. The authors should provide empirical evidence or theoretical analysis of CRReLU's performance under non-Gaussian data distributions, such as heavy-tailed or multimodal distributions, to address concerns about the robustness of the method.\n4. Lack of Diverse Experiments: \nWhile the experiments are comprehensive, they are limited to specific datasets and architectures. Additional experiments on diverse datasets, such as medical imaging (e.g., MICCAI) or remote sensing data (e.g., EuroSAT), and architectures like convolutional neural networks (e.g., ResNet) or graph neural networks, would strengthen the generalizability claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)